2023-10-16T20:19:05.713 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-16T20:19:05.714 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi/7430114 --name rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi --verbose --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --owner scheduled_rishabh@teuthology -- /home/teuthworker/archive/rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi/7430114/orig.config.yaml 2023-10-16T20:19:05.763 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-16T20:19:05.856 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi/7430114 branch: rishabh-2023oct13 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: null first_in_suite: false job_id: '7430114' kernel: flavor: default kdb: true sha1: 47672119149b103ec7cec2cc65c063ccbb001ac7 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: rishabh-2023oct13 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: false mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: false mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - Replacing daemon mds sha1: 4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-2023oct13-qa2 sha1: 19fd96fe3581520b6d9f50ffdabf29051aee5d4d owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 6006 sha1: 4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 sleep_before_teardown: 0 subset: 402/512 suite: fs suite_branch: rishabh-2023oct13-qa2 suite_path: /home/teuthworker/src/github.com_rishabh-d-dave_ceph_19fd96fe3581520b6d9f50ffdabf29051aee5d4d/qa suite_relpath: qa suite_repo: https://github.com/rishabh-d-dave/ceph suite_sha1: 19fd96fe3581520b6d9f50ffdabf29051aee5d4d syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO3ZZJl3J3J2FRWmXEHKzqpNP/UrrVcr/DKpKW9bbB30t5/mQLXo9MV/4v88nZ3Aidf1/I++8C8HmPysMI48YZU= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBENSQMED5hBpKUrlawu3SsqoqtNsEGYfWpGz3nN8acKssOxqU8AqvVaZ6pkCVsiVeDFvVF1lai5z+ACojtAJSDc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-16_18:49:15 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-16T20:19:05.857 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_rishabh-d-dave_ceph_19fd96fe3581520b6d9f50ffdabf29051aee5d4d/qa; will attempt to use it 2023-10-16T20:19:05.858 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_rishabh-d-dave_ceph_19fd96fe3581520b6d9f50ffdabf29051aee5d4d/qa/tasks 2023-10-16T20:19:05.858 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-16T20:19:05.860 INFO:teuthology.task.internal:Checking packages... 2023-10-16T20:19:05.887 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0' 2023-10-16T20:19:05.887 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-16T20:19:05.888 INFO:teuthology.packaging:ref: None 2023-10-16T20:19:05.888 INFO:teuthology.packaging:tag: None 2023-10-16T20:19:05.888 INFO:teuthology.packaging:branch: rishabh-2023oct13 2023-10-16T20:19:05.889 INFO:teuthology.packaging:sha1: 4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 2023-10-16T20:19:05.889 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=rishabh-2023oct13 2023-10-16T20:19:06.037 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6730-g4ba8a190-1jammy 2023-10-16T20:19:06.038 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-16T20:19:06.045 INFO:teuthology.task.internal:no buildpackages task found 2023-10-16T20:19:06.045 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-16T20:19:06.054 INFO:teuthology.task.internal:Saving configuration 2023-10-16T20:19:06.081 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-16T20:19:06.092 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-16T20:19:06.119 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi/7430114', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 20:13:16.460231', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGzpBDavTiAViq9QT28N4ygbKdeLqy7+Rk/DbNS4prsO'} 2023-10-16T20:19:06.143 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi/7430114', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 20:13:16.458789', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDE978WjoGLfflQlFt3+fRK6Bb4quF8VUB+6X1QP4Jca'} 2023-10-16T20:19:06.143 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-16T20:19:06.150 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-10-16T20:19:06.150 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-10-16T20:19:06.150 INFO:teuthology.run_tasks:Running task console_log... 2023-10-16T20:19:06.227 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6616d57a60>, signals=[15]) 2023-10-16T20:19:06.228 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-16T20:19:06.236 INFO:teuthology.task.internal:Opening connections... 2023-10-16T20:19:06.237 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2023-10-16T20:19:06.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T20:19:06.317 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2023-10-16T20:19:06.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T20:19:06.392 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-16T20:19:06.400 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2023-10-16T20:19:06.405 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2023-10-16T20:19:06.406 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2023-10-16T20:19:06.450 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-16T20:19:06.451 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="Ubuntu" 2023-10-16T20:19:06.451 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="22.04" 2023-10-16T20:19:06.451 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-16T20:19:06.451 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_CODENAME=jammy 2023-10-16T20:19:06.451 INFO:teuthology.orchestra.run.smithi088.stdout:ID=ubuntu 2023-10-16T20:19:06.452 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE=debian 2023-10-16T20:19:06.452 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-16T20:19:06.452 INFO:teuthology.orchestra.run.smithi088.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-16T20:19:06.452 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-16T20:19:06.452 INFO:teuthology.orchestra.run.smithi088.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-16T20:19:06.453 INFO:teuthology.orchestra.run.smithi088.stdout:UBUNTU_CODENAME=jammy 2023-10-16T20:19:06.453 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2023-10-16T20:19:06.483 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2023-10-16T20:19:06.488 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2023-10-16T20:19:06.488 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2023-10-16T20:19:06.533 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-16T20:19:06.534 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="Ubuntu" 2023-10-16T20:19:06.534 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="22.04" 2023-10-16T20:19:06.535 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-16T20:19:06.535 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_CODENAME=jammy 2023-10-16T20:19:06.535 INFO:teuthology.orchestra.run.smithi191.stdout:ID=ubuntu 2023-10-16T20:19:06.536 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE=debian 2023-10-16T20:19:06.536 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-16T20:19:06.536 INFO:teuthology.orchestra.run.smithi191.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-16T20:19:06.537 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-16T20:19:06.537 INFO:teuthology.orchestra.run.smithi191.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-16T20:19:06.537 INFO:teuthology.orchestra.run.smithi191.stdout:UBUNTU_CODENAME=jammy 2023-10-16T20:19:06.538 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2023-10-16T20:19:06.558 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-16T20:19:06.569 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-16T20:19:06.599 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-16T20:19:06.600 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2023-10-16T20:19:06.604 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2023-10-16T20:19:06.608 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-16T20:19:06.616 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-16T20:19:06.617 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2023-10-16T20:19:06.652 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2023-10-16T20:19:06.659 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-16T20:19:06.718 INFO:teuthology.run_tasks:Running task kernel... 2023-10-16T20:19:06.735 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '47672119149b103ec7cec2cc65c063ccbb001ac7'} 2023-10-16T20:19:06.736 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '47672119149b103ec7cec2cc65c063ccbb001ac7'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '47672119149b103ec7cec2cc65c063ccbb001ac7'}}, timeout 300 2023-10-16T20:19:06.736 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-16T20:19:06.737 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-16T20:19:06.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T20:19:06.742 DEBUG:teuthology.task.kernel:sha1 for mon.a is 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T20:19:06.742 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "47672119149b103ec7cec2cc65c063ccbb001ac7"... 2023-10-16T20:19:06.743 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2023-10-16T20:19:06.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T20:19:06.745 DEBUG:teuthology.task.kernel:sha1 for mon.b is 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T20:19:06.746 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "47672119149b103ec7cec2cc65c063ccbb001ac7"... 2023-10-16T20:19:06.746 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2023-10-16T20:19:06.791 INFO:teuthology.orchestra.run.smithi088.stdout:5.15.0-56-generic 2023-10-16T20:19:06.791 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T20:19:06.792 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-10-16T20:19:06.792 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T20:19:06.796 INFO:teuthology.orchestra.run.smithi191.stdout:5.15.0-56-generic 2023-10-16T20:19:06.796 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T20:19:06.797 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-10-16T20:19:06.797 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T20:19:06.942 INFO:teuthology.task.kernel:Installing linux-firmware on mon.b... 2023-10-16T20:19:06.942 DEBUG:teuthology.orchestra.run.smithi191:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2023-10-16T20:19:06.966 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2023-10-16T20:19:07.061 INFO:teuthology.task.kernel:Installing linux-firmware on mon.a... 2023-10-16T20:19:07.062 DEBUG:teuthology.orchestra.run.smithi088:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2023-10-16T20:19:07.089 INFO:teuthology.orchestra.run.smithi088.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2023-10-16T20:19:07.135 INFO:teuthology.orchestra.run.smithi191.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2023-10-16T20:19:07.135 INFO:teuthology.orchestra.run.smithi191.stderr:hint: is subject to change. To configure the initial branch name to use in all 2023-10-16T20:19:07.135 INFO:teuthology.orchestra.run.smithi191.stderr:hint: of your new repositories, which will suppress this warning, call: 2023-10-16T20:19:07.135 INFO:teuthology.orchestra.run.smithi191.stderr:hint: 2023-10-16T20:19:07.136 INFO:teuthology.orchestra.run.smithi191.stderr:hint: git config --global init.defaultBranch 2023-10-16T20:19:07.136 INFO:teuthology.orchestra.run.smithi191.stderr:hint: 2023-10-16T20:19:07.136 INFO:teuthology.orchestra.run.smithi191.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2023-10-16T20:19:07.136 INFO:teuthology.orchestra.run.smithi191.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2023-10-16T20:19:07.136 INFO:teuthology.orchestra.run.smithi191.stderr:hint: 2023-10-16T20:19:07.137 INFO:teuthology.orchestra.run.smithi191.stderr:hint: git branch -m 2023-10-16T20:19:07.138 INFO:teuthology.orchestra.run.smithi191.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2023-10-16T20:19:07.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2023-10-16T20:19:07.210 DEBUG:teuthology.orchestra.run.smithi191:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2023-10-16T20:19:07.222 INFO:teuthology.orchestra.run.smithi088.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2023-10-16T20:19:07.222 INFO:teuthology.orchestra.run.smithi088.stderr:hint: is subject to change. To configure the initial branch name to use in all 2023-10-16T20:19:07.223 INFO:teuthology.orchestra.run.smithi088.stderr:hint: of your new repositories, which will suppress this warning, call: 2023-10-16T20:19:07.223 INFO:teuthology.orchestra.run.smithi088.stderr:hint: 2023-10-16T20:19:07.223 INFO:teuthology.orchestra.run.smithi088.stderr:hint: git config --global init.defaultBranch 2023-10-16T20:19:07.223 INFO:teuthology.orchestra.run.smithi088.stderr:hint: 2023-10-16T20:19:07.224 INFO:teuthology.orchestra.run.smithi088.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2023-10-16T20:19:07.224 INFO:teuthology.orchestra.run.smithi088.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2023-10-16T20:19:07.224 INFO:teuthology.orchestra.run.smithi088.stderr:hint: 2023-10-16T20:19:07.224 INFO:teuthology.orchestra.run.smithi088.stderr:hint: git branch -m 2023-10-16T20:19:07.225 INFO:teuthology.orchestra.run.smithi088.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2023-10-16T20:19:07.226 DEBUG:teuthology.orchestra.run.smithi088:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2023-10-16T20:19:07.266 DEBUG:teuthology.orchestra.run.smithi191:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2023-10-16T20:19:07.290 DEBUG:teuthology.orchestra.run.smithi088:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2023-10-16T20:19:07.343 DEBUG:teuthology.orchestra.run.smithi088:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2023-10-16T20:21:24.071 INFO:teuthology.orchestra.run.smithi191.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2023-10-16T20:21:24.072 INFO:teuthology.orchestra.run.smithi191.stderr: * [new branch] main -> origin/main 2023-10-16T20:21:24.104 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20190312 -> 20190312 2023-10-16T20:21:24.104 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20190416 -> 20190416 2023-10-16T20:21:24.105 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20190514 -> 20190514 2023-10-16T20:21:24.105 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20190618 -> 20190618 2023-10-16T20:21:24.105 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20190717 -> 20190717 2023-10-16T20:21:24.106 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20190815 -> 20190815 2023-10-16T20:21:24.106 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20190923 -> 20190923 2023-10-16T20:21:24.106 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20191022 -> 20191022 2023-10-16T20:21:24.107 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20191215 -> 20191215 2023-10-16T20:21:24.107 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20200122 -> 20200122 2023-10-16T20:21:24.107 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20200316 -> 20200316 2023-10-16T20:21:24.107 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20200421 -> 20200421 2023-10-16T20:21:24.108 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20200519 -> 20200519 2023-10-16T20:21:24.108 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20200619 -> 20200619 2023-10-16T20:21:24.108 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20200721 -> 20200721 2023-10-16T20:21:24.108 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20200817 -> 20200817 2023-10-16T20:21:24.108 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20200918 -> 20200918 2023-10-16T20:21:24.109 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20201022 -> 20201022 2023-10-16T20:21:24.109 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20201118 -> 20201118 2023-10-16T20:21:24.109 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20201218 -> 20201218 2023-10-16T20:21:24.109 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20210208 -> 20210208 2023-10-16T20:21:24.110 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20210315 -> 20210315 2023-10-16T20:21:24.112 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20210511 -> 20210511 2023-10-16T20:21:24.113 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20210716 -> 20210716 2023-10-16T20:21:24.113 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20210818 -> 20210818 2023-10-16T20:21:24.113 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20210919 -> 20210919 2023-10-16T20:21:24.114 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20211027 -> 20211027 2023-10-16T20:21:24.114 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20211216 -> 20211216 2023-10-16T20:21:24.114 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20220209 -> 20220209 2023-10-16T20:21:24.115 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20220310 -> 20220310 2023-10-16T20:21:24.115 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20220411 -> 20220411 2023-10-16T20:21:24.115 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20220509 -> 20220509 2023-10-16T20:21:24.116 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20220610 -> 20220610 2023-10-16T20:21:24.116 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20220708 -> 20220708 2023-10-16T20:21:24.116 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20220815 -> 20220815 2023-10-16T20:21:24.117 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20220913 -> 20220913 2023-10-16T20:21:24.117 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20221012 -> 20221012 2023-10-16T20:21:24.117 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20221109 -> 20221109 2023-10-16T20:21:24.117 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20221214 -> 20221214 2023-10-16T20:21:24.118 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20230117 -> 20230117 2023-10-16T20:21:24.118 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20230210 -> 20230210 2023-10-16T20:21:24.119 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20230310 -> 20230310 2023-10-16T20:21:24.120 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20230404 -> 20230404 2023-10-16T20:21:24.120 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20230515 -> 20230515 2023-10-16T20:21:24.120 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20230625 -> 20230625 2023-10-16T20:21:24.121 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20230804 -> 20230804 2023-10-16T20:21:24.121 INFO:teuthology.orchestra.run.smithi191.stderr: * [new tag] 20230919 -> 20230919 2023-10-16T20:21:27.810 INFO:teuthology.orchestra.run.smithi191.stderr:Updating files: 51% (1624/3161) Updating files: 52% (1644/3161) Updating files: 53% (1676/3161) Updating files: 54% (1707/3161) Updating files: 55% (1739/3161) Updating files: 56% (1771/3161) Updating files: 57% (1802/3161) Updating files: 58% (1834/3161) Updating files: 59% (1865/3161) Updating files: 60% (1897/3161) Updating files: 61% (1929/3161) Updating files: 62% (1960/3161) Updating files: 63% (1992/3161) Updating files: 64% (2024/3161) Updating files: 65% (2055/3161) Updating files: 66% (2087/3161) Updating files: 66% (2106/3161) Updating files: 67% (2118/3161) Updating files: 68% (2150/3161) Updating files: 69% (2182/3161) Updating files: 70% (2213/3161) Updating files: 71% (2245/3161) Updating files: 72% (2276/3161) Updating files: 73% (2308/3161) Updating files: 74% (2340/3161) Updating files: 75% (2371/3161) Updating files: 76% (2403/3161) Updating files: 77% (2434/3161) Updating files: 78% (2466/3161) Updating files: 79% (2498/3161) Updating files: 80% (2529/3161) Updating files: 81% (2561/3161) Updating files: 81% (2589/3161) Updating files: 82% (2593/3161) Updating files: 83% (2624/3161) Updating files: 84% (2656/3161) Updating files: 85% (2687/3161) Updating files: 86% (2719/3161) Updating files: 87% (2751/3161) Updating files: 88% (2782/3161) Updating files: 89% (2814/3161) Updating files: 90% (2845/3161) Updating files: 91% (2877/3161) Updating files: 92% (2909/3161) Updating files: 93% (2940/3161) Updating files: 94% (2972/3161) Updating files: 95% (3003/3161) Updating files: 96% (3035/3161) Updating files: 97% (3067/3161) Updating files: 98% (3098/3161) Updating files: 99% (3130/3161) Updating files: 100% (3161/3161) Updating files: 100% (3161/3161), done. 2023-10-16T20:21:27.815 INFO:teuthology.orchestra.run.smithi191.stdout:HEAD is now at 63e8aa40 Merge branch 'main' into 'main' 2023-10-16T20:21:27.838 INFO:teuthology.task.kernel:Downloading kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on mon.b... 2023-10-16T20:21:27.839 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T20:21:28.109 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-g47672119149b/ 2023-10-16T20:21:28.109 DEBUG:teuthology.orchestra.run.smithi191:> rm -f /tmp/linux-image.deb && echo linux-image-6.6.0-rc5-g47672119149b_6.6.0-rc5-g47672119149b-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-g47672119149b/ --input-file=- 2023-10-16T20:21:28.112 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi191.front.sepia.ceph.com to complete... 2023-10-16T20:21:29.754 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16 20:21:29 URL:https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-g47672119149b/linux-image-6.6.0-rc5-g47672119149b_6.6.0-rc5-g47672119149b-1_amd64.deb [38909314/38909314] -> "/tmp/linux-image.deb" [1] 2023-10-16T20:21:29.755 INFO:teuthology.orchestra.run.smithi191.stderr:FINISHED --2023-10-16 20:21:29-- 2023-10-16T20:21:29.755 INFO:teuthology.orchestra.run.smithi191.stderr:Total wall clock time: 1.5s 2023-10-16T20:21:29.755 INFO:teuthology.orchestra.run.smithi191.stderr:Downloaded: 1 files, 37M in 1.4s (26.4 MB/s) 2023-10-16T20:21:29.758 INFO:teuthology.task.kernel:Installing kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on mon.b... 2023-10-16T20:21:29.758 DEBUG:teuthology.orchestra.run.smithi191:> sudo dpkg -i /tmp/linux-image.deb 2023-10-16T20:21:29.977 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package linux-image-6.6.0-rc5-g47672119149b. 2023-10-16T20:21:32.059 INFO:teuthology.orchestra.run.smithi191.stdout:(Reading database ... 129542 files and directories currently installed.) 2023-10-16T20:21:32.063 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack /tmp/linux-image.deb ... 2023-10-16T20:21:32.162 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking linux-image-6.6.0-rc5-g47672119149b (6.6.0-rc5-g47672119149b-1) ... 2023-10-16T20:21:35.559 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up linux-image-6.6.0-rc5-g47672119149b (6.6.0-rc5-g47672119149b-1) ... 2023-10-16T20:21:35.667 INFO:teuthology.orchestra.run.smithi191.stderr:update-initramfs: Generating /boot/initrd.img-6.6.0-rc5-g47672119149b 2023-10-16T20:21:47.472 INFO:teuthology.orchestra.run.smithi088.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2023-10-16T20:21:47.473 INFO:teuthology.orchestra.run.smithi088.stderr: * [new branch] main -> origin/main 2023-10-16T20:21:47.503 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20190312 -> 20190312 2023-10-16T20:21:47.504 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20190416 -> 20190416 2023-10-16T20:21:47.504 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20190514 -> 20190514 2023-10-16T20:21:47.504 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20190618 -> 20190618 2023-10-16T20:21:47.504 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20190717 -> 20190717 2023-10-16T20:21:47.504 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20190815 -> 20190815 2023-10-16T20:21:47.506 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20190923 -> 20190923 2023-10-16T20:21:47.506 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20191022 -> 20191022 2023-10-16T20:21:47.506 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20191215 -> 20191215 2023-10-16T20:21:47.506 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20200122 -> 20200122 2023-10-16T20:21:47.507 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20200316 -> 20200316 2023-10-16T20:21:47.507 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20200421 -> 20200421 2023-10-16T20:21:47.507 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20200519 -> 20200519 2023-10-16T20:21:47.507 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20200619 -> 20200619 2023-10-16T20:21:47.507 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20200721 -> 20200721 2023-10-16T20:21:47.510 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20200817 -> 20200817 2023-10-16T20:21:47.510 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20200918 -> 20200918 2023-10-16T20:21:47.511 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20201022 -> 20201022 2023-10-16T20:21:47.511 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20201118 -> 20201118 2023-10-16T20:21:47.511 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20201218 -> 20201218 2023-10-16T20:21:47.511 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20210208 -> 20210208 2023-10-16T20:21:47.512 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20210315 -> 20210315 2023-10-16T20:21:47.512 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20210511 -> 20210511 2023-10-16T20:21:47.512 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20210716 -> 20210716 2023-10-16T20:21:47.512 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20210818 -> 20210818 2023-10-16T20:21:47.512 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20210919 -> 20210919 2023-10-16T20:21:47.512 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20211027 -> 20211027 2023-10-16T20:21:47.513 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20211216 -> 20211216 2023-10-16T20:21:47.513 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20220209 -> 20220209 2023-10-16T20:21:47.513 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20220310 -> 20220310 2023-10-16T20:21:47.513 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20220411 -> 20220411 2023-10-16T20:21:47.513 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20220509 -> 20220509 2023-10-16T20:21:47.514 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20220610 -> 20220610 2023-10-16T20:21:47.514 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20220708 -> 20220708 2023-10-16T20:21:47.514 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20220815 -> 20220815 2023-10-16T20:21:47.514 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20220913 -> 20220913 2023-10-16T20:21:47.514 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20221012 -> 20221012 2023-10-16T20:21:47.515 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20221109 -> 20221109 2023-10-16T20:21:47.515 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20221214 -> 20221214 2023-10-16T20:21:47.515 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20230117 -> 20230117 2023-10-16T20:21:47.515 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20230210 -> 20230210 2023-10-16T20:21:47.515 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20230310 -> 20230310 2023-10-16T20:21:47.515 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20230404 -> 20230404 2023-10-16T20:21:47.516 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20230515 -> 20230515 2023-10-16T20:21:47.516 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20230625 -> 20230625 2023-10-16T20:21:47.516 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20230804 -> 20230804 2023-10-16T20:21:47.516 INFO:teuthology.orchestra.run.smithi088.stderr: * [new tag] 20230919 -> 20230919 2023-10-16T20:21:51.201 INFO:teuthology.orchestra.run.smithi088.stderr:Updating files: 51% (1624/3161) Updating files: 52% (1644/3161) Updating files: 53% (1676/3161) Updating files: 54% (1707/3161) Updating files: 55% (1739/3161) Updating files: 56% (1771/3161) Updating files: 57% (1802/3161) Updating files: 58% (1834/3161) Updating files: 59% (1865/3161) Updating files: 60% (1897/3161) Updating files: 61% (1929/3161) Updating files: 62% (1960/3161) Updating files: 63% (1992/3161) Updating files: 64% (2024/3161) Updating files: 65% (2055/3161) Updating files: 66% (2087/3161) Updating files: 66% (2106/3161) Updating files: 67% (2118/3161) Updating files: 68% (2150/3161) Updating files: 69% (2182/3161) Updating files: 70% (2213/3161) Updating files: 71% (2245/3161) Updating files: 72% (2276/3161) Updating files: 73% (2308/3161) Updating files: 74% (2340/3161) Updating files: 75% (2371/3161) Updating files: 76% (2403/3161) Updating files: 77% (2434/3161) Updating files: 78% (2466/3161) Updating files: 79% (2498/3161) Updating files: 80% (2529/3161) Updating files: 81% (2561/3161) Updating files: 81% (2589/3161) Updating files: 82% (2593/3161) Updating files: 83% (2624/3161) Updating files: 84% (2656/3161) Updating files: 85% (2687/3161) Updating files: 86% (2719/3161) Updating files: 87% (2751/3161) Updating files: 88% (2782/3161) Updating files: 89% (2814/3161) Updating files: 90% (2845/3161) Updating files: 91% (2877/3161) Updating files: 92% (2909/3161) Updating files: 93% (2940/3161) Updating files: 94% (2972/3161) Updating files: 95% (3003/3161) Updating files: 96% (3035/3161) Updating files: 97% (3067/3161) Updating files: 98% (3098/3161) Updating files: 99% (3130/3161) Updating files: 100% (3161/3161) Updating files: 100% (3161/3161), done. 2023-10-16T20:21:51.206 INFO:teuthology.orchestra.run.smithi088.stdout:HEAD is now at 63e8aa40 Merge branch 'main' into 'main' 2023-10-16T20:21:51.225 INFO:teuthology.task.kernel:Downloading kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on mon.a... 2023-10-16T20:21:51.226 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-16T20:21:51.371 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-g47672119149b/ 2023-10-16T20:21:51.371 DEBUG:teuthology.orchestra.run.smithi088:> rm -f /tmp/linux-image.deb && echo linux-image-6.6.0-rc5-g47672119149b_6.6.0-rc5-g47672119149b-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-g47672119149b/ --input-file=- 2023-10-16T20:21:51.374 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi088.front.sepia.ceph.com to complete... 2023-10-16T20:21:52.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16 20:21:52 URL:https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-g47672119149b/linux-image-6.6.0-rc5-g47672119149b_6.6.0-rc5-g47672119149b-1_amd64.deb [38909314/38909314] -> "/tmp/linux-image.deb" [1] 2023-10-16T20:21:52.894 INFO:teuthology.orchestra.run.smithi088.stderr:FINISHED --2023-10-16 20:21:52-- 2023-10-16T20:21:52.894 INFO:teuthology.orchestra.run.smithi088.stderr:Total wall clock time: 1.5s 2023-10-16T20:21:52.895 INFO:teuthology.orchestra.run.smithi088.stderr:Downloaded: 1 files, 37M in 1.4s (27.1 MB/s) 2023-10-16T20:21:52.897 INFO:teuthology.task.kernel:Installing kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on mon.a... 2023-10-16T20:21:52.897 DEBUG:teuthology.orchestra.run.smithi088:> sudo dpkg -i /tmp/linux-image.deb 2023-10-16T20:21:53.129 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-image-6.6.0-rc5-g47672119149b. 2023-10-16T20:21:55.123 INFO:teuthology.orchestra.run.smithi088.stdout:(Reading database ... 129542 files and directories currently installed.) 2023-10-16T20:21:55.128 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack /tmp/linux-image.deb ... 2023-10-16T20:21:55.235 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-image-6.6.0-rc5-g47672119149b (6.6.0-rc5-g47672119149b-1) ... 2023-10-16T20:21:55.734 INFO:teuthology.orchestra.run.smithi191.stderr:Sourcing file `/etc/default/grub' 2023-10-16T20:21:55.752 INFO:teuthology.orchestra.run.smithi191.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-16T20:21:55.784 INFO:teuthology.orchestra.run.smithi191.stderr:Generating grub configuration file ... 2023-10-16T20:21:56.074 INFO:teuthology.orchestra.run.smithi191.stderr:Found linux image: /boot/vmlinuz-6.6.0-rc5-g47672119149b 2023-10-16T20:21:56.095 INFO:teuthology.orchestra.run.smithi191.stderr:Found initrd image: /boot/initrd.img-6.6.0-rc5-g47672119149b 2023-10-16T20:21:56.378 INFO:teuthology.orchestra.run.smithi191.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-16T20:21:56.385 INFO:teuthology.orchestra.run.smithi191.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-16T20:21:56.450 INFO:teuthology.orchestra.run.smithi191.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-16T20:21:56.456 INFO:teuthology.orchestra.run.smithi191.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-16T20:21:56.607 INFO:teuthology.orchestra.run.smithi191.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-16T20:21:56.607 INFO:teuthology.orchestra.run.smithi191.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-16T20:21:56.607 INFO:teuthology.orchestra.run.smithi191.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-16T20:21:56.650 INFO:teuthology.orchestra.run.smithi191.stderr:done 2023-10-16T20:21:56.853 DEBUG:teuthology.orchestra.run.smithi191:> dpkg-deb -c /tmp/linux-image.deb 2023-10-16T20:21:56.861 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./ 2023-10-16T20:21:56.861 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./boot/ 2023-10-16T20:21:56.861 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 4415097 2023-10-10 19:56 ./boot/System.map-6.6.0-rc5-g47672119149b 2023-10-16T20:21:56.871 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 169298 2023-10-10 19:56 ./boot/config-6.6.0-rc5-g47672119149b 2023-10-16T20:21:56.871 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11352928 2023-10-10 19:56 ./boot/vmlinuz-6.6.0-rc5-g47672119149b 2023-10-16T20:21:56.890 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./etc/ 2023-10-16T20:21:56.891 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./etc/kernel/ 2023-10-16T20:21:56.891 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./etc/kernel/postinst.d/ 2023-10-16T20:21:56.891 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./etc/kernel/postrm.d/ 2023-10-16T20:21:56.891 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./etc/kernel/preinst.d/ 2023-10-16T20:21:56.891 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./etc/kernel/prerm.d/ 2023-10-16T20:21:56.892 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/ 2023-10-16T20:21:56.892 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/ 2023-10-16T20:21:56.892 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/ 2023-10-16T20:21:56.892 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/ 2023-10-16T20:21:56.893 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/ 2023-10-16T20:21:56.893 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/ 2023-10-16T20:21:56.893 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/ 2023-10-16T20:21:56.893 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 490233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/aesni-intel.ko 2023-10-16T20:21:56.893 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-10-16T20:21:56.894 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-10-16T20:21:56.894 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-10-16T20:21:56.894 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 93113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-10-16T20:21:56.894 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 106729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-10-16T20:21:56.895 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 152241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-10-16T20:21:56.895 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-10-16T20:21:56.895 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-10-16T20:21:56.895 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 83889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-10-16T20:21:56.895 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-10-16T20:21:56.896 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 47225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-10-16T20:21:56.896 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 47209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx2.ko 2023-10-16T20:21:56.896 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 51801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-10-16T20:21:56.896 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 38937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-10-16T20:21:56.896 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 40529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-10-16T20:21:56.897 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 58585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-10-16T20:21:56.897 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 50497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-10-16T20:21:56.897 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 32065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-10-16T20:21:56.898 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-10-16T20:21:56.898 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/ 2023-10-16T20:21:56.898 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/ 2023-10-16T20:21:56.898 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce/ 2023-10-16T20:21:56.899 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 40113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-10-16T20:21:56.899 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpuid.ko 2023-10-16T20:21:56.899 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/msr.ko 2023-10-16T20:21:56.899 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/ 2023-10-16T20:21:56.900 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 353761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-amd.ko 2023-10-16T20:21:56.900 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 881713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-intel.ko 2023-10-16T20:21:56.973 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 2276817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm.ko 2023-10-16T20:21:56.974 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ 2023-10-16T20:21:56.974 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 53761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/af_alg.ko 2023-10-16T20:21:56.974 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/algif_hash.ko 2023-10-16T20:21:56.974 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/algif_skcipher.ko 2023-10-16T20:21:56.975 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ansi_cprng.ko 2023-10-16T20:21:56.975 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/anubis.ko 2023-10-16T20:21:56.975 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/arc4.ko 2023-10-16T20:21:56.975 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/asymmetric_keys/ 2023-10-16T20:21:56.975 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2023-10-16T20:21:56.976 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/ 2023-10-16T20:21:56.976 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_memcpy.ko 2023-10-16T20:21:56.976 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_pq.ko 2023-10-16T20:21:56.976 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_raid6_recov.ko 2023-10-16T20:21:56.976 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_tx.ko 2023-10-16T20:21:56.976 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_xor.ko 2023-10-16T20:21:56.977 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/raid6test.ko 2023-10-16T20:21:56.977 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authenc.ko 2023-10-16T20:21:56.977 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authencesn.ko 2023-10-16T20:21:56.977 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blake2b_generic.ko 2023-10-16T20:21:56.977 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_common.ko 2023-10-16T20:21:56.978 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_generic.ko 2023-10-16T20:21:56.978 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 38873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/camellia_generic.ko 2023-10-16T20:21:56.978 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast5_generic.ko 2023-10-16T20:21:56.978 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast6_generic.ko 2023-10-16T20:21:56.978 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast_common.ko 2023-10-16T20:21:56.979 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ccm.ko 2023-10-16T20:21:56.979 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cmac.ko 2023-10-16T20:21:56.979 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crc32_generic.ko 2023-10-16T20:21:56.979 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 47505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cryptd.ko 2023-10-16T20:21:56.979 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 39505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_engine.ko 2023-10-16T20:21:56.980 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_simd.ko 2023-10-16T20:21:56.980 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_user.ko 2023-10-16T20:21:56.980 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cts.ko 2023-10-16T20:21:56.980 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/des_generic.ko 2023-10-16T20:21:56.980 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/dh_generic.ko 2023-10-16T20:21:56.980 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/echainiv.ko 2023-10-16T20:21:56.981 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/essiv.ko 2023-10-16T20:21:56.981 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/fcrypt.ko 2023-10-16T20:21:56.981 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/khazad.ko 2023-10-16T20:21:56.981 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lrw.ko 2023-10-16T20:21:56.981 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4.ko 2023-10-16T20:21:56.982 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4hc.ko 2023-10-16T20:21:56.982 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/md4.ko 2023-10-16T20:21:56.982 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/michael_mic.ko 2023-10-16T20:21:56.982 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcbc.ko 2023-10-16T20:21:56.982 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcrypt.ko 2023-10-16T20:21:56.983 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/rmd160.ko 2023-10-16T20:21:56.983 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/seed.ko 2023-10-16T20:21:56.983 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/serpent_generic.ko 2023-10-16T20:21:56.983 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 149193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tcrypt.ko 2023-10-16T20:21:56.983 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tea.ko 2023-10-16T20:21:56.984 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_common.ko 2023-10-16T20:21:56.984 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_generic.ko 2023-10-16T20:21:56.984 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/vmac.ko 2023-10-16T20:21:56.984 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/wp512.ko 2023-10-16T20:21:56.984 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xcbc.ko 2023-10-16T20:21:56.985 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xor.ko 2023-10-16T20:21:56.985 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xts.ko 2023-10-16T20:21:56.985 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xxhash_generic.ko 2023-10-16T20:21:56.985 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ 2023-10-16T20:21:56.985 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/ 2023-10-16T20:21:56.986 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_extlog.ko 2023-10-16T20:21:56.986 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_ipmi.ko 2023-10-16T20:21:56.986 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_pad.ko 2023-10-16T20:21:56.986 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/apei/ 2023-10-16T20:21:56.986 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/apei/einj.ko 2023-10-16T20:21:56.986 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/ec_sys.ko 2023-10-16T20:21:56.987 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/platform_profile.ko 2023-10-16T20:21:56.987 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbs.ko 2023-10-16T20:21:56.987 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbshc.ko 2023-10-16T20:21:56.987 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 103025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/video.ko 2023-10-16T20:21:56.987 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ 2023-10-16T20:21:56.988 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/acard-ahci.ko 2023-10-16T20:21:56.988 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 88865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ahci.ko 2023-10-16T20:21:56.988 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ahci_platform.ko 2023-10-16T20:21:56.988 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 88289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/libahci.ko 2023-10-16T20:21:56.988 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/libahci_platform.ko 2023-10-16T20:21:56.989 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_acpi.ko 2023-10-16T20:21:56.989 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ali.ko 2023-10-16T20:21:56.989 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_amd.ko 2023-10-16T20:21:56.991 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_artop.ko 2023-10-16T20:21:56.991 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atiixp.ko 2023-10-16T20:21:56.991 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atp867x.ko 2023-10-16T20:21:56.991 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd640.ko 2023-10-16T20:21:56.991 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd64x.ko 2023-10-16T20:21:56.992 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cypress.ko 2023-10-16T20:21:56.992 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_efar.ko 2023-10-16T20:21:56.992 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt366.ko 2023-10-16T20:21:56.992 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt37x.ko 2023-10-16T20:21:56.992 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x2n.ko 2023-10-16T20:21:56.993 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x3.ko 2023-10-16T20:21:56.993 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it8213.ko 2023-10-16T20:21:56.993 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it821x.ko 2023-10-16T20:21:56.993 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_jmicron.ko 2023-10-16T20:21:56.993 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 51481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_legacy.ko 2023-10-16T20:21:56.993 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_marvell.ko 2023-10-16T20:21:56.994 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16281 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_mpiix.ko 2023-10-16T20:21:56.994 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_netcell.ko 2023-10-16T20:21:56.994 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ninja32.ko 2023-10-16T20:21:56.994 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87410.ko 2023-10-16T20:21:56.994 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87415.ko 2023-10-16T20:21:56.995 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_oldpiix.ko 2023-10-16T20:21:56.995 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15377 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_opti.ko 2023-10-16T20:21:56.995 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_optidma.ko 2023-10-16T20:21:56.995 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 32585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pcmcia.ko 2023-10-16T20:21:56.995 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 37793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc2027x.ko 2023-10-16T20:21:56.996 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-10-16T20:21:56.996 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_piccolo.ko 2023-10-16T20:21:56.996 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_radisys.ko 2023-10-16T20:21:56.996 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_rdc.ko 2023-10-16T20:21:56.996 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_rz1000.ko 2023-10-16T20:21:56.997 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sch.ko 2023-10-16T20:21:56.997 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_serverworks.ko 2023-10-16T20:21:56.997 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sil680.ko 2023-10-16T20:21:56.997 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sl82c105.ko 2023-10-16T20:21:56.997 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_triflex.ko 2023-10-16T20:21:56.998 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_via.ko 2023-10-16T20:21:56.998 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pdc_adma.ko 2023-10-16T20:21:56.998 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_inic162x.ko 2023-10-16T20:21:56.998 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 83345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_mv.ko 2023-10-16T20:21:56.998 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 70393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_nv.ko 2023-10-16T20:21:56.999 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 37377 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_promise.ko 2023-10-16T20:21:56.999 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_qstor.ko 2023-10-16T20:21:56.999 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil.ko 2023-10-16T20:21:56.999 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 38337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil24.ko 2023-10-16T20:21:56.999 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sis.ko 2023-10-16T20:21:56.999 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_svw.ko 2023-10-16T20:21:57.000 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 44793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sx4.ko 2023-10-16T20:21:57.000 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_uli.ko 2023-10-16T20:21:57.000 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_via.ko 2023-10-16T20:21:57.000 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_vsc.ko 2023-10-16T20:21:57.001 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay/ 2023-10-16T20:21:57.001 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay/ks0108.ko 2023-10-16T20:21:57.001 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma/ 2023-10-16T20:21:57.001 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 124473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma/bcma.ko 2023-10-16T20:21:57.001 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/ 2023-10-16T20:21:57.002 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe/ 2023-10-16T20:21:57.002 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 113665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe/aoe.ko 2023-10-16T20:21:57.002 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd/ 2023-10-16T20:21:57.002 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 965585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd/drbd.ko 2023-10-16T20:21:57.002 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 173225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/floppy.ko 2023-10-16T20:21:57.003 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx/ 2023-10-16T20:21:57.003 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 94297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-10-16T20:21:57.003 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 115793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/nbd.ko 2023-10-16T20:21:57.003 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk/ 2023-10-16T20:21:57.003 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 110449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk/null_blk.ko 2023-10-16T20:21:57.004 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 98193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/pktcdvd.ko 2023-10-16T20:21:57.004 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 263769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/rbd.ko 2023-10-16T20:21:57.004 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback/ 2023-10-16T20:21:57.004 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 98145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-10-16T20:21:57.004 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram/ 2023-10-16T20:21:57.005 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 60217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram/zram.ko 2023-10-16T20:21:57.005 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ 2023-10-16T20:21:57.005 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30705 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/applicom.ko 2023-10-16T20:21:57.005 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hangcheck-timer.ko 2023-10-16T20:21:57.005 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/ 2023-10-16T20:21:57.005 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/amd-rng.ko 2023-10-16T20:21:57.006 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/intel-rng.ko 2023-10-16T20:21:57.006 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-10-16T20:21:57.006 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/via-rng.ko 2023-10-16T20:21:57.006 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/virtio-rng.ko 2023-10-16T20:21:57.006 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ 2023-10-16T20:21:57.007 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-10-16T20:21:57.007 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 148601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-10-16T20:21:57.007 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-10-16T20:21:57.007 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 139313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_si.ko 2023-10-16T20:21:57.007 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 70921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-10-16T20:21:57.008 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 53513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-10-16T20:21:57.008 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/lp.ko 2023-10-16T20:21:57.008 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave/ 2023-10-16T20:21:57.008 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 84097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave/mwave.ko 2023-10-16T20:21:57.008 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/nvram.ko 2023-10-16T20:21:57.009 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 32169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ppdev.ko 2023-10-16T20:21:57.009 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 51113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tlclk.ko 2023-10-16T20:21:57.009 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/ 2023-10-16T20:21:57.009 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_atmel.ko 2023-10-16T20:21:57.009 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2023-10-16T20:21:57.010 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2023-10-16T20:21:57.010 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 32985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2023-10-16T20:21:57.010 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_infineon.ko 2023-10-16T20:21:57.010 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_nsc.ko 2023-10-16T20:21:57.010 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/xen-tpmfront.ko 2023-10-16T20:21:57.010 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/xillybus/ 2023-10-16T20:21:57.011 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/xillybus/xillybus_class.ko 2023-10-16T20:21:57.011 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 50945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/xillybus/xillybus_core.ko 2023-10-16T20:21:57.011 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/xillybus/xillybus_pcie.ko 2023-10-16T20:21:57.011 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/clk/ 2023-10-16T20:21:57.011 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/clk/clk-palmas.ko 2023-10-16T20:21:57.012 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 47849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/clk/clk-si5351.ko 2023-10-16T20:21:57.012 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/clk/clk-twl6040.ko 2023-10-16T20:21:57.012 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/clk/clk-wm831x.ko 2023-10-16T20:21:57.012 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/ 2023-10-16T20:21:57.012 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-10-16T20:21:57.013 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/p4-clockmod.ko 2023-10-16T20:21:57.013 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/speedstep-lib.ko 2023-10-16T20:21:57.013 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ 2023-10-16T20:21:57.013 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ 2023-10-16T20:21:57.013 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 70801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-10-16T20:21:57.014 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 243025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp.ko 2023-10-16T20:21:57.014 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/ 2023-10-16T20:21:57.014 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/ 2023-10-16T20:21:57.014 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common/ 2023-10-16T20:21:57.014 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 438761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-10-16T20:21:57.015 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2023-10-16T20:21:57.015 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-10-16T20:21:57.015 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-aes.ko 2023-10-16T20:21:57.015 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-sha.ko 2023-10-16T20:21:57.015 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio/ 2023-10-16T20:21:57.015 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 68857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-10-16T20:21:57.016 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca/ 2023-10-16T20:21:57.016 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca/dca.ko 2023-10-16T20:21:57.016 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ 2023-10-16T20:21:57.016 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw/ 2023-10-16T20:21:57.016 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw/dw_dmac.ko 2023-10-16T20:21:57.017 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat/ 2023-10-16T20:21:57.017 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 122569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat/ioatdma.ko 2023-10-16T20:21:57.017 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/ 2023-10-16T20:21:57.017 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 79953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/amd64_edac.ko 2023-10-16T20:21:57.017 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/e752x_edac.ko 2023-10-16T20:21:57.018 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 63617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/edac_mce_amd.ko 2023-10-16T20:21:57.018 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3000_edac.ko 2023-10-16T20:21:57.018 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3200_edac.ko 2023-10-16T20:21:57.018 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5100_edac.ko 2023-10-16T20:21:57.018 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5400_edac.ko 2023-10-16T20:21:57.019 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7300_edac.ko 2023-10-16T20:21:57.019 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 52041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7core_edac.ko 2023-10-16T20:21:57.019 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i82975x_edac.ko 2023-10-16T20:21:57.019 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/ie31200_edac.ko 2023-10-16T20:21:57.019 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 56825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/sb_edac.ko 2023-10-16T20:21:57.020 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/x38_edac.ko 2023-10-16T20:21:57.020 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/extcon/ 2023-10-16T20:21:57.020 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/extcon/extcon-gpio.ko 2023-10-16T20:21:57.020 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/extcon/extcon-max14577.ko 2023-10-16T20:21:57.020 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/extcon/extcon-max77693.ko 2023-10-16T20:21:57.021 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/extcon/extcon-max8997.ko 2023-10-16T20:21:57.021 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/extcon/extcon-palmas.ko 2023-10-16T20:21:57.021 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/extcon/extcon-rt8973a.ko 2023-10-16T20:21:57.021 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/extcon/extcon-sm5502.ko 2023-10-16T20:21:57.021 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/ 2023-10-16T20:21:57.022 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/dmi-sysfs.ko 2023-10-16T20:21:57.022 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/efi/ 2023-10-16T20:21:57.022 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/efi/efi-pstore.ko 2023-10-16T20:21:57.022 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/iscsi_ibft.ko 2023-10-16T20:21:57.023 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/ 2023-10-16T20:21:57.023 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-adp5520.ko 2023-10-16T20:21:57.023 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-amd8111.ko 2023-10-16T20:21:57.023 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-arizona.ko 2023-10-16T20:21:57.023 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-da9052.ko 2023-10-16T20:21:57.023 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-da9055.ko 2023-10-16T20:21:57.024 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-dln2.ko 2023-10-16T20:21:57.024 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 77849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-f7188x.ko 2023-10-16T20:21:57.024 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-generic.ko 2023-10-16T20:21:57.024 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-ich.ko 2023-10-16T20:21:57.024 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-janz-ttl.ko 2023-10-16T20:21:57.025 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-kempld.ko 2023-10-16T20:21:57.025 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-lp3943.ko 2023-10-16T20:21:57.025 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-max7300.ko 2023-10-16T20:21:57.025 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-max730x.ko 2023-10-16T20:21:57.025 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-max732x.ko 2023-10-16T20:21:57.026 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-ml-ioh.ko 2023-10-16T20:21:57.026 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 47073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-pca953x.ko 2023-10-16T20:21:57.026 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-pcf857x.ko 2023-10-16T20:21:57.026 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14377 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-rdc321x.ko 2023-10-16T20:21:57.026 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-sch.ko 2023-10-16T20:21:57.027 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-sch311x.ko 2023-10-16T20:21:57.027 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-tps65912.ko 2023-10-16T20:21:57.027 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-twl4030.ko 2023-10-16T20:21:57.027 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-twl6040.ko 2023-10-16T20:21:57.027 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-viperboard.ko 2023-10-16T20:21:57.028 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15377 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-vx855.ko 2023-10-16T20:21:57.028 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16281 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-wm831x.ko 2023-10-16T20:21:57.028 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-wm8350.ko 2023-10-16T20:21:57.028 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-wm8994.ko 2023-10-16T20:21:57.028 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/ 2023-10-16T20:21:57.029 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-a4tech.ko 2023-10-16T20:21:57.029 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 39169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-apple.ko 2023-10-16T20:21:57.029 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-appleir.ko 2023-10-16T20:21:57.029 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-aureal.ko 2023-10-16T20:21:57.029 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-axff.ko 2023-10-16T20:21:57.030 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-belkin.ko 2023-10-16T20:21:57.030 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-cherry.ko 2023-10-16T20:21:57.030 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-chicony.ko 2023-10-16T20:21:57.030 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 60649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-cp2112.ko 2023-10-16T20:21:57.030 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-cypress.ko 2023-10-16T20:21:57.031 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-dr.ko 2023-10-16T20:21:57.031 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elecom.ko 2023-10-16T20:21:57.031 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elo.ko 2023-10-16T20:21:57.031 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-emsff.ko 2023-10-16T20:21:57.031 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-ezkey.ko 2023-10-16T20:21:57.032 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gaff.ko 2023-10-16T20:21:57.032 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-generic.ko 2023-10-16T20:21:57.032 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gt683r.ko 2023-10-16T20:21:57.032 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gyration.ko 2023-10-16T20:21:57.032 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-kbd.ko 2023-10-16T20:21:57.033 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-mouse.ko 2023-10-16T20:21:57.033 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtekff.ko 2023-10-16T20:21:57.033 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-hyperv.ko 2023-10-16T20:21:57.033 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-icade.ko 2023-10-16T20:21:57.033 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-kensington.ko 2023-10-16T20:21:57.034 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-keytouch.ko 2023-10-16T20:21:57.034 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-kye.ko 2023-10-16T20:21:57.034 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lcpower.ko 2023-10-16T20:21:57.034 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-led.ko 2023-10-16T20:21:57.034 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 42361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lenovo.ko 2023-10-16T20:21:57.034 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lg-g15.ko 2023-10-16T20:21:57.035 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 58185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-dj.ko 2023-10-16T20:21:57.035 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 101369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-10-16T20:21:57.035 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 80993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech.ko 2023-10-16T20:21:57.035 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-magicmouse.ko 2023-10-16T20:21:57.035 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-microsoft.ko 2023-10-16T20:21:57.036 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-monterey.ko 2023-10-16T20:21:57.036 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 50161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-multitouch.ko 2023-10-16T20:21:57.036 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 32593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-ntrig.ko 2023-10-16T20:21:57.036 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-ortek.ko 2023-10-16T20:21:57.036 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-penmount.ko 2023-10-16T20:21:57.037 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-petalynx.ko 2023-10-16T20:21:57.037 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 72937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-picolcd.ko 2023-10-16T20:21:57.037 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-pl.ko 2023-10-16T20:21:57.037 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-plantronics.ko 2023-10-16T20:21:57.037 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-primax.ko 2023-10-16T20:21:57.038 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-rmi.ko 2023-10-16T20:21:57.038 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-arvo.ko 2023-10-16T20:21:57.038 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-common.ko 2023-10-16T20:21:57.038 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-isku.ko 2023-10-16T20:21:57.038 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kone.ko 2023-10-16T20:21:57.039 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-10-16T20:21:57.039 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-konepure.ko 2023-10-16T20:21:57.039 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-10-16T20:21:57.039 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-lua.ko 2023-10-16T20:21:57.040 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-pyra.ko 2023-10-16T20:21:57.040 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-ryos.ko 2023-10-16T20:21:57.040 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-savu.ko 2023-10-16T20:21:57.040 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat.ko 2023-10-16T20:21:57.040 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-saitek.ko 2023-10-16T20:21:57.041 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-samsung.ko 2023-10-16T20:21:57.041 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sensor-hub.ko 2023-10-16T20:21:57.041 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sjoy.ko 2023-10-16T20:21:57.041 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 51249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sony.ko 2023-10-16T20:21:57.041 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-speedlink.ko 2023-10-16T20:21:57.042 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-steelseries.ko 2023-10-16T20:21:57.042 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sunplus.ko 2023-10-16T20:21:57.042 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-thrustmaster.ko 2023-10-16T20:21:57.042 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tivo.ko 2023-10-16T20:21:57.042 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tmff.ko 2023-10-16T20:21:57.043 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-topseed.ko 2023-10-16T20:21:57.043 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-twinhan.ko 2023-10-16T20:21:57.043 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 71073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-uclogic.ko 2023-10-16T20:21:57.043 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-waltop.ko 2023-10-16T20:21:57.043 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 143473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-wiimote.ko 2023-10-16T20:21:57.044 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-xinmo.ko 2023-10-16T20:21:57.044 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zpff.ko 2023-10-16T20:21:57.044 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zydacron.ko 2023-10-16T20:21:57.044 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 267825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid.ko 2023-10-16T20:21:57.044 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 32321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/uhid.ko 2023-10-16T20:21:57.045 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/usbhid/ 2023-10-16T20:21:57.045 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 125721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/usbhid/usbhid.ko 2023-10-16T20:21:57.045 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/usbhid/usbkbd.ko 2023-10-16T20:21:57.045 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/usbhid/usbmouse.ko 2023-10-16T20:21:57.045 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 217873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/wacom.ko 2023-10-16T20:21:57.045 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/ 2023-10-16T20:21:57.046 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 64201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_balloon.ko 2023-10-16T20:21:57.046 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 89625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_utils.ko 2023-10-16T20:21:57.046 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 305465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_vmbus.ko 2023-10-16T20:21:57.046 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ 2023-10-16T20:21:57.046 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 66521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru.ko 2023-10-16T20:21:57.047 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 61393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru3.ko 2023-10-16T20:21:57.047 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/acpi_power_meter.ko 2023-10-16T20:21:57.047 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7414.ko 2023-10-16T20:21:57.047 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7418.ko 2023-10-16T20:21:57.047 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adc128d818.ko 2023-10-16T20:21:57.047 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1025.ko 2023-10-16T20:21:57.048 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 97977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1026.ko 2023-10-16T20:21:57.048 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1029.ko 2023-10-16T20:21:57.048 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 45553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1031.ko 2023-10-16T20:21:57.048 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm9240.ko 2023-10-16T20:21:57.048 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ads7828.ko 2023-10-16T20:21:57.049 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7410.ko 2023-10-16T20:21:57.049 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7411.ko 2023-10-16T20:21:57.049 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 74433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7462.ko 2023-10-16T20:21:57.049 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 39585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7470.ko 2023-10-16T20:21:57.049 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 67929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7475.ko 2023-10-16T20:21:57.050 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7x10.ko 2023-10-16T20:21:57.050 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 39073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/amc6821.ko 2023-10-16T20:21:57.050 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 55265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/applesmc.ko 2023-10-16T20:21:57.050 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 49913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asb100.ko 2023-10-16T20:21:57.050 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 58617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asc7621.ko 2023-10-16T20:21:57.050 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 40889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asus_atk0110.ko 2023-10-16T20:21:57.051 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/atxp1.ko 2023-10-16T20:21:57.051 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28281 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/coretemp.ko 2023-10-16T20:21:57.051 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/da9052-hwmon.ko 2023-10-16T20:21:57.051 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/da9055-hwmon.ko 2023-10-16T20:21:57.051 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 44465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-10-16T20:21:57.052 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 99601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dme1737.ko 2023-10-16T20:21:57.052 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds1621.ko 2023-10-16T20:21:57.052 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds620.ko 2023-10-16T20:21:57.052 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc1403.ko 2023-10-16T20:21:57.052 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc2103.ko 2023-10-16T20:21:57.053 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc6w201.ko 2023-10-16T20:21:57.053 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 66985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71805f.ko 2023-10-16T20:21:57.053 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 94457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71882fg.ko 2023-10-16T20:21:57.053 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 41401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f75375s.ko 2023-10-16T20:21:57.053 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fam15h_power.ko 2023-10-16T20:21:57.053 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 55449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fschmd.ko 2023-10-16T20:21:57.054 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g760a.ko 2023-10-16T20:21:57.054 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g762.ko 2023-10-16T20:21:57.054 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 46801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl518sm.ko 2023-10-16T20:21:57.054 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 45529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl520sm.ko 2023-10-16T20:21:57.054 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/hih6130.ko 2023-10-16T20:21:57.055 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/hwmon-vid.ko 2023-10-16T20:21:57.055 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5500_temp.ko 2023-10-16T20:21:57.055 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5k_amb.ko 2023-10-16T20:21:57.055 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmaem.ko 2023-10-16T20:21:57.055 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmpex.ko 2023-10-16T20:21:57.056 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina209.ko 2023-10-16T20:21:57.056 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina2xx.ko 2023-10-16T20:21:57.056 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 130449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/it87.ko 2023-10-16T20:21:57.056 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/jc42.ko 2023-10-16T20:21:57.056 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k10temp.ko 2023-10-16T20:21:57.056 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k8temp.ko 2023-10-16T20:21:57.057 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lineage-pem.ko 2023-10-16T20:21:57.057 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 48601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm63.ko 2023-10-16T20:21:57.057 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm73.ko 2023-10-16T20:21:57.057 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm75.ko 2023-10-16T20:21:57.057 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm77.ko 2023-10-16T20:21:57.058 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm78.ko 2023-10-16T20:21:57.058 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm80.ko 2023-10-16T20:21:57.058 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm83.ko 2023-10-16T20:21:57.058 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 72833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm85.ko 2023-10-16T20:21:57.058 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 46153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm87.ko 2023-10-16T20:21:57.059 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 37913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm90.ko 2023-10-16T20:21:57.059 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm92.ko 2023-10-16T20:21:57.059 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 108841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm93.ko 2023-10-16T20:21:57.059 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95234.ko 2023-10-16T20:21:57.059 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95241.ko 2023-10-16T20:21:57.059 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95245.ko 2023-10-16T20:21:57.060 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc2945.ko 2023-10-16T20:21:57.060 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4151.ko 2023-10-16T20:21:57.060 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4215.ko 2023-10-16T20:21:57.060 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4222.ko 2023-10-16T20:21:57.060 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4245.ko 2023-10-16T20:21:57.061 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4260.ko 2023-10-16T20:21:57.061 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4261.ko 2023-10-16T20:21:57.061 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max16065.ko 2023-10-16T20:21:57.061 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1619.ko 2023-10-16T20:21:57.061 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1668.ko 2023-10-16T20:21:57.062 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max197.ko 2023-10-16T20:21:57.062 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6639.ko 2023-10-16T20:21:57.062 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6650.ko 2023-10-16T20:21:57.062 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6697.ko 2023-10-16T20:21:57.062 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/mc13783-adc.ko 2023-10-16T20:21:57.063 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/mcp3021.ko 2023-10-16T20:21:57.063 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2023-10-16T20:21:57.063 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 44129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6683.ko 2023-10-16T20:21:57.063 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 119817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775-core.ko 2023-10-16T20:21:57.063 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 60705 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775.ko 2023-10-16T20:21:57.064 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 59737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct7802.ko 2023-10-16T20:21:57.064 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 82337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87360.ko 2023-10-16T20:21:57.064 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 54433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87427.ko 2023-10-16T20:21:57.064 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pcf8591.ko 2023-10-16T20:21:57.064 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ 2023-10-16T20:21:57.064 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-10-16T20:21:57.065 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-10-16T20:21:57.065 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-10-16T20:21:57.065 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max16064.ko 2023-10-16T20:21:57.065 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max34440.ko 2023-10-16T20:21:57.065 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max8688.ko 2023-10-16T20:21:57.066 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-10-16T20:21:57.066 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 82577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-10-16T20:21:57.066 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-10-16T20:21:57.066 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-10-16T20:21:57.066 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-10-16T20:21:57.067 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-10-16T20:21:57.067 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/powr1220.ko 2023-10-16T20:21:57.067 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18705 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5627.ko 2023-10-16T20:21:57.067 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5636.ko 2023-10-16T20:21:57.067 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch56xx-common.ko 2023-10-16T20:21:57.068 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 32137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht15.ko 2023-10-16T20:21:57.068 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht21.ko 2023-10-16T20:21:57.068 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/shtc1.ko 2023-10-16T20:21:57.068 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 40105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sis5595.ko 2023-10-16T20:21:57.068 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47b397.ko 2023-10-16T20:21:57.069 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 39193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m1.ko 2023-10-16T20:21:57.069 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 37385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m192.ko 2023-10-16T20:21:57.069 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/thmc50.ko 2023-10-16T20:21:57.069 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp102.ko 2023-10-16T20:21:57.069 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp103.ko 2023-10-16T20:21:57.070 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp401.ko 2023-10-16T20:21:57.070 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp421.ko 2023-10-16T20:21:57.070 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via-cputemp.ko 2023-10-16T20:21:57.070 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 41945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via686a.ko 2023-10-16T20:21:57.070 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 52457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt1211.ko 2023-10-16T20:21:57.071 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 51545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt8231.ko 2023-10-16T20:21:57.071 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 57513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627ehf.ko 2023-10-16T20:21:57.071 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 78873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627hf.ko 2023-10-16T20:21:57.071 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 71345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83781d.ko 2023-10-16T20:21:57.071 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 63289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83791d.ko 2023-10-16T20:21:57.072 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 71321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83792d.ko 2023-10-16T20:21:57.072 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 101473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83793.ko 2023-10-16T20:21:57.072 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 119505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83795.ko 2023-10-16T20:21:57.072 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l785ts.ko 2023-10-16T20:21:57.072 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l786ng.ko 2023-10-16T20:21:57.073 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/wm831x-hwmon.ko 2023-10-16T20:21:57.073 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/wm8350-hwmon.ko 2023-10-16T20:21:57.073 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/ 2023-10-16T20:21:57.073 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/ 2023-10-16T20:21:57.073 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-10-16T20:21:57.074 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-10-16T20:21:57.074 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/ 2023-10-16T20:21:57.074 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ali1535.ko 2023-10-16T20:21:57.074 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ali1563.ko 2023-10-16T20:21:57.074 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2023-10-16T20:21:57.075 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-10-16T20:21:57.075 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-10-16T20:21:57.075 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-10-16T20:21:57.075 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2023-10-16T20:21:57.075 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 6929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-10-16T20:21:57.076 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2023-10-16T20:21:57.076 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 50201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-10-16T20:21:57.076 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-10-16T20:21:57.076 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-10-16T20:21:57.076 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-10-16T20:21:57.077 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-dln2.ko 2023-10-16T20:21:57.077 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-gpio.ko 2023-10-16T20:21:57.077 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 62817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-i801.ko 2023-10-16T20:21:57.077 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-isch.ko 2023-10-16T20:21:57.077 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 47889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-10-16T20:21:57.077 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-kempld.ko 2023-10-16T20:21:57.078 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-10-16T20:21:57.078 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-10-16T20:21:57.078 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ocores.ko 2023-10-16T20:21:57.078 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-parport.ko 2023-10-16T20:21:57.078 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-10-16T20:21:57.079 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 47241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-10-16T20:21:57.079 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2023-10-16T20:21:57.079 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-10-16T20:21:57.079 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-10-16T20:21:57.079 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-sis5595.ko 2023-10-16T20:21:57.080 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-sis630.ko 2023-10-16T20:21:57.080 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-10-16T20:21:57.080 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2023-10-16T20:21:57.080 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-10-16T20:21:57.080 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-via.ko 2023-10-16T20:21:57.080 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-10-16T20:21:57.081 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-10-16T20:21:57.081 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 47657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-xiic.ko 2023-10-16T20:21:57.081 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-mux.ko 2023-10-16T20:21:57.081 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-smbus.ko 2023-10-16T20:21:57.081 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-stub.ko 2023-10-16T20:21:57.082 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/muxes/ 2023-10-16T20:21:57.082 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2023-10-16T20:21:57.082 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2023-10-16T20:21:57.082 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2023-10-16T20:21:57.082 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ 2023-10-16T20:21:57.083 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ 2023-10-16T20:21:57.083 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 273609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_cm.ko 2023-10-16T20:21:57.083 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 879969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_core.ko 2023-10-16T20:21:57.083 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 69337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_umad.ko 2023-10-16T20:21:57.083 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 324217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-10-16T20:21:57.084 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 97985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/iw_cm.ko 2023-10-16T20:21:57.084 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 282273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_cm.ko 2023-10-16T20:21:57.084 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 66201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-10-16T20:21:57.084 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/ 2023-10-16T20:21:57.084 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4/ 2023-10-16T20:21:57.084 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 493073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-10-16T20:21:57.085 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4/ 2023-10-16T20:21:57.085 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 419737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-10-16T20:21:57.085 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5/ 2023-10-16T20:21:57.085 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 739025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-10-16T20:21:57.085 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca/ 2023-10-16T20:21:57.086 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 296625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-10-16T20:21:57.086 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/ocrdma/ 2023-10-16T20:21:57.086 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 153697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2023-10-16T20:21:57.086 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/usnic/ 2023-10-16T20:21:57.086 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 168969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2023-10-16T20:21:57.087 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ 2023-10-16T20:21:57.087 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib/ 2023-10-16T20:21:57.087 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 263081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-10-16T20:21:57.087 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser/ 2023-10-16T20:21:57.087 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 100241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-10-16T20:21:57.087 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert/ 2023-10-16T20:21:57.088 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 111425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-10-16T20:21:57.088 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp/ 2023-10-16T20:21:57.088 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 128753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-10-16T20:21:57.088 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt/ 2023-10-16T20:21:57.088 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 129529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-10-16T20:21:57.089 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/ 2023-10-16T20:21:57.089 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/evbug.ko 2023-10-16T20:21:57.089 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/ff-memless.ko 2023-10-16T20:21:57.089 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/ 2023-10-16T20:21:57.089 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-10-16T20:21:57.090 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/fm801-gp.ko 2023-10-16T20:21:57.090 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 41241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/gameport.ko 2023-10-16T20:21:57.090 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 41801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/joydev.ko 2023-10-16T20:21:57.090 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/ 2023-10-16T20:21:57.090 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/adp5520-keys.ko 2023-10-16T20:21:57.090 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/adp5588-keys.ko 2023-10-16T20:21:57.091 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/adp5589-keys.ko 2023-10-16T20:21:57.091 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2023-10-16T20:21:57.091 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/gpio_keys.ko 2023-10-16T20:21:57.091 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2023-10-16T20:21:57.091 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/lkkbd.ko 2023-10-16T20:21:57.092 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/lm8323.ko 2023-10-16T20:21:57.092 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/lm8333.ko 2023-10-16T20:21:57.092 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/matrix_keypad.ko 2023-10-16T20:21:57.092 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/max7359_keypad.ko 2023-10-16T20:21:57.092 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/mcs_touchkey.ko 2023-10-16T20:21:57.093 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2023-10-16T20:21:57.093 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/newtonkbd.ko 2023-10-16T20:21:57.093 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/opencores-kbd.ko 2023-10-16T20:21:57.093 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/qt1070.ko 2023-10-16T20:21:57.093 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/qt2160.ko 2023-10-16T20:21:57.094 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/samsung-keypad.ko 2023-10-16T20:21:57.094 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/stowaway.ko 2023-10-16T20:21:57.094 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/sunkbd.ko 2023-10-16T20:21:57.094 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/tca6416-keypad.ko 2023-10-16T20:21:57.094 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/tca8418_keypad.ko 2023-10-16T20:21:57.095 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/twl4030_keypad.ko 2023-10-16T20:21:57.095 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/xtkbd.ko 2023-10-16T20:21:57.095 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/matrix-keymap.ko 2023-10-16T20:21:57.095 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/ 2023-10-16T20:21:57.095 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/appletouch.ko 2023-10-16T20:21:57.096 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/bcm5974.ko 2023-10-16T20:21:57.096 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 123681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/cyapatp.ko 2023-10-16T20:21:57.096 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 89449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/elan_i2c.ko 2023-10-16T20:21:57.096 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/gpio_mouse.ko 2023-10-16T20:21:57.096 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 335977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/psmouse.ko 2023-10-16T20:21:57.096 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/sermouse.ko 2023-10-16T20:21:57.097 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-10-16T20:21:57.097 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_usb.ko 2023-10-16T20:21:57.097 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/vsxxxaa.ko 2023-10-16T20:21:57.097 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4/ 2023-10-16T20:21:57.097 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 109873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4/rmi_core.ko 2023-10-16T20:21:57.098 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/ 2023-10-16T20:21:57.098 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/altera_ps2.ko 2023-10-16T20:21:57.098 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/arc_ps2.ko 2023-10-16T20:21:57.098 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/ct82c710.ko 2023-10-16T20:21:57.098 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-10-16T20:21:57.099 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/parkbd.ko 2023-10-16T20:21:57.099 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/pcips2.ko 2023-10-16T20:21:57.099 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/ps2mult.ko 2023-10-16T20:21:57.099 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/serio_raw.ko 2023-10-16T20:21:57.099 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/serport.ko 2023-10-16T20:21:57.100 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/sparse-keymap.ko 2023-10-16T20:21:57.100 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/ 2023-10-16T20:21:57.100 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd/ 2023-10-16T20:21:57.100 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 39289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd/iommu_v2.ko 2023-10-16T20:21:57.100 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/ 2023-10-16T20:21:57.101 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-88pm860x.ko 2023-10-16T20:21:57.101 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-adp5520.ko 2023-10-16T20:21:57.101 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 59753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-bd2802.ko 2023-10-16T20:21:57.101 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-blinkm.ko 2023-10-16T20:21:57.101 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-da903x.ko 2023-10-16T20:21:57.101 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-da9052.ko 2023-10-16T20:21:57.102 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-gpio.ko 2023-10-16T20:21:57.102 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lm3530.ko 2023-10-16T20:21:57.102 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lm3533.ko 2023-10-16T20:21:57.102 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lm355x.ko 2023-10-16T20:21:57.102 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lm3642.ko 2023-10-16T20:21:57.103 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lp3944.ko 2023-10-16T20:21:57.103 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lp8788.ko 2023-10-16T20:21:57.103 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-max8997.ko 2023-10-16T20:21:57.103 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-mc13783.ko 2023-10-16T20:21:57.103 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-menf21bmc.ko 2023-10-16T20:21:57.104 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-pca9532.ko 2023-10-16T20:21:57.104 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-pca955x.ko 2023-10-16T20:21:57.104 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-pca963x.ko 2023-10-16T20:21:57.104 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-ss4200.ko 2023-10-16T20:21:57.104 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-tca6507.ko 2023-10-16T20:21:57.105 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-wm831x-status.ko 2023-10-16T20:21:57.105 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-wm8350.ko 2023-10-16T20:21:57.105 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ 2023-10-16T20:21:57.105 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-10-16T20:21:57.105 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-10-16T20:21:57.106 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-10-16T20:21:57.106 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-10-16T20:21:57.106 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-10-16T20:21:57.106 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-10-16T20:21:57.106 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-10-16T20:21:57.106 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-10-16T20:21:57.107 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/ 2023-10-16T20:21:57.107 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache/ 2023-10-16T20:21:57.107 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 654969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache/bcache.ko 2023-10-16T20:21:57.107 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 40577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-bio-prison.ko 2023-10-16T20:21:57.107 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 89601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-bufio.ko 2023-10-16T20:21:57.108 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 47529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache-smq.ko 2023-10-16T20:21:57.108 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 156009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache.ko 2023-10-16T20:21:57.108 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 106897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-crypt.ko 2023-10-16T20:21:57.108 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-delay.ko 2023-10-16T20:21:57.108 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 53857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-era.ko 2023-10-16T20:21:57.109 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-flakey.ko 2023-10-16T20:21:57.109 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 41593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-log-userspace.ko 2023-10-16T20:21:57.109 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 32737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-log.ko 2023-10-16T20:21:57.109 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 49545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-mirror.ko 2023-10-16T20:21:57.110 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 78033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-multipath.ko 2023-10-16T20:21:57.110 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-queue-length.ko 2023-10-16T20:21:57.110 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 70825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-raid.ko 2023-10-16T20:21:57.110 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-region-hash.ko 2023-10-16T20:21:57.110 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-round-robin.ko 2023-10-16T20:21:57.111 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-service-time.ko 2023-10-16T20:21:57.111 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 109593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-snapshot.ko 2023-10-16T20:21:57.111 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-switch.ko 2023-10-16T20:21:57.111 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 181865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-thin-pool.ko 2023-10-16T20:21:57.111 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 51737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-verity.ko 2023-10-16T20:21:57.112 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-zero.ko 2023-10-16T20:21:57.112 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/faulty.ko 2023-10-16T20:21:57.112 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/linear.ko 2023-10-16T20:21:57.112 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 32417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/multipath.ko 2023-10-16T20:21:57.112 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data/ 2023-10-16T20:21:57.112 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 199937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-10-16T20:21:57.113 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 37337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid0.ko 2023-10-16T20:21:57.113 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 113073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid1.ko 2023-10-16T20:21:57.113 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 158681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid10.ko 2023-10-16T20:21:57.113 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 395721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid456.ko 2023-10-16T20:21:57.113 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/ 2023-10-16T20:21:57.114 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/ 2023-10-16T20:21:57.114 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 178137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptbase.ko 2023-10-16T20:21:57.114 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 63385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptctl.ko 2023-10-16T20:21:57.114 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 44665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptfc.ko 2023-10-16T20:21:57.114 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 37225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptlan.ko 2023-10-16T20:21:57.115 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 123321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptsas.ko 2023-10-16T20:21:57.115 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 79041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptscsih.ko 2023-10-16T20:21:57.115 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 53081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptspi.ko 2023-10-16T20:21:57.115 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/ 2023-10-16T20:21:57.115 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/88pm800.ko 2023-10-16T20:21:57.116 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/88pm805.ko 2023-10-16T20:21:57.116 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/88pm80x.ko 2023-10-16T20:21:57.116 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/arizona-i2c.ko 2023-10-16T20:21:57.116 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 123721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/arizona.ko 2023-10-16T20:21:57.116 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/bcm590xx.ko 2023-10-16T20:21:57.116 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/cros_ec_dev.ko 2023-10-16T20:21:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/dln2.ko 2023-10-16T20:21:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/janz-cmodio.ko 2023-10-16T20:21:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 53585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/kempld-core.ko 2023-10-16T20:21:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lm3533-core.ko 2023-10-16T20:21:57.117 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lm3533-ctrlbank.ko 2023-10-16T20:21:57.118 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lp3943.ko 2023-10-16T20:21:57.118 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 54569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_ich.ko 2023-10-16T20:21:57.118 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_sch.ko 2023-10-16T20:21:57.118 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/max8907.ko 2023-10-16T20:21:57.118 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/mc13xxx-core.ko 2023-10-16T20:21:57.119 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/mc13xxx-i2c.ko 2023-10-16T20:21:57.119 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/menf21bmc.ko 2023-10-16T20:21:57.119 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/pcf50633-adc.ko 2023-10-16T20:21:57.119 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/pcf50633-gpio.ko 2023-10-16T20:21:57.119 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/pcf50633.ko 2023-10-16T20:21:57.120 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/rdc321x-southbridge.ko 2023-10-16T20:21:57.120 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/retu-mfd.ko 2023-10-16T20:21:57.120 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 80425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/si476x-core.ko 2023-10-16T20:21:57.120 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 48457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/sm501.ko 2023-10-16T20:21:57.120 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/ti_am335x_tscadc.ko 2023-10-16T20:21:57.121 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 51137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/tps65010.ko 2023-10-16T20:21:57.121 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/tps6507x.ko 2023-10-16T20:21:57.121 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/viperboard.ko 2023-10-16T20:21:57.121 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/vx855.ko 2023-10-16T20:21:57.121 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/wl1273-core.ko 2023-10-16T20:21:57.122 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 44345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/wm8994.ko 2023-10-16T20:21:57.122 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ 2023-10-16T20:21:57.122 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ad525x_dpot-i2c.ko 2023-10-16T20:21:57.122 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 48489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ad525x_dpot.ko 2023-10-16T20:21:57.122 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl/ 2023-10-16T20:21:57.122 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 46585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-10-16T20:21:57.123 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds9802als.ko 2023-10-16T20:21:57.123 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds990x.ko 2023-10-16T20:21:57.123 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 45593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/bh1770glc.ko 2023-10-16T20:21:57.123 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/c2port/ 2023-10-16T20:21:57.123 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2023-10-16T20:21:57.124 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35281 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/c2port/core.ko 2023-10-16T20:21:57.124 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710/ 2023-10-16T20:21:57.124 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710/cb710.ko 2023-10-16T20:21:57.124 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ds1682.ko 2023-10-16T20:21:57.124 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/dummy-irq.ko 2023-10-16T20:21:57.124 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/echo/ 2023-10-16T20:21:57.125 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/echo/echo.ko 2023-10-16T20:21:57.125 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/ 2023-10-16T20:21:57.125 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 37849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/at24.ko 2023-10-16T20:21:57.125 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom.ko 2023-10-16T20:21:57.125 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-10-16T20:21:57.126 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/max6875.ko 2023-10-16T20:21:57.126 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/enclosure.ko 2023-10-16T20:21:57.126 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/genwqe/ 2023-10-16T20:21:57.126 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 152201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/genwqe/genwqe_card.ko 2023-10-16T20:21:57.126 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/hmc6352.ko 2023-10-16T20:21:57.126 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/hpilo.ko 2023-10-16T20:21:57.127 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ibmasm/ 2023-10-16T20:21:57.127 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 72129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ibmasm/ibmasm.ko 2023-10-16T20:21:57.127 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ics932s401.ko 2023-10-16T20:21:57.127 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29003.ko 2023-10-16T20:21:57.127 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29020.ko 2023-10-16T20:21:57.128 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/ 2023-10-16T20:21:57.128 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 42513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-10-16T20:21:57.128 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-10-16T20:21:57.128 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/ 2023-10-16T20:21:57.128 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 125209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-me.ko 2023-10-16T20:21:57.128 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 56521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-txe.ko 2023-10-16T20:21:57.129 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 310113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei.ko 2023-10-16T20:21:57.129 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/phantom.ko 2023-10-16T20:21:57.129 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ti-st/ 2023-10-16T20:21:57.129 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 74265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ti-st/st_drv.ko 2023-10-16T20:21:57.129 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_7xx1.ko 2023-10-16T20:21:57.130 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_core.ko 2023-10-16T20:21:57.130 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tsl2550.ko 2023-10-16T20:21:57.130 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 48697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_balloon.ko 2023-10-16T20:21:57.130 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci/ 2023-10-16T20:21:57.130 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 183609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-10-16T20:21:57.131 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ 2023-10-16T20:21:57.131 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding/ 2023-10-16T20:21:57.131 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 490129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding/bonding.ko 2023-10-16T20:21:57.131 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/ 2023-10-16T20:21:57.131 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53/ 2023-10-16T20:21:57.132 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 106033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53/b53_common.ko 2023-10-16T20:21:57.132 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 40297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/bcm-sf2.ko 2023-10-16T20:21:57.132 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6060.ko 2023-10-16T20:21:57.132 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx/ 2023-10-16T20:21:57.132 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 453737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-10-16T20:21:57.133 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dummy.ko 2023-10-16T20:21:57.133 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/eql.ko 2023-10-16T20:21:57.133 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ 2023-10-16T20:21:57.133 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/ 2023-10-16T20:21:57.133 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 45785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-10-16T20:21:57.134 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 42105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-10-16T20:21:57.134 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 130553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-10-16T20:21:57.134 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 68849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-10-16T20:21:57.134 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/ 2023-10-16T20:21:57.134 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 37649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/8390.ko 2023-10-16T20:21:57.135 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 59689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-10-16T20:21:57.135 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-10-16T20:21:57.135 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 99113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-10-16T20:21:57.135 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec/ 2023-10-16T20:21:57.135 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 54521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-10-16T20:21:57.135 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere/ 2023-10-16T20:21:57.136 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 69201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere/et131x.ko 2023-10-16T20:21:57.136 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon/ 2023-10-16T20:21:57.136 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 59585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-10-16T20:21:57.136 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera/ 2023-10-16T20:21:57.136 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 41409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-10-16T20:21:57.137 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/ 2023-10-16T20:21:57.137 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 54609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-10-16T20:21:57.137 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 49289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-10-16T20:21:57.137 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 87297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-10-16T20:21:57.137 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/ 2023-10-16T20:21:57.138 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx/ 2023-10-16T20:21:57.138 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 97337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-10-16T20:21:57.138 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c/ 2023-10-16T20:21:57.138 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 104673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-10-16T20:21:57.138 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e/ 2023-10-16T20:21:57.138 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 90257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-10-16T20:21:57.139 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/ 2023-10-16T20:21:57.139 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 81569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-10-16T20:21:57.139 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 69729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-10-16T20:21:57.139 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/ 2023-10-16T20:21:57.139 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 78401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-10-16T20:21:57.140 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 176761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-10-16T20:21:57.140 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2023-10-16T20:21:57.140 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 1421497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-10-16T20:21:57.140 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 128569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-10-16T20:21:57.140 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet/ 2023-10-16T20:21:57.141 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 132089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-10-16T20:21:57.141 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 312081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-10-16T20:21:57.141 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/ 2023-10-16T20:21:57.141 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna/ 2023-10-16T20:21:57.141 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 323513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-10-16T20:21:57.141 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/ 2023-10-16T20:21:57.142 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb/ 2023-10-16T20:21:57.142 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 193497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-10-16T20:21:57.142 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2023-10-16T20:21:57.142 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 347417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-10-16T20:21:57.142 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2023-10-16T20:21:57.143 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 1044433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-10-16T20:21:57.143 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2023-10-16T20:21:57.143 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 166193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-10-16T20:21:57.143 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2023-10-16T20:21:57.143 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-10-16T20:21:57.143 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/ 2023-10-16T20:21:57.144 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic/ 2023-10-16T20:21:57.144 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 204345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-10-16T20:21:57.144 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/ 2023-10-16T20:21:57.144 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/ 2023-10-16T20:21:57.144 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 69449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-10-16T20:21:57.145 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 61489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-10-16T20:21:57.145 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 141225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-10-16T20:21:57.145 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 51497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-10-16T20:21:57.145 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 62817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-10-16T20:21:57.145 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-10-16T20:21:57.146 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/ 2023-10-16T20:21:57.146 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 60873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-10-16T20:21:57.146 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 62833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-10-16T20:21:57.146 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 45721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dnet.ko 2023-10-16T20:21:57.146 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ec_bhf.ko 2023-10-16T20:21:57.147 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/ 2023-10-16T20:21:57.147 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet/ 2023-10-16T20:21:57.147 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 281137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-10-16T20:21:57.147 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 49457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ethoc.ko 2023-10-16T20:21:57.147 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 51737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/fealnx.ko 2023-10-16T20:21:57.147 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/fujitsu/ 2023-10-16T20:21:57.148 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 45345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2023-10-16T20:21:57.148 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ 2023-10-16T20:21:57.148 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 90465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e100.ko 2023-10-16T20:21:57.148 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000/ 2023-10-16T20:21:57.148 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 295577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-10-16T20:21:57.149 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e/ 2023-10-16T20:21:57.149 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 584497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-10-16T20:21:57.149 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k/ 2023-10-16T20:21:57.149 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 246209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-10-16T20:21:57.149 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e/ 2023-10-16T20:21:57.149 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 925697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-10-16T20:21:57.150 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf/ 2023-10-16T20:21:57.150 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 302849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-10-16T20:21:57.150 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb/ 2023-10-16T20:21:57.150 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 514681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-10-16T20:21:57.150 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf/ 2023-10-16T20:21:57.151 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 93593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-10-16T20:21:57.151 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe/ 2023-10-16T20:21:57.151 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 734305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-10-16T20:21:57.255 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf/ 2023-10-16T20:21:57.255 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 151769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-10-16T20:21:57.255 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 86921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/jme.ko 2023-10-16T20:21:57.255 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/ 2023-10-16T20:21:57.256 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-10-16T20:21:57.256 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 102289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/skge.ko 2023-10-16T20:21:57.256 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 122649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-10-16T20:21:57.256 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/ 2023-10-16T20:21:57.256 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/ 2023-10-16T20:21:57.257 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 727601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-10-16T20:21:57.257 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 278937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-10-16T20:21:57.257 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/ 2023-10-16T20:21:57.257 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2023-10-16T20:21:57.257 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 1113993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-10-16T20:21:57.258 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel/ 2023-10-16T20:21:57.258 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 53105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel/ks8842.ko 2023-10-16T20:21:57.258 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 40993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2023-10-16T20:21:57.258 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2023-10-16T20:21:57.258 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 95665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-10-16T20:21:57.259 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/ 2023-10-16T20:21:57.259 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge/ 2023-10-16T20:21:57.259 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 108649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-10-16T20:21:57.259 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/ 2023-10-16T20:21:57.259 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 73441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-10-16T20:21:57.259 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 47777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-10-16T20:21:57.260 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion/ 2023-10-16T20:21:57.260 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 150905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-10-16T20:21:57.260 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia/ 2023-10-16T20:21:57.260 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 136721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-10-16T20:21:57.260 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/ 2023-10-16T20:21:57.261 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 50481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-10-16T20:21:57.261 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 43145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-10-16T20:21:57.261 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/ 2023-10-16T20:21:57.261 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen/ 2023-10-16T20:21:57.261 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 227897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-10-16T20:21:57.262 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 87081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-10-16T20:21:57.262 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2023-10-16T20:21:57.262 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 672049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-10-16T20:21:57.262 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc/ 2023-10-16T20:21:57.262 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 43017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-10-16T20:21:57.263 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/ 2023-10-16T20:21:57.263 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 67385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-10-16T20:21:57.263 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 83985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-10-16T20:21:57.263 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/atp.ko 2023-10-16T20:21:57.263 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 189801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-10-16T20:21:57.264 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/samsung/ 2023-10-16T20:21:57.264 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/samsung/sxgbe/ 2023-10-16T20:21:57.264 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 104241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2023-10-16T20:21:57.264 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc/ 2023-10-16T20:21:57.264 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 1053921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-10-16T20:21:57.264 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan/ 2023-10-16T20:21:57.265 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 47097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-10-16T20:21:57.265 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/ 2023-10-16T20:21:57.265 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 54817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis190.ko 2023-10-16T20:21:57.265 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 63337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis900.ko 2023-10-16T20:21:57.265 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/ 2023-10-16T20:21:57.266 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 61601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-10-16T20:21:57.266 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 67193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-10-16T20:21:57.266 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 73409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-10-16T20:21:57.266 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 68177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-10-16T20:21:57.266 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/ 2023-10-16T20:21:57.266 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/ 2023-10-16T20:21:57.267 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2023-10-16T20:21:57.267 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 39905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-10-16T20:21:57.267 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2023-10-16T20:21:57.267 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 486961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-10-16T20:21:57.267 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/ 2023-10-16T20:21:57.268 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 103337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/cassini.ko 2023-10-16T20:21:57.268 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 166161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/niu.ko 2023-10-16T20:21:57.268 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 66329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sungem.ko 2023-10-16T20:21:57.268 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 78801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-10-16T20:21:57.268 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti/ 2023-10-16T20:21:57.269 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 50057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-10-16T20:21:57.269 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti/ 2023-10-16T20:21:57.269 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 86529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti/tlan.ko 2023-10-16T20:21:57.269 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/ 2023-10-16T20:21:57.269 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 80273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-10-16T20:21:57.270 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 83633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-10-16T20:21:57.270 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/ 2023-10-16T20:21:57.270 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 46241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-10-16T20:21:57.270 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-10-16T20:21:57.270 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom/ 2023-10-16T20:21:57.271 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 60057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-10-16T20:21:57.271 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/fddi/ 2023-10-16T20:21:57.271 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 41065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/fddi/defxx.ko 2023-10-16T20:21:57.271 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/fddi/skfp/ 2023-10-16T20:21:57.271 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 216985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/fddi/skfp/skfp.ko 2023-10-16T20:21:57.271 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 77345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/geneve.ko 2023-10-16T20:21:57.272 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv/ 2023-10-16T20:21:57.272 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 235521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-10-16T20:21:57.272 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154/ 2023-10-16T20:21:57.272 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154/fakelb.ko 2023-10-16T20:21:57.272 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ifb.ko 2023-10-16T20:21:57.273 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan/ 2023-10-16T20:21:57.273 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 83769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan/ipvlan.ko 2023-10-16T20:21:57.273 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 62209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvlan.ko 2023-10-16T20:21:57.273 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvtap.ko 2023-10-16T20:21:57.273 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/ 2023-10-16T20:21:57.274 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-10-16T20:21:57.274 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-10-16T20:21:57.274 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-gpio.ko 2023-10-16T20:21:57.274 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-regmap.ko 2023-10-16T20:21:57.275 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio.ko 2023-10-16T20:21:57.275 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mii.ko 2023-10-16T20:21:57.275 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 43953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/netconsole.ko 2023-10-16T20:21:57.275 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/nlmon.ko 2023-10-16T20:21:57.275 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/ 2023-10-16T20:21:57.275 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs-lynx.ko 2023-10-16T20:21:57.276 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 38977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-10-16T20:21:57.276 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/ 2023-10-16T20:21:57.276 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/amd.ko 2023-10-16T20:21:57.276 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 50097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-10-16T20:21:57.276 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 44505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm7xxx.ko 2023-10-16T20:21:57.277 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm87xx.ko 2023-10-16T20:21:57.277 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 42241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/broadcom.ko 2023-10-16T20:21:57.277 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/cicada.ko 2023-10-16T20:21:57.277 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/davicom.ko 2023-10-16T20:21:57.277 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/et1011c.ko 2023-10-16T20:21:57.278 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/icplus.ko 2023-10-16T20:21:57.278 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/lxt.ko 2023-10-16T20:21:57.278 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 82049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/marvell.ko 2023-10-16T20:21:57.278 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 76177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/micrel.ko 2023-10-16T20:21:57.278 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/national.ko 2023-10-16T20:21:57.279 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 97857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/phylink.ko 2023-10-16T20:21:57.279 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/qsemi.ko 2023-10-16T20:21:57.279 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 50065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/realtek.ko 2023-10-16T20:21:57.279 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/smsc.ko 2023-10-16T20:21:57.279 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/ste10Xp.ko 2023-10-16T20:21:57.280 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/vitesse.ko 2023-10-16T20:21:57.280 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/plip/ 2023-10-16T20:21:57.280 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 43569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/plip/plip.ko 2023-10-16T20:21:57.280 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ 2023-10-16T20:21:57.280 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/bsd_comp.ko 2023-10-16T20:21:57.281 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_async.ko 2023-10-16T20:21:57.281 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_deflate.ko 2023-10-16T20:21:57.281 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_mppe.ko 2023-10-16T20:21:57.281 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_synctty.ko 2023-10-16T20:21:57.281 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 42545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppoe.ko 2023-10-16T20:21:57.282 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppox.ko 2023-10-16T20:21:57.282 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 32689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pptp.ko 2023-10-16T20:21:57.282 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/sb1000.ko 2023-10-16T20:21:57.282 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/ 2023-10-16T20:21:57.282 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 42465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/slip.ko 2023-10-16T20:21:57.283 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/sungem_phy.ko 2023-10-16T20:21:57.283 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 64385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/tap.ko 2023-10-16T20:21:57.283 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/ 2023-10-16T20:21:57.283 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 106169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team.ko 2023-10-16T20:21:57.283 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_activebackup.ko 2023-10-16T20:21:57.284 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_broadcast.ko 2023-10-16T20:21:57.284 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-10-16T20:21:57.284 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_random.ko 2023-10-16T20:21:57.284 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-10-16T20:21:57.284 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 87017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/veth.ko 2023-10-16T20:21:57.285 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3/ 2023-10-16T20:21:57.285 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 144297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-10-16T20:21:57.285 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan/ 2023-10-16T20:21:57.285 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 297329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan/vxlan.ko 2023-10-16T20:21:57.285 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback/ 2023-10-16T20:21:57.286 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 144649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback/xen-netback.ko 2023-10-16T20:21:57.286 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb/ 2023-10-16T20:21:57.286 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb/ntb.ko 2023-10-16T20:21:57.286 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/ 2023-10-16T20:21:57.286 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/ 2023-10-16T20:21:57.287 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 264937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme-core.ko 2023-10-16T20:21:57.287 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 102209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme.ko 2023-10-16T20:21:57.287 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/ 2023-10-16T20:21:57.287 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 133841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport.ko 2023-10-16T20:21:57.287 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_cs.ko 2023-10-16T20:21:57.288 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 84465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_pc.ko 2023-10-16T20:21:57.288 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_serial.ko 2023-10-16T20:21:57.288 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/ 2023-10-16T20:21:57.288 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug/ 2023-10-16T20:21:57.288 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-10-16T20:21:57.289 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2023-10-16T20:21:57.289 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2023-10-16T20:21:57.289 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/pci-stub.ko 2023-10-16T20:21:57.289 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 46529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/xen-pcifront.ko 2023-10-16T20:21:57.289 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/ 2023-10-16T20:21:57.289 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/i82092.ko 2023-10-16T20:21:57.290 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 145809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/pcmcia.ko 2023-10-16T20:21:57.290 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 62137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/pcmcia_core.ko 2023-10-16T20:21:57.290 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2023-10-16T20:21:57.290 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/pd6729.ko 2023-10-16T20:21:57.290 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 112889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/yenta_socket.ko 2023-10-16T20:21:57.291 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/phy/ 2023-10-16T20:21:57.291 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/phy/broadcom/ 2023-10-16T20:21:57.291 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2023-10-16T20:21:57.291 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/phy/samsung/ 2023-10-16T20:21:57.291 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2023-10-16T20:21:57.292 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/ 2023-10-16T20:21:57.292 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel/ 2023-10-16T20:21:57.292 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 70153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-10-16T20:21:57.292 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/ 2023-10-16T20:21:57.292 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/chrome/ 2023-10-16T20:21:57.292 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 50889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2023-10-16T20:21:57.293 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/chrome/cros_ec.ko 2023-10-16T20:21:57.293 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2023-10-16T20:21:57.293 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2023-10-16T20:21:57.293 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2023-10-16T20:21:57.293 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2023-10-16T20:21:57.294 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2023-10-16T20:21:57.294 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/ 2023-10-16T20:21:57.294 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 67097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acer-wmi.ko 2023-10-16T20:21:57.294 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 39145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acerhdf.ko 2023-10-16T20:21:57.294 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/apple-gmux.ko 2023-10-16T20:21:57.295 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 59665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-laptop.ko 2023-10-16T20:21:57.295 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-10-16T20:21:57.295 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 127569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-wmi.ko 2023-10-16T20:21:57.295 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/classmate-laptop.ko 2023-10-16T20:21:57.295 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 49121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-10-16T20:21:57.296 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-10-16T20:21:57.296 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 40265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-10-16T20:21:57.296 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-10-16T20:21:57.296 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/hdaps.ko 2023-10-16T20:21:57.296 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/ibm_rtl.ko 2023-10-16T20:21:57.297 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel/ 2023-10-16T20:21:57.297 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-10-16T20:21:57.297 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2023-10-16T20:21:57.297 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 43497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel_ips.ko 2023-10-16T20:21:57.297 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/msi-wmi.ko 2023-10-16T20:21:57.297 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10377 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/mxm-wmi.ko 2023-10-16T20:21:57.298 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-10-16T20:21:57.298 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 42329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-laptop.ko 2023-10-16T20:21:57.298 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-q10.ko 2023-10-16T20:21:57.298 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/topstar-laptop.ko 2023-10-16T20:21:57.298 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-10-16T20:21:57.299 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_haps.ko 2023-10-16T20:21:57.299 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi-bmof.ko 2023-10-16T20:21:57.299 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 57569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi.ko 2023-10-16T20:21:57.299 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/ 2023-10-16T20:21:57.299 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/ 2023-10-16T20:21:57.299 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/88pm860x_battery.ko 2023-10-16T20:21:57.300 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/88pm860x_charger.ko 2023-10-16T20:21:57.300 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 55057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/bq2415x_charger.ko 2023-10-16T20:21:57.300 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 54081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/bq24190_charger.ko 2023-10-16T20:21:57.300 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/bq24735-charger.ko 2023-10-16T20:21:57.300 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/cros_peripheral_charger.ko 2023-10-16T20:21:57.301 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/da9030_battery.ko 2023-10-16T20:21:57.301 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/da9052-battery.ko 2023-10-16T20:21:57.301 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29281 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/ds2760_battery.ko 2023-10-16T20:21:57.301 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/ds2780_battery.ko 2023-10-16T20:21:57.301 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/ds2781_battery.ko 2023-10-16T20:21:57.302 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/ds2782_battery.ko 2023-10-16T20:21:57.302 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/gpio-charger.ko 2023-10-16T20:21:57.302 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/isp1704_charger.ko 2023-10-16T20:21:57.302 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/lp8727_charger.ko 2023-10-16T20:21:57.302 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/max14577_charger.ko 2023-10-16T20:21:57.303 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/max17040_battery.ko 2023-10-16T20:21:57.303 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/max17042_battery.ko 2023-10-16T20:21:57.303 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/max8903_charger.ko 2023-10-16T20:21:57.303 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/max8925_power.ko 2023-10-16T20:21:57.303 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/pcf50633-charger.ko 2023-10-16T20:21:57.304 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 38009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/sbs-battery.ko 2023-10-16T20:21:57.304 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/test_power.ko 2023-10-16T20:21:57.304 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/tps65090-charger.ko 2023-10-16T20:21:57.304 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/wm831x_backup.ko 2023-10-16T20:21:57.304 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/wm831x_power.ko 2023-10-16T20:21:57.304 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/wm8350_power.ko 2023-10-16T20:21:57.305 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/ 2023-10-16T20:21:57.305 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 57729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_common.ko 2023-10-16T20:21:57.305 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_msr.ko 2023-10-16T20:21:57.305 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/ 2023-10-16T20:21:57.305 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/ 2023-10-16T20:21:57.306 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-gpio.ko 2023-10-16T20:21:57.306 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-ldisc.ko 2023-10-16T20:21:57.306 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps_parport.ko 2023-10-16T20:21:57.306 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 41873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/pps_core.ko 2023-10-16T20:21:57.306 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ 2023-10-16T20:21:57.307 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 66385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp.ko 2023-10-16T20:21:57.307 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp_kvm.ko 2023-10-16T20:21:57.307 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/ 2023-10-16T20:21:57.307 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-88pm80x.ko 2023-10-16T20:21:57.307 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-88pm860x.ko 2023-10-16T20:21:57.307 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-bq32k.ko 2023-10-16T20:21:57.308 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-da9052.ko 2023-10-16T20:21:57.308 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-da9055.ko 2023-10-16T20:21:57.308 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-da9063.ko 2023-10-16T20:21:57.308 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1286.ko 2023-10-16T20:21:57.308 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 68577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1307.ko 2023-10-16T20:21:57.309 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1374.ko 2023-10-16T20:21:57.309 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1511.ko 2023-10-16T20:21:57.309 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1553.ko 2023-10-16T20:21:57.309 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1672.ko 2023-10-16T20:21:57.309 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1742.ko 2023-10-16T20:21:57.310 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds2404.ko 2023-10-16T20:21:57.310 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23705 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds3232.ko 2023-10-16T20:21:57.310 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-em3027.ko 2023-10-16T20:21:57.310 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-fm3130.ko 2023-10-16T20:21:57.311 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl12022.ko 2023-10-16T20:21:57.311 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl1208.ko 2023-10-16T20:21:57.311 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-lp8788.ko 2023-10-16T20:21:57.311 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 38097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m41t80.ko 2023-10-16T20:21:57.311 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t35.ko 2023-10-16T20:21:57.311 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t59.ko 2023-10-16T20:21:57.312 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t86.ko 2023-10-16T20:21:57.312 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-max6900.ko 2023-10-16T20:21:57.312 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-max8907.ko 2023-10-16T20:21:57.312 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-max8925.ko 2023-10-16T20:21:57.312 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-max8997.ko 2023-10-16T20:21:57.313 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-max8998.ko 2023-10-16T20:21:57.313 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-mc13xxx.ko 2023-10-16T20:21:57.313 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-msm6242.ko 2023-10-16T20:21:57.313 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-palmas.ko 2023-10-16T20:21:57.313 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 46633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf2127.ko 2023-10-16T20:21:57.313 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf50633.ko 2023-10-16T20:21:57.314 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf85063.ko 2023-10-16T20:21:57.314 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8523.ko 2023-10-16T20:21:57.314 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8563.ko 2023-10-16T20:21:57.314 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8583.ko 2023-10-16T20:21:57.315 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rc5t583.ko 2023-10-16T20:21:57.315 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rp5c01.ko 2023-10-16T20:21:57.315 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rs5c372.ko 2023-10-16T20:21:57.315 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-10-16T20:21:57.315 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8025.ko 2023-10-16T20:21:57.315 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8581.ko 2023-10-16T20:21:57.316 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-s35390a.ko 2023-10-16T20:21:57.316 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-10-16T20:21:57.316 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-tps6586x.ko 2023-10-16T20:21:57.316 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-tps65910.ko 2023-10-16T20:21:57.316 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-wm831x.ko 2023-10-16T20:21:57.317 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-wm8350.ko 2023-10-16T20:21:57.317 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-x1205.ko 2023-10-16T20:21:57.317 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ 2023-10-16T20:21:57.317 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 77617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-9xxx.ko 2023-10-16T20:21:57.317 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 57665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-sas.ko 2023-10-16T20:21:57.318 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 55169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-xxxx.ko 2023-10-16T20:21:57.318 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 135185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/BusLogic.ko 2023-10-16T20:21:57.318 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/a100u2w.ko 2023-10-16T20:21:57.318 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid/ 2023-10-16T20:21:57.318 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 272305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid/aacraid.ko 2023-10-16T20:21:57.319 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 100617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/advansys.ko 2023-10-16T20:21:57.319 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/ 2023-10-16T20:21:57.319 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 356265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-10-16T20:21:57.319 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 295801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-10-16T20:21:57.319 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic94xx/ 2023-10-16T20:21:57.320 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 189185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic94xx/aic94xx.ko 2023-10-16T20:21:57.320 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/am53c974.ko 2023-10-16T20:21:57.320 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr/ 2023-10-16T20:21:57.320 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 106881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-10-16T20:21:57.320 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 38185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/atp870u.ko 2023-10-16T20:21:57.320 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi/ 2023-10-16T20:21:57.321 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 246985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-10-16T20:21:57.321 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa/ 2023-10-16T20:21:57.321 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 1171945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa/bfa.ko 2023-10-16T20:21:57.321 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc/ 2023-10-16T20:21:57.321 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 237273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-10-16T20:21:57.322 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i/ 2023-10-16T20:21:57.322 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 115369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-10-16T20:21:57.322 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 40585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ch.ko 2023-10-16T20:21:57.322 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor/ 2023-10-16T20:21:57.322 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 308921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor/csiostor.ko 2023-10-16T20:21:57.323 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/ 2023-10-16T20:21:57.323 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i/ 2023-10-16T20:21:57.323 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 76145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-10-16T20:21:57.323 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i/ 2023-10-16T20:21:57.323 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 101449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-10-16T20:21:57.323 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 146689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-10-16T20:21:57.324 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 75553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dc395x.ko 2023-10-16T20:21:57.324 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 38137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dmx3191d.ko 2023-10-16T20:21:57.324 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r/ 2023-10-16T20:21:57.324 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 212425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r/esas2r.ko 2023-10-16T20:21:57.324 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 60505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esp_scsi.ko 2023-10-16T20:21:57.325 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/ 2023-10-16T20:21:57.325 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 78209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/fcoe.ko 2023-10-16T20:21:57.325 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 158505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-10-16T20:21:57.325 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fdomain.ko 2023-10-16T20:21:57.325 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic/ 2023-10-16T20:21:57.325 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 263489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic/fnic.ko 2023-10-16T20:21:57.326 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 201817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hpsa.ko 2023-10-16T20:21:57.326 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 51209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hptiop.ko 2023-10-16T20:21:57.326 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 49041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hv_storvsc.ko 2023-10-16T20:21:57.326 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/imm.ko 2023-10-16T20:21:57.326 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 44057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/initio.ko 2023-10-16T20:21:57.327 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 266537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ipr.ko 2023-10-16T20:21:57.327 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 101881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ips.ko 2023-10-16T20:21:57.327 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci/ 2023-10-16T20:21:57.327 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 317153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci/isci.ko 2023-10-16T20:21:57.327 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-10-16T20:21:57.328 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 52961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_tcp.ko 2023-10-16T20:21:57.328 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc/ 2023-10-16T20:21:57.328 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 314865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc/libfc.ko 2023-10-16T20:21:57.328 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 145585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi.ko 2023-10-16T20:21:57.328 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 53521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi_tcp.ko 2023-10-16T20:21:57.329 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas/ 2023-10-16T20:21:57.329 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 216665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas/libsas.ko 2023-10-16T20:21:57.329 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc/ 2023-10-16T20:21:57.329 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 2197249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc/lpfc.ko 2023-10-16T20:21:57.329 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/ 2023-10-16T20:21:57.330 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 73569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-10-16T20:21:57.330 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-10-16T20:21:57.330 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 310617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-10-16T20:21:57.330 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 84721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid.ko 2023-10-16T20:21:57.330 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas/ 2023-10-16T20:21:57.331 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 659777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-10-16T20:21:57.331 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas/ 2023-10-16T20:21:57.331 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 119561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas/mvsas.ko 2023-10-16T20:21:57.331 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 78129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvumi.ko 2023-10-16T20:21:57.331 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pcmcia/ 2023-10-16T20:21:57.332 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 81625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2023-10-16T20:21:57.332 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2023-10-16T20:21:57.332 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2023-10-16T20:21:57.332 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2023-10-16T20:21:57.332 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001/ 2023-10-16T20:21:57.332 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 364105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-10-16T20:21:57.333 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 123617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pmcraid.ko 2023-10-16T20:21:57.333 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ppa.ko 2023-10-16T20:21:57.333 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 55361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla1280.ko 2023-10-16T20:21:57.333 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/ 2023-10-16T20:21:57.333 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 1876625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-10-16T20:21:57.334 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 85289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-10-16T20:21:57.334 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx/ 2023-10-16T20:21:57.334 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 559033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-10-16T20:21:57.334 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qlogicfas408.ko 2023-10-16T20:21:57.334 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/raid_class.ko 2023-10-16T20:21:57.335 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 238825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_debug.ko 2023-10-16T20:21:57.335 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 173937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_fc.ko 2023-10-16T20:21:57.335 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 279313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-10-16T20:21:57.335 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 97729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_sas.ko 2023-10-16T20:21:57.335 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 73881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_spi.ko 2023-10-16T20:21:57.335 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 44993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_srp.ko 2023-10-16T20:21:57.336 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ses.ko 2023-10-16T20:21:57.336 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 115889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/st.ko 2023-10-16T20:21:57.336 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 46825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/stex.ko 2023-10-16T20:21:57.336 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2/ 2023-10-16T20:21:57.336 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 162121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-10-16T20:21:57.337 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 42297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/virtio_scsi.ko 2023-10-16T20:21:57.337 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 50321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/vmw_pvscsi.ko 2023-10-16T20:21:57.337 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 40025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/wd719x.ko 2023-10-16T20:21:57.337 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 50025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/xen-scsifront.ko 2023-10-16T20:21:57.337 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb/ 2023-10-16T20:21:57.337 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 129441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb/ssb.ko 2023-10-16T20:21:57.338 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/ 2023-10-16T20:21:57.338 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi/ 2023-10-16T20:21:57.338 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 736601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-10-16T20:21:57.338 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback/ 2023-10-16T20:21:57.338 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 49985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback/tcm_loop.ko 2023-10-16T20:21:57.339 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 39713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_file.ko 2023-10-16T20:21:57.339 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 43025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_iblock.ko 2023-10-16T20:21:57.339 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 930321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_mod.ko 2023-10-16T20:21:57.339 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 44305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_pscsi.ko 2023-10-16T20:21:57.339 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 122521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_user.ko 2023-10-16T20:21:57.340 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc/ 2023-10-16T20:21:57.340 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 75593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-10-16T20:21:57.340 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/ 2023-10-16T20:21:57.340 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/ 2023-10-16T20:21:57.340 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/ 2023-10-16T20:21:57.340 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-10-16T20:21:57.341 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-10-16T20:21:57.341 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-10-16T20:21:57.341 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-10-16T20:21:57.341 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-10-16T20:21:57.341 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-10-16T20:21:57.342 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-10-16T20:21:57.342 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20281 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-10-16T20:21:57.342 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-10-16T20:21:57.342 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-10-16T20:21:57.342 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-10-16T20:21:57.343 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 37065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-10-16T20:21:57.343 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_powerclamp.ko 2023-10-16T20:21:57.343 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-10-16T20:21:57.343 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-10-16T20:21:57.343 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-10-16T20:21:57.343 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ 2023-10-16T20:21:57.344 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless/ 2023-10-16T20:21:57.344 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 78425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-10-16T20:21:57.344 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 57865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/moxa.ko 2023-10-16T20:21:57.344 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 50953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/mxser.ko 2023-10-16T20:21:57.344 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 108601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_gsm.ko 2023-10-16T20:21:57.345 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_hdlc.ko 2023-10-16T20:21:57.345 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 67929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/nozomi.ko 2023-10-16T20:21:57.345 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/ 2023-10-16T20:21:57.345 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250/ 2023-10-16T20:21:57.345 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250/8250_dw.ko 2023-10-16T20:21:57.345 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 59497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-10-16T20:21:57.346 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/altera_jtaguart.ko 2023-10-16T20:21:57.346 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/altera_uart.ko 2023-10-16T20:21:57.346 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/arc_uart.ko 2023-10-16T20:21:57.346 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 72425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/fsl_lpuart.ko 2023-10-16T20:21:57.346 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm/ 2023-10-16T20:21:57.347 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 103073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm/jsm.ko 2023-10-16T20:21:57.347 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/rp2.ko 2023-10-16T20:21:57.347 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 52817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/sc16is7xx.ko 2023-10-16T20:21:57.347 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/ 2023-10-16T20:21:57.347 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core/ 2023-10-16T20:21:57.348 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 531265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core/ufshcd-core.ko 2023-10-16T20:21:57.348 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host/ 2023-10-16T20:21:57.348 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-10-16T20:21:57.348 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2023-10-16T20:21:57.348 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/ 2023-10-16T20:21:57.348 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 44513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio.ko 2023-10-16T20:21:57.349 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_aec.ko 2023-10-16T20:21:57.349 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11281 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_cif.ko 2023-10-16T20:21:57.349 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_dmem_genirq.ko 2023-10-16T20:21:57.349 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_mf624.ko 2023-10-16T20:21:57.349 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_netx.ko 2023-10-16T20:21:57.350 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_pci_generic.ko 2023-10-16T20:21:57.350 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_pdrv_genirq.ko 2023-10-16T20:21:57.350 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_sercos3.ko 2023-10-16T20:21:57.350 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/ 2023-10-16T20:21:57.350 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/c67x00/ 2023-10-16T20:21:57.351 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 73297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/c67x00/c67x00.ko 2023-10-16T20:21:57.351 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/chipidea/ 2023-10-16T20:21:57.351 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 144377 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/chipidea/ci_hdrc.ko 2023-10-16T20:21:57.351 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2023-10-16T20:21:57.351 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2023-10-16T20:21:57.351 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2023-10-16T20:21:57.352 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/ 2023-10-16T20:21:57.352 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 87089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-acm.ko 2023-10-16T20:21:57.352 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 51409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-wdm.ko 2023-10-16T20:21:57.352 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 47369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usblp.ko 2023-10-16T20:21:57.352 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 89857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usbtmc.ko 2023-10-16T20:21:57.353 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/common/ 2023-10-16T20:21:57.353 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/common/ulpi.ko 2023-10-16T20:21:57.353 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/dwc2/ 2023-10-16T20:21:57.353 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/dwc2/dwc2_pci.ko 2023-10-16T20:21:57.353 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/dwc3/ 2023-10-16T20:21:57.353 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/dwc3/dwc3-haps.ko 2023-10-16T20:21:57.354 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/dwc3/dwc3-pci.ko 2023-10-16T20:21:57.354 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 444553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/dwc3/dwc3.ko 2023-10-16T20:21:57.354 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/ 2023-10-16T20:21:57.354 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/ 2023-10-16T20:21:57.354 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 60177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/u_ether.ko 2023-10-16T20:21:57.355 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 51881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/u_serial.ko 2023-10-16T20:21:57.355 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2023-10-16T20:21:57.355 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 42449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2023-10-16T20:21:57.355 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2023-10-16T20:21:57.355 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2023-10-16T20:21:57.355 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 92489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2023-10-16T20:21:57.356 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 51049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2023-10-16T20:21:57.356 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 122081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2023-10-16T20:21:57.356 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 57617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2023-10-16T20:21:57.356 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2023-10-16T20:21:57.356 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2023-10-16T20:21:57.357 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 57729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2023-10-16T20:21:57.357 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 85609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2023-10-16T20:21:57.357 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2023-10-16T20:21:57.357 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 58113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2023-10-16T20:21:57.357 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 67337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2023-10-16T20:21:57.358 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/ 2023-10-16T20:21:57.358 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2023-10-16T20:21:57.358 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2023-10-16T20:21:57.358 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2023-10-16T20:21:57.358 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_ether.ko 2023-10-16T20:21:57.359 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2023-10-16T20:21:57.359 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_hid.ko 2023-10-16T20:21:57.359 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2023-10-16T20:21:57.359 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2023-10-16T20:21:57.359 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 32073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2023-10-16T20:21:57.360 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_printer.ko 2023-10-16T20:21:57.360 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_serial.ko 2023-10-16T20:21:57.360 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_zero.ko 2023-10-16T20:21:57.360 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 62185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2023-10-16T20:21:57.360 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2023-10-16T20:21:57.361 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 161225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/libcomposite.ko 2023-10-16T20:21:57.361 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/ 2023-10-16T20:21:57.361 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17705 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2023-10-16T20:21:57.361 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/bdc/ 2023-10-16T20:21:57.361 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 157681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2023-10-16T20:21:57.361 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/goku_udc.ko 2023-10-16T20:21:57.362 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 52721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/gr_udc.ko 2023-10-16T20:21:57.362 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 69705 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2023-10-16T20:21:57.362 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 56033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/mv_udc.ko 2023-10-16T20:21:57.362 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 76057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/net2272.ko 2023-10-16T20:21:57.362 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 75073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/net2280.ko 2023-10-16T20:21:57.363 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 73169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/pch_udc.ko 2023-10-16T20:21:57.363 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 65809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2023-10-16T20:21:57.363 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 66097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2023-10-16T20:21:57.363 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 81505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2023-10-16T20:21:57.363 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 156073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/udc-core.ko 2023-10-16T20:21:57.364 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/ 2023-10-16T20:21:57.364 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/bcma-hcd.ko 2023-10-16T20:21:57.364 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 80113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/isp116x-hcd.ko 2023-10-16T20:21:57.364 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 87353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/oxu210hp-hcd.ko 2023-10-16T20:21:57.364 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 78873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/r8a66597-hcd.ko 2023-10-16T20:21:57.365 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 51329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/sl811-hcd.ko 2023-10-16T20:21:57.365 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/sl811_cs.ko 2023-10-16T20:21:57.365 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/ssb-hcd.ko 2023-10-16T20:21:57.365 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/xhci-plat-hcd.ko 2023-10-16T20:21:57.365 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/ 2023-10-16T20:21:57.365 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/mdc800.ko 2023-10-16T20:21:57.366 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/microtek.ko 2023-10-16T20:21:57.366 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ 2023-10-16T20:21:57.366 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 41721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/adutux.ko 2023-10-16T20:21:57.366 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/appledisplay.ko 2023-10-16T20:21:57.366 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/cypress_cy7c63.ko 2023-10-16T20:21:57.367 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/cytherm.ko 2023-10-16T20:21:57.367 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ehset.ko 2023-10-16T20:21:57.367 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi26.ko 2023-10-16T20:21:57.367 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi62.ko 2023-10-16T20:21:57.367 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ezusb.ko 2023-10-16T20:21:57.368 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/idmouse.ko 2023-10-16T20:21:57.368 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/iowarrior.ko 2023-10-16T20:21:57.368 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/isight_firmware.ko 2023-10-16T20:21:57.368 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ldusb.ko 2023-10-16T20:21:57.368 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/legousbtower.ko 2023-10-16T20:21:57.369 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/lvstest.ko 2023-10-16T20:21:57.369 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga/ 2023-10-16T20:21:57.369 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 64841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-10-16T20:21:57.369 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/trancevibrator.ko 2023-10-16T20:21:57.370 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usb3503.ko 2023-10-16T20:21:57.370 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usblcd.ko 2023-10-16T20:21:57.370 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usbsevseg.ko 2023-10-16T20:21:57.370 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 70873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usbtest.ko 2023-10-16T20:21:57.370 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 41969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/uss720.ko 2023-10-16T20:21:57.371 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/yurex.ko 2023-10-16T20:21:57.371 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/mon/ 2023-10-16T20:21:57.371 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 72953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/mon/usbmon.ko 2023-10-16T20:21:57.371 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/musb/ 2023-10-16T20:21:57.371 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 301545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/musb/musb_hdrc.ko 2023-10-16T20:21:57.372 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy/ 2023-10-16T20:21:57.372 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy/phy-generic.ko 2023-10-16T20:21:57.372 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2023-10-16T20:21:57.372 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy/phy-isp1301.ko 2023-10-16T20:21:57.372 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy/phy-tahvo.ko 2023-10-16T20:21:57.372 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ 2023-10-16T20:21:57.373 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/aircable.ko 2023-10-16T20:21:57.373 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ark3116.ko 2023-10-16T20:21:57.373 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/belkin_sa.ko 2023-10-16T20:21:57.373 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 38105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ch341.ko 2023-10-16T20:21:57.373 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 70257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cp210x.ko 2023-10-16T20:21:57.374 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cyberjack.ko 2023-10-16T20:21:57.374 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 58217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cypress_m8.ko 2023-10-16T20:21:57.374 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 49817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/digi_acceleport.ko 2023-10-16T20:21:57.374 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/empeg.ko 2023-10-16T20:21:57.374 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 44337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/f81232.ko 2023-10-16T20:21:57.375 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 175137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ftdi_sio.ko 2023-10-16T20:21:57.375 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 53113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/garmin_gps.ko 2023-10-16T20:21:57.375 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 137081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_edgeport.ko 2023-10-16T20:21:57.375 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 124833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_ti.ko 2023-10-16T20:21:57.375 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 67769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipaq.ko 2023-10-16T20:21:57.375 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipw.ko 2023-10-16T20:21:57.376 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ir-usb.ko 2023-10-16T20:21:57.377 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 58233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-10-16T20:21:57.377 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 107521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan.ko 2023-10-16T20:21:57.378 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan_pda.ko 2023-10-16T20:21:57.378 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kl5kusb105.ko 2023-10-16T20:21:57.378 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kobil_sct.ko 2023-10-16T20:21:57.378 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 32953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mct_u232.ko 2023-10-16T20:21:57.378 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/metro-usb.ko 2023-10-16T20:21:57.379 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 66449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7720.ko 2023-10-16T20:21:57.379 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 78217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7840.ko 2023-10-16T20:21:57.379 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 56337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mxuport.ko 2023-10-16T20:21:57.379 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/navman.ko 2023-10-16T20:21:57.379 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/omninet.ko 2023-10-16T20:21:57.380 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/opticon.ko 2023-10-16T20:21:57.380 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 190633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/option.ko 2023-10-16T20:21:57.380 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 39073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/oti6858.ko 2023-10-16T20:21:57.380 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 54353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/pl2303.ko 2023-10-16T20:21:57.380 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcaux.ko 2023-10-16T20:21:57.380 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 37825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcserial.ko 2023-10-16T20:21:57.381 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/quatech2.ko 2023-10-16T20:21:57.381 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/safe_serial.ko 2023-10-16T20:21:57.381 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 51625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/sierra.ko 2023-10-16T20:21:57.381 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/spcp8x5.ko 2023-10-16T20:21:57.381 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ssu100.ko 2023-10-16T20:21:57.382 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/symbolserial.ko 2023-10-16T20:21:57.382 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 61489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-10-16T20:21:57.382 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-10-16T20:21:57.382 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_debug.ko 2023-10-16T20:21:57.382 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_wwan.ko 2023-10-16T20:21:57.382 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 125609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usbserial.ko 2023-10-16T20:21:57.383 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/visor.ko 2023-10-16T20:21:57.383 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 37409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/whiteheat.ko 2023-10-16T20:21:57.383 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/wishbone-serial.ko 2023-10-16T20:21:57.383 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/xsens_mt.ko 2023-10-16T20:21:57.383 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ 2023-10-16T20:21:57.384 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 52913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/uas.ko 2023-10-16T20:21:57.384 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-alauda.ko 2023-10-16T20:21:57.384 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-cypress.ko 2023-10-16T20:21:57.384 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-datafab.ko 2023-10-16T20:21:57.384 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-10-16T20:21:57.385 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-freecom.ko 2023-10-16T20:21:57.385 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-isd200.ko 2023-10-16T20:21:57.385 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-10-16T20:21:57.385 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-karma.ko 2023-10-16T20:21:57.385 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-onetouch.ko 2023-10-16T20:21:57.385 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-realtek.ko 2023-10-16T20:21:57.386 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr09.ko 2023-10-16T20:21:57.386 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr55.ko 2023-10-16T20:21:57.386 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-usbat.ko 2023-10-16T20:21:57.386 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 179969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/usb-storage.ko 2023-10-16T20:21:57.386 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/ 2023-10-16T20:21:57.387 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 67481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-core.ko 2023-10-16T20:21:57.387 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 72177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-host.ko 2023-10-16T20:21:57.387 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 107001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-10-16T20:21:57.387 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/ 2023-10-16T20:21:57.387 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/ 2023-10-16T20:21:57.388 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 154297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-10-16T20:21:57.388 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci.ko 2023-10-16T20:21:57.388 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 108737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio.ko 2023-10-16T20:21:57.388 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 77609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-10-16T20:21:57.388 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/ 2023-10-16T20:21:57.388 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 113873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost.ko 2023-10-16T20:21:57.389 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_iotlb.ko 2023-10-16T20:21:57.389 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 58089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_net.ko 2023-10-16T20:21:57.389 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 89249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_scsi.ko 2023-10-16T20:21:57.389 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/ 2023-10-16T20:21:57.389 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/ 2023-10-16T20:21:57.390 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/88pm860x_bl.ko 2023-10-16T20:21:57.390 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/aat2870_bl.ko 2023-10-16T20:21:57.390 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/adp5520_bl.ko 2023-10-16T20:21:57.390 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 32609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/adp8860_bl.ko 2023-10-16T20:21:57.390 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 39993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/adp8870_bl.ko 2023-10-16T20:21:57.391 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/apple_bl.ko 2023-10-16T20:21:57.391 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/as3711_bl.ko 2023-10-16T20:21:57.391 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/bd6107.ko 2023-10-16T20:21:57.391 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/da903x_bl.ko 2023-10-16T20:21:57.391 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/da9052_bl.ko 2023-10-16T20:21:57.392 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/gpio_backlight.ko 2023-10-16T20:21:57.392 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/kb3886_bl.ko 2023-10-16T20:21:57.392 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/lcd.ko 2023-10-16T20:21:57.392 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/lm3533_bl.ko 2023-10-16T20:21:57.392 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/lm3639_bl.ko 2023-10-16T20:21:57.392 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/lv5207lp.ko 2023-10-16T20:21:57.393 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/max8925_bl.ko 2023-10-16T20:21:57.393 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/pandora_bl.ko 2023-10-16T20:21:57.393 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/pcf50633-backlight.ko 2023-10-16T20:21:57.393 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/platform_lcd.ko 2023-10-16T20:21:57.393 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/wm831x_bl.ko 2023-10-16T20:21:57.394 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/ 2023-10-16T20:21:57.394 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ 2023-10-16T20:21:57.394 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2482.ko 2023-10-16T20:21:57.394 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 38985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2490.ko 2023-10-16T20:21:57.394 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/matrox_w1.ko 2023-10-16T20:21:57.395 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/w1-gpio.ko 2023-10-16T20:21:57.395 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/ 2023-10-16T20:21:57.395 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-10-16T20:21:57.395 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-10-16T20:21:57.395 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-10-16T20:21:57.396 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-10-16T20:21:57.396 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-10-16T20:21:57.396 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-10-16T20:21:57.396 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-10-16T20:21:57.396 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-10-16T20:21:57.396 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-10-16T20:21:57.397 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_smem.ko 2023-10-16T20:21:57.397 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 56425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_therm.ko 2023-10-16T20:21:57.397 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 102673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/wire.ko 2023-10-16T20:21:57.397 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ 2023-10-16T20:21:57.397 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/acquirewdt.ko 2023-10-16T20:21:57.398 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/advantechwdt.ko 2023-10-16T20:21:57.398 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim1535_wdt.ko 2023-10-16T20:21:57.398 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim7101_wdt.ko 2023-10-16T20:21:57.398 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/cpu5wdt.ko 2023-10-16T20:21:57.398 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/da9052_wdt.ko 2023-10-16T20:21:57.398 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/da9055_wdt.ko 2023-10-16T20:21:57.399 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/da9063_wdt.ko 2023-10-16T20:21:57.399 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/dw_wdt.ko 2023-10-16T20:21:57.399 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/eurotechwdt.ko 2023-10-16T20:21:57.399 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/f71808e_wdt.ko 2023-10-16T20:21:57.399 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/hpwdt.ko 2023-10-16T20:21:57.400 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/i6300esb.ko 2023-10-16T20:21:57.400 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-10-16T20:21:57.400 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_wdt.ko 2023-10-16T20:21:57.400 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ib700wdt.ko 2023-10-16T20:21:57.400 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ibmasr.ko 2023-10-16T20:21:57.401 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-10-16T20:21:57.401 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it8712f_wdt.ko 2023-10-16T20:21:57.401 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it87_wdt.ko 2023-10-16T20:21:57.401 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/kempld_wdt.ko 2023-10-16T20:21:57.401 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/machzwd.ko 2023-10-16T20:21:57.401 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/mena21_wdt.ko 2023-10-16T20:21:57.402 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/menf21bmc_wdt.ko 2023-10-16T20:21:57.402 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/nv_tco.ko 2023-10-16T20:21:57.402 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/of_xilinx_wdt.ko 2023-10-16T20:21:57.402 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pc87413_wdt.ko 2023-10-16T20:21:57.402 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 38657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_pci.ko 2023-10-16T20:21:57.403 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_usb.ko 2023-10-16T20:21:57.403 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/retu_wdt.ko 2023-10-16T20:21:57.403 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sbc60xxwdt.ko 2023-10-16T20:21:57.403 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sbc_epx_c3.ko 2023-10-16T20:21:57.403 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-10-16T20:21:57.403 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sc1200wdt.ko 2023-10-16T20:21:57.404 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sch311x_wdt.ko 2023-10-16T20:21:57.404 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/smsc37b787_wdt.ko 2023-10-16T20:21:57.404 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/softdog.ko 2023-10-16T20:21:57.404 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sp5100_tco.ko 2023-10-16T20:21:57.404 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/twl4030_wdt.ko 2023-10-16T20:21:57.405 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/via_wdt.ko 2023-10-16T20:21:57.405 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-10-16T20:21:57.405 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83877f_wdt.ko 2023-10-16T20:21:57.405 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83977f_wdt.ko 2023-10-16T20:21:57.405 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/wafer5823wdt.ko 2023-10-16T20:21:57.405 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/wdt_pci.ko 2023-10-16T20:21:57.406 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/wm831x_wdt.ko 2023-10-16T20:21:57.406 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/wm8350_wdt.ko 2023-10-16T20:21:57.406 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/xen_wdt.ko 2023-10-16T20:21:57.406 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/ 2023-10-16T20:21:57.406 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-evtchn.ko 2023-10-16T20:21:57.407 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntalloc.ko 2023-10-16T20:21:57.407 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 47865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntdev.ko 2023-10-16T20:21:57.407 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback/ 2023-10-16T20:21:57.407 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 176441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-10-16T20:21:57.407 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 40121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-privcmd.ko 2023-10-16T20:21:57.408 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 71249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-scsiback.ko 2023-10-16T20:21:57.408 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs/ 2023-10-16T20:21:57.408 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs/xenfs.ko 2023-10-16T20:21:57.408 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ 2023-10-16T20:21:57.408 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p/ 2023-10-16T20:21:57.408 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 166425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p/9p.ko 2023-10-16T20:21:57.409 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/adfs/ 2023-10-16T20:21:57.409 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 61073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/adfs/adfs.ko 2023-10-16T20:21:57.409 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs/ 2023-10-16T20:21:57.409 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 163593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs/affs.ko 2023-10-16T20:21:57.409 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/afs/ 2023-10-16T20:21:57.410 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 932817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/afs/kafs.ko 2023-10-16T20:21:57.410 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/autofs/ 2023-10-16T20:21:57.410 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 101057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/autofs/autofs4.ko 2023-10-16T20:21:57.410 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs/ 2023-10-16T20:21:57.410 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 64545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs/befs.ko 2023-10-16T20:21:57.411 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/bfs/ 2023-10-16T20:21:57.411 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 41257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/bfs/bfs.ko 2023-10-16T20:21:57.411 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/binfmt_misc.ko 2023-10-16T20:21:57.411 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs/ 2023-10-16T20:21:57.411 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 3402585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs/btrfs.ko 2023-10-16T20:21:57.411 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles/ 2023-10-16T20:21:57.412 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 306073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles/cachefiles.ko 2023-10-16T20:21:57.412 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph/ 2023-10-16T20:21:57.412 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 1323873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph/ceph.ko 2023-10-16T20:21:57.412 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda/ 2023-10-16T20:21:57.412 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 100801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda/coda.ko 2023-10-16T20:21:57.413 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/configfs/ 2023-10-16T20:21:57.413 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 108393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/configfs/configfs.ko 2023-10-16T20:21:57.413 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs/ 2023-10-16T20:21:57.413 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 29625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs/cramfs.ko 2023-10-16T20:21:57.413 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm/ 2023-10-16T20:21:57.413 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 642577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm/dlm.ko 2023-10-16T20:21:57.414 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/efs/ 2023-10-16T20:21:57.414 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/efs/efs.ko 2023-10-16T20:21:57.414 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs/ 2023-10-16T20:21:57.414 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 1505025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs/f2fs.ko 2023-10-16T20:21:57.414 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/ 2023-10-16T20:21:57.415 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/msdos.ko 2023-10-16T20:21:57.415 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/freevxfs/ 2023-10-16T20:21:57.415 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 42545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/freevxfs/freevxfs.ko 2023-10-16T20:21:57.415 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache/ 2023-10-16T20:21:57.415 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 226481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache/fscache.ko 2023-10-16T20:21:57.416 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/ 2023-10-16T20:21:57.416 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/cuse.ko 2023-10-16T20:21:57.416 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2/ 2023-10-16T20:21:57.416 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 1231529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2/gfs2.ko 2023-10-16T20:21:57.416 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs/ 2023-10-16T20:21:57.416 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 141209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs/hfs.ko 2023-10-16T20:21:57.417 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus/ 2023-10-16T20:21:57.417 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 222817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus/hfsplus.ko 2023-10-16T20:21:57.417 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hpfs/ 2023-10-16T20:21:57.417 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 192329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hpfs/hpfs.ko 2023-10-16T20:21:57.417 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs/ 2023-10-16T20:21:57.418 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 74065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs/isofs.ko 2023-10-16T20:21:57.418 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs/ 2023-10-16T20:21:57.418 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 419497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs/jfs.ko 2023-10-16T20:21:57.418 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd/ 2023-10-16T20:21:57.418 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 280313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd/lockd.ko 2023-10-16T20:21:57.419 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix/ 2023-10-16T20:21:57.419 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 89193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix/minix.ko 2023-10-16T20:21:57.419 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs/ 2023-10-16T20:21:57.419 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 124649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs/netfs.ko 2023-10-16T20:21:57.419 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/ 2023-10-16T20:21:57.419 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout/ 2023-10-16T20:21:57.420 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 81449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-10-16T20:21:57.420 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout/ 2023-10-16T20:21:57.420 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 59449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-10-16T20:21:57.420 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout/ 2023-10-16T20:21:57.420 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 117513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-10-16T20:21:57.421 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 1055649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfs.ko 2023-10-16T20:21:57.421 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 66441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv2.ko 2023-10-16T20:21:57.421 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 120177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv3.ko 2023-10-16T20:21:57.421 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 2041689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv4.ko 2023-10-16T20:21:57.421 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/ 2023-10-16T20:21:57.421 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/grace.ko 2023-10-16T20:21:57.422 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/nfs_acl.ko 2023-10-16T20:21:57.422 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd/ 2023-10-16T20:21:57.422 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 1516809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd/nfsd.ko 2023-10-16T20:21:57.422 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2/ 2023-10-16T20:21:57.422 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 592833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2/nilfs2.ko 2023-10-16T20:21:57.423 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/ 2023-10-16T20:21:57.423 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-celtic.ko 2023-10-16T20:21:57.423 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-centeuro.ko 2023-10-16T20:21:57.423 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-croatian.ko 2023-10-16T20:21:57.423 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-cyrillic.ko 2023-10-16T20:21:57.424 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-gaelic.ko 2023-10-16T20:21:57.424 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-greek.ko 2023-10-16T20:21:57.424 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-iceland.ko 2023-10-16T20:21:57.424 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-inuit.ko 2023-10-16T20:21:57.424 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-roman.ko 2023-10-16T20:21:57.424 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-romanian.ko 2023-10-16T20:21:57.425 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-turkish.ko 2023-10-16T20:21:57.425 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_ascii.ko 2023-10-16T20:21:57.425 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1250.ko 2023-10-16T20:21:57.425 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1251.ko 2023-10-16T20:21:57.425 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1255.ko 2023-10-16T20:21:57.426 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp737.ko 2023-10-16T20:21:57.426 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp775.ko 2023-10-16T20:21:57.426 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp850.ko 2023-10-16T20:21:57.426 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp852.ko 2023-10-16T20:21:57.426 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp855.ko 2023-10-16T20:21:57.426 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp857.ko 2023-10-16T20:21:57.427 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp860.ko 2023-10-16T20:21:57.427 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp861.ko 2023-10-16T20:21:57.427 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp862.ko 2023-10-16T20:21:57.427 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp863.ko 2023-10-16T20:21:57.427 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp864.ko 2023-10-16T20:21:57.428 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp865.ko 2023-10-16T20:21:57.428 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp866.ko 2023-10-16T20:21:57.428 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp869.ko 2023-10-16T20:21:57.428 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp874.ko 2023-10-16T20:21:57.428 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 94137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp932.ko 2023-10-16T20:21:57.429 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 142673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp936.ko 2023-10-16T20:21:57.429 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 165745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp949.ko 2023-10-16T20:21:57.429 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 118121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp950.ko 2023-10-16T20:21:57.429 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_euc-jp.ko 2023-10-16T20:21:57.429 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-1.ko 2023-10-16T20:21:57.430 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-13.ko 2023-10-16T20:21:57.430 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-14.ko 2023-10-16T20:21:57.430 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-15.ko 2023-10-16T20:21:57.430 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-2.ko 2023-10-16T20:21:57.430 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-3.ko 2023-10-16T20:21:57.430 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-4.ko 2023-10-16T20:21:57.431 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-5.ko 2023-10-16T20:21:57.431 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-6.ko 2023-10-16T20:21:57.431 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-7.ko 2023-10-16T20:21:57.431 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-9.ko 2023-10-16T20:21:57.431 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-r.ko 2023-10-16T20:21:57.432 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-ru.ko 2023-10-16T20:21:57.432 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-u.ko 2023-10-16T20:21:57.432 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_ucs2_utils.ko 2023-10-16T20:21:57.432 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_utf8.ko 2023-10-16T20:21:57.432 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ntfs/ 2023-10-16T20:21:57.433 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 219417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ntfs/ntfs.ko 2023-10-16T20:21:57.433 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ 2023-10-16T20:21:57.433 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster/ 2023-10-16T20:21:57.433 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 244409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-10-16T20:21:57.433 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm/ 2023-10-16T20:21:57.434 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 440737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-10-16T20:21:57.434 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs/ 2023-10-16T20:21:57.434 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 54049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-10-16T20:21:57.434 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 2592433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2.ko 2023-10-16T20:21:57.434 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-10-16T20:21:57.435 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-10-16T20:21:57.435 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-10-16T20:21:57.435 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/omfs/ 2023-10-16T20:21:57.435 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 53433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/omfs/omfs.ko 2023-10-16T20:21:57.435 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs/ 2023-10-16T20:21:57.436 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 352153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs/overlay.ko 2023-10-16T20:21:57.436 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore/ 2023-10-16T20:21:57.436 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 54809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore/ramoops.ko 2023-10-16T20:21:57.436 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/qnx4/ 2023-10-16T20:21:57.436 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/qnx4/qnx4.ko 2023-10-16T20:21:57.437 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/qnx6/ 2023-10-16T20:21:57.437 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 46689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/qnx6/qnx6.ko 2023-10-16T20:21:57.437 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/quota/ 2023-10-16T20:21:57.437 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/quota/quota_tree.ko 2023-10-16T20:21:57.437 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/quota/quota_v1.ko 2023-10-16T20:21:57.437 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/quota/quota_v2.ko 2023-10-16T20:21:57.438 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs/ 2023-10-16T20:21:57.438 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 532673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs/reiserfs.ko 2023-10-16T20:21:57.438 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs/ 2023-10-16T20:21:57.438 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs/romfs.ko 2023-10-16T20:21:57.438 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/ 2023-10-16T20:21:57.439 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client/ 2023-10-16T20:21:57.439 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 2668121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client/cifs.ko 2023-10-16T20:21:57.439 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/ 2023-10-16T20:21:57.439 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_arc4.ko 2023-10-16T20:21:57.439 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_md4.ko 2023-10-16T20:21:57.440 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs/ 2023-10-16T20:21:57.440 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 134641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs/squashfs.ko 2023-10-16T20:21:57.440 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv/ 2023-10-16T20:21:57.440 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 90721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv/sysv.ko 2023-10-16T20:21:57.440 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf/ 2023-10-16T20:21:57.441 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 247737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf/udf.ko 2023-10-16T20:21:57.441 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs/ 2023-10-16T20:21:57.441 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 162497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs/ufs.ko 2023-10-16T20:21:57.441 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs/ 2023-10-16T20:21:57.441 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 4010729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs/xfs.ko 2023-10-16T20:21:57.441 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/ 2023-10-16T20:21:57.442 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/locking/ 2023-10-16T20:21:57.442 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 75489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/locking/locktorture.ko 2023-10-16T20:21:57.442 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/time/ 2023-10-16T20:21:57.442 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/time/test_udelay.ko 2023-10-16T20:21:57.442 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 60553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/torture.ko 2023-10-16T20:21:57.443 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ 2023-10-16T20:21:57.443 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 6833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc-itu-t.ko 2023-10-16T20:21:57.443 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 6529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc7.ko 2023-10-16T20:21:57.443 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc8.ko 2023-10-16T20:21:57.443 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/ 2023-10-16T20:21:57.443 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libarc4.ko 2023-10-16T20:21:57.444 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 37081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libdes.ko 2023-10-16T20:21:57.444 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/interval_tree_test.ko 2023-10-16T20:21:57.444 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/libcrc32c.ko 2023-10-16T20:21:57.444 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lru_cache.ko 2023-10-16T20:21:57.444 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/ 2023-10-16T20:21:57.445 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4_compress.ko 2023-10-16T20:21:57.445 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4hc_compress.ko 2023-10-16T20:21:57.445 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math/ 2023-10-16T20:21:57.445 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 6617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math/cordic.ko 2023-10-16T20:21:57.445 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/memory-notifier-error-inject.ko 2023-10-16T20:21:57.445 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/notifier-error-inject.ko 2023-10-16T20:21:57.446 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/percpu_test.ko 2023-10-16T20:21:57.446 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/pm-notifier-error-inject.ko 2023-10-16T20:21:57.446 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6/ 2023-10-16T20:21:57.446 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 143433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6/raid6_pq.ko 2023-10-16T20:21:57.446 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/rbtree_test.ko 2023-10-16T20:21:57.447 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon/ 2023-10-16T20:21:57.447 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon/reed_solomon.ko 2023-10-16T20:21:57.447 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 38841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/test-kstrtox.ko 2023-10-16T20:21:57.447 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/test-string_helpers.ko 2023-10-16T20:21:57.447 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 4760425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/test_bpf.ko 2023-10-16T20:21:57.448 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 49089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/test_firmware.ko 2023-10-16T20:21:57.448 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 6369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/test_module.ko 2023-10-16T20:21:57.448 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/test_user_copy.ko 2023-10-16T20:21:57.448 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_bm.ko 2023-10-16T20:21:57.448 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_fsm.ko 2023-10-16T20:21:57.448 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_kmp.ko 2023-10-16T20:21:57.449 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/xz/ 2023-10-16T20:21:57.449 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/xz/xz_dec_test.ko 2023-10-16T20:21:57.449 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/mm/ 2023-10-16T20:21:57.450 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/mm/hwpoison-inject.ko 2023-10-16T20:21:57.450 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ 2023-10-16T20:21:57.450 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/ 2023-10-16T20:21:57.450 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 55737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/6lowpan.ko 2023-10-16T20:21:57.450 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_dest.ko 2023-10-16T20:21:57.451 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_fragment.ko 2023-10-16T20:21:57.451 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_hop.ko 2023-10-16T20:21:57.451 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 6985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_ipv6.ko 2023-10-16T20:21:57.451 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_mobility.ko 2023-10-16T20:21:57.451 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_routing.ko 2023-10-16T20:21:57.452 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_udp.ko 2023-10-16T20:21:57.452 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/ 2023-10-16T20:21:57.452 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/garp.ko 2023-10-16T20:21:57.452 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/mrp.ko 2023-10-16T20:21:57.452 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/p8022.ko 2023-10-16T20:21:57.453 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/psnap.ko 2023-10-16T20:21:57.453 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/stp.ko 2023-10-16T20:21:57.453 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q/ 2023-10-16T20:21:57.453 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 81737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q/8021q.ko 2023-10-16T20:21:57.453 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/ 2023-10-16T20:21:57.453 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 198337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet.ko 2023-10-16T20:21:57.454 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_fd.ko 2023-10-16T20:21:57.454 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_rdma.ko 2023-10-16T20:21:57.454 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 32337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_virtio.ko 2023-10-16T20:21:57.454 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv/ 2023-10-16T20:21:57.454 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 649297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv/batman-adv.ko 2023-10-16T20:21:57.455 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/ 2023-10-16T20:21:57.455 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 63809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/br_netfilter.ko 2023-10-16T20:21:57.455 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 777689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/bridge.ko 2023-10-16T20:21:57.455 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ 2023-10-16T20:21:57.455 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-10-16T20:21:57.456 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_among.ko 2023-10-16T20:21:57.456 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arp.ko 2023-10-16T20:21:57.456 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-10-16T20:21:57.456 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-10-16T20:21:57.456 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip.ko 2023-10-16T20:21:57.457 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-10-16T20:21:57.457 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_limit.ko 2023-10-16T20:21:57.457 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_log.ko 2023-10-16T20:21:57.457 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark.ko 2023-10-16T20:21:57.457 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-10-16T20:21:57.457 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-10-16T20:21:57.458 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-10-16T20:21:57.458 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-10-16T20:21:57.458 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_snat.ko 2023-10-16T20:21:57.458 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_stp.ko 2023-10-16T20:21:57.458 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-10-16T20:21:57.459 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-10-16T20:21:57.459 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-10-16T20:21:57.459 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-10-16T20:21:57.459 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 55929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtables.ko 2023-10-16T20:21:57.459 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-10-16T20:21:57.459 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-10-16T20:21:57.460 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph/ 2023-10-16T20:21:57.460 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 1047017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph/libceph.ko 2023-10-16T20:21:57.460 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/ 2023-10-16T20:21:57.460 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 125697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/pktgen.ko 2023-10-16T20:21:57.460 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/ 2023-10-16T20:21:57.460 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 194209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp.ko 2023-10-16T20:21:57.461 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_diag.ko 2023-10-16T20:21:57.461 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 57881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv4.ko 2023-10-16T20:21:57.461 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 60337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv6.ko 2023-10-16T20:21:57.461 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/ 2023-10-16T20:21:57.461 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 390473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/dsa_core.ko 2023-10-16T20:21:57.462 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_brcm.ko 2023-10-16T20:21:57.462 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_dsa.ko 2023-10-16T20:21:57.462 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_none.ko 2023-10-16T20:21:57.462 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_trailer.ko 2023-10-16T20:21:57.462 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/hsr/ 2023-10-16T20:21:57.463 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 99729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/hsr/hsr.ko 2023-10-16T20:21:57.463 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ 2023-10-16T20:21:57.463 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan/ 2023-10-16T20:21:57.463 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 45233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-10-16T20:21:57.463 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 254625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154.ko 2023-10-16T20:21:57.464 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 47289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154_socket.ko 2023-10-16T20:21:57.464 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ 2023-10-16T20:21:57.464 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ah4.ko 2023-10-16T20:21:57.464 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 38905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/esp4.ko 2023-10-16T20:21:57.464 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 48401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/fou.ko 2023-10-16T20:21:57.465 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/gre.ko 2023-10-16T20:21:57.465 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 48657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/inet_diag.ko 2023-10-16T20:21:57.465 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 60633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_gre.ko 2023-10-16T20:21:57.465 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 61225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_tunnel.ko 2023-10-16T20:21:57.465 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_vti.ko 2023-10-16T20:21:57.466 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipcomp.ko 2023-10-16T20:21:57.466 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipip.ko 2023-10-16T20:21:57.466 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ 2023-10-16T20:21:57.466 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 37641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arp_tables.ko 2023-10-16T20:21:57.466 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-10-16T20:21:57.467 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-10-16T20:21:57.467 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 41673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ip_tables.ko 2023-10-16T20:21:57.467 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-10-16T20:21:57.467 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2023-10-16T20:21:57.467 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-10-16T20:21:57.467 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-10-16T20:21:57.468 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-10-16T20:21:57.468 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_filter.ko 2023-10-16T20:21:57.468 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-10-16T20:21:57.468 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-10-16T20:21:57.468 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-10-16T20:21:57.469 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_security.ko 2023-10-16T20:21:57.469 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-10-16T20:21:57.469 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-10-16T20:21:57.469 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-10-16T20:21:57.469 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-10-16T20:21:57.470 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-10-16T20:21:57.470 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2023-10-16T20:21:57.470 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-10-16T20:21:57.470 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-10-16T20:21:57.470 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-10-16T20:21:57.471 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-10-16T20:21:57.471 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-10-16T20:21:57.471 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_bic.ko 2023-10-16T20:21:57.471 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_dctcp.ko 2023-10-16T20:21:57.471 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_diag.ko 2023-10-16T20:21:57.472 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_highspeed.ko 2023-10-16T20:21:57.472 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_htcp.ko 2023-10-16T20:21:57.472 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_hybla.ko 2023-10-16T20:21:57.472 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_illinois.ko 2023-10-16T20:21:57.472 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_lp.ko 2023-10-16T20:21:57.473 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_scalable.ko 2023-10-16T20:21:57.473 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_vegas.ko 2023-10-16T20:21:57.473 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_veno.ko 2023-10-16T20:21:57.473 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_westwood.ko 2023-10-16T20:21:57.473 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_yeah.ko 2023-10-16T20:21:57.473 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tunnel4.ko 2023-10-16T20:21:57.474 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_diag.ko 2023-10-16T20:21:57.474 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 41665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_tunnel.ko 2023-10-16T20:21:57.474 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/xfrm4_tunnel.ko 2023-10-16T20:21:57.474 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ 2023-10-16T20:21:57.474 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ah6.ko 2023-10-16T20:21:57.475 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 39537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/esp6.ko 2023-10-16T20:21:57.475 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/fou6.ko 2023-10-16T20:21:57.475 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 80897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_gre.ko 2023-10-16T20:21:57.475 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 93577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_tunnel.ko 2023-10-16T20:21:57.475 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-10-16T20:21:57.475 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 53273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_vti.ko 2023-10-16T20:21:57.476 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ipcomp6.ko 2023-10-16T20:21:57.476 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/mip6.ko 2023-10-16T20:21:57.476 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ 2023-10-16T20:21:57.476 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 42169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-10-16T20:21:57.476 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2023-10-16T20:21:57.477 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-10-16T20:21:57.477 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-10-16T20:21:57.477 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-10-16T20:21:57.477 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-10-16T20:21:57.477 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-10-16T20:21:57.477 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-10-16T20:21:57.478 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-10-16T20:21:57.478 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-10-16T20:21:57.478 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-10-16T20:21:57.478 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-10-16T20:21:57.478 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-10-16T20:21:57.479 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-10-16T20:21:57.479 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-10-16T20:21:57.479 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-10-16T20:21:57.479 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-10-16T20:21:57.479 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 39113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-10-16T20:21:57.479 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-10-16T20:21:57.480 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-10-16T20:21:57.480 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-10-16T20:21:57.480 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-10-16T20:21:57.480 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-10-16T20:21:57.480 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-10-16T20:21:57.480 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-10-16T20:21:57.481 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 74073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/sit.ko 2023-10-16T20:21:57.481 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/tunnel6.ko 2023-10-16T20:21:57.481 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/xfrm6_tunnel.ko 2023-10-16T20:21:57.481 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key/ 2023-10-16T20:21:57.481 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 82833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key/af_key.ko 2023-10-16T20:21:57.482 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/ 2023-10-16T20:21:57.482 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 125281 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_core.ko 2023-10-16T20:21:57.482 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_debugfs.ko 2023-10-16T20:21:57.482 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 25953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_eth.ko 2023-10-16T20:21:57.482 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip.ko 2023-10-16T20:21:57.482 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip6.ko 2023-10-16T20:21:57.483 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_netlink.ko 2023-10-16T20:21:57.483 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 53241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ppp.ko 2023-10-16T20:21:57.483 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/lapb/ 2023-10-16T20:21:57.483 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 47977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/lapb/lapb.ko 2023-10-16T20:21:57.483 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc/ 2023-10-16T20:21:57.484 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc/llc.ko 2023-10-16T20:21:57.484 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 267777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc/llc2.ko 2023-10-16T20:21:57.484 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154/ 2023-10-16T20:21:57.484 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 290417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154/mac802154.ko 2023-10-16T20:21:57.484 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls/ 2023-10-16T20:21:57.485 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls/mpls_gso.ko 2023-10-16T20:21:57.485 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ 2023-10-16T20:21:57.485 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ 2023-10-16T20:21:57.485 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 119585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set.ko 2023-10-16T20:21:57.485 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-10-16T20:21:57.485 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-10-16T20:21:57.486 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-10-16T20:21:57.486 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 100297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-10-16T20:21:57.486 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 101441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-10-16T20:21:57.486 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 102817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-10-16T20:21:57.486 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 103481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-10-16T20:21:57.486 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 112849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-10-16T20:21:57.487 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 60329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-10-16T20:21:57.487 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 109593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-10-16T20:21:57.487 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 113129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-10-16T20:21:57.487 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 112585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-10-16T20:21:57.487 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 111585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-10-16T20:21:57.488 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 114465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-10-16T20:21:57.488 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-10-16T20:21:57.488 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ 2023-10-16T20:21:57.488 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 411729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs.ko 2023-10-16T20:21:57.488 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-10-16T20:21:57.489 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-10-16T20:21:57.489 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-10-16T20:21:57.489 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-10-16T20:21:57.489 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-10-16T20:21:57.489 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-10-16T20:21:57.489 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-10-16T20:21:57.490 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-10-16T20:21:57.490 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-10-16T20:21:57.490 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-10-16T20:21:57.490 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-10-16T20:21:57.490 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-10-16T20:21:57.491 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-10-16T20:21:57.491 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 49257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conncount.ko 2023-10-16T20:21:57.491 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 408257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack.ko 2023-10-16T20:21:57.491 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-10-16T20:21:57.491 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-10-16T20:21:57.491 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-10-16T20:21:57.492 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 118145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_h323.ko 2023-10-16T20:21:57.492 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_irc.ko 2023-10-16T20:21:57.492 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-10-16T20:21:57.492 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 107297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-10-16T20:21:57.492 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-10-16T20:21:57.493 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sane.ko 2023-10-16T20:21:57.493 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 66641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sip.ko 2023-10-16T20:21:57.493 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-10-16T20:21:57.493 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-10-16T20:21:57.493 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_dup_netdev.ko 2023-10-16T20:21:57.493 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 89473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table.ko 2023-10-16T20:21:57.494 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table_inet.ko 2023-10-16T20:21:57.494 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 37129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_log_syslog.ko 2023-10-16T20:21:57.494 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 95417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat.ko 2023-10-16T20:21:57.494 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_amanda.ko 2023-10-16T20:21:57.494 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_ftp.ko 2023-10-16T20:21:57.495 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_irc.ko 2023-10-16T20:21:57.495 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_sip.ko 2023-10-16T20:21:57.495 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_tftp.ko 2023-10-16T20:21:57.495 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 44065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_synproxy_core.ko 2023-10-16T20:21:57.495 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 615313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_tables.ko 2023-10-16T20:21:57.496 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 42561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink.ko 2023-10-16T20:21:57.496 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_acct.ko 2023-10-16T20:21:57.496 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_cttimeout.ko 2023-10-16T20:21:57.496 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 39785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_log.ko 2023-10-16T20:21:57.496 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_osf.ko 2023-10-16T20:21:57.497 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 49377 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_queue.ko 2023-10-16T20:21:57.497 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_chain_nat.ko 2023-10-16T20:21:57.497 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_compat.ko 2023-10-16T20:21:57.497 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 40721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_ct.ko 2023-10-16T20:21:57.497 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_dup_netdev.ko 2023-10-16T20:21:57.498 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fib.ko 2023-10-16T20:21:57.498 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fwd_netdev.ko 2023-10-16T20:21:57.498 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_hash.ko 2023-10-16T20:21:57.498 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_limit.ko 2023-10-16T20:21:57.498 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_log.ko 2023-10-16T20:21:57.498 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_masq.ko 2023-10-16T20:21:57.499 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_nat.ko 2023-10-16T20:21:57.499 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_numgen.ko 2023-10-16T20:21:57.499 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_osf.ko 2023-10-16T20:21:57.499 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_queue.ko 2023-10-16T20:21:57.499 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_redir.ko 2023-10-16T20:21:57.500 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject.ko 2023-10-16T20:21:57.500 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject_inet.ko 2023-10-16T20:21:57.500 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_socket.ko 2023-10-16T20:21:57.500 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_synproxy.ko 2023-10-16T20:21:57.500 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_tproxy.ko 2023-10-16T20:21:57.501 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 89313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/x_tables.ko 2023-10-16T20:21:57.501 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_AUDIT.ko 2023-10-16T20:21:57.501 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CHECKSUM.ko 2023-10-16T20:21:57.501 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CLASSIFY.ko 2023-10-16T20:21:57.501 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-10-16T20:21:57.501 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 18169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CT.ko 2023-10-16T20:21:57.502 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_DSCP.ko 2023-10-16T20:21:57.502 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HL.ko 2023-10-16T20:21:57.502 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HMARK.ko 2023-10-16T20:21:57.502 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 38873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_IDLETIMER.ko 2023-10-16T20:21:57.502 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LED.ko 2023-10-16T20:21:57.503 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LOG.ko 2023-10-16T20:21:57.503 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_MASQUERADE.ko 2023-10-16T20:21:57.503 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NETMAP.ko 2023-10-16T20:21:57.503 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFLOG.ko 2023-10-16T20:21:57.503 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFQUEUE.ko 2023-10-16T20:21:57.504 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_RATEEST.ko 2023-10-16T20:21:57.504 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_REDIRECT.ko 2023-10-16T20:21:57.504 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 14337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_SECMARK.ko 2023-10-16T20:21:57.504 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPMSS.ko 2023-10-16T20:21:57.504 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-10-16T20:21:57.505 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TEE.ko 2023-10-16T20:21:57.505 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TPROXY.ko 2023-10-16T20:21:57.505 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TRACE.ko 2023-10-16T20:21:57.505 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_addrtype.ko 2023-10-16T20:21:57.505 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_bpf.ko 2023-10-16T20:21:57.505 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cgroup.ko 2023-10-16T20:21:57.506 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cluster.ko 2023-10-16T20:21:57.506 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_comment.ko 2023-10-16T20:21:57.506 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connbytes.ko 2023-10-16T20:21:57.506 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlabel.ko 2023-10-16T20:21:57.506 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlimit.ko 2023-10-16T20:21:57.507 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connmark.ko 2023-10-16T20:21:57.507 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_conntrack.ko 2023-10-16T20:21:57.507 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cpu.ko 2023-10-16T20:21:57.507 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dccp.ko 2023-10-16T20:21:57.507 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_devgroup.ko 2023-10-16T20:21:57.507 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dscp.ko 2023-10-16T20:21:57.508 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ecn.ko 2023-10-16T20:21:57.508 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_esp.ko 2023-10-16T20:21:57.508 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 43289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hashlimit.ko 2023-10-16T20:21:57.508 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_helper.ko 2023-10-16T20:21:57.508 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hl.ko 2023-10-16T20:21:57.509 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipcomp.ko 2023-10-16T20:21:57.509 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_iprange.ko 2023-10-16T20:21:57.509 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipvs.ko 2023-10-16T20:21:57.509 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_l2tp.ko 2023-10-16T20:21:57.509 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_length.ko 2023-10-16T20:21:57.509 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_limit.ko 2023-10-16T20:21:57.510 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mac.ko 2023-10-16T20:21:57.510 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mark.ko 2023-10-16T20:21:57.510 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_multiport.ko 2023-10-16T20:21:57.510 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nat.ko 2023-10-16T20:21:57.510 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nfacct.ko 2023-10-16T20:21:57.511 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_osf.ko 2023-10-16T20:21:57.511 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_owner.ko 2023-10-16T20:21:57.511 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_physdev.ko 2023-10-16T20:21:57.511 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_pkttype.ko 2023-10-16T20:21:57.511 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_policy.ko 2023-10-16T20:21:57.511 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_quota.ko 2023-10-16T20:21:57.512 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_rateest.ko 2023-10-16T20:21:57.512 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_realm.ko 2023-10-16T20:21:57.512 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_recent.ko 2023-10-16T20:21:57.512 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_sctp.ko 2023-10-16T20:21:57.512 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_set.ko 2023-10-16T20:21:57.513 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 16089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_socket.ko 2023-10-16T20:21:57.513 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_state.ko 2023-10-16T20:21:57.513 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_statistic.ko 2023-10-16T20:21:57.513 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_string.ko 2023-10-16T20:21:57.513 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 8553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_tcpmss.ko 2023-10-16T20:21:57.513 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_tcpudp.ko 2023-10-16T20:21:57.514 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_time.ko 2023-10-16T20:21:57.514 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_u32.ko 2023-10-16T20:21:57.514 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink/ 2023-10-16T20:21:57.514 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink/netlink_diag.ko 2023-10-16T20:21:57.514 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh/ 2023-10-16T20:21:57.515 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 12425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh/nsh.ko 2023-10-16T20:21:57.515 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/ 2023-10-16T20:21:57.515 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 380033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/openvswitch.ko 2023-10-16T20:21:57.515 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-geneve.ko 2023-10-16T20:21:57.515 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-gre.ko 2023-10-16T20:21:57.516 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 11793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-vxlan.ko 2023-10-16T20:21:57.516 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet/ 2023-10-16T20:21:57.516 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet/af_packet_diag.ko 2023-10-16T20:21:57.516 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/phonet/ 2023-10-16T20:21:57.516 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 110201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/phonet/phonet.ko 2023-10-16T20:21:57.517 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 54697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/phonet/pn_pep.ko 2023-10-16T20:21:57.517 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/ 2023-10-16T20:21:57.517 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 251481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds.ko 2023-10-16T20:21:57.517 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 164937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_rdma.ko 2023-10-16T20:21:57.517 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 67377 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_tcp.ko 2023-10-16T20:21:57.517 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rxrpc/ 2023-10-16T20:21:57.518 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 819657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rxrpc/rxrpc.ko 2023-10-16T20:21:57.518 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/ 2023-10-16T20:21:57.518 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 32241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_csum.ko 2023-10-16T20:21:57.518 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_gact.ko 2023-10-16T20:21:57.518 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_ipt.ko 2023-10-16T20:21:57.519 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_mirred.ko 2023-10-16T20:21:57.519 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_nat.ko 2023-10-16T20:21:57.519 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 43449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_pedit.ko 2023-10-16T20:21:57.519 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_police.ko 2023-10-16T20:21:57.519 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_simple.ko 2023-10-16T20:21:57.519 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_skbedit.ko 2023-10-16T20:21:57.520 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_vlan.ko 2023-10-16T20:21:57.520 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_basic.ko 2023-10-16T20:21:57.520 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_bpf.ko 2023-10-16T20:21:57.520 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_cgroup.ko 2023-10-16T20:21:57.520 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_flow.ko 2023-10-16T20:21:57.521 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_fw.ko 2023-10-16T20:21:57.521 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 38489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_route.ko 2023-10-16T20:21:57.521 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 61409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_u32.ko 2023-10-16T20:21:57.521 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_cmp.ko 2023-10-16T20:21:57.521 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 13409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_ipset.ko 2023-10-16T20:21:57.522 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_meta.ko 2023-10-16T20:21:57.522 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_nbyte.ko 2023-10-16T20:21:57.522 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_text.ko 2023-10-16T20:21:57.522 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_u32.ko 2023-10-16T20:21:57.522 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 21649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_choke.ko 2023-10-16T20:21:57.523 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_codel.ko 2023-10-16T20:21:57.523 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 34633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_drr.ko 2023-10-16T20:21:57.523 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 33609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_fq.ko 2023-10-16T20:21:57.523 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_fq_codel.ko 2023-10-16T20:21:57.523 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 35433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_gred.ko 2023-10-16T20:21:57.524 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 49601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hfsc.ko 2023-10-16T20:21:57.524 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hhf.ko 2023-10-16T20:21:57.524 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 66393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_htb.ko 2023-10-16T20:21:57.524 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_ingress.ko 2023-10-16T20:21:57.524 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 30273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio.ko 2023-10-16T20:21:57.525 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio_lib.ko 2023-10-16T20:21:57.525 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_multiq.ko 2023-10-16T20:21:57.525 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 36137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_netem.ko 2023-10-16T20:21:57.525 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 27497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_pie.ko 2023-10-16T20:21:57.525 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_plug.ko 2023-10-16T20:21:57.525 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 24729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_prio.ko 2023-10-16T20:21:57.526 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 48793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_qfq.ko 2023-10-16T20:21:57.526 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 32905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_red.ko 2023-10-16T20:21:57.526 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 26881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfb.ko 2023-10-16T20:21:57.526 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 31353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfq.ko 2023-10-16T20:21:57.526 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_tbf.ko 2023-10-16T20:21:57.527 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 28105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_teql.ko 2023-10-16T20:21:57.527 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/ 2023-10-16T20:21:57.527 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 929745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp.ko 2023-10-16T20:21:57.527 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 19033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp_diag.ko 2023-10-16T20:21:57.527 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/ 2023-10-16T20:21:57.528 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/ 2023-10-16T20:21:57.528 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 336529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-10-16T20:21:57.528 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 54737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-10-16T20:21:57.528 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 1520161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/sunrpc.ko 2023-10-16T20:21:57.528 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma/ 2023-10-16T20:21:57.528 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 707721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-10-16T20:21:57.529 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/ 2023-10-16T20:21:57.529 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/diag.ko 2023-10-16T20:21:57.529 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 862641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/tipc.ko 2023-10-16T20:21:57.529 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix/ 2023-10-16T20:21:57.529 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix/unix_diag.ko 2023-10-16T20:21:57.530 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/ 2023-10-16T20:21:57.530 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 85337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-10-16T20:21:57.530 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 68049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-10-16T20:21:57.530 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 96193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock.ko 2023-10-16T20:21:57.530 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 10705 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_diag.ko 2023-10-16T20:21:57.531 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 17521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_loopback.ko 2023-10-16T20:21:57.531 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/x25/ 2023-10-16T20:21:57.531 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 125177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/x25/x25.ko 2023-10-16T20:21:57.531 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm/ 2023-10-16T20:21:57.531 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 22657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm/xfrm_algo.ko 2023-10-16T20:21:57.532 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 20649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm/xfrm_ipcomp.ko 2023-10-16T20:21:57.532 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 102257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm/xfrm_user.ko 2023-10-16T20:21:57.532 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/ 2023-10-16T20:21:57.532 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib/ 2023-10-16T20:21:57.532 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 15537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib/irqbypass.ko 2023-10-16T20:21:57.532 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 639863 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.alias 2023-10-16T20:21:57.533 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 623919 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.alias.bin 2023-10-16T20:21:57.533 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 7663 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.builtin 2023-10-16T20:21:57.533 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 23428 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.alias.bin 2023-10-16T20:21:57.533 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 9821 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.bin 2023-10-16T20:21:57.533 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 70004 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.modinfo 2023-10-16T20:21:57.534 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 146313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.dep 2023-10-16T20:21:57.534 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 232548 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.dep.bin 2023-10-16T20:21:57.534 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 210 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.devname 2023-10-16T20:21:57.534 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 72606 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.order 2023-10-16T20:21:57.534 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 879 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.softdep 2023-10-16T20:21:57.535 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 254438 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.symbols 2023-10-16T20:21:57.535 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 309363 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.symbols.bin 2023-10-16T20:21:57.535 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./usr/ 2023-10-16T20:21:57.535 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./usr/share/ 2023-10-16T20:21:57.535 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./usr/share/doc/ 2023-10-16T20:21:57.535 INFO:teuthology.orchestra.run.smithi191.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./usr/share/doc/linux-image-6.6.0-rc5-g47672119149b/ 2023-10-16T20:21:57.536 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 175 2023-10-10 19:56 ./usr/share/doc/linux-image-6.6.0-rc5-g47672119149b/changelog.Debian.gz 2023-10-16T20:21:57.536 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r-- root/root 691 2023-10-10 19:56 ./usr/share/doc/linux-image-6.6.0-rc5-g47672119149b/copyright 2023-10-16T20:21:57.537 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-g47672119149b 2023-10-16T20:21:57.537 INFO:teuthology.task.kernel:searching for kernel 6.6.0-rc5-g47672119149b 2023-10-16T20:21:57.537 DEBUG:teuthology.orchestra.run.smithi191:> egrep '(submenu|menuentry.*6.6.0-rc5-g47672119149b).*{' /boot/grub/grub.cfg 2023-10-16T20:21:57.544 INFO:teuthology.orchestra.run.smithi191.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-16T20:21:57.544 INFO:teuthology.orchestra.run.smithi191.stdout: menuentry 'Ubuntu, with Linux 6.6.0-rc5-g47672119149b' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-rc5-g47672119149b-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-16T20:21:57.544 INFO:teuthology.orchestra.run.smithi191.stdout: menuentry 'Ubuntu, with Linux 6.6.0-rc5-g47672119149b (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-rc5-g47672119149b-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-16T20:21:57.545 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2023-10-16T20:21:57.545 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.6.0-rc5-g47672119149b 2023-10-16T20:21:57.546 DEBUG:teuthology.orchestra.run.smithi191:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2023-10-16T20:21:57.589 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi191.front.sepia.ceph.com to complete... 2023-10-16T20:21:57.974 INFO:teuthology.orchestra.run.smithi191.stderr:Sourcing file `/etc/default/grub' 2023-10-16T20:21:57.975 INFO:teuthology.orchestra.run.smithi191.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-16T20:21:57.999 INFO:teuthology.orchestra.run.smithi191.stderr:Generating grub configuration file ... 2023-10-16T20:21:58.207 INFO:teuthology.orchestra.run.smithi191.stderr:Found linux image: /boot/vmlinuz-6.6.0-rc5-g47672119149b 2023-10-16T20:21:58.214 INFO:teuthology.orchestra.run.smithi191.stderr:Found initrd image: /boot/initrd.img-6.6.0-rc5-g47672119149b 2023-10-16T20:21:58.422 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-image-6.6.0-rc5-g47672119149b (6.6.0-rc5-g47672119149b-1) ... 2023-10-16T20:21:58.454 INFO:teuthology.orchestra.run.smithi191.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-16T20:21:58.459 INFO:teuthology.orchestra.run.smithi191.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-16T20:21:58.497 INFO:teuthology.orchestra.run.smithi191.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-16T20:21:58.501 INFO:teuthology.orchestra.run.smithi191.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-16T20:21:58.518 INFO:teuthology.orchestra.run.smithi088.stderr:update-initramfs: Generating /boot/initrd.img-6.6.0-rc5-g47672119149b 2023-10-16T20:21:58.611 INFO:teuthology.orchestra.run.smithi191.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-16T20:21:58.612 INFO:teuthology.orchestra.run.smithi191.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-16T20:21:58.612 INFO:teuthology.orchestra.run.smithi191.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-16T20:21:58.620 INFO:teuthology.orchestra.run.smithi191.stderr:done 2023-10-16T20:22:18.417 INFO:teuthology.orchestra.run.smithi088.stderr:Sourcing file `/etc/default/grub' 2023-10-16T20:22:18.436 INFO:teuthology.orchestra.run.smithi088.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-16T20:22:18.488 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2023-10-16T20:22:18.806 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-6.6.0-rc5-g47672119149b 2023-10-16T20:22:18.822 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initrd.img-6.6.0-rc5-g47672119149b 2023-10-16T20:22:19.095 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-16T20:22:19.107 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-16T20:22:19.182 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-16T20:22:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-16T20:22:19.330 INFO:teuthology.orchestra.run.smithi088.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-16T20:22:19.331 INFO:teuthology.orchestra.run.smithi088.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-16T20:22:19.331 INFO:teuthology.orchestra.run.smithi088.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-16T20:22:19.376 INFO:teuthology.orchestra.run.smithi088.stderr:done 2023-10-16T20:22:19.534 DEBUG:teuthology.orchestra.run.smithi088:> dpkg-deb -c /tmp/linux-image.deb 2023-10-16T20:22:19.542 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./ 2023-10-16T20:22:19.542 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./boot/ 2023-10-16T20:22:19.542 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 4415097 2023-10-10 19:56 ./boot/System.map-6.6.0-rc5-g47672119149b 2023-10-16T20:22:19.554 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 169298 2023-10-10 19:56 ./boot/config-6.6.0-rc5-g47672119149b 2023-10-16T20:22:19.555 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11352928 2023-10-10 19:56 ./boot/vmlinuz-6.6.0-rc5-g47672119149b 2023-10-16T20:22:19.573 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./etc/ 2023-10-16T20:22:19.573 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./etc/kernel/ 2023-10-16T20:22:19.573 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./etc/kernel/postinst.d/ 2023-10-16T20:22:19.573 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./etc/kernel/postrm.d/ 2023-10-16T20:22:19.574 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./etc/kernel/preinst.d/ 2023-10-16T20:22:19.574 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./etc/kernel/prerm.d/ 2023-10-16T20:22:19.574 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/ 2023-10-16T20:22:19.574 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/ 2023-10-16T20:22:19.574 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/ 2023-10-16T20:22:19.575 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/ 2023-10-16T20:22:19.575 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/ 2023-10-16T20:22:19.575 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/ 2023-10-16T20:22:19.575 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/ 2023-10-16T20:22:19.575 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 490233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/aesni-intel.ko 2023-10-16T20:22:19.578 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-10-16T20:22:19.578 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-10-16T20:22:19.578 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-10-16T20:22:19.578 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 93113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-10-16T20:22:19.578 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 106729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-10-16T20:22:19.579 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 152241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-10-16T20:22:19.579 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-10-16T20:22:19.579 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-10-16T20:22:19.579 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 83889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-10-16T20:22:19.579 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-10-16T20:22:19.579 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 47225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-10-16T20:22:19.580 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 47209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx2.ko 2023-10-16T20:22:19.580 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 51801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-10-16T20:22:19.580 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 38937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-10-16T20:22:19.580 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 40529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-10-16T20:22:19.580 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 58585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-10-16T20:22:19.581 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 50497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-10-16T20:22:19.581 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 32065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-10-16T20:22:19.581 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-10-16T20:22:19.581 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/ 2023-10-16T20:22:19.581 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/ 2023-10-16T20:22:19.581 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce/ 2023-10-16T20:22:19.582 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 40113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-10-16T20:22:19.582 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpuid.ko 2023-10-16T20:22:19.582 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/msr.ko 2023-10-16T20:22:19.582 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/ 2023-10-16T20:22:19.582 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 353761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-amd.ko 2023-10-16T20:22:19.583 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 881713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-intel.ko 2023-10-16T20:22:19.583 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 2276817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm.ko 2023-10-16T20:22:19.618 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ 2023-10-16T20:22:19.618 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 53761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/af_alg.ko 2023-10-16T20:22:19.618 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/algif_hash.ko 2023-10-16T20:22:19.619 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/algif_skcipher.ko 2023-10-16T20:22:19.619 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ansi_cprng.ko 2023-10-16T20:22:19.619 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/anubis.ko 2023-10-16T20:22:19.619 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/arc4.ko 2023-10-16T20:22:19.619 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/asymmetric_keys/ 2023-10-16T20:22:19.620 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2023-10-16T20:22:19.620 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/ 2023-10-16T20:22:19.620 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_memcpy.ko 2023-10-16T20:22:19.620 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_pq.ko 2023-10-16T20:22:19.620 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_raid6_recov.ko 2023-10-16T20:22:19.620 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_tx.ko 2023-10-16T20:22:19.621 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_xor.ko 2023-10-16T20:22:19.621 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/raid6test.ko 2023-10-16T20:22:19.621 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authenc.ko 2023-10-16T20:22:19.621 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authencesn.ko 2023-10-16T20:22:19.621 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blake2b_generic.ko 2023-10-16T20:22:19.621 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_common.ko 2023-10-16T20:22:19.622 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_generic.ko 2023-10-16T20:22:19.622 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 38873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/camellia_generic.ko 2023-10-16T20:22:19.622 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast5_generic.ko 2023-10-16T20:22:19.622 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast6_generic.ko 2023-10-16T20:22:19.622 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast_common.ko 2023-10-16T20:22:19.623 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ccm.ko 2023-10-16T20:22:19.623 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cmac.ko 2023-10-16T20:22:19.623 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crc32_generic.ko 2023-10-16T20:22:19.623 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 47505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cryptd.ko 2023-10-16T20:22:19.623 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 39505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_engine.ko 2023-10-16T20:22:19.623 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_simd.ko 2023-10-16T20:22:19.624 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_user.ko 2023-10-16T20:22:19.624 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cts.ko 2023-10-16T20:22:19.624 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/des_generic.ko 2023-10-16T20:22:19.624 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/dh_generic.ko 2023-10-16T20:22:19.624 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/echainiv.ko 2023-10-16T20:22:19.624 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/essiv.ko 2023-10-16T20:22:19.625 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/fcrypt.ko 2023-10-16T20:22:19.625 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/khazad.ko 2023-10-16T20:22:19.625 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lrw.ko 2023-10-16T20:22:19.625 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4.ko 2023-10-16T20:22:19.625 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4hc.ko 2023-10-16T20:22:19.625 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/md4.ko 2023-10-16T20:22:19.626 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/michael_mic.ko 2023-10-16T20:22:19.626 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcbc.ko 2023-10-16T20:22:19.626 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcrypt.ko 2023-10-16T20:22:19.626 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/rmd160.ko 2023-10-16T20:22:19.626 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/seed.ko 2023-10-16T20:22:19.627 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/serpent_generic.ko 2023-10-16T20:22:19.627 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 149193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tcrypt.ko 2023-10-16T20:22:19.627 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tea.ko 2023-10-16T20:22:19.627 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_common.ko 2023-10-16T20:22:19.627 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_generic.ko 2023-10-16T20:22:19.627 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/vmac.ko 2023-10-16T20:22:19.628 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/wp512.ko 2023-10-16T20:22:19.628 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xcbc.ko 2023-10-16T20:22:19.628 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xor.ko 2023-10-16T20:22:19.628 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xts.ko 2023-10-16T20:22:19.628 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xxhash_generic.ko 2023-10-16T20:22:19.628 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ 2023-10-16T20:22:19.629 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/ 2023-10-16T20:22:19.629 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_extlog.ko 2023-10-16T20:22:19.629 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_ipmi.ko 2023-10-16T20:22:19.629 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_pad.ko 2023-10-16T20:22:19.629 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/apei/ 2023-10-16T20:22:19.629 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/apei/einj.ko 2023-10-16T20:22:19.630 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/ec_sys.ko 2023-10-16T20:22:19.630 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/platform_profile.ko 2023-10-16T20:22:19.630 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbs.ko 2023-10-16T20:22:19.630 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbshc.ko 2023-10-16T20:22:19.630 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 103025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/video.ko 2023-10-16T20:22:19.631 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ 2023-10-16T20:22:19.631 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/acard-ahci.ko 2023-10-16T20:22:19.631 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 88865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ahci.ko 2023-10-16T20:22:19.631 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ahci_platform.ko 2023-10-16T20:22:19.631 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 88289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/libahci.ko 2023-10-16T20:22:19.632 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/libahci_platform.ko 2023-10-16T20:22:19.632 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_acpi.ko 2023-10-16T20:22:19.632 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ali.ko 2023-10-16T20:22:19.632 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_amd.ko 2023-10-16T20:22:19.632 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_artop.ko 2023-10-16T20:22:19.632 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atiixp.ko 2023-10-16T20:22:19.633 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atp867x.ko 2023-10-16T20:22:19.633 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd640.ko 2023-10-16T20:22:19.633 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd64x.ko 2023-10-16T20:22:19.633 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cypress.ko 2023-10-16T20:22:19.633 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_efar.ko 2023-10-16T20:22:19.634 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt366.ko 2023-10-16T20:22:19.634 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt37x.ko 2023-10-16T20:22:19.634 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x2n.ko 2023-10-16T20:22:19.634 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x3.ko 2023-10-16T20:22:19.634 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it8213.ko 2023-10-16T20:22:19.634 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it821x.ko 2023-10-16T20:22:19.635 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_jmicron.ko 2023-10-16T20:22:19.635 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 51481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_legacy.ko 2023-10-16T20:22:19.635 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_marvell.ko 2023-10-16T20:22:19.635 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16281 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_mpiix.ko 2023-10-16T20:22:19.635 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_netcell.ko 2023-10-16T20:22:19.635 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ninja32.ko 2023-10-16T20:22:19.636 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87410.ko 2023-10-16T20:22:19.636 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87415.ko 2023-10-16T20:22:19.636 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_oldpiix.ko 2023-10-16T20:22:19.636 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15377 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_opti.ko 2023-10-16T20:22:19.636 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_optidma.ko 2023-10-16T20:22:19.636 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 32585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pcmcia.ko 2023-10-16T20:22:19.637 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 37793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc2027x.ko 2023-10-16T20:22:19.637 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-10-16T20:22:19.637 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_piccolo.ko 2023-10-16T20:22:19.637 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_radisys.ko 2023-10-16T20:22:19.637 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_rdc.ko 2023-10-16T20:22:19.638 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_rz1000.ko 2023-10-16T20:22:19.638 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sch.ko 2023-10-16T20:22:19.638 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_serverworks.ko 2023-10-16T20:22:19.638 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sil680.ko 2023-10-16T20:22:19.638 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sl82c105.ko 2023-10-16T20:22:19.639 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_triflex.ko 2023-10-16T20:22:19.639 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_via.ko 2023-10-16T20:22:19.639 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pdc_adma.ko 2023-10-16T20:22:19.639 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_inic162x.ko 2023-10-16T20:22:19.639 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 83345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_mv.ko 2023-10-16T20:22:19.639 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 70393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_nv.ko 2023-10-16T20:22:19.640 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 37377 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_promise.ko 2023-10-16T20:22:19.640 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_qstor.ko 2023-10-16T20:22:19.640 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil.ko 2023-10-16T20:22:19.640 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 38337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil24.ko 2023-10-16T20:22:19.640 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sis.ko 2023-10-16T20:22:19.640 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_svw.ko 2023-10-16T20:22:19.641 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 44793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sx4.ko 2023-10-16T20:22:19.641 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_uli.ko 2023-10-16T20:22:19.641 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_via.ko 2023-10-16T20:22:19.641 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_vsc.ko 2023-10-16T20:22:19.641 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay/ 2023-10-16T20:22:19.642 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay/ks0108.ko 2023-10-16T20:22:19.642 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma/ 2023-10-16T20:22:19.642 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 124473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma/bcma.ko 2023-10-16T20:22:19.642 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/ 2023-10-16T20:22:19.642 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe/ 2023-10-16T20:22:19.642 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 113665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe/aoe.ko 2023-10-16T20:22:19.643 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd/ 2023-10-16T20:22:19.643 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 965585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd/drbd.ko 2023-10-16T20:22:19.643 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 173225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/floppy.ko 2023-10-16T20:22:19.643 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx/ 2023-10-16T20:22:19.643 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 94297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-10-16T20:22:19.644 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 115793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/nbd.ko 2023-10-16T20:22:19.644 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk/ 2023-10-16T20:22:19.644 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 110449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk/null_blk.ko 2023-10-16T20:22:19.644 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 98193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/pktcdvd.ko 2023-10-16T20:22:19.644 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 263769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/rbd.ko 2023-10-16T20:22:19.644 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback/ 2023-10-16T20:22:19.645 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 98145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-10-16T20:22:19.645 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram/ 2023-10-16T20:22:19.645 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 60217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram/zram.ko 2023-10-16T20:22:19.645 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ 2023-10-16T20:22:19.645 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30705 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/applicom.ko 2023-10-16T20:22:19.645 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hangcheck-timer.ko 2023-10-16T20:22:19.646 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/ 2023-10-16T20:22:19.646 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/amd-rng.ko 2023-10-16T20:22:19.646 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/intel-rng.ko 2023-10-16T20:22:19.646 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-10-16T20:22:19.646 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/via-rng.ko 2023-10-16T20:22:19.647 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/virtio-rng.ko 2023-10-16T20:22:19.647 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ 2023-10-16T20:22:19.647 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-10-16T20:22:19.647 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 148601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-10-16T20:22:19.647 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-10-16T20:22:19.647 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 139313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_si.ko 2023-10-16T20:22:19.648 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 70921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-10-16T20:22:19.648 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 53513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-10-16T20:22:19.648 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/lp.ko 2023-10-16T20:22:19.648 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave/ 2023-10-16T20:22:19.648 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 84097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave/mwave.ko 2023-10-16T20:22:19.648 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/nvram.ko 2023-10-16T20:22:19.649 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 32169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ppdev.ko 2023-10-16T20:22:19.649 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 51113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tlclk.ko 2023-10-16T20:22:19.649 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/ 2023-10-16T20:22:19.649 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_atmel.ko 2023-10-16T20:22:19.649 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2023-10-16T20:22:19.649 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2023-10-16T20:22:19.650 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 32985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2023-10-16T20:22:19.650 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_infineon.ko 2023-10-16T20:22:19.650 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_nsc.ko 2023-10-16T20:22:19.650 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/xen-tpmfront.ko 2023-10-16T20:22:19.650 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/xillybus/ 2023-10-16T20:22:19.650 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/xillybus/xillybus_class.ko 2023-10-16T20:22:19.651 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 50945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/xillybus/xillybus_core.ko 2023-10-16T20:22:19.651 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/xillybus/xillybus_pcie.ko 2023-10-16T20:22:19.651 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/clk/ 2023-10-16T20:22:19.651 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/clk/clk-palmas.ko 2023-10-16T20:22:19.651 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 47849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/clk/clk-si5351.ko 2023-10-16T20:22:19.652 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/clk/clk-twl6040.ko 2023-10-16T20:22:19.652 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/clk/clk-wm831x.ko 2023-10-16T20:22:19.652 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/ 2023-10-16T20:22:19.652 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-10-16T20:22:19.652 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/p4-clockmod.ko 2023-10-16T20:22:19.652 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/speedstep-lib.ko 2023-10-16T20:22:19.653 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ 2023-10-16T20:22:19.653 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ 2023-10-16T20:22:19.653 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 70801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-10-16T20:22:19.653 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 243025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp.ko 2023-10-16T20:22:19.653 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/ 2023-10-16T20:22:19.653 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/ 2023-10-16T20:22:19.654 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common/ 2023-10-16T20:22:19.654 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 438761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-10-16T20:22:19.654 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2023-10-16T20:22:19.654 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-10-16T20:22:19.654 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-aes.ko 2023-10-16T20:22:19.654 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-sha.ko 2023-10-16T20:22:19.655 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio/ 2023-10-16T20:22:19.655 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 68857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-10-16T20:22:19.655 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca/ 2023-10-16T20:22:19.655 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca/dca.ko 2023-10-16T20:22:19.655 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ 2023-10-16T20:22:19.656 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw/ 2023-10-16T20:22:19.656 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw/dw_dmac.ko 2023-10-16T20:22:19.656 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat/ 2023-10-16T20:22:19.656 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 122569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat/ioatdma.ko 2023-10-16T20:22:19.656 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/ 2023-10-16T20:22:19.656 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 79953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/amd64_edac.ko 2023-10-16T20:22:19.657 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/e752x_edac.ko 2023-10-16T20:22:19.657 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 63617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/edac_mce_amd.ko 2023-10-16T20:22:19.657 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3000_edac.ko 2023-10-16T20:22:19.657 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3200_edac.ko 2023-10-16T20:22:19.657 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5100_edac.ko 2023-10-16T20:22:19.657 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5400_edac.ko 2023-10-16T20:22:19.658 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7300_edac.ko 2023-10-16T20:22:19.658 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 52041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7core_edac.ko 2023-10-16T20:22:19.658 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i82975x_edac.ko 2023-10-16T20:22:19.658 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/ie31200_edac.ko 2023-10-16T20:22:19.658 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 56825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/sb_edac.ko 2023-10-16T20:22:19.658 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/x38_edac.ko 2023-10-16T20:22:19.659 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/extcon/ 2023-10-16T20:22:19.659 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/extcon/extcon-gpio.ko 2023-10-16T20:22:19.659 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/extcon/extcon-max14577.ko 2023-10-16T20:22:19.659 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/extcon/extcon-max77693.ko 2023-10-16T20:22:19.659 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/extcon/extcon-max8997.ko 2023-10-16T20:22:19.659 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/extcon/extcon-palmas.ko 2023-10-16T20:22:19.660 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/extcon/extcon-rt8973a.ko 2023-10-16T20:22:19.660 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/extcon/extcon-sm5502.ko 2023-10-16T20:22:19.660 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/ 2023-10-16T20:22:19.660 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/dmi-sysfs.ko 2023-10-16T20:22:19.660 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/efi/ 2023-10-16T20:22:19.660 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/efi/efi-pstore.ko 2023-10-16T20:22:19.661 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/iscsi_ibft.ko 2023-10-16T20:22:19.661 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/ 2023-10-16T20:22:19.661 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-adp5520.ko 2023-10-16T20:22:19.661 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-amd8111.ko 2023-10-16T20:22:19.661 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-arizona.ko 2023-10-16T20:22:19.662 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-da9052.ko 2023-10-16T20:22:19.662 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-da9055.ko 2023-10-16T20:22:19.662 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-dln2.ko 2023-10-16T20:22:19.662 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 77849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-f7188x.ko 2023-10-16T20:22:19.662 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-generic.ko 2023-10-16T20:22:19.663 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-ich.ko 2023-10-16T20:22:19.663 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-janz-ttl.ko 2023-10-16T20:22:19.663 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-kempld.ko 2023-10-16T20:22:19.663 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-lp3943.ko 2023-10-16T20:22:19.663 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-max7300.ko 2023-10-16T20:22:19.663 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-max730x.ko 2023-10-16T20:22:19.664 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-max732x.ko 2023-10-16T20:22:19.664 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-ml-ioh.ko 2023-10-16T20:22:19.664 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 47073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-pca953x.ko 2023-10-16T20:22:19.664 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-pcf857x.ko 2023-10-16T20:22:19.664 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14377 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-rdc321x.ko 2023-10-16T20:22:19.665 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-sch.ko 2023-10-16T20:22:19.665 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-sch311x.ko 2023-10-16T20:22:19.665 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-tps65912.ko 2023-10-16T20:22:19.665 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-twl4030.ko 2023-10-16T20:22:19.665 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-twl6040.ko 2023-10-16T20:22:19.665 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-viperboard.ko 2023-10-16T20:22:19.666 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15377 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-vx855.ko 2023-10-16T20:22:19.666 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16281 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-wm831x.ko 2023-10-16T20:22:19.666 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-wm8350.ko 2023-10-16T20:22:19.666 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-wm8994.ko 2023-10-16T20:22:19.666 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/ 2023-10-16T20:22:19.666 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-a4tech.ko 2023-10-16T20:22:19.667 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 39169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-apple.ko 2023-10-16T20:22:19.667 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-appleir.ko 2023-10-16T20:22:19.667 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-aureal.ko 2023-10-16T20:22:19.667 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-axff.ko 2023-10-16T20:22:19.667 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-belkin.ko 2023-10-16T20:22:19.668 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-cherry.ko 2023-10-16T20:22:19.668 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-chicony.ko 2023-10-16T20:22:19.668 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 60649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-cp2112.ko 2023-10-16T20:22:19.668 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-cypress.ko 2023-10-16T20:22:19.668 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-dr.ko 2023-10-16T20:22:19.669 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elecom.ko 2023-10-16T20:22:19.669 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elo.ko 2023-10-16T20:22:19.669 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-emsff.ko 2023-10-16T20:22:19.669 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-ezkey.ko 2023-10-16T20:22:19.669 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gaff.ko 2023-10-16T20:22:19.669 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-generic.ko 2023-10-16T20:22:19.670 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gt683r.ko 2023-10-16T20:22:19.670 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gyration.ko 2023-10-16T20:22:19.670 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-kbd.ko 2023-10-16T20:22:19.670 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-mouse.ko 2023-10-16T20:22:19.670 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtekff.ko 2023-10-16T20:22:19.670 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-hyperv.ko 2023-10-16T20:22:19.671 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-icade.ko 2023-10-16T20:22:19.671 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-kensington.ko 2023-10-16T20:22:19.671 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-keytouch.ko 2023-10-16T20:22:19.671 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-kye.ko 2023-10-16T20:22:19.671 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lcpower.ko 2023-10-16T20:22:19.672 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-led.ko 2023-10-16T20:22:19.672 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 42361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lenovo.ko 2023-10-16T20:22:19.672 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lg-g15.ko 2023-10-16T20:22:19.672 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 58185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-dj.ko 2023-10-16T20:22:19.672 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 101369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-10-16T20:22:19.672 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 80993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech.ko 2023-10-16T20:22:19.673 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-magicmouse.ko 2023-10-16T20:22:19.673 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-microsoft.ko 2023-10-16T20:22:19.673 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-monterey.ko 2023-10-16T20:22:19.673 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 50161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-multitouch.ko 2023-10-16T20:22:19.673 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 32593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-ntrig.ko 2023-10-16T20:22:19.673 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-ortek.ko 2023-10-16T20:22:19.674 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-penmount.ko 2023-10-16T20:22:19.674 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-petalynx.ko 2023-10-16T20:22:19.674 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 72937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-picolcd.ko 2023-10-16T20:22:19.674 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-pl.ko 2023-10-16T20:22:19.674 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-plantronics.ko 2023-10-16T20:22:19.674 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-primax.ko 2023-10-16T20:22:19.675 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-rmi.ko 2023-10-16T20:22:19.675 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-arvo.ko 2023-10-16T20:22:19.675 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-common.ko 2023-10-16T20:22:19.675 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-isku.ko 2023-10-16T20:22:19.675 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kone.ko 2023-10-16T20:22:19.675 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-10-16T20:22:19.676 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-konepure.ko 2023-10-16T20:22:19.676 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-10-16T20:22:19.676 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-lua.ko 2023-10-16T20:22:19.676 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-pyra.ko 2023-10-16T20:22:19.676 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-ryos.ko 2023-10-16T20:22:19.676 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-savu.ko 2023-10-16T20:22:19.677 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat.ko 2023-10-16T20:22:19.677 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-saitek.ko 2023-10-16T20:22:19.677 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-samsung.ko 2023-10-16T20:22:19.677 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sensor-hub.ko 2023-10-16T20:22:19.677 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sjoy.ko 2023-10-16T20:22:19.677 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 51249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sony.ko 2023-10-16T20:22:19.678 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-speedlink.ko 2023-10-16T20:22:19.678 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-steelseries.ko 2023-10-16T20:22:19.678 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sunplus.ko 2023-10-16T20:22:19.678 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-thrustmaster.ko 2023-10-16T20:22:19.678 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tivo.ko 2023-10-16T20:22:19.678 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tmff.ko 2023-10-16T20:22:19.679 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-topseed.ko 2023-10-16T20:22:19.679 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-twinhan.ko 2023-10-16T20:22:19.679 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 71073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-uclogic.ko 2023-10-16T20:22:19.679 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-waltop.ko 2023-10-16T20:22:19.679 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 143473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-wiimote.ko 2023-10-16T20:22:19.679 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-xinmo.ko 2023-10-16T20:22:19.680 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zpff.ko 2023-10-16T20:22:19.680 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zydacron.ko 2023-10-16T20:22:19.680 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 267825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid.ko 2023-10-16T20:22:19.680 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 32321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/uhid.ko 2023-10-16T20:22:19.680 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/usbhid/ 2023-10-16T20:22:19.681 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 125721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/usbhid/usbhid.ko 2023-10-16T20:22:19.681 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/usbhid/usbkbd.ko 2023-10-16T20:22:19.681 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/usbhid/usbmouse.ko 2023-10-16T20:22:19.681 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 217873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/wacom.ko 2023-10-16T20:22:19.681 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/ 2023-10-16T20:22:19.681 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 64201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_balloon.ko 2023-10-16T20:22:19.682 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 89625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_utils.ko 2023-10-16T20:22:19.682 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 305465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_vmbus.ko 2023-10-16T20:22:19.682 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ 2023-10-16T20:22:19.682 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 66521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru.ko 2023-10-16T20:22:19.682 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 61393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru3.ko 2023-10-16T20:22:19.682 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/acpi_power_meter.ko 2023-10-16T20:22:19.683 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7414.ko 2023-10-16T20:22:19.683 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7418.ko 2023-10-16T20:22:19.683 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adc128d818.ko 2023-10-16T20:22:19.683 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1025.ko 2023-10-16T20:22:19.683 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 97977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1026.ko 2023-10-16T20:22:19.683 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1029.ko 2023-10-16T20:22:19.684 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 45553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1031.ko 2023-10-16T20:22:19.684 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm9240.ko 2023-10-16T20:22:19.684 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ads7828.ko 2023-10-16T20:22:19.684 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7410.ko 2023-10-16T20:22:19.684 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7411.ko 2023-10-16T20:22:19.684 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 74433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7462.ko 2023-10-16T20:22:19.685 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 39585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7470.ko 2023-10-16T20:22:19.685 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 67929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7475.ko 2023-10-16T20:22:19.685 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7x10.ko 2023-10-16T20:22:19.685 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 39073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/amc6821.ko 2023-10-16T20:22:19.685 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 55265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/applesmc.ko 2023-10-16T20:22:19.685 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 49913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asb100.ko 2023-10-16T20:22:19.686 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 58617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asc7621.ko 2023-10-16T20:22:19.686 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 40889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asus_atk0110.ko 2023-10-16T20:22:19.686 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/atxp1.ko 2023-10-16T20:22:19.686 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28281 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/coretemp.ko 2023-10-16T20:22:19.686 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/da9052-hwmon.ko 2023-10-16T20:22:19.686 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/da9055-hwmon.ko 2023-10-16T20:22:19.687 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 44465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-10-16T20:22:19.687 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 99601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dme1737.ko 2023-10-16T20:22:19.687 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds1621.ko 2023-10-16T20:22:19.687 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds620.ko 2023-10-16T20:22:19.687 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc1403.ko 2023-10-16T20:22:19.687 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc2103.ko 2023-10-16T20:22:19.688 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc6w201.ko 2023-10-16T20:22:19.688 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 66985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71805f.ko 2023-10-16T20:22:19.688 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 94457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71882fg.ko 2023-10-16T20:22:19.688 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 41401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f75375s.ko 2023-10-16T20:22:19.688 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fam15h_power.ko 2023-10-16T20:22:19.688 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 55449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fschmd.ko 2023-10-16T20:22:19.689 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g760a.ko 2023-10-16T20:22:19.689 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g762.ko 2023-10-16T20:22:19.689 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 46801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl518sm.ko 2023-10-16T20:22:19.689 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 45529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl520sm.ko 2023-10-16T20:22:19.689 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/hih6130.ko 2023-10-16T20:22:19.690 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/hwmon-vid.ko 2023-10-16T20:22:19.690 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5500_temp.ko 2023-10-16T20:22:19.690 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5k_amb.ko 2023-10-16T20:22:19.690 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmaem.ko 2023-10-16T20:22:19.690 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmpex.ko 2023-10-16T20:22:19.691 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina209.ko 2023-10-16T20:22:19.691 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina2xx.ko 2023-10-16T20:22:19.691 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 130449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/it87.ko 2023-10-16T20:22:19.691 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/jc42.ko 2023-10-16T20:22:19.691 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k10temp.ko 2023-10-16T20:22:19.691 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k8temp.ko 2023-10-16T20:22:19.692 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lineage-pem.ko 2023-10-16T20:22:19.692 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 48601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm63.ko 2023-10-16T20:22:19.692 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm73.ko 2023-10-16T20:22:19.692 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm75.ko 2023-10-16T20:22:19.692 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm77.ko 2023-10-16T20:22:19.693 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm78.ko 2023-10-16T20:22:19.693 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm80.ko 2023-10-16T20:22:19.693 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm83.ko 2023-10-16T20:22:19.693 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 72833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm85.ko 2023-10-16T20:22:19.693 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 46153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm87.ko 2023-10-16T20:22:19.693 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 37913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm90.ko 2023-10-16T20:22:19.694 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm92.ko 2023-10-16T20:22:19.694 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 108841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm93.ko 2023-10-16T20:22:19.694 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95234.ko 2023-10-16T20:22:19.694 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95241.ko 2023-10-16T20:22:19.694 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95245.ko 2023-10-16T20:22:19.694 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc2945.ko 2023-10-16T20:22:19.695 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4151.ko 2023-10-16T20:22:19.695 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4215.ko 2023-10-16T20:22:19.695 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4222.ko 2023-10-16T20:22:19.695 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4245.ko 2023-10-16T20:22:19.695 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4260.ko 2023-10-16T20:22:19.695 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4261.ko 2023-10-16T20:22:19.696 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max16065.ko 2023-10-16T20:22:19.696 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1619.ko 2023-10-16T20:22:19.696 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1668.ko 2023-10-16T20:22:19.696 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max197.ko 2023-10-16T20:22:19.696 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6639.ko 2023-10-16T20:22:19.696 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6650.ko 2023-10-16T20:22:19.697 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6697.ko 2023-10-16T20:22:19.697 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/mc13783-adc.ko 2023-10-16T20:22:19.697 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/mcp3021.ko 2023-10-16T20:22:19.697 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2023-10-16T20:22:19.697 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 44129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6683.ko 2023-10-16T20:22:19.697 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 119817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775-core.ko 2023-10-16T20:22:19.698 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 60705 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775.ko 2023-10-16T20:22:19.698 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 59737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct7802.ko 2023-10-16T20:22:19.698 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 82337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87360.ko 2023-10-16T20:22:19.698 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 54433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87427.ko 2023-10-16T20:22:19.699 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pcf8591.ko 2023-10-16T20:22:19.699 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ 2023-10-16T20:22:19.699 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-10-16T20:22:19.699 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-10-16T20:22:19.699 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-10-16T20:22:19.700 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max16064.ko 2023-10-16T20:22:19.700 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max34440.ko 2023-10-16T20:22:19.700 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max8688.ko 2023-10-16T20:22:19.700 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-10-16T20:22:19.700 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 82577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-10-16T20:22:19.701 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-10-16T20:22:19.701 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-10-16T20:22:19.701 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-10-16T20:22:19.701 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-10-16T20:22:19.701 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/powr1220.ko 2023-10-16T20:22:19.702 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18705 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5627.ko 2023-10-16T20:22:19.702 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5636.ko 2023-10-16T20:22:19.702 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch56xx-common.ko 2023-10-16T20:22:19.702 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 32137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht15.ko 2023-10-16T20:22:19.702 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht21.ko 2023-10-16T20:22:19.703 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/shtc1.ko 2023-10-16T20:22:19.703 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 40105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sis5595.ko 2023-10-16T20:22:19.703 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47b397.ko 2023-10-16T20:22:19.703 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 39193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m1.ko 2023-10-16T20:22:19.703 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 37385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m192.ko 2023-10-16T20:22:19.704 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/thmc50.ko 2023-10-16T20:22:19.704 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp102.ko 2023-10-16T20:22:19.704 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp103.ko 2023-10-16T20:22:19.704 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp401.ko 2023-10-16T20:22:19.704 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp421.ko 2023-10-16T20:22:19.704 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via-cputemp.ko 2023-10-16T20:22:19.705 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 41945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via686a.ko 2023-10-16T20:22:19.705 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 52457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt1211.ko 2023-10-16T20:22:19.705 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 51545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt8231.ko 2023-10-16T20:22:19.705 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 57513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627ehf.ko 2023-10-16T20:22:19.705 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 78873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627hf.ko 2023-10-16T20:22:19.706 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 71345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83781d.ko 2023-10-16T20:22:19.706 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 63289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83791d.ko 2023-10-16T20:22:19.706 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 71321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83792d.ko 2023-10-16T20:22:19.706 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 101473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83793.ko 2023-10-16T20:22:19.706 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 119505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83795.ko 2023-10-16T20:22:19.707 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l785ts.ko 2023-10-16T20:22:19.707 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l786ng.ko 2023-10-16T20:22:19.707 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/wm831x-hwmon.ko 2023-10-16T20:22:19.707 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/wm8350-hwmon.ko 2023-10-16T20:22:19.707 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/ 2023-10-16T20:22:19.708 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/ 2023-10-16T20:22:19.708 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-10-16T20:22:19.708 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-10-16T20:22:19.708 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/ 2023-10-16T20:22:19.708 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ali1535.ko 2023-10-16T20:22:19.709 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ali1563.ko 2023-10-16T20:22:19.709 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2023-10-16T20:22:19.709 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-10-16T20:22:19.709 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-10-16T20:22:19.709 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-10-16T20:22:19.710 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2023-10-16T20:22:19.710 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 6929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-10-16T20:22:19.710 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2023-10-16T20:22:19.710 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 50201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-10-16T20:22:19.710 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-10-16T20:22:19.711 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-10-16T20:22:19.711 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-10-16T20:22:19.711 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-dln2.ko 2023-10-16T20:22:19.711 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-gpio.ko 2023-10-16T20:22:19.711 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 62817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-i801.ko 2023-10-16T20:22:19.712 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-isch.ko 2023-10-16T20:22:19.712 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 47889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-10-16T20:22:19.712 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-kempld.ko 2023-10-16T20:22:19.712 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-10-16T20:22:19.712 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-10-16T20:22:19.713 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ocores.ko 2023-10-16T20:22:19.713 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-parport.ko 2023-10-16T20:22:19.713 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-10-16T20:22:19.713 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 47241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-10-16T20:22:19.713 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2023-10-16T20:22:19.714 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-10-16T20:22:19.714 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-10-16T20:22:19.714 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-sis5595.ko 2023-10-16T20:22:19.714 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-sis630.ko 2023-10-16T20:22:19.714 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-10-16T20:22:19.715 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2023-10-16T20:22:19.715 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-10-16T20:22:19.715 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-via.ko 2023-10-16T20:22:19.715 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-10-16T20:22:19.715 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-10-16T20:22:19.715 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 47657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-xiic.ko 2023-10-16T20:22:19.716 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-mux.ko 2023-10-16T20:22:19.716 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-smbus.ko 2023-10-16T20:22:19.716 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-stub.ko 2023-10-16T20:22:19.716 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/muxes/ 2023-10-16T20:22:19.716 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2023-10-16T20:22:19.717 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2023-10-16T20:22:19.717 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2023-10-16T20:22:19.717 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ 2023-10-16T20:22:19.717 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ 2023-10-16T20:22:19.717 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 273609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_cm.ko 2023-10-16T20:22:19.718 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 879969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_core.ko 2023-10-16T20:22:19.718 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 69337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_umad.ko 2023-10-16T20:22:19.718 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 324217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-10-16T20:22:19.718 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 97985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/iw_cm.ko 2023-10-16T20:22:19.718 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 282273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_cm.ko 2023-10-16T20:22:19.809 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 66201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-10-16T20:22:19.809 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/ 2023-10-16T20:22:19.810 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4/ 2023-10-16T20:22:19.810 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 493073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-10-16T20:22:19.810 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4/ 2023-10-16T20:22:19.810 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 419737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-10-16T20:22:19.810 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5/ 2023-10-16T20:22:19.811 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 739025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-10-16T20:22:19.811 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca/ 2023-10-16T20:22:19.811 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 296625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-10-16T20:22:19.811 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/ocrdma/ 2023-10-16T20:22:19.811 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 153697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2023-10-16T20:22:19.812 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/usnic/ 2023-10-16T20:22:19.812 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 168969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2023-10-16T20:22:19.812 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ 2023-10-16T20:22:19.812 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib/ 2023-10-16T20:22:19.812 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 263081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-10-16T20:22:19.812 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser/ 2023-10-16T20:22:19.813 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 100241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-10-16T20:22:19.813 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert/ 2023-10-16T20:22:19.813 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 111425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-10-16T20:22:19.813 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp/ 2023-10-16T20:22:19.813 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 128753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-10-16T20:22:19.814 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt/ 2023-10-16T20:22:19.814 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 129529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-10-16T20:22:19.814 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/ 2023-10-16T20:22:19.814 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/evbug.ko 2023-10-16T20:22:19.814 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/ff-memless.ko 2023-10-16T20:22:19.815 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/ 2023-10-16T20:22:19.815 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-10-16T20:22:19.815 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/fm801-gp.ko 2023-10-16T20:22:19.815 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 41241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/gameport.ko 2023-10-16T20:22:19.815 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 41801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/joydev.ko 2023-10-16T20:22:19.816 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/ 2023-10-16T20:22:19.816 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/adp5520-keys.ko 2023-10-16T20:22:19.816 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/adp5588-keys.ko 2023-10-16T20:22:19.816 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/adp5589-keys.ko 2023-10-16T20:22:19.816 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2023-10-16T20:22:19.816 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/gpio_keys.ko 2023-10-16T20:22:19.817 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2023-10-16T20:22:19.817 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/lkkbd.ko 2023-10-16T20:22:19.817 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/lm8323.ko 2023-10-16T20:22:19.817 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/lm8333.ko 2023-10-16T20:22:19.817 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/matrix_keypad.ko 2023-10-16T20:22:19.818 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/max7359_keypad.ko 2023-10-16T20:22:19.818 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/mcs_touchkey.ko 2023-10-16T20:22:19.818 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2023-10-16T20:22:19.818 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/newtonkbd.ko 2023-10-16T20:22:19.818 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/opencores-kbd.ko 2023-10-16T20:22:19.819 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/qt1070.ko 2023-10-16T20:22:19.819 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/qt2160.ko 2023-10-16T20:22:19.819 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/samsung-keypad.ko 2023-10-16T20:22:19.819 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/stowaway.ko 2023-10-16T20:22:19.819 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/sunkbd.ko 2023-10-16T20:22:19.820 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/tca6416-keypad.ko 2023-10-16T20:22:19.820 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/tca8418_keypad.ko 2023-10-16T20:22:19.820 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/twl4030_keypad.ko 2023-10-16T20:22:19.820 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/xtkbd.ko 2023-10-16T20:22:19.820 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/matrix-keymap.ko 2023-10-16T20:22:19.820 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/ 2023-10-16T20:22:19.821 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/appletouch.ko 2023-10-16T20:22:19.821 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/bcm5974.ko 2023-10-16T20:22:19.821 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 123681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/cyapatp.ko 2023-10-16T20:22:19.821 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 89449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/elan_i2c.ko 2023-10-16T20:22:19.821 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/gpio_mouse.ko 2023-10-16T20:22:19.822 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 335977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/psmouse.ko 2023-10-16T20:22:19.822 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/sermouse.ko 2023-10-16T20:22:19.822 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-10-16T20:22:19.822 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_usb.ko 2023-10-16T20:22:19.822 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/vsxxxaa.ko 2023-10-16T20:22:19.823 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4/ 2023-10-16T20:22:19.823 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 109873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4/rmi_core.ko 2023-10-16T20:22:19.823 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/ 2023-10-16T20:22:19.823 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/altera_ps2.ko 2023-10-16T20:22:19.823 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/arc_ps2.ko 2023-10-16T20:22:19.824 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/ct82c710.ko 2023-10-16T20:22:19.824 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-10-16T20:22:19.824 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/parkbd.ko 2023-10-16T20:22:19.824 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/pcips2.ko 2023-10-16T20:22:19.824 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/ps2mult.ko 2023-10-16T20:22:19.824 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/serio_raw.ko 2023-10-16T20:22:19.825 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/serport.ko 2023-10-16T20:22:19.825 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/sparse-keymap.ko 2023-10-16T20:22:19.825 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/ 2023-10-16T20:22:19.825 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd/ 2023-10-16T20:22:19.825 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 39289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd/iommu_v2.ko 2023-10-16T20:22:19.826 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/ 2023-10-16T20:22:19.826 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-88pm860x.ko 2023-10-16T20:22:19.826 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-adp5520.ko 2023-10-16T20:22:19.826 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 59753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-bd2802.ko 2023-10-16T20:22:19.826 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-blinkm.ko 2023-10-16T20:22:19.827 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-da903x.ko 2023-10-16T20:22:19.827 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-da9052.ko 2023-10-16T20:22:19.827 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-gpio.ko 2023-10-16T20:22:19.827 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lm3530.ko 2023-10-16T20:22:19.827 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lm3533.ko 2023-10-16T20:22:19.827 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lm355x.ko 2023-10-16T20:22:19.828 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lm3642.ko 2023-10-16T20:22:19.828 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lp3944.ko 2023-10-16T20:22:19.828 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lp8788.ko 2023-10-16T20:22:19.828 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-max8997.ko 2023-10-16T20:22:19.828 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-mc13783.ko 2023-10-16T20:22:19.829 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-menf21bmc.ko 2023-10-16T20:22:19.829 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-pca9532.ko 2023-10-16T20:22:19.829 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-pca955x.ko 2023-10-16T20:22:19.829 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-pca963x.ko 2023-10-16T20:22:19.829 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-ss4200.ko 2023-10-16T20:22:19.830 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-tca6507.ko 2023-10-16T20:22:19.830 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-wm831x-status.ko 2023-10-16T20:22:19.830 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-wm8350.ko 2023-10-16T20:22:19.830 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ 2023-10-16T20:22:19.830 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-10-16T20:22:19.831 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-10-16T20:22:19.831 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-10-16T20:22:19.831 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-10-16T20:22:19.831 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-10-16T20:22:19.831 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-10-16T20:22:19.831 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-10-16T20:22:19.832 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-10-16T20:22:19.832 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/ 2023-10-16T20:22:19.832 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache/ 2023-10-16T20:22:19.832 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 654969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache/bcache.ko 2023-10-16T20:22:19.832 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 40577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-bio-prison.ko 2023-10-16T20:22:19.833 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 89601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-bufio.ko 2023-10-16T20:22:19.833 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 47529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache-smq.ko 2023-10-16T20:22:19.833 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 156009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache.ko 2023-10-16T20:22:19.833 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 106897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-crypt.ko 2023-10-16T20:22:19.833 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-delay.ko 2023-10-16T20:22:19.833 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 53857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-era.ko 2023-10-16T20:22:19.834 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-flakey.ko 2023-10-16T20:22:19.834 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 41593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-log-userspace.ko 2023-10-16T20:22:19.834 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 32737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-log.ko 2023-10-16T20:22:19.834 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 49545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-mirror.ko 2023-10-16T20:22:19.835 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 78033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-multipath.ko 2023-10-16T20:22:19.835 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-queue-length.ko 2023-10-16T20:22:19.835 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 70825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-raid.ko 2023-10-16T20:22:19.835 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-region-hash.ko 2023-10-16T20:22:19.835 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-round-robin.ko 2023-10-16T20:22:19.835 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-service-time.ko 2023-10-16T20:22:19.836 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 109593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-snapshot.ko 2023-10-16T20:22:19.836 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-switch.ko 2023-10-16T20:22:19.836 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 181865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-thin-pool.ko 2023-10-16T20:22:19.836 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 51737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-verity.ko 2023-10-16T20:22:19.836 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-zero.ko 2023-10-16T20:22:19.837 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/faulty.ko 2023-10-16T20:22:19.837 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/linear.ko 2023-10-16T20:22:19.837 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 32417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/multipath.ko 2023-10-16T20:22:19.837 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data/ 2023-10-16T20:22:19.837 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 199937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-10-16T20:22:19.838 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 37337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid0.ko 2023-10-16T20:22:19.838 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 113073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid1.ko 2023-10-16T20:22:19.838 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 158681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid10.ko 2023-10-16T20:22:19.838 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 395721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid456.ko 2023-10-16T20:22:19.838 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/ 2023-10-16T20:22:19.839 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/ 2023-10-16T20:22:19.839 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 178137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptbase.ko 2023-10-16T20:22:19.839 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 63385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptctl.ko 2023-10-16T20:22:19.839 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 44665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptfc.ko 2023-10-16T20:22:19.839 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 37225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptlan.ko 2023-10-16T20:22:19.840 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 123321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptsas.ko 2023-10-16T20:22:19.840 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 79041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptscsih.ko 2023-10-16T20:22:19.840 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 53081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptspi.ko 2023-10-16T20:22:19.840 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/ 2023-10-16T20:22:19.840 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/88pm800.ko 2023-10-16T20:22:19.841 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/88pm805.ko 2023-10-16T20:22:19.841 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/88pm80x.ko 2023-10-16T20:22:19.841 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/arizona-i2c.ko 2023-10-16T20:22:19.841 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 123721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/arizona.ko 2023-10-16T20:22:19.841 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/bcm590xx.ko 2023-10-16T20:22:19.842 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/cros_ec_dev.ko 2023-10-16T20:22:19.842 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/dln2.ko 2023-10-16T20:22:19.842 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/janz-cmodio.ko 2023-10-16T20:22:19.842 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 53585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/kempld-core.ko 2023-10-16T20:22:19.842 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lm3533-core.ko 2023-10-16T20:22:19.843 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lm3533-ctrlbank.ko 2023-10-16T20:22:19.843 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lp3943.ko 2023-10-16T20:22:19.843 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 54569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_ich.ko 2023-10-16T20:22:19.843 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_sch.ko 2023-10-16T20:22:19.843 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/max8907.ko 2023-10-16T20:22:19.844 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/mc13xxx-core.ko 2023-10-16T20:22:19.844 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/mc13xxx-i2c.ko 2023-10-16T20:22:19.844 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/menf21bmc.ko 2023-10-16T20:22:19.844 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/pcf50633-adc.ko 2023-10-16T20:22:19.844 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/pcf50633-gpio.ko 2023-10-16T20:22:19.844 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/pcf50633.ko 2023-10-16T20:22:19.845 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/rdc321x-southbridge.ko 2023-10-16T20:22:19.845 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/retu-mfd.ko 2023-10-16T20:22:19.845 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 80425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/si476x-core.ko 2023-10-16T20:22:19.845 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 48457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/sm501.ko 2023-10-16T20:22:19.845 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/ti_am335x_tscadc.ko 2023-10-16T20:22:19.846 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 51137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/tps65010.ko 2023-10-16T20:22:19.846 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/tps6507x.ko 2023-10-16T20:22:19.846 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/viperboard.ko 2023-10-16T20:22:19.846 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/vx855.ko 2023-10-16T20:22:19.846 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/wl1273-core.ko 2023-10-16T20:22:19.846 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 44345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/wm8994.ko 2023-10-16T20:22:19.847 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ 2023-10-16T20:22:19.847 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ad525x_dpot-i2c.ko 2023-10-16T20:22:19.847 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 48489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ad525x_dpot.ko 2023-10-16T20:22:19.847 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl/ 2023-10-16T20:22:19.847 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 46585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-10-16T20:22:19.848 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds9802als.ko 2023-10-16T20:22:19.848 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds990x.ko 2023-10-16T20:22:19.848 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 45593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/bh1770glc.ko 2023-10-16T20:22:19.848 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/c2port/ 2023-10-16T20:22:19.848 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2023-10-16T20:22:19.849 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35281 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/c2port/core.ko 2023-10-16T20:22:19.849 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710/ 2023-10-16T20:22:19.849 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710/cb710.ko 2023-10-16T20:22:19.849 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ds1682.ko 2023-10-16T20:22:19.849 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/dummy-irq.ko 2023-10-16T20:22:19.850 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/echo/ 2023-10-16T20:22:19.850 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/echo/echo.ko 2023-10-16T20:22:19.850 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/ 2023-10-16T20:22:19.850 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 37849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/at24.ko 2023-10-16T20:22:19.850 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom.ko 2023-10-16T20:22:19.851 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-10-16T20:22:19.851 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/max6875.ko 2023-10-16T20:22:19.851 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/enclosure.ko 2023-10-16T20:22:19.851 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/genwqe/ 2023-10-16T20:22:19.851 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 152201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/genwqe/genwqe_card.ko 2023-10-16T20:22:19.851 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/hmc6352.ko 2023-10-16T20:22:19.852 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/hpilo.ko 2023-10-16T20:22:19.852 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ibmasm/ 2023-10-16T20:22:19.852 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 72129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ibmasm/ibmasm.ko 2023-10-16T20:22:19.852 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ics932s401.ko 2023-10-16T20:22:19.852 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29003.ko 2023-10-16T20:22:19.853 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29020.ko 2023-10-16T20:22:19.853 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/ 2023-10-16T20:22:19.853 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 42513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-10-16T20:22:19.853 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-10-16T20:22:19.853 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/ 2023-10-16T20:22:19.853 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 125209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-me.ko 2023-10-16T20:22:19.854 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 56521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-txe.ko 2023-10-16T20:22:19.854 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 310113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei.ko 2023-10-16T20:22:19.854 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/phantom.ko 2023-10-16T20:22:19.854 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ti-st/ 2023-10-16T20:22:19.854 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 74265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/ti-st/st_drv.ko 2023-10-16T20:22:19.855 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_7xx1.ko 2023-10-16T20:22:19.855 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_core.ko 2023-10-16T20:22:19.855 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tsl2550.ko 2023-10-16T20:22:19.855 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 48697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_balloon.ko 2023-10-16T20:22:19.855 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci/ 2023-10-16T20:22:19.856 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 183609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-10-16T20:22:19.856 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ 2023-10-16T20:22:19.856 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding/ 2023-10-16T20:22:19.856 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 490129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding/bonding.ko 2023-10-16T20:22:19.856 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/ 2023-10-16T20:22:19.857 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53/ 2023-10-16T20:22:19.857 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 106033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53/b53_common.ko 2023-10-16T20:22:19.857 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 40297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/bcm-sf2.ko 2023-10-16T20:22:19.857 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6060.ko 2023-10-16T20:22:19.857 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx/ 2023-10-16T20:22:19.858 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 453737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-10-16T20:22:19.858 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dummy.ko 2023-10-16T20:22:19.858 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/eql.ko 2023-10-16T20:22:19.858 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ 2023-10-16T20:22:19.858 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/ 2023-10-16T20:22:19.859 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 45785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-10-16T20:22:19.859 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 42105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-10-16T20:22:19.859 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 130553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-10-16T20:22:19.859 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 68849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-10-16T20:22:19.859 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/ 2023-10-16T20:22:19.859 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 37649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/8390.ko 2023-10-16T20:22:19.860 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 59689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-10-16T20:22:19.860 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-10-16T20:22:19.860 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 99113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-10-16T20:22:19.860 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec/ 2023-10-16T20:22:19.861 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 54521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-10-16T20:22:19.861 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere/ 2023-10-16T20:22:19.861 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 69201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere/et131x.ko 2023-10-16T20:22:19.861 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon/ 2023-10-16T20:22:19.861 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 59585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-10-16T20:22:19.862 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera/ 2023-10-16T20:22:19.862 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 41409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-10-16T20:22:19.862 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/ 2023-10-16T20:22:19.862 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 54609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-10-16T20:22:19.862 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 49289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-10-16T20:22:19.862 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 87297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-10-16T20:22:19.863 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/ 2023-10-16T20:22:19.863 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx/ 2023-10-16T20:22:19.863 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 97337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-10-16T20:22:19.863 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c/ 2023-10-16T20:22:19.863 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 104673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-10-16T20:22:19.864 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e/ 2023-10-16T20:22:19.864 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 90257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-10-16T20:22:19.864 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/ 2023-10-16T20:22:19.864 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 81569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-10-16T20:22:19.864 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 69729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-10-16T20:22:19.864 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/ 2023-10-16T20:22:19.865 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 78401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-10-16T20:22:19.865 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 176761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-10-16T20:22:19.865 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2023-10-16T20:22:19.865 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 1421497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-10-16T20:22:19.865 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 128569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-10-16T20:22:19.865 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet/ 2023-10-16T20:22:19.866 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 132089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-10-16T20:22:19.866 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 312081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-10-16T20:22:19.866 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/ 2023-10-16T20:22:19.866 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna/ 2023-10-16T20:22:19.866 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 323513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-10-16T20:22:19.867 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/ 2023-10-16T20:22:19.867 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb/ 2023-10-16T20:22:19.867 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 193497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-10-16T20:22:19.867 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2023-10-16T20:22:19.867 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 347417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-10-16T20:22:19.868 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2023-10-16T20:22:19.868 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 1044433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-10-16T20:22:19.868 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2023-10-16T20:22:19.868 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 166193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-10-16T20:22:19.868 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2023-10-16T20:22:19.869 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-10-16T20:22:19.869 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/ 2023-10-16T20:22:19.869 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic/ 2023-10-16T20:22:19.869 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 204345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-10-16T20:22:19.869 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/ 2023-10-16T20:22:19.870 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/ 2023-10-16T20:22:19.870 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 69449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-10-16T20:22:19.870 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 61489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-10-16T20:22:19.870 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 141225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-10-16T20:22:19.870 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 51497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-10-16T20:22:19.870 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 62817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-10-16T20:22:19.871 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-10-16T20:22:19.871 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/ 2023-10-16T20:22:19.871 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 60873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-10-16T20:22:19.871 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 62833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-10-16T20:22:19.871 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 45721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dnet.ko 2023-10-16T20:22:19.872 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ec_bhf.ko 2023-10-16T20:22:19.872 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/ 2023-10-16T20:22:19.872 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet/ 2023-10-16T20:22:19.872 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 281137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-10-16T20:22:19.872 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 49457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ethoc.ko 2023-10-16T20:22:19.873 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 51737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/fealnx.ko 2023-10-16T20:22:19.873 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/fujitsu/ 2023-10-16T20:22:19.873 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 45345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2023-10-16T20:22:19.873 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ 2023-10-16T20:22:19.873 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 90465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e100.ko 2023-10-16T20:22:19.873 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000/ 2023-10-16T20:22:19.874 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 295577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-10-16T20:22:19.874 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e/ 2023-10-16T20:22:19.874 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 584497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-10-16T20:22:19.874 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k/ 2023-10-16T20:22:19.874 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 246209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-10-16T20:22:19.875 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e/ 2023-10-16T20:22:19.875 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 925697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-10-16T20:22:19.875 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf/ 2023-10-16T20:22:19.875 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 302849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-10-16T20:22:19.875 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb/ 2023-10-16T20:22:19.876 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 514681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-10-16T20:22:19.876 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf/ 2023-10-16T20:22:19.876 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 93593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-10-16T20:22:19.876 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe/ 2023-10-16T20:22:19.876 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 734305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-10-16T20:22:19.877 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf/ 2023-10-16T20:22:19.877 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 151769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-10-16T20:22:19.877 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 86921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/jme.ko 2023-10-16T20:22:19.877 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/ 2023-10-16T20:22:19.877 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-10-16T20:22:19.877 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 102289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/skge.ko 2023-10-16T20:22:19.878 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 122649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-10-16T20:22:19.878 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/ 2023-10-16T20:22:19.878 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/ 2023-10-16T20:22:19.878 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 727601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-10-16T20:22:19.878 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 278937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-10-16T20:22:19.879 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/ 2023-10-16T20:22:19.879 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2023-10-16T20:22:19.879 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 1113993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-10-16T20:22:19.879 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel/ 2023-10-16T20:22:19.879 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 53105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel/ks8842.ko 2023-10-16T20:22:19.880 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 40993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2023-10-16T20:22:19.880 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2023-10-16T20:22:19.880 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 95665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-10-16T20:22:19.880 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/ 2023-10-16T20:22:19.880 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge/ 2023-10-16T20:22:19.881 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 108649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-10-16T20:22:19.881 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/ 2023-10-16T20:22:19.881 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 73441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-10-16T20:22:19.881 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 47777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-10-16T20:22:19.881 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion/ 2023-10-16T20:22:19.881 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 150905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-10-16T20:22:19.882 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia/ 2023-10-16T20:22:19.882 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 136721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-10-16T20:22:19.882 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/ 2023-10-16T20:22:19.882 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 50481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-10-16T20:22:19.882 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 43145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-10-16T20:22:19.883 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/ 2023-10-16T20:22:19.883 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen/ 2023-10-16T20:22:19.883 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 227897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-10-16T20:22:19.883 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 87081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-10-16T20:22:19.883 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2023-10-16T20:22:19.884 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 672049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-10-16T20:22:19.884 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc/ 2023-10-16T20:22:19.884 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 43017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-10-16T20:22:19.884 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/ 2023-10-16T20:22:19.884 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 67385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-10-16T20:22:19.885 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 83985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-10-16T20:22:19.885 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/atp.ko 2023-10-16T20:22:19.885 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 189801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-10-16T20:22:19.885 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/samsung/ 2023-10-16T20:22:19.885 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/samsung/sxgbe/ 2023-10-16T20:22:19.886 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 104241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2023-10-16T20:22:19.886 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc/ 2023-10-16T20:22:19.886 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 1053921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-10-16T20:22:19.886 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan/ 2023-10-16T20:22:19.886 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 47097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-10-16T20:22:19.886 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/ 2023-10-16T20:22:19.887 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 54817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis190.ko 2023-10-16T20:22:19.887 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 63337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis900.ko 2023-10-16T20:22:19.887 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/ 2023-10-16T20:22:19.887 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 61601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-10-16T20:22:19.887 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 67193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-10-16T20:22:19.888 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 73409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-10-16T20:22:19.888 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 68177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-10-16T20:22:19.888 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/ 2023-10-16T20:22:19.888 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/ 2023-10-16T20:22:19.888 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2023-10-16T20:22:19.889 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 39905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-10-16T20:22:19.889 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2023-10-16T20:22:19.889 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 486961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-10-16T20:22:19.889 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/ 2023-10-16T20:22:19.889 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 103337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/cassini.ko 2023-10-16T20:22:19.890 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 166161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/niu.ko 2023-10-16T20:22:19.890 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 66329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sungem.ko 2023-10-16T20:22:19.890 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 78801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-10-16T20:22:19.890 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti/ 2023-10-16T20:22:19.890 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 50057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-10-16T20:22:19.891 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti/ 2023-10-16T20:22:19.891 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 86529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti/tlan.ko 2023-10-16T20:22:19.891 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/ 2023-10-16T20:22:19.891 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 80273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-10-16T20:22:19.891 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 83633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-10-16T20:22:19.892 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/ 2023-10-16T20:22:19.892 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 46241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-10-16T20:22:19.892 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-10-16T20:22:19.892 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom/ 2023-10-16T20:22:19.892 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 60057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-10-16T20:22:19.893 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/fddi/ 2023-10-16T20:22:19.893 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 41065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/fddi/defxx.ko 2023-10-16T20:22:19.893 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/fddi/skfp/ 2023-10-16T20:22:19.893 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 216985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/fddi/skfp/skfp.ko 2023-10-16T20:22:19.893 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 77345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/geneve.ko 2023-10-16T20:22:19.893 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv/ 2023-10-16T20:22:19.894 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 235521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-10-16T20:22:19.894 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154/ 2023-10-16T20:22:19.894 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154/fakelb.ko 2023-10-16T20:22:19.894 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ifb.ko 2023-10-16T20:22:19.894 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan/ 2023-10-16T20:22:19.895 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 83769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan/ipvlan.ko 2023-10-16T20:22:19.895 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 62209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvlan.ko 2023-10-16T20:22:19.895 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvtap.ko 2023-10-16T20:22:19.895 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/ 2023-10-16T20:22:19.895 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-10-16T20:22:19.896 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-10-16T20:22:19.896 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-gpio.ko 2023-10-16T20:22:19.896 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-regmap.ko 2023-10-16T20:22:19.896 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio.ko 2023-10-16T20:22:19.896 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mii.ko 2023-10-16T20:22:19.896 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 43953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/netconsole.ko 2023-10-16T20:22:19.897 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/nlmon.ko 2023-10-16T20:22:19.897 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/ 2023-10-16T20:22:19.897 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs-lynx.ko 2023-10-16T20:22:19.897 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 38977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-10-16T20:22:19.897 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/ 2023-10-16T20:22:19.898 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/amd.ko 2023-10-16T20:22:19.898 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 50097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-10-16T20:22:19.898 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 44505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm7xxx.ko 2023-10-16T20:22:19.898 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm87xx.ko 2023-10-16T20:22:19.898 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 42241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/broadcom.ko 2023-10-16T20:22:19.899 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/cicada.ko 2023-10-16T20:22:19.899 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/davicom.ko 2023-10-16T20:22:19.899 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/et1011c.ko 2023-10-16T20:22:19.899 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/icplus.ko 2023-10-16T20:22:19.899 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/lxt.ko 2023-10-16T20:22:19.900 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 82049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/marvell.ko 2023-10-16T20:22:19.900 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 76177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/micrel.ko 2023-10-16T20:22:19.900 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/national.ko 2023-10-16T20:22:19.900 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 97857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/phylink.ko 2023-10-16T20:22:19.900 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/qsemi.ko 2023-10-16T20:22:19.900 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 50065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/realtek.ko 2023-10-16T20:22:19.901 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/smsc.ko 2023-10-16T20:22:19.901 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/ste10Xp.ko 2023-10-16T20:22:19.901 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/vitesse.ko 2023-10-16T20:22:19.901 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/plip/ 2023-10-16T20:22:19.901 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 43569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/plip/plip.ko 2023-10-16T20:22:19.902 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ 2023-10-16T20:22:19.902 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/bsd_comp.ko 2023-10-16T20:22:19.902 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_async.ko 2023-10-16T20:22:19.902 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_deflate.ko 2023-10-16T20:22:19.902 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_mppe.ko 2023-10-16T20:22:19.903 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_synctty.ko 2023-10-16T20:22:19.903 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 42545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppoe.ko 2023-10-16T20:22:19.903 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppox.ko 2023-10-16T20:22:19.903 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 32689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pptp.ko 2023-10-16T20:22:19.903 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/sb1000.ko 2023-10-16T20:22:19.904 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/ 2023-10-16T20:22:19.904 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 42465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/slip.ko 2023-10-16T20:22:19.904 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/sungem_phy.ko 2023-10-16T20:22:19.904 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 64385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/tap.ko 2023-10-16T20:22:19.904 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/ 2023-10-16T20:22:19.904 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 106169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team.ko 2023-10-16T20:22:19.905 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_activebackup.ko 2023-10-16T20:22:19.905 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_broadcast.ko 2023-10-16T20:22:19.905 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-10-16T20:22:19.905 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_random.ko 2023-10-16T20:22:19.905 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-10-16T20:22:19.906 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 87017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/veth.ko 2023-10-16T20:22:19.906 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3/ 2023-10-16T20:22:19.906 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 144297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-10-16T20:22:19.906 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan/ 2023-10-16T20:22:19.906 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 297329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan/vxlan.ko 2023-10-16T20:22:19.907 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback/ 2023-10-16T20:22:19.907 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 144649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback/xen-netback.ko 2023-10-16T20:22:19.907 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb/ 2023-10-16T20:22:19.907 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb/ntb.ko 2023-10-16T20:22:19.907 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/ 2023-10-16T20:22:19.907 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/ 2023-10-16T20:22:19.908 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 264937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme-core.ko 2023-10-16T20:22:19.908 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 102209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme.ko 2023-10-16T20:22:19.908 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/ 2023-10-16T20:22:19.908 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 133841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport.ko 2023-10-16T20:22:19.908 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_cs.ko 2023-10-16T20:22:19.909 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 84465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_pc.ko 2023-10-16T20:22:19.909 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_serial.ko 2023-10-16T20:22:19.909 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/ 2023-10-16T20:22:19.909 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug/ 2023-10-16T20:22:19.909 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-10-16T20:22:19.910 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2023-10-16T20:22:19.910 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2023-10-16T20:22:19.910 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/pci-stub.ko 2023-10-16T20:22:19.910 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 46529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/xen-pcifront.ko 2023-10-16T20:22:19.910 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/ 2023-10-16T20:22:19.911 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/i82092.ko 2023-10-16T20:22:19.911 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 145809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/pcmcia.ko 2023-10-16T20:22:19.911 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 62137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/pcmcia_core.ko 2023-10-16T20:22:19.911 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2023-10-16T20:22:19.911 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/pd6729.ko 2023-10-16T20:22:19.911 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 112889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/yenta_socket.ko 2023-10-16T20:22:19.912 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/phy/ 2023-10-16T20:22:19.912 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/phy/broadcom/ 2023-10-16T20:22:19.912 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2023-10-16T20:22:19.912 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/phy/samsung/ 2023-10-16T20:22:19.912 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2023-10-16T20:22:19.913 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/ 2023-10-16T20:22:19.913 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel/ 2023-10-16T20:22:19.913 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 70153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-10-16T20:22:19.913 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/ 2023-10-16T20:22:19.913 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/chrome/ 2023-10-16T20:22:19.914 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 50889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2023-10-16T20:22:19.914 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/chrome/cros_ec.ko 2023-10-16T20:22:19.914 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2023-10-16T20:22:19.914 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2023-10-16T20:22:19.914 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2023-10-16T20:22:19.915 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2023-10-16T20:22:19.915 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2023-10-16T20:22:19.915 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/ 2023-10-16T20:22:19.915 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 67097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acer-wmi.ko 2023-10-16T20:22:19.915 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 39145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acerhdf.ko 2023-10-16T20:22:19.915 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/apple-gmux.ko 2023-10-16T20:22:19.916 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 59665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-laptop.ko 2023-10-16T20:22:19.916 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-10-16T20:22:19.916 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 127569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-wmi.ko 2023-10-16T20:22:19.916 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/classmate-laptop.ko 2023-10-16T20:22:19.916 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 49121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-10-16T20:22:19.917 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-10-16T20:22:19.917 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 40265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-10-16T20:22:19.917 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-10-16T20:22:19.917 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/hdaps.ko 2023-10-16T20:22:19.917 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/ibm_rtl.ko 2023-10-16T20:22:19.918 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel/ 2023-10-16T20:22:19.918 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-10-16T20:22:19.918 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2023-10-16T20:22:19.918 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 43497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel_ips.ko 2023-10-16T20:22:19.918 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/msi-wmi.ko 2023-10-16T20:22:19.919 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10377 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/mxm-wmi.ko 2023-10-16T20:22:19.919 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-10-16T20:22:19.919 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 42329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-laptop.ko 2023-10-16T20:22:19.919 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-q10.ko 2023-10-16T20:22:19.919 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/topstar-laptop.ko 2023-10-16T20:22:19.920 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-10-16T20:22:19.920 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_haps.ko 2023-10-16T20:22:19.920 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi-bmof.ko 2023-10-16T20:22:19.920 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 57569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi.ko 2023-10-16T20:22:19.920 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/ 2023-10-16T20:22:19.920 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/ 2023-10-16T20:22:19.921 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/88pm860x_battery.ko 2023-10-16T20:22:19.921 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/88pm860x_charger.ko 2023-10-16T20:22:19.921 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 55057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/bq2415x_charger.ko 2023-10-16T20:22:19.921 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 54081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/bq24190_charger.ko 2023-10-16T20:22:19.921 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/bq24735-charger.ko 2023-10-16T20:22:19.922 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/cros_peripheral_charger.ko 2023-10-16T20:22:19.922 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/da9030_battery.ko 2023-10-16T20:22:19.922 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/da9052-battery.ko 2023-10-16T20:22:19.922 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29281 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/ds2760_battery.ko 2023-10-16T20:22:19.922 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/ds2780_battery.ko 2023-10-16T20:22:19.923 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/ds2781_battery.ko 2023-10-16T20:22:19.923 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/ds2782_battery.ko 2023-10-16T20:22:19.923 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/gpio-charger.ko 2023-10-16T20:22:19.923 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/isp1704_charger.ko 2023-10-16T20:22:19.923 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/lp8727_charger.ko 2023-10-16T20:22:19.924 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/max14577_charger.ko 2023-10-16T20:22:19.924 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/max17040_battery.ko 2023-10-16T20:22:19.924 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/max17042_battery.ko 2023-10-16T20:22:19.924 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/max8903_charger.ko 2023-10-16T20:22:19.924 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/max8925_power.ko 2023-10-16T20:22:19.925 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/pcf50633-charger.ko 2023-10-16T20:22:19.925 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 38009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/sbs-battery.ko 2023-10-16T20:22:19.925 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/test_power.ko 2023-10-16T20:22:19.925 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/tps65090-charger.ko 2023-10-16T20:22:19.925 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/wm831x_backup.ko 2023-10-16T20:22:19.926 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/wm831x_power.ko 2023-10-16T20:22:19.926 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/power/supply/wm8350_power.ko 2023-10-16T20:22:19.926 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/ 2023-10-16T20:22:19.926 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 57729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_common.ko 2023-10-16T20:22:19.926 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_msr.ko 2023-10-16T20:22:19.927 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/ 2023-10-16T20:22:19.927 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/ 2023-10-16T20:22:19.927 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-gpio.ko 2023-10-16T20:22:19.927 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-ldisc.ko 2023-10-16T20:22:19.927 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps_parport.ko 2023-10-16T20:22:19.929 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 41873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/pps_core.ko 2023-10-16T20:22:19.929 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ 2023-10-16T20:22:19.929 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 66385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp.ko 2023-10-16T20:22:19.929 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp_kvm.ko 2023-10-16T20:22:19.930 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/ 2023-10-16T20:22:19.930 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-88pm80x.ko 2023-10-16T20:22:19.930 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-88pm860x.ko 2023-10-16T20:22:19.931 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-bq32k.ko 2023-10-16T20:22:19.931 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-da9052.ko 2023-10-16T20:22:19.931 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-da9055.ko 2023-10-16T20:22:19.931 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-da9063.ko 2023-10-16T20:22:19.931 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1286.ko 2023-10-16T20:22:19.932 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 68577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1307.ko 2023-10-16T20:22:19.932 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1374.ko 2023-10-16T20:22:19.932 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1511.ko 2023-10-16T20:22:19.932 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1553.ko 2023-10-16T20:22:19.932 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1672.ko 2023-10-16T20:22:19.933 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1742.ko 2023-10-16T20:22:19.933 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds2404.ko 2023-10-16T20:22:19.933 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23705 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds3232.ko 2023-10-16T20:22:19.933 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-em3027.ko 2023-10-16T20:22:19.933 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-fm3130.ko 2023-10-16T20:22:19.934 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl12022.ko 2023-10-16T20:22:19.934 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl1208.ko 2023-10-16T20:22:19.934 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-lp8788.ko 2023-10-16T20:22:19.934 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 38097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m41t80.ko 2023-10-16T20:22:19.934 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t35.ko 2023-10-16T20:22:19.935 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t59.ko 2023-10-16T20:22:19.935 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t86.ko 2023-10-16T20:22:19.935 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-max6900.ko 2023-10-16T20:22:19.935 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-max8907.ko 2023-10-16T20:22:19.935 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-max8925.ko 2023-10-16T20:22:19.935 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-max8997.ko 2023-10-16T20:22:19.936 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-max8998.ko 2023-10-16T20:22:19.936 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-mc13xxx.ko 2023-10-16T20:22:19.936 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-msm6242.ko 2023-10-16T20:22:19.936 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-palmas.ko 2023-10-16T20:22:19.936 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 46633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf2127.ko 2023-10-16T20:22:19.936 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf50633.ko 2023-10-16T20:22:19.937 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf85063.ko 2023-10-16T20:22:19.937 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8523.ko 2023-10-16T20:22:19.937 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8563.ko 2023-10-16T20:22:19.937 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8583.ko 2023-10-16T20:22:19.937 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rc5t583.ko 2023-10-16T20:22:19.938 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rp5c01.ko 2023-10-16T20:22:19.938 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rs5c372.ko 2023-10-16T20:22:19.938 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-10-16T20:22:19.938 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8025.ko 2023-10-16T20:22:19.938 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8581.ko 2023-10-16T20:22:19.939 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-s35390a.ko 2023-10-16T20:22:19.939 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-10-16T20:22:19.939 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-tps6586x.ko 2023-10-16T20:22:19.939 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-tps65910.ko 2023-10-16T20:22:19.939 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-wm831x.ko 2023-10-16T20:22:19.940 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-wm8350.ko 2023-10-16T20:22:19.940 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-x1205.ko 2023-10-16T20:22:19.940 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ 2023-10-16T20:22:19.940 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 77617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-9xxx.ko 2023-10-16T20:22:19.940 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 57665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-sas.ko 2023-10-16T20:22:19.940 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 55169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-xxxx.ko 2023-10-16T20:22:19.941 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 135185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/BusLogic.ko 2023-10-16T20:22:19.941 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/a100u2w.ko 2023-10-16T20:22:19.941 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid/ 2023-10-16T20:22:19.941 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 272305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid/aacraid.ko 2023-10-16T20:22:19.941 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 100617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/advansys.ko 2023-10-16T20:22:19.942 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/ 2023-10-16T20:22:19.942 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 356265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-10-16T20:22:19.942 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 295801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-10-16T20:22:19.942 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic94xx/ 2023-10-16T20:22:19.942 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 189185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic94xx/aic94xx.ko 2023-10-16T20:22:19.943 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/am53c974.ko 2023-10-16T20:22:19.943 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr/ 2023-10-16T20:22:19.943 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 106881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-10-16T20:22:19.943 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 38185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/atp870u.ko 2023-10-16T20:22:19.943 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi/ 2023-10-16T20:22:19.943 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 246985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-10-16T20:22:19.944 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa/ 2023-10-16T20:22:19.944 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 1171945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa/bfa.ko 2023-10-16T20:22:19.944 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc/ 2023-10-16T20:22:19.944 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 237273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-10-16T20:22:19.944 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i/ 2023-10-16T20:22:19.945 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 115369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-10-16T20:22:19.945 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 40585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ch.ko 2023-10-16T20:22:19.945 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor/ 2023-10-16T20:22:19.945 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 308921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor/csiostor.ko 2023-10-16T20:22:19.945 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/ 2023-10-16T20:22:19.945 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i/ 2023-10-16T20:22:19.946 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 76145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-10-16T20:22:19.946 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i/ 2023-10-16T20:22:19.946 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 101449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-10-16T20:22:19.946 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 146689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-10-16T20:22:19.946 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 75553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dc395x.ko 2023-10-16T20:22:19.947 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 38137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dmx3191d.ko 2023-10-16T20:22:19.947 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r/ 2023-10-16T20:22:19.947 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 212425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r/esas2r.ko 2023-10-16T20:22:19.947 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 60505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esp_scsi.ko 2023-10-16T20:22:19.947 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/ 2023-10-16T20:22:19.947 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 78209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/fcoe.ko 2023-10-16T20:22:19.948 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 158505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-10-16T20:22:19.948 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fdomain.ko 2023-10-16T20:22:19.948 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic/ 2023-10-16T20:22:19.948 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 263489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic/fnic.ko 2023-10-16T20:22:19.948 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 201817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hpsa.ko 2023-10-16T20:22:19.949 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 51209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hptiop.ko 2023-10-16T20:22:19.949 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 49041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hv_storvsc.ko 2023-10-16T20:22:19.949 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/imm.ko 2023-10-16T20:22:19.949 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 44057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/initio.ko 2023-10-16T20:22:19.949 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 266537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ipr.ko 2023-10-16T20:22:19.950 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 101881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ips.ko 2023-10-16T20:22:19.950 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci/ 2023-10-16T20:22:19.950 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 317153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci/isci.ko 2023-10-16T20:22:19.950 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-10-16T20:22:19.950 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 52961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_tcp.ko 2023-10-16T20:22:19.951 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc/ 2023-10-16T20:22:19.951 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 314865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc/libfc.ko 2023-10-16T20:22:19.951 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 145585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi.ko 2023-10-16T20:22:19.951 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 53521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi_tcp.ko 2023-10-16T20:22:19.951 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas/ 2023-10-16T20:22:19.952 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 216665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas/libsas.ko 2023-10-16T20:22:19.952 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc/ 2023-10-16T20:22:19.952 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 2197249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc/lpfc.ko 2023-10-16T20:22:19.952 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/ 2023-10-16T20:22:19.952 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 73569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-10-16T20:22:19.952 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-10-16T20:22:19.953 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 310617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-10-16T20:22:19.953 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 84721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid.ko 2023-10-16T20:22:19.953 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas/ 2023-10-16T20:22:19.953 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 659777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-10-16T20:22:19.953 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas/ 2023-10-16T20:22:19.954 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 119561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas/mvsas.ko 2023-10-16T20:22:19.954 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 78129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvumi.ko 2023-10-16T20:22:19.954 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pcmcia/ 2023-10-16T20:22:19.954 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 81625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2023-10-16T20:22:19.954 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2023-10-16T20:22:19.955 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2023-10-16T20:22:19.955 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2023-10-16T20:22:19.955 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001/ 2023-10-16T20:22:19.955 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 364105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-10-16T20:22:19.955 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 123617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pmcraid.ko 2023-10-16T20:22:19.955 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ppa.ko 2023-10-16T20:22:19.956 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 55361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla1280.ko 2023-10-16T20:22:19.956 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/ 2023-10-16T20:22:19.956 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 1876625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-10-16T20:22:19.956 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 85289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-10-16T20:22:19.956 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx/ 2023-10-16T20:22:19.956 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 559033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-10-16T20:22:19.957 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qlogicfas408.ko 2023-10-16T20:22:19.957 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/raid_class.ko 2023-10-16T20:22:19.957 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 238825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_debug.ko 2023-10-16T20:22:19.957 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 173937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_fc.ko 2023-10-16T20:22:19.957 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 279313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-10-16T20:22:19.958 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 97729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_sas.ko 2023-10-16T20:22:19.958 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 73881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_spi.ko 2023-10-16T20:22:19.958 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 44993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_srp.ko 2023-10-16T20:22:19.958 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ses.ko 2023-10-16T20:22:19.958 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 115889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/st.ko 2023-10-16T20:22:19.959 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 46825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/stex.ko 2023-10-16T20:22:19.959 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2/ 2023-10-16T20:22:19.959 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 162121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-10-16T20:22:19.959 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 42297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/virtio_scsi.ko 2023-10-16T20:22:19.959 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 50321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/vmw_pvscsi.ko 2023-10-16T20:22:19.959 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 40025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/wd719x.ko 2023-10-16T20:22:19.960 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 50025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/xen-scsifront.ko 2023-10-16T20:22:19.960 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb/ 2023-10-16T20:22:19.960 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 129441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb/ssb.ko 2023-10-16T20:22:19.960 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/ 2023-10-16T20:22:19.960 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi/ 2023-10-16T20:22:19.961 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 736601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-10-16T20:22:19.961 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback/ 2023-10-16T20:22:19.961 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 49985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback/tcm_loop.ko 2023-10-16T20:22:19.961 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 39713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_file.ko 2023-10-16T20:22:19.961 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 43025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_iblock.ko 2023-10-16T20:22:19.962 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 930321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_mod.ko 2023-10-16T20:22:19.962 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 44305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_pscsi.ko 2023-10-16T20:22:19.962 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 122521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_user.ko 2023-10-16T20:22:19.962 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc/ 2023-10-16T20:22:19.962 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 75593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-10-16T20:22:19.963 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/ 2023-10-16T20:22:19.963 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/ 2023-10-16T20:22:19.963 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/ 2023-10-16T20:22:19.963 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-10-16T20:22:19.963 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-10-16T20:22:19.963 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-10-16T20:22:19.964 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-10-16T20:22:19.964 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-10-16T20:22:19.964 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-10-16T20:22:19.964 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-10-16T20:22:19.964 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20281 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-10-16T20:22:19.965 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-10-16T20:22:19.965 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-10-16T20:22:19.965 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-10-16T20:22:19.965 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 37065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-10-16T20:22:19.965 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_powerclamp.ko 2023-10-16T20:22:19.965 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-10-16T20:22:19.966 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-10-16T20:22:19.966 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-10-16T20:22:19.966 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ 2023-10-16T20:22:19.966 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless/ 2023-10-16T20:22:19.966 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 78425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-10-16T20:22:19.967 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 57865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/moxa.ko 2023-10-16T20:22:19.967 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 50953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/mxser.ko 2023-10-16T20:22:19.967 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 108601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_gsm.ko 2023-10-16T20:22:19.967 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_hdlc.ko 2023-10-16T20:22:19.967 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 67929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/nozomi.ko 2023-10-16T20:22:19.968 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/ 2023-10-16T20:22:19.968 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250/ 2023-10-16T20:22:19.968 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250/8250_dw.ko 2023-10-16T20:22:19.968 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 59497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-10-16T20:22:19.968 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/altera_jtaguart.ko 2023-10-16T20:22:19.969 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/altera_uart.ko 2023-10-16T20:22:19.969 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/arc_uart.ko 2023-10-16T20:22:19.969 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 72425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/fsl_lpuart.ko 2023-10-16T20:22:19.969 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm/ 2023-10-16T20:22:19.969 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 103073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm/jsm.ko 2023-10-16T20:22:19.969 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/rp2.ko 2023-10-16T20:22:19.970 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 52817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/sc16is7xx.ko 2023-10-16T20:22:19.970 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/ 2023-10-16T20:22:19.970 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core/ 2023-10-16T20:22:19.970 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 531265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core/ufshcd-core.ko 2023-10-16T20:22:19.970 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host/ 2023-10-16T20:22:19.971 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-10-16T20:22:19.971 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2023-10-16T20:22:19.971 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/ 2023-10-16T20:22:19.971 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 44513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio.ko 2023-10-16T20:22:19.971 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_aec.ko 2023-10-16T20:22:19.972 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11281 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_cif.ko 2023-10-16T20:22:19.972 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_dmem_genirq.ko 2023-10-16T20:22:19.972 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_mf624.ko 2023-10-16T20:22:19.972 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_netx.ko 2023-10-16T20:22:19.972 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_pci_generic.ko 2023-10-16T20:22:19.973 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_pdrv_genirq.ko 2023-10-16T20:22:19.973 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_sercos3.ko 2023-10-16T20:22:19.973 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/ 2023-10-16T20:22:19.973 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/c67x00/ 2023-10-16T20:22:19.973 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 73297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/c67x00/c67x00.ko 2023-10-16T20:22:19.973 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/chipidea/ 2023-10-16T20:22:19.974 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 144377 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/chipidea/ci_hdrc.ko 2023-10-16T20:22:19.974 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2023-10-16T20:22:19.974 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2023-10-16T20:22:19.974 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2023-10-16T20:22:19.974 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/ 2023-10-16T20:22:19.975 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 87089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-acm.ko 2023-10-16T20:22:19.975 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 51409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-wdm.ko 2023-10-16T20:22:19.975 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 47369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usblp.ko 2023-10-16T20:22:19.975 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 89857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usbtmc.ko 2023-10-16T20:22:19.975 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/common/ 2023-10-16T20:22:19.976 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/common/ulpi.ko 2023-10-16T20:22:19.976 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/dwc2/ 2023-10-16T20:22:19.976 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/dwc2/dwc2_pci.ko 2023-10-16T20:22:19.976 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/dwc3/ 2023-10-16T20:22:19.976 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/dwc3/dwc3-haps.ko 2023-10-16T20:22:19.976 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/dwc3/dwc3-pci.ko 2023-10-16T20:22:19.977 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 444553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/dwc3/dwc3.ko 2023-10-16T20:22:19.977 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/ 2023-10-16T20:22:19.977 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/ 2023-10-16T20:22:19.977 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 60177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/u_ether.ko 2023-10-16T20:22:19.977 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 51881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/u_serial.ko 2023-10-16T20:22:19.978 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2023-10-16T20:22:19.978 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 42449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2023-10-16T20:22:19.978 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2023-10-16T20:22:19.978 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2023-10-16T20:22:19.978 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 92489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2023-10-16T20:22:19.979 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 51049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2023-10-16T20:22:19.979 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 122081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2023-10-16T20:22:19.979 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 57617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2023-10-16T20:22:19.979 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2023-10-16T20:22:19.979 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2023-10-16T20:22:19.979 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 57729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2023-10-16T20:22:19.980 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 85609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2023-10-16T20:22:19.980 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2023-10-16T20:22:19.980 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 58113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2023-10-16T20:22:19.980 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 67337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2023-10-16T20:22:19.980 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/ 2023-10-16T20:22:19.980 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2023-10-16T20:22:19.981 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2023-10-16T20:22:19.981 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2023-10-16T20:22:19.981 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_ether.ko 2023-10-16T20:22:19.981 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2023-10-16T20:22:19.981 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_hid.ko 2023-10-16T20:22:19.982 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2023-10-16T20:22:19.982 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2023-10-16T20:22:19.982 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 32073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2023-10-16T20:22:19.982 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_printer.ko 2023-10-16T20:22:19.982 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_serial.ko 2023-10-16T20:22:19.983 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/g_zero.ko 2023-10-16T20:22:19.983 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 62185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2023-10-16T20:22:19.983 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2023-10-16T20:22:19.983 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 161225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/libcomposite.ko 2023-10-16T20:22:19.983 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/ 2023-10-16T20:22:19.984 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17705 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2023-10-16T20:22:19.984 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/bdc/ 2023-10-16T20:22:19.984 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 157681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2023-10-16T20:22:19.984 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/goku_udc.ko 2023-10-16T20:22:19.984 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 52721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/gr_udc.ko 2023-10-16T20:22:19.985 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 69705 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2023-10-16T20:22:19.985 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 56033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/mv_udc.ko 2023-10-16T20:22:19.985 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 76057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/net2272.ko 2023-10-16T20:22:19.985 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 75073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/net2280.ko 2023-10-16T20:22:19.985 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 73169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/pch_udc.ko 2023-10-16T20:22:19.986 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 65809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2023-10-16T20:22:19.986 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 66097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2023-10-16T20:22:19.986 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 81505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2023-10-16T20:22:19.986 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 156073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/gadget/udc/udc-core.ko 2023-10-16T20:22:19.986 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/ 2023-10-16T20:22:19.986 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/bcma-hcd.ko 2023-10-16T20:22:19.987 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 80113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/isp116x-hcd.ko 2023-10-16T20:22:19.987 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 87353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/oxu210hp-hcd.ko 2023-10-16T20:22:19.987 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 78873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/r8a66597-hcd.ko 2023-10-16T20:22:19.987 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 51329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/sl811-hcd.ko 2023-10-16T20:22:19.987 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/sl811_cs.ko 2023-10-16T20:22:19.988 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/ssb-hcd.ko 2023-10-16T20:22:19.988 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/xhci-plat-hcd.ko 2023-10-16T20:22:19.988 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/ 2023-10-16T20:22:19.988 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/mdc800.ko 2023-10-16T20:22:19.988 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/microtek.ko 2023-10-16T20:22:19.989 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ 2023-10-16T20:22:19.989 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 41721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/adutux.ko 2023-10-16T20:22:19.989 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/appledisplay.ko 2023-10-16T20:22:19.989 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/cypress_cy7c63.ko 2023-10-16T20:22:19.989 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/cytherm.ko 2023-10-16T20:22:19.989 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ehset.ko 2023-10-16T20:22:19.990 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi26.ko 2023-10-16T20:22:19.990 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi62.ko 2023-10-16T20:22:19.990 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ezusb.ko 2023-10-16T20:22:19.990 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/idmouse.ko 2023-10-16T20:22:19.990 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/iowarrior.ko 2023-10-16T20:22:19.991 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/isight_firmware.ko 2023-10-16T20:22:19.991 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ldusb.ko 2023-10-16T20:22:19.991 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/legousbtower.ko 2023-10-16T20:22:19.991 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/lvstest.ko 2023-10-16T20:22:19.991 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga/ 2023-10-16T20:22:19.992 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 64841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-10-16T20:22:19.992 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/trancevibrator.ko 2023-10-16T20:22:19.992 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usb3503.ko 2023-10-16T20:22:19.992 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usblcd.ko 2023-10-16T20:22:19.992 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usbsevseg.ko 2023-10-16T20:22:19.993 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 70873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usbtest.ko 2023-10-16T20:22:19.993 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 41969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/uss720.ko 2023-10-16T20:22:19.993 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/yurex.ko 2023-10-16T20:22:19.993 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/mon/ 2023-10-16T20:22:19.993 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 72953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/mon/usbmon.ko 2023-10-16T20:22:19.993 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/musb/ 2023-10-16T20:22:19.994 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 301545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/musb/musb_hdrc.ko 2023-10-16T20:22:19.994 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy/ 2023-10-16T20:22:19.994 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy/phy-generic.ko 2023-10-16T20:22:19.994 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2023-10-16T20:22:19.994 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy/phy-isp1301.ko 2023-10-16T20:22:19.995 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy/phy-tahvo.ko 2023-10-16T20:22:19.995 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ 2023-10-16T20:22:19.995 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/aircable.ko 2023-10-16T20:22:19.995 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ark3116.ko 2023-10-16T20:22:19.995 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/belkin_sa.ko 2023-10-16T20:22:19.995 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 38105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ch341.ko 2023-10-16T20:22:19.996 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 70257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cp210x.ko 2023-10-16T20:22:19.996 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cyberjack.ko 2023-10-16T20:22:19.996 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 58217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cypress_m8.ko 2023-10-16T20:22:19.996 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 49817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/digi_acceleport.ko 2023-10-16T20:22:19.996 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/empeg.ko 2023-10-16T20:22:19.996 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 44337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/f81232.ko 2023-10-16T20:22:19.997 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 175137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ftdi_sio.ko 2023-10-16T20:22:19.997 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 53113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/garmin_gps.ko 2023-10-16T20:22:19.997 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 137081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_edgeport.ko 2023-10-16T20:22:19.997 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 124833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_ti.ko 2023-10-16T20:22:19.997 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 67769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipaq.ko 2023-10-16T20:22:19.998 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipw.ko 2023-10-16T20:22:19.998 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ir-usb.ko 2023-10-16T20:22:19.998 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 58233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-10-16T20:22:19.998 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 107521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan.ko 2023-10-16T20:22:19.998 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan_pda.ko 2023-10-16T20:22:19.998 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kl5kusb105.ko 2023-10-16T20:22:19.999 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kobil_sct.ko 2023-10-16T20:22:19.999 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 32953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mct_u232.ko 2023-10-16T20:22:19.999 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/metro-usb.ko 2023-10-16T20:22:19.999 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 66449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7720.ko 2023-10-16T20:22:19.999 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 78217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7840.ko 2023-10-16T20:22:20.000 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 56337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mxuport.ko 2023-10-16T20:22:20.000 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/navman.ko 2023-10-16T20:22:20.000 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/omninet.ko 2023-10-16T20:22:20.000 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/opticon.ko 2023-10-16T20:22:20.000 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 190633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/option.ko 2023-10-16T20:22:20.001 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 39073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/oti6858.ko 2023-10-16T20:22:20.001 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 54353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/pl2303.ko 2023-10-16T20:22:20.001 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcaux.ko 2023-10-16T20:22:20.001 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 37825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcserial.ko 2023-10-16T20:22:20.001 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/quatech2.ko 2023-10-16T20:22:20.001 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/safe_serial.ko 2023-10-16T20:22:20.002 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 51625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/sierra.ko 2023-10-16T20:22:20.002 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/spcp8x5.ko 2023-10-16T20:22:20.002 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ssu100.ko 2023-10-16T20:22:20.002 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/symbolserial.ko 2023-10-16T20:22:20.002 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 61489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-10-16T20:22:20.003 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-10-16T20:22:20.003 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_debug.ko 2023-10-16T20:22:20.003 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_wwan.ko 2023-10-16T20:22:20.003 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 125609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usbserial.ko 2023-10-16T20:22:20.003 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/visor.ko 2023-10-16T20:22:20.004 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 37409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/whiteheat.ko 2023-10-16T20:22:20.004 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/wishbone-serial.ko 2023-10-16T20:22:20.004 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/xsens_mt.ko 2023-10-16T20:22:20.004 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ 2023-10-16T20:22:20.004 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 52913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/uas.ko 2023-10-16T20:22:20.004 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-alauda.ko 2023-10-16T20:22:20.005 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-cypress.ko 2023-10-16T20:22:20.005 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-datafab.ko 2023-10-16T20:22:20.005 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-10-16T20:22:20.005 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-freecom.ko 2023-10-16T20:22:20.005 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-isd200.ko 2023-10-16T20:22:20.006 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-10-16T20:22:20.006 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-karma.ko 2023-10-16T20:22:20.006 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-onetouch.ko 2023-10-16T20:22:20.006 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-realtek.ko 2023-10-16T20:22:20.006 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr09.ko 2023-10-16T20:22:20.007 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr55.ko 2023-10-16T20:22:20.007 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-usbat.ko 2023-10-16T20:22:20.007 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 179969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/usb-storage.ko 2023-10-16T20:22:20.007 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/ 2023-10-16T20:22:20.007 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 67481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-core.ko 2023-10-16T20:22:20.007 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 72177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-host.ko 2023-10-16T20:22:20.008 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 107001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-10-16T20:22:20.008 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/ 2023-10-16T20:22:20.008 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/ 2023-10-16T20:22:20.008 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 154297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-10-16T20:22:20.008 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci.ko 2023-10-16T20:22:20.009 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 108737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio.ko 2023-10-16T20:22:20.009 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 77609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-10-16T20:22:20.009 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/ 2023-10-16T20:22:20.009 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 113873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost.ko 2023-10-16T20:22:20.009 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_iotlb.ko 2023-10-16T20:22:20.009 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 58089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_net.ko 2023-10-16T20:22:20.010 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 89249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_scsi.ko 2023-10-16T20:22:20.010 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/ 2023-10-16T20:22:20.010 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/ 2023-10-16T20:22:20.010 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/88pm860x_bl.ko 2023-10-16T20:22:20.010 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/aat2870_bl.ko 2023-10-16T20:22:20.011 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/adp5520_bl.ko 2023-10-16T20:22:20.011 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 32609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/adp8860_bl.ko 2023-10-16T20:22:20.011 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 39993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/adp8870_bl.ko 2023-10-16T20:22:20.011 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/apple_bl.ko 2023-10-16T20:22:20.011 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/as3711_bl.ko 2023-10-16T20:22:20.012 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/bd6107.ko 2023-10-16T20:22:20.012 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/da903x_bl.ko 2023-10-16T20:22:20.012 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/da9052_bl.ko 2023-10-16T20:22:20.012 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/gpio_backlight.ko 2023-10-16T20:22:20.013 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/kb3886_bl.ko 2023-10-16T20:22:20.013 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/lcd.ko 2023-10-16T20:22:20.013 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/lm3533_bl.ko 2023-10-16T20:22:20.013 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/lm3639_bl.ko 2023-10-16T20:22:20.013 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/lv5207lp.ko 2023-10-16T20:22:20.014 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/max8925_bl.ko 2023-10-16T20:22:20.014 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/pandora_bl.ko 2023-10-16T20:22:20.014 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/pcf50633-backlight.ko 2023-10-16T20:22:20.014 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/platform_lcd.ko 2023-10-16T20:22:20.014 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/wm831x_bl.ko 2023-10-16T20:22:20.015 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/ 2023-10-16T20:22:20.015 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ 2023-10-16T20:22:20.015 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2482.ko 2023-10-16T20:22:20.015 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 38985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2490.ko 2023-10-16T20:22:20.015 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/matrox_w1.ko 2023-10-16T20:22:20.015 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/w1-gpio.ko 2023-10-16T20:22:20.016 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/ 2023-10-16T20:22:20.016 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-10-16T20:22:20.016 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-10-16T20:22:20.016 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-10-16T20:22:20.016 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-10-16T20:22:20.017 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-10-16T20:22:20.017 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-10-16T20:22:20.017 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-10-16T20:22:20.017 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-10-16T20:22:20.017 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-10-16T20:22:20.017 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_smem.ko 2023-10-16T20:22:20.018 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 56425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_therm.ko 2023-10-16T20:22:20.018 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 102673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/wire.ko 2023-10-16T20:22:20.018 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ 2023-10-16T20:22:20.018 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/acquirewdt.ko 2023-10-16T20:22:20.018 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/advantechwdt.ko 2023-10-16T20:22:20.019 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim1535_wdt.ko 2023-10-16T20:22:20.019 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim7101_wdt.ko 2023-10-16T20:22:20.019 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/cpu5wdt.ko 2023-10-16T20:22:20.023 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/da9052_wdt.ko 2023-10-16T20:22:20.023 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/da9055_wdt.ko 2023-10-16T20:22:20.023 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/da9063_wdt.ko 2023-10-16T20:22:20.024 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/dw_wdt.ko 2023-10-16T20:22:20.024 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/eurotechwdt.ko 2023-10-16T20:22:20.024 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/f71808e_wdt.ko 2023-10-16T20:22:20.025 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/hpwdt.ko 2023-10-16T20:22:20.025 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/i6300esb.ko 2023-10-16T20:22:20.025 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-10-16T20:22:20.025 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_wdt.ko 2023-10-16T20:22:20.025 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ib700wdt.ko 2023-10-16T20:22:20.026 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ibmasr.ko 2023-10-16T20:22:20.026 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-10-16T20:22:20.026 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it8712f_wdt.ko 2023-10-16T20:22:20.027 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it87_wdt.ko 2023-10-16T20:22:20.027 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/kempld_wdt.ko 2023-10-16T20:22:20.027 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/machzwd.ko 2023-10-16T20:22:20.027 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/mena21_wdt.ko 2023-10-16T20:22:20.027 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/menf21bmc_wdt.ko 2023-10-16T20:22:20.028 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/nv_tco.ko 2023-10-16T20:22:20.028 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/of_xilinx_wdt.ko 2023-10-16T20:22:20.028 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22265 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pc87413_wdt.ko 2023-10-16T20:22:20.029 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 38657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_pci.ko 2023-10-16T20:22:20.029 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_usb.ko 2023-10-16T20:22:20.029 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/retu_wdt.ko 2023-10-16T20:22:20.029 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sbc60xxwdt.ko 2023-10-16T20:22:20.030 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sbc_epx_c3.ko 2023-10-16T20:22:20.030 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-10-16T20:22:20.030 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sc1200wdt.ko 2023-10-16T20:22:20.030 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sch311x_wdt.ko 2023-10-16T20:22:20.031 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/smsc37b787_wdt.ko 2023-10-16T20:22:20.031 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/softdog.ko 2023-10-16T20:22:20.031 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sp5100_tco.ko 2023-10-16T20:22:20.031 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/twl4030_wdt.ko 2023-10-16T20:22:20.031 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/via_wdt.ko 2023-10-16T20:22:20.032 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-10-16T20:22:20.032 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83877f_wdt.ko 2023-10-16T20:22:20.032 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83977f_wdt.ko 2023-10-16T20:22:20.032 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/wafer5823wdt.ko 2023-10-16T20:22:20.033 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/wdt_pci.ko 2023-10-16T20:22:20.033 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/wm831x_wdt.ko 2023-10-16T20:22:20.033 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/wm8350_wdt.ko 2023-10-16T20:22:20.033 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/xen_wdt.ko 2023-10-16T20:22:20.033 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/ 2023-10-16T20:22:20.034 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-evtchn.ko 2023-10-16T20:22:20.034 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntalloc.ko 2023-10-16T20:22:20.034 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 47865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntdev.ko 2023-10-16T20:22:20.034 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback/ 2023-10-16T20:22:20.034 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 176441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-10-16T20:22:20.035 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 40121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-privcmd.ko 2023-10-16T20:22:20.035 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 71249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-scsiback.ko 2023-10-16T20:22:20.035 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs/ 2023-10-16T20:22:20.035 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs/xenfs.ko 2023-10-16T20:22:20.035 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ 2023-10-16T20:22:20.036 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p/ 2023-10-16T20:22:20.036 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 166425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p/9p.ko 2023-10-16T20:22:20.036 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/adfs/ 2023-10-16T20:22:20.036 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 61073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/adfs/adfs.ko 2023-10-16T20:22:20.036 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs/ 2023-10-16T20:22:20.036 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 163593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs/affs.ko 2023-10-16T20:22:20.037 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/afs/ 2023-10-16T20:22:20.037 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 932817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/afs/kafs.ko 2023-10-16T20:22:20.037 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/autofs/ 2023-10-16T20:22:20.037 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 101057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/autofs/autofs4.ko 2023-10-16T20:22:20.037 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs/ 2023-10-16T20:22:20.038 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 64545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs/befs.ko 2023-10-16T20:22:20.038 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/bfs/ 2023-10-16T20:22:20.038 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 41257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/bfs/bfs.ko 2023-10-16T20:22:20.038 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/binfmt_misc.ko 2023-10-16T20:22:20.038 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs/ 2023-10-16T20:22:20.039 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 3402585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs/btrfs.ko 2023-10-16T20:22:20.039 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles/ 2023-10-16T20:22:20.039 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 306073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles/cachefiles.ko 2023-10-16T20:22:20.039 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph/ 2023-10-16T20:22:20.039 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 1323873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph/ceph.ko 2023-10-16T20:22:20.040 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda/ 2023-10-16T20:22:20.040 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 100801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda/coda.ko 2023-10-16T20:22:20.040 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/configfs/ 2023-10-16T20:22:20.040 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 108393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/configfs/configfs.ko 2023-10-16T20:22:20.040 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs/ 2023-10-16T20:22:20.041 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 29625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs/cramfs.ko 2023-10-16T20:22:20.041 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm/ 2023-10-16T20:22:20.041 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 642577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm/dlm.ko 2023-10-16T20:22:20.041 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/efs/ 2023-10-16T20:22:20.041 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/efs/efs.ko 2023-10-16T20:22:20.042 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs/ 2023-10-16T20:22:20.042 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 1505025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs/f2fs.ko 2023-10-16T20:22:20.042 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/ 2023-10-16T20:22:20.042 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/msdos.ko 2023-10-16T20:22:20.042 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/freevxfs/ 2023-10-16T20:22:20.043 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 42545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/freevxfs/freevxfs.ko 2023-10-16T20:22:20.043 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache/ 2023-10-16T20:22:20.043 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 226481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache/fscache.ko 2023-10-16T20:22:20.043 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/ 2023-10-16T20:22:20.043 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/cuse.ko 2023-10-16T20:22:20.044 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2/ 2023-10-16T20:22:20.044 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 1231529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2/gfs2.ko 2023-10-16T20:22:20.044 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs/ 2023-10-16T20:22:20.044 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 141209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs/hfs.ko 2023-10-16T20:22:20.044 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus/ 2023-10-16T20:22:20.045 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 222817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus/hfsplus.ko 2023-10-16T20:22:20.045 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hpfs/ 2023-10-16T20:22:20.045 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 192329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hpfs/hpfs.ko 2023-10-16T20:22:20.045 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs/ 2023-10-16T20:22:20.045 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 74065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs/isofs.ko 2023-10-16T20:22:20.045 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs/ 2023-10-16T20:22:20.046 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 419497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs/jfs.ko 2023-10-16T20:22:20.046 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd/ 2023-10-16T20:22:20.046 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 280313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd/lockd.ko 2023-10-16T20:22:20.046 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix/ 2023-10-16T20:22:20.046 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 89193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix/minix.ko 2023-10-16T20:22:20.047 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs/ 2023-10-16T20:22:20.047 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 124649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs/netfs.ko 2023-10-16T20:22:20.047 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/ 2023-10-16T20:22:20.047 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout/ 2023-10-16T20:22:20.047 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 81449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-10-16T20:22:20.048 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout/ 2023-10-16T20:22:20.048 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 59449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-10-16T20:22:20.048 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout/ 2023-10-16T20:22:20.048 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 117513 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-10-16T20:22:20.048 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 1055649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfs.ko 2023-10-16T20:22:20.049 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 66441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv2.ko 2023-10-16T20:22:20.049 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 120177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv3.ko 2023-10-16T20:22:20.049 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 2041689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv4.ko 2023-10-16T20:22:20.049 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/ 2023-10-16T20:22:20.049 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/grace.ko 2023-10-16T20:22:20.049 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/nfs_acl.ko 2023-10-16T20:22:20.050 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd/ 2023-10-16T20:22:20.050 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 1516809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd/nfsd.ko 2023-10-16T20:22:20.050 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2/ 2023-10-16T20:22:20.050 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 592833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2/nilfs2.ko 2023-10-16T20:22:20.050 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/ 2023-10-16T20:22:20.051 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-celtic.ko 2023-10-16T20:22:20.051 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-centeuro.ko 2023-10-16T20:22:20.051 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-croatian.ko 2023-10-16T20:22:20.051 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-cyrillic.ko 2023-10-16T20:22:20.051 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-gaelic.ko 2023-10-16T20:22:20.052 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-greek.ko 2023-10-16T20:22:20.052 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-iceland.ko 2023-10-16T20:22:20.052 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-inuit.ko 2023-10-16T20:22:20.052 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-roman.ko 2023-10-16T20:22:20.052 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-romanian.ko 2023-10-16T20:22:20.053 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-turkish.ko 2023-10-16T20:22:20.053 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_ascii.ko 2023-10-16T20:22:20.053 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1250.ko 2023-10-16T20:22:20.053 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1251.ko 2023-10-16T20:22:20.053 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1255.ko 2023-10-16T20:22:20.054 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp737.ko 2023-10-16T20:22:20.054 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp775.ko 2023-10-16T20:22:20.054 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp850.ko 2023-10-16T20:22:20.054 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp852.ko 2023-10-16T20:22:20.054 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp855.ko 2023-10-16T20:22:20.055 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp857.ko 2023-10-16T20:22:20.055 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp860.ko 2023-10-16T20:22:20.055 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp861.ko 2023-10-16T20:22:20.055 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp862.ko 2023-10-16T20:22:20.055 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp863.ko 2023-10-16T20:22:20.056 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp864.ko 2023-10-16T20:22:20.056 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp865.ko 2023-10-16T20:22:20.056 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp866.ko 2023-10-16T20:22:20.056 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp869.ko 2023-10-16T20:22:20.056 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp874.ko 2023-10-16T20:22:20.057 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 94137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp932.ko 2023-10-16T20:22:20.057 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 142673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp936.ko 2023-10-16T20:22:20.057 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 165745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp949.ko 2023-10-16T20:22:20.057 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 118121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp950.ko 2023-10-16T20:22:20.057 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_euc-jp.ko 2023-10-16T20:22:20.058 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-1.ko 2023-10-16T20:22:20.058 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-13.ko 2023-10-16T20:22:20.058 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-14.ko 2023-10-16T20:22:20.058 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-15.ko 2023-10-16T20:22:20.058 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-2.ko 2023-10-16T20:22:20.059 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-3.ko 2023-10-16T20:22:20.059 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-4.ko 2023-10-16T20:22:20.059 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-5.ko 2023-10-16T20:22:20.059 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-6.ko 2023-10-16T20:22:20.059 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-7.ko 2023-10-16T20:22:20.060 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-9.ko 2023-10-16T20:22:20.060 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-r.ko 2023-10-16T20:22:20.060 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-ru.ko 2023-10-16T20:22:20.060 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-u.ko 2023-10-16T20:22:20.060 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_ucs2_utils.ko 2023-10-16T20:22:20.060 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_utf8.ko 2023-10-16T20:22:20.061 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ntfs/ 2023-10-16T20:22:20.061 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 219417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ntfs/ntfs.ko 2023-10-16T20:22:20.061 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ 2023-10-16T20:22:20.061 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster/ 2023-10-16T20:22:20.061 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 244409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-10-16T20:22:20.062 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm/ 2023-10-16T20:22:20.062 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 440737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-10-16T20:22:20.062 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs/ 2023-10-16T20:22:20.062 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 54049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-10-16T20:22:20.062 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 2592433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2.ko 2023-10-16T20:22:20.063 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-10-16T20:22:20.063 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-10-16T20:22:20.063 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-10-16T20:22:20.063 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/omfs/ 2023-10-16T20:22:20.063 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 53433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/omfs/omfs.ko 2023-10-16T20:22:20.064 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs/ 2023-10-16T20:22:20.064 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 352153 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs/overlay.ko 2023-10-16T20:22:20.064 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore/ 2023-10-16T20:22:20.064 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 54809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore/ramoops.ko 2023-10-16T20:22:20.064 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/qnx4/ 2023-10-16T20:22:20.064 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/qnx4/qnx4.ko 2023-10-16T20:22:20.065 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/qnx6/ 2023-10-16T20:22:20.065 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 46689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/qnx6/qnx6.ko 2023-10-16T20:22:20.065 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/quota/ 2023-10-16T20:22:20.065 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/quota/quota_tree.ko 2023-10-16T20:22:20.065 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/quota/quota_v1.ko 2023-10-16T20:22:20.066 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/quota/quota_v2.ko 2023-10-16T20:22:20.066 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs/ 2023-10-16T20:22:20.066 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 532673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs/reiserfs.ko 2023-10-16T20:22:20.066 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs/ 2023-10-16T20:22:20.066 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs/romfs.ko 2023-10-16T20:22:20.066 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/ 2023-10-16T20:22:20.067 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client/ 2023-10-16T20:22:20.067 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 2668121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client/cifs.ko 2023-10-16T20:22:20.067 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/ 2023-10-16T20:22:20.067 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_arc4.ko 2023-10-16T20:22:20.067 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_md4.ko 2023-10-16T20:22:20.068 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs/ 2023-10-16T20:22:20.068 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 134641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs/squashfs.ko 2023-10-16T20:22:20.068 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv/ 2023-10-16T20:22:20.068 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 90721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv/sysv.ko 2023-10-16T20:22:20.068 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf/ 2023-10-16T20:22:20.069 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 247737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf/udf.ko 2023-10-16T20:22:20.069 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs/ 2023-10-16T20:22:20.069 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 162497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs/ufs.ko 2023-10-16T20:22:20.069 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs/ 2023-10-16T20:22:20.069 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 4010729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs/xfs.ko 2023-10-16T20:22:20.069 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/ 2023-10-16T20:22:20.070 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/locking/ 2023-10-16T20:22:20.070 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 75489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/locking/locktorture.ko 2023-10-16T20:22:20.070 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/time/ 2023-10-16T20:22:20.070 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/time/test_udelay.ko 2023-10-16T20:22:20.070 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 60553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/torture.ko 2023-10-16T20:22:20.071 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ 2023-10-16T20:22:20.071 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 6833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc-itu-t.ko 2023-10-16T20:22:20.071 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 6529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc7.ko 2023-10-16T20:22:20.071 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc8.ko 2023-10-16T20:22:20.071 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/ 2023-10-16T20:22:20.072 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libarc4.ko 2023-10-16T20:22:20.072 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 37081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libdes.ko 2023-10-16T20:22:20.072 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/interval_tree_test.ko 2023-10-16T20:22:20.072 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/libcrc32c.ko 2023-10-16T20:22:20.072 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lru_cache.ko 2023-10-16T20:22:20.072 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/ 2023-10-16T20:22:20.073 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4_compress.ko 2023-10-16T20:22:20.073 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4hc_compress.ko 2023-10-16T20:22:20.073 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math/ 2023-10-16T20:22:20.073 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 6617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math/cordic.ko 2023-10-16T20:22:20.073 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/memory-notifier-error-inject.ko 2023-10-16T20:22:20.074 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/notifier-error-inject.ko 2023-10-16T20:22:20.074 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/percpu_test.ko 2023-10-16T20:22:20.074 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/pm-notifier-error-inject.ko 2023-10-16T20:22:20.074 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6/ 2023-10-16T20:22:20.074 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 143433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6/raid6_pq.ko 2023-10-16T20:22:20.075 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/rbtree_test.ko 2023-10-16T20:22:20.075 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon/ 2023-10-16T20:22:20.075 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon/reed_solomon.ko 2023-10-16T20:22:20.075 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 38841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/test-kstrtox.ko 2023-10-16T20:22:20.075 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/test-string_helpers.ko 2023-10-16T20:22:20.075 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 4760425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/test_bpf.ko 2023-10-16T20:22:20.076 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 49089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/test_firmware.ko 2023-10-16T20:22:20.076 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 6369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/test_module.ko 2023-10-16T20:22:20.076 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/test_user_copy.ko 2023-10-16T20:22:20.076 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_bm.ko 2023-10-16T20:22:20.076 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_fsm.ko 2023-10-16T20:22:20.077 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_kmp.ko 2023-10-16T20:22:20.077 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/xz/ 2023-10-16T20:22:20.077 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/xz/xz_dec_test.ko 2023-10-16T20:22:20.077 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/mm/ 2023-10-16T20:22:20.077 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/mm/hwpoison-inject.ko 2023-10-16T20:22:20.077 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ 2023-10-16T20:22:20.078 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/ 2023-10-16T20:22:20.078 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 55737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/6lowpan.ko 2023-10-16T20:22:20.078 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_dest.ko 2023-10-16T20:22:20.078 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_fragment.ko 2023-10-16T20:22:20.078 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_hop.ko 2023-10-16T20:22:20.079 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 6985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_ipv6.ko 2023-10-16T20:22:20.079 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_mobility.ko 2023-10-16T20:22:20.079 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_routing.ko 2023-10-16T20:22:20.079 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_udp.ko 2023-10-16T20:22:20.079 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/ 2023-10-16T20:22:20.080 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/garp.ko 2023-10-16T20:22:20.080 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/mrp.ko 2023-10-16T20:22:20.080 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/p8022.ko 2023-10-16T20:22:20.080 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/psnap.ko 2023-10-16T20:22:20.080 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/stp.ko 2023-10-16T20:22:20.081 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q/ 2023-10-16T20:22:20.081 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 81737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q/8021q.ko 2023-10-16T20:22:20.081 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/ 2023-10-16T20:22:20.081 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 198337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet.ko 2023-10-16T20:22:20.081 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_fd.ko 2023-10-16T20:22:20.081 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26345 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_rdma.ko 2023-10-16T20:22:20.082 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 32337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_virtio.ko 2023-10-16T20:22:20.082 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv/ 2023-10-16T20:22:20.082 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 649297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv/batman-adv.ko 2023-10-16T20:22:20.082 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/ 2023-10-16T20:22:20.082 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 63809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/br_netfilter.ko 2023-10-16T20:22:20.083 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 777689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/bridge.ko 2023-10-16T20:22:20.110 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ 2023-10-16T20:22:20.111 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-10-16T20:22:20.111 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_among.ko 2023-10-16T20:22:20.111 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arp.ko 2023-10-16T20:22:20.111 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-10-16T20:22:20.111 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-10-16T20:22:20.112 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip.ko 2023-10-16T20:22:20.112 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-10-16T20:22:20.112 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_limit.ko 2023-10-16T20:22:20.112 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13217 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_log.ko 2023-10-16T20:22:20.112 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark.ko 2023-10-16T20:22:20.113 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-10-16T20:22:20.113 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8385 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-10-16T20:22:20.113 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-10-16T20:22:20.113 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-10-16T20:22:20.113 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_snat.ko 2023-10-16T20:22:20.114 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_stp.ko 2023-10-16T20:22:20.114 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-10-16T20:22:20.114 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-10-16T20:22:20.114 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-10-16T20:22:20.114 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-10-16T20:22:20.114 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 55929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtables.ko 2023-10-16T20:22:20.115 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-10-16T20:22:20.115 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-10-16T20:22:20.115 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph/ 2023-10-16T20:22:20.115 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 1047017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph/libceph.ko 2023-10-16T20:22:20.115 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/ 2023-10-16T20:22:20.116 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 125697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/pktgen.ko 2023-10-16T20:22:20.116 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/ 2023-10-16T20:22:20.116 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 194209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp.ko 2023-10-16T20:22:20.116 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_diag.ko 2023-10-16T20:22:20.116 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 57881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv4.ko 2023-10-16T20:22:20.117 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 60337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv6.ko 2023-10-16T20:22:20.117 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/ 2023-10-16T20:22:20.117 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 390473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/dsa_core.ko 2023-10-16T20:22:20.117 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_brcm.ko 2023-10-16T20:22:20.117 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_dsa.ko 2023-10-16T20:22:20.117 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_none.ko 2023-10-16T20:22:20.118 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_trailer.ko 2023-10-16T20:22:20.118 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/hsr/ 2023-10-16T20:22:20.118 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 99729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/hsr/hsr.ko 2023-10-16T20:22:20.118 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ 2023-10-16T20:22:20.118 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan/ 2023-10-16T20:22:20.119 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 45233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-10-16T20:22:20.119 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 254625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154.ko 2023-10-16T20:22:20.119 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 47289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154_socket.ko 2023-10-16T20:22:20.119 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ 2023-10-16T20:22:20.119 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ah4.ko 2023-10-16T20:22:20.119 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 38905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/esp4.ko 2023-10-16T20:22:20.120 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 48401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/fou.ko 2023-10-16T20:22:20.120 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/gre.ko 2023-10-16T20:22:20.120 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 48657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/inet_diag.ko 2023-10-16T20:22:20.120 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 60633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_gre.ko 2023-10-16T20:22:20.120 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 61225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_tunnel.ko 2023-10-16T20:22:20.121 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_vti.ko 2023-10-16T20:22:20.121 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipcomp.ko 2023-10-16T20:22:20.121 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipip.ko 2023-10-16T20:22:20.121 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ 2023-10-16T20:22:20.121 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 37641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arp_tables.ko 2023-10-16T20:22:20.121 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-10-16T20:22:20.122 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-10-16T20:22:20.122 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 41673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ip_tables.ko 2023-10-16T20:22:20.122 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-10-16T20:22:20.122 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2023-10-16T20:22:20.122 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-10-16T20:22:20.123 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-10-16T20:22:20.123 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-10-16T20:22:20.123 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11305 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_filter.ko 2023-10-16T20:22:20.123 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-10-16T20:22:20.123 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-10-16T20:22:20.124 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-10-16T20:22:20.124 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_security.ko 2023-10-16T20:22:20.124 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-10-16T20:22:20.124 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-10-16T20:22:20.124 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-10-16T20:22:20.125 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18225 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-10-16T20:22:20.125 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-10-16T20:22:20.125 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2023-10-16T20:22:20.125 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-10-16T20:22:20.125 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-10-16T20:22:20.125 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-10-16T20:22:20.126 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-10-16T20:22:20.126 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-10-16T20:22:20.126 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_bic.ko 2023-10-16T20:22:20.126 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_dctcp.ko 2023-10-16T20:22:20.127 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_diag.ko 2023-10-16T20:22:20.127 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_highspeed.ko 2023-10-16T20:22:20.127 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_htcp.ko 2023-10-16T20:22:20.127 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_hybla.ko 2023-10-16T20:22:20.127 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_illinois.ko 2023-10-16T20:22:20.127 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_lp.ko 2023-10-16T20:22:20.128 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_scalable.ko 2023-10-16T20:22:20.128 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_vegas.ko 2023-10-16T20:22:20.128 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_veno.ko 2023-10-16T20:22:20.128 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_westwood.ko 2023-10-16T20:22:20.128 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_yeah.ko 2023-10-16T20:22:20.129 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tunnel4.ko 2023-10-16T20:22:20.129 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_diag.ko 2023-10-16T20:22:20.129 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 41665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_tunnel.ko 2023-10-16T20:22:20.129 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/xfrm4_tunnel.ko 2023-10-16T20:22:20.129 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ 2023-10-16T20:22:20.130 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ah6.ko 2023-10-16T20:22:20.130 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 39537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/esp6.ko 2023-10-16T20:22:20.130 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/fou6.ko 2023-10-16T20:22:20.130 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 80897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_gre.ko 2023-10-16T20:22:20.130 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 93577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_tunnel.ko 2023-10-16T20:22:20.131 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-10-16T20:22:20.131 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 53273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_vti.ko 2023-10-16T20:22:20.131 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ipcomp6.ko 2023-10-16T20:22:20.131 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19017 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/mip6.ko 2023-10-16T20:22:20.131 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ 2023-10-16T20:22:20.133 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 42169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-10-16T20:22:20.133 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2023-10-16T20:22:20.133 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-10-16T20:22:20.133 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-10-16T20:22:20.134 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-10-16T20:22:20.134 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-10-16T20:22:20.134 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-10-16T20:22:20.134 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-10-16T20:22:20.134 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-10-16T20:22:20.135 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-10-16T20:22:20.135 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-10-16T20:22:20.135 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-10-16T20:22:20.135 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-10-16T20:22:20.135 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11121 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-10-16T20:22:20.135 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-10-16T20:22:20.136 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-10-16T20:22:20.136 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-10-16T20:22:20.136 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 39113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-10-16T20:22:20.136 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-10-16T20:22:20.136 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25945 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-10-16T20:22:20.137 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-10-16T20:22:20.137 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-10-16T20:22:20.137 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-10-16T20:22:20.137 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-10-16T20:22:20.137 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9009 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-10-16T20:22:20.138 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 74073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/sit.ko 2023-10-16T20:22:20.138 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/tunnel6.ko 2023-10-16T20:22:20.138 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/xfrm6_tunnel.ko 2023-10-16T20:22:20.138 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key/ 2023-10-16T20:22:20.138 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 82833 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key/af_key.ko 2023-10-16T20:22:20.138 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/ 2023-10-16T20:22:20.139 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 125281 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_core.ko 2023-10-16T20:22:20.139 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_debugfs.ko 2023-10-16T20:22:20.139 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 25953 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_eth.ko 2023-10-16T20:22:20.139 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip.ko 2023-10-16T20:22:20.139 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36713 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip6.ko 2023-10-16T20:22:20.140 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_netlink.ko 2023-10-16T20:22:20.140 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 53241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ppp.ko 2023-10-16T20:22:20.140 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/lapb/ 2023-10-16T20:22:20.140 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 47977 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/lapb/lapb.ko 2023-10-16T20:22:20.140 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc/ 2023-10-16T20:22:20.141 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc/llc.ko 2023-10-16T20:22:20.141 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 267777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc/llc2.ko 2023-10-16T20:22:20.141 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154/ 2023-10-16T20:22:20.141 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 290417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154/mac802154.ko 2023-10-16T20:22:20.141 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls/ 2023-10-16T20:22:20.141 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls/mpls_gso.ko 2023-10-16T20:22:20.142 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ 2023-10-16T20:22:20.142 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ 2023-10-16T20:22:20.142 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 119585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set.ko 2023-10-16T20:22:20.142 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-10-16T20:22:20.142 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26913 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-10-16T20:22:20.143 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-10-16T20:22:20.143 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 100297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-10-16T20:22:20.143 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 101441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-10-16T20:22:20.143 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 102817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-10-16T20:22:20.143 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 103481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-10-16T20:22:20.144 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 112849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-10-16T20:22:20.144 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 60329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-10-16T20:22:20.144 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 109593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-10-16T20:22:20.144 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 113129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-10-16T20:22:20.144 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 112585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-10-16T20:22:20.145 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 111585 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-10-16T20:22:20.145 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 114465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-10-16T20:22:20.145 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-10-16T20:22:20.145 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ 2023-10-16T20:22:20.145 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 411729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs.ko 2023-10-16T20:22:20.146 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-10-16T20:22:20.146 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-10-16T20:22:20.146 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-10-16T20:22:20.146 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-10-16T20:22:20.146 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-10-16T20:22:20.146 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-10-16T20:22:20.147 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7689 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-10-16T20:22:20.147 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10681 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-10-16T20:22:20.147 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-10-16T20:22:20.147 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-10-16T20:22:20.147 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-10-16T20:22:20.148 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-10-16T20:22:20.148 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10993 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-10-16T20:22:20.148 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 49257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conncount.ko 2023-10-16T20:22:20.148 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 408257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack.ko 2023-10-16T20:22:20.148 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-10-16T20:22:20.148 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-10-16T20:22:20.149 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33761 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-10-16T20:22:20.149 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 118145 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_h323.ko 2023-10-16T20:22:20.149 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_irc.ko 2023-10-16T20:22:20.149 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-10-16T20:22:20.149 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 107297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-10-16T20:22:20.150 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-10-16T20:22:20.150 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sane.ko 2023-10-16T20:22:20.150 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 66641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sip.ko 2023-10-16T20:22:20.150 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-10-16T20:22:20.150 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-10-16T20:22:20.151 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10113 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_dup_netdev.ko 2023-10-16T20:22:20.151 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 89473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table.ko 2023-10-16T20:22:20.151 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9593 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table_inet.ko 2023-10-16T20:22:20.151 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 37129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_log_syslog.ko 2023-10-16T20:22:20.151 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 95417 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat.ko 2023-10-16T20:22:20.151 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_amanda.ko 2023-10-16T20:22:20.152 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_ftp.ko 2023-10-16T20:22:20.152 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11857 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_irc.ko 2023-10-16T20:22:20.152 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28929 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_sip.ko 2023-10-16T20:22:20.152 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_tftp.ko 2023-10-16T20:22:20.152 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 44065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_synproxy_core.ko 2023-10-16T20:22:20.153 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 615313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_tables.ko 2023-10-16T20:22:20.153 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 42561 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink.ko 2023-10-16T20:22:20.153 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_acct.ko 2023-10-16T20:22:20.153 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30569 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_cttimeout.ko 2023-10-16T20:22:20.153 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 39785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_log.ko 2023-10-16T20:22:20.153 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_osf.ko 2023-10-16T20:22:20.154 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 49377 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_queue.ko 2023-10-16T20:22:20.154 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10457 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_chain_nat.ko 2023-10-16T20:22:20.154 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_compat.ko 2023-10-16T20:22:20.154 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 40721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_ct.ko 2023-10-16T20:22:20.154 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_dup_netdev.ko 2023-10-16T20:22:20.155 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fib.ko 2023-10-16T20:22:20.155 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fwd_netdev.ko 2023-10-16T20:22:20.155 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14185 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_hash.ko 2023-10-16T20:22:20.155 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_limit.ko 2023-10-16T20:22:20.155 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_log.ko 2023-10-16T20:22:20.156 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_masq.ko 2023-10-16T20:22:20.156 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14329 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_nat.ko 2023-10-16T20:22:20.156 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13673 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_numgen.ko 2023-10-16T20:22:20.156 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11841 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_osf.ko 2023-10-16T20:22:20.156 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_queue.ko 2023-10-16T20:22:20.156 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_redir.ko 2023-10-16T20:22:20.157 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject.ko 2023-10-16T20:22:20.157 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject_inet.ko 2023-10-16T20:22:20.157 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_socket.ko 2023-10-16T20:22:20.157 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20369 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_synproxy.ko 2023-10-16T20:22:20.157 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_tproxy.ko 2023-10-16T20:22:20.158 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 89313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/x_tables.ko 2023-10-16T20:22:20.158 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_AUDIT.ko 2023-10-16T20:22:20.158 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CHECKSUM.ko 2023-10-16T20:22:20.158 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CLASSIFY.ko 2023-10-16T20:22:20.158 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-10-16T20:22:20.159 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 18169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CT.ko 2023-10-16T20:22:20.159 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_DSCP.ko 2023-10-16T20:22:20.159 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HL.ko 2023-10-16T20:22:20.159 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HMARK.ko 2023-10-16T20:22:20.159 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 38873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_IDLETIMER.ko 2023-10-16T20:22:20.159 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13889 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LED.ko 2023-10-16T20:22:20.160 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LOG.ko 2023-10-16T20:22:20.160 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11753 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_MASQUERADE.ko 2023-10-16T20:22:20.160 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NETMAP.ko 2023-10-16T20:22:20.160 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFLOG.ko 2023-10-16T20:22:20.160 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFQUEUE.ko 2023-10-16T20:22:20.161 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_RATEEST.ko 2023-10-16T20:22:20.161 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_REDIRECT.ko 2023-10-16T20:22:20.161 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 14337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_SECMARK.ko 2023-10-16T20:22:20.161 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPMSS.ko 2023-10-16T20:22:20.161 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-10-16T20:22:20.161 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17361 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TEE.ko 2023-10-16T20:22:20.162 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TPROXY.ko 2023-10-16T20:22:20.162 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TRACE.ko 2023-10-16T20:22:20.162 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_addrtype.ko 2023-10-16T20:22:20.162 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_bpf.ko 2023-10-16T20:22:20.162 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cgroup.ko 2023-10-16T20:22:20.163 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11209 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cluster.ko 2023-10-16T20:22:20.163 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_comment.ko 2023-10-16T20:22:20.163 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10465 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connbytes.ko 2023-10-16T20:22:20.163 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11081 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlabel.ko 2023-10-16T20:22:20.163 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlimit.ko 2023-10-16T20:22:20.164 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connmark.ko 2023-10-16T20:22:20.164 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_conntrack.ko 2023-10-16T20:22:20.164 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cpu.ko 2023-10-16T20:22:20.164 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dccp.ko 2023-10-16T20:22:20.164 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_devgroup.ko 2023-10-16T20:22:20.164 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8801 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dscp.ko 2023-10-16T20:22:20.165 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ecn.ko 2023-10-16T20:22:20.165 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_esp.ko 2023-10-16T20:22:20.165 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 43289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hashlimit.ko 2023-10-16T20:22:20.165 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_helper.ko 2023-10-16T20:22:20.165 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7865 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hl.ko 2023-10-16T20:22:20.166 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11921 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipcomp.ko 2023-10-16T20:22:20.166 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_iprange.ko 2023-10-16T20:22:20.166 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11785 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipvs.ko 2023-10-16T20:22:20.166 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16041 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_l2tp.ko 2023-10-16T20:22:20.166 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8001 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_length.ko 2023-10-16T20:22:20.166 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_limit.ko 2023-10-16T20:22:20.167 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7545 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mac.ko 2023-10-16T20:22:20.167 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mark.ko 2023-10-16T20:22:20.167 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11825 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_multiport.ko 2023-10-16T20:22:20.167 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13817 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nat.ko 2023-10-16T20:22:20.167 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nfacct.ko 2023-10-16T20:22:20.168 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_osf.ko 2023-10-16T20:22:20.168 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_owner.ko 2023-10-16T20:22:20.168 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_physdev.ko 2023-10-16T20:22:20.168 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_pkttype.ko 2023-10-16T20:22:20.168 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11985 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_policy.ko 2023-10-16T20:22:20.168 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_quota.ko 2023-10-16T20:22:20.169 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_rateest.ko 2023-10-16T20:22:20.169 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_realm.ko 2023-10-16T20:22:20.169 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_recent.ko 2023-10-16T20:22:20.169 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_sctp.ko 2023-10-16T20:22:20.169 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_set.ko 2023-10-16T20:22:20.170 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 16089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_socket.ko 2023-10-16T20:22:20.170 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9505 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_state.ko 2023-10-16T20:22:20.170 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9473 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_statistic.ko 2023-10-16T20:22:20.170 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8809 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_string.ko 2023-10-16T20:22:20.170 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 8553 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_tcpmss.ko 2023-10-16T20:22:20.171 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15777 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_tcpudp.ko 2023-10-16T20:22:20.171 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11129 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_time.ko 2023-10-16T20:22:20.171 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_u32.ko 2023-10-16T20:22:20.171 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink/ 2023-10-16T20:22:20.171 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13969 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink/netlink_diag.ko 2023-10-16T20:22:20.171 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh/ 2023-10-16T20:22:20.172 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 12425 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh/nsh.ko 2023-10-16T20:22:20.172 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/ 2023-10-16T20:22:20.172 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 380033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/openvswitch.ko 2023-10-16T20:22:20.172 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10321 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-geneve.ko 2023-10-16T20:22:20.172 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-gre.ko 2023-10-16T20:22:20.173 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 11793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-vxlan.ko 2023-10-16T20:22:20.173 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet/ 2023-10-16T20:22:20.173 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13057 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet/af_packet_diag.ko 2023-10-16T20:22:20.173 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/phonet/ 2023-10-16T20:22:20.173 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 110201 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/phonet/phonet.ko 2023-10-16T20:22:20.173 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 54697 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/phonet/pn_pep.ko 2023-10-16T20:22:20.174 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/ 2023-10-16T20:22:20.174 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 251481 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds.ko 2023-10-16T20:22:20.174 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 164937 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_rdma.ko 2023-10-16T20:22:20.174 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 67377 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_tcp.ko 2023-10-16T20:22:20.174 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rxrpc/ 2023-10-16T20:22:20.175 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 819657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rxrpc/rxrpc.ko 2023-10-16T20:22:20.175 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/ 2023-10-16T20:22:20.175 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 32241 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_csum.ko 2023-10-16T20:22:20.175 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23577 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_gact.ko 2023-10-16T20:22:20.175 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26873 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_ipt.ko 2023-10-16T20:22:20.176 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34097 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_mirred.ko 2023-10-16T20:22:20.176 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_nat.ko 2023-10-16T20:22:20.176 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 43449 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_pedit.ko 2023-10-16T20:22:20.176 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_police.ko 2023-10-16T20:22:20.176 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_simple.ko 2023-10-16T20:22:20.176 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35297 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_skbedit.ko 2023-10-16T20:22:20.177 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36665 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_vlan.ko 2023-10-16T20:22:20.177 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26169 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_basic.ko 2023-10-16T20:22:20.177 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33961 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_bpf.ko 2023-10-16T20:22:20.177 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19233 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_cgroup.ko 2023-10-16T20:22:20.177 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31401 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_flow.ko 2023-10-16T20:22:20.178 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31769 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_fw.ko 2023-10-16T20:22:20.178 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 38489 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_route.ko 2023-10-16T20:22:20.178 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 61409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_u32.ko 2023-10-16T20:22:20.178 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7441 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_cmp.ko 2023-10-16T20:22:20.178 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 13409 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_ipset.ko 2023-10-16T20:22:20.179 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33289 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_meta.ko 2023-10-16T20:22:20.179 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_nbyte.ko 2023-10-16T20:22:20.179 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_text.ko 2023-10-16T20:22:20.179 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7249 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_u32.ko 2023-10-16T20:22:20.179 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 21649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_choke.ko 2023-10-16T20:22:20.179 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20849 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_codel.ko 2023-10-16T20:22:20.180 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 34633 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_drr.ko 2023-10-16T20:22:20.180 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 33609 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_fq.ko 2023-10-16T20:22:20.180 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31617 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_fq_codel.ko 2023-10-16T20:22:20.180 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 35433 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_gred.ko 2023-10-16T20:22:20.180 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 49601 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hfsc.ko 2023-10-16T20:22:20.180 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22089 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hhf.ko 2023-10-16T20:22:20.181 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 66393 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_htb.ko 2023-10-16T20:22:20.181 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26073 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_ingress.ko 2023-10-16T20:22:20.181 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 30273 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio.ko 2023-10-16T20:22:20.181 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio_lib.ko 2023-10-16T20:22:20.181 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26065 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_multiq.ko 2023-10-16T20:22:20.182 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 36137 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_netem.ko 2023-10-16T20:22:20.182 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 27497 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_pie.ko 2023-10-16T20:22:20.182 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10625 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_plug.ko 2023-10-16T20:22:20.182 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 24729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_prio.ko 2023-10-16T20:22:20.182 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 48793 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_qfq.ko 2023-10-16T20:22:20.183 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 32905 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_red.ko 2023-10-16T20:22:20.183 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 26881 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfb.ko 2023-10-16T20:22:20.183 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 31353 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfq.ko 2023-10-16T20:22:20.183 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28897 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_tbf.ko 2023-10-16T20:22:20.183 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 28105 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_teql.ko 2023-10-16T20:22:20.184 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/ 2023-10-16T20:22:20.184 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 929745 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp.ko 2023-10-16T20:22:20.184 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 19033 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp_diag.ko 2023-10-16T20:22:20.184 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/ 2023-10-16T20:22:20.184 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/ 2023-10-16T20:22:20.185 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 336529 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-10-16T20:22:20.185 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 54737 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-10-16T20:22:20.185 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 1520161 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/sunrpc.ko 2023-10-16T20:22:20.185 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma/ 2023-10-16T20:22:20.185 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 707721 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-10-16T20:22:20.186 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/ 2023-10-16T20:22:20.186 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10025 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/diag.ko 2023-10-16T20:22:20.186 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 862641 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/tipc.ko 2023-10-16T20:22:20.186 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix/ 2023-10-16T20:22:20.186 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15729 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix/unix_diag.ko 2023-10-16T20:22:20.187 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/ 2023-10-16T20:22:20.187 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 85337 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-10-16T20:22:20.187 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 68049 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-10-16T20:22:20.187 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 96193 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock.ko 2023-10-16T20:22:20.187 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 10705 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_diag.ko 2023-10-16T20:22:20.187 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 17521 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_loopback.ko 2023-10-16T20:22:20.188 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/x25/ 2023-10-16T20:22:20.188 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 125177 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/x25/x25.ko 2023-10-16T20:22:20.188 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm/ 2023-10-16T20:22:20.188 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 22657 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm/xfrm_algo.ko 2023-10-16T20:22:20.188 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 20649 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm/xfrm_ipcomp.ko 2023-10-16T20:22:20.189 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 102257 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm/xfrm_user.ko 2023-10-16T20:22:20.189 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/ 2023-10-16T20:22:20.189 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib/ 2023-10-16T20:22:20.189 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 15537 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib/irqbypass.ko 2023-10-16T20:22:20.189 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 639863 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.alias 2023-10-16T20:22:20.190 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 623919 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.alias.bin 2023-10-16T20:22:20.190 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 7663 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.builtin 2023-10-16T20:22:20.190 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 23428 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.alias.bin 2023-10-16T20:22:20.190 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 9821 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.bin 2023-10-16T20:22:20.190 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 70004 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.modinfo 2023-10-16T20:22:20.191 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 146313 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.dep 2023-10-16T20:22:20.191 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 232548 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.dep.bin 2023-10-16T20:22:20.191 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 210 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.devname 2023-10-16T20:22:20.191 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 72606 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.order 2023-10-16T20:22:20.191 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 879 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.softdep 2023-10-16T20:22:20.191 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 254438 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.symbols 2023-10-16T20:22:20.192 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 309363 2023-10-10 19:56 ./lib/modules/6.6.0-rc5-g47672119149b/modules.symbols.bin 2023-10-16T20:22:20.192 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./usr/ 2023-10-16T20:22:20.192 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./usr/share/ 2023-10-16T20:22:20.192 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./usr/share/doc/ 2023-10-16T20:22:20.192 INFO:teuthology.orchestra.run.smithi088.stdout:drwxr-xr-x root/root 0 2023-10-10 19:56 ./usr/share/doc/linux-image-6.6.0-rc5-g47672119149b/ 2023-10-16T20:22:20.193 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 175 2023-10-10 19:56 ./usr/share/doc/linux-image-6.6.0-rc5-g47672119149b/changelog.Debian.gz 2023-10-16T20:22:20.193 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r-- root/root 691 2023-10-10 19:56 ./usr/share/doc/linux-image-6.6.0-rc5-g47672119149b/copyright 2023-10-16T20:22:20.194 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-g47672119149b 2023-10-16T20:22:20.194 INFO:teuthology.task.kernel:searching for kernel 6.6.0-rc5-g47672119149b 2023-10-16T20:22:20.194 DEBUG:teuthology.orchestra.run.smithi088:> egrep '(submenu|menuentry.*6.6.0-rc5-g47672119149b).*{' /boot/grub/grub.cfg 2023-10-16T20:22:20.202 INFO:teuthology.orchestra.run.smithi088.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-16T20:22:20.202 INFO:teuthology.orchestra.run.smithi088.stdout: menuentry 'Ubuntu, with Linux 6.6.0-rc5-g47672119149b' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-rc5-g47672119149b-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-16T20:22:20.202 INFO:teuthology.orchestra.run.smithi088.stdout: menuentry 'Ubuntu, with Linux 6.6.0-rc5-g47672119149b (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-rc5-g47672119149b-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-16T20:22:20.203 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2023-10-16T20:22:20.203 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.6.0-rc5-g47672119149b 2023-10-16T20:22:20.204 DEBUG:teuthology.orchestra.run.smithi088:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2023-10-16T20:22:20.246 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi088.front.sepia.ceph.com to complete... 2023-10-16T20:22:20.688 INFO:teuthology.orchestra.run.smithi088.stderr:Sourcing file `/etc/default/grub' 2023-10-16T20:22:20.689 INFO:teuthology.orchestra.run.smithi088.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-16T20:22:20.708 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2023-10-16T20:22:20.955 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-6.6.0-rc5-g47672119149b 2023-10-16T20:22:20.961 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initrd.img-6.6.0-rc5-g47672119149b 2023-10-16T20:22:21.289 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-16T20:22:21.295 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-16T20:22:21.341 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-16T20:22:21.346 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-16T20:22:21.487 INFO:teuthology.orchestra.run.smithi088.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-16T20:22:21.487 INFO:teuthology.orchestra.run.smithi088.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-16T20:22:21.487 INFO:teuthology.orchestra.run.smithi088.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-16T20:22:21.499 INFO:teuthology.orchestra.run.smithi088.stderr:done 2023-10-16T20:22:29.774 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-10-16T20:22:29.775 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-10-16T20:22:29.776 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T20:22:52.629 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-16T20:22:52.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-10-16T20:22:52.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T20:23:27.099 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2023-10-16T20:23:29.780 DEBUG:teuthology.orchestra.remote:timed out 2023-10-16T20:23:36.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-10-16T20:23:36.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T20:23:38.785 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-10-16T20:23:38.785 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T20:23:39.166 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2023-10-16T20:23:39.404 DEBUG:teuthology.orchestra.run.smithi191:> true 2023-10-16T20:23:40.546 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-10-16T20:23:40.547 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.6.0-rc5-g47672119149b"... 2023-10-16T20:23:40.547 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2023-10-16T20:23:40.598 INFO:teuthology.orchestra.run.smithi191.stdout:6.6.0-rc5-g47672119149b 2023-10-16T20:23:40.599 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-g47672119149b vs 6.6.0-rc5-g47672119149b 2023-10-16T20:23:40.599 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-16T20:23:40.599 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-16T20:23:41.601 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-10-16T20:23:41.601 DEBUG:teuthology.orchestra.run.smithi191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-16T20:23:41.911 INFO:teuthology.orchestra.run.smithi191.stdout:ttyS1 2023-10-16T20:23:41.994 DEBUG:teuthology.parallel:result is None 2023-10-16T20:23:51.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-10-16T20:23:51.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T20:24:22.989 DEBUG:teuthology.orchestra.run.smithi088:> true 2023-10-16T20:24:23.680 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-10-16T20:24:23.680 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.6.0-rc5-g47672119149b"... 2023-10-16T20:24:23.680 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2023-10-16T20:24:23.730 INFO:teuthology.orchestra.run.smithi088.stdout:6.6.0-rc5-g47672119149b 2023-10-16T20:24:23.730 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-g47672119149b vs 6.6.0-rc5-g47672119149b 2023-10-16T20:24:23.731 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-16T20:24:23.731 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-16T20:24:24.732 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-16T20:24:24.732 DEBUG:teuthology.orchestra.run.smithi088:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-16T20:24:24.803 INFO:teuthology.orchestra.run.smithi088.stdout:ttyS1 2023-10-16T20:24:24.849 DEBUG:teuthology.parallel:result is None 2023-10-16T20:24:24.850 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-16T20:24:24.857 INFO:teuthology.task.internal:Creating test directory... 2023-10-16T20:24:24.858 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-16T20:24:24.862 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-16T20:24:24.872 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-16T20:24:24.878 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-16T20:24:24.885 INFO:teuthology.task.internal:Creating archive directory... 2023-10-16T20:24:24.886 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-16T20:24:24.914 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-16T20:24:24.938 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-16T20:24:24.945 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-16T20:24:24.945 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-16T20:24:24.977 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-16T20:24:25.014 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T20:24:25.016 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T20:24:25.029 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T20:24:25.031 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T20:24:25.033 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-16T20:24:25.040 INFO:teuthology.task.internal:Configuring sudo... 2023-10-16T20:24:25.040 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-16T20:24:25.077 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-16T20:24:25.099 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-16T20:24:25.108 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-16T20:24:25.108 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-16T20:24:25.141 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-16T20:24:25.151 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T20:24:25.199 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T20:24:25.251 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:24:25.252 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-16T20:24:25.321 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T20:24:25.331 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T20:24:25.383 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:24:25.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-16T20:24:25.453 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2023-10-16T20:24:25.456 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2023-10-16T20:24:25.576 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-16T20:24:25.584 INFO:teuthology.task.internal:Starting timer... 2023-10-16T20:24:25.584 INFO:teuthology.run_tasks:Running task pcp... 2023-10-16T20:24:25.597 INFO:teuthology.run_tasks:Running task selinux... 2023-10-16T20:24:25.605 DEBUG:teuthology.task.selinux:Excluding smithi088: OS 'ubuntu' does not support SELinux 2023-10-16T20:24:25.605 DEBUG:teuthology.task.selinux:Excluding smithi191: OS 'ubuntu' does not support SELinux 2023-10-16T20:24:25.606 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-16T20:24:25.606 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-16T20:24:25.606 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-16T20:24:25.607 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-16T20:24:25.615 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-16T20:24:25.621 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-16T20:24:25.764 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-16T20:24:25.780 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-16T20:24:25.782 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi088.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-16T20:32:59.495 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi088.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2023-10-16T20:32:59.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-10-16T20:32:59.498 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T20:32:59.578 DEBUG:teuthology.orchestra.run.smithi088:> true 2023-10-16T20:32:59.701 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-10-16T20:32:59.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-10-16T20:32:59.702 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T20:32:59.778 DEBUG:teuthology.orchestra.run.smithi191:> true 2023-10-16T20:32:59.893 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-10-16T20:32:59.893 INFO:teuthology.run_tasks:Running task clock... 2023-10-16T20:32:59.917 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-16T20:32:59.918 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-16T20:32:59.919 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T20:32:59.922 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-16T20:32:59.923 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T20:32:59.959 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-16T20:32:59.959 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: Command line: ntpd -gq 2023-10-16T20:32:59.959 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: ---------------------------------------------------- 2023-10-16T20:32:59.960 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: ntp-4 is maintained by Network Time Foundation, 2023-10-16T20:32:59.960 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-16T20:32:59.960 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: corporation. Support and training for ntp-4 are 2023-10-16T20:32:59.960 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: available at https://www.nwtime.org/support 2023-10-16T20:32:59.960 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: ---------------------------------------------------- 2023-10-16T20:32:59.961 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: proto: precision = 0.043 usec (-24) 2023-10-16T20:32:59.962 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: basedate set to 2022-02-04 2023-10-16T20:32:59.962 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: gps base set to 2022-02-06 (week 2196) 2023-10-16T20:32:59.962 INFO:teuthology.orchestra.run.smithi088.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-16T20:32:59.963 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-16T20:32:59.963 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: restrict ::: KOD does nothing without LIMITED. 2023-10-16T20:32:59.963 INFO:teuthology.orchestra.run.smithi088.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-16T20:32:59.964 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: Listen and drop on 0 v6wildcard [::]:123 2023-10-16T20:32:59.964 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-16T20:32:59.965 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: Listen normally on 2 lo 127.0.0.1:123 2023-10-16T20:32:59.965 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: Listen normally on 3 enp3s0f1 172.21.15.88:123 2023-10-16T20:32:59.965 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: Listen normally on 4 lo [::1]:123 2023-10-16T20:32:59.965 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d42d%5]:123 2023-10-16T20:32:59.965 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:32:59 ntpd[28025]: Listening on routing socket on fd #22 for interface updates 2023-10-16T20:32:59.977 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-16T20:32:59.977 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: Command line: ntpd -gq 2023-10-16T20:32:59.977 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: ---------------------------------------------------- 2023-10-16T20:32:59.978 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: ntp-4 is maintained by Network Time Foundation, 2023-10-16T20:32:59.978 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-16T20:32:59.978 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: corporation. Support and training for ntp-4 are 2023-10-16T20:32:59.978 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: available at https://www.nwtime.org/support 2023-10-16T20:32:59.978 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: ---------------------------------------------------- 2023-10-16T20:32:59.979 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: proto: precision = 0.044 usec (-24) 2023-10-16T20:32:59.979 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: basedate set to 2022-02-04 2023-10-16T20:32:59.980 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: gps base set to 2022-02-06 (week 2196) 2023-10-16T20:32:59.980 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-16T20:32:59.980 INFO:teuthology.orchestra.run.smithi191.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-16T20:32:59.981 INFO:teuthology.orchestra.run.smithi191.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-16T20:32:59.982 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: restrict ::: KOD does nothing without LIMITED. 2023-10-16T20:32:59.983 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: Listen and drop on 0 v6wildcard [::]:123 2023-10-16T20:32:59.984 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-16T20:32:59.984 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: Listen normally on 2 lo 127.0.0.1:123 2023-10-16T20:32:59.984 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: Listen normally on 3 enp3s0f1 172.21.15.191:123 2023-10-16T20:32:59.984 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: Listen normally on 4 lo [::1]:123 2023-10-16T20:32:59.984 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7211%5]:123 2023-10-16T20:32:59.985 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:32:59 ntpd[27982]: Listening on routing socket on fd #22 for interface updates 2023-10-16T20:33:00.959 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:00 ntpd[28025]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:00.959 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:00 ntpd[28025]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:00.960 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:00 ntpd[28025]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:00.960 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:00 ntpd[28025]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:00.960 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:00 ntpd[28025]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-16T20:33:00.961 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:00 ntpd[28025]: can't open /var/log/ntpstats/peerstats.20231016: Permission denied 2023-10-16T20:33:01.959 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:01 ntpd[28025]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:01.960 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:01 ntpd[28025]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:01.977 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:01 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:01.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:01 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:01.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:01 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:01.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:01 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:01.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:01 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-16T20:33:01.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:01 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231016: Permission denied 2023-10-16T20:33:02.960 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:02 ntpd[28025]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:02.960 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:02 ntpd[28025]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:02.961 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:02 ntpd[28025]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:02.961 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:02 ntpd[28025]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:02.961 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:02 ntpd[28025]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:02.961 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:02 ntpd[28025]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:02.962 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:02 ntpd[28025]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-16T20:33:02.962 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:02 ntpd[28025]: can't open /var/log/ntpstats/peerstats.20231016: Permission denied 2023-10-16T20:33:02.962 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:02 ntpd[28025]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:02.962 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:02 ntpd[28025]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:02.962 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:02 ntpd[28025]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-16T20:33:02.963 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:02 ntpd[28025]: can't open /var/log/ntpstats/peerstats.20231016: Permission denied 2023-10-16T20:33:02.977 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:02 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:02.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:02 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:02.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:02 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:02.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:02 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:02.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:02 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-16T20:33:02.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:02 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231016: Permission denied 2023-10-16T20:33:03.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:03 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:03.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:03 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:03.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:03 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:03.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:03 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:03.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:03 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:03.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:03 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:03.980 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:03 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-16T20:33:03.980 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:03 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231016: Permission denied 2023-10-16T20:33:03.980 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:03 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:03.980 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:03 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:03.980 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:03 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-16T20:33:03.982 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:03 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231016: Permission denied 2023-10-16T20:33:04.961 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:04.961 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:04.961 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:04.961 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:04.962 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:04.962 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:04.962 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:04.962 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:04.963 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-16T20:33:04.963 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: can't open /var/log/ntpstats/peerstats.20231016: Permission denied 2023-10-16T20:33:04.963 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:04.963 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:04.963 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-16T20:33:04.964 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: can't open /var/log/ntpstats/peerstats.20231016: Permission denied 2023-10-16T20:33:04.964 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:04.964 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:04.964 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-16T20:33:04.964 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:04 ntpd[28025]: can't open /var/log/ntpstats/peerstats.20231016: Permission denied 2023-10-16T20:33:04.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:04 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:04.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:04 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:04.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:04 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:04.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:04 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:04.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:04 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:04.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:04 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:04.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:04 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-16T20:33:04.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:04 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231016: Permission denied 2023-10-16T20:33:05.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:05 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:05.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:05 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:05.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:05 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:05.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:05 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:05.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:05 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:05.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:05 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:05.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:05 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-16T20:33:05.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:05 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231016: Permission denied 2023-10-16T20:33:05.980 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:05 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:05.980 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:05 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:05.980 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:05 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-16T20:33:05.980 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:05 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231016: Permission denied 2023-10-16T20:33:06.960 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:06 ntpd[28025]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:06.960 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:06 ntpd[28025]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:06.961 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:06 ntpd[28025]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:06.961 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:06 ntpd[28025]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:06.961 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:06 ntpd[28025]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:06.961 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:06 ntpd[28025]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:06.961 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:06 ntpd[28025]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:06.962 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:06 ntpd[28025]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:06.962 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:06 ntpd[28025]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-16T20:33:06.962 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:06 ntpd[28025]: can't open /var/log/ntpstats/peerstats.20231016: Permission denied 2023-10-16T20:33:06.962 INFO:teuthology.orchestra.run.smithi088.stdout:16 Oct 20:33:06 ntpd[28025]: ntpd: time slew -0.004284 s 2023-10-16T20:33:06.963 INFO:teuthology.orchestra.run.smithi088.stdout:ntpd: time slew -0.004284s 2023-10-16T20:33:06.963 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:06 ntpd[28025]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-16T20:33:06.963 INFO:teuthology.orchestra.run.smithi088.stderr:16 Oct 20:33:06 ntpd[28025]: can't open /var/log/ntpstats/loopstats.20231016: Permission denied 2023-10-16T20:33:06.977 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:06 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:06.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:06 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:06.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:06 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:06.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:06 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:06.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:06 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-16T20:33:06.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:06 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231016: Permission denied 2023-10-16T20:33:07.043 INFO:teuthology.orchestra.run.smithi088.stdout: remote refid st t when poll reach delay offset jitter 2023-10-16T20:33:07.044 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================== 2023-10-16T20:33:07.044 INFO:teuthology.orchestra.run.smithi088.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-16T20:33:07.044 INFO:teuthology.orchestra.run.smithi088.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-16T20:33:07.044 INFO:teuthology.orchestra.run.smithi088.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-16T20:33:07.045 INFO:teuthology.orchestra.run.smithi088.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-16T20:33:07.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:07 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:07.978 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:07 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:07.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:07 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:07.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:07 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:07.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:07 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:07.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:07 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:07.979 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:07 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-16T20:33:07.980 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:07 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231016: Permission denied 2023-10-16T20:33:07.980 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:07 ntpd[27982]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-16T20:33:07.980 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:07 ntpd[27982]: can't open /var/log/ntpstats/rawstats.20231016: Permission denied 2023-10-16T20:33:07.980 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:07 ntpd[27982]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-16T20:33:07.980 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:07 ntpd[27982]: can't open /var/log/ntpstats/peerstats.20231016: Permission denied 2023-10-16T20:33:07.981 INFO:teuthology.orchestra.run.smithi191.stdout:16 Oct 20:33:07 ntpd[27982]: ntpd: time slew -0.001352 s 2023-10-16T20:33:07.981 INFO:teuthology.orchestra.run.smithi191.stdout:ntpd: time slew -0.001352s 2023-10-16T20:33:07.982 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:07 ntpd[27982]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-16T20:33:07.982 INFO:teuthology.orchestra.run.smithi191.stderr:16 Oct 20:33:07 ntpd[27982]: can't open /var/log/ntpstats/loopstats.20231016: Permission denied 2023-10-16T20:33:08.075 INFO:teuthology.orchestra.run.smithi191.stdout: remote refid st t when poll reach delay offset jitter 2023-10-16T20:33:08.075 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================== 2023-10-16T20:33:08.075 INFO:teuthology.orchestra.run.smithi191.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-16T20:33:08.076 INFO:teuthology.orchestra.run.smithi191.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-16T20:33:08.076 INFO:teuthology.orchestra.run.smithi191.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-16T20:33:08.076 INFO:teuthology.orchestra.run.smithi191.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-16T20:33:08.077 INFO:teuthology.run_tasks:Running task install... 2023-10-16T20:33:08.086 DEBUG:teuthology.task.install:project ceph 2023-10-16T20:33:08.086 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0'}} 2023-10-16T20:33:08.087 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0'} 2023-10-16T20:33:08.087 INFO:teuthology.task.install:Using flavor: default 2023-10-16T20:33:08.093 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-16T20:33:08.093 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-16T20:33:08.094 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-key list | grep Ceph 2023-10-16T20:33:08.095 DEBUG:teuthology.orchestra.run.smithi191:> sudo apt-key list | grep Ceph 2023-10-16T20:33:08.203 INFO:teuthology.orchestra.run.smithi088.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-16T20:33:08.229 INFO:teuthology.orchestra.run.smithi191.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-16T20:33:08.256 INFO:teuthology.orchestra.run.smithi088.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-16T20:33:08.257 INFO:teuthology.orchestra.run.smithi088.stdout:uid [ unknown] Ceph.com (release key) 2023-10-16T20:33:08.258 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-16T20:33:08.258 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-10-16T20:33:08.258 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 2023-10-16T20:33:08.282 INFO:teuthology.orchestra.run.smithi191.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-16T20:33:08.282 INFO:teuthology.orchestra.run.smithi191.stdout:uid [ unknown] Ceph.com (release key) 2023-10-16T20:33:08.283 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-16T20:33:08.283 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-10-16T20:33:08.283 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 2023-10-16T20:33:08.412 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default/ 2023-10-16T20:33:08.413 INFO:teuthology.task.install.deb:Package version is 18.0.0-6730-g4ba8a190-1jammy 2023-10-16T20:33:08.520 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:33:08.520 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-16T20:33:08.533 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default/ 2023-10-16T20:33:08.533 INFO:teuthology.task.install.deb:Package version is 18.0.0-6730-g4ba8a190-1jammy 2023-10-16T20:33:08.540 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get update 2023-10-16T20:33:08.640 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:33:08.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-16T20:33:08.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo apt-get update 2023-10-16T20:33:08.748 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-16T20:33:08.776 INFO:teuthology.orchestra.run.smithi088.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy InRelease 2023-10-16T20:33:08.797 INFO:teuthology.orchestra.run.smithi088.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-16T20:33:08.819 INFO:teuthology.orchestra.run.smithi088.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-16T20:33:08.840 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-16T20:33:08.846 INFO:teuthology.orchestra.run.smithi191.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-16T20:33:08.847 INFO:teuthology.orchestra.run.smithi191.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-16T20:33:08.852 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-16T20:33:08.852 INFO:teuthology.orchestra.run.smithi191.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-16T20:33:08.869 INFO:teuthology.orchestra.run.smithi191.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-16T20:33:08.895 INFO:teuthology.orchestra.run.smithi088.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-10-16T20:33:08.916 INFO:teuthology.orchestra.run.smithi191.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy InRelease 2023-10-16T20:33:08.926 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-16T20:33:08.937 INFO:teuthology.orchestra.run.smithi191.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-16T20:33:08.959 INFO:teuthology.orchestra.run.smithi191.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-16T20:33:08.981 INFO:teuthology.orchestra.run.smithi191.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-16T20:33:09.008 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-16T20:33:09.014 INFO:teuthology.orchestra.run.smithi191.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-10-16T20:33:09.352 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 28.6 kB in 1s (48.4 kB/s) 2023-10-16T20:33:09.453 INFO:teuthology.orchestra.run.smithi191.stdout:Fetched 28.6 kB in 1s (50.1 kB/s) 2023-10-16T20:33:10.251 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2023-10-16T20:33:10.268 DEBUG:teuthology.orchestra.run.smithi088:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6730-g4ba8a190-1jammy cephadm=18.0.0-6730-g4ba8a190-1jammy ceph-mds=18.0.0-6730-g4ba8a190-1jammy ceph-mgr=18.0.0-6730-g4ba8a190-1jammy ceph-common=18.0.0-6730-g4ba8a190-1jammy ceph-fuse=18.0.0-6730-g4ba8a190-1jammy ceph-test=18.0.0-6730-g4ba8a190-1jammy ceph-volume=18.0.0-6730-g4ba8a190-1jammy radosgw=18.0.0-6730-g4ba8a190-1jammy python3-rados=18.0.0-6730-g4ba8a190-1jammy python3-rgw=18.0.0-6730-g4ba8a190-1jammy python3-cephfs=18.0.0-6730-g4ba8a190-1jammy python3-rbd=18.0.0-6730-g4ba8a190-1jammy libcephfs2=18.0.0-6730-g4ba8a190-1jammy libcephfs-dev=18.0.0-6730-g4ba8a190-1jammy librados2=18.0.0-6730-g4ba8a190-1jammy librbd1=18.0.0-6730-g4ba8a190-1jammy rbd-fuse=18.0.0-6730-g4ba8a190-1jammy python3-cephfs=18.0.0-6730-g4ba8a190-1jammy cephfs-shell=18.0.0-6730-g4ba8a190-1jammy cephfs-top=18.0.0-6730-g4ba8a190-1jammy cephfs-mirror=18.0.0-6730-g4ba8a190-1jammy 2023-10-16T20:33:10.344 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2023-10-16T20:33:10.350 INFO:teuthology.orchestra.run.smithi191.stdout:Reading package lists... 2023-10-16T20:33:10.368 DEBUG:teuthology.orchestra.run.smithi191:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6730-g4ba8a190-1jammy cephadm=18.0.0-6730-g4ba8a190-1jammy ceph-mds=18.0.0-6730-g4ba8a190-1jammy ceph-mgr=18.0.0-6730-g4ba8a190-1jammy ceph-common=18.0.0-6730-g4ba8a190-1jammy ceph-fuse=18.0.0-6730-g4ba8a190-1jammy ceph-test=18.0.0-6730-g4ba8a190-1jammy ceph-volume=18.0.0-6730-g4ba8a190-1jammy radosgw=18.0.0-6730-g4ba8a190-1jammy python3-rados=18.0.0-6730-g4ba8a190-1jammy python3-rgw=18.0.0-6730-g4ba8a190-1jammy python3-cephfs=18.0.0-6730-g4ba8a190-1jammy python3-rbd=18.0.0-6730-g4ba8a190-1jammy libcephfs2=18.0.0-6730-g4ba8a190-1jammy libcephfs-dev=18.0.0-6730-g4ba8a190-1jammy librados2=18.0.0-6730-g4ba8a190-1jammy librbd1=18.0.0-6730-g4ba8a190-1jammy rbd-fuse=18.0.0-6730-g4ba8a190-1jammy python3-cephfs=18.0.0-6730-g4ba8a190-1jammy cephfs-shell=18.0.0-6730-g4ba8a190-1jammy cephfs-top=18.0.0-6730-g4ba8a190-1jammy cephfs-mirror=18.0.0-6730-g4ba8a190-1jammy 2023-10-16T20:33:10.443 INFO:teuthology.orchestra.run.smithi191.stdout:Reading package lists... 2023-10-16T20:33:10.507 INFO:teuthology.orchestra.run.smithi088.stdout:Building dependency tree... 2023-10-16T20:33:10.507 INFO:teuthology.orchestra.run.smithi088.stdout:Reading state information... 2023-10-16T20:33:10.607 INFO:teuthology.orchestra.run.smithi191.stdout:Building dependency tree... 2023-10-16T20:33:10.608 INFO:teuthology.orchestra.run.smithi191.stdout:Reading state information... 2023-10-16T20:33:10.655 INFO:teuthology.orchestra.run.smithi088.stdout:The following packages were automatically installed and are no longer required: 2023-10-16T20:33:10.655 INFO:teuthology.orchestra.run.smithi088.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-16T20:33:10.655 INFO:teuthology.orchestra.run.smithi088.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-16T20:33:10.655 INFO:teuthology.orchestra.run.smithi088.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-16T20:33:10.656 INFO:teuthology.orchestra.run.smithi088.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-16T20:33:10.656 INFO:teuthology.orchestra.run.smithi088.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-16T20:33:10.656 INFO:teuthology.orchestra.run.smithi088.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-16T20:33:10.658 INFO:teuthology.orchestra.run.smithi088.stdout:The following additional packages will be installed: 2023-10-16T20:33:10.658 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-16T20:33:10.658 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-16T20:33:10.658 INFO:teuthology.orchestra.run.smithi088.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-16T20:33:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-16T20:33:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-16T20:33:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-16T20:33:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-16T20:33:10.659 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-16T20:33:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-10-16T20:33:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-16T20:33:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-16T20:33:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-16T20:33:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-16T20:33:10.660 INFO:teuthology.orchestra.run.smithi088.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-16T20:33:10.661 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-16T20:33:10.661 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-16T20:33:10.661 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-10-16T20:33:10.661 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-10-16T20:33:10.661 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-10-16T20:33:10.662 INFO:teuthology.orchestra.run.smithi088.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-10-16T20:33:10.662 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-10-16T20:33:10.662 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-10-16T20:33:10.662 INFO:teuthology.orchestra.run.smithi088.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-16T20:33:10.662 INFO:teuthology.orchestra.run.smithi088.stdout:Suggested packages: 2023-10-16T20:33:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-16T20:33:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-16T20:33:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-16T20:33:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-16T20:33:10.663 INFO:teuthology.orchestra.run.smithi088.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-16T20:33:10.664 INFO:teuthology.orchestra.run.smithi088.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-16T20:33:10.664 INFO:teuthology.orchestra.run.smithi088.stdout: python3-watchdog 2023-10-16T20:33:10.664 INFO:teuthology.orchestra.run.smithi088.stdout:Recommended packages: 2023-10-16T20:33:10.664 INFO:teuthology.orchestra.run.smithi088.stdout: btrfs-tools 2023-10-16T20:33:10.756 INFO:teuthology.orchestra.run.smithi191.stdout:The following packages were automatically installed and are no longer required: 2023-10-16T20:33:10.757 INFO:teuthology.orchestra.run.smithi191.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-16T20:33:10.757 INFO:teuthology.orchestra.run.smithi191.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-16T20:33:10.757 INFO:teuthology.orchestra.run.smithi191.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-16T20:33:10.757 INFO:teuthology.orchestra.run.smithi191.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-16T20:33:10.759 INFO:teuthology.orchestra.run.smithi191.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-16T20:33:10.759 INFO:teuthology.orchestra.run.smithi191.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-16T20:33:10.759 INFO:teuthology.orchestra.run.smithi191.stdout:The following additional packages will be installed: 2023-10-16T20:33:10.760 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-16T20:33:10.760 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-16T20:33:10.760 INFO:teuthology.orchestra.run.smithi191.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-16T20:33:10.760 INFO:teuthology.orchestra.run.smithi191.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-16T20:33:10.760 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-16T20:33:10.761 INFO:teuthology.orchestra.run.smithi191.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-16T20:33:10.761 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-16T20:33:10.761 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-16T20:33:10.761 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-10-16T20:33:10.761 INFO:teuthology.orchestra.run.smithi191.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-16T20:33:10.762 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-16T20:33:10.762 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-16T20:33:10.762 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-16T20:33:10.762 INFO:teuthology.orchestra.run.smithi191.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-16T20:33:10.762 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-16T20:33:10.763 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-16T20:33:10.763 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-10-16T20:33:10.763 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-10-16T20:33:10.763 INFO:teuthology.orchestra.run.smithi191.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-10-16T20:33:10.763 INFO:teuthology.orchestra.run.smithi191.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-10-16T20:33:10.764 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-10-16T20:33:10.764 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-10-16T20:33:10.764 INFO:teuthology.orchestra.run.smithi191.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-16T20:33:10.764 INFO:teuthology.orchestra.run.smithi191.stdout:Suggested packages: 2023-10-16T20:33:10.765 INFO:teuthology.orchestra.run.smithi191.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-16T20:33:10.765 INFO:teuthology.orchestra.run.smithi191.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-16T20:33:10.765 INFO:teuthology.orchestra.run.smithi191.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-16T20:33:10.765 INFO:teuthology.orchestra.run.smithi191.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-16T20:33:10.765 INFO:teuthology.orchestra.run.smithi191.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-16T20:33:10.766 INFO:teuthology.orchestra.run.smithi191.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-16T20:33:10.766 INFO:teuthology.orchestra.run.smithi191.stdout: python3-watchdog 2023-10-16T20:33:10.766 INFO:teuthology.orchestra.run.smithi191.stdout:Recommended packages: 2023-10-16T20:33:10.766 INFO:teuthology.orchestra.run.smithi191.stdout: btrfs-tools 2023-10-16T20:33:10.800 INFO:teuthology.orchestra.run.smithi088.stdout:The following NEW packages will be installed: 2023-10-16T20:33:10.800 INFO:teuthology.orchestra.run.smithi088.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-16T20:33:10.800 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-16T20:33:10.800 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-16T20:33:10.801 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-10-16T20:33:10.801 INFO:teuthology.orchestra.run.smithi088.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-16T20:33:10.801 INFO:teuthology.orchestra.run.smithi088.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-10-16T20:33:10.801 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-16T20:33:10.802 INFO:teuthology.orchestra.run.smithi088.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-16T20:33:10.803 INFO:teuthology.orchestra.run.smithi088.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-16T20:33:10.803 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-16T20:33:10.803 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-10-16T20:33:10.803 INFO:teuthology.orchestra.run.smithi088.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-10-16T20:33:10.804 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-16T20:33:10.804 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-16T20:33:10.804 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-16T20:33:10.804 INFO:teuthology.orchestra.run.smithi088.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-16T20:33:10.804 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-16T20:33:10.805 INFO:teuthology.orchestra.run.smithi088.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-16T20:33:10.805 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-10-16T20:33:10.805 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-16T20:33:10.805 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-16T20:33:10.805 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-16T20:33:10.806 INFO:teuthology.orchestra.run.smithi088.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-16T20:33:10.806 INFO:teuthology.orchestra.run.smithi088.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-16T20:33:10.806 INFO:teuthology.orchestra.run.smithi088.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-16T20:33:10.806 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-16T20:33:10.806 INFO:teuthology.orchestra.run.smithi088.stdout: socat xmlstarlet 2023-10-16T20:33:10.867 INFO:teuthology.orchestra.run.smithi088.stdout:0 upgraded, 115 newly installed, 0 to remove and 259 not upgraded. 2023-10-16T20:33:10.867 INFO:teuthology.orchestra.run.smithi088.stdout:Need to get 184 MB of archives. 2023-10-16T20:33:10.868 INFO:teuthology.orchestra.run.smithi088.stdout:After this operation, 826 MB of additional disk space will be used. 2023-10-16T20:33:10.868 INFO:teuthology.orchestra.run.smithi088.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-10-16T20:33:10.885 INFO:teuthology.orchestra.run.smithi088.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-16T20:33:10.894 INFO:teuthology.orchestra.run.smithi088.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-10-16T20:33:10.904 INFO:teuthology.orchestra.run.smithi191.stdout:The following NEW packages will be installed: 2023-10-16T20:33:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-16T20:33:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-16T20:33:10.904 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-16T20:33:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-10-16T20:33:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-16T20:33:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-10-16T20:33:10.905 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-16T20:33:10.906 INFO:teuthology.orchestra.run.smithi191.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-16T20:33:10.906 INFO:teuthology.orchestra.run.smithi191.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-16T20:33:10.906 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-16T20:33:10.906 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-10-16T20:33:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-10-16T20:33:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-16T20:33:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-16T20:33:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-16T20:33:10.907 INFO:teuthology.orchestra.run.smithi191.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-16T20:33:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-16T20:33:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-16T20:33:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-10-16T20:33:10.908 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-16T20:33:10.909 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-16T20:33:10.909 INFO:teuthology.orchestra.run.smithi191.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-16T20:33:10.909 INFO:teuthology.orchestra.run.smithi191.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-16T20:33:10.909 INFO:teuthology.orchestra.run.smithi191.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-16T20:33:10.910 INFO:teuthology.orchestra.run.smithi191.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-16T20:33:10.910 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-16T20:33:10.910 INFO:teuthology.orchestra.run.smithi191.stdout: socat xmlstarlet 2023-10-16T20:33:10.938 INFO:teuthology.orchestra.run.smithi088.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-16T20:33:10.945 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-16T20:33:10.961 INFO:teuthology.orchestra.run.smithi088.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-16T20:33:10.963 INFO:teuthology.orchestra.run.smithi088.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-16T20:33:10.974 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-16T20:33:11.009 INFO:teuthology.orchestra.run.smithi088.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-16T20:33:11.011 INFO:teuthology.orchestra.run.smithi088.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-16T20:33:11.018 INFO:teuthology.orchestra.run.smithi088.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-16T20:33:11.020 INFO:teuthology.orchestra.run.smithi088.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-16T20:33:11.021 INFO:teuthology.orchestra.run.smithi088.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-16T20:33:11.027 INFO:teuthology.orchestra.run.smithi088.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-16T20:33:11.028 INFO:teuthology.orchestra.run.smithi088.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-16T20:33:11.028 INFO:teuthology.orchestra.run.smithi088.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-16T20:33:11.029 INFO:teuthology.orchestra.run.smithi088.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-16T20:33:11.029 INFO:teuthology.orchestra.run.smithi088.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-16T20:33:11.044 INFO:teuthology.orchestra.run.smithi088.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-16T20:33:11.045 INFO:teuthology.orchestra.run.smithi088.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-16T20:33:11.045 INFO:teuthology.orchestra.run.smithi088.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-16T20:33:11.045 INFO:teuthology.orchestra.run.smithi088.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-16T20:33:11.061 INFO:teuthology.orchestra.run.smithi088.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-16T20:33:11.063 INFO:teuthology.orchestra.run.smithi088.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-16T20:33:11.063 INFO:teuthology.orchestra.run.smithi088.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-16T20:33:11.064 INFO:teuthology.orchestra.run.smithi088.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-16T20:33:11.064 INFO:teuthology.orchestra.run.smithi088.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-16T20:33:11.065 INFO:teuthology.orchestra.run.smithi088.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-16T20:33:11.077 INFO:teuthology.orchestra.run.smithi088.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-16T20:33:11.078 INFO:teuthology.orchestra.run.smithi088.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-16T20:33:11.079 INFO:teuthology.orchestra.run.smithi088.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-16T20:33:11.079 INFO:teuthology.orchestra.run.smithi088.stdout:Get:32 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6730-g4ba8a190-1jammy [3,449 kB] 2023-10-16T20:33:11.080 INFO:teuthology.orchestra.run.smithi088.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-16T20:33:11.094 INFO:teuthology.orchestra.run.smithi088.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-16T20:33:11.098 INFO:teuthology.orchestra.run.smithi088.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-16T20:33:11.099 INFO:teuthology.orchestra.run.smithi191.stdout:0 upgraded, 115 newly installed, 0 to remove and 259 not upgraded. 2023-10-16T20:33:11.100 INFO:teuthology.orchestra.run.smithi191.stdout:Need to get 184 MB of archives. 2023-10-16T20:33:11.100 INFO:teuthology.orchestra.run.smithi191.stdout:After this operation, 826 MB of additional disk space will be used. 2023-10-16T20:33:11.100 INFO:teuthology.orchestra.run.smithi191.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-10-16T20:33:11.100 INFO:teuthology.orchestra.run.smithi088.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-16T20:33:11.101 INFO:teuthology.orchestra.run.smithi088.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-16T20:33:11.101 INFO:teuthology.orchestra.run.smithi088.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-16T20:33:11.101 INFO:teuthology.orchestra.run.smithi088.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-16T20:33:11.111 INFO:teuthology.orchestra.run.smithi088.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-16T20:33:11.112 INFO:teuthology.orchestra.run.smithi088.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-16T20:33:11.113 INFO:teuthology.orchestra.run.smithi088.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-16T20:33:11.114 INFO:teuthology.orchestra.run.smithi088.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-16T20:33:11.128 INFO:teuthology.orchestra.run.smithi088.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-16T20:33:11.128 INFO:teuthology.orchestra.run.smithi088.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-16T20:33:11.130 INFO:teuthology.orchestra.run.smithi088.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-16T20:33:11.174 INFO:teuthology.orchestra.run.smithi088.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-16T20:33:11.175 INFO:teuthology.orchestra.run.smithi088.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-16T20:33:11.176 INFO:teuthology.orchestra.run.smithi088.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-16T20:33:11.177 INFO:teuthology.orchestra.run.smithi088.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-16T20:33:11.177 INFO:teuthology.orchestra.run.smithi088.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-16T20:33:11.179 INFO:teuthology.orchestra.run.smithi088.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-16T20:33:11.181 INFO:teuthology.orchestra.run.smithi191.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6730-g4ba8a190-1jammy [3,449 kB] 2023-10-16T20:33:11.182 INFO:teuthology.orchestra.run.smithi191.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-16T20:33:11.197 INFO:teuthology.orchestra.run.smithi088.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-16T20:33:11.199 INFO:teuthology.orchestra.run.smithi088.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-16T20:33:11.199 INFO:teuthology.orchestra.run.smithi088.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-16T20:33:11.216 INFO:teuthology.orchestra.run.smithi088.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-16T20:33:11.216 INFO:teuthology.orchestra.run.smithi088.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-16T20:33:11.217 INFO:teuthology.orchestra.run.smithi088.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-16T20:33:11.217 INFO:teuthology.orchestra.run.smithi088.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-16T20:33:11.218 INFO:teuthology.orchestra.run.smithi088.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-16T20:33:11.218 INFO:teuthology.orchestra.run.smithi088.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-16T20:33:11.227 INFO:teuthology.orchestra.run.smithi088.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-16T20:33:11.229 INFO:teuthology.orchestra.run.smithi191.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-10-16T20:33:11.230 INFO:teuthology.orchestra.run.smithi088.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-16T20:33:11.231 INFO:teuthology.orchestra.run.smithi088.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-16T20:33:11.244 INFO:teuthology.orchestra.run.smithi088.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-16T20:33:11.245 INFO:teuthology.orchestra.run.smithi088.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-16T20:33:11.246 INFO:teuthology.orchestra.run.smithi088.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-16T20:33:11.249 INFO:teuthology.orchestra.run.smithi088.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-16T20:33:11.252 INFO:teuthology.orchestra.run.smithi088.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-10-16T20:33:11.252 INFO:teuthology.orchestra.run.smithi088.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-16T20:33:11.257 INFO:teuthology.orchestra.run.smithi088.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-16T20:33:11.261 INFO:teuthology.orchestra.run.smithi088.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-16T20:33:11.261 INFO:teuthology.orchestra.run.smithi088.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-16T20:33:11.262 INFO:teuthology.orchestra.run.smithi088.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-16T20:33:11.277 INFO:teuthology.orchestra.run.smithi088.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-16T20:33:11.279 INFO:teuthology.orchestra.run.smithi088.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-16T20:33:11.280 INFO:teuthology.orchestra.run.smithi088.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-16T20:33:11.286 INFO:teuthology.orchestra.run.smithi088.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-16T20:33:11.287 INFO:teuthology.orchestra.run.smithi088.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-16T20:33:11.287 INFO:teuthology.orchestra.run.smithi088.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-16T20:33:11.289 INFO:teuthology.orchestra.run.smithi088.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-16T20:33:11.294 INFO:teuthology.orchestra.run.smithi088.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-16T20:33:11.433 INFO:teuthology.orchestra.run.smithi191.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-16T20:33:11.477 INFO:teuthology.orchestra.run.smithi191.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-16T20:33:11.553 INFO:teuthology.orchestra.run.smithi191.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-16T20:33:11.564 INFO:teuthology.orchestra.run.smithi191.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-16T20:33:11.610 INFO:teuthology.orchestra.run.smithi191.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-16T20:33:11.776 INFO:teuthology.orchestra.run.smithi191.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-16T20:33:11.783 INFO:teuthology.orchestra.run.smithi191.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-16T20:33:11.811 INFO:teuthology.orchestra.run.smithi191.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-16T20:33:11.983 INFO:teuthology.orchestra.run.smithi191.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-16T20:33:12.054 INFO:teuthology.orchestra.run.smithi191.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6730-g4ba8a190-1jammy [3,130 kB] 2023-10-16T20:33:12.175 INFO:teuthology.orchestra.run.smithi191.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-16T20:33:12.493 INFO:teuthology.orchestra.run.smithi191.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6730-g4ba8a190-1jammy [688 kB] 2023-10-16T20:33:12.592 INFO:teuthology.orchestra.run.smithi191.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-16T20:33:12.599 INFO:teuthology.orchestra.run.smithi191.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-16T20:33:12.600 INFO:teuthology.orchestra.run.smithi191.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-16T20:33:12.607 INFO:teuthology.orchestra.run.smithi191.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6730-g4ba8a190-1jammy [356 kB] 2023-10-16T20:33:12.612 INFO:teuthology.orchestra.run.smithi191.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-16T20:33:12.612 INFO:teuthology.orchestra.run.smithi191.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-16T20:33:12.614 INFO:teuthology.orchestra.run.smithi191.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-16T20:33:12.616 INFO:teuthology.orchestra.run.smithi191.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-16T20:33:12.618 INFO:teuthology.orchestra.run.smithi191.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-16T20:33:12.619 INFO:teuthology.orchestra.run.smithi191.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-16T20:33:12.645 INFO:teuthology.orchestra.run.smithi088.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6730-g4ba8a190-1jammy [3,130 kB] 2023-10-16T20:33:12.660 INFO:teuthology.orchestra.run.smithi191.stdout:Get:27 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6730-g4ba8a190-1jammy [32.8 kB] 2023-10-16T20:33:12.661 INFO:teuthology.orchestra.run.smithi191.stdout:Get:28 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6730-g4ba8a190-1jammy [180 kB] 2023-10-16T20:33:12.702 INFO:teuthology.orchestra.run.smithi191.stdout:Get:29 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6730-g4ba8a190-1jammy [62.4 kB] 2023-10-16T20:33:12.705 INFO:teuthology.orchestra.run.smithi191.stdout:Get:30 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6730-g4ba8a190-1jammy [329 kB] 2023-10-16T20:33:12.751 INFO:teuthology.orchestra.run.smithi191.stdout:Get:31 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6730-g4ba8a190-1jammy [6,222 kB] 2023-10-16T20:33:12.784 INFO:teuthology.orchestra.run.smithi191.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-16T20:33:13.228 INFO:teuthology.orchestra.run.smithi191.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-16T20:33:13.238 INFO:teuthology.orchestra.run.smithi191.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-16T20:33:13.244 INFO:teuthology.orchestra.run.smithi191.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-16T20:33:13.248 INFO:teuthology.orchestra.run.smithi191.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-16T20:33:13.266 INFO:teuthology.orchestra.run.smithi191.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-16T20:33:13.270 INFO:teuthology.orchestra.run.smithi191.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-16T20:33:13.272 INFO:teuthology.orchestra.run.smithi191.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-16T20:33:13.313 INFO:teuthology.orchestra.run.smithi191.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-16T20:33:13.327 INFO:teuthology.orchestra.run.smithi191.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-16T20:33:13.332 INFO:teuthology.orchestra.run.smithi191.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-16T20:33:13.430 INFO:teuthology.orchestra.run.smithi191.stdout:Get:43 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6730-g4ba8a190-1jammy [112 kB] 2023-10-16T20:33:13.432 INFO:teuthology.orchestra.run.smithi191.stdout:Get:44 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6730-g4ba8a190-1jammy [439 kB] 2023-10-16T20:33:13.457 INFO:teuthology.orchestra.run.smithi088.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6730-g4ba8a190-1jammy [688 kB] 2023-10-16T20:33:13.475 INFO:teuthology.orchestra.run.smithi191.stdout:Get:45 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6730-g4ba8a190-1jammy [22.6 MB] 2023-10-16T20:33:13.585 INFO:teuthology.orchestra.run.smithi088.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6730-g4ba8a190-1jammy [356 kB] 2023-10-16T20:33:13.586 INFO:teuthology.orchestra.run.smithi191.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-16T20:33:13.663 INFO:teuthology.orchestra.run.smithi088.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6730-g4ba8a190-1jammy [32.8 kB] 2023-10-16T20:33:13.668 INFO:teuthology.orchestra.run.smithi191.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-16T20:33:13.673 INFO:teuthology.orchestra.run.smithi088.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6730-g4ba8a190-1jammy [180 kB] 2023-10-16T20:33:13.719 INFO:teuthology.orchestra.run.smithi088.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6730-g4ba8a190-1jammy [62.4 kB] 2023-10-16T20:33:13.725 INFO:teuthology.orchestra.run.smithi088.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6730-g4ba8a190-1jammy [329 kB] 2023-10-16T20:33:13.761 INFO:teuthology.orchestra.run.smithi191.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-16T20:33:13.809 INFO:teuthology.orchestra.run.smithi088.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6730-g4ba8a190-1jammy [6,222 kB] 2023-10-16T20:33:13.837 INFO:teuthology.orchestra.run.smithi191.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-16T20:33:13.944 INFO:teuthology.orchestra.run.smithi191.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-16T20:33:14.034 INFO:teuthology.orchestra.run.smithi191.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-16T20:33:14.053 INFO:teuthology.orchestra.run.smithi191.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-16T20:33:14.083 INFO:teuthology.orchestra.run.smithi191.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-16T20:33:14.096 INFO:teuthology.orchestra.run.smithi191.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-16T20:33:14.134 INFO:teuthology.orchestra.run.smithi191.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-16T20:33:14.142 INFO:teuthology.orchestra.run.smithi191.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-16T20:33:14.332 INFO:teuthology.orchestra.run.smithi191.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-16T20:33:15.100 INFO:teuthology.orchestra.run.smithi088.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6730-g4ba8a190-1jammy [112 kB] 2023-10-16T20:33:15.112 INFO:teuthology.orchestra.run.smithi088.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6730-g4ba8a190-1jammy [439 kB] 2023-10-16T20:33:15.160 INFO:teuthology.orchestra.run.smithi191.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-16T20:33:15.161 INFO:teuthology.orchestra.run.smithi191.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-16T20:33:15.162 INFO:teuthology.orchestra.run.smithi191.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-16T20:33:15.163 INFO:teuthology.orchestra.run.smithi191.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-16T20:33:15.163 INFO:teuthology.orchestra.run.smithi191.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-16T20:33:15.164 INFO:teuthology.orchestra.run.smithi191.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-16T20:33:15.188 INFO:teuthology.orchestra.run.smithi191.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-16T20:33:15.188 INFO:teuthology.orchestra.run.smithi088.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6730-g4ba8a190-1jammy [22.6 MB] 2023-10-16T20:33:15.190 INFO:teuthology.orchestra.run.smithi191.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-16T20:33:15.190 INFO:teuthology.orchestra.run.smithi191.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-16T20:33:15.260 INFO:teuthology.orchestra.run.smithi191.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-16T20:33:15.425 INFO:teuthology.orchestra.run.smithi191.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-16T20:33:15.545 INFO:teuthology.orchestra.run.smithi191.stdout:Get:69 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6730-g4ba8a190-1jammy [6,851 kB] 2023-10-16T20:33:15.683 INFO:teuthology.orchestra.run.smithi191.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-16T20:33:15.715 INFO:teuthology.orchestra.run.smithi191.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-16T20:33:15.778 INFO:teuthology.orchestra.run.smithi191.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-16T20:33:15.813 INFO:teuthology.orchestra.run.smithi191.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-16T20:33:15.830 INFO:teuthology.orchestra.run.smithi191.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-16T20:33:15.859 INFO:teuthology.orchestra.run.smithi191.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-16T20:33:15.993 INFO:teuthology.orchestra.run.smithi191.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-16T20:33:16.027 INFO:teuthology.orchestra.run.smithi191.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-16T20:33:16.048 INFO:teuthology.orchestra.run.smithi191.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-16T20:33:16.056 INFO:teuthology.orchestra.run.smithi191.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-16T20:33:16.223 INFO:teuthology.orchestra.run.smithi191.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6730-g4ba8a190-1jammy [238 kB] 2023-10-16T20:33:16.250 INFO:teuthology.orchestra.run.smithi191.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6730-g4ba8a190-1jammy [128 kB] 2023-10-16T20:33:16.253 INFO:teuthology.orchestra.run.smithi191.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6730-g4ba8a190-1jammy [1,496 kB] 2023-10-16T20:33:16.259 INFO:teuthology.orchestra.run.smithi191.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-16T20:33:16.405 INFO:teuthology.orchestra.run.smithi191.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6730-g4ba8a190-1jammy [9,861 kB] 2023-10-16T20:33:16.739 INFO:teuthology.orchestra.run.smithi191.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-10-16T20:33:16.744 INFO:teuthology.orchestra.run.smithi191.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-16T20:33:16.873 INFO:teuthology.orchestra.run.smithi191.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-16T20:33:16.919 INFO:teuthology.orchestra.run.smithi191.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-16T20:33:16.920 INFO:teuthology.orchestra.run.smithi191.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-16T20:33:16.926 INFO:teuthology.orchestra.run.smithi191.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-16T20:33:16.927 INFO:teuthology.orchestra.run.smithi191.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-16T20:33:16.944 INFO:teuthology.orchestra.run.smithi191.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-16T20:33:16.952 INFO:teuthology.orchestra.run.smithi191.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-16T20:33:17.011 INFO:teuthology.orchestra.run.smithi191.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-16T20:33:17.176 INFO:teuthology.orchestra.run.smithi191.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-16T20:33:17.249 INFO:teuthology.orchestra.run.smithi191.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6730-g4ba8a190-1jammy [31.7 MB] 2023-10-16T20:33:17.341 INFO:teuthology.orchestra.run.smithi191.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-16T20:33:17.630 INFO:teuthology.orchestra.run.smithi191.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-16T20:33:17.656 INFO:teuthology.orchestra.run.smithi191.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-16T20:33:19.168 INFO:teuthology.orchestra.run.smithi088.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6730-g4ba8a190-1jammy [6,851 kB] 2023-10-16T20:33:19.450 INFO:teuthology.orchestra.run.smithi191.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6730-g4ba8a190-1jammy [14.0 kB] 2023-10-16T20:33:19.450 INFO:teuthology.orchestra.run.smithi191.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6730-g4ba8a190-1jammy [839 kB] 2023-10-16T20:33:19.499 INFO:teuthology.orchestra.run.smithi191.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6730-g4ba8a190-1jammy [2,146 kB] 2023-10-16T20:33:19.649 INFO:teuthology.orchestra.run.smithi191.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6730-g4ba8a190-1jammy [283 kB] 2023-10-16T20:33:19.666 INFO:teuthology.orchestra.run.smithi191.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6730-g4ba8a190-1jammy [138 kB] 2023-10-16T20:33:19.669 INFO:teuthology.orchestra.run.smithi191.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6730-g4ba8a190-1jammy [1,906 kB] 2023-10-16T20:33:19.814 INFO:teuthology.orchestra.run.smithi191.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6730-g4ba8a190-1jammy [8,625 kB] 2023-10-16T20:33:20.111 INFO:teuthology.orchestra.run.smithi088.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6730-g4ba8a190-1jammy [238 kB] 2023-10-16T20:33:20.130 INFO:teuthology.orchestra.run.smithi088.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6730-g4ba8a190-1jammy [128 kB] 2023-10-16T20:33:20.173 INFO:teuthology.orchestra.run.smithi088.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6730-g4ba8a190-1jammy [1,496 kB] 2023-10-16T20:33:20.389 INFO:teuthology.orchestra.run.smithi088.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6730-g4ba8a190-1jammy [9,861 kB] 2023-10-16T20:33:20.492 INFO:teuthology.orchestra.run.smithi191.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6730-g4ba8a190-1jammy [14.2 kB] 2023-10-16T20:33:20.492 INFO:teuthology.orchestra.run.smithi191.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6730-g4ba8a190-1jammy [47.8 MB] 2023-10-16T20:33:21.598 INFO:teuthology.orchestra.run.smithi088.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6730-g4ba8a190-1jammy [31.7 MB] 2023-10-16T20:33:23.811 INFO:teuthology.orchestra.run.smithi191.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6730-g4ba8a190-1jammy [120 kB] 2023-10-16T20:33:23.812 INFO:teuthology.orchestra.run.smithi191.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-6730-g4ba8a190-1jammy [227 kB] 2023-10-16T20:33:23.814 INFO:teuthology.orchestra.run.smithi191.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-6730-g4ba8a190-1jammy [33.9 kB] 2023-10-16T20:33:23.815 INFO:teuthology.orchestra.run.smithi191.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-6730-g4ba8a190-1jammy [23.4 kB] 2023-10-16T20:33:23.815 INFO:teuthology.orchestra.run.smithi191.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6730-g4ba8a190-1jammy [38.3 kB] 2023-10-16T20:33:23.816 INFO:teuthology.orchestra.run.smithi191.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6730-g4ba8a190-1jammy [12.3 MB] 2023-10-16T20:33:24.573 INFO:teuthology.orchestra.run.smithi191.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6730-g4ba8a190-1jammy [91.6 kB] 2023-10-16T20:33:24.970 INFO:teuthology.orchestra.run.smithi088.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6730-g4ba8a190-1jammy [14.0 kB] 2023-10-16T20:33:24.970 INFO:teuthology.orchestra.run.smithi088.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6730-g4ba8a190-1jammy [839 kB] 2023-10-16T20:33:25.037 INFO:teuthology.orchestra.run.smithi088.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6730-g4ba8a190-1jammy [2,146 kB] 2023-10-16T20:33:25.146 INFO:teuthology.orchestra.run.smithi191.stdout:Fetched 184 MB in 14s (13.5 MB/s) 2023-10-16T20:33:25.217 INFO:teuthology.orchestra.run.smithi088.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6730-g4ba8a190-1jammy [283 kB] 2023-10-16T20:33:25.242 INFO:teuthology.orchestra.run.smithi088.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6730-g4ba8a190-1jammy [138 kB] 2023-10-16T20:33:25.250 INFO:teuthology.orchestra.run.smithi088.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6730-g4ba8a190-1jammy [1,906 kB] 2023-10-16T20:33:25.273 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-16T20:33:25.341 INFO:teuthology.orchestra.run.smithi191.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132932 files and directories currently installed.) 2023-10-16T20:33:25.345 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-10-16T20:33:25.380 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-10-16T20:33:25.397 INFO:teuthology.orchestra.run.smithi088.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6730-g4ba8a190-1jammy [8,625 kB] 2023-10-16T20:33:25.658 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-16T20:33:25.671 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-16T20:33:25.716 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-16T20:33:26.036 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-cmd2. 2023-10-16T20:33:26.049 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-10-16T20:33:26.078 INFO:teuthology.orchestra.run.smithi088.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6730-g4ba8a190-1jammy [14.2 kB] 2023-10-16T20:33:26.079 INFO:teuthology.orchestra.run.smithi088.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6730-g4ba8a190-1jammy [47.8 MB] 2023-10-16T20:33:26.094 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-10-16T20:33:26.372 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-16T20:33:26.385 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-16T20:33:26.464 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-16T20:33:26.834 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-16T20:33:26.847 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-16T20:33:26.892 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-16T20:33:27.229 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-16T20:33:27.243 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-16T20:33:27.287 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-16T20:33:27.565 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-16T20:33:27.580 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-16T20:33:27.615 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-16T20:33:28.302 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-16T20:33:28.318 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-16T20:33:28.369 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-16T20:33:28.755 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-16T20:33:28.770 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-16T20:33:28.814 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-16T20:33:29.117 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-16T20:33:29.131 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-16T20:33:29.175 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-16T20:33:29.389 INFO:teuthology.orchestra.run.smithi088.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6730-g4ba8a190-1jammy [120 kB] 2023-10-16T20:33:29.390 INFO:teuthology.orchestra.run.smithi088.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-6730-g4ba8a190-1jammy [227 kB] 2023-10-16T20:33:29.392 INFO:teuthology.orchestra.run.smithi088.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-6730-g4ba8a190-1jammy [33.9 kB] 2023-10-16T20:33:29.392 INFO:teuthology.orchestra.run.smithi088.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-6730-g4ba8a190-1jammy [23.4 kB] 2023-10-16T20:33:29.392 INFO:teuthology.orchestra.run.smithi088.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6730-g4ba8a190-1jammy [38.3 kB] 2023-10-16T20:33:29.393 INFO:teuthology.orchestra.run.smithi088.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6730-g4ba8a190-1jammy [12.3 MB] 2023-10-16T20:33:29.520 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-16T20:33:29.535 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-16T20:33:29.578 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-16T20:33:29.877 INFO:teuthology.orchestra.run.smithi088.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/rishabh-2023oct13/4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6730-g4ba8a190-1jammy [91.6 kB] 2023-10-16T20:33:29.923 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package librados2. 2023-10-16T20:33:29.938 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../011-librados2_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:29.981 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking librados2 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:30.401 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package librbd1. 2023-10-16T20:33:30.416 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../012-librbd1_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:30.449 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 184 MB in 19s (9,654 kB/s) 2023-10-16T20:33:30.459 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking librbd1 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:30.596 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-16T20:33:30.665 INFO:teuthology.orchestra.run.smithi088.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132932 files and directories currently installed.) 2023-10-16T20:33:30.668 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-10-16T20:33:30.712 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-10-16T20:33:30.862 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libcephfs2. 2023-10-16T20:33:30.876 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../013-libcephfs2_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:30.945 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libcephfs2 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:31.031 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-16T20:33:31.044 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-16T20:33:31.081 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-16T20:33:31.231 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-rados. 2023-10-16T20:33:31.246 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../014-python3-rados_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:31.289 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-rados (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:31.383 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cmd2. 2023-10-16T20:33:31.397 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-10-16T20:33:31.425 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-10-16T20:33:31.584 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-16T20:33:31.599 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:33:31.634 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-ceph-argparse (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:31.712 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-16T20:33:31.725 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-16T20:33:31.795 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-16T20:33:31.878 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-cephfs. 2023-10-16T20:33:31.893 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../016-python3-cephfs_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:31.970 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-cephfs (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:32.123 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-16T20:33:32.136 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-16T20:33:32.181 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-16T20:33:32.281 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-16T20:33:32.296 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../017-python3-ceph-common_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:33:32.339 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-ceph-common (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:32.643 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-16T20:33:32.657 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-16T20:33:32.684 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-prettytable. 2023-10-16T20:33:32.699 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-10-16T20:33:32.710 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-16T20:33:32.742 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-16T20:33:33.045 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-rbd. 2023-10-16T20:33:33.060 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../019-python3-rbd_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:33.095 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-rbd (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:33.121 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-16T20:33:33.136 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-16T20:33:33.188 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-16T20:33:33.423 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-16T20:33:33.438 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-16T20:33:33.481 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-16T20:33:33.700 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-16T20:33:33.714 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-16T20:33:33.766 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-16T20:33:33.826 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package librgw2. 2023-10-16T20:33:33.841 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../021-librgw2_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:33.884 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking librgw2 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:34.245 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-16T20:33:34.259 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-16T20:33:34.311 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-16T20:33:34.362 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-rgw. 2023-10-16T20:33:34.376 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../022-python3-rgw_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:34.420 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-rgw (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:34.706 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-16T20:33:34.720 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-16T20:33:34.748 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-16T20:33:34.762 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-16T20:33:34.764 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-16T20:33:34.814 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-16T20:33:35.084 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libradosstriper1. 2023-10-16T20:33:35.098 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../024-libradosstriper1_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:35.143 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libradosstriper1 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:35.192 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-16T20:33:35.207 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-16T20:33:35.250 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-16T20:33:35.562 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-common. 2023-10-16T20:33:35.577 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../025-ceph-common_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:35.620 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-common (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:35.653 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librados2. 2023-10-16T20:33:35.668 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../011-librados2_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:35.720 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librados2 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:36.190 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librbd1. 2023-10-16T20:33:36.204 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../012-librbd1_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:36.256 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librbd1 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:36.743 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-base. 2023-10-16T20:33:36.750 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libcephfs2. 2023-10-16T20:33:36.758 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../026-ceph-base_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:36.765 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../013-libcephfs2_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:36.811 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-base (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:36.817 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libcephfs2 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:37.111 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rados. 2023-10-16T20:33:37.126 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../014-python3-rados_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:37.178 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rados (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:37.539 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-16T20:33:37.554 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:33:37.572 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-16T20:33:37.587 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-16T20:33:37.597 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-ceph-argparse (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:37.630 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-16T20:33:37.833 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cephfs. 2023-10-16T20:33:37.848 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../016-python3-cephfs_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:37.885 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cephfs (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:37.933 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-cheroot. 2023-10-16T20:33:37.949 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-16T20:33:38.000 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-16T20:33:38.161 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-16T20:33:38.175 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../017-python3-ceph-common_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:33:38.228 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-ceph-common (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:38.562 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-16T20:33:38.577 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-16T20:33:38.589 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-prettytable. 2023-10-16T20:33:38.603 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-10-16T20:33:38.620 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-16T20:33:38.647 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-16T20:33:38.881 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-16T20:33:38.896 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-16T20:33:38.925 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rbd. 2023-10-16T20:33:38.931 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-16T20:33:38.939 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../019-python3-rbd_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:38.983 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rbd (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:39.242 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-16T20:33:39.257 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-16T20:33:39.305 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-16T20:33:39.309 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-16T20:33:39.319 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-16T20:33:39.363 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-16T20:33:39.612 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-tempora. 2023-10-16T20:33:39.627 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-10-16T20:33:39.670 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-16T20:33:39.715 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librgw2. 2023-10-16T20:33:39.730 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../021-librgw2_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:39.774 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librgw2 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:39.965 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-portend. 2023-10-16T20:33:39.980 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-10-16T20:33:40.014 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-16T20:33:40.243 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rgw. 2023-10-16T20:33:40.257 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../022-python3-rgw_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:40.301 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rgw (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:40.320 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-16T20:33:40.335 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-16T20:33:40.378 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-16T20:33:40.629 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-16T20:33:40.643 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-16T20:33:40.665 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-16T20:33:40.679 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-16T20:33:40.680 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-16T20:33:40.715 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-16T20:33:40.948 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libradosstriper1. 2023-10-16T20:33:40.963 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../024-libradosstriper1_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:41.006 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libradosstriper1 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:41.070 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-natsort. 2023-10-16T20:33:41.085 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-10-16T20:33:41.128 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-16T20:33:41.465 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-logutils. 2023-10-16T20:33:41.468 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-common. 2023-10-16T20:33:41.480 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-10-16T20:33:41.483 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../025-ceph-common_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:41.515 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-16T20:33:41.526 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-common (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:41.786 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-16T20:33:41.801 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-16T20:33:41.852 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-16T20:33:42.181 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-mako. 2023-10-16T20:33:42.196 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-16T20:33:42.239 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-16T20:33:42.594 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-16T20:33:42.609 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-16T20:33:42.624 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-base. 2023-10-16T20:33:42.638 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../026-ceph-base_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:42.643 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-16T20:33:42.683 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-base (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:42.939 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-16T20:33:42.954 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-16T20:33:42.997 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-16T20:33:43.301 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-webob. 2023-10-16T20:33:43.316 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-16T20:33:43.384 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-16T20:33:43.721 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-waitress. 2023-10-16T20:33:43.737 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-16T20:33:43.774 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-16T20:33:44.078 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-16T20:33:44.092 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-16T20:33:44.128 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-16T20:33:44.430 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cheroot. 2023-10-16T20:33:44.445 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-16T20:33:44.489 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-16T20:33:44.659 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-tempita. 2023-10-16T20:33:44.674 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-16T20:33:44.717 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-16T20:33:44.816 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-16T20:33:44.831 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-16T20:33:44.892 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-16T20:33:45.029 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-paste. 2023-10-16T20:33:45.045 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-16T20:33:45.088 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-16T20:33:45.194 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-16T20:33:45.208 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-16T20:33:45.253 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-16T20:33:45.391 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-16T20:33:45.407 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-16T20:33:45.450 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-16T20:33:45.505 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-16T20:33:45.520 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-16T20:33:45.555 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-16T20:33:45.754 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-16T20:33:45.769 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-16T20:33:45.803 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-16T20:33:45.881 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-tempora. 2023-10-16T20:33:45.896 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-10-16T20:33:45.942 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-16T20:33:46.108 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-webtest. 2023-10-16T20:33:46.123 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-10-16T20:33:46.166 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-16T20:33:46.244 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-portend. 2023-10-16T20:33:46.259 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-10-16T20:33:46.303 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-16T20:33:46.478 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pecan. 2023-10-16T20:33:46.493 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-16T20:33:46.561 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-16T20:33:46.572 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-16T20:33:46.586 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-16T20:33:46.622 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-16T20:33:46.901 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-certifi. 2023-10-16T20:33:46.916 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-10-16T20:33:46.924 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-16T20:33:46.939 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-16T20:33:46.951 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-16T20:33:46.983 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-16T20:33:47.246 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-idna. 2023-10-16T20:33:47.260 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-10-16T20:33:47.304 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-16T20:33:47.328 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-natsort. 2023-10-16T20:33:47.342 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-10-16T20:33:47.386 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-16T20:33:47.633 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-urllib3. 2023-10-16T20:33:47.648 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-logutils. 2023-10-16T20:33:47.648 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-16T20:33:47.661 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-10-16T20:33:47.691 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-16T20:33:47.697 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-16T20:33:48.013 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-requests. 2023-10-16T20:33:48.017 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-16T20:33:48.028 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-16T20:33:48.031 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-16T20:33:48.071 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-16T20:33:48.075 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-16T20:33:48.361 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-mako. 2023-10-16T20:33:48.366 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-16T20:33:48.375 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-16T20:33:48.381 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-16T20:33:48.419 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-16T20:33:48.424 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-16T20:33:48.728 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-16T20:33:48.743 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:33:48.750 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-16T20:33:48.762 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-16T20:33:48.795 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:48.806 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-16T20:33:49.109 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-16T20:33:49.124 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-16T20:33:49.126 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-16T20:33:49.139 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:49.159 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-16T20:33:49.184 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:49.445 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-webob. 2023-10-16T20:33:49.460 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-16T20:33:49.471 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-openssl. 2023-10-16T20:33:49.485 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-10-16T20:33:49.502 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-16T20:33:49.530 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-16T20:33:49.873 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-waitress. 2023-10-16T20:33:49.887 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-16T20:33:49.934 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-16T20:33:49.958 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-mgr. 2023-10-16T20:33:49.973 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../058-ceph-mgr_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:50.009 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-mgr (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:50.259 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-tempita. 2023-10-16T20:33:50.272 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-16T20:33:50.317 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-16T20:33:50.371 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-mon. 2023-10-16T20:33:50.386 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../059-ceph-mon_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:50.429 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-mon (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:50.620 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-paste. 2023-10-16T20:33:50.633 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-16T20:33:50.678 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-16T20:33:51.031 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-16T20:33:51.045 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-16T20:33:51.089 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-16T20:33:51.334 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-osd. 2023-10-16T20:33:51.349 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../060-ceph-osd_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:51.392 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-16T20:33:51.392 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-osd (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:51.405 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-16T20:33:51.450 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-16T20:33:51.736 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-webtest. 2023-10-16T20:33:51.750 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-10-16T20:33:51.786 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-16T20:33:52.089 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pecan. 2023-10-16T20:33:52.102 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-16T20:33:52.139 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-16T20:33:52.655 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph. 2023-10-16T20:33:52.670 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../061-ceph_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:52.713 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:52.760 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-certifi. 2023-10-16T20:33:52.774 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-10-16T20:33:52.818 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-16T20:33:53.058 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-fuse. 2023-10-16T20:33:53.073 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../062-ceph-fuse_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:53.118 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-fuse (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:53.180 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-idna. 2023-10-16T20:33:53.193 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-10-16T20:33:53.238 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-16T20:33:53.429 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-mds. 2023-10-16T20:33:53.444 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../063-ceph-mds_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:53.488 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-mds (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:53.557 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-urllib3. 2023-10-16T20:33:53.572 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-16T20:33:53.616 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-16T20:33:53.944 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-requests. 2023-10-16T20:33:53.958 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-16T20:33:54.002 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-16T20:33:54.042 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package cephadm. 2023-10-16T20:33:54.056 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../064-cephadm_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:54.100 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking cephadm (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:54.330 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-16T20:33:54.345 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-16T20:33:54.380 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-16T20:33:54.387 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-16T20:33:54.402 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-16T20:33:54.437 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-16T20:33:54.674 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-16T20:33:54.689 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:33:54.724 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-16T20:33:54.732 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:54.738 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-16T20:33:54.782 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-16T20:33:55.154 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-16T20:33:55.168 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:55.221 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:55.540 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-openssl. 2023-10-16T20:33:55.554 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-10-16T20:33:55.598 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-16T20:33:55.781 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-babel. 2023-10-16T20:33:55.794 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-16T20:33:55.830 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-16T20:33:55.901 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr. 2023-10-16T20:33:55.915 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../058-ceph-mgr_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:55.960 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:56.151 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-jinja2. 2023-10-16T20:33:56.165 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-10-16T20:33:56.209 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-16T20:33:56.329 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mon. 2023-10-16T20:33:56.343 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../059-ceph-mon_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:56.396 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mon (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:56.547 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-16T20:33:56.561 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:33:56.605 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:56.977 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-16T20:33:56.990 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-16T20:33:57.035 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-16T20:33:57.233 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-osd. 2023-10-16T20:33:57.247 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../060-ceph-osd_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:57.290 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-osd (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:57.330 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-16T20:33:57.344 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../071-python3-repoze.lru_0.7-2_all.deb ... 2023-10-16T20:33:57.389 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-16T20:33:57.692 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-routes. 2023-10-16T20:33:57.706 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../072-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-16T20:33:57.751 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-16T20:33:58.089 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-16T20:33:58.102 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../073-ceph-mgr-dashboard_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:33:58.147 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:58.543 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-16T20:33:58.557 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-16T20:33:58.560 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph. 2023-10-16T20:33:58.575 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../061-ceph_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:58.601 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-16T20:33:58.619 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:59.021 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-fuse. 2023-10-16T20:33:59.036 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../062-ceph-fuse_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:59.088 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-fuse (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:59.114 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-joblib. 2023-10-16T20:33:59.128 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../075-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-16T20:33:59.173 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-16T20:33:59.550 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mds. 2023-10-16T20:33:59.552 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-16T20:33:59.564 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../063-ceph-mds_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:33:59.566 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../076-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-16T20:33:59.611 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-16T20:33:59.616 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mds (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:33:59.890 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-sklearn. 2023-10-16T20:33:59.904 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../077-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-16T20:33:59.940 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-16T20:34:00.061 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package cephadm. 2023-10-16T20:34:00.075 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../064-cephadm_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:34:00.119 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking cephadm (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:00.506 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-16T20:34:00.521 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-16T20:34:00.572 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-16T20:34:00.643 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-16T20:34:00.657 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../078-ceph-mgr-diskprediction-local_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:34:00.702 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:00.908 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-16T20:34:00.922 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-16T20:34:00.966 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-16T20:34:01.731 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-cachetools. 2023-10-16T20:34:01.746 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../079-python3-cachetools_5.0.0-1_all.deb ... 2023-10-16T20:34:01.789 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-16T20:34:01.938 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-babel. 2023-10-16T20:34:01.952 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-16T20:34:01.996 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-16T20:34:02.076 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-16T20:34:02.091 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../080-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-16T20:34:02.134 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-16T20:34:02.316 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jinja2. 2023-10-16T20:34:02.329 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-10-16T20:34:02.399 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-16T20:34:02.497 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-16T20:34:02.512 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../081-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-16T20:34:02.555 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-16T20:34:02.727 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-16T20:34:02.740 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:34:02.777 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:02.850 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-rsa. 2023-10-16T20:34:02.865 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../082-python3-rsa_4.8-1_all.deb ... 2023-10-16T20:34:02.908 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-16T20:34:03.122 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-16T20:34:03.136 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-16T20:34:03.172 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-16T20:34:03.340 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-google-auth. 2023-10-16T20:34:03.356 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../083-python3-google-auth_1.5.1-3_all.deb ... 2023-10-16T20:34:03.398 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-16T20:34:03.483 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-16T20:34:03.497 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../071-python3-repoze.lru_0.7-2_all.deb ... 2023-10-16T20:34:03.542 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-16T20:34:03.811 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-16T20:34:03.820 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-routes. 2023-10-16T20:34:03.826 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../084-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-16T20:34:03.833 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../072-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-16T20:34:03.869 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-16T20:34:03.878 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-16T20:34:04.165 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-websocket. 2023-10-16T20:34:04.180 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../085-python3-websocket_1.2.3-1_all.deb ... 2023-10-16T20:34:04.181 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-16T20:34:04.194 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../073-ceph-mgr-dashboard_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:34:04.223 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-16T20:34:04.239 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:04.561 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-16T20:34:04.576 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../086-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-16T20:34:04.619 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-16T20:34:04.784 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-16T20:34:04.798 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-16T20:34:04.842 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-16T20:34:05.338 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-joblib. 2023-10-16T20:34:05.351 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../075-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-16T20:34:05.368 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-16T20:34:05.382 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../087-ceph-mgr-k8sevents_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:34:05.396 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-16T20:34:05.427 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:05.738 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-16T20:34:05.752 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../088-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-16T20:34:05.766 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-16T20:34:05.779 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../076-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-16T20:34:05.796 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-16T20:34:05.824 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-16T20:34:06.094 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-sklearn. 2023-10-16T20:34:06.107 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../077-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-16T20:34:06.117 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-16T20:34:06.131 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../089-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-16T20:34:06.152 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-16T20:34:06.175 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-16T20:34:06.470 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package jq. 2023-10-16T20:34:06.485 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../090-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-16T20:34:06.529 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-16T20:34:06.757 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package socat. 2023-10-16T20:34:06.772 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../091-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-16T20:34:06.807 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-16T20:34:06.894 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-16T20:34:06.908 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../078-ceph-mgr-diskprediction-local_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:34:06.952 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:07.112 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package xmlstarlet. 2023-10-16T20:34:07.126 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../092-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-16T20:34:07.170 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-16T20:34:07.550 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-test. 2023-10-16T20:34:07.564 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../093-ceph-test_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:34:07.609 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-test (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:07.922 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cachetools. 2023-10-16T20:34:07.937 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../079-python3-cachetools_5.0.0-1_all.deb ... 2023-10-16T20:34:07.971 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-16T20:34:08.308 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-16T20:34:08.323 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../080-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-16T20:34:08.366 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-16T20:34:08.719 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-16T20:34:08.735 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../081-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-16T20:34:08.769 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-16T20:34:09.063 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rsa. 2023-10-16T20:34:09.078 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../082-python3-rsa_4.8-1_all.deb ... 2023-10-16T20:34:09.113 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-16T20:34:09.477 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-google-auth. 2023-10-16T20:34:09.492 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../083-python3-google-auth_1.5.1-3_all.deb ... 2023-10-16T20:34:09.535 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-16T20:34:09.856 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ceph-volume. 2023-10-16T20:34:09.871 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../094-ceph-volume_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:34:09.872 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-16T20:34:09.886 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../084-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-16T20:34:09.914 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ceph-volume (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:09.929 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-16T20:34:10.241 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-websocket. 2023-10-16T20:34:10.256 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../085-python3-websocket_1.2.3-1_all.deb ... 2023-10-16T20:34:10.286 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-16T20:34:10.291 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-16T20:34:10.301 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../095-cephfs-mirror_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:34:10.344 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking cephfs-mirror (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:10.586 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-16T20:34:10.601 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../086-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-16T20:34:10.635 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-16T20:34:10.682 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-colorama. 2023-10-16T20:34:10.698 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../096-python3-colorama_0.4.4-1_all.deb ... 2023-10-16T20:34:10.731 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-10-16T20:34:10.985 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package cephfs-shell. 2023-10-16T20:34:11.001 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../097-cephfs-shell_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:34:11.043 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking cephfs-shell (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:11.297 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package cephfs-top. 2023-10-16T20:34:11.312 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../098-cephfs-top_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:34:11.355 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking cephfs-top (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:11.357 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-16T20:34:11.372 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../087-ceph-mgr-k8sevents_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:34:11.415 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:11.634 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-16T20:34:11.649 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../099-libcephfs-dev_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:34:11.684 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libcephfs-dev (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:11.735 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-16T20:34:11.749 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../088-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-16T20:34:11.810 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-16T20:34:12.038 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package nvme-cli. 2023-10-16T20:34:12.054 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../100-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-16T20:34:12.097 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-16T20:34:12.113 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-16T20:34:12.127 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../089-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-16T20:34:12.163 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-16T20:34:12.415 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package jq. 2023-10-16T20:34:12.428 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-16T20:34:12.429 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../090-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-16T20:34:12.443 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../101-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-16T20:34:12.473 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-16T20:34:12.486 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-16T20:34:12.726 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package socat. 2023-10-16T20:34:12.740 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../091-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-16T20:34:12.784 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-16T20:34:12.906 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-16T20:34:12.921 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../102-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-16T20:34:12.965 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-16T20:34:13.121 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package xmlstarlet. 2023-10-16T20:34:13.136 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../092-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-16T20:34:13.179 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-16T20:34:13.268 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pastescript. 2023-10-16T20:34:13.285 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-4_all.deb ... 2023-10-16T20:34:13.326 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-16T20:34:13.534 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-test. 2023-10-16T20:34:13.547 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../093-ceph-test_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:34:13.591 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-test (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:13.692 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pluggy. 2023-10-16T20:34:13.708 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../104-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-16T20:34:13.756 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-16T20:34:14.060 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-psutil. 2023-10-16T20:34:14.075 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../105-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-16T20:34:14.118 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-16T20:34:14.463 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-py. 2023-10-16T20:34:14.479 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../106-python3-py_1.10.0-1_all.deb ... 2023-10-16T20:34:14.522 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-16T20:34:14.869 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pygments. 2023-10-16T20:34:14.884 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../107-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-16T20:34:14.927 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-16T20:34:15.398 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-16T20:34:15.412 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-16T20:34:15.455 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-16T20:34:15.495 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-volume. 2023-10-16T20:34:15.511 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../094-ceph-volume_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:34:15.562 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-volume (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:15.766 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-toml. 2023-10-16T20:34:15.782 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../109-python3-toml_0.10.2-1_all.deb ... 2023-10-16T20:34:15.816 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-16T20:34:16.008 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-16T20:34:16.023 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../095-cephfs-mirror_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:34:16.074 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking cephfs-mirror (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:16.094 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-pytest. 2023-10-16T20:34:16.110 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../110-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-16T20:34:16.152 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-16T20:34:16.420 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-colorama. 2023-10-16T20:34:16.435 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../096-python3-colorama_0.4.4-1_all.deb ... 2023-10-16T20:34:16.478 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-10-16T20:34:16.488 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package python3-simplejson. 2023-10-16T20:34:16.504 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-16T20:34:16.547 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-16T20:34:16.797 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package cephfs-shell. 2023-10-16T20:34:16.812 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../097-cephfs-shell_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:34:16.841 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-16T20:34:16.857 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-16T20:34:16.896 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking cephfs-shell (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:16.899 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-16T20:34:17.191 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package cephfs-top. 2023-10-16T20:34:17.207 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../098-cephfs-top_18.0.0-6730-g4ba8a190-1jammy_all.deb ... 2023-10-16T20:34:17.257 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking cephfs-top (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:17.543 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-16T20:34:17.559 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../099-libcephfs-dev_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:34:17.610 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libcephfs-dev (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:17.636 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package radosgw. 2023-10-16T20:34:17.651 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../113-radosgw_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:34:17.694 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking radosgw (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:18.065 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package nvme-cli. 2023-10-16T20:34:18.081 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../100-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-16T20:34:18.131 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-16T20:34:18.264 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package rbd-fuse. 2023-10-16T20:34:18.279 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../114-rbd-fuse_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:34:18.322 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking rbd-fuse (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:18.624 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-16T20:34:18.794 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-16T20:34:18.810 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../101-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-16T20:34:18.861 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-16T20:34:18.927 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-16T20:34:19.078 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-16T20:34:19.272 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-16T20:34:19.287 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../102-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-16T20:34:19.322 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-16T20:34:19.392 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-16T20:34:19.625 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pastescript. 2023-10-16T20:34:19.640 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-4_all.deb ... 2023-10-16T20:34:19.683 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-16T20:34:19.790 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-16T20:34:20.023 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pluggy. 2023-10-16T20:34:20.035 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../104-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-16T20:34:20.078 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-16T20:34:20.397 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-psutil. 2023-10-16T20:34:20.412 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../105-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-16T20:34:20.419 INFO:teuthology.orchestra.run.smithi191.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-16T20:34:20.456 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-16T20:34:20.477 INFO:teuthology.orchestra.run.smithi191.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2023-10-16T20:34:20.521 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up cephadm (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:20.817 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-py. 2023-10-16T20:34:20.832 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../106-python3-py_1.10.0-1_all.deb ... 2023-10-16T20:34:20.875 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-16T20:34:21.075 INFO:teuthology.orchestra.run.smithi191.stdout:Adding system user cephadm....done 2023-10-16T20:34:21.220 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pygments. 2023-10-16T20:34:21.235 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../107-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-16T20:34:21.236 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-16T20:34:21.278 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-16T20:34:21.512 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-16T20:34:21.741 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-16T20:34:21.755 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-16T20:34:21.771 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-16T20:34:21.799 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-16T20:34:21.922 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-16T20:34:22.152 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-toml. 2023-10-16T20:34:22.166 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../109-python3-toml_0.10.2-1_all.deb ... 2023-10-16T20:34:22.181 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-16T20:34:22.210 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-16T20:34:22.441 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-16T20:34:22.487 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pytest. 2023-10-16T20:34:22.502 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../110-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-16T20:34:22.545 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-16T20:34:22.567 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-16T20:34:22.824 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-simplejson. 2023-10-16T20:34:22.838 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-16T20:34:22.874 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-16T20:34:22.910 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-16T20:34:23.160 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-16T20:34:23.173 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-16T20:34:23.218 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-16T20:34:23.304 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-16T20:34:23.572 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-10-16T20:34:23.839 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-16T20:34:24.157 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-16T20:34:24.173 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package radosgw. 2023-10-16T20:34:24.186 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../113-radosgw_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:34:24.230 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking radosgw (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:24.417 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-16T20:34:24.826 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-16T20:34:24.841 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package rbd-fuse. 2023-10-16T20:34:24.856 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../114-rbd-fuse_18.0.0-6730-g4ba8a190-1jammy_amd64.deb ... 2023-10-16T20:34:24.899 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking rbd-fuse (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:25.144 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-ceph-argparse (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:25.218 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-16T20:34:25.420 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-16T20:34:25.504 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-16T20:34:25.546 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-10-16T20:34:25.639 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-16T20:34:25.813 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-16T20:34:25.894 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-16T20:34:26.073 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-16T20:34:26.182 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-16T20:34:26.308 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-16T20:34:26.558 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-16T20:34:26.684 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-16T20:34:26.915 INFO:teuthology.orchestra.run.smithi088.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-16T20:34:26.961 INFO:teuthology.orchestra.run.smithi088.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2023-10-16T20:34:26.993 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-16T20:34:26.998 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up cephadm (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:27.278 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-16T20:34:27.560 INFO:teuthology.orchestra.run.smithi088.stdout:Adding system user cephadm....done 2023-10-16T20:34:27.604 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-16T20:34:27.687 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-16T20:34:27.964 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-16T20:34:28.207 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-16T20:34:28.298 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-16T20:34:28.333 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-16T20:34:28.574 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-16T20:34:28.592 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-16T20:34:28.700 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-16T20:34:28.825 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-16T20:34:28.860 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-16T20:34:28.986 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-16T20:34:29.143 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-16T20:34:29.304 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-16T20:34:29.544 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-16T20:34:29.665 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-16T20:34:29.821 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-16T20:34:29.932 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-10-16T20:34:30.197 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-16T20:34:30.267 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-16T20:34:30.490 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-10-16T20:34:30.526 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-16T20:34:30.786 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-16T20:34:30.816 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-16T20:34:31.053 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-16T20:34:31.193 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-16T20:34:31.338 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-ceph-argparse (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:31.469 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-16T20:34:31.589 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-16T20:34:31.619 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-ceph-common (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:31.715 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-10-16T20:34:31.921 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-idna (3.3-1) ... 2023-10-16T20:34:31.974 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-16T20:34:32.217 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-16T20:34:32.263 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-16T20:34:32.368 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-16T20:34:32.727 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-16T20:34:32.853 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-16T20:34:33.138 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-16T20:34:33.405 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-16T20:34:33.519 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-16T20:34:33.653 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-16T20:34:33.707 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-16T20:34:33.921 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-16T20:34:34.030 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-16T20:34:34.181 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-16T20:34:34.400 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-16T20:34:34.500 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-16T20:34:34.659 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-16T20:34:34.759 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-16T20:34:34.785 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-16T20:34:34.911 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-16T20:34:35.018 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-16T20:34:35.194 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-16T20:34:35.229 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-16T20:34:35.320 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-16T20:34:35.630 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-16T20:34:35.655 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-16T20:34:35.898 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-16T20:34:35.914 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-16T20:34:36.213 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-16T20:34:36.300 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-16T20:34:36.483 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-16T20:34:36.609 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-10-16T20:34:36.751 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-16T20:34:36.927 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-16T20:34:37.028 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-16T20:34:37.170 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-16T20:34:37.286 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-16T20:34:37.496 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-16T20:34:37.571 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-16T20:34:37.613 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-16T20:34:37.697 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-ceph-common (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:37.898 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-16T20:34:38.006 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-idna (3.3-1) ... 2023-10-16T20:34:38.216 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-16T20:34:38.341 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-16T20:34:38.508 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-16T20:34:38.912 INFO:teuthology.orchestra.run.smithi191.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-16T20:34:39.169 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-16T20:34:39.329 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-16T20:34:39.580 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-16T20:34:39.689 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-16T20:34:39.730 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-16T20:34:39.956 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-16T20:34:39.998 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-16T20:34:40.066 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-16T20:34:40.191 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-16T20:34:40.493 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-16T20:34:40.641 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-16T20:34:40.766 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-16T20:34:40.777 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-16T20:34:41.037 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-16T20:34:41.101 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-16T20:34:41.163 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-16T20:34:41.289 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-16T20:34:41.435 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-16T20:34:41.607 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-16T20:34:41.720 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-16T20:34:41.837 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-16T20:34:41.866 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-16T20:34:42.113 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-16T20:34:42.117 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-16T20:34:42.385 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-16T20:34:42.389 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-16T20:34:42.678 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-16T20:34:42.699 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-16T20:34:42.954 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-16T20:34:43.080 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-16T20:34:43.084 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-16T20:34:43.364 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-16T20:34:43.385 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-16T20:34:43.509 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-16T20:34:43.762 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-16T20:34:43.808 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-16T20:34:43.888 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up librados2 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:44.014 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up librgw2 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:44.100 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-16T20:34:44.251 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:44.360 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-16T20:34:44.401 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libcephfs2 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:44.561 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libradosstriper1 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:44.687 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-16T20:34:44.756 INFO:teuthology.orchestra.run.smithi088.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-16T20:34:44.980 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up librbd1 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:45.022 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-16T20:34:45.098 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-mgr-modules-core (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:45.147 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-16T20:34:45.232 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-fuse (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:45.441 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-16T20:34:45.441 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-16T20:34:45.563 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-16T20:34:45.833 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-16T20:34:46.075 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libcephfs-dev (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:46.092 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-16T20:34:46.218 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-16T20:34:46.226 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-rados (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:46.368 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up cephfs-top (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:46.494 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-16T20:34:46.544 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-16T20:34:46.887 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-16T20:34:47.155 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-16T20:34:47.281 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-16T20:34:47.557 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-16T20:34:47.807 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-rbd (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:47.833 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-16T20:34:47.933 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up rbd-fuse (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:48.059 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-rgw (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:48.101 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-16T20:34:48.185 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up python3-cephfs (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:48.311 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up cephfs-shell (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:48.437 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-common (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:48.485 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-16T20:34:48.787 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-16T20:34:48.991 INFO:teuthology.orchestra.run.smithi191.stdout:Adding group ceph....done 2023-10-16T20:34:49.123 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-16T20:34:49.223 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librados2 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:49.324 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librgw2 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:49.375 INFO:teuthology.orchestra.run.smithi191.stdout:Adding system user ceph....done 2023-10-16T20:34:49.535 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:49.552 INFO:teuthology.orchestra.run.smithi191.stdout:Setting system user ceph properties....done 2023-10-16T20:34:49.574 INFO:teuthology.orchestra.run.smithi191.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-16T20:34:49.695 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libcephfs2 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:49.713 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-16T20:34:49.837 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libradosstriper1 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:49.988 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-16T20:34:50.080 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-16T20:34:50.282 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librbd1 (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:50.408 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-modules-core (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:50.550 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-fuse (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:50.730 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up cephfs-mirror (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:50.775 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-16T20:34:50.775 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-16T20:34:50.928 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-16T20:34:50.929 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-16T20:34:51.368 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libcephfs-dev (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:51.478 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rados (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:51.523 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-test (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:51.587 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up cephfs-top (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:51.649 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up radosgw (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:51.738 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-16T20:34:52.173 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-16T20:34:52.173 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-16T20:34:52.751 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-base (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:53.017 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rbd (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:53.109 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-16T20:34:53.176 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up rbd-fuse (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:53.328 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rgw (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:53.479 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cephfs (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:53.630 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up cephfs-shell (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:53.769 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-mds (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:53.822 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-common (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:53.999 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-16T20:34:53.999 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-16T20:34:54.401 INFO:teuthology.orchestra.run.smithi088.stdout:Adding group ceph....done 2023-10-16T20:34:54.587 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-mgr (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:54.800 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-16T20:34:54.800 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-16T20:34:55.102 INFO:teuthology.orchestra.run.smithi088.stdout:Adding system user ceph....done 2023-10-16T20:34:55.264 INFO:teuthology.orchestra.run.smithi088.stdout:Setting system user ceph properties....done 2023-10-16T20:34:55.285 INFO:teuthology.orchestra.run.smithi088.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-16T20:34:55.428 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-16T20:34:55.497 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-osd (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:55.819 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-16T20:34:55.868 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-16T20:34:55.868 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-16T20:34:56.443 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:56.515 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up cephfs-mirror (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:56.569 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:56.711 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-mon (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:56.731 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-16T20:34:56.732 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-16T20:34:56.946 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-16T20:34:56.946 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-16T20:34:57.325 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-test (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:57.476 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up radosgw (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:57.529 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-mgr-cephadm (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:57.664 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:57.790 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-mgr-dashboard (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:57.966 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ceph-volume (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:58.028 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-16T20:34:58.028 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-16T20:34:58.343 INFO:teuthology.orchestra.run.smithi191.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-16T20:34:58.620 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-base (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:59.014 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-16T20:34:59.511 INFO:teuthology.orchestra.run.smithi191.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-16T20:34:59.629 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mds (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:34:59.851 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-16T20:34:59.852 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-16T20:34:59.970 INFO:teuthology.orchestra.run.smithi191.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-16T20:34:59.975 DEBUG:teuthology.orchestra.run.smithi191:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-10-16T20:35:00.050 INFO:teuthology.orchestra.run.smithi191.stdout:Reading package lists... 2023-10-16T20:35:00.214 INFO:teuthology.orchestra.run.smithi191.stdout:Building dependency tree... 2023-10-16T20:35:00.214 INFO:teuthology.orchestra.run.smithi191.stdout:Reading state information... 2023-10-16T20:35:00.360 INFO:teuthology.orchestra.run.smithi191.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-10-16T20:35:00.360 INFO:teuthology.orchestra.run.smithi191.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-10-16T20:35:00.361 INFO:teuthology.orchestra.run.smithi191.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-10-16T20:35:00.361 INFO:teuthology.orchestra.run.smithi191.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-10-16T20:35:00.361 INFO:teuthology.orchestra.run.smithi191.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-10-16T20:35:00.361 INFO:teuthology.orchestra.run.smithi191.stdout:The following packages were automatically installed and are no longer required: 2023-10-16T20:35:00.361 INFO:teuthology.orchestra.run.smithi191.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-16T20:35:00.362 INFO:teuthology.orchestra.run.smithi191.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-16T20:35:00.362 INFO:teuthology.orchestra.run.smithi191.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-16T20:35:00.362 INFO:teuthology.orchestra.run.smithi191.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-16T20:35:00.363 INFO:teuthology.orchestra.run.smithi191.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-16T20:35:00.363 INFO:teuthology.orchestra.run.smithi191.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-16T20:35:00.364 INFO:teuthology.orchestra.run.smithi191.stdout:The following additional packages will be installed: 2023-10-16T20:35:00.364 INFO:teuthology.orchestra.run.smithi191.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-10-16T20:35:00.364 INFO:teuthology.orchestra.run.smithi191.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-10-16T20:35:00.364 INFO:teuthology.orchestra.run.smithi191.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-10-16T20:35:00.364 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-10-16T20:35:00.365 INFO:teuthology.orchestra.run.smithi191.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-16T20:35:00.365 INFO:teuthology.orchestra.run.smithi191.stdout:Suggested packages: 2023-10-16T20:35:00.365 INFO:teuthology.orchestra.run.smithi191.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-10-16T20:35:00.365 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-10-16T20:35:00.365 INFO:teuthology.orchestra.run.smithi191.stdout: libengine-pkcs11-openssl 2023-10-16T20:35:00.412 INFO:teuthology.orchestra.run.smithi191.stdout:The following NEW packages will be installed: 2023-10-16T20:35:00.412 INFO:teuthology.orchestra.run.smithi191.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-10-16T20:35:00.412 INFO:teuthology.orchestra.run.smithi191.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-10-16T20:35:00.413 INFO:teuthology.orchestra.run.smithi191.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-10-16T20:35:00.413 INFO:teuthology.orchestra.run.smithi191.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-10-16T20:35:00.413 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-10-16T20:35:00.413 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-10-16T20:35:00.414 INFO:teuthology.orchestra.run.smithi191.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-16T20:35:00.414 INFO:teuthology.orchestra.run.smithi191.stdout:The following packages will be upgraded: 2023-10-16T20:35:00.415 INFO:teuthology.orchestra.run.smithi191.stdout: libssl3 2023-10-16T20:35:00.477 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:35:00.598 INFO:teuthology.orchestra.run.smithi191.stdout:1 upgraded, 39 newly installed, 0 to remove and 258 not upgraded. 2023-10-16T20:35:00.598 INFO:teuthology.orchestra.run.smithi191.stdout:Need to get 54.6 MB of archives. 2023-10-16T20:35:00.598 INFO:teuthology.orchestra.run.smithi191.stdout:After this operation, 202 MB of additional disk space will be used. 2023-10-16T20:35:00.598 INFO:teuthology.orchestra.run.smithi191.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-10-16T20:35:00.705 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-16T20:35:00.706 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-16T20:35:01.094 INFO:teuthology.orchestra.run.smithi191.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.10 [1,901 kB] 2023-10-16T20:35:01.303 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-osd (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:35:01.352 INFO:teuthology.orchestra.run.smithi191.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-10-16T20:35:01.386 INFO:teuthology.orchestra.run.smithi191.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-10-16T20:35:01.388 INFO:teuthology.orchestra.run.smithi191.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-10-16T20:35:01.393 INFO:teuthology.orchestra.run.smithi191.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-10-16T20:35:01.396 INFO:teuthology.orchestra.run.smithi191.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-10-16T20:35:01.397 INFO:teuthology.orchestra.run.smithi191.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-10-16T20:35:01.401 INFO:teuthology.orchestra.run.smithi191.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-10-16T20:35:01.403 INFO:teuthology.orchestra.run.smithi191.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-10-16T20:35:01.568 INFO:teuthology.orchestra.run.smithi191.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-10-16T20:35:01.568 INFO:teuthology.orchestra.run.smithi191.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-10-16T20:35:01.666 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-16T20:35:01.667 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-16T20:35:01.829 INFO:teuthology.orchestra.run.smithi191.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-10-16T20:35:02.280 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:35:02.406 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:35:02.548 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mon (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:35:02.756 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-16T20:35:02.757 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-16T20:35:02.845 INFO:teuthology.orchestra.run.smithi191.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-10-16T20:35:02.847 INFO:teuthology.orchestra.run.smithi191.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-10-16T20:35:02.847 INFO:teuthology.orchestra.run.smithi191.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-10-16T20:35:02.872 INFO:teuthology.orchestra.run.smithi191.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-10-16T20:35:02.873 INFO:teuthology.orchestra.run.smithi191.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-10-16T20:35:02.877 INFO:teuthology.orchestra.run.smithi191.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-10-16T20:35:02.878 INFO:teuthology.orchestra.run.smithi191.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-10-16T20:35:02.886 INFO:teuthology.orchestra.run.smithi191.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-10-16T20:35:03.050 INFO:teuthology.orchestra.run.smithi191.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.10 [2,372 kB] 2023-10-16T20:35:03.366 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-cephadm (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:35:03.534 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:35:03.685 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-dashboard (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:35:03.790 INFO:teuthology.orchestra.run.smithi191.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-10-16T20:35:03.790 INFO:teuthology.orchestra.run.smithi191.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-10-16T20:35:03.816 INFO:teuthology.orchestra.run.smithi191.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-10-16T20:35:03.844 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-volume (18.0.0-6730-g4ba8a190-1jammy) ... 2023-10-16T20:35:03.894 INFO:teuthology.orchestra.run.smithi191.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-10-16T20:35:03.902 INFO:teuthology.orchestra.run.smithi191.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-10-16T20:35:03.903 INFO:teuthology.orchestra.run.smithi191.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-10-16T20:35:03.904 INFO:teuthology.orchestra.run.smithi191.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-10-16T20:35:03.905 INFO:teuthology.orchestra.run.smithi191.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-10-16T20:35:03.933 INFO:teuthology.orchestra.run.smithi191.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-10-16T20:35:03.936 INFO:teuthology.orchestra.run.smithi191.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-10-16T20:35:04.230 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-16T20:35:04.563 INFO:teuthology.orchestra.run.smithi191.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-10-16T20:35:04.646 INFO:teuthology.orchestra.run.smithi191.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-10-16T20:35:04.646 INFO:teuthology.orchestra.run.smithi191.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-10-16T20:35:04.646 INFO:teuthology.orchestra.run.smithi191.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-10-16T20:35:04.738 INFO:teuthology.orchestra.run.smithi191.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-10-16T20:35:04.741 INFO:teuthology.orchestra.run.smithi191.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-10-16T20:35:05.205 INFO:teuthology.orchestra.run.smithi191.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-10-16T20:35:05.207 INFO:teuthology.orchestra.run.smithi191.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-10-16T20:35:05.489 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-16T20:35:05.601 INFO:teuthology.orchestra.run.smithi191.stdout:Preconfiguring packages ... 2023-10-16T20:35:05.735 INFO:teuthology.orchestra.run.smithi191.stdout:Fetched 54.6 MB in 5s (11.4 MB/s) 2023-10-16T20:35:05.835 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package flex. 2023-10-16T20:35:05.908 INFO:teuthology.orchestra.run.smithi191.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 139851 files and directories currently installed.) 2023-10-16T20:35:05.912 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-10-16T20:35:05.951 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking flex (2.6.4-8build2) ... 2023-10-16T20:35:05.958 INFO:teuthology.orchestra.run.smithi088.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-16T20:35:05.962 DEBUG:teuthology.orchestra.run.smithi088:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-10-16T20:35:06.037 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2023-10-16T20:35:06.201 INFO:teuthology.orchestra.run.smithi088.stdout:Building dependency tree... 2023-10-16T20:35:06.202 INFO:teuthology.orchestra.run.smithi088.stdout:Reading state information... 2023-10-16T20:35:06.347 INFO:teuthology.orchestra.run.smithi088.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-10-16T20:35:06.347 INFO:teuthology.orchestra.run.smithi088.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-10-16T20:35:06.347 INFO:teuthology.orchestra.run.smithi088.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-10-16T20:35:06.348 INFO:teuthology.orchestra.run.smithi088.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-10-16T20:35:06.348 INFO:teuthology.orchestra.run.smithi088.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-10-16T20:35:06.348 INFO:teuthology.orchestra.run.smithi088.stdout:The following packages were automatically installed and are no longer required: 2023-10-16T20:35:06.348 INFO:teuthology.orchestra.run.smithi088.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-16T20:35:06.348 INFO:teuthology.orchestra.run.smithi088.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-16T20:35:06.349 INFO:teuthology.orchestra.run.smithi088.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-16T20:35:06.349 INFO:teuthology.orchestra.run.smithi088.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-16T20:35:06.350 INFO:teuthology.orchestra.run.smithi088.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-16T20:35:06.350 INFO:teuthology.orchestra.run.smithi088.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-16T20:35:06.351 INFO:teuthology.orchestra.run.smithi088.stdout:The following additional packages will be installed: 2023-10-16T20:35:06.351 INFO:teuthology.orchestra.run.smithi088.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-10-16T20:35:06.351 INFO:teuthology.orchestra.run.smithi088.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-10-16T20:35:06.351 INFO:teuthology.orchestra.run.smithi088.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-10-16T20:35:06.351 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-10-16T20:35:06.352 INFO:teuthology.orchestra.run.smithi088.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-16T20:35:06.352 INFO:teuthology.orchestra.run.smithi088.stdout:Suggested packages: 2023-10-16T20:35:06.352 INFO:teuthology.orchestra.run.smithi088.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-10-16T20:35:06.352 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-10-16T20:35:06.352 INFO:teuthology.orchestra.run.smithi088.stdout: libengine-pkcs11-openssl 2023-10-16T20:35:06.397 INFO:teuthology.orchestra.run.smithi088.stdout:The following NEW packages will be installed: 2023-10-16T20:35:06.398 INFO:teuthology.orchestra.run.smithi088.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-10-16T20:35:06.398 INFO:teuthology.orchestra.run.smithi088.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-10-16T20:35:06.399 INFO:teuthology.orchestra.run.smithi088.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-10-16T20:35:06.399 INFO:teuthology.orchestra.run.smithi088.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-10-16T20:35:06.399 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-10-16T20:35:06.399 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-10-16T20:35:06.400 INFO:teuthology.orchestra.run.smithi088.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-16T20:35:06.400 INFO:teuthology.orchestra.run.smithi088.stdout:The following packages will be upgraded: 2023-10-16T20:35:06.400 INFO:teuthology.orchestra.run.smithi088.stdout: libssl3 2023-10-16T20:35:06.478 INFO:teuthology.orchestra.run.smithi088.stdout:1 upgraded, 39 newly installed, 0 to remove and 258 not upgraded. 2023-10-16T20:35:06.478 INFO:teuthology.orchestra.run.smithi088.stdout:Need to get 54.6 MB of archives. 2023-10-16T20:35:06.479 INFO:teuthology.orchestra.run.smithi088.stdout:After this operation, 202 MB of additional disk space will be used. 2023-10-16T20:35:06.479 INFO:teuthology.orchestra.run.smithi088.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-10-16T20:35:06.520 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.10_amd64.deb ... 2023-10-16T20:35:06.690 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.10) over (3.0.2-0ubuntu1.7) ... 2023-10-16T20:35:06.764 INFO:teuthology.orchestra.run.smithi088.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.10 [1,901 kB] 2023-10-16T20:35:06.917 INFO:teuthology.orchestra.run.smithi088.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-10-16T20:35:06.945 INFO:teuthology.orchestra.run.smithi088.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-10-16T20:35:06.946 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-10-16T20:35:07.086 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.10) ... 2023-10-16T20:35:07.508 INFO:teuthology.orchestra.run.smithi088.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-10-16T20:35:07.509 INFO:teuthology.orchestra.run.smithi088.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-10-16T20:35:07.510 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-10-16T20:35:07.511 INFO:teuthology.orchestra.run.smithi088.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-10-16T20:35:07.512 INFO:teuthology.orchestra.run.smithi088.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-10-16T20:35:07.513 INFO:teuthology.orchestra.run.smithi088.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-10-16T20:35:07.513 INFO:teuthology.orchestra.run.smithi088.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-10-16T20:35:07.515 INFO:teuthology.orchestra.run.smithi088.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-10-16T20:35:07.640 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package bison. 2023-10-16T20:35:07.711 INFO:teuthology.orchestra.run.smithi191.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 139924 files and directories currently installed.) 2023-10-16T20:35:07.715 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-10-16T20:35:07.751 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-10-16T20:35:08.101 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-16T20:35:08.115 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-10-16T20:35:08.119 INFO:teuthology.orchestra.run.smithi088.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-10-16T20:35:08.121 INFO:teuthology.orchestra.run.smithi088.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-10-16T20:35:08.122 INFO:teuthology.orchestra.run.smithi088.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-10-16T20:35:08.124 INFO:teuthology.orchestra.run.smithi088.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-10-16T20:35:08.125 INFO:teuthology.orchestra.run.smithi088.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-10-16T20:35:08.136 INFO:teuthology.orchestra.run.smithi088.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-10-16T20:35:08.140 INFO:teuthology.orchestra.run.smithi088.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-10-16T20:35:08.156 INFO:teuthology.orchestra.run.smithi088.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-10-16T20:35:08.157 INFO:teuthology.orchestra.run.smithi088.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.10 [2,372 kB] 2023-10-16T20:35:08.184 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-10-16T20:35:08.214 INFO:teuthology.orchestra.run.smithi088.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-10-16T20:35:08.215 INFO:teuthology.orchestra.run.smithi088.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-10-16T20:35:08.244 INFO:teuthology.orchestra.run.smithi088.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-10-16T20:35:08.341 INFO:teuthology.orchestra.run.smithi088.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-10-16T20:35:08.344 INFO:teuthology.orchestra.run.smithi088.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-10-16T20:35:08.345 INFO:teuthology.orchestra.run.smithi088.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-10-16T20:35:08.346 INFO:teuthology.orchestra.run.smithi088.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-10-16T20:35:08.346 INFO:teuthology.orchestra.run.smithi088.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-10-16T20:35:08.373 INFO:teuthology.orchestra.run.smithi088.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-10-16T20:35:08.375 INFO:teuthology.orchestra.run.smithi088.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-10-16T20:35:08.495 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package dump. 2023-10-16T20:35:08.510 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-10-16T20:35:08.554 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking dump (0.4b47-3) ... 2023-10-16T20:35:08.779 INFO:teuthology.orchestra.run.smithi088.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-10-16T20:35:08.780 INFO:teuthology.orchestra.run.smithi088.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-10-16T20:35:08.780 INFO:teuthology.orchestra.run.smithi088.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-10-16T20:35:08.781 INFO:teuthology.orchestra.run.smithi088.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-10-16T20:35:08.781 INFO:teuthology.orchestra.run.smithi088.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-10-16T20:35:08.781 INFO:teuthology.orchestra.run.smithi088.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-10-16T20:35:08.804 INFO:teuthology.orchestra.run.smithi088.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-10-16T20:35:08.805 INFO:teuthology.orchestra.run.smithi088.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-10-16T20:35:08.824 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package indent. 2023-10-16T20:35:08.838 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-10-16T20:35:08.882 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-16T20:35:09.143 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-16T20:35:09.157 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-10-16T20:35:09.183 INFO:teuthology.orchestra.run.smithi088.stdout:Preconfiguring packages ... 2023-10-16T20:35:09.226 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-16T20:35:09.362 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 54.6 MB in 2s (22.8 MB/s) 2023-10-16T20:35:09.521 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package flex. 2023-10-16T20:35:09.554 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-16T20:35:09.569 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-10-16T20:35:09.593 INFO:teuthology.orchestra.run.smithi088.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 139851 files and directories currently installed.) 2023-10-16T20:35:09.596 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-10-16T20:35:09.613 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-16T20:35:09.652 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking flex (2.6.4-8build2) ... 2023-10-16T20:35:09.840 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-16T20:35:09.855 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-10-16T20:35:09.891 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-10-16T20:35:10.238 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.10_amd64.deb ... 2023-10-16T20:35:10.251 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libfl2:amd64. 2023-10-16T20:35:10.266 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-10-16T20:35:10.310 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-10-16T20:35:10.442 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.10) over (3.0.2-0ubuntu1.7) ... 2023-10-16T20:35:10.562 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-10-16T20:35:10.577 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-10-16T20:35:10.621 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-16T20:35:10.872 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.10) ... 2023-10-16T20:35:10.873 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libjson-perl. 2023-10-16T20:35:10.887 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-10-16T20:35:10.931 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-10-16T20:35:11.251 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libllvm14:amd64. 2023-10-16T20:35:11.265 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-10-16T20:35:11.301 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-16T20:35:11.484 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package bison. 2023-10-16T20:35:11.556 INFO:teuthology.orchestra.run.smithi088.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 139924 files and directories currently installed.) 2023-10-16T20:35:11.560 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-10-16T20:35:11.636 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-10-16T20:35:12.087 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-16T20:35:12.101 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-10-16T20:35:12.145 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-10-16T20:35:12.548 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package dump. 2023-10-16T20:35:12.562 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-10-16T20:35:12.615 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking dump (0.4b47-3) ... 2023-10-16T20:35:12.803 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-16T20:35:12.819 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-16T20:35:12.862 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-16T20:35:12.910 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package indent. 2023-10-16T20:35:12.924 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-10-16T20:35:12.960 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-16T20:35:13.131 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-16T20:35:13.146 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-16T20:35:13.179 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-16T20:35:13.189 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-16T20:35:13.193 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-10-16T20:35:13.237 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-16T20:35:13.550 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-16T20:35:13.566 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-16T20:35:13.591 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-16T20:35:13.606 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-10-16T20:35:13.608 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-16T20:35:13.649 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-16T20:35:13.903 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-16T20:35:13.910 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-16T20:35:13.918 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-10-16T20:35:13.926 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-10-16T20:35:13.953 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-10-16T20:35:13.960 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-16T20:35:14.280 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-16T20:35:14.295 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-10-16T20:35:14.306 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libfl2:amd64. 2023-10-16T20:35:14.321 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-10-16T20:35:14.347 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-16T20:35:14.365 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-10-16T20:35:14.634 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-10-16T20:35:14.649 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-10-16T20:35:14.692 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-16T20:35:14.716 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libpq5:amd64. 2023-10-16T20:35:14.732 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-16T20:35:14.774 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-16T20:35:14.953 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libjson-perl. 2023-10-16T20:35:14.968 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-10-16T20:35:15.003 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-10-16T20:35:15.118 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-16T20:35:15.134 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-16T20:35:15.176 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-16T20:35:15.281 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libllvm14:amd64. 2023-10-16T20:35:15.297 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-10-16T20:35:15.373 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-16T20:35:15.454 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-16T20:35:15.469 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-16T20:35:15.537 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-16T20:35:15.790 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-16T20:35:15.806 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.10_amd64.deb ... 2023-10-16T20:35:15.848 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.10) ... 2023-10-16T20:35:16.293 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-16T20:35:16.307 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-10-16T20:35:16.360 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-10-16T20:35:16.745 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package modemmanager. 2023-10-16T20:35:16.760 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-16T20:35:16.775 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-16T20:35:16.791 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-16T20:35:16.804 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-16T20:35:16.834 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-16T20:35:17.103 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-16T20:35:17.118 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-16T20:35:17.161 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-16T20:35:17.386 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package network-manager. 2023-10-16T20:35:17.401 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-10-16T20:35:17.453 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-10-16T20:35:17.463 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-16T20:35:17.480 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-16T20:35:17.522 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-16T20:35:17.858 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-16T20:35:17.874 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-10-16T20:35:17.916 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-16T20:35:18.076 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package ppp. 2023-10-16T20:35:18.091 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-10-16T20:35:18.252 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-16T20:35:18.268 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-10-16T20:35:18.311 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-16T20:35:18.416 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-10-16T20:35:18.655 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libpq5:amd64. 2023-10-16T20:35:18.671 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-16T20:35:18.705 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-16T20:35:18.756 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package pptp-linux. 2023-10-16T20:35:18.770 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-10-16T20:35:18.814 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-10-16T20:35:19.041 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-16T20:35:19.057 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-16T20:35:19.075 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-16T20:35:19.089 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-10-16T20:35:19.100 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-16T20:35:19.133 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-10-16T20:35:19.394 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-16T20:35:19.409 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-16T20:35:19.436 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package postgresql-client-common. 2023-10-16T20:35:19.450 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-10-16T20:35:19.452 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-16T20:35:19.494 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking postgresql-client-common (238) ... 2023-10-16T20:35:19.721 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-16T20:35:19.736 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.10_amd64.deb ... 2023-10-16T20:35:19.779 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.10) ... 2023-10-16T20:35:19.806 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package postgresql-client-14. 2023-10-16T20:35:19.820 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-16T20:35:19.856 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-16T20:35:20.241 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-16T20:35:20.256 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-10-16T20:35:20.299 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-10-16T20:35:20.435 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package postgresql-common. 2023-10-16T20:35:20.450 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-10-16T20:35:20.510 INFO:teuthology.orchestra.run.smithi191.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-10-16T20:35:20.542 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking postgresql-common (238) ... 2023-10-16T20:35:20.694 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package modemmanager. 2023-10-16T20:35:20.708 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-16T20:35:20.752 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-16T20:35:21.009 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package postgresql-14. 2023-10-16T20:35:21.024 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-16T20:35:21.070 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-16T20:35:21.358 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package network-manager. 2023-10-16T20:35:21.373 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-10-16T20:35:21.450 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-10-16T20:35:22.115 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ppp. 2023-10-16T20:35:22.130 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-10-16T20:35:22.212 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package postgresql. 2023-10-16T20:35:22.226 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-10-16T20:35:22.270 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking postgresql (14+238) ... 2023-10-16T20:35:22.484 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-10-16T20:35:22.497 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package postgresql-client. 2023-10-16T20:35:22.511 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-10-16T20:35:22.547 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking postgresql-client (14+238) ... 2023-10-16T20:35:22.799 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package postgresql-contrib. 2023-10-16T20:35:22.813 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-10-16T20:35:22.858 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking postgresql-contrib (14+238) ... 2023-10-16T20:35:22.870 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package pptp-linux. 2023-10-16T20:35:22.884 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-10-16T20:35:22.928 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-10-16T20:35:23.176 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package tcl8.6. 2023-10-16T20:35:23.181 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-16T20:35:23.191 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-10-16T20:35:23.194 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-10-16T20:35:23.235 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-16T20:35:23.239 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-10-16T20:35:23.521 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package tcl. 2023-10-16T20:35:23.525 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package postgresql-client-common. 2023-10-16T20:35:23.535 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-10-16T20:35:23.539 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-10-16T20:35:23.583 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-10-16T20:35:23.583 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking postgresql-client-common (238) ... 2023-10-16T20:35:23.879 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package postgresql-client-14. 2023-10-16T20:35:23.893 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-16T20:35:23.898 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package wpasupplicant. 2023-10-16T20:35:23.912 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-10-16T20:35:23.937 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-16T20:35:23.948 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-16T20:35:24.302 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-16T20:35:24.316 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-10-16T20:35:24.360 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-10-16T20:35:24.509 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package postgresql-common. 2023-10-16T20:35:24.523 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-10-16T20:35:24.584 INFO:teuthology.orchestra.run.smithi088.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-10-16T20:35:24.624 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking postgresql-common (238) ... 2023-10-16T20:35:24.722 INFO:teuthology.orchestra.run.smithi191.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-16T20:35:24.736 INFO:teuthology.orchestra.run.smithi191.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-10-16T20:35:24.804 INFO:teuthology.orchestra.run.smithi191.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-16T20:35:25.083 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package postgresql-14. 2023-10-16T20:35:25.095 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up postgresql-client-common (238) ... 2023-10-16T20:35:25.097 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-16T20:35:25.135 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-16T20:35:25.311 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up flex (2.6.4-8build2) ... 2023-10-16T20:35:25.462 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-16T20:35:25.604 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-16T20:35:25.730 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-10-16T20:35:25.882 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-16T20:35:26.007 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-16T20:35:26.269 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package postgresql. 2023-10-16T20:35:26.283 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-10-16T20:35:26.327 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking postgresql (14+238) ... 2023-10-16T20:35:26.571 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package postgresql-client. 2023-10-16T20:35:26.585 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-10-16T20:35:26.646 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking postgresql-client (14+238) ... 2023-10-16T20:35:26.915 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package postgresql-contrib. 2023-10-16T20:35:26.929 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-10-16T20:35:26.973 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking postgresql-contrib (14+238) ... 2023-10-16T20:35:27.014 INFO:teuthology.orchestra.run.smithi191.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-10-16T20:35:27.105 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-16T20:35:27.267 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package tcl8.6. 2023-10-16T20:35:27.282 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-10-16T20:35:27.318 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-16T20:35:27.482 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-16T20:35:27.482 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-16T20:35:27.570 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package tcl. 2023-10-16T20:35:27.584 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-10-16T20:35:27.633 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-10-16T20:35:27.956 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package wpasupplicant. 2023-10-16T20:35:27.971 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-10-16T20:35:28.006 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-16T20:35:28.183 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-10-16T20:35:28.308 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.10) ... 2023-10-16T20:35:28.360 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-16T20:35:28.375 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-10-16T20:35:28.419 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-10-16T20:35:28.434 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-10-16T20:35:28.560 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-10-16T20:35:28.686 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-10-16T20:35:28.797 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-16T20:35:28.812 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-16T20:35:28.813 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-10-16T20:35:28.880 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-16T20:35:28.963 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-10-16T20:35:29.163 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up postgresql-client-common (238) ... 2023-10-16T20:35:29.395 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up flex (2.6.4-8build2) ... 2023-10-16T20:35:29.546 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-16T20:35:29.685 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-16T20:35:29.697 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-16T20:35:29.811 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up dump (0.4b47-3) ... 2023-10-16T20:35:29.848 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-10-16T20:35:29.931 INFO:teuthology.orchestra.run.smithi191.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-16T20:35:29.991 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-16T20:35:29.996 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-16T20:35:30.122 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-16T20:35:30.125 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-16T20:35:30.223 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up tcl (8.6.11+1build2) ... 2023-10-16T20:35:30.340 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libjson-perl (4.04000-1) ... 2023-10-16T20:35:30.467 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-10-16T20:35:30.556 INFO:teuthology.orchestra.run.smithi191.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-10-16T20:35:30.626 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-16T20:35:30.753 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-16T20:35:30.953 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-16T20:35:31.080 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-16T20:35:31.171 INFO:teuthology.orchestra.run.smithi088.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-10-16T20:35:31.205 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-10-16T20:35:31.348 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up postgresql-client (14+238) ... 2023-10-16T20:35:31.434 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-16T20:35:31.819 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-16T20:35:31.819 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-16T20:35:31.824 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-10-16T20:35:32.567 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-10-16T20:35:32.718 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.10) ... 2023-10-16T20:35:32.815 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:32.815 INFO:teuthology.orchestra.run.smithi191.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-16T20:35:32.815 INFO:teuthology.orchestra.run.smithi191.stdout:which means they are currently configured by ifupdown: 2023-10-16T20:35:32.815 INFO:teuthology.orchestra.run.smithi191.stdout:- enp3s0f1 2023-10-16T20:35:32.815 INFO:teuthology.orchestra.run.smithi191.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-16T20:35:32.816 INFO:teuthology.orchestra.run.smithi191.stdout:remove their configuration from /etc/network/interfaces. 2023-10-16T20:35:32.816 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:32.878 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-10-16T20:35:32.947 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-16T20:35:33.054 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-10-16T20:35:33.171 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-10-16T20:35:33.339 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-16T20:35:33.364 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-16T20:35:33.490 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-10-16T20:35:33.727 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-16T20:35:34.179 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-16T20:35:34.289 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up dump (0.4b47-3) ... 2023-10-16T20:35:34.376 INFO:teuthology.orchestra.run.smithi088.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-16T20:35:34.410 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-16T20:35:34.440 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-16T20:35:34.553 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-16T20:35:34.566 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-16T20:35:34.679 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-10-16T20:35:34.692 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up tcl (8.6.11+1build2) ... 2023-10-16T20:35:34.788 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up postgresql-common (238) ... 2023-10-16T20:35:34.818 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libjson-perl (4.04000-1) ... 2023-10-16T20:35:34.944 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-10-16T20:35:35.059 INFO:teuthology.orchestra.run.smithi088.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-10-16T20:35:35.138 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-16T20:35:35.247 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-16T20:35:35.415 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-16T20:35:35.541 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-16T20:35:35.667 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-10-16T20:35:35.732 INFO:teuthology.orchestra.run.smithi191.stdout:Adding user postgres to group ssl-cert 2023-10-16T20:35:35.835 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up postgresql-client (14+238) ... 2023-10-16T20:35:35.962 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-10-16T20:35:35.994 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:35.994 INFO:teuthology.orchestra.run.smithi191.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-10-16T20:35:36.253 INFO:teuthology.orchestra.run.smithi191.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-10-16T20:35:36.254 INFO:teuthology.orchestra.run.smithi191.stdout:Removing obsolete dictionary files: 2023-10-16T20:35:36.923 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:36.923 INFO:teuthology.orchestra.run.smithi088.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-16T20:35:36.924 INFO:teuthology.orchestra.run.smithi088.stdout:which means they are currently configured by ifupdown: 2023-10-16T20:35:36.924 INFO:teuthology.orchestra.run.smithi088.stdout:- enp3s0f1 2023-10-16T20:35:36.924 INFO:teuthology.orchestra.run.smithi088.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-16T20:35:36.924 INFO:teuthology.orchestra.run.smithi088.stdout:remove their configuration from /etc/network/interfaces. 2023-10-16T20:35:36.924 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:36.996 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-10-16T20:35:37.055 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-16T20:35:37.447 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-16T20:35:37.464 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-16T20:35:37.847 INFO:teuthology.orchestra.run.smithi191.stdout:Creating new PostgreSQL cluster 14/main ... 2023-10-16T20:35:37.853 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-16T20:35:37.865 INFO:teuthology.orchestra.run.smithi191.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-10-16T20:35:37.922 INFO:teuthology.orchestra.run.smithi191.stdout:The files belonging to this database system will be owned by user "postgres". 2023-10-16T20:35:37.922 INFO:teuthology.orchestra.run.smithi191.stdout:This user must also own the server process. 2023-10-16T20:35:37.923 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:37.923 INFO:teuthology.orchestra.run.smithi191.stdout:The database cluster will be initialized with locale "en_US". 2023-10-16T20:35:37.923 INFO:teuthology.orchestra.run.smithi191.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-10-16T20:35:37.923 INFO:teuthology.orchestra.run.smithi191.stdout:The default text search configuration will be set to "english". 2023-10-16T20:35:37.923 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:37.923 INFO:teuthology.orchestra.run.smithi191.stdout:Data page checksums are disabled. 2023-10-16T20:35:37.924 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:37.924 INFO:teuthology.orchestra.run.smithi191.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-10-16T20:35:37.925 INFO:teuthology.orchestra.run.smithi191.stdout:creating subdirectories ... ok 2023-10-16T20:35:37.925 INFO:teuthology.orchestra.run.smithi191.stdout:selecting dynamic shared memory implementation ... posix 2023-10-16T20:35:37.948 INFO:teuthology.orchestra.run.smithi191.stdout:selecting default max_connections ... 100 2023-10-16T20:35:37.983 INFO:teuthology.orchestra.run.smithi191.stdout:selecting default shared_buffers ... 128MB 2023-10-16T20:35:37.997 INFO:teuthology.orchestra.run.smithi191.stdout:selecting default time zone ... Etc/UTC 2023-10-16T20:35:37.998 INFO:teuthology.orchestra.run.smithi191.stdout:creating configuration files ... ok 2023-10-16T20:35:38.166 INFO:teuthology.orchestra.run.smithi191.stdout:running bootstrap script ... ok 2023-10-16T20:35:38.548 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-16T20:35:38.733 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-16T20:35:38.769 INFO:teuthology.orchestra.run.smithi191.stdout:performing post-bootstrap initialization ... ok 2023-10-16T20:35:38.883 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-10-16T20:35:39.034 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up postgresql-common (238) ... 2023-10-16T20:35:39.429 INFO:teuthology.orchestra.run.smithi191.stdout:syncing data to disk ... ok 2023-10-16T20:35:39.951 INFO:teuthology.orchestra.run.smithi191.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-10-16T20:35:40.026 INFO:teuthology.orchestra.run.smithi088.stdout:Adding user postgres to group ssl-cert 2023-10-16T20:35:40.369 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:40.369 INFO:teuthology.orchestra.run.smithi088.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-10-16T20:35:40.629 INFO:teuthology.orchestra.run.smithi088.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-10-16T20:35:40.629 INFO:teuthology.orchestra.run.smithi088.stdout:Removing obsolete dictionary files: 2023-10-16T20:35:41.333 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-10-16T20:35:41.870 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-16T20:35:42.189 INFO:teuthology.orchestra.run.smithi088.stdout:Creating new PostgreSQL cluster 14/main ... 2023-10-16T20:35:42.213 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-10-16T20:35:42.259 INFO:teuthology.orchestra.run.smithi088.stdout:The files belonging to this database system will be owned by user "postgres". 2023-10-16T20:35:42.259 INFO:teuthology.orchestra.run.smithi088.stdout:This user must also own the server process. 2023-10-16T20:35:42.259 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:42.259 INFO:teuthology.orchestra.run.smithi088.stdout:The database cluster will be initialized with locale "en_US". 2023-10-16T20:35:42.260 INFO:teuthology.orchestra.run.smithi088.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-10-16T20:35:42.260 INFO:teuthology.orchestra.run.smithi088.stdout:The default text search configuration will be set to "english". 2023-10-16T20:35:42.260 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:42.260 INFO:teuthology.orchestra.run.smithi088.stdout:Data page checksums are disabled. 2023-10-16T20:35:42.260 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:42.261 INFO:teuthology.orchestra.run.smithi088.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-10-16T20:35:42.261 INFO:teuthology.orchestra.run.smithi088.stdout:creating subdirectories ... ok 2023-10-16T20:35:42.261 INFO:teuthology.orchestra.run.smithi088.stdout:selecting dynamic shared memory implementation ... posix 2023-10-16T20:35:42.284 INFO:teuthology.orchestra.run.smithi088.stdout:selecting default max_connections ... 100 2023-10-16T20:35:42.319 INFO:teuthology.orchestra.run.smithi088.stdout:selecting default shared_buffers ... 128MB 2023-10-16T20:35:42.333 INFO:teuthology.orchestra.run.smithi088.stdout:selecting default time zone ... Etc/UTC 2023-10-16T20:35:42.333 INFO:teuthology.orchestra.run.smithi088.stdout:creating configuration files ... ok 2023-10-16T20:35:42.502 INFO:teuthology.orchestra.run.smithi088.stdout:running bootstrap script ... ok 2023-10-16T20:35:42.997 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-16T20:35:43.119 INFO:teuthology.orchestra.run.smithi088.stdout:performing post-bootstrap initialization ... ok 2023-10-16T20:35:43.131 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up postgresql-contrib (14+238) ... 2023-10-16T20:35:43.265 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up postgresql (14+238) ... 2023-10-16T20:35:43.391 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-16T20:35:43.567 INFO:teuthology.orchestra.run.smithi191.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-16T20:35:43.656 INFO:teuthology.orchestra.run.smithi088.stdout:syncing data to disk ... ok 2023-10-16T20:35:43.853 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-16T20:35:43.853 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-16T20:35:44.169 INFO:teuthology.orchestra.run.smithi088.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-10-16T20:35:44.603 INFO:teuthology.orchestra.run.smithi191.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-16T20:35:44.929 INFO:teuthology.orchestra.run.smithi191.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-16T20:35:45.081 INFO:teuthology.orchestra.run.smithi191.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-16T20:35:46.492 INFO:teuthology.orchestra.run.smithi191.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-10-16T20:35:46.617 INFO:teuthology.orchestra.run.smithi191.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-10-16T20:35:47.210 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-16T20:35:47.296 INFO:teuthology.orchestra.run.smithi191.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-16T20:35:47.301 DEBUG:teuthology.parallel:result is None 2023-10-16T20:35:47.361 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up postgresql-contrib (14+238) ... 2023-10-16T20:35:47.504 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up postgresql (14+238) ... 2023-10-16T20:35:47.655 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-16T20:35:47.806 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-16T20:35:48.108 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-16T20:35:48.108 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-16T20:35:48.850 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-16T20:35:49.185 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-16T20:35:49.337 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-16T20:35:50.781 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-10-16T20:35:50.898 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-10-16T20:35:51.590 INFO:teuthology.orchestra.run.smithi088.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-16T20:35:51.594 DEBUG:teuthology.parallel:result is None 2023-10-16T20:35:51.595 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 2023-10-16T20:35:51.924 DEBUG:teuthology.orchestra.run.smithi088:> dpkg-query -W -f '${Version}' ceph 2023-10-16T20:35:51.947 INFO:teuthology.orchestra.run.smithi088.stdout:18.0.0-6730-g4ba8a190-1jammy 2023-10-16T20:35:51.947 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6730-g4ba8a190-1jammy 2023-10-16T20:35:51.947 INFO:teuthology.task.install:The correct ceph version 18.0.0-6730-g4ba8a190-1jammy is installed. 2023-10-16T20:35:51.948 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0 2023-10-16T20:35:52.091 DEBUG:teuthology.orchestra.run.smithi191:> dpkg-query -W -f '${Version}' ceph 2023-10-16T20:35:52.115 INFO:teuthology.orchestra.run.smithi191.stdout:18.0.0-6730-g4ba8a190-1jammy 2023-10-16T20:35:52.116 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6730-g4ba8a190-1jammy 2023-10-16T20:35:52.116 INFO:teuthology.task.install:The correct ceph version 18.0.0-6730-g4ba8a190-1jammy is installed. 2023-10-16T20:35:52.116 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-16T20:35:52.117 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:35:52.117 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-16T20:35:52.138 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:35:52.139 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-16T20:35:52.181 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-16T20:35:52.181 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:35:52.182 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2023-10-16T20:35:52.205 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-16T20:35:52.265 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:35:52.266 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2023-10-16T20:35:52.288 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-16T20:35:52.356 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-16T20:35:52.356 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:35:52.357 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-16T20:35:52.377 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-16T20:35:52.441 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:35:52.441 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-16T20:35:52.462 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-16T20:35:52.523 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-16T20:35:52.523 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:35:52.524 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2023-10-16T20:35:52.544 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-16T20:35:52.608 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:35:52.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2023-10-16T20:35:52.630 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-16T20:35:52.695 INFO:teuthology.run_tasks:Running task ceph... 2023-10-16T20:35:52.754 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-16T20:35:52.755 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /var/log/ceph 2023-10-16T20:35:52.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /var/log/ceph 2023-10-16T20:35:52.780 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-16T20:35:52.780 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-16T20:35:52.820 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-16T20:35:52.843 INFO:tasks.ceph:Creating extra log directories... 2023-10-16T20:35:52.843 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-16T20:35:52.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-16T20:35:52.913 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-16T20:35:52.914 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-16T20:35:52.914 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-16T20:35:52.914 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:35:52.915 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-16T20:35:52.949 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-16T20:35:53.037 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:35:53.037 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-16T20:35:53.047 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-16T20:35:53.147 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-16T20:35:53.147 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-16T20:35:53.148 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi/7430114', 'branch': 'rishabh-2023oct13', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': None, 'first_in_suite': False, 'job_id': '7430114', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '47672119149b103ec7cec2cc65c063ccbb001ac7'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'rishabh-2023oct13'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'rishabh-2023oct13-qa2', 'sha1': '19fd96fe3581520b6d9f50ffdabf29051aee5d4d'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 6006, 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0', 'sleep_before_teardown': 0, 'subset': '402/512', 'suite': 'fs', 'suite_branch': 'rishabh-2023oct13-qa2', 'suite_path': '/home/teuthworker/src/github.com_rishabh-d-dave_ceph_19fd96fe3581520b6d9f50ffdabf29051aee5d4d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/rishabh-d-dave/ceph', 'suite_sha1': '19fd96fe3581520b6d9f50ffdabf29051aee5d4d', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi088.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO3ZZJl3J3J2FRWmXEHKzqpNP/UrrVcr/DKpKW9bbB30t5/mQLXo9MV/4v88nZ3Aidf1/I++8C8HmPysMI48YZU=', 'smithi191.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBENSQMED5hBpKUrlawu3SsqoqtNsEGYfWpGz3nN8acKssOxqU8AqvVaZ6pkCVsiVeDFvVF1lai5z+ACojtAJSDc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '47672119149b103ec7cec2cc65c063ccbb001ac7'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': '4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-16_18:49:15', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-16T20:35:53.149 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-16T20:35:53.153 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-16T20:35:53.162 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/run/ceph 2023-10-16T20:35:53.204 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/run/ceph 2023-10-16T20:35:53.226 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:35:53.226 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2023-10-16T20:35:53.270 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-16T20:35:53.270 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_1 2023-10-16T20:35:53.326 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-16T20:35:53.326 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T20:35:53.326 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 901 Links: 1 2023-10-16T20:35:53.327 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T20:35:53.327 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-10-16 20:29:23.788840181 +0000 2023-10-16T20:35:53.327 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-10-16 20:29:23.376843861 +0000 2023-10-16T20:35:53.327 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-10-16 20:29:23.376843861 +0000 2023-10-16T20:35:53.327 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2023-10-16 20:29:23.376843861 +0000 2023-10-16T20:35:53.328 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-16T20:35:53.387 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-10-16T20:35:53.387 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-10-16T20:35:53.387 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000363677 s, 1.4 MB/s 2023-10-16T20:35:53.389 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-16T20:35:53.440 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_2 2023-10-16T20:35:53.490 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-16T20:35:53.490 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T20:35:53.490 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 920 Links: 1 2023-10-16T20:35:53.491 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T20:35:53.491 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-10-16 20:29:24.452834251 +0000 2023-10-16T20:35:53.491 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-10-16 20:29:24.072837646 +0000 2023-10-16T20:35:53.491 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-10-16 20:29:24.072837646 +0000 2023-10-16T20:35:53.491 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2023-10-16 20:29:24.072837646 +0000 2023-10-16T20:35:53.492 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-16T20:35:53.551 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-10-16T20:35:53.551 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-10-16T20:35:53.552 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000354284 s, 1.4 MB/s 2023-10-16T20:35:53.553 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-16T20:35:53.603 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_3 2023-10-16T20:35:53.658 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-16T20:35:53.658 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T20:35:53.658 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 936 Links: 1 2023-10-16T20:35:53.659 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T20:35:53.659 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-10-16 20:29:25.164827891 +0000 2023-10-16T20:35:53.659 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-10-16 20:29:24.760831499 +0000 2023-10-16T20:35:53.659 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-10-16 20:29:24.760831499 +0000 2023-10-16T20:35:53.660 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2023-10-16 20:29:24.760831499 +0000 2023-10-16T20:35:53.660 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-16T20:35:53.723 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-10-16T20:35:53.724 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-10-16T20:35:53.724 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000338889 s, 1.5 MB/s 2023-10-16T20:35:53.725 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-16T20:35:53.775 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_4 2023-10-16T20:35:53.830 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-16T20:35:53.830 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T20:35:53.830 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 955 Links: 1 2023-10-16T20:35:53.831 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T20:35:53.831 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-10-16 20:29:25.804822174 +0000 2023-10-16T20:35:53.831 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-10-16 20:29:25.412825676 +0000 2023-10-16T20:35:53.831 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-10-16 20:29:25.412825676 +0000 2023-10-16T20:35:53.831 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2023-10-16 20:29:25.412825676 +0000 2023-10-16T20:35:53.832 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-16T20:35:53.895 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-10-16T20:35:53.895 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-10-16T20:35:53.896 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000366443 s, 1.4 MB/s 2023-10-16T20:35:53.897 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-16T20:35:53.947 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T20:35:53.948 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:35:53.948 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2023-10-16T20:35:53.958 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-16T20:35:53.959 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2023-10-16T20:35:54.012 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-16T20:35:54.012 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T20:35:54.013 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 889 Links: 1 2023-10-16T20:35:54.013 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T20:35:54.013 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-10-16 20:29:25.902663605 +0000 2023-10-16T20:35:54.013 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-10-16 20:29:25.898663640 +0000 2023-10-16T20:35:54.013 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-10-16 20:29:25.898663640 +0000 2023-10-16T20:35:54.014 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2023-10-16 20:29:25.898663640 +0000 2023-10-16T20:35:54.014 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-16T20:35:54.082 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-10-16T20:35:54.082 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-10-16T20:35:54.083 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000396584 s, 1.3 MB/s 2023-10-16T20:35:54.084 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-16T20:35:54.142 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2023-10-16T20:35:54.192 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-16T20:35:54.192 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T20:35:54.192 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 908 Links: 1 2023-10-16T20:35:54.192 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T20:35:54.192 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-10-16 20:29:27.026653564 +0000 2023-10-16T20:35:54.193 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-10-16 20:29:26.610657280 +0000 2023-10-16T20:35:54.193 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-10-16 20:29:26.610657280 +0000 2023-10-16T20:35:54.193 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2023-10-16 20:29:26.610657280 +0000 2023-10-16T20:35:54.193 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-16T20:35:54.258 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-10-16T20:35:54.259 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-10-16T20:35:54.259 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000378696 s, 1.4 MB/s 2023-10-16T20:35:54.260 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-16T20:35:54.314 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2023-10-16T20:35:54.368 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-16T20:35:54.368 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T20:35:54.368 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 924 Links: 1 2023-10-16T20:35:54.369 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T20:35:54.369 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-10-16 20:29:27.690647633 +0000 2023-10-16T20:35:54.369 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-10-16 20:29:27.302651099 +0000 2023-10-16T20:35:54.369 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-10-16 20:29:27.302651099 +0000 2023-10-16T20:35:54.369 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2023-10-16 20:29:27.302651099 +0000 2023-10-16T20:35:54.370 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-16T20:35:54.434 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-10-16T20:35:54.435 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-10-16T20:35:54.435 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000388461 s, 1.3 MB/s 2023-10-16T20:35:54.436 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-16T20:35:54.494 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2023-10-16T20:35:54.548 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-16T20:35:54.548 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T20:35:54.549 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 943 Links: 1 2023-10-16T20:35:54.549 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T20:35:54.549 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-10-16 20:29:28.342641810 +0000 2023-10-16T20:35:54.549 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-10-16 20:29:27.942645383 +0000 2023-10-16T20:35:54.549 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-10-16 20:29:27.942645383 +0000 2023-10-16T20:35:54.550 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2023-10-16 20:29:27.942645383 +0000 2023-10-16T20:35:54.550 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-16T20:35:54.614 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-10-16T20:35:54.614 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-10-16T20:35:54.615 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000383884 s, 1.3 MB/s 2023-10-16T20:35:54.616 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-16T20:35:54.674 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T20:35:54.674 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-16T20:35:54.674 INFO:tasks.ceph:Generating config... 2023-10-16T20:35:54.675 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-16T20:35:54.676 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2023-10-16T20:35:54.676 INFO:tasks.ceph:[client] debug client = 20 2023-10-16T20:35:54.676 INFO:tasks.ceph:[client] debug ms = 1 2023-10-16T20:35:54.677 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-16T20:35:54.677 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-16T20:35:54.677 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-16T20:35:54.677 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-10-16T20:35:54.678 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-16T20:35:54.678 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-10-16T20:35:54.678 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-10-16T20:35:54.679 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-10-16T20:35:54.679 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-10-16T20:35:54.679 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-16T20:35:54.680 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-16T20:35:54.680 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = False 2023-10-16T20:35:54.680 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-16T20:35:54.680 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-16T20:35:54.681 INFO:tasks.ceph:[mds] mds_dir_prefetch = False 2023-10-16T20:35:54.681 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2023-10-16T20:35:54.681 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2023-10-16T20:35:54.682 INFO:tasks.ceph:[mds] ms inject delay max = 1 2023-10-16T20:35:54.684 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2023-10-16T20:35:54.684 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2023-10-16T20:35:54.684 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2023-10-16T20:35:54.684 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-16T20:35:54.685 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-16T20:35:54.685 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-16T20:35:54.685 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-16T20:35:54.685 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-16T20:35:54.685 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-16T20:35:54.686 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-16T20:35:54.686 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-16T20:35:54.686 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-16T20:35:54.687 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-16T20:35:54.687 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-16T20:35:54.687 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-10-16T20:35:54.687 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-10-16T20:35:54.688 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-16T20:35:54.688 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-16T20:35:54.688 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-16T20:35:54.688 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2023-10-16T20:35:54.688 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-16T20:35:54.689 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-16T20:35:54.689 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-16T20:35:54.689 INFO:tasks.ceph:[osd] ms inject delay max = 1 2023-10-16T20:35:54.689 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2023-10-16T20:35:54.690 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2023-10-16T20:35:54.690 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2023-10-16T20:35:54.690 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-10-16T20:35:54.690 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-16T20:35:54.691 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-10-16T20:35:54.691 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-16T20:35:54.691 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-16T20:35:54.692 INFO:tasks.ceph:Setting up mon.a... 2023-10-16T20:35:54.692 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-16T20:35:54.739 INFO:teuthology.orchestra.run.smithi088.stdout:creating /etc/ceph/ceph.keyring 2023-10-16T20:35:54.757 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-16T20:35:54.805 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-16T20:35:54.825 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.88'), ('mon.b', '172.21.15.191'), ('mon.c', '[v2:172.21.15.191:3301,v1:172.21.15.191:6790]')] 2023-10-16T20:35:54.826 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.88,172.21.15.191,[v2:172.21.15.191:3301,v1:172.21.15.191:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-16T20:35:54.827 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:35:54.828 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-16T20:35:54.877 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.88 --add b 172.21.15.191 --addv c '[v2:172.21.15.191:3301,v1:172.21.15.191:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-16T20:35:54.968 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-16T20:35:54.968 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: generated fsid 8cf7342f-a394-4b4c-ac35-9931ac9d9e57 2023-10-16T20:35:54.969 INFO:teuthology.orchestra.run.smithi088.stdout:setting min_mon_release = pacific 2023-10-16T20:35:54.969 INFO:teuthology.orchestra.run.smithi088.stdout:epoch 0 2023-10-16T20:35:54.969 INFO:teuthology.orchestra.run.smithi088.stdout:fsid 8cf7342f-a394-4b4c-ac35-9931ac9d9e57 2023-10-16T20:35:54.969 INFO:teuthology.orchestra.run.smithi088.stdout:last_changed 2023-10-16T20:35:54.963315+0000 2023-10-16T20:35:54.969 INFO:teuthology.orchestra.run.smithi088.stdout:created 2023-10-16T20:35:54.963315+0000 2023-10-16T20:35:54.970 INFO:teuthology.orchestra.run.smithi088.stdout:min_mon_release 16 (pacific) 2023-10-16T20:35:54.970 INFO:teuthology.orchestra.run.smithi088.stdout:election_strategy: 1 2023-10-16T20:35:54.970 INFO:teuthology.orchestra.run.smithi088.stdout:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2023-10-16T20:35:54.970 INFO:teuthology.orchestra.run.smithi088.stdout:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-10-16T20:35:54.970 INFO:teuthology.orchestra.run.smithi088.stdout:2: [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] mon.c 2023-10-16T20:35:54.971 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-16T20:35:54.972 DEBUG:teuthology.orchestra.run.smithi088:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-16T20:35:54.981 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8cf7342f-a394-4b4c-ac35-9931ac9d9e57... 2023-10-16T20:35:54.982 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-16T20:35:55.029 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-16T20:35:55.070 INFO:teuthology.orchestra.run.smithi088.stdout:[global] 2023-10-16T20:35:55.070 INFO:teuthology.orchestra.run.smithi088.stdout: chdir = "" 2023-10-16T20:35:55.070 INFO:teuthology.orchestra.run.smithi088.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-16T20:35:55.070 INFO:teuthology.orchestra.run.smithi088.stdout: auth supported = cephx 2023-10-16T20:35:55.071 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.071 INFO:teuthology.orchestra.run.smithi088.stdout: filestore xattr use omap = true 2023-10-16T20:35:55.071 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.071 INFO:teuthology.orchestra.run.smithi088.stdout: mon clock drift allowed = 1.000 2023-10-16T20:35:55.071 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.072 INFO:teuthology.orchestra.run.smithi088.stdout: osd crush chooseleaf type = 0 2023-10-16T20:35:55.072 INFO:teuthology.orchestra.run.smithi088.stdout: auth debug = true 2023-10-16T20:35:55.072 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.072 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on old message = true 2023-10-16T20:35:55.072 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on bug = true 2023-10-16T20:35:55.073 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.073 INFO:teuthology.orchestra.run.smithi088.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-16T20:35:55.073 INFO:teuthology.orchestra.run.smithi088.stdout: mon pg warn max object skew = 0 2023-10-16T20:35:55.073 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.073 INFO:teuthology.orchestra.run.smithi088.stdout: # disable pg_autoscaler by default for new pools 2023-10-16T20:35:55.074 INFO:teuthology.orchestra.run.smithi088.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-16T20:35:55.074 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.074 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default size = 2 2023-10-16T20:35:55.074 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.074 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow primary affinity = true 2023-10-16T20:35:55.075 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow pg remap = true 2023-10-16T20:35:55.075 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on legacy crush tunables = false 2023-10-16T20:35:55.075 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on crush straw calc version zero = false 2023-10-16T20:35:55.075 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on no sortbitwise = false 2023-10-16T20:35:55.075 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on osd down out interval zero = false 2023-10-16T20:35:55.075 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on too few osds = false 2023-10-16T20:35:55.076 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-16T20:35:55.076 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-16T20:35:55.076 INFO:teuthology.orchestra.run.smithi088.stdout: mon_allow_pool_size_one = true 2023-10-16T20:35:55.076 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.076 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-16T20:35:55.076 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.077 INFO:teuthology.orchestra.run.smithi088.stdout: osd default data pool replay window = 5 2023-10-16T20:35:55.077 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.077 INFO:teuthology.orchestra.run.smithi088.stdout: mon allow pool delete = true 2023-10-16T20:35:55.077 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.077 INFO:teuthology.orchestra.run.smithi088.stdout: mon cluster log file level = debug 2023-10-16T20:35:55.078 INFO:teuthology.orchestra.run.smithi088.stdout: debug asserts on shutdown = true 2023-10-16T20:35:55.078 INFO:teuthology.orchestra.run.smithi088.stdout: mon health detail to clog = false 2023-10-16T20:35:55.078 INFO:teuthology.orchestra.run.smithi088.stdout: mon host = "172.21.15.88,172.21.15.191,[v2:172.21.15.191:3301,v1:172.21.15.191:6790]" 2023-10-16T20:35:55.078 INFO:teuthology.orchestra.run.smithi088.stdout: fsid = 8cf7342f-a394-4b4c-ac35-9931ac9d9e57 2023-10-16T20:35:55.078 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.079 INFO:teuthology.orchestra.run.smithi088.stdout:[osd] 2023-10-16T20:35:55.079 INFO:teuthology.orchestra.run.smithi088.stdout: osd journal size = 100 2023-10-16T20:35:55.079 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.079 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub load threshold = 5.0 2023-10-16T20:35:55.079 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub max interval = 600 2023-10-16T20:35:55.079 INFO:teuthology.orchestra.run.smithi088.stdout: osd mclock profile = high_recovery_ops 2023-10-16T20:35:55.080 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.080 INFO:teuthology.orchestra.run.smithi088.stdout: osd recover clone overlap = true 2023-10-16T20:35:55.080 INFO:teuthology.orchestra.run.smithi088.stdout: osd recovery max chunk = 1048576 2023-10-16T20:35:55.080 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.080 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug shutdown = true 2023-10-16T20:35:55.080 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug op order = true 2023-10-16T20:35:55.081 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify stray on activate = true 2023-10-16T20:35:55.081 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.081 INFO:teuthology.orchestra.run.smithi088.stdout: osd open classes on start = true 2023-10-16T20:35:55.081 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug pg log writeout = true 2023-10-16T20:35:55.081 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.081 INFO:teuthology.orchestra.run.smithi088.stdout: osd deep scrub update digest min age = 30 2023-10-16T20:35:55.082 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.082 INFO:teuthology.orchestra.run.smithi088.stdout: osd map max advance = 10 2023-10-16T20:35:55.082 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.082 INFO:teuthology.orchestra.run.smithi088.stdout: journal zero on create = true 2023-10-16T20:35:55.082 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.083 INFO:teuthology.orchestra.run.smithi088.stdout: filestore ondisk finisher threads = 3 2023-10-16T20:35:55.083 INFO:teuthology.orchestra.run.smithi088.stdout: filestore apply finisher threads = 3 2023-10-16T20:35:55.083 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.083 INFO:teuthology.orchestra.run.smithi088.stdout: bdev debug aio = true 2023-10-16T20:35:55.083 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug misdirected ops = true 2023-10-16T20:35:55.083 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore block size = 96636764160 2023-10-16T20:35:55.084 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore fsck on mount = True 2023-10-16T20:35:55.084 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluefs = 20 2023-10-16T20:35:55.084 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluestore = 20 2023-10-16T20:35:55.084 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-10-16T20:35:55.084 INFO:teuthology.orchestra.run.smithi088.stdout: debug osd = 20 2023-10-16T20:35:55.085 INFO:teuthology.orchestra.run.smithi088.stdout: debug rocksdb = 10 2023-10-16T20:35:55.085 INFO:teuthology.orchestra.run.smithi088.stdout: mon client directed command retry = 5 2023-10-16T20:35:55.085 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd backfillfull_ratio = 0.85 2023-10-16T20:35:55.085 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd full ratio = 0.9 2023-10-16T20:35:55.085 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd nearfull ratio = 0.8 2023-10-16T20:35:55.085 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject delay max = 1 2023-10-16T20:35:55.086 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject delay probability = 0.005 2023-10-16T20:35:55.086 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject delay type = client mds 2023-10-16T20:35:55.086 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject socket failures = 2500 2023-10-16T20:35:55.086 INFO:teuthology.orchestra.run.smithi088.stdout: osd blocked scrub grace period = 3600 2023-10-16T20:35:55.086 INFO:teuthology.orchestra.run.smithi088.stdout: osd failsafe full ratio = 0.95 2023-10-16T20:35:55.087 INFO:teuthology.orchestra.run.smithi088.stdout: osd max markdown count = 1000 2023-10-16T20:35:55.087 INFO:teuthology.orchestra.run.smithi088.stdout: osd objectstore = bluestore 2023-10-16T20:35:55.087 INFO:teuthology.orchestra.run.smithi088.stdout: osd op complaint time = 180 2023-10-16T20:35:55.087 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.087 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr] 2023-10-16T20:35:55.087 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-10-16T20:35:55.088 INFO:teuthology.orchestra.run.smithi088.stdout: debug mgr = 20 2023-10-16T20:35:55.088 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2023-10-16T20:35:55.088 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2023-10-16T20:35:55.088 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2023-10-16T20:35:55.088 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2023-10-16T20:35:55.088 INFO:teuthology.orchestra.run.smithi088.stdout: mgr/telemetry/nag = false 2023-10-16T20:35:55.089 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.089 INFO:teuthology.orchestra.run.smithi088.stdout:[mon] 2023-10-16T20:35:55.089 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-10-16T20:35:55.089 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2023-10-16T20:35:55.089 INFO:teuthology.orchestra.run.smithi088.stdout: debug paxos = 20 2023-10-16T20:35:55.090 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2023-10-16T20:35:55.090 INFO:teuthology.orchestra.run.smithi088.stdout: mon data avail warn = 5 2023-10-16T20:35:55.090 INFO:teuthology.orchestra.run.smithi088.stdout: mon mgr mkfs grace = 240 2023-10-16T20:35:55.090 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2023-10-16T20:35:55.091 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd reporter subtree level = osd 2023-10-16T20:35:55.091 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd prime pg temp = true 2023-10-16T20:35:55.091 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2023-10-16T20:35:55.091 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.091 INFO:teuthology.orchestra.run.smithi088.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-16T20:35:55.092 INFO:teuthology.orchestra.run.smithi088.stdout: auth mon ticket ttl = 660 # 11m 2023-10-16T20:35:55.092 INFO:teuthology.orchestra.run.smithi088.stdout: auth service ticket ttl = 240 # 4m 2023-10-16T20:35:55.092 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.092 INFO:teuthology.orchestra.run.smithi088.stdout: # don't complain about insecure global_id in the test suite 2023-10-16T20:35:55.092 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-16T20:35:55.093 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-16T20:35:55.093 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.093 INFO:teuthology.orchestra.run.smithi088.stdout: # 1m isn't quite enough 2023-10-16T20:35:55.093 INFO:teuthology.orchestra.run.smithi088.stdout: mon_down_mkfs_grace = 2m 2023-10-16T20:35:55.093 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.093 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_filestore_osds = false 2023-10-16T20:35:55.094 INFO:teuthology.orchestra.run.smithi088.stdout: mon op complaint time = 120 2023-10-16T20:35:55.094 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:35:55.094 INFO:teuthology.orchestra.run.smithi088.stdout:[client] 2023-10-16T20:35:55.094 INFO:teuthology.orchestra.run.smithi088.stdout: rgw cache enabled = true 2023-10-16T20:35:55.094 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable ops log = true 2023-10-16T20:35:55.095 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable usage log = true 2023-10-16T20:35:55.095 INFO:teuthology.orchestra.run.smithi088.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-16T20:35:55.095 INFO:teuthology.orchestra.run.smithi088.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-16T20:35:55.095 INFO:teuthology.orchestra.run.smithi088.stdout: client mount timeout = 600 2023-10-16T20:35:55.095 INFO:teuthology.orchestra.run.smithi088.stdout: client_shutdown_timeout = 120 2023-10-16T20:35:55.096 INFO:teuthology.orchestra.run.smithi088.stdout: debug client = 20 2023-10-16T20:35:55.096 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-10-16T20:35:55.096 INFO:teuthology.orchestra.run.smithi088.stdout: rados mon op timeout = 900 2023-10-16T20:35:55.096 INFO:teuthology.orchestra.run.smithi088.stdout: rados osd op timeout = 900 2023-10-16T20:35:55.096 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.a] 2023-10-16T20:35:55.096 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.b] 2023-10-16T20:35:55.097 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.c] 2023-10-16T20:35:55.097 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.a] 2023-10-16T20:35:55.097 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.c] 2023-10-16T20:35:55.097 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.e] 2023-10-16T20:35:55.097 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.b] 2023-10-16T20:35:55.098 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.d] 2023-10-16T20:35:55.098 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.f] 2023-10-16T20:35:55.098 INFO:teuthology.orchestra.run.smithi088.stdout:[mds] 2023-10-16T20:35:55.098 INFO:teuthology.orchestra.run.smithi088.stdout: debug mds = 20 2023-10-16T20:35:55.098 INFO:teuthology.orchestra.run.smithi088.stdout: debug mds balancer = 20 2023-10-16T20:35:55.099 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2023-10-16T20:35:55.099 INFO:teuthology.orchestra.run.smithi088.stdout: mds bal fragment size max = 10000 2023-10-16T20:35:55.099 INFO:teuthology.orchestra.run.smithi088.stdout: mds bal merge size = 5 2023-10-16T20:35:55.099 INFO:teuthology.orchestra.run.smithi088.stdout: mds bal split bits = 3 2023-10-16T20:35:55.099 INFO:teuthology.orchestra.run.smithi088.stdout: mds bal split size = 100 2023-10-16T20:35:55.100 INFO:teuthology.orchestra.run.smithi088.stdout: mds debug frag = True 2023-10-16T20:35:55.100 INFO:teuthology.orchestra.run.smithi088.stdout: mds debug scatterstat = True 2023-10-16T20:35:55.100 INFO:teuthology.orchestra.run.smithi088.stdout: mds oft prefetch dirfrags = False 2023-10-16T20:35:55.100 INFO:teuthology.orchestra.run.smithi088.stdout: mds op complaint time = 180 2023-10-16T20:35:55.100 INFO:teuthology.orchestra.run.smithi088.stdout: mds verify scatter = True 2023-10-16T20:35:55.100 INFO:teuthology.orchestra.run.smithi088.stdout: mds_dir_prefetch = False 2023-10-16T20:35:55.101 INFO:teuthology.orchestra.run.smithi088.stdout: mds_sleep_rank_change = 5000000.0 2023-10-16T20:35:55.101 INFO:teuthology.orchestra.run.smithi088.stdout: mon client directed command retry = 5 2023-10-16T20:35:55.101 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject delay max = 1 2023-10-16T20:35:55.101 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject delay probability = 0.005 2023-10-16T20:35:55.101 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject delay type = client mds osd 2023-10-16T20:35:55.102 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject socket failures = 2500 2023-10-16T20:35:55.102 INFO:teuthology.orchestra.run.smithi088.stdout: osd op complaint time = 180 2023-10-16T20:35:55.102 INFO:teuthology.orchestra.run.smithi088.stdout: rados mon op timeout = 900 2023-10-16T20:35:55.102 INFO:teuthology.orchestra.run.smithi088.stdout: rados osd op timeout = 900 2023-10-16T20:35:55.104 INFO:teuthology.orchestra.run.smithi191.stdout:[global] 2023-10-16T20:35:55.104 INFO:teuthology.orchestra.run.smithi191.stdout: chdir = "" 2023-10-16T20:35:55.104 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-16T20:35:55.104 INFO:teuthology.orchestra.run.smithi191.stdout: auth supported = cephx 2023-10-16T20:35:55.104 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.105 INFO:teuthology.orchestra.run.smithi191.stdout: filestore xattr use omap = true 2023-10-16T20:35:55.105 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.105 INFO:teuthology.orchestra.run.smithi191.stdout: mon clock drift allowed = 1.000 2023-10-16T20:35:55.105 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.105 INFO:teuthology.orchestra.run.smithi191.stdout: osd crush chooseleaf type = 0 2023-10-16T20:35:55.106 INFO:teuthology.orchestra.run.smithi191.stdout: auth debug = true 2023-10-16T20:35:55.106 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.106 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on old message = true 2023-10-16T20:35:55.106 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on bug = true 2023-10-16T20:35:55.106 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.107 INFO:teuthology.orchestra.run.smithi191.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-16T20:35:55.107 INFO:teuthology.orchestra.run.smithi191.stdout: mon pg warn max object skew = 0 2023-10-16T20:35:55.107 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.107 INFO:teuthology.orchestra.run.smithi191.stdout: # disable pg_autoscaler by default for new pools 2023-10-16T20:35:55.107 INFO:teuthology.orchestra.run.smithi191.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-16T20:35:55.108 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.108 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default size = 2 2023-10-16T20:35:55.108 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.108 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow primary affinity = true 2023-10-16T20:35:55.108 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow pg remap = true 2023-10-16T20:35:55.108 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on legacy crush tunables = false 2023-10-16T20:35:55.109 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on crush straw calc version zero = false 2023-10-16T20:35:55.109 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on no sortbitwise = false 2023-10-16T20:35:55.109 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on osd down out interval zero = false 2023-10-16T20:35:55.109 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on too few osds = false 2023-10-16T20:35:55.109 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-16T20:35:55.110 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-16T20:35:55.110 INFO:teuthology.orchestra.run.smithi191.stdout: mon_allow_pool_size_one = true 2023-10-16T20:35:55.110 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.110 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-16T20:35:55.110 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.111 INFO:teuthology.orchestra.run.smithi191.stdout: osd default data pool replay window = 5 2023-10-16T20:35:55.111 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.111 INFO:teuthology.orchestra.run.smithi191.stdout: mon allow pool delete = true 2023-10-16T20:35:55.111 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.111 INFO:teuthology.orchestra.run.smithi191.stdout: mon cluster log file level = debug 2023-10-16T20:35:55.111 INFO:teuthology.orchestra.run.smithi191.stdout: debug asserts on shutdown = true 2023-10-16T20:35:55.112 INFO:teuthology.orchestra.run.smithi191.stdout: mon health detail to clog = false 2023-10-16T20:35:55.112 INFO:teuthology.orchestra.run.smithi191.stdout: mon host = "172.21.15.88,172.21.15.191,[v2:172.21.15.191:3301,v1:172.21.15.191:6790]" 2023-10-16T20:35:55.112 INFO:teuthology.orchestra.run.smithi191.stdout: fsid = 8cf7342f-a394-4b4c-ac35-9931ac9d9e57 2023-10-16T20:35:55.112 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.112 INFO:teuthology.orchestra.run.smithi191.stdout:[osd] 2023-10-16T20:35:55.112 INFO:teuthology.orchestra.run.smithi191.stdout: osd journal size = 100 2023-10-16T20:35:55.113 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.113 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub load threshold = 5.0 2023-10-16T20:35:55.113 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub max interval = 600 2023-10-16T20:35:55.113 INFO:teuthology.orchestra.run.smithi191.stdout: osd mclock profile = high_recovery_ops 2023-10-16T20:35:55.113 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.113 INFO:teuthology.orchestra.run.smithi191.stdout: osd recover clone overlap = true 2023-10-16T20:35:55.114 INFO:teuthology.orchestra.run.smithi191.stdout: osd recovery max chunk = 1048576 2023-10-16T20:35:55.114 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.114 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug shutdown = true 2023-10-16T20:35:55.114 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug op order = true 2023-10-16T20:35:55.114 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify stray on activate = true 2023-10-16T20:35:55.115 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.115 INFO:teuthology.orchestra.run.smithi191.stdout: osd open classes on start = true 2023-10-16T20:35:55.115 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug pg log writeout = true 2023-10-16T20:35:55.115 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.115 INFO:teuthology.orchestra.run.smithi191.stdout: osd deep scrub update digest min age = 30 2023-10-16T20:35:55.116 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.116 INFO:teuthology.orchestra.run.smithi191.stdout: osd map max advance = 10 2023-10-16T20:35:55.116 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.116 INFO:teuthology.orchestra.run.smithi191.stdout: journal zero on create = true 2023-10-16T20:35:55.116 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.116 INFO:teuthology.orchestra.run.smithi191.stdout: filestore ondisk finisher threads = 3 2023-10-16T20:35:55.117 INFO:teuthology.orchestra.run.smithi191.stdout: filestore apply finisher threads = 3 2023-10-16T20:35:55.117 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.117 INFO:teuthology.orchestra.run.smithi191.stdout: bdev debug aio = true 2023-10-16T20:35:55.117 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug misdirected ops = true 2023-10-16T20:35:55.117 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore block size = 96636764160 2023-10-16T20:35:55.118 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore fsck on mount = True 2023-10-16T20:35:55.118 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluefs = 20 2023-10-16T20:35:55.118 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluestore = 20 2023-10-16T20:35:55.118 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-10-16T20:35:55.118 INFO:teuthology.orchestra.run.smithi191.stdout: debug osd = 20 2023-10-16T20:35:55.119 INFO:teuthology.orchestra.run.smithi191.stdout: debug rocksdb = 10 2023-10-16T20:35:55.119 INFO:teuthology.orchestra.run.smithi191.stdout: mon client directed command retry = 5 2023-10-16T20:35:55.119 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd backfillfull_ratio = 0.85 2023-10-16T20:35:55.119 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd full ratio = 0.9 2023-10-16T20:35:55.119 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd nearfull ratio = 0.8 2023-10-16T20:35:55.120 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay max = 1 2023-10-16T20:35:55.120 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay probability = 0.005 2023-10-16T20:35:55.120 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay type = client mds 2023-10-16T20:35:55.120 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject socket failures = 2500 2023-10-16T20:35:55.120 INFO:teuthology.orchestra.run.smithi191.stdout: osd blocked scrub grace period = 3600 2023-10-16T20:35:55.120 INFO:teuthology.orchestra.run.smithi191.stdout: osd failsafe full ratio = 0.95 2023-10-16T20:35:55.121 INFO:teuthology.orchestra.run.smithi191.stdout: osd max markdown count = 1000 2023-10-16T20:35:55.121 INFO:teuthology.orchestra.run.smithi191.stdout: osd objectstore = bluestore 2023-10-16T20:35:55.121 INFO:teuthology.orchestra.run.smithi191.stdout: osd op complaint time = 180 2023-10-16T20:35:55.121 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.121 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr] 2023-10-16T20:35:55.122 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-10-16T20:35:55.122 INFO:teuthology.orchestra.run.smithi191.stdout: debug mgr = 20 2023-10-16T20:35:55.122 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2023-10-16T20:35:55.122 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2023-10-16T20:35:55.122 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2023-10-16T20:35:55.123 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2023-10-16T20:35:55.123 INFO:teuthology.orchestra.run.smithi191.stdout: mgr/telemetry/nag = false 2023-10-16T20:35:55.123 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.123 INFO:teuthology.orchestra.run.smithi191.stdout:[mon] 2023-10-16T20:35:55.123 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-10-16T20:35:55.123 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2023-10-16T20:35:55.124 INFO:teuthology.orchestra.run.smithi191.stdout: debug paxos = 20 2023-10-16T20:35:55.124 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2023-10-16T20:35:55.124 INFO:teuthology.orchestra.run.smithi191.stdout: mon data avail warn = 5 2023-10-16T20:35:55.124 INFO:teuthology.orchestra.run.smithi191.stdout: mon mgr mkfs grace = 240 2023-10-16T20:35:55.124 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2023-10-16T20:35:55.125 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd reporter subtree level = osd 2023-10-16T20:35:55.125 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd prime pg temp = true 2023-10-16T20:35:55.125 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2023-10-16T20:35:55.125 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.125 INFO:teuthology.orchestra.run.smithi191.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-16T20:35:55.126 INFO:teuthology.orchestra.run.smithi191.stdout: auth mon ticket ttl = 660 # 11m 2023-10-16T20:35:55.126 INFO:teuthology.orchestra.run.smithi191.stdout: auth service ticket ttl = 240 # 4m 2023-10-16T20:35:55.126 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.126 INFO:teuthology.orchestra.run.smithi191.stdout: # don't complain about insecure global_id in the test suite 2023-10-16T20:35:55.126 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-16T20:35:55.126 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-16T20:35:55.127 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.127 INFO:teuthology.orchestra.run.smithi191.stdout: # 1m isn't quite enough 2023-10-16T20:35:55.127 INFO:teuthology.orchestra.run.smithi191.stdout: mon_down_mkfs_grace = 2m 2023-10-16T20:35:55.127 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.127 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_filestore_osds = false 2023-10-16T20:35:55.128 INFO:teuthology.orchestra.run.smithi191.stdout: mon op complaint time = 120 2023-10-16T20:35:55.128 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-10-16T20:35:55.128 INFO:teuthology.orchestra.run.smithi191.stdout:[client] 2023-10-16T20:35:55.128 INFO:teuthology.orchestra.run.smithi191.stdout: rgw cache enabled = true 2023-10-16T20:35:55.128 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable ops log = true 2023-10-16T20:35:55.129 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable usage log = true 2023-10-16T20:35:55.129 INFO:teuthology.orchestra.run.smithi191.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-16T20:35:55.129 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-16T20:35:55.129 INFO:teuthology.orchestra.run.smithi191.stdout: client mount timeout = 600 2023-10-16T20:35:55.129 INFO:teuthology.orchestra.run.smithi191.stdout: client_shutdown_timeout = 120 2023-10-16T20:35:55.129 INFO:teuthology.orchestra.run.smithi191.stdout: debug client = 20 2023-10-16T20:35:55.130 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-10-16T20:35:55.130 INFO:teuthology.orchestra.run.smithi191.stdout: rados mon op timeout = 900 2023-10-16T20:35:55.130 INFO:teuthology.orchestra.run.smithi191.stdout: rados osd op timeout = 900 2023-10-16T20:35:55.130 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.a] 2023-10-16T20:35:55.130 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.b] 2023-10-16T20:35:55.131 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.c] 2023-10-16T20:35:55.131 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.a] 2023-10-16T20:35:55.131 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.c] 2023-10-16T20:35:55.131 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.e] 2023-10-16T20:35:55.131 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.b] 2023-10-16T20:35:55.132 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.d] 2023-10-16T20:35:55.132 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.f] 2023-10-16T20:35:55.132 INFO:teuthology.orchestra.run.smithi191.stdout:[mds] 2023-10-16T20:35:55.132 INFO:teuthology.orchestra.run.smithi191.stdout: debug mds = 20 2023-10-16T20:35:55.132 INFO:teuthology.orchestra.run.smithi191.stdout: debug mds balancer = 20 2023-10-16T20:35:55.132 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-10-16T20:35:55.133 INFO:teuthology.orchestra.run.smithi191.stdout: mds bal fragment size max = 10000 2023-10-16T20:35:55.133 INFO:teuthology.orchestra.run.smithi191.stdout: mds bal merge size = 5 2023-10-16T20:35:55.133 INFO:teuthology.orchestra.run.smithi191.stdout: mds bal split bits = 3 2023-10-16T20:35:55.133 INFO:teuthology.orchestra.run.smithi191.stdout: mds bal split size = 100 2023-10-16T20:35:55.133 INFO:teuthology.orchestra.run.smithi191.stdout: mds debug frag = True 2023-10-16T20:35:55.133 INFO:teuthology.orchestra.run.smithi191.stdout: mds debug scatterstat = True 2023-10-16T20:35:55.134 INFO:teuthology.orchestra.run.smithi191.stdout: mds oft prefetch dirfrags = False 2023-10-16T20:35:55.134 INFO:teuthology.orchestra.run.smithi191.stdout: mds op complaint time = 180 2023-10-16T20:35:55.134 INFO:teuthology.orchestra.run.smithi191.stdout: mds verify scatter = True 2023-10-16T20:35:55.134 INFO:teuthology.orchestra.run.smithi191.stdout: mds_dir_prefetch = False 2023-10-16T20:35:55.134 INFO:teuthology.orchestra.run.smithi191.stdout: mds_sleep_rank_change = 5000000.0 2023-10-16T20:35:55.135 INFO:teuthology.orchestra.run.smithi191.stdout: mon client directed command retry = 5 2023-10-16T20:35:55.135 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay max = 1 2023-10-16T20:35:55.135 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay probability = 0.005 2023-10-16T20:35:55.135 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject delay type = client mds osd 2023-10-16T20:35:55.135 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject socket failures = 2500 2023-10-16T20:35:55.135 INFO:teuthology.orchestra.run.smithi191.stdout: osd op complaint time = 180 2023-10-16T20:35:55.136 INFO:teuthology.orchestra.run.smithi191.stdout: rados mon op timeout = 900 2023-10-16T20:35:55.136 INFO:teuthology.orchestra.run.smithi191.stdout: rados osd op timeout = 900 2023-10-16T20:35:55.136 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-16T20:35:55.137 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-16T20:35:55.202 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-16T20:35:55.202 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:35:55.203 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-16T20:35:55.212 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:35:55.212 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-16T20:35:55.261 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi088.front.sepia.ceph.com 2023-10-16T20:35:55.263 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:35:55.263 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-16T20:35:55.263 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-16T20:35:55.338 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:35:55.339 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-16T20:35:55.389 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi191.front.sepia.ceph.com 2023-10-16T20:35:55.390 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:35:55.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-16T20:35:55.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-16T20:35:55.427 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:35:55.428 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-16T20:35:55.479 INFO:tasks.ceph:Setting up mon nodes... 2023-10-16T20:35:55.480 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-16T20:35:55.480 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-16T20:35:55.536 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-16T20:35:55.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-16T20:35:55.601 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-16T20:35:55.616 INFO:tasks.ceph:Setting up mds nodes... 2023-10-16T20:35:55.616 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-16T20:35:55.672 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-16T20:35:55.676 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-16T20:35:55.696 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-16T20:35:55.795 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-16T20:35:55.800 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-16T20:35:55.821 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2023-10-16T20:35:55.916 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2023-10-16T20:35:55.920 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2023-10-16T20:35:55.940 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-16T20:35:56.002 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-16T20:35:56.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-16T20:35:56.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-16T20:35:56.127 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-16T20:35:56.131 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-16T20:35:56.151 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2023-10-16T20:35:56.249 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2023-10-16T20:35:56.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2023-10-16T20:35:56.274 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-16T20:35:56.275 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-16T20:35:56.318 INFO:teuthology.orchestra.run.smithi088.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-16T20:35:56.336 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-16T20:35:56.336 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-16T20:35:56.337 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-16T20:35:56.397 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T20:35:56.398 INFO:tasks.ceph:role: osd.0 2023-10-16T20:35:56.398 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-16T20:35:56.398 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-16T20:35:56.469 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T20:35:56.469 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T20:35:56.470 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T20:35:56.470 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T20:35:56.470 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T20:35:56.471 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-10-16T20:35:56.471 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T20:35:56.471 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T20:35:56.471 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T20:35:56.472 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T20:35:56.940 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2023-10-16T20:35:56.943 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-10-16T20:35:56.944 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-16T20:35:57.083 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-16T20:35:57.101 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: /sbin/restorecon: command not found 2023-10-16T20:35:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T20:35:57.102 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-16T20:35:57.167 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T20:35:57.167 INFO:tasks.ceph:role: osd.1 2023-10-16T20:35:57.168 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-16T20:35:57.168 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-16T20:35:57.238 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T20:35:57.238 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T20:35:57.238 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T20:35:57.239 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T20:35:57.239 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T20:35:57.239 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-10-16T20:35:57.239 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T20:35:57.239 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T20:35:57.240 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T20:35:57.240 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T20:35:57.680 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2023-10-16T20:35:57.683 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-10-16T20:35:57.684 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-16T20:35:57.781 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-16T20:35:57.798 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: /sbin/restorecon: command not found 2023-10-16T20:35:57.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T20:35:57.800 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-16T20:35:57.865 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T20:35:57.866 INFO:tasks.ceph:role: osd.2 2023-10-16T20:35:57.866 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-16T20:35:57.866 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-16T20:35:57.938 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T20:35:57.938 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T20:35:57.938 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T20:35:57.938 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T20:35:57.939 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T20:35:57.939 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-10-16T20:35:57.939 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T20:35:57.939 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T20:35:57.940 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T20:35:57.940 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T20:35:58.400 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2023-10-16T20:35:58.402 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-10-16T20:35:58.403 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-16T20:35:58.549 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-16T20:35:58.566 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: /sbin/restorecon: command not found 2023-10-16T20:35:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T20:35:58.568 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-16T20:35:58.634 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-16T20:35:58.635 INFO:tasks.ceph:role: osd.3 2023-10-16T20:35:58.635 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-16T20:35:58.635 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-16T20:35:58.705 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T20:35:58.706 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T20:35:58.706 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T20:35:58.706 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T20:35:58.706 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T20:35:58.706 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2023-10-16T20:35:58.707 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T20:35:58.707 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T20:35:58.707 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T20:35:58.707 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T20:35:59.189 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2023-10-16T20:35:59.192 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2023-10-16T20:35:59.193 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-16T20:35:59.335 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-16T20:35:59.353 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: /sbin/restorecon: command not found 2023-10-16T20:35:59.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T20:35:59.355 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T20:35:59.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:35:59.474+0000 7f0e59824640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-16T20:35:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:35:59.474+0000 7f0e59824640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-16T20:35:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:35:59.474+0000 7f0e59824640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-16T20:35:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:35:59.474+0000 7f0e59824640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-16T20:35:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:35:59.474+0000 7f0e59824640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-16T20:35:59.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:35:59.478+0000 7f0e59824640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-16T20:36:02.466 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-16T20:36:02.487 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T20:36:02.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:02.578+0000 7f8d3cd75640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-16T20:36:02.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:02.578+0000 7f8d3cd75640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-16T20:36:02.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:02.578+0000 7f8d3cd75640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-16T20:36:02.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:02.578+0000 7f8d3cd75640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-16T20:36:02.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:02.578+0000 7f8d3cd75640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-16T20:36:02.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:02.578+0000 7f8d3cd75640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-16T20:36:05.577 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-16T20:36:05.599 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T20:36:05.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:05.690+0000 7f69bf375640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-16T20:36:05.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:05.690+0000 7f69bf375640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-16T20:36:05.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:05.690+0000 7f69bf375640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-16T20:36:05.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:05.690+0000 7f69bf375640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-16T20:36:05.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:05.690+0000 7f69bf375640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-16T20:36:05.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:05.694+0000 7f69bf375640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-16T20:36:08.868 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-16T20:36:08.888 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T20:36:08.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:08.978+0000 7f8d0302f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-16T20:36:08.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:08.978+0000 7f8d0302f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-16T20:36:08.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:08.978+0000 7f8d0302f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-16T20:36:08.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:08.978+0000 7f8d0302f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-16T20:36:08.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:08.978+0000 7f8d0302f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-16T20:36:08.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:08.978+0000 7f8d0302f640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-16T20:36:11.985 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-16T20:36:12.006 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-16T20:36:12.029 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T20:36:12.029 INFO:tasks.ceph:role: osd.4 2023-10-16T20:36:12.030 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com 2023-10-16T20:36:12.030 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-16T20:36:12.104 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T20:36:12.105 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T20:36:12.105 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T20:36:12.105 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T20:36:12.105 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T20:36:12.105 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-10-16T20:36:12.106 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T20:36:12.106 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T20:36:12.106 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T20:36:12.106 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T20:36:12.583 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-10-16T20:36:12.586 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-10-16T20:36:12.586 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-16T20:36:12.696 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-16T20:36:12.714 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: /sbin/restorecon: command not found 2023-10-16T20:36:12.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T20:36:12.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-16T20:36:12.779 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T20:36:12.779 INFO:tasks.ceph:role: osd.5 2023-10-16T20:36:12.780 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com 2023-10-16T20:36:12.780 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-16T20:36:12.851 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T20:36:12.851 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T20:36:12.852 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T20:36:12.852 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T20:36:12.852 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T20:36:12.852 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-10-16T20:36:12.853 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T20:36:12.853 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T20:36:12.853 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T20:36:12.853 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T20:36:13.339 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-10-16T20:36:13.342 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-10-16T20:36:13.342 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-16T20:36:13.471 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-16T20:36:13.489 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: /sbin/restorecon: command not found 2023-10-16T20:36:13.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T20:36:13.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-16T20:36:13.551 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T20:36:13.552 INFO:tasks.ceph:role: osd.6 2023-10-16T20:36:13.552 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com 2023-10-16T20:36:13.552 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-16T20:36:13.623 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T20:36:13.623 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T20:36:13.623 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T20:36:13.624 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T20:36:13.624 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T20:36:13.624 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-10-16T20:36:13.624 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T20:36:13.624 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T20:36:13.624 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T20:36:13.625 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T20:36:14.105 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-10-16T20:36:14.108 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-10-16T20:36:14.109 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-16T20:36:14.253 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-16T20:36:14.272 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: /sbin/restorecon: command not found 2023-10-16T20:36:14.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T20:36:14.274 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-16T20:36:14.336 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-16T20:36:14.336 INFO:tasks.ceph:role: osd.7 2023-10-16T20:36:14.337 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com 2023-10-16T20:36:14.337 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-16T20:36:14.411 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T20:36:14.411 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T20:36:14.411 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T20:36:14.411 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-16T20:36:14.412 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T20:36:14.412 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-10-16T20:36:14.412 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T20:36:14.412 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-16T20:36:14.412 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T20:36:14.413 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T20:36:14.863 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-10-16T20:36:14.867 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-10-16T20:36:14.867 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-16T20:36:14.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-16T20:36:14.989 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: /sbin/restorecon: command not found 2023-10-16T20:36:14.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T20:36:14.991 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T20:36:15.135 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:15.133+0000 7f40432a9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-16T20:36:15.139 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:15.137+0000 7f40432a9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-16T20:36:15.139 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:15.137+0000 7f40432a9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-16T20:36:15.140 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:15.137+0000 7f40432a9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-16T20:36:15.140 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:15.137+0000 7f40432a9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-16T20:36:15.144 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:15.141+0000 7f40432a9640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-16T20:36:18.133 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-16T20:36:18.154 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T20:36:18.255 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:18.253+0000 7f4ca3184640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-16T20:36:18.256 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:18.253+0000 7f4ca3184640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-16T20:36:18.256 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:18.253+0000 7f4ca3184640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-16T20:36:18.256 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:18.253+0000 7f4ca3184640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-16T20:36:18.257 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:18.253+0000 7f4ca3184640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-16T20:36:18.257 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:18.253+0000 7f4ca3184640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-16T20:36:21.242 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-16T20:36:21.263 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T20:36:21.360 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:21.357+0000 7fba2c2c1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-16T20:36:21.361 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:21.357+0000 7fba2c2c1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-16T20:36:21.361 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:21.357+0000 7fba2c2c1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-16T20:36:21.361 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:21.357+0000 7fba2c2c1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-16T20:36:21.362 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:21.357+0000 7fba2c2c1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-16T20:36:21.362 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:21.357+0000 7fba2c2c1640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-16T20:36:23.151 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:36:23.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:36:24.360 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-16T20:36:24.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T20:36:24.480 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:24.477+0000 7f994847e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-16T20:36:24.481 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:24.477+0000 7f994847e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-16T20:36:24.481 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:24.477+0000 7f994847e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-16T20:36:24.481 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:24.477+0000 7f994847e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-16T20:36:24.481 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:24.477+0000 7f994847e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-16T20:36:24.482 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:24.477+0000 7f994847e640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-16T20:36:27.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-16T20:36:27.499 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-16T20:36:27.501 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:36:27.501 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-16T20:36:27.524 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:36:27.525 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-16T20:36:27.586 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:36:27.587 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-16T20:36:27.650 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:36:27.651 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2023-10-16T20:36:27.714 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:36:27.715 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-16T20:36:27.778 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:36:27.778 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-16T20:36:27.842 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:36:27.842 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-16T20:36:27.906 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:36:27.906 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-16T20:36:27.974 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:36:27.974 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-16T20:36:27.996 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:36:27.996 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-16T20:36:28.059 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:36:28.059 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-16T20:36:28.123 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:36:28.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2023-10-16T20:36:28.186 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:36:28.187 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-16T20:36:28.251 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:36:28.251 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-16T20:36:28.315 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:36:28.315 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-16T20:36:28.379 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:36:28.379 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-16T20:36:28.442 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:36:28.443 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-16T20:36:28.452 INFO:tasks.ceph:Adding keys to all mons... 2023-10-16T20:36:28.453 DEBUG:teuthology.orchestra.run.smithi088:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-16T20:36:28.496 DEBUG:teuthology.orchestra.run.smithi191:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-16T20:36:28.512 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.y] 2023-10-16T20:36:28.512 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCrni1lSkCrHxAAqNCs8VQgcJWqldSRVtZiAA== 2023-10-16T20:36:28.513 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.a] 2023-10-16T20:36:28.513 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCrni1lmzXJJxAAcO7cvtifwPm8rBF5d2kciQ== 2023-10-16T20:36:28.513 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.c] 2023-10-16T20:36:28.513 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCrni1lDGwlLxAAhYQNSSeKhkenBEpPBCiLVA== 2023-10-16T20:36:28.513 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.e] 2023-10-16T20:36:28.513 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCrni1legpTNhAAEoHRVhZJ0slbj9QHi6iDBQ== 2023-10-16T20:36:28.514 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.0] 2023-10-16T20:36:28.514 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCvni1lGJJ5HBAAqDzBqhELxl8AIpvwQH/WJw== 2023-10-16T20:36:28.514 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.1] 2023-10-16T20:36:28.514 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCyni1lqgS3IhAAkIlUp1SwRgvI3ATtdBF6xA== 2023-10-16T20:36:28.515 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.2] 2023-10-16T20:36:28.515 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQC1ni1lvRN0KRAAf2i84K7EXSBnKP37nLOnDQ== 2023-10-16T20:36:28.515 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.3] 2023-10-16T20:36:28.515 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQC4ni1l1d6SOhAACRmNxzbfJnZX394YVxbTCg== 2023-10-16T20:36:28.515 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.x] 2023-10-16T20:36:28.516 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCrni1l6Y/WIxAAKTA4Xk1PzURzGgURhIU5Ww== 2023-10-16T20:36:28.516 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.b] 2023-10-16T20:36:28.516 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCsni1lLY0pABAAKhSbYrdbKddV6PkAhWADPg== 2023-10-16T20:36:28.516 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.d] 2023-10-16T20:36:28.516 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCsni1l3pmTBxAAFHROUzP6GmVVg0oizka53g== 2023-10-16T20:36:28.517 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.f] 2023-10-16T20:36:28.517 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCsni1lKw3iDhAAMtRjt5qDqnpoXP//Y4uRiw== 2023-10-16T20:36:28.517 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.4] 2023-10-16T20:36:28.517 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQC/ni1lwNIaCBAAdo2sFFIkskFjolDWRV30wQ== 2023-10-16T20:36:28.518 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.5] 2023-10-16T20:36:28.518 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDCni1lxSc9DxAA2BDuiGqX6l0D2PmGejE67Q== 2023-10-16T20:36:28.518 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.6] 2023-10-16T20:36:28.518 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDFni1lu455FRAAXOUJ6j2GhweAVLVnXov/KQ== 2023-10-16T20:36:28.518 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.7] 2023-10-16T20:36:28.519 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDIni1l6vSmHBAAuEdEhH5U9BeGwdskNRWrQg== 2023-10-16T20:36:28.519 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2023-10-16T20:36:28.519 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQCsni1lxXK4EhAAZ9RKoptvWqujxxV+EMQoWA== 2023-10-16T20:36:28.521 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.y] 2023-10-16T20:36:28.521 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCrni1lSkCrHxAAqNCs8VQgcJWqldSRVtZiAA== 2023-10-16T20:36:28.521 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.a] 2023-10-16T20:36:28.521 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCrni1lmzXJJxAAcO7cvtifwPm8rBF5d2kciQ== 2023-10-16T20:36:28.522 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.c] 2023-10-16T20:36:28.522 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCrni1lDGwlLxAAhYQNSSeKhkenBEpPBCiLVA== 2023-10-16T20:36:28.522 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.e] 2023-10-16T20:36:28.522 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCrni1legpTNhAAEoHRVhZJ0slbj9QHi6iDBQ== 2023-10-16T20:36:28.522 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.0] 2023-10-16T20:36:28.523 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCvni1lGJJ5HBAAqDzBqhELxl8AIpvwQH/WJw== 2023-10-16T20:36:28.523 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.1] 2023-10-16T20:36:28.523 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCyni1lqgS3IhAAkIlUp1SwRgvI3ATtdBF6xA== 2023-10-16T20:36:28.523 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.2] 2023-10-16T20:36:28.524 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQC1ni1lvRN0KRAAf2i84K7EXSBnKP37nLOnDQ== 2023-10-16T20:36:28.525 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.3] 2023-10-16T20:36:28.525 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQC4ni1l1d6SOhAACRmNxzbfJnZX394YVxbTCg== 2023-10-16T20:36:28.525 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.x] 2023-10-16T20:36:28.525 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCrni1l6Y/WIxAAKTA4Xk1PzURzGgURhIU5Ww== 2023-10-16T20:36:28.526 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.b] 2023-10-16T20:36:28.526 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCsni1lLY0pABAAKhSbYrdbKddV6PkAhWADPg== 2023-10-16T20:36:28.526 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.d] 2023-10-16T20:36:28.526 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCsni1l3pmTBxAAFHROUzP6GmVVg0oizka53g== 2023-10-16T20:36:28.526 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.f] 2023-10-16T20:36:28.527 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCsni1lKw3iDhAAMtRjt5qDqnpoXP//Y4uRiw== 2023-10-16T20:36:28.527 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.4] 2023-10-16T20:36:28.527 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQC/ni1lwNIaCBAAdo2sFFIkskFjolDWRV30wQ== 2023-10-16T20:36:28.527 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.5] 2023-10-16T20:36:28.527 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDCni1lxSc9DxAA2BDuiGqX6l0D2PmGejE67Q== 2023-10-16T20:36:28.528 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.6] 2023-10-16T20:36:28.528 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDFni1lu455FRAAXOUJ6j2GhweAVLVnXov/KQ== 2023-10-16T20:36:28.528 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.7] 2023-10-16T20:36:28.528 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDIni1l6vSmHBAAuEdEhH5U9BeGwdskNRWrQg== 2023-10-16T20:36:28.528 INFO:teuthology.orchestra.run.smithi191.stdout:[client.0] 2023-10-16T20:36:28.528 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCsni1lxXK4EhAAZ9RKoptvWqujxxV+EMQoWA== 2023-10-16T20:36:28.529 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T20:36:28.564 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T20:36:28.619 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T20:36:28.623 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T20:36:28.681 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T20:36:28.686 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T20:36:28.742 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T20:36:28.746 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T20:36:28.803 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T20:36:28.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T20:36:28.861 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T20:36:28.865 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T20:36:28.919 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T20:36:28.922 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T20:36:28.977 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T20:36:28.981 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T20:36:29.036 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T20:36:29.040 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T20:36:29.095 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T20:36:29.099 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T20:36:29.156 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T20:36:29.160 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T20:36:29.218 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T20:36:29.222 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-16T20:36:29.280 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T20:36:29.284 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T20:36:29.338 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T20:36:29.342 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T20:36:29.396 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T20:36:29.400 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T20:36:29.454 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T20:36:29.458 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T20:36:29.513 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T20:36:29.518 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T20:36:29.575 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-16T20:36:29.575 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-16T20:36:29.595 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-16T20:36:29.711 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-16T20:36:29.731 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-16T20:36:29.752 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-16T20:36:29.872 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-16T20:36:29.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-16T20:36:29.958 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-16T20:36:30.070 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-16T20:36:30.091 DEBUG:teuthology.orchestra.run.smithi088:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-16T20:36:30.094 DEBUG:teuthology.orchestra.run.smithi191:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-16T20:36:30.142 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-16T20:36:30.143 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-16T20:36:30.143 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-16T20:36:30.147 INFO:tasks.ceph.mon.a:Started 2023-10-16T20:36:30.148 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-16T20:36:30.148 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-16T20:36:30.190 INFO:tasks.ceph.mon.b:Started 2023-10-16T20:36:30.190 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-16T20:36:30.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-16T20:36:30.194 INFO:tasks.ceph.mon.c:Started 2023-10-16T20:36:30.194 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-16T20:36:30.195 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-16T20:36:30.195 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-16T20:36:30.199 INFO:tasks.ceph.mgr.y:Started 2023-10-16T20:36:30.199 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-16T20:36:30.200 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-16T20:36:30.203 INFO:tasks.ceph.mgr.x:Started 2023-10-16T20:36:30.204 INFO:tasks.ceph:Setting crush tunables to default 2023-10-16T20:36:30.204 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd crush tunables default 2023-10-16T20:36:30.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.334+0000 7f1db473e640 1 Processor -- start 2023-10-16T20:36:30.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.334+0000 7f1db473e640 1 -- start start 2023-10-16T20:36:30.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.334+0000 7f1db473e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dac159490 0x7f1dac0c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:30.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.334+0000 7f1db473e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac159d90 con 0x7f1dac0c5930 2023-10-16T20:36:30.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.334+0000 7f1db473e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac15a070 con 0x7f1dac153780 2023-10-16T20:36:30.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.334+0000 7f1db473e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac15a350 con 0x7f1dac159490 2023-10-16T20:36:30.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.334+0000 7f1db1cd9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dac159490 0x7f1dac0c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:30.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.334+0000 7f1db24da640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f1dac153780 0x7f1dac0c9be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:48592/0 (socket says 172.21.15.88:48592) 2023-10-16T20:36:30.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.334+0000 7f1db24da640 1 -- 172.21.15.88:0/789276455 learned_addr learned my addr 172.21.15.88:0/789276455 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:36:30.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.334+0000 7f1db2cdb640 1 --1- 172.21.15.88:0/789276455 >> v1:172.21.15.88:6789/0 conn(0x7f1dac0c5930 0x7f1dac0ced80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34768/0 (socket says 172.21.15.88:34768) 2023-10-16T20:36:30.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.334+0000 7f1db1cd9640 1 -- 172.21.15.88:0/789276455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dac159490 msgr2=0x7f1dac0c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-16T20:36:30.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.334+0000 7f1db1cd9640 1 -- 172.21.15.88:0/789276455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dac159490 msgr2=0x7f1dac0c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T20:36:30.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.334+0000 7f1db1cd9640 1 --2- 172.21.15.88:0/789276455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dac159490 0x7f1dac0c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T20:36:30.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.334+0000 7f1db1cd9640 1 --2- 172.21.15.88:0/789276455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dac159490 0x7f1dac0c53f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T20:36:30.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.490+0000 7f1db14d8640 1 -- 172.21.15.88:0/789276455 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948155339 0 0) 0x7f1dac159d90 con 0x7f1dac0c5930 2023-10-16T20:36:30.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.490+0000 7f1db14d8640 1 -- 172.21.15.88:0/789276455 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d98003530 con 0x7f1dac0c5930 2023-10-16T20:36:30.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db14d8640 1 -- 172.21.15.88:0/789276455 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2202921457 0 0) 0x7f1d98003530 con 0x7f1dac0c5930 2023-10-16T20:36:30.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db14d8640 1 -- 172.21.15.88:0/789276455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dac159490 msgr2=0x7f1dac0c53f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T20:36:30.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db14d8640 1 --2- 172.21.15.88:0/789276455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dac159490 0x7f1dac0c53f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:30.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db14d8640 1 -- 172.21.15.88:0/789276455 >> v1:172.21.15.191:6789/0 conn(0x7f1dac153780 legacy=0x7f1dac0c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:30.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db14d8640 1 -- 172.21.15.88:0/789276455 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac0c4910 con 0x7f1dac0c5930 2023-10-16T20:36:30.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db14d8640 1 -- 172.21.15.88:0/789276455 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f1da4001930 con 0x7f1dac0c5930 2023-10-16T20:36:30.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db14d8640 1 -- 172.21.15.88:0/789276455 >> v1:172.21.15.88:6789/0 conn(0x7f1dac0c5930 legacy=0x7f1dac0ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:30.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db14d8640 1 --2- 172.21.15.88:0/789276455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d980039b0 0x7f1d98003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:30.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db14d8640 1 --2- 172.21.15.88:0/789276455 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d980047d0 0x7f1d98004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:30.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db14d8640 1 --2- 172.21.15.88:0/789276455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dac159490 0x7f1d98005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:30.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db14d8640 1 -- 172.21.15.88:0/789276455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac15a350 con 0x7f1d980039b0 2023-10-16T20:36:30.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db14d8640 1 -- 172.21.15.88:0/789276455 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9800bbb0 con 0x7f1d980047d0 2023-10-16T20:36:30.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db14d8640 1 -- 172.21.15.88:0/789276455 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9800bd70 con 0x7f1dac159490 2023-10-16T20:36:30.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db24da640 1 --2- 172.21.15.88:0/789276455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d980039b0 0x7f1d98003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:30.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db1cd9640 1 --2- 172.21.15.88:0/789276455 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d980047d0 0x7f1d98004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:30.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db2cdb640 1 --2- 172.21.15.88:0/789276455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dac159490 0x7f1d98005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:30.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db24da640 1 -- 172.21.15.88:0/789276455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dac159490 msgr2=0x7f1d98005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:30.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db24da640 1 --2- 172.21.15.88:0/789276455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dac159490 0x7f1d98005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:30.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db24da640 1 -- 172.21.15.88:0/789276455 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d980047d0 msgr2=0x7f1d98004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:30.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db24da640 1 --2- 172.21.15.88:0/789276455 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d980047d0 0x7f1d98004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:30.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db24da640 1 -- 172.21.15.88:0/789276455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d98003530 con 0x7f1d980039b0 2023-10-16T20:36:30.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.494+0000 7f1db24da640 1 --2- 172.21.15.88:0/789276455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d980039b0 0x7f1d98003dd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1da8001890 tx=0x7f1da801c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8649bd3e8d5e553 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:30.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.498+0000 7f1db14d8640 1 -- 172.21.15.88:0/789276455 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da8004b00 con 0x7f1d980039b0 2023-10-16T20:36:30.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.498+0000 7f1db14d8640 1 -- 172.21.15.88:0/789276455 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da80a7070 con 0x7f1d980039b0 2023-10-16T20:36:30.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.498+0000 7f1db14d8640 1 -- 172.21.15.88:0/789276455 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da8007070 con 0x7f1d980039b0 2023-10-16T20:36:30.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.498+0000 7f1db473e640 1 -- 172.21.15.88:0/789276455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d980039b0 msgr2=0x7f1d98003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:30.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.498+0000 7f1db473e640 1 --2- 172.21.15.88:0/789276455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d980039b0 0x7f1d98003dd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1da8001890 tx=0x7f1da801c4c0 comp rx=0 tx=0).stop 2023-10-16T20:36:30.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.498+0000 7f1db2cdb640 1 -- 172.21.15.88:0/789276455 reap_dead start 2023-10-16T20:36:30.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.498+0000 7f1db473e640 1 -- 172.21.15.88:0/789276455 shutdown_connections 2023-10-16T20:36:30.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.498+0000 7f1db473e640 1 -- 172.21.15.88:0/789276455 >> 172.21.15.88:0/789276455 conn(0x7f1dac06bcb0 msgr2=0x7f1dac06c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:30.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.498+0000 7f1db473e640 1 -- 172.21.15.88:0/789276455 shutdown_connections 2023-10-16T20:36:30.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.498+0000 7f1db473e640 1 -- 172.21.15.88:0/789276455 wait complete. 2023-10-16T20:36:30.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db473e640 1 Processor -- start 2023-10-16T20:36:30.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db473e640 1 -- start start 2023-10-16T20:36:30.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db473e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d980039b0 0x7f1dac148d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:30.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db473e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dac1485c0 0x7f1dac144ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:30.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db473e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dac1509b0 0x7f1dac0c7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:30.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db473e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac147ca0 con 0x7f1dac1485c0 2023-10-16T20:36:30.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db473e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac148a60 con 0x7f1d980039b0 2023-10-16T20:36:30.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db473e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac1ef9a0 con 0x7f1dac1509b0 2023-10-16T20:36:30.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db1cd9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dac1485c0 0x7f1dac144ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:30.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db24da640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d980039b0 0x7f1dac148d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:30.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db1cd9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dac1485c0 0x7f1dac144ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56820/0 (socket says 172.21.15.88:56820) 2023-10-16T20:36:30.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db1cd9640 1 -- 172.21.15.88:0/3097012813 learned_addr learned my addr 172.21.15.88:0/3097012813 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:30.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db2cdb640 1 --2- 172.21.15.88:0/3097012813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dac1509b0 0x7f1dac0c7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:30.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db1cd9640 1 -- 172.21.15.88:0/3097012813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dac1509b0 msgr2=0x7f1dac0c7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:30.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db1cd9640 1 --2- 172.21.15.88:0/3097012813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dac1509b0 0x7f1dac0c7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:30.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db1cd9640 1 -- 172.21.15.88:0/3097012813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d980039b0 msgr2=0x7f1dac148d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:30.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db1cd9640 1 --2- 172.21.15.88:0/3097012813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d980039b0 0x7f1dac148d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:30.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db1cd9640 1 -- 172.21.15.88:0/3097012813 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac1efc80 con 0x7f1dac1485c0 2023-10-16T20:36:30.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db1cd9640 1 --2- 172.21.15.88:0/3097012813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dac1485c0 0x7f1dac144ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c011600 tx=0x7f1d9c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:30.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3097012813 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c004230 con 0x7f1dac1485c0 2023-10-16T20:36:30.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3097012813 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d9c004390 con 0x7f1dac1485c0 2023-10-16T20:36:30.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1db473e640 1 -- 172.21.15.88:0/3097012813 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac147300 con 0x7f1dac1485c0 2023-10-16T20:36:30.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.502+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3097012813 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c00f940 con 0x7f1dac1485c0 2023-10-16T20:36:30.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.506+0000 7f1db473e640 1 -- 172.21.15.88:0/3097012813 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac1478c0 con 0x7f1dac1485c0 2023-10-16T20:36:30.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.506+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3097012813 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1d9c00faa0 con 0x7f1dac1485c0 2023-10-16T20:36:30.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.506+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3097012813 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1d9c015480 con 0x7f1dac1485c0 2023-10-16T20:36:30.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.506+0000 7f1db473e640 1 -- 172.21.15.88:0/3097012813 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dac147a80 con 0x7f1dac1485c0 2023-10-16T20:36:30.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.522+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3097012813 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f1d9c022060 con 0x7f1dac1485c0 2023-10-16T20:36:30.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.582+0000 7f1db473e640 1 -- 172.21.15.88:0/3097012813 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f1dac147de0 con 0x7f1dac1485c0 2023-10-16T20:36:30.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.586+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3097012813 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1d9c02b020 con 0x7f1dac1485c0 2023-10-16T20:36:30.595 INFO:teuthology.orchestra.run.smithi088.stderr:adjusted tunables profile to default 2023-10-16T20:36:30.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.586+0000 7f1db473e640 1 -- 172.21.15.88:0/3097012813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dac1485c0 msgr2=0x7f1dac144ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:30.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.586+0000 7f1db473e640 1 --2- 172.21.15.88:0/3097012813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dac1485c0 0x7f1dac144ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c011600 tx=0x7f1d9c011ac0 comp rx=0 tx=0).stop 2023-10-16T20:36:30.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.586+0000 7f1db473e640 1 -- 172.21.15.88:0/3097012813 shutdown_connections 2023-10-16T20:36:30.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.586+0000 7f1db473e640 1 --2- 172.21.15.88:0/3097012813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1dac1509b0 0x7f1dac0c7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:30.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.586+0000 7f1db473e640 1 --2- 172.21.15.88:0/3097012813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dac1485c0 0x7f1dac144ce0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:30.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.586+0000 7f1db473e640 1 --2- 172.21.15.88:0/3097012813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d980039b0 0x7f1dac148d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:30.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.586+0000 7f1db473e640 1 -- 172.21.15.88:0/3097012813 >> 172.21.15.88:0/3097012813 conn(0x7f1dac06bcb0 msgr2=0x7f1dac0c3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:30.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.586+0000 7f1db473e640 1 -- 172.21.15.88:0/3097012813 shutdown_connections 2023-10-16T20:36:30.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:30.586+0000 7f1db473e640 1 -- 172.21.15.88:0/3097012813 wait complete. 2023-10-16T20:36:30.619 INFO:tasks.ceph:check_enable_crimson: False 2023-10-16T20:36:30.620 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-16T20:36:30.620 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:36:30.620 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-16T20:36:30.641 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:36:30.641 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-16T20:36:30.705 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:36:30.705 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-16T20:36:30.706 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:30.701+0000 7f0f19b7e180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T20:36:30.734 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:30.726+0000 7fd450c13180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T20:36:30.755 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:36:30.756 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-16T20:36:30.818 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:36:30.819 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-16T20:36:30.840 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:36:30.840 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-16T20:36:30.902 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:36:30.903 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-16T20:36:30.966 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-10-16T20:36:30.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-16T20:36:31.031 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 8f96fb46-c190-4aae-85f6-7fc0bf73047b 0 2023-10-16T20:36:31.210 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.205+0000 7f54298ea640 1 Processor -- start 2023-10-16T20:36:31.210 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.205+0000 7f54298ea640 1 -- start start 2023-10-16T20:36:31.212 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.209+0000 7f54298ea640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5424153780 0x7f54240c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.212 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.209+0000 7f54298ea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f54241598c0 0x7f54240ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.212 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.209+0000 7f54298ea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5424159d90 con 0x7f5424159490 2023-10-16T20:36:31.213 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.209+0000 7f54298ea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f542415a070 con 0x7f54241598c0 2023-10-16T20:36:31.213 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.209+0000 7f54298ea640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f542415a350 con 0x7f5424153780 2023-10-16T20:36:31.213 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.209+0000 7f5422ffd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5424153780 0x7f54240c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.213 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.209+0000 7f5422ffd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5424153780 0x7f54240c9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:33502/0 (socket says 172.21.15.191:33502) 2023-10-16T20:36:31.213 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.209+0000 7f5422ffd640 1 -- 172.21.15.191:0/1971276917 learned_addr learned my addr 172.21.15.191:0/1971276917 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-16T20:36:31.213 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.209+0000 7f54237fe640 1 --2- 172.21.15.191:0/1971276917 >> v2:172.21.15.88:3300/0 conn(0x7f54241598c0 0x7f54240ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.214 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.209+0000 7f5422ffd640 1 -- 172.21.15.191:0/1971276917 >> v2:172.21.15.88:3300/0 conn(0x7f54241598c0 msgr2=0x7f54240ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:31.214 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.209+0000 7f5422ffd640 1 --2- 172.21.15.191:0/1971276917 >> v2:172.21.15.88:3300/0 conn(0x7f54241598c0 0x7f54240ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.214 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.209+0000 7f5422ffd640 1 -- 172.21.15.191:0/1971276917 >> v1:172.21.15.88:6789/0 conn(0x7f5424159490 legacy=0x7f54240c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:31.214 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.209+0000 7f5422ffd640 1 -- 172.21.15.191:0/1971276917 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54240c4910 con 0x7f5424153780 2023-10-16T20:36:31.215 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f5422ffd640 1 --2- 172.21.15.191:0/1971276917 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5424153780 0x7f54240c9be0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f54240c9900 tx=0x7f541401ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=924a1241b0c32d08 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:31.215 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54227fc640 1 -- 172.21.15.191:0/1971276917 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54140a89b0 con 0x7f5424153780 2023-10-16T20:36:31.216 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54227fc640 1 -- 172.21.15.191:0/1971276917 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54140a8b10 con 0x7f5424153780 2023-10-16T20:36:31.216 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54227fc640 1 -- 172.21.15.191:0/1971276917 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54140a8ea0 con 0x7f5424153780 2023-10-16T20:36:31.217 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54298ea640 1 -- 172.21.15.191:0/1971276917 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5424153780 msgr2=0x7f54240c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.217 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54298ea640 1 --2- 172.21.15.191:0/1971276917 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5424153780 0x7f54240c9be0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f54240c9900 tx=0x7f541401ecb0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.217 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54298ea640 1 -- 172.21.15.191:0/1971276917 shutdown_connections 2023-10-16T20:36:31.217 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54298ea640 1 --2- 172.21.15.191:0/1971276917 >> v2:172.21.15.88:3300/0 conn(0x7f54241598c0 0x7f54240ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.217 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54298ea640 1 --2- 172.21.15.191:0/1971276917 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5424153780 0x7f54240c9be0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.218 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54298ea640 1 -- 172.21.15.191:0/1971276917 >> 172.21.15.191:0/1971276917 conn(0x7f542406bcb0 msgr2=0x7f542406c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:31.218 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54298ea640 1 -- 172.21.15.191:0/1971276917 shutdown_connections 2023-10-16T20:36:31.218 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54298ea640 1 -- 172.21.15.191:0/1971276917 wait complete. 2023-10-16T20:36:31.220 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54298ea640 1 Processor -- start 2023-10-16T20:36:31.221 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54298ea640 1 -- start start 2023-10-16T20:36:31.221 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54298ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5424153780 0x7f54240b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.221 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54298ea640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5424159490 0x7f542415ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.221 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54298ea640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54241598c0 0x7f542415b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.221 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54298ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542415a860 con 0x7f5424153780 2023-10-16T20:36:31.222 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54298ea640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54241ef400 con 0x7f5424159490 2023-10-16T20:36:31.222 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.213+0000 7f54298ea640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54241ef930 con 0x7f54241598c0 2023-10-16T20:36:31.222 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f541bfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5424159490 0x7f542415ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.222 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f541bfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5424159490 0x7f542415ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:33508/0 (socket says 172.21.15.191:33508) 2023-10-16T20:36:31.222 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f5422ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5424153780 0x7f54240b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.223 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f541bfff640 1 -- 172.21.15.191:0/1968803616 learned_addr learned my addr 172.21.15.191:0/1968803616 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-16T20:36:31.223 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f54237fe640 1 --2- 172.21.15.191:0/1968803616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54241598c0 0x7f542415b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.223 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f5422ffd640 1 -- 172.21.15.191:0/1968803616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54241598c0 msgr2=0x7f542415b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.223 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f5422ffd640 1 --2- 172.21.15.191:0/1968803616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54241598c0 0x7f542415b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.223 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f5422ffd640 1 -- 172.21.15.191:0/1968803616 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5424159490 msgr2=0x7f542415ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.224 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f5422ffd640 1 --2- 172.21.15.191:0/1968803616 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5424159490 0x7f542415ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.224 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f5422ffd640 1 -- 172.21.15.191:0/1968803616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5414009480 con 0x7f5424153780 2023-10-16T20:36:31.224 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f541bfff640 1 --2- 172.21.15.191:0/1968803616 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5424159490 0x7f542415ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:36:31.224 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f5422ffd640 1 --2- 172.21.15.191:0/1968803616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5424153780 0x7f54240b4130 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f54140af000 tx=0x7f541401ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:31.224 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f5420ff9640 1 -- 172.21.15.191:0/1968803616 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5414003a30 con 0x7f5424153780 2023-10-16T20:36:31.225 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f5420ff9640 1 -- 172.21.15.191:0/1968803616 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5414003bd0 con 0x7f5424153780 2023-10-16T20:36:31.225 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f54298ea640 1 -- 172.21.15.191:0/1968803616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54240ccf40 con 0x7f5424153780 2023-10-16T20:36:31.225 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f5420ff9640 1 -- 172.21.15.191:0/1968803616 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54140a8970 con 0x7f5424153780 2023-10-16T20:36:31.225 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f54298ea640 1 -- 172.21.15.191:0/1968803616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54241ef650 con 0x7f5424153780 2023-10-16T20:36:31.225 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f5420ff9640 1 -- 172.21.15.191:0/1968803616 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f54140b7070 con 0x7f5424153780 2023-10-16T20:36:31.225 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f5420ff9640 1 -- 172.21.15.191:0/1968803616 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f54140b9760 con 0x7f5424153780 2023-10-16T20:36:31.226 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.217+0000 7f54298ea640 1 -- 172.21.15.191:0/1968803616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f542415aad0 con 0x7f5424153780 2023-10-16T20:36:31.226 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.221+0000 7f5420ff9640 1 -- 172.21.15.191:0/1968803616 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f54140b5070 con 0x7f5424153780 2023-10-16T20:36:31.253 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:31.249+0000 7f0f19b7e180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T20:36:31.275 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:31.266+0000 7fd450c13180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T20:36:31.289 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.285+0000 7f54298ea640 1 -- 172.21.15.191:0/1968803616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8f96fb46-c190-4aae-85f6-7fc0bf73047b", "id": 0} v 0) v1 -- 0x7f54240c4ea0 con 0x7f5424153780 2023-10-16T20:36:31.291 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.289+0000 7f5420ff9640 1 -- 172.21.15.191:0/1968803616 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8f96fb46-c190-4aae-85f6-7fc0bf73047b", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f54140b0830 con 0x7f5424153780 2023-10-16T20:36:31.291 INFO:teuthology.orchestra.run.smithi191.stdout:0 2023-10-16T20:36:31.292 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.289+0000 7f54298ea640 1 -- 172.21.15.191:0/1968803616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5424153780 msgr2=0x7f54240b4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.292 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.289+0000 7f54298ea640 1 --2- 172.21.15.191:0/1968803616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5424153780 0x7f54240b4130 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f54140af000 tx=0x7f541401ba50 comp rx=0 tx=0).stop 2023-10-16T20:36:31.293 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.289+0000 7f54298ea640 1 -- 172.21.15.191:0/1968803616 shutdown_connections 2023-10-16T20:36:31.293 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.289+0000 7f54298ea640 1 --2- 172.21.15.191:0/1968803616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54241598c0 0x7f542415b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.293 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.289+0000 7f54298ea640 1 --2- 172.21.15.191:0/1968803616 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5424159490 0x7f542415ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.293 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.289+0000 7f54298ea640 1 --2- 172.21.15.191:0/1968803616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5424153780 0x7f54240b4130 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.293 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.289+0000 7f54298ea640 1 -- 172.21.15.191:0/1968803616 >> 172.21.15.191:0/1968803616 conn(0x7f542406bcb0 msgr2=0x7f54240ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:31.294 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.289+0000 7f54298ea640 1 -- 172.21.15.191:0/1968803616 shutdown_connections 2023-10-16T20:36:31.294 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.289+0000 7f54298ea640 1 -- 172.21.15.191:0/1968803616 wait complete. 2023-10-16T20:36:31.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new f4a636df-8b38-4af2-a0a2-59dc457d22bc 1 2023-10-16T20:36:31.449 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.445+0000 7f52f3274640 1 Processor -- start 2023-10-16T20:36:31.449 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.445+0000 7f52f3274640 1 -- start start 2023-10-16T20:36:31.451 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.445+0000 7f52f3274640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f52ec0720b0 0x7f52ec15ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.451 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.445+0000 7f52f3274640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec0cb4a0 con 0x7f52ec0715c0 2023-10-16T20:36:31.451 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.445+0000 7f52f3274640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec1594b0 con 0x7f52ec0caf40 2023-10-16T20:36:31.451 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.445+0000 7f52f3274640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52ec159790 con 0x7f52ec0720b0 2023-10-16T20:36:31.451 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.445+0000 7f52f2a73640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f52ec0caf40 0x7f52ec163c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:53308/0 (socket says 172.21.15.191:53308) 2023-10-16T20:36:31.452 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.445+0000 7f52f2a73640 1 -- 172.21.15.191:0/3671296921 learned_addr learned my addr 172.21.15.191:0/3671296921 (peer_addr_for_me v1:172.21.15.191:0/0) 2023-10-16T20:36:31.452 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.449+0000 7f52f1a71640 1 --2- 172.21.15.191:0/3671296921 >> v2:172.21.15.88:3300/0 conn(0x7f52ec0720b0 0x7f52ec15ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.453 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.449+0000 7f52f1270640 1 -- 172.21.15.191:0/3671296921 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255865259 0 0) 0x7f52ec1594b0 con 0x7f52ec0caf40 2023-10-16T20:36:31.453 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.449+0000 7f52f1270640 1 -- 172.21.15.191:0/3671296921 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d8003530 con 0x7f52ec0caf40 2023-10-16T20:36:31.453 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.449+0000 7f52f1270640 1 -- 172.21.15.191:0/3671296921 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249521969 0 0) 0x7f52ec0cb4a0 con 0x7f52ec0715c0 2023-10-16T20:36:31.453 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.449+0000 7f52f1270640 1 -- 172.21.15.191:0/3671296921 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52ec1594b0 con 0x7f52ec0715c0 2023-10-16T20:36:31.454 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.449+0000 7f52f1a71640 1 -- 172.21.15.191:0/3671296921 >> v1:172.21.15.191:6789/0 conn(0x7f52ec0caf40 legacy=0x7f52ec163c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.454 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.449+0000 7f52f1a71640 1 -- 172.21.15.191:0/3671296921 >> v1:172.21.15.88:6789/0 conn(0x7f52ec0715c0 legacy=0x7f52ec0719a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.454 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.449+0000 7f52f1a71640 1 -- 172.21.15.191:0/3671296921 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec159a70 con 0x7f52ec0720b0 2023-10-16T20:36:31.455 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.449+0000 7f52f1a71640 1 --2- 172.21.15.191:0/3671296921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec0720b0 0x7f52ec15ca00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f52dc0097a0 tx=0x7f52dc01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f01697c0f64497b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:31.455 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.449+0000 7f52f1270640 1 -- 172.21.15.191:0/3671296921 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52dc0a7070 con 0x7f52ec0720b0 2023-10-16T20:36:31.455 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.449+0000 7f52f1270640 1 -- 172.21.15.191:0/3671296921 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52dc0ab430 con 0x7f52ec0720b0 2023-10-16T20:36:31.455 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.449+0000 7f52f1270640 1 -- 172.21.15.191:0/3671296921 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52dc0ab7c0 con 0x7f52ec0720b0 2023-10-16T20:36:31.456 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f3274640 1 -- 172.21.15.191:0/3671296921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec0720b0 msgr2=0x7f52ec15ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.457 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f3274640 1 --2- 172.21.15.191:0/3671296921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec0720b0 0x7f52ec15ca00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f52dc0097a0 tx=0x7f52dc01c430 comp rx=0 tx=0).stop 2023-10-16T20:36:31.457 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f3274640 1 -- 172.21.15.191:0/3671296921 shutdown_connections 2023-10-16T20:36:31.457 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f3274640 1 --2- 172.21.15.191:0/3671296921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec0720b0 0x7f52ec15ca00 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.457 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f3274640 1 -- 172.21.15.191:0/3671296921 >> 172.21.15.191:0/3671296921 conn(0x7f52ec0cec20 msgr2=0x7f52ec0cf030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:31.457 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f3274640 1 -- 172.21.15.191:0/3671296921 shutdown_connections 2023-10-16T20:36:31.458 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f3274640 1 -- 172.21.15.191:0/3671296921 wait complete. 2023-10-16T20:36:31.458 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f0f19b7e180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T20:36:31.460 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f3274640 1 Processor -- start 2023-10-16T20:36:31.460 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f3274640 1 -- start start 2023-10-16T20:36:31.461 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f3274640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52ec0715c0 0x7f52ec1f4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.461 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f3274640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52ec0720b0 0x7f52ec1f3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.461 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f3274640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec0caf40 0x7f52ec1f3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.461 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f3274640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec0c4320 con 0x7f52ec0caf40 2023-10-16T20:36:31.461 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f3274640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec1f2b50 con 0x7f52ec0715c0 2023-10-16T20:36:31.462 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f3274640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec1f58c0 con 0x7f52ec0720b0 2023-10-16T20:36:31.462 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f2272640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52ec0715c0 0x7f52ec1f4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.462 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f2272640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52ec0715c0 0x7f52ec1f4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:33514/0 (socket says 172.21.15.191:33514) 2023-10-16T20:36:31.462 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f2272640 1 -- 172.21.15.191:0/3858377586 learned_addr learned my addr 172.21.15.191:0/3858377586 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-16T20:36:31.462 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f1a71640 1 --2- 172.21.15.191:0/3858377586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52ec0720b0 0x7f52ec1f3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.463 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.453+0000 7f52f2a73640 1 --2- 172.21.15.191:0/3858377586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec0caf40 0x7f52ec1f3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.463 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.457+0000 7f52f2272640 1 -- 172.21.15.191:0/3858377586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52ec0720b0 msgr2=0x7f52ec1f3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.463 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.457+0000 7f52f2272640 1 --2- 172.21.15.191:0/3858377586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52ec0720b0 0x7f52ec1f3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.463 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.457+0000 7f52f2272640 1 -- 172.21.15.191:0/3858377586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec0caf40 msgr2=0x7f52ec1f3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.463 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.457+0000 7f52f2272640 1 --2- 172.21.15.191:0/3858377586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec0caf40 0x7f52ec1f3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.464 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.457+0000 7f52f2272640 1 -- 172.21.15.191:0/3858377586 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52dc0093b0 con 0x7f52ec0715c0 2023-10-16T20:36:31.464 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.457+0000 7f52f2272640 1 --2- 172.21.15.191:0/3858377586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52ec0715c0 0x7f52ec1f4e40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f52e800c4c0 tx=0x7f52e800c980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:31.464 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.457+0000 7f52e2ffd640 1 -- 172.21.15.191:0/3858377586 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e80052d0 con 0x7f52ec0715c0 2023-10-16T20:36:31.464 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.457+0000 7f52e2ffd640 1 -- 172.21.15.191:0/3858377586 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e8005430 con 0x7f52ec0715c0 2023-10-16T20:36:31.465 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.457+0000 7f52f3274640 1 -- 172.21.15.191:0/3858377586 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ec0c4500 con 0x7f52ec0715c0 2023-10-16T20:36:31.465 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.457+0000 7f52e2ffd640 1 -- 172.21.15.191:0/3858377586 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e8012870 con 0x7f52ec0715c0 2023-10-16T20:36:31.465 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.457+0000 7f52f3274640 1 -- 172.21.15.191:0/3858377586 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ec06af10 con 0x7f52ec0715c0 2023-10-16T20:36:31.465 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.457+0000 7f52e2ffd640 1 -- 172.21.15.191:0/3858377586 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f52e8012cd0 con 0x7f52ec0715c0 2023-10-16T20:36:31.465 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.457+0000 7f52e2ffd640 1 -- 172.21.15.191:0/3858377586 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f52e801b490 con 0x7f52ec0715c0 2023-10-16T20:36:31.466 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.457+0000 7f52f3274640 1 -- 172.21.15.191:0/3858377586 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52ec153720 con 0x7f52ec0715c0 2023-10-16T20:36:31.471 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.469+0000 7f52e2ffd640 1 -- 172.21.15.191:0/3858377586 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f52e8003650 con 0x7f52ec0715c0 2023-10-16T20:36:31.495 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:31.486+0000 7fd450c13180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T20:36:31.538 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.533+0000 7f52f3274640 1 -- 172.21.15.191:0/3858377586 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f4a636df-8b38-4af2-a0a2-59dc457d22bc", "id": 1} v 0) v1 -- 0x7f52ec1f2cc0 con 0x7f52ec0715c0 2023-10-16T20:36:31.545 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.541+0000 7f52e2ffd640 1 -- 172.21.15.191:0/3858377586 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f4a636df-8b38-4af2-a0a2-59dc457d22bc", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f52e80038c0 con 0x7f52ec0715c0 2023-10-16T20:36:31.545 INFO:teuthology.orchestra.run.smithi191.stdout:1 2023-10-16T20:36:31.546 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.541+0000 7f52f3274640 1 -- 172.21.15.191:0/3858377586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52ec0715c0 msgr2=0x7f52ec1f4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.547 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.541+0000 7f52f3274640 1 --2- 172.21.15.191:0/3858377586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52ec0715c0 0x7f52ec1f4e40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f52e800c4c0 tx=0x7f52e800c980 comp rx=0 tx=0).stop 2023-10-16T20:36:31.547 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.541+0000 7f52f3274640 1 -- 172.21.15.191:0/3858377586 shutdown_connections 2023-10-16T20:36:31.547 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.541+0000 7f52f3274640 1 --2- 172.21.15.191:0/3858377586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec0caf40 0x7f52ec1f3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.547 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.541+0000 7f52f3274640 1 --2- 172.21.15.191:0/3858377586 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f52ec0720b0 0x7f52ec1f3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.548 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.541+0000 7f52f3274640 1 --2- 172.21.15.191:0/3858377586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52ec0715c0 0x7f52ec1f4e40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.548 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.541+0000 7f52f3274640 1 -- 172.21.15.191:0/3858377586 >> 172.21.15.191:0/3858377586 conn(0x7f52ec0cec20 msgr2=0x7f52ec0719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:31.548 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.541+0000 7f52f3274640 1 -- 172.21.15.191:0/3858377586 shutdown_connections 2023-10-16T20:36:31.548 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.541+0000 7f52f3274640 1 -- 172.21.15.191:0/3858377586 wait complete. 2023-10-16T20:36:31.564 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 38af5d72-deaa-45f1-86b8-920f2698202c 2 2023-10-16T20:36:31.699 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.693+0000 7f2d5359e640 1 Processor -- start 2023-10-16T20:36:31.699 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.697+0000 7f2d5359e640 1 -- start start 2023-10-16T20:36:31.701 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.697+0000 7f2d5359e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2d540715c0 0x7f2d540719a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.701 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.697+0000 7f2d5359e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2d5415d110 0x7f2d54071ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.701 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.697+0000 7f2d5359e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d540cb4a0 con 0x7f2d540caf40 2023-10-16T20:36:31.702 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.697+0000 7f2d5359e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d541594b0 con 0x7f2d540715c0 2023-10-16T20:36:31.702 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.697+0000 7f2d5359e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d54159790 con 0x7f2d5415d110 2023-10-16T20:36:31.702 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.697+0000 7f2d52d9d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2d5415d110 0x7f2d54071ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.702 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.697+0000 7f2d52d9d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2d5415d110 0x7f2d54071ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:33516/0 (socket says 172.21.15.191:33516) 2023-10-16T20:36:31.702 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.697+0000 7f2d51d9b640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f2d540caf40 0x7f2d5415ca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:53316/0 (socket says 172.21.15.191:53316) 2023-10-16T20:36:31.703 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.697+0000 7f2d52d9d640 1 -- 172.21.15.191:0/2447743995 learned_addr learned my addr 172.21.15.191:0/2447743995 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-16T20:36:31.703 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.697+0000 7f2d5259c640 1 --2- 172.21.15.191:0/2447743995 >> v2:172.21.15.88:3300/0 conn(0x7f2d540715c0 0x7f2d540719a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.703 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.697+0000 7f2d52d9d640 1 -- 172.21.15.191:0/2447743995 >> v2:172.21.15.88:3300/0 conn(0x7f2d540715c0 msgr2=0x7f2d540719a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:31.703 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.697+0000 7f2d52d9d640 1 --2- 172.21.15.191:0/2447743995 >> v2:172.21.15.88:3300/0 conn(0x7f2d540715c0 0x7f2d540719a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.704 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.697+0000 7f2d52d9d640 1 -- 172.21.15.191:0/2447743995 >> v1:172.21.15.191:6789/0 conn(0x7f2d540caf40 legacy=0x7f2d5415ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.704 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.697+0000 7f2d52d9d640 1 -- 172.21.15.191:0/2447743995 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d54159a70 con 0x7f2d5415d110 2023-10-16T20:36:31.704 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.697+0000 7f2d5159a640 1 -- 172.21.15.191:0/2447743995 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390830199 0 0) 0x7f2d540cb4a0 con 0x7f2d540caf40 2023-10-16T20:36:31.705 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.701+0000 7f2d52d9d640 1 --2- 172.21.15.191:0/2447743995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d5415d110 0x7f2d54071ee0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2d54153b80 tx=0x7f2d4c01c9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd6c162750981188 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:31.705 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.701+0000 7f2d5159a640 1 -- 172.21.15.191:0/2447743995 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c0a5780 con 0x7f2d5415d110 2023-10-16T20:36:31.705 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.701+0000 7f2d5159a640 1 -- 172.21.15.191:0/2447743995 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4c0a58e0 con 0x7f2d5415d110 2023-10-16T20:36:31.706 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.701+0000 7f2d5159a640 1 -- 172.21.15.191:0/2447743995 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c0a5c70 con 0x7f2d5415d110 2023-10-16T20:36:31.706 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.701+0000 7f2d5359e640 1 -- 172.21.15.191:0/2447743995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d5415d110 msgr2=0x7f2d54071ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.706 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.701+0000 7f2d5359e640 1 --2- 172.21.15.191:0/2447743995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d5415d110 0x7f2d54071ee0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2d54153b80 tx=0x7f2d4c01c9d0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.706 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.701+0000 7f2d5359e640 1 -- 172.21.15.191:0/2447743995 shutdown_connections 2023-10-16T20:36:31.707 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.701+0000 7f2d5359e640 1 --2- 172.21.15.191:0/2447743995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d5415d110 0x7f2d54071ee0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.707 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.701+0000 7f2d5359e640 1 --2- 172.21.15.191:0/2447743995 >> v2:172.21.15.88:3300/0 conn(0x7f2d540715c0 0x7f2d540719a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.707 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.701+0000 7f2d5359e640 1 -- 172.21.15.191:0/2447743995 >> 172.21.15.191:0/2447743995 conn(0x7f2d540cec20 msgr2=0x7f2d540cf030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:31.707 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.701+0000 7f2d5359e640 1 -- 172.21.15.191:0/2447743995 shutdown_connections 2023-10-16T20:36:31.707 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.701+0000 7f2d5359e640 1 -- 172.21.15.191:0/2447743995 wait complete. 2023-10-16T20:36:31.709 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5359e640 1 Processor -- start 2023-10-16T20:36:31.710 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5359e640 1 -- start start 2023-10-16T20:36:31.710 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5359e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d540715c0 0x7f2d541f1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.710 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5359e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d540caf40 0x7f2d54157f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.710 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5359e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d5415d110 0x7f2d5415b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.710 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5359e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d541f1ca0 con 0x7f2d5415d110 2023-10-16T20:36:31.711 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5359e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d54145e90 con 0x7f2d540caf40 2023-10-16T20:36:31.711 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5359e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d540c8ec0 con 0x7f2d540715c0 2023-10-16T20:36:31.711 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d51d9b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d540caf40 0x7f2d54157f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.711 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5259c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d540715c0 0x7f2d541f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.711 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5259c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d540715c0 0x7f2d541f1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:35142/0 (socket says 172.21.15.191:35142) 2023-10-16T20:36:31.712 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5259c640 1 -- 172.21.15.191:0/986341116 learned_addr learned my addr 172.21.15.191:0/986341116 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-16T20:36:31.712 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d52d9d640 1 --2- 172.21.15.191:0/986341116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d5415d110 0x7f2d5415b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.712 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5259c640 1 -- 172.21.15.191:0/986341116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d540caf40 msgr2=0x7f2d54157f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.712 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5259c640 1 --2- 172.21.15.191:0/986341116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d540caf40 0x7f2d54157f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.712 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5259c640 1 -- 172.21.15.191:0/986341116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d5415d110 msgr2=0x7f2d5415b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.713 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5259c640 1 --2- 172.21.15.191:0/986341116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d5415d110 0x7f2d5415b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.713 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5259c640 1 -- 172.21.15.191:0/986341116 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d4c009b30 con 0x7f2d540715c0 2023-10-16T20:36:31.713 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d51d9b640 1 --2- 172.21.15.191:0/986341116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d540caf40 0x7f2d54157f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:36:31.713 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5259c640 1 --2- 172.21.15.191:0/986341116 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d540715c0 0x7f2d541f1760 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400e930 tx=0x7f2d4400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:31.713 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d42ffd640 1 -- 172.21.15.191:0/986341116 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d44009690 con 0x7f2d540715c0 2023-10-16T20:36:31.714 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d42ffd640 1 -- 172.21.15.191:0/986341116 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4400ca00 con 0x7f2d540715c0 2023-10-16T20:36:31.714 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d42ffd640 1 -- 172.21.15.191:0/986341116 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4400cd70 con 0x7f2d540715c0 2023-10-16T20:36:31.714 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5359e640 1 -- 172.21.15.191:0/986341116 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d54145bb0 con 0x7f2d540715c0 2023-10-16T20:36:31.714 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5359e640 1 -- 172.21.15.191:0/986341116 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d541f5810 con 0x7f2d540715c0 2023-10-16T20:36:31.714 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d42ffd640 1 -- 172.21.15.191:0/986341116 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2d44013770 con 0x7f2d540715c0 2023-10-16T20:36:31.715 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d42ffd640 1 -- 172.21.15.191:0/986341116 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f2d44013e40 con 0x7f2d540715c0 2023-10-16T20:36:31.715 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.705+0000 7f2d5359e640 1 -- 172.21.15.191:0/986341116 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5404e9c0 con 0x7f2d540715c0 2023-10-16T20:36:31.715 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.709+0000 7f2d42ffd640 1 -- 172.21.15.191:0/986341116 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f2d44021060 con 0x7f2d540715c0 2023-10-16T20:36:31.779 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.777+0000 7f2d5359e640 1 -- 172.21.15.191:0/986341116 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd new", "uuid": "38af5d72-deaa-45f1-86b8-920f2698202c", "id": 2} v 0) v1 -- 0x7f2d540c9cd0 con 0x7f2d540715c0 2023-10-16T20:36:31.785 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.781+0000 7f2d42ffd640 1 -- 172.21.15.191:0/986341116 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "38af5d72-deaa-45f1-86b8-920f2698202c", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2d4402a020 con 0x7f2d540715c0 2023-10-16T20:36:31.786 INFO:teuthology.orchestra.run.smithi191.stdout:2 2023-10-16T20:36:31.787 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.781+0000 7f2d5359e640 1 -- 172.21.15.191:0/986341116 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d540715c0 msgr2=0x7f2d541f1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.787 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.781+0000 7f2d5359e640 1 --2- 172.21.15.191:0/986341116 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d540715c0 0x7f2d541f1760 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400e930 tx=0x7f2d4400edf0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.787 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.781+0000 7f2d5359e640 1 -- 172.21.15.191:0/986341116 shutdown_connections 2023-10-16T20:36:31.787 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.781+0000 7f2d5359e640 1 --2- 172.21.15.191:0/986341116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d5415d110 0x7f2d5415b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.788 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.781+0000 7f2d5359e640 1 --2- 172.21.15.191:0/986341116 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d540caf40 0x7f2d54157f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.788 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.781+0000 7f2d5359e640 1 --2- 172.21.15.191:0/986341116 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2d540715c0 0x7f2d541f1760 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.788 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.781+0000 7f2d5359e640 1 -- 172.21.15.191:0/986341116 >> 172.21.15.191:0/986341116 conn(0x7f2d540cec20 msgr2=0x7f2d5406a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:31.788 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.785+0000 7f2d5359e640 1 -- 172.21.15.191:0/986341116 shutdown_connections 2023-10-16T20:36:31.788 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.785+0000 7f2d5359e640 1 -- 172.21.15.191:0/986341116 wait complete. 2023-10-16T20:36:31.805 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 9cd29885-fd6d-4419-aae7-e9d4065bbc31 3 2023-10-16T20:36:31.942 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.937+0000 7f2de2e46640 1 Processor -- start 2023-10-16T20:36:31.943 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.937+0000 7f2de2e46640 1 -- start start 2023-10-16T20:36:31.944 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.941+0000 7f2de2e46640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2ddc159490 0x7f2ddc0c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.944 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.941+0000 7f2de2e46640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc159d90 con 0x7f2ddc0c5930 2023-10-16T20:36:31.944 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.941+0000 7f2de2e46640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc15a070 con 0x7f2ddc153780 2023-10-16T20:36:31.944 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.941+0000 7f2de2e46640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ddc15a350 con 0x7f2ddc159490 2023-10-16T20:36:31.945 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.941+0000 7f2ddbfff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2ddc159490 0x7f2ddc0c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.945 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.941+0000 7f2ddbfff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2ddc159490 0x7f2ddc0c53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:33542/0 (socket says 172.21.15.191:33542) 2023-10-16T20:36:31.945 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.941+0000 7f2ddbfff640 1 -- 172.21.15.191:0/3196914201 learned_addr learned my addr 172.21.15.191:0/3196914201 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-16T20:36:31.945 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.941+0000 7f2de0be2640 1 --1- 172.21.15.191:0/3196914201 >> v1:172.21.15.191:6789/0 conn(0x7f2ddc153780 0x7f2ddc0c9be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:53324/0 (socket says 172.21.15.191:53324) 2023-10-16T20:36:31.946 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.941+0000 7f2ddbfff640 1 -- 172.21.15.191:0/3196914201 >> v1:172.21.15.191:6789/0 conn(0x7f2ddc153780 legacy=0x7f2ddc0c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.946 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.941+0000 7f2ddbfff640 1 -- 172.21.15.191:0/3196914201 >> v1:172.21.15.88:6789/0 conn(0x7f2ddc0c5930 legacy=0x7f2ddc0ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.946 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.941+0000 7f2ddbfff640 1 -- 172.21.15.191:0/3196914201 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc0c4910 con 0x7f2ddc159490 2023-10-16T20:36:31.946 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.941+0000 7f2ddb7fe640 1 -- 172.21.15.191:0/3196914201 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837754503 0 0) 0x7f2ddc15a070 con 0x7f2ddc153780 2023-10-16T20:36:31.947 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.945+0000 7f2ddbfff640 1 --2- 172.21.15.191:0/3196914201 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ddc159490 0x7f2ddc0c53f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8009870 tx=0x7f2dc801ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bda957e87afa259 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:31.948 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.945+0000 7f2ddb7fe640 1 -- 172.21.15.191:0/3196914201 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc80a8a30 con 0x7f2ddc159490 2023-10-16T20:36:31.948 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.945+0000 7f2ddb7fe640 1 -- 172.21.15.191:0/3196914201 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc80a8b90 con 0x7f2ddc159490 2023-10-16T20:36:31.948 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.945+0000 7f2ddb7fe640 1 -- 172.21.15.191:0/3196914201 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc80b0440 con 0x7f2ddc159490 2023-10-16T20:36:31.949 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.945+0000 7f2de2e46640 1 -- 172.21.15.191:0/3196914201 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ddc159490 msgr2=0x7f2ddc0c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.949 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.945+0000 7f2de2e46640 1 --2- 172.21.15.191:0/3196914201 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ddc159490 0x7f2ddc0c53f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8009870 tx=0x7f2dc801ed30 comp rx=0 tx=0).stop 2023-10-16T20:36:31.949 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.945+0000 7f2de2e46640 1 -- 172.21.15.191:0/3196914201 shutdown_connections 2023-10-16T20:36:31.949 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.945+0000 7f2de2e46640 1 --2- 172.21.15.191:0/3196914201 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ddc159490 0x7f2ddc0c53f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.950 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.945+0000 7f2de2e46640 1 -- 172.21.15.191:0/3196914201 >> 172.21.15.191:0/3196914201 conn(0x7f2ddc06bcb0 msgr2=0x7f2ddc06c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:31.950 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.945+0000 7f2de2e46640 1 -- 172.21.15.191:0/3196914201 shutdown_connections 2023-10-16T20:36:31.950 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.945+0000 7f2de2e46640 1 -- 172.21.15.191:0/3196914201 wait complete. 2023-10-16T20:36:31.953 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.945+0000 7f2de2e46640 1 Processor -- start 2023-10-16T20:36:31.953 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.945+0000 7f2de2e46640 1 -- start start 2023-10-16T20:36:31.953 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2de2e46640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ddc0c5930 0x7f2ddc1ef8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.953 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2de2e46640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ddc153780 0x7f2ddc150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.953 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2de2e46640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ddc159490 0x7f2ddc14eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:31.954 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2de2e46640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc15a630 con 0x7f2ddc159490 2023-10-16T20:36:31.954 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2de2e46640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc15b2e0 con 0x7f2ddc153780 2023-10-16T20:36:31.954 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2de2e46640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc15a840 con 0x7f2ddc0c5930 2023-10-16T20:36:31.954 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2ddbfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ddc153780 0x7f2ddc150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.955 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2de0be2640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ddc0c5930 0x7f2ddc1ef8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.955 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2ddbfff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ddc153780 0x7f2ddc150d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:33556/0 (socket says 172.21.15.191:33556) 2023-10-16T20:36:31.955 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2ddbfff640 1 -- 172.21.15.191:0/1164800261 learned_addr learned my addr 172.21.15.191:0/1164800261 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-16T20:36:31.955 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2de13e3640 1 --2- 172.21.15.191:0/1164800261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ddc159490 0x7f2ddc14eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:31.955 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2ddbfff640 1 -- 172.21.15.191:0/1164800261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ddc0c5930 msgr2=0x7f2ddc1ef8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.956 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2ddbfff640 1 --2- 172.21.15.191:0/1164800261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ddc0c5930 0x7f2ddc1ef8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.956 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2ddbfff640 1 -- 172.21.15.191:0/1164800261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ddc159490 msgr2=0x7f2ddc14eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:31.956 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2ddbfff640 1 --2- 172.21.15.191:0/1164800261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ddc159490 0x7f2ddc14eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:31.956 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2ddbfff640 1 -- 172.21.15.191:0/1164800261 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc80094d0 con 0x7f2ddc153780 2023-10-16T20:36:31.956 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2de0be2640 1 --2- 172.21.15.191:0/1164800261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ddc0c5930 0x7f2ddc1ef8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:36:31.957 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2ddbfff640 1 --2- 172.21.15.191:0/1164800261 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ddc153780 0x7f2ddc150d80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2dc80b3040 tx=0x7f2dc801bad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:31.957 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2dd97fa640 1 -- 172.21.15.191:0/1164800261 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc80a8a30 con 0x7f2ddc153780 2023-10-16T20:36:31.957 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2dd97fa640 1 -- 172.21.15.191:0/1164800261 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dc80bb480 con 0x7f2ddc153780 2023-10-16T20:36:31.957 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2de2e46640 1 -- 172.21.15.191:0/1164800261 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ddc1ef0a0 con 0x7f2ddc153780 2023-10-16T20:36:31.957 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2dd97fa640 1 -- 172.21.15.191:0/1164800261 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc80bb850 con 0x7f2ddc153780 2023-10-16T20:36:31.958 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2de2e46640 1 -- 172.21.15.191:0/1164800261 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ddc0cbad0 con 0x7f2ddc153780 2023-10-16T20:36:31.958 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2dd97fa640 1 -- 172.21.15.191:0/1164800261 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2dc80bb9b0 con 0x7f2ddc153780 2023-10-16T20:36:31.958 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2dd97fa640 1 -- 172.21.15.191:0/1164800261 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f2dc80c5460 con 0x7f2ddc153780 2023-10-16T20:36:31.959 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.949+0000 7f2de2e46640 1 -- 172.21.15.191:0/1164800261 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ddc0c4ba0 con 0x7f2ddc153780 2023-10-16T20:36:31.959 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:31.953+0000 7f2dd97fa640 1 -- 172.21.15.191:0/1164800261 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f2dc80b5070 con 0x7f2ddc153780 2023-10-16T20:36:32.022 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.017+0000 7f2de2e46640 1 -- 172.21.15.191:0/1164800261 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9cd29885-fd6d-4419-aae7-e9d4065bbc31", "id": 3} v 0) v1 -- 0x7f2ddc0ca7d0 con 0x7f2ddc153780 2023-10-16T20:36:32.028 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.025+0000 7f2dd97fa640 1 -- 172.21.15.191:0/1164800261 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9cd29885-fd6d-4419-aae7-e9d4065bbc31", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2dc80c5910 con 0x7f2ddc153780 2023-10-16T20:36:32.029 INFO:teuthology.orchestra.run.smithi191.stdout:3 2023-10-16T20:36:32.030 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.025+0000 7f2de2e46640 1 -- 172.21.15.191:0/1164800261 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ddc153780 msgr2=0x7f2ddc150d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.030 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.025+0000 7f2de2e46640 1 --2- 172.21.15.191:0/1164800261 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ddc153780 0x7f2ddc150d80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2dc80b3040 tx=0x7f2dc801bad0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.030 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.025+0000 7f2de2e46640 1 -- 172.21.15.191:0/1164800261 shutdown_connections 2023-10-16T20:36:32.030 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.025+0000 7f2de2e46640 1 --2- 172.21.15.191:0/1164800261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ddc159490 0x7f2ddc14eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.031 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.025+0000 7f2de2e46640 1 --2- 172.21.15.191:0/1164800261 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ddc153780 0x7f2ddc150d80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.031 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.025+0000 7f2de2e46640 1 --2- 172.21.15.191:0/1164800261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2ddc0c5930 0x7f2ddc1ef8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.031 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.025+0000 7f2de2e46640 1 -- 172.21.15.191:0/1164800261 >> 172.21.15.191:0/1164800261 conn(0x7f2ddc06bcb0 msgr2=0x7f2ddc0ce540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:32.031 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.025+0000 7f2de2e46640 1 -- 172.21.15.191:0/1164800261 shutdown_connections 2023-10-16T20:36:32.031 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.025+0000 7f2de2e46640 1 -- 172.21.15.191:0/1164800261 wait complete. 2023-10-16T20:36:32.048 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 81025152-01c0-4624-be28-68cc5c853e79 4 2023-10-16T20:36:32.121 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:32.117+0000 7f0f19b7e180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T20:36:32.135 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:32.126+0000 7fd450c13180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T20:36:32.184 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.181+0000 7f958c13e640 1 Processor -- start 2023-10-16T20:36:32.185 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:32.181+0000 7f0f19b7e180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T20:36:32.185 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.181+0000 7f958c13e640 1 -- start start 2023-10-16T20:36:32.186 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.181+0000 7f958c13e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9584159490 0x7f95840c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.186 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.181+0000 7f958c13e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f95840c5930 0x7f95840ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.186 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.181+0000 7f958c13e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9584159d90 con 0x7f9584153780 2023-10-16T20:36:32.187 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.181+0000 7f958c13e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f958415a070 con 0x7f95840c5930 2023-10-16T20:36:32.187 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.181+0000 7f958c13e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f958415a350 con 0x7f9584159490 2023-10-16T20:36:32.188 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.181+0000 7f95896d9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9584159490 0x7f95840c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.188 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.181+0000 7f95896d9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9584159490 0x7f95840c53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:33558/0 (socket says 172.21.15.191:33558) 2023-10-16T20:36:32.188 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.181+0000 7f95896d9640 1 -- 172.21.15.191:0/431305094 learned_addr learned my addr 172.21.15.191:0/431305094 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-16T20:36:32.188 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.185+0000 7f95896d9640 1 -- 172.21.15.191:0/431305094 >> v2:172.21.15.88:3300/0 conn(0x7f95840c5930 msgr2=0x7f95840ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:32.188 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.185+0000 7f95896d9640 1 --2- 172.21.15.191:0/431305094 >> v2:172.21.15.88:3300/0 conn(0x7f95840c5930 0x7f95840ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.189 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.185+0000 7f95896d9640 1 -- 172.21.15.191:0/431305094 >> v1:172.21.15.191:6789/0 conn(0x7f9584153780 legacy=0x7f95840c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.189 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.185+0000 7f9588ed8640 1 -- 172.21.15.191:0/431305094 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190969556 0 0) 0x7f9584159d90 con 0x7f9584153780 2023-10-16T20:36:32.189 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.185+0000 7f95896d9640 1 -- 172.21.15.191:0/431305094 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95840c4910 con 0x7f9584159490 2023-10-16T20:36:32.190 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.185+0000 7f95896d9640 1 --2- 172.21.15.191:0/431305094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9584159490 0x7f95840c53f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f95840cfef0 tx=0x7f957401ecf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1294abdef2ed202c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:32.190 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.185+0000 7f9588ed8640 1 -- 172.21.15.191:0/431305094 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95740a89f0 con 0x7f9584159490 2023-10-16T20:36:32.190 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.185+0000 7f9588ed8640 1 -- 172.21.15.191:0/431305094 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95740a8b50 con 0x7f9584159490 2023-10-16T20:36:32.190 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.185+0000 7f9588ed8640 1 -- 172.21.15.191:0/431305094 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95740b0440 con 0x7f9584159490 2023-10-16T20:36:32.191 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.185+0000 7f958c13e640 1 -- 172.21.15.191:0/431305094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9584159490 msgr2=0x7f95840c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.191 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.185+0000 7f958c13e640 1 --2- 172.21.15.191:0/431305094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9584159490 0x7f95840c53f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f95840cfef0 tx=0x7f957401ecf0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.192 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.185+0000 7f958c13e640 1 -- 172.21.15.191:0/431305094 shutdown_connections 2023-10-16T20:36:32.192 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.185+0000 7f958c13e640 1 --2- 172.21.15.191:0/431305094 >> v2:172.21.15.88:3300/0 conn(0x7f95840c5930 0x7f95840ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.192 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.185+0000 7f958c13e640 1 --2- 172.21.15.191:0/431305094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9584159490 0x7f95840c53f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.192 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.185+0000 7f958c13e640 1 -- 172.21.15.191:0/431305094 >> 172.21.15.191:0/431305094 conn(0x7f958406bcb0 msgr2=0x7f958406c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:32.192 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f958c13e640 1 -- 172.21.15.191:0/431305094 shutdown_connections 2023-10-16T20:36:32.193 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f958c13e640 1 -- 172.21.15.191:0/431305094 wait complete. 2023-10-16T20:36:32.195 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f958c13e640 1 Processor -- start 2023-10-16T20:36:32.195 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f958c13e640 1 -- start start 2023-10-16T20:36:32.195 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f958c13e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95840c5930 0x7f9584142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.195 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f958c13e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9584153780 0x7f95840c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.196 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f958c13e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9584159490 0x7f95840c4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.196 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f958c13e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95840b18d0 con 0x7f9584153780 2023-10-16T20:36:32.196 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f958c13e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9584143020 con 0x7f95840c5930 2023-10-16T20:36:32.196 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f958c13e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9584144d10 con 0x7f9584159490 2023-10-16T20:36:32.196 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f9589eda640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95840c5930 0x7f9584142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.196 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f9589eda640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95840c5930 0x7f9584142ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:33572/0 (socket says 172.21.15.191:33572) 2023-10-16T20:36:32.197 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f9589eda640 1 -- 172.21.15.191:0/2061961991 learned_addr learned my addr 172.21.15.191:0/2061961991 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-16T20:36:32.197 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f958a6db640 1 --2- 172.21.15.191:0/2061961991 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9584159490 0x7f95840c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.197 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f95896d9640 1 --2- 172.21.15.191:0/2061961991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9584153780 0x7f95840c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.197 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f9589eda640 1 -- 172.21.15.191:0/2061961991 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9584159490 msgr2=0x7f95840c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.197 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f9589eda640 1 --2- 172.21.15.191:0/2061961991 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9584159490 0x7f95840c4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.198 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f9589eda640 1 -- 172.21.15.191:0/2061961991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9584153780 msgr2=0x7f95840c3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.198 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f9589eda640 1 --2- 172.21.15.191:0/2061961991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9584153780 0x7f95840c3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.198 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f9589eda640 1 -- 172.21.15.191:0/2061961991 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95740094d0 con 0x7f95840c5930 2023-10-16T20:36:32.198 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f9589eda640 1 --2- 172.21.15.191:0/2061961991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95840c5930 0x7f9584142ae0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f958000c520 tx=0x7f958000c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:32.198 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f957a7fc640 1 -- 172.21.15.191:0/2061961991 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9580005480 con 0x7f95840c5930 2023-10-16T20:36:32.199 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f957a7fc640 1 -- 172.21.15.191:0/2061961991 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9580003600 con 0x7f95840c5930 2023-10-16T20:36:32.199 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f957a7fc640 1 -- 172.21.15.191:0/2061961991 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95800039d0 con 0x7f95840c5930 2023-10-16T20:36:32.199 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.189+0000 7f958c13e640 1 -- 172.21.15.191:0/2061961991 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9584150e20 con 0x7f95840c5930 2023-10-16T20:36:32.199 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.193+0000 7f958c13e640 1 -- 172.21.15.191:0/2061961991 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9584144fb0 con 0x7f95840c5930 2023-10-16T20:36:32.199 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.193+0000 7f957a7fc640 1 -- 172.21.15.191:0/2061961991 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f958001c440 con 0x7f95840c5930 2023-10-16T20:36:32.199 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.193+0000 7f957a7fc640 1 -- 172.21.15.191:0/2061961991 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f9580012910 con 0x7f95840c5930 2023-10-16T20:36:32.200 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:32.190+0000 7fd450c13180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T20:36:32.200 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.193+0000 7f958c13e640 1 -- 172.21.15.191:0/2061961991 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f958414d5c0 con 0x7f95840c5930 2023-10-16T20:36:32.201 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.193+0000 7f957a7fc640 1 -- 172.21.15.191:0/2061961991 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f9580017070 con 0x7f95840c5930 2023-10-16T20:36:32.248 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:32.245+0000 7f0f19b7e180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T20:36:32.258 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:32.250+0000 7fd450c13180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T20:36:32.264 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.261+0000 7f958c13e640 1 -- 172.21.15.191:0/2061961991 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "81025152-01c0-4624-be28-68cc5c853e79", "id": 4} v 0) v1 -- 0x7f95840ca7d0 con 0x7f95840c5930 2023-10-16T20:36:32.270 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.265+0000 7f957a7fc640 1 -- 172.21.15.191:0/2061961991 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "81025152-01c0-4624-be28-68cc5c853e79", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9580012c90 con 0x7f95840c5930 2023-10-16T20:36:32.271 INFO:teuthology.orchestra.run.smithi191.stdout:4 2023-10-16T20:36:32.273 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.269+0000 7f958c13e640 1 -- 172.21.15.191:0/2061961991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95840c5930 msgr2=0x7f9584142ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.273 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.269+0000 7f958c13e640 1 --2- 172.21.15.191:0/2061961991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95840c5930 0x7f9584142ae0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f958000c520 tx=0x7f958000c9e0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.274 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.269+0000 7f958c13e640 1 -- 172.21.15.191:0/2061961991 shutdown_connections 2023-10-16T20:36:32.274 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.269+0000 7f958c13e640 1 --2- 172.21.15.191:0/2061961991 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9584159490 0x7f95840c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.274 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.269+0000 7f958c13e640 1 --2- 172.21.15.191:0/2061961991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9584153780 0x7f95840c3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.274 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.269+0000 7f958c13e640 1 --2- 172.21.15.191:0/2061961991 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f95840c5930 0x7f9584142ae0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.274 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.269+0000 7f958c13e640 1 -- 172.21.15.191:0/2061961991 >> 172.21.15.191:0/2061961991 conn(0x7f958406bcb0 msgr2=0x7f95841483d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:32.275 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.269+0000 7f958c13e640 1 -- 172.21.15.191:0/2061961991 shutdown_connections 2023-10-16T20:36:32.275 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.269+0000 7f958c13e640 1 -- 172.21.15.191:0/2061961991 wait complete. 2023-10-16T20:36:32.290 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 6e32466e-2f1f-4220-9f02-30708ad99202 5 2023-10-16T20:36:32.342 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:32.337+0000 7f0f19b7e180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T20:36:32.347 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:32.338+0000 7fd450c13180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T20:36:32.403 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:32.401+0000 7f0f19b7e180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T20:36:32.407 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:32.398+0000 7fd450c13180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T20:36:32.426 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.421+0000 7f7524232640 1 Processor -- start 2023-10-16T20:36:32.426 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.421+0000 7f7524232640 1 -- start start 2023-10-16T20:36:32.428 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.425+0000 7f7524232640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f751c153780 0x7f751c0c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.428 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.425+0000 7f7524232640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f751c159490 0x7f751c0c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.429 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.425+0000 7f7524232640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f751c159d90 con 0x7f751c0c5930 2023-10-16T20:36:32.429 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.425+0000 7f7524232640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f751c15a070 con 0x7f751c153780 2023-10-16T20:36:32.429 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.425+0000 7f7524232640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f751c15a350 con 0x7f751c159490 2023-10-16T20:36:32.429 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.425+0000 7f75217cd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f751c159490 0x7f751c0c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.429 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.425+0000 7f75217cd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f751c159490 0x7f751c0c53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:33586/0 (socket says 172.21.15.191:33586) 2023-10-16T20:36:32.430 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.425+0000 7f75227cf640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f751c0c5930 0x7f751c0ced80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:53344/0 (socket says 172.21.15.191:53344) 2023-10-16T20:36:32.430 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.425+0000 7f75217cd640 1 -- 172.21.15.191:0/1082074303 learned_addr learned my addr 172.21.15.191:0/1082074303 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-16T20:36:32.430 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.425+0000 7f7521fce640 1 --2- 172.21.15.191:0/1082074303 >> v2:172.21.15.88:3300/0 conn(0x7f751c153780 0x7f751c0c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.430 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.425+0000 7f75217cd640 1 -- 172.21.15.191:0/1082074303 >> v2:172.21.15.88:3300/0 conn(0x7f751c153780 msgr2=0x7f751c0c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:32.430 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.425+0000 7f75217cd640 1 --2- 172.21.15.191:0/1082074303 >> v2:172.21.15.88:3300/0 conn(0x7f751c153780 0x7f751c0c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.431 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.425+0000 7f75217cd640 1 -- 172.21.15.191:0/1082074303 >> v1:172.21.15.191:6789/0 conn(0x7f751c0c5930 legacy=0x7f751c0ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.431 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.425+0000 7f75217cd640 1 -- 172.21.15.191:0/1082074303 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751c0c4910 con 0x7f751c159490 2023-10-16T20:36:32.431 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.425+0000 7f7520fcc640 1 -- 172.21.15.191:0/1082074303 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641357351 0 0) 0x7f751c159d90 con 0x7f751c0c5930 2023-10-16T20:36:32.432 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.429+0000 7f75217cd640 1 --2- 172.21.15.191:0/1082074303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f751c159490 0x7f751c0c53f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f751c0cfef0 tx=0x7f750c01ecf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41654ebc3e944005 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:32.432 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.429+0000 7f7520fcc640 1 -- 172.21.15.191:0/1082074303 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750c0a89f0 con 0x7f751c159490 2023-10-16T20:36:32.432 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.429+0000 7f7520fcc640 1 -- 172.21.15.191:0/1082074303 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750c0a8b50 con 0x7f751c159490 2023-10-16T20:36:32.432 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.429+0000 7f7520fcc640 1 -- 172.21.15.191:0/1082074303 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750c0b0440 con 0x7f751c159490 2023-10-16T20:36:32.433 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.429+0000 7f7524232640 1 -- 172.21.15.191:0/1082074303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f751c159490 msgr2=0x7f751c0c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.433 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.429+0000 7f7524232640 1 --2- 172.21.15.191:0/1082074303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f751c159490 0x7f751c0c53f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f751c0cfef0 tx=0x7f750c01ecf0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.434 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.429+0000 7f7524232640 1 -- 172.21.15.191:0/1082074303 shutdown_connections 2023-10-16T20:36:32.434 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.429+0000 7f7524232640 1 --2- 172.21.15.191:0/1082074303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f751c159490 0x7f751c0c53f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.434 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.429+0000 7f7524232640 1 --2- 172.21.15.191:0/1082074303 >> v2:172.21.15.88:3300/0 conn(0x7f751c153780 0x7f751c0c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.434 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.429+0000 7f7524232640 1 -- 172.21.15.191:0/1082074303 >> 172.21.15.191:0/1082074303 conn(0x7f751c06bcb0 msgr2=0x7f751c06c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:32.434 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.429+0000 7f7524232640 1 -- 172.21.15.191:0/1082074303 shutdown_connections 2023-10-16T20:36:32.435 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.429+0000 7f7524232640 1 -- 172.21.15.191:0/1082074303 wait complete. 2023-10-16T20:36:32.437 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.429+0000 7f7524232640 1 Processor -- start 2023-10-16T20:36:32.437 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.429+0000 7f7524232640 1 -- start start 2023-10-16T20:36:32.437 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7524232640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f751c0c5930 0x7f751c157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.437 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7524232640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f751c153780 0x7f751c0c6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.437 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7524232640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f751c159490 0x7f751c0c6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.438 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7524232640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751c157630 con 0x7f751c153780 2023-10-16T20:36:32.438 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7524232640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751c1ee950 con 0x7f751c0c5930 2023-10-16T20:36:32.438 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7524232640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f751c1ef920 con 0x7f751c159490 2023-10-16T20:36:32.438 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7521fce640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f751c0c5930 0x7f751c157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.438 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7521fce640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f751c0c5930 0x7f751c157ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:33600/0 (socket says 172.21.15.191:33600) 2023-10-16T20:36:32.439 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7521fce640 1 -- 172.21.15.191:0/3391423007 learned_addr learned my addr 172.21.15.191:0/3391423007 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-16T20:36:32.439 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f75227cf640 1 --2- 172.21.15.191:0/3391423007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f751c159490 0x7f751c0c6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.439 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f75217cd640 1 --2- 172.21.15.191:0/3391423007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f751c153780 0x7f751c0c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.439 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7521fce640 1 -- 172.21.15.191:0/3391423007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f751c159490 msgr2=0x7f751c0c6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.439 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7521fce640 1 --2- 172.21.15.191:0/3391423007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f751c159490 0x7f751c0c6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.440 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7521fce640 1 -- 172.21.15.191:0/3391423007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f751c153780 msgr2=0x7f751c0c6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.440 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7521fce640 1 --2- 172.21.15.191:0/3391423007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f751c153780 0x7f751c0c6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.440 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7521fce640 1 -- 172.21.15.191:0/3391423007 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750c0094d0 con 0x7f751c0c5930 2023-10-16T20:36:32.440 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7521fce640 1 --2- 172.21.15.191:0/3391423007 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f751c0c5930 0x7f751c157ee0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f751800e9e0 tx=0x7f751800eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:32.440 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f75127fc640 1 -- 172.21.15.191:0/3391423007 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7518009670 con 0x7f751c0c5930 2023-10-16T20:36:32.440 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f75127fc640 1 -- 172.21.15.191:0/3391423007 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751800c9d0 con 0x7f751c0c5930 2023-10-16T20:36:32.441 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f75127fc640 1 -- 172.21.15.191:0/3391423007 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751800cd40 con 0x7f751c0c5930 2023-10-16T20:36:32.441 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7524232640 1 -- 172.21.15.191:0/3391423007 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f751c157350 con 0x7f751c0c5930 2023-10-16T20:36:32.441 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7524232640 1 -- 172.21.15.191:0/3391423007 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f751c1eee40 con 0x7f751c0c5930 2023-10-16T20:36:32.441 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f75127fc640 1 -- 172.21.15.191:0/3391423007 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7518013770 con 0x7f751c0c5930 2023-10-16T20:36:32.441 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f75127fc640 1 -- 172.21.15.191:0/3391423007 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f75180123d0 con 0x7f751c0c5930 2023-10-16T20:36:32.442 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.433+0000 7f7524232640 1 -- 172.21.15.191:0/3391423007 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f751c074700 con 0x7f751c0c5930 2023-10-16T20:36:32.442 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.437+0000 7f75127fc640 1 -- 172.21.15.191:0/3391423007 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f7518021060 con 0x7f751c0c5930 2023-10-16T20:36:32.497 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:32.490+0000 7fd450c13180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T20:36:32.498 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:32.493+0000 7f0f19b7e180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T20:36:32.508 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.505+0000 7f7524232640 1 -- 172.21.15.191:0/3391423007 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6e32466e-2f1f-4220-9f02-30708ad99202", "id": 5} v 0) v1 -- 0x7f751c0ca7d0 con 0x7f751c0c5930 2023-10-16T20:36:32.515 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.509+0000 7f75127fc640 1 -- 172.21.15.191:0/3391423007 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6e32466e-2f1f-4220-9f02-30708ad99202", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7518017070 con 0x7f751c0c5930 2023-10-16T20:36:32.515 INFO:teuthology.orchestra.run.smithi191.stdout:5 2023-10-16T20:36:32.516 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.513+0000 7f7524232640 1 -- 172.21.15.191:0/3391423007 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f751c0c5930 msgr2=0x7f751c157ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.516 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.513+0000 7f7524232640 1 --2- 172.21.15.191:0/3391423007 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f751c0c5930 0x7f751c157ee0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f751800e9e0 tx=0x7f751800eea0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.516 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.513+0000 7f7524232640 1 -- 172.21.15.191:0/3391423007 shutdown_connections 2023-10-16T20:36:32.516 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.513+0000 7f7524232640 1 --2- 172.21.15.191:0/3391423007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f751c159490 0x7f751c0c6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.517 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.513+0000 7f7524232640 1 --2- 172.21.15.191:0/3391423007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f751c153780 0x7f751c0c6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.517 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.513+0000 7f7524232640 1 --2- 172.21.15.191:0/3391423007 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f751c0c5930 0x7f751c157ee0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.517 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.513+0000 7f7524232640 1 -- 172.21.15.191:0/3391423007 >> 172.21.15.191:0/3391423007 conn(0x7f751c06bcb0 msgr2=0x7f751c0cbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:32.517 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.513+0000 7f7524232640 1 -- 172.21.15.191:0/3391423007 shutdown_connections 2023-10-16T20:36:32.517 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.513+0000 7f7524232640 1 -- 172.21.15.191:0/3391423007 wait complete. 2023-10-16T20:36:32.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new d82bbafe-7801-495b-95ce-768a512980fa 6 2023-10-16T20:36:32.558 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:32.550+0000 7fd450c13180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T20:36:32.564 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:32.561+0000 7f0f19b7e180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T20:36:32.671 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.669+0000 7fa989b2a640 1 Processor -- start 2023-10-16T20:36:32.672 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.669+0000 7fa989b2a640 1 -- start start 2023-10-16T20:36:32.673 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.669+0000 7fa989b2a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa984153780 0x7fa9840c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.674 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.669+0000 7fa989b2a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa984159490 0x7fa9840c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.674 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.669+0000 7fa989b2a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa984159d90 con 0x7fa9840c5930 2023-10-16T20:36:32.674 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.669+0000 7fa989b2a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98415a070 con 0x7fa984153780 2023-10-16T20:36:32.674 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.669+0000 7fa989b2a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98415a350 con 0x7fa984159490 2023-10-16T20:36:32.674 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.669+0000 7fa982ffd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa984159490 0x7fa9840c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.675 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.669+0000 7fa982ffd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa984159490 0x7fa9840c53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:33604/0 (socket says 172.21.15.191:33604) 2023-10-16T20:36:32.675 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.669+0000 7fa982ffd640 1 -- 172.21.15.191:0/2257430720 learned_addr learned my addr 172.21.15.191:0/2257430720 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-16T20:36:32.675 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.669+0000 7fa9837fe640 1 --2- 172.21.15.191:0/2257430720 >> v2:172.21.15.88:3300/0 conn(0x7fa984153780 0x7fa9840c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.675 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.669+0000 7fa982ffd640 1 -- 172.21.15.191:0/2257430720 >> v2:172.21.15.88:3300/0 conn(0x7fa984153780 msgr2=0x7fa9840c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:32.675 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.669+0000 7fa982ffd640 1 --2- 172.21.15.191:0/2257430720 >> v2:172.21.15.88:3300/0 conn(0x7fa984153780 0x7fa9840c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.675 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.669+0000 7fa982ffd640 1 -- 172.21.15.191:0/2257430720 >> v1:172.21.15.88:6789/0 conn(0x7fa9840c5930 legacy=0x7fa9840ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.676 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.669+0000 7fa982ffd640 1 -- 172.21.15.191:0/2257430720 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9840c4910 con 0x7fa984159490 2023-10-16T20:36:32.676 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.673+0000 7fa982ffd640 1 --2- 172.21.15.191:0/2257430720 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa984159490 0x7fa9840c53f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa9840cfef0 tx=0x7fa97801ecf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95d21adfd27ec9f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:32.677 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.673+0000 7fa9827fc640 1 -- 172.21.15.191:0/2257430720 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9780a89f0 con 0x7fa984159490 2023-10-16T20:36:32.677 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.673+0000 7fa9827fc640 1 -- 172.21.15.191:0/2257430720 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9780a8b50 con 0x7fa984159490 2023-10-16T20:36:32.677 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.673+0000 7fa9827fc640 1 -- 172.21.15.191:0/2257430720 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9780b0440 con 0x7fa984159490 2023-10-16T20:36:32.678 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.673+0000 7fa989b2a640 1 -- 172.21.15.191:0/2257430720 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa984159490 msgr2=0x7fa9840c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.678 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.673+0000 7fa989b2a640 1 --2- 172.21.15.191:0/2257430720 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa984159490 0x7fa9840c53f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa9840cfef0 tx=0x7fa97801ecf0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.678 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.673+0000 7fa989b2a640 1 -- 172.21.15.191:0/2257430720 shutdown_connections 2023-10-16T20:36:32.679 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.673+0000 7fa989b2a640 1 --2- 172.21.15.191:0/2257430720 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa984159490 0x7fa9840c53f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.679 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.673+0000 7fa989b2a640 1 --2- 172.21.15.191:0/2257430720 >> v2:172.21.15.88:3300/0 conn(0x7fa984153780 0x7fa9840c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.679 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.673+0000 7fa989b2a640 1 -- 172.21.15.191:0/2257430720 >> 172.21.15.191:0/2257430720 conn(0x7fa98406bcb0 msgr2=0x7fa98406c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:32.679 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.673+0000 7fa989b2a640 1 -- 172.21.15.191:0/2257430720 shutdown_connections 2023-10-16T20:36:32.679 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.673+0000 7fa989b2a640 1 -- 172.21.15.191:0/2257430720 wait complete. 2023-10-16T20:36:32.681 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa989b2a640 1 Processor -- start 2023-10-16T20:36:32.681 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa989b2a640 1 -- start start 2023-10-16T20:36:32.681 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa989b2a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9840c5930 0x7fa9840c8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.681 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa989b2a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa984153780 0x7fa98414edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.682 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa989b2a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa984159490 0x7fa9840b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.682 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa989b2a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9841ef970 con 0x7fa984159490 2023-10-16T20:36:32.682 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa989b2a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9840c89d0 con 0x7fa984153780 2023-10-16T20:36:32.682 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa989b2a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9841eecf0 con 0x7fa9840c5930 2023-10-16T20:36:32.682 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa982ffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa984153780 0x7fa98414edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.683 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa9837fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9840c5930 0x7fa9840c8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.683 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa982ffd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa984153780 0x7fa98414edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:33606/0 (socket says 172.21.15.191:33606) 2023-10-16T20:36:32.683 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa982ffd640 1 -- 172.21.15.191:0/3185415087 learned_addr learned my addr 172.21.15.191:0/3185415087 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-16T20:36:32.683 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa983fff640 1 --2- 172.21.15.191:0/3185415087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa984159490 0x7fa9840b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.683 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa9837fe640 1 -- 172.21.15.191:0/3185415087 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa984153780 msgr2=0x7fa98414edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.683 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa9837fe640 1 --2- 172.21.15.191:0/3185415087 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa984153780 0x7fa98414edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa9837fe640 1 -- 172.21.15.191:0/3185415087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa984159490 msgr2=0x7fa9840b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa9837fe640 1 --2- 172.21.15.191:0/3185415087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa984159490 0x7fa9840b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa9837fe640 1 -- 172.21.15.191:0/3185415087 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9780094d0 con 0x7fa9840c5930 2023-10-16T20:36:32.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa982ffd640 1 --2- 172.21.15.191:0/3185415087 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa984153780 0x7fa98414edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:36:32.685 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa9837fe640 1 --2- 172.21.15.191:0/3185415087 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9840c5930 0x7fa9840c8490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa97400d5a0 tx=0x7fa97400da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:32.685 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa96bfff640 1 -- 172.21.15.191:0/3185415087 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa974005bb0 con 0x7fa9840c5930 2023-10-16T20:36:32.686 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa96bfff640 1 -- 172.21.15.191:0/3185415087 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa974005d10 con 0x7fa9840c5930 2023-10-16T20:36:32.686 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa96bfff640 1 -- 172.21.15.191:0/3185415087 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa974006290 con 0x7fa9840c5930 2023-10-16T20:36:32.686 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa989b2a640 1 -- 172.21.15.191:0/3185415087 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9841eefd0 con 0x7fa9840c5930 2023-10-16T20:36:32.686 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa989b2a640 1 -- 172.21.15.191:0/3185415087 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9841ef590 con 0x7fa9840c5930 2023-10-16T20:36:32.686 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa96bfff640 1 -- 172.21.15.191:0/3185415087 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa9740066f0 con 0x7fa9840c5930 2023-10-16T20:36:32.687 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa96bfff640 1 -- 172.21.15.191:0/3185415087 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fa974013cf0 con 0x7fa9840c5930 2023-10-16T20:36:32.687 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.677+0000 7fa989b2a640 1 -- 172.21.15.191:0/3185415087 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa984074700 con 0x7fa9840c5930 2023-10-16T20:36:32.687 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.685+0000 7fa96bfff640 1 -- 172.21.15.191:0/3185415087 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fa974013480 con 0x7fa9840c5930 2023-10-16T20:36:32.714 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:32.706+0000 7fd450c13180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T20:36:32.728 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:32.725+0000 7f0f19b7e180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T20:36:32.752 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.749+0000 7fa989b2a640 1 -- 172.21.15.191:0/3185415087 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d82bbafe-7801-495b-95ce-768a512980fa", "id": 6} v 0) v1 -- 0x7fa984150560 con 0x7fa9840c5930 2023-10-16T20:36:32.758 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.753+0000 7fa96bfff640 1 -- 172.21.15.191:0/3185415087 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d82bbafe-7801-495b-95ce-768a512980fa", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa974013620 con 0x7fa9840c5930 2023-10-16T20:36:32.758 INFO:teuthology.orchestra.run.smithi191.stdout:6 2023-10-16T20:36:32.759 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.753+0000 7fa989b2a640 1 -- 172.21.15.191:0/3185415087 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9840c5930 msgr2=0x7fa9840c8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.759 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.753+0000 7fa989b2a640 1 --2- 172.21.15.191:0/3185415087 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9840c5930 0x7fa9840c8490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa97400d5a0 tx=0x7fa97400da60 comp rx=0 tx=0).stop 2023-10-16T20:36:32.760 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.757+0000 7fa989b2a640 1 -- 172.21.15.191:0/3185415087 shutdown_connections 2023-10-16T20:36:32.760 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.757+0000 7fa989b2a640 1 --2- 172.21.15.191:0/3185415087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa984159490 0x7fa9840b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.760 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.757+0000 7fa989b2a640 1 --2- 172.21.15.191:0/3185415087 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa984153780 0x7fa98414edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.760 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.757+0000 7fa989b2a640 1 --2- 172.21.15.191:0/3185415087 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9840c5930 0x7fa9840c8490 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.760 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.757+0000 7fa989b2a640 1 -- 172.21.15.191:0/3185415087 >> 172.21.15.191:0/3185415087 conn(0x7fa98406bcb0 msgr2=0x7fa9840cdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:32.761 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.757+0000 7fa989b2a640 1 -- 172.21.15.191:0/3185415087 shutdown_connections 2023-10-16T20:36:32.761 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.757+0000 7fa989b2a640 1 -- 172.21.15.191:0/3185415087 wait complete. 2023-10-16T20:36:32.777 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new d10aba86-e78a-4f6d-a15c-fe5dffd0aac6 7 2023-10-16T20:36:32.784 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:32.774+0000 7fd450c13180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T20:36:32.801 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:32.797+0000 7f0f19b7e180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T20:36:32.938 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.933+0000 7fb4c49f8640 1 Processor -- start 2023-10-16T20:36:32.939 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.933+0000 7fb4c49f8640 1 -- start start 2023-10-16T20:36:32.940 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.937+0000 7fb4c49f8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4c0153780 0x7fb4c00c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.941 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.937+0000 7fb4c49f8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb4c0159890 0x7fb4c00ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.941 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.937+0000 7fb4c49f8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c0159d90 con 0x7fb4c0159490 2023-10-16T20:36:32.941 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.937+0000 7fb4c49f8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c015a070 con 0x7fb4c0159890 2023-10-16T20:36:32.941 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.937+0000 7fb4c49f8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c015a350 con 0x7fb4c0153780 2023-10-16T20:36:32.941 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.937+0000 7fb4be59c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4c0153780 0x7fb4c00c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.942 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.937+0000 7fb4bdd9b640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fb4c0159490 0x7fb4c00c53f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:53358/0 (socket says 172.21.15.191:53358) 2023-10-16T20:36:32.942 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.937+0000 7fb4bdd9b640 1 -- 172.21.15.191:0/3358084753 learned_addr learned my addr 172.21.15.191:0/3358084753 (peer_addr_for_me v1:172.21.15.191:0/0) 2023-10-16T20:36:32.942 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.937+0000 7fb4bed9d640 1 --2- 172.21.15.191:0/3358084753 >> v2:172.21.15.88:3300/0 conn(0x7fb4c0159890 0x7fb4c00ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.942 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.937+0000 7fb4be59c640 1 -- 172.21.15.191:0/3358084753 >> v2:172.21.15.88:3300/0 conn(0x7fb4c0159890 msgr2=0x7fb4c00ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:32.942 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.937+0000 7fb4be59c640 1 --2- 172.21.15.191:0/3358084753 >> v2:172.21.15.88:3300/0 conn(0x7fb4c0159890 0x7fb4c00ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.943 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.937+0000 7fb4be59c640 1 -- 172.21.15.191:0/3358084753 >> v1:172.21.15.191:6789/0 conn(0x7fb4c0159490 legacy=0x7fb4c00c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.943 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.937+0000 7fb4be59c640 1 -- 172.21.15.191:0/3358084753 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c00c4910 con 0x7fb4c0153780 2023-10-16T20:36:32.943 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.937+0000 7fb4bd59a640 1 -- 172.21.15.191:0/3358084753 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703973244 0 0) 0x7fb4c0159d90 con 0x7fb4c0159490 2023-10-16T20:36:32.944 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.941+0000 7fb4be59c640 1 --2- 172.21.15.191:0/3358084753 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4c0153780 0x7fb4c00c9be0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00097a0 tx=0x7fb4b001ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5a0f8a9dfcaad3b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:32.945 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.941+0000 7fb4bd59a640 1 -- 172.21.15.191:0/3358084753 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b00a8950 con 0x7fb4c0153780 2023-10-16T20:36:32.945 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.941+0000 7fb4bd59a640 1 -- 172.21.15.191:0/3358084753 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b00a8ab0 con 0x7fb4c0153780 2023-10-16T20:36:32.945 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.941+0000 7fb4bd59a640 1 -- 172.21.15.191:0/3358084753 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b00a8e40 con 0x7fb4c0153780 2023-10-16T20:36:32.945 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.941+0000 7fb4c49f8640 1 -- 172.21.15.191:0/3358084753 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4c0153780 msgr2=0x7fb4c00c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.945 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.941+0000 7fb4c49f8640 1 --2- 172.21.15.191:0/3358084753 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4c0153780 0x7fb4c00c9be0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00097a0 tx=0x7fb4b001ec50 comp rx=0 tx=0).stop 2023-10-16T20:36:32.946 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.941+0000 7fb4c49f8640 1 -- 172.21.15.191:0/3358084753 shutdown_connections 2023-10-16T20:36:32.946 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.941+0000 7fb4c49f8640 1 --2- 172.21.15.191:0/3358084753 >> v2:172.21.15.88:3300/0 conn(0x7fb4c0159890 0x7fb4c00ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.946 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.941+0000 7fb4c49f8640 1 --2- 172.21.15.191:0/3358084753 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4c0153780 0x7fb4c00c9be0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.946 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.941+0000 7fb4c49f8640 1 -- 172.21.15.191:0/3358084753 >> 172.21.15.191:0/3358084753 conn(0x7fb4c006bcb0 msgr2=0x7fb4c006c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:32.946 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.941+0000 7fb4c49f8640 1 -- 172.21.15.191:0/3358084753 shutdown_connections 2023-10-16T20:36:32.947 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.941+0000 7fb4c49f8640 1 -- 172.21.15.191:0/3358084753 wait complete. 2023-10-16T20:36:32.949 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.941+0000 7fb4c49f8640 1 Processor -- start 2023-10-16T20:36:32.949 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.941+0000 7fb4c49f8640 1 -- start start 2023-10-16T20:36:32.949 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4c49f8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4c0153780 0x7fb4c01f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.949 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4c49f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4c0159490 0x7fb4c01ee930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.950 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4c49f8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4c0159890 0x7fb4c01ef400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:32.950 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4c49f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c015c180 con 0x7fb4c0159490 2023-10-16T20:36:32.950 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4c49f8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c015aea0 con 0x7fb4c0153780 2023-10-16T20:36:32.950 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4c49f8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c01f2610 con 0x7fb4c0159890 2023-10-16T20:36:32.951 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4bed9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4c0159890 0x7fb4c01ef400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.951 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4be59c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4c0153780 0x7fb4c01f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.951 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4bed9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4c0159890 0x7fb4c01ef400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.191:35194/0 (socket says 172.21.15.191:35194) 2023-10-16T20:36:32.951 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4bed9d640 1 -- 172.21.15.191:0/986889685 learned_addr learned my addr 172.21.15.191:0/986889685 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-10-16T20:36:32.951 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4bdd9b640 1 --2- 172.21.15.191:0/986889685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4c0159490 0x7fb4c01ee930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:32.952 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4bed9d640 1 -- 172.21.15.191:0/986889685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4c0153780 msgr2=0x7fb4c01f2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.952 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4bed9d640 1 --2- 172.21.15.191:0/986889685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4c0153780 0x7fb4c01f2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.952 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4bed9d640 1 -- 172.21.15.191:0/986889685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4c0159490 msgr2=0x7fb4c01ee930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:32.952 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4bed9d640 1 --2- 172.21.15.191:0/986889685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4c0159490 0x7fb4c01ee930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:32.952 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4bed9d640 1 -- 172.21.15.191:0/986889685 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b00093b0 con 0x7fb4c0159890 2023-10-16T20:36:32.952 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4be59c640 1 --2- 172.21.15.191:0/986889685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4c0153780 0x7fb4c01f2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:36:32.953 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4bed9d640 1 --2- 172.21.15.191:0/986889685 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4c0159890 0x7fb4c01ef400 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8007d70 tx=0x7fb4b800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:32.953 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4aeffd640 1 -- 172.21.15.191:0/986889685 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8014420 con 0x7fb4c0159890 2023-10-16T20:36:32.953 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4aeffd640 1 -- 172.21.15.191:0/986889685 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b80145a0 con 0x7fb4c0159890 2023-10-16T20:36:32.953 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4c49f8640 1 -- 172.21.15.191:0/986889685 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c01ef940 con 0x7fb4c0159890 2023-10-16T20:36:32.953 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4aeffd640 1 -- 172.21.15.191:0/986889685 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8014950 con 0x7fb4c0159890 2023-10-16T20:36:32.954 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4c49f8640 1 -- 172.21.15.191:0/986889685 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c00c7b00 con 0x7fb4c0159890 2023-10-16T20:36:32.954 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4aeffd640 1 -- 172.21.15.191:0/986889685 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb4b801e460 con 0x7fb4c0159890 2023-10-16T20:36:32.954 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4aeffd640 1 -- 172.21.15.191:0/986889685 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fb4b801da20 con 0x7fb4c0159890 2023-10-16T20:36:32.955 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.945+0000 7fb4c49f8640 1 -- 172.21.15.191:0/986889685 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c0074700 con 0x7fb4c0159890 2023-10-16T20:36:32.955 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:32.949+0000 7fb4aeffd640 1 -- 172.21.15.191:0/986889685 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fb4b801dd00 con 0x7fb4c0159890 2023-10-16T20:36:33.017 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:33.013+0000 7fb4c49f8640 1 -- 172.21.15.191:0/986889685 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d10aba86-e78a-4f6d-a15c-fe5dffd0aac6", "id": 7} v 0) v1 -- 0x7fb4c01f2750 con 0x7fb4c0159890 2023-10-16T20:36:33.022 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:33.017+0000 7fb4aeffd640 1 -- 172.21.15.191:0/986889685 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d10aba86-e78a-4f6d-a15c-fe5dffd0aac6", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb4b801d3c0 con 0x7fb4c0159890 2023-10-16T20:36:33.022 INFO:teuthology.orchestra.run.smithi191.stdout:7 2023-10-16T20:36:33.023 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:33.021+0000 7fb4c49f8640 1 -- 172.21.15.191:0/986889685 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4c0159890 msgr2=0x7fb4c01ef400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:33.023 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:33.021+0000 7fb4c49f8640 1 --2- 172.21.15.191:0/986889685 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4c0159890 0x7fb4c01ef400 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8007d70 tx=0x7fb4b800e4b0 comp rx=0 tx=0).stop 2023-10-16T20:36:33.023 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:33.021+0000 7fb4c49f8640 1 -- 172.21.15.191:0/986889685 shutdown_connections 2023-10-16T20:36:33.024 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:33.021+0000 7fb4c49f8640 1 --2- 172.21.15.191:0/986889685 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb4c0159890 0x7fb4c01ef400 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:33.024 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:33.021+0000 7fb4c49f8640 1 --2- 172.21.15.191:0/986889685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4c0159490 0x7fb4c01ee930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:33.024 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:33.021+0000 7fb4c49f8640 1 --2- 172.21.15.191:0/986889685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb4c0153780 0x7fb4c01f2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:33.024 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:33.021+0000 7fb4c49f8640 1 -- 172.21.15.191:0/986889685 >> 172.21.15.191:0/986889685 conn(0x7fb4c006bcb0 msgr2=0x7fb4c00c3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:33.025 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:33.021+0000 7fb4c49f8640 1 -- 172.21.15.191:0/986889685 shutdown_connections 2023-10-16T20:36:33.025 INFO:teuthology.orchestra.run.smithi191.stderr:2023-10-16T20:36:33.021+0000 7fb4c49f8640 1 -- 172.21.15.191:0/986889685 wait complete. 2023-10-16T20:36:33.040 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-16T20:36:33.041 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-16T20:36:33.045 INFO:tasks.ceph.osd.0:Started 2023-10-16T20:36:33.045 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-16T20:36:33.045 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-16T20:36:33.049 INFO:tasks.ceph.osd.1:Started 2023-10-16T20:36:33.049 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-16T20:36:33.049 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-16T20:36:33.053 INFO:tasks.ceph.osd.2:Started 2023-10-16T20:36:33.053 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-16T20:36:33.053 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-16T20:36:33.059 INFO:tasks.ceph.osd.3:Started 2023-10-16T20:36:33.059 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-16T20:36:33.060 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-16T20:36:33.063 INFO:tasks.ceph.osd.4:Started 2023-10-16T20:36:33.064 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-16T20:36:33.064 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-16T20:36:33.068 INFO:tasks.ceph.osd.5:Started 2023-10-16T20:36:33.068 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-16T20:36:33.068 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-16T20:36:33.072 INFO:tasks.ceph.osd.6:Started 2023-10-16T20:36:33.073 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-16T20:36:33.073 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-16T20:36:33.078 INFO:tasks.ceph.osd.7:Started 2023-10-16T20:36:33.079 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:36:33.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.298+0000 7f4c4a93e640 1 Processor -- start 2023-10-16T20:36:33.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.298+0000 7f4c4a93e640 1 -- start start 2023-10-16T20:36:33.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.302+0000 7f4c4a93e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4c440c9bb0 0x7f4c440c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:33.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.302+0000 7f4c4a93e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c44153b50 con 0x7f4c440c41f0 2023-10-16T20:36:33.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.302+0000 7f4c4a93e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c440cb2f0 con 0x7f4c44153770 2023-10-16T20:36:33.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.302+0000 7f4c4a93e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c440cc250 con 0x7f4c440c9bb0 2023-10-16T20:36:33.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.302+0000 7f4c437fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4c440c9bb0 0x7f4c440c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:33.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.302+0000 7f4c437fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4c440c9bb0 0x7f4c440c3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56860/0 (socket says 172.21.15.88:56860) 2023-10-16T20:36:33.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.302+0000 7f4c437fe640 1 -- 172.21.15.88:0/3749500360 learned_addr learned my addr 172.21.15.88:0/3749500360 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:33.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.302+0000 7f4c437fe640 1 -- 172.21.15.88:0/3749500360 >> v1:172.21.15.191:6789/0 conn(0x7f4c44153770 legacy=0x7f4c44159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:33.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.302+0000 7f4c437fe640 1 -- 172.21.15.88:0/3749500360 >> v1:172.21.15.88:6789/0 conn(0x7f4c440c41f0 legacy=0x7f4c440cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:33.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.302+0000 7f4c437fe640 1 -- 172.21.15.88:0/3749500360 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c440cb5d0 con 0x7f4c440c9bb0 2023-10-16T20:36:33.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c437fe640 1 --2- 172.21.15.88:0/3749500360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c440c9bb0 0x7f4c440c3cb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4c38009870 tx=0x7f4c3801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd7c50a0a34bb65 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:33.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c42ffd640 1 -- 172.21.15.88:0/3749500360 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c380ab650 con 0x7f4c440c9bb0 2023-10-16T20:36:33.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c42ffd640 1 -- 172.21.15.88:0/3749500360 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c380ab7b0 con 0x7f4c440c9bb0 2023-10-16T20:36:33.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c42ffd640 1 -- 172.21.15.88:0/3749500360 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c380abb40 con 0x7f4c440c9bb0 2023-10-16T20:36:33.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c4a93e640 1 -- 172.21.15.88:0/3749500360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c440c9bb0 msgr2=0x7f4c440c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:33.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c4a93e640 1 --2- 172.21.15.88:0/3749500360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c440c9bb0 0x7f4c440c3cb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4c38009870 tx=0x7f4c3801c540 comp rx=0 tx=0).stop 2023-10-16T20:36:33.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c4a93e640 1 -- 172.21.15.88:0/3749500360 shutdown_connections 2023-10-16T20:36:33.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c4a93e640 1 --2- 172.21.15.88:0/3749500360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c440c9bb0 0x7f4c440c3cb0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:33.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c4a93e640 1 -- 172.21.15.88:0/3749500360 >> 172.21.15.88:0/3749500360 conn(0x7f4c440715c0 msgr2=0x7f4c440719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:33.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c4a93e640 1 -- 172.21.15.88:0/3749500360 shutdown_connections 2023-10-16T20:36:33.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c4a93e640 1 -- 172.21.15.88:0/3749500360 wait complete. 2023-10-16T20:36:33.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c4a93e640 1 Processor -- start 2023-10-16T20:36:33.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c4a93e640 1 -- start start 2023-10-16T20:36:33.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c4a93e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c440c41f0 0x7f4c44152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:33.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c4a93e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4c440c9bb0 0x7f4c4415c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:33.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c4a93e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c44153770 0x7f4c441f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:33.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c4a93e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c441f1ff0 con 0x7f4c440c41f0 2023-10-16T20:36:33.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c4a93e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c441f2520 con 0x7f4c44153770 2023-10-16T20:36:33.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.306+0000 7f4c4a93e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c440cea30 con 0x7f4c440c9bb0 2023-10-16T20:36:33.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c43fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c440c41f0 0x7f4c44152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:33.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c43fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c440c41f0 0x7f4c44152440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56866/0 (socket says 172.21.15.88:56866) 2023-10-16T20:36:33.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c43fff640 1 -- 172.21.15.88:0/3266170616 learned_addr learned my addr 172.21.15.88:0/3266170616 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:33.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c48edb640 1 --2- 172.21.15.88:0/3266170616 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c44153770 0x7f4c441f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:33.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c437fe640 1 --2- 172.21.15.88:0/3266170616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4c440c9bb0 0x7f4c4415c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:33.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c43fff640 1 -- 172.21.15.88:0/3266170616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4c440c9bb0 msgr2=0x7f4c4415c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:33.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c43fff640 1 --2- 172.21.15.88:0/3266170616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4c440c9bb0 0x7f4c4415c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:33.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c43fff640 1 -- 172.21.15.88:0/3266170616 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c44153770 msgr2=0x7f4c441f4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:33.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c43fff640 1 --2- 172.21.15.88:0/3266170616 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c44153770 0x7f4c441f4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:33.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c43fff640 1 -- 172.21.15.88:0/3266170616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c380094d0 con 0x7f4c440c41f0 2023-10-16T20:36:33.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c43fff640 1 --2- 172.21.15.88:0/3266170616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c440c41f0 0x7f4c44152440 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4c3400c400 tx=0x7f4c3400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:33.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c40ff9640 1 -- 172.21.15.88:0/3266170616 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c340052d0 con 0x7f4c440c41f0 2023-10-16T20:36:33.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c40ff9640 1 -- 172.21.15.88:0/3266170616 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c34005430 con 0x7f4c440c41f0 2023-10-16T20:36:33.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c4a93e640 1 -- 172.21.15.88:0/3266170616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4414bc10 con 0x7f4c440c41f0 2023-10-16T20:36:33.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c40ff9640 1 -- 172.21.15.88:0/3266170616 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c34003890 con 0x7f4c440c41f0 2023-10-16T20:36:33.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c4a93e640 1 -- 172.21.15.88:0/3266170616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4414ae50 con 0x7f4c440c41f0 2023-10-16T20:36:33.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c40ff9640 1 -- 172.21.15.88:0/3266170616 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4c34003cf0 con 0x7f4c440c41f0 2023-10-16T20:36:33.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c40ff9640 1 -- 172.21.15.88:0/3266170616 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f4c34012420 con 0x7f4c440c41f0 2023-10-16T20:36:33.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.310+0000 7f4c4a93e640 1 -- 172.21.15.88:0/3266170616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c44074700 con 0x7f4c440c41f0 2023-10-16T20:36:33.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.314+0000 7f4c40ff9640 1 -- 172.21.15.88:0/3266170616 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f4c34017070 con 0x7f4c440c41f0 2023-10-16T20:36:33.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.370+0000 7f4c4a93e640 1 -- 172.21.15.88:0/3266170616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c440ca740 con 0x7f4c440c41f0 2023-10-16T20:36:33.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.370+0000 7f4c40ff9640 1 -- 172.21.15.88:0/3266170616 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f4c34011590 con 0x7f4c440c41f0 2023-10-16T20:36:33.378 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:36:33.379 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":10,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:36:33.014460+0000","last_up_change":"0.000000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:36:33.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.370+0000 7f4c4a93e640 1 -- 172.21.15.88:0/3266170616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c440c41f0 msgr2=0x7f4c44152440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:33.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.370+0000 7f4c4a93e640 1 --2- 172.21.15.88:0/3266170616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c440c41f0 0x7f4c44152440 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4c3400c400 tx=0x7f4c3400c8c0 comp rx=0 tx=0).stop 2023-10-16T20:36:33.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.370+0000 7f4c4a93e640 1 -- 172.21.15.88:0/3266170616 shutdown_connections 2023-10-16T20:36:33.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.370+0000 7f4c4a93e640 1 --2- 172.21.15.88:0/3266170616 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4c44153770 0x7f4c441f4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:33.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.370+0000 7f4c4a93e640 1 --2- 172.21.15.88:0/3266170616 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4c440c9bb0 0x7f4c4415c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:33.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.370+0000 7f4c4a93e640 1 --2- 172.21.15.88:0/3266170616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c440c41f0 0x7f4c44152440 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:33.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.370+0000 7f4c4a93e640 1 -- 172.21.15.88:0/3266170616 >> 172.21.15.88:0/3266170616 conn(0x7f4c440715c0 msgr2=0x7f4c440719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:33.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.370+0000 7f4c4a93e640 1 -- 172.21.15.88:0/3266170616 shutdown_connections 2023-10-16T20:36:33.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:33.370+0000 7f4c4a93e640 1 -- 172.21.15.88:0/3266170616 wait complete. 2023-10-16T20:36:33.398 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-16T20:36:33.399 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-16T20:36:33.471 INFO:tasks.ceph.mgr.y.smithi088.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-16T20:36:33.471 INFO:tasks.ceph.mgr.y.smithi088.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-16T20:36:33.472 INFO:tasks.ceph.mgr.y.smithi088.stderr: from numpy import show_config as show_numpy_config 2023-10-16T20:36:33.569 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:33.562+0000 7fd450c13180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T20:36:33.577 INFO:tasks.ceph.mgr.x.smithi191.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-16T20:36:33.577 INFO:tasks.ceph.mgr.x.smithi191.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-16T20:36:33.577 INFO:tasks.ceph.mgr.x.smithi191.stderr: from numpy import show_config as show_numpy_config 2023-10-16T20:36:33.625 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:33.621+0000 7f0f19b7e180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T20:36:33.705 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:33.694+0000 7fd450c13180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T20:36:33.752 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-10-16T20:36:33.749+0000 7f7cdfa2d640 -1 Falling back to public interface 2023-10-16T20:36:33.754 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-10-16T20:36:33.746+0000 7f430cd4a640 -1 Falling back to public interface 2023-10-16T20:36:33.755 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-10-16T20:36:33.746+0000 7f5fd71a1640 -1 Falling back to public interface 2023-10-16T20:36:33.755 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-10-16T20:36:33.746+0000 7f2021586640 -1 Falling back to public interface 2023-10-16T20:36:33.756 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-10-16T20:36:33.746+0000 7f5fd6162640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6808/0: (98) Address already in use 2023-10-16T20:36:33.760 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:33.757+0000 7f0f19b7e180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T20:36:33.762 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-10-16T20:36:33.754+0000 7f7672246640 -1 Falling back to public interface 2023-10-16T20:36:33.772 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-10-16T20:36:33.769+0000 7fafef365640 -1 Falling back to public interface 2023-10-16T20:36:33.788 INFO:tasks.ceph.osd.6.smithi191.stderr:2023-10-16T20:36:33.785+0000 7fd54bf94640 -1 Falling back to public interface 2023-10-16T20:36:33.808 INFO:tasks.ceph.osd.7.smithi191.stderr:2023-10-16T20:36:33.805+0000 7faf15680640 -1 Falling back to public interface 2023-10-16T20:36:33.889 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:33.878+0000 7fd450c13180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T20:36:33.947 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:33.941+0000 7f0f19b7e180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T20:36:33.962 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:33.954+0000 7fd450c13180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T20:36:34.019 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:34.013+0000 7f0f19b7e180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T20:36:34.101 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:34.094+0000 7fd450c13180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T20:36:34.157 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:34.153+0000 7f0f19b7e180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T20:36:34.204 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-16T20:36:34.331 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.322+0000 7f9fd4854640 1 Processor -- start 2023-10-16T20:36:34.331 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.322+0000 7f9fd4854640 1 -- start start 2023-10-16T20:36:34.333 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.322+0000 7f9fd4854640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fd01537e0 0x7f9fd00cba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:34.333 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.322+0000 7f9fd4854640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9fd0153bc0 0x7f9fd00cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:34.334 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.322+0000 7f9fd4854640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd0072470 con 0x7f9fd00b21a0 2023-10-16T20:36:34.334 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.322+0000 7f9fd4854640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fd00b3100 con 0x7f9fd0153bc0 2023-10-16T20:36:34.334 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.322+0000 7f9fd4854640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd00b1dd0 con 0x7f9fd01537e0 2023-10-16T20:36:34.334 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.322+0000 7f9fced9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9fd00b21a0 0x7f9fd01596c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34796/0 (socket says 172.21.15.88:34796) 2023-10-16T20:36:34.334 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.322+0000 7f9fced9d640 1 -- 172.21.15.88:0/1671091400 learned_addr learned my addr 172.21.15.88:0/1671091400 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:36:34.335 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.326+0000 7f9fce59c640 1 --2- 172.21.15.88:0/1671091400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fd01537e0 0x7f9fd00cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:34.335 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.326+0000 7f9fc7fff640 1 --2- 172.21.15.88:0/1671091400 >> v2:172.21.15.191:3300/0 conn(0x7f9fd0153bc0 0x7f9fd00cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:34.335 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.326+0000 7f9fcdd9b640 1 -- 172.21.15.88:0/1671091400 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470687609 0 0) 0x7f9fd0072470 con 0x7f9fd00b21a0 2023-10-16T20:36:34.335 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.326+0000 7f9fcdd9b640 1 -- 172.21.15.88:0/1671091400 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fbc003530 con 0x7f9fd00b21a0 2023-10-16T20:36:34.335 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.326+0000 7f9fce59c640 1 -- 172.21.15.88:0/1671091400 >> v2:172.21.15.191:3300/0 conn(0x7f9fd0153bc0 msgr2=0x7f9fd00cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:34.335 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.326+0000 7f9fce59c640 1 --2- 172.21.15.88:0/1671091400 >> v2:172.21.15.191:3300/0 conn(0x7f9fd0153bc0 0x7f9fd00cbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:34.336 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.326+0000 7f9fce59c640 1 -- 172.21.15.88:0/1671091400 >> v1:172.21.15.88:6789/0 conn(0x7f9fd00b21a0 legacy=0x7f9fd01596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:34.336 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.326+0000 7f9fce59c640 1 -- 172.21.15.88:0/1671091400 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd00b3310 con 0x7f9fd01537e0 2023-10-16T20:36:34.336 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.326+0000 7f9fce59c640 1 --2- 172.21.15.88:0/1671091400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fd01537e0 0x7f9fd00cba00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00097c0 tx=0x7f9fc001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc374f5ef59e00d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:34.337 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.326+0000 7f9fcdd9b640 1 -- 172.21.15.88:0/1671091400 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc00a8970 con 0x7f9fd01537e0 2023-10-16T20:36:34.337 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.326+0000 7f9fcdd9b640 1 -- 172.21.15.88:0/1671091400 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc00a8ad0 con 0x7f9fd01537e0 2023-10-16T20:36:34.337 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.326+0000 7f9fcdd9b640 1 -- 172.21.15.88:0/1671091400 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc00a8e60 con 0x7f9fd01537e0 2023-10-16T20:36:34.338 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fd4854640 1 -- 172.21.15.88:0/1671091400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fd01537e0 msgr2=0x7f9fd00cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:34.338 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fd4854640 1 --2- 172.21.15.88:0/1671091400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fd01537e0 0x7f9fd00cba00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00097c0 tx=0x7f9fc001ec70 comp rx=0 tx=0).stop 2023-10-16T20:36:34.338 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fd4854640 1 -- 172.21.15.88:0/1671091400 shutdown_connections 2023-10-16T20:36:34.338 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fd4854640 1 --2- 172.21.15.88:0/1671091400 >> v2:172.21.15.191:3300/0 conn(0x7f9fd0153bc0 0x7f9fd00cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:34.339 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fd4854640 1 --2- 172.21.15.88:0/1671091400 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fd01537e0 0x7f9fd00cba00 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:34.339 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fd4854640 1 -- 172.21.15.88:0/1671091400 >> 172.21.15.88:0/1671091400 conn(0x7f9fd00715c0 msgr2=0x7f9fd00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:34.339 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fd4854640 1 -- 172.21.15.88:0/1671091400 shutdown_connections 2023-10-16T20:36:34.339 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fd4854640 1 -- 172.21.15.88:0/1671091400 wait complete. 2023-10-16T20:36:34.341 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fd4854640 1 Processor -- start 2023-10-16T20:36:34.341 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fd4854640 1 -- start start 2023-10-16T20:36:34.341 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fd4854640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fd00b21a0 0x7f9fd00ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:34.341 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fd4854640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fd01537e0 0x7f9fd01e0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:34.342 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fd4854640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fd0153bc0 0x7f9fd01f19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:34.342 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fd4854640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd00c6c50 con 0x7f9fd01537e0 2023-10-16T20:36:34.342 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fd4854640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd00c66b0 con 0x7f9fd00b21a0 2023-10-16T20:36:34.342 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fd4854640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd00c6970 con 0x7f9fd0153bc0 2023-10-16T20:36:34.342 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fc7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fd01537e0 0x7f9fd01e0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:34.342 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fc7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fd01537e0 0x7f9fd01e0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56872/0 (socket says 172.21.15.88:56872) 2023-10-16T20:36:34.343 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fc7fff640 1 -- 172.21.15.88:0/291788568 learned_addr learned my addr 172.21.15.88:0/291788568 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:34.343 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fced9d640 1 --2- 172.21.15.88:0/291788568 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fd0153bc0 0x7f9fd01f19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:34.343 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.330+0000 7f9fce59c640 1 --2- 172.21.15.88:0/291788568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fd00b21a0 0x7f9fd00ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:34.343 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.334+0000 7f9fc7fff640 1 -- 172.21.15.88:0/291788568 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fd0153bc0 msgr2=0x7f9fd01f19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:34.343 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.334+0000 7f9fc7fff640 1 --2- 172.21.15.88:0/291788568 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fd0153bc0 0x7f9fd01f19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:34.344 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.334+0000 7f9fc7fff640 1 -- 172.21.15.88:0/291788568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fd00b21a0 msgr2=0x7f9fd00ccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:34.344 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.334+0000 7f9fc7fff640 1 --2- 172.21.15.88:0/291788568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fd00b21a0 0x7f9fd00ccc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:34.344 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.334+0000 7f9fc7fff640 1 -- 172.21.15.88:0/291788568 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc0009420 con 0x7f9fd01537e0 2023-10-16T20:36:34.345 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.334+0000 7f9fc7fff640 1 --2- 172.21.15.88:0/291788568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fd01537e0 0x7f9fd01e0a20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9fb800e870 tx=0x7f9fb800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:34.345 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.334+0000 7f9fc6ffd640 1 -- 172.21.15.88:0/291788568 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb8009510 con 0x7f9fd01537e0 2023-10-16T20:36:34.345 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.334+0000 7f9fc6ffd640 1 -- 172.21.15.88:0/291788568 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb800eec0 con 0x7f9fd01537e0 2023-10-16T20:36:34.346 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.334+0000 7f9fc6ffd640 1 -- 172.21.15.88:0/291788568 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb800cb40 con 0x7f9fd01537e0 2023-10-16T20:36:34.346 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.334+0000 7f9fd4854640 1 -- 172.21.15.88:0/291788568 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd00c76b0 con 0x7f9fd01537e0 2023-10-16T20:36:34.346 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.334+0000 7f9fd4854640 1 -- 172.21.15.88:0/291788568 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd00c7c70 con 0x7f9fd01537e0 2023-10-16T20:36:34.346 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.334+0000 7f9fc6ffd640 1 -- 172.21.15.88:0/291788568 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9fb80059a0 con 0x7f9fd01537e0 2023-10-16T20:36:34.346 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.334+0000 7f9fc6ffd640 1 -- 172.21.15.88:0/291788568 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f9fb80067a0 con 0x7f9fd01537e0 2023-10-16T20:36:34.347 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.334+0000 7f9fd4854640 1 -- 172.21.15.88:0/291788568 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fd00c7e30 con 0x7f9fd01537e0 2023-10-16T20:36:34.347 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.338+0000 7f9fc6ffd640 1 -- 172.21.15.88:0/291788568 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f9fb8021060 con 0x7f9fd01537e0 2023-10-16T20:36:34.400 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.390+0000 7f9fd4854640 1 -- 172.21.15.88:0/291788568 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fd004e9c0 con 0x7f9fd01537e0 2023-10-16T20:36:34.401 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.394+0000 7f9fc6ffd640 1 -- 172.21.15.88:0/291788568 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f9fb8017070 con 0x7f9fd01537e0 2023-10-16T20:36:34.402 INFO:teuthology.misc.health.smithi088.stdout: 2023-10-16T20:36:34.402 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:36:33.014460+0000","last_up_change":"0.000000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:36:34.403 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.394+0000 7f9fd4854640 1 -- 172.21.15.88:0/291788568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fd01537e0 msgr2=0x7f9fd01e0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:34.403 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.394+0000 7f9fd4854640 1 --2- 172.21.15.88:0/291788568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fd01537e0 0x7f9fd01e0a20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9fb800e870 tx=0x7f9fb800ec70 comp rx=0 tx=0).stop 2023-10-16T20:36:34.403 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.394+0000 7f9fd4854640 1 -- 172.21.15.88:0/291788568 shutdown_connections 2023-10-16T20:36:34.403 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.394+0000 7f9fd4854640 1 --2- 172.21.15.88:0/291788568 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9fd0153bc0 0x7f9fd01f19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:34.404 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.394+0000 7f9fd4854640 1 --2- 172.21.15.88:0/291788568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fd01537e0 0x7f9fd01e0a20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:34.404 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.394+0000 7f9fd4854640 1 --2- 172.21.15.88:0/291788568 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9fd00b21a0 0x7f9fd00ccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:34.404 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.394+0000 7f9fd4854640 1 -- 172.21.15.88:0/291788568 >> 172.21.15.88:0/291788568 conn(0x7f9fd00715c0 msgr2=0x7f9fd0150f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:34.404 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.394+0000 7f9fd4854640 1 -- 172.21.15.88:0/291788568 shutdown_connections 2023-10-16T20:36:34.404 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:34.394+0000 7f9fd4854640 1 -- 172.21.15.88:0/291788568 wait complete. 2023-10-16T20:36:34.419 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-16T20:36:34.811 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:34.802+0000 7fd450c13180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T20:36:34.848 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:34.845+0000 7f0f19b7e180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T20:36:34.898 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:34.890+0000 7fd450c13180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T20:36:34.933 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:34.929+0000 7f0f19b7e180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T20:36:35.091 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:35.082+0000 7fd450c13180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T20:36:35.124 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:35.121+0000 7f0f19b7e180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T20:36:35.176 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:35.166+0000 7fd450c13180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T20:36:35.207 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:35.205+0000 7f0f19b7e180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T20:36:35.506 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:35.498+0000 7fd450c13180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T20:36:35.535 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:35.533+0000 7f0f19b7e180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T20:36:35.574 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:35.566+0000 7fd450c13180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T20:36:35.606 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-10-16T20:36:35.601+0000 7f0f19b7e180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T20:36:36.598 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:36.590+0000 7fd3f847a640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-16T20:36:36.598 INFO:tasks.ceph.mgr.y.smithi088.stderr:2023-10-16T20:36:36.590+0000 7fd3f847a640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-16T20:36:37.571 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-10-16T20:36:37.562+0000 7f430cd4a640 -1 osd.0 0 log_to_monitors true 2023-10-16T20:36:37.606 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-10-16T20:36:37.601+0000 7f7cdfa2d640 -1 osd.4 0 log_to_monitors true 2023-10-16T20:36:37.751 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-10-16T20:36:37.749+0000 7fafef365640 -1 osd.5 0 log_to_monitors true 2023-10-16T20:36:37.761 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-10-16T20:36:37.754+0000 7f2021586640 -1 osd.2 0 log_to_monitors true 2023-10-16T20:36:37.766 INFO:tasks.ceph.osd.7.smithi191.stderr:2023-10-16T20:36:37.761+0000 7faf15680640 -1 osd.7 0 log_to_monitors true 2023-10-16T20:36:37.780 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-10-16T20:36:37.770+0000 7f5fd71a1640 -1 osd.1 0 log_to_monitors true 2023-10-16T20:36:37.783 INFO:tasks.ceph.osd.6.smithi191.stderr:2023-10-16T20:36:37.781+0000 7fd54bf94640 -1 osd.6 0 log_to_monitors true 2023-10-16T20:36:37.801 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-10-16T20:36:37.794+0000 7f7672246640 -1 osd.3 0 log_to_monitors true 2023-10-16T20:36:39.519 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-10-16T20:36:39.511+0000 7f4308cf2640 -1 osd.0 0 waiting for initial osdmap 2023-10-16T20:36:39.545 INFO:tasks.ceph.osd.0.smithi088.stderr:2023-10-16T20:36:39.539+0000 7f4300b27640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T20:36:41.225 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-16T20:36:41.409 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-10-16T20:36:41.399+0000 7f5fd3149640 -1 osd.1 0 waiting for initial osdmap 2023-10-16T20:36:41.435 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.427+0000 7fe1e92c9640 1 Processor -- start 2023-10-16T20:36:41.435 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.427+0000 7fe1e92c9640 1 -- start start 2023-10-16T20:36:41.436 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.427+0000 7fe1e92c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e4153bc0 0x7fe1e415cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:41.436 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.431+0000 7fe1e92c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe1e40cc110 0x7fe1e415d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:41.437 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.431+0000 7fe1e92c9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e4072470 con 0x7fe1e41537e0 2023-10-16T20:36:41.437 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.431+0000 7fe1e92c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1e40b3100 con 0x7fe1e40cc110 2023-10-16T20:36:41.437 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.431+0000 7fe1e92c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e40b1dd0 con 0x7fe1e4153bc0 2023-10-16T20:36:41.438 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.431+0000 7fe1e37fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe1e40cc110 0x7fe1e415d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:41.438 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.431+0000 7fe1e37fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe1e40cc110 0x7fe1e415d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49544/0 (socket says 172.21.15.88:49544) 2023-10-16T20:36:41.438 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.431+0000 7fe1e37fe640 1 -- 172.21.15.88:0/3698736047 learned_addr learned my addr 172.21.15.88:0/3698736047 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:41.438 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.431+0000 7fe1e27fc640 1 --2- 172.21.15.88:0/3698736047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e4153bc0 0x7fe1e415cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:41.439 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.431+0000 7fe1e27fc640 1 -- 172.21.15.88:0/3698736047 >> v2:172.21.15.88:3300/0 conn(0x7fe1e40cc110 msgr2=0x7fe1e415d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:41.439 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.431+0000 7fe1e27fc640 1 --2- 172.21.15.88:0/3698736047 >> v2:172.21.15.88:3300/0 conn(0x7fe1e40cc110 0x7fe1e415d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:41.439 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.431+0000 7fe1e27fc640 1 -- 172.21.15.88:0/3698736047 >> v1:172.21.15.191:6789/0 conn(0x7fe1e41537e0 legacy=0x7fe1e40cba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:41.439 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.431+0000 7fe1e27fc640 1 -- 172.21.15.88:0/3698736047 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e40b3310 con 0x7fe1e4153bc0 2023-10-16T20:36:41.441 INFO:tasks.ceph.osd.1.smithi088.stderr:2023-10-16T20:36:41.435+0000 7f5fcaf7e640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T20:36:41.443 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.435+0000 7fe1e27fc640 1 --2- 172.21.15.88:0/3698736047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e4153bc0 0x7fe1e415cc00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe1d80097c0 tx=0x7fe1d801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dad4256bd0a1f6e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:41.443 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.435+0000 7fe1e1ffb640 1 -- 172.21.15.88:0/3698736047 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d80a8970 con 0x7fe1e4153bc0 2023-10-16T20:36:41.444 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.435+0000 7fe1e1ffb640 1 -- 172.21.15.88:0/3698736047 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d80a8ad0 con 0x7fe1e4153bc0 2023-10-16T20:36:41.444 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.435+0000 7fe1e1ffb640 1 -- 172.21.15.88:0/3698736047 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d80a8e60 con 0x7fe1e4153bc0 2023-10-16T20:36:41.444 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.439+0000 7fe1e92c9640 1 -- 172.21.15.88:0/3698736047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e4153bc0 msgr2=0x7fe1e415cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:41.445 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.439+0000 7fe1e92c9640 1 --2- 172.21.15.88:0/3698736047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e4153bc0 0x7fe1e415cc00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe1d80097c0 tx=0x7fe1d801ec70 comp rx=0 tx=0).stop 2023-10-16T20:36:41.445 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.439+0000 7fe1e92c9640 1 -- 172.21.15.88:0/3698736047 shutdown_connections 2023-10-16T20:36:41.445 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.439+0000 7fe1e92c9640 1 --2- 172.21.15.88:0/3698736047 >> v2:172.21.15.88:3300/0 conn(0x7fe1e40cc110 0x7fe1e415d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:41.445 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.439+0000 7fe1e92c9640 1 --2- 172.21.15.88:0/3698736047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e4153bc0 0x7fe1e415cc00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:41.446 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.439+0000 7fe1e92c9640 1 -- 172.21.15.88:0/3698736047 >> 172.21.15.88:0/3698736047 conn(0x7fe1e40715c0 msgr2=0x7fe1e40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:41.446 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.439+0000 7fe1e92c9640 1 -- 172.21.15.88:0/3698736047 shutdown_connections 2023-10-16T20:36:41.446 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.439+0000 7fe1e92c9640 1 -- 172.21.15.88:0/3698736047 wait complete. 2023-10-16T20:36:41.447 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.439+0000 7fe1e92c9640 1 Processor -- start 2023-10-16T20:36:41.447 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.439+0000 7fe1e92c9640 1 -- start start 2023-10-16T20:36:41.449 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1e92c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1e40cc110 0x7fe1e41f63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:41.449 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1e92c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e41537e0 0x7fe1e41f6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:41.449 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1e92c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1e4153bc0 0x7fe1e4159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:41.449 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1e92c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e4158db0 con 0x7fe1e40cc110 2023-10-16T20:36:41.449 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1e92c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e4157a20 con 0x7fe1e4153bc0 2023-10-16T20:36:41.450 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1e92c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e41563f0 con 0x7fe1e41537e0 2023-10-16T20:36:41.450 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1e27fc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e41537e0 0x7fe1e41f6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:41.450 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1e27fc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e41537e0 0x7fe1e41f6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:45262/0 (socket says 172.21.15.88:45262) 2023-10-16T20:36:41.450 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1e27fc640 1 -- 172.21.15.88:0/2092049865 learned_addr learned my addr 172.21.15.88:0/2092049865 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:41.451 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1e27fc640 1 -- 172.21.15.88:0/2092049865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1e4153bc0 msgr2=0x7fe1e4159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:41.451 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1e27fc640 1 --2- 172.21.15.88:0/2092049865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1e4153bc0 0x7fe1e4159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:41.452 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1e27fc640 1 -- 172.21.15.88:0/2092049865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1e40cc110 msgr2=0x7fe1e41f63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:41.452 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1e27fc640 1 --2- 172.21.15.88:0/2092049865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1e40cc110 0x7fe1e41f63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:41.452 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1e27fc640 1 -- 172.21.15.88:0/2092049865 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d8009420 con 0x7fe1e41537e0 2023-10-16T20:36:41.452 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1e27fc640 1 --2- 172.21.15.88:0/2092049865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e41537e0 0x7fe1e41f6900 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe1d801d320 tx=0x7fe1d801ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:41.452 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1cb7fe640 1 -- 172.21.15.88:0/2092049865 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d80b0cf0 con 0x7fe1e41537e0 2023-10-16T20:36:41.452 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1cb7fe640 1 -- 172.21.15.88:0/2092049865 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d80b0e50 con 0x7fe1e41537e0 2023-10-16T20:36:41.453 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1cb7fe640 1 -- 172.21.15.88:0/2092049865 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d80ba660 con 0x7fe1e41537e0 2023-10-16T20:36:41.453 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1e92c9640 1 -- 172.21.15.88:0/2092049865 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e4158810 con 0x7fe1e41537e0 2023-10-16T20:36:41.453 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1e92c9640 1 -- 172.21.15.88:0/2092049865 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e4158ad0 con 0x7fe1e41537e0 2023-10-16T20:36:41.454 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.443+0000 7fe1c97fa640 1 -- 172.21.15.88:0/2092049865 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1bc005260 con 0x7fe1e41537e0 2023-10-16T20:36:41.454 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.447+0000 7fe1cb7fe640 1 -- 172.21.15.88:0/2092049865 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 4) v1 ==== 85794+0+0 (secure 0 0 0) 0x7fe1d80ba7c0 con 0x7fe1e41537e0 2023-10-16T20:36:41.455 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.447+0000 7fe1cb7fe640 1 --2- 172.21.15.88:0/2092049865 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe1d00663e0 0x7fe1d00688a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:41.455 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.447+0000 7fe1cb7fe640 1 -- 172.21.15.88:0/2092049865 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2511+0+0 (secure 0 0 0) 0x7fe1d8121960 con 0x7fe1e41537e0 2023-10-16T20:36:41.456 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.447+0000 7fe1e2ffd640 1 --2- 172.21.15.88:0/2092049865 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe1d00663e0 0x7fe1d00688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:41.457 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.451+0000 7fe1e2ffd640 1 --2- 172.21.15.88:0/2092049865 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe1d00663e0 0x7fe1d00688a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4000f70 tx=0x7fe1d40079e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:41.461 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.455+0000 7fe1cb7fe640 1 -- 172.21.15.88:0/2092049865 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe1d80efd40 con 0x7fe1e41537e0 2023-10-16T20:36:41.546 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-10-16T20:36:41.539+0000 7f766e1ee640 -1 osd.3 0 waiting for initial osdmap 2023-10-16T20:36:41.568 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-10-16T20:36:41.559+0000 7f201d52e640 -1 osd.2 0 waiting for initial osdmap 2023-10-16T20:36:41.589 INFO:tasks.ceph.osd.3.smithi088.stderr:2023-10-16T20:36:41.583+0000 7f7666023640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T20:36:41.596 INFO:tasks.ceph.osd.2.smithi088.stderr:2023-10-16T20:36:41.591+0000 7f2015363640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T20:36:41.634 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.627+0000 7fe1c97fa640 1 -- 172.21.15.88:0/2092049865 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1bc0057e0 con 0x7fe1e41537e0 2023-10-16T20:36:41.635 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.627+0000 7fe1cb7fe640 1 -- 172.21.15.88:0/2092049865 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+7577 (secure 0 0 0) 0x7fe1d80b5070 con 0x7fe1e41537e0 2023-10-16T20:36:41.636 INFO:teuthology.misc.health.smithi088.stdout: 2023-10-16T20:36:41.636 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":14,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:36:41.353337+0000","last_up_change":"2023-10-16T20:36:39.593768+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:36:41.637 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.631+0000 7fe1c97fa640 1 -- 172.21.15.88:0/2092049865 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe1d00663e0 msgr2=0x7fe1d00688a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:41.638 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.631+0000 7fe1c97fa640 1 --2- 172.21.15.88:0/2092049865 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe1d00663e0 0x7fe1d00688a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4000f70 tx=0x7fe1d40079e0 comp rx=0 tx=0).stop 2023-10-16T20:36:41.638 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.631+0000 7fe1c97fa640 1 -- 172.21.15.88:0/2092049865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e41537e0 msgr2=0x7fe1e41f6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:41.638 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.631+0000 7fe1c97fa640 1 --2- 172.21.15.88:0/2092049865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e41537e0 0x7fe1e41f6900 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe1d801d320 tx=0x7fe1d801ba10 comp rx=0 tx=0).stop 2023-10-16T20:36:41.638 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.631+0000 7fe1c97fa640 1 -- 172.21.15.88:0/2092049865 shutdown_connections 2023-10-16T20:36:41.639 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.631+0000 7fe1c97fa640 1 --2- 172.21.15.88:0/2092049865 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe1d00663e0 0x7fe1d00688a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:41.639 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.631+0000 7fe1c97fa640 1 --2- 172.21.15.88:0/2092049865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1e4153bc0 0x7fe1e4159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:41.639 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.631+0000 7fe1c97fa640 1 --2- 172.21.15.88:0/2092049865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe1e41537e0 0x7fe1e41f6900 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:41.639 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.631+0000 7fe1c97fa640 1 --2- 172.21.15.88:0/2092049865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1e40cc110 0x7fe1e41f63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:41.639 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.631+0000 7fe1c97fa640 1 -- 172.21.15.88:0/2092049865 >> 172.21.15.88:0/2092049865 conn(0x7fe1e40715c0 msgr2=0x7fe1e4155be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:41.640 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.631+0000 7fe1c97fa640 1 -- 172.21.15.88:0/2092049865 shutdown_connections 2023-10-16T20:36:41.640 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:41.631+0000 7fe1c97fa640 1 -- 172.21.15.88:0/2092049865 wait complete. 2023-10-16T20:36:41.655 DEBUG:teuthology.misc:1 of 8 OSDs are up 2023-10-16T20:36:41.997 INFO:tasks.ceph.osd.7.smithi191.stderr:2023-10-16T20:36:41.993+0000 7faf1162a640 -1 osd.7 0 waiting for initial osdmap 2023-10-16T20:36:42.040 INFO:tasks.ceph.osd.7.smithi191.stderr:2023-10-16T20:36:42.037+0000 7faf0945f640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T20:36:42.064 INFO:tasks.ceph.osd.6.smithi191.stderr:2023-10-16T20:36:42.061+0000 7fd547f3c640 -1 osd.6 0 waiting for initial osdmap 2023-10-16T20:36:42.097 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-10-16T20:36:42.093+0000 7f7cdb9d5640 -1 osd.4 0 waiting for initial osdmap 2023-10-16T20:36:42.103 INFO:tasks.ceph.osd.6.smithi191.stderr:2023-10-16T20:36:42.101+0000 7fd53fd71640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T20:36:42.130 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-10-16T20:36:42.125+0000 7fafeb30d640 -1 osd.5 0 waiting for initial osdmap 2023-10-16T20:36:42.139 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-10-16T20:36:42.137+0000 7f7cd380a640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T20:36:42.159 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-10-16T20:36:42.157+0000 7fafe3142640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-16T20:36:48.461 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-16T20:36:48.593 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f259d30f640 1 Processor -- start 2023-10-16T20:36:48.593 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f259d30f640 1 -- start start 2023-10-16T20:36:48.595 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f259d30f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f25981537e0 0x7f25980cba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:48.595 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f259d30f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2598153bc0 0x7f25980cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:48.596 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f259d30f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f25980b21a0 0x7f25981596c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:48.596 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f259d30f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25980b3100 con 0x7f25981537e0 2023-10-16T20:36:48.596 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f259d30f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2598072470 con 0x7f2598153bc0 2023-10-16T20:36:48.596 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f259d30f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25980b1dd0 con 0x7f25980b21a0 2023-10-16T20:36:48.596 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f2596ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f25981537e0 0x7f25980cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:48.597 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f2596ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f25981537e0 0x7f25980cba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49584/0 (socket says 172.21.15.88:49584) 2023-10-16T20:36:48.597 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f2596ffd640 1 -- 172.21.15.88:0/17042948 learned_addr learned my addr 172.21.15.88:0/17042948 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:48.597 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f258e7fc640 1 --2- 172.21.15.88:0/17042948 >> v2:172.21.15.191:3300/0 conn(0x7f2598153bc0 0x7f25980cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:48.597 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f2596ffd640 1 -- 172.21.15.88:0/17042948 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f25980b21a0 msgr2=0x7f25981596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:48.597 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f2596ffd640 1 --2- 172.21.15.88:0/17042948 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f25980b21a0 0x7f25981596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:48.598 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f2596ffd640 1 -- 172.21.15.88:0/17042948 >> v2:172.21.15.191:3300/0 conn(0x7f2598153bc0 msgr2=0x7f25980cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:48.598 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f2596ffd640 1 --2- 172.21.15.88:0/17042948 >> v2:172.21.15.191:3300/0 conn(0x7f2598153bc0 0x7f25980cbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:48.598 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f2596ffd640 1 -- 172.21.15.88:0/17042948 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25980b3310 con 0x7f25981537e0 2023-10-16T20:36:48.598 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.587+0000 7f258e7fc640 1 --2- 172.21.15.88:0/17042948 >> v2:172.21.15.191:3300/0 conn(0x7f2598153bc0 0x7f25980cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:36:48.600 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.591+0000 7f2596ffd640 1 --2- 172.21.15.88:0/17042948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25981537e0 0x7f25980cba00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f25981572f0 tx=0x7f258801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607499878c4662da server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:48.600 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.591+0000 7f25967fc640 1 -- 172.21.15.88:0/17042948 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25880a89b0 con 0x7f25981537e0 2023-10-16T20:36:48.600 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.591+0000 7f25967fc640 1 -- 172.21.15.88:0/17042948 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25880a8b10 con 0x7f25981537e0 2023-10-16T20:36:48.600 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.591+0000 7f25967fc640 1 -- 172.21.15.88:0/17042948 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25880a8ea0 con 0x7f25981537e0 2023-10-16T20:36:48.601 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.591+0000 7f259d30f640 1 -- 172.21.15.88:0/17042948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25981537e0 msgr2=0x7f25980cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:48.601 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.591+0000 7f259d30f640 1 --2- 172.21.15.88:0/17042948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25981537e0 0x7f25980cba00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f25981572f0 tx=0x7f258801ecf0 comp rx=0 tx=0).stop 2023-10-16T20:36:48.601 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.591+0000 7f259d30f640 1 -- 172.21.15.88:0/17042948 shutdown_connections 2023-10-16T20:36:48.601 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.591+0000 7f259d30f640 1 --2- 172.21.15.88:0/17042948 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f25980b21a0 0x7f25981596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:48.602 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.591+0000 7f259d30f640 1 --2- 172.21.15.88:0/17042948 >> v2:172.21.15.191:3300/0 conn(0x7f2598153bc0 0x7f25980cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:48.602 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.591+0000 7f259d30f640 1 --2- 172.21.15.88:0/17042948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25981537e0 0x7f25980cba00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:48.602 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.591+0000 7f259d30f640 1 -- 172.21.15.88:0/17042948 >> 172.21.15.88:0/17042948 conn(0x7f25980715c0 msgr2=0x7f25980719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:48.602 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.591+0000 7f259d30f640 1 -- 172.21.15.88:0/17042948 shutdown_connections 2023-10-16T20:36:48.603 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.591+0000 7f259d30f640 1 -- 172.21.15.88:0/17042948 wait complete. 2023-10-16T20:36:48.606 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f259d30f640 1 Processor -- start 2023-10-16T20:36:48.606 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f259d30f640 1 -- start start 2023-10-16T20:36:48.606 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f259d30f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f25980b21a0 0x7f25980c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:48.607 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f259d30f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f25981537e0 0x7f25980c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:48.607 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f259d30f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2598153bc0 0x7f25980c7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:48.607 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f259d30f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25980c63f0 con 0x7f2598153bc0 2023-10-16T20:36:48.607 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f259d30f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25980c6f50 con 0x7f25980b21a0 2023-10-16T20:36:48.607 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f259d30f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25980c6620 con 0x7f25981537e0 2023-10-16T20:36:48.608 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f25977fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2598153bc0 0x7f25980c7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:48.608 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f25977fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2598153bc0 0x7f25980c7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49596/0 (socket says 172.21.15.88:49596) 2023-10-16T20:36:48.608 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f25977fe640 1 -- 172.21.15.88:0/3037068655 learned_addr learned my addr 172.21.15.88:0/3037068655 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:48.608 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f2596ffd640 1 --2- 172.21.15.88:0/3037068655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f25980b21a0 0x7f25980c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:48.608 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f258e7fc640 1 --2- 172.21.15.88:0/3037068655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f25981537e0 0x7f25980c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:48.609 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f25977fe640 1 -- 172.21.15.88:0/3037068655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f25981537e0 msgr2=0x7f25980c73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:48.609 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f25977fe640 1 --2- 172.21.15.88:0/3037068655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f25981537e0 0x7f25980c73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:48.609 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f25977fe640 1 -- 172.21.15.88:0/3037068655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f25980b21a0 msgr2=0x7f25980c86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:48.609 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f25977fe640 1 --2- 172.21.15.88:0/3037068655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f25980b21a0 0x7f25980c86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:48.610 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f25977fe640 1 -- 172.21.15.88:0/3037068655 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25880094d0 con 0x7f2598153bc0 2023-10-16T20:36:48.610 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f25977fe640 1 --2- 172.21.15.88:0/3037068655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2598153bc0 0x7f25980c7920 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f259000b3c0 tx=0x7f259000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:48.610 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f258ffff640 1 -- 172.21.15.88:0/3037068655 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2590011830 con 0x7f2598153bc0 2023-10-16T20:36:48.610 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f258ffff640 1 -- 172.21.15.88:0/3037068655 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2590011990 con 0x7f2598153bc0 2023-10-16T20:36:48.611 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f258ffff640 1 -- 172.21.15.88:0/3037068655 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2590011d40 con 0x7f2598153bc0 2023-10-16T20:36:48.611 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f259d30f640 1 -- 172.21.15.88:0/3037068655 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25980c6900 con 0x7f2598153bc0 2023-10-16T20:36:48.611 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.595+0000 7f259d30f640 1 -- 172.21.15.88:0/3037068655 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25981eea40 con 0x7f2598153bc0 2023-10-16T20:36:48.611 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.599+0000 7f259d30f640 1 -- 172.21.15.88:0/3037068655 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25980c7100 con 0x7f2598153bc0 2023-10-16T20:36:48.611 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.599+0000 7f258ffff640 1 -- 172.21.15.88:0/3037068655 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f259001b460 con 0x7f2598153bc0 2023-10-16T20:36:48.612 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.599+0000 7f258ffff640 1 --2- 172.21.15.88:0/3037068655 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f25780664a0 0x7f2578068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:48.612 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.599+0000 7f258ffff640 1 -- 172.21.15.88:0/3037068655 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f25900838b0 con 0x7f2598153bc0 2023-10-16T20:36:48.612 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.599+0000 7f2596ffd640 1 --2- 172.21.15.88:0/3037068655 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f25780664a0 0x7f2578068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:48.613 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.599+0000 7f2596ffd640 1 --2- 172.21.15.88:0/3037068655 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f25780664a0 0x7f2578068960 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f25880b2000 tx=0x7f2588009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:48.613 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.603+0000 7f258ffff640 1 -- 172.21.15.88:0/3037068655 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f25980c7100 con 0x7f2598153bc0 2023-10-16T20:36:48.763 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.755+0000 7f259d30f640 1 -- 172.21.15.88:0/3037068655 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25980c7100 con 0x7f2598153bc0 2023-10-16T20:36:48.764 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.759+0000 7f258ffff640 1 -- 172.21.15.88:0/3037068655 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+12826 (secure 0 0 0) 0x7f25980c7100 con 0x7f2598153bc0 2023-10-16T20:36:48.764 INFO:teuthology.misc.health.smithi088.stdout: 2023-10-16T20:36:48.765 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":19,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:36:47.255863+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:36:48.766 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.759+0000 7f259d30f640 1 -- 172.21.15.88:0/3037068655 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f25780664a0 msgr2=0x7f2578068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:48.767 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.759+0000 7f259d30f640 1 --2- 172.21.15.88:0/3037068655 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f25780664a0 0x7f2578068960 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f25880b2000 tx=0x7f2588009040 comp rx=0 tx=0).stop 2023-10-16T20:36:48.767 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.759+0000 7f259d30f640 1 -- 172.21.15.88:0/3037068655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2598153bc0 msgr2=0x7f25980c7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:48.767 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.759+0000 7f259d30f640 1 --2- 172.21.15.88:0/3037068655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2598153bc0 0x7f25980c7920 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f259000b3c0 tx=0x7f259000b880 comp rx=0 tx=0).stop 2023-10-16T20:36:48.767 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.759+0000 7f259d30f640 1 -- 172.21.15.88:0/3037068655 shutdown_connections 2023-10-16T20:36:48.768 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.759+0000 7f259d30f640 1 --2- 172.21.15.88:0/3037068655 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f25780664a0 0x7f2578068960 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:48.768 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.759+0000 7f259d30f640 1 --2- 172.21.15.88:0/3037068655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2598153bc0 0x7f25980c7920 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:48.768 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.759+0000 7f259d30f640 1 --2- 172.21.15.88:0/3037068655 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f25981537e0 0x7f25980c73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:48.768 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.759+0000 7f259d30f640 1 --2- 172.21.15.88:0/3037068655 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f25980b21a0 0x7f25980c86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:48.769 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.759+0000 7f259d30f640 1 -- 172.21.15.88:0/3037068655 >> 172.21.15.88:0/3037068655 conn(0x7f25980715c0 msgr2=0x7f25980ca110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:48.769 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.759+0000 7f259d30f640 1 -- 172.21.15.88:0/3037068655 shutdown_connections 2023-10-16T20:36:48.769 INFO:teuthology.misc.health.smithi088.stderr:2023-10-16T20:36:48.759+0000 7f259d30f640 1 -- 172.21.15.88:0/3037068655 wait complete. 2023-10-16T20:36:48.784 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-16T20:36:48.785 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-16T20:36:48.785 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T20:36:48.786 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T20:36:48.789 INFO:tasks.ceph.mds.a:Started 2023-10-16T20:36:48.790 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T20:36:48.790 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T20:36:48.794 INFO:tasks.ceph.mds.c:Started 2023-10-16T20:36:48.794 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-16T20:36:48.794 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-16T20:36:48.798 INFO:tasks.ceph.mds.e:Started 2023-10-16T20:36:48.798 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T20:36:48.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T20:36:48.803 INFO:tasks.ceph.mds.b:Started 2023-10-16T20:36:48.803 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T20:36:48.804 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T20:36:48.808 INFO:tasks.ceph.mds.d:Started 2023-10-16T20:36:48.808 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-16T20:36:48.809 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-16T20:36:48.812 INFO:tasks.ceph.mds.f:Started 2023-10-16T20:36:48.813 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-16T20:36:48.813 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:36:48.895 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2023-10-16T20:36:48.908 INFO:tasks.ceph.mds.d.smithi191.stdout:starting mds.d at 2023-10-16T20:36:48.910 INFO:tasks.ceph.mds.e.smithi088.stdout:starting mds.e at 2023-10-16T20:36:48.912 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2023-10-16T20:36:48.915 INFO:tasks.ceph.mds.f.smithi191.stdout:starting mds.f at 2023-10-16T20:36:48.920 INFO:tasks.ceph.mds.b.smithi191.stdout:starting mds.b at 2023-10-16T20:36:48.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.959+0000 7f898f1e4640 1 Processor -- start 2023-10-16T20:36:48.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.963+0000 7f898f1e4640 1 -- start start 2023-10-16T20:36:48.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.963+0000 7f898f1e4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89880cead0 0x7f89880cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:48.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.963+0000 7f898f1e4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89880cae10 con 0x7f89880ca2f0 2023-10-16T20:36:48.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.963+0000 7f898f1e4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89880cac30 con 0x7f89881537c0 2023-10-16T20:36:48.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.963+0000 7f898f1e4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898815d2b0 con 0x7f89880cead0 2023-10-16T20:36:48.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.963+0000 7f898d9e1640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f89880ca2f0 0x7f89880c4c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46848/0 (socket says 172.21.15.88:46848) 2023-10-16T20:36:48.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.963+0000 7f898d9e1640 1 -- 172.21.15.88:0/758122858 learned_addr learned my addr 172.21.15.88:0/758122858 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:36:48.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.963+0000 7f898e9e3640 1 --2- 172.21.15.88:0/758122858 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89880cead0 0x7f89880cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:48.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.963+0000 7f898d1e0640 1 -- 172.21.15.88:0/758122858 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549508130 0 0) 0x7f89880cae10 con 0x7f89880ca2f0 2023-10-16T20:36:48.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.963+0000 7f898d1e0640 1 -- 172.21.15.88:0/758122858 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8974003530 con 0x7f89880ca2f0 2023-10-16T20:36:48.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.963+0000 7f898d1e0640 1 -- 172.21.15.88:0/758122858 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261797680 0 0) 0x7f89880cac30 con 0x7f89881537c0 2023-10-16T20:36:48.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.963+0000 7f898d1e0640 1 -- 172.21.15.88:0/758122858 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89880cae10 con 0x7f89881537c0 2023-10-16T20:36:48.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.963+0000 7f898e9e3640 1 -- 172.21.15.88:0/758122858 >> v1:172.21.15.191:6789/0 conn(0x7f89881537c0 legacy=0x7f898815c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:48.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.963+0000 7f898e9e3640 1 -- 172.21.15.88:0/758122858 >> v1:172.21.15.88:6789/0 conn(0x7f89880ca2f0 legacy=0x7f89880c4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:48.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.963+0000 7f898e9e3640 1 -- 172.21.15.88:0/758122858 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898815d590 con 0x7f89880cead0 2023-10-16T20:36:48.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.967+0000 7f898e9e3640 1 --2- 172.21.15.88:0/758122858 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89880cead0 0x7f89880cef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8980009dc0 tx=0x7f898001c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a4e9b6cf443ab8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:48.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.967+0000 7f898d1e0640 1 -- 172.21.15.88:0/758122858 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89800a5a00 con 0x7f89880cead0 2023-10-16T20:36:48.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.967+0000 7f898d1e0640 1 -- 172.21.15.88:0/758122858 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89800a5b60 con 0x7f89880cead0 2023-10-16T20:36:48.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.967+0000 7f898d1e0640 1 -- 172.21.15.88:0/758122858 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89800ad440 con 0x7f89880cead0 2023-10-16T20:36:48.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.967+0000 7f898f1e4640 1 -- 172.21.15.88:0/758122858 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89880cead0 msgr2=0x7f89880cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:48.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.967+0000 7f898f1e4640 1 --2- 172.21.15.88:0/758122858 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89880cead0 0x7f89880cef20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8980009dc0 tx=0x7f898001c940 comp rx=0 tx=0).stop 2023-10-16T20:36:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.967+0000 7f898f1e4640 1 -- 172.21.15.88:0/758122858 shutdown_connections 2023-10-16T20:36:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.967+0000 7f898f1e4640 1 --2- 172.21.15.88:0/758122858 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89880cead0 0x7f89880cef20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.967+0000 7f898f1e4640 1 -- 172.21.15.88:0/758122858 >> 172.21.15.88:0/758122858 conn(0x7f898806c170 msgr2=0x7f89880715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.967+0000 7f898f1e4640 1 -- 172.21.15.88:0/758122858 shutdown_connections 2023-10-16T20:36:48.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.967+0000 7f898f1e4640 1 -- 172.21.15.88:0/758122858 wait complete. 2023-10-16T20:36:48.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898f1e4640 1 Processor -- start 2023-10-16T20:36:48.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898f1e4640 1 -- start start 2023-10-16T20:36:48.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898f1e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89880ca2f0 0x7f89881417a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:48.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898f1e4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89880cead0 0x7f8988143300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:48.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898f1e4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89881537c0 0x7f8988143840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:48.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898f1e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89880cd370 con 0x7f89880ca2f0 2023-10-16T20:36:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898f1e4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898815a500 con 0x7f89880cead0 2023-10-16T20:36:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898f1e4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898814d6a0 con 0x7f89881537c0 2023-10-16T20:36:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898e1e2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89880ca2f0 0x7f89881417a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898e1e2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89880ca2f0 0x7f89881417a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33468/0 (socket says 172.21.15.88:33468) 2023-10-16T20:36:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898e1e2640 1 -- 172.21.15.88:0/1822950543 learned_addr learned my addr 172.21.15.88:0/1822950543 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898d9e1640 1 --2- 172.21.15.88:0/1822950543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89880cead0 0x7f8988143300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898e9e3640 1 --2- 172.21.15.88:0/1822950543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89881537c0 0x7f8988143840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:48.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898e1e2640 1 -- 172.21.15.88:0/1822950543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89881537c0 msgr2=0x7f8988143840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:48.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898e1e2640 1 --2- 172.21.15.88:0/1822950543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89881537c0 0x7f8988143840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:48.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898e1e2640 1 -- 172.21.15.88:0/1822950543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89880cead0 msgr2=0x7f8988143300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:48.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898e1e2640 1 --2- 172.21.15.88:0/1822950543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89880cead0 0x7f8988143300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:48.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898e1e2640 1 -- 172.21.15.88:0/1822950543 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8980009a70 con 0x7f89880ca2f0 2023-10-16T20:36:48.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898e1e2640 1 --2- 172.21.15.88:0/1822950543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89880ca2f0 0x7f89881417a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f898400c430 tx=0x7f898400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:48.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f897effd640 1 -- 172.21.15.88:0/1822950543 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89840052d0 con 0x7f89880ca2f0 2023-10-16T20:36:48.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f897effd640 1 -- 172.21.15.88:0/1822950543 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8984005450 con 0x7f89880ca2f0 2023-10-16T20:36:48.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f897effd640 1 -- 172.21.15.88:0/1822950543 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8984003840 con 0x7f89880ca2f0 2023-10-16T20:36:48.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898f1e4640 1 -- 172.21.15.88:0/1822950543 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89881f1db0 con 0x7f89880ca2f0 2023-10-16T20:36:48.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898f1e4640 1 -- 172.21.15.88:0/1822950543 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89881f2090 con 0x7f89880ca2f0 2023-10-16T20:36:48.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.971+0000 7f898f1e4640 1 -- 172.21.15.88:0/1822950543 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f898815a9e0 con 0x7f89880ca2f0 2023-10-16T20:36:48.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.975+0000 7f897effd640 1 -- 172.21.15.88:0/1822950543 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f89840039a0 con 0x7f89880ca2f0 2023-10-16T20:36:48.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.975+0000 7f897effd640 1 --2- 172.21.15.88:0/1822950543 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f896c066470 0x7f896c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:48.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.975+0000 7f897effd640 1 -- 172.21.15.88:0/1822950543 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f89840831c0 con 0x7f89880ca2f0 2023-10-16T20:36:48.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.975+0000 7f898d9e1640 1 --2- 172.21.15.88:0/1822950543 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f896c066470 0x7f896c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:48.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.975+0000 7f898d9e1640 1 --2- 172.21.15.88:0/1822950543 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f896c066470 0x7f896c068930 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f89880cdaf0 tx=0x7f8978007d40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:48.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:48.979+0000 7f897effd640 1 -- 172.21.15.88:0/1822950543 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8984050920 con 0x7f89880ca2f0 2023-10-16T20:36:49.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.131+0000 7f898f1e4640 1 -- 172.21.15.88:0/1822950543 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f898804e9c0 con 0x7f89880ca2f0 2023-10-16T20:36:49.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.131+0000 7f897effd640 1 -- 172.21.15.88:0/1822950543 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+12826 (secure 0 0 0) 0x7f8984053f40 con 0x7f89880ca2f0 2023-10-16T20:36:49.137 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:36:49.138 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":19,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:36:47.255863+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:36:49.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.131+0000 7f898f1e4640 1 -- 172.21.15.88:0/1822950543 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f896c066470 msgr2=0x7f896c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:49.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.131+0000 7f898f1e4640 1 --2- 172.21.15.88:0/1822950543 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f896c066470 0x7f896c068930 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f89880cdaf0 tx=0x7f8978007d40 comp rx=0 tx=0).stop 2023-10-16T20:36:49.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.131+0000 7f898f1e4640 1 -- 172.21.15.88:0/1822950543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89880ca2f0 msgr2=0x7f89881417a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:49.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.131+0000 7f898f1e4640 1 --2- 172.21.15.88:0/1822950543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89880ca2f0 0x7f89881417a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f898400c430 tx=0x7f898400c8f0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.131+0000 7f898f1e4640 1 -- 172.21.15.88:0/1822950543 shutdown_connections 2023-10-16T20:36:49.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.131+0000 7f898f1e4640 1 --2- 172.21.15.88:0/1822950543 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f896c066470 0x7f896c068930 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.131+0000 7f898f1e4640 1 --2- 172.21.15.88:0/1822950543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89881537c0 0x7f8988143840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.131+0000 7f898f1e4640 1 --2- 172.21.15.88:0/1822950543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89880cead0 0x7f8988143300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.131+0000 7f898f1e4640 1 --2- 172.21.15.88:0/1822950543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89880ca2f0 0x7f89881417a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.131+0000 7f898f1e4640 1 -- 172.21.15.88:0/1822950543 >> 172.21.15.88:0/1822950543 conn(0x7f898806c170 msgr2=0x7f89880cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:49.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.131+0000 7f898f1e4640 1 -- 172.21.15.88:0/1822950543 shutdown_connections 2023-10-16T20:36:49.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.131+0000 7f898f1e4640 1 -- 172.21.15.88:0/1822950543 wait complete. 2023-10-16T20:36:49.158 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T20:36:43.646418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T20:36:49.158 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T20:36:49.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.291+0000 7f9cd359e640 1 Processor -- start 2023-10-16T20:36:49.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.291+0000 7f9cd359e640 1 -- start start 2023-10-16T20:36:49.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.295+0000 7f9cd359e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9cd41535d0 0x7f9cd40c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:49.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.295+0000 7f9cd359e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cd40cebd0 0x7f9cd40cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:49.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.295+0000 7f9cd359e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd40caf10 con 0x7f9cd4159e20 2023-10-16T20:36:49.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.295+0000 7f9cd359e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cd40cc4c0 con 0x7f9cd41535d0 2023-10-16T20:36:49.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.295+0000 7f9cd359e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd40cb140 con 0x7f9cd40cebd0 2023-10-16T20:36:49.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.295+0000 7f9cd259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9cd41535d0 0x7f9cd40c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:49.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.295+0000 7f9cd259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9cd41535d0 0x7f9cd40c9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33472/0 (socket says 172.21.15.88:33472) 2023-10-16T20:36:49.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.295+0000 7f9cd259c640 1 -- 172.21.15.88:0/1344351194 learned_addr learned my addr 172.21.15.88:0/1344351194 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:49.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.295+0000 7f9cd2d9d640 1 --2- 172.21.15.88:0/1344351194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cd40cebd0 0x7f9cd40cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:49.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.295+0000 7f9cd259c640 1 -- 172.21.15.88:0/1344351194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cd40cebd0 msgr2=0x7f9cd40cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:49.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.295+0000 7f9cd259c640 1 --2- 172.21.15.88:0/1344351194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cd40cebd0 0x7f9cd40cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.295+0000 7f9cd259c640 1 -- 172.21.15.88:0/1344351194 >> v1:172.21.15.191:6789/0 conn(0x7f9cd4159e20 legacy=0x7f9cd40c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:49.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.295+0000 7f9cd259c640 1 -- 172.21.15.88:0/1344351194 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd40cb420 con 0x7f9cd41535d0 2023-10-16T20:36:49.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.299+0000 7f9cd259c640 1 --2- 172.21.15.88:0/1344351194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9cd41535d0 0x7f9cd40c9be0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4009870 tx=0x7f9cc401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c36e4841ecddc062 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:49.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.299+0000 7f9cd159a640 1 -- 172.21.15.88:0/1344351194 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc40a8a30 con 0x7f9cd41535d0 2023-10-16T20:36:49.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.299+0000 7f9cd159a640 1 -- 172.21.15.88:0/1344351194 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc40a8b90 con 0x7f9cd41535d0 2023-10-16T20:36:49.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.299+0000 7f9cd159a640 1 -- 172.21.15.88:0/1344351194 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc40b0440 con 0x7f9cd41535d0 2023-10-16T20:36:49.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.299+0000 7f9cd359e640 1 -- 172.21.15.88:0/1344351194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9cd41535d0 msgr2=0x7f9cd40c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:49.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.299+0000 7f9cd359e640 1 --2- 172.21.15.88:0/1344351194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9cd41535d0 0x7f9cd40c9be0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4009870 tx=0x7f9cc401ed30 comp rx=0 tx=0).stop 2023-10-16T20:36:49.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.299+0000 7f9cd359e640 1 -- 172.21.15.88:0/1344351194 shutdown_connections 2023-10-16T20:36:49.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.299+0000 7f9cd359e640 1 --2- 172.21.15.88:0/1344351194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cd40cebd0 0x7f9cd40cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.299+0000 7f9cd359e640 1 --2- 172.21.15.88:0/1344351194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9cd41535d0 0x7f9cd40c9be0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.299+0000 7f9cd359e640 1 -- 172.21.15.88:0/1344351194 >> 172.21.15.88:0/1344351194 conn(0x7f9cd406bbe0 msgr2=0x7f9cd406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:49.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.299+0000 7f9cd359e640 1 -- 172.21.15.88:0/1344351194 shutdown_connections 2023-10-16T20:36:49.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.299+0000 7f9cd359e640 1 -- 172.21.15.88:0/1344351194 wait complete. 2023-10-16T20:36:49.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.299+0000 7f9cd359e640 1 Processor -- start 2023-10-16T20:36:49.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.299+0000 7f9cd359e640 1 -- start start 2023-10-16T20:36:49.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd359e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd40cebd0 0x7f9cd41f29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:49.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd359e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9cd41535d0 0x7f9cd41ee990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:49.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd359e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cd4159e20 0x7f9cd40c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:49.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd359e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd4154620 con 0x7f9cd41535d0 2023-10-16T20:36:49.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd359e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd40c76c0 con 0x7f9cd40cebd0 2023-10-16T20:36:49.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd359e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd40b14e0 con 0x7f9cd4159e20 2023-10-16T20:36:49.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd1d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9cd41535d0 0x7f9cd41ee990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:49.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd1d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9cd41535d0 0x7f9cd41ee990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33474/0 (socket says 172.21.15.88:33474) 2023-10-16T20:36:49.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd259c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd40cebd0 0x7f9cd41f29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:49.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd1d9b640 1 -- 172.21.15.88:0/2856244100 learned_addr learned my addr 172.21.15.88:0/2856244100 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd2d9d640 1 --2- 172.21.15.88:0/2856244100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cd4159e20 0x7f9cd40c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd1d9b640 1 -- 172.21.15.88:0/2856244100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cd4159e20 msgr2=0x7f9cd40c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd1d9b640 1 --2- 172.21.15.88:0/2856244100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cd4159e20 0x7f9cd40c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd1d9b640 1 -- 172.21.15.88:0/2856244100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd40cebd0 msgr2=0x7f9cd41f29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd1d9b640 1 --2- 172.21.15.88:0/2856244100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd40cebd0 0x7f9cd41f29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd1d9b640 1 -- 172.21.15.88:0/2856244100 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc40094d0 con 0x7f9cd41535d0 2023-10-16T20:36:49.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd259c640 1 --2- 172.21.15.88:0/2856244100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd40cebd0 0x7f9cd41f29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:36:49.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd1d9b640 1 --2- 172.21.15.88:0/2856244100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9cd41535d0 0x7f9cd41ee990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9cc800c3c0 tx=0x7f9cc800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:49.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cc2ffd640 1 -- 172.21.15.88:0/2856244100 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc80052d0 con 0x7f9cd41535d0 2023-10-16T20:36:49.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cc2ffd640 1 -- 172.21.15.88:0/2856244100 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc8005430 con 0x7f9cd41535d0 2023-10-16T20:36:49.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cc2ffd640 1 -- 172.21.15.88:0/2856244100 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8003840 con 0x7f9cd41535d0 2023-10-16T20:36:49.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd359e640 1 -- 172.21.15.88:0/2856244100 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd4153d90 con 0x7f9cd41535d0 2023-10-16T20:36:49.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd359e640 1 -- 172.21.15.88:0/2856244100 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd40cbf00 con 0x7f9cd41535d0 2023-10-16T20:36:49.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cd359e640 1 -- 172.21.15.88:0/2856244100 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd404e9c0 con 0x7f9cd41535d0 2023-10-16T20:36:49.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.303+0000 7f9cc2ffd640 1 -- 172.21.15.88:0/2856244100 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9cc80039a0 con 0x7f9cd41535d0 2023-10-16T20:36:49.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.307+0000 7f9cc2ffd640 1 --2- 172.21.15.88:0/2856244100 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9cb40664d0 0x7f9cb4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:49.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.307+0000 7f9cc2ffd640 1 -- 172.21.15.88:0/2856244100 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9cc8082f80 con 0x7f9cd41535d0 2023-10-16T20:36:49.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.307+0000 7f9cd259c640 1 --2- 172.21.15.88:0/2856244100 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9cb40664d0 0x7f9cb4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:49.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.307+0000 7f9cd259c640 1 --2- 172.21.15.88:0/2856244100 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9cb40664d0 0x7f9cb4068990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9cc40b2000 tx=0x7f9cc4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:49.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.311+0000 7f9cc2ffd640 1 -- 172.21.15.88:0/2856244100 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9cc80507f0 con 0x7f9cd41535d0 2023-10-16T20:36:49.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.455+0000 7f9cd359e640 1 -- 172.21.15.88:0/2856244100 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9cd40ca600 con 0x7f9cd41535d0 2023-10-16T20:36:49.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.455+0000 7f9cc2ffd640 1 -- 172.21.15.88:0/2856244100 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9cc8053e10 con 0x7f9cd41535d0 2023-10-16T20:36:49.464 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-10-16T20:36:49.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.459+0000 7f9cd359e640 1 -- 172.21.15.88:0/2856244100 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9cb40664d0 msgr2=0x7f9cb4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:49.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.459+0000 7f9cd359e640 1 --2- 172.21.15.88:0/2856244100 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9cb40664d0 0x7f9cb4068990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9cc40b2000 tx=0x7f9cc4009040 comp rx=0 tx=0).stop 2023-10-16T20:36:49.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.459+0000 7f9cd359e640 1 -- 172.21.15.88:0/2856244100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9cd41535d0 msgr2=0x7f9cd41ee990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:49.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.459+0000 7f9cd359e640 1 --2- 172.21.15.88:0/2856244100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9cd41535d0 0x7f9cd41ee990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9cc800c3c0 tx=0x7f9cc800c880 comp rx=0 tx=0).stop 2023-10-16T20:36:49.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.459+0000 7f9cd359e640 1 -- 172.21.15.88:0/2856244100 shutdown_connections 2023-10-16T20:36:49.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.459+0000 7f9cd359e640 1 --2- 172.21.15.88:0/2856244100 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9cb40664d0 0x7f9cb4068990 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.459+0000 7f9cd359e640 1 --2- 172.21.15.88:0/2856244100 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9cd4159e20 0x7f9cd40c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.459+0000 7f9cd359e640 1 --2- 172.21.15.88:0/2856244100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9cd41535d0 0x7f9cd41ee990 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.459+0000 7f9cd359e640 1 --2- 172.21.15.88:0/2856244100 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd40cebd0 0x7f9cd41f29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.459+0000 7f9cd359e640 1 -- 172.21.15.88:0/2856244100 >> 172.21.15.88:0/2856244100 conn(0x7f9cd406bbe0 msgr2=0x7f9cd40c3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:49.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.459+0000 7f9cd359e640 1 -- 172.21.15.88:0/2856244100 shutdown_connections 2023-10-16T20:36:49.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.459+0000 7f9cd359e640 1 -- 172.21.15.88:0/2856244100 wait complete. 2023-10-16T20:36:49.484 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:36:49.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.615+0000 7f2dcaf2a640 1 Processor -- start 2023-10-16T20:36:49.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.615+0000 7f2dcaf2a640 1 -- start start 2023-10-16T20:36:49.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.615+0000 7f2dcaf2a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2dc415a810 0x7f2dc415ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:49.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.615+0000 7f2dcaf2a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc415b3c0 con 0x7f2dc406a970 2023-10-16T20:36:49.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.615+0000 7f2dcaf2a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc415b680 con 0x7f2dc41406d0 2023-10-16T20:36:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.615+0000 7f2dcaf2a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc415b1c0 con 0x7f2dc415a810 2023-10-16T20:36:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.615+0000 7f2dc3fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2dc406a970 0x7f2dc4071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46856/0 (socket says 172.21.15.88:46856) 2023-10-16T20:36:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.615+0000 7f2dc3fff640 1 -- 172.21.15.88:0/1155441589 learned_addr learned my addr 172.21.15.88:0/1155441589 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:36:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.619+0000 7f2dc94c7640 1 --2- 172.21.15.88:0/1155441589 >> v2:172.21.15.191:3300/0 conn(0x7f2dc415a810 0x7f2dc415ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.619+0000 7f2dc37fe640 1 -- 172.21.15.88:0/1155441589 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006419736 0 0) 0x7f2dc415b3c0 con 0x7f2dc406a970 2023-10-16T20:36:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.619+0000 7f2dc37fe640 1 -- 172.21.15.88:0/1155441589 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db4003530 con 0x7f2dc406a970 2023-10-16T20:36:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.619+0000 7f2dc94c7640 1 -- 172.21.15.88:0/1155441589 >> v1:172.21.15.191:6789/0 conn(0x7f2dc41406d0 legacy=0x7f2dc40c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.619+0000 7f2dc94c7640 1 -- 172.21.15.88:0/1155441589 >> v1:172.21.15.88:6789/0 conn(0x7f2dc406a970 legacy=0x7f2dc4071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.619+0000 7f2dc94c7640 1 -- 172.21.15.88:0/1155441589 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc4157570 con 0x7f2dc415a810 2023-10-16T20:36:49.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.619+0000 7f2dc94c7640 1 --2- 172.21.15.88:0/1155441589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2dc415a810 0x7f2dc415ac80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc009d90 tx=0x7f2dbc01c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94db9fc9ea42f2a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:49.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.619+0000 7f2dc37fe640 1 -- 172.21.15.88:0/1155441589 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc0a5900 con 0x7f2dc415a810 2023-10-16T20:36:49.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.619+0000 7f2dc37fe640 1 -- 172.21.15.88:0/1155441589 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dbc0a5a60 con 0x7f2dc415a810 2023-10-16T20:36:49.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.619+0000 7f2dc37fe640 1 -- 172.21.15.88:0/1155441589 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc0a5dd0 con 0x7f2dc415a810 2023-10-16T20:36:49.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dcaf2a640 1 -- 172.21.15.88:0/1155441589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2dc415a810 msgr2=0x7f2dc415ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:49.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dcaf2a640 1 --2- 172.21.15.88:0/1155441589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2dc415a810 0x7f2dc415ac80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc009d90 tx=0x7f2dbc01c950 comp rx=0 tx=0).stop 2023-10-16T20:36:49.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dcaf2a640 1 -- 172.21.15.88:0/1155441589 shutdown_connections 2023-10-16T20:36:49.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dcaf2a640 1 --2- 172.21.15.88:0/1155441589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2dc415a810 0x7f2dc415ac80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dcaf2a640 1 -- 172.21.15.88:0/1155441589 >> 172.21.15.88:0/1155441589 conn(0x7f2dc40cad50 msgr2=0x7f2dc40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:49.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dcaf2a640 1 -- 172.21.15.88:0/1155441589 shutdown_connections 2023-10-16T20:36:49.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dcaf2a640 1 -- 172.21.15.88:0/1155441589 wait complete. 2023-10-16T20:36:49.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dcaf2a640 1 Processor -- start 2023-10-16T20:36:49.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dcaf2a640 1 -- start start 2023-10-16T20:36:49.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dcaf2a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2dc406a970 0x7f2dc4150f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:49.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dcaf2a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dc41406d0 0x7f2dc413b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:49.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dcaf2a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2dc415a810 0x7f2dc413b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:49.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dcaf2a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc4151480 con 0x7f2dc41406d0 2023-10-16T20:36:49.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dcaf2a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc41590a0 con 0x7f2dc415a810 2023-10-16T20:36:49.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dcaf2a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc4159380 con 0x7f2dc406a970 2023-10-16T20:36:49.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dc3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dc41406d0 0x7f2dc413b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:49.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dc3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dc41406d0 0x7f2dc413b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33482/0 (socket says 172.21.15.88:33482) 2023-10-16T20:36:49.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dc3fff640 1 -- 172.21.15.88:0/716053221 learned_addr learned my addr 172.21.15.88:0/716053221 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:49.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dc94c7640 1 --2- 172.21.15.88:0/716053221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2dc415a810 0x7f2dc413b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:49.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dc8cc6640 1 --2- 172.21.15.88:0/716053221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2dc406a970 0x7f2dc4150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:49.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dc3fff640 1 -- 172.21.15.88:0/716053221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2dc406a970 msgr2=0x7f2dc4150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:49.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dc3fff640 1 --2- 172.21.15.88:0/716053221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2dc406a970 0x7f2dc4150f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dc3fff640 1 -- 172.21.15.88:0/716053221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2dc415a810 msgr2=0x7f2dc413b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:49.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dc3fff640 1 --2- 172.21.15.88:0/716053221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2dc415a810 0x7f2dc413b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dc3fff640 1 -- 172.21.15.88:0/716053221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc009a70 con 0x7f2dc41406d0 2023-10-16T20:36:49.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.623+0000 7f2dc8cc6640 1 --2- 172.21.15.88:0/716053221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2dc406a970 0x7f2dc4150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:36:49.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.627+0000 7f2dc3fff640 1 --2- 172.21.15.88:0/716053221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dc41406d0 0x7f2dc413b080 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2db000c420 tx=0x7f2db000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:49.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.627+0000 7f2dc17fa640 1 -- 172.21.15.88:0/716053221 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db0005cf0 con 0x7f2dc41406d0 2023-10-16T20:36:49.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.627+0000 7f2dc17fa640 1 -- 172.21.15.88:0/716053221 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db0005e50 con 0x7f2dc41406d0 2023-10-16T20:36:49.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.627+0000 7f2dcaf2a640 1 -- 172.21.15.88:0/716053221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc4159660 con 0x7f2dc41406d0 2023-10-16T20:36:49.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.627+0000 7f2dc17fa640 1 -- 172.21.15.88:0/716053221 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db00072d0 con 0x7f2dc41406d0 2023-10-16T20:36:49.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.627+0000 7f2dcaf2a640 1 -- 172.21.15.88:0/716053221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc4159c20 con 0x7f2dc41406d0 2023-10-16T20:36:49.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.627+0000 7f2dcaf2a640 1 -- 172.21.15.88:0/716053221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc4074700 con 0x7f2dc41406d0 2023-10-16T20:36:49.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.627+0000 7f2dc17fa640 1 -- 172.21.15.88:0/716053221 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2db0007430 con 0x7f2dc41406d0 2023-10-16T20:36:49.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.627+0000 7f2dc17fa640 1 --2- 172.21.15.88:0/716053221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2da80664d0 0x7f2da8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:49.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.627+0000 7f2dc17fa640 1 -- 172.21.15.88:0/716053221 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2db0083180 con 0x7f2dc41406d0 2023-10-16T20:36:49.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.627+0000 7f2dc8cc6640 1 --2- 172.21.15.88:0/716053221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2da80664d0 0x7f2da8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:49.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.627+0000 7f2dc8cc6640 1 --2- 172.21.15.88:0/716053221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2da80664d0 0x7f2da8068990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2db8001710 tx=0x7f2db8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:49.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.631+0000 7f2dc17fa640 1 -- 172.21.15.88:0/716053221 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2db0050850 con 0x7f2dc41406d0 2023-10-16T20:36:49.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.815+0000 7f2dcaf2a640 1 -- 172.21.15.88:0/716053221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2dc4154c40 con 0x7f2dc41406d0 2023-10-16T20:36:49.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.815+0000 7f2dc17fa640 1 -- 172.21.15.88:0/716053221 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1321 (secure 0 0 0) 0x7f2db0053e70 con 0x7f2dc41406d0 2023-10-16T20:36:49.822 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:36:49.822 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-10-16T20:36:49.823 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 2 2023-10-16T20:36:49.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.815+0000 7f2dcaf2a640 1 -- 172.21.15.88:0/716053221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2da80664d0 msgr2=0x7f2da8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:49.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.815+0000 7f2dcaf2a640 1 --2- 172.21.15.88:0/716053221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2da80664d0 0x7f2da8068990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2db8001710 tx=0x7f2db8009040 comp rx=0 tx=0).stop 2023-10-16T20:36:49.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.815+0000 7f2dcaf2a640 1 -- 172.21.15.88:0/716053221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dc41406d0 msgr2=0x7f2dc413b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:49.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.815+0000 7f2dcaf2a640 1 --2- 172.21.15.88:0/716053221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dc41406d0 0x7f2dc413b080 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2db000c420 tx=0x7f2db000c8e0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.815+0000 7f2dcaf2a640 1 -- 172.21.15.88:0/716053221 shutdown_connections 2023-10-16T20:36:49.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.815+0000 7f2dcaf2a640 1 --2- 172.21.15.88:0/716053221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2da80664d0 0x7f2da8068990 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.815+0000 7f2dcaf2a640 1 --2- 172.21.15.88:0/716053221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2dc415a810 0x7f2dc413b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.815+0000 7f2dcaf2a640 1 --2- 172.21.15.88:0/716053221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dc41406d0 0x7f2dc413b080 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.815+0000 7f2dcaf2a640 1 --2- 172.21.15.88:0/716053221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2dc406a970 0x7f2dc4150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:49.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.815+0000 7f2dcaf2a640 1 -- 172.21.15.88:0/716053221 >> 172.21.15.88:0/716053221 conn(0x7f2dc40cad50 msgr2=0x7f2dc413e430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:49.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.819+0000 7f2dcaf2a640 1 -- 172.21.15.88:0/716053221 shutdown_connections 2023-10-16T20:36:49.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:49.819+0000 7f2dcaf2a640 1 -- 172.21.15.88:0/716053221 wait complete. 2023-10-16T20:36:51.843 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:36:51.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.983+0000 7fcb57f65640 1 Processor -- start 2023-10-16T20:36:51.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.983+0000 7fcb57f65640 1 -- start start 2023-10-16T20:36:51.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.987+0000 7fcb57f65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb500c58d0 0x7fcb5006d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:51.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.987+0000 7fcb57f65640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcb5006d6b0 0x7fcb50160e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:51.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.987+0000 7fcb57f65640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb500c5e10 con 0x7fcb50157b10 2023-10-16T20:36:51.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.987+0000 7fcb57f65640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb5006f840 con 0x7fcb5006d6b0 2023-10-16T20:36:51.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.987+0000 7fcb57f65640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5006fb20 con 0x7fcb500c58d0 2023-10-16T20:36:51.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.987+0000 7fcb56502640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcb5006d6b0 0x7fcb50160e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:51.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.987+0000 7fcb56502640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcb5006d6b0 0x7fcb50160e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33486/0 (socket says 172.21.15.88:33486) 2023-10-16T20:36:51.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.987+0000 7fcb56502640 1 -- 172.21.15.88:0/2744572320 learned_addr learned my addr 172.21.15.88:0/2744572320 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:51.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.987+0000 7fcb55500640 1 --2- 172.21.15.88:0/2744572320 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb500c58d0 0x7fcb5006d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:51.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.987+0000 7fcb56502640 1 -- 172.21.15.88:0/2744572320 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb500c58d0 msgr2=0x7fcb5006d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:51.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.987+0000 7fcb56502640 1 --2- 172.21.15.88:0/2744572320 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb500c58d0 0x7fcb5006d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:51.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.987+0000 7fcb56502640 1 -- 172.21.15.88:0/2744572320 >> v1:172.21.15.191:6789/0 conn(0x7fcb50157b10 legacy=0x7fcb5015e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:51.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.987+0000 7fcb56502640 1 -- 172.21.15.88:0/2744572320 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb500c67b0 con 0x7fcb5006d6b0 2023-10-16T20:36:51.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb56502640 1 --2- 172.21.15.88:0/2744572320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb5006d6b0 0x7fcb50160e70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009e50 tx=0x7fcb4801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9e0dbc50b8c434 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:51.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb54cff640 1 -- 172.21.15.88:0/2744572320 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb480a5830 con 0x7fcb5006d6b0 2023-10-16T20:36:51.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb54cff640 1 -- 172.21.15.88:0/2744572320 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb480a5990 con 0x7fcb5006d6b0 2023-10-16T20:36:51.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb54cff640 1 -- 172.21.15.88:0/2744572320 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb480a5d20 con 0x7fcb5006d6b0 2023-10-16T20:36:51.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb57f65640 1 -- 172.21.15.88:0/2744572320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb5006d6b0 msgr2=0x7fcb50160e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:51.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb57f65640 1 --2- 172.21.15.88:0/2744572320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb5006d6b0 0x7fcb50160e70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009e50 tx=0x7fcb4801c950 comp rx=0 tx=0).stop 2023-10-16T20:36:51.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb57f65640 1 -- 172.21.15.88:0/2744572320 shutdown_connections 2023-10-16T20:36:51.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb57f65640 1 --2- 172.21.15.88:0/2744572320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb5006d6b0 0x7fcb50160e70 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:51.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb57f65640 1 --2- 172.21.15.88:0/2744572320 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb500c58d0 0x7fcb5006d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:51.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb57f65640 1 -- 172.21.15.88:0/2744572320 >> 172.21.15.88:0/2744572320 conn(0x7fcb500cad50 msgr2=0x7fcb500cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:52.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb57f65640 1 -- 172.21.15.88:0/2744572320 shutdown_connections 2023-10-16T20:36:52.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb57f65640 1 -- 172.21.15.88:0/2744572320 wait complete. 2023-10-16T20:36:52.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb57f65640 1 Processor -- start 2023-10-16T20:36:52.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb57f65640 1 -- start start 2023-10-16T20:36:52.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb57f65640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb5006d6b0 0x7fcb501f6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb57f65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb500c58d0 0x7fcb50153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb57f65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb50157b10 0x7fcb501f9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb57f65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5015ae00 con 0x7fcb500c58d0 2023-10-16T20:36:52.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb57f65640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb500c3f40 con 0x7fcb5006d6b0 2023-10-16T20:36:52.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.991+0000 7fcb57f65640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb500c31b0 con 0x7fcb50157b10 2023-10-16T20:36:52.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb55500640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb500c58d0 0x7fcb50153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb55500640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb500c58d0 0x7fcb50153e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33496/0 (socket says 172.21.15.88:33496) 2023-10-16T20:36:52.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb55500640 1 -- 172.21.15.88:0/3928845595 learned_addr learned my addr 172.21.15.88:0/3928845595 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:52.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb55d01640 1 --2- 172.21.15.88:0/3928845595 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb5006d6b0 0x7fcb501f6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb56502640 1 --2- 172.21.15.88:0/3928845595 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb50157b10 0x7fcb501f9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb55500640 1 -- 172.21.15.88:0/3928845595 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb50157b10 msgr2=0x7fcb501f9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:52.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb55500640 1 --2- 172.21.15.88:0/3928845595 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb50157b10 0x7fcb501f9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb55500640 1 -- 172.21.15.88:0/3928845595 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb5006d6b0 msgr2=0x7fcb501f6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:52.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb55500640 1 --2- 172.21.15.88:0/3928845595 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb5006d6b0 0x7fcb501f6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb55500640 1 -- 172.21.15.88:0/3928845595 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb48009b30 con 0x7fcb500c58d0 2023-10-16T20:36:52.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb55500640 1 --2- 172.21.15.88:0/3928845595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb500c58d0 0x7fcb50153e90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000e960 tx=0x7fcb4000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:52.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb467fc640 1 -- 172.21.15.88:0/3928845595 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb40009690 con 0x7fcb500c58d0 2023-10-16T20:36:52.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb467fc640 1 -- 172.21.15.88:0/3928845595 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4000ca00 con 0x7fcb500c58d0 2023-10-16T20:36:52.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb57f65640 1 -- 172.21.15.88:0/3928845595 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb501f7080 con 0x7fcb500c58d0 2023-10-16T20:36:52.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb467fc640 1 -- 172.21.15.88:0/3928845595 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4000cd70 con 0x7fcb500c58d0 2023-10-16T20:36:52.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb57f65640 1 -- 172.21.15.88:0/3928845595 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb501f97d0 con 0x7fcb500c58d0 2023-10-16T20:36:52.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb57f65640 1 -- 172.21.15.88:0/3928845595 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb50074700 con 0x7fcb500c58d0 2023-10-16T20:36:52.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb467fc640 1 -- 172.21.15.88:0/3928845595 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcb40013470 con 0x7fcb500c58d0 2023-10-16T20:36:52.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb467fc640 1 --2- 172.21.15.88:0/3928845595 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb340664a0 0x7fcb34068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.995+0000 7fcb467fc640 1 -- 172.21.15.88:0/3928845595 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcb40083ff0 con 0x7fcb500c58d0 2023-10-16T20:36:52.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.999+0000 7fcb55d01640 1 --2- 172.21.15.88:0/3928845595 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb340664a0 0x7fcb34068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:51.999+0000 7fcb55d01640 1 --2- 172.21.15.88:0/3928845595 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb340664a0 0x7fcb34068960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c007fd0 tx=0x7fcb4c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:52.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.003+0000 7fcb467fc640 1 -- 172.21.15.88:0/3928845595 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcb400b5090 con 0x7fcb500c58d0 2023-10-16T20:36:52.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.179+0000 7fcb57f65640 1 -- 172.21.15.88:0/3928845595 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb501f7080 con 0x7fcb500c58d0 2023-10-16T20:36:52.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.183+0000 7fcb467fc640 1 -- 172.21.15.88:0/3928845595 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5277 (secure 0 0 0) 0x7fcb501f7080 con 0x7fcb500c58d0 2023-10-16T20:36:52.189 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:36:52.189 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4275,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4288,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-10-16T20:36:52.191 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 3 2023-10-16T20:36:52.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.183+0000 7fcb57f65640 1 -- 172.21.15.88:0/3928845595 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb340664a0 msgr2=0x7fcb34068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:52.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.183+0000 7fcb57f65640 1 --2- 172.21.15.88:0/3928845595 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb340664a0 0x7fcb34068960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c007fd0 tx=0x7fcb4c009040 comp rx=0 tx=0).stop 2023-10-16T20:36:52.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.183+0000 7fcb57f65640 1 -- 172.21.15.88:0/3928845595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb500c58d0 msgr2=0x7fcb50153e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:52.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.183+0000 7fcb57f65640 1 --2- 172.21.15.88:0/3928845595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb500c58d0 0x7fcb50153e90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000e960 tx=0x7fcb4000ee20 comp rx=0 tx=0).stop 2023-10-16T20:36:52.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.183+0000 7fcb57f65640 1 -- 172.21.15.88:0/3928845595 shutdown_connections 2023-10-16T20:36:52.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.183+0000 7fcb57f65640 1 --2- 172.21.15.88:0/3928845595 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb340664a0 0x7fcb34068960 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.183+0000 7fcb57f65640 1 --2- 172.21.15.88:0/3928845595 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb50157b10 0x7fcb501f9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.183+0000 7fcb57f65640 1 --2- 172.21.15.88:0/3928845595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb500c58d0 0x7fcb50153e90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.183+0000 7fcb57f65640 1 --2- 172.21.15.88:0/3928845595 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb5006d6b0 0x7fcb501f6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.183+0000 7fcb57f65640 1 -- 172.21.15.88:0/3928845595 >> 172.21.15.88:0/3928845595 conn(0x7fcb500cad50 msgr2=0x7fcb500cb130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:52.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.183+0000 7fcb57f65640 1 -- 172.21.15.88:0/3928845595 shutdown_connections 2023-10-16T20:36:52.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.183+0000 7fcb57f65640 1 -- 172.21.15.88:0/3928845595 wait complete. 2023-10-16T20:36:52.209 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:36:52.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.339+0000 7fc3ded9d640 1 Processor -- start 2023-10-16T20:36:52.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.339+0000 7fc3ded9d640 1 -- start start 2023-10-16T20:36:52.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.339+0000 7fc3ded9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc3d8150660 0x7fc3d80c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.339+0000 7fc3ded9d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc3d80c6bc0 0x7fc3d80c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.339+0000 7fc3ded9d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d80c5c60 con 0x7fc3d8150280 2023-10-16T20:36:52.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.339+0000 7fc3ded9d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d80c5370 con 0x7fc3d8150660 2023-10-16T20:36:52.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.339+0000 7fc3ded9d640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d8156d10 con 0x7fc3d80c6bc0 2023-10-16T20:36:52.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.339+0000 7fc3cffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc3d8150660 0x7fc3d80c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.339+0000 7fc3cffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc3d8150660 0x7fc3d80c60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33502/0 (socket says 172.21.15.88:33502) 2023-10-16T20:36:52.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.339+0000 7fc3cffff640 1 -- 172.21.15.88:0/1684356939 learned_addr learned my addr 172.21.15.88:0/1684356939 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.339+0000 7fc3dd33a640 1 --2- 172.21.15.88:0/1684356939 >> v2:172.21.15.191:3300/0 conn(0x7fc3d80c6bc0 0x7fc3d80c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.343+0000 7fc3cffff640 1 -- 172.21.15.88:0/1684356939 >> v2:172.21.15.191:3300/0 conn(0x7fc3d80c6bc0 msgr2=0x7fc3d80c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.343+0000 7fc3cffff640 1 --2- 172.21.15.88:0/1684356939 >> v2:172.21.15.191:3300/0 conn(0x7fc3d80c6bc0 0x7fc3d80c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.343+0000 7fc3cffff640 1 -- 172.21.15.88:0/1684356939 >> v1:172.21.15.191:6789/0 conn(0x7fc3d8150280 legacy=0x7fc3d806d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:52.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.343+0000 7fc3cffff640 1 -- 172.21.15.88:0/1684356939 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d80c5610 con 0x7fc3d8150660 2023-10-16T20:36:52.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.343+0000 7fc3cffff640 1 --2- 172.21.15.88:0/1684356939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d8150660 0x7fc3d80c60f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc3d806e030 tx=0x7fc3c801cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d119e984b45100 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:52.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.343+0000 7fc3cf7fe640 1 -- 172.21.15.88:0/1684356939 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c80aa610 con 0x7fc3d8150660 2023-10-16T20:36:52.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.343+0000 7fc3cf7fe640 1 -- 172.21.15.88:0/1684356939 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c80a6040 con 0x7fc3d8150660 2023-10-16T20:36:52.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.343+0000 7fc3cf7fe640 1 -- 172.21.15.88:0/1684356939 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c80aa990 con 0x7fc3d8150660 2023-10-16T20:36:52.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.343+0000 7fc3ded9d640 1 -- 172.21.15.88:0/1684356939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d8150660 msgr2=0x7fc3d80c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:52.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.343+0000 7fc3ded9d640 1 --2- 172.21.15.88:0/1684356939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d8150660 0x7fc3d80c60f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc3d806e030 tx=0x7fc3c801cef0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.343+0000 7fc3ded9d640 1 -- 172.21.15.88:0/1684356939 shutdown_connections 2023-10-16T20:36:52.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.343+0000 7fc3ded9d640 1 --2- 172.21.15.88:0/1684356939 >> v2:172.21.15.191:3300/0 conn(0x7fc3d80c6bc0 0x7fc3d80c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.343+0000 7fc3ded9d640 1 --2- 172.21.15.88:0/1684356939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d8150660 0x7fc3d80c60f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.343+0000 7fc3ded9d640 1 -- 172.21.15.88:0/1684356939 >> 172.21.15.88:0/1684356939 conn(0x7fc3d80cad50 msgr2=0x7fc3d80cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:52.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.343+0000 7fc3ded9d640 1 -- 172.21.15.88:0/1684356939 shutdown_connections 2023-10-16T20:36:52.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3ded9d640 1 -- 172.21.15.88:0/1684356939 wait complete. 2023-10-16T20:36:52.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3ded9d640 1 Processor -- start 2023-10-16T20:36:52.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3ded9d640 1 -- start start 2023-10-16T20:36:52.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3ded9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d80c6bc0 0x7fc3d8141b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3ded9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3d8150280 0x7fc3d813bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3ded9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3d8150660 0x7fc3d813e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3ded9d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d813dea0 con 0x7fc3d80c6bc0 2023-10-16T20:36:52.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3ded9d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d81376a0 con 0x7fc3d8150660 2023-10-16T20:36:52.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3ded9d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d813b050 con 0x7fc3d8150280 2023-10-16T20:36:52.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3dcb39640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d80c6bc0 0x7fc3d8141b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3dcb39640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d80c6bc0 0x7fc3d8141b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33506/0 (socket says 172.21.15.88:33506) 2023-10-16T20:36:52.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3dcb39640 1 -- 172.21.15.88:0/744335065 learned_addr learned my addr 172.21.15.88:0/744335065 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:52.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3cffff640 1 --2- 172.21.15.88:0/744335065 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3d8150280 0x7fc3d813bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3dd33a640 1 --2- 172.21.15.88:0/744335065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3d8150660 0x7fc3d813e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3dcb39640 1 -- 172.21.15.88:0/744335065 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3d8150280 msgr2=0x7fc3d813bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:52.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3dcb39640 1 --2- 172.21.15.88:0/744335065 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3d8150280 0x7fc3d813bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3dcb39640 1 -- 172.21.15.88:0/744335065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3d8150660 msgr2=0x7fc3d813e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:52.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3dcb39640 1 --2- 172.21.15.88:0/744335065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3d8150660 0x7fc3d813e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3dcb39640 1 -- 172.21.15.88:0/744335065 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c80094d0 con 0x7fc3d80c6bc0 2023-10-16T20:36:52.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3dcb39640 1 --2- 172.21.15.88:0/744335065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d80c6bc0 0x7fc3d8141b40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc3d400c450 tx=0x7fc3d400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:52.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3cd7fa640 1 -- 172.21.15.88:0/744335065 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4012830 con 0x7fc3d80c6bc0 2023-10-16T20:36:52.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3cd7fa640 1 -- 172.21.15.88:0/744335065 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d40129b0 con 0x7fc3d80c6bc0 2023-10-16T20:36:52.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.347+0000 7fc3ded9d640 1 -- 172.21.15.88:0/744335065 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d81373c0 con 0x7fc3d80c6bc0 2023-10-16T20:36:52.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.351+0000 7fc3cd7fa640 1 -- 172.21.15.88:0/744335065 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4012d60 con 0x7fc3d80c6bc0 2023-10-16T20:36:52.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.351+0000 7fc3ded9d640 1 -- 172.21.15.88:0/744335065 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d80c0e90 con 0x7fc3d80c6bc0 2023-10-16T20:36:52.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.351+0000 7fc3ded9d640 1 -- 172.21.15.88:0/744335065 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d8067200 con 0x7fc3d80c6bc0 2023-10-16T20:36:52.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.351+0000 7fc3cd7fa640 1 -- 172.21.15.88:0/744335065 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc3d401c460 con 0x7fc3d80c6bc0 2023-10-16T20:36:52.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.351+0000 7fc3cd7fa640 1 --2- 172.21.15.88:0/744335065 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc3c40664a0 0x7fc3c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.351+0000 7fc3cd7fa640 1 -- 172.21.15.88:0/744335065 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc3d4083a40 con 0x7fc3d80c6bc0 2023-10-16T20:36:52.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.351+0000 7fc3cffff640 1 --2- 172.21.15.88:0/744335065 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc3c40664a0 0x7fc3c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.355+0000 7fc3cffff640 1 --2- 172.21.15.88:0/744335065 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc3c40664a0 0x7fc3c4068960 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc3c80ae000 tx=0x7fc3c8004740 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:52.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.355+0000 7fc3cd7fa640 1 -- 172.21.15.88:0/744335065 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc3d4051170 con 0x7fc3d80c6bc0 2023-10-16T20:36:52.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.507+0000 7fc3ded9d640 1 -- 172.21.15.88:0/744335065 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3d8074700 con 0x7fc3d80c6bc0 2023-10-16T20:36:52.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.507+0000 7fc3cd7fa640 1 -- 172.21.15.88:0/744335065 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+12826 (secure 0 0 0) 0x7fc3d4054790 con 0x7fc3d80c6bc0 2023-10-16T20:36:52.514 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:36:52.514 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":19,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:36:47.255863+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:36:52.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.507+0000 7fc3ded9d640 1 -- 172.21.15.88:0/744335065 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc3c40664a0 msgr2=0x7fc3c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:52.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.507+0000 7fc3ded9d640 1 --2- 172.21.15.88:0/744335065 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc3c40664a0 0x7fc3c4068960 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc3c80ae000 tx=0x7fc3c8004740 comp rx=0 tx=0).stop 2023-10-16T20:36:52.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.507+0000 7fc3ded9d640 1 -- 172.21.15.88:0/744335065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d80c6bc0 msgr2=0x7fc3d8141b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:52.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.507+0000 7fc3ded9d640 1 --2- 172.21.15.88:0/744335065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d80c6bc0 0x7fc3d8141b40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc3d400c450 tx=0x7fc3d400c910 comp rx=0 tx=0).stop 2023-10-16T20:36:52.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.507+0000 7fc3ded9d640 1 -- 172.21.15.88:0/744335065 shutdown_connections 2023-10-16T20:36:52.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.507+0000 7fc3ded9d640 1 --2- 172.21.15.88:0/744335065 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc3c40664a0 0x7fc3c4068960 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.507+0000 7fc3ded9d640 1 --2- 172.21.15.88:0/744335065 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3d8150660 0x7fc3d813e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.507+0000 7fc3ded9d640 1 --2- 172.21.15.88:0/744335065 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3d8150280 0x7fc3d813bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.507+0000 7fc3ded9d640 1 --2- 172.21.15.88:0/744335065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d80c6bc0 0x7fc3d8141b40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.507+0000 7fc3ded9d640 1 -- 172.21.15.88:0/744335065 >> 172.21.15.88:0/744335065 conn(0x7fc3d80cad50 msgr2=0x7fc3d806f710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:52.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.507+0000 7fc3ded9d640 1 -- 172.21.15.88:0/744335065 shutdown_connections 2023-10-16T20:36:52.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.511+0000 7fc3ded9d640 1 -- 172.21.15.88:0/744335065 wait complete. 2023-10-16T20:36:52.534 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T20:36:43.646418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-16T20:36:52.535 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T20:36:52.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b3a49e640 1 Processor -- start 2023-10-16T20:36:52.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b3a49e640 1 -- start start 2023-10-16T20:36:52.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b3a49e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b34071450 0x7f8b340c1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b3a49e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b341596e0 con 0x7f8b34153a30 2023-10-16T20:36:52.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b3a49e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b340c2da0 con 0x7f8b34071f90 2023-10-16T20:36:52.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b3a49e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b340c6cc0 con 0x7f8b34071450 2023-10-16T20:36:52.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b33fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8b34153a30 0x7f8b340c65b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46868/0 (socket says 172.21.15.88:46868) 2023-10-16T20:36:52.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b33fff640 1 -- 172.21.15.88:0/1235947036 learned_addr learned my addr 172.21.15.88:0/1235947036 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:36:52.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b337fe640 1 --2- 172.21.15.88:0/1235947036 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b34071450 0x7f8b340c1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b32ffd640 1 -- 172.21.15.88:0/1235947036 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860767433 0 0) 0x7f8b341596e0 con 0x7f8b34153a30 2023-10-16T20:36:52.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b32ffd640 1 -- 172.21.15.88:0/1235947036 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b1c003530 con 0x7f8b34153a30 2023-10-16T20:36:52.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b32ffd640 1 -- 172.21.15.88:0/1235947036 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109430115 0 0) 0x7f8b340c2da0 con 0x7f8b34071f90 2023-10-16T20:36:52.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b32ffd640 1 -- 172.21.15.88:0/1235947036 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b341596e0 con 0x7f8b34071f90 2023-10-16T20:36:52.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b32ffd640 1 -- 172.21.15.88:0/1235947036 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1289655903 0 0) 0x7f8b1c003530 con 0x7f8b34153a30 2023-10-16T20:36:52.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b32ffd640 1 -- 172.21.15.88:0/1235947036 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b34071450 msgr2=0x7f8b340c1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:52.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b32ffd640 1 --2- 172.21.15.88:0/1235947036 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b34071450 0x7f8b340c1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b32ffd640 1 -- 172.21.15.88:0/1235947036 >> v1:172.21.15.191:6789/0 conn(0x7f8b34071f90 legacy=0x7f8b34163f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:52.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b32ffd640 1 -- 172.21.15.88:0/1235947036 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b340c1bb0 con 0x7f8b34153a30 2023-10-16T20:36:52.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b32ffd640 1 -- 172.21.15.88:0/1235947036 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f8b24001680 con 0x7f8b34153a30 2023-10-16T20:36:52.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b32ffd640 1 -- 172.21.15.88:0/1235947036 >> v1:172.21.15.88:6789/0 conn(0x7f8b34153a30 legacy=0x7f8b340c65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:52.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b32ffd640 1 --2- 172.21.15.88:0/1235947036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b1c003b40 0x7f8b1c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b32ffd640 1 --2- 172.21.15.88:0/1235947036 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b34071450 0x7f8b1c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b32ffd640 1 --2- 172.21.15.88:0/1235947036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b1c004d10 0x7f8b1c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b32ffd640 1 -- 172.21.15.88:0/1235947036 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b340c6cc0 con 0x7f8b1c004d10 2023-10-16T20:36:52.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b32ffd640 1 -- 172.21.15.88:0/1235947036 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c00ba40 con 0x7f8b1c003b40 2023-10-16T20:36:52.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b32ffd640 1 -- 172.21.15.88:0/1235947036 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c00bc00 con 0x7f8b34071450 2023-10-16T20:36:52.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b38a3b640 1 --2- 172.21.15.88:0/1235947036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b1c004d10 0x7f8b1c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b33fff640 1 --2- 172.21.15.88:0/1235947036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b1c003b40 0x7f8b1c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.671+0000 7f8b337fe640 1 --2- 172.21.15.88:0/1235947036 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b34071450 0x7f8b1c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.675+0000 7f8b38a3b640 1 -- 172.21.15.88:0/1235947036 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b34071450 msgr2=0x7f8b1c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:52.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.675+0000 7f8b38a3b640 1 --2- 172.21.15.88:0/1235947036 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b34071450 0x7f8b1c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.675+0000 7f8b38a3b640 1 -- 172.21.15.88:0/1235947036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b1c003b40 msgr2=0x7f8b1c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:52.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.675+0000 7f8b38a3b640 1 --2- 172.21.15.88:0/1235947036 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b1c003b40 0x7f8b1c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.675+0000 7f8b38a3b640 1 -- 172.21.15.88:0/1235947036 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1c003530 con 0x7f8b1c004d10 2023-10-16T20:36:52.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.675+0000 7f8b38a3b640 1 --2- 172.21.15.88:0/1235947036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b1c004d10 0x7f8b1c005190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c001fd0 tx=0x7f8b2c01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df10631bdd8234a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:52.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.675+0000 7f8b32ffd640 1 -- 172.21.15.88:0/1235947036 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b24007ea0 con 0x7f8b1c004d10 2023-10-16T20:36:52.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.675+0000 7f8b32ffd640 1 -- 172.21.15.88:0/1235947036 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2c003ba0 con 0x7f8b1c004d10 2023-10-16T20:36:52.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.675+0000 7f8b32ffd640 1 -- 172.21.15.88:0/1235947036 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c0050d0 con 0x7f8b1c004d10 2023-10-16T20:36:52.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.675+0000 7f8b3a49e640 1 -- 172.21.15.88:0/1235947036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b1c004d10 msgr2=0x7f8b1c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:52.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.675+0000 7f8b3a49e640 1 --2- 172.21.15.88:0/1235947036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b1c004d10 0x7f8b1c005190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c001fd0 tx=0x7f8b2c01c8e0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.675+0000 7f8b38a3b640 1 -- 172.21.15.88:0/1235947036 reap_dead start 2023-10-16T20:36:52.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.675+0000 7f8b3a49e640 1 -- 172.21.15.88:0/1235947036 shutdown_connections 2023-10-16T20:36:52.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.675+0000 7f8b3a49e640 1 -- 172.21.15.88:0/1235947036 >> 172.21.15.88:0/1235947036 conn(0x7f8b340c86b0 msgr2=0x7f8b340c8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:52.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b3a49e640 1 -- 172.21.15.88:0/1235947036 shutdown_connections 2023-10-16T20:36:52.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b3a49e640 1 -- 172.21.15.88:0/1235947036 wait complete. 2023-10-16T20:36:52.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b3a49e640 1 Processor -- start 2023-10-16T20:36:52.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b3a49e640 1 -- start start 2023-10-16T20:36:52.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b3a49e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b3414c8c0 0x7f8b341f2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b3a49e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b3414cca0 0x7f8b3406a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b3a49e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b340c2f90 0x7f8b3406aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b3a49e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b340c39d0 con 0x7f8b3414c8c0 2023-10-16T20:36:52.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b3a49e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b340c4590 con 0x7f8b340c2f90 2023-10-16T20:36:52.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b3a49e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b341557b0 con 0x7f8b3414cca0 2023-10-16T20:36:52.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b33fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b3414c8c0 0x7f8b341f2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b33fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b3414c8c0 0x7f8b341f2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33528/0 (socket says 172.21.15.88:33528) 2023-10-16T20:36:52.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b33fff640 1 -- 172.21.15.88:0/3954732200 learned_addr learned my addr 172.21.15.88:0/3954732200 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:52.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b337fe640 1 --2- 172.21.15.88:0/3954732200 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b3414cca0 0x7f8b3406a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b38a3b640 1 --2- 172.21.15.88:0/3954732200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b340c2f90 0x7f8b3406aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b33fff640 1 -- 172.21.15.88:0/3954732200 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b3414cca0 msgr2=0x7f8b3406a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:52.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b33fff640 1 --2- 172.21.15.88:0/3954732200 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b3414cca0 0x7f8b3406a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b33fff640 1 -- 172.21.15.88:0/3954732200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b340c2f90 msgr2=0x7f8b3406aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:52.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b33fff640 1 --2- 172.21.15.88:0/3954732200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b340c2f90 0x7f8b3406aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b33fff640 1 -- 172.21.15.88:0/3954732200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b34154060 con 0x7f8b3414c8c0 2023-10-16T20:36:52.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.679+0000 7f8b33fff640 1 --2- 172.21.15.88:0/3954732200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b3414c8c0 0x7f8b341f2c80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8b24004850 tx=0x7f8b240129a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:52.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.683+0000 7f8b30ff9640 1 -- 172.21.15.88:0/3954732200 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b240148d0 con 0x7f8b3414c8c0 2023-10-16T20:36:52.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.683+0000 7f8b30ff9640 1 -- 172.21.15.88:0/3954732200 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b24014a50 con 0x7f8b3414c8c0 2023-10-16T20:36:52.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.683+0000 7f8b3a49e640 1 -- 172.21.15.88:0/3954732200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b34154340 con 0x7f8b3414c8c0 2023-10-16T20:36:52.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.683+0000 7f8b30ff9640 1 -- 172.21.15.88:0/3954732200 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b24014e00 con 0x7f8b3414c8c0 2023-10-16T20:36:52.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.683+0000 7f8b3a49e640 1 -- 172.21.15.88:0/3954732200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b34154900 con 0x7f8b3414c8c0 2023-10-16T20:36:52.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.683+0000 7f8b3a49e640 1 -- 172.21.15.88:0/3954732200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b34071230 con 0x7f8b3414c8c0 2023-10-16T20:36:52.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.683+0000 7f8b30ff9640 1 -- 172.21.15.88:0/3954732200 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8b240153e0 con 0x7f8b3414c8c0 2023-10-16T20:36:52.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.683+0000 7f8b30ff9640 1 --2- 172.21.15.88:0/3954732200 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8b200664d0 0x7f8b20068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:52.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.683+0000 7f8b30ff9640 1 -- 172.21.15.88:0/3954732200 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8b24086330 con 0x7f8b3414c8c0 2023-10-16T20:36:52.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.683+0000 7f8b337fe640 1 --2- 172.21.15.88:0/3954732200 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8b200664d0 0x7f8b20068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:52.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.683+0000 7f8b337fe640 1 --2- 172.21.15.88:0/3954732200 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8b200664d0 0x7f8b20068990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8b34158370 tx=0x7f8b28008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:52.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.687+0000 7f8b30ff9640 1 -- 172.21.15.88:0/3954732200 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8b24053af0 con 0x7f8b3414c8c0 2023-10-16T20:36:52.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.843+0000 7f8b3a49e640 1 -- 172.21.15.88:0/3954732200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8b340c9050 con 0x7f8b3414c8c0 2023-10-16T20:36:52.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.843+0000 7f8b30ff9640 1 -- 172.21.15.88:0/3954732200 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8b2401c020 con 0x7f8b3414c8c0 2023-10-16T20:36:52.849 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-10-16T20:36:52.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.843+0000 7f8b3a49e640 1 -- 172.21.15.88:0/3954732200 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8b200664d0 msgr2=0x7f8b20068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:52.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.843+0000 7f8b3a49e640 1 --2- 172.21.15.88:0/3954732200 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8b200664d0 0x7f8b20068990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8b34158370 tx=0x7f8b28008040 comp rx=0 tx=0).stop 2023-10-16T20:36:52.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.843+0000 7f8b3a49e640 1 -- 172.21.15.88:0/3954732200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b3414c8c0 msgr2=0x7f8b341f2c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:52.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.843+0000 7f8b3a49e640 1 --2- 172.21.15.88:0/3954732200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b3414c8c0 0x7f8b341f2c80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8b24004850 tx=0x7f8b240129a0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.843+0000 7f8b3a49e640 1 -- 172.21.15.88:0/3954732200 shutdown_connections 2023-10-16T20:36:52.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.843+0000 7f8b3a49e640 1 --2- 172.21.15.88:0/3954732200 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8b200664d0 0x7f8b20068990 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.843+0000 7f8b3a49e640 1 --2- 172.21.15.88:0/3954732200 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b340c2f90 0x7f8b3406aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.843+0000 7f8b3a49e640 1 --2- 172.21.15.88:0/3954732200 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b3414cca0 0x7f8b3406a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.843+0000 7f8b3a49e640 1 --2- 172.21.15.88:0/3954732200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b3414c8c0 0x7f8b341f2c80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:52.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.843+0000 7f8b3a49e640 1 -- 172.21.15.88:0/3954732200 >> 172.21.15.88:0/3954732200 conn(0x7f8b340c86b0 msgr2=0x7f8b340c9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:52.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.843+0000 7f8b3a49e640 1 -- 172.21.15.88:0/3954732200 shutdown_connections 2023-10-16T20:36:52.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.843+0000 7f8b3a49e640 1 -- 172.21.15.88:0/3954732200 wait complete. 2023-10-16T20:36:52.869 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-16T20:36:53.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:52.999+0000 7f764f668640 1 Processor -- start 2023-10-16T20:36:53.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.003+0000 7f764f668640 1 -- start start 2023-10-16T20:36:53.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.003+0000 7f764f668640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7648150660 0x7f76480c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.003+0000 7f764f668640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76480c6bc0 0x7f76480c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.003+0000 7f764f668640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76480c5c60 con 0x7f7648150280 2023-10-16T20:36:53.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.003+0000 7f764f668640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76480c5370 con 0x7f7648150660 2023-10-16T20:36:53.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.003+0000 7f764f668640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7648156d10 con 0x7f76480c6bc0 2023-10-16T20:36:53.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.003+0000 7f764cc03640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7648150660 0x7f76480c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.003+0000 7f764d404640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7648150280 0x7f764806d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46884/0 (socket says 172.21.15.88:46884) 2023-10-16T20:36:53.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.003+0000 7f764d404640 1 -- 172.21.15.88:0/2179528877 learned_addr learned my addr 172.21.15.88:0/2179528877 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:36:53.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.003+0000 7f764dc05640 1 --2- 172.21.15.88:0/2179528877 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76480c6bc0 0x7f76480c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.003+0000 7f764cc03640 1 -- 172.21.15.88:0/2179528877 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76480c6bc0 msgr2=0x7f76480c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:53.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.003+0000 7f764cc03640 1 --2- 172.21.15.88:0/2179528877 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76480c6bc0 0x7f76480c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.003+0000 7f764cc03640 1 -- 172.21.15.88:0/2179528877 >> v1:172.21.15.88:6789/0 conn(0x7f7648150280 legacy=0x7f764806d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:53.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.003+0000 7f764cc03640 1 -- 172.21.15.88:0/2179528877 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76480c5610 con 0x7f7648150660 2023-10-16T20:36:53.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.003+0000 7f763ffff640 1 -- 172.21.15.88:0/2179528877 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168348276 0 0) 0x7f76480c5c60 con 0x7f7648150280 2023-10-16T20:36:53.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.007+0000 7f764cc03640 1 --2- 172.21.15.88:0/2179528877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648150660 0x7f76480c60f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f76380097c0 tx=0x7f763801ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f2d0a930fa962a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:53.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.007+0000 7f763ffff640 1 -- 172.21.15.88:0/2179528877 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76380ab670 con 0x7f7648150660 2023-10-16T20:36:53.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.007+0000 7f763ffff640 1 -- 172.21.15.88:0/2179528877 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76380ab7d0 con 0x7f7648150660 2023-10-16T20:36:53.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.007+0000 7f763ffff640 1 -- 172.21.15.88:0/2179528877 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76380abb60 con 0x7f7648150660 2023-10-16T20:36:53.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.007+0000 7f764f668640 1 -- 172.21.15.88:0/2179528877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648150660 msgr2=0x7f76480c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:53.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.007+0000 7f764f668640 1 --2- 172.21.15.88:0/2179528877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648150660 0x7f76480c60f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f76380097c0 tx=0x7f763801ceb0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.007+0000 7f764f668640 1 -- 172.21.15.88:0/2179528877 shutdown_connections 2023-10-16T20:36:53.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.007+0000 7f764f668640 1 --2- 172.21.15.88:0/2179528877 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76480c6bc0 0x7f76480c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.007+0000 7f764f668640 1 --2- 172.21.15.88:0/2179528877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648150660 0x7f76480c60f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.007+0000 7f764f668640 1 -- 172.21.15.88:0/2179528877 >> 172.21.15.88:0/2179528877 conn(0x7f76480cad50 msgr2=0x7f76480cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:53.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.007+0000 7f764f668640 1 -- 172.21.15.88:0/2179528877 shutdown_connections 2023-10-16T20:36:53.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.007+0000 7f764f668640 1 -- 172.21.15.88:0/2179528877 wait complete. 2023-10-16T20:36:53.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.007+0000 7f764f668640 1 Processor -- start 2023-10-16T20:36:53.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764f668640 1 -- start start 2023-10-16T20:36:53.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764f668640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76480c6bc0 0x7f7648071ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764f668640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648150280 0x7f7648144f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764f668640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7648150660 0x7f76481f2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764f668640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76481f5200 con 0x7f7648150280 2023-10-16T20:36:53.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764f668640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76481f1d90 con 0x7f7648150660 2023-10-16T20:36:53.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764f668640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7648142ba0 con 0x7f76480c6bc0 2023-10-16T20:36:53.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764cc03640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648150280 0x7f7648144f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764cc03640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648150280 0x7f7648144f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33548/0 (socket says 172.21.15.88:33548) 2023-10-16T20:36:53.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764cc03640 1 -- 172.21.15.88:0/10488564 learned_addr learned my addr 172.21.15.88:0/10488564 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:53.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764d404640 1 --2- 172.21.15.88:0/10488564 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76480c6bc0 0x7f7648071ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764dc05640 1 --2- 172.21.15.88:0/10488564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7648150660 0x7f76481f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764cc03640 1 -- 172.21.15.88:0/10488564 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76480c6bc0 msgr2=0x7f7648071ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:53.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764cc03640 1 --2- 172.21.15.88:0/10488564 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76480c6bc0 0x7f7648071ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764cc03640 1 -- 172.21.15.88:0/10488564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7648150660 msgr2=0x7f76481f2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:53.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764cc03640 1 --2- 172.21.15.88:0/10488564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7648150660 0x7f76481f2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764cc03640 1 -- 172.21.15.88:0/10488564 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7638009420 con 0x7f7648150280 2023-10-16T20:36:53.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764cc03640 1 --2- 172.21.15.88:0/10488564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648150280 0x7f7648144f90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f76380ad040 tx=0x7f763801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:53.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f763dffb640 1 -- 172.21.15.88:0/10488564 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76380bf610 con 0x7f7648150280 2023-10-16T20:36:53.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f763dffb640 1 -- 172.21.15.88:0/10488564 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76380bf790 con 0x7f7648150280 2023-10-16T20:36:53.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f763dffb640 1 -- 172.21.15.88:0/10488564 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76380bfb40 con 0x7f7648150280 2023-10-16T20:36:53.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764f668640 1 -- 172.21.15.88:0/10488564 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76481f5430 con 0x7f7648150280 2023-10-16T20:36:53.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764f668640 1 -- 172.21.15.88:0/10488564 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76481f59f0 con 0x7f7648150280 2023-10-16T20:36:53.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.011+0000 7f764f668640 1 -- 172.21.15.88:0/10488564 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76481f1f40 con 0x7f7648150280 2023-10-16T20:36:53.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.015+0000 7f763dffb640 1 -- 172.21.15.88:0/10488564 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f76380b5400 con 0x7f7648150280 2023-10-16T20:36:53.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.015+0000 7f763dffb640 1 --2- 172.21.15.88:0/10488564 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f762c0664a0 0x7f762c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.015+0000 7f763dffb640 1 -- 172.21.15.88:0/10488564 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f763811e410 con 0x7f7648150280 2023-10-16T20:36:53.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.015+0000 7f764d404640 1 --2- 172.21.15.88:0/10488564 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f762c0664a0 0x7f762c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.015+0000 7f764d404640 1 --2- 172.21.15.88:0/10488564 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f762c0664a0 0x7f762c068960 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7644007fd0 tx=0x7f7644007d60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:53.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.019+0000 7f763dffb640 1 -- 172.21.15.88:0/10488564 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f76380ebb60 con 0x7f7648150280 2023-10-16T20:36:53.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.171+0000 7f764f668640 1 -- 172.21.15.88:0/10488564 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7648074700 con 0x7f7648150280 2023-10-16T20:36:53.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.171+0000 7f763dffb640 1 -- 172.21.15.88:0/10488564 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v19) v1 ==== 84+0+65 (secure 0 0 0) 0x7f76380ef180 con 0x7f7648150280 2023-10-16T20:36:53.176 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:36:53.176 INFO:teuthology.orchestra.run.smithi088.stdout:[ 2023-10-16T20:36:53.177 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-10-16T20:36:53.177 INFO:teuthology.orchestra.run.smithi088.stdout: "poolnum": 1, 2023-10-16T20:36:53.177 INFO:teuthology.orchestra.run.smithi088.stdout: "poolname": ".mgr" 2023-10-16T20:36:53.177 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-10-16T20:36:53.178 INFO:teuthology.orchestra.run.smithi088.stdout:] 2023-10-16T20:36:53.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.171+0000 7f764f668640 1 -- 172.21.15.88:0/10488564 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f762c0664a0 msgr2=0x7f762c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:53.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.171+0000 7f764f668640 1 --2- 172.21.15.88:0/10488564 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f762c0664a0 0x7f762c068960 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7644007fd0 tx=0x7f7644007d60 comp rx=0 tx=0).stop 2023-10-16T20:36:53.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.171+0000 7f764f668640 1 -- 172.21.15.88:0/10488564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648150280 msgr2=0x7f7648144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:53.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.171+0000 7f764f668640 1 --2- 172.21.15.88:0/10488564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648150280 0x7f7648144f90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f76380ad040 tx=0x7f763801ac90 comp rx=0 tx=0).stop 2023-10-16T20:36:53.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.171+0000 7f764f668640 1 -- 172.21.15.88:0/10488564 shutdown_connections 2023-10-16T20:36:53.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.171+0000 7f764f668640 1 --2- 172.21.15.88:0/10488564 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f762c0664a0 0x7f762c068960 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.171+0000 7f764f668640 1 --2- 172.21.15.88:0/10488564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7648150660 0x7f76481f2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.171+0000 7f764f668640 1 --2- 172.21.15.88:0/10488564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648150280 0x7f7648144f90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.171+0000 7f764f668640 1 --2- 172.21.15.88:0/10488564 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f76480c6bc0 0x7f7648071ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.171+0000 7f764f668640 1 -- 172.21.15.88:0/10488564 >> 172.21.15.88:0/10488564 conn(0x7f76480cad50 msgr2=0x7f76480c36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:53.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.171+0000 7f764f668640 1 -- 172.21.15.88:0/10488564 shutdown_connections 2023-10-16T20:36:53.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.171+0000 7f764f668640 1 -- 172.21.15.88:0/10488564 wait complete. 2023-10-16T20:36:53.197 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-16T20:36:53.197 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-16T20:36:53.267 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:36:53.271 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:36:53.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.331+0000 7fcb2cf3e640 1 Processor -- start 2023-10-16T20:36:53.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.331+0000 7fcb2cf3e640 1 -- start start 2023-10-16T20:36:53.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb2cf3e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcb28153620 0x7fcb280c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb2cf3e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcb28153a00 0x7fcb280c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb2cf3e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb280ad870 0x7fcb280cb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb2cf3e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb280c0230 con 0x7fcb28153620 2023-10-16T20:36:53.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb2cf3e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb281594f0 con 0x7fcb28153a00 2023-10-16T20:36:53.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb2cf3e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb280c0480 con 0x7fcb280ad870 2023-10-16T20:36:53.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb2659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcb28153620 0x7fcb280c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb2659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcb28153620 0x7fcb280c5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33564/0 (socket says 172.21.15.88:33564) 2023-10-16T20:36:53.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb2659c640 1 -- 172.21.15.88:0/2227955560 learned_addr learned my addr 172.21.15.88:0/2227955560 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:53.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb1dd9b640 1 --2- 172.21.15.88:0/2227955560 >> v2:172.21.15.191:3300/0 conn(0x7fcb28153a00 0x7fcb280c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb2659c640 1 -- 172.21.15.88:0/2227955560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb280ad870 msgr2=0x7fcb280cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:53.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb26d9d640 1 --2- 172.21.15.88:0/2227955560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb280ad870 0x7fcb280cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb2659c640 1 --2- 172.21.15.88:0/2227955560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb280ad870 0x7fcb280cb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb2659c640 1 -- 172.21.15.88:0/2227955560 >> v2:172.21.15.191:3300/0 conn(0x7fcb28153a00 msgr2=0x7fcb280c61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:53.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb2659c640 1 --2- 172.21.15.88:0/2227955560 >> v2:172.21.15.191:3300/0 conn(0x7fcb28153a00 0x7fcb280c61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb2659c640 1 -- 172.21.15.88:0/2227955560 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb280c0760 con 0x7fcb28153620 2023-10-16T20:36:53.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb1dd9b640 1 --2- 172.21.15.88:0/2227955560 >> v2:172.21.15.191:3300/0 conn(0x7fcb28153a00 0x7fcb280c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:36:53.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.335+0000 7fcb26d9d640 1 --2- 172.21.15.88:0/2227955560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb280ad870 0x7fcb280cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:36:53.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.339+0000 7fcb2659c640 1 --2- 172.21.15.88:0/2227955560 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb28153620 0x7fcb280c5c70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcb28142bc0 tx=0x7fcb1801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162a48fc8b92a6e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:53.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.339+0000 7fcb25d9b640 1 -- 172.21.15.88:0/2227955560 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb180a55c0 con 0x7fcb28153620 2023-10-16T20:36:53.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.339+0000 7fcb25d9b640 1 -- 172.21.15.88:0/2227955560 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb180a5720 con 0x7fcb28153620 2023-10-16T20:36:53.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.339+0000 7fcb25d9b640 1 -- 172.21.15.88:0/2227955560 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb180a5ab0 con 0x7fcb28153620 2023-10-16T20:36:53.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.339+0000 7fcb2cf3e640 1 -- 172.21.15.88:0/2227955560 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb28153620 msgr2=0x7fcb280c5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:53.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.339+0000 7fcb2cf3e640 1 --2- 172.21.15.88:0/2227955560 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb28153620 0x7fcb280c5c70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcb28142bc0 tx=0x7fcb1801c940 comp rx=0 tx=0).stop 2023-10-16T20:36:53.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.339+0000 7fcb2cf3e640 1 -- 172.21.15.88:0/2227955560 shutdown_connections 2023-10-16T20:36:53.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.339+0000 7fcb2cf3e640 1 --2- 172.21.15.88:0/2227955560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb280ad870 0x7fcb280cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.339+0000 7fcb2cf3e640 1 --2- 172.21.15.88:0/2227955560 >> v2:172.21.15.191:3300/0 conn(0x7fcb28153a00 0x7fcb280c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.339+0000 7fcb2cf3e640 1 --2- 172.21.15.88:0/2227955560 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb28153620 0x7fcb280c5c70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.339+0000 7fcb2cf3e640 1 -- 172.21.15.88:0/2227955560 >> 172.21.15.88:0/2227955560 conn(0x7fcb28069e90 msgr2=0x7fcb2806a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:53.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.339+0000 7fcb2cf3e640 1 -- 172.21.15.88:0/2227955560 shutdown_connections 2023-10-16T20:36:53.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.339+0000 7fcb2cf3e640 1 -- 172.21.15.88:0/2227955560 wait complete. 2023-10-16T20:36:53.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.339+0000 7fcb2cf3e640 1 Processor -- start 2023-10-16T20:36:53.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.339+0000 7fcb2cf3e640 1 -- start start 2023-10-16T20:36:53.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb2cf3e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb280ad870 0x7fcb280c3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb2cf3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb28153620 0x7fcb280c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb2cf3e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb28153a00 0x7fcb2813da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb2cf3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb280c3960 con 0x7fcb28153620 2023-10-16T20:36:53.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb2cf3e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb281548c0 con 0x7fcb280ad870 2023-10-16T20:36:53.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb2cf3e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb281f5ae0 con 0x7fcb28153a00 2023-10-16T20:36:53.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb1dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb28153620 0x7fcb280c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb1dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb28153620 0x7fcb280c7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33574/0 (socket says 172.21.15.88:33574) 2023-10-16T20:36:53.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb1dd9b640 1 -- 172.21.15.88:0/893215450 learned_addr learned my addr 172.21.15.88:0/893215450 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:53.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb2659c640 1 --2- 172.21.15.88:0/893215450 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb280ad870 0x7fcb280c3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb26d9d640 1 --2- 172.21.15.88:0/893215450 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb28153a00 0x7fcb2813da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb1dd9b640 1 -- 172.21.15.88:0/893215450 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb28153a00 msgr2=0x7fcb2813da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:53.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb1dd9b640 1 --2- 172.21.15.88:0/893215450 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb28153a00 0x7fcb2813da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb1dd9b640 1 -- 172.21.15.88:0/893215450 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb280ad870 msgr2=0x7fcb280c3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:53.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb1dd9b640 1 --2- 172.21.15.88:0/893215450 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb280ad870 0x7fcb280c3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb1dd9b640 1 -- 172.21.15.88:0/893215450 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb18009b30 con 0x7fcb28153620 2023-10-16T20:36:53.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb1dd9b640 1 --2- 172.21.15.88:0/893215450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb28153620 0x7fcb280c7b10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcb1000c7f0 tx=0x7fcb1000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:53.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb1f7fe640 1 -- 172.21.15.88:0/893215450 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb100077c0 con 0x7fcb28153620 2023-10-16T20:36:53.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb1f7fe640 1 -- 172.21.15.88:0/893215450 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb10007920 con 0x7fcb28153620 2023-10-16T20:36:53.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb1f7fe640 1 -- 172.21.15.88:0/893215450 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb10007cd0 con 0x7fcb28153620 2023-10-16T20:36:53.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb2cf3e640 1 -- 172.21.15.88:0/893215450 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb281f5da0 con 0x7fcb28153620 2023-10-16T20:36:53.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb2cf3e640 1 -- 172.21.15.88:0/893215450 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb281f6360 con 0x7fcb28153620 2023-10-16T20:36:53.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb2cf3e640 1 -- 172.21.15.88:0/893215450 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb280b15f0 con 0x7fcb28153620 2023-10-16T20:36:53.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.343+0000 7fcb1f7fe640 1 -- 172.21.15.88:0/893215450 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcb100113f0 con 0x7fcb28153620 2023-10-16T20:36:53.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.347+0000 7fcb1f7fe640 1 --2- 172.21.15.88:0/893215450 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb080664d0 0x7fcb08068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.347+0000 7fcb1f7fe640 1 -- 172.21.15.88:0/893215450 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcb10083cd0 con 0x7fcb28153620 2023-10-16T20:36:53.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.347+0000 7fcb2659c640 1 --2- 172.21.15.88:0/893215450 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb080664d0 0x7fcb08068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.347+0000 7fcb2659c640 1 --2- 172.21.15.88:0/893215450 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb080664d0 0x7fcb08068990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcb180ae000 tx=0x7fcb180a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:53.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.351+0000 7fcb1f7fe640 1 -- 172.21.15.88:0/893215450 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcb10051540 con 0x7fcb28153620 2023-10-16T20:36:53.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.499+0000 7fcb2cf3e640 1 -- 172.21.15.88:0/893215450 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fcb28074700 con 0x7fcb28153620 2023-10-16T20:36:53.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.715+0000 7fcb1f7fe640 1 -- 172.21.15.88:0/893215450 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v20) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcb10054b60 con 0x7fcb28153620 2023-10-16T20:36:53.722 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' created 2023-10-16T20:36:53.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.715+0000 7fcb2cf3e640 1 -- 172.21.15.88:0/893215450 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb080664d0 msgr2=0x7fcb08068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:53.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.715+0000 7fcb2cf3e640 1 --2- 172.21.15.88:0/893215450 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb080664d0 0x7fcb08068990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcb180ae000 tx=0x7fcb180a7070 comp rx=0 tx=0).stop 2023-10-16T20:36:53.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.715+0000 7fcb2cf3e640 1 -- 172.21.15.88:0/893215450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb28153620 msgr2=0x7fcb280c7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:53.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.715+0000 7fcb2cf3e640 1 --2- 172.21.15.88:0/893215450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb28153620 0x7fcb280c7b10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcb1000c7f0 tx=0x7fcb1000ccb0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.719+0000 7fcb2cf3e640 1 -- 172.21.15.88:0/893215450 shutdown_connections 2023-10-16T20:36:53.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.719+0000 7fcb2cf3e640 1 --2- 172.21.15.88:0/893215450 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb080664d0 0x7fcb08068990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.719+0000 7fcb2cf3e640 1 --2- 172.21.15.88:0/893215450 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb28153a00 0x7fcb2813da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.719+0000 7fcb2cf3e640 1 --2- 172.21.15.88:0/893215450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb28153620 0x7fcb280c7b10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.719+0000 7fcb2cf3e640 1 --2- 172.21.15.88:0/893215450 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb280ad870 0x7fcb280c3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.719+0000 7fcb2cf3e640 1 -- 172.21.15.88:0/893215450 >> 172.21.15.88:0/893215450 conn(0x7fcb28069e90 msgr2=0x7fcb2815add0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:53.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.719+0000 7fcb2cf3e640 1 -- 172.21.15.88:0/893215450 shutdown_connections 2023-10-16T20:36:53.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.719+0000 7fcb2cf3e640 1 -- 172.21.15.88:0/893215450 wait complete. 2023-10-16T20:36:53.750 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-10-16T20:36:53.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.927+0000 7f86ab0d6640 1 Processor -- start 2023-10-16T20:36:53.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.927+0000 7f86ab0d6640 1 -- start start 2023-10-16T20:36:53.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.927+0000 7f86ab0d6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f86a40c3b70 0x7f86a415d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.927+0000 7f86ab0d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a40ca090 con 0x7f86a415d610 2023-10-16T20:36:53.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.927+0000 7f86ab0d6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a4153c40 con 0x7f86a4153860 2023-10-16T20:36:53.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f86ab0d6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a40c4250 con 0x7f86a40c3b70 2023-10-16T20:36:53.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f86a9673640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f86a415d610 0x7f86a40cebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46886/0 (socket says 172.21.15.88:46886) 2023-10-16T20:36:53.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f86a9673640 1 -- 172.21.15.88:0/1050247715 learned_addr learned my addr 172.21.15.88:0/1050247715 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:36:53.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869bfff640 1 --2- 172.21.15.88:0/1050247715 >> v2:172.21.15.191:3300/0 conn(0x7f86a40c3b70 0x7f86a415d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869b7fe640 1 -- 172.21.15.88:0/1050247715 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988671527 0 0) 0x7f86a40ca090 con 0x7f86a415d610 2023-10-16T20:36:53.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869b7fe640 1 -- 172.21.15.88:0/1050247715 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f868c003530 con 0x7f86a415d610 2023-10-16T20:36:53.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869b7fe640 1 -- 172.21.15.88:0/1050247715 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156757774 0 0) 0x7f86a4153c40 con 0x7f86a4153860 2023-10-16T20:36:53.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869b7fe640 1 -- 172.21.15.88:0/1050247715 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a40ca090 con 0x7f86a4153860 2023-10-16T20:36:53.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869b7fe640 1 -- 172.21.15.88:0/1050247715 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2937933106 0 0) 0x7f868c003530 con 0x7f86a415d610 2023-10-16T20:36:53.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869b7fe640 1 -- 172.21.15.88:0/1050247715 >> v2:172.21.15.191:3300/0 conn(0x7f86a40c3b70 msgr2=0x7f86a415d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:53.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869b7fe640 1 --2- 172.21.15.88:0/1050247715 >> v2:172.21.15.191:3300/0 conn(0x7f86a40c3b70 0x7f86a415d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869b7fe640 1 -- 172.21.15.88:0/1050247715 >> v1:172.21.15.191:6789/0 conn(0x7f86a4153860 legacy=0x7f86a4159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:53.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869b7fe640 1 -- 172.21.15.88:0/1050247715 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a40c9db0 con 0x7f86a415d610 2023-10-16T20:36:53.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869b7fe640 1 -- 172.21.15.88:0/1050247715 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f869c001a00 con 0x7f86a415d610 2023-10-16T20:36:53.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869b7fe640 1 -- 172.21.15.88:0/1050247715 >> v1:172.21.15.88:6789/0 conn(0x7f86a415d610 legacy=0x7f86a40cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:53.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869b7fe640 1 --2- 172.21.15.88:0/1050247715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f868c003b10 0x7f868c003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869b7fe640 1 --2- 172.21.15.88:0/1050247715 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f868c0047d0 0x7f868c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869b7fe640 1 --2- 172.21.15.88:0/1050247715 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f868c005190 0x7f868c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869b7fe640 1 -- 172.21.15.88:0/1050247715 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a40c4250 con 0x7f868c003b10 2023-10-16T20:36:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869b7fe640 1 -- 172.21.15.88:0/1050247715 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c00c0c0 con 0x7f868c005190 2023-10-16T20:36:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869b7fe640 1 -- 172.21.15.88:0/1050247715 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868c00c280 con 0x7f868c0047d0 2023-10-16T20:36:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f86a8e72640 1 --2- 172.21.15.88:0/1050247715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f868c003b10 0x7f868c003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f86a9673640 1 --2- 172.21.15.88:0/1050247715 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f868c005190 0x7f868c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869bfff640 1 --2- 172.21.15.88:0/1050247715 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f868c0047d0 0x7f868c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f86a8e72640 1 -- 172.21.15.88:0/1050247715 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f868c0047d0 msgr2=0x7f868c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:53.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f86a8e72640 1 --2- 172.21.15.88:0/1050247715 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f868c0047d0 0x7f868c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f86a8e72640 1 -- 172.21.15.88:0/1050247715 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f868c005190 msgr2=0x7f868c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:53.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f86a8e72640 1 --2- 172.21.15.88:0/1050247715 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f868c005190 0x7f868c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f86a8e72640 1 -- 172.21.15.88:0/1050247715 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868c003530 con 0x7f868c003b10 2023-10-16T20:36:53.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f86a9673640 1 -- 172.21.15.88:0/1050247715 reap_dead start 2023-10-16T20:36:53.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.931+0000 7f869bfff640 1 --2- 172.21.15.88:0/1050247715 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f868c0047d0 0x7f868c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:36:53.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.935+0000 7f86a8e72640 1 --2- 172.21.15.88:0/1050247715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f868c003b10 0x7f868c003f10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f86a00018d0 tx=0x7f86a001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22369febd7a062f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:53.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.935+0000 7f869b7fe640 1 -- 172.21.15.88:0/1050247715 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a00052d0 con 0x7f868c003b10 2023-10-16T20:36:53.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.935+0000 7f869b7fe640 1 -- 172.21.15.88:0/1050247715 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a00a7050 con 0x7f868c003b10 2023-10-16T20:36:53.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.935+0000 7f869b7fe640 1 -- 172.21.15.88:0/1050247715 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a0003600 con 0x7f868c003b10 2023-10-16T20:36:53.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.935+0000 7f86ab0d6640 1 -- 172.21.15.88:0/1050247715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f868c003b10 msgr2=0x7f868c003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:53.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.935+0000 7f86ab0d6640 1 --2- 172.21.15.88:0/1050247715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f868c003b10 0x7f868c003f10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f86a00018d0 tx=0x7f86a001c540 comp rx=0 tx=0).stop 2023-10-16T20:36:53.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.935+0000 7f86ab0d6640 1 -- 172.21.15.88:0/1050247715 shutdown_connections 2023-10-16T20:36:53.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.935+0000 7f86ab0d6640 1 --2- 172.21.15.88:0/1050247715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f868c003b10 0x7f868c003f10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.935+0000 7f86ab0d6640 1 -- 172.21.15.88:0/1050247715 >> 172.21.15.88:0/1050247715 conn(0x7f86a40715c0 msgr2=0x7f86a40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:53.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.935+0000 7f86ab0d6640 1 -- 172.21.15.88:0/1050247715 shutdown_connections 2023-10-16T20:36:53.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.935+0000 7f86ab0d6640 1 -- 172.21.15.88:0/1050247715 wait complete. 2023-10-16T20:36:53.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.935+0000 7f86ab0d6640 1 Processor -- start 2023-10-16T20:36:53.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.935+0000 7f86ab0d6640 1 -- start start 2023-10-16T20:36:53.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86ab0d6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f868c003b10 0x7f86a414b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86ab0d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86a40cd930 0x7f86a40b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86ab0d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a41594e0 0x7f86a41488a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86ab0d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a40c8520 con 0x7f86a41594e0 2023-10-16T20:36:53.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86ab0d6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a4152730 con 0x7f868c003b10 2023-10-16T20:36:53.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86ab0d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a4159050 con 0x7f86a40cd930 2023-10-16T20:36:53.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86a9673640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a41594e0 0x7f86a41488a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86a9673640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a41594e0 0x7f86a41488a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33598/0 (socket says 172.21.15.88:33598) 2023-10-16T20:36:53.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86a9673640 1 -- 172.21.15.88:0/642410758 learned_addr learned my addr 172.21.15.88:0/642410758 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:53.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86a8e72640 1 --2- 172.21.15.88:0/642410758 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f868c003b10 0x7f86a414b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f869bfff640 1 --2- 172.21.15.88:0/642410758 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86a40cd930 0x7f86a40b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86a9673640 1 -- 172.21.15.88:0/642410758 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86a40cd930 msgr2=0x7f86a40b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:53.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86a9673640 1 --2- 172.21.15.88:0/642410758 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86a40cd930 0x7f86a40b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86a9673640 1 -- 172.21.15.88:0/642410758 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f868c003b10 msgr2=0x7f86a414b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:53.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86a9673640 1 --2- 172.21.15.88:0/642410758 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f868c003b10 0x7f86a414b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:53.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86a9673640 1 -- 172.21.15.88:0/642410758 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a4152960 con 0x7f86a41594e0 2023-10-16T20:36:53.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86a9673640 1 --2- 172.21.15.88:0/642410758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a41594e0 0x7f86a41488a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f869c001340 tx=0x7f869c007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:53.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86997fa640 1 -- 172.21.15.88:0/642410758 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c00ab30 con 0x7f86a41594e0 2023-10-16T20:36:53.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86997fa640 1 -- 172.21.15.88:0/642410758 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869c00ac90 con 0x7f86a41594e0 2023-10-16T20:36:53.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86997fa640 1 -- 172.21.15.88:0/642410758 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c0133f0 con 0x7f86a41594e0 2023-10-16T20:36:53.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86ab0d6640 1 -- 172.21.15.88:0/642410758 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a40c8240 con 0x7f86a41594e0 2023-10-16T20:36:53.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86ab0d6640 1 -- 172.21.15.88:0/642410758 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a4150bd0 con 0x7f86a41594e0 2023-10-16T20:36:53.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86ab0d6640 1 -- 172.21.15.88:0/642410758 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a4150750 con 0x7f86a41594e0 2023-10-16T20:36:53.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.939+0000 7f86997fa640 1 -- 172.21.15.88:0/642410758 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f869c013550 con 0x7f86a41594e0 2023-10-16T20:36:53.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.943+0000 7f86997fa640 1 --2- 172.21.15.88:0/642410758 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f86900664a0 0x7f8690068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:53.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.943+0000 7f86997fa640 1 -- 172.21.15.88:0/642410758 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f869c0841a0 con 0x7f86a41594e0 2023-10-16T20:36:53.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.943+0000 7f86a8e72640 1 --2- 172.21.15.88:0/642410758 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f86900664a0 0x7f8690068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:53.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.943+0000 7f86a8e72640 1 --2- 172.21.15.88:0/642410758 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f86900664a0 0x7f8690068960 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f86a00ac000 tx=0x7f86a00a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:53.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:53.947+0000 7f86997fa640 1 -- 172.21.15.88:0/642410758 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f86a4150750 con 0x7f86a41594e0 2023-10-16T20:36:54.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.095+0000 7f86ab0d6640 1 -- 172.21.15.88:0/642410758 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f86a4150750 con 0x7f86a41594e0 2023-10-16T20:36:54.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.719+0000 7f86997fa640 1 -- 172.21.15.88:0/642410758 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v21) v1 ==== 171+0+0 (secure 0 0 0) 0x7f86a4150750 con 0x7f86a41594e0 2023-10-16T20:36:54.725 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' created 2023-10-16T20:36:54.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.723+0000 7f86ab0d6640 1 -- 172.21.15.88:0/642410758 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f86900664a0 msgr2=0x7f8690068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:54.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.723+0000 7f86ab0d6640 1 --2- 172.21.15.88:0/642410758 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f86900664a0 0x7f8690068960 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f86a00ac000 tx=0x7f86a00a6000 comp rx=0 tx=0).stop 2023-10-16T20:36:54.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.723+0000 7f86ab0d6640 1 -- 172.21.15.88:0/642410758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a41594e0 msgr2=0x7f86a41488a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:54.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.723+0000 7f86ab0d6640 1 --2- 172.21.15.88:0/642410758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a41594e0 0x7f86a41488a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f869c001340 tx=0x7f869c007b20 comp rx=0 tx=0).stop 2023-10-16T20:36:54.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.723+0000 7f86ab0d6640 1 -- 172.21.15.88:0/642410758 shutdown_connections 2023-10-16T20:36:54.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.723+0000 7f86ab0d6640 1 --2- 172.21.15.88:0/642410758 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f86900664a0 0x7f8690068960 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:54.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.723+0000 7f86ab0d6640 1 --2- 172.21.15.88:0/642410758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a41594e0 0x7f86a41488a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:54.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.723+0000 7f86ab0d6640 1 --2- 172.21.15.88:0/642410758 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f86a40cd930 0x7f86a40b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:54.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.723+0000 7f86ab0d6640 1 --2- 172.21.15.88:0/642410758 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f868c003b10 0x7f86a414b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:54.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.723+0000 7f86ab0d6640 1 -- 172.21.15.88:0/642410758 >> 172.21.15.88:0/642410758 conn(0x7f86a40715c0 msgr2=0x7f86a406b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:54.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.723+0000 7f86ab0d6640 1 -- 172.21.15.88:0/642410758 shutdown_connections 2023-10-16T20:36:54.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.723+0000 7f86ab0d6640 1 -- 172.21.15.88:0/642410758 wait complete. 2023-10-16T20:36:54.772 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-16T20:36:54.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.907+0000 7f353ac09640 1 Processor -- start 2023-10-16T20:36:54.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.907+0000 7f353ac09640 1 -- start start 2023-10-16T20:36:54.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.911+0000 7f353ac09640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3534159e20 0x7f3534164d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:54.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.911+0000 7f353ac09640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35340caf10 con 0x7f35340c9be0 2023-10-16T20:36:54.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.911+0000 7f353ac09640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35340cc4c0 con 0x7f35341535d0 2023-10-16T20:36:54.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.911+0000 7f353ac09640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35340cb140 con 0x7f3534159e20 2023-10-16T20:36:54.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.911+0000 7f3539406640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f35340c9be0 0x7f35340cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46902/0 (socket says 172.21.15.88:46902) 2023-10-16T20:36:54.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.911+0000 7f3539406640 1 -- 172.21.15.88:0/2002666724 learned_addr learned my addr 172.21.15.88:0/2002666724 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:36:54.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.911+0000 7f353a408640 1 --2- 172.21.15.88:0/2002666724 >> v2:172.21.15.191:3300/0 conn(0x7f3534159e20 0x7f3534164d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:54.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.911+0000 7f3538c05640 1 -- 172.21.15.88:0/2002666724 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437011235 0 0) 0x7f35340caf10 con 0x7f35340c9be0 2023-10-16T20:36:54.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.911+0000 7f3538c05640 1 -- 172.21.15.88:0/2002666724 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3520003530 con 0x7f35340c9be0 2023-10-16T20:36:54.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.911+0000 7f3538c05640 1 -- 172.21.15.88:0/2002666724 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252582411 0 0) 0x7f35340cc4c0 con 0x7f35341535d0 2023-10-16T20:36:54.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.911+0000 7f3538c05640 1 -- 172.21.15.88:0/2002666724 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35340caf10 con 0x7f35341535d0 2023-10-16T20:36:54.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.911+0000 7f353a408640 1 -- 172.21.15.88:0/2002666724 >> v1:172.21.15.191:6789/0 conn(0x7f35341535d0 legacy=0x7f35340c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:54.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.911+0000 7f3538c05640 1 -- 172.21.15.88:0/2002666724 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1161833465 0 0) 0x7f3520003530 con 0x7f35340c9be0 2023-10-16T20:36:54.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.911+0000 7f353a408640 1 -- 172.21.15.88:0/2002666724 >> v1:172.21.15.88:6789/0 conn(0x7f35340c9be0 legacy=0x7f35340cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:54.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.911+0000 7f353a408640 1 -- 172.21.15.88:0/2002666724 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35340cb420 con 0x7f3534159e20 2023-10-16T20:36:54.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.915+0000 7f353a408640 1 --2- 172.21.15.88:0/2002666724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3534159e20 0x7f3534164d30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f352c009d90 tx=0x7f352c01f060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32c39829c3683183 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:54.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.915+0000 7f3538c05640 1 -- 172.21.15.88:0/2002666724 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3524001680 con 0x7f3534159e20 2023-10-16T20:36:54.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.915+0000 7f3538c05640 1 -- 172.21.15.88:0/2002666724 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352c0a8bf0 con 0x7f3534159e20 2023-10-16T20:36:54.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.915+0000 7f3538c05640 1 -- 172.21.15.88:0/2002666724 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352c0b0440 con 0x7f3534159e20 2023-10-16T20:36:54.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.915+0000 7f353ac09640 1 -- 172.21.15.88:0/2002666724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3534159e20 msgr2=0x7f3534164d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:54.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.915+0000 7f353ac09640 1 --2- 172.21.15.88:0/2002666724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3534159e20 0x7f3534164d30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f352c009d90 tx=0x7f352c01f060 comp rx=0 tx=0).stop 2023-10-16T20:36:54.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.915+0000 7f353ac09640 1 -- 172.21.15.88:0/2002666724 shutdown_connections 2023-10-16T20:36:54.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.915+0000 7f353ac09640 1 --2- 172.21.15.88:0/2002666724 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3534159e20 0x7f3534164d30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:54.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.915+0000 7f353ac09640 1 -- 172.21.15.88:0/2002666724 >> 172.21.15.88:0/2002666724 conn(0x7f353406bbe0 msgr2=0x7f353406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:54.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.915+0000 7f353ac09640 1 -- 172.21.15.88:0/2002666724 shutdown_connections 2023-10-16T20:36:54.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.915+0000 7f353ac09640 1 -- 172.21.15.88:0/2002666724 wait complete. 2023-10-16T20:36:54.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.915+0000 7f353ac09640 1 Processor -- start 2023-10-16T20:36:54.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.915+0000 7f353ac09640 1 -- start start 2023-10-16T20:36:54.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f353ac09640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35340c9be0 0x7f35340b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:54.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f353ac09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35341535d0 0x7f35341ef2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:54.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f353ac09640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3534159e20 0x7f353414ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:54.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f353ac09640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35340c8600 con 0x7f35341535d0 2023-10-16T20:36:54.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f353ac09640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35340b4ab0 con 0x7f35340c9be0 2023-10-16T20:36:54.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f353ac09640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35341f2890 con 0x7f3534159e20 2023-10-16T20:36:54.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f3539406640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35341535d0 0x7f35341ef2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:54.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f3539406640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35341535d0 0x7f35341ef2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33606/0 (socket says 172.21.15.88:33606) 2023-10-16T20:36:54.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f3539406640 1 -- 172.21.15.88:0/2734581001 learned_addr learned my addr 172.21.15.88:0/2734581001 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:54.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f353a408640 1 --2- 172.21.15.88:0/2734581001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3534159e20 0x7f353414ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:54.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f3539c07640 1 --2- 172.21.15.88:0/2734581001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35340c9be0 0x7f35340b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:54.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f3539406640 1 -- 172.21.15.88:0/2734581001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3534159e20 msgr2=0x7f353414ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:54.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f3539406640 1 --2- 172.21.15.88:0/2734581001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3534159e20 0x7f353414ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:54.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f3539406640 1 -- 172.21.15.88:0/2734581001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35340c9be0 msgr2=0x7f35340b3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:54.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f3539406640 1 --2- 172.21.15.88:0/2734581001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35340c9be0 0x7f35340b3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:54.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f3539406640 1 -- 172.21.15.88:0/2734581001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352c009a70 con 0x7f35341535d0 2023-10-16T20:36:54.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f3539c07640 1 --2- 172.21.15.88:0/2734581001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35340c9be0 0x7f35340b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:36:54.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f3539406640 1 --2- 172.21.15.88:0/2734581001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35341535d0 0x7f35341ef2a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f352400c780 tx=0x7f352400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:54.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f352a7fc640 1 -- 172.21.15.88:0/2734581001 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3524005ae0 con 0x7f35341535d0 2023-10-16T20:36:54.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f352a7fc640 1 -- 172.21.15.88:0/2734581001 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3524005c40 con 0x7f35341535d0 2023-10-16T20:36:54.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f353ac09640 1 -- 172.21.15.88:0/2734581001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35341eee10 con 0x7f35341535d0 2023-10-16T20:36:54.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f352a7fc640 1 -- 172.21.15.88:0/2734581001 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3524007070 con 0x7f35341535d0 2023-10-16T20:36:54.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f353ac09640 1 -- 172.21.15.88:0/2734581001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35341eeae0 con 0x7f35341535d0 2023-10-16T20:36:54.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.919+0000 7f353ac09640 1 -- 172.21.15.88:0/2734581001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35341eefa0 con 0x7f35341535d0 2023-10-16T20:36:54.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.923+0000 7f352a7fc640 1 -- 172.21.15.88:0/2734581001 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f35240071f0 con 0x7f35341535d0 2023-10-16T20:36:54.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.923+0000 7f352a7fc640 1 --2- 172.21.15.88:0/2734581001 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f35180664a0 0x7f3518068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:54.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.923+0000 7f352a7fc640 1 -- 172.21.15.88:0/2734581001 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f3524083a00 con 0x7f35341535d0 2023-10-16T20:36:54.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.923+0000 7f3539c07640 1 --2- 172.21.15.88:0/2734581001 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f35180664a0 0x7f3518068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:54.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.923+0000 7f3539c07640 1 --2- 172.21.15.88:0/2734581001 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f35180664a0 0x7f3518068960 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f35300011f0 tx=0x7f3530009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:54.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:54.927+0000 7f352a7fc640 1 -- 172.21.15.88:0/2734581001 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3524054990 con 0x7f35341535d0 2023-10-16T20:36:55.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.107+0000 7f353ac09640 1 -- 172.21.15.88:0/2734581001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f353404e9c0 con 0x7f35341535d0 2023-10-16T20:36:55.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.735+0000 7f352a7fc640 1 -- 172.21.15.88:0/2734581001 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f3524017020 con 0x7f35341535d0 2023-10-16T20:36:55.743 INFO:teuthology.orchestra.run.smithi088.stderr:new fs with metadata pool 2 and data pool 3 2023-10-16T20:36:55.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.739+0000 7f353ac09640 1 -- 172.21.15.88:0/2734581001 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f35180664a0 msgr2=0x7f3518068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:55.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.739+0000 7f353ac09640 1 --2- 172.21.15.88:0/2734581001 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f35180664a0 0x7f3518068960 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f35300011f0 tx=0x7f3530009040 comp rx=0 tx=0).stop 2023-10-16T20:36:55.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.739+0000 7f353ac09640 1 -- 172.21.15.88:0/2734581001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35341535d0 msgr2=0x7f35341ef2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:55.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.739+0000 7f353ac09640 1 --2- 172.21.15.88:0/2734581001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35341535d0 0x7f35341ef2a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f352400c780 tx=0x7f352400cc40 comp rx=0 tx=0).stop 2023-10-16T20:36:55.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.739+0000 7f353ac09640 1 -- 172.21.15.88:0/2734581001 shutdown_connections 2023-10-16T20:36:55.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.739+0000 7f353ac09640 1 --2- 172.21.15.88:0/2734581001 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f35180664a0 0x7f3518068960 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:55.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.739+0000 7f353ac09640 1 --2- 172.21.15.88:0/2734581001 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3534159e20 0x7f353414ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:55.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.739+0000 7f353ac09640 1 --2- 172.21.15.88:0/2734581001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35341535d0 0x7f35341ef2a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:55.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.739+0000 7f353ac09640 1 --2- 172.21.15.88:0/2734581001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f35340c9be0 0x7f35340b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:55.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.739+0000 7f353ac09640 1 -- 172.21.15.88:0/2734581001 >> 172.21.15.88:0/2734581001 conn(0x7f353406bbe0 msgr2=0x7f35340c4370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:55.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.739+0000 7f353ac09640 1 -- 172.21.15.88:0/2734581001 shutdown_connections 2023-10-16T20:36:55.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.739+0000 7f353ac09640 1 -- 172.21.15.88:0/2734581001 wait complete. 2023-10-16T20:36:55.765 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-10-16T20:36:55.765 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-10-16T20:36:55.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.903+0000 7f9e2fe6b640 1 Processor -- start 2023-10-16T20:36:55.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.903+0000 7f9e2fe6b640 1 -- start start 2023-10-16T20:36:55.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.903+0000 7f9e2fe6b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9e2806ae00 0x7f9e280c20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:55.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.903+0000 7f9e2fe6b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2815dba0 con 0x7f9e28157b20 2023-10-16T20:36:55.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.903+0000 7f9e2fe6b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2815de80 con 0x7f9e28157f00 2023-10-16T20:36:55.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.903+0000 7f9e2fe6b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e2815d9b0 con 0x7f9e2806ae00 2023-10-16T20:36:55.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.903+0000 7f9e2dc07640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9e28157b20 0x7f9e28069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46906/0 (socket says 172.21.15.88:46906) 2023-10-16T20:36:55.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.903+0000 7f9e2dc07640 1 -- 172.21.15.88:0/2160869246 learned_addr learned my addr 172.21.15.88:0/2160869246 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:36:55.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.903+0000 7f9e2d406640 1 --2- 172.21.15.88:0/2160869246 >> v2:172.21.15.88:3300/0 conn(0x7f9e2806ae00 0x7f9e280c20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:55.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.903+0000 7f9e2cc05640 1 -- 172.21.15.88:0/2160869246 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853434569 0 0) 0x7f9e2815dba0 con 0x7f9e28157b20 2023-10-16T20:36:55.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.903+0000 7f9e2cc05640 1 -- 172.21.15.88:0/2160869246 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e14003530 con 0x7f9e28157b20 2023-10-16T20:36:55.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.903+0000 7f9e2d406640 1 -- 172.21.15.88:0/2160869246 >> v1:172.21.15.191:6789/0 conn(0x7f9e28157f00 legacy=0x7f9e280c5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:55.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.903+0000 7f9e2d406640 1 -- 172.21.15.88:0/2160869246 >> v1:172.21.15.88:6789/0 conn(0x7f9e28157b20 legacy=0x7f9e28069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:55.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.903+0000 7f9e2d406640 1 -- 172.21.15.88:0/2160869246 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2815e160 con 0x7f9e2806ae00 2023-10-16T20:36:55.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.907+0000 7f9e2d406640 1 --2- 172.21.15.88:0/2160869246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e2806ae00 0x7f9e280c20e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9e180097c0 tx=0x7f9e1801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc3898075e4db8d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:55.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.907+0000 7f9e2cc05640 1 -- 172.21.15.88:0/2160869246 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e180a8970 con 0x7f9e2806ae00 2023-10-16T20:36:55.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.907+0000 7f9e2cc05640 1 -- 172.21.15.88:0/2160869246 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e180a8ad0 con 0x7f9e2806ae00 2023-10-16T20:36:55.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.907+0000 7f9e2cc05640 1 -- 172.21.15.88:0/2160869246 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e180a8e60 con 0x7f9e2806ae00 2023-10-16T20:36:55.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.907+0000 7f9e2fe6b640 1 -- 172.21.15.88:0/2160869246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e2806ae00 msgr2=0x7f9e280c20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:55.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.907+0000 7f9e2fe6b640 1 --2- 172.21.15.88:0/2160869246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e2806ae00 0x7f9e280c20e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9e180097c0 tx=0x7f9e1801ec70 comp rx=0 tx=0).stop 2023-10-16T20:36:55.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.907+0000 7f9e2fe6b640 1 -- 172.21.15.88:0/2160869246 shutdown_connections 2023-10-16T20:36:55.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.907+0000 7f9e2fe6b640 1 --2- 172.21.15.88:0/2160869246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e2806ae00 0x7f9e280c20e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:55.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.907+0000 7f9e2fe6b640 1 -- 172.21.15.88:0/2160869246 >> 172.21.15.88:0/2160869246 conn(0x7f9e280ca9f0 msgr2=0x7f9e280cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:55.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.907+0000 7f9e2fe6b640 1 -- 172.21.15.88:0/2160869246 shutdown_connections 2023-10-16T20:36:55.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.907+0000 7f9e2fe6b640 1 -- 172.21.15.88:0/2160869246 wait complete. 2023-10-16T20:36:55.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2fe6b640 1 Processor -- start 2023-10-16T20:36:55.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2fe6b640 1 -- start start 2023-10-16T20:36:55.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2fe6b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e2806ae00 0x7f9e280c9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:55.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2fe6b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e28157b20 0x7f9e2815c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:55.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2fe6b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e28157f00 0x7f9e2815a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:55.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2fe6b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e28159e20 con 0x7f9e28157f00 2023-10-16T20:36:55.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2fe6b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e280b8300 con 0x7f9e28157b20 2023-10-16T20:36:55.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2fe6b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2815b9d0 con 0x7f9e2806ae00 2023-10-16T20:36:55.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2e408640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e28157f00 0x7f9e2815a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:55.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2e408640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e28157f00 0x7f9e2815a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33618/0 (socket says 172.21.15.88:33618) 2023-10-16T20:36:55.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2e408640 1 -- 172.21.15.88:0/1148479887 learned_addr learned my addr 172.21.15.88:0/1148479887 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:55.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2d406640 1 --2- 172.21.15.88:0/1148479887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e28157b20 0x7f9e2815c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:55.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2dc07640 1 --2- 172.21.15.88:0/1148479887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e2806ae00 0x7f9e280c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:55.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2e408640 1 -- 172.21.15.88:0/1148479887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e2806ae00 msgr2=0x7f9e280c9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:55.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2e408640 1 --2- 172.21.15.88:0/1148479887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e2806ae00 0x7f9e280c9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:55.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2e408640 1 -- 172.21.15.88:0/1148479887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e28157b20 msgr2=0x7f9e2815c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:55.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2e408640 1 --2- 172.21.15.88:0/1148479887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e28157b20 0x7f9e2815c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:55.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2e408640 1 -- 172.21.15.88:0/1148479887 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e18009420 con 0x7f9e28157f00 2023-10-16T20:36:55.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2e408640 1 --2- 172.21.15.88:0/1148479887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e28157f00 0x7f9e2815a2b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000c730 tx=0x7f9e2000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:55.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e1e7fc640 1 -- 172.21.15.88:0/1148479887 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e20003b80 con 0x7f9e28157f00 2023-10-16T20:36:55.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e1e7fc640 1 -- 172.21.15.88:0/1148479887 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e20003ce0 con 0x7f9e28157f00 2023-10-16T20:36:55.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e1e7fc640 1 -- 172.21.15.88:0/1148479887 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2000aa30 con 0x7f9e28157f00 2023-10-16T20:36:55.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2fe6b640 1 -- 172.21.15.88:0/1148479887 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e281f67d0 con 0x7f9e28157f00 2023-10-16T20:36:55.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2fe6b640 1 -- 172.21.15.88:0/1148479887 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e281597d0 con 0x7f9e28157f00 2023-10-16T20:36:55.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.911+0000 7f9e2fe6b640 1 -- 172.21.15.88:0/1148479887 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e280729f0 con 0x7f9e28157f00 2023-10-16T20:36:55.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.915+0000 7f9e1e7fc640 1 -- 172.21.15.88:0/1148479887 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9e2000ab90 con 0x7f9e28157f00 2023-10-16T20:36:55.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.915+0000 7f9e1e7fc640 1 --2- 172.21.15.88:0/1148479887 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9e0c0664a0 0x7f9e0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:55.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.915+0000 7f9e1e7fc640 1 -- 172.21.15.88:0/1148479887 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e20083c20 con 0x7f9e28157f00 2023-10-16T20:36:55.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.915+0000 7f9e2dc07640 1 --2- 172.21.15.88:0/1148479887 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9e0c0664a0 0x7f9e0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:55.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.915+0000 7f9e2dc07640 1 --2- 172.21.15.88:0/1148479887 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9e0c0664a0 0x7f9e0c068960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9e240011f0 tx=0x7f9e24007d40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:55.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:55.919+0000 7f9e1e7fc640 1 -- 172.21.15.88:0/1148479887 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9e280729f0 con 0x7f9e28157f00 2023-10-16T20:36:56.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.079+0000 7f9e2fe6b640 1 -- 172.21.15.88:0/1148479887 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f9e280729f0 con 0x7f9e28157f00 2023-10-16T20:36:56.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.259+0000 7f9e1e7fc640 1 -- 172.21.15.88:0/1148479887 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v23) v1 ==== 157+0+0 (secure 0 0 0) 0x7f9e280729f0 con 0x7f9e28157f00 2023-10-16T20:36:56.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.263+0000 7f9e2fe6b640 1 -- 172.21.15.88:0/1148479887 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9e0c0664a0 msgr2=0x7f9e0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:56.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.263+0000 7f9e2fe6b640 1 --2- 172.21.15.88:0/1148479887 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9e0c0664a0 0x7f9e0c068960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9e240011f0 tx=0x7f9e24007d40 comp rx=0 tx=0).stop 2023-10-16T20:36:56.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.263+0000 7f9e2fe6b640 1 -- 172.21.15.88:0/1148479887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e28157f00 msgr2=0x7f9e2815a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:56.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.263+0000 7f9e2fe6b640 1 --2- 172.21.15.88:0/1148479887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e28157f00 0x7f9e2815a2b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9e2000c730 tx=0x7f9e2000cbf0 comp rx=0 tx=0).stop 2023-10-16T20:36:56.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.263+0000 7f9e2fe6b640 1 -- 172.21.15.88:0/1148479887 shutdown_connections 2023-10-16T20:36:56.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.263+0000 7f9e2fe6b640 1 --2- 172.21.15.88:0/1148479887 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9e0c0664a0 0x7f9e0c068960 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9e240011f0 tx=0x7f9e24007d40 comp rx=0 tx=0).stop 2023-10-16T20:36:56.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.263+0000 7f9e2fe6b640 1 --2- 172.21.15.88:0/1148479887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e28157f00 0x7f9e2815a2b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:56.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.263+0000 7f9e2fe6b640 1 --2- 172.21.15.88:0/1148479887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e28157b20 0x7f9e2815c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:56.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.263+0000 7f9e2fe6b640 1 --2- 172.21.15.88:0/1148479887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e2806ae00 0x7f9e280c9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:56.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.263+0000 7f9e2fe6b640 1 -- 172.21.15.88:0/1148479887 >> 172.21.15.88:0/1148479887 conn(0x7f9e280ca9f0 msgr2=0x7f9e280c3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:56.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.263+0000 7f9e2fe6b640 1 -- 172.21.15.88:0/1148479887 shutdown_connections 2023-10-16T20:36:56.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.263+0000 7f9e2fe6b640 1 -- 172.21.15.88:0/1148479887 wait complete. 2023-10-16T20:36:56.289 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-10-16T20:36:56.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.427+0000 7f57de90f640 1 Processor -- start 2023-10-16T20:36:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.427+0000 7f57de90f640 1 -- start start 2023-10-16T20:36:56.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.431+0000 7f57de90f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57d8150140 0x7f57d8156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:56.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.431+0000 7f57de90f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f57d8150520 0x7f57d80c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.431+0000 7f57de90f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d806d150 con 0x7f57d80c1e60 2023-10-16T20:36:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.431+0000 7f57de90f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d806d380 con 0x7f57d8150520 2023-10-16T20:36:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.431+0000 7f57de90f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d806d620 con 0x7f57d8150140 2023-10-16T20:36:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.431+0000 7f57d77fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f57d8150520 0x7f57d80c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.431+0000 7f57dceac640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f57d80c1e60 0x7f57d8070e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46916/0 (socket says 172.21.15.88:46916) 2023-10-16T20:36:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.431+0000 7f57d77fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f57d8150520 0x7f57d80c64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33622/0 (socket says 172.21.15.88:33622) 2023-10-16T20:36:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.431+0000 7f57dceac640 1 -- 172.21.15.88:0/801201837 learned_addr learned my addr 172.21.15.88:0/801201837 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:36:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.431+0000 7f57d7fff640 1 --2- 172.21.15.88:0/801201837 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57d8150140 0x7f57d8156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.431+0000 7f57d77fe640 1 -- 172.21.15.88:0/801201837 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57d8150140 msgr2=0x7f57d8156ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.431+0000 7f57d77fe640 1 --2- 172.21.15.88:0/801201837 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57d8150140 0x7f57d8156ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.431+0000 7f57d77fe640 1 -- 172.21.15.88:0/801201837 >> v1:172.21.15.88:6789/0 conn(0x7f57d80c1e60 legacy=0x7f57d8070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.431+0000 7f57d77fe640 1 -- 172.21.15.88:0/801201837 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d8157220 con 0x7f57d8150520 2023-10-16T20:36:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.431+0000 7f57d7fff640 1 --2- 172.21.15.88:0/801201837 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57d8150140 0x7f57d8156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:36:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.431+0000 7f57d77fe640 1 --2- 172.21.15.88:0/801201837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57d8150520 0x7f57d80c64f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0097c0 tx=0x7f57cc01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=751261b7bf532785 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57d6ffd640 1 -- 172.21.15.88:0/801201837 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc0a8970 con 0x7f57d8150520 2023-10-16T20:36:56.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57d6ffd640 1 -- 172.21.15.88:0/801201837 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57cc0a8ad0 con 0x7f57d8150520 2023-10-16T20:36:56.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57d6ffd640 1 -- 172.21.15.88:0/801201837 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc0a8e60 con 0x7f57d8150520 2023-10-16T20:36:56.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57de90f640 1 -- 172.21.15.88:0/801201837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57d8150520 msgr2=0x7f57d80c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:56.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57de90f640 1 --2- 172.21.15.88:0/801201837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57d8150520 0x7f57d80c64f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0097c0 tx=0x7f57cc01ec70 comp rx=0 tx=0).stop 2023-10-16T20:36:56.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57de90f640 1 -- 172.21.15.88:0/801201837 shutdown_connections 2023-10-16T20:36:56.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57de90f640 1 --2- 172.21.15.88:0/801201837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57d8150520 0x7f57d80c64f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:56.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57de90f640 1 --2- 172.21.15.88:0/801201837 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57d8150140 0x7f57d8156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:56.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57de90f640 1 -- 172.21.15.88:0/801201837 >> 172.21.15.88:0/801201837 conn(0x7f57d80caa60 msgr2=0x7f57d80cae40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:56.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57de90f640 1 -- 172.21.15.88:0/801201837 shutdown_connections 2023-10-16T20:36:56.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57de90f640 1 -- 172.21.15.88:0/801201837 wait complete. 2023-10-16T20:36:56.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57de90f640 1 Processor -- start 2023-10-16T20:36:56.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57de90f640 1 -- start start 2023-10-16T20:36:56.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57de90f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57d80c1e60 0x7f57d80bd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:56.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57de90f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57d8150140 0x7f57d8155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:56.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57de90f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57d8150520 0x7f57d8149370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:56.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57de90f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d80c47f0 con 0x7f57d80c1e60 2023-10-16T20:36:56.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57de90f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d80bd8d0 con 0x7f57d8150520 2023-10-16T20:36:56.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57de90f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d80c4530 con 0x7f57d8150140 2023-10-16T20:36:56.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57d7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57d80c1e60 0x7f57d80bd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:56.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57d7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57d80c1e60 0x7f57d80bd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33624/0 (socket says 172.21.15.88:33624) 2023-10-16T20:36:56.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57d7fff640 1 -- 172.21.15.88:0/679757067 learned_addr learned my addr 172.21.15.88:0/679757067 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:56.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.435+0000 7f57d77fe640 1 --2- 172.21.15.88:0/679757067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57d8150140 0x7f57d8155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:56.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.439+0000 7f57dceac640 1 --2- 172.21.15.88:0/679757067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57d8150520 0x7f57d8149370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.439+0000 7f57d7fff640 1 -- 172.21.15.88:0/679757067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57d8150140 msgr2=0x7f57d8155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.439+0000 7f57d7fff640 1 --2- 172.21.15.88:0/679757067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57d8150140 0x7f57d8155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.439+0000 7f57d7fff640 1 -- 172.21.15.88:0/679757067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57d8150520 msgr2=0x7f57d8149370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.439+0000 7f57d7fff640 1 --2- 172.21.15.88:0/679757067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57d8150520 0x7f57d8149370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.439+0000 7f57d7fff640 1 -- 172.21.15.88:0/679757067 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc009420 con 0x7f57d80c1e60 2023-10-16T20:36:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.439+0000 7f57d7fff640 1 --2- 172.21.15.88:0/679757067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57d80c1e60 0x7f57d80bd390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f57c800e810 tx=0x7f57c800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:56.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.439+0000 7f57d4ff9640 1 -- 172.21.15.88:0/679757067 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c800c8d0 con 0x7f57d80c1e60 2023-10-16T20:36:56.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.439+0000 7f57d4ff9640 1 -- 172.21.15.88:0/679757067 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c800ca30 con 0x7f57d80c1e60 2023-10-16T20:36:56.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.439+0000 7f57d4ff9640 1 -- 172.21.15.88:0/679757067 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c800cdc0 con 0x7f57d80c1e60 2023-10-16T20:36:56.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.439+0000 7f57de90f640 1 -- 172.21.15.88:0/679757067 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d80c2a60 con 0x7f57d80c1e60 2023-10-16T20:36:56.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.439+0000 7f57de90f640 1 -- 172.21.15.88:0/679757067 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d80bc6a0 con 0x7f57d80c1e60 2023-10-16T20:36:56.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.439+0000 7f57de90f640 1 -- 172.21.15.88:0/679757067 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d8074700 con 0x7f57d80c1e60 2023-10-16T20:36:56.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.439+0000 7f57d4ff9640 1 -- 172.21.15.88:0/679757067 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f57c8013410 con 0x7f57d80c1e60 2023-10-16T20:36:56.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.439+0000 7f57d4ff9640 1 --2- 172.21.15.88:0/679757067 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57c40664a0 0x7f57c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:56.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.439+0000 7f57d4ff9640 1 -- 172.21.15.88:0/679757067 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5935+0+0 (secure 0 0 0) 0x7f57c8083d40 con 0x7f57d80c1e60 2023-10-16T20:36:56.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.439+0000 7f57d77fe640 1 --2- 172.21.15.88:0/679757067 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57c40664a0 0x7f57c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:56.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.443+0000 7f57d77fe640 1 --2- 172.21.15.88:0/679757067 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57c40664a0 0x7f57c4068960 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0b2000 tx=0x7f57cc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:56.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.443+0000 7f57d4ff9640 1 -- 172.21.15.88:0/679757067 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f57c8051100 con 0x7f57d80c1e60 2023-10-16T20:36:56.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:56.595+0000 7f57de90f640 1 -- 172.21.15.88:0/679757067 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f57d80cb860 con 0x7f57d80c1e60 2023-10-16T20:36:58.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.271+0000 7f57d4ff9640 1 -- 172.21.15.88:0/679757067 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v25) v1 ==== 228+0+0 (secure 0 0 0) 0x7f57c8021100 con 0x7f57d80c1e60 2023-10-16T20:36:58.276 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data_ec' created 2023-10-16T20:36:58.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.271+0000 7f57de90f640 1 -- 172.21.15.88:0/679757067 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57c40664a0 msgr2=0x7f57c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:58.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.271+0000 7f57de90f640 1 --2- 172.21.15.88:0/679757067 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57c40664a0 0x7f57c4068960 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0b2000 tx=0x7f57cc008040 comp rx=0 tx=0).stop 2023-10-16T20:36:58.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.271+0000 7f57de90f640 1 -- 172.21.15.88:0/679757067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57d80c1e60 msgr2=0x7f57d80bd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:58.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.271+0000 7f57de90f640 1 --2- 172.21.15.88:0/679757067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57d80c1e60 0x7f57d80bd390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f57c800e810 tx=0x7f57c800ecd0 comp rx=0 tx=0).stop 2023-10-16T20:36:58.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.275+0000 7f57de90f640 1 -- 172.21.15.88:0/679757067 shutdown_connections 2023-10-16T20:36:58.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.275+0000 7f57de90f640 1 --2- 172.21.15.88:0/679757067 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57c40664a0 0x7f57c4068960 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:58.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.275+0000 7f57de90f640 1 --2- 172.21.15.88:0/679757067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57d8150520 0x7f57d8149370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:58.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.275+0000 7f57de90f640 1 --2- 172.21.15.88:0/679757067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57d8150140 0x7f57d8155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:58.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.275+0000 7f57de90f640 1 --2- 172.21.15.88:0/679757067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57d80c1e60 0x7f57d80bd390 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:58.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.275+0000 7f57de90f640 1 -- 172.21.15.88:0/679757067 >> 172.21.15.88:0/679757067 conn(0x7f57d80caa60 msgr2=0x7f57d81de750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:58.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.275+0000 7f57de90f640 1 -- 172.21.15.88:0/679757067 shutdown_connections 2023-10-16T20:36:58.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.275+0000 7f57de90f640 1 -- 172.21.15.88:0/679757067 wait complete. 2023-10-16T20:36:58.312 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-10-16T20:36:58.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.451+0000 7f3ef433e640 1 Processor -- start 2023-10-16T20:36:58.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.451+0000 7f3ef433e640 1 -- start start 2023-10-16T20:36:58.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.451+0000 7f3ef433e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3eec0c66e0 0x7f3eec15a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:58.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.451+0000 7f3ef433e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eec070f40 con 0x7f3eec0c6ac0 2023-10-16T20:36:58.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef433e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eec15a790 con 0x7f3eec0cb5c0 2023-10-16T20:36:58.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef433e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eec06a2d0 con 0x7f3eec0c66e0 2023-10-16T20:36:58.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef18d9640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3eec0c6ac0 0x7f3eec150150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46928/0 (socket says 172.21.15.88:46928) 2023-10-16T20:36:58.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef18d9640 1 -- 172.21.15.88:0/3464301573 learned_addr learned my addr 172.21.15.88:0/3464301573 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:36:58.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef20da640 1 --2- 172.21.15.88:0/3464301573 >> v2:172.21.15.191:3300/0 conn(0x7f3eec0c66e0 0x7f3eec15a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:58.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef10d8640 1 -- 172.21.15.88:0/3464301573 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126022646 0 0) 0x7f3eec070f40 con 0x7f3eec0c6ac0 2023-10-16T20:36:58.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef10d8640 1 -- 172.21.15.88:0/3464301573 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed8003530 con 0x7f3eec0c6ac0 2023-10-16T20:36:58.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef10d8640 1 -- 172.21.15.88:0/3464301573 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3550951463 0 0) 0x7f3ed8003530 con 0x7f3eec0c6ac0 2023-10-16T20:36:58.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef10d8640 1 -- 172.21.15.88:0/3464301573 >> v2:172.21.15.191:3300/0 conn(0x7f3eec0c66e0 msgr2=0x7f3eec15a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:58.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef10d8640 1 --2- 172.21.15.88:0/3464301573 >> v2:172.21.15.191:3300/0 conn(0x7f3eec0c66e0 0x7f3eec15a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:58.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef10d8640 1 -- 172.21.15.88:0/3464301573 >> v1:172.21.15.191:6789/0 conn(0x7f3eec0cb5c0 legacy=0x7f3eec071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:58.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef10d8640 1 -- 172.21.15.88:0/3464301573 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eec06a5b0 con 0x7f3eec0c6ac0 2023-10-16T20:36:58.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef10d8640 1 -- 172.21.15.88:0/3464301573 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f3edc001680 con 0x7f3eec0c6ac0 2023-10-16T20:36:58.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef10d8640 1 -- 172.21.15.88:0/3464301573 >> v1:172.21.15.88:6789/0 conn(0x7f3eec0c6ac0 legacy=0x7f3eec150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:58.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef10d8640 1 --2- 172.21.15.88:0/3464301573 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ed8003980 0x7f3ed8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:58.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef10d8640 1 --2- 172.21.15.88:0/3464301573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ed80047d0 0x7f3ed8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:58.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef10d8640 1 --2- 172.21.15.88:0/3464301573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ed8005190 0x7f3ed800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:58.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef10d8640 1 -- 172.21.15.88:0/3464301573 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec06a2d0 con 0x7f3ed8005190 2023-10-16T20:36:58.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef10d8640 1 -- 172.21.15.88:0/3464301573 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed800c130 con 0x7f3ed8003980 2023-10-16T20:36:58.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef10d8640 1 -- 172.21.15.88:0/3464301573 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed800e500 con 0x7f3ed80047d0 2023-10-16T20:36:58.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef28db640 1 --2- 172.21.15.88:0/3464301573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ed8005190 0x7f3ed800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:58.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef20da640 1 --2- 172.21.15.88:0/3464301573 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ed8003980 0x7f3ed8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:58.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef18d9640 1 --2- 172.21.15.88:0/3464301573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ed80047d0 0x7f3ed8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:58.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef20da640 1 -- 172.21.15.88:0/3464301573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ed80047d0 msgr2=0x7f3ed8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:58.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef20da640 1 --2- 172.21.15.88:0/3464301573 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ed80047d0 0x7f3ed8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:58.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef20da640 1 -- 172.21.15.88:0/3464301573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ed8005190 msgr2=0x7f3ed800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:58.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef20da640 1 --2- 172.21.15.88:0/3464301573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ed8005190 0x7f3ed800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:58.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef20da640 1 -- 172.21.15.88:0/3464301573 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed8003530 con 0x7f3ed8003980 2023-10-16T20:36:58.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.455+0000 7f3ef28db640 1 -- 172.21.15.88:0/3464301573 reap_dead start 2023-10-16T20:36:58.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.459+0000 7f3ef20da640 1 --2- 172.21.15.88:0/3464301573 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ed8003980 0x7f3ed8003da0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8009ae0 tx=0x7f3ee801ca50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1a32e2b19cec0f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:58.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.459+0000 7f3ef10d8640 1 -- 172.21.15.88:0/3464301573 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee80a6440 con 0x7f3ed8003980 2023-10-16T20:36:58.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.459+0000 7f3ef10d8640 1 -- 172.21.15.88:0/3464301573 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee80a65a0 con 0x7f3ed8003980 2023-10-16T20:36:58.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.459+0000 7f3ef10d8640 1 -- 172.21.15.88:0/3464301573 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee80a6950 con 0x7f3ed8003980 2023-10-16T20:36:58.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.459+0000 7f3ef433e640 1 -- 172.21.15.88:0/3464301573 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ed8003980 msgr2=0x7f3ed8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:58.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.459+0000 7f3ef433e640 1 --2- 172.21.15.88:0/3464301573 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ed8003980 0x7f3ed8003da0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8009ae0 tx=0x7f3ee801ca50 comp rx=0 tx=0).stop 2023-10-16T20:36:58.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.459+0000 7f3ef433e640 1 -- 172.21.15.88:0/3464301573 shutdown_connections 2023-10-16T20:36:58.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.459+0000 7f3ef433e640 1 --2- 172.21.15.88:0/3464301573 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ed8003980 0x7f3ed8003da0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:58.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.459+0000 7f3ef433e640 1 -- 172.21.15.88:0/3464301573 >> 172.21.15.88:0/3464301573 conn(0x7f3eec0ca9f0 msgr2=0x7f3eec0cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:58.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.459+0000 7f3ef433e640 1 -- 172.21.15.88:0/3464301573 shutdown_connections 2023-10-16T20:36:58.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.459+0000 7f3ef433e640 1 -- 172.21.15.88:0/3464301573 wait complete. 2023-10-16T20:36:58.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.459+0000 7f3ef433e640 1 Processor -- start 2023-10-16T20:36:58.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.459+0000 7f3ef433e640 1 -- start start 2023-10-16T20:36:58.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef433e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ed8003980 0x7f3eec149380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:58.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef433e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3eec1579e0 0x7f3eec1472e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:58.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef433e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3eec147820 0x7f3eec14a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:58.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef433e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec1480d0 con 0x7f3eec147820 2023-10-16T20:36:58.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef433e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec147ca0 con 0x7f3eec1579e0 2023-10-16T20:36:58.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef433e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec1482e0 con 0x7f3ed8003980 2023-10-16T20:36:58.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef28db640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3eec147820 0x7f3eec14a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:58.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef28db640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3eec147820 0x7f3eec14a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33636/0 (socket says 172.21.15.88:33636) 2023-10-16T20:36:58.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef28db640 1 -- 172.21.15.88:0/2057693424 learned_addr learned my addr 172.21.15.88:0/2057693424 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:58.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef20da640 1 --2- 172.21.15.88:0/2057693424 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ed8003980 0x7f3eec149380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:58.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef18d9640 1 --2- 172.21.15.88:0/2057693424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3eec1579e0 0x7f3eec1472e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:58.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef28db640 1 -- 172.21.15.88:0/2057693424 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ed8003980 msgr2=0x7f3eec149380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:58.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef28db640 1 --2- 172.21.15.88:0/2057693424 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ed8003980 0x7f3eec149380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:58.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef28db640 1 -- 172.21.15.88:0/2057693424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3eec1579e0 msgr2=0x7f3eec1472e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:58.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef28db640 1 --2- 172.21.15.88:0/2057693424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3eec1579e0 0x7f3eec1472e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:58.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef28db640 1 -- 172.21.15.88:0/2057693424 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eec1485c0 con 0x7f3eec147820 2023-10-16T20:36:58.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef18d9640 1 --2- 172.21.15.88:0/2057693424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3eec1579e0 0x7f3eec1472e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:36:58.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef20da640 1 --2- 172.21.15.88:0/2057693424 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ed8003980 0x7f3eec149380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:36:58.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef28db640 1 --2- 172.21.15.88:0/2057693424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3eec147820 0x7f3eec14a660 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3eec0c1cc0 tx=0x7f3ee400fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:58.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ee2ffd640 1 -- 172.21.15.88:0/2057693424 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4006d70 con 0x7f3eec147820 2023-10-16T20:36:58.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ee2ffd640 1 -- 172.21.15.88:0/2057693424 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee4003980 con 0x7f3eec147820 2023-10-16T20:36:58.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ee2ffd640 1 -- 172.21.15.88:0/2057693424 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4003d30 con 0x7f3eec147820 2023-10-16T20:36:58.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef433e640 1 -- 172.21.15.88:0/2057693424 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eec1488a0 con 0x7f3eec147820 2023-10-16T20:36:58.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef433e640 1 -- 172.21.15.88:0/2057693424 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eec148e60 con 0x7f3eec147820 2023-10-16T20:36:58.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ef433e640 1 -- 172.21.15.88:0/2057693424 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eec0c55e0 con 0x7f3eec147820 2023-10-16T20:36:58.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.463+0000 7f3ee2ffd640 1 -- 172.21.15.88:0/2057693424 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3ee4005070 con 0x7f3eec147820 2023-10-16T20:36:58.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.467+0000 7f3ee2ffd640 1 --2- 172.21.15.88:0/2057693424 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3ed00664a0 0x7f3ed0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:58.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.467+0000 7f3ee2ffd640 1 -- 172.21.15.88:0/2057693424 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6428+0+0 (secure 0 0 0) 0x7f3ee4083680 con 0x7f3eec147820 2023-10-16T20:36:58.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.467+0000 7f3ef20da640 1 --2- 172.21.15.88:0/2057693424 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3ed00664a0 0x7f3ed0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:58.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.467+0000 7f3ef20da640 1 --2- 172.21.15.88:0/2057693424 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3ed00664a0 0x7f3ed0068960 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3ee80ad000 tx=0x7f3ee80b2040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:58.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.471+0000 7f3ee2ffd640 1 -- 172.21.15.88:0/2057693424 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3ee4050700 con 0x7f3eec147820 2023-10-16T20:36:58.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:58.619+0000 7f3ef433e640 1 -- 172.21.15.88:0/2057693424 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f3eec06ab40 con 0x7f3eec147820 2023-10-16T20:36:59.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.271+0000 7f3ee2ffd640 1 -- 172.21.15.88:0/2057693424 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v26) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3ee4053d20 con 0x7f3eec147820 2023-10-16T20:36:59.278 INFO:teuthology.orchestra.run.smithi088.stderr:set pool 4 allow_ec_overwrites to true 2023-10-16T20:36:59.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.275+0000 7f3ef433e640 1 -- 172.21.15.88:0/2057693424 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3ed00664a0 msgr2=0x7f3ed0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:59.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.275+0000 7f3ef433e640 1 --2- 172.21.15.88:0/2057693424 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3ed00664a0 0x7f3ed0068960 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3ee80ad000 tx=0x7f3ee80b2040 comp rx=0 tx=0).stop 2023-10-16T20:36:59.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.275+0000 7f3ef433e640 1 -- 172.21.15.88:0/2057693424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3eec147820 msgr2=0x7f3eec14a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:59.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.275+0000 7f3ef433e640 1 --2- 172.21.15.88:0/2057693424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3eec147820 0x7f3eec14a660 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3eec0c1cc0 tx=0x7f3ee400fc70 comp rx=0 tx=0).stop 2023-10-16T20:36:59.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.275+0000 7f3ef433e640 1 -- 172.21.15.88:0/2057693424 shutdown_connections 2023-10-16T20:36:59.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.275+0000 7f3ef433e640 1 --2- 172.21.15.88:0/2057693424 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3ed00664a0 0x7f3ed0068960 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:59.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.275+0000 7f3ef433e640 1 --2- 172.21.15.88:0/2057693424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3eec147820 0x7f3eec14a660 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:59.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.275+0000 7f3ef433e640 1 --2- 172.21.15.88:0/2057693424 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3eec1579e0 0x7f3eec1472e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:59.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.275+0000 7f3ef433e640 1 --2- 172.21.15.88:0/2057693424 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ed8003980 0x7f3eec149380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:59.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.275+0000 7f3ef433e640 1 -- 172.21.15.88:0/2057693424 >> 172.21.15.88:0/2057693424 conn(0x7f3eec0ca9f0 msgr2=0x7f3eec073250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:59.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.275+0000 7f3ef433e640 1 -- 172.21.15.88:0/2057693424 shutdown_connections 2023-10-16T20:36:59.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.275+0000 7f3ef433e640 1 -- 172.21.15.88:0/2057693424 wait complete. 2023-10-16T20:36:59.327 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-10-16T20:36:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.463+0000 7f15eebc4640 1 Processor -- start 2023-10-16T20:36:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.463+0000 7f15eebc4640 1 -- start start 2023-10-16T20:36:59.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15eebc4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f15e81598c0 0x7f15e80ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:59.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15eebc4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e8159d90 con 0x7f15e8153780 2023-10-16T20:36:59.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15eebc4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e815a070 con 0x7f15e8159490 2023-10-16T20:36:59.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15eebc4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e815a350 con 0x7f15e81598c0 2023-10-16T20:36:59.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e7fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f15e8159490 0x7f15e80c53f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:39282/0 (socket says 172.21.15.88:39282) 2023-10-16T20:36:59.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e7fff640 1 -- 172.21.15.88:0/1631354550 learned_addr learned my addr 172.21.15.88:0/1631354550 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:36:59.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15ed161640 1 --2- 172.21.15.88:0/1631354550 >> v2:172.21.15.191:3300/0 conn(0x7f15e81598c0 0x7f15e80ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:59.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e77fe640 1 -- 172.21.15.88:0/1631354550 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942273872 0 0) 0x7f15e8159d90 con 0x7f15e8153780 2023-10-16T20:36:59.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e77fe640 1 -- 172.21.15.88:0/1631354550 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15d0003530 con 0x7f15e8153780 2023-10-16T20:36:59.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e77fe640 1 -- 172.21.15.88:0/1631354550 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959469339 0 0) 0x7f15e815a070 con 0x7f15e8159490 2023-10-16T20:36:59.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e77fe640 1 -- 172.21.15.88:0/1631354550 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15e8159d90 con 0x7f15e8159490 2023-10-16T20:36:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e77fe640 1 -- 172.21.15.88:0/1631354550 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 47416764 0 0) 0x7f15d0003530 con 0x7f15e8153780 2023-10-16T20:36:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e77fe640 1 -- 172.21.15.88:0/1631354550 >> v2:172.21.15.191:3300/0 conn(0x7f15e81598c0 msgr2=0x7f15e80ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e77fe640 1 --2- 172.21.15.88:0/1631354550 >> v2:172.21.15.191:3300/0 conn(0x7f15e81598c0 0x7f15e80ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e77fe640 1 -- 172.21.15.88:0/1631354550 >> v1:172.21.15.191:6789/0 conn(0x7f15e8159490 legacy=0x7f15e80c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e77fe640 1 -- 172.21.15.88:0/1631354550 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e80c4910 con 0x7f15e8153780 2023-10-16T20:36:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e77fe640 1 -- 172.21.15.88:0/1631354550 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f15d80016f0 con 0x7f15e8153780 2023-10-16T20:36:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e77fe640 1 -- 172.21.15.88:0/1631354550 >> v1:172.21.15.88:6789/0 conn(0x7f15e8153780 legacy=0x7f15e80c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e77fe640 1 --2- 172.21.15.88:0/1631354550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15d0003b10 0x7f15d0003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e77fe640 1 --2- 172.21.15.88:0/1631354550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15d00047d0 0x7f15d0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:59.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e77fe640 1 --2- 172.21.15.88:0/1631354550 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15d0005190 0x7f15d000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:59.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e77fe640 1 -- 172.21.15.88:0/1631354550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e815a350 con 0x7f15d0003b10 2023-10-16T20:36:59.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e77fe640 1 -- 172.21.15.88:0/1631354550 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d000bf00 con 0x7f15d00047d0 2023-10-16T20:36:59.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e77fe640 1 -- 172.21.15.88:0/1631354550 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d000c0c0 con 0x7f15d0005190 2023-10-16T20:36:59.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15ec960640 1 --2- 172.21.15.88:0/1631354550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15d0003b10 0x7f15d0003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:59.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15e7fff640 1 --2- 172.21.15.88:0/1631354550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15d00047d0 0x7f15d0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:59.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15ed161640 1 --2- 172.21.15.88:0/1631354550 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15d0005190 0x7f15d000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:59.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15ec960640 1 -- 172.21.15.88:0/1631354550 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15d0005190 msgr2=0x7f15d000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:59.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15ec960640 1 --2- 172.21.15.88:0/1631354550 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15d0005190 0x7f15d000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:59.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15ec960640 1 -- 172.21.15.88:0/1631354550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15d00047d0 msgr2=0x7f15d0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:36:59.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15ec960640 1 --2- 172.21.15.88:0/1631354550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15d00047d0 0x7f15d0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:59.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15ec960640 1 -- 172.21.15.88:0/1631354550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e80c4910 con 0x7f15d0003b10 2023-10-16T20:36:59.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.467+0000 7f15ed161640 1 -- 172.21.15.88:0/1631354550 reap_dead start 2023-10-16T20:36:59.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.471+0000 7f15ec960640 1 --2- 172.21.15.88:0/1631354550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15d0003b10 0x7f15d0003ef0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f15d800c160 tx=0x7f15d801ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22669376ef1cea56 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:59.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.471+0000 7f15e77fe640 1 -- 172.21.15.88:0/1631354550 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d8005270 con 0x7f15d0003b10 2023-10-16T20:36:59.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.471+0000 7f15e77fe640 1 -- 172.21.15.88:0/1631354550 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d80053d0 con 0x7f15d0003b10 2023-10-16T20:36:59.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.471+0000 7f15e77fe640 1 -- 172.21.15.88:0/1631354550 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d8003860 con 0x7f15d0003b10 2023-10-16T20:36:59.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.471+0000 7f15eebc4640 1 -- 172.21.15.88:0/1631354550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15d0003b10 msgr2=0x7f15d0003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:59.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.471+0000 7f15eebc4640 1 --2- 172.21.15.88:0/1631354550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15d0003b10 0x7f15d0003ef0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f15d800c160 tx=0x7f15d801ca60 comp rx=0 tx=0).stop 2023-10-16T20:36:59.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.471+0000 7f15eebc4640 1 -- 172.21.15.88:0/1631354550 shutdown_connections 2023-10-16T20:36:59.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.471+0000 7f15eebc4640 1 --2- 172.21.15.88:0/1631354550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15d0003b10 0x7f15d0003ef0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:59.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.471+0000 7f15eebc4640 1 -- 172.21.15.88:0/1631354550 >> 172.21.15.88:0/1631354550 conn(0x7f15e806bcb0 msgr2=0x7f15e806c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:36:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.471+0000 7f15eebc4640 1 -- 172.21.15.88:0/1631354550 shutdown_connections 2023-10-16T20:36:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.471+0000 7f15eebc4640 1 -- 172.21.15.88:0/1631354550 wait complete. 2023-10-16T20:36:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15eebc4640 1 Processor -- start 2023-10-16T20:36:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15eebc4640 1 -- start start 2023-10-16T20:36:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15eebc4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15d0003b10 0x7f15e8144530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:59.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15eebc4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15e8152520 0x7f15e8157800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:59.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15eebc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15e8151980 0x7f15e8157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:59.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15eebc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80b4010 con 0x7f15e8151980 2023-10-16T20:36:59.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15eebc4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e8152230 con 0x7f15d0003b10 2023-10-16T20:36:59.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15eebc4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80b4420 con 0x7f15e8152520 2023-10-16T20:36:59.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15ed161640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15e8151980 0x7f15e8157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:59.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15ed161640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15e8151980 0x7f15e8157d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34380/0 (socket says 172.21.15.88:34380) 2023-10-16T20:36:59.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15ed161640 1 -- 172.21.15.88:0/3412473566 learned_addr learned my addr 172.21.15.88:0/3412473566 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:36:59.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15e7fff640 1 --2- 172.21.15.88:0/3412473566 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15e8152520 0x7f15e8157800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:59.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15ec960640 1 --2- 172.21.15.88:0/3412473566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15d0003b10 0x7f15e8144530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:59.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15ed161640 1 -- 172.21.15.88:0/3412473566 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15e8152520 msgr2=0x7f15e8157800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:59.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15ed161640 1 --2- 172.21.15.88:0/3412473566 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15e8152520 0x7f15e8157800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:59.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15ed161640 1 -- 172.21.15.88:0/3412473566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15d0003b10 msgr2=0x7f15e8144530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:36:59.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15ed161640 1 --2- 172.21.15.88:0/3412473566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15d0003b10 0x7f15e8144530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:36:59.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15ed161640 1 -- 172.21.15.88:0/3412473566 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e8151f50 con 0x7f15e8151980 2023-10-16T20:36:59.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15ed161640 1 --2- 172.21.15.88:0/3412473566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15e8151980 0x7f15e8157d40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f15e80ccca0 tx=0x7f15e000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:59.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15e57fa640 1 -- 172.21.15.88:0/3412473566 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e000e040 con 0x7f15e8151980 2023-10-16T20:36:59.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15e57fa640 1 -- 172.21.15.88:0/3412473566 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e000f040 con 0x7f15e8151980 2023-10-16T20:36:59.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15e57fa640 1 -- 172.21.15.88:0/3412473566 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e0015430 con 0x7f15e8151980 2023-10-16T20:36:59.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15eebc4640 1 -- 172.21.15.88:0/3412473566 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e80b46c0 con 0x7f15e8151980 2023-10-16T20:36:59.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.475+0000 7f15eebc4640 1 -- 172.21.15.88:0/3412473566 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e80b4c80 con 0x7f15e8151980 2023-10-16T20:36:59.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.479+0000 7f15eebc4640 1 -- 172.21.15.88:0/3412473566 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e81575c0 con 0x7f15e8151980 2023-10-16T20:36:59.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.479+0000 7f15e57fa640 1 -- 172.21.15.88:0/3412473566 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f15e0015590 con 0x7f15e8151980 2023-10-16T20:36:59.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.479+0000 7f15e57fa640 1 --2- 172.21.15.88:0/3412473566 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f15d40664d0 0x7f15d4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:36:59.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.479+0000 7f15e57fa640 1 -- 172.21.15.88:0/3412473566 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6428+0+0 (secure 0 0 0) 0x7f15e0084840 con 0x7f15e8151980 2023-10-16T20:36:59.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.479+0000 7f15ec960640 1 --2- 172.21.15.88:0/3412473566 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f15d40664d0 0x7f15d4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:36:59.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.479+0000 7f15ec960640 1 --2- 172.21.15.88:0/3412473566 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f15d40664d0 0x7f15d4068990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f15d80ac000 tx=0x7f15d80b0040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:36:59.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.483+0000 7f15e57fa640 1 -- 172.21.15.88:0/3412473566 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f15e0051a10 con 0x7f15e8151980 2023-10-16T20:36:59.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:36:59.667+0000 7f15eebc4640 1 -- 172.21.15.88:0/3412473566 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f15e80ca7d0 con 0x7f15e8151980 2023-10-16T20:37:00.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.295+0000 7f15e57fa640 1 -- 172.21.15.88:0/3412473566 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v5) v1 ==== 137+0+0 (secure 0 0 0) 0x7f15e0055030 con 0x7f15e8151980 2023-10-16T20:37:00.300 INFO:teuthology.orchestra.run.smithi088.stderr:added data pool 4 to fsmap 2023-10-16T20:37:00.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.295+0000 7f15eebc4640 1 -- 172.21.15.88:0/3412473566 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f15d40664d0 msgr2=0x7f15d4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:00.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.295+0000 7f15eebc4640 1 --2- 172.21.15.88:0/3412473566 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f15d40664d0 0x7f15d4068990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f15d80ac000 tx=0x7f15d80b0040 comp rx=0 tx=0).stop 2023-10-16T20:37:00.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.295+0000 7f15eebc4640 1 -- 172.21.15.88:0/3412473566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15e8151980 msgr2=0x7f15e8157d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:00.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.295+0000 7f15eebc4640 1 --2- 172.21.15.88:0/3412473566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15e8151980 0x7f15e8157d40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f15e80ccca0 tx=0x7f15e000f580 comp rx=0 tx=0).stop 2023-10-16T20:37:00.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.295+0000 7f15eebc4640 1 -- 172.21.15.88:0/3412473566 shutdown_connections 2023-10-16T20:37:00.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.295+0000 7f15eebc4640 1 --2- 172.21.15.88:0/3412473566 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f15d40664d0 0x7f15d4068990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.295+0000 7f15eebc4640 1 --2- 172.21.15.88:0/3412473566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15e8151980 0x7f15e8157d40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.295+0000 7f15eebc4640 1 --2- 172.21.15.88:0/3412473566 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15e8152520 0x7f15e8157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.295+0000 7f15eebc4640 1 --2- 172.21.15.88:0/3412473566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15d0003b10 0x7f15e8144530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.295+0000 7f15eebc4640 1 -- 172.21.15.88:0/3412473566 >> 172.21.15.88:0/3412473566 conn(0x7f15e806bcb0 msgr2=0x7f15e80ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:37:00.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.295+0000 7f15eebc4640 1 -- 172.21.15.88:0/3412473566 shutdown_connections 2023-10-16T20:37:00.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.295+0000 7f15eebc4640 1 -- 172.21.15.88:0/3412473566 wait complete. 2023-10-16T20:37:00.323 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:37:00.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.455+0000 7f551a0c9640 1 Processor -- start 2023-10-16T20:37:00.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.455+0000 7f551a0c9640 1 -- start start 2023-10-16T20:37:00.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.455+0000 7f551a0c9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f55140c7dc0 0x7f5514159450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.455+0000 7f551a0c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f55140c81a0 0x7f551406b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.455+0000 7f551a0c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55140c8810 con 0x7f55140c76c0 2023-10-16T20:37:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.455+0000 7f551a0c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f551406b400 con 0x7f55140c81a0 2023-10-16T20:37:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.455+0000 7f551a0c9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f551406b0d0 con 0x7f55140c7dc0 2023-10-16T20:37:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.455+0000 7f5513fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f55140c81a0 0x7f551406b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.455+0000 7f5512ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f55140c76c0 0x7f551406d650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60860/0 (socket says 172.21.15.88:60860) 2023-10-16T20:37:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.455+0000 7f5512ffd640 1 -- 172.21.15.88:0/2920725354 learned_addr learned my addr 172.21.15.88:0/2920725354 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:37:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.455+0000 7f55137fe640 1 --2- 172.21.15.88:0/2920725354 >> v2:172.21.15.191:3300/0 conn(0x7f55140c7dc0 0x7f5514159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.459+0000 7f5513fff640 1 -- 172.21.15.88:0/2920725354 >> v2:172.21.15.191:3300/0 conn(0x7f55140c7dc0 msgr2=0x7f5514159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:37:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.459+0000 7f5513fff640 1 --2- 172.21.15.88:0/2920725354 >> v2:172.21.15.191:3300/0 conn(0x7f55140c7dc0 0x7f5514159450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.459+0000 7f5513fff640 1 -- 172.21.15.88:0/2920725354 >> v1:172.21.15.88:6789/0 conn(0x7f55140c76c0 legacy=0x7f551406d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:00.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.459+0000 7f5513fff640 1 -- 172.21.15.88:0/2920725354 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551406e6a0 con 0x7f55140c81a0 2023-10-16T20:37:00.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.459+0000 7f55127fc640 1 -- 172.21.15.88:0/2920725354 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785578768 0 0) 0x7f55140c8810 con 0x7f55140c76c0 2023-10-16T20:37:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.459+0000 7f5513fff640 1 --2- 172.21.15.88:0/2920725354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55140c81a0 0x7f551406b890 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f55140c7310 tx=0x7f550c01c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f76eeead4a5afa2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.459+0000 7f55127fc640 1 -- 172.21.15.88:0/2920725354 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c0a5740 con 0x7f55140c81a0 2023-10-16T20:37:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.459+0000 7f55127fc640 1 -- 172.21.15.88:0/2920725354 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550c0a58a0 con 0x7f55140c81a0 2023-10-16T20:37:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.459+0000 7f55127fc640 1 -- 172.21.15.88:0/2920725354 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c0a5c30 con 0x7f55140c81a0 2023-10-16T20:37:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.459+0000 7f551a0c9640 1 -- 172.21.15.88:0/2920725354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55140c81a0 msgr2=0x7f551406b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.459+0000 7f551a0c9640 1 --2- 172.21.15.88:0/2920725354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55140c81a0 0x7f551406b890 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f55140c7310 tx=0x7f550c01c880 comp rx=0 tx=0).stop 2023-10-16T20:37:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.459+0000 7f551a0c9640 1 -- 172.21.15.88:0/2920725354 shutdown_connections 2023-10-16T20:37:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.459+0000 7f551a0c9640 1 --2- 172.21.15.88:0/2920725354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55140c81a0 0x7f551406b890 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.459+0000 7f551a0c9640 1 --2- 172.21.15.88:0/2920725354 >> v2:172.21.15.191:3300/0 conn(0x7f55140c7dc0 0x7f5514159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.459+0000 7f551a0c9640 1 -- 172.21.15.88:0/2920725354 >> 172.21.15.88:0/2920725354 conn(0x7f5514071a60 msgr2=0x7f5514071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:37:00.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.459+0000 7f551a0c9640 1 -- 172.21.15.88:0/2920725354 shutdown_connections 2023-10-16T20:37:00.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.459+0000 7f551a0c9640 1 -- 172.21.15.88:0/2920725354 wait complete. 2023-10-16T20:37:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f551a0c9640 1 Processor -- start 2023-10-16T20:37:00.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f551a0c9640 1 -- start start 2023-10-16T20:37:00.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f551a0c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55140c76c0 0x7f55141468d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:00.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f551a0c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55140c7dc0 0x7f551414fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:00.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f551a0c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55140c81a0 0x7f5514150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:00.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f551a0c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5514150e60 con 0x7f55140c7dc0 2023-10-16T20:37:00.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f551a0c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5514151670 con 0x7f55140c76c0 2023-10-16T20:37:00.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f551a0c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55141f60b0 con 0x7f55140c81a0 2023-10-16T20:37:00.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f5512ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55140c7dc0 0x7f551414fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:00.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f55137fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55140c76c0 0x7f55141468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:00.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f5512ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55140c7dc0 0x7f551414fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34392/0 (socket says 172.21.15.88:34392) 2023-10-16T20:37:00.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f5512ffd640 1 -- 172.21.15.88:0/1232436504 learned_addr learned my addr 172.21.15.88:0/1232436504 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:37:00.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f5513fff640 1 --2- 172.21.15.88:0/1232436504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55140c81a0 0x7f5514150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:00.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f5512ffd640 1 -- 172.21.15.88:0/1232436504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55140c81a0 msgr2=0x7f5514150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:00.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f5512ffd640 1 --2- 172.21.15.88:0/1232436504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55140c81a0 0x7f5514150060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f5512ffd640 1 -- 172.21.15.88:0/1232436504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55140c76c0 msgr2=0x7f55141468d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:00.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f5512ffd640 1 --2- 172.21.15.88:0/1232436504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55140c76c0 0x7f55141468d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f5512ffd640 1 -- 172.21.15.88:0/1232436504 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550c009a70 con 0x7f55140c7dc0 2023-10-16T20:37:00.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f5512ffd640 1 --2- 172.21.15.88:0/1232436504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55140c7dc0 0x7f551414fb20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f550800c4b0 tx=0x7f550800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:00.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f54fbfff640 1 -- 172.21.15.88:0/1232436504 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55080074e0 con 0x7f55140c7dc0 2023-10-16T20:37:00.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f54fbfff640 1 -- 172.21.15.88:0/1232436504 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5508007660 con 0x7f55140c7dc0 2023-10-16T20:37:00.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f54fbfff640 1 -- 172.21.15.88:0/1232436504 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5508004a30 con 0x7f55140c7dc0 2023-10-16T20:37:00.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.463+0000 7f551a0c9640 1 -- 172.21.15.88:0/1232436504 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55141510b0 con 0x7f55140c7dc0 2023-10-16T20:37:00.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.467+0000 7f551a0c9640 1 -- 172.21.15.88:0/1232436504 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55141573f0 con 0x7f55140c7dc0 2023-10-16T20:37:00.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.467+0000 7f551a0c9640 1 -- 172.21.15.88:0/1232436504 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5514157610 con 0x7f55140c7dc0 2023-10-16T20:37:00.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.467+0000 7f54fbfff640 1 -- 172.21.15.88:0/1232436504 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5508004b90 con 0x7f55140c7dc0 2023-10-16T20:37:00.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.467+0000 7f54fbfff640 1 --2- 172.21.15.88:0/1232436504 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55000664a0 0x7f5500068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:00.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.467+0000 7f54fbfff640 1 -- 172.21.15.88:0/1232436504 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5508083250 con 0x7f55140c7dc0 2023-10-16T20:37:00.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.467+0000 7f55137fe640 1 --2- 172.21.15.88:0/1232436504 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55000664a0 0x7f5500068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:00.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.467+0000 7f55137fe640 1 --2- 172.21.15.88:0/1232436504 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55000664a0 0x7f5500068960 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5504009b60 tx=0x7f5504009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:00.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.471+0000 7f54fbfff640 1 -- 172.21.15.88:0/1232436504 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5508050480 con 0x7f55140c7dc0 2023-10-16T20:37:00.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.655+0000 7f551a0c9640 1 -- 172.21.15.88:0/1232436504 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5514074700 con 0x7f55140c7dc0 2023-10-16T20:37:00.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.655+0000 7f54fbfff640 1 -- 172.21.15.88:0/1232436504 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+6494 (secure 0 0 0) 0x7f5508053aa0 con 0x7f55140c7dc0 2023-10-16T20:37:00.661 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:37:00.662 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":5,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4275,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:36:59.673657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:37:00.662 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 5 2023-10-16T20:37:00.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.655+0000 7f551a0c9640 1 -- 172.21.15.88:0/1232436504 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55000664a0 msgr2=0x7f5500068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:00.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.655+0000 7f551a0c9640 1 --2- 172.21.15.88:0/1232436504 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55000664a0 0x7f5500068960 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5504009b60 tx=0x7f5504009190 comp rx=0 tx=0).stop 2023-10-16T20:37:00.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.655+0000 7f551a0c9640 1 -- 172.21.15.88:0/1232436504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55140c7dc0 msgr2=0x7f551414fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:00.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.655+0000 7f551a0c9640 1 --2- 172.21.15.88:0/1232436504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55140c7dc0 0x7f551414fb20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f550800c4b0 tx=0x7f550800c970 comp rx=0 tx=0).stop 2023-10-16T20:37:00.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.659+0000 7f551a0c9640 1 -- 172.21.15.88:0/1232436504 shutdown_connections 2023-10-16T20:37:00.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.659+0000 7f551a0c9640 1 --2- 172.21.15.88:0/1232436504 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55000664a0 0x7f5500068960 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.659+0000 7f551a0c9640 1 --2- 172.21.15.88:0/1232436504 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55140c81a0 0x7f5514150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.659+0000 7f551a0c9640 1 --2- 172.21.15.88:0/1232436504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55140c7dc0 0x7f551414fb20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.659+0000 7f551a0c9640 1 --2- 172.21.15.88:0/1232436504 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f55140c76c0 0x7f55141468d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.659+0000 7f551a0c9640 1 -- 172.21.15.88:0/1232436504 >> 172.21.15.88:0/1232436504 conn(0x7f5514071a60 msgr2=0x7f5514155680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:37:00.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.659+0000 7f551a0c9640 1 -- 172.21.15.88:0/1232436504 shutdown_connections 2023-10-16T20:37:00.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.659+0000 7f551a0c9640 1 -- 172.21.15.88:0/1232436504 wait complete. 2023-10-16T20:37:00.683 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:37:00.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.819+0000 7fc7bb491640 1 Processor -- start 2023-10-16T20:37:00.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.819+0000 7fc7bb491640 1 -- start start 2023-10-16T20:37:00.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.823+0000 7fc7bb491640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc7b4160e70 0x7fc7b41612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:00.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.823+0000 7fc7bb491640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b4161800 con 0x7fc7b4157b10 2023-10-16T20:37:00.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.823+0000 7fc7bb491640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b40c5e10 con 0x7fc7b40c58d0 2023-10-16T20:37:00.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.823+0000 7fc7bb491640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7b406f840 con 0x7fc7b4160e70 2023-10-16T20:37:00.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.823+0000 7fc7b922d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc7b4157b10 0x7fc7b415e2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60862/0 (socket says 172.21.15.88:60862) 2023-10-16T20:37:00.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.823+0000 7fc7b922d640 1 -- 172.21.15.88:0/81467211 learned_addr learned my addr 172.21.15.88:0/81467211 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:37:00.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.823+0000 7fc7b9a2e640 1 --2- 172.21.15.88:0/81467211 >> v2:172.21.15.88:3300/0 conn(0x7fc7b4160e70 0x7fc7b41612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:00.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.823+0000 7fc7abfff640 1 -- 172.21.15.88:0/81467211 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005342615 0 0) 0x7fc7b4161800 con 0x7fc7b4157b10 2023-10-16T20:37:00.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.823+0000 7fc7abfff640 1 -- 172.21.15.88:0/81467211 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a0003530 con 0x7fc7b4157b10 2023-10-16T20:37:00.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.823+0000 7fc7b9a2e640 1 -- 172.21.15.88:0/81467211 >> v1:172.21.15.191:6789/0 conn(0x7fc7b40c58d0 legacy=0x7fc7b406d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:00.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.823+0000 7fc7abfff640 1 -- 172.21.15.88:0/81467211 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519378546 0 0) 0x7fc7b40c5e10 con 0x7fc7b40c58d0 2023-10-16T20:37:00.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.823+0000 7fc7b9a2e640 1 -- 172.21.15.88:0/81467211 >> v1:172.21.15.88:6789/0 conn(0x7fc7b4157b10 legacy=0x7fc7b415e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:00.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.823+0000 7fc7b9a2e640 1 -- 172.21.15.88:0/81467211 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b406fb20 con 0x7fc7b4160e70 2023-10-16T20:37:00.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.823+0000 7fc7b9a2e640 1 --2- 172.21.15.88:0/81467211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7b4160e70 0x7fc7b41612c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009d90 tx=0x7fc7ac01c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b364bbc7e7386504 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:00.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7abfff640 1 -- 172.21.15.88:0/81467211 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0a5950 con 0x7fc7b4160e70 2023-10-16T20:37:00.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7abfff640 1 -- 172.21.15.88:0/81467211 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac0a5ab0 con 0x7fc7b4160e70 2023-10-16T20:37:00.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7abfff640 1 -- 172.21.15.88:0/81467211 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0a5e20 con 0x7fc7b4160e70 2023-10-16T20:37:00.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7bb491640 1 -- 172.21.15.88:0/81467211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7b4160e70 msgr2=0x7fc7b41612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:00.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7bb491640 1 --2- 172.21.15.88:0/81467211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7b4160e70 0x7fc7b41612c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009d90 tx=0x7fc7ac01c890 comp rx=0 tx=0).stop 2023-10-16T20:37:00.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7bb491640 1 -- 172.21.15.88:0/81467211 shutdown_connections 2023-10-16T20:37:00.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7bb491640 1 --2- 172.21.15.88:0/81467211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7b4160e70 0x7fc7b41612c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7bb491640 1 -- 172.21.15.88:0/81467211 >> 172.21.15.88:0/81467211 conn(0x7fc7b40cad50 msgr2=0x7fc7b40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:37:00.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7bb491640 1 -- 172.21.15.88:0/81467211 shutdown_connections 2023-10-16T20:37:00.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7bb491640 1 -- 172.21.15.88:0/81467211 wait complete. 2023-10-16T20:37:00.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7bb491640 1 Processor -- start 2023-10-16T20:37:00.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7bb491640 1 -- start start 2023-10-16T20:37:00.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7bb491640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7b40c58d0 0x7fc7b40c32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:00.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7bb491640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7b4157b10 0x7fc7b4151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:00.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7bb491640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7b4160e70 0x7fc7b415c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:00.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7bb491640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b40716a0 con 0x7fc7b4160e70 2023-10-16T20:37:00.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7bb491640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b4151c60 con 0x7fc7b4157b10 2023-10-16T20:37:00.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7bb491640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b415cab0 con 0x7fc7b40c58d0 2023-10-16T20:37:00.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7b9a2e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7b4160e70 0x7fc7b415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:00.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7b9a2e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7b4160e70 0x7fc7b415c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34408/0 (socket says 172.21.15.88:34408) 2023-10-16T20:37:00.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7b9a2e640 1 -- 172.21.15.88:0/3516528621 learned_addr learned my addr 172.21.15.88:0/3516528621 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:37:00.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7b8a2c640 1 --2- 172.21.15.88:0/3516528621 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7b4157b10 0x7fc7b4151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:00.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.827+0000 7fc7b922d640 1 --2- 172.21.15.88:0/3516528621 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7b40c58d0 0x7fc7b40c32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:00.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.831+0000 7fc7b9a2e640 1 -- 172.21.15.88:0/3516528621 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7b40c58d0 msgr2=0x7fc7b40c32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:00.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.831+0000 7fc7b9a2e640 1 --2- 172.21.15.88:0/3516528621 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7b40c58d0 0x7fc7b40c32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.831+0000 7fc7b9a2e640 1 -- 172.21.15.88:0/3516528621 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7b4157b10 msgr2=0x7fc7b4151720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:00.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.831+0000 7fc7b9a2e640 1 --2- 172.21.15.88:0/3516528621 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7b4157b10 0x7fc7b4151720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.831+0000 7fc7b9a2e640 1 -- 172.21.15.88:0/3516528621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac009a70 con 0x7fc7b4160e70 2023-10-16T20:37:00.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.831+0000 7fc7b9a2e640 1 --2- 172.21.15.88:0/3516528621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7b4160e70 0x7fc7b415c570 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009d90 tx=0x7fc7ac019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:00.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.831+0000 7fc7a9ffb640 1 -- 172.21.15.88:0/3516528621 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0aede0 con 0x7fc7b4160e70 2023-10-16T20:37:00.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.831+0000 7fc7a9ffb640 1 -- 172.21.15.88:0/3516528621 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac009450 con 0x7fc7b4160e70 2023-10-16T20:37:00.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.831+0000 7fc7a9ffb640 1 -- 172.21.15.88:0/3516528621 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0b7610 con 0x7fc7b4160e70 2023-10-16T20:37:00.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.831+0000 7fc7bb491640 1 -- 172.21.15.88:0/3516528621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b40c2700 con 0x7fc7b4160e70 2023-10-16T20:37:00.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.831+0000 7fc7bb491640 1 -- 172.21.15.88:0/3516528621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b4071420 con 0x7fc7b4160e70 2023-10-16T20:37:00.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.831+0000 7fc7bb491640 1 -- 172.21.15.88:0/3516528621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b4074700 con 0x7fc7b4160e70 2023-10-16T20:37:00.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.831+0000 7fc7a9ffb640 1 -- 172.21.15.88:0/3516528621 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc7ac0b4050 con 0x7fc7b4160e70 2023-10-16T20:37:00.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.831+0000 7fc7a9ffb640 1 --2- 172.21.15.88:0/3516528621 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc7980664a0 0x7fc798068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:00.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.831+0000 7fc7a9ffb640 1 -- 172.21.15.88:0/3516528621 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc7ac11f600 con 0x7fc7b4160e70 2023-10-16T20:37:00.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.831+0000 7fc7b922d640 1 --2- 172.21.15.88:0/3516528621 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc7980664a0 0x7fc798068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:00.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.835+0000 7fc7b922d640 1 --2- 172.21.15.88:0/3516528621 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc7980664a0 0x7fc798068960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0005890 tx=0x7fc7b0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:00.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.835+0000 7fc7a9ffb640 1 -- 172.21.15.88:0/3516528621 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc7ac0ec950 con 0x7fc7b4160e70 2023-10-16T20:37:00.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.987+0000 7fc7bb491640 1 -- 172.21.15.88:0/3516528621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7b40c6410 con 0x7fc7b4160e70 2023-10-16T20:37:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.991+0000 7fc7a9ffb640 1 -- 172.21.15.88:0/3516528621 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18322 (secure 0 0 0) 0x7fc7ac0eff70 con 0x7fc7b4160e70 2023-10-16T20:37:00.996 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:37:00.996 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":27,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:36:59.673478+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:37:00.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.991+0000 7fc7bb491640 1 -- 172.21.15.88:0/3516528621 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc7980664a0 msgr2=0x7fc798068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:00.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.991+0000 7fc7bb491640 1 --2- 172.21.15.88:0/3516528621 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc7980664a0 0x7fc798068960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0005890 tx=0x7fc7b0009040 comp rx=0 tx=0).stop 2023-10-16T20:37:00.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.991+0000 7fc7bb491640 1 -- 172.21.15.88:0/3516528621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7b4160e70 msgr2=0x7fc7b415c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.991+0000 7fc7bb491640 1 --2- 172.21.15.88:0/3516528621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7b4160e70 0x7fc7b415c570 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009d90 tx=0x7fc7ac019630 comp rx=0 tx=0).stop 2023-10-16T20:37:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.991+0000 7fc7bb491640 1 -- 172.21.15.88:0/3516528621 shutdown_connections 2023-10-16T20:37:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.991+0000 7fc7bb491640 1 --2- 172.21.15.88:0/3516528621 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc7980664a0 0x7fc798068960 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.991+0000 7fc7bb491640 1 --2- 172.21.15.88:0/3516528621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7b4160e70 0x7fc7b415c570 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.991+0000 7fc7bb491640 1 --2- 172.21.15.88:0/3516528621 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7b4157b10 0x7fc7b4151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.991+0000 7fc7bb491640 1 --2- 172.21.15.88:0/3516528621 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7b40c58d0 0x7fc7b40c32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.991+0000 7fc7bb491640 1 -- 172.21.15.88:0/3516528621 >> 172.21.15.88:0/3516528621 conn(0x7fc7b40cad50 msgr2=0x7fc7b4161280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:37:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.991+0000 7fc7bb491640 1 -- 172.21.15.88:0/3516528621 shutdown_connections 2023-10-16T20:37:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:00.991+0000 7fc7bb491640 1 -- 172.21.15.88:0/3516528621 wait complete. 2023-10-16T20:37:01.017 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:37:01.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.147+0000 7fb5f5409640 1 Processor -- start 2023-10-16T20:37:01.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.151+0000 7fb5f5409640 1 -- start start 2023-10-16T20:37:01.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.151+0000 7fb5f5409640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb5f0153770 0x7fb5f0159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.151+0000 7fb5f5409640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb5f00c9bb0 0x7fb5f00c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.151+0000 7fb5f5409640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f015a410 con 0x7fb5f00c41f0 2023-10-16T20:37:01.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.151+0000 7fb5f5409640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5f00cc250 con 0x7fb5f00c9bb0 2023-10-16T20:37:01.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.151+0000 7fb5f5409640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5f00cc670 con 0x7fb5f0153770 2023-10-16T20:37:01.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.151+0000 7fb5ee7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb5f00c9bb0 0x7fb5f00c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.151+0000 7fb5ee7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb5f00c9bb0 0x7fb5f00c3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34412/0 (socket says 172.21.15.88:34412) 2023-10-16T20:37:01.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.151+0000 7fb5ee7fc640 1 -- 172.21.15.88:0/2630778208 learned_addr learned my addr 172.21.15.88:0/2630778208 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:37:01.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.151+0000 7fb5eeffd640 1 --2- 172.21.15.88:0/2630778208 >> v2:172.21.15.191:3300/0 conn(0x7fb5f0153770 0x7fb5f0159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.151+0000 7fb5ee7fc640 1 -- 172.21.15.88:0/2630778208 >> v2:172.21.15.191:3300/0 conn(0x7fb5f0153770 msgr2=0x7fb5f0159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:37:01.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.151+0000 7fb5ee7fc640 1 --2- 172.21.15.88:0/2630778208 >> v2:172.21.15.191:3300/0 conn(0x7fb5f0153770 0x7fb5f0159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.151+0000 7fb5ee7fc640 1 -- 172.21.15.88:0/2630778208 >> v1:172.21.15.191:6789/0 conn(0x7fb5f00c41f0 legacy=0x7fb5f00cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:01.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.151+0000 7fb5ee7fc640 1 -- 172.21.15.88:0/2630778208 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f0153b50 con 0x7fb5f00c9bb0 2023-10-16T20:37:01.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.155+0000 7fb5ee7fc640 1 --2- 172.21.15.88:0/2630778208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5f00c9bb0 0x7fb5f00c3cb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb5f00cc7b0 tx=0x7fb5e401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896827380431a59 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:01.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.155+0000 7fb5edffb640 1 -- 172.21.15.88:0/2630778208 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e40ab610 con 0x7fb5f00c9bb0 2023-10-16T20:37:01.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.155+0000 7fb5edffb640 1 -- 172.21.15.88:0/2630778208 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e40ab770 con 0x7fb5f00c9bb0 2023-10-16T20:37:01.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.155+0000 7fb5edffb640 1 -- 172.21.15.88:0/2630778208 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e40abb00 con 0x7fb5f00c9bb0 2023-10-16T20:37:01.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.155+0000 7fb5f5409640 1 -- 172.21.15.88:0/2630778208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5f00c9bb0 msgr2=0x7fb5f00c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:01.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.155+0000 7fb5f5409640 1 --2- 172.21.15.88:0/2630778208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5f00c9bb0 0x7fb5f00c3cb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb5f00cc7b0 tx=0x7fb5e401c500 comp rx=0 tx=0).stop 2023-10-16T20:37:01.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.155+0000 7fb5f5409640 1 -- 172.21.15.88:0/2630778208 shutdown_connections 2023-10-16T20:37:01.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.155+0000 7fb5f5409640 1 --2- 172.21.15.88:0/2630778208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5f00c9bb0 0x7fb5f00c3cb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.155+0000 7fb5f5409640 1 --2- 172.21.15.88:0/2630778208 >> v2:172.21.15.191:3300/0 conn(0x7fb5f0153770 0x7fb5f0159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.155+0000 7fb5f5409640 1 -- 172.21.15.88:0/2630778208 >> 172.21.15.88:0/2630778208 conn(0x7fb5f00715c0 msgr2=0x7fb5f00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:37:01.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.155+0000 7fb5f5409640 1 -- 172.21.15.88:0/2630778208 shutdown_connections 2023-10-16T20:37:01.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.155+0000 7fb5f5409640 1 -- 172.21.15.88:0/2630778208 wait complete. 2023-10-16T20:37:01.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.155+0000 7fb5f5409640 1 Processor -- start 2023-10-16T20:37:01.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.155+0000 7fb5f5409640 1 -- start start 2023-10-16T20:37:01.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5f5409640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb5f00c41f0 0x7fb5f00c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5f5409640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5f00c9bb0 0x7fb5f0159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5f5409640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5f0153770 0x7fb5f01584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5f5409640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f00c6e90 con 0x7fb5f0153770 2023-10-16T20:37:01.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5f5409640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f00c62a0 con 0x7fb5f00c9bb0 2023-10-16T20:37:01.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5f5409640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f0157f60 con 0x7fb5f00c41f0 2023-10-16T20:37:01.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5ef7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5f0153770 0x7fb5f01584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5ef7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5f0153770 0x7fb5f01584d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34426/0 (socket says 172.21.15.88:34426) 2023-10-16T20:37:01.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5ef7fe640 1 -- 172.21.15.88:0/408612208 learned_addr learned my addr 172.21.15.88:0/408612208 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:37:01.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5eeffd640 1 --2- 172.21.15.88:0/408612208 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb5f00c41f0 0x7fb5f00c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5ee7fc640 1 --2- 172.21.15.88:0/408612208 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5f00c9bb0 0x7fb5f0159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5ef7fe640 1 -- 172.21.15.88:0/408612208 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb5f00c41f0 msgr2=0x7fb5f00c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5ef7fe640 1 --2- 172.21.15.88:0/408612208 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb5f00c41f0 0x7fb5f00c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5ef7fe640 1 -- 172.21.15.88:0/408612208 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5f00c9bb0 msgr2=0x7fb5f0159170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5ef7fe640 1 --2- 172.21.15.88:0/408612208 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5f00c9bb0 0x7fb5f0159170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5ef7fe640 1 -- 172.21.15.88:0/408612208 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e40094d0 con 0x7fb5f0153770 2023-10-16T20:37:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5ef7fe640 1 --2- 172.21.15.88:0/408612208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5f0153770 0x7fb5f01584d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800c770 tx=0x7fb5e800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5d77fe640 1 -- 172.21.15.88:0/408612208 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8003b80 con 0x7fb5f0153770 2023-10-16T20:37:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5d77fe640 1 -- 172.21.15.88:0/408612208 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e8003ce0 con 0x7fb5f0153770 2023-10-16T20:37:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5f5409640 1 -- 172.21.15.88:0/408612208 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f00c64f0 con 0x7fb5f0153770 2023-10-16T20:37:01.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5d77fe640 1 -- 172.21.15.88:0/408612208 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e800aa90 con 0x7fb5f0153770 2023-10-16T20:37:01.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5f5409640 1 -- 172.21.15.88:0/408612208 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f00c6ab0 con 0x7fb5f0153770 2023-10-16T20:37:01.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5f5409640 1 -- 172.21.15.88:0/408612208 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f00c6fd0 con 0x7fb5f0153770 2023-10-16T20:37:01.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.159+0000 7fb5d77fe640 1 -- 172.21.15.88:0/408612208 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb5e800abf0 con 0x7fb5f0153770 2023-10-16T20:37:01.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.163+0000 7fb5d77fe640 1 --2- 172.21.15.88:0/408612208 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb5dc0664a0 0x7fb5dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.163+0000 7fb5d77fe640 1 -- 172.21.15.88:0/408612208 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb5e8083e70 con 0x7fb5f0153770 2023-10-16T20:37:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.163+0000 7fb5eeffd640 1 --2- 172.21.15.88:0/408612208 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb5dc0664a0 0x7fb5dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.163+0000 7fb5eeffd640 1 --2- 172.21.15.88:0/408612208 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb5dc0664a0 0x7fb5dc068960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb5e00079a0 tx=0x7fb5e0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.167+0000 7fb5d77fe640 1 -- 172.21.15.88:0/408612208 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb5f00c6fd0 con 0x7fb5f0153770 2023-10-16T20:37:01.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.315+0000 7fb5f5409640 1 -- 172.21.15.88:0/408612208 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5f00c6fd0 con 0x7fb5f0153770 2023-10-16T20:37:01.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.315+0000 7fb5d77fe640 1 -- 172.21.15.88:0/408612208 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18322 (secure 0 0 0) 0x7fb5f00c6fd0 con 0x7fb5f0153770 2023-10-16T20:37:01.322 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:37:01.323 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":27,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:36:59.673478+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:37:01.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.319+0000 7fb5f5409640 1 -- 172.21.15.88:0/408612208 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb5dc0664a0 msgr2=0x7fb5dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:01.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.319+0000 7fb5f5409640 1 --2- 172.21.15.88:0/408612208 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb5dc0664a0 0x7fb5dc068960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb5e00079a0 tx=0x7fb5e0008040 comp rx=0 tx=0).stop 2023-10-16T20:37:01.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.319+0000 7fb5f5409640 1 -- 172.21.15.88:0/408612208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5f0153770 msgr2=0x7fb5f01584d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:01.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.319+0000 7fb5f5409640 1 --2- 172.21.15.88:0/408612208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5f0153770 0x7fb5f01584d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800c770 tx=0x7fb5e800cc30 comp rx=0 tx=0).stop 2023-10-16T20:37:01.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.319+0000 7fb5f5409640 1 -- 172.21.15.88:0/408612208 shutdown_connections 2023-10-16T20:37:01.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.319+0000 7fb5f5409640 1 --2- 172.21.15.88:0/408612208 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb5dc0664a0 0x7fb5dc068960 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.319+0000 7fb5f5409640 1 --2- 172.21.15.88:0/408612208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5f0153770 0x7fb5f01584d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.319+0000 7fb5f5409640 1 --2- 172.21.15.88:0/408612208 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb5f00c9bb0 0x7fb5f0159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.319+0000 7fb5f5409640 1 --2- 172.21.15.88:0/408612208 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb5f00c41f0 0x7fb5f00c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.319+0000 7fb5f5409640 1 -- 172.21.15.88:0/408612208 >> 172.21.15.88:0/408612208 conn(0x7fb5f00715c0 msgr2=0x7fb5f006bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:37:01.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.319+0000 7fb5f5409640 1 -- 172.21.15.88:0/408612208 shutdown_connections 2023-10-16T20:37:01.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.319+0000 7fb5f5409640 1 -- 172.21.15.88:0/408612208 wait complete. 2023-10-16T20:37:01.357 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:37:01.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.487+0000 7f8389e54640 1 Processor -- start 2023-10-16T20:37:01.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.487+0000 7f8389e54640 1 -- start start 2023-10-16T20:37:01.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.491+0000 7f8389e54640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8384150280 0x7f83840c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.491+0000 7f8389e54640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8384150660 0x7f83840c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.491+0000 7f8389e54640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83840c5c60 con 0x7f83840c6bc0 2023-10-16T20:37:01.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.491+0000 7f8389e54640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83840c5370 con 0x7f8384150660 2023-10-16T20:37:01.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.491+0000 7f8389e54640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8384156d10 con 0x7f8384150280 2023-10-16T20:37:01.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.491+0000 7f837bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8384150660 0x7f83840c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.491+0000 7f837bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8384150660 0x7f83840c3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34442/0 (socket says 172.21.15.88:34442) 2023-10-16T20:37:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.491+0000 7f837bfff640 1 -- 172.21.15.88:0/41146270 learned_addr learned my addr 172.21.15.88:0/41146270 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:37:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.491+0000 7f83837fe640 1 --2- 172.21.15.88:0/41146270 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8384150280 0x7f83840c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.491+0000 7f837bfff640 1 -- 172.21.15.88:0/41146270 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8384150280 msgr2=0x7f83840c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:37:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.491+0000 7f837bfff640 1 --2- 172.21.15.88:0/41146270 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8384150280 0x7f83840c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.491+0000 7f837bfff640 1 -- 172.21.15.88:0/41146270 >> v1:172.21.15.191:6789/0 conn(0x7f83840c6bc0 legacy=0x7f838406d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:37:01.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.491+0000 7f837bfff640 1 -- 172.21.15.88:0/41146270 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83840c5610 con 0x7f8384150660 2023-10-16T20:37:01.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.491+0000 7f83837fe640 1 --2- 172.21.15.88:0/41146270 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8384150280 0x7f83840c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:37:01.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.495+0000 7f837bfff640 1 --2- 172.21.15.88:0/41146270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8384150660 0x7f83840c3110 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f836c00bb10 tx=0x7f836c01f570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae92982f7a455b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:01.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.495+0000 7f8382ffd640 1 -- 172.21.15.88:0/41146270 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836c0a7bc0 con 0x7f8384150660 2023-10-16T20:37:01.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.495+0000 7f8382ffd640 1 -- 172.21.15.88:0/41146270 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f836c0a7d20 con 0x7f8384150660 2023-10-16T20:37:01.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.495+0000 7f8382ffd640 1 -- 172.21.15.88:0/41146270 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836c0af660 con 0x7f8384150660 2023-10-16T20:37:01.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.495+0000 7f8389e54640 1 -- 172.21.15.88:0/41146270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8384150660 msgr2=0x7f83840c3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:01.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.495+0000 7f8389e54640 1 --2- 172.21.15.88:0/41146270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8384150660 0x7f83840c3110 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f836c00bb10 tx=0x7f836c01f570 comp rx=0 tx=0).stop 2023-10-16T20:37:01.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.495+0000 7f8389e54640 1 -- 172.21.15.88:0/41146270 shutdown_connections 2023-10-16T20:37:01.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.495+0000 7f8389e54640 1 --2- 172.21.15.88:0/41146270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8384150660 0x7f83840c3110 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.495+0000 7f8389e54640 1 --2- 172.21.15.88:0/41146270 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8384150280 0x7f83840c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.495+0000 7f8389e54640 1 -- 172.21.15.88:0/41146270 >> 172.21.15.88:0/41146270 conn(0x7f83840cad50 msgr2=0x7f83840cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:37:01.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.495+0000 7f8389e54640 1 -- 172.21.15.88:0/41146270 shutdown_connections 2023-10-16T20:37:01.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.495+0000 7f8389e54640 1 -- 172.21.15.88:0/41146270 wait complete. 2023-10-16T20:37:01.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.495+0000 7f8389e54640 1 Processor -- start 2023-10-16T20:37:01.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.495+0000 7f8389e54640 1 -- start start 2023-10-16T20:37:01.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f8389e54640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83840c6bc0 0x7f8384143540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f8389e54640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8384150280 0x7f83841423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f8389e54640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8384150660 0x7f838413a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f8389e54640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8384142910 con 0x7f8384150280 2023-10-16T20:37:01.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f8389e54640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838414ecf0 con 0x7f83840c6bc0 2023-10-16T20:37:01.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f8389e54640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83841f56a0 con 0x7f8384150660 2023-10-16T20:37:01.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f837bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8384150280 0x7f83841423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f837bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8384150280 0x7f83841423d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34446/0 (socket says 172.21.15.88:34446) 2023-10-16T20:37:01.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f837bfff640 1 -- 172.21.15.88:0/95392535 learned_addr learned my addr 172.21.15.88:0/95392535 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:37:01.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f83837fe640 1 --2- 172.21.15.88:0/95392535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83840c6bc0 0x7f8384143540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f8383fff640 1 --2- 172.21.15.88:0/95392535 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8384150660 0x7f838413a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f837bfff640 1 -- 172.21.15.88:0/95392535 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8384150660 msgr2=0x7f838413a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:01.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f837bfff640 1 --2- 172.21.15.88:0/95392535 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8384150660 0x7f838413a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f837bfff640 1 -- 172.21.15.88:0/95392535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83840c6bc0 msgr2=0x7f8384143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:01.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f837bfff640 1 --2- 172.21.15.88:0/95392535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83840c6bc0 0x7f8384143540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f837bfff640 1 -- 172.21.15.88:0/95392535 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f836c00b770 con 0x7f8384150280 2023-10-16T20:37:01.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f83837fe640 1 --2- 172.21.15.88:0/95392535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83840c6bc0 0x7f8384143540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:37:01.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f8383fff640 1 --2- 172.21.15.88:0/95392535 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8384150660 0x7f838413a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:37:01.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f837bfff640 1 --2- 172.21.15.88:0/95392535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8384150280 0x7f83841423d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f836c01d1c0 tx=0x7f836c01b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:01.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f8380ff9640 1 -- 172.21.15.88:0/95392535 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836c0ba660 con 0x7f8384150280 2023-10-16T20:37:01.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f8380ff9640 1 -- 172.21.15.88:0/95392535 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f836c0a7e90 con 0x7f8384150280 2023-10-16T20:37:01.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f8380ff9640 1 -- 172.21.15.88:0/95392535 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836c0ba9e0 con 0x7f8384150280 2023-10-16T20:37:01.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f8389e54640 1 -- 172.21.15.88:0/95392535 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83841f4a90 con 0x7f8384150280 2023-10-16T20:37:01.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f8389e54640 1 -- 172.21.15.88:0/95392535 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83841f5050 con 0x7f8384150280 2023-10-16T20:37:01.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f8389e54640 1 -- 172.21.15.88:0/95392535 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8384074700 con 0x7f8384150280 2023-10-16T20:37:01.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.499+0000 7f8380ff9640 1 -- 172.21.15.88:0/95392535 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f836c0c2360 con 0x7f8384150280 2023-10-16T20:37:01.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.503+0000 7f8380ff9640 1 --2- 172.21.15.88:0/95392535 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f83640664a0 0x7f8364068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.503+0000 7f8380ff9640 1 -- 172.21.15.88:0/95392535 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f836c11ec90 con 0x7f8384150280 2023-10-16T20:37:01.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.503+0000 7f83837fe640 1 --2- 172.21.15.88:0/95392535 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f83640664a0 0x7f8364068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.503+0000 7f83837fe640 1 --2- 172.21.15.88:0/95392535 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f83640664a0 0x7f8364068960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8374006fd0 tx=0x7f8374008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:01.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.507+0000 7f8380ff9640 1 -- 172.21.15.88:0/95392535 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f836c0ebd80 con 0x7f8384150280 2023-10-16T20:37:01.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.687+0000 7f8389e54640 1 -- 172.21.15.88:0/95392535 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83841f57e0 con 0x7f8384150280 2023-10-16T20:37:01.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.687+0000 7f8380ff9640 1 -- 172.21.15.88:0/95392535 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+6494 (secure 0 0 0) 0x7f836c0bf090 con 0x7f8384150280 2023-10-16T20:37:01.695 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:37:01.695 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":5,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4275,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:36:59.673657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:37:01.697 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 5 2023-10-16T20:37:01.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.691+0000 7f8389e54640 1 -- 172.21.15.88:0/95392535 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f83640664a0 msgr2=0x7f8364068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:01.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.691+0000 7f8389e54640 1 --2- 172.21.15.88:0/95392535 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f83640664a0 0x7f8364068960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8374006fd0 tx=0x7f8374008040 comp rx=0 tx=0).stop 2023-10-16T20:37:01.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.691+0000 7f8389e54640 1 -- 172.21.15.88:0/95392535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8384150280 msgr2=0x7f83841423d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:01.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.691+0000 7f8389e54640 1 --2- 172.21.15.88:0/95392535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8384150280 0x7f83841423d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f836c01d1c0 tx=0x7f836c01b8b0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.691+0000 7f8389e54640 1 -- 172.21.15.88:0/95392535 shutdown_connections 2023-10-16T20:37:01.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.691+0000 7f8389e54640 1 --2- 172.21.15.88:0/95392535 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f83640664a0 0x7f8364068960 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.691+0000 7f8389e54640 1 --2- 172.21.15.88:0/95392535 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8384150660 0x7f838413a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.691+0000 7f8389e54640 1 --2- 172.21.15.88:0/95392535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8384150280 0x7f83841423d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.691+0000 7f8389e54640 1 --2- 172.21.15.88:0/95392535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f83840c6bc0 0x7f8384143540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.691+0000 7f8389e54640 1 -- 172.21.15.88:0/95392535 >> 172.21.15.88:0/95392535 conn(0x7f83840cad50 msgr2=0x7f83841558b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:37:01.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.691+0000 7f8389e54640 1 -- 172.21.15.88:0/95392535 shutdown_connections 2023-10-16T20:37:01.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.691+0000 7f8389e54640 1 -- 172.21.15.88:0/95392535 wait complete. 2023-10-16T20:37:01.715 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:37:01.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.847+0000 7f0fd6b80640 1 Processor -- start 2023-10-16T20:37:01.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.847+0000 7f0fd6b80640 1 -- start start 2023-10-16T20:37:01.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.847+0000 7f0fd6b80640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0fd00c9bb0 0x7f0fd00c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.847+0000 7f0fd6b80640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0fd00c41f0 0x7f0fd00cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.847+0000 7f0fd6b80640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd0153b50 con 0x7f0fd0153770 2023-10-16T20:37:01.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.847+0000 7f0fd6b80640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd00cc250 con 0x7f0fd00c41f0 2023-10-16T20:37:01.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.847+0000 7f0fd6b80640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd00cc670 con 0x7f0fd00c9bb0 2023-10-16T20:37:01.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.847+0000 7f0fd491c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0fd0153770 0x7f0fd0159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60878/0 (socket says 172.21.15.88:60878) 2023-10-16T20:37:01.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.847+0000 7f0fd511d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0fd00c41f0 0x7f0fd00cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.847+0000 7f0fd491c640 1 -- 172.21.15.88:0/1710940670 learned_addr learned my addr 172.21.15.88:0/1710940670 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:37:01.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.847+0000 7f0fcffff640 1 --2- 172.21.15.88:0/1710940670 >> v2:172.21.15.191:3300/0 conn(0x7f0fd00c9bb0 0x7f0fd00c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.847+0000 7f0fd511d640 1 -- 172.21.15.88:0/1710940670 >> v2:172.21.15.191:3300/0 conn(0x7f0fd00c9bb0 msgr2=0x7f0fd00c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:37:01.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.847+0000 7f0fd511d640 1 --2- 172.21.15.88:0/1710940670 >> v2:172.21.15.191:3300/0 conn(0x7f0fd00c9bb0 0x7f0fd00c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.847+0000 7f0fd511d640 1 -- 172.21.15.88:0/1710940670 >> v1:172.21.15.88:6789/0 conn(0x7f0fd0153770 legacy=0x7f0fd0159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:01.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.847+0000 7f0fd511d640 1 -- 172.21.15.88:0/1710940670 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd00cb2f0 con 0x7f0fd00c41f0 2023-10-16T20:37:01.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.847+0000 7f0fcf7fe640 1 -- 172.21.15.88:0/1710940670 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423202267 0 0) 0x7f0fd0153b50 con 0x7f0fd0153770 2023-10-16T20:37:01.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.851+0000 7f0fd511d640 1 --2- 172.21.15.88:0/1710940670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fd00c41f0 0x7f0fd00cece0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00cfe50 tx=0x7f0fc801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e953a5003fa07378 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:01.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.851+0000 7f0fcf7fe640 1 -- 172.21.15.88:0/1710940670 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc80a58e0 con 0x7f0fd00c41f0 2023-10-16T20:37:01.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.851+0000 7f0fcf7fe640 1 -- 172.21.15.88:0/1710940670 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc80a5a40 con 0x7f0fd00c41f0 2023-10-16T20:37:01.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.851+0000 7f0fcf7fe640 1 -- 172.21.15.88:0/1710940670 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc80a5dd0 con 0x7f0fd00c41f0 2023-10-16T20:37:01.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.851+0000 7f0fd6b80640 1 -- 172.21.15.88:0/1710940670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fd00c41f0 msgr2=0x7f0fd00cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:01.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.851+0000 7f0fd6b80640 1 --2- 172.21.15.88:0/1710940670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fd00c41f0 0x7f0fd00cece0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00cfe50 tx=0x7f0fc801c910 comp rx=0 tx=0).stop 2023-10-16T20:37:01.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.851+0000 7f0fd6b80640 1 -- 172.21.15.88:0/1710940670 shutdown_connections 2023-10-16T20:37:01.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.851+0000 7f0fd6b80640 1 --2- 172.21.15.88:0/1710940670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fd00c41f0 0x7f0fd00cece0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.851+0000 7f0fd6b80640 1 --2- 172.21.15.88:0/1710940670 >> v2:172.21.15.191:3300/0 conn(0x7f0fd00c9bb0 0x7f0fd00c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.851+0000 7f0fd6b80640 1 -- 172.21.15.88:0/1710940670 >> 172.21.15.88:0/1710940670 conn(0x7f0fd00715c0 msgr2=0x7f0fd00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:37:01.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.851+0000 7f0fd6b80640 1 -- 172.21.15.88:0/1710940670 shutdown_connections 2023-10-16T20:37:01.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.851+0000 7f0fd6b80640 1 -- 172.21.15.88:0/1710940670 wait complete. 2023-10-16T20:37:01.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd6b80640 1 Processor -- start 2023-10-16T20:37:01.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd6b80640 1 -- start start 2023-10-16T20:37:01.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd6b80640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fd00c41f0 0x7f0fd0146c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd6b80640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fd00c9bb0 0x7f0fd01582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd6b80640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fd0153770 0x7f0fd014fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd6b80640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd014f150 con 0x7f0fd0153770 2023-10-16T20:37:01.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd6b80640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd0147150 con 0x7f0fd00c41f0 2023-10-16T20:37:01.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd6b80640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd014d120 con 0x7f0fd00c9bb0 2023-10-16T20:37:01.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd511d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fd0153770 0x7f0fd014fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd511d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fd0153770 0x7f0fd014fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34456/0 (socket says 172.21.15.88:34456) 2023-10-16T20:37:01.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd511d640 1 -- 172.21.15.88:0/4080814812 learned_addr learned my addr 172.21.15.88:0/4080814812 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:37:01.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd491c640 1 --2- 172.21.15.88:0/4080814812 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fd00c41f0 0x7f0fd0146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fcffff640 1 --2- 172.21.15.88:0/4080814812 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fd00c9bb0 0x7f0fd01582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd511d640 1 -- 172.21.15.88:0/4080814812 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fd00c9bb0 msgr2=0x7f0fd01582f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:01.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd511d640 1 --2- 172.21.15.88:0/4080814812 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fd00c9bb0 0x7f0fd01582f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd511d640 1 -- 172.21.15.88:0/4080814812 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fd00c41f0 msgr2=0x7f0fd0146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:01.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd511d640 1 --2- 172.21.15.88:0/4080814812 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fd00c41f0 0x7f0fd0146c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:01.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd511d640 1 -- 172.21.15.88:0/4080814812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc8009a70 con 0x7f0fd0153770 2023-10-16T20:37:01.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd491c640 1 --2- 172.21.15.88:0/4080814812 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fd00c41f0 0x7f0fd0146c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:37:01.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd511d640 1 --2- 172.21.15.88:0/4080814812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fd0153770 0x7f0fd014fa10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0fc80b0040 tx=0x7f0fc80196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:01.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fcd7fa640 1 -- 172.21.15.88:0/4080814812 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc80b9860 con 0x7f0fd0153770 2023-10-16T20:37:01.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fcd7fa640 1 -- 172.21.15.88:0/4080814812 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc80b99c0 con 0x7f0fd0153770 2023-10-16T20:37:01.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fcd7fa640 1 -- 172.21.15.88:0/4080814812 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc80b9d70 con 0x7f0fd0153770 2023-10-16T20:37:01.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd6b80640 1 -- 172.21.15.88:0/4080814812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd014d400 con 0x7f0fd0153770 2023-10-16T20:37:01.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.855+0000 7f0fd6b80640 1 -- 172.21.15.88:0/4080814812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd01f20e0 con 0x7f0fd0153770 2023-10-16T20:37:01.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.859+0000 7f0fd6b80640 1 -- 172.21.15.88:0/4080814812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd0074700 con 0x7f0fd0153770 2023-10-16T20:37:01.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.859+0000 7f0fcd7fa640 1 -- 172.21.15.88:0/4080814812 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0fc80c4460 con 0x7f0fd0153770 2023-10-16T20:37:01.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.859+0000 7f0fcd7fa640 1 --2- 172.21.15.88:0/4080814812 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0fb40664a0 0x7f0fb4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:01.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.859+0000 7f0fcd7fa640 1 -- 172.21.15.88:0/4080814812 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0fc811f5d0 con 0x7f0fd0153770 2023-10-16T20:37:01.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.859+0000 7f0fd491c640 1 --2- 172.21.15.88:0/4080814812 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0fb40664a0 0x7f0fb4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:01.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.859+0000 7f0fd491c640 1 --2- 172.21.15.88:0/4080814812 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0fb40664a0 0x7f0fb4068960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4001740 tx=0x7f0fc4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:01.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:01.863+0000 7f0fcd7fa640 1 -- 172.21.15.88:0/4080814812 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0fc80ec6c0 con 0x7f0fd0153770 2023-10-16T20:37:02.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:02.015+0000 7f0fd6b80640 1 -- 172.21.15.88:0/4080814812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fd00cd2a0 con 0x7f0fd0153770 2023-10-16T20:37:02.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:02.015+0000 7f0fcd7fa640 1 -- 172.21.15.88:0/4080814812 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18322 (secure 0 0 0) 0x7f0fc80efce0 con 0x7f0fd0153770 2023-10-16T20:37:02.022 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:37:02.023 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":27,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:36:59.673478+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:37:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:02.019+0000 7f0fd6b80640 1 -- 172.21.15.88:0/4080814812 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0fb40664a0 msgr2=0x7f0fb4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:02.019+0000 7f0fd6b80640 1 --2- 172.21.15.88:0/4080814812 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0fb40664a0 0x7f0fb4068960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4001740 tx=0x7f0fc4009040 comp rx=0 tx=0).stop 2023-10-16T20:37:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:02.019+0000 7f0fd6b80640 1 -- 172.21.15.88:0/4080814812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fd0153770 msgr2=0x7f0fd014fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:02.019+0000 7f0fd6b80640 1 --2- 172.21.15.88:0/4080814812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fd0153770 0x7f0fd014fa10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0fc80b0040 tx=0x7f0fc80196b0 comp rx=0 tx=0).stop 2023-10-16T20:37:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:02.019+0000 7f0fd6b80640 1 -- 172.21.15.88:0/4080814812 shutdown_connections 2023-10-16T20:37:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:02.019+0000 7f0fd6b80640 1 --2- 172.21.15.88:0/4080814812 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0fb40664a0 0x7f0fb4068960 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:02.019+0000 7f0fd6b80640 1 --2- 172.21.15.88:0/4080814812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fd0153770 0x7f0fd014fa10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:02.019+0000 7f0fd6b80640 1 --2- 172.21.15.88:0/4080814812 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0fd00c9bb0 0x7f0fd01582f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:02.019+0000 7f0fd6b80640 1 --2- 172.21.15.88:0/4080814812 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0fd00c41f0 0x7f0fd0146c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:02.019+0000 7f0fd6b80640 1 -- 172.21.15.88:0/4080814812 >> 172.21.15.88:0/4080814812 conn(0x7f0fd00715c0 msgr2=0x7f0fd0149ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:37:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:02.019+0000 7f0fd6b80640 1 -- 172.21.15.88:0/4080814812 shutdown_connections 2023-10-16T20:37:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:02.019+0000 7f0fd6b80640 1 -- 172.21.15.88:0/4080814812 wait complete. 2023-10-16T20:37:02.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 5, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T20:36:55.111692+0000', 'modified': '2023-10-16T20:36:59.673657+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4288}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4288': {'gid': 4288, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.191:6835/3059699574', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 3059699574}, {'type': 'v1', 'addr': '172.21.15.191:6835', 'nonce': 3059699574}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T20:37:02.648 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4288:up:creating 2023-10-16T20:37:03.649 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:37:03.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.792+0000 7fc9cbc54640 1 Processor -- start 2023-10-16T20:37:03.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.792+0000 7fc9cbc54640 1 -- start start 2023-10-16T20:37:03.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.792+0000 7fc9cbc54640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9c4153670 0x7fc9c406a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:03.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.792+0000 7fc9cbc54640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c40c5e80 con 0x7fc9c406f810 2023-10-16T20:37:03.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.792+0000 7fc9cbc54640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c415cb30 con 0x7fc9c40c58d0 2023-10-16T20:37:03.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.792+0000 7fc9cbc54640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c40c6860 con 0x7fc9c4153670 2023-10-16T20:37:03.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.792+0000 7fc9ca1f1640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc9c406f810 0x7fc9c406fc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60886/0 (socket says 172.21.15.88:60886) 2023-10-16T20:37:03.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.792+0000 7fc9ca1f1640 1 -- 172.21.15.88:0/4152393184 learned_addr learned my addr 172.21.15.88:0/4152393184 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:37:03.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.796+0000 7fc9c99f0640 1 --2- 172.21.15.88:0/4152393184 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9c4153670 0x7fc9c406a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:03.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.796+0000 7fc9c89ee640 1 -- 172.21.15.88:0/4152393184 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575393105 0 0) 0x7fc9bc0014a0 con 0x7fc9c406f810 2023-10-16T20:37:03.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.796+0000 7fc9c89ee640 1 -- 172.21.15.88:0/4152393184 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9b0003530 con 0x7fc9c406f810 2023-10-16T20:37:03.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.796+0000 7fc9c89ee640 1 -- 172.21.15.88:0/4152393184 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286699972 0 0) 0x7fc9c415cb30 con 0x7fc9c40c58d0 2023-10-16T20:37:03.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.796+0000 7fc9c89ee640 1 -- 172.21.15.88:0/4152393184 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9bc0014a0 con 0x7fc9c40c58d0 2023-10-16T20:37:03.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.796+0000 7fc9c99f0640 1 -- 172.21.15.88:0/4152393184 >> v1:172.21.15.191:6789/0 conn(0x7fc9c40c58d0 legacy=0x7fc9c406d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:03.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.796+0000 7fc9c99f0640 1 -- 172.21.15.88:0/4152393184 >> v1:172.21.15.88:6789/0 conn(0x7fc9c406f810 legacy=0x7fc9c406fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:03.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.796+0000 7fc9c99f0640 1 -- 172.21.15.88:0/4152393184 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c415ce10 con 0x7fc9c4153670 2023-10-16T20:37:03.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.796+0000 7fc9c99f0640 1 --2- 172.21.15.88:0/4152393184 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9c4153670 0x7fc9c406a8a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc9c00097f0 tx=0x7fc9c001c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c15d472e20667421 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:03.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.796+0000 7fc9c89ee640 1 -- 172.21.15.88:0/4152393184 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c00ab600 con 0x7fc9c4153670 2023-10-16T20:37:03.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.796+0000 7fc9c89ee640 1 -- 172.21.15.88:0/4152393184 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c00a7050 con 0x7fc9c4153670 2023-10-16T20:37:03.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9c89ee640 1 -- 172.21.15.88:0/4152393184 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c00ab980 con 0x7fc9c4153670 2023-10-16T20:37:03.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9cbc54640 1 -- 172.21.15.88:0/4152393184 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9c4153670 msgr2=0x7fc9c406a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:03.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9cbc54640 1 --2- 172.21.15.88:0/4152393184 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9c4153670 0x7fc9c406a8a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc9c00097f0 tx=0x7fc9c001c4c0 comp rx=0 tx=0).stop 2023-10-16T20:37:03.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9cbc54640 1 -- 172.21.15.88:0/4152393184 shutdown_connections 2023-10-16T20:37:03.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9cbc54640 1 --2- 172.21.15.88:0/4152393184 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9c4153670 0x7fc9c406a8a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:03.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9cbc54640 1 -- 172.21.15.88:0/4152393184 >> 172.21.15.88:0/4152393184 conn(0x7fc9c40cad50 msgr2=0x7fc9c40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:37:03.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9cbc54640 1 -- 172.21.15.88:0/4152393184 shutdown_connections 2023-10-16T20:37:03.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9cbc54640 1 -- 172.21.15.88:0/4152393184 wait complete. 2023-10-16T20:37:03.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9cbc54640 1 Processor -- start 2023-10-16T20:37:03.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9cbc54640 1 -- start start 2023-10-16T20:37:03.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9cbc54640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9c406f810 0x7fc9c41f17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:03.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9cbc54640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9c40c58d0 0x7fc9c406d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:03.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9cbc54640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9c4153670 0x7fc9c41456f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:03.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9cbc54640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c4148360 con 0x7fc9c40c58d0 2023-10-16T20:37:03.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9cbc54640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c41491a0 con 0x7fc9c4153670 2023-10-16T20:37:03.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9cbc54640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c4148590 con 0x7fc9c406f810 2023-10-16T20:37:03.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9c91ef640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9c40c58d0 0x7fc9c406d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:03.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9c91ef640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9c40c58d0 0x7fc9c406d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34458/0 (socket says 172.21.15.88:34458) 2023-10-16T20:37:03.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9c91ef640 1 -- 172.21.15.88:0/1201368098 learned_addr learned my addr 172.21.15.88:0/1201368098 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:37:03.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9c99f0640 1 --2- 172.21.15.88:0/1201368098 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9c406f810 0x7fc9c41f17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:03.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.800+0000 7fc9ca1f1640 1 --2- 172.21.15.88:0/1201368098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9c4153670 0x7fc9c41456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:03.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9c91ef640 1 -- 172.21.15.88:0/1201368098 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9c406f810 msgr2=0x7fc9c41f17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:03.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9c91ef640 1 --2- 172.21.15.88:0/1201368098 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9c406f810 0x7fc9c41f17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:03.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9c91ef640 1 -- 172.21.15.88:0/1201368098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9c4153670 msgr2=0x7fc9c41456f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:03.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9c91ef640 1 --2- 172.21.15.88:0/1201368098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9c4153670 0x7fc9c41456f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:03.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9c91ef640 1 -- 172.21.15.88:0/1201368098 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c0009420 con 0x7fc9c40c58d0 2023-10-16T20:37:03.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9ca1f1640 1 --2- 172.21.15.88:0/1201368098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9c4153670 0x7fc9c41456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:37:03.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9c99f0640 1 --2- 172.21.15.88:0/1201368098 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9c406f810 0x7fc9c41f17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:37:03.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9c91ef640 1 --2- 172.21.15.88:0/1201368098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9c40c58d0 0x7fc9c406d400 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc9b400c490 tx=0x7fc9b400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:03.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9ba7fc640 1 -- 172.21.15.88:0/1201368098 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b40052d0 con 0x7fc9c40c58d0 2023-10-16T20:37:03.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9ba7fc640 1 -- 172.21.15.88:0/1201368098 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b4005430 con 0x7fc9c40c58d0 2023-10-16T20:37:03.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9ba7fc640 1 -- 172.21.15.88:0/1201368098 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4012870 con 0x7fc9c40c58d0 2023-10-16T20:37:03.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9cbc54640 1 -- 172.21.15.88:0/1201368098 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c4148870 con 0x7fc9c40c58d0 2023-10-16T20:37:03.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9cbc54640 1 -- 172.21.15.88:0/1201368098 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c4148e30 con 0x7fc9c40c58d0 2023-10-16T20:37:03.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9cbc54640 1 -- 172.21.15.88:0/1201368098 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c40c1910 con 0x7fc9c40c58d0 2023-10-16T20:37:03.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9ba7fc640 1 -- 172.21.15.88:0/1201368098 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc9b40129d0 con 0x7fc9c40c58d0 2023-10-16T20:37:03.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9ba7fc640 1 --2- 172.21.15.88:0/1201368098 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc9a80664d0 0x7fc9a8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:03.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9ba7fc640 1 -- 172.21.15.88:0/1201368098 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc9b4083e20 con 0x7fc9c40c58d0 2023-10-16T20:37:03.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.804+0000 7fc9c99f0640 1 --2- 172.21.15.88:0/1201368098 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc9a80664d0 0x7fc9a8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:03.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.808+0000 7fc9c99f0640 1 --2- 172.21.15.88:0/1201368098 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc9a80664d0 0x7fc9a8068990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc9c00ad000 tx=0x7fc9c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:03.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.808+0000 7fc9ba7fc640 1 -- 172.21.15.88:0/1201368098 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc9b4050e80 con 0x7fc9c40c58d0 2023-10-16T20:37:03.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.992+0000 7fc9cbc54640 1 -- 172.21.15.88:0/1201368098 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9c4074700 con 0x7fc9c40c58d0 2023-10-16T20:37:03.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.992+0000 7fc9ba7fc640 1 -- 172.21.15.88:0/1201368098 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+6492 (secure 0 0 0) 0x7fc9b40544a0 con 0x7fc9c40c58d0 2023-10-16T20:37:03.997 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:37:03.997 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4275,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:37:01.681065+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-10-16T20:37:03.997 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 7 2023-10-16T20:37:03.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.992+0000 7fc9cbc54640 1 -- 172.21.15.88:0/1201368098 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc9a80664d0 msgr2=0x7fc9a8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:03.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.992+0000 7fc9cbc54640 1 --2- 172.21.15.88:0/1201368098 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc9a80664d0 0x7fc9a8068990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc9c00ad000 tx=0x7fc9c0008040 comp rx=0 tx=0).stop 2023-10-16T20:37:03.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.992+0000 7fc9cbc54640 1 -- 172.21.15.88:0/1201368098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9c40c58d0 msgr2=0x7fc9c406d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:03.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.992+0000 7fc9cbc54640 1 --2- 172.21.15.88:0/1201368098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9c40c58d0 0x7fc9c406d400 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc9b400c490 tx=0x7fc9b400c950 comp rx=0 tx=0).stop 2023-10-16T20:37:03.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.992+0000 7fc9cbc54640 1 -- 172.21.15.88:0/1201368098 shutdown_connections 2023-10-16T20:37:04.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.992+0000 7fc9cbc54640 1 --2- 172.21.15.88:0/1201368098 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc9a80664d0 0x7fc9a8068990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:04.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.992+0000 7fc9cbc54640 1 --2- 172.21.15.88:0/1201368098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9c4153670 0x7fc9c41456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:04.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.992+0000 7fc9cbc54640 1 --2- 172.21.15.88:0/1201368098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9c40c58d0 0x7fc9c406d400 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:04.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.992+0000 7fc9cbc54640 1 --2- 172.21.15.88:0/1201368098 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9c406f810 0x7fc9c41f17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:04.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.992+0000 7fc9cbc54640 1 -- 172.21.15.88:0/1201368098 >> 172.21.15.88:0/1201368098 conn(0x7fc9c40cad50 msgr2=0x7fc9c406fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:37:04.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.992+0000 7fc9cbc54640 1 -- 172.21.15.88:0/1201368098 shutdown_connections 2023-10-16T20:37:04.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:03.992+0000 7fc9cbc54640 1 -- 172.21.15.88:0/1201368098 wait complete. 2023-10-16T20:37:04.620 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T20:36:55.111692+0000', 'modified': '2023-10-16T20:37:01.681065+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4288}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4288': {'gid': 4288, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.191:6835/3059699574', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 3059699574}, {'type': 'v1', 'addr': '172.21.15.191:6835', 'nonce': 3059699574}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-10-16T20:37:04.621 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T20:37:04.621 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-10-16T20:37:04.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.756+0000 7f7b179a1640 1 Processor -- start 2023-10-16T20:37:04.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.756+0000 7f7b179a1640 1 -- start start 2023-10-16T20:37:04.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.760+0000 7f7b179a1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7b10153780 0x7f7b100c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:04.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.760+0000 7f7b179a1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b10159d90 con 0x7f7b10159490 2023-10-16T20:37:04.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.760+0000 7f7b179a1640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b1015a070 con 0x7f7b101598c0 2023-10-16T20:37:04.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.760+0000 7f7b179a1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b1015a350 con 0x7f7b10153780 2023-10-16T20:37:04.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.760+0000 7f7b1573d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7b10153780 0x7f7b100c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:04.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.760+0000 7f7b1573d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7b10153780 0x7f7b100c9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34470/0 (socket says 172.21.15.88:34470) 2023-10-16T20:37:04.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.760+0000 7f7b1573d640 1 -- 172.21.15.88:0/1666742336 learned_addr learned my addr 172.21.15.88:0/1666742336 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:37:04.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.760+0000 7f7b14f3c640 1 --1- 172.21.15.88:0/1666742336 >> v1:172.21.15.88:6789/0 conn(0x7f7b10159490 0x7f7b100c53f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60898/0 (socket says 172.21.15.88:60898) 2023-10-16T20:37:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.760+0000 7f7b1573d640 1 -- 172.21.15.88:0/1666742336 >> v1:172.21.15.191:6789/0 conn(0x7f7b101598c0 legacy=0x7f7b100ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:37:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.760+0000 7f7b1573d640 1 -- 172.21.15.88:0/1666742336 >> v1:172.21.15.88:6789/0 conn(0x7f7b10159490 legacy=0x7f7b100c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.760+0000 7f7b1573d640 1 -- 172.21.15.88:0/1666742336 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b100c4910 con 0x7f7b10153780 2023-10-16T20:37:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.760+0000 7f7b07fff640 1 -- 172.21.15.88:0/1666742336 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636297762 0 0) 0x7f7b10159d90 con 0x7f7b10159490 2023-10-16T20:37:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.764+0000 7f7b1573d640 1 --2- 172.21.15.88:0/1666742336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b10153780 0x7f7b100c9be0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c006fd0 tx=0x7f7b0c01ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0315c78d099b9f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.764+0000 7f7b07fff640 1 -- 172.21.15.88:0/1666742336 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c0a89d0 con 0x7f7b10153780 2023-10-16T20:37:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.764+0000 7f7b07fff640 1 -- 172.21.15.88:0/1666742336 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0c0a8b30 con 0x7f7b10153780 2023-10-16T20:37:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.764+0000 7f7b07fff640 1 -- 172.21.15.88:0/1666742336 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c0b0440 con 0x7f7b10153780 2023-10-16T20:37:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.764+0000 7f7b179a1640 1 -- 172.21.15.88:0/1666742336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b10153780 msgr2=0x7f7b100c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.764+0000 7f7b179a1640 1 --2- 172.21.15.88:0/1666742336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b10153780 0x7f7b100c9be0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c006fd0 tx=0x7f7b0c01ecd0 comp rx=0 tx=0).stop 2023-10-16T20:37:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.764+0000 7f7b179a1640 1 -- 172.21.15.88:0/1666742336 shutdown_connections 2023-10-16T20:37:04.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.764+0000 7f7b179a1640 1 --2- 172.21.15.88:0/1666742336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b10153780 0x7f7b100c9be0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:04.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.764+0000 7f7b179a1640 1 -- 172.21.15.88:0/1666742336 >> 172.21.15.88:0/1666742336 conn(0x7f7b1006bcb0 msgr2=0x7f7b1006c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:37:04.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.764+0000 7f7b179a1640 1 -- 172.21.15.88:0/1666742336 shutdown_connections 2023-10-16T20:37:04.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.764+0000 7f7b179a1640 1 -- 172.21.15.88:0/1666742336 wait complete. 2023-10-16T20:37:04.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.764+0000 7f7b179a1640 1 Processor -- start 2023-10-16T20:37:04.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.764+0000 7f7b179a1640 1 -- start start 2023-10-16T20:37:04.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b179a1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b10153780 0x7f7b10152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b179a1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b10159490 0x7f7b101f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b179a1640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b101598c0 0x7f7b101f2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b179a1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b10154920 con 0x7f7b10153780 2023-10-16T20:37:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b179a1640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b10158530 con 0x7f7b10159490 2023-10-16T20:37:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b179a1640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b100ce320 con 0x7f7b101598c0 2023-10-16T20:37:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b1573d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b10153780 0x7f7b10152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b1573d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b10153780 0x7f7b10152960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34472/0 (socket says 172.21.15.88:34472) 2023-10-16T20:37:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b1573d640 1 -- 172.21.15.88:0/592169478 learned_addr learned my addr 172.21.15.88:0/592169478 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:37:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b1573d640 1 -- 172.21.15.88:0/592169478 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b101598c0 msgr2=0x7f7b101f2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b1573d640 1 --2- 172.21.15.88:0/592169478 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b101598c0 0x7f7b101f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b1573d640 1 -- 172.21.15.88:0/592169478 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b10159490 msgr2=0x7f7b101f2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b1573d640 1 --2- 172.21.15.88:0/592169478 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b10159490 0x7f7b101f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b1573d640 1 -- 172.21.15.88:0/592169478 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0c009480 con 0x7f7b10153780 2023-10-16T20:37:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b1573d640 1 --2- 172.21.15.88:0/592169478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b10153780 0x7f7b10152960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c01b8e0 tx=0x7f7b0c01ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b05ffb640 1 -- 172.21.15.88:0/592169478 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c0bf070 con 0x7f7b10153780 2023-10-16T20:37:04.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b05ffb640 1 -- 172.21.15.88:0/592169478 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0c0a9e70 con 0x7f7b10153780 2023-10-16T20:37:04.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b05ffb640 1 -- 172.21.15.88:0/592169478 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c003820 con 0x7f7b10153780 2023-10-16T20:37:04.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b179a1640 1 -- 172.21.15.88:0/592169478 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b101f1b60 con 0x7f7b10153780 2023-10-16T20:37:04.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b179a1640 1 -- 172.21.15.88:0/592169478 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b10154080 con 0x7f7b10153780 2023-10-16T20:37:04.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.768+0000 7f7b05ffb640 1 -- 172.21.15.88:0/592169478 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7b0c0b7070 con 0x7f7b10153780 2023-10-16T20:37:04.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b05ffb640 1 --2- 172.21.15.88:0/592169478 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7af40664d0 0x7f7af4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:04.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b05ffb640 1 -- 172.21.15.88:0/592169478 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7b0c121fe0 con 0x7f7b10153780 2023-10-16T20:37:04.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b14f3c640 1 --2- 172.21.15.88:0/592169478 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7af40664d0 0x7f7af4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:04.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b1ad1f000 1 Processor -- start 2023-10-16T20:37:04.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b1ad1f000 1 -- start start 2023-10-16T20:37:04.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b14f3c640 1 --2- 172.21.15.88:0/592169478 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7af40664d0 0x7f7af4068990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7b000011f0 tx=0x7f7b00007930 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:04.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b1ad1f000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b27f810 0x555d1b27fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:04.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b1ad1f000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b280130 0x555d1b280510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:04.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b1ad1f000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280a50 0x555d1b28aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:04.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b1ad1f000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1b033cc0 con 0x555d1b280a50 2023-10-16T20:37:04.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b1ad1f000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1b096920 con 0x555d1b27f810 2023-10-16T20:37:04.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b1ad1f000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x555d1b093e50 con 0x555d1b280130 2023-10-16T20:37:04.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b15f3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280a50 0x555d1b28aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:04.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b1573d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b27f810 0x555d1b27fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:04.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b15f3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280a50 0x555d1b28aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34488/0 (socket says 172.21.15.88:34488) 2023-10-16T20:37:04.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b15f3e640 1 -- 172.21.15.88:0/2583224226 learned_addr learned my addr 172.21.15.88:0/2583224226 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:37:04.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b14f3c640 1 --2- 172.21.15.88:0/2583224226 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b280130 0x555d1b280510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:04.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b15f3e640 1 -- 172.21.15.88:0/2583224226 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b280130 msgr2=0x555d1b280510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:37:04.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b15f3e640 1 --2- 172.21.15.88:0/2583224226 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b280130 0x555d1b280510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:04.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b15f3e640 1 -- 172.21.15.88:0/2583224226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b27f810 msgr2=0x555d1b27fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:37:04.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b15f3e640 1 --2- 172.21.15.88:0/2583224226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b27f810 0x555d1b27fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:04.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b15f3e640 1 -- 172.21.15.88:0/2583224226 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d1ad88890 con 0x555d1b280a50 2023-10-16T20:37:04.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b1573d640 1 --2- 172.21.15.88:0/2583224226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b27f810 0x555d1b27fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:37:04.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7b15f3e640 1 --2- 172.21.15.88:0/2583224226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280a50 0x555d1b28aaa0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009d00 tx=0x7f7b0800b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26935fa3108ab8fc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:04.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7af2ffd640 1 -- 172.21.15.88:0/2583224226 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08016070 con 0x555d1b280a50 2023-10-16T20:37:04.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7af2ffd640 1 -- 172.21.15.88:0/2583224226 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0800bd40 con 0x555d1b280a50 2023-10-16T20:37:04.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.772+0000 7f7af2ffd640 1 -- 172.21.15.88:0/2583224226 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0800e070 con 0x555d1b280a50 2023-10-16T20:37:04.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/2583224226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280a50 msgr2=0x555d1b28aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:04.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 --2- 172.21.15.88:0/2583224226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280a50 0x555d1b28aaa0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7b08009d00 tx=0x7f7b0800b420 comp rx=0 tx=0).stop 2023-10-16T20:37:04.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/2583224226 shutdown_connections 2023-10-16T20:37:04.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 --2- 172.21.15.88:0/2583224226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280a50 0x555d1b28aaa0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:04.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 --2- 172.21.15.88:0/2583224226 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b280130 0x555d1b280510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:04.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 --2- 172.21.15.88:0/2583224226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b27f810 0x555d1b27fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:04.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/2583224226 >> 172.21.15.88:0/2583224226 conn(0x555d1b235fc0 msgr2=0x555d1b27d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:37:04.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/2583224226 shutdown_connections 2023-10-16T20:37:04.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/2583224226 wait complete. 2023-10-16T20:37:04.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 -- start start 2023-10-16T20:37:04.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 Processor -- start 2023-10-16T20:37:04.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b27f810 0x555d1b279f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:04.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 0x555d1b27a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:04.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b280a50 0x555d1b285c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:04.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1b08f740 con 0x555d1b280130 2023-10-16T20:37:04.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1b04ed10 con 0x555d1b280a50 2023-10-16T20:37:04.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x555d1b04e9b0 con 0x555d1b27f810 2023-10-16T20:37:04.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 10 client.0 _ll_register_callbacks cb 0x555d1b22a390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T20:37:04.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b14f3c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 0x555d1b27a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:04.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b14f3c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 0x555d1b27a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34498/0 (socket says 172.21.15.88:34498) 2023-10-16T20:37:04.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b14f3c640 1 -- 172.21.15.88:0/2790672348 learned_addr learned my addr 172.21.15.88:0/2790672348 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:37:04.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b15f3e640 1 --2- 172.21.15.88:0/2790672348 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b280a50 0x555d1b285c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:04.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1573d640 1 --2- 172.21.15.88:0/2790672348 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b27f810 0x555d1b279f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:04.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b14f3c640 1 -- 172.21.15.88:0/2790672348 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b27f810 msgr2=0x555d1b279f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:37:04.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b14f3c640 1 --2- 172.21.15.88:0/2790672348 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b27f810 0x555d1b279f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:04.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b14f3c640 1 -- 172.21.15.88:0/2790672348 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b280a50 msgr2=0x555d1b285c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:37:04.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b14f3c640 1 --2- 172.21.15.88:0/2790672348 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b280a50 0x555d1b285c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:04.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b14f3c640 1 -- 172.21.15.88:0/2790672348 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b10154080 con 0x555d1b280130 2023-10-16T20:37:04.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b14f3c640 1 --2- 172.21.15.88:0/2790672348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 0x555d1b27a490 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7b00005db0 tx=0x7f7b00001090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ef7f4e8562af97 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:04.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7aebfff640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-10-16T20:37:04.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7aebfff640 1 -- 172.21.15.88:0/2790672348 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b00016b50 con 0x555d1b280130 2023-10-16T20:37:04.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/2790672348 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555d1b043370 con 0x555d1b280130 2023-10-16T20:37:04.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7aebfff640 1 -- 172.21.15.88:0/2790672348 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(27..27 src has 1..27) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7b00023db0 con 0x555d1b280130 2023-10-16T20:37:04.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7aebfff640 1 -- 172.21.15.88:0/2790672348 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b000076b0 con 0x555d1b280130 2023-10-16T20:37:04.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 10 client.4437 fetch_fsmap learned FSMap version 7 2023-10-16T20:37:04.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.776+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/2790672348 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x7f7b080099b0 con 0x555d1b280130 2023-10-16T20:37:04.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.780+0000 7f7aebfff640 1 -- 172.21.15.88:0/2790672348 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555d1b043370 con 0x555d1b280130 2023-10-16T20:37:04.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.780+0000 7f7aebfff640 1 -- 172.21.15.88:0/2790672348 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 7) v1 ==== 4452+0+0 (secure 0 0 0) 0x7f7b0001f070 con 0x555d1b280130 2023-10-16T20:37:04.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.780+0000 7f7b1ad1f000 10 client.4437 fetch_fsmap finished waiting for FSMap version 7 2023-10-16T20:37:04.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.780+0000 7f7b1ad1f000 10 client.4437 resolve_mds: resolved name 'f' to daemon mds.f 2023-10-16T20:37:04.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.780+0000 7f7b1ad1f000 20 client.4437 populate_metadata read hostname 'smithi088' 2023-10-16T20:37:04.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.780+0000 7f7b1ad1f000 1 --2- 172.21.15.88:0/2790672348 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x555d1b286910 0x555d1b286cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:04.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.780+0000 7f7b1ad1f000 4 client.4437 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T20:37:04.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.780+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/2790672348 --> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d1b0292d0 con 0x555d1b286910 2023-10-16T20:37:04.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.780+0000 7f7b1573d640 1 --2- 172.21.15.88:0/2790672348 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x555d1b286910 0x555d1b286cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:04.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.780+0000 7f7aebfff640 10 client.4437 ms_handle_connect on v2:172.21.15.191:6834/3059699574 2023-10-16T20:37:04.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:04.780+0000 7f7b1573d640 1 --2- 172.21.15.88:0/2790672348 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x555d1b286910 0x555d1b286cf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edaf1dbdd0da98dd server_cookie=47c2de5e761ff306 in_seq=0 out_seq=0 2023-10-16T20:37:07.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.312+0000 7f7aebfff640 1 -- 172.21.15.88:0/2790672348 <== mds.0 v2:172.21.15.191:6834/3059699574 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555d1b0292d0 con 0x555d1b286910 2023-10-16T20:37:07.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.312+0000 7f7aebfff640 10 client.4437 handle_command_reply: tid=0 2023-10-16T20:37:07.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.312+0000 7f7b1ad1f000 1 client.4437 shutdown 2023-10-16T20:37:07.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.312+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/2790672348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 msgr2=0x555d1b27a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:07.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.312+0000 7f7b1ad1f000 1 --2- 172.21.15.88:0/2790672348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 0x555d1b27a490 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7b00005db0 tx=0x7f7b00001090 comp rx=0 tx=0).stop 2023-10-16T20:37:07.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.312+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/2790672348 shutdown_connections 2023-10-16T20:37:07.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.312+0000 7f7b1ad1f000 1 --2- 172.21.15.88:0/2790672348 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b280a50 0x555d1b285c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:07.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.312+0000 7f7b1ad1f000 1 --2- 172.21.15.88:0/2790672348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 0x555d1b27a490 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:07.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.312+0000 7f7b1ad1f000 1 --2- 172.21.15.88:0/2790672348 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x555d1b286910 0x555d1b286cf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:07.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.312+0000 7f7b1ad1f000 1 --2- 172.21.15.88:0/2790672348 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b27f810 0x555d1b279f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:07.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.312+0000 7f7aebfff640 0 client.4437 ms_handle_reset on v2:172.21.15.191:6834/3059699574 2023-10-16T20:37:07.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.312+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/2790672348 >> 172.21.15.88:0/2790672348 conn(0x555d1b235fc0 msgr2=0x555d1b27f090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:37:07.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.312+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/2790672348 shutdown_connections 2023-10-16T20:37:07.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.312+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/2790672348 wait complete. 2023-10-16T20:37:07.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.312+0000 7f7b1ad1f000 20 client.4437 trim_cache size 0 max 16384 2023-10-16T20:37:07.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b1ad1f000 1 Processor -- start 2023-10-16T20:37:07.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b1ad1f000 1 -- start start 2023-10-16T20:37:07.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b1ad1f000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b27f810 0x555d1b243c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:07.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b1ad1f000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 0x555d1b244180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:07.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b1ad1f000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b280a50 0x555d1b2834e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:07.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b1ad1f000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1ad49940 con 0x555d1b280130 2023-10-16T20:37:07.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b1ad1f000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1ad891b0 con 0x555d1b280a50 2023-10-16T20:37:07.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b1ad1f000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x555d1b08cb40 con 0x555d1b27f810 2023-10-16T20:37:07.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b14f3c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 0x555d1b244180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:07.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b14f3c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 0x555d1b244180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34506/0 (socket says 172.21.15.88:34506) 2023-10-16T20:37:07.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b14f3c640 1 -- 172.21.15.88:0/261118354 learned_addr learned my addr 172.21.15.88:0/261118354 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:37:07.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b15f3e640 1 --2- 172.21.15.88:0/261118354 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b280a50 0x555d1b2834e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:07.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b1573d640 1 --2- 172.21.15.88:0/261118354 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b27f810 0x555d1b243c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:07.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b14f3c640 1 -- 172.21.15.88:0/261118354 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b27f810 msgr2=0x555d1b243c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:37:07.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b14f3c640 1 --2- 172.21.15.88:0/261118354 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b27f810 0x555d1b243c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:07.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b14f3c640 1 -- 172.21.15.88:0/261118354 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b280a50 msgr2=0x555d1b2834e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:37:07.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b14f3c640 1 --2- 172.21.15.88:0/261118354 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b280a50 0x555d1b2834e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:07.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b14f3c640 1 -- 172.21.15.88:0/261118354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d1b051460 con 0x555d1b280130 2023-10-16T20:37:07.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.328+0000 7f7b1573d640 1 --2- 172.21.15.88:0/261118354 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b27f810 0x555d1b243c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:37:07.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b14f3c640 1 --2- 172.21.15.88:0/261118354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 0x555d1b244180 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7b00007e10 tx=0x7f7b0001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bcc0023ad0c3da3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:07.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7af37fe640 1 -- 172.21.15.88:0/261118354 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b00007600 con 0x555d1b280130 2023-10-16T20:37:07.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7af37fe640 1 -- 172.21.15.88:0/261118354 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0001c040 con 0x555d1b280130 2023-10-16T20:37:07.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7af37fe640 1 -- 172.21.15.88:0/261118354 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b00016820 con 0x555d1b280130 2023-10-16T20:37:07.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/261118354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 msgr2=0x555d1b244180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:37:07.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 1 --2- 172.21.15.88:0/261118354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 0x555d1b244180 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7b00007e10 tx=0x7f7b0001a000 comp rx=0 tx=0).stop 2023-10-16T20:37:07.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/261118354 shutdown_connections 2023-10-16T20:37:07.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 1 --2- 172.21.15.88:0/261118354 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b280a50 0x555d1b2834e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:07.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 1 --2- 172.21.15.88:0/261118354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 0x555d1b244180 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:07.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 1 --2- 172.21.15.88:0/261118354 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b27f810 0x555d1b243c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:07.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/261118354 >> 172.21.15.88:0/261118354 conn(0x555d1b235fc0 msgr2=0x555d1b287f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:37:07.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/261118354 shutdown_connections 2023-10-16T20:37:07.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/261118354 wait complete. 2023-10-16T20:37:07.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 1 -- start start 2023-10-16T20:37:07.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 1 Processor -- start 2023-10-16T20:37:07.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b27f810 0x555d1b2e29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:07.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 0x555d1b2e2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:07.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b280a50 0x555d1b2a2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:07.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1b0d0ba0 con 0x555d1b280130 2023-10-16T20:37:07.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1b08e5d0 con 0x555d1b280a50 2023-10-16T20:37:07.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x555d1b093740 con 0x555d1b27f810 2023-10-16T20:37:07.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1ad1f000 10 client.0 _ll_register_callbacks cb 0x7f7b08009510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T20:37:07.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b14f3c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 0x555d1b2e2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:07.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b14f3c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 0x555d1b2e2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34510/0 (socket says 172.21.15.88:34510) 2023-10-16T20:37:07.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b14f3c640 1 -- 172.21.15.88:0/487865633 learned_addr learned my addr 172.21.15.88:0/487865633 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:37:07.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b1573d640 1 --2- 172.21.15.88:0/487865633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b27f810 0x555d1b2e29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:07.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.332+0000 7f7b15f3e640 1 --2- 172.21.15.88:0/487865633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b280a50 0x555d1b2a2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:07.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b14f3c640 1 -- 172.21.15.88:0/487865633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b27f810 msgr2=0x555d1b2e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:37:07.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b14f3c640 1 --2- 172.21.15.88:0/487865633 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x555d1b27f810 0x555d1b2e29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:07.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b14f3c640 1 -- 172.21.15.88:0/487865633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b280a50 msgr2=0x555d1b2a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:37:07.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b14f3c640 1 --2- 172.21.15.88:0/487865633 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x555d1b280a50 0x555d1b2a2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:37:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b14f3c640 1 -- 172.21.15.88:0/487865633 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b080099b0 con 0x555d1b280130 2023-10-16T20:37:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b14f3c640 1 --2- 172.21.15.88:0/487865633 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d1b280130 0x555d1b2e2ee0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7b00033040 tx=0x7f7b00011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e4956e6c8b679c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:37:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7aebfff640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-10-16T20:37:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7aebfff640 1 -- 172.21.15.88:0/487865633 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b00016c30 con 0x555d1b280130 2023-10-16T20:37:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7aebfff640 1 -- 172.21.15.88:0/487865633 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(27..27 src has 1..27) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7b0003f470 con 0x555d1b280130 2023-10-16T20:37:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/487865633 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555d1b0276f0 con 0x555d1b280130 2023-10-16T20:37:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7aebfff640 1 -- 172.21.15.88:0/487865633 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b0003f630 con 0x555d1b280130 2023-10-16T20:37:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1ad1f000 10 client.4443 fetch_fsmap learned FSMap version 7 2023-10-16T20:37:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/487865633 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x555d1b037760 con 0x555d1b280130 2023-10-16T20:37:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7aebfff640 1 -- 172.21.15.88:0/487865633 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555d1b0276f0 con 0x555d1b280130 2023-10-16T20:37:07.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7aebfff640 1 -- 172.21.15.88:0/487865633 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 7) v1 ==== 4452+0+0 (secure 0 0 0) 0x7f7b0001f070 con 0x555d1b280130 2023-10-16T20:37:07.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1ad1f000 10 client.4443 fetch_fsmap finished waiting for FSMap version 7 2023-10-16T20:37:07.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1ad1f000 10 client.4443 resolve_mds: resolved name 'f' to daemon mds.f 2023-10-16T20:37:07.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1ad1f000 20 client.4443 populate_metadata read hostname 'smithi088' 2023-10-16T20:37:07.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1ad1f000 1 --2- 172.21.15.88:0/487865633 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x555d1b2a30f0 0x555d1b2a34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:37:07.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1ad1f000 4 client.4443 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T20:37:07.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1ad1f000 1 -- 172.21.15.88:0/487865633 --> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555d1b088720 con 0x555d1b2a30f0 2023-10-16T20:37:07.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1573d640 1 --2- 172.21.15.88:0/487865633 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x555d1b2a30f0 0x555d1b2a34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:07.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1573d640 1 --2- 172.21.15.88:0/487865633 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x555d1b2a30f0 0x555d1b2a34d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18c7c4b563681809 server_cookie=decbf60fc5e2c5a6 in_seq=0 out_seq=0 2023-10-16T20:37:07.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7aebfff640 10 client.4443 ms_handle_connect on v2:172.21.15.191:6834/3059699574 2023-10-16T20:37:07.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1573d640 1 -- 172.21.15.88:0/487865633 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x555d1b2a30f0 msgr2=0x555d1b2a34d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2023-10-16T20:37:07.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1573d640 1 -- 172.21.15.88:0/487865633 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x555d1b2a30f0 msgr2=0x555d1b2a34d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-16T20:37:07.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1573d640 1 --2- 172.21.15.88:0/487865633 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x555d1b2a30f0 0x555d1b2a34d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-16T20:37:07.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1573d640 1 --2- 172.21.15.88:0/487865633 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x555d1b2a30f0 0x555d1b2a34d0 unknown :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2023-10-16T20:37:07.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1573d640 1 --2- 172.21.15.88:0/487865633 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x555d1b2a30f0 0x555d1b2a34d0 unknown :-1 s=BANNER_CONNECTING pgs=14 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:37:07.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1573d640 1 --2- 172.21.15.88:0/487865633 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x555d1b2a30f0 0x555d1b2a34d0 crc :-1 s=SESSION_RECONNECTING pgs=14 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_session_reset received session reset full=1 2023-10-16T20:37:07.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1573d640 1 --2- 172.21.15.88:0/487865633 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x555d1b2a30f0 0x555d1b2a34d0 crc :-1 s=SESSION_RECONNECTING pgs=14 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).reset_session 2023-10-16T20:37:07.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7aebfff640 0 client.4443 ms_handle_remote_reset on v2:172.21.15.191:6834/3059699574 2023-10-16T20:37:07.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7aebfff640 10 client.4443 ms_handle_connect on v2:172.21.15.191:6834/3059699574 2023-10-16T20:37:07.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:37:07.336+0000 7f7b1573d640 1 --2- 172.21.15.88:0/487865633 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x555d1b2a30f0 0x555d1b2a34d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea5368d8eac71c39 server_cookie=559560d0adba4dcd in_seq=0 out_seq=0 2023-10-16T20:37:09.922 INFO:tasks.ceph.mds.f.smithi191.stderr:2023-10-16T20:37:09.916+0000 7fe37f078640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T20:37:23.333 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:37:23.337 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:37:53.456 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:37:53.460 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:38:23.543 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:38:23.547 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:38:53.700 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:38:53.704 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:39:04.652 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-10-16T20:39:05.654 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:39:05.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.798+0000 7f3f86e79640 1 Processor -- start 2023-10-16T20:39:05.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.798+0000 7f3f86e79640 1 -- start start 2023-10-16T20:39:05.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.798+0000 7f3f86e79640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3f800c9bb0 0x7f3f800c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:05.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.798+0000 7f3f86e79640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f800c41f0 0x7f3f800cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:05.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.798+0000 7f3f86e79640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f80153b50 con 0x7f3f80153770 2023-10-16T20:39:05.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.798+0000 7f3f86e79640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f800cc250 con 0x7f3f800c9bb0 2023-10-16T20:39:05.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.798+0000 7f3f86e79640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f800cc670 con 0x7f3f800c41f0 2023-10-16T20:39:05.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.798+0000 7f3f77fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3f800c9bb0 0x7f3f800c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:05.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.798+0000 7f3f84c15640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3f80153770 0x7f3f80159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40942/0 (socket says 172.21.15.88:40942) 2023-10-16T20:39:05.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.798+0000 7f3f84c15640 1 -- 172.21.15.88:0/1388588603 learned_addr learned my addr 172.21.15.88:0/1388588603 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:05.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.798+0000 7f3f85416640 1 --2- 172.21.15.88:0/1388588603 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f800c41f0 0x7f3f800cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:05.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.798+0000 7f3f77fff640 1 -- 172.21.15.88:0/1388588603 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f800c41f0 msgr2=0x7f3f800cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:05.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.798+0000 7f3f77fff640 1 --2- 172.21.15.88:0/1388588603 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f800c41f0 0x7f3f800cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:05.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.798+0000 7f3f77fff640 1 -- 172.21.15.88:0/1388588603 >> v1:172.21.15.88:6789/0 conn(0x7f3f80153770 legacy=0x7f3f80159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:05.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.798+0000 7f3f777fe640 1 -- 172.21.15.88:0/1388588603 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224910196 0 0) 0x7f3f80153b50 con 0x7f3f80153770 2023-10-16T20:39:05.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.798+0000 7f3f77fff640 1 -- 172.21.15.88:0/1388588603 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f800cb2f0 con 0x7f3f800c9bb0 2023-10-16T20:39:05.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.802+0000 7f3f77fff640 1 --2- 172.21.15.88:0/1388588603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f800c9bb0 0x7f3f800c3cb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3f700097c0 tx=0x7f3f7001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e11eea1be4dfc74 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:05.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.802+0000 7f3f777fe640 1 -- 172.21.15.88:0/1388588603 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f700ad640 con 0x7f3f800c9bb0 2023-10-16T20:39:05.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.802+0000 7f3f777fe640 1 -- 172.21.15.88:0/1388588603 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f700ad7a0 con 0x7f3f800c9bb0 2023-10-16T20:39:05.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.802+0000 7f3f777fe640 1 -- 172.21.15.88:0/1388588603 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f700adb30 con 0x7f3f800c9bb0 2023-10-16T20:39:05.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.802+0000 7f3f86e79640 1 -- 172.21.15.88:0/1388588603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f800c9bb0 msgr2=0x7f3f800c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:05.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.802+0000 7f3f86e79640 1 --2- 172.21.15.88:0/1388588603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f800c9bb0 0x7f3f800c3cb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3f700097c0 tx=0x7f3f7001c480 comp rx=0 tx=0).stop 2023-10-16T20:39:05.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.802+0000 7f3f86e79640 1 -- 172.21.15.88:0/1388588603 shutdown_connections 2023-10-16T20:39:05.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.802+0000 7f3f86e79640 1 --2- 172.21.15.88:0/1388588603 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f800c41f0 0x7f3f800cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:05.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.802+0000 7f3f86e79640 1 --2- 172.21.15.88:0/1388588603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f800c9bb0 0x7f3f800c3cb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:05.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.802+0000 7f3f86e79640 1 -- 172.21.15.88:0/1388588603 >> 172.21.15.88:0/1388588603 conn(0x7f3f800715c0 msgr2=0x7f3f800719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:05.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.802+0000 7f3f86e79640 1 -- 172.21.15.88:0/1388588603 shutdown_connections 2023-10-16T20:39:05.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.802+0000 7f3f86e79640 1 -- 172.21.15.88:0/1388588603 wait complete. 2023-10-16T20:39:05.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f86e79640 1 Processor -- start 2023-10-16T20:39:05.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f86e79640 1 -- start start 2023-10-16T20:39:05.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f86e79640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f800c41f0 0x7f3f800b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:05.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f86e79640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f800c9bb0 0x7f3f801416b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:05.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f86e79640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f80153770 0x7f3f80141bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:05.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f86e79640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f801eb660 con 0x7f3f800c41f0 2023-10-16T20:39:05.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f86e79640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f801f3c80 con 0x7f3f80153770 2023-10-16T20:39:05.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f86e79640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f801f3070 con 0x7f3f800c9bb0 2023-10-16T20:39:05.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f84c15640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f800c41f0 0x7f3f800b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:05.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f84c15640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f800c41f0 0x7f3f800b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44878/0 (socket says 172.21.15.88:44878) 2023-10-16T20:39:05.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f84c15640 1 -- 172.21.15.88:0/2843336390 learned_addr learned my addr 172.21.15.88:0/2843336390 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:05.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f77fff640 1 --2- 172.21.15.88:0/2843336390 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f800c9bb0 0x7f3f801416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:05.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f85416640 1 --2- 172.21.15.88:0/2843336390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f80153770 0x7f3f80141bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:05.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f84c15640 1 -- 172.21.15.88:0/2843336390 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f800c9bb0 msgr2=0x7f3f801416b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:05.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f84c15640 1 --2- 172.21.15.88:0/2843336390 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f800c9bb0 0x7f3f801416b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:05.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f84c15640 1 -- 172.21.15.88:0/2843336390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f80153770 msgr2=0x7f3f80141bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:05.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f84c15640 1 --2- 172.21.15.88:0/2843336390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f80153770 0x7f3f80141bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:05.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f84c15640 1 -- 172.21.15.88:0/2843336390 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f70009420 con 0x7f3f800c41f0 2023-10-16T20:39:05.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f84c15640 1 --2- 172.21.15.88:0/2843336390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f800c41f0 0x7f3f800b3120 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c0010f0 tx=0x7f3f7c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:05.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f757fa640 1 -- 172.21.15.88:0/2843336390 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c0074e0 con 0x7f3f800c41f0 2023-10-16T20:39:05.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f757fa640 1 -- 172.21.15.88:0/2843336390 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7c007640 con 0x7f3f800c41f0 2023-10-16T20:39:05.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f86e79640 1 -- 172.21.15.88:0/2843336390 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f801f3350 con 0x7f3f800c41f0 2023-10-16T20:39:05.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f757fa640 1 -- 172.21.15.88:0/2843336390 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c0059d0 con 0x7f3f800c41f0 2023-10-16T20:39:05.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f86e79640 1 -- 172.21.15.88:0/2843336390 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f801f3910 con 0x7f3f800c41f0 2023-10-16T20:39:05.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.806+0000 7f3f86e79640 1 -- 172.21.15.88:0/2843336390 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f80074700 con 0x7f3f800c41f0 2023-10-16T20:39:05.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.810+0000 7f3f757fa640 1 -- 172.21.15.88:0/2843336390 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3f7c005b30 con 0x7f3f800c41f0 2023-10-16T20:39:05.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.810+0000 7f3f757fa640 1 --2- 172.21.15.88:0/2843336390 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3f6c0664a0 0x7f3f6c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:05.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.810+0000 7f3f757fa640 1 -- 172.21.15.88:0/2843336390 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3f7c083840 con 0x7f3f800c41f0 2023-10-16T20:39:05.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.810+0000 7f3f77fff640 1 --2- 172.21.15.88:0/2843336390 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3f6c0664a0 0x7f3f6c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:05.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.810+0000 7f3f77fff640 1 --2- 172.21.15.88:0/2843336390 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3f6c0664a0 0x7f3f6c068960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3f700af000 tx=0x7f3f700a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:05.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.814+0000 7f3f757fa640 1 -- 172.21.15.88:0/2843336390 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3f7c050910 con 0x7f3f800c41f0 2023-10-16T20:39:06.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.994+0000 7f3f86e79640 1 -- 172.21.15.88:0/2843336390 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f801570a0 con 0x7f3f800c41f0 2023-10-16T20:39:06.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.994+0000 7f3f757fa640 1 -- 172.21.15.88:0/2843336390 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+6492 (secure 0 0 0) 0x7f3f7c053f30 con 0x7f3f800c41f0 2023-10-16T20:39:06.002 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:06.002 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4275,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:37:01.681065+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-10-16T20:39:06.003 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 7 2023-10-16T20:39:06.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.998+0000 7f3f86e79640 1 -- 172.21.15.88:0/2843336390 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3f6c0664a0 msgr2=0x7f3f6c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:06.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.998+0000 7f3f86e79640 1 --2- 172.21.15.88:0/2843336390 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3f6c0664a0 0x7f3f6c068960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3f700af000 tx=0x7f3f700a7070 comp rx=0 tx=0).stop 2023-10-16T20:39:06.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.998+0000 7f3f86e79640 1 -- 172.21.15.88:0/2843336390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f800c41f0 msgr2=0x7f3f800b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:06.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.998+0000 7f3f86e79640 1 --2- 172.21.15.88:0/2843336390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f800c41f0 0x7f3f800b3120 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c0010f0 tx=0x7f3f7c00c8b0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.998+0000 7f3f86e79640 1 -- 172.21.15.88:0/2843336390 shutdown_connections 2023-10-16T20:39:06.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.998+0000 7f3f86e79640 1 --2- 172.21.15.88:0/2843336390 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3f6c0664a0 0x7f3f6c068960 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.998+0000 7f3f86e79640 1 --2- 172.21.15.88:0/2843336390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f80153770 0x7f3f80141bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.998+0000 7f3f86e79640 1 --2- 172.21.15.88:0/2843336390 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f800c9bb0 0x7f3f801416b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.998+0000 7f3f86e79640 1 --2- 172.21.15.88:0/2843336390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f800c41f0 0x7f3f800b3120 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.998+0000 7f3f86e79640 1 -- 172.21.15.88:0/2843336390 >> 172.21.15.88:0/2843336390 conn(0x7f3f800715c0 msgr2=0x7f3f80151ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:06.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.998+0000 7f3f86e79640 1 -- 172.21.15.88:0/2843336390 shutdown_connections 2023-10-16T20:39:06.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:05.998+0000 7f3f86e79640 1 -- 172.21.15.88:0/2843336390 wait complete. 2023-10-16T20:39:06.626 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T20:36:55.111692+0000', 'modified': '2023-10-16T20:37:01.681065+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4288}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4288': {'gid': 4288, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.191:6835/3059699574', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 3059699574}, {'type': 'v1', 'addr': '172.21.15.191:6835', 'nonce': 3059699574}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-10-16T20:39:06.626 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-16T20:39:06.627 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-10-16T20:39:06.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.766+0000 7effe14c9640 1 Processor -- start 2023-10-16T20:39:06.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.766+0000 7effe14c9640 1 -- start start 2023-10-16T20:39:06.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.766+0000 7effe14c9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7effdc159490 0x7effdc0c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.766+0000 7effe14c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effdc0c5930 0x7effdc0ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.766+0000 7effe14c9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effdc159d90 con 0x7effdc153780 2023-10-16T20:39:06.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.766+0000 7effe14c9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effdc15a070 con 0x7effdc159490 2023-10-16T20:39:06.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.766+0000 7effe14c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effdc15a350 con 0x7effdc0c5930 2023-10-16T20:39:06.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.766+0000 7effdb7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effdc0c5930 0x7effdc0ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.766+0000 7effdb7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effdc0c5930 0x7effdc0ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:36642/0 (socket says 172.21.15.88:36642) 2023-10-16T20:39:06.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.766+0000 7effdb7fe640 1 -- 172.21.15.88:0/1379885528 learned_addr learned my addr 172.21.15.88:0/1379885528 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:06.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.766+0000 7effd27fc640 1 --2- 172.21.15.88:0/1379885528 >> v2:172.21.15.191:3300/0 conn(0x7effdc159490 0x7effdc0c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.770+0000 7effda7fc640 1 -- 172.21.15.88:0/1379885528 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324151744 0 0) 0x7effdc159d90 con 0x7effdc153780 2023-10-16T20:39:06.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.770+0000 7effda7fc640 1 -- 172.21.15.88:0/1379885528 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc8003530 con 0x7effdc153780 2023-10-16T20:39:06.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.770+0000 7effd27fc640 1 -- 172.21.15.88:0/1379885528 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effdc0c5930 msgr2=0x7effdc0ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:06.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.770+0000 7effd27fc640 1 --2- 172.21.15.88:0/1379885528 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effdc0c5930 0x7effdc0ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.770+0000 7effd27fc640 1 -- 172.21.15.88:0/1379885528 >> v1:172.21.15.191:6789/0 conn(0x7effdc153780 legacy=0x7effdc0c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:06.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.770+0000 7effd27fc640 1 -- 172.21.15.88:0/1379885528 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc0c4910 con 0x7effdc159490 2023-10-16T20:39:06.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.770+0000 7effdb7fe640 1 --2- 172.21.15.88:0/1379885528 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effdc0c5930 0x7effdc0ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:39:06.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.770+0000 7effd27fc640 1 --2- 172.21.15.88:0/1379885528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effdc159490 0x7effdc0c53f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7effc4009d90 tx=0x7effc401f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=361067fa05f6479 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:06.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.770+0000 7effda7fc640 1 -- 172.21.15.88:0/1379885528 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc40af600 con 0x7effdc159490 2023-10-16T20:39:06.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.770+0000 7effda7fc640 1 -- 172.21.15.88:0/1379885528 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc40ab050 con 0x7effdc159490 2023-10-16T20:39:06.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.770+0000 7effda7fc640 1 -- 172.21.15.88:0/1379885528 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc40af980 con 0x7effdc159490 2023-10-16T20:39:06.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effe14c9640 1 -- 172.21.15.88:0/1379885528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effdc159490 msgr2=0x7effdc0c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:06.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effe14c9640 1 --2- 172.21.15.88:0/1379885528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effdc159490 0x7effdc0c53f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7effc4009d90 tx=0x7effc401f130 comp rx=0 tx=0).stop 2023-10-16T20:39:06.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effe14c9640 1 -- 172.21.15.88:0/1379885528 shutdown_connections 2023-10-16T20:39:06.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effe14c9640 1 --2- 172.21.15.88:0/1379885528 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effdc0c5930 0x7effdc0ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effe14c9640 1 --2- 172.21.15.88:0/1379885528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effdc159490 0x7effdc0c53f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effe14c9640 1 -- 172.21.15.88:0/1379885528 >> 172.21.15.88:0/1379885528 conn(0x7effdc06bcb0 msgr2=0x7effdc06c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:06.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effe14c9640 1 -- 172.21.15.88:0/1379885528 shutdown_connections 2023-10-16T20:39:06.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effe14c9640 1 -- 172.21.15.88:0/1379885528 wait complete. 2023-10-16T20:39:06.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effe14c9640 1 Processor -- start 2023-10-16T20:39:06.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effe14c9640 1 -- start start 2023-10-16T20:39:06.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effe14c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effdc0c5930 0x7effdc071820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effe14c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effdc153780 0x7effdc14c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effe14c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effdc159490 0x7effdc14c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effe14c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc147d60 con 0x7effdc0c5930 2023-10-16T20:39:06.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effe14c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc14d050 con 0x7effdc159490 2023-10-16T20:39:06.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effe14c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effdc1ef580 con 0x7effdc153780 2023-10-16T20:39:06.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effdaffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effdc0c5930 0x7effdc071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effdaffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effdc0c5930 0x7effdc071820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44888/0 (socket says 172.21.15.88:44888) 2023-10-16T20:39:06.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effdaffd640 1 -- 172.21.15.88:0/3147122038 learned_addr learned my addr 172.21.15.88:0/3147122038 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:06.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effd27fc640 1 --2- 172.21.15.88:0/3147122038 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effdc153780 0x7effdc14c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effdb7fe640 1 --2- 172.21.15.88:0/3147122038 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effdc159490 0x7effdc14c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effdaffd640 1 -- 172.21.15.88:0/3147122038 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effdc153780 msgr2=0x7effdc14c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:06.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effdaffd640 1 --2- 172.21.15.88:0/3147122038 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effdc153780 0x7effdc14c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effdaffd640 1 -- 172.21.15.88:0/3147122038 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effdc159490 msgr2=0x7effdc14c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:06.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effdaffd640 1 --2- 172.21.15.88:0/3147122038 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effdc159490 0x7effdc14c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effdaffd640 1 -- 172.21.15.88:0/3147122038 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd40094d0 con 0x7effdc0c5930 2023-10-16T20:39:06.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.774+0000 7effdaffd640 1 --2- 172.21.15.88:0/3147122038 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effdc0c5930 0x7effdc071820 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7effcc00c4d0 tx=0x7effcc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:06.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.778+0000 7effd3fff640 1 -- 172.21.15.88:0/3147122038 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc0052d0 con 0x7effdc0c5930 2023-10-16T20:39:06.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.778+0000 7effd3fff640 1 -- 172.21.15.88:0/3147122038 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effcc005430 con 0x7effdc0c5930 2023-10-16T20:39:06.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.778+0000 7effd3fff640 1 -- 172.21.15.88:0/3147122038 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc003850 con 0x7effdc0c5930 2023-10-16T20:39:06.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.778+0000 7effe14c9640 1 -- 172.21.15.88:0/3147122038 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc4009a70 con 0x7effdc0c5930 2023-10-16T20:39:06.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.778+0000 7effe14c9640 1 -- 172.21.15.88:0/3147122038 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effdc071d60 con 0x7effdc0c5930 2023-10-16T20:39:06.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.778+0000 7effd3fff640 1 -- 172.21.15.88:0/3147122038 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7effcc0039b0 con 0x7effdc0c5930 2023-10-16T20:39:06.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.778+0000 7effd3fff640 1 --2- 172.21.15.88:0/3147122038 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7effbc0664a0 0x7effbc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.778+0000 7effd3fff640 1 -- 172.21.15.88:0/3147122038 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7effcc083eb0 con 0x7effdc0c5930 2023-10-16T20:39:06.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.778+0000 7effd27fc640 1 --2- 172.21.15.88:0/3147122038 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7effbc0664a0 0x7effbc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.778+0000 7effe483b000 1 Processor -- start 2023-10-16T20:39:06.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.778+0000 7effe483b000 1 -- start start 2023-10-16T20:39:06.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effd27fc640 1 --2- 172.21.15.88:0/3147122038 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7effbc0664a0 0x7effbc068960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7effc40b1000 tx=0x7effc40aa000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:06.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effe483b000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4bf810 0x56048b4bfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effe483b000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4c0130 0x56048b4c0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effe483b000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4c0a50 0x56048b4caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effe483b000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048b273cc0 con 0x56048b4c0a50 2023-10-16T20:39:06.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effe483b000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048b2d6920 con 0x56048b4c0130 2023-10-16T20:39:06.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effe483b000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x56048b2d3e50 con 0x56048b4bf810 2023-10-16T20:39:06.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effdb7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4c0a50 0x56048b4caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effdb7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4c0a50 0x56048b4caaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44904/0 (socket says 172.21.15.88:44904) 2023-10-16T20:39:06.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effdb7fe640 1 -- 172.21.15.88:0/452531498 learned_addr learned my addr 172.21.15.88:0/452531498 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:06.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effdaffd640 1 --2- 172.21.15.88:0/452531498 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4bf810 0x56048b4bfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effd27fc640 1 --2- 172.21.15.88:0/452531498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4c0130 0x56048b4c0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effdb7fe640 1 -- 172.21.15.88:0/452531498 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4bf810 msgr2=0x56048b4bfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:06.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effdb7fe640 1 --2- 172.21.15.88:0/452531498 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4bf810 0x56048b4bfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effdb7fe640 1 -- 172.21.15.88:0/452531498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4c0130 msgr2=0x56048b4c0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:06.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effdb7fe640 1 --2- 172.21.15.88:0/452531498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4c0130 0x56048b4c0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effdb7fe640 1 -- 172.21.15.88:0/452531498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56048afc8890 con 0x56048b4c0a50 2023-10-16T20:39:06.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effd27fc640 1 --2- 172.21.15.88:0/452531498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4c0130 0x56048b4c0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:06.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effdb7fe640 1 --2- 172.21.15.88:0/452531498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4c0a50 0x56048b4caaa0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7effd4009c70 tx=0x7effd4009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3cd55db0c4dae85 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:06.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effd0ff9640 1 -- 172.21.15.88:0/452531498 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4011ed0 con 0x56048b4c0a50 2023-10-16T20:39:06.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effd0ff9640 1 -- 172.21.15.88:0/452531498 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd4009e80 con 0x56048b4c0a50 2023-10-16T20:39:06.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effd0ff9640 1 -- 172.21.15.88:0/452531498 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4015680 con 0x56048b4c0a50 2023-10-16T20:39:06.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effe483b000 1 -- 172.21.15.88:0/452531498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4c0a50 msgr2=0x56048b4caaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:06.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effe483b000 1 --2- 172.21.15.88:0/452531498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4c0a50 0x56048b4caaa0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7effd4009c70 tx=0x7effd4009240 comp rx=0 tx=0).stop 2023-10-16T20:39:06.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effe483b000 1 -- 172.21.15.88:0/452531498 shutdown_connections 2023-10-16T20:39:06.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effe483b000 1 --2- 172.21.15.88:0/452531498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4c0a50 0x56048b4caaa0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effe483b000 1 --2- 172.21.15.88:0/452531498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4c0130 0x56048b4c0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effe483b000 1 --2- 172.21.15.88:0/452531498 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4bf810 0x56048b4bfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effe483b000 1 -- 172.21.15.88:0/452531498 >> 172.21.15.88:0/452531498 conn(0x56048b475fc0 msgr2=0x56048b4bd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:06.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effe483b000 1 -- 172.21.15.88:0/452531498 shutdown_connections 2023-10-16T20:39:06.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effe483b000 1 -- 172.21.15.88:0/452531498 wait complete. 2023-10-16T20:39:06.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effe483b000 1 -- start start 2023-10-16T20:39:06.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.782+0000 7effe483b000 1 Processor -- start 2023-10-16T20:39:06.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effe483b000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4bf810 0x56048b4b9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effe483b000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4c0130 0x56048b4ba490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effe483b000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4c0a50 0x56048b4c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effe483b000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048b2cf740 con 0x56048b4bf810 2023-10-16T20:39:06.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effe483b000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048b28ed10 con 0x56048b4c0130 2023-10-16T20:39:06.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effe483b000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x56048b28e9b0 con 0x56048b4c0a50 2023-10-16T20:39:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effe483b000 10 client.0 _ll_register_callbacks cb 0x56048b46a390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T20:39:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effdaffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4bf810 0x56048b4b9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effdaffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4bf810 0x56048b4b9f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44912/0 (socket says 172.21.15.88:44912) 2023-10-16T20:39:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effdaffd640 1 -- 172.21.15.88:0/3803658531 learned_addr learned my addr 172.21.15.88:0/3803658531 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effd27fc640 1 --2- 172.21.15.88:0/3803658531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4c0130 0x56048b4ba490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effdaffd640 1 -- 172.21.15.88:0/3803658531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4c0a50 msgr2=0x56048b4c5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:06.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effdaffd640 1 --2- 172.21.15.88:0/3803658531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4c0a50 0x56048b4c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effdaffd640 1 -- 172.21.15.88:0/3803658531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4c0130 msgr2=0x56048b4ba490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:06.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effdaffd640 1 --2- 172.21.15.88:0/3803658531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4c0130 0x56048b4ba490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effdaffd640 1 -- 172.21.15.88:0/3803658531 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd4009950 con 0x56048b4bf810 2023-10-16T20:39:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effdaffd640 1 --2- 172.21.15.88:0/3803658531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4bf810 0x56048b4b9f50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7effcc054920 tx=0x7effcc054c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9913a6534e04180 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-10-16T20:39:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effb9ffb640 1 -- 172.21.15.88:0/3803658531 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc05cf20 con 0x56048b4bf810 2023-10-16T20:39:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effb9ffb640 1 -- 172.21.15.88:0/3803658531 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7effcc05ca90 con 0x56048b4bf810 2023-10-16T20:39:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effe483b000 1 -- 172.21.15.88:0/3803658531 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56048b283370 con 0x56048b4bf810 2023-10-16T20:39:06.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effb9ffb640 1 -- 172.21.15.88:0/3803658531 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7effcc05cc50 con 0x56048b4bf810 2023-10-16T20:39:06.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effe483b000 10 client.4461 fetch_fsmap learned FSMap version 7 2023-10-16T20:39:06.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effe483b000 1 -- 172.21.15.88:0/3803658531 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x56048b291460 con 0x56048b4bf810 2023-10-16T20:39:06.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effb9ffb640 1 -- 172.21.15.88:0/3803658531 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56048b283370 con 0x56048b4bf810 2023-10-16T20:39:06.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effb9ffb640 1 -- 172.21.15.88:0/3803658531 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 7) v1 ==== 4452+0+0 (secure 0 0 0) 0x7effcc019020 con 0x56048b4bf810 2023-10-16T20:39:06.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effe483b000 10 client.4461 fetch_fsmap finished waiting for FSMap version 7 2023-10-16T20:39:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effe483b000 10 client.4461 resolve_mds: resolved name 'f' to daemon mds.f 2023-10-16T20:39:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effe483b000 20 client.4461 populate_metadata read hostname 'smithi088' 2023-10-16T20:39:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effe483b000 1 --2- 172.21.15.88:0/3803658531 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x56048b4c69a0 0x56048b4c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effe483b000 4 client.4461 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T20:39:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effe483b000 1 -- 172.21.15.88:0/3803658531 --> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56048b2692d0 con 0x56048b4c69a0 2023-10-16T20:39:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effd27fc640 1 --2- 172.21.15.88:0/3803658531 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x56048b4c69a0 0x56048b4c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effd27fc640 1 --2- 172.21.15.88:0/3803658531 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x56048b4c69a0 0x56048b4c6d80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75aa20a411e31c58 server_cookie=520e73c31d6b9a22 in_seq=0 out_seq=0 2023-10-16T20:39:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.786+0000 7effb9ffb640 10 client.4461 ms_handle_connect on v2:172.21.15.191:6834/3059699574 2023-10-16T20:39:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.790+0000 7effb9ffb640 1 -- 172.21.15.88:0/3803658531 <== mds.0 v2:172.21.15.191:6834/3059699574 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56048b2692d0 con 0x56048b4c69a0 2023-10-16T20:39:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.790+0000 7effb9ffb640 10 client.4461 handle_command_reply: tid=0 2023-10-16T20:39:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.790+0000 7effe483b000 1 client.4461 shutdown 2023-10-16T20:39:06.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.790+0000 7effe483b000 1 -- 172.21.15.88:0/3803658531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4bf810 msgr2=0x56048b4b9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:06.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.790+0000 7effe483b000 1 --2- 172.21.15.88:0/3803658531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4bf810 0x56048b4b9f50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7effcc054920 tx=0x7effcc054c20 comp rx=0 tx=0).stop 2023-10-16T20:39:06.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.790+0000 7effe483b000 1 -- 172.21.15.88:0/3803658531 shutdown_connections 2023-10-16T20:39:06.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.790+0000 7effe483b000 1 --2- 172.21.15.88:0/3803658531 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x56048b4c69a0 0x56048b4c6d80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.790+0000 7effe483b000 1 --2- 172.21.15.88:0/3803658531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4c0a50 0x56048b4c5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.790+0000 7effe483b000 1 --2- 172.21.15.88:0/3803658531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4c0130 0x56048b4ba490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.790+0000 7effe483b000 1 --2- 172.21.15.88:0/3803658531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4bf810 0x56048b4b9f50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.790+0000 7effb9ffb640 0 client.4461 ms_handle_reset on v2:172.21.15.191:6834/3059699574 2023-10-16T20:39:06.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.790+0000 7effe483b000 1 -- 172.21.15.88:0/3803658531 >> 172.21.15.88:0/3803658531 conn(0x56048b475fc0 msgr2=0x56048b4bf090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:06.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.790+0000 7effe483b000 1 -- 172.21.15.88:0/3803658531 shutdown_connections 2023-10-16T20:39:06.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.790+0000 7effe483b000 1 -- 172.21.15.88:0/3803658531 wait complete. 2023-10-16T20:39:06.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.790+0000 7effe483b000 20 client.4461 trim_cache size 0 max 16384 2023-10-16T20:39:06.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.802+0000 7effe483b000 1 Processor -- start 2023-10-16T20:39:06.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.802+0000 7effe483b000 1 -- start start 2023-10-16T20:39:06.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4bf810 0x56048b4c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4c0130 0x56048b4c3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4c0a50 0x56048b4c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048b4c14d0 con 0x56048b4bf810 2023-10-16T20:39:06.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048af89940 con 0x56048b4c0130 2023-10-16T20:39:06.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x56048b28e4d0 con 0x56048b4c0a50 2023-10-16T20:39:06.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effdaffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4bf810 0x56048b4c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effdaffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4bf810 0x56048b4c56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44924/0 (socket says 172.21.15.88:44924) 2023-10-16T20:39:06.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effdaffd640 1 -- 172.21.15.88:0/1126319178 learned_addr learned my addr 172.21.15.88:0/1126319178 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:06.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effdaffd640 1 -- 172.21.15.88:0/1126319178 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4c0a50 msgr2=0x56048b4c3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:06.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effdaffd640 1 --2- 172.21.15.88:0/1126319178 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4c0a50 0x56048b4c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effdaffd640 1 -- 172.21.15.88:0/1126319178 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4c0130 msgr2=0x56048b4c3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:06.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effdaffd640 1 --2- 172.21.15.88:0/1126319178 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4c0130 0x56048b4c3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effdaffd640 1 -- 172.21.15.88:0/1126319178 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56048b4d1f50 con 0x56048b4bf810 2023-10-16T20:39:06.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effdaffd640 1 --2- 172.21.15.88:0/1126319178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4bf810 0x56048b4c56f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7effcc054b60 tx=0x7effcc058000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551d16377cc83072 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effd17fa640 1 -- 172.21.15.88:0/1126319178 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc05cf20 con 0x56048b4bf810 2023-10-16T20:39:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effd17fa640 1 -- 172.21.15.88:0/1126319178 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effcc00e040 con 0x56048b4bf810 2023-10-16T20:39:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effd17fa640 1 -- 172.21.15.88:0/1126319178 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc0182b0 con 0x56048b4bf810 2023-10-16T20:39:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 -- 172.21.15.88:0/1126319178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4bf810 msgr2=0x56048b4c56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 --2- 172.21.15.88:0/1126319178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4bf810 0x56048b4c56f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7effcc054b60 tx=0x7effcc058000 comp rx=0 tx=0).stop 2023-10-16T20:39:06.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 -- 172.21.15.88:0/1126319178 shutdown_connections 2023-10-16T20:39:06.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 --2- 172.21.15.88:0/1126319178 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4c0a50 0x56048b4c3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 --2- 172.21.15.88:0/1126319178 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4c0130 0x56048b4c3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 --2- 172.21.15.88:0/1126319178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4bf810 0x56048b4c56f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 -- 172.21.15.88:0/1126319178 >> 172.21.15.88:0/1126319178 conn(0x56048b475fc0 msgr2=0x56048b484470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 -- 172.21.15.88:0/1126319178 shutdown_connections 2023-10-16T20:39:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 -- 172.21.15.88:0/1126319178 wait complete. 2023-10-16T20:39:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 -- start start 2023-10-16T20:39:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 Processor -- start 2023-10-16T20:39:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4bf810 0x56048b51ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4c0130 0x56048b51cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4c0a50 0x56048b51d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048b2d3740 con 0x56048b4c0a50 2023-10-16T20:39:06.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x56048b310ba0 con 0x56048b4bf810 2023-10-16T20:39:06.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x56048b2a5a10 con 0x56048b4c0130 2023-10-16T20:39:06.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.806+0000 7effe483b000 10 client.0 _ll_register_callbacks cb 0x7effd4009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T20:39:06.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effdb7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4c0a50 0x56048b51d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effdb7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4c0a50 0x56048b51d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44928/0 (socket says 172.21.15.88:44928) 2023-10-16T20:39:06.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effdb7fe640 1 -- 172.21.15.88:0/2638833070 learned_addr learned my addr 172.21.15.88:0/2638833070 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:06.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effdaffd640 1 --2- 172.21.15.88:0/2638833070 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4bf810 0x56048b51ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effd27fc640 1 --2- 172.21.15.88:0/2638833070 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4c0130 0x56048b51cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effdb7fe640 1 -- 172.21.15.88:0/2638833070 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4c0130 msgr2=0x56048b51cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:06.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effdb7fe640 1 --2- 172.21.15.88:0/2638833070 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4c0130 0x56048b51cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effdb7fe640 1 -- 172.21.15.88:0/2638833070 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4bf810 msgr2=0x56048b51ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:06.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effdb7fe640 1 --2- 172.21.15.88:0/2638833070 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4bf810 0x56048b51ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effdb7fe640 1 -- 172.21.15.88:0/2638833070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56048b291460 con 0x56048b4c0a50 2023-10-16T20:39:06.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effdb7fe640 1 --2- 172.21.15.88:0/2638833070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4c0a50 0x56048b51d730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7effd4000c10 tx=0x7effd4015690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4014cf54ac1d7441 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:06.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-10-16T20:39:06.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effb9ffb640 1 -- 172.21.15.88:0/2638833070 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4015cd0 con 0x56048b4c0a50 2023-10-16T20:39:06.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effb9ffb640 1 -- 172.21.15.88:0/2638833070 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7effd4015e30 con 0x56048b4c0a50 2023-10-16T20:39:06.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 1 -- 172.21.15.88:0/2638833070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56048b2c8720 con 0x56048b4c0a50 2023-10-16T20:39:06.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effb9ffb640 1 -- 172.21.15.88:0/2638833070 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7effd40059a0 con 0x56048b4c0a50 2023-10-16T20:39:06.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 10 client.4467 fetch_fsmap learned FSMap version 7 2023-10-16T20:39:06.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 1 -- 172.21.15.88:0/2638833070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x56048b277760 con 0x56048b4c0a50 2023-10-16T20:39:06.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effb9ffb640 1 -- 172.21.15.88:0/2638833070 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56048b2c8720 con 0x56048b4c0a50 2023-10-16T20:39:06.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effb9ffb640 1 -- 172.21.15.88:0/2638833070 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 7) v1 ==== 4452+0+0 (secure 0 0 0) 0x7effd4005b00 con 0x56048b4c0a50 2023-10-16T20:39:06.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 10 client.4467 fetch_fsmap finished waiting for FSMap version 7 2023-10-16T20:39:06.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 10 client.4467 resolve_mds: resolved name 'f' to daemon mds.f 2023-10-16T20:39:06.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 20 client.4467 populate_metadata read hostname 'smithi088' 2023-10-16T20:39:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 1 --2- 172.21.15.88:0/2638833070 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x56048b529be0 0x56048b51e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 4 client.4467 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T20:39:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 1 -- 172.21.15.88:0/2638833070 --> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56048b2c8440 con 0x56048b529be0 2023-10-16T20:39:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effdaffd640 1 --2- 172.21.15.88:0/2638833070 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x56048b529be0 0x56048b51e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effdaffd640 1 --2- 172.21.15.88:0/2638833070 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x56048b529be0 0x56048b51e550 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e400ebe760ff21c8 server_cookie=e3dc1441fbff5cf1 in_seq=0 out_seq=0 2023-10-16T20:39:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effb9ffb640 10 client.4467 ms_handle_connect on v2:172.21.15.191:6834/3059699574 2023-10-16T20:39:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effb9ffb640 1 -- 172.21.15.88:0/2638833070 <== mds.0 v2:172.21.15.191:6834/3059699574 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x56048b2c8440 con 0x56048b529be0 2023-10-16T20:39:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effb9ffb640 10 client.4467 handle_command_reply: tid=0 2023-10-16T20:39:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 1 client.4467 shutdown 2023-10-16T20:39:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 1 -- 172.21.15.88:0/2638833070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4c0a50 msgr2=0x56048b51d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 1 --2- 172.21.15.88:0/2638833070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4c0a50 0x56048b51d730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7effd4000c10 tx=0x7effd4015690 comp rx=0 tx=0).stop 2023-10-16T20:39:06.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 1 -- 172.21.15.88:0/2638833070 shutdown_connections 2023-10-16T20:39:06.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 1 --2- 172.21.15.88:0/2638833070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56048b4c0a50 0x56048b51d730 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 1 --2- 172.21.15.88:0/2638833070 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x56048b529be0 0x56048b51e550 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 1 --2- 172.21.15.88:0/2638833070 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x56048b4c0130 0x56048b51cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 1 --2- 172.21.15.88:0/2638833070 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56048b4bf810 0x56048b51ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effb9ffb640 0 client.4467 ms_handle_reset on v2:172.21.15.191:6834/3059699574 2023-10-16T20:39:06.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 1 -- 172.21.15.88:0/2638833070 >> 172.21.15.88:0/2638833070 conn(0x56048b475fc0 msgr2=0x56048b483be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:06.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 1 -- 172.21.15.88:0/2638833070 shutdown_connections 2023-10-16T20:39:06.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 1 -- 172.21.15.88:0/2638833070 wait complete. 2023-10-16T20:39:06.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe483b000 20 client.4467 trim_cache size 0 max 16384 2023-10-16T20:39:06.829 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-10-16T20:39:06.830 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "8cf7342f-a394-4b4c-ac35-9931ac9d9e57", 2023-10-16T20:39:06.830 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2023-10-16T20:39:06.830 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 4288, 2023-10-16T20:39:06.830 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2023-10-16T20:39:06.830 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2023-10-16T20:39:06.831 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2023-10-16T20:39:06.831 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 131.07246069600001, 2023-10-16T20:39:06.831 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 6, 2023-10-16T20:39:06.831 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 29, 2023-10-16T20:39:06.831 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 27, 2023-10-16T20:39:06.831 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 137.89871439199999 2023-10-16T20:39:06.832 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-10-16T20:39:06.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe14c9640 1 -- 172.21.15.88:0/3147122038 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7effbc0664a0 msgr2=0x7effbc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:06.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe14c9640 1 --2- 172.21.15.88:0/3147122038 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7effbc0664a0 0x7effbc068960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7effc40b1000 tx=0x7effc40aa000 comp rx=0 tx=0).stop 2023-10-16T20:39:06.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe14c9640 1 -- 172.21.15.88:0/3147122038 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effdc0c5930 msgr2=0x7effdc071820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:06.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe14c9640 1 --2- 172.21.15.88:0/3147122038 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effdc0c5930 0x7effdc071820 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7effcc00c4d0 tx=0x7effcc00c990 comp rx=0 tx=0).stop 2023-10-16T20:39:06.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe14c9640 1 -- 172.21.15.88:0/3147122038 shutdown_connections 2023-10-16T20:39:06.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe14c9640 1 --2- 172.21.15.88:0/3147122038 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7effbc0664a0 0x7effbc068960 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe14c9640 1 --2- 172.21.15.88:0/3147122038 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effdc159490 0x7effdc14c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.810+0000 7effe14c9640 1 --2- 172.21.15.88:0/3147122038 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effdc153780 0x7effdc14c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.814+0000 7effe14c9640 1 --2- 172.21.15.88:0/3147122038 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effdc0c5930 0x7effdc071820 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.814+0000 7effe14c9640 1 -- 172.21.15.88:0/3147122038 >> 172.21.15.88:0/3147122038 conn(0x7effdc06bcb0 msgr2=0x7effdc0cbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:06.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.814+0000 7effe14c9640 1 -- 172.21.15.88:0/3147122038 shutdown_connections 2023-10-16T20:39:06.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.814+0000 7effe14c9640 1 -- 172.21.15.88:0/3147122038 wait complete. 2023-10-16T20:39:06.837 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T20:39:06.837 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-10-16T20:39:06.838 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T20:39:06.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.970+0000 7f1425405640 1 Processor -- start 2023-10-16T20:39:06.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.970+0000 7f1425405640 1 -- start start 2023-10-16T20:39:06.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.970+0000 7f1425405640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1420153770 0x7f1420159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.970+0000 7f1425405640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14200c9bb0 0x7f14200c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.970+0000 7f1425405640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142015a410 con 0x7f14200c41f0 2023-10-16T20:39:06.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.970+0000 7f1425405640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14200cc250 con 0x7f1420153770 2023-10-16T20:39:06.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.970+0000 7f1425405640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14200cc670 con 0x7f14200c9bb0 2023-10-16T20:39:06.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.970+0000 7f141effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1420153770 0x7f1420159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.970+0000 7f141effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1420153770 0x7f1420159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44930/0 (socket says 172.21.15.88:44930) 2023-10-16T20:39:06.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.970+0000 7f141effd640 1 -- 172.21.15.88:0/173846860 learned_addr learned my addr 172.21.15.88:0/173846860 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:06.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.970+0000 7f1417fff640 1 --2- 172.21.15.88:0/173846860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14200c9bb0 0x7f14200c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.970+0000 7f141effd640 1 -- 172.21.15.88:0/173846860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14200c9bb0 msgr2=0x7f14200c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:06.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.970+0000 7f141effd640 1 --2- 172.21.15.88:0/173846860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14200c9bb0 0x7f14200c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.970+0000 7f141effd640 1 -- 172.21.15.88:0/173846860 >> v1:172.21.15.191:6789/0 conn(0x7f14200c41f0 legacy=0x7f14200cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:06.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.970+0000 7f141effd640 1 -- 172.21.15.88:0/173846860 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1420153b50 con 0x7f1420153770 2023-10-16T20:39:06.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.974+0000 7f141effd640 1 --2- 172.21.15.88:0/173846860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1420153770 0x7f1420159ed0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1410006fd0 tx=0x7f141001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=545d890a402b59ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:06.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.974+0000 7f141e7fc640 1 -- 172.21.15.88:0/173846860 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14100a7070 con 0x7f1420153770 2023-10-16T20:39:06.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.974+0000 7f141e7fc640 1 -- 172.21.15.88:0/173846860 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14100ab430 con 0x7f1420153770 2023-10-16T20:39:06.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.974+0000 7f141e7fc640 1 -- 172.21.15.88:0/173846860 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14100ab7c0 con 0x7f1420153770 2023-10-16T20:39:06.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.974+0000 7f1425405640 1 -- 172.21.15.88:0/173846860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1420153770 msgr2=0x7f1420159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:06.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.974+0000 7f1425405640 1 --2- 172.21.15.88:0/173846860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1420153770 0x7f1420159ed0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1410006fd0 tx=0x7f141001c520 comp rx=0 tx=0).stop 2023-10-16T20:39:06.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.974+0000 7f1425405640 1 -- 172.21.15.88:0/173846860 shutdown_connections 2023-10-16T20:39:06.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.974+0000 7f1425405640 1 --2- 172.21.15.88:0/173846860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14200c9bb0 0x7f14200c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.974+0000 7f1425405640 1 --2- 172.21.15.88:0/173846860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1420153770 0x7f1420159ed0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.974+0000 7f1425405640 1 -- 172.21.15.88:0/173846860 >> 172.21.15.88:0/173846860 conn(0x7f14200715c0 msgr2=0x7f14200719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:06.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.974+0000 7f1425405640 1 -- 172.21.15.88:0/173846860 shutdown_connections 2023-10-16T20:39:06.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.974+0000 7f1425405640 1 -- 172.21.15.88:0/173846860 wait complete. 2023-10-16T20:39:06.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1425405640 1 Processor -- start 2023-10-16T20:39:06.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1425405640 1 -- start start 2023-10-16T20:39:06.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1425405640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14200c41f0 0x7f1420158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1425405640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14200c9bb0 0x7f14200ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1425405640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1420153770 0x7f14201444b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1425405640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1420159470 con 0x7f14200c9bb0 2023-10-16T20:39:06.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1425405640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14200c80d0 con 0x7f1420153770 2023-10-16T20:39:06.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1425405640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14200c6db0 con 0x7f14200c41f0 2023-10-16T20:39:06.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1417fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14200c9bb0 0x7f14200ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1417fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14200c9bb0 0x7f14200ce4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44942/0 (socket says 172.21.15.88:44942) 2023-10-16T20:39:06.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1417fff640 1 -- 172.21.15.88:0/4095873048 learned_addr learned my addr 172.21.15.88:0/4095873048 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:06.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f141effd640 1 --2- 172.21.15.88:0/4095873048 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14200c41f0 0x7f1420158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f141f7fe640 1 --2- 172.21.15.88:0/4095873048 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1420153770 0x7f14201444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1417fff640 1 -- 172.21.15.88:0/4095873048 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14200c41f0 msgr2=0x7f1420158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:06.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1417fff640 1 --2- 172.21.15.88:0/4095873048 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14200c41f0 0x7f1420158ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1417fff640 1 -- 172.21.15.88:0/4095873048 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1420153770 msgr2=0x7f14201444b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:06.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1417fff640 1 --2- 172.21.15.88:0/4095873048 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1420153770 0x7f14201444b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:06.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1417fff640 1 -- 172.21.15.88:0/4095873048 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1410009480 con 0x7f14200c9bb0 2023-10-16T20:39:06.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1417fff640 1 --2- 172.21.15.88:0/4095873048 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14200c9bb0 0x7f14200ce4c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f140800d800 tx=0x7f140800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:06.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f14177fe640 1 -- 172.21.15.88:0/4095873048 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140800b6c0 con 0x7f14200c9bb0 2023-10-16T20:39:06.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f14177fe640 1 -- 172.21.15.88:0/4095873048 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140800b820 con 0x7f14200c9bb0 2023-10-16T20:39:06.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f14177fe640 1 -- 172.21.15.88:0/4095873048 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140800bb90 con 0x7f14200c9bb0 2023-10-16T20:39:06.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1425405640 1 -- 172.21.15.88:0/4095873048 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14201591e0 con 0x7f14200c9bb0 2023-10-16T20:39:06.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.978+0000 7f1425405640 1 -- 172.21.15.88:0/4095873048 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14200c6510 con 0x7f14200c9bb0 2023-10-16T20:39:06.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.982+0000 7f1425405640 1 -- 172.21.15.88:0/4095873048 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1420074700 con 0x7f14200c9bb0 2023-10-16T20:39:06.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.982+0000 7f14177fe640 1 -- 172.21.15.88:0/4095873048 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f140800bcf0 con 0x7f14200c9bb0 2023-10-16T20:39:06.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.982+0000 7f14177fe640 1 --2- 172.21.15.88:0/4095873048 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f14000664a0 0x7f1400068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:06.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.982+0000 7f14177fe640 1 -- 172.21.15.88:0/4095873048 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f1408084790 con 0x7f14200c9bb0 2023-10-16T20:39:06.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.982+0000 7f141effd640 1 --2- 172.21.15.88:0/4095873048 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f14000664a0 0x7f1400068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:06.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.982+0000 7f141effd640 1 --2- 172.21.15.88:0/4095873048 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f14000664a0 0x7f1400068960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f14100af000 tx=0x7f14100a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:06.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:06.986+0000 7f14177fe640 1 -- 172.21.15.88:0/4095873048 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f14080517b0 con 0x7f14200c9bb0 2023-10-16T20:39:07.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:07.166+0000 7f1425405640 1 -- 172.21.15.88:0/4095873048 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f14200ca740 con 0x7f14200c9bb0 2023-10-16T20:39:07.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:07.166+0000 7f14177fe640 1 -- 172.21.15.88:0/4095873048 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7f1408054dd0 con 0x7f14200c9bb0 2023-10-16T20:39:07.175 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-16T20:39:07.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:07.170+0000 7f1425405640 1 -- 172.21.15.88:0/4095873048 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f14000664a0 msgr2=0x7f1400068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:07.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:07.170+0000 7f1425405640 1 --2- 172.21.15.88:0/4095873048 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f14000664a0 0x7f1400068960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f14100af000 tx=0x7f14100a7000 comp rx=0 tx=0).stop 2023-10-16T20:39:07.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:07.170+0000 7f1425405640 1 -- 172.21.15.88:0/4095873048 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14200c9bb0 msgr2=0x7f14200ce4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:07.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:07.170+0000 7f1425405640 1 --2- 172.21.15.88:0/4095873048 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14200c9bb0 0x7f14200ce4c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f140800d800 tx=0x7f140800dcc0 comp rx=0 tx=0).stop 2023-10-16T20:39:07.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:07.170+0000 7f1425405640 1 -- 172.21.15.88:0/4095873048 shutdown_connections 2023-10-16T20:39:07.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:07.170+0000 7f1425405640 1 --2- 172.21.15.88:0/4095873048 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f14000664a0 0x7f1400068960 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:07.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:07.170+0000 7f1425405640 1 --2- 172.21.15.88:0/4095873048 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1420153770 0x7f14201444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:07.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:07.170+0000 7f1425405640 1 --2- 172.21.15.88:0/4095873048 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14200c9bb0 0x7f14200ce4c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:07.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:07.170+0000 7f1425405640 1 --2- 172.21.15.88:0/4095873048 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14200c41f0 0x7f1420158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:07.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:07.170+0000 7f1425405640 1 -- 172.21.15.88:0/4095873048 >> 172.21.15.88:0/4095873048 conn(0x7f14200715c0 msgr2=0x7f14200b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:07.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:07.170+0000 7f1425405640 1 -- 172.21.15.88:0/4095873048 shutdown_connections 2023-10-16T20:39:07.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:07.170+0000 7f1425405640 1 -- 172.21.15.88:0/4095873048 wait complete. 2023-10-16T20:39:07.196 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-16T20:39:07.197 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T20:39:07.197 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2023-10-16T20:39:07.197 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-10-16T20:39:07.198 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-10-16T20:39:07.198 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-16T20:39:07.198 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T20:39:07.199 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T20:39:07.199 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-10-16T20:39:07.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:07.200 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2023-10-16T20:39:07.209 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T20:39:07.209 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T20:39:07.209 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T20:39:07.209 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2023-10-16T20:39:07.210 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2023-10-16T20:39:07.210 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2023-10-16T20:39:07.210 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-16T20:39:07.210 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2023-10-16T20:39:07.211 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2023-10-16T20:39:07.211 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-16T20:39:07.211 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2023-10-16T20:39:07.211 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2023-10-16T20:39:07.213 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-16T20:39:07.213 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T20:39:07.214 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T20:39:07.214 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2023-10-16T20:39:07.214 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-10-16T20:39:07.214 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42308sec preferred_lft 42308sec 2023-10-16T20:39:07.214 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link 2023-10-16T20:39:07.215 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2023-10-16T20:39:07.215 INFO:teuthology.orchestra.run.smithi088.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-10-16T20:39:07.215 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 02:42:34:f9:43:24 brd ff:ff:ff:ff:ff:ff 2023-10-16T20:39:07.215 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-10-16T20:39:07.215 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2023-10-16T20:39:07.217 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-16T20:39:07.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:07.217 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-10-16T20:39:07.218 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-10-16T20:39:07.218 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link add name ceph-brx type bridge 2023-10-16T20:39:07.219 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip addr flush dev ceph-brx 2023-10-16T20:39:07.219 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set ceph-brx up 2023-10-16T20:39:07.219 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-16T20:39:07.220 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-10-16T20:39:07.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:07.331 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T20:39:07.351 INFO:teuthology.orchestra.run.smithi088.stdout:1 2023-10-16T20:39:07.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:07.353 DEBUG:teuthology.orchestra.run.smithi088:> route 2023-10-16T20:39:07.423 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2023-10-16T20:39:07.423 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T20:39:07.423 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-10-16T20:39:07.424 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T20:39:07.424 INFO:teuthology.orchestra.run.smithi088.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-10-16T20:39:07.424 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T20:39:07.424 INFO:teuthology.orchestra.run.smithi088.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-16T20:39:07.424 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-16T20:39:07.425 INFO:teuthology.orchestra.run.smithi088.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-16T20:39:07.425 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T20:39:07.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:07.426 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-10-16T20:39:07.426 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-10-16T20:39:07.426 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T20:39:07.427 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T20:39:07.427 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T20:39:07.427 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-10-16T20:39:07.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:07.518 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2023-10-16T20:39:07.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:07.528 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list-id 2023-10-16T20:39:07.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:07.579 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-10-16T20:39:07.579 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-10-16T20:39:07.580 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T20:39:07.580 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-10-16T20:39:07.580 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-10-16T20:39:07.663 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-10-16T20:39:07.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:07.664 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-10-16T20:39:07.664 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-10-16T20:39:07.665 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-10-16T20:39:07.665 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-16T20:39:07.665 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-10-16T20:39:07.666 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-10-16T20:39:07.666 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-10-16T20:39:07.666 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-10-16T20:39:07.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:07.820 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-10-16T20:39:07.821 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-10-16T20:39:07.821 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2023-10-16T20:39:07.821 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set dev brx.0 master ceph-brx 2023-10-16T20:39:07.822 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-10-16T20:39:07.865 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-10-16T20:39:07.865 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T20:39:07.866 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-10-16T20:39:07.919 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-10-16T20:39:07.919 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T20:39:07.919 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-10-16T20:39:07.973 DEBUG:teuthology.orchestra.run.smithi088:> sudo modprobe fuse 2023-10-16T20:39:08.034 DEBUG:teuthology.orchestra.run.smithi088:> cat /proc/self/mounts | awk '{print $2}' 2023-10-16T20:39:08.088 INFO:teuthology.orchestra.run.smithi088.stdout:/sys 2023-10-16T20:39:08.088 INFO:teuthology.orchestra.run.smithi088.stdout:/proc 2023-10-16T20:39:08.088 INFO:teuthology.orchestra.run.smithi088.stdout:/dev 2023-10-16T20:39:08.088 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/pts 2023-10-16T20:39:08.088 INFO:teuthology.orchestra.run.smithi088.stdout:/run 2023-10-16T20:39:08.089 INFO:teuthology.orchestra.run.smithi088.stdout:/ 2023-10-16T20:39:08.089 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/security 2023-10-16T20:39:08.089 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/shm 2023-10-16T20:39:08.089 INFO:teuthology.orchestra.run.smithi088.stdout:/run/lock 2023-10-16T20:39:08.089 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/cgroup 2023-10-16T20:39:08.090 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/pstore 2023-10-16T20:39:08.090 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/bpf 2023-10-16T20:39:08.090 INFO:teuthology.orchestra.run.smithi088.stdout:/proc/sys/fs/binfmt_misc 2023-10-16T20:39:08.090 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/hugepages 2023-10-16T20:39:08.090 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/mqueue 2023-10-16T20:39:08.091 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/debug 2023-10-16T20:39:08.091 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/tracing 2023-10-16T20:39:08.091 INFO:teuthology.orchestra.run.smithi088.stdout:/proc/fs/nfsd 2023-10-16T20:39:08.091 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/fuse/connections 2023-10-16T20:39:08.091 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/config 2023-10-16T20:39:08.092 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-sysusers.service 2023-10-16T20:39:08.092 INFO:teuthology.orchestra.run.smithi088.stdout:/run/rpc_pipefs 2023-10-16T20:39:08.092 INFO:teuthology.orchestra.run.smithi088.stdout:/run/user/1000 2023-10-16T20:39:08.092 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph 2023-10-16T20:39:08.092 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-0 2023-10-16T20:39:08.093 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-1 2023-10-16T20:39:08.093 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-2 2023-10-16T20:39:08.093 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-3 2023-10-16T20:39:08.093 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns 2023-10-16T20:39:08.093 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T20:39:08.094 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T20:39:08.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:08.095 DEBUG:teuthology.orchestra.run.smithi088:> ls /sys/fs/fuse/connections 2023-10-16T20:39:08.146 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-16T20:39:08.146 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-10-16T20:39:08.193 DEBUG:teuthology.orchestra.run.smithi088:> sudo modprobe fuse 2023-10-16T20:39:08.214 DEBUG:teuthology.orchestra.run.smithi088:> cat /proc/self/mounts | awk '{print $2}' 2023-10-16T20:39:08.268 INFO:teuthology.orchestra.run.smithi088.stdout:/sys 2023-10-16T20:39:08.268 INFO:teuthology.orchestra.run.smithi088.stdout:/proc 2023-10-16T20:39:08.268 INFO:teuthology.orchestra.run.smithi088.stdout:/dev 2023-10-16T20:39:08.269 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/pts 2023-10-16T20:39:08.269 INFO:teuthology.orchestra.run.smithi088.stdout:/run 2023-10-16T20:39:08.269 INFO:teuthology.orchestra.run.smithi088.stdout:/ 2023-10-16T20:39:08.269 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/security 2023-10-16T20:39:08.269 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/shm 2023-10-16T20:39:08.270 INFO:teuthology.orchestra.run.smithi088.stdout:/run/lock 2023-10-16T20:39:08.270 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/cgroup 2023-10-16T20:39:08.270 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/pstore 2023-10-16T20:39:08.270 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/bpf 2023-10-16T20:39:08.270 INFO:teuthology.orchestra.run.smithi088.stdout:/proc/sys/fs/binfmt_misc 2023-10-16T20:39:08.271 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/hugepages 2023-10-16T20:39:08.271 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/mqueue 2023-10-16T20:39:08.271 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/debug 2023-10-16T20:39:08.271 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/tracing 2023-10-16T20:39:08.271 INFO:teuthology.orchestra.run.smithi088.stdout:/proc/fs/nfsd 2023-10-16T20:39:08.272 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/fuse/connections 2023-10-16T20:39:08.272 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/config 2023-10-16T20:39:08.272 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-sysusers.service 2023-10-16T20:39:08.272 INFO:teuthology.orchestra.run.smithi088.stdout:/run/rpc_pipefs 2023-10-16T20:39:08.272 INFO:teuthology.orchestra.run.smithi088.stdout:/run/user/1000 2023-10-16T20:39:08.272 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph 2023-10-16T20:39:08.273 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-0 2023-10-16T20:39:08.273 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-1 2023-10-16T20:39:08.273 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-2 2023-10-16T20:39:08.273 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-3 2023-10-16T20:39:08.273 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns 2023-10-16T20:39:08.274 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T20:39:08.274 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T20:39:08.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:08.275 DEBUG:teuthology.orchestra.run.smithi088:> ls /sys/fs/fuse/connections 2023-10-16T20:39:08.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi088.stderr:2023-10-16T20:39:08.297+0000 7f55fc066580 -1 init, newargv = 0x55dff4bd3840 newargc=15 2023-10-16T20:39:08.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi088.stderr:ceph-fuse[44187]: starting ceph client 2023-10-16T20:39:08.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi088.stderr:ceph-fuse[44187]: starting fuse 2023-10-16T20:39:09.416 DEBUG:teuthology.orchestra.run.smithi088:> sudo modprobe fuse 2023-10-16T20:39:09.441 DEBUG:teuthology.orchestra.run.smithi088:> cat /proc/self/mounts | awk '{print $2}' 2023-10-16T20:39:09.493 INFO:teuthology.orchestra.run.smithi088.stdout:/sys 2023-10-16T20:39:09.493 INFO:teuthology.orchestra.run.smithi088.stdout:/proc 2023-10-16T20:39:09.493 INFO:teuthology.orchestra.run.smithi088.stdout:/dev 2023-10-16T20:39:09.493 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/pts 2023-10-16T20:39:09.494 INFO:teuthology.orchestra.run.smithi088.stdout:/run 2023-10-16T20:39:09.494 INFO:teuthology.orchestra.run.smithi088.stdout:/ 2023-10-16T20:39:09.494 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/security 2023-10-16T20:39:09.494 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/shm 2023-10-16T20:39:09.494 INFO:teuthology.orchestra.run.smithi088.stdout:/run/lock 2023-10-16T20:39:09.495 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/cgroup 2023-10-16T20:39:09.495 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/pstore 2023-10-16T20:39:09.495 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/bpf 2023-10-16T20:39:09.495 INFO:teuthology.orchestra.run.smithi088.stdout:/proc/sys/fs/binfmt_misc 2023-10-16T20:39:09.495 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/hugepages 2023-10-16T20:39:09.496 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/mqueue 2023-10-16T20:39:09.496 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/debug 2023-10-16T20:39:09.496 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/tracing 2023-10-16T20:39:09.496 INFO:teuthology.orchestra.run.smithi088.stdout:/proc/fs/nfsd 2023-10-16T20:39:09.496 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/fuse/connections 2023-10-16T20:39:09.497 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/config 2023-10-16T20:39:09.497 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-sysusers.service 2023-10-16T20:39:09.497 INFO:teuthology.orchestra.run.smithi088.stdout:/run/rpc_pipefs 2023-10-16T20:39:09.497 INFO:teuthology.orchestra.run.smithi088.stdout:/run/user/1000 2023-10-16T20:39:09.497 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph 2023-10-16T20:39:09.498 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-0 2023-10-16T20:39:09.498 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-1 2023-10-16T20:39:09.498 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-2 2023-10-16T20:39:09.498 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-3 2023-10-16T20:39:09.498 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns 2023-10-16T20:39:09.499 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T20:39:09.499 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T20:39:09.499 INFO:teuthology.orchestra.run.smithi088.stdout:/home/ubuntu/cephtest/mnt.admin 2023-10-16T20:39:09.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:09.500 DEBUG:teuthology.orchestra.run.smithi088:> ls /sys/fs/fuse/connections 2023-10-16T20:39:09.551 INFO:teuthology.orchestra.run.smithi088.stdout:43 2023-10-16T20:39:09.551 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-16T20:39:09.552 DEBUG:teuthology.orchestra.run.smithi088:> sudo stdin-killer -- python3 -c ' 2023-10-16T20:39:09.552 DEBUG:teuthology.orchestra.run.smithi088:> import glob 2023-10-16T20:39:09.553 DEBUG:teuthology.orchestra.run.smithi088:> import re 2023-10-16T20:39:09.553 DEBUG:teuthology.orchestra.run.smithi088:> import os 2023-10-16T20:39:09.553 DEBUG:teuthology.orchestra.run.smithi088:> import subprocess 2023-10-16T20:39:09.554 DEBUG:teuthology.orchestra.run.smithi088:> 2023-10-16T20:39:09.554 DEBUG:teuthology.orchestra.run.smithi088:> def _find_admin_socket(client_name): 2023-10-16T20:39:09.554 DEBUG:teuthology.orchestra.run.smithi088:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-10-16T20:39:09.555 DEBUG:teuthology.orchestra.run.smithi088:> files = glob.glob(asok_path) 2023-10-16T20:39:09.555 DEBUG:teuthology.orchestra.run.smithi088:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-10-16T20:39:09.555 DEBUG:teuthology.orchestra.run.smithi088:> 2023-10-16T20:39:09.556 DEBUG:teuthology.orchestra.run.smithi088:> # Given a non-glob path, it better be there 2023-10-16T20:39:09.556 DEBUG:teuthology.orchestra.run.smithi088:> if "*" not in asok_path: 2023-10-16T20:39:09.556 DEBUG:teuthology.orchestra.run.smithi088:> assert(len(files) == 1) 2023-10-16T20:39:09.557 DEBUG:teuthology.orchestra.run.smithi088:> return files[0] 2023-10-16T20:39:09.557 DEBUG:teuthology.orchestra.run.smithi088:> 2023-10-16T20:39:09.558 DEBUG:teuthology.orchestra.run.smithi088:> for f in files: 2023-10-16T20:39:09.558 DEBUG:teuthology.orchestra.run.smithi088:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-16T20:39:09.558 DEBUG:teuthology.orchestra.run.smithi088:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-16T20:39:09.559 DEBUG:teuthology.orchestra.run.smithi088:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-16T20:39:09.559 DEBUG:teuthology.orchestra.run.smithi088:> contents = proc_f.read() 2023-10-16T20:39:09.559 DEBUG:teuthology.orchestra.run.smithi088:> if mountpoint in contents: 2023-10-16T20:39:09.560 DEBUG:teuthology.orchestra.run.smithi088:> return f 2023-10-16T20:39:09.560 DEBUG:teuthology.orchestra.run.smithi088:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-16T20:39:09.561 DEBUG:teuthology.orchestra.run.smithi088:> 2023-10-16T20:39:09.561 DEBUG:teuthology.orchestra.run.smithi088:> print(_find_admin_socket("client.admin")) 2023-10-16T20:39:09.561 DEBUG:teuthology.orchestra.run.smithi088:> ' 2023-10-16T20:39:09.684 INFO:teuthology.orchestra.run.smithi088.stdout:/var/run/ceph/ceph-client.admin.44187.asok 2023-10-16T20:39:09.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-16T20:39:09.696 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.44187.asok 2023-10-16T20:39:09.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:09.697 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.44187.asok status 2023-10-16T20:39:09.874 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-10-16T20:39:09.874 INFO:teuthology.orchestra.run.smithi088.stdout: "metadata": { 2023-10-16T20:39:09.875 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_sha1": "4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0", 2023-10-16T20:39:09.875 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_version": "ceph version 18.0.0-6730-g4ba8a190 (4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0) reef (dev)", 2023-10-16T20:39:09.875 INFO:teuthology.orchestra.run.smithi088.stdout: "entity_id": "admin", 2023-10-16T20:39:09.876 INFO:teuthology.orchestra.run.smithi088.stdout: "hostname": "smithi088", 2023-10-16T20:39:09.876 INFO:teuthology.orchestra.run.smithi088.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-10-16T20:39:09.876 INFO:teuthology.orchestra.run.smithi088.stdout: "pid": "44187", 2023-10-16T20:39:09.876 INFO:teuthology.orchestra.run.smithi088.stdout: "root": "/" 2023-10-16T20:39:09.877 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-10-16T20:39:09.877 INFO:teuthology.orchestra.run.smithi088.stdout: "dentry_count": 0, 2023-10-16T20:39:09.877 INFO:teuthology.orchestra.run.smithi088.stdout: "dentry_pinned_count": 0, 2023-10-16T20:39:09.877 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 4482, 2023-10-16T20:39:09.878 INFO:teuthology.orchestra.run.smithi088.stdout: "inst": { 2023-10-16T20:39:09.878 INFO:teuthology.orchestra.run.smithi088.stdout: "name": { 2023-10-16T20:39:09.878 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "client", 2023-10-16T20:39:09.878 INFO:teuthology.orchestra.run.smithi088.stdout: "num": 4482 2023-10-16T20:39:09.878 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-10-16T20:39:09.879 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": { 2023-10-16T20:39:09.879 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2023-10-16T20:39:09.879 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "192.168.0.1:0", 2023-10-16T20:39:09.879 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2879963654 2023-10-16T20:39:09.880 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-10-16T20:39:09.880 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-10-16T20:39:09.880 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": { 2023-10-16T20:39:09.880 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2023-10-16T20:39:09.881 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "192.168.0.1:0", 2023-10-16T20:39:09.881 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 2879963654 2023-10-16T20:39:09.882 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-10-16T20:39:09.882 INFO:teuthology.orchestra.run.smithi088.stdout: "inst_str": "client.4482 192.168.0.1:0/2879963654", 2023-10-16T20:39:09.882 INFO:teuthology.orchestra.run.smithi088.stdout: "addr_str": "192.168.0.1:0/2879963654", 2023-10-16T20:39:09.883 INFO:teuthology.orchestra.run.smithi088.stdout: "inode_count": 1, 2023-10-16T20:39:09.883 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_epoch": 6, 2023-10-16T20:39:09.883 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_epoch": 29, 2023-10-16T20:39:09.884 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_epoch_barrier": 0, 2023-10-16T20:39:09.884 INFO:teuthology.orchestra.run.smithi088.stdout: "blocklisted": false, 2023-10-16T20:39:09.884 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs" 2023-10-16T20:39:09.885 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-10-16T20:39:09.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:09.889 DEBUG:teuthology.orchestra.run.smithi088:> stat --file-system '--printf=%T 2023-10-16T20:39:09.889 DEBUG:teuthology.orchestra.run.smithi088:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-10-16T20:39:09.899 INFO:teuthology.orchestra.run.smithi088.stdout:fuseblk 2023-10-16T20:39:09.899 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-10-16T20:39:09.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:09.900 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-10-16T20:39:09.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:09.964 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-10-16T20:39:10.045 INFO:teuthology.orchestra.run.smithi088.stdout:# file: . 2023-10-16T20:39:10.046 INFO:teuthology.orchestra.run.smithi088.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-10-16T20:39:10.046 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:10.046 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:39:10.047 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T20:39:10.094 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:39:10.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.269+0000 7f3fc2e6b640 1 Processor -- start 2023-10-16T20:39:10.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.269+0000 7f3fc2e6b640 1 -- start start 2023-10-16T20:39:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.269+0000 7f3fc2e6b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3fbc153770 0x7f3fbc159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.269+0000 7f3fc2e6b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fbc0c9bb0 0x7f3fbc0c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.269+0000 7f3fc2e6b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc15a410 con 0x7f3fbc0c41f0 2023-10-16T20:39:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.269+0000 7f3fc2e6b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc0cc250 con 0x7f3fbc153770 2023-10-16T20:39:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.269+0000 7f3fc2e6b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0cc670 con 0x7f3fbc0c9bb0 2023-10-16T20:39:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.269+0000 7f3fc0c07640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3fbc153770 0x7f3fbc159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.269+0000 7f3fc0c07640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3fbc153770 0x7f3fbc159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52562/0 (socket says 172.21.15.88:52562) 2023-10-16T20:39:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.269+0000 7f3fc0c07640 1 -- 172.21.15.88:0/3043533170 learned_addr learned my addr 172.21.15.88:0/3043533170 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.269+0000 7f3fbbfff640 1 --2- 172.21.15.88:0/3043533170 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fbc0c9bb0 0x7f3fbc0c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.269+0000 7f3fc0c07640 1 -- 172.21.15.88:0/3043533170 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fbc0c9bb0 msgr2=0x7f3fbc0c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.269+0000 7f3fc0c07640 1 --2- 172.21.15.88:0/3043533170 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fbc0c9bb0 0x7f3fbc0c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.269+0000 7f3fc0c07640 1 -- 172.21.15.88:0/3043533170 >> v1:172.21.15.191:6789/0 conn(0x7f3fbc0c41f0 legacy=0x7f3fbc0cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:10.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.269+0000 7f3fc0c07640 1 -- 172.21.15.88:0/3043533170 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc153b50 con 0x7f3fbc153770 2023-10-16T20:39:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.273+0000 7f3fc0c07640 1 --2- 172.21.15.88:0/3043533170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fbc153770 0x7f3fbc159ed0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0006fd0 tx=0x7f3fb001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7813d5a8dc130144 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.273+0000 7f3fbb7fe640 1 -- 172.21.15.88:0/3043533170 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb00a7070 con 0x7f3fbc153770 2023-10-16T20:39:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.273+0000 7f3fbb7fe640 1 -- 172.21.15.88:0/3043533170 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb00ab430 con 0x7f3fbc153770 2023-10-16T20:39:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.273+0000 7f3fbb7fe640 1 -- 172.21.15.88:0/3043533170 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb00ab7c0 con 0x7f3fbc153770 2023-10-16T20:39:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.273+0000 7f3fc2e6b640 1 -- 172.21.15.88:0/3043533170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fbc153770 msgr2=0x7f3fbc159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.273+0000 7f3fc2e6b640 1 --2- 172.21.15.88:0/3043533170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fbc153770 0x7f3fbc159ed0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0006fd0 tx=0x7f3fb001c520 comp rx=0 tx=0).stop 2023-10-16T20:39:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.273+0000 7f3fc2e6b640 1 -- 172.21.15.88:0/3043533170 shutdown_connections 2023-10-16T20:39:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.273+0000 7f3fc2e6b640 1 --2- 172.21.15.88:0/3043533170 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fbc0c9bb0 0x7f3fbc0c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.273+0000 7f3fc2e6b640 1 --2- 172.21.15.88:0/3043533170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fbc153770 0x7f3fbc159ed0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.273+0000 7f3fc2e6b640 1 -- 172.21.15.88:0/3043533170 >> 172.21.15.88:0/3043533170 conn(0x7f3fbc0715c0 msgr2=0x7f3fbc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.273+0000 7f3fc2e6b640 1 -- 172.21.15.88:0/3043533170 shutdown_connections 2023-10-16T20:39:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.273+0000 7f3fc2e6b640 1 -- 172.21.15.88:0/3043533170 wait complete. 2023-10-16T20:39:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc2e6b640 1 Processor -- start 2023-10-16T20:39:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc2e6b640 1 -- start start 2023-10-16T20:39:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc2e6b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fbc0c41f0 0x7f3fbc158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc2e6b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fbc0c9bb0 0x7f3fbc0ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc2e6b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fbc153770 0x7f3fbc1444b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc2e6b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc159470 con 0x7f3fbc0c41f0 2023-10-16T20:39:10.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc2e6b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0c80d0 con 0x7f3fbc0c9bb0 2023-10-16T20:39:10.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc2e6b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0c6db0 con 0x7f3fbc153770 2023-10-16T20:39:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc0c07640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fbc0c41f0 0x7f3fbc158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc0c07640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fbc0c41f0 0x7f3fbc158ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52574/0 (socket says 172.21.15.88:52574) 2023-10-16T20:39:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc0c07640 1 -- 172.21.15.88:0/2662629717 learned_addr learned my addr 172.21.15.88:0/2662629717 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc0c07640 1 -- 172.21.15.88:0/2662629717 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fbc153770 msgr2=0x7f3fbc1444b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc0c07640 1 --2- 172.21.15.88:0/2662629717 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fbc153770 0x7f3fbc1444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc0c07640 1 -- 172.21.15.88:0/2662629717 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fbc0c9bb0 msgr2=0x7f3fbc0ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc0c07640 1 --2- 172.21.15.88:0/2662629717 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fbc0c9bb0 0x7f3fbc0ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc0c07640 1 -- 172.21.15.88:0/2662629717 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb0009480 con 0x7f3fbc0c41f0 2023-10-16T20:39:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc0c07640 1 --2- 172.21.15.88:0/2662629717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fbc0c41f0 0x7f3fbc158ca0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0006fd0 tx=0x7f3fb00192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:10.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fb97fa640 1 -- 172.21.15.88:0/2662629717 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb0003bb0 con 0x7f3fbc0c41f0 2023-10-16T20:39:10.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fb97fa640 1 -- 172.21.15.88:0/2662629717 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb0003d30 con 0x7f3fbc0c41f0 2023-10-16T20:39:10.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc2e6b640 1 -- 172.21.15.88:0/2662629717 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fbc1591e0 con 0x7f3fbc0c41f0 2023-10-16T20:39:10.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fb97fa640 1 -- 172.21.15.88:0/2662629717 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb0003ea0 con 0x7f3fbc0c41f0 2023-10-16T20:39:10.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc2e6b640 1 -- 172.21.15.88:0/2662629717 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fbc0c6510 con 0x7f3fbc0c41f0 2023-10-16T20:39:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.277+0000 7f3fc2e6b640 1 -- 172.21.15.88:0/2662629717 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fbc15c890 con 0x7f3fbc0c41f0 2023-10-16T20:39:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.281+0000 7f3fb97fa640 1 -- 172.21.15.88:0/2662629717 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3fb00ab7e0 con 0x7f3fbc0c41f0 2023-10-16T20:39:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.281+0000 7f3fb97fa640 1 --2- 172.21.15.88:0/2662629717 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3fa00664a0 0x7f3fa0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.281+0000 7f3fb97fa640 1 -- 172.21.15.88:0/2662629717 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3fb011d9c0 con 0x7f3fbc0c41f0 2023-10-16T20:39:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.281+0000 7f3fbbfff640 1 --2- 172.21.15.88:0/2662629717 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3fa00664a0 0x7f3fa0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:10.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.281+0000 7f3fbbfff640 1 --2- 172.21.15.88:0/2662629717 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3fa00664a0 0x7f3fa0068960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc1f2960 tx=0x7f3fa8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:10.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.285+0000 7f3fb97fa640 1 -- 172.21.15.88:0/2662629717 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3fbc15c890 con 0x7f3fbc0c41f0 2023-10-16T20:39:10.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.441+0000 7f3fc2e6b640 1 -- 172.21.15.88:0/2662629717 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fbc15c890 con 0x7f3fbc0c41f0 2023-10-16T20:39:10.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.441+0000 7f3fb97fa640 1 -- 172.21.15.88:0/2662629717 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18710 (secure 0 0 0) 0x7f3fbc15c890 con 0x7f3fbc0c41f0 2023-10-16T20:39:10.448 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:10.449 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:37:37.392549+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:39:10.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.441+0000 7f3fc2e6b640 1 -- 172.21.15.88:0/2662629717 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3fa00664a0 msgr2=0x7f3fa0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:10.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.441+0000 7f3fc2e6b640 1 --2- 172.21.15.88:0/2662629717 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3fa00664a0 0x7f3fa0068960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc1f2960 tx=0x7f3fa8008040 comp rx=0 tx=0).stop 2023-10-16T20:39:10.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.441+0000 7f3fc2e6b640 1 -- 172.21.15.88:0/2662629717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fbc0c41f0 msgr2=0x7f3fbc158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:10.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.441+0000 7f3fc2e6b640 1 --2- 172.21.15.88:0/2662629717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fbc0c41f0 0x7f3fbc158ca0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0006fd0 tx=0x7f3fb00192c0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.441+0000 7f3fc2e6b640 1 -- 172.21.15.88:0/2662629717 shutdown_connections 2023-10-16T20:39:10.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.441+0000 7f3fc2e6b640 1 --2- 172.21.15.88:0/2662629717 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3fa00664a0 0x7f3fa0068960 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.441+0000 7f3fc2e6b640 1 --2- 172.21.15.88:0/2662629717 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3fbc153770 0x7f3fbc1444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.441+0000 7f3fc2e6b640 1 --2- 172.21.15.88:0/2662629717 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3fbc0c9bb0 0x7f3fbc0ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.441+0000 7f3fc2e6b640 1 --2- 172.21.15.88:0/2662629717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fbc0c41f0 0x7f3fbc158ca0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.441+0000 7f3fc2e6b640 1 -- 172.21.15.88:0/2662629717 >> 172.21.15.88:0/2662629717 conn(0x7f3fbc0715c0 msgr2=0x7f3fbc0b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:10.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.441+0000 7f3fc2e6b640 1 -- 172.21.15.88:0/2662629717 shutdown_connections 2023-10-16T20:39:10.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.445+0000 7f3fc2e6b640 1 -- 172.21.15.88:0/2662629717 wait complete. 2023-10-16T20:39:10.470 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T20:36:43.646418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T20:36:53.546513+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T20:36:54.143703+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T20:36:57.314642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T20:39:10.470 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T20:39:10.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.601+0000 7f5330808640 1 Processor -- start 2023-10-16T20:39:10.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f5330808640 1 -- start start 2023-10-16T20:39:10.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f5330808640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f532c0c3ea0 0x7f532c0c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f5330808640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532c0ca5e0 con 0x7f532c0ca190 2023-10-16T20:39:10.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f5330808640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532c0c4350 con 0x7f532c153630 2023-10-16T20:39:10.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f5330808640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f532c0c4ed0 con 0x7f532c0c3ea0 2023-10-16T20:39:10.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532b59e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f532c0ca190 0x7f532c06a1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50400/0 (socket says 172.21.15.88:50400) 2023-10-16T20:39:10.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532b59e640 1 -- 172.21.15.88:0/2729724580 learned_addr learned my addr 172.21.15.88:0/2729724580 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:10.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532a59c640 1 -- 172.21.15.88:0/2729724580 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886799920 0 0) 0x7f532c0ca5e0 con 0x7f532c0ca190 2023-10-16T20:39:10.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532a59c640 1 -- 172.21.15.88:0/2729724580 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5318003530 con 0x7f532c0ca190 2023-10-16T20:39:10.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f5323fff640 1 --2- 172.21.15.88:0/2729724580 >> v2:172.21.15.191:3300/0 conn(0x7f532c0c3ea0 0x7f532c0c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:10.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532a59c640 1 -- 172.21.15.88:0/2729724580 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4047003244 0 0) 0x7f5318003530 con 0x7f532c0ca190 2023-10-16T20:39:10.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532a59c640 1 -- 172.21.15.88:0/2729724580 >> v2:172.21.15.191:3300/0 conn(0x7f532c0c3ea0 msgr2=0x7f532c0c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:10.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532a59c640 1 --2- 172.21.15.88:0/2729724580 >> v2:172.21.15.191:3300/0 conn(0x7f532c0c3ea0 0x7f532c0c9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532a59c640 1 -- 172.21.15.88:0/2729724580 >> v1:172.21.15.191:6789/0 conn(0x7f532c153630 legacy=0x7f532c159dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:10.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532a59c640 1 -- 172.21.15.88:0/2729724580 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532c0c4630 con 0x7f532c0ca190 2023-10-16T20:39:10.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532a59c640 1 -- 172.21.15.88:0/2729724580 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f5324001a00 con 0x7f532c0ca190 2023-10-16T20:39:10.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532a59c640 1 -- 172.21.15.88:0/2729724580 >> v1:172.21.15.88:6789/0 conn(0x7f532c0ca190 legacy=0x7f532c06a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:10.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532a59c640 1 --2- 172.21.15.88:0/2729724580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5318003980 0x7f5318003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532a59c640 1 --2- 172.21.15.88:0/2729724580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f53180047d0 0x7f5318004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532a59c640 1 --2- 172.21.15.88:0/2729724580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5318005190 0x7f531800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532a59c640 1 -- 172.21.15.88:0/2729724580 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c0c4ed0 con 0x7f5318005190 2023-10-16T20:39:10.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532a59c640 1 -- 172.21.15.88:0/2729724580 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531800c0e0 con 0x7f53180047d0 2023-10-16T20:39:10.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532a59c640 1 -- 172.21.15.88:0/2729724580 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531800c2a0 con 0x7f5318003980 2023-10-16T20:39:10.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532b59e640 1 --2- 172.21.15.88:0/2729724580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5318005190 0x7f531800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:10.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532ad9d640 1 --2- 172.21.15.88:0/2729724580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5318003980 0x7f5318003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:10.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f5323fff640 1 --2- 172.21.15.88:0/2729724580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f53180047d0 0x7f5318004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:10.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532b59e640 1 -- 172.21.15.88:0/2729724580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5318003980 msgr2=0x7f5318003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:10.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532b59e640 1 --2- 172.21.15.88:0/2729724580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5318003980 0x7f5318003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532b59e640 1 -- 172.21.15.88:0/2729724580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f53180047d0 msgr2=0x7f5318004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:10.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532b59e640 1 --2- 172.21.15.88:0/2729724580 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f53180047d0 0x7f5318004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.605+0000 7f532b59e640 1 -- 172.21.15.88:0/2729724580 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5318003530 con 0x7f5318005190 2023-10-16T20:39:10.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.609+0000 7f532b59e640 1 -- 172.21.15.88:0/2729724580 reap_dead start 2023-10-16T20:39:10.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.609+0000 7f532b59e640 1 --2- 172.21.15.88:0/2729724580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5318005190 0x7f531800b990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f532400a830 tx=0x7f532401d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd3fa9b24867f3f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:10.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.609+0000 7f532a59c640 1 -- 172.21.15.88:0/2729724580 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53240072d0 con 0x7f5318005190 2023-10-16T20:39:10.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.609+0000 7f532a59c640 1 -- 172.21.15.88:0/2729724580 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53240a8070 con 0x7f5318005190 2023-10-16T20:39:10.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.609+0000 7f532a59c640 1 -- 172.21.15.88:0/2729724580 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53240076b0 con 0x7f5318005190 2023-10-16T20:39:10.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.609+0000 7f5330808640 1 -- 172.21.15.88:0/2729724580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5318005190 msgr2=0x7f531800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:10.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.609+0000 7f5330808640 1 --2- 172.21.15.88:0/2729724580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5318005190 0x7f531800b990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f532400a830 tx=0x7f532401d340 comp rx=0 tx=0).stop 2023-10-16T20:39:10.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.609+0000 7f5330808640 1 -- 172.21.15.88:0/2729724580 shutdown_connections 2023-10-16T20:39:10.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.609+0000 7f5330808640 1 --2- 172.21.15.88:0/2729724580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5318005190 0x7f531800b990 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.609+0000 7f5330808640 1 -- 172.21.15.88:0/2729724580 >> 172.21.15.88:0/2729724580 conn(0x7f532c06bc00 msgr2=0x7f532c06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:10.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.609+0000 7f5330808640 1 -- 172.21.15.88:0/2729724580 shutdown_connections 2023-10-16T20:39:10.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.609+0000 7f5330808640 1 -- 172.21.15.88:0/2729724580 wait complete. 2023-10-16T20:39:10.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f5330808640 1 Processor -- start 2023-10-16T20:39:10.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f5330808640 1 -- start start 2023-10-16T20:39:10.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f5330808640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5318005190 0x7f532c14e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f5330808640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f532c0c8bc0 0x7f532c14d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f5330808640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f532c1eea60 0x7f532c1eeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f5330808640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c0c9830 con 0x7f532c1eea60 2023-10-16T20:39:10.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f5330808640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c150e00 con 0x7f532c0c8bc0 2023-10-16T20:39:10.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f5330808640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532c1ee630 con 0x7f5318005190 2023-10-16T20:39:10.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f532b59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f532c1eea60 0x7f532c1eeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:10.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f532b59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f532c1eea60 0x7f532c1eeeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52598/0 (socket says 172.21.15.88:52598) 2023-10-16T20:39:10.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f532b59e640 1 -- 172.21.15.88:0/421180051 learned_addr learned my addr 172.21.15.88:0/421180051 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:10.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f532ad9d640 1 --2- 172.21.15.88:0/421180051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5318005190 0x7f532c14e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:10.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f5323fff640 1 --2- 172.21.15.88:0/421180051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f532c0c8bc0 0x7f532c14d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:10.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f532b59e640 1 -- 172.21.15.88:0/421180051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5318005190 msgr2=0x7f532c14e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:10.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f532b59e640 1 --2- 172.21.15.88:0/421180051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5318005190 0x7f532c14e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f532b59e640 1 -- 172.21.15.88:0/421180051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f532c0c8bc0 msgr2=0x7f532c14d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:10.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f532b59e640 1 --2- 172.21.15.88:0/421180051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f532c0c8bc0 0x7f532c14d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f532b59e640 1 -- 172.21.15.88:0/421180051 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532c1ef3f0 con 0x7f532c1eea60 2023-10-16T20:39:10.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f532b59e640 1 --2- 172.21.15.88:0/421180051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f532c1eea60 0x7f532c1eeeb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f532401dbe0 tx=0x7f5318005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:10.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f53237fe640 1 -- 172.21.15.88:0/421180051 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53240019c0 con 0x7f532c1eea60 2023-10-16T20:39:10.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f53237fe640 1 -- 172.21.15.88:0/421180051 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5324007450 con 0x7f532c1eea60 2023-10-16T20:39:10.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f53237fe640 1 -- 172.21.15.88:0/421180051 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5324007800 con 0x7f532c1eea60 2023-10-16T20:39:10.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f5330808640 1 -- 172.21.15.88:0/421180051 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532c0c7dc0 con 0x7f532c1eea60 2023-10-16T20:39:10.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f5330808640 1 -- 172.21.15.88:0/421180051 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f532c151c90 con 0x7f532c1eea60 2023-10-16T20:39:10.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.613+0000 7f5330808640 1 -- 172.21.15.88:0/421180051 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f532c04e9c0 con 0x7f532c1eea60 2023-10-16T20:39:10.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.617+0000 7f53237fe640 1 -- 172.21.15.88:0/421180051 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f53240aa070 con 0x7f532c1eea60 2023-10-16T20:39:10.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.617+0000 7f53237fe640 1 --2- 172.21.15.88:0/421180051 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f530c0664a0 0x7f530c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.617+0000 7f53237fe640 1 -- 172.21.15.88:0/421180051 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f532411d9a0 con 0x7f532c1eea60 2023-10-16T20:39:10.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.617+0000 7f532ad9d640 1 --2- 172.21.15.88:0/421180051 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f530c0664a0 0x7f530c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:10.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.617+0000 7f532ad9d640 1 --2- 172.21.15.88:0/421180051 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f530c0664a0 0x7f530c068960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f531c001660 tx=0x7f531c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:10.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.621+0000 7f53237fe640 1 -- 172.21.15.88:0/421180051 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f53240eaa80 con 0x7f532c1eea60 2023-10-16T20:39:10.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.773+0000 7f5330808640 1 -- 172.21.15.88:0/421180051 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f532c06b2d0 con 0x7f532c1eea60 2023-10-16T20:39:10.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.773+0000 7f53237fe640 1 -- 172.21.15.88:0/421180051 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f53240ee0a0 con 0x7f532c1eea60 2023-10-16T20:39:10.781 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-10-16T20:39:10.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.773+0000 7f5330808640 1 -- 172.21.15.88:0/421180051 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f530c0664a0 msgr2=0x7f530c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:10.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.773+0000 7f5330808640 1 --2- 172.21.15.88:0/421180051 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f530c0664a0 0x7f530c068960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f531c001660 tx=0x7f531c009040 comp rx=0 tx=0).stop 2023-10-16T20:39:10.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.773+0000 7f5330808640 1 -- 172.21.15.88:0/421180051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f532c1eea60 msgr2=0x7f532c1eeeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:10.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.773+0000 7f5330808640 1 --2- 172.21.15.88:0/421180051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f532c1eea60 0x7f532c1eeeb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f532401dbe0 tx=0x7f5318005800 comp rx=0 tx=0).stop 2023-10-16T20:39:10.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.773+0000 7f5330808640 1 -- 172.21.15.88:0/421180051 shutdown_connections 2023-10-16T20:39:10.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.773+0000 7f5330808640 1 --2- 172.21.15.88:0/421180051 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f530c0664a0 0x7f530c068960 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.773+0000 7f5330808640 1 --2- 172.21.15.88:0/421180051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f532c1eea60 0x7f532c1eeeb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.773+0000 7f5330808640 1 --2- 172.21.15.88:0/421180051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f532c0c8bc0 0x7f532c14d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.773+0000 7f5330808640 1 --2- 172.21.15.88:0/421180051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5318005190 0x7f532c14e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.773+0000 7f5330808640 1 -- 172.21.15.88:0/421180051 >> 172.21.15.88:0/421180051 conn(0x7f532c06bc00 msgr2=0x7f532c0cbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:10.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.777+0000 7f5330808640 1 -- 172.21.15.88:0/421180051 shutdown_connections 2023-10-16T20:39:10.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.777+0000 7f5330808640 1 -- 172.21.15.88:0/421180051 wait complete. 2023-10-16T20:39:10.802 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T20:39:10.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.933+0000 7fa1c40ea640 1 Processor -- start 2023-10-16T20:39:10.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.937+0000 7fa1c40ea640 1 -- start start 2023-10-16T20:39:10.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.937+0000 7fa1c40ea640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1bc1537d0 0x7fa1bc15aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.937+0000 7fa1c40ea640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa1bc06dd70 0x7fa1bc15afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.937+0000 7fa1c40ea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc06cbe0 con 0x7fa1bc0cdb20 2023-10-16T20:39:10.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.937+0000 7fa1c40ea640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1bc06f5f0 con 0x7fa1bc06dd70 2023-10-16T20:39:10.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.937+0000 7fa1c40ea640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc15d8b0 con 0x7fa1bc1537d0 2023-10-16T20:39:10.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.937+0000 7fa1c1685640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa1bc0cdb20 0x7fa1bc0ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50406/0 (socket says 172.21.15.88:50406) 2023-10-16T20:39:10.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.937+0000 7fa1c1685640 1 -- 172.21.15.88:0/801431081 learned_addr learned my addr 172.21.15.88:0/801431081 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:10.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.937+0000 7fa1c1e86640 1 --2- 172.21.15.88:0/801431081 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1bc1537d0 0x7fa1bc15aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:10.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.937+0000 7fa1c2687640 1 --2- 172.21.15.88:0/801431081 >> v2:172.21.15.191:3300/0 conn(0x7fa1bc06dd70 0x7fa1bc15afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:10.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.937+0000 7fa1c0e84640 1 -- 172.21.15.88:0/801431081 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092079678 0 0) 0x7fa1bc06cbe0 con 0x7fa1bc0cdb20 2023-10-16T20:39:10.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.937+0000 7fa1c0e84640 1 -- 172.21.15.88:0/801431081 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1a8003530 con 0x7fa1bc0cdb20 2023-10-16T20:39:10.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.937+0000 7fa1c1e86640 1 -- 172.21.15.88:0/801431081 >> v2:172.21.15.191:3300/0 conn(0x7fa1bc06dd70 msgr2=0x7fa1bc15afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:10.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.937+0000 7fa1c1e86640 1 --2- 172.21.15.88:0/801431081 >> v2:172.21.15.191:3300/0 conn(0x7fa1bc06dd70 0x7fa1bc15afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.937+0000 7fa1c1e86640 1 -- 172.21.15.88:0/801431081 >> v1:172.21.15.88:6789/0 conn(0x7fa1bc0cdb20 legacy=0x7fa1bc0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:10.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.937+0000 7fa1c1e86640 1 -- 172.21.15.88:0/801431081 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc06e830 con 0x7fa1bc1537d0 2023-10-16T20:39:10.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.941+0000 7fa1c1e86640 1 --2- 172.21.15.88:0/801431081 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1bc1537d0 0x7fa1bc15aa60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa1b80097c0 tx=0x7fa1b801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dec4f7782f95a754 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:10.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.941+0000 7fa1c0e84640 1 -- 172.21.15.88:0/801431081 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b80ab600 con 0x7fa1bc1537d0 2023-10-16T20:39:10.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.941+0000 7fa1c0e84640 1 -- 172.21.15.88:0/801431081 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b80a7050 con 0x7fa1bc1537d0 2023-10-16T20:39:10.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.941+0000 7fa1c0e84640 1 -- 172.21.15.88:0/801431081 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b80ab980 con 0x7fa1bc1537d0 2023-10-16T20:39:10.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.941+0000 7fa1c40ea640 1 -- 172.21.15.88:0/801431081 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1bc1537d0 msgr2=0x7fa1bc15aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:10.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.941+0000 7fa1c40ea640 1 --2- 172.21.15.88:0/801431081 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1bc1537d0 0x7fa1bc15aa60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa1b80097c0 tx=0x7fa1b801c480 comp rx=0 tx=0).stop 2023-10-16T20:39:10.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.941+0000 7fa1c40ea640 1 -- 172.21.15.88:0/801431081 shutdown_connections 2023-10-16T20:39:10.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.941+0000 7fa1c40ea640 1 --2- 172.21.15.88:0/801431081 >> v2:172.21.15.191:3300/0 conn(0x7fa1bc06dd70 0x7fa1bc15afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.941+0000 7fa1c40ea640 1 --2- 172.21.15.88:0/801431081 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1bc1537d0 0x7fa1bc15aa60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.941+0000 7fa1c40ea640 1 -- 172.21.15.88:0/801431081 >> 172.21.15.88:0/801431081 conn(0x7fa1bc06f7d0 msgr2=0x7fa1bc06fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:10.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.941+0000 7fa1c40ea640 1 -- 172.21.15.88:0/801431081 shutdown_connections 2023-10-16T20:39:10.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.941+0000 7fa1c40ea640 1 -- 172.21.15.88:0/801431081 wait complete. 2023-10-16T20:39:10.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.941+0000 7fa1c40ea640 1 Processor -- start 2023-10-16T20:39:10.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c40ea640 1 -- start start 2023-10-16T20:39:10.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c40ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1bc06dd70 0x7fa1bc14ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c40ea640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1bc0cdb20 0x7fa1bc1f5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c40ea640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1bc1537d0 0x7fa1bc1f6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c40ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc1f5ba0 con 0x7fa1bc06dd70 2023-10-16T20:39:10.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c40ea640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc153bb0 con 0x7fa1bc1537d0 2023-10-16T20:39:10.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c40ea640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc153e00 con 0x7fa1bc0cdb20 2023-10-16T20:39:10.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c1e86640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1bc06dd70 0x7fa1bc14ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:10.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c1e86640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1bc06dd70 0x7fa1bc14ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52610/0 (socket says 172.21.15.88:52610) 2023-10-16T20:39:10.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c1e86640 1 -- 172.21.15.88:0/3527213639 learned_addr learned my addr 172.21.15.88:0/3527213639 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:10.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c1e86640 1 -- 172.21.15.88:0/3527213639 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1bc0cdb20 msgr2=0x7fa1bc1f5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:10.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c1e86640 1 --2- 172.21.15.88:0/3527213639 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1bc0cdb20 0x7fa1bc1f5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c1e86640 1 -- 172.21.15.88:0/3527213639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1bc1537d0 msgr2=0x7fa1bc1f6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:10.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c1e86640 1 --2- 172.21.15.88:0/3527213639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1bc1537d0 0x7fa1bc1f6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:10.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c1e86640 1 -- 172.21.15.88:0/3527213639 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b8009420 con 0x7fa1bc06dd70 2023-10-16T20:39:10.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c1e86640 1 --2- 172.21.15.88:0/3527213639 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1bc06dd70 0x7fa1bc14ce30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa1b801a0b0 tx=0x7fa1b8019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:10.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1b27fc640 1 -- 172.21.15.88:0/3527213639 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b80abe90 con 0x7fa1bc06dd70 2023-10-16T20:39:10.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1b27fc640 1 -- 172.21.15.88:0/3527213639 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b80b4400 con 0x7fa1bc06dd70 2023-10-16T20:39:10.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1b27fc640 1 -- 172.21.15.88:0/3527213639 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b80b4790 con 0x7fa1bc06dd70 2023-10-16T20:39:10.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c40ea640 1 -- 172.21.15.88:0/3527213639 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1bc1540e0 con 0x7fa1bc06dd70 2023-10-16T20:39:10.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c40ea640 1 -- 172.21.15.88:0/3527213639 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1bc1546a0 con 0x7fa1bc06dd70 2023-10-16T20:39:10.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.945+0000 7fa1c40ea640 1 -- 172.21.15.88:0/3527213639 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1bc1f5900 con 0x7fa1bc06dd70 2023-10-16T20:39:10.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.949+0000 7fa1b27fc640 1 -- 172.21.15.88:0/3527213639 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa1b80b2040 con 0x7fa1bc06dd70 2023-10-16T20:39:10.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.949+0000 7fa1b27fc640 1 --2- 172.21.15.88:0/3527213639 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa1a00664a0 0x7fa1a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:10.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.949+0000 7fa1b27fc640 1 -- 172.21.15.88:0/3527213639 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa1b811e180 con 0x7fa1bc06dd70 2023-10-16T20:39:10.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.949+0000 7fa1c1685640 1 --2- 172.21.15.88:0/3527213639 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa1a00664a0 0x7fa1a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:10.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.949+0000 7fa1c1685640 1 --2- 172.21.15.88:0/3527213639 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa1a00664a0 0x7fa1a0068960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac007fd0 tx=0x7fa1ac007940 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:10.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:10.953+0000 7fa1b27fc640 1 -- 172.21.15.88:0/3527213639 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa1b80eb1a0 con 0x7fa1bc06dd70 2023-10-16T20:39:11.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.101+0000 7fa1c40ea640 1 -- 172.21.15.88:0/3527213639 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa1bc074700 con 0x7fa1bc06dd70 2023-10-16T20:39:11.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.101+0000 7fa1b27fc640 1 -- 172.21.15.88:0/3527213639 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa1b80ee7c0 con 0x7fa1bc06dd70 2023-10-16T20:39:11.110 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:39:11.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.105+0000 7fa1c40ea640 1 -- 172.21.15.88:0/3527213639 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa1a00664a0 msgr2=0x7fa1a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.105+0000 7fa1c40ea640 1 --2- 172.21.15.88:0/3527213639 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa1a00664a0 0x7fa1a0068960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac007fd0 tx=0x7fa1ac007940 comp rx=0 tx=0).stop 2023-10-16T20:39:11.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.105+0000 7fa1c40ea640 1 -- 172.21.15.88:0/3527213639 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1bc06dd70 msgr2=0x7fa1bc14ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.105+0000 7fa1c40ea640 1 --2- 172.21.15.88:0/3527213639 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1bc06dd70 0x7fa1bc14ce30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa1b801a0b0 tx=0x7fa1b8019220 comp rx=0 tx=0).stop 2023-10-16T20:39:11.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.105+0000 7fa1c40ea640 1 -- 172.21.15.88:0/3527213639 shutdown_connections 2023-10-16T20:39:11.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.105+0000 7fa1c40ea640 1 --2- 172.21.15.88:0/3527213639 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa1a00664a0 0x7fa1a0068960 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.105+0000 7fa1c40ea640 1 --2- 172.21.15.88:0/3527213639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1bc1537d0 0x7fa1bc1f6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.105+0000 7fa1c40ea640 1 --2- 172.21.15.88:0/3527213639 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1bc0cdb20 0x7fa1bc1f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.105+0000 7fa1c40ea640 1 --2- 172.21.15.88:0/3527213639 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1bc06dd70 0x7fa1bc14ce30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.105+0000 7fa1c40ea640 1 -- 172.21.15.88:0/3527213639 >> 172.21.15.88:0/3527213639 conn(0x7fa1bc06f7d0 msgr2=0x7fa1bc14e420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:11.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.105+0000 7fa1c40ea640 1 -- 172.21.15.88:0/3527213639 shutdown_connections 2023-10-16T20:39:11.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.105+0000 7fa1c40ea640 1 -- 172.21.15.88:0/3527213639 wait complete. 2023-10-16T20:39:11.130 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T20:39:11.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee90bd9640 1 Processor -- start 2023-10-16T20:39:11.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee90bd9640 1 -- start start 2023-10-16T20:39:11.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee90bd9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fee8c0cdb20 0x7fee8c0ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee90bd9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee8c06cbe0 con 0x7fee8c06dd70 2023-10-16T20:39:11.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee90bd9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee8c15d8b0 con 0x7fee8c1537d0 2023-10-16T20:39:11.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee90bd9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee8c06f5f0 con 0x7fee8c0cdb20 2023-10-16T20:39:11.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee8ad9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fee8c06dd70 0x7fee8c163ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50412/0 (socket says 172.21.15.88:50412) 2023-10-16T20:39:11.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee8ad9d640 1 -- 172.21.15.88:0/3745367531 learned_addr learned my addr 172.21.15.88:0/3745367531 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:11.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee81d9b640 1 --2- 172.21.15.88:0/3745367531 >> v2:172.21.15.191:3300/0 conn(0x7fee8c0cdb20 0x7fee8c0ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee89d9b640 1 -- 172.21.15.88:0/3745367531 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846108619 0 0) 0x7fee8c06cbe0 con 0x7fee8c06dd70 2023-10-16T20:39:11.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee89d9b640 1 -- 172.21.15.88:0/3745367531 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee78003530 con 0x7fee8c06dd70 2023-10-16T20:39:11.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee89d9b640 1 -- 172.21.15.88:0/3745367531 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3995091127 0 0) 0x7fee78003530 con 0x7fee8c06dd70 2023-10-16T20:39:11.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee89d9b640 1 -- 172.21.15.88:0/3745367531 >> v2:172.21.15.191:3300/0 conn(0x7fee8c0cdb20 msgr2=0x7fee8c0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:11.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee89d9b640 1 --2- 172.21.15.88:0/3745367531 >> v2:172.21.15.191:3300/0 conn(0x7fee8c0cdb20 0x7fee8c0ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee89d9b640 1 -- 172.21.15.88:0/3745367531 >> v1:172.21.15.191:6789/0 conn(0x7fee8c1537d0 legacy=0x7fee8c15aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee89d9b640 1 -- 172.21.15.88:0/3745367531 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c06e830 con 0x7fee8c06dd70 2023-10-16T20:39:11.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee89d9b640 1 -- 172.21.15.88:0/3745367531 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7fee84001a00 con 0x7fee8c06dd70 2023-10-16T20:39:11.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee89d9b640 1 -- 172.21.15.88:0/3745367531 >> v1:172.21.15.88:6789/0 conn(0x7fee8c06dd70 legacy=0x7fee8c163ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee89d9b640 1 --2- 172.21.15.88:0/3745367531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee78003980 0x7fee78003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee89d9b640 1 --2- 172.21.15.88:0/3745367531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee780047d0 0x7fee78004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee89d9b640 1 --2- 172.21.15.88:0/3745367531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee78005190 0x7fee7800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee89d9b640 1 -- 172.21.15.88:0/3745367531 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c06f5f0 con 0x7fee78003980 2023-10-16T20:39:11.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee89d9b640 1 -- 172.21.15.88:0/3745367531 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7800c030 con 0x7fee780047d0 2023-10-16T20:39:11.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee89d9b640 1 -- 172.21.15.88:0/3745367531 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee7800c1f0 con 0x7fee78005190 2023-10-16T20:39:11.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee8a59c640 1 --2- 172.21.15.88:0/3745367531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee78003980 0x7fee78003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee81d9b640 1 --2- 172.21.15.88:0/3745367531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee780047d0 0x7fee78004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.265+0000 7fee8ad9d640 1 --2- 172.21.15.88:0/3745367531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee78005190 0x7fee7800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.269+0000 7fee8a59c640 1 -- 172.21.15.88:0/3745367531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee78005190 msgr2=0x7fee7800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:11.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.269+0000 7fee8a59c640 1 --2- 172.21.15.88:0/3745367531 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee78005190 0x7fee7800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.269+0000 7fee8a59c640 1 -- 172.21.15.88:0/3745367531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee780047d0 msgr2=0x7fee78004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:11.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.269+0000 7fee8a59c640 1 --2- 172.21.15.88:0/3745367531 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee780047d0 0x7fee78004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.269+0000 7fee8a59c640 1 -- 172.21.15.88:0/3745367531 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee78003530 con 0x7fee78003980 2023-10-16T20:39:11.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.269+0000 7fee8ad9d640 1 -- 172.21.15.88:0/3745367531 reap_dead start 2023-10-16T20:39:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.269+0000 7fee8a59c640 1 --2- 172.21.15.88:0/3745367531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee78003980 0x7fee78003da0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fee7c001780 tx=0x7fee7c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c20bbd561a17b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:11.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.269+0000 7fee89d9b640 1 -- 172.21.15.88:0/3745367531 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c0052d0 con 0x7fee78003980 2023-10-16T20:39:11.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.269+0000 7fee89d9b640 1 -- 172.21.15.88:0/3745367531 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7c0a7050 con 0x7fee78003980 2023-10-16T20:39:11.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.269+0000 7fee89d9b640 1 -- 172.21.15.88:0/3745367531 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c003600 con 0x7fee78003980 2023-10-16T20:39:11.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.273+0000 7fee90bd9640 1 -- 172.21.15.88:0/3745367531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee78003980 msgr2=0x7fee78003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.273+0000 7fee90bd9640 1 --2- 172.21.15.88:0/3745367531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee78003980 0x7fee78003da0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fee7c001780 tx=0x7fee7c01c3e0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.273+0000 7fee90bd9640 1 -- 172.21.15.88:0/3745367531 shutdown_connections 2023-10-16T20:39:11.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.273+0000 7fee90bd9640 1 --2- 172.21.15.88:0/3745367531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee78003980 0x7fee78003da0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.273+0000 7fee90bd9640 1 -- 172.21.15.88:0/3745367531 >> 172.21.15.88:0/3745367531 conn(0x7fee8c06f7d0 msgr2=0x7fee8c06fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:11.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.273+0000 7fee90bd9640 1 -- 172.21.15.88:0/3745367531 shutdown_connections 2023-10-16T20:39:11.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.273+0000 7fee90bd9640 1 -- 172.21.15.88:0/3745367531 wait complete. 2023-10-16T20:39:11.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.273+0000 7fee90bd9640 1 Processor -- start 2023-10-16T20:39:11.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.273+0000 7fee90bd9640 1 -- start start 2023-10-16T20:39:11.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.273+0000 7fee90bd9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee78003980 0x7fee8c148a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.273+0000 7fee90bd9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee8c0cbfe0 0x7fee8c150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.273+0000 7fee90bd9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee8c150d40 0x7fee8c13d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.273+0000 7fee90bd9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c147a90 con 0x7fee78003980 2023-10-16T20:39:11.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.273+0000 7fee90bd9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c150370 con 0x7fee8c150d40 2023-10-16T20:39:11.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.273+0000 7fee90bd9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c1482a0 con 0x7fee8c0cbfe0 2023-10-16T20:39:11.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee8a59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee78003980 0x7fee8c148a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee8a59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee78003980 0x7fee8c148a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52638/0 (socket says 172.21.15.88:52638) 2023-10-16T20:39:11.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee8a59c640 1 -- 172.21.15.88:0/972891945 learned_addr learned my addr 172.21.15.88:0/972891945 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:11.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee81d9b640 1 --2- 172.21.15.88:0/972891945 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee8c0cbfe0 0x7fee8c150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee8ad9d640 1 --2- 172.21.15.88:0/972891945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee8c150d40 0x7fee8c13d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee8a59c640 1 -- 172.21.15.88:0/972891945 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee8c0cbfe0 msgr2=0x7fee8c150800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee8a59c640 1 --2- 172.21.15.88:0/972891945 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee8c0cbfe0 0x7fee8c150800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee8a59c640 1 -- 172.21.15.88:0/972891945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee8c150d40 msgr2=0x7fee8c13d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee8a59c640 1 --2- 172.21.15.88:0/972891945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee8c150d40 0x7fee8c13d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee8a59c640 1 -- 172.21.15.88:0/972891945 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c147ce0 con 0x7fee78003980 2023-10-16T20:39:11.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee8a59c640 1 --2- 172.21.15.88:0/972891945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee78003980 0x7fee8c148a10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fee7c001780 tx=0x7fee78005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:11.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee837fe640 1 -- 172.21.15.88:0/972891945 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c0070c0 con 0x7fee78003980 2023-10-16T20:39:11.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee837fe640 1 -- 172.21.15.88:0/972891945 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7c007220 con 0x7fee78003980 2023-10-16T20:39:11.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee837fe640 1 -- 172.21.15.88:0/972891945 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c007590 con 0x7fee78003980 2023-10-16T20:39:11.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee90bd9640 1 -- 172.21.15.88:0/972891945 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8c147fc0 con 0x7fee78003980 2023-10-16T20:39:11.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee90bd9640 1 -- 172.21.15.88:0/972891945 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee8c150000 con 0x7fee78003980 2023-10-16T20:39:11.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee90bd9640 1 -- 172.21.15.88:0/972891945 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8c06e470 con 0x7fee78003980 2023-10-16T20:39:11.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.277+0000 7fee837fe640 1 -- 172.21.15.88:0/972891945 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fee7c0067e0 con 0x7fee78003980 2023-10-16T20:39:11.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.281+0000 7fee837fe640 1 --2- 172.21.15.88:0/972891945 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee6c0664d0 0x7fee6c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.281+0000 7fee837fe640 1 -- 172.21.15.88:0/972891945 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fee7c11e3d0 con 0x7fee78003980 2023-10-16T20:39:11.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.281+0000 7fee81d9b640 1 --2- 172.21.15.88:0/972891945 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee6c0664d0 0x7fee6c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.281+0000 7fee81d9b640 1 --2- 172.21.15.88:0/972891945 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee6c0664d0 0x7fee6c068990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fee8c1437f0 tx=0x7fee74009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:11.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.285+0000 7fee837fe640 1 -- 172.21.15.88:0/972891945 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fee7c0eb5b0 con 0x7fee78003980 2023-10-16T20:39:11.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.441+0000 7fee90bd9640 1 -- 172.21.15.88:0/972891945 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fee8c1528e0 con 0x7fee78003980 2023-10-16T20:39:11.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.441+0000 7fee837fe640 1 -- 172.21.15.88:0/972891945 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fee7c0eebd0 con 0x7fee78003980 2023-10-16T20:39:11.450 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2023-10-16T20:39:11.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.445+0000 7fee90bd9640 1 -- 172.21.15.88:0/972891945 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee6c0664d0 msgr2=0x7fee6c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.445+0000 7fee90bd9640 1 --2- 172.21.15.88:0/972891945 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee6c0664d0 0x7fee6c068990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fee8c1437f0 tx=0x7fee74009190 comp rx=0 tx=0).stop 2023-10-16T20:39:11.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.445+0000 7fee90bd9640 1 -- 172.21.15.88:0/972891945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee78003980 msgr2=0x7fee8c148a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.445+0000 7fee90bd9640 1 --2- 172.21.15.88:0/972891945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee78003980 0x7fee8c148a10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fee7c001780 tx=0x7fee78005800 comp rx=0 tx=0).stop 2023-10-16T20:39:11.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.445+0000 7fee90bd9640 1 -- 172.21.15.88:0/972891945 shutdown_connections 2023-10-16T20:39:11.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.445+0000 7fee90bd9640 1 --2- 172.21.15.88:0/972891945 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee6c0664d0 0x7fee6c068990 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.445+0000 7fee90bd9640 1 --2- 172.21.15.88:0/972891945 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee8c150d40 0x7fee8c13d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.445+0000 7fee90bd9640 1 --2- 172.21.15.88:0/972891945 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee8c0cbfe0 0x7fee8c150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.445+0000 7fee90bd9640 1 --2- 172.21.15.88:0/972891945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee78003980 0x7fee8c148a10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.445+0000 7fee90bd9640 1 -- 172.21.15.88:0/972891945 >> 172.21.15.88:0/972891945 conn(0x7fee8c06f7d0 msgr2=0x7fee8c0b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:11.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.445+0000 7fee90bd9640 1 -- 172.21.15.88:0/972891945 shutdown_connections 2023-10-16T20:39:11.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.445+0000 7fee90bd9640 1 -- 172.21.15.88:0/972891945 wait complete. 2023-10-16T20:39:11.470 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T20:39:11.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.605+0000 7f66eb446640 1 Processor -- start 2023-10-16T20:39:11.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.605+0000 7f66eb446640 1 -- start start 2023-10-16T20:39:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.605+0000 7f66eb446640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66e40c66e0 0x7f66e415a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.605+0000 7f66eb446640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f66e40c6ac0 0x7f66e4150150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.605+0000 7f66eb446640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66e4070f40 con 0x7f66e40cb5c0 2023-10-16T20:39:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.605+0000 7f66eb446640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e415a790 con 0x7f66e40c66e0 2023-10-16T20:39:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.605+0000 7f66eb446640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e406a2d0 con 0x7f66e40c6ac0 2023-10-16T20:39:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.605+0000 7f66e91e2640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66e40c66e0 0x7f66e415a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.605+0000 7f66e91e2640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66e40c66e0 0x7f66e415a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52648/0 (socket says 172.21.15.88:52648) 2023-10-16T20:39:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.605+0000 7f66e91e2640 1 -- 172.21.15.88:0/3822959401 learned_addr learned my addr 172.21.15.88:0/3822959401 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:11.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.605+0000 7f66e89e1640 1 --2- 172.21.15.88:0/3822959401 >> v2:172.21.15.191:3300/0 conn(0x7f66e40c6ac0 0x7f66e4150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.605+0000 7f66e91e2640 1 -- 172.21.15.88:0/3822959401 >> v2:172.21.15.191:3300/0 conn(0x7f66e40c6ac0 msgr2=0x7f66e4150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:11.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.605+0000 7f66e91e2640 1 --2- 172.21.15.88:0/3822959401 >> v2:172.21.15.191:3300/0 conn(0x7f66e40c6ac0 0x7f66e4150150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.605+0000 7f66e91e2640 1 -- 172.21.15.88:0/3822959401 >> v1:172.21.15.191:6789/0 conn(0x7f66e40cb5c0 legacy=0x7f66e4071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.605+0000 7f66e91e2640 1 -- 172.21.15.88:0/3822959401 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e406a5b0 con 0x7f66e40c66e0 2023-10-16T20:39:11.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.609+0000 7f66e91e2640 1 --2- 172.21.15.88:0/3822959401 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66e40c66e0 0x7f66e415a250 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f66e4073ed0 tx=0x7f66e001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d05a5e0e7a9b4f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:11.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.609+0000 7f66dbfff640 1 -- 172.21.15.88:0/3822959401 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e00ad660 con 0x7f66e40c66e0 2023-10-16T20:39:11.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.609+0000 7f66dbfff640 1 -- 172.21.15.88:0/3822959401 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e00ad7c0 con 0x7f66e40c66e0 2023-10-16T20:39:11.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.609+0000 7f66dbfff640 1 -- 172.21.15.88:0/3822959401 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e00adb50 con 0x7f66e40c66e0 2023-10-16T20:39:11.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.609+0000 7f66eb446640 1 -- 172.21.15.88:0/3822959401 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66e40c66e0 msgr2=0x7f66e415a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.609+0000 7f66eb446640 1 --2- 172.21.15.88:0/3822959401 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66e40c66e0 0x7f66e415a250 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f66e4073ed0 tx=0x7f66e001c470 comp rx=0 tx=0).stop 2023-10-16T20:39:11.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.609+0000 7f66eb446640 1 -- 172.21.15.88:0/3822959401 shutdown_connections 2023-10-16T20:39:11.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.609+0000 7f66eb446640 1 --2- 172.21.15.88:0/3822959401 >> v2:172.21.15.191:3300/0 conn(0x7f66e40c6ac0 0x7f66e4150150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.609+0000 7f66eb446640 1 --2- 172.21.15.88:0/3822959401 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66e40c66e0 0x7f66e415a250 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.609+0000 7f66eb446640 1 -- 172.21.15.88:0/3822959401 >> 172.21.15.88:0/3822959401 conn(0x7f66e40ca9f0 msgr2=0x7f66e40cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:11.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.609+0000 7f66eb446640 1 -- 172.21.15.88:0/3822959401 shutdown_connections 2023-10-16T20:39:11.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.609+0000 7f66eb446640 1 -- 172.21.15.88:0/3822959401 wait complete. 2023-10-16T20:39:11.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66eb446640 1 Processor -- start 2023-10-16T20:39:11.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66eb446640 1 -- start start 2023-10-16T20:39:11.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66eb446640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66e40c66e0 0x7f66e414f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66eb446640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66e40c6ac0 0x7f66e41490e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66eb446640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66e40cb5c0 0x7f66e4147350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66eb446640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e40c3040 con 0x7f66e40c66e0 2023-10-16T20:39:11.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66eb446640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e4148c50 con 0x7f66e40cb5c0 2023-10-16T20:39:11.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66eb446640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e414ceb0 con 0x7f66e40c6ac0 2023-10-16T20:39:11.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66e91e2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66e40c66e0 0x7f66e414f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66e91e2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66e40c66e0 0x7f66e414f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52662/0 (socket says 172.21.15.88:52662) 2023-10-16T20:39:11.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66e91e2640 1 -- 172.21.15.88:0/1457244364 learned_addr learned my addr 172.21.15.88:0/1457244364 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:11.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66e89e1640 1 --2- 172.21.15.88:0/1457244364 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66e40c6ac0 0x7f66e41490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66e99e3640 1 --2- 172.21.15.88:0/1457244364 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66e40cb5c0 0x7f66e4147350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66e91e2640 1 -- 172.21.15.88:0/1457244364 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66e40c6ac0 msgr2=0x7f66e41490e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66e91e2640 1 --2- 172.21.15.88:0/1457244364 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66e40c6ac0 0x7f66e41490e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66e91e2640 1 -- 172.21.15.88:0/1457244364 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66e40cb5c0 msgr2=0x7f66e4147350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66e91e2640 1 --2- 172.21.15.88:0/1457244364 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66e40cb5c0 0x7f66e4147350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66e91e2640 1 -- 172.21.15.88:0/1457244364 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e00094d0 con 0x7f66e40c66e0 2023-10-16T20:39:11.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66e99e3640 1 --2- 172.21.15.88:0/1457244364 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66e40cb5c0 0x7f66e4147350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:11.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66e89e1640 1 --2- 172.21.15.88:0/1457244364 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66e40c6ac0 0x7f66e41490e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:11.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66e91e2640 1 --2- 172.21.15.88:0/1457244364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66e40c66e0 0x7f66e414f1d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f66e001a0a0 tx=0x7f66e0019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:11.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66d9ffb640 1 -- 172.21.15.88:0/1457244364 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e00b6640 con 0x7f66e40c66e0 2023-10-16T20:39:11.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66d9ffb640 1 -- 172.21.15.88:0/1457244364 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e00b67a0 con 0x7f66e40c66e0 2023-10-16T20:39:11.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66d9ffb640 1 -- 172.21.15.88:0/1457244364 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e00b6b30 con 0x7f66e40c66e0 2023-10-16T20:39:11.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66eb446640 1 -- 172.21.15.88:0/1457244364 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e414d150 con 0x7f66e40c66e0 2023-10-16T20:39:11.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66eb446640 1 -- 172.21.15.88:0/1457244364 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e414d710 con 0x7f66e40c66e0 2023-10-16T20:39:11.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.613+0000 7f66eb446640 1 -- 172.21.15.88:0/1457244364 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66e4074700 con 0x7f66e40c66e0 2023-10-16T20:39:11.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.617+0000 7f66d9ffb640 1 -- 172.21.15.88:0/1457244364 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f66e00b4070 con 0x7f66e40c66e0 2023-10-16T20:39:11.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.617+0000 7f66d9ffb640 1 --2- 172.21.15.88:0/1457244364 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f66c80664d0 0x7f66c8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.617+0000 7f66d9ffb640 1 -- 172.21.15.88:0/1457244364 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f66e011e600 con 0x7f66e40c66e0 2023-10-16T20:39:11.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.617+0000 7f66e89e1640 1 --2- 172.21.15.88:0/1457244364 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f66c80664d0 0x7f66c8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.617+0000 7f66e89e1640 1 --2- 172.21.15.88:0/1457244364 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f66c80664d0 0x7f66c8068990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f66e4156e60 tx=0x7f66d4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:11.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.621+0000 7f66d9ffb640 1 -- 172.21.15.88:0/1457244364 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f66e00eb590 con 0x7f66e40c66e0 2023-10-16T20:39:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.769+0000 7f66eb446640 1 -- 172.21.15.88:0/1457244364 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f66e40715c0 con 0x7f66e40c66e0 2023-10-16T20:39:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.769+0000 7f66d9ffb640 1 -- 172.21.15.88:0/1457244364 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f66e00eebb0 con 0x7f66e40c66e0 2023-10-16T20:39:11.779 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:39:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.773+0000 7f66eb446640 1 -- 172.21.15.88:0/1457244364 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f66c80664d0 msgr2=0x7f66c8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.773+0000 7f66eb446640 1 --2- 172.21.15.88:0/1457244364 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f66c80664d0 0x7f66c8068990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f66e4156e60 tx=0x7f66d4008040 comp rx=0 tx=0).stop 2023-10-16T20:39:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.773+0000 7f66eb446640 1 -- 172.21.15.88:0/1457244364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66e40c66e0 msgr2=0x7f66e414f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.773+0000 7f66eb446640 1 --2- 172.21.15.88:0/1457244364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66e40c66e0 0x7f66e414f1d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f66e001a0a0 tx=0x7f66e0019210 comp rx=0 tx=0).stop 2023-10-16T20:39:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.773+0000 7f66eb446640 1 -- 172.21.15.88:0/1457244364 shutdown_connections 2023-10-16T20:39:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.773+0000 7f66eb446640 1 --2- 172.21.15.88:0/1457244364 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f66c80664d0 0x7f66c8068990 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.773+0000 7f66eb446640 1 --2- 172.21.15.88:0/1457244364 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f66e40cb5c0 0x7f66e4147350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.773+0000 7f66eb446640 1 --2- 172.21.15.88:0/1457244364 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f66e40c6ac0 0x7f66e41490e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.773+0000 7f66eb446640 1 --2- 172.21.15.88:0/1457244364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66e40c66e0 0x7f66e414f1d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.773+0000 7f66eb446640 1 -- 172.21.15.88:0/1457244364 >> 172.21.15.88:0/1457244364 conn(0x7f66e40ca9f0 msgr2=0x7f66e4159e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.773+0000 7f66eb446640 1 -- 172.21.15.88:0/1457244364 shutdown_connections 2023-10-16T20:39:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.773+0000 7f66eb446640 1 -- 172.21.15.88:0/1457244364 wait complete. 2023-10-16T20:39:11.799 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T20:39:11.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.929+0000 7fbd95009640 1 Processor -- start 2023-10-16T20:39:11.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.929+0000 7fbd95009640 1 -- start start 2023-10-16T20:39:11.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd95009640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd901537d0 0x7fbd9015aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd95009640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbd900cdb20 0x7fbd9015afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd95009640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbd9006dd70 0x7fbd900ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd95009640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd9006f5f0 con 0x7fbd900cdb20 2023-10-16T20:39:11.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd95009640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd9006cbe0 con 0x7fbd9006dd70 2023-10-16T20:39:11.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd95009640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9015d8b0 con 0x7fbd901537d0 2023-10-16T20:39:11.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd8e59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbd900cdb20 0x7fbd9015afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd8e59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbd900cdb20 0x7fbd9015afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52672/0 (socket says 172.21.15.88:52672) 2023-10-16T20:39:11.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd8e59c640 1 -- 172.21.15.88:0/47173135 learned_addr learned my addr 172.21.15.88:0/47173135 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:11.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd8f59e640 1 --2- 172.21.15.88:0/47173135 >> v2:172.21.15.191:3300/0 conn(0x7fbd9006dd70 0x7fbd900ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd8ed9d640 1 --2- 172.21.15.88:0/47173135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd901537d0 0x7fbd9015aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd8e59c640 1 -- 172.21.15.88:0/47173135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd901537d0 msgr2=0x7fbd9015aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:11.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd8e59c640 1 --2- 172.21.15.88:0/47173135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd901537d0 0x7fbd9015aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd8e59c640 1 -- 172.21.15.88:0/47173135 >> v2:172.21.15.191:3300/0 conn(0x7fbd9006dd70 msgr2=0x7fbd900ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:11.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd8e59c640 1 --2- 172.21.15.88:0/47173135 >> v2:172.21.15.191:3300/0 conn(0x7fbd9006dd70 0x7fbd900ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd8e59c640 1 -- 172.21.15.88:0/47173135 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9006e830 con 0x7fbd900cdb20 2023-10-16T20:39:11.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd8f59e640 1 --2- 172.21.15.88:0/47173135 >> v2:172.21.15.191:3300/0 conn(0x7fbd9006dd70 0x7fbd900ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:11.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd8ed9d640 1 --2- 172.21.15.88:0/47173135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd901537d0 0x7fbd9015aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd8e59c640 1 --2- 172.21.15.88:0/47173135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd900cdb20 0x7fbd9015afa0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbd9006e3e0 tx=0x7fbd8401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a1dcf716df9c88 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.933+0000 7fbd8dd9b640 1 -- 172.21.15.88:0/47173135 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd840ab600 con 0x7fbd900cdb20 2023-10-16T20:39:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd8dd9b640 1 -- 172.21.15.88:0/47173135 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd840a7050 con 0x7fbd900cdb20 2023-10-16T20:39:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd8dd9b640 1 -- 172.21.15.88:0/47173135 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd840ab980 con 0x7fbd900cdb20 2023-10-16T20:39:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd95009640 1 -- 172.21.15.88:0/47173135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd900cdb20 msgr2=0x7fbd9015afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd95009640 1 --2- 172.21.15.88:0/47173135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd900cdb20 0x7fbd9015afa0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbd9006e3e0 tx=0x7fbd8401c500 comp rx=0 tx=0).stop 2023-10-16T20:39:11.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd95009640 1 -- 172.21.15.88:0/47173135 shutdown_connections 2023-10-16T20:39:11.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd95009640 1 --2- 172.21.15.88:0/47173135 >> v2:172.21.15.191:3300/0 conn(0x7fbd9006dd70 0x7fbd900ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd95009640 1 --2- 172.21.15.88:0/47173135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd900cdb20 0x7fbd9015afa0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd95009640 1 --2- 172.21.15.88:0/47173135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd901537d0 0x7fbd9015aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd95009640 1 -- 172.21.15.88:0/47173135 >> 172.21.15.88:0/47173135 conn(0x7fbd9006f7d0 msgr2=0x7fbd9006fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:11.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd95009640 1 -- 172.21.15.88:0/47173135 shutdown_connections 2023-10-16T20:39:11.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd95009640 1 -- 172.21.15.88:0/47173135 wait complete. 2023-10-16T20:39:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd95009640 1 Processor -- start 2023-10-16T20:39:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd95009640 1 -- start start 2023-10-16T20:39:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd95009640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd9006dd70 0x7fbd90148770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd95009640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd900cdb20 0x7fbd90148cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd95009640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd901537d0 0x7fbd901557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd95009640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd901f2fe0 con 0x7fbd900cdb20 2023-10-16T20:39:11.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd95009640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd901f2180 con 0x7fbd9006dd70 2023-10-16T20:39:11.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd95009640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd901f2460 con 0x7fbd901537d0 2023-10-16T20:39:11.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd8e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd900cdb20 0x7fbd90148cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd8e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd900cdb20 0x7fbd90148cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52680/0 (socket says 172.21.15.88:52680) 2023-10-16T20:39:11.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd8e59c640 1 -- 172.21.15.88:0/4054550550 learned_addr learned my addr 172.21.15.88:0/4054550550 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:11.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd8ed9d640 1 --2- 172.21.15.88:0/4054550550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd9006dd70 0x7fbd90148770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.937+0000 7fbd8f59e640 1 --2- 172.21.15.88:0/4054550550 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd901537d0 0x7fbd901557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.941+0000 7fbd8e59c640 1 -- 172.21.15.88:0/4054550550 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd901537d0 msgr2=0x7fbd901557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.941+0000 7fbd8e59c640 1 --2- 172.21.15.88:0/4054550550 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd901537d0 0x7fbd901557e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.941+0000 7fbd8e59c640 1 -- 172.21.15.88:0/4054550550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd9006dd70 msgr2=0x7fbd90148770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.941+0000 7fbd8e59c640 1 --2- 172.21.15.88:0/4054550550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd9006dd70 0x7fbd90148770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.941+0000 7fbd8e59c640 1 -- 172.21.15.88:0/4054550550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd840094d0 con 0x7fbd900cdb20 2023-10-16T20:39:11.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.941+0000 7fbd8e59c640 1 --2- 172.21.15.88:0/4054550550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd900cdb20 0x7fbd90148cb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbd8401abb0 tx=0x7fbd9015a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:11.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.941+0000 7fbd777fe640 1 -- 172.21.15.88:0/4054550550 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd840b37d0 con 0x7fbd900cdb20 2023-10-16T20:39:11.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.941+0000 7fbd777fe640 1 -- 172.21.15.88:0/4054550550 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd840b3930 con 0x7fbd900cdb20 2023-10-16T20:39:11.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.941+0000 7fbd95009640 1 -- 172.21.15.88:0/4054550550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd901f2740 con 0x7fbd900cdb20 2023-10-16T20:39:11.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.941+0000 7fbd777fe640 1 -- 172.21.15.88:0/4054550550 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd840b3ce0 con 0x7fbd900cdb20 2023-10-16T20:39:11.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.941+0000 7fbd95009640 1 -- 172.21.15.88:0/4054550550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd901f2d00 con 0x7fbd900cdb20 2023-10-16T20:39:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.941+0000 7fbd95009640 1 -- 172.21.15.88:0/4054550550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd90074700 con 0x7fbd900cdb20 2023-10-16T20:39:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.941+0000 7fbd777fe640 1 -- 172.21.15.88:0/4054550550 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbd840b2070 con 0x7fbd900cdb20 2023-10-16T20:39:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.941+0000 7fbd777fe640 1 --2- 172.21.15.88:0/4054550550 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbd7c0664d0 0x7fbd7c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.945+0000 7fbd777fe640 1 -- 172.21.15.88:0/4054550550 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fbd8411e540 con 0x7fbd900cdb20 2023-10-16T20:39:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.945+0000 7fbd8ed9d640 1 --2- 172.21.15.88:0/4054550550 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbd7c0664d0 0x7fbd7c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.945+0000 7fbd8ed9d640 1 --2- 172.21.15.88:0/4054550550 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbd7c0664d0 0x7fbd7c068990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbd80006fd0 tx=0x7fbd80008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:11.945+0000 7fbd777fe640 1 -- 172.21.15.88:0/4054550550 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbd840eb580 con 0x7fbd900cdb20 2023-10-16T20:39:12.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.105+0000 7fbd95009640 1 -- 172.21.15.88:0/4054550550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbd9006e470 con 0x7fbd900cdb20 2023-10-16T20:39:12.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.105+0000 7fbd777fe640 1 -- 172.21.15.88:0/4054550550 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbd840eeba0 con 0x7fbd900cdb20 2023-10-16T20:39:12.115 INFO:teuthology.orchestra.run.smithi088.stderr:listed 0 entries 2023-10-16T20:39:12.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.109+0000 7fbd95009640 1 -- 172.21.15.88:0/4054550550 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbd7c0664d0 msgr2=0x7fbd7c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:12.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.109+0000 7fbd95009640 1 --2- 172.21.15.88:0/4054550550 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbd7c0664d0 0x7fbd7c068990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbd80006fd0 tx=0x7fbd80008040 comp rx=0 tx=0).stop 2023-10-16T20:39:12.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.109+0000 7fbd95009640 1 -- 172.21.15.88:0/4054550550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd900cdb20 msgr2=0x7fbd90148cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:12.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.109+0000 7fbd95009640 1 --2- 172.21.15.88:0/4054550550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd900cdb20 0x7fbd90148cb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbd8401abb0 tx=0x7fbd9015a600 comp rx=0 tx=0).stop 2023-10-16T20:39:12.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.109+0000 7fbd95009640 1 -- 172.21.15.88:0/4054550550 shutdown_connections 2023-10-16T20:39:12.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.109+0000 7fbd95009640 1 --2- 172.21.15.88:0/4054550550 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbd7c0664d0 0x7fbd7c068990 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.109+0000 7fbd95009640 1 --2- 172.21.15.88:0/4054550550 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbd901537d0 0x7fbd901557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.109+0000 7fbd95009640 1 --2- 172.21.15.88:0/4054550550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd900cdb20 0x7fbd90148cb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.109+0000 7fbd95009640 1 --2- 172.21.15.88:0/4054550550 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbd9006dd70 0x7fbd90148770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.109+0000 7fbd95009640 1 -- 172.21.15.88:0/4054550550 >> 172.21.15.88:0/4054550550 conn(0x7fbd9006f7d0 msgr2=0x7fbd90152490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:12.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.109+0000 7fbd95009640 1 -- 172.21.15.88:0/4054550550 shutdown_connections 2023-10-16T20:39:12.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.109+0000 7fbd95009640 1 -- 172.21.15.88:0/4054550550 wait complete. 2023-10-16T20:39:12.135 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:39:12.136 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T20:39:12.147 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:39:12.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.325+0000 7ff98df6e640 1 Processor -- start 2023-10-16T20:39:12.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.325+0000 7ff98df6e640 1 -- start start 2023-10-16T20:39:12.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.325+0000 7ff98df6e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff9880cb580 0x7ff9880cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:12.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.325+0000 7ff98df6e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9880b1a60 0x7ff988163b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:12.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.325+0000 7ff98df6e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9880c9fc0 con 0x7ff9880c9af0 2023-10-16T20:39:12.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.325+0000 7ff98df6e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9880b20c0 con 0x7ff9880cb580 2023-10-16T20:39:12.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.325+0000 7ff98df6e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9880cbea0 con 0x7ff9880b1a60 2023-10-16T20:39:12.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.325+0000 7ff98cf6c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff9880cb580 0x7ff9880cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:12.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.325+0000 7ff98cf6c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff9880cb580 0x7ff9880cb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52690/0 (socket says 172.21.15.88:52690) 2023-10-16T20:39:12.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.325+0000 7ff98cf6c640 1 -- 172.21.15.88:0/2786793578 learned_addr learned my addr 172.21.15.88:0/2786793578 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:12.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.325+0000 7ff987fff640 1 --1- 172.21.15.88:0/2786793578 >> v1:172.21.15.88:6789/0 conn(0x7ff9880c9af0 0x7ff988159300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50418/0 (socket says 172.21.15.88:50418) 2023-10-16T20:39:12.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.325+0000 7ff98d76d640 1 --2- 172.21.15.88:0/2786793578 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9880b1a60 0x7ff988163b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:12.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.325+0000 7ff98cf6c640 1 -- 172.21.15.88:0/2786793578 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9880b1a60 msgr2=0x7ff988163b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:12.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.325+0000 7ff98cf6c640 1 --2- 172.21.15.88:0/2786793578 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9880b1a60 0x7ff988163b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.325+0000 7ff98cf6c640 1 -- 172.21.15.88:0/2786793578 >> v1:172.21.15.88:6789/0 conn(0x7ff9880c9af0 legacy=0x7ff988159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:12.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.325+0000 7ff98cf6c640 1 -- 172.21.15.88:0/2786793578 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9880c47d0 con 0x7ff9880cb580 2023-10-16T20:39:12.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.329+0000 7ff98cf6c640 1 --2- 172.21.15.88:0/2786793578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9880cb580 0x7ff9880cb960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009870 tx=0x7ff97c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb9ef3bd32af8cfc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:12.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.329+0000 7ff9877fe640 1 -- 172.21.15.88:0/2786793578 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97c0ab680 con 0x7ff9880cb580 2023-10-16T20:39:12.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.329+0000 7ff9877fe640 1 -- 172.21.15.88:0/2786793578 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97c0ab7e0 con 0x7ff9880cb580 2023-10-16T20:39:12.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.329+0000 7ff9877fe640 1 -- 172.21.15.88:0/2786793578 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97c0abb70 con 0x7ff9880cb580 2023-10-16T20:39:12.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.329+0000 7ff98df6e640 1 -- 172.21.15.88:0/2786793578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9880cb580 msgr2=0x7ff9880cb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:12.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.329+0000 7ff98df6e640 1 --2- 172.21.15.88:0/2786793578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9880cb580 0x7ff9880cb960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009870 tx=0x7ff97c01c480 comp rx=0 tx=0).stop 2023-10-16T20:39:12.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.329+0000 7ff98df6e640 1 -- 172.21.15.88:0/2786793578 shutdown_connections 2023-10-16T20:39:12.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.329+0000 7ff98df6e640 1 --2- 172.21.15.88:0/2786793578 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9880b1a60 0x7ff988163b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.329+0000 7ff98df6e640 1 --2- 172.21.15.88:0/2786793578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9880cb580 0x7ff9880cb960 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.329+0000 7ff98df6e640 1 -- 172.21.15.88:0/2786793578 >> 172.21.15.88:0/2786793578 conn(0x7ff98806bb40 msgr2=0x7ff98806bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:12.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.329+0000 7ff98df6e640 1 -- 172.21.15.88:0/2786793578 shutdown_connections 2023-10-16T20:39:12.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.329+0000 7ff98df6e640 1 -- 172.21.15.88:0/2786793578 wait complete. 2023-10-16T20:39:12.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.329+0000 7ff98df6e640 1 Processor -- start 2023-10-16T20:39:12.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff98df6e640 1 -- start start 2023-10-16T20:39:12.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff98df6e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9880b1a60 0x7ff988147110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:12.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff98df6e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9880c9af0 0x7ff9881f5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:12.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff98df6e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9880cb580 0x7ff98814a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:12.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff98df6e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98814bb70 con 0x7ff9880c9af0 2023-10-16T20:39:12.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff98df6e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff988147650 con 0x7ff9880b1a60 2023-10-16T20:39:12.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff98df6e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98814d9b0 con 0x7ff9880cb580 2023-10-16T20:39:12.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff987fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9880c9af0 0x7ff9881f5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:12.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff987fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9880c9af0 0x7ff9881f5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52696/0 (socket says 172.21.15.88:52696) 2023-10-16T20:39:12.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff987fff640 1 -- 172.21.15.88:0/3998845469 learned_addr learned my addr 172.21.15.88:0/3998845469 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:12.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff98cf6c640 1 --2- 172.21.15.88:0/3998845469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9880b1a60 0x7ff988147110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:12.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff98d76d640 1 --2- 172.21.15.88:0/3998845469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9880cb580 0x7ff98814a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:12.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff987fff640 1 -- 172.21.15.88:0/3998845469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9880cb580 msgr2=0x7ff98814a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:12.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff987fff640 1 --2- 172.21.15.88:0/3998845469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9880cb580 0x7ff98814a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff987fff640 1 -- 172.21.15.88:0/3998845469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9880b1a60 msgr2=0x7ff988147110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:12.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff987fff640 1 --2- 172.21.15.88:0/3998845469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9880b1a60 0x7ff988147110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff987fff640 1 -- 172.21.15.88:0/3998845469 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97c0094d0 con 0x7ff9880c9af0 2023-10-16T20:39:12.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff987fff640 1 --2- 172.21.15.88:0/3998845469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9880c9af0 0x7ff9881f5ae0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff97400c500 tx=0x7ff97400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:12.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff9857fa640 1 -- 172.21.15.88:0/3998845469 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff974007500 con 0x7ff9880c9af0 2023-10-16T20:39:12.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff9857fa640 1 -- 172.21.15.88:0/3998845469 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff974007660 con 0x7ff9880c9af0 2023-10-16T20:39:12.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff9857fa640 1 -- 172.21.15.88:0/3998845469 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff974004a10 con 0x7ff9880c9af0 2023-10-16T20:39:12.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff98df6e640 1 -- 172.21.15.88:0/3998845469 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98814c5e0 con 0x7ff9880c9af0 2023-10-16T20:39:12.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff98df6e640 1 -- 172.21.15.88:0/3998845469 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98814cba0 con 0x7ff9880c9af0 2023-10-16T20:39:12.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.333+0000 7ff98df6e640 1 -- 172.21.15.88:0/3998845469 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9880ca2c0 con 0x7ff9880c9af0 2023-10-16T20:39:12.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.337+0000 7ff9857fa640 1 -- 172.21.15.88:0/3998845469 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff974004b70 con 0x7ff9880c9af0 2023-10-16T20:39:12.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.337+0000 7ff9857fa640 1 --2- 172.21.15.88:0/3998845469 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff96c0664a0 0x7ff96c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:12.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.337+0000 7ff9857fa640 1 -- 172.21.15.88:0/3998845469 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff9740833f0 con 0x7ff9880c9af0 2023-10-16T20:39:12.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.337+0000 7ff98cf6c640 1 --2- 172.21.15.88:0/3998845469 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff96c0664a0 0x7ff96c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:12.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.337+0000 7ff98cf6c640 1 --2- 172.21.15.88:0/3998845469 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff96c0664a0 0x7ff96c068960 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff97c0af000 tx=0x7ff97c0b6040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:12.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.341+0000 7ff9857fa640 1 -- 172.21.15.88:0/3998845469 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff974050550 con 0x7ff9880c9af0 2023-10-16T20:39:12.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.497+0000 7ff98df6e640 1 -- 172.21.15.88:0/3998845469 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff98804e9c0 con 0x7ff9880c9af0 2023-10-16T20:39:12.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.497+0000 7ff9857fa640 1 -- 172.21.15.88:0/3998845469 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18710 (secure 0 0 0) 0x7ff974053b70 con 0x7ff9880c9af0 2023-10-16T20:39:12.504 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:12.504 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:37:37.392549+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:39:12.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.497+0000 7ff98df6e640 1 -- 172.21.15.88:0/3998845469 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff96c0664a0 msgr2=0x7ff96c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:12.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.497+0000 7ff98df6e640 1 --2- 172.21.15.88:0/3998845469 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff96c0664a0 0x7ff96c068960 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff97c0af000 tx=0x7ff97c0b6040 comp rx=0 tx=0).stop 2023-10-16T20:39:12.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.497+0000 7ff98df6e640 1 -- 172.21.15.88:0/3998845469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9880c9af0 msgr2=0x7ff9881f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:12.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.497+0000 7ff98df6e640 1 --2- 172.21.15.88:0/3998845469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9880c9af0 0x7ff9881f5ae0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff97400c500 tx=0x7ff97400c9c0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.497+0000 7ff98df6e640 1 -- 172.21.15.88:0/3998845469 shutdown_connections 2023-10-16T20:39:12.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.497+0000 7ff98df6e640 1 --2- 172.21.15.88:0/3998845469 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff96c0664a0 0x7ff96c068960 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.497+0000 7ff98df6e640 1 --2- 172.21.15.88:0/3998845469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9880cb580 0x7ff98814a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.497+0000 7ff98df6e640 1 --2- 172.21.15.88:0/3998845469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9880c9af0 0x7ff9881f5ae0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.497+0000 7ff98df6e640 1 --2- 172.21.15.88:0/3998845469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9880b1a60 0x7ff988147110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.497+0000 7ff98df6e640 1 -- 172.21.15.88:0/3998845469 >> 172.21.15.88:0/3998845469 conn(0x7ff98806bb40 msgr2=0x7ff988158e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:12.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.497+0000 7ff98df6e640 1 -- 172.21.15.88:0/3998845469 shutdown_connections 2023-10-16T20:39:12.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.497+0000 7ff98df6e640 1 -- 172.21.15.88:0/3998845469 wait complete. 2023-10-16T20:39:12.526 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T20:36:43.646418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T20:36:53.546513+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T20:36:54.143703+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T20:36:57.314642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T20:39:12.527 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T20:39:12.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.661+0000 7f790f209640 1 Processor -- start 2023-10-16T20:39:12.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.661+0000 7f790f209640 1 -- start start 2023-10-16T20:39:12.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.665+0000 7f790f209640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f79081517e0 0x7f79081573e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:12.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.665+0000 7f790f209640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7908069e90 con 0x7f79080c4640 2023-10-16T20:39:12.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.665+0000 7f790f209640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790806aea0 con 0x7f79080c4260 2023-10-16T20:39:12.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.665+0000 7f790f209640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79080716a0 con 0x7f79081517e0 2023-10-16T20:39:12.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.665+0000 7f790d7a6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f79081517e0 0x7f79081573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:12.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.665+0000 7f78fffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f79080c4640 0x7f79081510d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50420/0 (socket says 172.21.15.88:50420) 2023-10-16T20:39:12.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.665+0000 7f78fffff640 1 -- 172.21.15.88:0/1512819675 learned_addr learned my addr 172.21.15.88:0/1512819675 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:12.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.665+0000 7f790d7a6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f79081517e0 0x7f79081573e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52706/0 (socket says 172.21.15.88:52706) 2023-10-16T20:39:12.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.665+0000 7f790d7a6640 1 -- 172.21.15.88:0/1512819675 >> v1:172.21.15.191:6789/0 conn(0x7f79080c4260 legacy=0x7f79080c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:12.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.665+0000 7f790d7a6640 1 -- 172.21.15.88:0/1512819675 >> v1:172.21.15.88:6789/0 conn(0x7f79080c4640 legacy=0x7f79081510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:12.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.665+0000 7f790d7a6640 1 -- 172.21.15.88:0/1512819675 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79081581e0 con 0x7f79081517e0 2023-10-16T20:39:12.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.665+0000 7f790d7a6640 1 --2- 172.21.15.88:0/1512819675 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79081517e0 0x7f79081573e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7900009d90 tx=0x7f790001c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98da4d11e48ce589 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:12.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.665+0000 7f78ff7fe640 1 -- 172.21.15.88:0/1512819675 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79000a58b0 con 0x7f79081517e0 2023-10-16T20:39:12.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.665+0000 7f78ff7fe640 1 -- 172.21.15.88:0/1512819675 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79000a5a10 con 0x7f79081517e0 2023-10-16T20:39:12.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.665+0000 7f78ff7fe640 1 -- 172.21.15.88:0/1512819675 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79000a5da0 con 0x7f79081517e0 2023-10-16T20:39:12.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790f209640 1 -- 172.21.15.88:0/1512819675 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79081517e0 msgr2=0x7f79081573e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:12.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790f209640 1 --2- 172.21.15.88:0/1512819675 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79081517e0 0x7f79081573e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7900009d90 tx=0x7f790001c7f0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790f209640 1 -- 172.21.15.88:0/1512819675 shutdown_connections 2023-10-16T20:39:12.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790f209640 1 --2- 172.21.15.88:0/1512819675 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79081517e0 0x7f79081573e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790f209640 1 -- 172.21.15.88:0/1512819675 >> 172.21.15.88:0/1512819675 conn(0x7f79080ca9f0 msgr2=0x7f79080cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:12.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790f209640 1 -- 172.21.15.88:0/1512819675 shutdown_connections 2023-10-16T20:39:12.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790f209640 1 -- 172.21.15.88:0/1512819675 wait complete. 2023-10-16T20:39:12.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790f209640 1 Processor -- start 2023-10-16T20:39:12.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790f209640 1 -- start start 2023-10-16T20:39:12.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790f209640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79080c4260 0x7f79081496f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:12.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790f209640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79080c4640 0x7f7908149c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:12.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790f209640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79081517e0 0x7f790814b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:12.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790f209640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79081601c0 con 0x7f79081517e0 2023-10-16T20:39:12.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790f209640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790814fd70 con 0x7f79080c4260 2023-10-16T20:39:12.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790f209640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7908149260 con 0x7f79080c4640 2023-10-16T20:39:12.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790d7a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79081517e0 0x7f790814b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:12.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790d7a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79081517e0 0x7f790814b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52720/0 (socket says 172.21.15.88:52720) 2023-10-16T20:39:12.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f790d7a6640 1 -- 172.21.15.88:0/2948250118 learned_addr learned my addr 172.21.15.88:0/2948250118 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:12.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.669+0000 7f78fffff640 1 --2- 172.21.15.88:0/2948250118 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79080c4640 0x7f7908149c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:12.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.673+0000 7f790d7a6640 1 -- 172.21.15.88:0/2948250118 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79080c4640 msgr2=0x7f7908149c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:12.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.673+0000 7f790d7a6640 1 --2- 172.21.15.88:0/2948250118 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79080c4640 0x7f7908149c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.673+0000 7f790cfa5640 1 --2- 172.21.15.88:0/2948250118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79080c4260 0x7f79081496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:12.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.673+0000 7f790d7a6640 1 -- 172.21.15.88:0/2948250118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79080c4260 msgr2=0x7f79081496f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:12.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.673+0000 7f790d7a6640 1 --2- 172.21.15.88:0/2948250118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79080c4260 0x7f79081496f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.673+0000 7f790d7a6640 1 -- 172.21.15.88:0/2948250118 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7900009a70 con 0x7f79081517e0 2023-10-16T20:39:12.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.673+0000 7f78fffff640 1 --2- 172.21.15.88:0/2948250118 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79080c4640 0x7f7908149c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:12.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.673+0000 7f790cfa5640 1 --2- 172.21.15.88:0/2948250118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79080c4260 0x7f79081496f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:12.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.673+0000 7f790d7a6640 1 --2- 172.21.15.88:0/2948250118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79081517e0 0x7f790814b1a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f790001a420 tx=0x7f7900019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:12.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.673+0000 7f78fd7fa640 1 -- 172.21.15.88:0/2948250118 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79000ad650 con 0x7f79081517e0 2023-10-16T20:39:12.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.673+0000 7f78fd7fa640 1 -- 172.21.15.88:0/2948250118 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79000ad7b0 con 0x7f79081517e0 2023-10-16T20:39:12.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.673+0000 7f78fd7fa640 1 -- 172.21.15.88:0/2948250118 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79000adb40 con 0x7f79081517e0 2023-10-16T20:39:12.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.673+0000 7f790f209640 1 -- 172.21.15.88:0/2948250118 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79080c3e40 con 0x7f79081517e0 2023-10-16T20:39:12.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.673+0000 7f790f209640 1 -- 172.21.15.88:0/2948250118 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79080c3880 con 0x7f79081517e0 2023-10-16T20:39:12.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.673+0000 7f790f209640 1 -- 172.21.15.88:0/2948250118 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79081493a0 con 0x7f79081517e0 2023-10-16T20:39:12.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.673+0000 7f78fd7fa640 1 -- 172.21.15.88:0/2948250118 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f79000adca0 con 0x7f79081517e0 2023-10-16T20:39:12.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.673+0000 7f78fd7fa640 1 --2- 172.21.15.88:0/2948250118 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f78f40664a0 0x7f78f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:12.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.677+0000 7f78fd7fa640 1 -- 172.21.15.88:0/2948250118 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f790011f560 con 0x7f79081517e0 2023-10-16T20:39:12.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.677+0000 7f790cfa5640 1 --2- 172.21.15.88:0/2948250118 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f78f40664a0 0x7f78f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:12.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.677+0000 7f790cfa5640 1 --2- 172.21.15.88:0/2948250118 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f78f40664a0 0x7f78f4068960 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7904003e40 tx=0x7f7904009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:12.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.677+0000 7f78fd7fa640 1 -- 172.21.15.88:0/2948250118 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f79000ec580 con 0x7f79081517e0 2023-10-16T20:39:12.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.829+0000 7f790f209640 1 -- 172.21.15.88:0/2948250118 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f79080c6e80 con 0x7f79081517e0 2023-10-16T20:39:12.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.829+0000 7f78fd7fa640 1 -- 172.21.15.88:0/2948250118 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f79000bc020 con 0x7f79081517e0 2023-10-16T20:39:12.838 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-10-16T20:39:12.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.833+0000 7f790f209640 1 -- 172.21.15.88:0/2948250118 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f78f40664a0 msgr2=0x7f78f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:12.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.833+0000 7f790f209640 1 --2- 172.21.15.88:0/2948250118 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f78f40664a0 0x7f78f4068960 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7904003e40 tx=0x7f7904009040 comp rx=0 tx=0).stop 2023-10-16T20:39:12.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.833+0000 7f790f209640 1 -- 172.21.15.88:0/2948250118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79081517e0 msgr2=0x7f790814b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:12.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.833+0000 7f790f209640 1 --2- 172.21.15.88:0/2948250118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79081517e0 0x7f790814b1a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f790001a420 tx=0x7f7900019590 comp rx=0 tx=0).stop 2023-10-16T20:39:12.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.833+0000 7f790f209640 1 -- 172.21.15.88:0/2948250118 shutdown_connections 2023-10-16T20:39:12.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.833+0000 7f790f209640 1 --2- 172.21.15.88:0/2948250118 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f78f40664a0 0x7f78f4068960 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.833+0000 7f790f209640 1 --2- 172.21.15.88:0/2948250118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79081517e0 0x7f790814b1a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.833+0000 7f790f209640 1 --2- 172.21.15.88:0/2948250118 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f79080c4640 0x7f7908149c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.833+0000 7f790f209640 1 --2- 172.21.15.88:0/2948250118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f79080c4260 0x7f79081496f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:12.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.833+0000 7f790f209640 1 -- 172.21.15.88:0/2948250118 >> 172.21.15.88:0/2948250118 conn(0x7f79080ca9f0 msgr2=0x7f79080c2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:12.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.833+0000 7f790f209640 1 -- 172.21.15.88:0/2948250118 shutdown_connections 2023-10-16T20:39:12.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.833+0000 7f790f209640 1 -- 172.21.15.88:0/2948250118 wait complete. 2023-10-16T20:39:12.858 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T20:39:12.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.993+0000 7ff61490e640 1 Processor -- start 2023-10-16T20:39:13.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.993+0000 7ff61490e640 1 -- start start 2023-10-16T20:39:13.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.993+0000 7ff61490e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6101537d0 0x7ff61015aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.993+0000 7ff61490e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61006cbe0 con 0x7ff6100cdb20 2023-10-16T20:39:13.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.993+0000 7ff61490e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61015d8b0 con 0x7ff61006dd70 2023-10-16T20:39:13.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.993+0000 7ff61490e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61006f5f0 con 0x7ff6101537d0 2023-10-16T20:39:13.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.993+0000 7ff605d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff6100cdb20 0x7ff6100ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50430/0 (socket says 172.21.15.88:50430) 2023-10-16T20:39:13.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.993+0000 7ff605d9b640 1 -- 172.21.15.88:0/266083390 learned_addr learned my addr 172.21.15.88:0/266083390 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:13.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.993+0000 7ff60e59c640 1 --2- 172.21.15.88:0/266083390 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6101537d0 0x7ff61015aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.993+0000 7ff60dd9b640 1 -- 172.21.15.88:0/266083390 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216124958 0 0) 0x7ff61006cbe0 con 0x7ff6100cdb20 2023-10-16T20:39:13.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.993+0000 7ff60dd9b640 1 -- 172.21.15.88:0/266083390 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5fc003530 con 0x7ff6100cdb20 2023-10-16T20:39:13.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.993+0000 7ff60dd9b640 1 -- 172.21.15.88:0/266083390 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515838966 0 0) 0x7ff61015d8b0 con 0x7ff61006dd70 2023-10-16T20:39:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.993+0000 7ff60dd9b640 1 -- 172.21.15.88:0/266083390 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff61006cbe0 con 0x7ff61006dd70 2023-10-16T20:39:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.993+0000 7ff60e59c640 1 -- 172.21.15.88:0/266083390 >> v1:172.21.15.191:6789/0 conn(0x7ff61006dd70 legacy=0x7ff610163ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.993+0000 7ff60dd9b640 1 -- 172.21.15.88:0/266083390 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3335627632 0 0) 0x7ff5fc003530 con 0x7ff6100cdb20 2023-10-16T20:39:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.993+0000 7ff60e59c640 1 -- 172.21.15.88:0/266083390 >> v1:172.21.15.88:6789/0 conn(0x7ff6100cdb20 legacy=0x7ff6100ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.993+0000 7ff60e59c640 1 -- 172.21.15.88:0/266083390 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61006e830 con 0x7ff6101537d0 2023-10-16T20:39:13.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.997+0000 7ff60e59c640 1 --2- 172.21.15.88:0/266083390 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6101537d0 0x7ff61015aa60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff6000097f0 tx=0x7ff60001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdfc86c1a4150c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:13.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.997+0000 7ff60dd9b640 1 -- 172.21.15.88:0/266083390 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8001a00 con 0x7ff6101537d0 2023-10-16T20:39:13.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.997+0000 7ff60dd9b640 1 -- 172.21.15.88:0/266083390 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6000a7050 con 0x7ff6101537d0 2023-10-16T20:39:13.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.997+0000 7ff60dd9b640 1 -- 172.21.15.88:0/266083390 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6000ab620 con 0x7ff6101537d0 2023-10-16T20:39:13.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.997+0000 7ff61490e640 1 -- 172.21.15.88:0/266083390 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6101537d0 msgr2=0x7ff61015aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.997+0000 7ff61490e640 1 --2- 172.21.15.88:0/266083390 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6101537d0 0x7ff61015aa60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff6000097f0 tx=0x7ff60001c3e0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.997+0000 7ff61490e640 1 -- 172.21.15.88:0/266083390 shutdown_connections 2023-10-16T20:39:13.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.997+0000 7ff61490e640 1 --2- 172.21.15.88:0/266083390 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6101537d0 0x7ff61015aa60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.997+0000 7ff61490e640 1 -- 172.21.15.88:0/266083390 >> 172.21.15.88:0/266083390 conn(0x7ff61006f7d0 msgr2=0x7ff61006fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:13.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.997+0000 7ff61490e640 1 -- 172.21.15.88:0/266083390 shutdown_connections 2023-10-16T20:39:13.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:12.997+0000 7ff61490e640 1 -- 172.21.15.88:0/266083390 wait complete. 2023-10-16T20:39:13.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff61490e640 1 Processor -- start 2023-10-16T20:39:13.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff61490e640 1 -- start start 2023-10-16T20:39:13.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff61490e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff61006dd70 0x7ff610146e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff61490e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6100cdb20 0x7ff61014e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff61490e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6101537d0 0x7ff61014e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff61490e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61014fe40 con 0x7ff6101537d0 2023-10-16T20:39:13.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff61490e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6101f5980 con 0x7ff61006dd70 2023-10-16T20:39:13.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff61490e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6101f5c60 con 0x7ff6100cdb20 2023-10-16T20:39:13.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff60ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6101537d0 0x7ff61014e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff60ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6101537d0 0x7ff61014e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52722/0 (socket says 172.21.15.88:52722) 2023-10-16T20:39:13.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff60ed9d640 1 -- 172.21.15.88:0/114293744 learned_addr learned my addr 172.21.15.88:0/114293744 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:13.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff605d9b640 1 --2- 172.21.15.88:0/114293744 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6100cdb20 0x7ff61014e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff60e59c640 1 --2- 172.21.15.88:0/114293744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff61006dd70 0x7ff610146e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff60ed9d640 1 -- 172.21.15.88:0/114293744 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6100cdb20 msgr2=0x7ff61014e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff60ed9d640 1 --2- 172.21.15.88:0/114293744 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6100cdb20 0x7ff61014e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff60ed9d640 1 -- 172.21.15.88:0/114293744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff61006dd70 msgr2=0x7ff610146e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff60ed9d640 1 --2- 172.21.15.88:0/114293744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff61006dd70 0x7ff610146e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff60ed9d640 1 -- 172.21.15.88:0/114293744 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff600009420 con 0x7ff6101537d0 2023-10-16T20:39:13.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff60ed9d640 1 --2- 172.21.15.88:0/114293744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6101537d0 0x7ff61014e680 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff60800c410 tx=0x7ff60800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:13.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff6077fe640 1 -- 172.21.15.88:0/114293744 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6080052d0 con 0x7ff6101537d0 2023-10-16T20:39:13.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff6077fe640 1 -- 172.21.15.88:0/114293744 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff608005430 con 0x7ff6101537d0 2023-10-16T20:39:13.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff6077fe640 1 -- 172.21.15.88:0/114293744 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608003800 con 0x7ff6101537d0 2023-10-16T20:39:13.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff61490e640 1 -- 172.21.15.88:0/114293744 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6101f5f40 con 0x7ff6101537d0 2023-10-16T20:39:13.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.001+0000 7ff61490e640 1 -- 172.21.15.88:0/114293744 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff61014dc20 con 0x7ff6101537d0 2023-10-16T20:39:13.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.005+0000 7ff61490e640 1 -- 172.21.15.88:0/114293744 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6101f6460 con 0x7ff6101537d0 2023-10-16T20:39:13.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.005+0000 7ff6077fe640 1 -- 172.21.15.88:0/114293744 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff608003960 con 0x7ff6101537d0 2023-10-16T20:39:13.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.005+0000 7ff6077fe640 1 --2- 172.21.15.88:0/114293744 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff5f00664a0 0x7ff5f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.005+0000 7ff6077fe640 1 -- 172.21.15.88:0/114293744 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff6080837c0 con 0x7ff6101537d0 2023-10-16T20:39:13.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.005+0000 7ff60e59c640 1 --2- 172.21.15.88:0/114293744 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff5f00664a0 0x7ff5f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.005+0000 7ff60e59c640 1 --2- 172.21.15.88:0/114293744 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff5f00664a0 0x7ff5f0068960 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff6000ad000 tx=0x7ff6000a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:13.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.009+0000 7ff6077fe640 1 -- 172.21.15.88:0/114293744 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff6101f6460 con 0x7ff6101537d0 2023-10-16T20:39:13.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.161+0000 7ff61490e640 1 -- 172.21.15.88:0/114293744 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff6101f6460 con 0x7ff6101537d0 2023-10-16T20:39:13.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.165+0000 7ff6077fe640 1 -- 172.21.15.88:0/114293744 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff6101f6460 con 0x7ff6101537d0 2023-10-16T20:39:13.172 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:39:13.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.165+0000 7ff61490e640 1 -- 172.21.15.88:0/114293744 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff5f00664a0 msgr2=0x7ff5f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.165+0000 7ff61490e640 1 --2- 172.21.15.88:0/114293744 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff5f00664a0 0x7ff5f0068960 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff6000ad000 tx=0x7ff6000a6000 comp rx=0 tx=0).stop 2023-10-16T20:39:13.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.165+0000 7ff61490e640 1 -- 172.21.15.88:0/114293744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6101537d0 msgr2=0x7ff61014e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.165+0000 7ff61490e640 1 --2- 172.21.15.88:0/114293744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6101537d0 0x7ff61014e680 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff60800c410 tx=0x7ff60800c8d0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.165+0000 7ff61490e640 1 -- 172.21.15.88:0/114293744 shutdown_connections 2023-10-16T20:39:13.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.165+0000 7ff61490e640 1 --2- 172.21.15.88:0/114293744 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff5f00664a0 0x7ff5f0068960 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.165+0000 7ff61490e640 1 --2- 172.21.15.88:0/114293744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6101537d0 0x7ff61014e680 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.165+0000 7ff61490e640 1 --2- 172.21.15.88:0/114293744 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6100cdb20 0x7ff61014e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.165+0000 7ff61490e640 1 --2- 172.21.15.88:0/114293744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff61006dd70 0x7ff610146e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.165+0000 7ff61490e640 1 -- 172.21.15.88:0/114293744 >> 172.21.15.88:0/114293744 conn(0x7ff61006f7d0 msgr2=0x7ff610151e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:13.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.165+0000 7ff61490e640 1 -- 172.21.15.88:0/114293744 shutdown_connections 2023-10-16T20:39:13.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.165+0000 7ff61490e640 1 -- 172.21.15.88:0/114293744 wait complete. 2023-10-16T20:39:13.191 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T20:39:13.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.325+0000 7fb0a9fc5640 1 Processor -- start 2023-10-16T20:39:13.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.325+0000 7fb0a9fc5640 1 -- start start 2023-10-16T20:39:13.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.325+0000 7fb0a9fc5640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb0a4159490 0x7fb0a40c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.325+0000 7fb0a9fc5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a4159d90 con 0x7fb0a40c5930 2023-10-16T20:39:13.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.325+0000 7fb0a9fc5640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a415a070 con 0x7fb0a4153780 2023-10-16T20:39:13.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.325+0000 7fb0a9fc5640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a415a350 con 0x7fb0a4159490 2023-10-16T20:39:13.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.325+0000 7fb0a3fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb0a40c5930 0x7fb0a40ced80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50434/0 (socket says 172.21.15.88:50434) 2023-10-16T20:39:13.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.325+0000 7fb0a3fff640 1 -- 172.21.15.88:0/925731928 learned_addr learned my addr 172.21.15.88:0/925731928 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:13.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.325+0000 7fb0a2ffd640 1 --2- 172.21.15.88:0/925731928 >> v2:172.21.15.191:3300/0 conn(0x7fb0a4159490 0x7fb0a40c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.325+0000 7fb0a27fc640 1 -- 172.21.15.88:0/925731928 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290206013 0 0) 0x7fb0a4159d90 con 0x7fb0a40c5930 2023-10-16T20:39:13.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.325+0000 7fb0a27fc640 1 -- 172.21.15.88:0/925731928 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb08c003530 con 0x7fb0a40c5930 2023-10-16T20:39:13.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.325+0000 7fb0a27fc640 1 -- 172.21.15.88:0/925731928 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794491659 0 0) 0x7fb0a415a070 con 0x7fb0a4153780 2023-10-16T20:39:13.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.329+0000 7fb0a27fc640 1 -- 172.21.15.88:0/925731928 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0a4159d90 con 0x7fb0a4153780 2023-10-16T20:39:13.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.329+0000 7fb0a2ffd640 1 -- 172.21.15.88:0/925731928 >> v1:172.21.15.191:6789/0 conn(0x7fb0a4153780 legacy=0x7fb0a40c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.329+0000 7fb0a2ffd640 1 -- 172.21.15.88:0/925731928 >> v1:172.21.15.88:6789/0 conn(0x7fb0a40c5930 legacy=0x7fb0a40ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.329+0000 7fb0a2ffd640 1 -- 172.21.15.88:0/925731928 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a40c4910 con 0x7fb0a4159490 2023-10-16T20:39:13.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.329+0000 7fb0a2ffd640 1 --2- 172.21.15.88:0/925731928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0a4159490 0x7fb0a40c53f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb0980097c0 tx=0x7fb09801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2173f34581018834 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:13.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.329+0000 7fb0a27fc640 1 -- 172.21.15.88:0/925731928 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0980a8970 con 0x7fb0a4159490 2023-10-16T20:39:13.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.329+0000 7fb0a27fc640 1 -- 172.21.15.88:0/925731928 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0980a8ad0 con 0x7fb0a4159490 2023-10-16T20:39:13.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.329+0000 7fb0a27fc640 1 -- 172.21.15.88:0/925731928 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0980a8e60 con 0x7fb0a4159490 2023-10-16T20:39:13.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.329+0000 7fb0a9fc5640 1 -- 172.21.15.88:0/925731928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0a4159490 msgr2=0x7fb0a40c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.329+0000 7fb0a9fc5640 1 --2- 172.21.15.88:0/925731928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0a4159490 0x7fb0a40c53f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb0980097c0 tx=0x7fb09801ec70 comp rx=0 tx=0).stop 2023-10-16T20:39:13.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.329+0000 7fb0a9fc5640 1 -- 172.21.15.88:0/925731928 shutdown_connections 2023-10-16T20:39:13.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.329+0000 7fb0a9fc5640 1 --2- 172.21.15.88:0/925731928 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0a4159490 0x7fb0a40c53f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.329+0000 7fb0a9fc5640 1 -- 172.21.15.88:0/925731928 >> 172.21.15.88:0/925731928 conn(0x7fb0a406bcb0 msgr2=0x7fb0a406c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:13.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.329+0000 7fb0a9fc5640 1 -- 172.21.15.88:0/925731928 shutdown_connections 2023-10-16T20:39:13.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a9fc5640 1 -- 172.21.15.88:0/925731928 wait complete. 2023-10-16T20:39:13.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a9fc5640 1 Processor -- start 2023-10-16T20:39:13.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a9fc5640 1 -- start start 2023-10-16T20:39:13.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a9fc5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0a40c5930 0x7fb0a4152980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a9fc5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0a4153780 0x7fb0a4152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a9fc5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0a4159490 0x7fb0a414a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a9fc5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a414ecc0 con 0x7fb0a4159490 2023-10-16T20:39:13.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a9fc5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a414a470 con 0x7fb0a40c5930 2023-10-16T20:39:13.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a9fc5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a414dad0 con 0x7fb0a4153780 2023-10-16T20:39:13.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0a4159490 0x7fb0a414a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0a4159490 0x7fb0a414a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52730/0 (socket says 172.21.15.88:52730) 2023-10-16T20:39:13.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a3fff640 1 -- 172.21.15.88:0/357805307 learned_addr learned my addr 172.21.15.88:0/357805307 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:13.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a37fe640 1 --2- 172.21.15.88:0/357805307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0a40c5930 0x7fb0a4152980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a2ffd640 1 --2- 172.21.15.88:0/357805307 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0a4153780 0x7fb0a4152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a3fff640 1 -- 172.21.15.88:0/357805307 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0a4153780 msgr2=0x7fb0a4152030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a3fff640 1 --2- 172.21.15.88:0/357805307 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0a4153780 0x7fb0a4152030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a3fff640 1 -- 172.21.15.88:0/357805307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0a40c5930 msgr2=0x7fb0a4152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a3fff640 1 --2- 172.21.15.88:0/357805307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0a40c5930 0x7fb0a4152980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a3fff640 1 -- 172.21.15.88:0/357805307 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb098009420 con 0x7fb0a4159490 2023-10-16T20:39:13.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a37fe640 1 --2- 172.21.15.88:0/357805307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0a40c5930 0x7fb0a4152980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:13.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a3fff640 1 --2- 172.21.15.88:0/357805307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0a4159490 0x7fb0a414a900 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00cb90 tx=0x7fb09c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:13.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb08bfff640 1 -- 172.21.15.88:0/357805307 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c00ad40 con 0x7fb0a4159490 2023-10-16T20:39:13.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb08bfff640 1 -- 172.21.15.88:0/357805307 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c012400 con 0x7fb0a4159490 2023-10-16T20:39:13.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb08bfff640 1 -- 172.21.15.88:0/357805307 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c00aea0 con 0x7fb0a4159490 2023-10-16T20:39:13.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.333+0000 7fb0a9fc5640 1 -- 172.21.15.88:0/357805307 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a414ddb0 con 0x7fb0a4159490 2023-10-16T20:39:13.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.337+0000 7fb0a9fc5640 1 -- 172.21.15.88:0/357805307 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a414e370 con 0x7fb0a4159490 2023-10-16T20:39:13.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.337+0000 7fb0a9fc5640 1 -- 172.21.15.88:0/357805307 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a414a5e0 con 0x7fb0a4159490 2023-10-16T20:39:13.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.337+0000 7fb08bfff640 1 -- 172.21.15.88:0/357805307 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb09c007070 con 0x7fb0a4159490 2023-10-16T20:39:13.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.337+0000 7fb08bfff640 1 --2- 172.21.15.88:0/357805307 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb0900664a0 0x7fb090068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.337+0000 7fb08bfff640 1 -- 172.21.15.88:0/357805307 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb09c084180 con 0x7fb0a4159490 2023-10-16T20:39:13.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.337+0000 7fb0a37fe640 1 --2- 172.21.15.88:0/357805307 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb0900664a0 0x7fb090068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.337+0000 7fb0a37fe640 1 --2- 172.21.15.88:0/357805307 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb0900664a0 0x7fb090068960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb094001240 tx=0x7fb094009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:13.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.341+0000 7fb08bfff640 1 -- 172.21.15.88:0/357805307 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb09c051250 con 0x7fb0a4159490 2023-10-16T20:39:13.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.493+0000 7fb0a9fc5640 1 -- 172.21.15.88:0/357805307 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb0a4159ad0 con 0x7fb0a4159490 2023-10-16T20:39:13.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.493+0000 7fb08bfff640 1 -- 172.21.15.88:0/357805307 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb09c054870 con 0x7fb0a4159490 2023-10-16T20:39:13.501 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2023-10-16T20:39:13.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.493+0000 7fb0a9fc5640 1 -- 172.21.15.88:0/357805307 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb0900664a0 msgr2=0x7fb090068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.493+0000 7fb0a9fc5640 1 --2- 172.21.15.88:0/357805307 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb0900664a0 0x7fb090068960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb094001240 tx=0x7fb094009040 comp rx=0 tx=0).stop 2023-10-16T20:39:13.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.493+0000 7fb0a9fc5640 1 -- 172.21.15.88:0/357805307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0a4159490 msgr2=0x7fb0a414a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.493+0000 7fb0a9fc5640 1 --2- 172.21.15.88:0/357805307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0a4159490 0x7fb0a414a900 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00cb90 tx=0x7fb09c00a430 comp rx=0 tx=0).stop 2023-10-16T20:39:13.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.493+0000 7fb0a9fc5640 1 -- 172.21.15.88:0/357805307 shutdown_connections 2023-10-16T20:39:13.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.493+0000 7fb0a9fc5640 1 --2- 172.21.15.88:0/357805307 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb0900664a0 0x7fb090068960 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.493+0000 7fb0a9fc5640 1 --2- 172.21.15.88:0/357805307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0a4159490 0x7fb0a414a900 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.493+0000 7fb0a9fc5640 1 --2- 172.21.15.88:0/357805307 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0a4153780 0x7fb0a4152030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.493+0000 7fb0a9fc5640 1 --2- 172.21.15.88:0/357805307 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0a40c5930 0x7fb0a4152980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.493+0000 7fb0a9fc5640 1 -- 172.21.15.88:0/357805307 >> 172.21.15.88:0/357805307 conn(0x7fb0a406bcb0 msgr2=0x7fb0a4149af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:13.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.497+0000 7fb0a9fc5640 1 -- 172.21.15.88:0/357805307 shutdown_connections 2023-10-16T20:39:13.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.497+0000 7fb0a9fc5640 1 -- 172.21.15.88:0/357805307 wait complete. 2023-10-16T20:39:13.522 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T20:39:13.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc8baa640 1 Processor -- start 2023-10-16T20:39:13.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc8baa640 1 -- start start 2023-10-16T20:39:13.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc8baa640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bc4071450 0x7f1bc40c1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc8baa640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1bc4071f90 0x7f1bc4072400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc8baa640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc41596e0 con 0x7f1bc4153a30 2023-10-16T20:39:13.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc8baa640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bc40c6cc0 con 0x7f1bc4071f90 2023-10-16T20:39:13.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc8baa640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc40c2da0 con 0x7f1bc4071450 2023-10-16T20:39:13.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc259c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1bc4153a30 0x7f1bc40c65b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50438/0 (socket says 172.21.15.88:50438) 2023-10-16T20:39:13.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc259c640 1 -- 172.21.15.88:0/2564140669 learned_addr learned my addr 172.21.15.88:0/2564140669 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:13.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc1d9b640 1 --2- 172.21.15.88:0/2564140669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bc4071450 0x7f1bc40c1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc2d9d640 1 --2- 172.21.15.88:0/2564140669 >> v2:172.21.15.191:3300/0 conn(0x7f1bc4071f90 0x7f1bc4072400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc159a640 1 -- 172.21.15.88:0/2564140669 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063573559 0 0) 0x7f1bc41596e0 con 0x7f1bc4153a30 2023-10-16T20:39:13.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc159a640 1 -- 172.21.15.88:0/2564140669 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bac003530 con 0x7f1bc4153a30 2023-10-16T20:39:13.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc159a640 1 -- 172.21.15.88:0/2564140669 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4007626385 0 0) 0x7f1bac003530 con 0x7f1bc4153a30 2023-10-16T20:39:13.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc159a640 1 -- 172.21.15.88:0/2564140669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bc4071450 msgr2=0x7f1bc40c1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:13.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc159a640 1 --2- 172.21.15.88:0/2564140669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bc4071450 0x7f1bc40c1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc159a640 1 -- 172.21.15.88:0/2564140669 >> v2:172.21.15.191:3300/0 conn(0x7f1bc4071f90 msgr2=0x7f1bc4072400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:13.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc159a640 1 --2- 172.21.15.88:0/2564140669 >> v2:172.21.15.191:3300/0 conn(0x7f1bc4071f90 0x7f1bc4072400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc159a640 1 -- 172.21.15.88:0/2564140669 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc40c1bb0 con 0x7f1bc4153a30 2023-10-16T20:39:13.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc159a640 1 -- 172.21.15.88:0/2564140669 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f1bb4001680 con 0x7f1bc4153a30 2023-10-16T20:39:13.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc159a640 1 -- 172.21.15.88:0/2564140669 >> v1:172.21.15.88:6789/0 conn(0x7f1bc4153a30 legacy=0x7f1bc40c65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc159a640 1 --2- 172.21.15.88:0/2564140669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac003980 0x7f1bac003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc159a640 1 --2- 172.21.15.88:0/2564140669 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bac0047d0 0x7f1bac004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc159a640 1 --2- 172.21.15.88:0/2564140669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bc4071450 0x7f1bac005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc159a640 1 -- 172.21.15.88:0/2564140669 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc40c6cc0 con 0x7f1bac003980 2023-10-16T20:39:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc159a640 1 -- 172.21.15.88:0/2564140669 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc40c2da0 con 0x7f1bac0047d0 2023-10-16T20:39:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc159a640 1 -- 172.21.15.88:0/2564140669 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac00b990 con 0x7f1bc4071450 2023-10-16T20:39:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc259c640 1 --2- 172.21.15.88:0/2564140669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac003980 0x7f1bac003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc1d9b640 1 --2- 172.21.15.88:0/2564140669 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bac0047d0 0x7f1bac004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc2d9d640 1 --2- 172.21.15.88:0/2564140669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bc4071450 0x7f1bac005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc259c640 1 -- 172.21.15.88:0/2564140669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bc4071450 msgr2=0x7f1bac005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:13.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc259c640 1 --2- 172.21.15.88:0/2564140669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bc4071450 0x7f1bac005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc259c640 1 -- 172.21.15.88:0/2564140669 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bac0047d0 msgr2=0x7f1bac004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:13.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc259c640 1 --2- 172.21.15.88:0/2564140669 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bac0047d0 0x7f1bac004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.657+0000 7f1bc259c640 1 -- 172.21.15.88:0/2564140669 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bac003530 con 0x7f1bac003980 2023-10-16T20:39:13.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.661+0000 7f1bc259c640 1 --2- 172.21.15.88:0/2564140669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac003980 0x7f1bac003da0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1bb400c240 tx=0x7f1bb401cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7aa88b49b78e7f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:13.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.661+0000 7f1bc159a640 1 -- 172.21.15.88:0/2564140669 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb40058e0 con 0x7f1bac003980 2023-10-16T20:39:13.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.661+0000 7f1bc159a640 1 -- 172.21.15.88:0/2564140669 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb4005a40 con 0x7f1bac003980 2023-10-16T20:39:13.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.661+0000 7f1bc159a640 1 -- 172.21.15.88:0/2564140669 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb4005dd0 con 0x7f1bac003980 2023-10-16T20:39:13.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.661+0000 7f1bc8baa640 1 -- 172.21.15.88:0/2564140669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac003980 msgr2=0x7f1bac003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.661+0000 7f1bc8baa640 1 --2- 172.21.15.88:0/2564140669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac003980 0x7f1bac003da0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1bb400c240 tx=0x7f1bb401cc30 comp rx=0 tx=0).stop 2023-10-16T20:39:13.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.661+0000 7f1bc2d9d640 1 -- 172.21.15.88:0/2564140669 reap_dead start 2023-10-16T20:39:13.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.661+0000 7f1bc8baa640 1 -- 172.21.15.88:0/2564140669 shutdown_connections 2023-10-16T20:39:13.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.661+0000 7f1bc8baa640 1 -- 172.21.15.88:0/2564140669 >> 172.21.15.88:0/2564140669 conn(0x7f1bc40c86b0 msgr2=0x7f1bc40c8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:13.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.661+0000 7f1bc8baa640 1 -- 172.21.15.88:0/2564140669 shutdown_connections 2023-10-16T20:39:13.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.661+0000 7f1bc8baa640 1 -- 172.21.15.88:0/2564140669 wait complete. 2023-10-16T20:39:13.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc8baa640 1 Processor -- start 2023-10-16T20:39:13.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc8baa640 1 -- start start 2023-10-16T20:39:13.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc8baa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac003980 0x7f1bc41f2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc8baa640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bc40c2f10 0x7f1bc40c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc8baa640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bc414dce0 0x7f1bc414e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc8baa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc41f2c90 con 0x7f1bac003980 2023-10-16T20:39:13.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc8baa640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc414e6b0 con 0x7f1bc40c2f10 2023-10-16T20:39:13.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc8baa640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc40c33c0 con 0x7f1bc414dce0 2023-10-16T20:39:13.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac003980 0x7f1bc41f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac003980 0x7f1bc41f2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52762/0 (socket says 172.21.15.88:52762) 2023-10-16T20:39:13.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc259c640 1 -- 172.21.15.88:0/1808014604 learned_addr learned my addr 172.21.15.88:0/1808014604 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:13.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc1d9b640 1 --2- 172.21.15.88:0/1808014604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bc40c2f10 0x7f1bc40c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc2d9d640 1 --2- 172.21.15.88:0/1808014604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bc414dce0 0x7f1bc414e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc259c640 1 -- 172.21.15.88:0/1808014604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bc414dce0 msgr2=0x7f1bc414e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc259c640 1 --2- 172.21.15.88:0/1808014604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bc414dce0 0x7f1bc414e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc259c640 1 -- 172.21.15.88:0/1808014604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bc40c2f10 msgr2=0x7f1bc40c5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc259c640 1 --2- 172.21.15.88:0/1808014604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bc40c2f10 0x7f1bc40c5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc259c640 1 -- 172.21.15.88:0/1808014604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc40c36a0 con 0x7f1bac003980 2023-10-16T20:39:13.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc259c640 1 --2- 172.21.15.88:0/1808014604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac003980 0x7f1bc41f2750 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1bb40ab040 tx=0x7f1bac005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:13.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bb2ffd640 1 -- 172.21.15.88:0/1808014604 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb4005e30 con 0x7f1bac003980 2023-10-16T20:39:13.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bb2ffd640 1 -- 172.21.15.88:0/1808014604 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb40047f0 con 0x7f1bac003980 2023-10-16T20:39:13.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bc8baa640 1 -- 172.21.15.88:0/1808014604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc40c3980 con 0x7f1bac003980 2023-10-16T20:39:13.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.665+0000 7f1bb2ffd640 1 -- 172.21.15.88:0/1808014604 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb4004b60 con 0x7f1bac003980 2023-10-16T20:39:13.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.669+0000 7f1bc8baa640 1 -- 172.21.15.88:0/1808014604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc414d4e0 con 0x7f1bac003980 2023-10-16T20:39:13.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.669+0000 7f1bc8baa640 1 -- 172.21.15.88:0/1808014604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc40c9240 con 0x7f1bac003980 2023-10-16T20:39:13.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.669+0000 7f1bb2ffd640 1 -- 172.21.15.88:0/1808014604 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1bb4004cc0 con 0x7f1bac003980 2023-10-16T20:39:13.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.669+0000 7f1bb2ffd640 1 --2- 172.21.15.88:0/1808014604 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1ba40664d0 0x7f1ba4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.669+0000 7f1bb2ffd640 1 -- 172.21.15.88:0/1808014604 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f1bb411cf80 con 0x7f1bac003980 2023-10-16T20:39:13.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.669+0000 7f1bc1d9b640 1 --2- 172.21.15.88:0/1808014604 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1ba40664d0 0x7f1ba4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.669+0000 7f1bc1d9b640 1 --2- 172.21.15.88:0/1808014604 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1ba40664d0 0x7f1ba4068990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1bc40c32f0 tx=0x7f1bb8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:13.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.673+0000 7f1bb2ffd640 1 -- 172.21.15.88:0/1808014604 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1bb40e9f10 con 0x7f1bac003980 2023-10-16T20:39:13.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.821+0000 7f1bc8baa640 1 -- 172.21.15.88:0/1808014604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1bc4071230 con 0x7f1bac003980 2023-10-16T20:39:13.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.821+0000 7f1bb2ffd640 1 -- 172.21.15.88:0/1808014604 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1bb40ed530 con 0x7f1bac003980 2023-10-16T20:39:13.831 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:39:13.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.825+0000 7f1bc8baa640 1 -- 172.21.15.88:0/1808014604 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1ba40664d0 msgr2=0x7f1ba4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.825+0000 7f1bc8baa640 1 --2- 172.21.15.88:0/1808014604 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1ba40664d0 0x7f1ba4068990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1bc40c32f0 tx=0x7f1bb8009190 comp rx=0 tx=0).stop 2023-10-16T20:39:13.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.825+0000 7f1bc8baa640 1 -- 172.21.15.88:0/1808014604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac003980 msgr2=0x7f1bc41f2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.825+0000 7f1bc8baa640 1 --2- 172.21.15.88:0/1808014604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac003980 0x7f1bc41f2750 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1bb40ab040 tx=0x7f1bac005800 comp rx=0 tx=0).stop 2023-10-16T20:39:13.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.825+0000 7f1bc8baa640 1 -- 172.21.15.88:0/1808014604 shutdown_connections 2023-10-16T20:39:13.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.825+0000 7f1bc8baa640 1 --2- 172.21.15.88:0/1808014604 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1ba40664d0 0x7f1ba4068990 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.825+0000 7f1bc8baa640 1 --2- 172.21.15.88:0/1808014604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bc414dce0 0x7f1bc414e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.825+0000 7f1bc8baa640 1 --2- 172.21.15.88:0/1808014604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bc40c2f10 0x7f1bc40c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.825+0000 7f1bc8baa640 1 --2- 172.21.15.88:0/1808014604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac003980 0x7f1bc41f2750 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.825+0000 7f1bc8baa640 1 -- 172.21.15.88:0/1808014604 >> 172.21.15.88:0/1808014604 conn(0x7f1bc40c86b0 msgr2=0x7f1bc414c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:13.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.825+0000 7f1bc8baa640 1 -- 172.21.15.88:0/1808014604 shutdown_connections 2023-10-16T20:39:13.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.825+0000 7f1bc8baa640 1 -- 172.21.15.88:0/1808014604 wait complete. 2023-10-16T20:39:13.851 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T20:39:13.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.981+0000 7fee93f1d640 1 Processor -- start 2023-10-16T20:39:13.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.981+0000 7fee93f1d640 1 -- start start 2023-10-16T20:39:13.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.985+0000 7fee93f1d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fee8c1537d0 0x7fee8c15aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.985+0000 7fee93f1d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fee8c06dd70 0x7fee8c15afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:13.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.985+0000 7fee93f1d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee8c06cbe0 con 0x7fee8c0cdb20 2023-10-16T20:39:13.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.985+0000 7fee93f1d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee8c06f5f0 con 0x7fee8c1537d0 2023-10-16T20:39:13.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.985+0000 7fee93f1d640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee8c15d8b0 con 0x7fee8c06dd70 2023-10-16T20:39:13.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.985+0000 7fee91cb9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fee8c1537d0 0x7fee8c15aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.985+0000 7fee91cb9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fee8c1537d0 0x7fee8c15aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52772/0 (socket says 172.21.15.88:52772) 2023-10-16T20:39:13.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.985+0000 7fee91cb9640 1 -- 172.21.15.88:0/3210314054 learned_addr learned my addr 172.21.15.88:0/3210314054 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:13.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.985+0000 7fee924ba640 1 --2- 172.21.15.88:0/3210314054 >> v2:172.21.15.191:3300/0 conn(0x7fee8c06dd70 0x7fee8c15afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:13.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.985+0000 7fee91cb9640 1 -- 172.21.15.88:0/3210314054 >> v2:172.21.15.191:3300/0 conn(0x7fee8c06dd70 msgr2=0x7fee8c15afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:13.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.985+0000 7fee91cb9640 1 --2- 172.21.15.88:0/3210314054 >> v2:172.21.15.191:3300/0 conn(0x7fee8c06dd70 0x7fee8c15afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.985+0000 7fee91cb9640 1 -- 172.21.15.88:0/3210314054 >> v1:172.21.15.191:6789/0 conn(0x7fee8c0cdb20 legacy=0x7fee8c0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.985+0000 7fee91cb9640 1 -- 172.21.15.88:0/3210314054 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c06e830 con 0x7fee8c1537d0 2023-10-16T20:39:13.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee91cb9640 1 --2- 172.21.15.88:0/3210314054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee8c1537d0 0x7fee8c15aa60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fee8c0b1ae0 tx=0x7fee8801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb2fc6f72ea3494e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:13.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee90cb7640 1 -- 172.21.15.88:0/3210314054 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee880ab5f0 con 0x7fee8c1537d0 2023-10-16T20:39:13.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee90cb7640 1 -- 172.21.15.88:0/3210314054 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee880ab750 con 0x7fee8c1537d0 2023-10-16T20:39:13.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee90cb7640 1 -- 172.21.15.88:0/3210314054 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee880abae0 con 0x7fee8c1537d0 2023-10-16T20:39:13.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee93f1d640 1 -- 172.21.15.88:0/3210314054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee8c1537d0 msgr2=0x7fee8c15aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:13.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee93f1d640 1 --2- 172.21.15.88:0/3210314054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee8c1537d0 0x7fee8c15aa60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fee8c0b1ae0 tx=0x7fee8801c500 comp rx=0 tx=0).stop 2023-10-16T20:39:13.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee93f1d640 1 -- 172.21.15.88:0/3210314054 shutdown_connections 2023-10-16T20:39:13.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee93f1d640 1 --2- 172.21.15.88:0/3210314054 >> v2:172.21.15.191:3300/0 conn(0x7fee8c06dd70 0x7fee8c15afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee93f1d640 1 --2- 172.21.15.88:0/3210314054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee8c1537d0 0x7fee8c15aa60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:13.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee93f1d640 1 -- 172.21.15.88:0/3210314054 >> 172.21.15.88:0/3210314054 conn(0x7fee8c06f7d0 msgr2=0x7fee8c06fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:13.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee93f1d640 1 -- 172.21.15.88:0/3210314054 shutdown_connections 2023-10-16T20:39:13.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee93f1d640 1 -- 172.21.15.88:0/3210314054 wait complete. 2023-10-16T20:39:14.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee93f1d640 1 Processor -- start 2023-10-16T20:39:14.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee93f1d640 1 -- start start 2023-10-16T20:39:14.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee93f1d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee8c06dd70 0x7fee8c1f2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee93f1d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee8c0cdb20 0x7fee8c1f4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee93f1d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee8c1537d0 0x7fee8c1f52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee93f1d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c155ce0 con 0x7fee8c06dd70 2023-10-16T20:39:14.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee93f1d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c156820 con 0x7fee8c0cdb20 2023-10-16T20:39:14.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.989+0000 7fee93f1d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c1f2f80 con 0x7fee8c1537d0 2023-10-16T20:39:14.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee91cb9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee8c06dd70 0x7fee8c1f2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee91cb9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee8c06dd70 0x7fee8c1f2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52776/0 (socket says 172.21.15.88:52776) 2023-10-16T20:39:14.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee91cb9640 1 -- 172.21.15.88:0/3454662648 learned_addr learned my addr 172.21.15.88:0/3454662648 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:14.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee914b8640 1 --2- 172.21.15.88:0/3454662648 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee8c0cdb20 0x7fee8c1f4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee924ba640 1 --2- 172.21.15.88:0/3454662648 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee8c1537d0 0x7fee8c1f52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee91cb9640 1 -- 172.21.15.88:0/3454662648 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee8c1537d0 msgr2=0x7fee8c1f52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:14.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee91cb9640 1 --2- 172.21.15.88:0/3454662648 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee8c1537d0 0x7fee8c1f52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee91cb9640 1 -- 172.21.15.88:0/3454662648 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee8c0cdb20 msgr2=0x7fee8c1f4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:14.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee91cb9640 1 --2- 172.21.15.88:0/3454662648 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee8c0cdb20 0x7fee8c1f4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee91cb9640 1 -- 172.21.15.88:0/3454662648 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee880094d0 con 0x7fee8c06dd70 2023-10-16T20:39:14.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee924ba640 1 --2- 172.21.15.88:0/3454662648 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee8c1537d0 0x7fee8c1f52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:14.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee91cb9640 1 --2- 172.21.15.88:0/3454662648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee8c06dd70 0x7fee8c1f2a40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fee88019110 tx=0x7fee8801ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:14.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee827fc640 1 -- 172.21.15.88:0/3454662648 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee880be5f0 con 0x7fee8c06dd70 2023-10-16T20:39:14.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee827fc640 1 -- 172.21.15.88:0/3454662648 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee880be770 con 0x7fee8c06dd70 2023-10-16T20:39:14.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee827fc640 1 -- 172.21.15.88:0/3454662648 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee880beb20 con 0x7fee8c06dd70 2023-10-16T20:39:14.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee93f1d640 1 -- 172.21.15.88:0/3454662648 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8c15ce40 con 0x7fee8c06dd70 2023-10-16T20:39:14.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee93f1d640 1 -- 172.21.15.88:0/3454662648 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee8c1561d0 con 0x7fee8c06dd70 2023-10-16T20:39:14.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee93f1d640 1 -- 172.21.15.88:0/3454662648 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8c15d120 con 0x7fee8c06dd70 2023-10-16T20:39:14.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.993+0000 7fee827fc640 1 -- 172.21.15.88:0/3454662648 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fee880b4600 con 0x7fee8c06dd70 2023-10-16T20:39:14.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.997+0000 7fee827fc640 1 --2- 172.21.15.88:0/3454662648 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee700664a0 0x7fee70068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.997+0000 7fee827fc640 1 -- 172.21.15.88:0/3454662648 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fee880a8050 con 0x7fee8c06dd70 2023-10-16T20:39:14.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.997+0000 7fee914b8640 1 --2- 172.21.15.88:0/3454662648 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee700664a0 0x7fee70068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:13.997+0000 7fee914b8640 1 --2- 172.21.15.88:0/3454662648 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee700664a0 0x7fee70068960 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fee8c14e0f0 tx=0x7fee7c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:14.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.001+0000 7fee827fc640 1 -- 172.21.15.88:0/3454662648 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fee880ec000 con 0x7fee8c06dd70 2023-10-16T20:39:14.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.157+0000 7fee93f1d640 1 -- 172.21.15.88:0/3454662648 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fee8c074700 con 0x7fee8c06dd70 2023-10-16T20:39:14.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.157+0000 7fee827fc640 1 -- 172.21.15.88:0/3454662648 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7fee880c4020 con 0x7fee8c06dd70 2023-10-16T20:39:14.166 INFO:teuthology.orchestra.run.smithi088.stderr:listed 0 entries 2023-10-16T20:39:14.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.161+0000 7fee93f1d640 1 -- 172.21.15.88:0/3454662648 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee700664a0 msgr2=0x7fee70068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:14.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.161+0000 7fee93f1d640 1 --2- 172.21.15.88:0/3454662648 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee700664a0 0x7fee70068960 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fee8c14e0f0 tx=0x7fee7c009040 comp rx=0 tx=0).stop 2023-10-16T20:39:14.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.161+0000 7fee93f1d640 1 -- 172.21.15.88:0/3454662648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee8c06dd70 msgr2=0x7fee8c1f2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:14.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.161+0000 7fee93f1d640 1 --2- 172.21.15.88:0/3454662648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee8c06dd70 0x7fee8c1f2a40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fee88019110 tx=0x7fee8801ad40 comp rx=0 tx=0).stop 2023-10-16T20:39:14.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.161+0000 7fee93f1d640 1 -- 172.21.15.88:0/3454662648 shutdown_connections 2023-10-16T20:39:14.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.161+0000 7fee93f1d640 1 --2- 172.21.15.88:0/3454662648 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee700664a0 0x7fee70068960 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.161+0000 7fee93f1d640 1 --2- 172.21.15.88:0/3454662648 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee8c1537d0 0x7fee8c1f52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.161+0000 7fee93f1d640 1 --2- 172.21.15.88:0/3454662648 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee8c0cdb20 0x7fee8c1f4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.161+0000 7fee93f1d640 1 --2- 172.21.15.88:0/3454662648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee8c06dd70 0x7fee8c1f2a40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.161+0000 7fee93f1d640 1 -- 172.21.15.88:0/3454662648 >> 172.21.15.88:0/3454662648 conn(0x7fee8c06f7d0 msgr2=0x7fee8c0cb220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:14.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.161+0000 7fee93f1d640 1 -- 172.21.15.88:0/3454662648 shutdown_connections 2023-10-16T20:39:14.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.161+0000 7fee93f1d640 1 -- 172.21.15.88:0/3454662648 wait complete. 2023-10-16T20:39:14.187 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi088.front.sepia.ceph.com... 2023-10-16T20:39:14.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:14.188 DEBUG:teuthology.orchestra.run.smithi088:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-10-16T20:39:14.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi088.stderr:ceph-fuse[44187]: fuse finished with error 0 and tester_r 0 2023-10-16T20:39:14.220 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T20:39:14.292 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2023-10-16T20:39:14.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:14.293 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-10-16T20:39:14.308 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:39:14.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90dc1c4640 1 Processor -- start 2023-10-16T20:39:14.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90dc1c4640 1 -- start start 2023-10-16T20:39:14.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90dc1c4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f90d4153770 0x7f90d4159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90dc1c4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f90d40c9bb0 0x7f90d40c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90dc1c4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90d40c41f0 0x7f90d40cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90dc1c4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d40cc250 con 0x7f90d4153770 2023-10-16T20:39:14.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90dc1c4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d40cc670 con 0x7f90d40c9bb0 2023-10-16T20:39:14.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90dc1c4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d415a410 con 0x7f90d40c41f0 2023-10-16T20:39:14.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90d9f60640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f90d4153770 0x7f90d4159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90d9f60640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f90d4153770 0x7f90d4159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52792/0 (socket says 172.21.15.88:52792) 2023-10-16T20:39:14.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90d9f60640 1 -- 172.21.15.88:0/2508677906 learned_addr learned my addr 172.21.15.88:0/2508677906 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:14.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90d975f640 1 --2- 172.21.15.88:0/2508677906 >> v2:172.21.15.191:3300/0 conn(0x7f90d40c9bb0 0x7f90d40c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90da761640 1 --2- 172.21.15.88:0/2508677906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90d40c41f0 0x7f90d40cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90d9f60640 1 -- 172.21.15.88:0/2508677906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90d40c41f0 msgr2=0x7f90d40cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:14.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90d9f60640 1 --2- 172.21.15.88:0/2508677906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90d40c41f0 0x7f90d40cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90d9f60640 1 -- 172.21.15.88:0/2508677906 >> v2:172.21.15.191:3300/0 conn(0x7f90d40c9bb0 msgr2=0x7f90d40c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:14.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90d9f60640 1 --2- 172.21.15.88:0/2508677906 >> v2:172.21.15.191:3300/0 conn(0x7f90d40c9bb0 0x7f90d40c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90d9f60640 1 -- 172.21.15.88:0/2508677906 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d4153b50 con 0x7f90d4153770 2023-10-16T20:39:14.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.481+0000 7f90da761640 1 --2- 172.21.15.88:0/2508677906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90d40c41f0 0x7f90d40cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:14.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.485+0000 7f90d9f60640 1 --2- 172.21.15.88:0/2508677906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d4153770 0x7f90d4159ed0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f90d40cfe50 tx=0x7f90d001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d8ff1f43479c3e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:14.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.485+0000 7f90d8f5e640 1 -- 172.21.15.88:0/2508677906 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d00ac630 con 0x7f90d4153770 2023-10-16T20:39:14.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.485+0000 7f90d8f5e640 1 -- 172.21.15.88:0/2508677906 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d00ac790 con 0x7f90d4153770 2023-10-16T20:39:14.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.485+0000 7f90d8f5e640 1 -- 172.21.15.88:0/2508677906 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d00acb20 con 0x7f90d4153770 2023-10-16T20:39:14.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.485+0000 7f90dc1c4640 1 -- 172.21.15.88:0/2508677906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d4153770 msgr2=0x7f90d4159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:14.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.485+0000 7f90dc1c4640 1 --2- 172.21.15.88:0/2508677906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d4153770 0x7f90d4159ed0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f90d40cfe50 tx=0x7f90d001c500 comp rx=0 tx=0).stop 2023-10-16T20:39:14.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.485+0000 7f90dc1c4640 1 -- 172.21.15.88:0/2508677906 shutdown_connections 2023-10-16T20:39:14.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.485+0000 7f90dc1c4640 1 --2- 172.21.15.88:0/2508677906 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90d40c41f0 0x7f90d40cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.485+0000 7f90dc1c4640 1 --2- 172.21.15.88:0/2508677906 >> v2:172.21.15.191:3300/0 conn(0x7f90d40c9bb0 0x7f90d40c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.485+0000 7f90dc1c4640 1 --2- 172.21.15.88:0/2508677906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d4153770 0x7f90d4159ed0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.485+0000 7f90dc1c4640 1 -- 172.21.15.88:0/2508677906 >> 172.21.15.88:0/2508677906 conn(0x7f90d40715c0 msgr2=0x7f90d40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:14.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.485+0000 7f90dc1c4640 1 -- 172.21.15.88:0/2508677906 shutdown_connections 2023-10-16T20:39:14.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.485+0000 7f90dc1c4640 1 -- 172.21.15.88:0/2508677906 wait complete. 2023-10-16T20:39:14.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90dc1c4640 1 Processor -- start 2023-10-16T20:39:14.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90dc1c4640 1 -- start start 2023-10-16T20:39:14.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90dc1c4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90d40c41f0 0x7f90d40c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90dc1c4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90d40c9bb0 0x7f90d41f24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90dc1c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d4153770 0x7f90d40c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90dc1c4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d41f2010 con 0x7f90d4153770 2023-10-16T20:39:14.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90dc1c4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d414a9d0 con 0x7f90d40c41f0 2023-10-16T20:39:14.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90dc1c4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d41f1ce0 con 0x7f90d40c9bb0 2023-10-16T20:39:14.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90da761640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d4153770 0x7f90d40c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90da761640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d4153770 0x7f90d40c8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52796/0 (socket says 172.21.15.88:52796) 2023-10-16T20:39:14.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90da761640 1 -- 172.21.15.88:0/241831244 learned_addr learned my addr 172.21.15.88:0/241831244 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:14.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90d9f60640 1 --2- 172.21.15.88:0/241831244 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90d40c41f0 0x7f90d40c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90d975f640 1 --2- 172.21.15.88:0/241831244 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90d40c9bb0 0x7f90d41f24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90da761640 1 -- 172.21.15.88:0/241831244 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90d40c9bb0 msgr2=0x7f90d41f24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:14.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90da761640 1 --2- 172.21.15.88:0/241831244 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90d40c9bb0 0x7f90d41f24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90da761640 1 -- 172.21.15.88:0/241831244 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90d40c41f0 msgr2=0x7f90d40c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:14.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90da761640 1 --2- 172.21.15.88:0/241831244 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90d40c41f0 0x7f90d40c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90da761640 1 -- 172.21.15.88:0/241831244 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d0009480 con 0x7f90d4153770 2023-10-16T20:39:14.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90da761640 1 --2- 172.21.15.88:0/241831244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d4153770 0x7f90d40c8260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00cef0 tx=0x7f90cc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:14.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90ca7fc640 1 -- 172.21.15.88:0/241831244 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc00d840 con 0x7f90d4153770 2023-10-16T20:39:14.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90ca7fc640 1 -- 172.21.15.88:0/241831244 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90cc00d9a0 con 0x7f90d4153770 2023-10-16T20:39:14.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90dc1c4640 1 -- 172.21.15.88:0/241831244 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d41f5cf0 con 0x7f90d4153770 2023-10-16T20:39:14.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90ca7fc640 1 -- 172.21.15.88:0/241831244 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc00dd50 con 0x7f90d4153770 2023-10-16T20:39:14.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90dc1c4640 1 -- 172.21.15.88:0/241831244 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d414a4a0 con 0x7f90d4153770 2023-10-16T20:39:14.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.489+0000 7f90dc1c4640 1 -- 172.21.15.88:0/241831244 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d41f21c0 con 0x7f90d4153770 2023-10-16T20:39:14.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.493+0000 7f90ca7fc640 1 -- 172.21.15.88:0/241831244 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f90cc01f440 con 0x7f90d4153770 2023-10-16T20:39:14.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.493+0000 7f90ca7fc640 1 --2- 172.21.15.88:0/241831244 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f90b80664a0 0x7f90b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.493+0000 7f90ca7fc640 1 -- 172.21.15.88:0/241831244 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f90cc084b90 con 0x7f90d4153770 2023-10-16T20:39:14.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.493+0000 7f90d9f60640 1 --2- 172.21.15.88:0/241831244 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f90b80664a0 0x7f90b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.493+0000 7f90d9f60640 1 --2- 172.21.15.88:0/241831244 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f90b80664a0 0x7f90b8068960 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f90d00ae000 tx=0x7f90d00a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:14.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.497+0000 7f90ca7fc640 1 -- 172.21.15.88:0/241831244 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f90d41f21c0 con 0x7f90d4153770 2023-10-16T20:39:14.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.649+0000 7f90dc1c4640 1 -- 172.21.15.88:0/241831244 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90d41f21c0 con 0x7f90d4153770 2023-10-16T20:39:14.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.649+0000 7f90ca7fc640 1 -- 172.21.15.88:0/241831244 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18710 (secure 0 0 0) 0x7f90d41f21c0 con 0x7f90d4153770 2023-10-16T20:39:14.657 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:14.657 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:37:37.392549+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:39:14.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.653+0000 7f90dc1c4640 1 -- 172.21.15.88:0/241831244 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f90b80664a0 msgr2=0x7f90b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:14.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.653+0000 7f90dc1c4640 1 --2- 172.21.15.88:0/241831244 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f90b80664a0 0x7f90b8068960 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f90d00ae000 tx=0x7f90d00a7000 comp rx=0 tx=0).stop 2023-10-16T20:39:14.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.653+0000 7f90dc1c4640 1 -- 172.21.15.88:0/241831244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d4153770 msgr2=0x7f90d40c8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:14.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.653+0000 7f90dc1c4640 1 --2- 172.21.15.88:0/241831244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d4153770 0x7f90d40c8260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00cef0 tx=0x7f90cc00a710 comp rx=0 tx=0).stop 2023-10-16T20:39:14.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.653+0000 7f90dc1c4640 1 -- 172.21.15.88:0/241831244 shutdown_connections 2023-10-16T20:39:14.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.653+0000 7f90dc1c4640 1 --2- 172.21.15.88:0/241831244 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f90b80664a0 0x7f90b8068960 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.653+0000 7f90dc1c4640 1 --2- 172.21.15.88:0/241831244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d4153770 0x7f90d40c8260 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.653+0000 7f90dc1c4640 1 --2- 172.21.15.88:0/241831244 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90d40c9bb0 0x7f90d41f24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.653+0000 7f90dc1c4640 1 --2- 172.21.15.88:0/241831244 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90d40c41f0 0x7f90d40c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.653+0000 7f90dc1c4640 1 -- 172.21.15.88:0/241831244 >> 172.21.15.88:0/241831244 conn(0x7f90d40715c0 msgr2=0x7f90d40719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:14.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.653+0000 7f90dc1c4640 1 -- 172.21.15.88:0/241831244 shutdown_connections 2023-10-16T20:39:14.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.653+0000 7f90dc1c4640 1 -- 172.21.15.88:0/241831244 wait complete. 2023-10-16T20:39:14.679 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:39:14.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.809+0000 7faaa42ea640 1 Processor -- start 2023-10-16T20:39:14.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.809+0000 7faaa42ea640 1 -- start start 2023-10-16T20:39:14.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.809+0000 7faaa42ea640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa9c0c81a0 0x7faa9c159450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.809+0000 7faaa42ea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa9c0c8810 con 0x7faa9c0c76c0 2023-10-16T20:39:14.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.809+0000 7faaa42ea640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa9c06b400 con 0x7faa9c0c7dc0 2023-10-16T20:39:14.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.809+0000 7faaa42ea640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c06b0d0 con 0x7faa9c0c81a0 2023-10-16T20:39:14.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1885640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7faa9c0c76c0 0x7faa9c06b890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50446/0 (socket says 172.21.15.88:50446) 2023-10-16T20:39:14.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1885640 1 -- 172.21.15.88:0/3327196660 learned_addr learned my addr 172.21.15.88:0/3327196660 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:14.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa2887640 1 --2- 172.21.15.88:0/3327196660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa9c0c81a0 0x7faa9c159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1084640 1 -- 172.21.15.88:0/3327196660 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748515119 0 0) 0x7faa9c0c8810 con 0x7faa9c0c76c0 2023-10-16T20:39:14.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1084640 1 -- 172.21.15.88:0/3327196660 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa88003530 con 0x7faa9c0c76c0 2023-10-16T20:39:14.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1084640 1 -- 172.21.15.88:0/3327196660 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916028313 0 0) 0x7faa9c06b400 con 0x7faa9c0c7dc0 2023-10-16T20:39:14.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1084640 1 -- 172.21.15.88:0/3327196660 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa9c0c8810 con 0x7faa9c0c7dc0 2023-10-16T20:39:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1084640 1 -- 172.21.15.88:0/3327196660 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4129418992 0 0) 0x7faa88003530 con 0x7faa9c0c76c0 2023-10-16T20:39:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1084640 1 -- 172.21.15.88:0/3327196660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa9c0c81a0 msgr2=0x7faa9c159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1084640 1 --2- 172.21.15.88:0/3327196660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa9c0c81a0 0x7faa9c159450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1084640 1 -- 172.21.15.88:0/3327196660 >> v1:172.21.15.191:6789/0 conn(0x7faa9c0c7dc0 legacy=0x7faa9c06d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1084640 1 -- 172.21.15.88:0/3327196660 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c06e6a0 con 0x7faa9c0c76c0 2023-10-16T20:39:14.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1084640 1 -- 172.21.15.88:0/3327196660 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7faa8c001680 con 0x7faa9c0c76c0 2023-10-16T20:39:14.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1084640 1 -- 172.21.15.88:0/3327196660 >> v1:172.21.15.88:6789/0 conn(0x7faa9c0c76c0 legacy=0x7faa9c06b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:14.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1084640 1 --2- 172.21.15.88:0/3327196660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa88003b40 0x7faa88003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1084640 1 --2- 172.21.15.88:0/3327196660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa880047d0 0x7faa88004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1084640 1 --2- 172.21.15.88:0/3327196660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa88005190 0x7faa8800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1084640 1 -- 172.21.15.88:0/3327196660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c06b0d0 con 0x7faa880047d0 2023-10-16T20:39:14.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1084640 1 -- 172.21.15.88:0/3327196660 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8800c000 con 0x7faa88003b40 2023-10-16T20:39:14.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1084640 1 -- 172.21.15.88:0/3327196660 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8800c1c0 con 0x7faa88005190 2023-10-16T20:39:14.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa2887640 1 --2- 172.21.15.88:0/3327196660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa9c0c81a0 0x7faa9c159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:39:14.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1885640 1 --2- 172.21.15.88:0/3327196660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa880047d0 0x7faa88004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa2086640 1 --2- 172.21.15.88:0/3327196660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa88003b40 0x7faa88003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa2887640 1 --2- 172.21.15.88:0/3327196660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa88005190 0x7faa8800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1885640 1 -- 172.21.15.88:0/3327196660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa88005190 msgr2=0x7faa8800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:14.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1885640 1 --2- 172.21.15.88:0/3327196660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa88005190 0x7faa8800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1885640 1 -- 172.21.15.88:0/3327196660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa88003b40 msgr2=0x7faa88003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:14.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1885640 1 --2- 172.21.15.88:0/3327196660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa88003b40 0x7faa88003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.813+0000 7faaa1885640 1 -- 172.21.15.88:0/3327196660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa88003530 con 0x7faa880047d0 2023-10-16T20:39:14.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.817+0000 7faaa1885640 1 --2- 172.21.15.88:0/3327196660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa880047d0 0x7faa88004c50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faa8c00c160 tx=0x7faa8c01cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a02dc60db9bb026 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:14.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.817+0000 7faaa1084640 1 -- 172.21.15.88:0/3327196660 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c0058e0 con 0x7faa880047d0 2023-10-16T20:39:14.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.817+0000 7faaa1084640 1 -- 172.21.15.88:0/3327196660 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8c005a60 con 0x7faa880047d0 2023-10-16T20:39:14.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.817+0000 7faaa1084640 1 -- 172.21.15.88:0/3327196660 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c005e10 con 0x7faa880047d0 2023-10-16T20:39:14.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.817+0000 7faaa42ea640 1 -- 172.21.15.88:0/3327196660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa880047d0 msgr2=0x7faa88004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:14.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.817+0000 7faaa42ea640 1 --2- 172.21.15.88:0/3327196660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa880047d0 0x7faa88004c50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faa8c00c160 tx=0x7faa8c01cba0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.817+0000 7faaa2887640 1 -- 172.21.15.88:0/3327196660 reap_dead start 2023-10-16T20:39:14.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.817+0000 7faaa42ea640 1 -- 172.21.15.88:0/3327196660 shutdown_connections 2023-10-16T20:39:14.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.817+0000 7faaa42ea640 1 -- 172.21.15.88:0/3327196660 >> 172.21.15.88:0/3327196660 conn(0x7faa9c071a60 msgr2=0x7faa9c071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:14.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.817+0000 7faaa42ea640 1 -- 172.21.15.88:0/3327196660 shutdown_connections 2023-10-16T20:39:14.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.817+0000 7faaa42ea640 1 -- 172.21.15.88:0/3327196660 wait complete. 2023-10-16T20:39:14.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.817+0000 7faaa42ea640 1 Processor -- start 2023-10-16T20:39:14.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.817+0000 7faaa42ea640 1 -- start start 2023-10-16T20:39:14.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa42ea640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa880047d0 0x7faa9c14dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa42ea640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa9c1f1e40 0x7faa9c13e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa42ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa9c14efa0 0x7faa9c13ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa42ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c13e4c0 con 0x7faa9c14efa0 2023-10-16T20:39:14.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa42ea640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c141840 con 0x7faa9c1f1e40 2023-10-16T20:39:14.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa42ea640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c141ae0 con 0x7faa880047d0 2023-10-16T20:39:14.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa2887640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa9c14efa0 0x7faa9c13ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa2887640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa9c14efa0 0x7faa9c13ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52820/0 (socket says 172.21.15.88:52820) 2023-10-16T20:39:14.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa2086640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa880047d0 0x7faa9c14dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa2887640 1 -- 172.21.15.88:0/1662856247 learned_addr learned my addr 172.21.15.88:0/1662856247 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:14.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa1885640 1 --2- 172.21.15.88:0/1662856247 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa9c1f1e40 0x7faa9c13e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa2887640 1 -- 172.21.15.88:0/1662856247 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa880047d0 msgr2=0x7faa9c14dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:14.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa2887640 1 --2- 172.21.15.88:0/1662856247 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa880047d0 0x7faa9c14dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa2887640 1 -- 172.21.15.88:0/1662856247 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa9c1f1e40 msgr2=0x7faa9c13e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:14.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa2887640 1 --2- 172.21.15.88:0/1662856247 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa9c1f1e40 0x7faa9c13e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa2887640 1 -- 172.21.15.88:0/1662856247 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c141dc0 con 0x7faa9c14efa0 2023-10-16T20:39:14.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa2887640 1 --2- 172.21.15.88:0/1662856247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa9c14efa0 0x7faa9c13ee20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faa9c0719f0 tx=0x7faa940117d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:14.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faa92ffd640 1 -- 172.21.15.88:0/1662856247 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa9400e070 con 0x7faa9c14efa0 2023-10-16T20:39:14.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faa92ffd640 1 -- 172.21.15.88:0/1662856247 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa940039a0 con 0x7faa9c14efa0 2023-10-16T20:39:14.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa42ea640 1 -- 172.21.15.88:0/1662856247 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa9c1420a0 con 0x7faa9c14efa0 2023-10-16T20:39:14.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faa92ffd640 1 -- 172.21.15.88:0/1662856247 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa94003b50 con 0x7faa9c14efa0 2023-10-16T20:39:14.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa42ea640 1 -- 172.21.15.88:0/1662856247 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9c142660 con 0x7faa9c14efa0 2023-10-16T20:39:14.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faaa42ea640 1 -- 172.21.15.88:0/1662856247 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9c13e600 con 0x7faa9c14efa0 2023-10-16T20:39:14.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.821+0000 7faa92ffd640 1 -- 172.21.15.88:0/1662856247 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faa94003cb0 con 0x7faa9c14efa0 2023-10-16T20:39:14.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.825+0000 7faa92ffd640 1 --2- 172.21.15.88:0/1662856247 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faa800664a0 0x7faa80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:14.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.825+0000 7faa92ffd640 1 -- 172.21.15.88:0/1662856247 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7faa94084be0 con 0x7faa9c14efa0 2023-10-16T20:39:14.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.825+0000 7faaa2086640 1 --2- 172.21.15.88:0/1662856247 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faa800664a0 0x7faa80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:14.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.825+0000 7faaa2086640 1 --2- 172.21.15.88:0/1662856247 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faa800664a0 0x7faa80068960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faa98007fd0 tx=0x7faa98009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:14.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.829+0000 7faa92ffd640 1 -- 172.21.15.88:0/1662856247 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faa9c13e600 con 0x7faa9c14efa0 2023-10-16T20:39:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.977+0000 7faaa42ea640 1 -- 172.21.15.88:0/1662856247 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa9c13e600 con 0x7faa9c14efa0 2023-10-16T20:39:14.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.977+0000 7faa92ffd640 1 -- 172.21.15.88:0/1662856247 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18710 (secure 0 0 0) 0x7faa9c13e600 con 0x7faa9c14efa0 2023-10-16T20:39:14.986 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:14.986 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:37:37.392549+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:39:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.981+0000 7faaa42ea640 1 -- 172.21.15.88:0/1662856247 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faa800664a0 msgr2=0x7faa80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.981+0000 7faaa42ea640 1 --2- 172.21.15.88:0/1662856247 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faa800664a0 0x7faa80068960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faa98007fd0 tx=0x7faa98009040 comp rx=0 tx=0).stop 2023-10-16T20:39:14.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.981+0000 7faaa42ea640 1 -- 172.21.15.88:0/1662856247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa9c14efa0 msgr2=0x7faa9c13ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:14.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.981+0000 7faaa42ea640 1 --2- 172.21.15.88:0/1662856247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa9c14efa0 0x7faa9c13ee20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faa9c0719f0 tx=0x7faa940117d0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.981+0000 7faaa42ea640 1 -- 172.21.15.88:0/1662856247 shutdown_connections 2023-10-16T20:39:14.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.981+0000 7faaa42ea640 1 --2- 172.21.15.88:0/1662856247 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faa800664a0 0x7faa80068960 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.981+0000 7faaa42ea640 1 --2- 172.21.15.88:0/1662856247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa9c14efa0 0x7faa9c13ee20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.981+0000 7faaa42ea640 1 --2- 172.21.15.88:0/1662856247 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa9c1f1e40 0x7faa9c13e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.981+0000 7faaa42ea640 1 --2- 172.21.15.88:0/1662856247 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faa880047d0 0x7faa9c14dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:14.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.981+0000 7faaa42ea640 1 -- 172.21.15.88:0/1662856247 >> 172.21.15.88:0/1662856247 conn(0x7faa9c071a60 msgr2=0x7faa9c0c5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:14.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.981+0000 7faaa42ea640 1 -- 172.21.15.88:0/1662856247 shutdown_connections 2023-10-16T20:39:14.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:14.981+0000 7faaa42ea640 1 -- 172.21.15.88:0/1662856247 wait complete. 2023-10-16T20:39:15.008 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-16T20:39:15.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef6b772640 1 Processor -- start 2023-10-16T20:39:15.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef6b772640 1 -- start start 2023-10-16T20:39:15.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef6b772640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef641517e0 0x7fef641573e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:15.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef6b772640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef64069e90 con 0x7fef640c4640 2023-10-16T20:39:15.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef6b772640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef6406aea0 con 0x7fef640c4260 2023-10-16T20:39:15.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef6b772640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef640716a0 con 0x7fef641517e0 2023-10-16T20:39:15.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef68d0d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fef640c4640 0x7fef641510d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50462/0 (socket says 172.21.15.88:50462) 2023-10-16T20:39:15.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef68d0d640 1 -- 172.21.15.88:0/2615549813 learned_addr learned my addr 172.21.15.88:0/2615549813 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:15.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef69d0f640 1 --2- 172.21.15.88:0/2615549813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef641517e0 0x7fef641573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:15.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef5bfff640 1 -- 172.21.15.88:0/2615549813 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837517291 0 0) 0x7fef64069e90 con 0x7fef640c4640 2023-10-16T20:39:15.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef5bfff640 1 -- 172.21.15.88:0/2615549813 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef50003530 con 0x7fef640c4640 2023-10-16T20:39:15.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef5bfff640 1 -- 172.21.15.88:0/2615549813 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3078769132 0 0) 0x7fef50003530 con 0x7fef640c4640 2023-10-16T20:39:15.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef5bfff640 1 -- 172.21.15.88:0/2615549813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef641517e0 msgr2=0x7fef641573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:15.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef5bfff640 1 --2- 172.21.15.88:0/2615549813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef641517e0 0x7fef641573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:15.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef5bfff640 1 -- 172.21.15.88:0/2615549813 >> v1:172.21.15.191:6789/0 conn(0x7fef640c4260 legacy=0x7fef640c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:15.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef5bfff640 1 -- 172.21.15.88:0/2615549813 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef641581e0 con 0x7fef640c4640 2023-10-16T20:39:15.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef5bfff640 1 -- 172.21.15.88:0/2615549813 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7fef54001680 con 0x7fef640c4640 2023-10-16T20:39:15.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef5bfff640 1 -- 172.21.15.88:0/2615549813 >> v1:172.21.15.88:6789/0 conn(0x7fef640c4640 legacy=0x7fef641510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:15.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef5bfff640 1 --2- 172.21.15.88:0/2615549813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef500039b0 0x7fef50003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:15.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef5bfff640 1 --2- 172.21.15.88:0/2615549813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef641517e0 0x7fef500047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:15.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef5bfff640 1 --2- 172.21.15.88:0/2615549813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef50004d10 0x7fef50005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:15.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef5bfff640 1 -- 172.21.15.88:0/2615549813 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5000bb20 con 0x7fef50004d10 2023-10-16T20:39:15.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef5bfff640 1 -- 172.21.15.88:0/2615549813 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5000bc60 con 0x7fef500039b0 2023-10-16T20:39:15.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef5bfff640 1 -- 172.21.15.88:0/2615549813 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5000be20 con 0x7fef641517e0 2023-10-16T20:39:15.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef69d0f640 1 --2- 172.21.15.88:0/2615549813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef50004d10 0x7fef50005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:15.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef68d0d640 1 --2- 172.21.15.88:0/2615549813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef641517e0 0x7fef500047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:15.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef6950e640 1 --2- 172.21.15.88:0/2615549813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef500039b0 0x7fef50003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:15.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef69d0f640 1 -- 172.21.15.88:0/2615549813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef641517e0 msgr2=0x7fef500047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:15.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef69d0f640 1 --2- 172.21.15.88:0/2615549813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef641517e0 0x7fef500047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:15.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef69d0f640 1 -- 172.21.15.88:0/2615549813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef500039b0 msgr2=0x7fef50003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:15.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef69d0f640 1 --2- 172.21.15.88:0/2615549813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef500039b0 0x7fef50003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:15.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.141+0000 7fef69d0f640 1 -- 172.21.15.88:0/2615549813 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef50003530 con 0x7fef50004d10 2023-10-16T20:39:15.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.145+0000 7fef69d0f640 1 --2- 172.21.15.88:0/2615549813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef50004d10 0x7fef50005190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fef5c009fc0 tx=0x7fef5c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef3ddbc9cf8f2c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:15.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.145+0000 7fef5bfff640 1 -- 172.21.15.88:0/2615549813 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5c0a5c70 con 0x7fef50004d10 2023-10-16T20:39:15.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.145+0000 7fef5bfff640 1 -- 172.21.15.88:0/2615549813 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5c00f040 con 0x7fef50004d10 2023-10-16T20:39:15.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.145+0000 7fef5bfff640 1 -- 172.21.15.88:0/2615549813 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5c0ad440 con 0x7fef50004d10 2023-10-16T20:39:15.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.145+0000 7fef6b772640 1 -- 172.21.15.88:0/2615549813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef50004d10 msgr2=0x7fef50005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:15.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.145+0000 7fef6b772640 1 --2- 172.21.15.88:0/2615549813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef50004d10 0x7fef50005190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fef5c009fc0 tx=0x7fef5c01cc60 comp rx=0 tx=0).stop 2023-10-16T20:39:15.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.145+0000 7fef69d0f640 1 -- 172.21.15.88:0/2615549813 reap_dead start 2023-10-16T20:39:15.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.145+0000 7fef6b772640 1 -- 172.21.15.88:0/2615549813 shutdown_connections 2023-10-16T20:39:15.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.145+0000 7fef6b772640 1 -- 172.21.15.88:0/2615549813 >> 172.21.15.88:0/2615549813 conn(0x7fef640ca9f0 msgr2=0x7fef640cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:15.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.145+0000 7fef6b772640 1 -- 172.21.15.88:0/2615549813 shutdown_connections 2023-10-16T20:39:15.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.145+0000 7fef6b772640 1 -- 172.21.15.88:0/2615549813 wait complete. 2023-10-16T20:39:15.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef6b772640 1 Processor -- start 2023-10-16T20:39:15.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef6b772640 1 -- start start 2023-10-16T20:39:15.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef6b772640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef50004d10 0x7fef640c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:15.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef6b772640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef641543a0 0x7fef641e0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:15.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef6b772640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef6414e350 0x7fef6414e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:15.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef6b772640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef64152a40 con 0x7fef641543a0 2023-10-16T20:39:15.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef6b772640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6414ed10 con 0x7fef50004d10 2023-10-16T20:39:15.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef6b772640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef640c5040 con 0x7fef6414e350 2023-10-16T20:39:15.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef68d0d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef641543a0 0x7fef641e0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:15.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef68d0d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef641543a0 0x7fef641e0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52834/0 (socket says 172.21.15.88:52834) 2023-10-16T20:39:15.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef6950e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef50004d10 0x7fef640c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:15.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef68d0d640 1 -- 172.21.15.88:0/3940512408 learned_addr learned my addr 172.21.15.88:0/3940512408 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:15.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef69d0f640 1 --2- 172.21.15.88:0/3940512408 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef6414e350 0x7fef6414e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:15.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef68d0d640 1 -- 172.21.15.88:0/3940512408 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef6414e350 msgr2=0x7fef6414e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:15.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef68d0d640 1 --2- 172.21.15.88:0/3940512408 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef6414e350 0x7fef6414e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:15.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef68d0d640 1 -- 172.21.15.88:0/3940512408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef50004d10 msgr2=0x7fef640c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:15.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef68d0d640 1 --2- 172.21.15.88:0/3940512408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef50004d10 0x7fef640c9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:15.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef68d0d640 1 -- 172.21.15.88:0/3940512408 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef640c9580 con 0x7fef641543a0 2023-10-16T20:39:15.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef68d0d640 1 --2- 172.21.15.88:0/3940512408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef641543a0 0x7fef641e0a70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fef5400e820 tx=0x7fef5400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:15.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef59ffb640 1 -- 172.21.15.88:0/3940512408 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5400c8d0 con 0x7fef641543a0 2023-10-16T20:39:15.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef59ffb640 1 -- 172.21.15.88:0/3940512408 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5400ca30 con 0x7fef641543a0 2023-10-16T20:39:15.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef59ffb640 1 -- 172.21.15.88:0/3940512408 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5400cde0 con 0x7fef641543a0 2023-10-16T20:39:15.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef6b772640 1 -- 172.21.15.88:0/3940512408 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef64154970 con 0x7fef641543a0 2023-10-16T20:39:15.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.149+0000 7fef6b772640 1 -- 172.21.15.88:0/3940512408 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef640c5290 con 0x7fef641543a0 2023-10-16T20:39:15.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.153+0000 7fef6b772640 1 -- 172.21.15.88:0/3940512408 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef6406c990 con 0x7fef641543a0 2023-10-16T20:39:15.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.153+0000 7fef59ffb640 1 -- 172.21.15.88:0/3940512408 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fef5401d460 con 0x7fef641543a0 2023-10-16T20:39:15.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.153+0000 7fef59ffb640 1 --2- 172.21.15.88:0/3940512408 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fef480664d0 0x7fef48068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:15.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.153+0000 7fef59ffb640 1 -- 172.21.15.88:0/3940512408 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fef54084110 con 0x7fef641543a0 2023-10-16T20:39:15.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.153+0000 7fef6950e640 1 --2- 172.21.15.88:0/3940512408 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fef480664d0 0x7fef48068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:15.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.153+0000 7fef6950e640 1 --2- 172.21.15.88:0/3940512408 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fef480664d0 0x7fef48068990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fef6000ad20 tx=0x7fef6000acb0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:15.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.157+0000 7fef59ffb640 1 -- 172.21.15.88:0/3940512408 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fef54051280 con 0x7fef641543a0 2023-10-16T20:39:15.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:15.341+0000 7fef6b772640 1 -- 172.21.15.88:0/3940512408 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fef640c7d00 con 0x7fef641543a0 2023-10-16T20:39:16.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.289+0000 7fef59ffb640 1 -- 172.21.15.88:0/3940512408 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7fef540548a0 con 0x7fef641543a0 2023-10-16T20:39:16.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.293+0000 7fef6b772640 1 -- 172.21.15.88:0/3940512408 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fef480664d0 msgr2=0x7fef48068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:16.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.293+0000 7fef6b772640 1 --2- 172.21.15.88:0/3940512408 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fef480664d0 0x7fef48068990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fef6000ad20 tx=0x7fef6000acb0 comp rx=0 tx=0).stop 2023-10-16T20:39:16.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.293+0000 7fef6b772640 1 -- 172.21.15.88:0/3940512408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef641543a0 msgr2=0x7fef641e0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:16.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.293+0000 7fef6b772640 1 --2- 172.21.15.88:0/3940512408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef641543a0 0x7fef641e0a70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fef5400e820 tx=0x7fef5400ece0 comp rx=0 tx=0).stop 2023-10-16T20:39:16.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.293+0000 7fef6b772640 1 -- 172.21.15.88:0/3940512408 shutdown_connections 2023-10-16T20:39:16.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.293+0000 7fef6b772640 1 --2- 172.21.15.88:0/3940512408 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fef480664d0 0x7fef48068990 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:16.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.293+0000 7fef6b772640 1 --2- 172.21.15.88:0/3940512408 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fef6414e350 0x7fef6414e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:16.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.293+0000 7fef6b772640 1 --2- 172.21.15.88:0/3940512408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef641543a0 0x7fef641e0a70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:16.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.293+0000 7fef6b772640 1 --2- 172.21.15.88:0/3940512408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fef50004d10 0x7fef640c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:16.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.293+0000 7fef6b772640 1 -- 172.21.15.88:0/3940512408 >> 172.21.15.88:0/3940512408 conn(0x7fef640ca9f0 msgr2=0x7fef640c6470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:16.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.293+0000 7fef6b772640 1 -- 172.21.15.88:0/3940512408 shutdown_connections 2023-10-16T20:39:16.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.293+0000 7fef6b772640 1 -- 172.21.15.88:0/3940512408 wait complete. 2023-10-16T20:39:16.323 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300} 2023-10-16T20:39:16.323 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-10-16T20:39:16.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.461+0000 7fb431279640 1 Processor -- start 2023-10-16T20:39:16.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.461+0000 7fb431279640 1 -- start start 2023-10-16T20:39:16.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.461+0000 7fb431279640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb42c153780 0x7fb42c0c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:16.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.461+0000 7fb431279640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42c159d90 con 0x7fb42c1598c0 2023-10-16T20:39:16.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.461+0000 7fb431279640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42c15a070 con 0x7fb42c159490 2023-10-16T20:39:16.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.461+0000 7fb431279640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb42c15a350 con 0x7fb42c153780 2023-10-16T20:39:16.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.465+0000 7fb42affd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb42c153780 0x7fb42c0c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:16.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.465+0000 7fb42affd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb42c153780 0x7fb42c0c9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52836/0 (socket says 172.21.15.88:52836) 2023-10-16T20:39:16.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.465+0000 7fb42affd640 1 -- 172.21.15.88:0/2392731086 learned_addr learned my addr 172.21.15.88:0/2392731086 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:16.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.465+0000 7fb42b7fe640 1 --1- 172.21.15.88:0/2392731086 >> v1:172.21.15.88:6789/0 conn(0x7fb42c1598c0 0x7fb42c0ced80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50468/0 (socket says 172.21.15.88:50468) 2023-10-16T20:39:16.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.465+0000 7fb42affd640 1 -- 172.21.15.88:0/2392731086 >> v1:172.21.15.191:6789/0 conn(0x7fb42c159490 legacy=0x7fb42c0c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:16.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.465+0000 7fb42affd640 1 -- 172.21.15.88:0/2392731086 >> v1:172.21.15.88:6789/0 conn(0x7fb42c1598c0 legacy=0x7fb42c0ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:16.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.465+0000 7fb42affd640 1 -- 172.21.15.88:0/2392731086 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c0c4910 con 0x7fb42c153780 2023-10-16T20:39:16.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.465+0000 7fb42a7fc640 1 -- 172.21.15.88:0/2392731086 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931537933 0 0) 0x7fb42c159d90 con 0x7fb42c1598c0 2023-10-16T20:39:16.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.465+0000 7fb42affd640 1 --2- 172.21.15.88:0/2392731086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb42c153780 0x7fb42c0c9be0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb41c006fd0 tx=0x7fb41c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a2fd88aada52cb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:16.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.465+0000 7fb42a7fc640 1 -- 172.21.15.88:0/2392731086 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c0a8a10 con 0x7fb42c153780 2023-10-16T20:39:16.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.465+0000 7fb42a7fc640 1 -- 172.21.15.88:0/2392731086 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41c0a8b70 con 0x7fb42c153780 2023-10-16T20:39:16.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.465+0000 7fb42a7fc640 1 -- 172.21.15.88:0/2392731086 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c0b0440 con 0x7fb42c153780 2023-10-16T20:39:16.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb431279640 1 -- 172.21.15.88:0/2392731086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb42c153780 msgr2=0x7fb42c0c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:16.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb431279640 1 --2- 172.21.15.88:0/2392731086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb42c153780 0x7fb42c0c9be0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb41c006fd0 tx=0x7fb41c01ed10 comp rx=0 tx=0).stop 2023-10-16T20:39:16.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb431279640 1 -- 172.21.15.88:0/2392731086 shutdown_connections 2023-10-16T20:39:16.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb431279640 1 --2- 172.21.15.88:0/2392731086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb42c153780 0x7fb42c0c9be0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:16.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb431279640 1 -- 172.21.15.88:0/2392731086 >> 172.21.15.88:0/2392731086 conn(0x7fb42c06bcb0 msgr2=0x7fb42c06c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:16.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb431279640 1 -- 172.21.15.88:0/2392731086 shutdown_connections 2023-10-16T20:39:16.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb431279640 1 -- 172.21.15.88:0/2392731086 wait complete. 2023-10-16T20:39:16.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb431279640 1 Processor -- start 2023-10-16T20:39:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb431279640 1 -- start start 2023-10-16T20:39:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb431279640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb42c153780 0x7fb42c1f2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb431279640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb42c159490 0x7fb42c1eec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb431279640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb42c1598c0 0x7fb42c1582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb431279640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c0c8e00 con 0x7fb42c1598c0 2023-10-16T20:39:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb431279640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c1f28d0 con 0x7fb42c159490 2023-10-16T20:39:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb431279640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42c1436c0 con 0x7fb42c153780 2023-10-16T20:39:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb42b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb42c1598c0 0x7fb42c1582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb42b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb42c1598c0 0x7fb42c1582d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52846/0 (socket says 172.21.15.88:52846) 2023-10-16T20:39:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb42b7fe640 1 -- 172.21.15.88:0/3975944032 learned_addr learned my addr 172.21.15.88:0/3975944032 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb423fff640 1 --2- 172.21.15.88:0/3975944032 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb42c159490 0x7fb42c1eec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:16.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb42affd640 1 --2- 172.21.15.88:0/3975944032 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb42c153780 0x7fb42c1f2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:16.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb42b7fe640 1 -- 172.21.15.88:0/3975944032 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb42c153780 msgr2=0x7fb42c1f2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:16.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb42b7fe640 1 --2- 172.21.15.88:0/3975944032 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb42c153780 0x7fb42c1f2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:16.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb42b7fe640 1 -- 172.21.15.88:0/3975944032 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb42c159490 msgr2=0x7fb42c1eec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:16.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb42b7fe640 1 --2- 172.21.15.88:0/3975944032 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb42c159490 0x7fb42c1eec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:16.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb42b7fe640 1 -- 172.21.15.88:0/3975944032 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41c009480 con 0x7fb42c1598c0 2023-10-16T20:39:16.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.469+0000 7fb42b7fe640 1 --2- 172.21.15.88:0/3975944032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb42c1598c0 0x7fb42c1582d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb42400c7b0 tx=0x7fb42400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:16.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.473+0000 7fb4237fe640 1 -- 172.21.15.88:0/3975944032 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42400a640 con 0x7fb42c1598c0 2023-10-16T20:39:16.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.473+0000 7fb4237fe640 1 -- 172.21.15.88:0/3975944032 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42400a7c0 con 0x7fb42c1598c0 2023-10-16T20:39:16.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.473+0000 7fb4237fe640 1 -- 172.21.15.88:0/3975944032 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42400ab70 con 0x7fb42c1598c0 2023-10-16T20:39:16.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.473+0000 7fb431279640 1 -- 172.21.15.88:0/3975944032 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42c1f2280 con 0x7fb42c1598c0 2023-10-16T20:39:16.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.473+0000 7fb431279640 1 -- 172.21.15.88:0/3975944032 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42c145d50 con 0x7fb42c1598c0 2023-10-16T20:39:16.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.473+0000 7fb431279640 1 -- 172.21.15.88:0/3975944032 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42c1f2a10 con 0x7fb42c1598c0 2023-10-16T20:39:16.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.473+0000 7fb4237fe640 1 -- 172.21.15.88:0/3975944032 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb42400acd0 con 0x7fb42c1598c0 2023-10-16T20:39:16.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.473+0000 7fb4237fe640 1 --2- 172.21.15.88:0/3975944032 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb40c0664a0 0x7fb40c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:16.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.473+0000 7fb4237fe640 1 -- 172.21.15.88:0/3975944032 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb4240835e0 con 0x7fb42c1598c0 2023-10-16T20:39:16.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.473+0000 7fb42affd640 1 --2- 172.21.15.88:0/3975944032 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb40c0664a0 0x7fb40c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:16.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.477+0000 7fb42affd640 1 --2- 172.21.15.88:0/3975944032 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb40c0664a0 0x7fb40c068960 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb41c0b2000 tx=0x7fb41c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:16.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.477+0000 7fb4237fe640 1 -- 172.21.15.88:0/3975944032 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb424050740 con 0x7fb42c1598c0 2023-10-16T20:39:16.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:16.665+0000 7fb431279640 1 -- 172.21.15.88:0/3975944032 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7fb42c14d810 con 0x7fb42c1598c0 2023-10-16T20:39:17.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.381+0000 7fb4237fe640 1 -- 172.21.15.88:0/3975944032 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v9) v1 ==== 105+0+0 (secure 0 0 0) 0x7fb424053d60 con 0x7fb42c1598c0 2023-10-16T20:39:17.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.381+0000 7fb431279640 1 -- 172.21.15.88:0/3975944032 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb40c0664a0 msgr2=0x7fb40c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:17.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.381+0000 7fb431279640 1 --2- 172.21.15.88:0/3975944032 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb40c0664a0 0x7fb40c068960 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb41c0b2000 tx=0x7fb41c009040 comp rx=0 tx=0).stop 2023-10-16T20:39:17.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.381+0000 7fb431279640 1 -- 172.21.15.88:0/3975944032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb42c1598c0 msgr2=0x7fb42c1582d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:17.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.381+0000 7fb431279640 1 --2- 172.21.15.88:0/3975944032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb42c1598c0 0x7fb42c1582d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb42400c7b0 tx=0x7fb42400cc70 comp rx=0 tx=0).stop 2023-10-16T20:39:17.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.381+0000 7fb431279640 1 -- 172.21.15.88:0/3975944032 shutdown_connections 2023-10-16T20:39:17.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.381+0000 7fb431279640 1 --2- 172.21.15.88:0/3975944032 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb40c0664a0 0x7fb40c068960 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:17.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.381+0000 7fb431279640 1 --2- 172.21.15.88:0/3975944032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb42c1598c0 0x7fb42c1582d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:17.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.381+0000 7fb431279640 1 --2- 172.21.15.88:0/3975944032 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb42c159490 0x7fb42c1eec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:17.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.381+0000 7fb431279640 1 --2- 172.21.15.88:0/3975944032 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb42c153780 0x7fb42c1f2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:17.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.381+0000 7fb431279640 1 -- 172.21.15.88:0/3975944032 >> 172.21.15.88:0/3975944032 conn(0x7fb42c06bcb0 msgr2=0x7fb42c0cbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:17.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.381+0000 7fb431279640 1 -- 172.21.15.88:0/3975944032 shutdown_connections 2023-10-16T20:39:17.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.381+0000 7fb431279640 1 -- 172.21.15.88:0/3975944032 wait complete. 2023-10-16T20:39:17.414 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-16T20:39:17.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.549+0000 7fa684b01640 1 Processor -- start 2023-10-16T20:39:17.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.549+0000 7fa684b01640 1 -- start start 2023-10-16T20:39:17.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.553+0000 7fa684b01640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6800c6ac0 0x7fa680150150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:17.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.553+0000 7fa684b01640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa6800cb5c0 0x7fa680150690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:17.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.553+0000 7fa684b01640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa680070f40 con 0x7fa6800c66e0 2023-10-16T20:39:17.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.553+0000 7fa684b01640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa68006a2d0 con 0x7fa6800cb5c0 2023-10-16T20:39:17.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.553+0000 7fa684b01640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa68006a5b0 con 0x7fa6800c6ac0 2023-10-16T20:39:17.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.553+0000 7fa67e59c640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fa6800c66e0 0x7fa68015a250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:59992/0 (socket says 172.21.15.88:59992) 2023-10-16T20:39:17.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.553+0000 7fa675d9b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6800c6ac0 0x7fa680150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:17.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.553+0000 7fa67e59c640 1 -- 172.21.15.88:0/98337508 learned_addr learned my addr 172.21.15.88:0/98337508 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:17.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.553+0000 7fa67ed9d640 1 --2- 172.21.15.88:0/98337508 >> v2:172.21.15.191:3300/0 conn(0x7fa6800cb5c0 0x7fa680150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:17.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.553+0000 7fa67dd9b640 1 -- 172.21.15.88:0/98337508 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456329231 0 0) 0x7fa680070f40 con 0x7fa6800c66e0 2023-10-16T20:39:17.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.553+0000 7fa67dd9b640 1 -- 172.21.15.88:0/98337508 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa66c003530 con 0x7fa6800c66e0 2023-10-16T20:39:17.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.553+0000 7fa675d9b640 1 -- 172.21.15.88:0/98337508 >> v2:172.21.15.191:3300/0 conn(0x7fa6800cb5c0 msgr2=0x7fa680150690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:17.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.553+0000 7fa675d9b640 1 --2- 172.21.15.88:0/98337508 >> v2:172.21.15.191:3300/0 conn(0x7fa6800cb5c0 0x7fa680150690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:17.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.553+0000 7fa675d9b640 1 -- 172.21.15.88:0/98337508 >> v1:172.21.15.191:6789/0 conn(0x7fa6800c66e0 legacy=0x7fa68015a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:17.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.553+0000 7fa675d9b640 1 -- 172.21.15.88:0/98337508 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68006a890 con 0x7fa6800c6ac0 2023-10-16T20:39:17.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.553+0000 7fa67ed9d640 1 --2- 172.21.15.88:0/98337508 >> v2:172.21.15.191:3300/0 conn(0x7fa6800cb5c0 0x7fa680150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:39:17.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.557+0000 7fa675d9b640 1 --2- 172.21.15.88:0/98337508 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6800c6ac0 0x7fa680150150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa67000ba60 tx=0x7fa67001f520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dbeff5075e0482 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:17.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.557+0000 7fa67dd9b640 1 -- 172.21.15.88:0/98337508 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6700a7aa0 con 0x7fa6800c6ac0 2023-10-16T20:39:17.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.557+0000 7fa67dd9b640 1 -- 172.21.15.88:0/98337508 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6700a7c00 con 0x7fa6800c6ac0 2023-10-16T20:39:17.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.557+0000 7fa67dd9b640 1 -- 172.21.15.88:0/98337508 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6700af440 con 0x7fa6800c6ac0 2023-10-16T20:39:17.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.557+0000 7fa684b01640 1 -- 172.21.15.88:0/98337508 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6800c6ac0 msgr2=0x7fa680150150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:17.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.557+0000 7fa684b01640 1 --2- 172.21.15.88:0/98337508 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6800c6ac0 0x7fa680150150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa67000ba60 tx=0x7fa67001f520 comp rx=0 tx=0).stop 2023-10-16T20:39:17.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.557+0000 7fa684b01640 1 -- 172.21.15.88:0/98337508 shutdown_connections 2023-10-16T20:39:17.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.557+0000 7fa684b01640 1 --2- 172.21.15.88:0/98337508 >> v2:172.21.15.191:3300/0 conn(0x7fa6800cb5c0 0x7fa680150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:17.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.557+0000 7fa684b01640 1 --2- 172.21.15.88:0/98337508 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6800c6ac0 0x7fa680150150 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:17.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.557+0000 7fa684b01640 1 -- 172.21.15.88:0/98337508 >> 172.21.15.88:0/98337508 conn(0x7fa6800ca9f0 msgr2=0x7fa6800cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:17.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.557+0000 7fa684b01640 1 -- 172.21.15.88:0/98337508 shutdown_connections 2023-10-16T20:39:17.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.557+0000 7fa684b01640 1 -- 172.21.15.88:0/98337508 wait complete. 2023-10-16T20:39:17.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.557+0000 7fa684b01640 1 Processor -- start 2023-10-16T20:39:17.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.557+0000 7fa684b01640 1 -- start start 2023-10-16T20:39:17.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa684b01640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6800c66e0 0x7fa68014cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:17.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa684b01640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6800c6ac0 0x7fa680150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:17.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa684b01640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6800cb5c0 0x7fa680156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:17.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa684b01640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68014f0b0 con 0x7fa6800c6ac0 2023-10-16T20:39:17.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa684b01640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6801537f0 con 0x7fa6800cb5c0 2023-10-16T20:39:17.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa684b01640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa680153a90 con 0x7fa6800c66e0 2023-10-16T20:39:17.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa675d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6800c6ac0 0x7fa680150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:17.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa675d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6800c6ac0 0x7fa680150180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52858/0 (socket says 172.21.15.88:52858) 2023-10-16T20:39:17.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa675d9b640 1 -- 172.21.15.88:0/1418609097 learned_addr learned my addr 172.21.15.88:0/1418609097 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:17.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa67e59c640 1 --2- 172.21.15.88:0/1418609097 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6800c66e0 0x7fa68014cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:17.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa67ed9d640 1 --2- 172.21.15.88:0/1418609097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6800cb5c0 0x7fa680156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:17.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa675d9b640 1 -- 172.21.15.88:0/1418609097 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6800c66e0 msgr2=0x7fa68014cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:17.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa675d9b640 1 --2- 172.21.15.88:0/1418609097 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6800c66e0 0x7fa68014cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:17.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa675d9b640 1 -- 172.21.15.88:0/1418609097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6800cb5c0 msgr2=0x7fa680156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:17.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa675d9b640 1 --2- 172.21.15.88:0/1418609097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6800cb5c0 0x7fa680156080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:17.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa675d9b640 1 -- 172.21.15.88:0/1418609097 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa678009a70 con 0x7fa6800c6ac0 2023-10-16T20:39:17.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa675d9b640 1 --2- 172.21.15.88:0/1418609097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6800c6ac0 0x7fa680150180 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa67001c6f0 tx=0x7fa67001b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:17.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa6777fe640 1 -- 172.21.15.88:0/1418609097 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6700afbc0 con 0x7fa6800c6ac0 2023-10-16T20:39:17.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa6777fe640 1 -- 172.21.15.88:0/1418609097 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6700afd20 con 0x7fa6800c6ac0 2023-10-16T20:39:17.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa6777fe640 1 -- 172.21.15.88:0/1418609097 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6700ba690 con 0x7fa6800c6ac0 2023-10-16T20:39:17.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa684b01640 1 -- 172.21.15.88:0/1418609097 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa67000b6c0 con 0x7fa6800c6ac0 2023-10-16T20:39:17.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa684b01640 1 -- 172.21.15.88:0/1418609097 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6801f17f0 con 0x7fa6800c6ac0 2023-10-16T20:39:17.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa684b01640 1 -- 172.21.15.88:0/1418609097 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa680074700 con 0x7fa6800c6ac0 2023-10-16T20:39:17.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.561+0000 7fa6777fe640 1 -- 172.21.15.88:0/1418609097 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa6700b6070 con 0x7fa6800c6ac0 2023-10-16T20:39:17.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.565+0000 7fa6777fe640 1 --2- 172.21.15.88:0/1418609097 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6600664a0 0x7fa660068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:17.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.565+0000 7fa6777fe640 1 -- 172.21.15.88:0/1418609097 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa67011f340 con 0x7fa6800c6ac0 2023-10-16T20:39:17.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.565+0000 7fa67e59c640 1 --2- 172.21.15.88:0/1418609097 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6600664a0 0x7fa660068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:17.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.565+0000 7fa67e59c640 1 --2- 172.21.15.88:0/1418609097 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6600664a0 0x7fa660068960 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa6680011f0 tx=0x7fa668009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:17.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.569+0000 7fa6777fe640 1 -- 172.21.15.88:0/1418609097 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa6700ec360 con 0x7fa6800c6ac0 2023-10-16T20:39:17.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:17.749+0000 7fa684b01640 1 -- 172.21.15.88:0/1418609097 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa6800c6260 con 0x7fa6800c6ac0 2023-10-16T20:39:18.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.393+0000 7fa6777fe640 1 -- 172.21.15.88:0/1418609097 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v11) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa6700ef980 con 0x7fa6800c6ac0 2023-10-16T20:39:18.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.397+0000 7fa684b01640 1 -- 172.21.15.88:0/1418609097 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6600664a0 msgr2=0x7fa660068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:18.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.397+0000 7fa684b01640 1 --2- 172.21.15.88:0/1418609097 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6600664a0 0x7fa660068960 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa6680011f0 tx=0x7fa668009040 comp rx=0 tx=0).stop 2023-10-16T20:39:18.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.397+0000 7fa684b01640 1 -- 172.21.15.88:0/1418609097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6800c6ac0 msgr2=0x7fa680150180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:18.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.397+0000 7fa684b01640 1 --2- 172.21.15.88:0/1418609097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6800c6ac0 0x7fa680150180 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa67001c6f0 tx=0x7fa67001b860 comp rx=0 tx=0).stop 2023-10-16T20:39:18.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.397+0000 7fa684b01640 1 -- 172.21.15.88:0/1418609097 shutdown_connections 2023-10-16T20:39:18.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.397+0000 7fa684b01640 1 --2- 172.21.15.88:0/1418609097 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6600664a0 0x7fa660068960 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:18.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.397+0000 7fa684b01640 1 --2- 172.21.15.88:0/1418609097 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6800cb5c0 0x7fa680156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:18.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.397+0000 7fa684b01640 1 --2- 172.21.15.88:0/1418609097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6800c6ac0 0x7fa680150180 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:18.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.397+0000 7fa684b01640 1 --2- 172.21.15.88:0/1418609097 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6800c66e0 0x7fa68014cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:18.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.397+0000 7fa684b01640 1 -- 172.21.15.88:0/1418609097 >> 172.21.15.88:0/1418609097 conn(0x7fa6800ca9f0 msgr2=0x7fa680159630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:18.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.397+0000 7fa684b01640 1 -- 172.21.15.88:0/1418609097 shutdown_connections 2023-10-16T20:39:18.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.397+0000 7fa684b01640 1 -- 172.21.15.88:0/1418609097 wait complete. 2023-10-16T20:39:18.425 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-10-16T20:39:18.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.561+0000 7f21a73d9640 1 Processor -- start 2023-10-16T20:39:18.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.561+0000 7f21a73d9640 1 -- start start 2023-10-16T20:39:18.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.561+0000 7f21a73d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21a0157b20 0x7f21a0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:18.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.561+0000 7f21a73d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f21a006ae00 0x7f21a00c20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:18.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.561+0000 7f21a73d9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a015dba0 con 0x7f21a0157f00 2023-10-16T20:39:18.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.561+0000 7f21a73d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a015d9b0 con 0x7f21a006ae00 2023-10-16T20:39:18.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.561+0000 7f21a73d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a0072a10 con 0x7f21a0157b20 2023-10-16T20:39:18.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.561+0000 7f21a4974640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f21a006ae00 0x7f21a00c20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:18.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.561+0000 7f21a4974640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f21a006ae00 0x7f21a00c20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52870/0 (socket says 172.21.15.88:52870) 2023-10-16T20:39:18.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.561+0000 7f21a5976640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f21a0157f00 0x7f21a00c5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50480/0 (socket says 172.21.15.88:50480) 2023-10-16T20:39:18.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.561+0000 7f21a4974640 1 -- 172.21.15.88:0/3780226992 learned_addr learned my addr 172.21.15.88:0/3780226992 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:18.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.561+0000 7f21a5175640 1 --2- 172.21.15.88:0/3780226992 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21a0157b20 0x7f21a0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:18.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.561+0000 7f21a4974640 1 -- 172.21.15.88:0/3780226992 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21a0157b20 msgr2=0x7f21a0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:18.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.561+0000 7f21a4974640 1 --2- 172.21.15.88:0/3780226992 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21a0157b20 0x7f21a0069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:18.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.561+0000 7f2197fff640 1 -- 172.21.15.88:0/3780226992 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052532755 0 0) 0x7f21a015dba0 con 0x7f21a0157f00 2023-10-16T20:39:18.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.561+0000 7f21a4974640 1 -- 172.21.15.88:0/3780226992 >> v1:172.21.15.88:6789/0 conn(0x7f21a0157f00 legacy=0x7f21a00c5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:18.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.561+0000 7f21a4974640 1 -- 172.21.15.88:0/3780226992 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a015de80 con 0x7f21a006ae00 2023-10-16T20:39:18.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.565+0000 7f21a4974640 1 --2- 172.21.15.88:0/3780226992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21a006ae00 0x7f21a00c20e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f21900098a0 tx=0x7f219001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb074c27ec17dc4f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:18.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.565+0000 7f2197fff640 1 -- 172.21.15.88:0/3780226992 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21900a8a10 con 0x7f21a006ae00 2023-10-16T20:39:18.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.565+0000 7f2197fff640 1 -- 172.21.15.88:0/3780226992 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21900a8b70 con 0x7f21a006ae00 2023-10-16T20:39:18.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.565+0000 7f2197fff640 1 -- 172.21.15.88:0/3780226992 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21900b0440 con 0x7f21a006ae00 2023-10-16T20:39:18.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.565+0000 7f21a73d9640 1 -- 172.21.15.88:0/3780226992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21a006ae00 msgr2=0x7f21a00c20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:18.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.565+0000 7f21a73d9640 1 --2- 172.21.15.88:0/3780226992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21a006ae00 0x7f21a00c20e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f21900098a0 tx=0x7f219001ed10 comp rx=0 tx=0).stop 2023-10-16T20:39:18.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.565+0000 7f21a73d9640 1 -- 172.21.15.88:0/3780226992 shutdown_connections 2023-10-16T20:39:18.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.565+0000 7f21a73d9640 1 --2- 172.21.15.88:0/3780226992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21a006ae00 0x7f21a00c20e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:18.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.565+0000 7f21a73d9640 1 --2- 172.21.15.88:0/3780226992 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21a0157b20 0x7f21a0069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:18.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.565+0000 7f21a73d9640 1 -- 172.21.15.88:0/3780226992 >> 172.21.15.88:0/3780226992 conn(0x7f21a00ca9f0 msgr2=0x7f21a00cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:18.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.565+0000 7f21a73d9640 1 -- 172.21.15.88:0/3780226992 shutdown_connections 2023-10-16T20:39:18.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.565+0000 7f21a73d9640 1 -- 172.21.15.88:0/3780226992 wait complete. 2023-10-16T20:39:18.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a73d9640 1 Processor -- start 2023-10-16T20:39:18.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a73d9640 1 -- start start 2023-10-16T20:39:18.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a73d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21a006ae00 0x7f21a0152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:18.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a73d9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21a0157b20 0x7f21a01f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:18.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a73d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21a0157f00 0x7f21a015af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:18.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a73d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a01f6620 con 0x7f21a0157f00 2023-10-16T20:39:18.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a73d9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a01517c0 con 0x7f21a0157b20 2023-10-16T20:39:18.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a73d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a01ea1f0 con 0x7f21a006ae00 2023-10-16T20:39:18.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a5976640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21a0157f00 0x7f21a015af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:18.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a5976640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21a0157f00 0x7f21a015af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52874/0 (socket says 172.21.15.88:52874) 2023-10-16T20:39:18.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a5976640 1 -- 172.21.15.88:0/1989517902 learned_addr learned my addr 172.21.15.88:0/1989517902 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:18.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a5175640 1 --2- 172.21.15.88:0/1989517902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21a006ae00 0x7f21a0152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:18.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a4974640 1 --2- 172.21.15.88:0/1989517902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21a0157b20 0x7f21a01f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:18.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a5976640 1 -- 172.21.15.88:0/1989517902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21a006ae00 msgr2=0x7f21a0152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:18.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a5976640 1 --2- 172.21.15.88:0/1989517902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21a006ae00 0x7f21a0152600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:18.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a5976640 1 -- 172.21.15.88:0/1989517902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21a0157b20 msgr2=0x7f21a01f6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:18.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a5976640 1 --2- 172.21.15.88:0/1989517902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21a0157b20 0x7f21a01f6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:18.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a5976640 1 -- 172.21.15.88:0/1989517902 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21900094d0 con 0x7f21a0157f00 2023-10-16T20:39:18.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a5976640 1 --2- 172.21.15.88:0/1989517902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21a0157f00 0x7f21a015af80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f219800c780 tx=0x7f219800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:18.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f2195ffb640 1 -- 172.21.15.88:0/1989517902 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219800ce90 con 0x7f21a0157f00 2023-10-16T20:39:18.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f2195ffb640 1 -- 172.21.15.88:0/1989517902 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219800a640 con 0x7f21a0157f00 2023-10-16T20:39:18.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f2195ffb640 1 -- 172.21.15.88:0/1989517902 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219800a9b0 con 0x7f21a0157f00 2023-10-16T20:39:18.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a73d9640 1 -- 172.21.15.88:0/1989517902 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a01ea4d0 con 0x7f21a0157f00 2023-10-16T20:39:18.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.569+0000 7f21a73d9640 1 -- 172.21.15.88:0/1989517902 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a01f5f60 con 0x7f21a0157f00 2023-10-16T20:39:18.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.573+0000 7f21a73d9640 1 -- 172.21.15.88:0/1989517902 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a01f6760 con 0x7f21a0157f00 2023-10-16T20:39:18.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.573+0000 7f2195ffb640 1 -- 172.21.15.88:0/1989517902 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f219800ab10 con 0x7f21a0157f00 2023-10-16T20:39:18.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.573+0000 7f2195ffb640 1 --2- 172.21.15.88:0/1989517902 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21840664a0 0x7f2184068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:18.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.573+0000 7f2195ffb640 1 -- 172.21.15.88:0/1989517902 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f21980835d0 con 0x7f21a0157f00 2023-10-16T20:39:18.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.573+0000 7f21a5175640 1 --2- 172.21.15.88:0/1989517902 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21840664a0 0x7f2184068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:18.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.573+0000 7f21a5175640 1 --2- 172.21.15.88:0/1989517902 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21840664a0 0x7f2184068960 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f219c007a60 tx=0x7f219c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:18.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.577+0000 7f2195ffb640 1 -- 172.21.15.88:0/1989517902 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f21a01f6760 con 0x7f21a0157f00 2023-10-16T20:39:18.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:18.761+0000 7f21a73d9640 1 -- 172.21.15.88:0/1989517902 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f21a01f6760 con 0x7f21a0157f00 2023-10-16T20:39:19.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.397+0000 7f2195ffb640 1 -- 172.21.15.88:0/1989517902 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v12) v1 ==== 115+0+0 (secure 0 0 0) 0x7f21a01f6760 con 0x7f21a0157f00 2023-10-16T20:39:19.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.397+0000 7f21a73d9640 1 -- 172.21.15.88:0/1989517902 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21840664a0 msgr2=0x7f2184068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:19.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.397+0000 7f21a73d9640 1 --2- 172.21.15.88:0/1989517902 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21840664a0 0x7f2184068960 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f219c007a60 tx=0x7f219c008040 comp rx=0 tx=0).stop 2023-10-16T20:39:19.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.397+0000 7f21a73d9640 1 -- 172.21.15.88:0/1989517902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21a0157f00 msgr2=0x7f21a015af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:19.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.397+0000 7f21a73d9640 1 --2- 172.21.15.88:0/1989517902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21a0157f00 0x7f21a015af80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f219800c780 tx=0x7f219800cc40 comp rx=0 tx=0).stop 2023-10-16T20:39:19.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.397+0000 7f21a73d9640 1 -- 172.21.15.88:0/1989517902 shutdown_connections 2023-10-16T20:39:19.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.397+0000 7f21a73d9640 1 --2- 172.21.15.88:0/1989517902 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21840664a0 0x7f2184068960 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.397+0000 7f21a73d9640 1 --2- 172.21.15.88:0/1989517902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21a0157f00 0x7f21a015af80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.397+0000 7f21a73d9640 1 --2- 172.21.15.88:0/1989517902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21a0157b20 0x7f21a01f6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.397+0000 7f21a73d9640 1 --2- 172.21.15.88:0/1989517902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21a006ae00 0x7f21a0152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.397+0000 7f21a73d9640 1 -- 172.21.15.88:0/1989517902 >> 172.21.15.88:0/1989517902 conn(0x7f21a00ca9f0 msgr2=0x7f21a00cadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:19.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.397+0000 7f21a73d9640 1 -- 172.21.15.88:0/1989517902 shutdown_connections 2023-10-16T20:39:19.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.397+0000 7f21a73d9640 1 -- 172.21.15.88:0/1989517902 wait complete. 2023-10-16T20:39:19.428 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-16T20:39:19.428 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:39:19.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.561+0000 7f689159d640 1 Processor -- start 2023-10-16T20:39:19.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.561+0000 7f689159d640 1 -- start start 2023-10-16T20:39:19.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.561+0000 7f689159d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f688c153770 0x7f688c159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:19.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.561+0000 7f689159d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f688c0c9bb0 0x7f688c0c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:19.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.561+0000 7f689159d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f688c15a410 con 0x7f688c0c41f0 2023-10-16T20:39:19.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.561+0000 7f689159d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f688c0cc250 con 0x7f688c153770 2023-10-16T20:39:19.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.561+0000 7f689159d640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f688c0cc670 con 0x7f688c0c9bb0 2023-10-16T20:39:19.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.561+0000 7f688affd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f688c153770 0x7f688c159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:19.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.561+0000 7f688affd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f688c153770 0x7f688c159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56218/0 (socket says 172.21.15.88:56218) 2023-10-16T20:39:19.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.561+0000 7f688affd640 1 -- 172.21.15.88:0/3594930331 learned_addr learned my addr 172.21.15.88:0/3594930331 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:19.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.561+0000 7f688affd640 1 -- 172.21.15.88:0/3594930331 >> v2:172.21.15.191:3300/0 conn(0x7f688c0c9bb0 msgr2=0x7f688c0c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:19.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.561+0000 7f68827fc640 1 --2- 172.21.15.88:0/3594930331 >> v2:172.21.15.191:3300/0 conn(0x7f688c0c9bb0 0x7f688c0c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:19.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.561+0000 7f688affd640 1 --2- 172.21.15.88:0/3594930331 >> v2:172.21.15.191:3300/0 conn(0x7f688c0c9bb0 0x7f688c0c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.561+0000 7f688affd640 1 -- 172.21.15.88:0/3594930331 >> v1:172.21.15.191:6789/0 conn(0x7f688c0c41f0 legacy=0x7f688c0cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:19.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.561+0000 7f688affd640 1 -- 172.21.15.88:0/3594930331 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688c153b50 con 0x7f688c153770 2023-10-16T20:39:19.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.561+0000 7f68827fc640 1 --2- 172.21.15.88:0/3594930331 >> v2:172.21.15.191:3300/0 conn(0x7f688c0c9bb0 0x7f688c0c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:19.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.565+0000 7f688affd640 1 --2- 172.21.15.88:0/3594930331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f688c153770 0x7f688c159ed0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f688c0cc7b0 tx=0x7f687c01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce48053bfb829b6d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:19.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.565+0000 7f688a7fc640 1 -- 172.21.15.88:0/3594930331 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c0ab5f0 con 0x7f688c153770 2023-10-16T20:39:19.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.565+0000 7f688a7fc640 1 -- 172.21.15.88:0/3594930331 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687c0ab750 con 0x7f688c153770 2023-10-16T20:39:19.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.565+0000 7f688a7fc640 1 -- 172.21.15.88:0/3594930331 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c0abae0 con 0x7f688c153770 2023-10-16T20:39:19.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.565+0000 7f689159d640 1 -- 172.21.15.88:0/3594930331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f688c153770 msgr2=0x7f688c159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:19.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.565+0000 7f689159d640 1 --2- 172.21.15.88:0/3594930331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f688c153770 0x7f688c159ed0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f688c0cc7b0 tx=0x7f687c01c4e0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.565+0000 7f689159d640 1 -- 172.21.15.88:0/3594930331 shutdown_connections 2023-10-16T20:39:19.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.565+0000 7f689159d640 1 --2- 172.21.15.88:0/3594930331 >> v2:172.21.15.191:3300/0 conn(0x7f688c0c9bb0 0x7f688c0c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.565+0000 7f689159d640 1 --2- 172.21.15.88:0/3594930331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f688c153770 0x7f688c159ed0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.565+0000 7f689159d640 1 -- 172.21.15.88:0/3594930331 >> 172.21.15.88:0/3594930331 conn(0x7f688c0715c0 msgr2=0x7f688c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:19.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.565+0000 7f689159d640 1 -- 172.21.15.88:0/3594930331 shutdown_connections 2023-10-16T20:39:19.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.565+0000 7f689159d640 1 -- 172.21.15.88:0/3594930331 wait complete. 2023-10-16T20:39:19.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f689159d640 1 Processor -- start 2023-10-16T20:39:19.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f689159d640 1 -- start start 2023-10-16T20:39:19.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f689159d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f688c0c41f0 0x7f688c15d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:19.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f689159d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f688c0c9bb0 0x7f688c0c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:19.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f689159d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f688c153770 0x7f688c154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:19.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f689159d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688c154080 con 0x7f688c0c41f0 2023-10-16T20:39:19.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f689159d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688c1f1920 con 0x7f688c153770 2023-10-16T20:39:19.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f689159d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688c1f2480 con 0x7f688c0c9bb0 2023-10-16T20:39:19.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f688affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f688c0c41f0 0x7f688c15d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:19.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f688affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f688c0c41f0 0x7f688c15d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56224/0 (socket says 172.21.15.88:56224) 2023-10-16T20:39:19.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f688affd640 1 -- 172.21.15.88:0/3628973817 learned_addr learned my addr 172.21.15.88:0/3628973817 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:19.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f688b7fe640 1 --2- 172.21.15.88:0/3628973817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f688c153770 0x7f688c154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:19.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f68827fc640 1 --2- 172.21.15.88:0/3628973817 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f688c0c9bb0 0x7f688c0c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:19.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f688affd640 1 -- 172.21.15.88:0/3628973817 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f688c0c9bb0 msgr2=0x7f688c0c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:19.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f688affd640 1 --2- 172.21.15.88:0/3628973817 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f688c0c9bb0 0x7f688c0c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f688affd640 1 -- 172.21.15.88:0/3628973817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f688c153770 msgr2=0x7f688c154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:19.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f688affd640 1 --2- 172.21.15.88:0/3628973817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f688c153770 0x7f688c154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f688affd640 1 -- 172.21.15.88:0/3628973817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687c009480 con 0x7f688c0c41f0 2023-10-16T20:39:19.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f688affd640 1 --2- 172.21.15.88:0/3628973817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f688c0c41f0 0x7f688c15d300 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f687c01a110 tx=0x7f687c019280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:19.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f6883fff640 1 -- 172.21.15.88:0/3628973817 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c0abcd0 con 0x7f688c0c41f0 2023-10-16T20:39:19.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f6883fff640 1 -- 172.21.15.88:0/3628973817 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687c0abe30 con 0x7f688c0c41f0 2023-10-16T20:39:19.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f6883fff640 1 -- 172.21.15.88:0/3628973817 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c003860 con 0x7f688c0c41f0 2023-10-16T20:39:19.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f689159d640 1 -- 172.21.15.88:0/3628973817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f688c1f1be0 con 0x7f688c0c41f0 2023-10-16T20:39:19.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.569+0000 7f689159d640 1 -- 172.21.15.88:0/3628973817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f688c1f21a0 con 0x7f688c0c41f0 2023-10-16T20:39:19.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.573+0000 7f689159d640 1 -- 172.21.15.88:0/3628973817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f688c0cbb70 con 0x7f688c0c41f0 2023-10-16T20:39:19.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.573+0000 7f6883fff640 1 -- 172.21.15.88:0/3628973817 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f687c0039c0 con 0x7f688c0c41f0 2023-10-16T20:39:19.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.573+0000 7f6883fff640 1 --2- 172.21.15.88:0/3628973817 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f686c0664a0 0x7f686c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:19.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.573+0000 7f6883fff640 1 -- 172.21.15.88:0/3628973817 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f687c0a8050 con 0x7f688c0c41f0 2023-10-16T20:39:19.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.573+0000 7f68827fc640 1 --2- 172.21.15.88:0/3628973817 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f686c0664a0 0x7f686c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:19.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.573+0000 7f68827fc640 1 --2- 172.21.15.88:0/3628973817 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f686c0664a0 0x7f686c068960 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f688c15d1d0 tx=0x7f6874007810 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:19.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.577+0000 7f6883fff640 1 -- 172.21.15.88:0/3628973817 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f688c0cbb70 con 0x7f688c0c41f0 2023-10-16T20:39:19.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.757+0000 7f689159d640 1 -- 172.21.15.88:0/3628973817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f688c0cbb70 con 0x7f688c0c41f0 2023-10-16T20:39:19.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.761+0000 7f6883fff640 1 -- 172.21.15.88:0/3628973817 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6512 (secure 0 0 0) 0x7f688c0cbb70 con 0x7f688c0c41f0 2023-10-16T20:39:19.767 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:19.768 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4275,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:39:18.766425+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1],"up":{"mds_0":4288,"mds_1":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:creating","state_seq":1,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:39:19.768 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 12 2023-10-16T20:39:19.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.761+0000 7f689159d640 1 -- 172.21.15.88:0/3628973817 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f686c0664a0 msgr2=0x7f686c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:19.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.761+0000 7f689159d640 1 --2- 172.21.15.88:0/3628973817 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f686c0664a0 0x7f686c068960 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f688c15d1d0 tx=0x7f6874007810 comp rx=0 tx=0).stop 2023-10-16T20:39:19.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.761+0000 7f689159d640 1 -- 172.21.15.88:0/3628973817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f688c0c41f0 msgr2=0x7f688c15d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:19.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.761+0000 7f689159d640 1 --2- 172.21.15.88:0/3628973817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f688c0c41f0 0x7f688c15d300 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f687c01a110 tx=0x7f687c019280 comp rx=0 tx=0).stop 2023-10-16T20:39:19.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.761+0000 7f689159d640 1 -- 172.21.15.88:0/3628973817 shutdown_connections 2023-10-16T20:39:19.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.761+0000 7f689159d640 1 --2- 172.21.15.88:0/3628973817 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f686c0664a0 0x7f686c068960 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.761+0000 7f689159d640 1 --2- 172.21.15.88:0/3628973817 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f688c153770 0x7f688c154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.761+0000 7f689159d640 1 --2- 172.21.15.88:0/3628973817 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f688c0c9bb0 0x7f688c0c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.761+0000 7f689159d640 1 --2- 172.21.15.88:0/3628973817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f688c0c41f0 0x7f688c15d300 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.761+0000 7f689159d640 1 -- 172.21.15.88:0/3628973817 >> 172.21.15.88:0/3628973817 conn(0x7f688c0715c0 msgr2=0x7f688c0cd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:19.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.761+0000 7f689159d640 1 -- 172.21.15.88:0/3628973817 shutdown_connections 2023-10-16T20:39:19.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.761+0000 7f689159d640 1 -- 172.21.15.88:0/3628973817 wait complete. 2023-10-16T20:39:19.789 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:39:19.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.917+0000 7f8e1d10f640 1 Processor -- start 2023-10-16T20:39:19.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.917+0000 7f8e1d10f640 1 -- start start 2023-10-16T20:39:19.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1d10f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8e18153770 0x7f8e18159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:19.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1d10f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e180c9bb0 0x7f8e180c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:19.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1d10f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1815a410 con 0x7f8e180c41f0 2023-10-16T20:39:19.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1d10f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e180cc250 con 0x7f8e18153770 2023-10-16T20:39:19.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1d10f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e180cc670 con 0x7f8e180c9bb0 2023-10-16T20:39:19.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1759e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8e180c41f0 0x7f8e180cece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36848/0 (socket says 172.21.15.88:36848) 2023-10-16T20:39:19.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1759e640 1 -- 172.21.15.88:0/3333672517 learned_addr learned my addr 172.21.15.88:0/3333672517 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:19.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e16d9d640 1 --2- 172.21.15.88:0/3333672517 >> v2:172.21.15.191:3300/0 conn(0x7f8e18153770 0x7f8e18159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:19.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e0e59c640 1 --2- 172.21.15.88:0/3333672517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e180c9bb0 0x7f8e180c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:19.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1659c640 1 -- 172.21.15.88:0/3333672517 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564346387 0 0) 0x7f8e1815a410 con 0x7f8e180c41f0 2023-10-16T20:39:19.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1659c640 1 -- 172.21.15.88:0/3333672517 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e04003530 con 0x7f8e180c41f0 2023-10-16T20:39:19.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1659c640 1 -- 172.21.15.88:0/3333672517 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 850070378 0 0) 0x7f8e04003530 con 0x7f8e180c41f0 2023-10-16T20:39:19.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1659c640 1 -- 172.21.15.88:0/3333672517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e180c9bb0 msgr2=0x7f8e180c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:19.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1659c640 1 --2- 172.21.15.88:0/3333672517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e180c9bb0 0x7f8e180c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1659c640 1 -- 172.21.15.88:0/3333672517 >> v2:172.21.15.191:3300/0 conn(0x7f8e18153770 msgr2=0x7f8e18159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:19.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1659c640 1 --2- 172.21.15.88:0/3333672517 >> v2:172.21.15.191:3300/0 conn(0x7f8e18153770 0x7f8e18159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1659c640 1 -- 172.21.15.88:0/3333672517 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e18153b50 con 0x7f8e180c41f0 2023-10-16T20:39:19.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1659c640 1 -- 172.21.15.88:0/3333672517 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f8e10001a00 con 0x7f8e180c41f0 2023-10-16T20:39:19.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1659c640 1 -- 172.21.15.88:0/3333672517 >> v1:172.21.15.88:6789/0 conn(0x7f8e180c41f0 legacy=0x7f8e180cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:19.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1659c640 1 --2- 172.21.15.88:0/3333672517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e04003980 0x7f8e04003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:19.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1659c640 1 --2- 172.21.15.88:0/3333672517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e180c9bb0 0x7f8e040047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:19.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1659c640 1 --2- 172.21.15.88:0/3333672517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e04004d10 0x7f8e04005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:19.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1659c640 1 -- 172.21.15.88:0/3333672517 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e180cc250 con 0x7f8e04003980 2023-10-16T20:39:19.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1659c640 1 -- 172.21.15.88:0/3333672517 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e180cc670 con 0x7f8e04004d10 2023-10-16T20:39:19.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1659c640 1 -- 172.21.15.88:0/3333672517 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0400b990 con 0x7f8e180c9bb0 2023-10-16T20:39:19.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e16d9d640 1 --2- 172.21.15.88:0/3333672517 >> v2:172.21.15.191:3300/0 conn(0x7f8e18153770 0x7f8e18159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:39:19.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e16d9d640 1 --2- 172.21.15.88:0/3333672517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e04003980 0x7f8e04003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:19.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e0e59c640 1 --2- 172.21.15.88:0/3333672517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e180c9bb0 0x7f8e040047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:19.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e1759e640 1 --2- 172.21.15.88:0/3333672517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e04004d10 0x7f8e04005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:19.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e16d9d640 1 -- 172.21.15.88:0/3333672517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e180c9bb0 msgr2=0x7f8e040047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:19.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e16d9d640 1 --2- 172.21.15.88:0/3333672517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e180c9bb0 0x7f8e040047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e16d9d640 1 -- 172.21.15.88:0/3333672517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e04004d10 msgr2=0x7f8e04005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:19.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e16d9d640 1 --2- 172.21.15.88:0/3333672517 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e04004d10 0x7f8e04005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.921+0000 7f8e16d9d640 1 -- 172.21.15.88:0/3333672517 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e04003530 con 0x7f8e04003980 2023-10-16T20:39:19.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.925+0000 7f8e16d9d640 1 --2- 172.21.15.88:0/3333672517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e04003980 0x7f8e04003da0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8e08009ca0 tx=0x7f8e0801cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=add82763dca64e6b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:19.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.925+0000 7f8e1659c640 1 -- 172.21.15.88:0/3333672517 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e08004020 con 0x7f8e04003980 2023-10-16T20:39:19.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.925+0000 7f8e1659c640 1 -- 172.21.15.88:0/3333672517 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e08004180 con 0x7f8e04003980 2023-10-16T20:39:19.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.925+0000 7f8e1659c640 1 -- 172.21.15.88:0/3333672517 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e080a57c0 con 0x7f8e04003980 2023-10-16T20:39:19.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.925+0000 7f8e1d10f640 1 -- 172.21.15.88:0/3333672517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e04003980 msgr2=0x7f8e04003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:19.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.925+0000 7f8e1d10f640 1 --2- 172.21.15.88:0/3333672517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e04003980 0x7f8e04003da0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8e08009ca0 tx=0x7f8e0801cc10 comp rx=0 tx=0).stop 2023-10-16T20:39:19.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.925+0000 7f8e1759e640 1 -- 172.21.15.88:0/3333672517 reap_dead start 2023-10-16T20:39:19.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.925+0000 7f8e1d10f640 1 -- 172.21.15.88:0/3333672517 shutdown_connections 2023-10-16T20:39:19.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.925+0000 7f8e1d10f640 1 -- 172.21.15.88:0/3333672517 >> 172.21.15.88:0/3333672517 conn(0x7f8e180715c0 msgr2=0x7f8e180719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:19.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.925+0000 7f8e1d10f640 1 -- 172.21.15.88:0/3333672517 shutdown_connections 2023-10-16T20:39:19.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.925+0000 7f8e1d10f640 1 -- 172.21.15.88:0/3333672517 wait complete. 2023-10-16T20:39:19.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1d10f640 1 Processor -- start 2023-10-16T20:39:19.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1d10f640 1 -- start start 2023-10-16T20:39:19.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1d10f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e04003980 0x7f8e180c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:19.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1d10f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e180b4b50 0x7f8e181528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:19.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1d10f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e18152de0 0x7f8e181456f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:19.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1d10f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e18151ef0 con 0x7f8e18152de0 2023-10-16T20:39:19.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1d10f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e18152160 con 0x7f8e180b4b50 2023-10-16T20:39:19.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1d10f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e180b4070 con 0x7f8e04003980 2023-10-16T20:39:19.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e18152de0 0x7f8e181456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:19.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e18152de0 0x7f8e181456f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56232/0 (socket says 172.21.15.88:56232) 2023-10-16T20:39:19.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1759e640 1 -- 172.21.15.88:0/4271031108 learned_addr learned my addr 172.21.15.88:0/4271031108 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:19.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e0e59c640 1 --2- 172.21.15.88:0/4271031108 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e180b4b50 0x7f8e181528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:19.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e16d9d640 1 --2- 172.21.15.88:0/4271031108 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e04003980 0x7f8e180c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:19.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1759e640 1 -- 172.21.15.88:0/4271031108 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e04003980 msgr2=0x7f8e180c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:19.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1759e640 1 --2- 172.21.15.88:0/4271031108 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e04003980 0x7f8e180c6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1759e640 1 -- 172.21.15.88:0/4271031108 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e180b4b50 msgr2=0x7f8e181528a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:19.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1759e640 1 --2- 172.21.15.88:0/4271031108 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e180b4b50 0x7f8e181528a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:19.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1759e640 1 -- 172.21.15.88:0/4271031108 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e180b4350 con 0x7f8e18152de0 2023-10-16T20:39:19.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1759e640 1 --2- 172.21.15.88:0/4271031108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e18152de0 0x7f8e181456f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8e1808cd30 tx=0x7f8e10007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:19.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e0ffff640 1 -- 172.21.15.88:0/4271031108 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1000ac00 con 0x7f8e18152de0 2023-10-16T20:39:19.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e0ffff640 1 -- 172.21.15.88:0/4271031108 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1000ad60 con 0x7f8e18152de0 2023-10-16T20:39:19.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1d10f640 1 -- 172.21.15.88:0/4271031108 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e18151530 con 0x7f8e18152de0 2023-10-16T20:39:19.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e0ffff640 1 -- 172.21.15.88:0/4271031108 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e10013630 con 0x7f8e18152de0 2023-10-16T20:39:19.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1d10f640 1 -- 172.21.15.88:0/4271031108 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1815f660 con 0x7f8e18152de0 2023-10-16T20:39:19.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.929+0000 7f8e1d10f640 1 -- 172.21.15.88:0/4271031108 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e18074700 con 0x7f8e18152de0 2023-10-16T20:39:19.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.933+0000 7f8e0ffff640 1 -- 172.21.15.88:0/4271031108 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8e10013790 con 0x7f8e18152de0 2023-10-16T20:39:19.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.933+0000 7f8e0ffff640 1 --2- 172.21.15.88:0/4271031108 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8df80664a0 0x7f8df8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:19.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.933+0000 7f8e0ffff640 1 -- 172.21.15.88:0/4271031108 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8e10084b00 con 0x7f8e18152de0 2023-10-16T20:39:19.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.933+0000 7f8e16d9d640 1 --2- 172.21.15.88:0/4271031108 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8df80664a0 0x7f8df8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:19.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.933+0000 7f8e16d9d640 1 --2- 172.21.15.88:0/4271031108 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8df80664a0 0x7f8df8068960 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8e080ae000 tx=0x7f8e080046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:19.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:19.937+0000 7f8e0ffff640 1 -- 172.21.15.88:0/4271031108 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8e10086e60 con 0x7f8e18152de0 2023-10-16T20:39:20.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.089+0000 7f8e1d10f640 1 -- 172.21.15.88:0/4271031108 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e180cbe40 con 0x7f8e18152de0 2023-10-16T20:39:20.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.089+0000 7f8e0ffff640 1 -- 172.21.15.88:0/4271031108 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18710 (secure 0 0 0) 0x7f8e10017070 con 0x7f8e18152de0 2023-10-16T20:39:20.096 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:20.097 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:37:37.392549+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:39:20.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.089+0000 7f8e1d10f640 1 -- 172.21.15.88:0/4271031108 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8df80664a0 msgr2=0x7f8df8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:20.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.089+0000 7f8e1d10f640 1 --2- 172.21.15.88:0/4271031108 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8df80664a0 0x7f8df8068960 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8e080ae000 tx=0x7f8e080046d0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.089+0000 7f8e1d10f640 1 -- 172.21.15.88:0/4271031108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e18152de0 msgr2=0x7f8e181456f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:20.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.089+0000 7f8e1d10f640 1 --2- 172.21.15.88:0/4271031108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e18152de0 0x7f8e181456f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8e1808cd30 tx=0x7f8e10007b10 comp rx=0 tx=0).stop 2023-10-16T20:39:20.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.089+0000 7f8e1d10f640 1 -- 172.21.15.88:0/4271031108 shutdown_connections 2023-10-16T20:39:20.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.089+0000 7f8e1d10f640 1 --2- 172.21.15.88:0/4271031108 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8df80664a0 0x7f8df8068960 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.089+0000 7f8e1d10f640 1 --2- 172.21.15.88:0/4271031108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e18152de0 0x7f8e181456f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.089+0000 7f8e1d10f640 1 --2- 172.21.15.88:0/4271031108 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8e180b4b50 0x7f8e181528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.089+0000 7f8e1d10f640 1 --2- 172.21.15.88:0/4271031108 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8e04003980 0x7f8e180c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.089+0000 7f8e1d10f640 1 -- 172.21.15.88:0/4271031108 >> 172.21.15.88:0/4271031108 conn(0x7f8e180715c0 msgr2=0x7f8e18162080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:20.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.089+0000 7f8e1d10f640 1 -- 172.21.15.88:0/4271031108 shutdown_connections 2023-10-16T20:39:20.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.089+0000 7f8e1d10f640 1 -- 172.21.15.88:0/4271031108 wait complete. 2023-10-16T20:39:20.118 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-16T20:39:20.119 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T20:39:20.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.249+0000 7f1268846640 1 Processor -- start 2023-10-16T20:39:20.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.249+0000 7f1268846640 1 -- start start 2023-10-16T20:39:20.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.249+0000 7f1268846640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1264153770 0x7f1264159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:20.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.249+0000 7f1268846640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126415a410 con 0x7f12640c9bb0 2023-10-16T20:39:20.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.249+0000 7f1268846640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1264153b50 con 0x7f12640cece0 2023-10-16T20:39:20.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.249+0000 7f1268846640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12640cc250 con 0x7f1264153770 2023-10-16T20:39:20.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.249+0000 7f1259d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f12640c9bb0 0x7f12640c3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36862/0 (socket says 172.21.15.88:36862) 2023-10-16T20:39:20.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.249+0000 7f1259d9b640 1 -- 172.21.15.88:0/1943911520 learned_addr learned my addr 172.21.15.88:0/1943911520 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:20.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.249+0000 7f126259c640 1 --2- 172.21.15.88:0/1943911520 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1264153770 0x7f1264159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:20.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.249+0000 7f1261d9b640 1 -- 172.21.15.88:0/1943911520 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533700143 0 0) 0x7f126415a410 con 0x7f12640c9bb0 2023-10-16T20:39:20.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.249+0000 7f1261d9b640 1 -- 172.21.15.88:0/1943911520 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1250003530 con 0x7f12640c9bb0 2023-10-16T20:39:20.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.249+0000 7f126259c640 1 -- 172.21.15.88:0/1943911520 >> v1:172.21.15.191:6789/0 conn(0x7f12640cece0 legacy=0x7f1264163dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:20.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.249+0000 7f126259c640 1 -- 172.21.15.88:0/1943911520 >> v1:172.21.15.88:6789/0 conn(0x7f12640c9bb0 legacy=0x7f12640c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:20.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.249+0000 7f126259c640 1 -- 172.21.15.88:0/1943911520 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12640cb2f0 con 0x7f1264153770 2023-10-16T20:39:20.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.253+0000 7f126259c640 1 --2- 172.21.15.88:0/1943911520 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1264153770 0x7f1264159ed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f12540097d0 tx=0x7f125401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=372d7086b3b857ec server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:20.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.253+0000 7f1261d9b640 1 -- 172.21.15.88:0/1943911520 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12540ac410 con 0x7f1264153770 2023-10-16T20:39:20.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.253+0000 7f1261d9b640 1 -- 172.21.15.88:0/1943911520 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12540ac570 con 0x7f1264153770 2023-10-16T20:39:20.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.253+0000 7f1261d9b640 1 -- 172.21.15.88:0/1943911520 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12540ac900 con 0x7f1264153770 2023-10-16T20:39:20.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.253+0000 7f1268846640 1 -- 172.21.15.88:0/1943911520 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1264153770 msgr2=0x7f1264159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:20.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.253+0000 7f1268846640 1 --2- 172.21.15.88:0/1943911520 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1264153770 0x7f1264159ed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f12540097d0 tx=0x7f125401c4a0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.253+0000 7f1268846640 1 -- 172.21.15.88:0/1943911520 shutdown_connections 2023-10-16T20:39:20.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.253+0000 7f1268846640 1 --2- 172.21.15.88:0/1943911520 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1264153770 0x7f1264159ed0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.253+0000 7f1268846640 1 -- 172.21.15.88:0/1943911520 >> 172.21.15.88:0/1943911520 conn(0x7f12640715c0 msgr2=0x7f12640719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:20.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.253+0000 7f1268846640 1 -- 172.21.15.88:0/1943911520 shutdown_connections 2023-10-16T20:39:20.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.253+0000 7f1268846640 1 -- 172.21.15.88:0/1943911520 wait complete. 2023-10-16T20:39:20.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f1268846640 1 Processor -- start 2023-10-16T20:39:20.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f1268846640 1 -- start start 2023-10-16T20:39:20.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f1268846640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12640c9bb0 0x7f126415c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:20.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f1268846640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f12640cece0 0x7f1264154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:20.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f1268846640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1264153770 0x7f12641553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:20.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f1268846640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12641f2950 con 0x7f12640c9bb0 2023-10-16T20:39:20.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f1268846640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12641f1ae0 con 0x7f1264153770 2023-10-16T20:39:20.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f1268846640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12641f1cd0 con 0x7f12640cece0 2023-10-16T20:39:20.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f126259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12640c9bb0 0x7f126415c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:20.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f126259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12640c9bb0 0x7f126415c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56234/0 (socket says 172.21.15.88:56234) 2023-10-16T20:39:20.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f1259d9b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f12640cece0 0x7f1264154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:20.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f126259c640 1 -- 172.21.15.88:0/2360083507 learned_addr learned my addr 172.21.15.88:0/2360083507 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:20.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f1262d9d640 1 --2- 172.21.15.88:0/2360083507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1264153770 0x7f12641553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:20.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f126259c640 1 -- 172.21.15.88:0/2360083507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f12640cece0 msgr2=0x7f1264154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:20.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f126259c640 1 --2- 172.21.15.88:0/2360083507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f12640cece0 0x7f1264154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f126259c640 1 -- 172.21.15.88:0/2360083507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1264153770 msgr2=0x7f12641553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:20.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f126259c640 1 --2- 172.21.15.88:0/2360083507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1264153770 0x7f12641553b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f126259c640 1 -- 172.21.15.88:0/2360083507 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12540093b0 con 0x7f12640c9bb0 2023-10-16T20:39:20.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f126259c640 1 --2- 172.21.15.88:0/2360083507 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12640c9bb0 0x7f126415c120 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f12540af040 tx=0x7f1254019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:20.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f125b7fe640 1 -- 172.21.15.88:0/2360083507 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12540ac410 con 0x7f12640c9bb0 2023-10-16T20:39:20.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f125b7fe640 1 -- 172.21.15.88:0/2360083507 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1254006dc0 con 0x7f12640c9bb0 2023-10-16T20:39:20.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f125b7fe640 1 -- 172.21.15.88:0/2360083507 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12540b5870 con 0x7f12640c9bb0 2023-10-16T20:39:20.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f1268846640 1 -- 172.21.15.88:0/2360083507 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12641f1fb0 con 0x7f12640c9bb0 2023-10-16T20:39:20.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.257+0000 7f1268846640 1 -- 172.21.15.88:0/2360083507 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12641f2570 con 0x7f12640c9bb0 2023-10-16T20:39:20.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.261+0000 7f1268846640 1 -- 172.21.15.88:0/2360083507 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1264074700 con 0x7f12640c9bb0 2023-10-16T20:39:20.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.261+0000 7f125b7fe640 1 -- 172.21.15.88:0/2360083507 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f12540b3040 con 0x7f12640c9bb0 2023-10-16T20:39:20.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.261+0000 7f125b7fe640 1 --2- 172.21.15.88:0/2360083507 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f12440664a0 0x7f1244068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:20.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.261+0000 7f125b7fe640 1 -- 172.21.15.88:0/2360083507 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f125411dfe0 con 0x7f12640c9bb0 2023-10-16T20:39:20.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.261+0000 7f1259d9b640 1 --2- 172.21.15.88:0/2360083507 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f12440664a0 0x7f1244068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:20.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.261+0000 7f1259d9b640 1 --2- 172.21.15.88:0/2360083507 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f12440664a0 0x7f1244068960 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f12640b4c40 tx=0x7f124c007d40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:20.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.265+0000 7f125b7fe640 1 -- 172.21.15.88:0/2360083507 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f12540eb160 con 0x7f12640c9bb0 2023-10-16T20:39:20.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.413+0000 7f1268846640 1 -- 172.21.15.88:0/2360083507 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f12640cbe70 con 0x7f12440664a0 2023-10-16T20:39:20.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.425+0000 7f125b7fe640 1 -- 172.21.15.88:0/2360083507 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+369665 (secure 0 0 0) 0x7f124c005860 con 0x7f12440664a0 2023-10-16T20:39:20.437 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:20.440 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-16T20:39:20.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.429+0000 7f1268846640 1 -- 172.21.15.88:0/2360083507 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f12440664a0 msgr2=0x7f1244068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:20.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.429+0000 7f1268846640 1 --2- 172.21.15.88:0/2360083507 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f12440664a0 0x7f1244068960 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f12640b4c40 tx=0x7f124c007d40 comp rx=0 tx=0).stop 2023-10-16T20:39:20.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.429+0000 7f1268846640 1 -- 172.21.15.88:0/2360083507 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12640c9bb0 msgr2=0x7f126415c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:20.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.429+0000 7f1268846640 1 --2- 172.21.15.88:0/2360083507 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12640c9bb0 0x7f126415c120 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f12540af040 tx=0x7f1254019240 comp rx=0 tx=0).stop 2023-10-16T20:39:20.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.429+0000 7f1268846640 1 -- 172.21.15.88:0/2360083507 shutdown_connections 2023-10-16T20:39:20.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.429+0000 7f1268846640 1 --2- 172.21.15.88:0/2360083507 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f12440664a0 0x7f1244068960 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.429+0000 7f1268846640 1 --2- 172.21.15.88:0/2360083507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1264153770 0x7f12641553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.429+0000 7f1268846640 1 --2- 172.21.15.88:0/2360083507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f12640cece0 0x7f1264154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.429+0000 7f1268846640 1 --2- 172.21.15.88:0/2360083507 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12640c9bb0 0x7f126415c120 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.429+0000 7f1268846640 1 -- 172.21.15.88:0/2360083507 >> 172.21.15.88:0/2360083507 conn(0x7f12640715c0 msgr2=0x7f1264152890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:20.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.429+0000 7f1268846640 1 -- 172.21.15.88:0/2360083507 shutdown_connections 2023-10-16T20:39:20.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.429+0000 7f1268846640 1 -- 172.21.15.88:0/2360083507 wait complete. 2023-10-16T20:39:20.458 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2023-10-16T20:39:19.627547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595007,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219792,"kb_used_data":3624,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754928,"statfs":{"total":773094113280,"available":772869046272,"internally_reserved":0,"allocated":3710976,"data_stored":2183974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2229,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002567"},"pg_stats":[{"pgid":"2.1f","version":"26'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435615+0000","last_change":"2023-10-16T20:36:54.763744+0000","last_active":"2023-10-16T20:37:37.435615+0000","last_peered":"2023-10-16T20:37:37.435615+0000","last_clean":"2023-10-16T20:37:37.435615+0000","last_became_active":"2023-10-16T20:36:54.763457+0000","last_became_peered":"2023-10-16T20:36:54.763457+0000","last_unstale":"2023-10-16T20:37:37.435615+0000","last_undegraded":"2023-10-16T20:37:37.435615+0000","last_fullsized":"2023-10-16T20:37:37.435615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:05.774485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465462+0000","last_change":"2023-10-16T20:36:55.302740+0000","last_active":"2023-10-16T20:37:37.465462+0000","last_peered":"2023-10-16T20:37:37.465462+0000","last_clean":"2023-10-16T20:37:37.465462+0000","last_became_active":"2023-10-16T20:36:55.302519+0000","last_became_peered":"2023-10-16T20:36:55.302519+0000","last_unstale":"2023-10-16T20:37:37.465462+0000","last_undegraded":"2023-10-16T20:37:37.465462+0000","last_fullsized":"2023-10-16T20:37:37.465462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:36.762521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404215+0000","last_change":"2023-10-16T20:36:59.466679+0000","last_active":"2023-10-16T20:37:36.404215+0000","last_peered":"2023-10-16T20:37:36.404215+0000","last_clean":"2023-10-16T20:37:36.404215+0000","last_became_active":"2023-10-16T20:36:59.466306+0000","last_became_peered":"2023-10-16T20:36:59.466306+0000","last_unstale":"2023-10-16T20:37:36.404215+0000","last_undegraded":"2023-10-16T20:37:36.404215+0000","last_fullsized":"2023-10-16T20:37:36.404215+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:14.874622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"26'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465585+0000","last_change":"2023-10-16T20:36:54.759491+0000","last_active":"2023-10-16T20:37:37.465585+0000","last_peered":"2023-10-16T20:37:37.465585+0000","last_clean":"2023-10-16T20:37:37.465585+0000","last_became_active":"2023-10-16T20:36:54.759177+0000","last_became_peered":"2023-10-16T20:36:54.759177+0000","last_unstale":"2023-10-16T20:37:37.465585+0000","last_undegraded":"2023-10-16T20:37:37.465585+0000","last_fullsized":"2023-10-16T20:37:37.465585+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:43:36.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435464+0000","last_change":"2023-10-16T20:36:55.291745+0000","last_active":"2023-10-16T20:37:37.435464+0000","last_peered":"2023-10-16T20:37:37.435464+0000","last_clean":"2023-10-16T20:37:37.435464+0000","last_became_active":"2023-10-16T20:36:55.291458+0000","last_became_peered":"2023-10-16T20:36:55.291458+0000","last_unstale":"2023-10-16T20:37:37.435464+0000","last_undegraded":"2023-10-16T20:37:37.435464+0000","last_fullsized":"2023-10-16T20:37:37.435464+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:01.291335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.401937+0000","last_change":"2023-10-16T20:36:59.303600+0000","last_active":"2023-10-16T20:37:36.401937+0000","last_peered":"2023-10-16T20:37:36.401937+0000","last_clean":"2023-10-16T20:37:36.401937+0000","last_became_active":"2023-10-16T20:36:59.302968+0000","last_became_peered":"2023-10-16T20:36:59.302968+0000","last_unstale":"2023-10-16T20:37:36.401937+0000","last_undegraded":"2023-10-16T20:37:36.401937+0000","last_fullsized":"2023-10-16T20:37:36.401937+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:11:28.322824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"27'2","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.412675+0000","last_change":"2023-10-16T20:36:54.762600+0000","last_active":"2023-10-16T20:37:36.412675+0000","last_peered":"2023-10-16T20:37:36.412675+0000","last_clean":"2023-10-16T20:37:36.412675+0000","last_became_active":"2023-10-16T20:36:54.762278+0000","last_became_peered":"2023-10-16T20:36:54.762278+0000","last_unstale":"2023-10-16T20:37:36.412675+0000","last_undegraded":"2023-10-16T20:37:36.412675+0000","last_fullsized":"2023-10-16T20:37:36.412675+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:46:03.836415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404637+0000","last_change":"2023-10-16T20:37:37.403448+0000","last_active":"2023-10-16T20:37:37.404637+0000","last_peered":"2023-10-16T20:37:37.404637+0000","last_clean":"2023-10-16T20:37:37.404637+0000","last_became_active":"2023-10-16T20:37:37.402987+0000","last_became_peered":"2023-10-16T20:37:37.402987+0000","last_unstale":"2023-10-16T20:37:37.404637+0000","last_undegraded":"2023-10-16T20:37:37.404637+0000","last_fullsized":"2023-10-16T20:37:37.404637+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:39:05.729682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.402113+0000","last_change":"2023-10-16T20:36:59.304684+0000","last_active":"2023-10-16T20:37:36.402113+0000","last_peered":"2023-10-16T20:37:36.402113+0000","last_clean":"2023-10-16T20:37:36.402113+0000","last_became_active":"2023-10-16T20:36:59.303805+0000","last_became_peered":"2023-10-16T20:36:59.303805+0000","last_unstale":"2023-10-16T20:37:36.402113+0000","last_undegraded":"2023-10-16T20:37:36.402113+0000","last_fullsized":"2023-10-16T20:37:36.402113+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:39:54.406763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"27'2","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.412121+0000","last_change":"2023-10-16T20:36:54.749849+0000","last_active":"2023-10-16T20:37:36.412121+0000","last_peered":"2023-10-16T20:37:36.412121+0000","last_clean":"2023-10-16T20:37:36.412121+0000","last_became_active":"2023-10-16T20:36:54.749316+0000","last_became_peered":"2023-10-16T20:36:54.749316+0000","last_unstale":"2023-10-16T20:37:36.412121+0000","last_undegraded":"2023-10-16T20:37:36.412121+0000","last_fullsized":"2023-10-16T20:37:36.412121+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:49:29.077126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.409911+0000","last_change":"2023-10-16T20:36:55.302838+0000","last_active":"2023-10-16T20:37:36.409911+0000","last_peered":"2023-10-16T20:37:36.409911+0000","last_clean":"2023-10-16T20:37:36.409911+0000","last_became_active":"2023-10-16T20:36:55.302387+0000","last_became_peered":"2023-10-16T20:36:55.302387+0000","last_unstale":"2023-10-16T20:37:36.409911+0000","last_undegraded":"2023-10-16T20:37:36.409911+0000","last_fullsized":"2023-10-16T20:37:36.409911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:55:56.517950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.402135+0000","last_change":"2023-10-16T20:36:59.305750+0000","last_active":"2023-10-16T20:37:36.402135+0000","last_peered":"2023-10-16T20:37:36.402135+0000","last_clean":"2023-10-16T20:37:36.402135+0000","last_became_active":"2023-10-16T20:36:59.305228+0000","last_became_peered":"2023-10-16T20:36:59.305228+0000","last_unstale":"2023-10-16T20:37:36.402135+0000","last_undegraded":"2023-10-16T20:37:36.402135+0000","last_fullsized":"2023-10-16T20:37:36.402135+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:00:18.194057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407634+0000","last_change":"2023-10-16T20:36:55.306712+0000","last_active":"2023-10-16T20:37:37.407634+0000","last_peered":"2023-10-16T20:37:37.407634+0000","last_clean":"2023-10-16T20:37:37.407634+0000","last_became_active":"2023-10-16T20:36:55.306427+0000","last_became_peered":"2023-10-16T20:36:55.306427+0000","last_unstale":"2023-10-16T20:37:37.407634+0000","last_undegraded":"2023-10-16T20:37:37.407634+0000","last_fullsized":"2023-10-16T20:37:37.407634+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:42:03.769890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"27'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466293+0000","last_change":"2023-10-16T20:36:54.768375+0000","last_active":"2023-10-16T20:37:37.466293+0000","last_peered":"2023-10-16T20:37:37.466293+0000","last_clean":"2023-10-16T20:37:37.466293+0000","last_became_active":"2023-10-16T20:36:54.767996+0000","last_became_peered":"2023-10-16T20:36:54.767996+0000","last_unstale":"2023-10-16T20:37:37.466293+0000","last_undegraded":"2023-10-16T20:37:37.466293+0000","last_fullsized":"2023-10-16T20:37:37.466293+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:55.303942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404253+0000","last_change":"2023-10-16T20:36:59.307130+0000","last_active":"2023-10-16T20:37:37.404253+0000","last_peered":"2023-10-16T20:37:37.404253+0000","last_clean":"2023-10-16T20:37:37.404253+0000","last_became_active":"2023-10-16T20:36:59.306640+0000","last_became_peered":"2023-10-16T20:36:59.306640+0000","last_unstale":"2023-10-16T20:37:37.404253+0000","last_undegraded":"2023-10-16T20:37:37.404253+0000","last_fullsized":"2023-10-16T20:37:37.404253+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:19:18.664630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465074+0000","last_change":"2023-10-16T20:36:55.275436+0000","last_active":"2023-10-16T20:37:37.465074+0000","last_peered":"2023-10-16T20:37:37.465074+0000","last_clean":"2023-10-16T20:37:37.465074+0000","last_became_active":"2023-10-16T20:36:55.275085+0000","last_became_peered":"2023-10-16T20:36:55.275085+0000","last_unstale":"2023-10-16T20:37:37.465074+0000","last_undegraded":"2023-10-16T20:37:37.465074+0000","last_fullsized":"2023-10-16T20:37:37.465074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:37:37.814511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402406+0000","last_change":"2023-10-16T20:36:54.756174+0000","last_active":"2023-10-16T20:37:37.402406+0000","last_peered":"2023-10-16T20:37:37.402406+0000","last_clean":"2023-10-16T20:37:37.402406+0000","last_became_active":"2023-10-16T20:36:54.755872+0000","last_became_peered":"2023-10-16T20:36:54.755872+0000","last_unstale":"2023-10-16T20:37:37.402406+0000","last_undegraded":"2023-10-16T20:37:37.402406+0000","last_fullsized":"2023-10-16T20:37:37.402406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:32.741671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403750+0000","last_change":"2023-10-16T20:36:59.303855+0000","last_active":"2023-10-16T20:37:37.403750+0000","last_peered":"2023-10-16T20:37:37.403750+0000","last_clean":"2023-10-16T20:37:37.403750+0000","last_became_active":"2023-10-16T20:36:59.302919+0000","last_became_peered":"2023-10-16T20:36:59.302919+0000","last_unstale":"2023-10-16T20:37:37.403750+0000","last_undegraded":"2023-10-16T20:37:37.403750+0000","last_fullsized":"2023-10-16T20:37:37.403750+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:38:02.591237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464820+0000","last_change":"2023-10-16T20:36:55.270717+0000","last_active":"2023-10-16T20:37:37.464820+0000","last_peered":"2023-10-16T20:37:37.464820+0000","last_clean":"2023-10-16T20:37:37.464820+0000","last_became_active":"2023-10-16T20:36:55.270455+0000","last_became_peered":"2023-10-16T20:36:55.270455+0000","last_unstale":"2023-10-16T20:37:37.464820+0000","last_undegraded":"2023-10-16T20:37:37.464820+0000","last_fullsized":"2023-10-16T20:37:37.464820+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:16:44.485783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401912+0000","last_change":"2023-10-16T20:36:54.744128+0000","last_active":"2023-10-16T20:37:37.401912+0000","last_peered":"2023-10-16T20:37:37.401912+0000","last_clean":"2023-10-16T20:37:37.401912+0000","last_became_active":"2023-10-16T20:36:54.743805+0000","last_became_peered":"2023-10-16T20:36:54.743805+0000","last_unstale":"2023-10-16T20:37:37.401912+0000","last_undegraded":"2023-10-16T20:37:37.401912+0000","last_fullsized":"2023-10-16T20:37:37.401912+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:54:35.819820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402668+0000","last_change":"2023-10-16T20:36:59.465650+0000","last_active":"2023-10-16T20:37:37.402668+0000","last_peered":"2023-10-16T20:37:37.402668+0000","last_clean":"2023-10-16T20:37:37.402668+0000","last_became_active":"2023-10-16T20:36:59.465172+0000","last_became_peered":"2023-10-16T20:36:59.465172+0000","last_unstale":"2023-10-16T20:37:37.402668+0000","last_undegraded":"2023-10-16T20:37:37.402668+0000","last_fullsized":"2023-10-16T20:37:37.402668+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:12.845517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403561+0000","last_change":"2023-10-16T20:36:55.278974+0000","last_active":"2023-10-16T20:37:37.403561+0000","last_peered":"2023-10-16T20:37:37.403561+0000","last_clean":"2023-10-16T20:37:37.403561+0000","last_became_active":"2023-10-16T20:36:55.278568+0000","last_became_peered":"2023-10-16T20:36:55.278568+0000","last_unstale":"2023-10-16T20:37:37.403561+0000","last_undegraded":"2023-10-16T20:37:37.403561+0000","last_fullsized":"2023-10-16T20:37:37.403561+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:26.756114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465927+0000","last_change":"2023-10-16T20:36:54.758778+0000","last_active":"2023-10-16T20:37:37.465927+0000","last_peered":"2023-10-16T20:37:37.465927+0000","last_clean":"2023-10-16T20:37:37.465927+0000","last_became_active":"2023-10-16T20:36:54.758496+0000","last_became_peered":"2023-10-16T20:36:54.758496+0000","last_unstale":"2023-10-16T20:37:37.465927+0000","last_undegraded":"2023-10-16T20:37:37.465927+0000","last_fullsized":"2023-10-16T20:37:37.465927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:36:00.515106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404277+0000","last_change":"2023-10-16T20:36:59.309382+0000","last_active":"2023-10-16T20:37:37.404277+0000","last_peered":"2023-10-16T20:37:37.404277+0000","last_clean":"2023-10-16T20:37:37.404277+0000","last_became_active":"2023-10-16T20:36:59.309068+0000","last_became_peered":"2023-10-16T20:36:59.309068+0000","last_unstale":"2023-10-16T20:37:37.404277+0000","last_undegraded":"2023-10-16T20:37:37.404277+0000","last_fullsized":"2023-10-16T20:37:37.404277+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:46:02.601985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435429+0000","last_change":"2023-10-16T20:36:55.273089+0000","last_active":"2023-10-16T20:37:37.435429+0000","last_peered":"2023-10-16T20:37:37.435429+0000","last_clean":"2023-10-16T20:37:37.435429+0000","last_became_active":"2023-10-16T20:36:55.272840+0000","last_became_peered":"2023-10-16T20:36:55.272840+0000","last_unstale":"2023-10-16T20:37:37.435429+0000","last_undegraded":"2023-10-16T20:37:37.435429+0000","last_fullsized":"2023-10-16T20:37:37.435429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:37.330924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"27'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408542+0000","last_change":"2023-10-16T20:36:54.760681+0000","last_active":"2023-10-16T20:37:37.408542+0000","last_peered":"2023-10-16T20:37:37.408542+0000","last_clean":"2023-10-16T20:37:37.408542+0000","last_became_active":"2023-10-16T20:36:54.760397+0000","last_became_peered":"2023-10-16T20:36:54.760397+0000","last_unstale":"2023-10-16T20:37:37.408542+0000","last_undegraded":"2023-10-16T20:37:37.408542+0000","last_fullsized":"2023-10-16T20:37:37.408542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:47:16.510382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403318+0000","last_change":"2023-10-16T20:36:59.465626+0000","last_active":"2023-10-16T20:37:37.403318+0000","last_peered":"2023-10-16T20:37:37.403318+0000","last_clean":"2023-10-16T20:37:37.403318+0000","last_became_active":"2023-10-16T20:36:59.465152+0000","last_became_peered":"2023-10-16T20:36:59.465152+0000","last_unstale":"2023-10-16T20:37:37.403318+0000","last_undegraded":"2023-10-16T20:37:37.403318+0000","last_fullsized":"2023-10-16T20:37:37.403318+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:27:51.088333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465673+0000","last_change":"2023-10-16T20:36:55.285796+0000","last_active":"2023-10-16T20:37:37.465673+0000","last_peered":"2023-10-16T20:37:37.465673+0000","last_clean":"2023-10-16T20:37:37.465673+0000","last_became_active":"2023-10-16T20:36:55.285566+0000","last_became_peered":"2023-10-16T20:36:55.285566+0000","last_unstale":"2023-10-16T20:37:37.465673+0000","last_undegraded":"2023-10-16T20:37:37.465673+0000","last_fullsized":"2023-10-16T20:37:37.465673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:22:21.950705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"27'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402756+0000","last_change":"2023-10-16T20:36:54.758062+0000","last_active":"2023-10-16T20:37:37.402756+0000","last_peered":"2023-10-16T20:37:37.402756+0000","last_clean":"2023-10-16T20:37:37.402756+0000","last_became_active":"2023-10-16T20:36:54.757796+0000","last_became_peered":"2023-10-16T20:36:54.757796+0000","last_unstale":"2023-10-16T20:37:37.402756+0000","last_undegraded":"2023-10-16T20:37:37.402756+0000","last_fullsized":"2023-10-16T20:37:37.402756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:02:09.953795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464463+0000","last_change":"2023-10-16T20:36:59.468529+0000","last_active":"2023-10-16T20:37:37.464463+0000","last_peered":"2023-10-16T20:37:37.464463+0000","last_clean":"2023-10-16T20:37:37.464463+0000","last_became_active":"2023-10-16T20:36:59.465221+0000","last_became_peered":"2023-10-16T20:36:59.465221+0000","last_unstale":"2023-10-16T20:37:37.464463+0000","last_undegraded":"2023-10-16T20:37:37.464463+0000","last_fullsized":"2023-10-16T20:37:37.464463+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:58:08.457686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401257+0000","last_change":"2023-10-16T20:36:55.275759+0000","last_active":"2023-10-16T20:37:37.401257+0000","last_peered":"2023-10-16T20:37:37.401257+0000","last_clean":"2023-10-16T20:37:37.401257+0000","last_became_active":"2023-10-16T20:36:55.274432+0000","last_became_peered":"2023-10-16T20:36:55.274432+0000","last_unstale":"2023-10-16T20:37:37.401257+0000","last_undegraded":"2023-10-16T20:37:37.401257+0000","last_fullsized":"2023-10-16T20:37:37.401257+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:53:08.896731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466573+0000","last_change":"2023-10-16T20:36:54.762356+0000","last_active":"2023-10-16T20:37:37.466573+0000","last_peered":"2023-10-16T20:37:37.466573+0000","last_clean":"2023-10-16T20:37:37.466573+0000","last_became_active":"2023-10-16T20:36:54.762167+0000","last_became_peered":"2023-10-16T20:36:54.762167+0000","last_unstale":"2023-10-16T20:37:37.466573+0000","last_undegraded":"2023-10-16T20:37:37.466573+0000","last_fullsized":"2023-10-16T20:37:37.466573+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:45.164625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434946+0000","last_change":"2023-10-16T20:36:59.467492+0000","last_active":"2023-10-16T20:37:37.434946+0000","last_peered":"2023-10-16T20:37:37.434946+0000","last_clean":"2023-10-16T20:37:37.434946+0000","last_became_active":"2023-10-16T20:36:59.466876+0000","last_became_peered":"2023-10-16T20:36:59.466876+0000","last_unstale":"2023-10-16T20:37:37.434946+0000","last_undegraded":"2023-10-16T20:37:37.434946+0000","last_fullsized":"2023-10-16T20:37:37.434946+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:29:01.292167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409570+0000","last_change":"2023-10-16T20:36:55.293537+0000","last_active":"2023-10-16T20:37:37.409570+0000","last_peered":"2023-10-16T20:37:37.409570+0000","last_clean":"2023-10-16T20:37:37.409570+0000","last_became_active":"2023-10-16T20:36:55.293228+0000","last_became_peered":"2023-10-16T20:36:55.293228+0000","last_unstale":"2023-10-16T20:37:37.409570+0000","last_undegraded":"2023-10-16T20:37:37.409570+0000","last_fullsized":"2023-10-16T20:37:37.409570+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:50:13.226645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401381+0000","last_change":"2023-10-16T20:36:54.757588+0000","last_active":"2023-10-16T20:37:37.401381+0000","last_peered":"2023-10-16T20:37:37.401381+0000","last_clean":"2023-10-16T20:37:37.401381+0000","last_became_active":"2023-10-16T20:36:54.757308+0000","last_became_peered":"2023-10-16T20:36:54.757308+0000","last_unstale":"2023-10-16T20:37:37.401381+0000","last_undegraded":"2023-10-16T20:37:37.401381+0000","last_fullsized":"2023-10-16T20:37:37.401381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:51.298239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402099+0000","last_change":"2023-10-16T20:36:59.303844+0000","last_active":"2023-10-16T20:37:37.402099+0000","last_peered":"2023-10-16T20:37:37.402099+0000","last_clean":"2023-10-16T20:37:37.402099+0000","last_became_active":"2023-10-16T20:36:59.303224+0000","last_became_peered":"2023-10-16T20:36:59.303224+0000","last_unstale":"2023-10-16T20:37:37.402099+0000","last_undegraded":"2023-10-16T20:37:37.402099+0000","last_fullsized":"2023-10-16T20:37:37.402099+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:52.447785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403798+0000","last_change":"2023-10-16T20:36:55.278310+0000","last_active":"2023-10-16T20:37:37.403798+0000","last_peered":"2023-10-16T20:37:37.403798+0000","last_clean":"2023-10-16T20:37:37.403798+0000","last_became_active":"2023-10-16T20:36:55.277872+0000","last_became_peered":"2023-10-16T20:36:55.277872+0000","last_unstale":"2023-10-16T20:37:37.403798+0000","last_undegraded":"2023-10-16T20:37:37.403798+0000","last_fullsized":"2023-10-16T20:37:37.403798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:37.320469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408898+0000","last_change":"2023-10-16T20:36:54.740644+0000","last_active":"2023-10-16T20:37:37.408898+0000","last_peered":"2023-10-16T20:37:37.408898+0000","last_clean":"2023-10-16T20:37:37.408898+0000","last_became_active":"2023-10-16T20:36:54.740340+0000","last_became_peered":"2023-10-16T20:36:54.740340+0000","last_unstale":"2023-10-16T20:37:37.408898+0000","last_undegraded":"2023-10-16T20:37:37.408898+0000","last_fullsized":"2023-10-16T20:37:37.408898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:22.221727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401155+0000","last_change":"2023-10-16T20:36:59.294914+0000","last_active":"2023-10-16T20:37:37.401155+0000","last_peered":"2023-10-16T20:37:37.401155+0000","last_clean":"2023-10-16T20:37:37.401155+0000","last_became_active":"2023-10-16T20:36:59.294548+0000","last_became_peered":"2023-10-16T20:36:59.294548+0000","last_unstale":"2023-10-16T20:37:37.401155+0000","last_undegraded":"2023-10-16T20:37:37.401155+0000","last_fullsized":"2023-10-16T20:37:37.401155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:52:15.265262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401383+0000","last_change":"2023-10-16T20:36:55.270992+0000","last_active":"2023-10-16T20:37:37.401383+0000","last_peered":"2023-10-16T20:37:37.401383+0000","last_clean":"2023-10-16T20:37:37.401383+0000","last_became_active":"2023-10-16T20:36:55.269749+0000","last_became_peered":"2023-10-16T20:36:55.269749+0000","last_unstale":"2023-10-16T20:37:37.401383+0000","last_undegraded":"2023-10-16T20:37:37.401383+0000","last_fullsized":"2023-10-16T20:37:37.401383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:30.925314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.399629+0000","last_change":"2023-10-16T20:36:54.757903+0000","last_active":"2023-10-16T20:37:37.399629+0000","last_peered":"2023-10-16T20:37:37.399629+0000","last_clean":"2023-10-16T20:37:37.399629+0000","last_became_active":"2023-10-16T20:36:54.757549+0000","last_became_peered":"2023-10-16T20:36:54.757549+0000","last_unstale":"2023-10-16T20:37:37.399629+0000","last_undegraded":"2023-10-16T20:37:37.399629+0000","last_fullsized":"2023-10-16T20:37:37.399629+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:02:42.090848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402786+0000","last_change":"2023-10-16T20:36:59.302063+0000","last_active":"2023-10-16T20:37:37.402786+0000","last_peered":"2023-10-16T20:37:37.402786+0000","last_clean":"2023-10-16T20:37:37.402786+0000","last_became_active":"2023-10-16T20:36:59.301693+0000","last_became_peered":"2023-10-16T20:36:59.301693+0000","last_unstale":"2023-10-16T20:37:37.402786+0000","last_undegraded":"2023-10-16T20:37:37.402786+0000","last_fullsized":"2023-10-16T20:37:37.402786+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:29:44.812282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434713+0000","last_change":"2023-10-16T20:36:55.278132+0000","last_active":"2023-10-16T20:37:37.434713+0000","last_peered":"2023-10-16T20:37:37.434713+0000","last_clean":"2023-10-16T20:37:37.434713+0000","last_became_active":"2023-10-16T20:36:55.276649+0000","last_became_peered":"2023-10-16T20:36:55.276649+0000","last_unstale":"2023-10-16T20:37:37.434713+0000","last_undegraded":"2023-10-16T20:37:37.434713+0000","last_fullsized":"2023-10-16T20:37:37.434713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:30:53.299514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465960+0000","last_change":"2023-10-16T20:36:54.753592+0000","last_active":"2023-10-16T20:37:37.465960+0000","last_peered":"2023-10-16T20:37:37.465960+0000","last_clean":"2023-10-16T20:37:37.465960+0000","last_became_active":"2023-10-16T20:36:54.753237+0000","last_became_peered":"2023-10-16T20:36:54.753237+0000","last_unstale":"2023-10-16T20:37:37.465960+0000","last_undegraded":"2023-10-16T20:37:37.465960+0000","last_fullsized":"2023-10-16T20:37:37.465960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:53.005979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435360+0000","last_change":"2023-10-16T20:36:59.468431+0000","last_active":"2023-10-16T20:37:37.435360+0000","last_peered":"2023-10-16T20:37:37.435360+0000","last_clean":"2023-10-16T20:37:37.435360+0000","last_became_active":"2023-10-16T20:36:59.466708+0000","last_became_peered":"2023-10-16T20:36:59.466708+0000","last_unstale":"2023-10-16T20:37:37.435360+0000","last_undegraded":"2023-10-16T20:37:37.435360+0000","last_fullsized":"2023-10-16T20:37:37.435360+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:42:20.607519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464817+0000","last_change":"2023-10-16T20:36:55.294825+0000","last_active":"2023-10-16T20:37:37.464817+0000","last_peered":"2023-10-16T20:37:37.464817+0000","last_clean":"2023-10-16T20:37:37.464817+0000","last_became_active":"2023-10-16T20:36:55.294610+0000","last_became_peered":"2023-10-16T20:36:55.294610+0000","last_unstale":"2023-10-16T20:37:37.464817+0000","last_undegraded":"2023-10-16T20:37:37.464817+0000","last_fullsized":"2023-10-16T20:37:37.464817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:43.442939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408668+0000","last_change":"2023-10-16T20:36:54.741191+0000","last_active":"2023-10-16T20:37:37.408668+0000","last_peered":"2023-10-16T20:37:37.408668+0000","last_clean":"2023-10-16T20:37:37.408668+0000","last_became_active":"2023-10-16T20:36:54.740944+0000","last_became_peered":"2023-10-16T20:36:54.740944+0000","last_unstale":"2023-10-16T20:37:37.408668+0000","last_undegraded":"2023-10-16T20:37:37.408668+0000","last_fullsized":"2023-10-16T20:37:37.408668+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:05.961850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":172,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466038+0000","last_change":"2023-10-16T20:36:46.390001+0000","last_active":"2023-10-16T20:37:37.466038+0000","last_peered":"2023-10-16T20:37:37.466038+0000","last_clean":"2023-10-16T20:37:37.466038+0000","last_became_active":"2023-10-16T20:36:46.388824+0000","last_became_peered":"2023-10-16T20:36:46.388824+0000","last_unstale":"2023-10-16T20:37:37.466038+0000","last_undegraded":"2023-10-16T20:37:37.466038+0000","last_fullsized":"2023-10-16T20:37:37.466038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_clean_scrub_stamp":"2023-10-16T20:36:44.575506+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:31.004318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407653+0000","last_change":"2023-10-16T20:36:59.468528+0000","last_active":"2023-10-16T20:37:37.407653+0000","last_peered":"2023-10-16T20:37:37.407653+0000","last_clean":"2023-10-16T20:37:37.407653+0000","last_became_active":"2023-10-16T20:36:59.468082+0000","last_became_peered":"2023-10-16T20:36:59.468082+0000","last_unstale":"2023-10-16T20:37:37.407653+0000","last_undegraded":"2023-10-16T20:37:37.407653+0000","last_fullsized":"2023-10-16T20:37:37.407653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:00.920004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410690+0000","last_change":"2023-10-16T20:37:37.409578+0000","last_active":"2023-10-16T20:37:37.410690+0000","last_peered":"2023-10-16T20:37:37.410690+0000","last_clean":"2023-10-16T20:37:37.410690+0000","last_became_active":"2023-10-16T20:37:37.409004+0000","last_became_peered":"2023-10-16T20:37:37.409004+0000","last_unstale":"2023-10-16T20:37:37.410690+0000","last_undegraded":"2023-10-16T20:37:37.410690+0000","last_fullsized":"2023-10-16T20:37:37.410690+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:28:57.364794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.400188+0000","last_change":"2023-10-16T20:36:54.746600+0000","last_active":"2023-10-16T20:37:37.400188+0000","last_peered":"2023-10-16T20:37:37.400188+0000","last_clean":"2023-10-16T20:37:37.400188+0000","last_became_active":"2023-10-16T20:36:54.746247+0000","last_became_peered":"2023-10-16T20:36:54.746247+0000","last_unstale":"2023-10-16T20:37:37.400188+0000","last_undegraded":"2023-10-16T20:37:37.400188+0000","last_fullsized":"2023-10-16T20:37:37.400188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:28:57.949487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.470531+0000","last_change":"2023-10-16T20:37:37.469427+0000","last_active":"2023-10-16T20:37:37.470531+0000","last_peered":"2023-10-16T20:37:37.470531+0000","last_clean":"2023-10-16T20:37:37.470531+0000","last_became_active":"2023-10-16T20:37:37.467402+0000","last_became_peered":"2023-10-16T20:37:37.467402+0000","last_unstale":"2023-10-16T20:37:37.470531+0000","last_undegraded":"2023-10-16T20:37:37.470531+0000","last_fullsized":"2023-10-16T20:37:37.470531+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:00:19.171972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409113+0000","last_change":"2023-10-16T20:36:55.301234+0000","last_active":"2023-10-16T20:37:37.409113+0000","last_peered":"2023-10-16T20:37:37.409113+0000","last_clean":"2023-10-16T20:37:37.409113+0000","last_became_active":"2023-10-16T20:36:55.300958+0000","last_became_peered":"2023-10-16T20:36:55.300958+0000","last_unstale":"2023-10-16T20:37:37.409113+0000","last_undegraded":"2023-10-16T20:37:37.409113+0000","last_fullsized":"2023-10-16T20:37:37.409113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:16.784124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"27'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402899+0000","last_change":"2023-10-16T20:36:54.753770+0000","last_active":"2023-10-16T20:37:37.402899+0000","last_peered":"2023-10-16T20:37:37.402899+0000","last_clean":"2023-10-16T20:37:37.402899+0000","last_became_active":"2023-10-16T20:36:54.753443+0000","last_became_peered":"2023-10-16T20:36:54.753443+0000","last_unstale":"2023-10-16T20:37:37.402899+0000","last_undegraded":"2023-10-16T20:37:37.402899+0000","last_fullsized":"2023-10-16T20:37:37.402899+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:21:26.392480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435046+0000","last_change":"2023-10-16T20:36:59.303351+0000","last_active":"2023-10-16T20:37:37.435046+0000","last_peered":"2023-10-16T20:37:37.435046+0000","last_clean":"2023-10-16T20:37:37.435046+0000","last_became_active":"2023-10-16T20:36:59.302869+0000","last_became_peered":"2023-10-16T20:36:59.302869+0000","last_unstale":"2023-10-16T20:37:37.435046+0000","last_undegraded":"2023-10-16T20:37:37.435046+0000","last_fullsized":"2023-10-16T20:37:37.435046+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:26:35.886399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466830+0000","last_change":"2023-10-16T20:36:55.291947+0000","last_active":"2023-10-16T20:37:37.466830+0000","last_peered":"2023-10-16T20:37:37.466830+0000","last_clean":"2023-10-16T20:37:37.466830+0000","last_became_active":"2023-10-16T20:36:55.291633+0000","last_became_peered":"2023-10-16T20:36:55.291633+0000","last_unstale":"2023-10-16T20:37:37.466830+0000","last_undegraded":"2023-10-16T20:37:37.466830+0000","last_fullsized":"2023-10-16T20:37:37.466830+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:00:14.349339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"27'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.400583+0000","last_change":"2023-10-16T20:36:54.748196+0000","last_active":"2023-10-16T20:37:37.400583+0000","last_peered":"2023-10-16T20:37:37.400583+0000","last_clean":"2023-10-16T20:37:37.400583+0000","last_became_active":"2023-10-16T20:36:54.747704+0000","last_became_peered":"2023-10-16T20:36:54.747704+0000","last_unstale":"2023-10-16T20:37:37.400583+0000","last_undegraded":"2023-10-16T20:37:37.400583+0000","last_fullsized":"2023-10-16T20:37:37.400583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:55.471916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410236+0000","last_change":"2023-10-16T20:36:59.470368+0000","last_active":"2023-10-16T20:37:37.410236+0000","last_peered":"2023-10-16T20:37:37.410236+0000","last_clean":"2023-10-16T20:37:37.410236+0000","last_became_active":"2023-10-16T20:36:59.467291+0000","last_became_peered":"2023-10-16T20:36:59.467291+0000","last_unstale":"2023-10-16T20:37:37.410236+0000","last_undegraded":"2023-10-16T20:37:37.410236+0000","last_fullsized":"2023-10-16T20:37:37.410236+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:18.312394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466658+0000","last_change":"2023-10-16T20:36:55.291451+0000","last_active":"2023-10-16T20:37:37.466658+0000","last_peered":"2023-10-16T20:37:37.466658+0000","last_clean":"2023-10-16T20:37:37.466658+0000","last_became_active":"2023-10-16T20:36:55.290998+0000","last_became_peered":"2023-10-16T20:36:55.290998+0000","last_unstale":"2023-10-16T20:37:37.466658+0000","last_undegraded":"2023-10-16T20:37:37.466658+0000","last_fullsized":"2023-10-16T20:37:37.466658+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:05:01.634971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"26'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404148+0000","last_change":"2023-10-16T20:36:54.763818+0000","last_active":"2023-10-16T20:37:37.404148+0000","last_peered":"2023-10-16T20:37:37.404148+0000","last_clean":"2023-10-16T20:37:37.404148+0000","last_became_active":"2023-10-16T20:36:54.761337+0000","last_became_peered":"2023-10-16T20:36:54.761337+0000","last_unstale":"2023-10-16T20:37:37.404148+0000","last_undegraded":"2023-10-16T20:37:37.404148+0000","last_fullsized":"2023-10-16T20:37:37.404148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:03.093189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.470301+0000","last_change":"2023-10-16T20:37:37.468803+0000","last_active":"2023-10-16T20:37:37.470301+0000","last_peered":"2023-10-16T20:37:37.470301+0000","last_clean":"2023-10-16T20:37:37.470301+0000","last_became_active":"2023-10-16T20:37:37.468309+0000","last_became_peered":"2023-10-16T20:37:37.468309+0000","last_unstale":"2023-10-16T20:37:37.470301+0000","last_undegraded":"2023-10-16T20:37:37.470301+0000","last_fullsized":"2023-10-16T20:37:37.470301+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:11.761661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466101+0000","last_change":"2023-10-16T20:36:55.288895+0000","last_active":"2023-10-16T20:37:37.466101+0000","last_peered":"2023-10-16T20:37:37.466101+0000","last_clean":"2023-10-16T20:37:37.466101+0000","last_became_active":"2023-10-16T20:36:55.288354+0000","last_became_peered":"2023-10-16T20:36:55.288354+0000","last_unstale":"2023-10-16T20:37:37.466101+0000","last_undegraded":"2023-10-16T20:37:37.466101+0000","last_fullsized":"2023-10-16T20:37:37.466101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:53:29.192406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"27'8","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.399457+0000","last_change":"2023-10-16T20:36:54.738216+0000","last_active":"2023-10-16T20:37:37.399457+0000","last_peered":"2023-10-16T20:37:37.399457+0000","last_clean":"2023-10-16T20:37:37.399457+0000","last_became_active":"2023-10-16T20:36:54.737876+0000","last_became_peered":"2023-10-16T20:36:54.737876+0000","last_unstale":"2023-10-16T20:37:37.399457+0000","last_undegraded":"2023-10-16T20:37:37.399457+0000","last_fullsized":"2023-10-16T20:37:37.399457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:28.214601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.397491+0000","last_change":"2023-10-16T20:36:59.307215+0000","last_active":"2023-10-16T20:37:36.397491+0000","last_peered":"2023-10-16T20:37:36.397491+0000","last_clean":"2023-10-16T20:37:36.397491+0000","last_became_active":"2023-10-16T20:36:59.306801+0000","last_became_peered":"2023-10-16T20:36:59.306801+0000","last_unstale":"2023-10-16T20:37:36.397491+0000","last_undegraded":"2023-10-16T20:37:36.397491+0000","last_fullsized":"2023-10-16T20:37:36.397491+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:41:42.656134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407054+0000","last_change":"2023-10-16T20:36:55.287714+0000","last_active":"2023-10-16T20:37:37.407054+0000","last_peered":"2023-10-16T20:37:37.407054+0000","last_clean":"2023-10-16T20:37:37.407054+0000","last_became_active":"2023-10-16T20:36:55.287066+0000","last_became_peered":"2023-10-16T20:36:55.287066+0000","last_unstale":"2023-10-16T20:37:37.407054+0000","last_undegraded":"2023-10-16T20:37:37.407054+0000","last_fullsized":"2023-10-16T20:37:37.407054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:45:21.058260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407609+0000","last_change":"2023-10-16T20:36:54.764693+0000","last_active":"2023-10-16T20:37:37.407609+0000","last_peered":"2023-10-16T20:37:37.407609+0000","last_clean":"2023-10-16T20:37:37.407609+0000","last_became_active":"2023-10-16T20:36:54.764417+0000","last_became_peered":"2023-10-16T20:36:54.764417+0000","last_unstale":"2023-10-16T20:37:37.407609+0000","last_undegraded":"2023-10-16T20:37:37.407609+0000","last_fullsized":"2023-10-16T20:37:37.407609+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:56.653149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404514+0000","last_change":"2023-10-16T20:36:59.465145+0000","last_active":"2023-10-16T20:37:36.404514+0000","last_peered":"2023-10-16T20:37:36.404514+0000","last_clean":"2023-10-16T20:37:36.404514+0000","last_became_active":"2023-10-16T20:36:59.464626+0000","last_became_peered":"2023-10-16T20:36:59.464626+0000","last_unstale":"2023-10-16T20:37:36.404514+0000","last_undegraded":"2023-10-16T20:37:36.404514+0000","last_fullsized":"2023-10-16T20:37:36.404514+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:38.928616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466290+0000","last_change":"2023-10-16T20:36:55.291718+0000","last_active":"2023-10-16T20:37:37.466290+0000","last_peered":"2023-10-16T20:37:37.466290+0000","last_clean":"2023-10-16T20:37:37.466290+0000","last_became_active":"2023-10-16T20:36:55.291396+0000","last_became_peered":"2023-10-16T20:36:55.291396+0000","last_unstale":"2023-10-16T20:37:37.466290+0000","last_undegraded":"2023-10-16T20:37:37.466290+0000","last_fullsized":"2023-10-16T20:37:37.466290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:29:49.138380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408792+0000","last_change":"2023-10-16T20:36:54.750814+0000","last_active":"2023-10-16T20:37:37.408792+0000","last_peered":"2023-10-16T20:37:37.408792+0000","last_clean":"2023-10-16T20:37:37.408792+0000","last_became_active":"2023-10-16T20:36:54.750448+0000","last_became_peered":"2023-10-16T20:36:54.750448+0000","last_unstale":"2023-10-16T20:37:37.408792+0000","last_undegraded":"2023-10-16T20:37:37.408792+0000","last_fullsized":"2023-10-16T20:37:37.408792+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:41.981408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404254+0000","last_change":"2023-10-16T20:36:59.464962+0000","last_active":"2023-10-16T20:37:36.404254+0000","last_peered":"2023-10-16T20:37:36.404254+0000","last_clean":"2023-10-16T20:37:36.404254+0000","last_became_active":"2023-10-16T20:36:59.464578+0000","last_became_peered":"2023-10-16T20:36:59.464578+0000","last_unstale":"2023-10-16T20:37:36.404254+0000","last_undegraded":"2023-10-16T20:37:36.404254+0000","last_fullsized":"2023-10-16T20:37:36.404254+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:12:27.738364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435179+0000","last_change":"2023-10-16T20:36:55.285675+0000","last_active":"2023-10-16T20:37:37.435179+0000","last_peered":"2023-10-16T20:37:37.435179+0000","last_clean":"2023-10-16T20:37:37.435179+0000","last_became_active":"2023-10-16T20:36:55.285407+0000","last_became_peered":"2023-10-16T20:36:55.285407+0000","last_unstale":"2023-10-16T20:37:37.435179+0000","last_undegraded":"2023-10-16T20:37:37.435179+0000","last_fullsized":"2023-10-16T20:37:37.435179+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:31:10.259938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"26'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435225+0000","last_change":"2023-10-16T20:36:54.757012+0000","last_active":"2023-10-16T20:37:37.435225+0000","last_peered":"2023-10-16T20:37:37.435225+0000","last_clean":"2023-10-16T20:37:37.435225+0000","last_became_active":"2023-10-16T20:36:54.756697+0000","last_became_peered":"2023-10-16T20:36:54.756697+0000","last_unstale":"2023-10-16T20:37:37.435225+0000","last_undegraded":"2023-10-16T20:37:37.435225+0000","last_fullsized":"2023-10-16T20:37:37.435225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:20.259580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.409816+0000","last_change":"2023-10-16T20:36:59.469134+0000","last_active":"2023-10-16T20:37:36.409816+0000","last_peered":"2023-10-16T20:37:36.409816+0000","last_clean":"2023-10-16T20:37:36.409816+0000","last_became_active":"2023-10-16T20:36:59.468254+0000","last_became_peered":"2023-10-16T20:36:59.468254+0000","last_unstale":"2023-10-16T20:37:36.409816+0000","last_undegraded":"2023-10-16T20:37:36.409816+0000","last_fullsized":"2023-10-16T20:37:36.409816+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:12.444815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.412512+0000","last_change":"2023-10-16T20:36:55.289133+0000","last_active":"2023-10-16T20:37:36.412512+0000","last_peered":"2023-10-16T20:37:36.412512+0000","last_clean":"2023-10-16T20:37:36.412512+0000","last_became_active":"2023-10-16T20:36:55.288488+0000","last_became_peered":"2023-10-16T20:36:55.288488+0000","last_unstale":"2023-10-16T20:37:36.412512+0000","last_undegraded":"2023-10-16T20:37:36.412512+0000","last_fullsized":"2023-10-16T20:37:36.412512+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:27:22.022319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'10","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:14.222665+0000","last_change":"2023-10-16T20:36:54.765361+0000","last_active":"2023-10-16T20:39:14.222665+0000","last_peered":"2023-10-16T20:39:14.222665+0000","last_clean":"2023-10-16T20:39:14.222665+0000","last_became_active":"2023-10-16T20:36:54.764969+0000","last_became_peered":"2023-10-16T20:36:54.764969+0000","last_unstale":"2023-10-16T20:39:14.222665+0000","last_undegraded":"2023-10-16T20:39:14.222665+0000","last_fullsized":"2023-10-16T20:39:14.222665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:21:21.183612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.400125+0000","last_change":"2023-10-16T20:36:59.303588+0000","last_active":"2023-10-16T20:37:36.400125+0000","last_peered":"2023-10-16T20:37:36.400125+0000","last_clean":"2023-10-16T20:37:36.400125+0000","last_became_active":"2023-10-16T20:36:59.302698+0000","last_became_peered":"2023-10-16T20:36:59.302698+0000","last_unstale":"2023-10-16T20:37:36.400125+0000","last_undegraded":"2023-10-16T20:37:36.400125+0000","last_fullsized":"2023-10-16T20:37:36.400125+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:08:28.644565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403928+0000","last_change":"2023-10-16T20:36:55.290812+0000","last_active":"2023-10-16T20:37:37.403928+0000","last_peered":"2023-10-16T20:37:37.403928+0000","last_clean":"2023-10-16T20:37:37.403928+0000","last_became_active":"2023-10-16T20:36:55.290127+0000","last_became_peered":"2023-10-16T20:36:55.290127+0000","last_unstale":"2023-10-16T20:37:37.403928+0000","last_undegraded":"2023-10-16T20:37:37.403928+0000","last_fullsized":"2023-10-16T20:37:37.403928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:28:27.930123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"27'3","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.400784+0000","last_change":"2023-10-16T20:36:54.749120+0000","last_active":"2023-10-16T20:37:36.400784+0000","last_peered":"2023-10-16T20:37:36.400784+0000","last_clean":"2023-10-16T20:37:36.400784+0000","last_became_active":"2023-10-16T20:36:54.748788+0000","last_became_peered":"2023-10-16T20:36:54.748788+0000","last_unstale":"2023-10-16T20:37:36.400784+0000","last_undegraded":"2023-10-16T20:37:36.400784+0000","last_fullsized":"2023-10-16T20:37:36.400784+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:11:07.257343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.402309+0000","last_change":"2023-10-16T20:36:59.303546+0000","last_active":"2023-10-16T20:37:36.402309+0000","last_peered":"2023-10-16T20:37:36.402309+0000","last_clean":"2023-10-16T20:37:36.402309+0000","last_became_active":"2023-10-16T20:36:59.302916+0000","last_became_peered":"2023-10-16T20:36:59.302916+0000","last_unstale":"2023-10-16T20:37:36.402309+0000","last_undegraded":"2023-10-16T20:37:36.402309+0000","last_fullsized":"2023-10-16T20:37:36.402309+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:17:16.174476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.398104+0000","last_change":"2023-10-16T20:36:55.291739+0000","last_active":"2023-10-16T20:37:36.398104+0000","last_peered":"2023-10-16T20:37:36.398104+0000","last_clean":"2023-10-16T20:37:36.398104+0000","last_became_active":"2023-10-16T20:36:55.291448+0000","last_became_peered":"2023-10-16T20:36:55.291448+0000","last_unstale":"2023-10-16T20:37:36.398104+0000","last_undegraded":"2023-10-16T20:37:36.398104+0000","last_fullsized":"2023-10-16T20:37:36.398104+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:41:15.380678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.408759+0000","last_change":"2023-10-16T20:36:54.761824+0000","last_active":"2023-10-16T20:37:36.408759+0000","last_peered":"2023-10-16T20:37:36.408759+0000","last_clean":"2023-10-16T20:37:36.408759+0000","last_became_active":"2023-10-16T20:36:54.761541+0000","last_became_peered":"2023-10-16T20:36:54.761541+0000","last_unstale":"2023-10-16T20:37:36.408759+0000","last_undegraded":"2023-10-16T20:37:36.408759+0000","last_fullsized":"2023-10-16T20:37:36.408759+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:34.512207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.403494+0000","last_change":"2023-10-16T20:36:59.304165+0000","last_active":"2023-10-16T20:37:36.403494+0000","last_peered":"2023-10-16T20:37:36.403494+0000","last_clean":"2023-10-16T20:37:36.403494+0000","last_became_active":"2023-10-16T20:36:59.303773+0000","last_became_peered":"2023-10-16T20:36:59.303773+0000","last_unstale":"2023-10-16T20:37:36.403494+0000","last_undegraded":"2023-10-16T20:37:36.403494+0000","last_fullsized":"2023-10-16T20:37:36.403494+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:09.710698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.408640+0000","last_change":"2023-10-16T20:36:55.300829+0000","last_active":"2023-10-16T20:37:36.408640+0000","last_peered":"2023-10-16T20:37:36.408640+0000","last_clean":"2023-10-16T20:37:36.408640+0000","last_became_active":"2023-10-16T20:36:55.300535+0000","last_became_peered":"2023-10-16T20:36:55.300535+0000","last_unstale":"2023-10-16T20:37:36.408640+0000","last_undegraded":"2023-10-16T20:37:36.408640+0000","last_fullsized":"2023-10-16T20:37:36.408640+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:34.910378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.405392+0000","last_change":"2023-10-16T20:36:54.758140+0000","last_active":"2023-10-16T20:37:36.405392+0000","last_peered":"2023-10-16T20:37:36.405392+0000","last_clean":"2023-10-16T20:37:36.405392+0000","last_became_active":"2023-10-16T20:36:54.757303+0000","last_became_peered":"2023-10-16T20:36:54.757303+0000","last_unstale":"2023-10-16T20:37:36.405392+0000","last_undegraded":"2023-10-16T20:37:36.405392+0000","last_fullsized":"2023-10-16T20:37:36.405392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:45.944191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404201+0000","last_change":"2023-10-16T20:36:59.471158+0000","last_active":"2023-10-16T20:37:36.404201+0000","last_peered":"2023-10-16T20:37:36.404201+0000","last_clean":"2023-10-16T20:37:36.404201+0000","last_became_active":"2023-10-16T20:36:59.465844+0000","last_became_peered":"2023-10-16T20:36:59.465844+0000","last_unstale":"2023-10-16T20:37:36.404201+0000","last_undegraded":"2023-10-16T20:37:36.404201+0000","last_fullsized":"2023-10-16T20:37:36.404201+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:12.133525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435182+0000","last_change":"2023-10-16T20:36:59.468843+0000","last_active":"2023-10-16T20:37:37.435182+0000","last_peered":"2023-10-16T20:37:37.435182+0000","last_clean":"2023-10-16T20:37:37.435182+0000","last_became_active":"2023-10-16T20:36:59.467356+0000","last_became_peered":"2023-10-16T20:36:59.467356+0000","last_unstale":"2023-10-16T20:37:37.435182+0000","last_undegraded":"2023-10-16T20:37:37.435182+0000","last_fullsized":"2023-10-16T20:37:37.435182+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:47:02.923540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.399733+0000","last_change":"2023-10-16T20:36:55.281355+0000","last_active":"2023-10-16T20:37:36.399733+0000","last_peered":"2023-10-16T20:37:36.399733+0000","last_clean":"2023-10-16T20:37:36.399733+0000","last_became_active":"2023-10-16T20:36:55.280903+0000","last_became_peered":"2023-10-16T20:36:55.280903+0000","last_unstale":"2023-10-16T20:37:36.399733+0000","last_undegraded":"2023-10-16T20:37:36.399733+0000","last_fullsized":"2023-10-16T20:37:36.399733+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:15:25.813826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.408338+0000","last_change":"2023-10-16T20:36:54.757589+0000","last_active":"2023-10-16T20:37:36.408338+0000","last_peered":"2023-10-16T20:37:36.408338+0000","last_clean":"2023-10-16T20:37:36.408338+0000","last_became_active":"2023-10-16T20:36:54.756775+0000","last_became_peered":"2023-10-16T20:36:54.756775+0000","last_unstale":"2023-10-16T20:37:36.408338+0000","last_undegraded":"2023-10-16T20:37:36.408338+0000","last_fullsized":"2023-10-16T20:37:36.408338+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:40.629540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.441661+0000","last_change":"2023-10-16T20:37:37.438581+0000","last_active":"2023-10-16T20:37:37.441661+0000","last_peered":"2023-10-16T20:37:37.441661+0000","last_clean":"2023-10-16T20:37:37.441661+0000","last_became_active":"2023-10-16T20:37:37.438140+0000","last_became_peered":"2023-10-16T20:37:37.438140+0000","last_unstale":"2023-10-16T20:37:37.441661+0000","last_undegraded":"2023-10-16T20:37:37.441661+0000","last_fullsized":"2023-10-16T20:37:37.441661+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:55.110618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.403530+0000","last_change":"2023-10-16T20:36:55.274207+0000","last_active":"2023-10-16T20:37:36.403530+0000","last_peered":"2023-10-16T20:37:36.403530+0000","last_clean":"2023-10-16T20:37:36.403530+0000","last_became_active":"2023-10-16T20:36:55.273959+0000","last_became_peered":"2023-10-16T20:36:55.273959+0000","last_unstale":"2023-10-16T20:37:36.403530+0000","last_undegraded":"2023-10-16T20:37:36.403530+0000","last_fullsized":"2023-10-16T20:37:36.403530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:23:09.275091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404500+0000","last_change":"2023-10-16T20:36:54.759513+0000","last_active":"2023-10-16T20:37:36.404500+0000","last_peered":"2023-10-16T20:37:36.404500+0000","last_clean":"2023-10-16T20:37:36.404500+0000","last_became_active":"2023-10-16T20:36:54.759059+0000","last_became_peered":"2023-10-16T20:36:54.759059+0000","last_unstale":"2023-10-16T20:37:36.404500+0000","last_undegraded":"2023-10-16T20:37:36.404500+0000","last_fullsized":"2023-10-16T20:37:36.404500+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:36.696053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"27'1","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404990+0000","last_change":"2023-10-16T20:36:54.764780+0000","last_active":"2023-10-16T20:37:36.404990+0000","last_peered":"2023-10-16T20:37:36.404990+0000","last_clean":"2023-10-16T20:37:36.404990+0000","last_became_active":"2023-10-16T20:36:54.764436+0000","last_became_peered":"2023-10-16T20:36:54.764436+0000","last_unstale":"2023-10-16T20:37:36.404990+0000","last_undegraded":"2023-10-16T20:37:36.404990+0000","last_fullsized":"2023-10-16T20:37:36.404990+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:55:31.816599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.398400+0000","last_change":"2023-10-16T20:36:55.278134+0000","last_active":"2023-10-16T20:37:36.398400+0000","last_peered":"2023-10-16T20:37:36.398400+0000","last_clean":"2023-10-16T20:37:36.398400+0000","last_became_active":"2023-10-16T20:36:55.277873+0000","last_became_peered":"2023-10-16T20:36:55.277873+0000","last_unstale":"2023-10-16T20:37:36.398400+0000","last_undegraded":"2023-10-16T20:37:36.398400+0000","last_fullsized":"2023-10-16T20:37:36.398400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:59:43.701800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.407258+0000","last_change":"2023-10-16T20:36:59.302774+0000","last_active":"2023-10-16T20:37:36.407258+0000","last_peered":"2023-10-16T20:37:36.407258+0000","last_clean":"2023-10-16T20:37:36.407258+0000","last_became_active":"2023-10-16T20:36:59.302435+0000","last_became_peered":"2023-10-16T20:36:59.302435+0000","last_unstale":"2023-10-16T20:37:36.407258+0000","last_undegraded":"2023-10-16T20:37:36.407258+0000","last_fullsized":"2023-10-16T20:37:36.407258+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:11:53.740982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"26'1","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404112+0000","last_change":"2023-10-16T20:36:54.756190+0000","last_active":"2023-10-16T20:37:36.404112+0000","last_peered":"2023-10-16T20:37:36.404112+0000","last_clean":"2023-10-16T20:37:36.404112+0000","last_became_active":"2023-10-16T20:36:54.755834+0000","last_became_peered":"2023-10-16T20:36:54.755834+0000","last_unstale":"2023-10-16T20:37:36.404112+0000","last_undegraded":"2023-10-16T20:37:36.404112+0000","last_fullsized":"2023-10-16T20:37:36.404112+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:31.012441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.402297+0000","last_change":"2023-10-16T20:36:55.279822+0000","last_active":"2023-10-16T20:37:36.402297+0000","last_peered":"2023-10-16T20:37:36.402297+0000","last_clean":"2023-10-16T20:37:36.402297+0000","last_became_active":"2023-10-16T20:36:55.279333+0000","last_became_peered":"2023-10-16T20:36:55.279333+0000","last_unstale":"2023-10-16T20:37:36.402297+0000","last_undegraded":"2023-10-16T20:37:36.402297+0000","last_fullsized":"2023-10-16T20:37:36.402297+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:02:21.471812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.403977+0000","last_change":"2023-10-16T20:36:59.302190+0000","last_active":"2023-10-16T20:37:36.403977+0000","last_peered":"2023-10-16T20:37:36.403977+0000","last_clean":"2023-10-16T20:37:36.403977+0000","last_became_active":"2023-10-16T20:36:59.301864+0000","last_became_peered":"2023-10-16T20:36:59.301864+0000","last_unstale":"2023-10-16T20:37:36.403977+0000","last_undegraded":"2023-10-16T20:37:36.403977+0000","last_fullsized":"2023-10-16T20:37:36.403977+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:37:55.282815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.406436+0000","last_change":"2023-10-16T20:36:55.302476+0000","last_active":"2023-10-16T20:37:37.406436+0000","last_peered":"2023-10-16T20:37:37.406436+0000","last_clean":"2023-10-16T20:37:37.406436+0000","last_became_active":"2023-10-16T20:36:55.287442+0000","last_became_peered":"2023-10-16T20:36:55.287442+0000","last_unstale":"2023-10-16T20:37:37.406436+0000","last_undegraded":"2023-10-16T20:37:37.406436+0000","last_fullsized":"2023-10-16T20:37:37.406436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:19.511156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.399520+0000","last_change":"2023-10-16T20:36:55.276521+0000","last_active":"2023-10-16T20:37:37.399520+0000","last_peered":"2023-10-16T20:37:37.399520+0000","last_clean":"2023-10-16T20:37:37.399520+0000","last_became_active":"2023-10-16T20:36:55.276276+0000","last_became_peered":"2023-10-16T20:36:55.276276+0000","last_unstale":"2023-10-16T20:37:37.399520+0000","last_undegraded":"2023-10-16T20:37:37.399520+0000","last_fullsized":"2023-10-16T20:37:37.399520+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:38.218242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.415548+0000","last_change":"2023-10-16T20:37:37.413429+0000","last_active":"2023-10-16T20:37:37.415548+0000","last_peered":"2023-10-16T20:37:37.415548+0000","last_clean":"2023-10-16T20:37:37.415548+0000","last_became_active":"2023-10-16T20:37:37.413040+0000","last_became_peered":"2023-10-16T20:37:37.413040+0000","last_unstale":"2023-10-16T20:37:37.415548+0000","last_undegraded":"2023-10-16T20:37:37.415548+0000","last_fullsized":"2023-10-16T20:37:37.415548+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:39:10.897056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435108+0000","last_change":"2023-10-16T20:36:55.276258+0000","last_active":"2023-10-16T20:37:37.435108+0000","last_peered":"2023-10-16T20:37:37.435108+0000","last_clean":"2023-10-16T20:37:37.435108+0000","last_became_active":"2023-10-16T20:36:55.275349+0000","last_became_peered":"2023-10-16T20:36:55.275349+0000","last_unstale":"2023-10-16T20:37:37.435108+0000","last_undegraded":"2023-10-16T20:37:37.435108+0000","last_fullsized":"2023-10-16T20:37:37.435108+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:59.007782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465455+0000","last_change":"2023-10-16T20:36:55.277332+0000","last_active":"2023-10-16T20:37:37.465455+0000","last_peered":"2023-10-16T20:37:37.465455+0000","last_clean":"2023-10-16T20:37:37.465455+0000","last_became_active":"2023-10-16T20:36:55.277043+0000","last_became_peered":"2023-10-16T20:36:55.277043+0000","last_unstale":"2023-10-16T20:37:37.465455+0000","last_undegraded":"2023-10-16T20:37:37.465455+0000","last_fullsized":"2023-10-16T20:37:37.465455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:46:07.227813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401690+0000","last_change":"2023-10-16T20:36:55.279003+0000","last_active":"2023-10-16T20:37:37.401690+0000","last_peered":"2023-10-16T20:37:37.401690+0000","last_clean":"2023-10-16T20:37:37.401690+0000","last_became_active":"2023-10-16T20:36:55.278715+0000","last_became_peered":"2023-10-16T20:36:55.278715+0000","last_unstale":"2023-10-16T20:37:37.401690+0000","last_undegraded":"2023-10-16T20:37:37.401690+0000","last_fullsized":"2023-10-16T20:37:37.401690+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:58.274009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408078+0000","last_change":"2023-10-16T20:36:55.303936+0000","last_active":"2023-10-16T20:37:37.408078+0000","last_peered":"2023-10-16T20:37:37.408078+0000","last_clean":"2023-10-16T20:37:37.408078+0000","last_became_active":"2023-10-16T20:36:55.301604+0000","last_became_peered":"2023-10-16T20:36:55.301604+0000","last_unstale":"2023-10-16T20:37:37.408078+0000","last_undegraded":"2023-10-16T20:37:37.408078+0000","last_fullsized":"2023-10-16T20:37:37.408078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:48:14.922695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465625+0000","last_change":"2023-10-16T20:36:55.277329+0000","last_active":"2023-10-16T20:37:37.465625+0000","last_peered":"2023-10-16T20:37:37.465625+0000","last_clean":"2023-10-16T20:37:37.465625+0000","last_became_active":"2023-10-16T20:36:55.276984+0000","last_became_peered":"2023-10-16T20:36:55.276984+0000","last_unstale":"2023-10-16T20:37:37.465625+0000","last_undegraded":"2023-10-16T20:37:37.465625+0000","last_fullsized":"2023-10-16T20:37:37.465625+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:43:52.763586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408404+0000","last_change":"2023-10-16T20:36:55.290545+0000","last_active":"2023-10-16T20:37:37.408404+0000","last_peered":"2023-10-16T20:37:37.408404+0000","last_clean":"2023-10-16T20:37:37.408404+0000","last_became_active":"2023-10-16T20:36:55.290023+0000","last_became_peered":"2023-10-16T20:36:55.290023+0000","last_unstale":"2023-10-16T20:37:37.408404+0000","last_undegraded":"2023-10-16T20:37:37.408404+0000","last_fullsized":"2023-10-16T20:37:37.408404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:04:20.914854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464978+0000","last_change":"2023-10-16T20:36:55.280456+0000","last_active":"2023-10-16T20:37:37.464978+0000","last_peered":"2023-10-16T20:37:37.464978+0000","last_clean":"2023-10-16T20:37:37.464978+0000","last_became_active":"2023-10-16T20:36:55.280186+0000","last_became_peered":"2023-10-16T20:36:55.280186+0000","last_unstale":"2023-10-16T20:37:37.464978+0000","last_undegraded":"2023-10-16T20:37:37.464978+0000","last_fullsized":"2023-10-16T20:37:37.464978+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:52.469768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408252+0000","last_change":"2023-10-16T20:36:55.297917+0000","last_active":"2023-10-16T20:37:37.408252+0000","last_peered":"2023-10-16T20:37:37.408252+0000","last_clean":"2023-10-16T20:37:37.408252+0000","last_became_active":"2023-10-16T20:36:55.297253+0000","last_became_peered":"2023-10-16T20:36:55.297253+0000","last_unstale":"2023-10-16T20:37:37.408252+0000","last_undegraded":"2023-10-16T20:37:37.408252+0000","last_fullsized":"2023-10-16T20:37:37.408252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:14.977093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434979+0000","last_change":"2023-10-16T20:36:55.286327+0000","last_active":"2023-10-16T20:37:37.434979+0000","last_peered":"2023-10-16T20:37:37.434979+0000","last_clean":"2023-10-16T20:37:37.434979+0000","last_became_active":"2023-10-16T20:36:55.286044+0000","last_became_peered":"2023-10-16T20:36:55.286044+0000","last_unstale":"2023-10-16T20:37:37.434979+0000","last_undegraded":"2023-10-16T20:37:37.434979+0000","last_fullsized":"2023-10-16T20:37:37.434979+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:30:56.000502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409227+0000","last_change":"2023-10-16T20:36:55.298742+0000","last_active":"2023-10-16T20:37:37.409227+0000","last_peered":"2023-10-16T20:37:37.409227+0000","last_clean":"2023-10-16T20:37:37.409227+0000","last_became_active":"2023-10-16T20:36:55.286864+0000","last_became_peered":"2023-10-16T20:36:55.286864+0000","last_unstale":"2023-10-16T20:37:37.409227+0000","last_undegraded":"2023-10-16T20:37:37.409227+0000","last_fullsized":"2023-10-16T20:37:37.409227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:24:27.598910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466521+0000","last_change":"2023-10-16T20:36:55.292541+0000","last_active":"2023-10-16T20:37:37.466521+0000","last_peered":"2023-10-16T20:37:37.466521+0000","last_clean":"2023-10-16T20:37:37.466521+0000","last_became_active":"2023-10-16T20:36:55.292262+0000","last_became_peered":"2023-10-16T20:36:55.292262+0000","last_unstale":"2023-10-16T20:37:37.466521+0000","last_undegraded":"2023-10-16T20:37:37.466521+0000","last_fullsized":"2023-10-16T20:37:37.466521+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:10:08.644265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.400632+0000","last_change":"2023-10-16T20:36:55.283056+0000","last_active":"2023-10-16T20:37:37.400632+0000","last_peered":"2023-10-16T20:37:37.400632+0000","last_clean":"2023-10-16T20:37:37.400632+0000","last_became_active":"2023-10-16T20:36:55.282806+0000","last_became_peered":"2023-10-16T20:36:55.282806+0000","last_unstale":"2023-10-16T20:37:37.400632+0000","last_undegraded":"2023-10-16T20:37:37.400632+0000","last_fullsized":"2023-10-16T20:37:37.400632+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:21:24.681696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410649+0000","last_change":"2023-10-16T20:36:55.303385+0000","last_active":"2023-10-16T20:37:37.410649+0000","last_peered":"2023-10-16T20:37:37.410649+0000","last_clean":"2023-10-16T20:37:37.410649+0000","last_became_active":"2023-10-16T20:36:55.303095+0000","last_became_peered":"2023-10-16T20:36:55.303095+0000","last_unstale":"2023-10-16T20:37:37.410649+0000","last_undegraded":"2023-10-16T20:37:37.410649+0000","last_fullsized":"2023-10-16T20:37:37.410649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:26.368480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408312+0000","last_change":"2023-10-16T20:36:55.291852+0000","last_active":"2023-10-16T20:37:37.408312+0000","last_peered":"2023-10-16T20:37:37.408312+0000","last_clean":"2023-10-16T20:37:37.408312+0000","last_became_active":"2023-10-16T20:36:55.291447+0000","last_became_peered":"2023-10-16T20:36:55.291447+0000","last_unstale":"2023-10-16T20:37:37.408312+0000","last_undegraded":"2023-10-16T20:37:37.408312+0000","last_fullsized":"2023-10-16T20:37:37.408312+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:07.195372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401838+0000","last_change":"2023-10-16T20:36:55.281169+0000","last_active":"2023-10-16T20:37:37.401838+0000","last_peered":"2023-10-16T20:37:37.401838+0000","last_clean":"2023-10-16T20:37:37.401838+0000","last_became_active":"2023-10-16T20:36:55.280605+0000","last_became_peered":"2023-10-16T20:36:55.280605+0000","last_unstale":"2023-10-16T20:37:37.401838+0000","last_undegraded":"2023-10-16T20:37:37.401838+0000","last_fullsized":"2023-10-16T20:37:37.401838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:03.294173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435040+0000","last_change":"2023-10-16T20:36:55.277395+0000","last_active":"2023-10-16T20:37:37.435040+0000","last_peered":"2023-10-16T20:37:37.435040+0000","last_clean":"2023-10-16T20:37:37.435040+0000","last_became_active":"2023-10-16T20:36:55.277094+0000","last_became_peered":"2023-10-16T20:36:55.277094+0000","last_unstale":"2023-10-16T20:37:37.435040+0000","last_undegraded":"2023-10-16T20:37:37.435040+0000","last_fullsized":"2023-10-16T20:37:37.435040+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:23:03.400384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.403871+0000","last_change":"2023-10-16T20:36:55.297429+0000","last_active":"2023-10-16T20:37:36.403871+0000","last_peered":"2023-10-16T20:37:36.403871+0000","last_clean":"2023-10-16T20:37:36.403871+0000","last_became_active":"2023-10-16T20:36:55.297199+0000","last_became_peered":"2023-10-16T20:36:55.297199+0000","last_unstale":"2023-10-16T20:37:36.403871+0000","last_undegraded":"2023-10-16T20:37:36.403871+0000","last_fullsized":"2023-10-16T20:37:36.403871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:43.860251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.403290+0000","last_change":"2023-10-16T20:36:55.287421+0000","last_active":"2023-10-16T20:37:36.403290+0000","last_peered":"2023-10-16T20:37:36.403290+0000","last_clean":"2023-10-16T20:37:36.403290+0000","last_became_active":"2023-10-16T20:36:55.286964+0000","last_became_peered":"2023-10-16T20:36:55.286964+0000","last_unstale":"2023-10-16T20:37:36.403290+0000","last_undegraded":"2023-10-16T20:37:36.403290+0000","last_fullsized":"2023-10-16T20:37:36.403290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:15:54.970100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.413891+0000","last_change":"2023-10-16T20:37:37.413891+0000","last_active":"2023-10-16T20:37:37.413891+0000","last_peered":"2023-10-16T20:37:37.413891+0000","last_clean":"2023-10-16T20:37:37.413891+0000","last_became_active":"2023-10-16T20:37:37.413492+0000","last_became_peered":"2023-10-16T20:37:37.413492+0000","last_unstale":"2023-10-16T20:37:37.413891+0000","last_undegraded":"2023-10-16T20:37:37.413891+0000","last_fullsized":"2023-10-16T20:37:37.413891+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:03:46.658556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.405037+0000","last_change":"2023-10-16T20:36:55.303950+0000","last_active":"2023-10-16T20:37:36.405037+0000","last_peered":"2023-10-16T20:37:36.405037+0000","last_clean":"2023-10-16T20:37:36.405037+0000","last_became_active":"2023-10-16T20:36:55.302468+0000","last_became_peered":"2023-10-16T20:36:55.302468+0000","last_unstale":"2023-10-16T20:37:36.405037+0000","last_undegraded":"2023-10-16T20:37:36.405037+0000","last_fullsized":"2023-10-16T20:37:36.405037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:02.962031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.413370+0000","last_change":"2023-10-16T20:36:55.297558+0000","last_active":"2023-10-16T20:37:36.413370+0000","last_peered":"2023-10-16T20:37:36.413370+0000","last_clean":"2023-10-16T20:37:36.413370+0000","last_became_active":"2023-10-16T20:36:55.296476+0000","last_became_peered":"2023-10-16T20:36:55.296476+0000","last_unstale":"2023-10-16T20:37:36.413370+0000","last_undegraded":"2023-10-16T20:37:36.413370+0000","last_fullsized":"2023-10-16T20:37:36.413370+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:04:38.235205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.403439+0000","last_change":"2023-10-16T20:36:55.293957+0000","last_active":"2023-10-16T20:37:36.403439+0000","last_peered":"2023-10-16T20:37:36.403439+0000","last_clean":"2023-10-16T20:37:36.403439+0000","last_became_active":"2023-10-16T20:36:55.293687+0000","last_became_peered":"2023-10-16T20:36:55.293687+0000","last_unstale":"2023-10-16T20:37:36.403439+0000","last_undegraded":"2023-10-16T20:37:36.403439+0000","last_fullsized":"2023-10-16T20:37:36.403439+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:54.623691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.411550+0000","last_change":"2023-10-16T20:36:55.280728+0000","last_active":"2023-10-16T20:37:36.411550+0000","last_peered":"2023-10-16T20:37:36.411550+0000","last_clean":"2023-10-16T20:37:36.411550+0000","last_became_active":"2023-10-16T20:36:55.280444+0000","last_became_peered":"2023-10-16T20:36:55.280444+0000","last_unstale":"2023-10-16T20:37:36.411550+0000","last_undegraded":"2023-10-16T20:37:36.411550+0000","last_fullsized":"2023-10-16T20:37:36.411550+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:44.044563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.408400+0000","last_change":"2023-10-16T20:36:55.297802+0000","last_active":"2023-10-16T20:37:36.408400+0000","last_peered":"2023-10-16T20:37:36.408400+0000","last_clean":"2023-10-16T20:37:36.408400+0000","last_became_active":"2023-10-16T20:36:55.296694+0000","last_became_peered":"2023-10-16T20:36:55.296694+0000","last_unstale":"2023-10-16T20:37:36.408400+0000","last_undegraded":"2023-10-16T20:37:36.408400+0000","last_fullsized":"2023-10-16T20:37:36.408400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:00:26.797387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.411707+0000","last_change":"2023-10-16T20:36:55.291347+0000","last_active":"2023-10-16T20:37:36.411707+0000","last_peered":"2023-10-16T20:37:36.411707+0000","last_clean":"2023-10-16T20:37:36.411707+0000","last_became_active":"2023-10-16T20:36:55.291052+0000","last_became_peered":"2023-10-16T20:36:55.291052+0000","last_unstale":"2023-10-16T20:37:36.411707+0000","last_undegraded":"2023-10-16T20:37:36.411707+0000","last_fullsized":"2023-10-16T20:37:36.411707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:40:51.331446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409006+0000","last_change":"2023-10-16T20:36:55.290870+0000","last_active":"2023-10-16T20:37:37.409006+0000","last_peered":"2023-10-16T20:37:37.409006+0000","last_clean":"2023-10-16T20:37:37.409006+0000","last_became_active":"2023-10-16T20:36:55.290386+0000","last_became_peered":"2023-10-16T20:36:55.290386+0000","last_unstale":"2023-10-16T20:37:37.409006+0000","last_undegraded":"2023-10-16T20:37:37.409006+0000","last_fullsized":"2023-10-16T20:37:37.409006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:20.006864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435199+0000","last_change":"2023-10-16T20:36:55.271041+0000","last_active":"2023-10-16T20:37:37.435199+0000","last_peered":"2023-10-16T20:37:37.435199+0000","last_clean":"2023-10-16T20:37:37.435199+0000","last_became_active":"2023-10-16T20:36:55.270787+0000","last_became_peered":"2023-10-16T20:36:55.270787+0000","last_unstale":"2023-10-16T20:37:37.435199+0000","last_undegraded":"2023-10-16T20:37:37.435199+0000","last_fullsized":"2023-10-16T20:37:37.435199+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:07.944778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402678+0000","last_change":"2023-10-16T20:36:55.271632+0000","last_active":"2023-10-16T20:37:37.402678+0000","last_peered":"2023-10-16T20:37:37.402678+0000","last_clean":"2023-10-16T20:37:37.402678+0000","last_became_active":"2023-10-16T20:36:55.271373+0000","last_became_peered":"2023-10-16T20:36:55.271373+0000","last_unstale":"2023-10-16T20:37:37.402678+0000","last_undegraded":"2023-10-16T20:37:37.402678+0000","last_fullsized":"2023-10-16T20:37:37.402678+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:07.458540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466741+0000","last_change":"2023-10-16T20:36:55.291343+0000","last_active":"2023-10-16T20:37:37.466741+0000","last_peered":"2023-10-16T20:37:37.466741+0000","last_clean":"2023-10-16T20:37:37.466741+0000","last_became_active":"2023-10-16T20:36:55.291016+0000","last_became_peered":"2023-10-16T20:36:55.291016+0000","last_unstale":"2023-10-16T20:37:37.466741+0000","last_undegraded":"2023-10-16T20:37:37.466741+0000","last_fullsized":"2023-10-16T20:37:37.466741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:13:40.287782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4639,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509473,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343936,"statfs":{"total":96636764160,"available":96608190464,"internally_reserved":0,"allocated":905216,"data_stored":715195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.74299999999999999,"15min":0.748},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.628},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.75700000000000001,"15min":0.753},"min":{"1min":0.58099999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94599999999999995,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.76800000000000002}]},{"osd":1,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.74299999999999999,"15min":0.73999999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.91600000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.52100000000000002,"5min":0.497,"15min":0.497},"max":{"1min":0.91500000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.68899999999999995}]},{"osd":2,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.73199999999999998,"15min":0.72599999999999998},"min":{"1min":0.624,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.748,"15min":0.745},"min":{"1min":0.60699999999999998,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.89700000000000002,"5min":1.083,"15min":1.083},"last":0.66800000000000004}]},{"osd":3,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.77600000000000002,"15min":0.77200000000000002},"min":{"1min":0.62,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0069999999999999,"5min":1.234,"15min":1.234},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.93200000000000005,"5min":1.024,"15min":1.024},"last":0.63800000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63100000000000001,"15min":0.63300000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.92500000000000004,"5min":1.3,"15min":1.3},"last":0.496},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60499999999999998,"15min":0.60999999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76500000000000001,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.65600000000000003}]},{"osd":5,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.47599999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.78600000000000003,"15min":0.79500000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.006,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.72199999999999998}]},{"osd":6,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.749,"15min":0.755},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.97399999999999998,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.70399999999999996,"15min":0.70599999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.88300000000000001,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.51900000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509473,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":128086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.76800000000000002,"15min":0.77100000000000002},"min":{"1min":0.41899999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88,"5min":1.131,"15min":1.131},"last":0.751},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.748,"15min":0.75800000000000001},"min":{"1min":0.48899999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.875,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.69199999999999995}]},{"osd":1,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.751,"15min":0.76300000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.81799999999999995,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.624},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.77000000000000002,"15min":0.77600000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.90400000000000003,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.76900000000000002}]},{"osd":2,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.78000000000000003,"15min":0.79200000000000004},"min":{"1min":0.48599999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.90300000000000002,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.73899999999999999,"15min":0.746},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.86899999999999999,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.71099999999999997}]},{"osd":3,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.84999999999999998,"15min":0.87},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.90600000000000003,"5min":1.611,"15min":1.611},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.749,"5min":0.80900000000000005,"15min":0.81899999999999995},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.89700000000000002,"5min":1.163,"15min":1.163},"last":0.64400000000000002}]},{"osd":4,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.66700000000000004,"15min":0.67900000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.79500000000000004,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65700000000000003,"15min":0.66700000000000004},"min":{"1min":0.49299999999999999,"5min":0.435,"15min":0.435},"max":{"1min":0.69499999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.497}]},{"osd":5,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86199999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69899999999999995,"15min":0.70599999999999996},"min":{"1min":0.54900000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.82399999999999995,"5min":1.016,"15min":1.016},"last":0.73399999999999999}]},{"osd":7,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.748,"15min":0.754},"min":{"1min":0.52200000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.93100000000000005,"5min":1.292,"15min":1.292},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.75900000000000001,"15min":0.77500000000000002},"min":{"1min":0.52800000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.92100000000000004,"5min":1.569,"15min":1.569},"last":0.78800000000000003}]}]},{"osd":5,"up_from":15,"seq":64424509475,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":311296,"data_stored":124335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.70099999999999996,"15min":0.69599999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.505},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.56899999999999995}]},{"osd":1,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70699999999999996,"15min":0.70499999999999996},"min":{"1min":0.52100000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.752,"5min":0.71499999999999997,"15min":0.70899999999999996},"min":{"1min":0.57399999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.63200000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.72199999999999998,"15min":0.73199999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92300000000000004,"5min":1.022,"15min":1.022},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.74299999999999999,"15min":0.73699999999999999},"min":{"1min":0.60799999999999998,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.91700000000000004,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65600000000000003}]},{"osd":3,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79200000000000004,"15min":0.79100000000000004},"min":{"1min":0.66700000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81000000000000005,"15min":0.81599999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.70799999999999996}]},{"osd":4,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65100000000000002,"15min":0.65900000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.90300000000000002,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.73399999999999999,"15min":0.75800000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.80500000000000005,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.61599999999999999}]},{"osd":6,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.70999999999999996,"15min":0.72799999999999998},"min":{"1min":0.45500000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.79700000000000004,"5min":1.069,"15min":1.069},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.495,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.038,"5min":1.264,"15min":1.264},"last":0.67500000000000004}]},{"osd":7,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.78000000000000003,"15min":0.79600000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.89200000000000002,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.79100000000000004,"15min":0.80700000000000005},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.163,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.76900000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509473,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":319488,"data_stored":124371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.72199999999999998,"15min":0.72399999999999998},"min":{"1min":0.45200000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.872,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.59599999999999997,"5min":0.5,"15min":0.5},"max":{"1min":0.88200000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.80800000000000005}]},{"osd":1,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.57499999999999996,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92500000000000004,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.748,"15min":0.748},"min":{"1min":0.54500000000000004,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.69099999999999995}]},{"osd":2,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.76400000000000001,"15min":0.76200000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77700000000000002,"15min":0.77800000000000002},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.746}]},{"osd":3,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.747,"15min":0.73899999999999999},"min":{"1min":0.58899999999999997,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.78200000000000003,"15min":0.78600000000000003},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.86399999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.83299999999999996}]},{"osd":5,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.63600000000000001,"15min":0.64800000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.75600000000000001,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.44800000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95899999999999996,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.91700000000000004}]},{"osd":6,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.66500000000000004,"15min":0.66700000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73499999999999999,"15min":0.73799999999999999},"min":{"1min":0.495,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.85999999999999999}]},{"osd":7,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.747,"15min":0.751},"min":{"1min":0.52000000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.94599999999999995,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.79300000000000004,"15min":0.80300000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.96599999999999997,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.77100000000000002}]}]},{"osd":3,"up_from":15,"seq":64424509473,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":128086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.46999999999999997,"15min":0.47199999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.69499999999999995,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.498},{"interface":"front","average":{"1min":0.5,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.53400000000000003}]},{"osd":1,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.41499999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71699999999999997,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.56299999999999994}]},{"osd":2,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.53800000000000003,"15min":0.54400000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.67600000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.626,"15min":0.61699999999999999},"min":{"1min":0.51600000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.58299999999999996}]},{"osd":4,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.80200000000000005,"15min":0.80000000000000004},"min":{"1min":0.56200000000000006,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.054,"5min":1.181,"15min":1.181},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.77300000000000002,"15min":0.76500000000000001},"min":{"1min":0.57499999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.052,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.88600000000000001}]},{"osd":5,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.85999999999999999,"15min":0.873},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.008,"5min":1.544,"15min":1.544},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.323,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.79100000000000004}]},{"osd":6,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.85499999999999998,"15min":0.85799999999999998},"min":{"1min":0.64900000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.3480000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.755}]},{"osd":7,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.80600000000000005,"15min":0.80000000000000004},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0880000000000001,"5min":1.139,"15min":1.139},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.0029999999999999,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.76400000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509473,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":124245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.47899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71099999999999997,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.58899999999999997}]},{"osd":1,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77400000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55100000000000005,"15min":0.54900000000000004},"min":{"1min":0.38700000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69399999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44600000000000001}]},{"osd":3,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54400000000000004,"15min":0.53600000000000003},"min":{"1min":0.41299999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76700000000000002,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.46999999999999997}]},{"osd":4,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.82299999999999995,"15min":0.82899999999999996},"min":{"1min":0.60799999999999998,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.97499999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.64600000000000002,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.98299999999999998,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.63900000000000001}]},{"osd":5,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.60999999999999999,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.96999999999999997,"5min":1.444,"15min":1.444},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.83899999999999997,"15min":0.84199999999999997},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":0.998,"5min":1.585,"15min":1.585},"last":0.79700000000000004}]},{"osd":6,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.84799999999999998,"15min":0.85199999999999998},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.1020000000000001,"5min":1.335,"15min":1.335},"last":0.754},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.871,"15min":0.88200000000000001},"min":{"1min":0.68999999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":0.95099999999999996,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.69799999999999995}]},{"osd":7,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.86499999999999999,"15min":0.875},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.046,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.88300000000000001,"15min":0.89300000000000002},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.0589999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.78900000000000003}]}]},{"osd":1,"up_from":15,"seq":64424509473,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":315392,"data_stored":124405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.46400000000000002,"15min":0.46700000000000003},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.64700000000000002,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.32600000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.69899999999999995,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.35099999999999998}]},{"osd":2,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.375,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82499999999999996,"5min":1.046,"15min":1.046},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.70699999999999996,"5min":1.115,"15min":1.115},"last":0.373}]},{"osd":3,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60599999999999998,"15min":0.60899999999999999},"min":{"1min":0.40799999999999997,"5min":0.312,"15min":0.312},"max":{"1min":0.78500000000000003,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.60799999999999998,"15min":0.61799999999999999},"min":{"1min":0.41499999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.80400000000000005,"5min":1.431,"15min":1.431},"last":0.48199999999999998}]},{"osd":4,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.83399999999999996,"15min":0.83599999999999997},"min":{"1min":0.59199999999999997,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.242,"15min":1.242},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.81299999999999994,"15min":0.81100000000000005},"min":{"1min":0.60099999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.054,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.67500000000000004}]},{"osd":5,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.80800000000000005,"15min":0.81999999999999995},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90800000000000003,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.871,"15min":0.872},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.1160000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.68500000000000005}]},{"osd":6,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.874,"15min":0.88100000000000001},"min":{"1min":0.65300000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.123,"5min":2.004,"15min":2.004},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.876,"15min":0.88300000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.0880000000000001,"5min":1.635,"15min":1.635},"last":0.72499999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.878,"15min":0.89200000000000002},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.99099999999999999,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.871,"5min":0.90700000000000003,"15min":0.91300000000000003},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.0760000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.66500000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343928,"statfs":{"total":96636764160,"available":96608182272,"internally_reserved":0,"allocated":913408,"data_stored":715251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.439,"15min":0.438},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.495,"15min":0.496},"min":{"1min":0.374,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.58999999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.313}]},{"osd":2,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52300000000000002,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54100000000000004,"15min":0.54600000000000004},"min":{"1min":0.40400000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.68899999999999995,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.46999999999999997}]},{"osd":3,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54800000000000004,"15min":0.54500000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73399999999999999,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.56499999999999995,"5min":0.497,"15min":0.497},"max":{"1min":0.84999999999999998,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.442}]},{"osd":4,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.83799999999999997,"15min":0.82999999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0600000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.85999999999999999,"15min":0.84899999999999998},"min":{"1min":0.61399999999999999,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.59599999999999997}]},{"osd":5,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.80800000000000005,"15min":0.80300000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.67300000000000004}]},{"osd":6,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.86299999999999999,"15min":0.85899999999999999},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.746},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88400000000000001,"15min":0.88300000000000001},"min":{"1min":0.60899999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.56599999999999995}]},{"osd":7,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.64900000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.83499999999999996,"15min":0.82299999999999995},"min":{"1min":0.66500000000000004,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.69899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T20:39:20.468 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T20:39:20.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.629+0000 7efc9b93e640 1 Processor -- start 2023-10-16T20:39:20.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.629+0000 7efc9b93e640 1 -- start start 2023-10-16T20:39:20.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.629+0000 7efc9b93e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efc94153770 0x7efc94159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:20.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.629+0000 7efc9b93e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9415a410 con 0x7efc940c9bb0 2023-10-16T20:39:20.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.629+0000 7efc9b93e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc94153b50 con 0x7efc940cece0 2023-10-16T20:39:20.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.629+0000 7efc9b93e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc940cc250 con 0x7efc94153770 2023-10-16T20:39:20.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.629+0000 7efc996da640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efc94153770 0x7efc94159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:20.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.629+0000 7efc98ed9640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7efc940c9bb0 0x7efc940c3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36872/0 (socket says 172.21.15.88:36872) 2023-10-16T20:39:20.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.629+0000 7efc98ed9640 1 -- 172.21.15.88:0/2978081205 learned_addr learned my addr 172.21.15.88:0/2978081205 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:20.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.629+0000 7efc996da640 1 -- 172.21.15.88:0/2978081205 >> v1:172.21.15.191:6789/0 conn(0x7efc940cece0 legacy=0x7efc94163dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:20.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.629+0000 7efc996da640 1 -- 172.21.15.88:0/2978081205 >> v1:172.21.15.88:6789/0 conn(0x7efc940c9bb0 legacy=0x7efc940c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:20.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.629+0000 7efc996da640 1 -- 172.21.15.88:0/2978081205 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc940cb2f0 con 0x7efc94153770 2023-10-16T20:39:20.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.629+0000 7efc8bfff640 1 -- 172.21.15.88:0/2978081205 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935618203 0 0) 0x7efc9415a410 con 0x7efc940c9bb0 2023-10-16T20:39:20.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.633+0000 7efc996da640 1 --2- 172.21.15.88:0/2978081205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc94153770 0x7efc94159ed0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efc900097a0 tx=0x7efc9001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5811ecdda165731c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:20.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.633+0000 7efc8bfff640 1 -- 172.21.15.88:0/2978081205 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc900ab600 con 0x7efc94153770 2023-10-16T20:39:20.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.633+0000 7efc8bfff640 1 -- 172.21.15.88:0/2978081205 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc900a7050 con 0x7efc94153770 2023-10-16T20:39:20.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.633+0000 7efc8bfff640 1 -- 172.21.15.88:0/2978081205 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc900ab980 con 0x7efc94153770 2023-10-16T20:39:20.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.633+0000 7efc9b93e640 1 -- 172.21.15.88:0/2978081205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc94153770 msgr2=0x7efc94159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:20.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.633+0000 7efc9b93e640 1 --2- 172.21.15.88:0/2978081205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc94153770 0x7efc94159ed0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efc900097a0 tx=0x7efc9001c460 comp rx=0 tx=0).stop 2023-10-16T20:39:20.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.633+0000 7efc9b93e640 1 -- 172.21.15.88:0/2978081205 shutdown_connections 2023-10-16T20:39:20.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.633+0000 7efc9b93e640 1 --2- 172.21.15.88:0/2978081205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc94153770 0x7efc94159ed0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.633+0000 7efc9b93e640 1 -- 172.21.15.88:0/2978081205 >> 172.21.15.88:0/2978081205 conn(0x7efc940715c0 msgr2=0x7efc940719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:20.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.633+0000 7efc9b93e640 1 -- 172.21.15.88:0/2978081205 shutdown_connections 2023-10-16T20:39:20.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.633+0000 7efc9b93e640 1 -- 172.21.15.88:0/2978081205 wait complete. 2023-10-16T20:39:20.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc9b93e640 1 Processor -- start 2023-10-16T20:39:20.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc9b93e640 1 -- start start 2023-10-16T20:39:20.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc9b93e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc940c9bb0 0x7efc9413d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:20.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc9b93e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc940cece0 0x7efc9413f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:20.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc9b93e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc94153770 0x7efc9413f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:20.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc9b93e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9413e330 con 0x7efc940c9bb0 2023-10-16T20:39:20.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc9b93e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc941427b0 con 0x7efc940cece0 2023-10-16T20:39:20.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc9b93e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9413dc10 con 0x7efc94153770 2023-10-16T20:39:20.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc996da640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc940c9bb0 0x7efc9413d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:20.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc996da640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc940c9bb0 0x7efc9413d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56252/0 (socket says 172.21.15.88:56252) 2023-10-16T20:39:20.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc996da640 1 -- 172.21.15.88:0/1109206205 learned_addr learned my addr 172.21.15.88:0/1109206205 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:20.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc98ed9640 1 --2- 172.21.15.88:0/1109206205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc940cece0 0x7efc9413f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:20.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc99edb640 1 --2- 172.21.15.88:0/1109206205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc94153770 0x7efc9413f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:20.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc996da640 1 -- 172.21.15.88:0/1109206205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc94153770 msgr2=0x7efc9413f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:20.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc996da640 1 --2- 172.21.15.88:0/1109206205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc94153770 0x7efc9413f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc996da640 1 -- 172.21.15.88:0/1109206205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc940cece0 msgr2=0x7efc9413f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:20.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc996da640 1 --2- 172.21.15.88:0/1109206205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc940cece0 0x7efc9413f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc996da640 1 -- 172.21.15.88:0/1109206205 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc900093b0 con 0x7efc940c9bb0 2023-10-16T20:39:20.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc98ed9640 1 --2- 172.21.15.88:0/1109206205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc940cece0 0x7efc9413f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:20.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc996da640 1 --2- 172.21.15.88:0/1109206205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc940c9bb0 0x7efc9413d3b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efc9001a090 tx=0x7efc90019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:20.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc89ffb640 1 -- 172.21.15.88:0/1109206205 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc90004030 con 0x7efc940c9bb0 2023-10-16T20:39:20.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc89ffb640 1 -- 172.21.15.88:0/1109206205 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc90004190 con 0x7efc940c9bb0 2023-10-16T20:39:20.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc89ffb640 1 -- 172.21.15.88:0/1109206205 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc90004500 con 0x7efc940c9bb0 2023-10-16T20:39:20.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc9b93e640 1 -- 172.21.15.88:0/1109206205 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc940b3bb0 con 0x7efc940c9bb0 2023-10-16T20:39:20.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc9b93e640 1 -- 172.21.15.88:0/1109206205 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc940b4170 con 0x7efc940c9bb0 2023-10-16T20:39:20.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.637+0000 7efc9b93e640 1 -- 172.21.15.88:0/1109206205 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc94074700 con 0x7efc940c9bb0 2023-10-16T20:39:20.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.641+0000 7efc89ffb640 1 -- 172.21.15.88:0/1109206205 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efc900b2040 con 0x7efc940c9bb0 2023-10-16T20:39:20.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.641+0000 7efc89ffb640 1 --2- 172.21.15.88:0/1109206205 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efc780664a0 0x7efc78068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:20.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.641+0000 7efc89ffb640 1 -- 172.21.15.88:0/1109206205 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7efc9011d430 con 0x7efc940c9bb0 2023-10-16T20:39:20.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.641+0000 7efc98ed9640 1 --2- 172.21.15.88:0/1109206205 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efc780664a0 0x7efc78068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:20.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.641+0000 7efc98ed9640 1 --2- 172.21.15.88:0/1109206205 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efc780664a0 0x7efc78068960 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efc94142310 tx=0x7efc84007c60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:20.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.645+0000 7efc89ffb640 1 -- 172.21.15.88:0/1109206205 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efc900ea4f0 con 0x7efc940c9bb0 2023-10-16T20:39:20.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.793+0000 7efc9b93e640 1 -- 172.21.15.88:0/1109206205 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efc940cbe70 con 0x7efc780664a0 2023-10-16T20:39:20.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.801+0000 7efc89ffb640 1 -- 172.21.15.88:0/1109206205 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+369665 (secure 0 0 0) 0x7efc84007270 con 0x7efc780664a0 2023-10-16T20:39:20.813 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:20.815 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-16T20:39:20.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.805+0000 7efc9b93e640 1 -- 172.21.15.88:0/1109206205 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efc780664a0 msgr2=0x7efc78068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:20.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.805+0000 7efc9b93e640 1 --2- 172.21.15.88:0/1109206205 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efc780664a0 0x7efc78068960 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efc94142310 tx=0x7efc84007c60 comp rx=0 tx=0).stop 2023-10-16T20:39:20.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.805+0000 7efc9b93e640 1 -- 172.21.15.88:0/1109206205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc940c9bb0 msgr2=0x7efc9413d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:20.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.805+0000 7efc9b93e640 1 --2- 172.21.15.88:0/1109206205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc940c9bb0 0x7efc9413d3b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efc9001a090 tx=0x7efc90019200 comp rx=0 tx=0).stop 2023-10-16T20:39:20.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.805+0000 7efc9b93e640 1 -- 172.21.15.88:0/1109206205 shutdown_connections 2023-10-16T20:39:20.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.805+0000 7efc9b93e640 1 --2- 172.21.15.88:0/1109206205 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efc780664a0 0x7efc78068960 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.805+0000 7efc9b93e640 1 --2- 172.21.15.88:0/1109206205 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efc94153770 0x7efc9413f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.805+0000 7efc9b93e640 1 --2- 172.21.15.88:0/1109206205 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efc940cece0 0x7efc9413f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.805+0000 7efc9b93e640 1 --2- 172.21.15.88:0/1109206205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc940c9bb0 0x7efc9413d3b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:20.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.805+0000 7efc9b93e640 1 -- 172.21.15.88:0/1109206205 >> 172.21.15.88:0/1109206205 conn(0x7efc940715c0 msgr2=0x7efc94152460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:20.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.805+0000 7efc9b93e640 1 -- 172.21.15.88:0/1109206205 shutdown_connections 2023-10-16T20:39:20.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:20.805+0000 7efc9b93e640 1 -- 172.21.15.88:0/1109206205 wait complete. 2023-10-16T20:39:20.832 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2023-10-16T20:39:19.627547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595007,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219792,"kb_used_data":3624,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754928,"statfs":{"total":773094113280,"available":772869046272,"internally_reserved":0,"allocated":3710976,"data_stored":2183974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2229,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002567"},"pg_stats":[{"pgid":"2.1f","version":"26'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435615+0000","last_change":"2023-10-16T20:36:54.763744+0000","last_active":"2023-10-16T20:37:37.435615+0000","last_peered":"2023-10-16T20:37:37.435615+0000","last_clean":"2023-10-16T20:37:37.435615+0000","last_became_active":"2023-10-16T20:36:54.763457+0000","last_became_peered":"2023-10-16T20:36:54.763457+0000","last_unstale":"2023-10-16T20:37:37.435615+0000","last_undegraded":"2023-10-16T20:37:37.435615+0000","last_fullsized":"2023-10-16T20:37:37.435615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:05.774485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465462+0000","last_change":"2023-10-16T20:36:55.302740+0000","last_active":"2023-10-16T20:37:37.465462+0000","last_peered":"2023-10-16T20:37:37.465462+0000","last_clean":"2023-10-16T20:37:37.465462+0000","last_became_active":"2023-10-16T20:36:55.302519+0000","last_became_peered":"2023-10-16T20:36:55.302519+0000","last_unstale":"2023-10-16T20:37:37.465462+0000","last_undegraded":"2023-10-16T20:37:37.465462+0000","last_fullsized":"2023-10-16T20:37:37.465462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:36.762521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404215+0000","last_change":"2023-10-16T20:36:59.466679+0000","last_active":"2023-10-16T20:37:36.404215+0000","last_peered":"2023-10-16T20:37:36.404215+0000","last_clean":"2023-10-16T20:37:36.404215+0000","last_became_active":"2023-10-16T20:36:59.466306+0000","last_became_peered":"2023-10-16T20:36:59.466306+0000","last_unstale":"2023-10-16T20:37:36.404215+0000","last_undegraded":"2023-10-16T20:37:36.404215+0000","last_fullsized":"2023-10-16T20:37:36.404215+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:14.874622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"26'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465585+0000","last_change":"2023-10-16T20:36:54.759491+0000","last_active":"2023-10-16T20:37:37.465585+0000","last_peered":"2023-10-16T20:37:37.465585+0000","last_clean":"2023-10-16T20:37:37.465585+0000","last_became_active":"2023-10-16T20:36:54.759177+0000","last_became_peered":"2023-10-16T20:36:54.759177+0000","last_unstale":"2023-10-16T20:37:37.465585+0000","last_undegraded":"2023-10-16T20:37:37.465585+0000","last_fullsized":"2023-10-16T20:37:37.465585+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:43:36.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435464+0000","last_change":"2023-10-16T20:36:55.291745+0000","last_active":"2023-10-16T20:37:37.435464+0000","last_peered":"2023-10-16T20:37:37.435464+0000","last_clean":"2023-10-16T20:37:37.435464+0000","last_became_active":"2023-10-16T20:36:55.291458+0000","last_became_peered":"2023-10-16T20:36:55.291458+0000","last_unstale":"2023-10-16T20:37:37.435464+0000","last_undegraded":"2023-10-16T20:37:37.435464+0000","last_fullsized":"2023-10-16T20:37:37.435464+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:01.291335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.401937+0000","last_change":"2023-10-16T20:36:59.303600+0000","last_active":"2023-10-16T20:37:36.401937+0000","last_peered":"2023-10-16T20:37:36.401937+0000","last_clean":"2023-10-16T20:37:36.401937+0000","last_became_active":"2023-10-16T20:36:59.302968+0000","last_became_peered":"2023-10-16T20:36:59.302968+0000","last_unstale":"2023-10-16T20:37:36.401937+0000","last_undegraded":"2023-10-16T20:37:36.401937+0000","last_fullsized":"2023-10-16T20:37:36.401937+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:11:28.322824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"27'2","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.412675+0000","last_change":"2023-10-16T20:36:54.762600+0000","last_active":"2023-10-16T20:37:36.412675+0000","last_peered":"2023-10-16T20:37:36.412675+0000","last_clean":"2023-10-16T20:37:36.412675+0000","last_became_active":"2023-10-16T20:36:54.762278+0000","last_became_peered":"2023-10-16T20:36:54.762278+0000","last_unstale":"2023-10-16T20:37:36.412675+0000","last_undegraded":"2023-10-16T20:37:36.412675+0000","last_fullsized":"2023-10-16T20:37:36.412675+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:46:03.836415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404637+0000","last_change":"2023-10-16T20:37:37.403448+0000","last_active":"2023-10-16T20:37:37.404637+0000","last_peered":"2023-10-16T20:37:37.404637+0000","last_clean":"2023-10-16T20:37:37.404637+0000","last_became_active":"2023-10-16T20:37:37.402987+0000","last_became_peered":"2023-10-16T20:37:37.402987+0000","last_unstale":"2023-10-16T20:37:37.404637+0000","last_undegraded":"2023-10-16T20:37:37.404637+0000","last_fullsized":"2023-10-16T20:37:37.404637+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:39:05.729682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.402113+0000","last_change":"2023-10-16T20:36:59.304684+0000","last_active":"2023-10-16T20:37:36.402113+0000","last_peered":"2023-10-16T20:37:36.402113+0000","last_clean":"2023-10-16T20:37:36.402113+0000","last_became_active":"2023-10-16T20:36:59.303805+0000","last_became_peered":"2023-10-16T20:36:59.303805+0000","last_unstale":"2023-10-16T20:37:36.402113+0000","last_undegraded":"2023-10-16T20:37:36.402113+0000","last_fullsized":"2023-10-16T20:37:36.402113+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:39:54.406763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"27'2","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.412121+0000","last_change":"2023-10-16T20:36:54.749849+0000","last_active":"2023-10-16T20:37:36.412121+0000","last_peered":"2023-10-16T20:37:36.412121+0000","last_clean":"2023-10-16T20:37:36.412121+0000","last_became_active":"2023-10-16T20:36:54.749316+0000","last_became_peered":"2023-10-16T20:36:54.749316+0000","last_unstale":"2023-10-16T20:37:36.412121+0000","last_undegraded":"2023-10-16T20:37:36.412121+0000","last_fullsized":"2023-10-16T20:37:36.412121+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:49:29.077126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.409911+0000","last_change":"2023-10-16T20:36:55.302838+0000","last_active":"2023-10-16T20:37:36.409911+0000","last_peered":"2023-10-16T20:37:36.409911+0000","last_clean":"2023-10-16T20:37:36.409911+0000","last_became_active":"2023-10-16T20:36:55.302387+0000","last_became_peered":"2023-10-16T20:36:55.302387+0000","last_unstale":"2023-10-16T20:37:36.409911+0000","last_undegraded":"2023-10-16T20:37:36.409911+0000","last_fullsized":"2023-10-16T20:37:36.409911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:55:56.517950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.402135+0000","last_change":"2023-10-16T20:36:59.305750+0000","last_active":"2023-10-16T20:37:36.402135+0000","last_peered":"2023-10-16T20:37:36.402135+0000","last_clean":"2023-10-16T20:37:36.402135+0000","last_became_active":"2023-10-16T20:36:59.305228+0000","last_became_peered":"2023-10-16T20:36:59.305228+0000","last_unstale":"2023-10-16T20:37:36.402135+0000","last_undegraded":"2023-10-16T20:37:36.402135+0000","last_fullsized":"2023-10-16T20:37:36.402135+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:00:18.194057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407634+0000","last_change":"2023-10-16T20:36:55.306712+0000","last_active":"2023-10-16T20:37:37.407634+0000","last_peered":"2023-10-16T20:37:37.407634+0000","last_clean":"2023-10-16T20:37:37.407634+0000","last_became_active":"2023-10-16T20:36:55.306427+0000","last_became_peered":"2023-10-16T20:36:55.306427+0000","last_unstale":"2023-10-16T20:37:37.407634+0000","last_undegraded":"2023-10-16T20:37:37.407634+0000","last_fullsized":"2023-10-16T20:37:37.407634+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:42:03.769890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"27'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466293+0000","last_change":"2023-10-16T20:36:54.768375+0000","last_active":"2023-10-16T20:37:37.466293+0000","last_peered":"2023-10-16T20:37:37.466293+0000","last_clean":"2023-10-16T20:37:37.466293+0000","last_became_active":"2023-10-16T20:36:54.767996+0000","last_became_peered":"2023-10-16T20:36:54.767996+0000","last_unstale":"2023-10-16T20:37:37.466293+0000","last_undegraded":"2023-10-16T20:37:37.466293+0000","last_fullsized":"2023-10-16T20:37:37.466293+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:55.303942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404253+0000","last_change":"2023-10-16T20:36:59.307130+0000","last_active":"2023-10-16T20:37:37.404253+0000","last_peered":"2023-10-16T20:37:37.404253+0000","last_clean":"2023-10-16T20:37:37.404253+0000","last_became_active":"2023-10-16T20:36:59.306640+0000","last_became_peered":"2023-10-16T20:36:59.306640+0000","last_unstale":"2023-10-16T20:37:37.404253+0000","last_undegraded":"2023-10-16T20:37:37.404253+0000","last_fullsized":"2023-10-16T20:37:37.404253+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:19:18.664630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465074+0000","last_change":"2023-10-16T20:36:55.275436+0000","last_active":"2023-10-16T20:37:37.465074+0000","last_peered":"2023-10-16T20:37:37.465074+0000","last_clean":"2023-10-16T20:37:37.465074+0000","last_became_active":"2023-10-16T20:36:55.275085+0000","last_became_peered":"2023-10-16T20:36:55.275085+0000","last_unstale":"2023-10-16T20:37:37.465074+0000","last_undegraded":"2023-10-16T20:37:37.465074+0000","last_fullsized":"2023-10-16T20:37:37.465074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:37:37.814511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402406+0000","last_change":"2023-10-16T20:36:54.756174+0000","last_active":"2023-10-16T20:37:37.402406+0000","last_peered":"2023-10-16T20:37:37.402406+0000","last_clean":"2023-10-16T20:37:37.402406+0000","last_became_active":"2023-10-16T20:36:54.755872+0000","last_became_peered":"2023-10-16T20:36:54.755872+0000","last_unstale":"2023-10-16T20:37:37.402406+0000","last_undegraded":"2023-10-16T20:37:37.402406+0000","last_fullsized":"2023-10-16T20:37:37.402406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:32.741671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403750+0000","last_change":"2023-10-16T20:36:59.303855+0000","last_active":"2023-10-16T20:37:37.403750+0000","last_peered":"2023-10-16T20:37:37.403750+0000","last_clean":"2023-10-16T20:37:37.403750+0000","last_became_active":"2023-10-16T20:36:59.302919+0000","last_became_peered":"2023-10-16T20:36:59.302919+0000","last_unstale":"2023-10-16T20:37:37.403750+0000","last_undegraded":"2023-10-16T20:37:37.403750+0000","last_fullsized":"2023-10-16T20:37:37.403750+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:38:02.591237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464820+0000","last_change":"2023-10-16T20:36:55.270717+0000","last_active":"2023-10-16T20:37:37.464820+0000","last_peered":"2023-10-16T20:37:37.464820+0000","last_clean":"2023-10-16T20:37:37.464820+0000","last_became_active":"2023-10-16T20:36:55.270455+0000","last_became_peered":"2023-10-16T20:36:55.270455+0000","last_unstale":"2023-10-16T20:37:37.464820+0000","last_undegraded":"2023-10-16T20:37:37.464820+0000","last_fullsized":"2023-10-16T20:37:37.464820+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:16:44.485783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401912+0000","last_change":"2023-10-16T20:36:54.744128+0000","last_active":"2023-10-16T20:37:37.401912+0000","last_peered":"2023-10-16T20:37:37.401912+0000","last_clean":"2023-10-16T20:37:37.401912+0000","last_became_active":"2023-10-16T20:36:54.743805+0000","last_became_peered":"2023-10-16T20:36:54.743805+0000","last_unstale":"2023-10-16T20:37:37.401912+0000","last_undegraded":"2023-10-16T20:37:37.401912+0000","last_fullsized":"2023-10-16T20:37:37.401912+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:54:35.819820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402668+0000","last_change":"2023-10-16T20:36:59.465650+0000","last_active":"2023-10-16T20:37:37.402668+0000","last_peered":"2023-10-16T20:37:37.402668+0000","last_clean":"2023-10-16T20:37:37.402668+0000","last_became_active":"2023-10-16T20:36:59.465172+0000","last_became_peered":"2023-10-16T20:36:59.465172+0000","last_unstale":"2023-10-16T20:37:37.402668+0000","last_undegraded":"2023-10-16T20:37:37.402668+0000","last_fullsized":"2023-10-16T20:37:37.402668+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:12.845517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403561+0000","last_change":"2023-10-16T20:36:55.278974+0000","last_active":"2023-10-16T20:37:37.403561+0000","last_peered":"2023-10-16T20:37:37.403561+0000","last_clean":"2023-10-16T20:37:37.403561+0000","last_became_active":"2023-10-16T20:36:55.278568+0000","last_became_peered":"2023-10-16T20:36:55.278568+0000","last_unstale":"2023-10-16T20:37:37.403561+0000","last_undegraded":"2023-10-16T20:37:37.403561+0000","last_fullsized":"2023-10-16T20:37:37.403561+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:26.756114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465927+0000","last_change":"2023-10-16T20:36:54.758778+0000","last_active":"2023-10-16T20:37:37.465927+0000","last_peered":"2023-10-16T20:37:37.465927+0000","last_clean":"2023-10-16T20:37:37.465927+0000","last_became_active":"2023-10-16T20:36:54.758496+0000","last_became_peered":"2023-10-16T20:36:54.758496+0000","last_unstale":"2023-10-16T20:37:37.465927+0000","last_undegraded":"2023-10-16T20:37:37.465927+0000","last_fullsized":"2023-10-16T20:37:37.465927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:36:00.515106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404277+0000","last_change":"2023-10-16T20:36:59.309382+0000","last_active":"2023-10-16T20:37:37.404277+0000","last_peered":"2023-10-16T20:37:37.404277+0000","last_clean":"2023-10-16T20:37:37.404277+0000","last_became_active":"2023-10-16T20:36:59.309068+0000","last_became_peered":"2023-10-16T20:36:59.309068+0000","last_unstale":"2023-10-16T20:37:37.404277+0000","last_undegraded":"2023-10-16T20:37:37.404277+0000","last_fullsized":"2023-10-16T20:37:37.404277+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:46:02.601985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435429+0000","last_change":"2023-10-16T20:36:55.273089+0000","last_active":"2023-10-16T20:37:37.435429+0000","last_peered":"2023-10-16T20:37:37.435429+0000","last_clean":"2023-10-16T20:37:37.435429+0000","last_became_active":"2023-10-16T20:36:55.272840+0000","last_became_peered":"2023-10-16T20:36:55.272840+0000","last_unstale":"2023-10-16T20:37:37.435429+0000","last_undegraded":"2023-10-16T20:37:37.435429+0000","last_fullsized":"2023-10-16T20:37:37.435429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:37.330924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"27'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408542+0000","last_change":"2023-10-16T20:36:54.760681+0000","last_active":"2023-10-16T20:37:37.408542+0000","last_peered":"2023-10-16T20:37:37.408542+0000","last_clean":"2023-10-16T20:37:37.408542+0000","last_became_active":"2023-10-16T20:36:54.760397+0000","last_became_peered":"2023-10-16T20:36:54.760397+0000","last_unstale":"2023-10-16T20:37:37.408542+0000","last_undegraded":"2023-10-16T20:37:37.408542+0000","last_fullsized":"2023-10-16T20:37:37.408542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:47:16.510382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403318+0000","last_change":"2023-10-16T20:36:59.465626+0000","last_active":"2023-10-16T20:37:37.403318+0000","last_peered":"2023-10-16T20:37:37.403318+0000","last_clean":"2023-10-16T20:37:37.403318+0000","last_became_active":"2023-10-16T20:36:59.465152+0000","last_became_peered":"2023-10-16T20:36:59.465152+0000","last_unstale":"2023-10-16T20:37:37.403318+0000","last_undegraded":"2023-10-16T20:37:37.403318+0000","last_fullsized":"2023-10-16T20:37:37.403318+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:27:51.088333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465673+0000","last_change":"2023-10-16T20:36:55.285796+0000","last_active":"2023-10-16T20:37:37.465673+0000","last_peered":"2023-10-16T20:37:37.465673+0000","last_clean":"2023-10-16T20:37:37.465673+0000","last_became_active":"2023-10-16T20:36:55.285566+0000","last_became_peered":"2023-10-16T20:36:55.285566+0000","last_unstale":"2023-10-16T20:37:37.465673+0000","last_undegraded":"2023-10-16T20:37:37.465673+0000","last_fullsized":"2023-10-16T20:37:37.465673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:22:21.950705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"27'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402756+0000","last_change":"2023-10-16T20:36:54.758062+0000","last_active":"2023-10-16T20:37:37.402756+0000","last_peered":"2023-10-16T20:37:37.402756+0000","last_clean":"2023-10-16T20:37:37.402756+0000","last_became_active":"2023-10-16T20:36:54.757796+0000","last_became_peered":"2023-10-16T20:36:54.757796+0000","last_unstale":"2023-10-16T20:37:37.402756+0000","last_undegraded":"2023-10-16T20:37:37.402756+0000","last_fullsized":"2023-10-16T20:37:37.402756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:02:09.953795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464463+0000","last_change":"2023-10-16T20:36:59.468529+0000","last_active":"2023-10-16T20:37:37.464463+0000","last_peered":"2023-10-16T20:37:37.464463+0000","last_clean":"2023-10-16T20:37:37.464463+0000","last_became_active":"2023-10-16T20:36:59.465221+0000","last_became_peered":"2023-10-16T20:36:59.465221+0000","last_unstale":"2023-10-16T20:37:37.464463+0000","last_undegraded":"2023-10-16T20:37:37.464463+0000","last_fullsized":"2023-10-16T20:37:37.464463+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:58:08.457686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401257+0000","last_change":"2023-10-16T20:36:55.275759+0000","last_active":"2023-10-16T20:37:37.401257+0000","last_peered":"2023-10-16T20:37:37.401257+0000","last_clean":"2023-10-16T20:37:37.401257+0000","last_became_active":"2023-10-16T20:36:55.274432+0000","last_became_peered":"2023-10-16T20:36:55.274432+0000","last_unstale":"2023-10-16T20:37:37.401257+0000","last_undegraded":"2023-10-16T20:37:37.401257+0000","last_fullsized":"2023-10-16T20:37:37.401257+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:53:08.896731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466573+0000","last_change":"2023-10-16T20:36:54.762356+0000","last_active":"2023-10-16T20:37:37.466573+0000","last_peered":"2023-10-16T20:37:37.466573+0000","last_clean":"2023-10-16T20:37:37.466573+0000","last_became_active":"2023-10-16T20:36:54.762167+0000","last_became_peered":"2023-10-16T20:36:54.762167+0000","last_unstale":"2023-10-16T20:37:37.466573+0000","last_undegraded":"2023-10-16T20:37:37.466573+0000","last_fullsized":"2023-10-16T20:37:37.466573+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:45.164625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434946+0000","last_change":"2023-10-16T20:36:59.467492+0000","last_active":"2023-10-16T20:37:37.434946+0000","last_peered":"2023-10-16T20:37:37.434946+0000","last_clean":"2023-10-16T20:37:37.434946+0000","last_became_active":"2023-10-16T20:36:59.466876+0000","last_became_peered":"2023-10-16T20:36:59.466876+0000","last_unstale":"2023-10-16T20:37:37.434946+0000","last_undegraded":"2023-10-16T20:37:37.434946+0000","last_fullsized":"2023-10-16T20:37:37.434946+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:29:01.292167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409570+0000","last_change":"2023-10-16T20:36:55.293537+0000","last_active":"2023-10-16T20:37:37.409570+0000","last_peered":"2023-10-16T20:37:37.409570+0000","last_clean":"2023-10-16T20:37:37.409570+0000","last_became_active":"2023-10-16T20:36:55.293228+0000","last_became_peered":"2023-10-16T20:36:55.293228+0000","last_unstale":"2023-10-16T20:37:37.409570+0000","last_undegraded":"2023-10-16T20:37:37.409570+0000","last_fullsized":"2023-10-16T20:37:37.409570+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:50:13.226645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401381+0000","last_change":"2023-10-16T20:36:54.757588+0000","last_active":"2023-10-16T20:37:37.401381+0000","last_peered":"2023-10-16T20:37:37.401381+0000","last_clean":"2023-10-16T20:37:37.401381+0000","last_became_active":"2023-10-16T20:36:54.757308+0000","last_became_peered":"2023-10-16T20:36:54.757308+0000","last_unstale":"2023-10-16T20:37:37.401381+0000","last_undegraded":"2023-10-16T20:37:37.401381+0000","last_fullsized":"2023-10-16T20:37:37.401381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:51.298239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402099+0000","last_change":"2023-10-16T20:36:59.303844+0000","last_active":"2023-10-16T20:37:37.402099+0000","last_peered":"2023-10-16T20:37:37.402099+0000","last_clean":"2023-10-16T20:37:37.402099+0000","last_became_active":"2023-10-16T20:36:59.303224+0000","last_became_peered":"2023-10-16T20:36:59.303224+0000","last_unstale":"2023-10-16T20:37:37.402099+0000","last_undegraded":"2023-10-16T20:37:37.402099+0000","last_fullsized":"2023-10-16T20:37:37.402099+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:52.447785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403798+0000","last_change":"2023-10-16T20:36:55.278310+0000","last_active":"2023-10-16T20:37:37.403798+0000","last_peered":"2023-10-16T20:37:37.403798+0000","last_clean":"2023-10-16T20:37:37.403798+0000","last_became_active":"2023-10-16T20:36:55.277872+0000","last_became_peered":"2023-10-16T20:36:55.277872+0000","last_unstale":"2023-10-16T20:37:37.403798+0000","last_undegraded":"2023-10-16T20:37:37.403798+0000","last_fullsized":"2023-10-16T20:37:37.403798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:37.320469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408898+0000","last_change":"2023-10-16T20:36:54.740644+0000","last_active":"2023-10-16T20:37:37.408898+0000","last_peered":"2023-10-16T20:37:37.408898+0000","last_clean":"2023-10-16T20:37:37.408898+0000","last_became_active":"2023-10-16T20:36:54.740340+0000","last_became_peered":"2023-10-16T20:36:54.740340+0000","last_unstale":"2023-10-16T20:37:37.408898+0000","last_undegraded":"2023-10-16T20:37:37.408898+0000","last_fullsized":"2023-10-16T20:37:37.408898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:22.221727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401155+0000","last_change":"2023-10-16T20:36:59.294914+0000","last_active":"2023-10-16T20:37:37.401155+0000","last_peered":"2023-10-16T20:37:37.401155+0000","last_clean":"2023-10-16T20:37:37.401155+0000","last_became_active":"2023-10-16T20:36:59.294548+0000","last_became_peered":"2023-10-16T20:36:59.294548+0000","last_unstale":"2023-10-16T20:37:37.401155+0000","last_undegraded":"2023-10-16T20:37:37.401155+0000","last_fullsized":"2023-10-16T20:37:37.401155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:52:15.265262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401383+0000","last_change":"2023-10-16T20:36:55.270992+0000","last_active":"2023-10-16T20:37:37.401383+0000","last_peered":"2023-10-16T20:37:37.401383+0000","last_clean":"2023-10-16T20:37:37.401383+0000","last_became_active":"2023-10-16T20:36:55.269749+0000","last_became_peered":"2023-10-16T20:36:55.269749+0000","last_unstale":"2023-10-16T20:37:37.401383+0000","last_undegraded":"2023-10-16T20:37:37.401383+0000","last_fullsized":"2023-10-16T20:37:37.401383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:30.925314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.399629+0000","last_change":"2023-10-16T20:36:54.757903+0000","last_active":"2023-10-16T20:37:37.399629+0000","last_peered":"2023-10-16T20:37:37.399629+0000","last_clean":"2023-10-16T20:37:37.399629+0000","last_became_active":"2023-10-16T20:36:54.757549+0000","last_became_peered":"2023-10-16T20:36:54.757549+0000","last_unstale":"2023-10-16T20:37:37.399629+0000","last_undegraded":"2023-10-16T20:37:37.399629+0000","last_fullsized":"2023-10-16T20:37:37.399629+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:02:42.090848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402786+0000","last_change":"2023-10-16T20:36:59.302063+0000","last_active":"2023-10-16T20:37:37.402786+0000","last_peered":"2023-10-16T20:37:37.402786+0000","last_clean":"2023-10-16T20:37:37.402786+0000","last_became_active":"2023-10-16T20:36:59.301693+0000","last_became_peered":"2023-10-16T20:36:59.301693+0000","last_unstale":"2023-10-16T20:37:37.402786+0000","last_undegraded":"2023-10-16T20:37:37.402786+0000","last_fullsized":"2023-10-16T20:37:37.402786+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:29:44.812282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434713+0000","last_change":"2023-10-16T20:36:55.278132+0000","last_active":"2023-10-16T20:37:37.434713+0000","last_peered":"2023-10-16T20:37:37.434713+0000","last_clean":"2023-10-16T20:37:37.434713+0000","last_became_active":"2023-10-16T20:36:55.276649+0000","last_became_peered":"2023-10-16T20:36:55.276649+0000","last_unstale":"2023-10-16T20:37:37.434713+0000","last_undegraded":"2023-10-16T20:37:37.434713+0000","last_fullsized":"2023-10-16T20:37:37.434713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:30:53.299514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465960+0000","last_change":"2023-10-16T20:36:54.753592+0000","last_active":"2023-10-16T20:37:37.465960+0000","last_peered":"2023-10-16T20:37:37.465960+0000","last_clean":"2023-10-16T20:37:37.465960+0000","last_became_active":"2023-10-16T20:36:54.753237+0000","last_became_peered":"2023-10-16T20:36:54.753237+0000","last_unstale":"2023-10-16T20:37:37.465960+0000","last_undegraded":"2023-10-16T20:37:37.465960+0000","last_fullsized":"2023-10-16T20:37:37.465960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:53.005979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435360+0000","last_change":"2023-10-16T20:36:59.468431+0000","last_active":"2023-10-16T20:37:37.435360+0000","last_peered":"2023-10-16T20:37:37.435360+0000","last_clean":"2023-10-16T20:37:37.435360+0000","last_became_active":"2023-10-16T20:36:59.466708+0000","last_became_peered":"2023-10-16T20:36:59.466708+0000","last_unstale":"2023-10-16T20:37:37.435360+0000","last_undegraded":"2023-10-16T20:37:37.435360+0000","last_fullsized":"2023-10-16T20:37:37.435360+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:42:20.607519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464817+0000","last_change":"2023-10-16T20:36:55.294825+0000","last_active":"2023-10-16T20:37:37.464817+0000","last_peered":"2023-10-16T20:37:37.464817+0000","last_clean":"2023-10-16T20:37:37.464817+0000","last_became_active":"2023-10-16T20:36:55.294610+0000","last_became_peered":"2023-10-16T20:36:55.294610+0000","last_unstale":"2023-10-16T20:37:37.464817+0000","last_undegraded":"2023-10-16T20:37:37.464817+0000","last_fullsized":"2023-10-16T20:37:37.464817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:43.442939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408668+0000","last_change":"2023-10-16T20:36:54.741191+0000","last_active":"2023-10-16T20:37:37.408668+0000","last_peered":"2023-10-16T20:37:37.408668+0000","last_clean":"2023-10-16T20:37:37.408668+0000","last_became_active":"2023-10-16T20:36:54.740944+0000","last_became_peered":"2023-10-16T20:36:54.740944+0000","last_unstale":"2023-10-16T20:37:37.408668+0000","last_undegraded":"2023-10-16T20:37:37.408668+0000","last_fullsized":"2023-10-16T20:37:37.408668+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:05.961850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":172,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466038+0000","last_change":"2023-10-16T20:36:46.390001+0000","last_active":"2023-10-16T20:37:37.466038+0000","last_peered":"2023-10-16T20:37:37.466038+0000","last_clean":"2023-10-16T20:37:37.466038+0000","last_became_active":"2023-10-16T20:36:46.388824+0000","last_became_peered":"2023-10-16T20:36:46.388824+0000","last_unstale":"2023-10-16T20:37:37.466038+0000","last_undegraded":"2023-10-16T20:37:37.466038+0000","last_fullsized":"2023-10-16T20:37:37.466038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_clean_scrub_stamp":"2023-10-16T20:36:44.575506+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:31.004318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407653+0000","last_change":"2023-10-16T20:36:59.468528+0000","last_active":"2023-10-16T20:37:37.407653+0000","last_peered":"2023-10-16T20:37:37.407653+0000","last_clean":"2023-10-16T20:37:37.407653+0000","last_became_active":"2023-10-16T20:36:59.468082+0000","last_became_peered":"2023-10-16T20:36:59.468082+0000","last_unstale":"2023-10-16T20:37:37.407653+0000","last_undegraded":"2023-10-16T20:37:37.407653+0000","last_fullsized":"2023-10-16T20:37:37.407653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:00.920004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410690+0000","last_change":"2023-10-16T20:37:37.409578+0000","last_active":"2023-10-16T20:37:37.410690+0000","last_peered":"2023-10-16T20:37:37.410690+0000","last_clean":"2023-10-16T20:37:37.410690+0000","last_became_active":"2023-10-16T20:37:37.409004+0000","last_became_peered":"2023-10-16T20:37:37.409004+0000","last_unstale":"2023-10-16T20:37:37.410690+0000","last_undegraded":"2023-10-16T20:37:37.410690+0000","last_fullsized":"2023-10-16T20:37:37.410690+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:28:57.364794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.400188+0000","last_change":"2023-10-16T20:36:54.746600+0000","last_active":"2023-10-16T20:37:37.400188+0000","last_peered":"2023-10-16T20:37:37.400188+0000","last_clean":"2023-10-16T20:37:37.400188+0000","last_became_active":"2023-10-16T20:36:54.746247+0000","last_became_peered":"2023-10-16T20:36:54.746247+0000","last_unstale":"2023-10-16T20:37:37.400188+0000","last_undegraded":"2023-10-16T20:37:37.400188+0000","last_fullsized":"2023-10-16T20:37:37.400188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:28:57.949487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.470531+0000","last_change":"2023-10-16T20:37:37.469427+0000","last_active":"2023-10-16T20:37:37.470531+0000","last_peered":"2023-10-16T20:37:37.470531+0000","last_clean":"2023-10-16T20:37:37.470531+0000","last_became_active":"2023-10-16T20:37:37.467402+0000","last_became_peered":"2023-10-16T20:37:37.467402+0000","last_unstale":"2023-10-16T20:37:37.470531+0000","last_undegraded":"2023-10-16T20:37:37.470531+0000","last_fullsized":"2023-10-16T20:37:37.470531+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:00:19.171972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409113+0000","last_change":"2023-10-16T20:36:55.301234+0000","last_active":"2023-10-16T20:37:37.409113+0000","last_peered":"2023-10-16T20:37:37.409113+0000","last_clean":"2023-10-16T20:37:37.409113+0000","last_became_active":"2023-10-16T20:36:55.300958+0000","last_became_peered":"2023-10-16T20:36:55.300958+0000","last_unstale":"2023-10-16T20:37:37.409113+0000","last_undegraded":"2023-10-16T20:37:37.409113+0000","last_fullsized":"2023-10-16T20:37:37.409113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:16.784124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"27'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402899+0000","last_change":"2023-10-16T20:36:54.753770+0000","last_active":"2023-10-16T20:37:37.402899+0000","last_peered":"2023-10-16T20:37:37.402899+0000","last_clean":"2023-10-16T20:37:37.402899+0000","last_became_active":"2023-10-16T20:36:54.753443+0000","last_became_peered":"2023-10-16T20:36:54.753443+0000","last_unstale":"2023-10-16T20:37:37.402899+0000","last_undegraded":"2023-10-16T20:37:37.402899+0000","last_fullsized":"2023-10-16T20:37:37.402899+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:21:26.392480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435046+0000","last_change":"2023-10-16T20:36:59.303351+0000","last_active":"2023-10-16T20:37:37.435046+0000","last_peered":"2023-10-16T20:37:37.435046+0000","last_clean":"2023-10-16T20:37:37.435046+0000","last_became_active":"2023-10-16T20:36:59.302869+0000","last_became_peered":"2023-10-16T20:36:59.302869+0000","last_unstale":"2023-10-16T20:37:37.435046+0000","last_undegraded":"2023-10-16T20:37:37.435046+0000","last_fullsized":"2023-10-16T20:37:37.435046+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:26:35.886399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466830+0000","last_change":"2023-10-16T20:36:55.291947+0000","last_active":"2023-10-16T20:37:37.466830+0000","last_peered":"2023-10-16T20:37:37.466830+0000","last_clean":"2023-10-16T20:37:37.466830+0000","last_became_active":"2023-10-16T20:36:55.291633+0000","last_became_peered":"2023-10-16T20:36:55.291633+0000","last_unstale":"2023-10-16T20:37:37.466830+0000","last_undegraded":"2023-10-16T20:37:37.466830+0000","last_fullsized":"2023-10-16T20:37:37.466830+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:00:14.349339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"27'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.400583+0000","last_change":"2023-10-16T20:36:54.748196+0000","last_active":"2023-10-16T20:37:37.400583+0000","last_peered":"2023-10-16T20:37:37.400583+0000","last_clean":"2023-10-16T20:37:37.400583+0000","last_became_active":"2023-10-16T20:36:54.747704+0000","last_became_peered":"2023-10-16T20:36:54.747704+0000","last_unstale":"2023-10-16T20:37:37.400583+0000","last_undegraded":"2023-10-16T20:37:37.400583+0000","last_fullsized":"2023-10-16T20:37:37.400583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:55.471916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410236+0000","last_change":"2023-10-16T20:36:59.470368+0000","last_active":"2023-10-16T20:37:37.410236+0000","last_peered":"2023-10-16T20:37:37.410236+0000","last_clean":"2023-10-16T20:37:37.410236+0000","last_became_active":"2023-10-16T20:36:59.467291+0000","last_became_peered":"2023-10-16T20:36:59.467291+0000","last_unstale":"2023-10-16T20:37:37.410236+0000","last_undegraded":"2023-10-16T20:37:37.410236+0000","last_fullsized":"2023-10-16T20:37:37.410236+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:18.312394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466658+0000","last_change":"2023-10-16T20:36:55.291451+0000","last_active":"2023-10-16T20:37:37.466658+0000","last_peered":"2023-10-16T20:37:37.466658+0000","last_clean":"2023-10-16T20:37:37.466658+0000","last_became_active":"2023-10-16T20:36:55.290998+0000","last_became_peered":"2023-10-16T20:36:55.290998+0000","last_unstale":"2023-10-16T20:37:37.466658+0000","last_undegraded":"2023-10-16T20:37:37.466658+0000","last_fullsized":"2023-10-16T20:37:37.466658+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:05:01.634971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"26'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404148+0000","last_change":"2023-10-16T20:36:54.763818+0000","last_active":"2023-10-16T20:37:37.404148+0000","last_peered":"2023-10-16T20:37:37.404148+0000","last_clean":"2023-10-16T20:37:37.404148+0000","last_became_active":"2023-10-16T20:36:54.761337+0000","last_became_peered":"2023-10-16T20:36:54.761337+0000","last_unstale":"2023-10-16T20:37:37.404148+0000","last_undegraded":"2023-10-16T20:37:37.404148+0000","last_fullsized":"2023-10-16T20:37:37.404148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:03.093189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.470301+0000","last_change":"2023-10-16T20:37:37.468803+0000","last_active":"2023-10-16T20:37:37.470301+0000","last_peered":"2023-10-16T20:37:37.470301+0000","last_clean":"2023-10-16T20:37:37.470301+0000","last_became_active":"2023-10-16T20:37:37.468309+0000","last_became_peered":"2023-10-16T20:37:37.468309+0000","last_unstale":"2023-10-16T20:37:37.470301+0000","last_undegraded":"2023-10-16T20:37:37.470301+0000","last_fullsized":"2023-10-16T20:37:37.470301+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:11.761661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466101+0000","last_change":"2023-10-16T20:36:55.288895+0000","last_active":"2023-10-16T20:37:37.466101+0000","last_peered":"2023-10-16T20:37:37.466101+0000","last_clean":"2023-10-16T20:37:37.466101+0000","last_became_active":"2023-10-16T20:36:55.288354+0000","last_became_peered":"2023-10-16T20:36:55.288354+0000","last_unstale":"2023-10-16T20:37:37.466101+0000","last_undegraded":"2023-10-16T20:37:37.466101+0000","last_fullsized":"2023-10-16T20:37:37.466101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:53:29.192406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"27'8","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.399457+0000","last_change":"2023-10-16T20:36:54.738216+0000","last_active":"2023-10-16T20:37:37.399457+0000","last_peered":"2023-10-16T20:37:37.399457+0000","last_clean":"2023-10-16T20:37:37.399457+0000","last_became_active":"2023-10-16T20:36:54.737876+0000","last_became_peered":"2023-10-16T20:36:54.737876+0000","last_unstale":"2023-10-16T20:37:37.399457+0000","last_undegraded":"2023-10-16T20:37:37.399457+0000","last_fullsized":"2023-10-16T20:37:37.399457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:28.214601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.397491+0000","last_change":"2023-10-16T20:36:59.307215+0000","last_active":"2023-10-16T20:37:36.397491+0000","last_peered":"2023-10-16T20:37:36.397491+0000","last_clean":"2023-10-16T20:37:36.397491+0000","last_became_active":"2023-10-16T20:36:59.306801+0000","last_became_peered":"2023-10-16T20:36:59.306801+0000","last_unstale":"2023-10-16T20:37:36.397491+0000","last_undegraded":"2023-10-16T20:37:36.397491+0000","last_fullsized":"2023-10-16T20:37:36.397491+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:41:42.656134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407054+0000","last_change":"2023-10-16T20:36:55.287714+0000","last_active":"2023-10-16T20:37:37.407054+0000","last_peered":"2023-10-16T20:37:37.407054+0000","last_clean":"2023-10-16T20:37:37.407054+0000","last_became_active":"2023-10-16T20:36:55.287066+0000","last_became_peered":"2023-10-16T20:36:55.287066+0000","last_unstale":"2023-10-16T20:37:37.407054+0000","last_undegraded":"2023-10-16T20:37:37.407054+0000","last_fullsized":"2023-10-16T20:37:37.407054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:45:21.058260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407609+0000","last_change":"2023-10-16T20:36:54.764693+0000","last_active":"2023-10-16T20:37:37.407609+0000","last_peered":"2023-10-16T20:37:37.407609+0000","last_clean":"2023-10-16T20:37:37.407609+0000","last_became_active":"2023-10-16T20:36:54.764417+0000","last_became_peered":"2023-10-16T20:36:54.764417+0000","last_unstale":"2023-10-16T20:37:37.407609+0000","last_undegraded":"2023-10-16T20:37:37.407609+0000","last_fullsized":"2023-10-16T20:37:37.407609+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:56.653149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404514+0000","last_change":"2023-10-16T20:36:59.465145+0000","last_active":"2023-10-16T20:37:36.404514+0000","last_peered":"2023-10-16T20:37:36.404514+0000","last_clean":"2023-10-16T20:37:36.404514+0000","last_became_active":"2023-10-16T20:36:59.464626+0000","last_became_peered":"2023-10-16T20:36:59.464626+0000","last_unstale":"2023-10-16T20:37:36.404514+0000","last_undegraded":"2023-10-16T20:37:36.404514+0000","last_fullsized":"2023-10-16T20:37:36.404514+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:38.928616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466290+0000","last_change":"2023-10-16T20:36:55.291718+0000","last_active":"2023-10-16T20:37:37.466290+0000","last_peered":"2023-10-16T20:37:37.466290+0000","last_clean":"2023-10-16T20:37:37.466290+0000","last_became_active":"2023-10-16T20:36:55.291396+0000","last_became_peered":"2023-10-16T20:36:55.291396+0000","last_unstale":"2023-10-16T20:37:37.466290+0000","last_undegraded":"2023-10-16T20:37:37.466290+0000","last_fullsized":"2023-10-16T20:37:37.466290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:29:49.138380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408792+0000","last_change":"2023-10-16T20:36:54.750814+0000","last_active":"2023-10-16T20:37:37.408792+0000","last_peered":"2023-10-16T20:37:37.408792+0000","last_clean":"2023-10-16T20:37:37.408792+0000","last_became_active":"2023-10-16T20:36:54.750448+0000","last_became_peered":"2023-10-16T20:36:54.750448+0000","last_unstale":"2023-10-16T20:37:37.408792+0000","last_undegraded":"2023-10-16T20:37:37.408792+0000","last_fullsized":"2023-10-16T20:37:37.408792+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:41.981408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404254+0000","last_change":"2023-10-16T20:36:59.464962+0000","last_active":"2023-10-16T20:37:36.404254+0000","last_peered":"2023-10-16T20:37:36.404254+0000","last_clean":"2023-10-16T20:37:36.404254+0000","last_became_active":"2023-10-16T20:36:59.464578+0000","last_became_peered":"2023-10-16T20:36:59.464578+0000","last_unstale":"2023-10-16T20:37:36.404254+0000","last_undegraded":"2023-10-16T20:37:36.404254+0000","last_fullsized":"2023-10-16T20:37:36.404254+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:12:27.738364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435179+0000","last_change":"2023-10-16T20:36:55.285675+0000","last_active":"2023-10-16T20:37:37.435179+0000","last_peered":"2023-10-16T20:37:37.435179+0000","last_clean":"2023-10-16T20:37:37.435179+0000","last_became_active":"2023-10-16T20:36:55.285407+0000","last_became_peered":"2023-10-16T20:36:55.285407+0000","last_unstale":"2023-10-16T20:37:37.435179+0000","last_undegraded":"2023-10-16T20:37:37.435179+0000","last_fullsized":"2023-10-16T20:37:37.435179+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:31:10.259938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"26'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435225+0000","last_change":"2023-10-16T20:36:54.757012+0000","last_active":"2023-10-16T20:37:37.435225+0000","last_peered":"2023-10-16T20:37:37.435225+0000","last_clean":"2023-10-16T20:37:37.435225+0000","last_became_active":"2023-10-16T20:36:54.756697+0000","last_became_peered":"2023-10-16T20:36:54.756697+0000","last_unstale":"2023-10-16T20:37:37.435225+0000","last_undegraded":"2023-10-16T20:37:37.435225+0000","last_fullsized":"2023-10-16T20:37:37.435225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:20.259580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.409816+0000","last_change":"2023-10-16T20:36:59.469134+0000","last_active":"2023-10-16T20:37:36.409816+0000","last_peered":"2023-10-16T20:37:36.409816+0000","last_clean":"2023-10-16T20:37:36.409816+0000","last_became_active":"2023-10-16T20:36:59.468254+0000","last_became_peered":"2023-10-16T20:36:59.468254+0000","last_unstale":"2023-10-16T20:37:36.409816+0000","last_undegraded":"2023-10-16T20:37:36.409816+0000","last_fullsized":"2023-10-16T20:37:36.409816+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:12.444815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.412512+0000","last_change":"2023-10-16T20:36:55.289133+0000","last_active":"2023-10-16T20:37:36.412512+0000","last_peered":"2023-10-16T20:37:36.412512+0000","last_clean":"2023-10-16T20:37:36.412512+0000","last_became_active":"2023-10-16T20:36:55.288488+0000","last_became_peered":"2023-10-16T20:36:55.288488+0000","last_unstale":"2023-10-16T20:37:36.412512+0000","last_undegraded":"2023-10-16T20:37:36.412512+0000","last_fullsized":"2023-10-16T20:37:36.412512+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:27:22.022319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'10","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:14.222665+0000","last_change":"2023-10-16T20:36:54.765361+0000","last_active":"2023-10-16T20:39:14.222665+0000","last_peered":"2023-10-16T20:39:14.222665+0000","last_clean":"2023-10-16T20:39:14.222665+0000","last_became_active":"2023-10-16T20:36:54.764969+0000","last_became_peered":"2023-10-16T20:36:54.764969+0000","last_unstale":"2023-10-16T20:39:14.222665+0000","last_undegraded":"2023-10-16T20:39:14.222665+0000","last_fullsized":"2023-10-16T20:39:14.222665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:21:21.183612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.400125+0000","last_change":"2023-10-16T20:36:59.303588+0000","last_active":"2023-10-16T20:37:36.400125+0000","last_peered":"2023-10-16T20:37:36.400125+0000","last_clean":"2023-10-16T20:37:36.400125+0000","last_became_active":"2023-10-16T20:36:59.302698+0000","last_became_peered":"2023-10-16T20:36:59.302698+0000","last_unstale":"2023-10-16T20:37:36.400125+0000","last_undegraded":"2023-10-16T20:37:36.400125+0000","last_fullsized":"2023-10-16T20:37:36.400125+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:08:28.644565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403928+0000","last_change":"2023-10-16T20:36:55.290812+0000","last_active":"2023-10-16T20:37:37.403928+0000","last_peered":"2023-10-16T20:37:37.403928+0000","last_clean":"2023-10-16T20:37:37.403928+0000","last_became_active":"2023-10-16T20:36:55.290127+0000","last_became_peered":"2023-10-16T20:36:55.290127+0000","last_unstale":"2023-10-16T20:37:37.403928+0000","last_undegraded":"2023-10-16T20:37:37.403928+0000","last_fullsized":"2023-10-16T20:37:37.403928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:28:27.930123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"27'3","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.400784+0000","last_change":"2023-10-16T20:36:54.749120+0000","last_active":"2023-10-16T20:37:36.400784+0000","last_peered":"2023-10-16T20:37:36.400784+0000","last_clean":"2023-10-16T20:37:36.400784+0000","last_became_active":"2023-10-16T20:36:54.748788+0000","last_became_peered":"2023-10-16T20:36:54.748788+0000","last_unstale":"2023-10-16T20:37:36.400784+0000","last_undegraded":"2023-10-16T20:37:36.400784+0000","last_fullsized":"2023-10-16T20:37:36.400784+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:11:07.257343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.402309+0000","last_change":"2023-10-16T20:36:59.303546+0000","last_active":"2023-10-16T20:37:36.402309+0000","last_peered":"2023-10-16T20:37:36.402309+0000","last_clean":"2023-10-16T20:37:36.402309+0000","last_became_active":"2023-10-16T20:36:59.302916+0000","last_became_peered":"2023-10-16T20:36:59.302916+0000","last_unstale":"2023-10-16T20:37:36.402309+0000","last_undegraded":"2023-10-16T20:37:36.402309+0000","last_fullsized":"2023-10-16T20:37:36.402309+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:17:16.174476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.398104+0000","last_change":"2023-10-16T20:36:55.291739+0000","last_active":"2023-10-16T20:37:36.398104+0000","last_peered":"2023-10-16T20:37:36.398104+0000","last_clean":"2023-10-16T20:37:36.398104+0000","last_became_active":"2023-10-16T20:36:55.291448+0000","last_became_peered":"2023-10-16T20:36:55.291448+0000","last_unstale":"2023-10-16T20:37:36.398104+0000","last_undegraded":"2023-10-16T20:37:36.398104+0000","last_fullsized":"2023-10-16T20:37:36.398104+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:41:15.380678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.408759+0000","last_change":"2023-10-16T20:36:54.761824+0000","last_active":"2023-10-16T20:37:36.408759+0000","last_peered":"2023-10-16T20:37:36.408759+0000","last_clean":"2023-10-16T20:37:36.408759+0000","last_became_active":"2023-10-16T20:36:54.761541+0000","last_became_peered":"2023-10-16T20:36:54.761541+0000","last_unstale":"2023-10-16T20:37:36.408759+0000","last_undegraded":"2023-10-16T20:37:36.408759+0000","last_fullsized":"2023-10-16T20:37:36.408759+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:34.512207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.403494+0000","last_change":"2023-10-16T20:36:59.304165+0000","last_active":"2023-10-16T20:37:36.403494+0000","last_peered":"2023-10-16T20:37:36.403494+0000","last_clean":"2023-10-16T20:37:36.403494+0000","last_became_active":"2023-10-16T20:36:59.303773+0000","last_became_peered":"2023-10-16T20:36:59.303773+0000","last_unstale":"2023-10-16T20:37:36.403494+0000","last_undegraded":"2023-10-16T20:37:36.403494+0000","last_fullsized":"2023-10-16T20:37:36.403494+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:09.710698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.408640+0000","last_change":"2023-10-16T20:36:55.300829+0000","last_active":"2023-10-16T20:37:36.408640+0000","last_peered":"2023-10-16T20:37:36.408640+0000","last_clean":"2023-10-16T20:37:36.408640+0000","last_became_active":"2023-10-16T20:36:55.300535+0000","last_became_peered":"2023-10-16T20:36:55.300535+0000","last_unstale":"2023-10-16T20:37:36.408640+0000","last_undegraded":"2023-10-16T20:37:36.408640+0000","last_fullsized":"2023-10-16T20:37:36.408640+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:34.910378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.405392+0000","last_change":"2023-10-16T20:36:54.758140+0000","last_active":"2023-10-16T20:37:36.405392+0000","last_peered":"2023-10-16T20:37:36.405392+0000","last_clean":"2023-10-16T20:37:36.405392+0000","last_became_active":"2023-10-16T20:36:54.757303+0000","last_became_peered":"2023-10-16T20:36:54.757303+0000","last_unstale":"2023-10-16T20:37:36.405392+0000","last_undegraded":"2023-10-16T20:37:36.405392+0000","last_fullsized":"2023-10-16T20:37:36.405392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:45.944191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404201+0000","last_change":"2023-10-16T20:36:59.471158+0000","last_active":"2023-10-16T20:37:36.404201+0000","last_peered":"2023-10-16T20:37:36.404201+0000","last_clean":"2023-10-16T20:37:36.404201+0000","last_became_active":"2023-10-16T20:36:59.465844+0000","last_became_peered":"2023-10-16T20:36:59.465844+0000","last_unstale":"2023-10-16T20:37:36.404201+0000","last_undegraded":"2023-10-16T20:37:36.404201+0000","last_fullsized":"2023-10-16T20:37:36.404201+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:12.133525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435182+0000","last_change":"2023-10-16T20:36:59.468843+0000","last_active":"2023-10-16T20:37:37.435182+0000","last_peered":"2023-10-16T20:37:37.435182+0000","last_clean":"2023-10-16T20:37:37.435182+0000","last_became_active":"2023-10-16T20:36:59.467356+0000","last_became_peered":"2023-10-16T20:36:59.467356+0000","last_unstale":"2023-10-16T20:37:37.435182+0000","last_undegraded":"2023-10-16T20:37:37.435182+0000","last_fullsized":"2023-10-16T20:37:37.435182+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:47:02.923540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.399733+0000","last_change":"2023-10-16T20:36:55.281355+0000","last_active":"2023-10-16T20:37:36.399733+0000","last_peered":"2023-10-16T20:37:36.399733+0000","last_clean":"2023-10-16T20:37:36.399733+0000","last_became_active":"2023-10-16T20:36:55.280903+0000","last_became_peered":"2023-10-16T20:36:55.280903+0000","last_unstale":"2023-10-16T20:37:36.399733+0000","last_undegraded":"2023-10-16T20:37:36.399733+0000","last_fullsized":"2023-10-16T20:37:36.399733+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:15:25.813826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.408338+0000","last_change":"2023-10-16T20:36:54.757589+0000","last_active":"2023-10-16T20:37:36.408338+0000","last_peered":"2023-10-16T20:37:36.408338+0000","last_clean":"2023-10-16T20:37:36.408338+0000","last_became_active":"2023-10-16T20:36:54.756775+0000","last_became_peered":"2023-10-16T20:36:54.756775+0000","last_unstale":"2023-10-16T20:37:36.408338+0000","last_undegraded":"2023-10-16T20:37:36.408338+0000","last_fullsized":"2023-10-16T20:37:36.408338+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:40.629540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.441661+0000","last_change":"2023-10-16T20:37:37.438581+0000","last_active":"2023-10-16T20:37:37.441661+0000","last_peered":"2023-10-16T20:37:37.441661+0000","last_clean":"2023-10-16T20:37:37.441661+0000","last_became_active":"2023-10-16T20:37:37.438140+0000","last_became_peered":"2023-10-16T20:37:37.438140+0000","last_unstale":"2023-10-16T20:37:37.441661+0000","last_undegraded":"2023-10-16T20:37:37.441661+0000","last_fullsized":"2023-10-16T20:37:37.441661+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:55.110618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.403530+0000","last_change":"2023-10-16T20:36:55.274207+0000","last_active":"2023-10-16T20:37:36.403530+0000","last_peered":"2023-10-16T20:37:36.403530+0000","last_clean":"2023-10-16T20:37:36.403530+0000","last_became_active":"2023-10-16T20:36:55.273959+0000","last_became_peered":"2023-10-16T20:36:55.273959+0000","last_unstale":"2023-10-16T20:37:36.403530+0000","last_undegraded":"2023-10-16T20:37:36.403530+0000","last_fullsized":"2023-10-16T20:37:36.403530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:23:09.275091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404500+0000","last_change":"2023-10-16T20:36:54.759513+0000","last_active":"2023-10-16T20:37:36.404500+0000","last_peered":"2023-10-16T20:37:36.404500+0000","last_clean":"2023-10-16T20:37:36.404500+0000","last_became_active":"2023-10-16T20:36:54.759059+0000","last_became_peered":"2023-10-16T20:36:54.759059+0000","last_unstale":"2023-10-16T20:37:36.404500+0000","last_undegraded":"2023-10-16T20:37:36.404500+0000","last_fullsized":"2023-10-16T20:37:36.404500+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:36.696053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"27'1","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404990+0000","last_change":"2023-10-16T20:36:54.764780+0000","last_active":"2023-10-16T20:37:36.404990+0000","last_peered":"2023-10-16T20:37:36.404990+0000","last_clean":"2023-10-16T20:37:36.404990+0000","last_became_active":"2023-10-16T20:36:54.764436+0000","last_became_peered":"2023-10-16T20:36:54.764436+0000","last_unstale":"2023-10-16T20:37:36.404990+0000","last_undegraded":"2023-10-16T20:37:36.404990+0000","last_fullsized":"2023-10-16T20:37:36.404990+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:55:31.816599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.398400+0000","last_change":"2023-10-16T20:36:55.278134+0000","last_active":"2023-10-16T20:37:36.398400+0000","last_peered":"2023-10-16T20:37:36.398400+0000","last_clean":"2023-10-16T20:37:36.398400+0000","last_became_active":"2023-10-16T20:36:55.277873+0000","last_became_peered":"2023-10-16T20:36:55.277873+0000","last_unstale":"2023-10-16T20:37:36.398400+0000","last_undegraded":"2023-10-16T20:37:36.398400+0000","last_fullsized":"2023-10-16T20:37:36.398400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:59:43.701800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.407258+0000","last_change":"2023-10-16T20:36:59.302774+0000","last_active":"2023-10-16T20:37:36.407258+0000","last_peered":"2023-10-16T20:37:36.407258+0000","last_clean":"2023-10-16T20:37:36.407258+0000","last_became_active":"2023-10-16T20:36:59.302435+0000","last_became_peered":"2023-10-16T20:36:59.302435+0000","last_unstale":"2023-10-16T20:37:36.407258+0000","last_undegraded":"2023-10-16T20:37:36.407258+0000","last_fullsized":"2023-10-16T20:37:36.407258+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:11:53.740982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"26'1","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404112+0000","last_change":"2023-10-16T20:36:54.756190+0000","last_active":"2023-10-16T20:37:36.404112+0000","last_peered":"2023-10-16T20:37:36.404112+0000","last_clean":"2023-10-16T20:37:36.404112+0000","last_became_active":"2023-10-16T20:36:54.755834+0000","last_became_peered":"2023-10-16T20:36:54.755834+0000","last_unstale":"2023-10-16T20:37:36.404112+0000","last_undegraded":"2023-10-16T20:37:36.404112+0000","last_fullsized":"2023-10-16T20:37:36.404112+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:31.012441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.402297+0000","last_change":"2023-10-16T20:36:55.279822+0000","last_active":"2023-10-16T20:37:36.402297+0000","last_peered":"2023-10-16T20:37:36.402297+0000","last_clean":"2023-10-16T20:37:36.402297+0000","last_became_active":"2023-10-16T20:36:55.279333+0000","last_became_peered":"2023-10-16T20:36:55.279333+0000","last_unstale":"2023-10-16T20:37:36.402297+0000","last_undegraded":"2023-10-16T20:37:36.402297+0000","last_fullsized":"2023-10-16T20:37:36.402297+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:02:21.471812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.403977+0000","last_change":"2023-10-16T20:36:59.302190+0000","last_active":"2023-10-16T20:37:36.403977+0000","last_peered":"2023-10-16T20:37:36.403977+0000","last_clean":"2023-10-16T20:37:36.403977+0000","last_became_active":"2023-10-16T20:36:59.301864+0000","last_became_peered":"2023-10-16T20:36:59.301864+0000","last_unstale":"2023-10-16T20:37:36.403977+0000","last_undegraded":"2023-10-16T20:37:36.403977+0000","last_fullsized":"2023-10-16T20:37:36.403977+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:37:55.282815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.406436+0000","last_change":"2023-10-16T20:36:55.302476+0000","last_active":"2023-10-16T20:37:37.406436+0000","last_peered":"2023-10-16T20:37:37.406436+0000","last_clean":"2023-10-16T20:37:37.406436+0000","last_became_active":"2023-10-16T20:36:55.287442+0000","last_became_peered":"2023-10-16T20:36:55.287442+0000","last_unstale":"2023-10-16T20:37:37.406436+0000","last_undegraded":"2023-10-16T20:37:37.406436+0000","last_fullsized":"2023-10-16T20:37:37.406436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:19.511156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.399520+0000","last_change":"2023-10-16T20:36:55.276521+0000","last_active":"2023-10-16T20:37:37.399520+0000","last_peered":"2023-10-16T20:37:37.399520+0000","last_clean":"2023-10-16T20:37:37.399520+0000","last_became_active":"2023-10-16T20:36:55.276276+0000","last_became_peered":"2023-10-16T20:36:55.276276+0000","last_unstale":"2023-10-16T20:37:37.399520+0000","last_undegraded":"2023-10-16T20:37:37.399520+0000","last_fullsized":"2023-10-16T20:37:37.399520+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:38.218242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.415548+0000","last_change":"2023-10-16T20:37:37.413429+0000","last_active":"2023-10-16T20:37:37.415548+0000","last_peered":"2023-10-16T20:37:37.415548+0000","last_clean":"2023-10-16T20:37:37.415548+0000","last_became_active":"2023-10-16T20:37:37.413040+0000","last_became_peered":"2023-10-16T20:37:37.413040+0000","last_unstale":"2023-10-16T20:37:37.415548+0000","last_undegraded":"2023-10-16T20:37:37.415548+0000","last_fullsized":"2023-10-16T20:37:37.415548+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:39:10.897056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435108+0000","last_change":"2023-10-16T20:36:55.276258+0000","last_active":"2023-10-16T20:37:37.435108+0000","last_peered":"2023-10-16T20:37:37.435108+0000","last_clean":"2023-10-16T20:37:37.435108+0000","last_became_active":"2023-10-16T20:36:55.275349+0000","last_became_peered":"2023-10-16T20:36:55.275349+0000","last_unstale":"2023-10-16T20:37:37.435108+0000","last_undegraded":"2023-10-16T20:37:37.435108+0000","last_fullsized":"2023-10-16T20:37:37.435108+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:59.007782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465455+0000","last_change":"2023-10-16T20:36:55.277332+0000","last_active":"2023-10-16T20:37:37.465455+0000","last_peered":"2023-10-16T20:37:37.465455+0000","last_clean":"2023-10-16T20:37:37.465455+0000","last_became_active":"2023-10-16T20:36:55.277043+0000","last_became_peered":"2023-10-16T20:36:55.277043+0000","last_unstale":"2023-10-16T20:37:37.465455+0000","last_undegraded":"2023-10-16T20:37:37.465455+0000","last_fullsized":"2023-10-16T20:37:37.465455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:46:07.227813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401690+0000","last_change":"2023-10-16T20:36:55.279003+0000","last_active":"2023-10-16T20:37:37.401690+0000","last_peered":"2023-10-16T20:37:37.401690+0000","last_clean":"2023-10-16T20:37:37.401690+0000","last_became_active":"2023-10-16T20:36:55.278715+0000","last_became_peered":"2023-10-16T20:36:55.278715+0000","last_unstale":"2023-10-16T20:37:37.401690+0000","last_undegraded":"2023-10-16T20:37:37.401690+0000","last_fullsized":"2023-10-16T20:37:37.401690+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:58.274009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408078+0000","last_change":"2023-10-16T20:36:55.303936+0000","last_active":"2023-10-16T20:37:37.408078+0000","last_peered":"2023-10-16T20:37:37.408078+0000","last_clean":"2023-10-16T20:37:37.408078+0000","last_became_active":"2023-10-16T20:36:55.301604+0000","last_became_peered":"2023-10-16T20:36:55.301604+0000","last_unstale":"2023-10-16T20:37:37.408078+0000","last_undegraded":"2023-10-16T20:37:37.408078+0000","last_fullsized":"2023-10-16T20:37:37.408078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:48:14.922695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465625+0000","last_change":"2023-10-16T20:36:55.277329+0000","last_active":"2023-10-16T20:37:37.465625+0000","last_peered":"2023-10-16T20:37:37.465625+0000","last_clean":"2023-10-16T20:37:37.465625+0000","last_became_active":"2023-10-16T20:36:55.276984+0000","last_became_peered":"2023-10-16T20:36:55.276984+0000","last_unstale":"2023-10-16T20:37:37.465625+0000","last_undegraded":"2023-10-16T20:37:37.465625+0000","last_fullsized":"2023-10-16T20:37:37.465625+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:43:52.763586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408404+0000","last_change":"2023-10-16T20:36:55.290545+0000","last_active":"2023-10-16T20:37:37.408404+0000","last_peered":"2023-10-16T20:37:37.408404+0000","last_clean":"2023-10-16T20:37:37.408404+0000","last_became_active":"2023-10-16T20:36:55.290023+0000","last_became_peered":"2023-10-16T20:36:55.290023+0000","last_unstale":"2023-10-16T20:37:37.408404+0000","last_undegraded":"2023-10-16T20:37:37.408404+0000","last_fullsized":"2023-10-16T20:37:37.408404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:04:20.914854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464978+0000","last_change":"2023-10-16T20:36:55.280456+0000","last_active":"2023-10-16T20:37:37.464978+0000","last_peered":"2023-10-16T20:37:37.464978+0000","last_clean":"2023-10-16T20:37:37.464978+0000","last_became_active":"2023-10-16T20:36:55.280186+0000","last_became_peered":"2023-10-16T20:36:55.280186+0000","last_unstale":"2023-10-16T20:37:37.464978+0000","last_undegraded":"2023-10-16T20:37:37.464978+0000","last_fullsized":"2023-10-16T20:37:37.464978+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:52.469768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408252+0000","last_change":"2023-10-16T20:36:55.297917+0000","last_active":"2023-10-16T20:37:37.408252+0000","last_peered":"2023-10-16T20:37:37.408252+0000","last_clean":"2023-10-16T20:37:37.408252+0000","last_became_active":"2023-10-16T20:36:55.297253+0000","last_became_peered":"2023-10-16T20:36:55.297253+0000","last_unstale":"2023-10-16T20:37:37.408252+0000","last_undegraded":"2023-10-16T20:37:37.408252+0000","last_fullsized":"2023-10-16T20:37:37.408252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:14.977093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434979+0000","last_change":"2023-10-16T20:36:55.286327+0000","last_active":"2023-10-16T20:37:37.434979+0000","last_peered":"2023-10-16T20:37:37.434979+0000","last_clean":"2023-10-16T20:37:37.434979+0000","last_became_active":"2023-10-16T20:36:55.286044+0000","last_became_peered":"2023-10-16T20:36:55.286044+0000","last_unstale":"2023-10-16T20:37:37.434979+0000","last_undegraded":"2023-10-16T20:37:37.434979+0000","last_fullsized":"2023-10-16T20:37:37.434979+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:30:56.000502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409227+0000","last_change":"2023-10-16T20:36:55.298742+0000","last_active":"2023-10-16T20:37:37.409227+0000","last_peered":"2023-10-16T20:37:37.409227+0000","last_clean":"2023-10-16T20:37:37.409227+0000","last_became_active":"2023-10-16T20:36:55.286864+0000","last_became_peered":"2023-10-16T20:36:55.286864+0000","last_unstale":"2023-10-16T20:37:37.409227+0000","last_undegraded":"2023-10-16T20:37:37.409227+0000","last_fullsized":"2023-10-16T20:37:37.409227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:24:27.598910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466521+0000","last_change":"2023-10-16T20:36:55.292541+0000","last_active":"2023-10-16T20:37:37.466521+0000","last_peered":"2023-10-16T20:37:37.466521+0000","last_clean":"2023-10-16T20:37:37.466521+0000","last_became_active":"2023-10-16T20:36:55.292262+0000","last_became_peered":"2023-10-16T20:36:55.292262+0000","last_unstale":"2023-10-16T20:37:37.466521+0000","last_undegraded":"2023-10-16T20:37:37.466521+0000","last_fullsized":"2023-10-16T20:37:37.466521+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:10:08.644265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.400632+0000","last_change":"2023-10-16T20:36:55.283056+0000","last_active":"2023-10-16T20:37:37.400632+0000","last_peered":"2023-10-16T20:37:37.400632+0000","last_clean":"2023-10-16T20:37:37.400632+0000","last_became_active":"2023-10-16T20:36:55.282806+0000","last_became_peered":"2023-10-16T20:36:55.282806+0000","last_unstale":"2023-10-16T20:37:37.400632+0000","last_undegraded":"2023-10-16T20:37:37.400632+0000","last_fullsized":"2023-10-16T20:37:37.400632+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:21:24.681696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410649+0000","last_change":"2023-10-16T20:36:55.303385+0000","last_active":"2023-10-16T20:37:37.410649+0000","last_peered":"2023-10-16T20:37:37.410649+0000","last_clean":"2023-10-16T20:37:37.410649+0000","last_became_active":"2023-10-16T20:36:55.303095+0000","last_became_peered":"2023-10-16T20:36:55.303095+0000","last_unstale":"2023-10-16T20:37:37.410649+0000","last_undegraded":"2023-10-16T20:37:37.410649+0000","last_fullsized":"2023-10-16T20:37:37.410649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:26.368480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408312+0000","last_change":"2023-10-16T20:36:55.291852+0000","last_active":"2023-10-16T20:37:37.408312+0000","last_peered":"2023-10-16T20:37:37.408312+0000","last_clean":"2023-10-16T20:37:37.408312+0000","last_became_active":"2023-10-16T20:36:55.291447+0000","last_became_peered":"2023-10-16T20:36:55.291447+0000","last_unstale":"2023-10-16T20:37:37.408312+0000","last_undegraded":"2023-10-16T20:37:37.408312+0000","last_fullsized":"2023-10-16T20:37:37.408312+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:07.195372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401838+0000","last_change":"2023-10-16T20:36:55.281169+0000","last_active":"2023-10-16T20:37:37.401838+0000","last_peered":"2023-10-16T20:37:37.401838+0000","last_clean":"2023-10-16T20:37:37.401838+0000","last_became_active":"2023-10-16T20:36:55.280605+0000","last_became_peered":"2023-10-16T20:36:55.280605+0000","last_unstale":"2023-10-16T20:37:37.401838+0000","last_undegraded":"2023-10-16T20:37:37.401838+0000","last_fullsized":"2023-10-16T20:37:37.401838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:03.294173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435040+0000","last_change":"2023-10-16T20:36:55.277395+0000","last_active":"2023-10-16T20:37:37.435040+0000","last_peered":"2023-10-16T20:37:37.435040+0000","last_clean":"2023-10-16T20:37:37.435040+0000","last_became_active":"2023-10-16T20:36:55.277094+0000","last_became_peered":"2023-10-16T20:36:55.277094+0000","last_unstale":"2023-10-16T20:37:37.435040+0000","last_undegraded":"2023-10-16T20:37:37.435040+0000","last_fullsized":"2023-10-16T20:37:37.435040+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:23:03.400384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.403871+0000","last_change":"2023-10-16T20:36:55.297429+0000","last_active":"2023-10-16T20:37:36.403871+0000","last_peered":"2023-10-16T20:37:36.403871+0000","last_clean":"2023-10-16T20:37:36.403871+0000","last_became_active":"2023-10-16T20:36:55.297199+0000","last_became_peered":"2023-10-16T20:36:55.297199+0000","last_unstale":"2023-10-16T20:37:36.403871+0000","last_undegraded":"2023-10-16T20:37:36.403871+0000","last_fullsized":"2023-10-16T20:37:36.403871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:43.860251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.403290+0000","last_change":"2023-10-16T20:36:55.287421+0000","last_active":"2023-10-16T20:37:36.403290+0000","last_peered":"2023-10-16T20:37:36.403290+0000","last_clean":"2023-10-16T20:37:36.403290+0000","last_became_active":"2023-10-16T20:36:55.286964+0000","last_became_peered":"2023-10-16T20:36:55.286964+0000","last_unstale":"2023-10-16T20:37:36.403290+0000","last_undegraded":"2023-10-16T20:37:36.403290+0000","last_fullsized":"2023-10-16T20:37:36.403290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:15:54.970100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.413891+0000","last_change":"2023-10-16T20:37:37.413891+0000","last_active":"2023-10-16T20:37:37.413891+0000","last_peered":"2023-10-16T20:37:37.413891+0000","last_clean":"2023-10-16T20:37:37.413891+0000","last_became_active":"2023-10-16T20:37:37.413492+0000","last_became_peered":"2023-10-16T20:37:37.413492+0000","last_unstale":"2023-10-16T20:37:37.413891+0000","last_undegraded":"2023-10-16T20:37:37.413891+0000","last_fullsized":"2023-10-16T20:37:37.413891+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:03:46.658556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.405037+0000","last_change":"2023-10-16T20:36:55.303950+0000","last_active":"2023-10-16T20:37:36.405037+0000","last_peered":"2023-10-16T20:37:36.405037+0000","last_clean":"2023-10-16T20:37:36.405037+0000","last_became_active":"2023-10-16T20:36:55.302468+0000","last_became_peered":"2023-10-16T20:36:55.302468+0000","last_unstale":"2023-10-16T20:37:36.405037+0000","last_undegraded":"2023-10-16T20:37:36.405037+0000","last_fullsized":"2023-10-16T20:37:36.405037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:02.962031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.413370+0000","last_change":"2023-10-16T20:36:55.297558+0000","last_active":"2023-10-16T20:37:36.413370+0000","last_peered":"2023-10-16T20:37:36.413370+0000","last_clean":"2023-10-16T20:37:36.413370+0000","last_became_active":"2023-10-16T20:36:55.296476+0000","last_became_peered":"2023-10-16T20:36:55.296476+0000","last_unstale":"2023-10-16T20:37:36.413370+0000","last_undegraded":"2023-10-16T20:37:36.413370+0000","last_fullsized":"2023-10-16T20:37:36.413370+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:04:38.235205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.403439+0000","last_change":"2023-10-16T20:36:55.293957+0000","last_active":"2023-10-16T20:37:36.403439+0000","last_peered":"2023-10-16T20:37:36.403439+0000","last_clean":"2023-10-16T20:37:36.403439+0000","last_became_active":"2023-10-16T20:36:55.293687+0000","last_became_peered":"2023-10-16T20:36:55.293687+0000","last_unstale":"2023-10-16T20:37:36.403439+0000","last_undegraded":"2023-10-16T20:37:36.403439+0000","last_fullsized":"2023-10-16T20:37:36.403439+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:54.623691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.411550+0000","last_change":"2023-10-16T20:36:55.280728+0000","last_active":"2023-10-16T20:37:36.411550+0000","last_peered":"2023-10-16T20:37:36.411550+0000","last_clean":"2023-10-16T20:37:36.411550+0000","last_became_active":"2023-10-16T20:36:55.280444+0000","last_became_peered":"2023-10-16T20:36:55.280444+0000","last_unstale":"2023-10-16T20:37:36.411550+0000","last_undegraded":"2023-10-16T20:37:36.411550+0000","last_fullsized":"2023-10-16T20:37:36.411550+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:44.044563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.408400+0000","last_change":"2023-10-16T20:36:55.297802+0000","last_active":"2023-10-16T20:37:36.408400+0000","last_peered":"2023-10-16T20:37:36.408400+0000","last_clean":"2023-10-16T20:37:36.408400+0000","last_became_active":"2023-10-16T20:36:55.296694+0000","last_became_peered":"2023-10-16T20:36:55.296694+0000","last_unstale":"2023-10-16T20:37:36.408400+0000","last_undegraded":"2023-10-16T20:37:36.408400+0000","last_fullsized":"2023-10-16T20:37:36.408400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:00:26.797387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.411707+0000","last_change":"2023-10-16T20:36:55.291347+0000","last_active":"2023-10-16T20:37:36.411707+0000","last_peered":"2023-10-16T20:37:36.411707+0000","last_clean":"2023-10-16T20:37:36.411707+0000","last_became_active":"2023-10-16T20:36:55.291052+0000","last_became_peered":"2023-10-16T20:36:55.291052+0000","last_unstale":"2023-10-16T20:37:36.411707+0000","last_undegraded":"2023-10-16T20:37:36.411707+0000","last_fullsized":"2023-10-16T20:37:36.411707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:40:51.331446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409006+0000","last_change":"2023-10-16T20:36:55.290870+0000","last_active":"2023-10-16T20:37:37.409006+0000","last_peered":"2023-10-16T20:37:37.409006+0000","last_clean":"2023-10-16T20:37:37.409006+0000","last_became_active":"2023-10-16T20:36:55.290386+0000","last_became_peered":"2023-10-16T20:36:55.290386+0000","last_unstale":"2023-10-16T20:37:37.409006+0000","last_undegraded":"2023-10-16T20:37:37.409006+0000","last_fullsized":"2023-10-16T20:37:37.409006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:20.006864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435199+0000","last_change":"2023-10-16T20:36:55.271041+0000","last_active":"2023-10-16T20:37:37.435199+0000","last_peered":"2023-10-16T20:37:37.435199+0000","last_clean":"2023-10-16T20:37:37.435199+0000","last_became_active":"2023-10-16T20:36:55.270787+0000","last_became_peered":"2023-10-16T20:36:55.270787+0000","last_unstale":"2023-10-16T20:37:37.435199+0000","last_undegraded":"2023-10-16T20:37:37.435199+0000","last_fullsized":"2023-10-16T20:37:37.435199+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:07.944778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402678+0000","last_change":"2023-10-16T20:36:55.271632+0000","last_active":"2023-10-16T20:37:37.402678+0000","last_peered":"2023-10-16T20:37:37.402678+0000","last_clean":"2023-10-16T20:37:37.402678+0000","last_became_active":"2023-10-16T20:36:55.271373+0000","last_became_peered":"2023-10-16T20:36:55.271373+0000","last_unstale":"2023-10-16T20:37:37.402678+0000","last_undegraded":"2023-10-16T20:37:37.402678+0000","last_fullsized":"2023-10-16T20:37:37.402678+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:07.458540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466741+0000","last_change":"2023-10-16T20:36:55.291343+0000","last_active":"2023-10-16T20:37:37.466741+0000","last_peered":"2023-10-16T20:37:37.466741+0000","last_clean":"2023-10-16T20:37:37.466741+0000","last_became_active":"2023-10-16T20:36:55.291016+0000","last_became_peered":"2023-10-16T20:36:55.291016+0000","last_unstale":"2023-10-16T20:37:37.466741+0000","last_undegraded":"2023-10-16T20:37:37.466741+0000","last_fullsized":"2023-10-16T20:37:37.466741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:13:40.287782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4639,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509473,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343936,"statfs":{"total":96636764160,"available":96608190464,"internally_reserved":0,"allocated":905216,"data_stored":715195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.74299999999999999,"15min":0.748},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.628},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.75700000000000001,"15min":0.753},"min":{"1min":0.58099999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94599999999999995,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.76800000000000002}]},{"osd":1,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.74299999999999999,"15min":0.73999999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.91600000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.52100000000000002,"5min":0.497,"15min":0.497},"max":{"1min":0.91500000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.68899999999999995}]},{"osd":2,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.73199999999999998,"15min":0.72599999999999998},"min":{"1min":0.624,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.748,"15min":0.745},"min":{"1min":0.60699999999999998,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.89700000000000002,"5min":1.083,"15min":1.083},"last":0.66800000000000004}]},{"osd":3,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.77600000000000002,"15min":0.77200000000000002},"min":{"1min":0.62,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0069999999999999,"5min":1.234,"15min":1.234},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.93200000000000005,"5min":1.024,"15min":1.024},"last":0.63800000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63100000000000001,"15min":0.63300000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.92500000000000004,"5min":1.3,"15min":1.3},"last":0.496},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60499999999999998,"15min":0.60999999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76500000000000001,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.65600000000000003}]},{"osd":5,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.47599999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.78600000000000003,"15min":0.79500000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.006,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.72199999999999998}]},{"osd":6,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.749,"15min":0.755},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.97399999999999998,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.70399999999999996,"15min":0.70599999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.88300000000000001,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.51900000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509473,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":128086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.76800000000000002,"15min":0.77100000000000002},"min":{"1min":0.41899999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88,"5min":1.131,"15min":1.131},"last":0.751},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.748,"15min":0.75800000000000001},"min":{"1min":0.48899999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.875,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.69199999999999995}]},{"osd":1,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.751,"15min":0.76300000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.81799999999999995,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.624},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.77000000000000002,"15min":0.77600000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.90400000000000003,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.76900000000000002}]},{"osd":2,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.78000000000000003,"15min":0.79200000000000004},"min":{"1min":0.48599999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.90300000000000002,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.73899999999999999,"15min":0.746},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.86899999999999999,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.71099999999999997}]},{"osd":3,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.84999999999999998,"15min":0.87},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.90600000000000003,"5min":1.611,"15min":1.611},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.749,"5min":0.80900000000000005,"15min":0.81899999999999995},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.89700000000000002,"5min":1.163,"15min":1.163},"last":0.64400000000000002}]},{"osd":4,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.66700000000000004,"15min":0.67900000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.79500000000000004,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65700000000000003,"15min":0.66700000000000004},"min":{"1min":0.49299999999999999,"5min":0.435,"15min":0.435},"max":{"1min":0.69499999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.497}]},{"osd":5,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86199999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69899999999999995,"15min":0.70599999999999996},"min":{"1min":0.54900000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.82399999999999995,"5min":1.016,"15min":1.016},"last":0.73399999999999999}]},{"osd":7,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.748,"15min":0.754},"min":{"1min":0.52200000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.93100000000000005,"5min":1.292,"15min":1.292},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.75900000000000001,"15min":0.77500000000000002},"min":{"1min":0.52800000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.92100000000000004,"5min":1.569,"15min":1.569},"last":0.78800000000000003}]}]},{"osd":5,"up_from":15,"seq":64424509475,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":311296,"data_stored":124335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.70099999999999996,"15min":0.69599999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.505},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.56899999999999995}]},{"osd":1,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70699999999999996,"15min":0.70499999999999996},"min":{"1min":0.52100000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.752,"5min":0.71499999999999997,"15min":0.70899999999999996},"min":{"1min":0.57399999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.63200000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.72199999999999998,"15min":0.73199999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92300000000000004,"5min":1.022,"15min":1.022},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.74299999999999999,"15min":0.73699999999999999},"min":{"1min":0.60799999999999998,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.91700000000000004,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65600000000000003}]},{"osd":3,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79200000000000004,"15min":0.79100000000000004},"min":{"1min":0.66700000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81000000000000005,"15min":0.81599999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.70799999999999996}]},{"osd":4,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65100000000000002,"15min":0.65900000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.90300000000000002,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.73399999999999999,"15min":0.75800000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.80500000000000005,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.61599999999999999}]},{"osd":6,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.70999999999999996,"15min":0.72799999999999998},"min":{"1min":0.45500000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.79700000000000004,"5min":1.069,"15min":1.069},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.495,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.038,"5min":1.264,"15min":1.264},"last":0.67500000000000004}]},{"osd":7,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.78000000000000003,"15min":0.79600000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.89200000000000002,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.79100000000000004,"15min":0.80700000000000005},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.163,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.76900000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509473,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":319488,"data_stored":124371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.72199999999999998,"15min":0.72399999999999998},"min":{"1min":0.45200000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.872,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.59599999999999997,"5min":0.5,"15min":0.5},"max":{"1min":0.88200000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.80800000000000005}]},{"osd":1,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.57499999999999996,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92500000000000004,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.748,"15min":0.748},"min":{"1min":0.54500000000000004,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.69099999999999995}]},{"osd":2,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.76400000000000001,"15min":0.76200000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77700000000000002,"15min":0.77800000000000002},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.746}]},{"osd":3,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.747,"15min":0.73899999999999999},"min":{"1min":0.58899999999999997,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.78200000000000003,"15min":0.78600000000000003},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.86399999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.83299999999999996}]},{"osd":5,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.63600000000000001,"15min":0.64800000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.75600000000000001,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.44800000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95899999999999996,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.91700000000000004}]},{"osd":6,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.66500000000000004,"15min":0.66700000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73499999999999999,"15min":0.73799999999999999},"min":{"1min":0.495,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.85999999999999999}]},{"osd":7,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.747,"15min":0.751},"min":{"1min":0.52000000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.94599999999999995,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.79300000000000004,"15min":0.80300000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.96599999999999997,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.77100000000000002}]}]},{"osd":3,"up_from":15,"seq":64424509473,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":128086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.46999999999999997,"15min":0.47199999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.69499999999999995,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.498},{"interface":"front","average":{"1min":0.5,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.53400000000000003}]},{"osd":1,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.41499999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71699999999999997,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.56299999999999994}]},{"osd":2,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.53800000000000003,"15min":0.54400000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.67600000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.626,"15min":0.61699999999999999},"min":{"1min":0.51600000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.58299999999999996}]},{"osd":4,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.80200000000000005,"15min":0.80000000000000004},"min":{"1min":0.56200000000000006,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.054,"5min":1.181,"15min":1.181},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.77300000000000002,"15min":0.76500000000000001},"min":{"1min":0.57499999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.052,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.88600000000000001}]},{"osd":5,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.85999999999999999,"15min":0.873},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.008,"5min":1.544,"15min":1.544},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.323,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.79100000000000004}]},{"osd":6,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.85499999999999998,"15min":0.85799999999999998},"min":{"1min":0.64900000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.3480000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.755}]},{"osd":7,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.80600000000000005,"15min":0.80000000000000004},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0880000000000001,"5min":1.139,"15min":1.139},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.0029999999999999,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.76400000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509473,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":124245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.47899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71099999999999997,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.58899999999999997}]},{"osd":1,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77400000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55100000000000005,"15min":0.54900000000000004},"min":{"1min":0.38700000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69399999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44600000000000001}]},{"osd":3,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54400000000000004,"15min":0.53600000000000003},"min":{"1min":0.41299999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76700000000000002,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.46999999999999997}]},{"osd":4,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.82299999999999995,"15min":0.82899999999999996},"min":{"1min":0.60799999999999998,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.97499999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.64600000000000002,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.98299999999999998,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.63900000000000001}]},{"osd":5,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.60999999999999999,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.96999999999999997,"5min":1.444,"15min":1.444},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.83899999999999997,"15min":0.84199999999999997},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":0.998,"5min":1.585,"15min":1.585},"last":0.79700000000000004}]},{"osd":6,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.84799999999999998,"15min":0.85199999999999998},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.1020000000000001,"5min":1.335,"15min":1.335},"last":0.754},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.871,"15min":0.88200000000000001},"min":{"1min":0.68999999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":0.95099999999999996,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.69799999999999995}]},{"osd":7,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.86499999999999999,"15min":0.875},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.046,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.88300000000000001,"15min":0.89300000000000002},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.0589999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.78900000000000003}]}]},{"osd":1,"up_from":15,"seq":64424509473,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":315392,"data_stored":124405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.46400000000000002,"15min":0.46700000000000003},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.64700000000000002,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.32600000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.69899999999999995,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.35099999999999998}]},{"osd":2,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.375,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82499999999999996,"5min":1.046,"15min":1.046},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.70699999999999996,"5min":1.115,"15min":1.115},"last":0.373}]},{"osd":3,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60599999999999998,"15min":0.60899999999999999},"min":{"1min":0.40799999999999997,"5min":0.312,"15min":0.312},"max":{"1min":0.78500000000000003,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.60799999999999998,"15min":0.61799999999999999},"min":{"1min":0.41499999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.80400000000000005,"5min":1.431,"15min":1.431},"last":0.48199999999999998}]},{"osd":4,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.83399999999999996,"15min":0.83599999999999997},"min":{"1min":0.59199999999999997,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.242,"15min":1.242},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.81299999999999994,"15min":0.81100000000000005},"min":{"1min":0.60099999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.054,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.67500000000000004}]},{"osd":5,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.80800000000000005,"15min":0.81999999999999995},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90800000000000003,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.871,"15min":0.872},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.1160000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.68500000000000005}]},{"osd":6,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.874,"15min":0.88100000000000001},"min":{"1min":0.65300000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.123,"5min":2.004,"15min":2.004},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.876,"15min":0.88300000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.0880000000000001,"5min":1.635,"15min":1.635},"last":0.72499999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.878,"15min":0.89200000000000002},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.99099999999999999,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.871,"5min":0.90700000000000003,"15min":0.91300000000000003},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.0760000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.66500000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343928,"statfs":{"total":96636764160,"available":96608182272,"internally_reserved":0,"allocated":913408,"data_stored":715251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.439,"15min":0.438},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.495,"15min":0.496},"min":{"1min":0.374,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.58999999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.313}]},{"osd":2,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52300000000000002,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54100000000000004,"15min":0.54600000000000004},"min":{"1min":0.40400000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.68899999999999995,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.46999999999999997}]},{"osd":3,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54800000000000004,"15min":0.54500000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73399999999999999,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.56499999999999995,"5min":0.497,"15min":0.497},"max":{"1min":0.84999999999999998,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.442}]},{"osd":4,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.83799999999999997,"15min":0.82999999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0600000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.85999999999999999,"15min":0.84899999999999998},"min":{"1min":0.61399999999999999,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.59599999999999997}]},{"osd":5,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.80800000000000005,"15min":0.80300000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.67300000000000004}]},{"osd":6,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.86299999999999999,"15min":0.85899999999999999},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.746},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88400000000000001,"15min":0.88300000000000001},"min":{"1min":0.60899999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.56599999999999995}]},{"osd":7,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.64900000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.83499999999999996,"15min":0.82299999999999995},"min":{"1min":0.66500000000000004,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.69899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T20:39:20.840 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-16T20:39:20.841 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-16T20:39:20.841 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-16T20:39:20.842 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-16T20:39:20.842 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-16T20:39:21.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75abb3e640 1 Processor -- start 2023-10-16T20:39:21.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75abb3e640 1 -- start start 2023-10-16T20:39:21.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75abb3e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a4151410 0x7f75a40c4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75abb3e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f75a41517f0 0x7f75a40c6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75abb3e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a415b740 con 0x7f75a40c7bd0 2023-10-16T20:39:21.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75abb3e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a415b460 con 0x7f75a41517f0 2023-10-16T20:39:21.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75abb3e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a406d780 con 0x7f75a4151410 2023-10-16T20:39:21.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75aa0db640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f75a40c7bd0 0x7f75a415a840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36884/0 (socket says 172.21.15.88:36884) 2023-10-16T20:39:21.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75aa0db640 1 -- 172.21.15.88:0/1556049268 learned_addr learned my addr 172.21.15.88:0/1556049268 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:21.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a98da640 1 --2- 172.21.15.88:0/1556049268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a4151410 0x7f75a40c4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a90d9640 1 --2- 172.21.15.88:0/1556049268 >> v2:172.21.15.191:3300/0 conn(0x7f75a41517f0 0x7f75a40c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a88d8640 1 -- 172.21.15.88:0/1556049268 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442597954 0 0) 0x7f75a415b740 con 0x7f75a40c7bd0 2023-10-16T20:39:21.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a88d8640 1 -- 172.21.15.88:0/1556049268 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7590003530 con 0x7f75a40c7bd0 2023-10-16T20:39:21.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a88d8640 1 -- 172.21.15.88:0/1556049268 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1667311536 0 0) 0x7f7590003530 con 0x7f75a40c7bd0 2023-10-16T20:39:21.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a88d8640 1 -- 172.21.15.88:0/1556049268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a4151410 msgr2=0x7f75a40c4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:21.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a88d8640 1 --2- 172.21.15.88:0/1556049268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a4151410 0x7f75a40c4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a88d8640 1 -- 172.21.15.88:0/1556049268 >> v2:172.21.15.191:3300/0 conn(0x7f75a41517f0 msgr2=0x7f75a40c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:21.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a88d8640 1 --2- 172.21.15.88:0/1556049268 >> v2:172.21.15.191:3300/0 conn(0x7f75a41517f0 0x7f75a40c6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a88d8640 1 -- 172.21.15.88:0/1556049268 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a406da60 con 0x7f75a40c7bd0 2023-10-16T20:39:21.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a88d8640 1 -- 172.21.15.88:0/1556049268 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f759c001a00 con 0x7f75a40c7bd0 2023-10-16T20:39:21.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a88d8640 1 -- 172.21.15.88:0/1556049268 >> v1:172.21.15.88:6789/0 conn(0x7f75a40c7bd0 legacy=0x7f75a415a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:21.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a88d8640 1 --2- 172.21.15.88:0/1556049268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7590003980 0x7f7590003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a88d8640 1 --2- 172.21.15.88:0/1556049268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75900047d0 0x7f7590004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a88d8640 1 --2- 172.21.15.88:0/1556049268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a4151410 0x7f7590005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a88d8640 1 -- 172.21.15.88:0/1556049268 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a415b460 con 0x7f7590003980 2023-10-16T20:39:21.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a88d8640 1 -- 172.21.15.88:0/1556049268 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a406d780 con 0x7f75900047d0 2023-10-16T20:39:21.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a88d8640 1 -- 172.21.15.88:0/1556049268 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759000b990 con 0x7f75a4151410 2023-10-16T20:39:21.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a98da640 1 --2- 172.21.15.88:0/1556049268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7590003980 0x7f7590003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a90d9640 1 --2- 172.21.15.88:0/1556049268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75900047d0 0x7f7590004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75aa0db640 1 --2- 172.21.15.88:0/1556049268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a4151410 0x7f7590005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a98da640 1 -- 172.21.15.88:0/1556049268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a4151410 msgr2=0x7f7590005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:21.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a98da640 1 --2- 172.21.15.88:0/1556049268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a4151410 0x7f7590005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a98da640 1 -- 172.21.15.88:0/1556049268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75900047d0 msgr2=0x7f7590004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:21.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.005+0000 7f75a98da640 1 --2- 172.21.15.88:0/1556049268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75900047d0 0x7f7590004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.009+0000 7f75a98da640 1 -- 172.21.15.88:0/1556049268 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7590003530 con 0x7f7590003980 2023-10-16T20:39:21.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.009+0000 7f75a98da640 1 --2- 172.21.15.88:0/1556049268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7590003980 0x7f7590003da0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009ae0 tx=0x7f75a001ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83c56c091d5c284f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:21.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.009+0000 7f75a88d8640 1 -- 172.21.15.88:0/1556049268 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a00a5510 con 0x7f7590003980 2023-10-16T20:39:21.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.009+0000 7f75a88d8640 1 -- 172.21.15.88:0/1556049268 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a00a5670 con 0x7f7590003980 2023-10-16T20:39:21.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.009+0000 7f75a88d8640 1 -- 172.21.15.88:0/1556049268 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a00a5a20 con 0x7f7590003980 2023-10-16T20:39:21.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.009+0000 7f75abb3e640 1 -- 172.21.15.88:0/1556049268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7590003980 msgr2=0x7f7590003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:21.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.009+0000 7f75abb3e640 1 --2- 172.21.15.88:0/1556049268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7590003980 0x7f7590003da0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009ae0 tx=0x7f75a001ca20 comp rx=0 tx=0).stop 2023-10-16T20:39:21.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.009+0000 7f75aa0db640 1 -- 172.21.15.88:0/1556049268 reap_dead start 2023-10-16T20:39:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.009+0000 7f75abb3e640 1 -- 172.21.15.88:0/1556049268 shutdown_connections 2023-10-16T20:39:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.009+0000 7f75abb3e640 1 -- 172.21.15.88:0/1556049268 >> 172.21.15.88:0/1556049268 conn(0x7f75a4071a60 msgr2=0x7f75a4071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.009+0000 7f75abb3e640 1 -- 172.21.15.88:0/1556049268 shutdown_connections 2023-10-16T20:39:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.009+0000 7f75abb3e640 1 -- 172.21.15.88:0/1556049268 wait complete. 2023-10-16T20:39:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75abb3e640 1 Processor -- start 2023-10-16T20:39:21.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75abb3e640 1 -- start start 2023-10-16T20:39:21.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75abb3e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7590003980 0x7f75a4151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75abb3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75a4154100 0x7f75a4154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75abb3e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a41f38e0 0x7f75a41f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75abb3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a41f34c0 con 0x7f75a4154100 2023-10-16T20:39:21.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75abb3e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a414a3f0 con 0x7f7590003980 2023-10-16T20:39:21.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75abb3e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a41f42a0 con 0x7f75a41f38e0 2023-10-16T20:39:21.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75a90d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75a4154100 0x7f75a4154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75a90d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75a4154100 0x7f75a4154560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56266/0 (socket says 172.21.15.88:56266) 2023-10-16T20:39:21.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75a90d9640 1 -- 172.21.15.88:0/2276799689 learned_addr learned my addr 172.21.15.88:0/2276799689 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:21.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75a98da640 1 --2- 172.21.15.88:0/2276799689 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7590003980 0x7f75a4151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75aa0db640 1 --2- 172.21.15.88:0/2276799689 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a41f38e0 0x7f75a41f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75a90d9640 1 -- 172.21.15.88:0/2276799689 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a41f38e0 msgr2=0x7f75a41f3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:21.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75a90d9640 1 --2- 172.21.15.88:0/2276799689 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a41f38e0 0x7f75a41f3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75a90d9640 1 -- 172.21.15.88:0/2276799689 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7590003980 msgr2=0x7f75a4151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:21.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75a90d9640 1 --2- 172.21.15.88:0/2276799689 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7590003980 0x7f75a4151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75a90d9640 1 -- 172.21.15.88:0/2276799689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a415ac50 con 0x7f75a4154100 2023-10-16T20:39:21.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75a90d9640 1 --2- 172.21.15.88:0/2276799689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75a4154100 0x7f75a4154560 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7594011ad0 tx=0x7f7594011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:21.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f759a7fc640 1 -- 172.21.15.88:0/2276799689 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759400faf0 con 0x7f75a4154100 2023-10-16T20:39:21.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f759a7fc640 1 -- 172.21.15.88:0/2276799689 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759400fc50 con 0x7f75a4154100 2023-10-16T20:39:21.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f75abb3e640 1 -- 172.21.15.88:0/2276799689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a41530f0 con 0x7f75a4154100 2023-10-16T20:39:21.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.013+0000 7f759a7fc640 1 -- 172.21.15.88:0/2276799689 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7594014440 con 0x7f75a4154100 2023-10-16T20:39:21.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.017+0000 7f75abb3e640 1 -- 172.21.15.88:0/2276799689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a41536a0 con 0x7f75a4154100 2023-10-16T20:39:21.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.017+0000 7f75abb3e640 1 -- 172.21.15.88:0/2276799689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a41f3600 con 0x7f75a4154100 2023-10-16T20:39:21.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.017+0000 7f759a7fc640 1 -- 172.21.15.88:0/2276799689 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f75940145a0 con 0x7f75a4154100 2023-10-16T20:39:21.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.017+0000 7f759a7fc640 1 --2- 172.21.15.88:0/2276799689 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7588066520 0x7f75880689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.017+0000 7f759a7fc640 1 -- 172.21.15.88:0/2276799689 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7594084910 con 0x7f75a4154100 2023-10-16T20:39:21.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.017+0000 7f75a98da640 1 --2- 172.21.15.88:0/2276799689 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7588066520 0x7f75880689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.017+0000 7f75a98da640 1 --2- 172.21.15.88:0/2276799689 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7588066520 0x7f75880689e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f75a00af000 tx=0x7f75a00046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:21.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.021+0000 7f759a7fc640 1 -- 172.21.15.88:0/2276799689 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f75a41f3600 con 0x7f75a4154100 2023-10-16T20:39:21.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.209+0000 7f75abb3e640 1 -- 172.21.15.88:0/2276799689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f75a41f3600 con 0x7f75a4154100 2023-10-16T20:39:21.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.209+0000 7f759a7fc640 1 -- 172.21.15.88:0/2276799689 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+164429 (secure 0 0 0) 0x7f75a41f3600 con 0x7f75a4154100 2023-10-16T20:39:21.219 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:21.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.213+0000 7f75abb3e640 1 -- 172.21.15.88:0/2276799689 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7588066520 msgr2=0x7f75880689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:21.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.213+0000 7f75abb3e640 1 --2- 172.21.15.88:0/2276799689 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7588066520 0x7f75880689e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f75a00af000 tx=0x7f75a00046d0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.213+0000 7f75abb3e640 1 -- 172.21.15.88:0/2276799689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75a4154100 msgr2=0x7f75a4154560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:21.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.213+0000 7f75abb3e640 1 --2- 172.21.15.88:0/2276799689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75a4154100 0x7f75a4154560 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7594011ad0 tx=0x7f7594011ed0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.213+0000 7f75abb3e640 1 -- 172.21.15.88:0/2276799689 shutdown_connections 2023-10-16T20:39:21.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.213+0000 7f75abb3e640 1 --2- 172.21.15.88:0/2276799689 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7588066520 0x7f75880689e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.213+0000 7f75abb3e640 1 --2- 172.21.15.88:0/2276799689 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75a41f38e0 0x7f75a41f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.213+0000 7f75abb3e640 1 --2- 172.21.15.88:0/2276799689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75a4154100 0x7f75a4154560 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.213+0000 7f75abb3e640 1 --2- 172.21.15.88:0/2276799689 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7590003980 0x7f75a4151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.213+0000 7f75abb3e640 1 -- 172.21.15.88:0/2276799689 >> 172.21.15.88:0/2276799689 conn(0x7f75a4071a60 msgr2=0x7f75a406c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:21.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.213+0000 7f75abb3e640 1 -- 172.21.15.88:0/2276799689 shutdown_connections 2023-10-16T20:39:21.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.213+0000 7f75abb3e640 1 -- 172.21.15.88:0/2276799689 wait complete. 2023-10-16T20:39:21.239 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":5,"active_gid":4110,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":1504518179},{"type":"v1","addr":"172.21.15.88:6833","nonce":1504518179}]},"active_addr":"172.21.15.88:6833/1504518179","active_change":"2023-10-16T20:36:35.572934+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4105,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":1566315103}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":214895095}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":3069273656}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":1725049522}]}]} 2023-10-16T20:39:21.244 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-16T20:39:21.245 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-16T20:39:21.245 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:39:21.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.373+0000 7fdd510c9640 1 Processor -- start 2023-10-16T20:39:21.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.373+0000 7fdd510c9640 1 -- start start 2023-10-16T20:39:21.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.377+0000 7fdd510c9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdd4c153770 0x7fdd4c159f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.377+0000 7fdd510c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdd4c0c9040 0x7fdd4c0c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.377+0000 7fdd510c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd4c0c6a20 0x7fdd4c071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.377+0000 7fdd510c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd4c069e90 con 0x7fdd4c0c9040 2023-10-16T20:39:21.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.377+0000 7fdd510c9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd4c06aef0 con 0x7fdd4c153770 2023-10-16T20:39:21.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.377+0000 7fdd510c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c06a0c0 con 0x7fdd4c0c6a20 2023-10-16T20:39:21.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.377+0000 7fdd4259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdd4c0c9040 0x7fdd4c0c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.377+0000 7fdd4ad9d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdd4c153770 0x7fdd4c159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.377+0000 7fdd4259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdd4c0c9040 0x7fdd4c0c64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56274/0 (socket says 172.21.15.88:56274) 2023-10-16T20:39:21.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.377+0000 7fdd4259c640 1 -- 172.21.15.88:0/310360723 learned_addr learned my addr 172.21.15.88:0/310360723 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:21.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.377+0000 7fdd4b59e640 1 --2- 172.21.15.88:0/310360723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd4c0c6a20 0x7fdd4c071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.377+0000 7fdd4259c640 1 -- 172.21.15.88:0/310360723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd4c0c6a20 msgr2=0x7fdd4c071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:21.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.377+0000 7fdd4259c640 1 --2- 172.21.15.88:0/310360723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd4c0c6a20 0x7fdd4c071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.377+0000 7fdd4259c640 1 -- 172.21.15.88:0/310360723 >> v2:172.21.15.191:3300/0 conn(0x7fdd4c153770 msgr2=0x7fdd4c159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:21.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.377+0000 7fdd4259c640 1 --2- 172.21.15.88:0/310360723 >> v2:172.21.15.191:3300/0 conn(0x7fdd4c153770 0x7fdd4c159f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.377+0000 7fdd4259c640 1 -- 172.21.15.88:0/310360723 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4c06a360 con 0x7fdd4c0c9040 2023-10-16T20:39:21.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.377+0000 7fdd4ad9d640 1 --2- 172.21.15.88:0/310360723 >> v2:172.21.15.191:3300/0 conn(0x7fdd4c153770 0x7fdd4c159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:39:21.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd4259c640 1 --2- 172.21.15.88:0/310360723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd4c0c9040 0x7fdd4c0c64e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c06f390 tx=0x7fdd3401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85145b3a629360d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:21.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd4a59c640 1 -- 172.21.15.88:0/310360723 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd340ac410 con 0x7fdd4c0c9040 2023-10-16T20:39:21.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd4a59c640 1 -- 172.21.15.88:0/310360723 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd340ac570 con 0x7fdd4c0c9040 2023-10-16T20:39:21.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd4a59c640 1 -- 172.21.15.88:0/310360723 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd340ac900 con 0x7fdd4c0c9040 2023-10-16T20:39:21.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd510c9640 1 -- 172.21.15.88:0/310360723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd4c0c9040 msgr2=0x7fdd4c0c64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:21.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd510c9640 1 --2- 172.21.15.88:0/310360723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd4c0c9040 0x7fdd4c0c64e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c06f390 tx=0x7fdd3401c4e0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd510c9640 1 -- 172.21.15.88:0/310360723 shutdown_connections 2023-10-16T20:39:21.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd510c9640 1 --2- 172.21.15.88:0/310360723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd4c0c6a20 0x7fdd4c071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd510c9640 1 --2- 172.21.15.88:0/310360723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd4c0c9040 0x7fdd4c0c64e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd510c9640 1 --2- 172.21.15.88:0/310360723 >> v2:172.21.15.191:3300/0 conn(0x7fdd4c153770 0x7fdd4c159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd510c9640 1 -- 172.21.15.88:0/310360723 >> 172.21.15.88:0/310360723 conn(0x7fdd4c0c84e0 msgr2=0x7fdd4c0c88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:21.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd510c9640 1 -- 172.21.15.88:0/310360723 shutdown_connections 2023-10-16T20:39:21.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd510c9640 1 -- 172.21.15.88:0/310360723 wait complete. 2023-10-16T20:39:21.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd510c9640 1 Processor -- start 2023-10-16T20:39:21.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd510c9640 1 -- start start 2023-10-16T20:39:21.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd510c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd4c0c6a20 0x7fdd4c1f5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd510c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd4c0c9040 0x7fdd4c152410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd510c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd4c153770 0x7fdd4c1f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd510c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c1f5b00 con 0x7fdd4c0c6a20 2023-10-16T20:39:21.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd510c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c1517c0 con 0x7fdd4c0c9040 2023-10-16T20:39:21.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.381+0000 7fdd510c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c150970 con 0x7fdd4c153770 2023-10-16T20:39:21.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd4ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd4c0c6a20 0x7fdd4c1f5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd4ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd4c0c6a20 0x7fdd4c1f5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56276/0 (socket says 172.21.15.88:56276) 2023-10-16T20:39:21.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd4ad9d640 1 -- 172.21.15.88:0/1430827803 learned_addr learned my addr 172.21.15.88:0/1430827803 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:21.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd4b59e640 1 --2- 172.21.15.88:0/1430827803 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd4c153770 0x7fdd4c1f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd4259c640 1 --2- 172.21.15.88:0/1430827803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd4c0c9040 0x7fdd4c152410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd4ad9d640 1 -- 172.21.15.88:0/1430827803 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd4c153770 msgr2=0x7fdd4c1f1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:21.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd4ad9d640 1 --2- 172.21.15.88:0/1430827803 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd4c153770 0x7fdd4c1f1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd4ad9d640 1 -- 172.21.15.88:0/1430827803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd4c0c9040 msgr2=0x7fdd4c152410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:21.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd4ad9d640 1 --2- 172.21.15.88:0/1430827803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd4c0c9040 0x7fdd4c152410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd4ad9d640 1 -- 172.21.15.88:0/1430827803 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c009420 con 0x7fdd4c0c6a20 2023-10-16T20:39:21.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd4ad9d640 1 --2- 172.21.15.88:0/1430827803 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd4c0c6a20 0x7fdd4c1f5f90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00e980 tx=0x7fdd3c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:21.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd43fff640 1 -- 172.21.15.88:0/1430827803 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c00ca00 con 0x7fdd4c0c6a20 2023-10-16T20:39:21.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd43fff640 1 -- 172.21.15.88:0/1430827803 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3c00cb60 con 0x7fdd4c0c6a20 2023-10-16T20:39:21.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd510c9640 1 -- 172.21.15.88:0/1430827803 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd340094d0 con 0x7fdd4c0c6a20 2023-10-16T20:39:21.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd43fff640 1 -- 172.21.15.88:0/1430827803 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c013470 con 0x7fdd4c0c6a20 2023-10-16T20:39:21.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd510c9640 1 -- 172.21.15.88:0/1430827803 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd4c1514e0 con 0x7fdd4c0c6a20 2023-10-16T20:39:21.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd510c9640 1 -- 172.21.15.88:0/1430827803 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4c074700 con 0x7fdd4c0c6a20 2023-10-16T20:39:21.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd43fff640 1 -- 172.21.15.88:0/1430827803 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdd3c0135d0 con 0x7fdd4c0c6a20 2023-10-16T20:39:21.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.385+0000 7fdd43fff640 1 --2- 172.21.15.88:0/1430827803 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdd2c0664a0 0x7fdd2c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.389+0000 7fdd43fff640 1 -- 172.21.15.88:0/1430827803 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fdd3c084860 con 0x7fdd4c0c6a20 2023-10-16T20:39:21.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.389+0000 7fdd4259c640 1 --2- 172.21.15.88:0/1430827803 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdd2c0664a0 0x7fdd2c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.389+0000 7fdd4259c640 1 --2- 172.21.15.88:0/1430827803 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdd2c0664a0 0x7fdd2c068960 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdd340ae000 tx=0x7fdd34009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:21.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.389+0000 7fdd43fff640 1 -- 172.21.15.88:0/1430827803 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdd3c0518a0 con 0x7fdd4c0c6a20 2023-10-16T20:39:21.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.545+0000 7fdd510c9640 1 -- 172.21.15.88:0/1430827803 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd4c0c9af0 con 0x7fdd4c0c6a20 2023-10-16T20:39:21.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.545+0000 7fdd43fff640 1 -- 172.21.15.88:0/1430827803 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18710 (secure 0 0 0) 0x7fdd3c054ec0 con 0x7fdd4c0c6a20 2023-10-16T20:39:21.554 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:21.554 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:37:37.392549+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:39:21.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.549+0000 7fdd510c9640 1 -- 172.21.15.88:0/1430827803 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdd2c0664a0 msgr2=0x7fdd2c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:21.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.549+0000 7fdd510c9640 1 --2- 172.21.15.88:0/1430827803 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdd2c0664a0 0x7fdd2c068960 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdd340ae000 tx=0x7fdd34009040 comp rx=0 tx=0).stop 2023-10-16T20:39:21.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.549+0000 7fdd510c9640 1 -- 172.21.15.88:0/1430827803 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd4c0c6a20 msgr2=0x7fdd4c1f5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:21.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.549+0000 7fdd510c9640 1 --2- 172.21.15.88:0/1430827803 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd4c0c6a20 0x7fdd4c1f5f90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00e980 tx=0x7fdd3c00ee40 comp rx=0 tx=0).stop 2023-10-16T20:39:21.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.549+0000 7fdd510c9640 1 -- 172.21.15.88:0/1430827803 shutdown_connections 2023-10-16T20:39:21.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.549+0000 7fdd510c9640 1 --2- 172.21.15.88:0/1430827803 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdd2c0664a0 0x7fdd2c068960 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.549+0000 7fdd510c9640 1 --2- 172.21.15.88:0/1430827803 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdd4c153770 0x7fdd4c1f1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.549+0000 7fdd510c9640 1 --2- 172.21.15.88:0/1430827803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd4c0c9040 0x7fdd4c152410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.549+0000 7fdd510c9640 1 --2- 172.21.15.88:0/1430827803 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd4c0c6a20 0x7fdd4c1f5f90 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.549+0000 7fdd510c9640 1 -- 172.21.15.88:0/1430827803 >> 172.21.15.88:0/1430827803 conn(0x7fdd4c0c84e0 msgr2=0x7fdd4c14f840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:21.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.549+0000 7fdd510c9640 1 -- 172.21.15.88:0/1430827803 shutdown_connections 2023-10-16T20:39:21.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.549+0000 7fdd510c9640 1 -- 172.21.15.88:0/1430827803 wait complete. 2023-10-16T20:39:21.575 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-16T20:39:21.575 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:39:21.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.705+0000 7f0980217640 1 Processor -- start 2023-10-16T20:39:21.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.705+0000 7f0980217640 1 -- start start 2023-10-16T20:39:21.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.709+0000 7f0980217640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0978150280 0x7f09780c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.709+0000 7f0980217640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0978150660 0x7f09780c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.709+0000 7f0980217640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09780c5c60 con 0x7f09780c6bc0 2023-10-16T20:39:21.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.709+0000 7f0980217640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09780c5370 con 0x7f0978150280 2023-10-16T20:39:21.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.709+0000 7f0980217640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0978156d10 con 0x7f0978150660 2023-10-16T20:39:21.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.709+0000 7f097dfb3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0978150280 0x7f09780c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.709+0000 7f097e7b4640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f09780c6bc0 0x7f097806d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36900/0 (socket says 172.21.15.88:36900) 2023-10-16T20:39:21.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.709+0000 7f097dfb3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0978150280 0x7f09780c60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56278/0 (socket says 172.21.15.88:56278) 2023-10-16T20:39:21.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.709+0000 7f097e7b4640 1 -- 172.21.15.88:0/447290019 learned_addr learned my addr 172.21.15.88:0/447290019 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:21.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.709+0000 7f097d7b2640 1 --2- 172.21.15.88:0/447290019 >> v2:172.21.15.191:3300/0 conn(0x7f0978150660 0x7f09780c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.709+0000 7f097dfb3640 1 -- 172.21.15.88:0/447290019 >> v2:172.21.15.191:3300/0 conn(0x7f0978150660 msgr2=0x7f09780c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:21.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.709+0000 7f097dfb3640 1 --2- 172.21.15.88:0/447290019 >> v2:172.21.15.191:3300/0 conn(0x7f0978150660 0x7f09780c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.709+0000 7f097dfb3640 1 -- 172.21.15.88:0/447290019 >> v1:172.21.15.88:6789/0 conn(0x7f09780c6bc0 legacy=0x7f097806d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:21.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.709+0000 7f097dfb3640 1 -- 172.21.15.88:0/447290019 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09780c5610 con 0x7f0978150280 2023-10-16T20:39:21.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.709+0000 7f097cfb1640 1 -- 172.21.15.88:0/447290019 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352491806 0 0) 0x7f09780c5c60 con 0x7f09780c6bc0 2023-10-16T20:39:21.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.713+0000 7f097dfb3640 1 --2- 172.21.15.88:0/447290019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0978150280 0x7f09780c60f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f09780c1290 tx=0x7f097401ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b907ca5278cb7e55 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:21.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.713+0000 7f097cfb1640 1 -- 172.21.15.88:0/447290019 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09740ab650 con 0x7f0978150280 2023-10-16T20:39:21.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.713+0000 7f097cfb1640 1 -- 172.21.15.88:0/447290019 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09740ab7b0 con 0x7f0978150280 2023-10-16T20:39:21.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.713+0000 7f097cfb1640 1 -- 172.21.15.88:0/447290019 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09740abb40 con 0x7f0978150280 2023-10-16T20:39:21.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.713+0000 7f0980217640 1 -- 172.21.15.88:0/447290019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0978150280 msgr2=0x7f09780c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:21.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.713+0000 7f0980217640 1 --2- 172.21.15.88:0/447290019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0978150280 0x7f09780c60f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f09780c1290 tx=0x7f097401ce90 comp rx=0 tx=0).stop 2023-10-16T20:39:21.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.713+0000 7f0980217640 1 -- 172.21.15.88:0/447290019 shutdown_connections 2023-10-16T20:39:21.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.713+0000 7f0980217640 1 --2- 172.21.15.88:0/447290019 >> v2:172.21.15.191:3300/0 conn(0x7f0978150660 0x7f09780c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.713+0000 7f0980217640 1 --2- 172.21.15.88:0/447290019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0978150280 0x7f09780c60f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.713+0000 7f0980217640 1 -- 172.21.15.88:0/447290019 >> 172.21.15.88:0/447290019 conn(0x7f09780cad50 msgr2=0x7f09780cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:21.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.713+0000 7f0980217640 1 -- 172.21.15.88:0/447290019 shutdown_connections 2023-10-16T20:39:21.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.713+0000 7f0980217640 1 -- 172.21.15.88:0/447290019 wait complete. 2023-10-16T20:39:21.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.713+0000 7f0980217640 1 Processor -- start 2023-10-16T20:39:21.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.713+0000 7f0980217640 1 -- start start 2023-10-16T20:39:21.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f0980217640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780c6bc0 0x7f097814c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f0980217640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0978150280 0x7f09781430f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f0980217640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0978150660 0x7f0978143630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f0980217640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0978142300 con 0x7f09780c6bc0 2023-10-16T20:39:21.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f0980217640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097814e310 con 0x7f0978150280 2023-10-16T20:39:21.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f0980217640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097814e5f0 con 0x7f0978150660 2023-10-16T20:39:21.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f097dfb3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780c6bc0 0x7f097814c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f097dfb3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780c6bc0 0x7f097814c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56282/0 (socket says 172.21.15.88:56282) 2023-10-16T20:39:21.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f097dfb3640 1 -- 172.21.15.88:0/355434778 learned_addr learned my addr 172.21.15.88:0/355434778 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:21.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f097e7b4640 1 --2- 172.21.15.88:0/355434778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0978150660 0x7f0978143630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f097d7b2640 1 --2- 172.21.15.88:0/355434778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0978150280 0x7f09781430f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f097dfb3640 1 -- 172.21.15.88:0/355434778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0978150660 msgr2=0x7f0978143630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:21.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f097dfb3640 1 --2- 172.21.15.88:0/355434778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0978150660 0x7f0978143630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f097dfb3640 1 -- 172.21.15.88:0/355434778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0978150280 msgr2=0x7f09781430f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:21.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f097dfb3640 1 --2- 172.21.15.88:0/355434778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0978150280 0x7f09781430f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f097dfb3640 1 -- 172.21.15.88:0/355434778 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0974009420 con 0x7f09780c6bc0 2023-10-16T20:39:21.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f097dfb3640 1 --2- 172.21.15.88:0/355434778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780c6bc0 0x7f097814c930 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f09740ad040 tx=0x7f097401ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:21.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f096e7fc640 1 -- 172.21.15.88:0/355434778 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09740b5810 con 0x7f09780c6bc0 2023-10-16T20:39:21.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f096e7fc640 1 -- 172.21.15.88:0/355434778 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09740b5970 con 0x7f09780c6bc0 2023-10-16T20:39:21.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f0980217640 1 -- 172.21.15.88:0/355434778 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f097814e8d0 con 0x7f09780c6bc0 2023-10-16T20:39:21.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f096e7fc640 1 -- 172.21.15.88:0/355434778 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09740b5d00 con 0x7f09780c6bc0 2023-10-16T20:39:21.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f0980217640 1 -- 172.21.15.88:0/355434778 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09781425e0 con 0x7f09780c6bc0 2023-10-16T20:39:21.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f0980217640 1 -- 172.21.15.88:0/355434778 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0978074700 con 0x7f09780c6bc0 2023-10-16T20:39:21.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.717+0000 7f096e7fc640 1 -- 172.21.15.88:0/355434778 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f09740c83f0 con 0x7f09780c6bc0 2023-10-16T20:39:21.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.721+0000 7f096e7fc640 1 --2- 172.21.15.88:0/355434778 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f095c0664d0 0x7f095c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:21.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.721+0000 7f096e7fc640 1 -- 172.21.15.88:0/355434778 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f097411f430 con 0x7f09780c6bc0 2023-10-16T20:39:21.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.721+0000 7f097d7b2640 1 --2- 172.21.15.88:0/355434778 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f095c0664d0 0x7f095c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:21.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.721+0000 7f097d7b2640 1 --2- 172.21.15.88:0/355434778 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f095c0664d0 0x7f095c068990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f09780725a0 tx=0x7f0968008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:21.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.725+0000 7f096e7fc640 1 -- 172.21.15.88:0/355434778 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f09740ec6b0 con 0x7f09780c6bc0 2023-10-16T20:39:21.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.873+0000 7f0980217640 1 -- 172.21.15.88:0/355434778 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0978142da0 con 0x7f09780c6bc0 2023-10-16T20:39:21.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.877+0000 7f096e7fc640 1 -- 172.21.15.88:0/355434778 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18710 (secure 0 0 0) 0x7f09740b5e60 con 0x7f09780c6bc0 2023-10-16T20:39:21.883 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:21.884 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:37:37.392549+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:39:21.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.877+0000 7f0980217640 1 -- 172.21.15.88:0/355434778 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f095c0664d0 msgr2=0x7f095c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:21.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.877+0000 7f0980217640 1 --2- 172.21.15.88:0/355434778 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f095c0664d0 0x7f095c068990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f09780725a0 tx=0x7f0968008040 comp rx=0 tx=0).stop 2023-10-16T20:39:21.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.877+0000 7f0980217640 1 -- 172.21.15.88:0/355434778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780c6bc0 msgr2=0x7f097814c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:21.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.877+0000 7f0980217640 1 --2- 172.21.15.88:0/355434778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780c6bc0 0x7f097814c930 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f09740ad040 tx=0x7f097401ac70 comp rx=0 tx=0).stop 2023-10-16T20:39:21.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.877+0000 7f0980217640 1 -- 172.21.15.88:0/355434778 shutdown_connections 2023-10-16T20:39:21.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.877+0000 7f0980217640 1 --2- 172.21.15.88:0/355434778 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f095c0664d0 0x7f095c068990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.877+0000 7f0980217640 1 --2- 172.21.15.88:0/355434778 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0978150660 0x7f0978143630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.877+0000 7f0980217640 1 --2- 172.21.15.88:0/355434778 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0978150280 0x7f09781430f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.877+0000 7f0980217640 1 --2- 172.21.15.88:0/355434778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780c6bc0 0x7f097814c930 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:21.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.877+0000 7f0980217640 1 -- 172.21.15.88:0/355434778 >> 172.21.15.88:0/355434778 conn(0x7f09780cad50 msgr2=0x7f097806d200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:21.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.877+0000 7f0980217640 1 -- 172.21.15.88:0/355434778 shutdown_connections 2023-10-16T20:39:21.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:21.877+0000 7f0980217640 1 -- 172.21.15.88:0/355434778 wait complete. 2023-10-16T20:39:21.904 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-16T20:39:21.905 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-16T20:39:21.906 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-16T20:39:21.906 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-16T20:39:21.907 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-16T20:39:21.907 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-16T20:39:21.908 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-16T20:39:21.908 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-16T20:39:22.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.137+0000 7ff8fd796640 1 Processor -- start 2023-10-16T20:39:22.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.137+0000 7ff8fd796640 1 -- start start 2023-10-16T20:39:22.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7f3b7a606640 1 Processor -- start 2023-10-16T20:39:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7f3b7a606640 1 -- start start 2023-10-16T20:39:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7ff8fd796640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff8f00dade0 0x7ff8f00e6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7ff8f77fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff8f0005ff0 0x7ff8f004b1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36902/0 (socket says 172.21.15.88:36902) 2023-10-16T20:39:22.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7ff8f77fe640 1 -- 172.21.15.88:0/2852215865 learned_addr learned my addr 172.21.15.88:0/2852215865 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:22.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7ff8fd796640 1 -- 172.21.15.88:0/2852215865 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f0007400 con 0x7ff8f0005ff0 2023-10-16T20:39:22.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7ff8fd796640 1 -- 172.21.15.88:0/2852215865 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f0006530 con 0x7ff8f00d1760 2023-10-16T20:39:22.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7ff8fd796640 1 -- 172.21.15.88:0/2852215865 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f0007890 con 0x7ff8f00dade0 2023-10-16T20:39:22.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7f3b7a606640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3b740bcc80 0x7f3b7406a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7f3b7a606640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b74069e90 con 0x7f3b741537b0 2023-10-16T20:39:22.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7ff8fcf95640 1 --2- 172.21.15.88:0/2852215865 >> v2:172.21.15.88:3300/0 conn(0x7ff8f00dade0 0x7ff8f00e6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7f3b7a606640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b7406a450 con 0x7f3b740c5bb0 2023-10-16T20:39:22.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7f3b7a606640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b7406ae20 con 0x7f3b740bcc80 2023-10-16T20:39:22.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7f3b78e03640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3b740bcc80 0x7f3b7406a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7f3b79e05640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f3b740c5bb0 0x7f3b7406d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:40052/0 (socket says 172.21.15.88:40052) 2023-10-16T20:39:22.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7f3b79e05640 1 -- 172.21.15.88:0/3783842225 learned_addr learned my addr 172.21.15.88:0/3783842225 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:22.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7ff8f6ffd640 1 -- 172.21.15.88:0/2852215865 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291617612 0 0) 0x7ff8f0006530 con 0x7ff8f00d1760 2023-10-16T20:39:22.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7ff8f6ffd640 1 -- 172.21.15.88:0/2852215865 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8e8003530 con 0x7ff8f00d1760 2023-10-16T20:39:22.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8f6ffd640 1 -- 172.21.15.88:0/2852215865 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3288314358 0 0) 0x7ff8e8003530 con 0x7ff8f00d1760 2023-10-16T20:39:22.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7f3b6bfff640 1 -- 172.21.15.88:0/3783842225 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21949426 0 0) 0x7f3b7406a450 con 0x7f3b740c5bb0 2023-10-16T20:39:22.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.141+0000 7f3b6bfff640 1 -- 172.21.15.88:0/3783842225 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b60003530 con 0x7f3b740c5bb0 2023-10-16T20:39:22.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8f6ffd640 1 -- 172.21.15.88:0/2852215865 >> v2:172.21.15.88:3300/0 conn(0x7ff8f00dade0 msgr2=0x7ff8f00e6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8f6ffd640 1 --2- 172.21.15.88:0/2852215865 >> v2:172.21.15.88:3300/0 conn(0x7ff8f00dade0 0x7ff8f00e6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8f6ffd640 1 -- 172.21.15.88:0/2852215865 >> v1:172.21.15.88:6789/0 conn(0x7ff8f0005ff0 legacy=0x7ff8f004b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8f6ffd640 1 -- 172.21.15.88:0/2852215865 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f00067f0 con 0x7ff8f00d1760 2023-10-16T20:39:22.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8f6ffd640 1 -- 172.21.15.88:0/2852215865 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7ff8f8068d50 con 0x7ff8f00d1760 2023-10-16T20:39:22.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8f6ffd640 1 -- 172.21.15.88:0/2852215865 >> v1:172.21.15.191:6789/0 conn(0x7ff8f00d1760 legacy=0x7ff8f0045280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8f6ffd640 1 --2- 172.21.15.88:0/2852215865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8e8003980 0x7ff8e8003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8f6ffd640 1 --2- 172.21.15.88:0/2852215865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff8e8004300 0x7ff8e80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8f6ffd640 1 --2- 172.21.15.88:0/2852215865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8e8004f80 0x7ff8e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8f6ffd640 1 -- 172.21.15.88:0/2852215865 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f0007890 con 0x7ff8e8003980 2023-10-16T20:39:22.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8f6ffd640 1 -- 172.21.15.88:0/2852215865 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e800bec0 con 0x7ff8e8004f80 2023-10-16T20:39:22.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8f6ffd640 1 -- 172.21.15.88:0/2852215865 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e800c080 con 0x7ff8e8004300 2023-10-16T20:39:22.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7f3b78e03640 1 -- 172.21.15.88:0/3783842225 >> v1:172.21.15.191:6789/0 conn(0x7f3b740c5bb0 legacy=0x7f3b7406d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7f3b78e03640 1 -- 172.21.15.88:0/3783842225 >> v1:172.21.15.88:6789/0 conn(0x7f3b741537b0 legacy=0x7f3b74159f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7f3b78e03640 1 -- 172.21.15.88:0/3783842225 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b740bc2e0 con 0x7f3b740bcc80 2023-10-16T20:39:22.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8fcf95640 1 --2- 172.21.15.88:0/2852215865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8e8004f80 0x7ff8e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8f77fe640 1 --2- 172.21.15.88:0/2852215865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff8e8004300 0x7ff8e80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8fcf95640 1 -- 172.21.15.88:0/2852215865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff8e8004300 msgr2=0x7ff8e80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8fcf95640 1 --2- 172.21.15.88:0/2852215865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff8e8004300 0x7ff8e80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8fcf95640 1 -- 172.21.15.88:0/2852215865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8e8003980 msgr2=0x7ff8e8003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8fcf95640 1 --2- 172.21.15.88:0/2852215865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8e8003980 0x7ff8e8003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.145+0000 7ff8fcf95640 1 -- 172.21.15.88:0/2852215865 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f00067f0 con 0x7ff8e8004f80 2023-10-16T20:39:22.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.149+0000 7ff8fcf95640 1 -- 172.21.15.88:0/2852215865 reap_dead start 2023-10-16T20:39:22.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.149+0000 7f01a58ea640 1 Processor -- start 2023-10-16T20:39:22.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.149+0000 7f3b78e03640 1 --2- 172.21.15.88:0/3783842225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b740bcc80 0x7f3b7406a8e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3b700097c0 tx=0x7f3b7001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c98612f9dc8bb65 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.149+0000 7ff8fcf95640 1 --2- 172.21.15.88:0/2852215865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8e8004f80 0x7ff8e800b980 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4009ae0 tx=0x7ff8e401ca90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8d5b1db27cb34ad server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.149+0000 7f3b6bfff640 1 -- 172.21.15.88:0/3783842225 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b700ab610 con 0x7f3b740bcc80 2023-10-16T20:39:22.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.149+0000 7f3b6bfff640 1 -- 172.21.15.88:0/3783842225 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b700ab770 con 0x7f3b740bcc80 2023-10-16T20:39:22.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.149+0000 7f3b6bfff640 1 -- 172.21.15.88:0/3783842225 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b700abb00 con 0x7f3b740bcc80 2023-10-16T20:39:22.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.149+0000 7f01a58ea640 1 -- start start 2023-10-16T20:39:22.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.149+0000 7ff8f6ffd640 1 -- 172.21.15.88:0/2852215865 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4004020 con 0x7ff8e8004f80 2023-10-16T20:39:22.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.149+0000 7ff8f6ffd640 1 -- 172.21.15.88:0/2852215865 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e4004180 con 0x7ff8e8004f80 2023-10-16T20:39:22.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.149+0000 7ff8f6ffd640 1 -- 172.21.15.88:0/2852215865 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4003620 con 0x7ff8e8004f80 2023-10-16T20:39:22.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7ff8fd796640 1 -- 172.21.15.88:0/2852215865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8e8004f80 msgr2=0x7ff8e800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7ff8fd796640 1 --2- 172.21.15.88:0/2852215865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8e8004f80 0x7ff8e800b980 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4009ae0 tx=0x7ff8e401ca90 comp rx=0 tx=0).stop 2023-10-16T20:39:22.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f3b7a606640 1 -- 172.21.15.88:0/3783842225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b740bcc80 msgr2=0x7f3b7406a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f3b7a606640 1 --2- 172.21.15.88:0/3783842225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b740bcc80 0x7f3b7406a8e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3b700097c0 tx=0x7f3b7001c410 comp rx=0 tx=0).stop 2023-10-16T20:39:22.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f01a58ea640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f01a0153780 0x7f01a00c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f01a58ea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f01a0159490 0x7f01a00c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7ff8fd796640 1 -- 172.21.15.88:0/2852215865 shutdown_connections 2023-10-16T20:39:22.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7ff8fd796640 1 --2- 172.21.15.88:0/2852215865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8e8004f80 0x7ff8e800b980 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7ff8fd796640 1 -- 172.21.15.88:0/2852215865 >> 172.21.15.88:0/2852215865 conn(0x7ff8f00502e0 msgr2=0x7ff8f00506f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f01a58ea640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a0159d90 con 0x7f01a00c5930 2023-10-16T20:39:22.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f01a58ea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a015a070 con 0x7f01a0159490 2023-10-16T20:39:22.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f01a58ea640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a015a350 con 0x7f01a0153780 2023-10-16T20:39:22.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019f7fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f01a00c5930 0x7f01a00ced80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:40066/0 (socket says 172.21.15.88:40066) 2023-10-16T20:39:22.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019f7fe640 1 -- 172.21.15.88:0/1039552163 learned_addr learned my addr 172.21.15.88:0/1039552163 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:22.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019effd640 1 --2- 172.21.15.88:0/1039552163 >> v2:172.21.15.191:3300/0 conn(0x7f01a0153780 0x7f01a00c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f3b7a606640 1 -- 172.21.15.88:0/3783842225 shutdown_connections 2023-10-16T20:39:22.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f3b7a606640 1 --2- 172.21.15.88:0/3783842225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b740bcc80 0x7f3b7406a8e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f3b7a606640 1 -- 172.21.15.88:0/3783842225 >> 172.21.15.88:0/3783842225 conn(0x7f3b740caee0 msgr2=0x7f3b740cb2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019e7fc640 1 --2- 172.21.15.88:0/1039552163 >> v2:172.21.15.88:3300/0 conn(0x7f01a0159490 0x7f01a00c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019dffb640 1 -- 172.21.15.88:0/1039552163 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3915310279 0 0) 0x7f01a0159d90 con 0x7f01a00c5930 2023-10-16T20:39:22.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019dffb640 1 -- 172.21.15.88:0/1039552163 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0188003530 con 0x7f01a00c5930 2023-10-16T20:39:22.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019dffb640 1 -- 172.21.15.88:0/1039552163 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 467462317 0 0) 0x7f0188003530 con 0x7f01a00c5930 2023-10-16T20:39:22.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019dffb640 1 -- 172.21.15.88:0/1039552163 >> v2:172.21.15.191:3300/0 conn(0x7f01a0153780 msgr2=0x7f01a00c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019dffb640 1 --2- 172.21.15.88:0/1039552163 >> v2:172.21.15.191:3300/0 conn(0x7f01a0153780 0x7f01a00c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019dffb640 1 -- 172.21.15.88:0/1039552163 >> v2:172.21.15.88:3300/0 conn(0x7f01a0159490 msgr2=0x7f01a00c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019dffb640 1 --2- 172.21.15.88:0/1039552163 >> v2:172.21.15.88:3300/0 conn(0x7f01a0159490 0x7f01a00c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019dffb640 1 -- 172.21.15.88:0/1039552163 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a00c4910 con 0x7f01a00c5930 2023-10-16T20:39:22.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019dffb640 1 -- 172.21.15.88:0/1039552163 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f0198001a20 con 0x7f01a00c5930 2023-10-16T20:39:22.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019dffb640 1 -- 172.21.15.88:0/1039552163 >> v1:172.21.15.191:6789/0 conn(0x7f01a00c5930 legacy=0x7f01a00ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019dffb640 1 --2- 172.21.15.88:0/1039552163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0188003980 0x7f0188003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019dffb640 1 --2- 172.21.15.88:0/1039552163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01880047d0 0x7f0188004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019dffb640 1 --2- 172.21.15.88:0/1039552163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0188005190 0x7f018800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019dffb640 1 -- 172.21.15.88:0/1039552163 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a015a070 con 0x7f0188003980 2023-10-16T20:39:22.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019dffb640 1 -- 172.21.15.88:0/1039552163 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a015a350 con 0x7f01880047d0 2023-10-16T20:39:22.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019dffb640 1 -- 172.21.15.88:0/1039552163 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018800bf00 con 0x7f0188005190 2023-10-16T20:39:22.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019e7fc640 1 --2- 172.21.15.88:0/1039552163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01880047d0 0x7f0188004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f3b7a606640 1 -- 172.21.15.88:0/3783842225 shutdown_connections 2023-10-16T20:39:22.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7f019f7fe640 1 --2- 172.21.15.88:0/1039552163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0188005190 0x7f018800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7f019effd640 1 --2- 172.21.15.88:0/1039552163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0188003980 0x7f0188003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7f019f7fe640 1 -- 172.21.15.88:0/1039552163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01880047d0 msgr2=0x7f0188004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7f019f7fe640 1 --2- 172.21.15.88:0/1039552163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01880047d0 0x7f0188004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7f019f7fe640 1 -- 172.21.15.88:0/1039552163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0188003980 msgr2=0x7f0188003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7f019f7fe640 1 --2- 172.21.15.88:0/1039552163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0188003980 0x7f0188003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7f019f7fe640 1 -- 172.21.15.88:0/1039552163 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0188003530 con 0x7f0188005190 2023-10-16T20:39:22.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7ff8fd796640 1 -- 172.21.15.88:0/2852215865 shutdown_connections 2023-10-16T20:39:22.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.153+0000 7ff8fd796640 1 -- 172.21.15.88:0/2852215865 wait complete. 2023-10-16T20:39:22.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7f3b7a606640 1 -- 172.21.15.88:0/3783842225 wait complete. 2023-10-16T20:39:22.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7ff8fd796640 1 Processor -- start 2023-10-16T20:39:22.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7ff8fd796640 1 -- start start 2023-10-16T20:39:22.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7f3b7a606640 1 Processor -- start 2023-10-16T20:39:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7f81ceb72640 1 Processor -- start 2023-10-16T20:39:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7ff4ebc9a640 1 Processor -- start 2023-10-16T20:39:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7f81ceb72640 1 -- start start 2023-10-16T20:39:22.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7ff8fd796640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff8e8004f80 0x7ff8f00d46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7f3eeec9c640 1 Processor -- start 2023-10-16T20:39:22.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7ff8fd796640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8f00d6340 0x7ff8f00d3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7ff4ebc9a640 1 -- start start 2023-10-16T20:39:22.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7ff8fd796640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8f004ed60 0x7ff8f004f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7ff8fd796640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00d5b90 con 0x7ff8f00d6340 2023-10-16T20:39:22.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8fd796640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f004f750 con 0x7ff8f004ed60 2023-10-16T20:39:22.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8fd796640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00d5f60 con 0x7ff8e8004f80 2023-10-16T20:39:22.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.157+0000 7f3eeec9c640 1 -- start start 2023-10-16T20:39:22.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8fcf95640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8f004ed60 0x7ff8f004f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8fcf95640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8f004ed60 0x7ff8f004f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:40104/0 (socket says 172.21.15.88:40104) 2023-10-16T20:39:22.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8fcf95640 1 -- 172.21.15.88:0/2793680559 learned_addr learned my addr 172.21.15.88:0/2793680559 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8f77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8f00d6340 0x7ff8f00d3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f019f7fe640 1 -- 172.21.15.88:0/1039552163 reap_dead start 2023-10-16T20:39:22.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8fcf95640 1 -- 172.21.15.88:0/2793680559 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff8e8004f80 msgr2=0x7ff8f00d46c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T20:39:22.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8fcf95640 1 --2- 172.21.15.88:0/2793680559 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff8e8004f80 0x7ff8f00d46c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8fcf95640 1 -- 172.21.15.88:0/2793680559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8f00d6340 msgr2=0x7ff8f00d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f81ceb72640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81c8159490 0x7f81c80c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f81ceb72640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c8159d90 con 0x7f81c8153780 2023-10-16T20:39:22.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f81ceb72640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c815a070 con 0x7f81c80c5930 2023-10-16T20:39:22.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f81ceb72640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c815a350 con 0x7f81c8159490 2023-10-16T20:39:22.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff4ebc9a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff4e406a4d0 0x7ff4e4153410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff4ebc9a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff4e406ad20 0x7ff4e4153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff4ebc9a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e415cd00 con 0x7ff4e406a0f0 2023-10-16T20:39:22.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff4ebc9a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e415d980 con 0x7ff4e406ad20 2023-10-16T20:39:22.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff4ebc9a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e40c2000 con 0x7ff4e406a4d0 2023-10-16T20:39:22.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff4e9235640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff4e406a4d0 0x7ff4e4153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f3b7a606640 1 -- start start 2023-10-16T20:39:22.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f019f7fe640 1 --2- 172.21.15.88:0/1039552163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0188005190 0x7f018800b990 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f01a00c98d0 tx=0x7f0198028270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d3ec9a0d557818 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff4e9235640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff4e406a4d0 0x7ff4e4153410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:40108/0 (socket says 172.21.15.88:40108) 2023-10-16T20:39:22.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff4e9235640 1 -- 172.21.15.88:0/3725697539 learned_addr learned my addr 172.21.15.88:0/3725697539 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f81c7fff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81c8159490 0x7f81c80c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f81c7fff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81c8159490 0x7f81c80c53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:37572/0 (socket says 172.21.15.88:37572) 2023-10-16T20:39:22.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f81c7fff640 1 -- 172.21.15.88:0/2477366952 learned_addr learned my addr 172.21.15.88:0/2477366952 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f019dffb640 1 -- 172.21.15.88:0/1039552163 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01980b1e90 con 0x7f0188005190 2023-10-16T20:39:22.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8fcf95640 1 --2- 172.21.15.88:0/2793680559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8f00d6340 0x7ff8f00d3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8fcf95640 1 -- 172.21.15.88:0/2793680559 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f00d55f0 con 0x7ff8f004ed60 2023-10-16T20:39:22.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8f77fe640 1 --2- 172.21.15.88:0/2793680559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8f00d6340 0x7ff8f00d3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:22.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8fcf95640 1 --2- 172.21.15.88:0/2793680559 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8f004ed60 0x7ff8f004f210 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff8e401d330 tx=0x7ff8e4019830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8f4ff9640 1 -- 172.21.15.88:0/2793680559 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e40044f0 con 0x7ff8f004ed60 2023-10-16T20:39:22.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8f4ff9640 1 -- 172.21.15.88:0/2793680559 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e40b2040 con 0x7ff8f004ed60 2023-10-16T20:39:22.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8f4ff9640 1 -- 172.21.15.88:0/2793680559 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e40a5c70 con 0x7ff8f004ed60 2023-10-16T20:39:22.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8fd796640 1 -- 172.21.15.88:0/2793680559 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f00d58b0 con 0x7ff8f004ed60 2023-10-16T20:39:22.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f019dffb640 1 -- 172.21.15.88:0/1039552163 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01980070d0 con 0x7f0188005190 2023-10-16T20:39:22.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f019dffb640 1 -- 172.21.15.88:0/1039552163 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0198007480 con 0x7f0188005190 2023-10-16T20:39:22.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f3b7a606640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b740bcc80 0x7f3b74150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f3b7a606640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b740c5bb0 0x7f3b7414a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f3b7a606640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b741537b0 0x7f3b741f1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f3b7a606640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b741f5b60 con 0x7f3b741537b0 2023-10-16T20:39:22.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f3b7a606640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b741513c0 con 0x7f3b740bcc80 2023-10-16T20:39:22.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f3b7a606640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b741f4fe0 con 0x7f3b740c5bb0 2023-10-16T20:39:22.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff4ea237640 1 --2- 172.21.15.88:0/3725697539 >> v2:172.21.15.88:3300/0 conn(0x7ff4e406ad20 0x7ff4e4153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f3b78e03640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b740c5bb0 0x7f3b7414a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f3b78e03640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b740c5bb0 0x7f3b7414a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:37580/0 (socket says 172.21.15.88:37580) 2023-10-16T20:39:22.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f3b78e03640 1 -- 172.21.15.88:0/3768034351 learned_addr learned my addr 172.21.15.88:0/3768034351 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f3b79604640 1 --2- 172.21.15.88:0/3768034351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b740bcc80 0x7f3b74150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff4e9235640 1 -- 172.21.15.88:0/3725697539 >> v2:172.21.15.88:3300/0 conn(0x7ff4e406ad20 msgr2=0x7ff4e4153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff4e9235640 1 --2- 172.21.15.88:0/3725697539 >> v2:172.21.15.88:3300/0 conn(0x7ff4e406ad20 0x7ff4e4153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff4e9235640 1 -- 172.21.15.88:0/3725697539 >> v1:172.21.15.191:6789/0 conn(0x7ff4e406a0f0 legacy=0x7ff4e40c2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff4e9235640 1 -- 172.21.15.88:0/3725697539 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e415cfe0 con 0x7ff4e406a4d0 2023-10-16T20:39:22.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f81c7fff640 1 -- 172.21.15.88:0/2477366952 >> v1:172.21.15.191:6789/0 conn(0x7f81c80c5930 legacy=0x7f81c80ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f81c7fff640 1 -- 172.21.15.88:0/2477366952 >> v1:172.21.15.88:6789/0 conn(0x7f81c8153780 legacy=0x7f81c80c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f81c7fff640 1 -- 172.21.15.88:0/2477366952 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c80c4910 con 0x7f81c8159490 2023-10-16T20:39:22.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7f81c77fe640 1 -- 172.21.15.88:0/2477366952 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115783761 0 0) 0x7f81c815a070 con 0x7f81c80c5930 2023-10-16T20:39:22.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3b79604640 1 -- 172.21.15.88:0/3768034351 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b740c5bb0 msgr2=0x7f3b7414a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3b79e05640 1 --2- 172.21.15.88:0/3768034351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b741537b0 0x7f3b741f1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3b79604640 1 --2- 172.21.15.88:0/3768034351 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3b740c5bb0 0x7f3b7414a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3b79604640 1 -- 172.21.15.88:0/3768034351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b741537b0 msgr2=0x7f3b741f1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f01a58ea640 1 -- 172.21.15.88:0/1039552163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0188005190 msgr2=0x7f018800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f01a58ea640 1 --2- 172.21.15.88:0/1039552163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0188005190 0x7f018800b990 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f01a00c98d0 tx=0x7f0198028270 comp rx=0 tx=0).stop 2023-10-16T20:39:22.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff4ea237640 1 --2- 172.21.15.88:0/3725697539 >> v2:172.21.15.88:3300/0 conn(0x7ff4e406ad20 0x7ff4e4153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:22.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3b79604640 1 --2- 172.21.15.88:0/3768034351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b741537b0 0x7f3b741f1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3b79604640 1 -- 172.21.15.88:0/3768034351 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b70009420 con 0x7f3b740bcc80 2023-10-16T20:39:22.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f01a58ea640 1 -- 172.21.15.88:0/1039552163 shutdown_connections 2023-10-16T20:39:22.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f01a58ea640 1 --2- 172.21.15.88:0/1039552163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0188005190 0x7f018800b990 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f01a58ea640 1 -- 172.21.15.88:0/1039552163 >> 172.21.15.88:0/1039552163 conn(0x7f01a006bcb0 msgr2=0x7f01a006c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.161+0000 7ff8fd796640 1 -- 172.21.15.88:0/2793680559 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f004e560 con 0x7ff8f004ed60 2023-10-16T20:39:22.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f01a58ea640 1 -- 172.21.15.88:0/1039552163 shutdown_connections 2023-10-16T20:39:22.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f01a58ea640 1 -- 172.21.15.88:0/1039552163 wait complete. 2023-10-16T20:39:22.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3eeec9c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ee8159490 0x7f3ee80c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3eeec9c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee8159d90 con 0x7f3ee80c5930 2023-10-16T20:39:22.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3eeec9c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee815a070 con 0x7f3ee8153780 2023-10-16T20:39:22.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3eeec9c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee815a350 con 0x7f3ee8159490 2023-10-16T20:39:22.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3eeca38640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f3ee8153780 0x7f3ee80c9be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:40088/0 (socket says 172.21.15.88:40088) 2023-10-16T20:39:22.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3eeca38640 1 -- 172.21.15.88:0/3033946398 learned_addr learned my addr 172.21.15.88:0/3033946398 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:22.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3b79604640 1 --2- 172.21.15.88:0/3768034351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b740bcc80 0x7f3b74150e80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3b6400b410 tx=0x7f3b6400b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3ee7fff640 1 --2- 172.21.15.88:0/3033946398 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ee8159490 0x7f3ee80c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3ee77fe640 1 -- 172.21.15.88:0/3033946398 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234745333 0 0) 0x7f3ee815a070 con 0x7f3ee8153780 2023-10-16T20:39:22.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3ee77fe640 1 -- 172.21.15.88:0/3033946398 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed0003530 con 0x7f3ee8153780 2023-10-16T20:39:22.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7ff8e27fc640 1 -- 172.21.15.88:0/2793680559 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff8f0006db0 con 0x7ff8f004ed60 2023-10-16T20:39:22.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3b69ffb640 1 -- 172.21.15.88:0/3768034351 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b64004020 con 0x7f3b740bcc80 2023-10-16T20:39:22.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3b69ffb640 1 -- 172.21.15.88:0/3768034351 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b64004180 con 0x7f3b740bcc80 2023-10-16T20:39:22.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3b69ffb640 1 -- 172.21.15.88:0/3768034351 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b64011630 con 0x7f3b740bcc80 2023-10-16T20:39:22.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3b7a606640 1 -- 172.21.15.88:0/3768034351 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b741f52c0 con 0x7f3b740bcc80 2023-10-16T20:39:22.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3b7a606640 1 -- 172.21.15.88:0/3768034351 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b741f5880 con 0x7f3b740bcc80 2023-10-16T20:39:22.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3b7a606640 1 -- 172.21.15.88:0/3768034351 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3b74153b90 con 0x7f3b740bcc80 2023-10-16T20:39:22.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f01a58ea640 1 Processor -- start 2023-10-16T20:39:22.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7ff4e9235640 1 --2- 172.21.15.88:0/3725697539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4e406a4d0 0x7ff4e4153410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4067260 tx=0x7ff4d801c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5eb492c5c74f26fa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3ee7fff640 1 -- 172.21.15.88:0/3033946398 >> v1:172.21.15.191:6789/0 conn(0x7f3ee8153780 legacy=0x7f3ee80c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3ee7fff640 1 -- 172.21.15.88:0/3033946398 >> v1:172.21.15.88:6789/0 conn(0x7f3ee80c5930 legacy=0x7f3ee80ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.165+0000 7f3ee7fff640 1 -- 172.21.15.88:0/3033946398 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee80c4910 con 0x7f3ee8159490 2023-10-16T20:39:22.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff4e8a34640 1 -- 172.21.15.88:0/3725697539 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d80ac630 con 0x7ff4e406a4d0 2023-10-16T20:39:22.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff4e8a34640 1 -- 172.21.15.88:0/3725697539 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d80ac790 con 0x7ff4e406a4d0 2023-10-16T20:39:22.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff4e8a34640 1 -- 172.21.15.88:0/3725697539 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d80acb20 con 0x7ff4e406a4d0 2023-10-16T20:39:22.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f81c7fff640 1 --2- 172.21.15.88:0/2477366952 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81c8159490 0x7f81c80c53f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f81c0009e80 tx=0x7f81c001f1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30c8180dfe26d362 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff8f4ff9640 1 -- 172.21.15.88:0/2793680559 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff8e4003620 con 0x7ff8f004ed60 2023-10-16T20:39:22.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f01a58ea640 1 -- start start 2023-10-16T20:39:22.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f01a58ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0188005190 0x7f01a00b3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f01a58ea640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01a014da30 0x7f01a01eb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f01a58ea640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01a014e100 0x7f01a0140760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f01a58ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a01eaea0 con 0x7f0188005190 2023-10-16T20:39:22.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f01a58ea640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a0146950 con 0x7f01a014da30 2023-10-16T20:39:22.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f01a58ea640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a0146bf0 con 0x7f01a014e100 2023-10-16T20:39:22.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff4ebc9a640 1 -- 172.21.15.88:0/3725697539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4e406a4d0 msgr2=0x7ff4e4153410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff4ebc9a640 1 --2- 172.21.15.88:0/3725697539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4e406a4d0 0x7ff4e4153410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4067260 tx=0x7ff4d801c520 comp rx=0 tx=0).stop 2023-10-16T20:39:22.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f81c77fe640 1 -- 172.21.15.88:0/2477366952 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c00af400 con 0x7f81c8159490 2023-10-16T20:39:22.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f81c77fe640 1 -- 172.21.15.88:0/2477366952 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c00ab050 con 0x7f81c8159490 2023-10-16T20:39:22.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f81c77fe640 1 -- 172.21.15.88:0/2477366952 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c00af400 con 0x7f81c8159490 2023-10-16T20:39:22.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff8f4ff9640 1 --2- 172.21.15.88:0/2793680559 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff8dc06a9b0 0x7ff8dc06ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff8f4ff9640 1 -- 172.21.15.88:0/2793680559 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff8e411de70 con 0x7ff8f004ed60 2023-10-16T20:39:22.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f3b69ffb640 1 -- 172.21.15.88:0/3768034351 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3b64011790 con 0x7f3b740bcc80 2023-10-16T20:39:22.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff4ebc9a640 1 -- 172.21.15.88:0/3725697539 shutdown_connections 2023-10-16T20:39:22.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff4ebc9a640 1 --2- 172.21.15.88:0/3725697539 >> v2:172.21.15.88:3300/0 conn(0x7ff4e406ad20 0x7ff4e4153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff4ebc9a640 1 --2- 172.21.15.88:0/3725697539 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4e406a4d0 0x7ff4e4153410 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff4ebc9a640 1 -- 172.21.15.88:0/3725697539 >> 172.21.15.88:0/3725697539 conn(0x7ff4e40ca9f0 msgr2=0x7ff4e40cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff8f4ff9640 1 --2- 172.21.15.88:0/2793680559 >> [v2:172.21.15.88:6800/1130215823,v1:172.21.15.88:6801/1130215823] conn(0x7ff8dc074310 0x7ff8dc0767b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff8f4ff9640 1 -- 172.21.15.88:0/2793680559 --> [v2:172.21.15.88:6800/1130215823,v1:172.21.15.88:6801/1130215823] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff8dc076e80 con 0x7ff8dc074310 2023-10-16T20:39:22.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff8f4ff9640 1 -- 172.21.15.88:0/2793680559 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8e411e220 con 0x7ff8f004ed60 2023-10-16T20:39:22.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff4ebc9a640 1 -- 172.21.15.88:0/3725697539 shutdown_connections 2023-10-16T20:39:22.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff8f77fe640 1 --2- 172.21.15.88:0/2793680559 >> [v2:172.21.15.88:6800/1130215823,v1:172.21.15.88:6801/1130215823] conn(0x7ff8dc074310 0x7ff8dc0767b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff8f7fff640 1 --2- 172.21.15.88:0/2793680559 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff8dc06a9b0 0x7ff8dc06ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f81ceb72640 1 -- 172.21.15.88:0/2477366952 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81c8159490 msgr2=0x7f81c80c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f81ceb72640 1 --2- 172.21.15.88:0/2477366952 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81c8159490 0x7f81c80c53f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f81c0009e80 tx=0x7f81c001f1e0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f019effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0188005190 0x7f01a00b3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f019effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0188005190 0x7f01a00b3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56366/0 (socket says 172.21.15.88:56366) 2023-10-16T20:39:22.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f019effd640 1 -- 172.21.15.88:0/3186187548 learned_addr learned my addr 172.21.15.88:0/3186187548 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7ff4ebc9a640 1 -- 172.21.15.88:0/3725697539 wait complete. 2023-10-16T20:39:22.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f3ee7fff640 1 --2- 172.21.15.88:0/3033946398 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ee8159490 0x7f3ee80c53f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8009dc0 tx=0x7f3ed801f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfbb8155ff65ca6e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f3b69ffb640 1 --2- 172.21.15.88:0/3768034351 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3b58066570 0x7f3b58068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f81ceb72640 1 -- 172.21.15.88:0/2477366952 shutdown_connections 2023-10-16T20:39:22.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f81ceb72640 1 --2- 172.21.15.88:0/2477366952 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81c8159490 0x7f81c80c53f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f019f7fe640 1 --2- 172.21.15.88:0/3186187548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01a014e100 0x7f01a0140760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f019e7fc640 1 --2- 172.21.15.88:0/3186187548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01a014da30 0x7f01a01eb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f81ceb72640 1 -- 172.21.15.88:0/2477366952 >> 172.21.15.88:0/2477366952 conn(0x7f81c806bcb0 msgr2=0x7f81c806c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.169+0000 7f3b69ffb640 1 -- 172.21.15.88:0/3768034351 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3b640847e0 con 0x7f3b740bcc80 2023-10-16T20:39:22.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f3b69ffb640 1 --2- 172.21.15.88:0/3768034351 >> [v2:172.21.15.88:6804/3386972440,v1:172.21.15.88:6809/3386972440] conn(0x7f3b5806fed0 0x7f3b58072370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f3b69ffb640 1 -- 172.21.15.88:0/3768034351 --> [v2:172.21.15.88:6804/3386972440,v1:172.21.15.88:6809/3386972440] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b58072a40 con 0x7f3b5806fed0 2023-10-16T20:39:22.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f3b69ffb640 1 -- 172.21.15.88:0/3768034351 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b64084b90 con 0x7f3b740bcc80 2023-10-16T20:39:22.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7ff8f77fe640 1 --2- 172.21.15.88:0/2793680559 >> [v2:172.21.15.88:6800/1130215823,v1:172.21.15.88:6801/1130215823] conn(0x7ff8dc074310 0x7ff8dc0767b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f019effd640 1 -- 172.21.15.88:0/3186187548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01a014e100 msgr2=0x7f01a0140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f019effd640 1 --2- 172.21.15.88:0/3186187548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01a014e100 0x7f01a0140760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f019effd640 1 -- 172.21.15.88:0/3186187548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01a014da30 msgr2=0x7f01a01eb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f019effd640 1 --2- 172.21.15.88:0/3186187548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01a014da30 0x7f01a01eb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f019effd640 1 -- 172.21.15.88:0/3186187548 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a0146ed0 con 0x7f0188005190 2023-10-16T20:39:22.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f3ee77fe640 1 -- 172.21.15.88:0/3033946398 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed80af600 con 0x7f3ee8159490 2023-10-16T20:39:22.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f3ee77fe640 1 -- 172.21.15.88:0/3033946398 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed80ab050 con 0x7f3ee8159490 2023-10-16T20:39:22.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f3ee77fe640 1 -- 172.21.15.88:0/3033946398 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed80af980 con 0x7f3ee8159490 2023-10-16T20:39:22.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f019e7fc640 1 --2- 172.21.15.88:0/3186187548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01a014da30 0x7f01a01eb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:39:22.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f3b79e05640 1 --2- 172.21.15.88:0/3768034351 >> [v2:172.21.15.88:6804/3386972440,v1:172.21.15.88:6809/3386972440] conn(0x7f3b5806fed0 0x7f3b58072370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f3b78e03640 1 --2- 172.21.15.88:0/3768034351 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3b58066570 0x7f3b58068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f019effd640 1 --2- 172.21.15.88:0/3186187548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0188005190 0x7f01a00b3000 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0190011590 tx=0x7f0190011990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7ff8f4ff9640 1 -- 172.21.15.88:0/2793680559 <== osd.0 v2:172.21.15.88:6800/1130215823 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7ff8dc076e80 con 0x7ff8dc074310 2023-10-16T20:39:22.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f81ceb72640 1 -- 172.21.15.88:0/2477366952 shutdown_connections 2023-10-16T20:39:22.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f81ceb72640 1 -- 172.21.15.88:0/2477366952 wait complete. 2023-10-16T20:39:22.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f0187fff640 1 -- 172.21.15.88:0/3186187548 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0190015870 con 0x7f0188005190 2023-10-16T20:39:22.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f0187fff640 1 -- 172.21.15.88:0/3186187548 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01900159d0 con 0x7f0188005190 2023-10-16T20:39:22.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f0187fff640 1 -- 172.21.15.88:0/3186187548 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0190015da0 con 0x7f0188005190 2023-10-16T20:39:22.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f3eeec9c640 1 -- 172.21.15.88:0/3033946398 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ee8159490 msgr2=0x7f3ee80c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f3eeec9c640 1 --2- 172.21.15.88:0/3033946398 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ee8159490 0x7f3ee80c53f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8009dc0 tx=0x7f3ed801f150 comp rx=0 tx=0).stop 2023-10-16T20:39:22.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f3b79e05640 1 --2- 172.21.15.88:0/3768034351 >> [v2:172.21.15.88:6804/3386972440,v1:172.21.15.88:6809/3386972440] conn(0x7f3b5806fed0 0x7f3b58072370 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f3b78e03640 1 --2- 172.21.15.88:0/3768034351 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3b58066570 0x7f3b58068a30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3b7001c450 tx=0x7f3b700a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7ff4ebc9a640 1 Processor -- start 2023-10-16T20:39:22.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f01a58ea640 1 -- 172.21.15.88:0/3186187548 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a01eaac0 con 0x7f0188005190 2023-10-16T20:39:22.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f01a58ea640 1 -- 172.21.15.88:0/3186187548 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a01eed70 con 0x7f0188005190 2023-10-16T20:39:22.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7ff8f7fff640 1 --2- 172.21.15.88:0/2793680559 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff8dc06a9b0 0x7ff8dc06ce70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff8f804fb10 tx=0x7ff8f8068de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f3eeec9c640 1 -- 172.21.15.88:0/3033946398 shutdown_connections 2023-10-16T20:39:22.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f3eeec9c640 1 --2- 172.21.15.88:0/3033946398 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ee8159490 0x7f3ee80c53f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f3eeec9c640 1 -- 172.21.15.88:0/3033946398 >> 172.21.15.88:0/3033946398 conn(0x7f3ee806bcb0 msgr2=0x7f3ee806c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.173+0000 7f01a58ea640 1 -- 172.21.15.88:0/3186187548 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f01a00c4ed0 con 0x7f0188005190 2023-10-16T20:39:22.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f81ceb72640 1 Processor -- start 2023-10-16T20:39:22.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4ebc9a640 1 -- start start 2023-10-16T20:39:22.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f3eeec9c640 1 -- 172.21.15.88:0/3033946398 shutdown_connections 2023-10-16T20:39:22.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f3b69ffb640 1 -- 172.21.15.88:0/3768034351 <== osd.1 v2:172.21.15.88:6804/3386972440 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f3b58072a40 con 0x7f3b5806fed0 2023-10-16T20:39:22.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f3eeec9c640 1 -- 172.21.15.88:0/3033946398 wait complete. 2023-10-16T20:39:22.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f62ff9f1640 1 Processor -- start 2023-10-16T20:39:22.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f81ceb72640 1 -- start start 2023-10-16T20:39:22.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f3401eea640 1 Processor -- start 2023-10-16T20:39:22.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f62ff9f1640 1 -- start start 2023-10-16T20:39:22.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f0187fff640 1 -- 172.21.15.88:0/3186187548 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0190013400 con 0x7f0188005190 2023-10-16T20:39:22.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f3401eea640 1 -- start start 2023-10-16T20:39:22.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4ebc9a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4e406a0f0 0x7ff4e4151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4ebc9a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4e406a4d0 0x7ff4e4152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4ebc9a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4e406ad20 0x7ff4e41f5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4ebc9a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e4157e80 con 0x7ff4e406a4d0 2023-10-16T20:39:22.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4ebc9a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e414c950 con 0x7ff4e406a0f0 2023-10-16T20:39:22.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4ebc9a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e414cef0 con 0x7ff4e406ad20 2023-10-16T20:39:22.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4e9a36640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4e406a0f0 0x7ff4e4151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4e9a36640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4e406a0f0 0x7ff4e4151fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:40138/0 (socket says 172.21.15.88:40138) 2023-10-16T20:39:22.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4e9a36640 1 -- 172.21.15.88:0/2722262418 learned_addr learned my addr 172.21.15.88:0/2722262418 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4ea237640 1 --2- 172.21.15.88:0/2722262418 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4e406ad20 0x7ff4e41f5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4e9235640 1 --2- 172.21.15.88:0/2722262418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4e406a4d0 0x7ff4e4152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f81ceb72640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81c80c5930 0x7f81c81529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f81ceb72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81c8153780 0x7f81c8144b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f81ceb72640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81c8159490 0x7f81c81574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f81ceb72640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c8152ee0 con 0x7f81c8153780 2023-10-16T20:39:22.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f81ceb72640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c81f5ec0 con 0x7f81c80c5930 2023-10-16T20:39:22.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f81ceb72640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c81f6160 con 0x7f81c8159490 2023-10-16T20:39:22.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f0187fff640 1 --2- 172.21.15.88:0/3186187548 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f018c0664a0 0x7f018c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4e9a36640 1 -- 172.21.15.88:0/2722262418 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4e406ad20 msgr2=0x7ff4e41f5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4e9a36640 1 --2- 172.21.15.88:0/2722262418 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4e406ad20 0x7ff4e41f5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4e9a36640 1 -- 172.21.15.88:0/2722262418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4e406a4d0 msgr2=0x7ff4e4152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4e9a36640 1 --2- 172.21.15.88:0/2722262418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4e406a4d0 0x7ff4e4152510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4e9a36640 1 -- 172.21.15.88:0/2722262418 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d80094d0 con 0x7ff4e406a0f0 2023-10-16T20:39:22.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f3eeec9c640 1 Processor -- start 2023-10-16T20:39:22.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f0187fff640 1 -- 172.21.15.88:0/3186187548 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0190085080 con 0x7f0188005190 2023-10-16T20:39:22.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f3eeec9c640 1 -- start start 2023-10-16T20:39:22.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7f3401eea640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f33fc153780 0x7f33fc0c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.177+0000 7ff4e9a36640 1 --2- 172.21.15.88:0/2722262418 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4e406a0f0 0x7ff4e4151fd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff4d400c7f0 tx=0x7ff4d400ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f0187fff640 1 --2- 172.21.15.88:0/3186187548 >> [v2:172.21.15.88:6802/360619264,v1:172.21.15.88:6805/360619264] conn(0x7f018c06fd80 0x7f018c072220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7ff4e27fc640 1 -- 172.21.15.88:0/2722262418 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4003b80 con 0x7ff4e406a0f0 2023-10-16T20:39:22.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7ff4e27fc640 1 -- 172.21.15.88:0/2722262418 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d4003ce0 con 0x7ff4e406a0f0 2023-10-16T20:39:22.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7ff4ebc9a640 1 -- 172.21.15.88:0/2722262418 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e414cc10 con 0x7ff4e406a0f0 2023-10-16T20:39:22.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62ff9f1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f62f8071550 0x7f62f8071cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62ff9f1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f8070d90 con 0x7f62f8150420 2023-10-16T20:39:22.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62ff9f1640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f80c7ed0 con 0x7f62f80c2af0 2023-10-16T20:39:22.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62ff9f1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62f8070810 con 0x7f62f8071550 2023-10-16T20:39:22.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3401eea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33fc159d90 con 0x7f33fc159490 2023-10-16T20:39:22.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3401eea640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33fc15a070 con 0x7f33fc1598c0 2023-10-16T20:39:22.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3401eea640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33fc15a350 con 0x7f33fc153780 2023-10-16T20:39:22.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7ff4ebc9a640 1 -- 172.21.15.88:0/2722262418 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e41577c0 con 0x7ff4e406a0f0 2023-10-16T20:39:22.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f019e7fc640 1 --2- 172.21.15.88:0/3186187548 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f018c0664a0 0x7f018c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f0187fff640 1 -- 172.21.15.88:0/3186187548 --> [v2:172.21.15.88:6802/360619264,v1:172.21.15.88:6805/360619264] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f018c0728f0 con 0x7f018c06fd80 2023-10-16T20:39:22.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f0187fff640 1 -- 172.21.15.88:0/3186187548 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01900854d0 con 0x7f0188005190 2023-10-16T20:39:22.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7ff4e27fc640 1 -- 172.21.15.88:0/2722262418 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d400aa90 con 0x7ff4e406a0f0 2023-10-16T20:39:22.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62fcf8c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f62f8071550 0x7f62f8071cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62fcf8c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f62f8071550 0x7f62f8071cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56378/0 (socket says 172.21.15.88:56378) 2023-10-16T20:39:22.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62fcf8c640 1 -- 172.21.15.88:0/3678426109 learned_addr learned my addr 172.21.15.88:0/3678426109 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f33fb7fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f33fc153780 0x7f33fc0c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f33fbfff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f33fc1598c0 0x7f33fc0ced80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:40096/0 (socket says 172.21.15.88:40096) 2023-10-16T20:39:22.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f33fbfff640 1 -- 172.21.15.88:0/2294048803 learned_addr learned my addr 172.21.15.88:0/2294048803 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:22.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f81cc90e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81c80c5930 0x7f81c81529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f019e7fc640 1 --2- 172.21.15.88:0/3186187548 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f018c0664a0 0x7f018c068960 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f01a014fc20 tx=0x7f0194009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f81cc90e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81c80c5930 0x7f81c81529a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:40156/0 (socket says 172.21.15.88:40156) 2023-10-16T20:39:22.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f81cc90e640 1 -- 172.21.15.88:0/3542551029 learned_addr learned my addr 172.21.15.88:0/3542551029 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62effff640 1 -- 172.21.15.88:0/3678426109 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952988072 0 0) 0x7f62f80c7ed0 con 0x7f62f80c2af0 2023-10-16T20:39:22.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3eeec9c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ee80c5930 0x7f3ee80cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62effff640 1 -- 172.21.15.88:0/3678426109 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e4003530 con 0x7f62f80c2af0 2023-10-16T20:39:22.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f81cc90e640 1 -- 172.21.15.88:0/3542551029 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81c8159490 msgr2=0x7f81c81574d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T20:39:22.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f81cc90e640 1 --2- 172.21.15.88:0/3542551029 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81c8159490 0x7f81c81574d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3eeec9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ee8153780 0x7f3ee81f6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f81cc90e640 1 -- 172.21.15.88:0/3542551029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81c8153780 msgr2=0x7f81c8144b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f81cc90e640 1 --2- 172.21.15.88:0/3542551029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81c8153780 0x7f81c8144b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f81cc90e640 1 -- 172.21.15.88:0/3542551029 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c0009b30 con 0x7f81c80c5930 2023-10-16T20:39:22.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f33fb7fe640 1 -- 172.21.15.88:0/2294048803 >> v1:172.21.15.191:6789/0 conn(0x7f33fc1598c0 legacy=0x7f33fc0ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f33fb7fe640 1 -- 172.21.15.88:0/2294048803 >> v1:172.21.15.88:6789/0 conn(0x7f33fc159490 legacy=0x7f33fc0c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f33fb7fe640 1 -- 172.21.15.88:0/2294048803 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33fc0c4910 con 0x7f33fc153780 2023-10-16T20:39:22.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62effff640 1 -- 172.21.15.88:0/3678426109 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1181863172 0 0) 0x7f62e4003530 con 0x7f62f80c2af0 2023-10-16T20:39:22.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62effff640 1 -- 172.21.15.88:0/3678426109 >> v2:172.21.15.88:3300/0 conn(0x7f62f8071550 msgr2=0x7f62f8071cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62effff640 1 --2- 172.21.15.88:0/3678426109 >> v2:172.21.15.88:3300/0 conn(0x7f62f8071550 0x7f62f8071cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62effff640 1 -- 172.21.15.88:0/3678426109 >> v1:172.21.15.88:6789/0 conn(0x7f62f8150420 legacy=0x7f62f806afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62effff640 1 -- 172.21.15.88:0/3678426109 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f806bea0 con 0x7f62f80c2af0 2023-10-16T20:39:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62effff640 1 -- 172.21.15.88:0/3678426109 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f62f0001a00 con 0x7f62f80c2af0 2023-10-16T20:39:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3eeca38640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ee80c5930 0x7f3ee80cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3eeca38640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ee80c5930 0x7f3ee80cefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:40164/0 (socket says 172.21.15.88:40164) 2023-10-16T20:39:22.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3eeec9c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ee8159490 0x7f3ee81f6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3eeca38640 1 -- 172.21.15.88:0/1108072887 learned_addr learned my addr 172.21.15.88:0/1108072887 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3eeec9c640 1 -- 172.21.15.88:0/1108072887 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee8155430 con 0x7f3ee8153780 2023-10-16T20:39:22.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3eeec9c640 1 -- 172.21.15.88:0/1108072887 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee81517f0 con 0x7f3ee80c5930 2023-10-16T20:39:22.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3eeec9c640 1 -- 172.21.15.88:0/1108072887 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee8150fb0 con 0x7f3ee8159490 2023-10-16T20:39:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62effff640 1 -- 172.21.15.88:0/3678426109 >> v1:172.21.15.191:6789/0 conn(0x7f62f80c2af0 legacy=0x7f62f8160b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62effff640 1 --2- 172.21.15.88:0/3678426109 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62e4003980 0x7f62e4003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62effff640 1 --2- 172.21.15.88:0/3678426109 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62e40047d0 0x7f62e4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62effff640 1 --2- 172.21.15.88:0/3678426109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62e40052b0 0x7f62e400fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62effff640 1 -- 172.21.15.88:0/3678426109 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f8070810 con 0x7f62e40052b0 2023-10-16T20:39:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62effff640 1 -- 172.21.15.88:0/3678426109 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e4010570 con 0x7f62e40047d0 2023-10-16T20:39:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62effff640 1 -- 172.21.15.88:0/3678426109 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e4009980 con 0x7f62e4003980 2023-10-16T20:39:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3eed239640 1 --2- 172.21.15.88:0/1108072887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ee8159490 0x7f3ee81f6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f81cc90e640 1 --2- 172.21.15.88:0/3542551029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81c80c5930 0x7f81c81529a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f81bc00c480 tx=0x7f81bc00c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f019f7fe640 1 --2- 172.21.15.88:0/3186187548 >> [v2:172.21.15.88:6802/360619264,v1:172.21.15.88:6805/360619264] conn(0x7f018c06fd80 0x7f018c072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f81c57fa640 1 -- 172.21.15.88:0/3542551029 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc012830 con 0x7f81c80c5930 2023-10-16T20:39:22.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f019f7fe640 1 --2- 172.21.15.88:0/3186187548 >> [v2:172.21.15.88:6802/360619264,v1:172.21.15.88:6805/360619264] conn(0x7f018c06fd80 0x7f018c072220 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3eeca38640 1 -- 172.21.15.88:0/1108072887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ee8159490 msgr2=0x7f3ee81f6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3eeca38640 1 --2- 172.21.15.88:0/1108072887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3ee8159490 0x7f3ee81f6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3eeca38640 1 -- 172.21.15.88:0/1108072887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ee8153780 msgr2=0x7f3ee81f6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3eeca38640 1 --2- 172.21.15.88:0/1108072887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ee8153780 0x7f3ee81f6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3eeca38640 1 -- 172.21.15.88:0/1108072887 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed8009a70 con 0x7f3ee80c5930 2023-10-16T20:39:22.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62fcf8c640 1 --2- 172.21.15.88:0/3678426109 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62e40047d0 0x7f62e4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f81c57fa640 1 -- 172.21.15.88:0/3542551029 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81bc012990 con 0x7f81c80c5930 2023-10-16T20:39:22.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f81c57fa640 1 -- 172.21.15.88:0/3542551029 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc012d40 con 0x7f81c80c5930 2023-10-16T20:39:22.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f62fd78d640 1 --2- 172.21.15.88:0/3678426109 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62e4003980 0x7f62e4003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7f3eeca38640 1 --2- 172.21.15.88:0/1108072887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ee80c5930 0x7f3ee80cefc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00c4f0 tx=0x7f3edc00c9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f0187fff640 1 -- 172.21.15.88:0/3186187548 <== osd.2 v2:172.21.15.88:6802/360619264 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f018c0728f0 con 0x7f018c06fd80 2023-10-16T20:39:22.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.181+0000 7ff4ebc9a640 1 -- 172.21.15.88:0/2722262418 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff4c4000fd0 con 0x7ff4e406a0f0 2023-10-16T20:39:22.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f81ceb72640 1 -- 172.21.15.88:0/3542551029 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c8150e20 con 0x7f81c80c5930 2023-10-16T20:39:22.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f81ceb72640 1 -- 172.21.15.88:0/3542551029 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c81513e0 con 0x7f81c80c5930 2023-10-16T20:39:22.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f3ee57fa640 1 -- 172.21.15.88:0/1108072887 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc005cf0 con 0x7f3ee80c5930 2023-10-16T20:39:22.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f3ee57fa640 1 -- 172.21.15.88:0/1108072887 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3edc005e50 con 0x7f3ee80c5930 2023-10-16T20:39:22.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f3ee57fa640 1 -- 172.21.15.88:0/1108072887 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc007270 con 0x7f3ee80c5930 2023-10-16T20:39:22.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f3eeec9c640 1 -- 172.21.15.88:0/1108072887 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee81515d0 con 0x7f3ee80c5930 2023-10-16T20:39:22.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f3eeec9c640 1 -- 172.21.15.88:0/1108072887 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee80b4670 con 0x7f3ee80c5930 2023-10-16T20:39:22.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f3eeec9c640 1 -- 172.21.15.88:0/1108072887 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3ee80c4ed0 con 0x7f3ee80c5930 2023-10-16T20:39:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f62fdf8e640 1 --2- 172.21.15.88:0/3678426109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62e40052b0 0x7f62e400fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f62fcf8c640 1 -- 172.21.15.88:0/3678426109 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62e4003980 msgr2=0x7f62e4003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f62fcf8c640 1 --2- 172.21.15.88:0/3678426109 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62e4003980 0x7f62e4003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f62fcf8c640 1 -- 172.21.15.88:0/3678426109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62e40052b0 msgr2=0x7f62e400fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f62fcf8c640 1 --2- 172.21.15.88:0/3678426109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62e40052b0 0x7f62e400fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f62fcf8c640 1 -- 172.21.15.88:0/3678426109 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f806bea0 con 0x7f62e40047d0 2023-10-16T20:39:22.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f62fdf8e640 1 --2- 172.21.15.88:0/3678426109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62e40052b0 0x7f62e400fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:22.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f62fdf8e640 1 -- 172.21.15.88:0/3678426109 reap_dead start 2023-10-16T20:39:22.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f81ceb72640 1 -- 172.21.15.88:0/3542551029 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f81c80c4ed0 con 0x7f81c80c5930 2023-10-16T20:39:22.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7ff4e27fc640 1 -- 172.21.15.88:0/2722262418 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff4d400abf0 con 0x7ff4e406a0f0 2023-10-16T20:39:22.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f33fb7fe640 1 --2- 172.21.15.88:0/2294048803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33fc153780 0x7f33fc0c9be0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f33ec0097a0 tx=0x7f33ec01ec10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dda283b93a6573f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7ff4e27fc640 1 --2- 172.21.15.88:0/2722262418 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff4d0066570 0x7ff4d0068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7ff4e27fc640 1 -- 172.21.15.88:0/2722262418 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff4d4084180 con 0x7ff4e406a0f0 2023-10-16T20:39:22.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7ff4e27fc640 1 --2- 172.21.15.88:0/2722262418 >> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] conn(0x7ff4d006fed0 0x7ff4d0072370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7ff4e27fc640 1 -- 172.21.15.88:0/2722262418 --> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4d0072a40 con 0x7ff4d006fed0 2023-10-16T20:39:22.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7ff4e27fc640 1 -- 172.21.15.88:0/2722262418 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4d40867a0 con 0x7ff4e406a0f0 2023-10-16T20:39:22.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f33faffd640 1 -- 172.21.15.88:0/2294048803 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec0a8910 con 0x7f33fc153780 2023-10-16T20:39:22.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f33faffd640 1 -- 172.21.15.88:0/2294048803 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec0a8a70 con 0x7f33fc153780 2023-10-16T20:39:22.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.185+0000 7f33faffd640 1 -- 172.21.15.88:0/2294048803 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec0a8e00 con 0x7f33fc153780 2023-10-16T20:39:22.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f3401eea640 1 -- 172.21.15.88:0/2294048803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33fc153780 msgr2=0x7f33fc0c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f3401eea640 1 --2- 172.21.15.88:0/2294048803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33fc153780 0x7f33fc0c9be0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f33ec0097a0 tx=0x7f33ec01ec10 comp rx=0 tx=0).stop 2023-10-16T20:39:22.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f81c57fa640 1 -- 172.21.15.88:0/3542551029 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f81bc01c460 con 0x7f81c80c5930 2023-10-16T20:39:22.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f3401eea640 1 -- 172.21.15.88:0/2294048803 shutdown_connections 2023-10-16T20:39:22.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f3401eea640 1 --2- 172.21.15.88:0/2294048803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33fc153780 0x7f33fc0c9be0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f3401eea640 1 -- 172.21.15.88:0/2294048803 >> 172.21.15.88:0/2294048803 conn(0x7f33fc06bcb0 msgr2=0x7f33fc06c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7ff4ea237640 1 --2- 172.21.15.88:0/2722262418 >> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] conn(0x7ff4d006fed0 0x7ff4d0072370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f3ee57fa640 1 -- 172.21.15.88:0/1108072887 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3edc0073d0 con 0x7f3ee80c5930 2023-10-16T20:39:22.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7ff4e9235640 1 --2- 172.21.15.88:0/2722262418 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff4d0066570 0x7ff4d0068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f3ee57fa640 1 --2- 172.21.15.88:0/1108072887 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3ed4066570 0x7f3ed4068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f81c57fa640 1 --2- 172.21.15.88:0/3542551029 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f81b40665a0 0x7f81b4068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7ff4ea237640 1 --2- 172.21.15.88:0/2722262418 >> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] conn(0x7ff4d006fed0 0x7ff4d0072370 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f81c57fa640 1 -- 172.21.15.88:0/3542551029 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f81bc0840a0 con 0x7f81c80c5930 2023-10-16T20:39:22.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f3ee57fa640 1 -- 172.21.15.88:0/1108072887 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3edc083900 con 0x7f3ee80c5930 2023-10-16T20:39:22.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7ff4e9235640 1 --2- 172.21.15.88:0/2722262418 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff4d0066570 0x7ff4d0068a30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff4d80ae000 tx=0x7ff4d80a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f81c57fa640 1 --2- 172.21.15.88:0/3542551029 >> [v2:172.21.15.191:6800/584586645,v1:172.21.15.191:6801/584586645] conn(0x7f81b406ff00 0x7f81b40723a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f3ee57fa640 1 --2- 172.21.15.88:0/1108072887 >> [v2:172.21.15.191:6808/3826846674,v1:172.21.15.191:6809/3826846674] conn(0x7f3ed406fed0 0x7f3ed4072370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f3ee57fa640 1 -- 172.21.15.88:0/1108072887 --> [v2:172.21.15.191:6808/3826846674,v1:172.21.15.191:6809/3826846674] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ed4072a40 con 0x7f3ed406fed0 2023-10-16T20:39:22.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f3ee57fa640 1 -- 172.21.15.88:0/1108072887 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3edc083cb0 con 0x7f3ee80c5930 2023-10-16T20:39:22.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f62fcf8c640 1 --2- 172.21.15.88:0/3678426109 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62e40047d0 0x7f62e4004c50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f62f8071990 tx=0x7f62e801cb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0adb0deb3ba625b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f3401eea640 1 -- 172.21.15.88:0/2294048803 shutdown_connections 2023-10-16T20:39:22.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f3401eea640 1 -- 172.21.15.88:0/2294048803 wait complete. 2023-10-16T20:39:22.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f62effff640 1 -- 172.21.15.88:0/3678426109 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e80a5a20 con 0x7f62e40047d0 2023-10-16T20:39:22.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f3eed239640 1 --2- 172.21.15.88:0/1108072887 >> [v2:172.21.15.191:6808/3826846674,v1:172.21.15.191:6809/3826846674] conn(0x7f3ed406fed0 0x7f3ed4072370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f62effff640 1 -- 172.21.15.88:0/3678426109 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f0007e90 con 0x7f62e40047d0 2023-10-16T20:39:22.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f62effff640 1 -- 172.21.15.88:0/3678426109 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e80a5da0 con 0x7f62e40047d0 2023-10-16T20:39:22.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f3ee7fff640 1 --2- 172.21.15.88:0/1108072887 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3ed4066570 0x7f3ed4068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.193+0000 7ff4e27fc640 1 -- 172.21.15.88:0/2722262418 <== osd.3 v2:172.21.15.88:6823/1519973662 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7ff4d0072a40 con 0x7ff4d006fed0 2023-10-16T20:39:22.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.193+0000 7f3eed239640 1 --2- 172.21.15.88:0/1108072887 >> [v2:172.21.15.191:6808/3826846674,v1:172.21.15.191:6809/3826846674] conn(0x7f3ed406fed0 0x7f3ed4072370 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.193+0000 7f62ff9f1640 1 -- 172.21.15.88:0/3678426109 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62e40047d0 msgr2=0x7f62e4004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.193+0000 7f62ff9f1640 1 --2- 172.21.15.88:0/3678426109 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62e40047d0 0x7f62e4004c50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f62f8071990 tx=0x7f62e801cb10 comp rx=0 tx=0).stop 2023-10-16T20:39:22.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.193+0000 7f3ee7fff640 1 --2- 172.21.15.88:0/1108072887 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3ed4066570 0x7f3ed4068a30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3ed80b1000 tx=0x7f3ed8004ac0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.193+0000 7f3ee57fa640 1 -- 172.21.15.88:0/1108072887 <== osd.5 v2:172.21.15.191:6808/3826846674 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f3ed4072a40 con 0x7f3ed406fed0 2023-10-16T20:39:22.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.193+0000 7f62ff9f1640 1 -- 172.21.15.88:0/3678426109 shutdown_connections 2023-10-16T20:39:22.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f81c57fa640 1 -- 172.21.15.88:0/3542551029 --> [v2:172.21.15.191:6800/584586645,v1:172.21.15.191:6801/584586645] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f81b4072a70 con 0x7f81b406ff00 2023-10-16T20:39:22.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f81c57fa640 1 -- 172.21.15.88:0/3542551029 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f81bc086600 con 0x7f81c80c5930 2023-10-16T20:39:22.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.189+0000 7f81c7fff640 1 --2- 172.21.15.88:0/3542551029 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f81b40665a0 0x7f81b4068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.193+0000 7f81cd10f640 1 --2- 172.21.15.88:0/3542551029 >> [v2:172.21.15.191:6800/584586645,v1:172.21.15.191:6801/584586645] conn(0x7f81b406ff00 0x7f81b40723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.193+0000 7f81cd10f640 1 --2- 172.21.15.88:0/3542551029 >> [v2:172.21.15.191:6800/584586645,v1:172.21.15.191:6801/584586645] conn(0x7f81b406ff00 0x7f81b40723a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.193+0000 7f81c7fff640 1 --2- 172.21.15.88:0/3542551029 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f81b40665a0 0x7f81b4068a60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f81c00b1000 tx=0x7f81c0004ac0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.193+0000 7f62ff9f1640 1 --2- 172.21.15.88:0/3678426109 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62e40047d0 0x7f62e4004c50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.193+0000 7f62ff9f1640 1 -- 172.21.15.88:0/3678426109 >> 172.21.15.88:0/3678426109 conn(0x7f62f80c86b0 msgr2=0x7f62f80c8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.193+0000 7f62ff9f1640 1 -- 172.21.15.88:0/3678426109 shutdown_connections 2023-10-16T20:39:22.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.193+0000 7f62ff9f1640 1 -- 172.21.15.88:0/3678426109 wait complete. 2023-10-16T20:39:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.193+0000 7f81c57fa640 1 -- 172.21.15.88:0/3542551029 <== osd.4 v2:172.21.15.191:6800/584586645 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f81b4072a70 con 0x7f81b406ff00 2023-10-16T20:39:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.193+0000 7f3401eea640 1 Processor -- start 2023-10-16T20:39:22.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f62ff9f1640 1 Processor -- start 2023-10-16T20:39:22.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f62ff9f1640 1 -- start start 2023-10-16T20:39:22.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.193+0000 7f3401eea640 1 -- start start 2023-10-16T20:39:22.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f3401eea640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33fc153780 0x7f33fc0b4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f3401eea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33fc159490 0x7f33fc1583a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f33fb7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33fc153780 0x7f33fc0b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f33fb7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33fc153780 0x7f33fc0b4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:40182/0 (socket says 172.21.15.88:40182) 2023-10-16T20:39:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f3401eea640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33fc1598c0 0x7f33fc1588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f3401eea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc0c89d0 con 0x7f33fc159490 2023-10-16T20:39:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f3401eea640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc15acb0 con 0x7f33fc153780 2023-10-16T20:39:22.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f3401eea640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc15d200 con 0x7f33fc1598c0 2023-10-16T20:39:22.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f33fb7fe640 1 -- 172.21.15.88:0/785747284 learned_addr learned my addr 172.21.15.88:0/785747284 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f62ff9f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62e40047d0 0x7f62f8159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f33fbfff640 1 --2- 172.21.15.88:0/785747284 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33fc1598c0 0x7f33fc1588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f33f3fff640 1 --2- 172.21.15.88:0/785747284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33fc159490 0x7f33fc1583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f33fb7fe640 1 -- 172.21.15.88:0/785747284 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33fc1598c0 msgr2=0x7f33fc1588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f62ff9f1640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62f8154930 0x7f62f8157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f62ff9f1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62f815a190 0x7f62f8157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f62ff9f1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f80cb360 con 0x7f62e40047d0 2023-10-16T20:39:22.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f62ff9f1640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f8151470 con 0x7f62f815a190 2023-10-16T20:39:22.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f62ff9f1640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f81564d0 con 0x7f62f8154930 2023-10-16T20:39:22.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f62fdf8e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62f815a190 0x7f62f8157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f62fdf8e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62f815a190 0x7f62f8157b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:40186/0 (socket says 172.21.15.88:40186) 2023-10-16T20:39:22.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f62fdf8e640 1 -- 172.21.15.88:0/4187397412 learned_addr learned my addr 172.21.15.88:0/4187397412 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f62fd78d640 1 --2- 172.21.15.88:0/4187397412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62e40047d0 0x7f62f8159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f62fdf8e640 1 -- 172.21.15.88:0/4187397412 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62f8154930 msgr2=0x7f62f8157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f62fcf8c640 1 --2- 172.21.15.88:0/4187397412 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62f8154930 0x7f62f8157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f33fb7fe640 1 --2- 172.21.15.88:0/785747284 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33fc1598c0 0x7f33fc1588e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f33fb7fe640 1 -- 172.21.15.88:0/785747284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33fc159490 msgr2=0x7f33fc1583a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f33fb7fe640 1 --2- 172.21.15.88:0/785747284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33fc159490 0x7f33fc1583a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.197+0000 7f33fb7fe640 1 -- 172.21.15.88:0/785747284 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec0093b0 con 0x7f33fc153780 2023-10-16T20:39:22.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f62fdf8e640 1 --2- 172.21.15.88:0/4187397412 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62f8154930 0x7f62f8157600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f62fdf8e640 1 -- 172.21.15.88:0/4187397412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62e40047d0 msgr2=0x7f62f8159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f62fdf8e640 1 --2- 172.21.15.88:0/4187397412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62e40047d0 0x7f62f8159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f62fdf8e640 1 -- 172.21.15.88:0/4187397412 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f8155600 con 0x7f62f815a190 2023-10-16T20:39:22.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f33fb7fe640 1 --2- 172.21.15.88:0/785747284 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33fc153780 0x7f33fc0b4260 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f33ec01b820 tx=0x7f33ec01b9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f62fd78d640 1 --2- 172.21.15.88:0/4187397412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62e40047d0 0x7f62f8159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:22.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f62fcf8c640 1 --2- 172.21.15.88:0/4187397412 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f62f8154930 0x7f62f8157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:22.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f62fdf8e640 1 --2- 172.21.15.88:0/4187397412 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62f815a190 0x7f62f8157b40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f62f806f350 tx=0x7f62f0010ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f33f8ff9640 1 -- 172.21.15.88:0/785747284 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec0b1aa0 con 0x7f33fc153780 2023-10-16T20:39:22.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f33f8ff9640 1 -- 172.21.15.88:0/785747284 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec0b1c00 con 0x7f33fc153780 2023-10-16T20:39:22.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f33f8ff9640 1 -- 172.21.15.88:0/785747284 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec0b9410 con 0x7f33fc153780 2023-10-16T20:39:22.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f3401eea640 1 -- 172.21.15.88:0/785747284 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33fc15d4c0 con 0x7f33fc153780 2023-10-16T20:39:22.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f3401eea640 1 -- 172.21.15.88:0/785747284 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33fc0c8380 con 0x7f33fc153780 2023-10-16T20:39:22.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f62edffb640 1 -- 172.21.15.88:0/4187397412 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0019070 con 0x7f62f815a190 2023-10-16T20:39:22.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f62edffb640 1 -- 172.21.15.88:0/4187397412 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f000e6a0 con 0x7f62f815a190 2023-10-16T20:39:22.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f62edffb640 1 -- 172.21.15.88:0/4187397412 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f000ea30 con 0x7f62f815a190 2023-10-16T20:39:22.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f62ff9f1640 1 -- 172.21.15.88:0/4187397412 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f81558c0 con 0x7f62f815a190 2023-10-16T20:39:22.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.201+0000 7f62ff9f1640 1 -- 172.21.15.88:0/4187397412 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f8155e80 con 0x7f62f815a190 2023-10-16T20:39:22.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.205+0000 7f62db7fe640 1 -- 172.21.15.88:0/4187397412 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f62d4000fd0 con 0x7f62f815a190 2023-10-16T20:39:22.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.205+0000 7f62edffb640 1 -- 172.21.15.88:0/4187397412 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f62f000eb90 con 0x7f62f815a190 2023-10-16T20:39:22.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.205+0000 7f33f8ff9640 1 -- 172.21.15.88:0/785747284 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f33ec0b7040 con 0x7f33fc153780 2023-10-16T20:39:22.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.205+0000 7f33f8ff9640 1 --2- 172.21.15.88:0/785747284 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f33e00664d0 0x7f33e0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.205+0000 7f33f8ff9640 1 -- 172.21.15.88:0/785747284 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f33ec121ab0 con 0x7f33fc153780 2023-10-16T20:39:22.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.205+0000 7f3401eea640 1 --2- 172.21.15.88:0/785747284 >> [v2:172.21.15.191:6824/4079140119,v1:172.21.15.191:6825/4079140119] conn(0x7f33fc0c9000 0x7f33fc13d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.205+0000 7f3401eea640 1 -- 172.21.15.88:0/785747284 --> [v2:172.21.15.191:6824/4079140119,v1:172.21.15.191:6825/4079140119] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33fc0c4bf0 con 0x7f33fc0c9000 2023-10-16T20:39:22.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.205+0000 7f33f3fff640 1 --2- 172.21.15.88:0/785747284 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f33e00664d0 0x7f33e0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.205+0000 7f33fbfff640 1 --2- 172.21.15.88:0/785747284 >> [v2:172.21.15.191:6824/4079140119,v1:172.21.15.191:6825/4079140119] conn(0x7f33fc0c9000 0x7f33fc13d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7ff8e27fc640 1 -- 172.21.15.88:0/2793680559 --> [v2:172.21.15.88:6800/1130215823,v1:172.21.15.88:6801/1130215823] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff8f0006ad0 con 0x7ff8dc074310 2023-10-16T20:39:22.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7f62edffb640 1 --2- 172.21.15.88:0/4187397412 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f62dc0665a0 0x7f62dc068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7f62edffb640 1 -- 172.21.15.88:0/4187397412 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f62f00869d0 con 0x7f62f815a190 2023-10-16T20:39:22.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7f62edffb640 1 --2- 172.21.15.88:0/4187397412 >> [v2:172.21.15.191:6816/292421315,v1:172.21.15.191:6817/292421315] conn(0x7f62dc06fd90 0x7f62dc072230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7f62edffb640 1 -- 172.21.15.88:0/4187397412 --> [v2:172.21.15.191:6816/292421315,v1:172.21.15.191:6817/292421315] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62dc072900 con 0x7f62dc06fd90 2023-10-16T20:39:22.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7f62edffb640 1 -- 172.21.15.88:0/4187397412 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62f008b240 con 0x7f62f815a190 2023-10-16T20:39:22.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7f62fd78d640 1 --2- 172.21.15.88:0/4187397412 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f62dc0665a0 0x7f62dc068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7f33f3fff640 1 --2- 172.21.15.88:0/785747284 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f33e00664d0 0x7f33e0068990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f33fc0725a0 tx=0x7f33e4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7f33fbfff640 1 --2- 172.21.15.88:0/785747284 >> [v2:172.21.15.191:6824/4079140119,v1:172.21.15.191:6825/4079140119] conn(0x7f33fc0c9000 0x7f33fc13d470 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7f62fcf8c640 1 --2- 172.21.15.88:0/4187397412 >> [v2:172.21.15.191:6816/292421315,v1:172.21.15.191:6817/292421315] conn(0x7f62dc06fd90 0x7f62dc072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7ff8f4ff9640 1 -- 172.21.15.88:0/2793680559 <== osd.0 v2:172.21.15.88:6800/1130215823 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff8dc076e80 con 0x7ff8dc074310 2023-10-16T20:39:22.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7f62fcf8c640 1 --2- 172.21.15.88:0/4187397412 >> [v2:172.21.15.191:6816/292421315,v1:172.21.15.191:6817/292421315] conn(0x7f62dc06fd90 0x7f62dc072230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7f62fd78d640 1 --2- 172.21.15.88:0/4187397412 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f62dc0665a0 0x7f62dc068a60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f62f4007fd0 tx=0x7f62f4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7f33f8ff9640 1 -- 172.21.15.88:0/785747284 <== osd.7 v2:172.21.15.191:6824/4079140119 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f33f4001c40 con 0x7f33fc0c9000 2023-10-16T20:39:22.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7ff8e27fc640 1 -- 172.21.15.88:0/2793680559 >> [v2:172.21.15.88:6800/1130215823,v1:172.21.15.88:6801/1130215823] conn(0x7ff8dc074310 msgr2=0x7ff8dc0767b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7ff8e27fc640 1 --2- 172.21.15.88:0/2793680559 >> [v2:172.21.15.88:6800/1130215823,v1:172.21.15.88:6801/1130215823] conn(0x7ff8dc074310 0x7ff8dc0767b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7f3b7a606640 1 -- 172.21.15.88:0/3768034351 --> [v2:172.21.15.88:6804/3386972440,v1:172.21.15.88:6809/3386972440] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3b7406a0e0 con 0x7f3b5806fed0 2023-10-16T20:39:22.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7ff8e27fc640 1 -- 172.21.15.88:0/2793680559 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff8dc06a9b0 msgr2=0x7ff8dc06ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7ff8e27fc640 1 --2- 172.21.15.88:0/2793680559 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff8dc06a9b0 0x7ff8dc06ce70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff8f804fb10 tx=0x7ff8f8068de0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7ff8e27fc640 1 -- 172.21.15.88:0/2793680559 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8f004ed60 msgr2=0x7ff8f004f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7ff8e27fc640 1 --2- 172.21.15.88:0/2793680559 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff8f004ed60 0x7ff8f004f210 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff8e401d330 tx=0x7ff8e4019830 comp rx=0 tx=0).stop 2023-10-16T20:39:22.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7ff8fcf95640 1 -- 172.21.15.88:0/2793680559 reap_dead start 2023-10-16T20:39:22.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7f62edffb640 1 -- 172.21.15.88:0/4187397412 <== osd.6 v2:172.21.15.191:6816/292421315 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f62dc072900 con 0x7f62dc06fd90 2023-10-16T20:39:22.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7ff8e27fc640 1 -- 172.21.15.88:0/2793680559 shutdown_connections 2023-10-16T20:39:22.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.209+0000 7ff8e27fc640 1 -- 172.21.15.88:0/2793680559 >> 172.21.15.88:0/2793680559 conn(0x7ff8f00502e0 msgr2=0x7ff8f004af60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.213+0000 7ff8e27fc640 1 -- 172.21.15.88:0/2793680559 shutdown_connections 2023-10-16T20:39:22.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.213+0000 7ff8e27fc640 1 -- 172.21.15.88:0/2793680559 wait complete. 2023-10-16T20:39:22.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.213+0000 7f3b69ffb640 1 -- 172.21.15.88:0/3768034351 <== osd.1 v2:172.21.15.88:6804/3386972440 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3b6c001cf0 con 0x7f3b5806fed0 2023-10-16T20:39:22.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.213+0000 7f3b7a606640 1 -- 172.21.15.88:0/3768034351 >> [v2:172.21.15.88:6804/3386972440,v1:172.21.15.88:6809/3386972440] conn(0x7f3b5806fed0 msgr2=0x7f3b58072370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.213+0000 7f3b7a606640 1 --2- 172.21.15.88:0/3768034351 >> [v2:172.21.15.88:6804/3386972440,v1:172.21.15.88:6809/3386972440] conn(0x7f3b5806fed0 0x7f3b58072370 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.213+0000 7f3b7a606640 1 -- 172.21.15.88:0/3768034351 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3b58066570 msgr2=0x7f3b58068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.213+0000 7f3b7a606640 1 --2- 172.21.15.88:0/3768034351 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3b58066570 0x7f3b58068a30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3b7001c450 tx=0x7f3b700a7000 comp rx=0 tx=0).stop 2023-10-16T20:39:22.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.213+0000 7f3b7a606640 1 -- 172.21.15.88:0/3768034351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b740bcc80 msgr2=0x7f3b74150e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.213+0000 7f3b7a606640 1 --2- 172.21.15.88:0/3768034351 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b740bcc80 0x7f3b74150e80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3b6400b410 tx=0x7f3b6400b8d0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.213+0000 7f3b79e05640 1 -- 172.21.15.88:0/3768034351 reap_dead start 2023-10-16T20:39:22.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.213+0000 7f3b7a606640 1 -- 172.21.15.88:0/3768034351 shutdown_connections 2023-10-16T20:39:22.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.213+0000 7f01a58ea640 1 -- 172.21.15.88:0/3186187548 --> [v2:172.21.15.88:6802/360619264,v1:172.21.15.88:6805/360619264] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f01a00c4bf0 con 0x7f018c06fd80 2023-10-16T20:39:22.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.213+0000 7f3b7a606640 1 -- 172.21.15.88:0/3768034351 >> 172.21.15.88:0/3768034351 conn(0x7f3b740caee0 msgr2=0x7f3b740c72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.213+0000 7f3b7a606640 1 -- 172.21.15.88:0/3768034351 shutdown_connections 2023-10-16T20:39:22.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.217+0000 7f3b7a606640 1 -- 172.21.15.88:0/3768034351 wait complete. 2023-10-16T20:39:22.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.217+0000 7f0187fff640 1 -- 172.21.15.88:0/3186187548 <== osd.2 v2:172.21.15.88:6802/360619264 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0198001900 con 0x7f018c06fd80 2023-10-16T20:39:22.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.217+0000 7f01a58ea640 1 -- 172.21.15.88:0/3186187548 >> [v2:172.21.15.88:6802/360619264,v1:172.21.15.88:6805/360619264] conn(0x7f018c06fd80 msgr2=0x7f018c072220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.217+0000 7f01a58ea640 1 --2- 172.21.15.88:0/3186187548 >> [v2:172.21.15.88:6802/360619264,v1:172.21.15.88:6805/360619264] conn(0x7f018c06fd80 0x7f018c072220 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.217+0000 7f01a58ea640 1 -- 172.21.15.88:0/3186187548 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f018c0664a0 msgr2=0x7f018c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.217+0000 7f01a58ea640 1 --2- 172.21.15.88:0/3186187548 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f018c0664a0 0x7f018c068960 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f01a014fc20 tx=0x7f0194009190 comp rx=0 tx=0).stop 2023-10-16T20:39:22.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.217+0000 7f01a58ea640 1 -- 172.21.15.88:0/3186187548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0188005190 msgr2=0x7f01a00b3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.217+0000 7f01a58ea640 1 --2- 172.21.15.88:0/3186187548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0188005190 0x7f01a00b3000 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0190011590 tx=0x7f0190011990 comp rx=0 tx=0).stop 2023-10-16T20:39:22.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.217+0000 7f019f7fe640 1 -- 172.21.15.88:0/3186187548 reap_dead start 2023-10-16T20:39:22.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.217+0000 7f01a58ea640 1 -- 172.21.15.88:0/3186187548 shutdown_connections 2023-10-16T20:39:22.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.217+0000 7f01a58ea640 1 -- 172.21.15.88:0/3186187548 >> 172.21.15.88:0/3186187548 conn(0x7f01a006bcb0 msgr2=0x7f01a01519c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.217+0000 7f01a58ea640 1 -- 172.21.15.88:0/3186187548 shutdown_connections 2023-10-16T20:39:22.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.217+0000 7f01a58ea640 1 -- 172.21.15.88:0/3186187548 wait complete. 2023-10-16T20:39:22.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.221+0000 7ff4ebc9a640 1 -- 172.21.15.88:0/2722262418 --> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff4c4002cc0 con 0x7ff4d006fed0 2023-10-16T20:39:22.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.221+0000 7f81ceb72640 1 -- 172.21.15.88:0/3542551029 --> [v2:172.21.15.191:6800/584586645,v1:172.21.15.191:6801/584586645] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f81c80c4bf0 con 0x7f81b406ff00 2023-10-16T20:39:22.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.221+0000 7f81c57fa640 1 -- 172.21.15.88:0/3542551029 <== osd.4 v2:172.21.15.191:6800/584586645 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f81b80050b0 con 0x7f81b406ff00 2023-10-16T20:39:22.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.221+0000 7f3eeec9c640 1 -- 172.21.15.88:0/1108072887 --> [v2:172.21.15.191:6808/3826846674,v1:172.21.15.191:6809/3826846674] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3ee80c4bf0 con 0x7f3ed406fed0 2023-10-16T20:39:22.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.221+0000 7ff4e27fc640 1 -- 172.21.15.88:0/2722262418 <== osd.3 v2:172.21.15.88:6823/1519973662 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff4c4002cc0 con 0x7ff4d006fed0 2023-10-16T20:39:22.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.221+0000 7f81aeffd640 1 -- 172.21.15.88:0/3542551029 >> [v2:172.21.15.191:6800/584586645,v1:172.21.15.191:6801/584586645] conn(0x7f81b406ff00 msgr2=0x7f81b40723a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.221+0000 7f81aeffd640 1 --2- 172.21.15.88:0/3542551029 >> [v2:172.21.15.191:6800/584586645,v1:172.21.15.191:6801/584586645] conn(0x7f81b406ff00 0x7f81b40723a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.221+0000 7f81aeffd640 1 -- 172.21.15.88:0/3542551029 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f81b40665a0 msgr2=0x7f81b4068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.221+0000 7f81aeffd640 1 --2- 172.21.15.88:0/3542551029 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f81b40665a0 0x7f81b4068a60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f81c00b1000 tx=0x7f81c0004ac0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.221+0000 7f81aeffd640 1 -- 172.21.15.88:0/3542551029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81c80c5930 msgr2=0x7f81c81529a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.221+0000 7f81aeffd640 1 --2- 172.21.15.88:0/3542551029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f81c80c5930 0x7f81c81529a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f81bc00c480 tx=0x7f81bc00c940 comp rx=0 tx=0).stop 2023-10-16T20:39:22.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7f81cd10f640 1 -- 172.21.15.88:0/3542551029 reap_dead start 2023-10-16T20:39:22.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7ff4ebc9a640 1 -- 172.21.15.88:0/2722262418 >> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] conn(0x7ff4d006fed0 msgr2=0x7ff4d0072370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7ff4ebc9a640 1 --2- 172.21.15.88:0/2722262418 >> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] conn(0x7ff4d006fed0 0x7ff4d0072370 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7f81aeffd640 1 -- 172.21.15.88:0/3542551029 shutdown_connections 2023-10-16T20:39:22.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7f81aeffd640 1 -- 172.21.15.88:0/3542551029 >> 172.21.15.88:0/3542551029 conn(0x7f81c806bcb0 msgr2=0x7f81c80cce40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7f81aeffd640 1 -- 172.21.15.88:0/3542551029 shutdown_connections 2023-10-16T20:39:22.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7f81aeffd640 1 -- 172.21.15.88:0/3542551029 wait complete. 2023-10-16T20:39:22.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7f3ee57fa640 1 -- 172.21.15.88:0/1108072887 <== osd.5 v2:172.21.15.191:6808/3826846674 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3ee0001250 con 0x7f3ed406fed0 2023-10-16T20:39:22.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7f3eeec9c640 1 -- 172.21.15.88:0/1108072887 >> [v2:172.21.15.191:6808/3826846674,v1:172.21.15.191:6809/3826846674] conn(0x7f3ed406fed0 msgr2=0x7f3ed4072370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7f3eeec9c640 1 --2- 172.21.15.88:0/1108072887 >> [v2:172.21.15.191:6808/3826846674,v1:172.21.15.191:6809/3826846674] conn(0x7f3ed406fed0 0x7f3ed4072370 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7f3eeec9c640 1 -- 172.21.15.88:0/1108072887 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3ed4066570 msgr2=0x7f3ed4068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7f3eeec9c640 1 --2- 172.21.15.88:0/1108072887 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3ed4066570 0x7f3ed4068a30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3ed80b1000 tx=0x7f3ed8004ac0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7f3eeec9c640 1 -- 172.21.15.88:0/1108072887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ee80c5930 msgr2=0x7f3ee80cefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7f3eeec9c640 1 --2- 172.21.15.88:0/1108072887 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ee80c5930 0x7f3ee80cefc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00c4f0 tx=0x7f3edc00c9b0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7f3eed239640 1 -- 172.21.15.88:0/1108072887 reap_dead start 2023-10-16T20:39:22.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7f3eeec9c640 1 -- 172.21.15.88:0/1108072887 shutdown_connections 2023-10-16T20:39:22.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7f3eeec9c640 1 -- 172.21.15.88:0/1108072887 >> 172.21.15.88:0/1108072887 conn(0x7f3ee806bcb0 msgr2=0x7f3ee80ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7f3eeec9c640 1 -- 172.21.15.88:0/1108072887 shutdown_connections 2023-10-16T20:39:22.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7f3eeec9c640 1 -- 172.21.15.88:0/1108072887 wait complete. 2023-10-16T20:39:22.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7ff4ebc9a640 1 -- 172.21.15.88:0/2722262418 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff4d0066570 msgr2=0x7ff4d0068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7ff4ebc9a640 1 --2- 172.21.15.88:0/2722262418 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff4d0066570 0x7ff4d0068a30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff4d80ae000 tx=0x7ff4d80a7000 comp rx=0 tx=0).stop 2023-10-16T20:39:22.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7ff4ebc9a640 1 -- 172.21.15.88:0/2722262418 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4e406a0f0 msgr2=0x7ff4e4151fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7ff4ebc9a640 1 --2- 172.21.15.88:0/2722262418 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4e406a0f0 0x7ff4e4151fd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff4d400c7f0 tx=0x7ff4d400ccb0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7ff4ea237640 1 -- 172.21.15.88:0/2722262418 reap_dead start 2023-10-16T20:39:22.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7ff4ebc9a640 1 -- 172.21.15.88:0/2722262418 shutdown_connections 2023-10-16T20:39:22.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.225+0000 7ff4ebc9a640 1 -- 172.21.15.88:0/2722262418 >> 172.21.15.88:0/2722262418 conn(0x7ff4e40ca9f0 msgr2=0x7ff4e40c9080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.233+0000 7ff4ebc9a640 1 -- 172.21.15.88:0/2722262418 shutdown_connections 2023-10-16T20:39:22.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.233+0000 7ff4ebc9a640 1 -- 172.21.15.88:0/2722262418 wait complete. 2023-10-16T20:39:22.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.233+0000 7f62db7fe640 1 -- 172.21.15.88:0/4187397412 --> [v2:172.21.15.191:6816/292421315,v1:172.21.15.191:6817/292421315] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f62d4002cc0 con 0x7f62dc06fd90 2023-10-16T20:39:22.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.237+0000 7f62edffb640 1 -- 172.21.15.88:0/4187397412 <== osd.6 v2:172.21.15.191:6816/292421315 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f62d4002cc0 con 0x7f62dc06fd90 2023-10-16T20:39:22.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.237+0000 7f3401eea640 1 -- 172.21.15.88:0/785747284 --> [v2:172.21.15.191:6824/4079140119,v1:172.21.15.191:6825/4079140119] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f33fc0c5b00 con 0x7f33fc0c9000 2023-10-16T20:39:22.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f62db7fe640 1 -- 172.21.15.88:0/4187397412 >> [v2:172.21.15.191:6816/292421315,v1:172.21.15.191:6817/292421315] conn(0x7f62dc06fd90 msgr2=0x7f62dc072230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f62db7fe640 1 --2- 172.21.15.88:0/4187397412 >> [v2:172.21.15.191:6816/292421315,v1:172.21.15.191:6817/292421315] conn(0x7f62dc06fd90 0x7f62dc072230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f33f8ff9640 1 -- 172.21.15.88:0/785747284 <== osd.7 v2:172.21.15.191:6824/4079140119 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f33f4005be0 con 0x7f33fc0c9000 2023-10-16T20:39:22.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f62db7fe640 1 -- 172.21.15.88:0/4187397412 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f62dc0665a0 msgr2=0x7f62dc068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f62db7fe640 1 --2- 172.21.15.88:0/4187397412 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f62dc0665a0 0x7f62dc068a60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f62f4007fd0 tx=0x7f62f4009040 comp rx=0 tx=0).stop 2023-10-16T20:39:22.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f62db7fe640 1 -- 172.21.15.88:0/4187397412 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62f815a190 msgr2=0x7f62f8157b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f62db7fe640 1 --2- 172.21.15.88:0/4187397412 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f62f815a190 0x7f62f8157b40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f62f806f350 tx=0x7f62f0010ce0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f62fdf8e640 1 -- 172.21.15.88:0/4187397412 reap_dead start 2023-10-16T20:39:22.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f62db7fe640 1 -- 172.21.15.88:0/4187397412 shutdown_connections 2023-10-16T20:39:22.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f62db7fe640 1 -- 172.21.15.88:0/4187397412 >> 172.21.15.88:0/4187397412 conn(0x7f62f80c86b0 msgr2=0x7f62f806acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f3401eea640 1 -- 172.21.15.88:0/785747284 >> [v2:172.21.15.191:6824/4079140119,v1:172.21.15.191:6825/4079140119] conn(0x7f33fc0c9000 msgr2=0x7f33fc13d470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f3401eea640 1 --2- 172.21.15.88:0/785747284 >> [v2:172.21.15.191:6824/4079140119,v1:172.21.15.191:6825/4079140119] conn(0x7f33fc0c9000 0x7f33fc13d470 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f3401eea640 1 -- 172.21.15.88:0/785747284 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f33e00664d0 msgr2=0x7f33e0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f3401eea640 1 --2- 172.21.15.88:0/785747284 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f33e00664d0 0x7f33e0068990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f33fc0725a0 tx=0x7f33e4009040 comp rx=0 tx=0).stop 2023-10-16T20:39:22.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f3401eea640 1 -- 172.21.15.88:0/785747284 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33fc153780 msgr2=0x7f33fc0b4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f3401eea640 1 --2- 172.21.15.88:0/785747284 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f33fc153780 0x7f33fc0b4260 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f33ec01b820 tx=0x7f33ec01b9b0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f33fbfff640 1 -- 172.21.15.88:0/785747284 reap_dead start 2023-10-16T20:39:22.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.241+0000 7f62db7fe640 1 -- 172.21.15.88:0/4187397412 shutdown_connections 2023-10-16T20:39:22.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.245+0000 7f62db7fe640 1 -- 172.21.15.88:0/4187397412 wait complete. 2023-10-16T20:39:22.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.245+0000 7f3401eea640 1 -- 172.21.15.88:0/785747284 shutdown_connections 2023-10-16T20:39:22.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.245+0000 7f3401eea640 1 -- 172.21.15.88:0/785747284 >> 172.21.15.88:0/785747284 conn(0x7f33fc06bcb0 msgr2=0x7f33fc0c9900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.245+0000 7f3401eea640 1 -- 172.21.15.88:0/785747284 shutdown_connections 2023-10-16T20:39:22.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.245+0000 7f3401eea640 1 -- 172.21.15.88:0/785747284 wait complete. 2023-10-16T20:39:22.368 INFO:teuthology.orchestra.run.smithi088.stdout:55834574883 2023-10-16T20:39:22.369 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-16T20:39:22.370 INFO:teuthology.orchestra.run.smithi088.stdout:64424509474 2023-10-16T20:39:22.370 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-16T20:39:22.371 INFO:teuthology.orchestra.run.smithi088.stdout:64424509474 2023-10-16T20:39:22.371 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-16T20:39:22.372 INFO:teuthology.orchestra.run.smithi088.stdout:64424509476 2023-10-16T20:39:22.373 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-16T20:39:22.373 INFO:teuthology.orchestra.run.smithi088.stdout:64424509474 2023-10-16T20:39:22.374 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-16T20:39:22.375 INFO:teuthology.orchestra.run.smithi088.stdout:64424509474 2023-10-16T20:39:22.375 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-16T20:39:22.376 INFO:teuthology.orchestra.run.smithi088.stdout:64424509474 2023-10-16T20:39:22.376 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-16T20:39:22.377 INFO:teuthology.orchestra.run.smithi088.stdout:64424509474 2023-10-16T20:39:22.378 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-16T20:39:22.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.653+0000 7f01d759e640 1 Processor -- start 2023-10-16T20:39:22.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.653+0000 7f01d759e640 1 -- start start 2023-10-16T20:39:22.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.653+0000 7fe8f50dc640 1 Processor -- start 2023-10-16T20:39:22.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.653+0000 7fe8f50dc640 1 -- start start 2023-10-16T20:39:22.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7f01d759e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01d80c7570 0x7f01d80c7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7f01d759e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f01d8155c80 0x7f01d81591d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7f01d759e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d80c6ff0 con 0x7f01d80c2050 2023-10-16T20:39:22.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7f01d759e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d80c6510 con 0x7f01d8155c80 2023-10-16T20:39:22.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7f01d759e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d8156200 con 0x7f01d80c7570 2023-10-16T20:39:22.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7fe8f50dc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe8e800e400 0x7fe8e80dade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7fe8f50dc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8e8005ff0 0x7fe8e8045280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7f01d6d9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f01d80c2050 0x7f01d8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36950/0 (socket says 172.21.15.88:36950) 2023-10-16T20:39:22.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7fe8f50dc640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe8e80457c0 0x7fe8e804b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7f2ca359e640 1 Processor -- start 2023-10-16T20:39:22.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7f01d6d9d640 1 -- 172.21.15.88:0/2752408007 learned_addr learned my addr 172.21.15.88:0/2752408007 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:22.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7f01d5d9b640 1 --2- 172.21.15.88:0/2752408007 >> v2:172.21.15.88:3300/0 conn(0x7f01d8155c80 0x7f01d81591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7fe8effff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe8e800e400 0x7fe8e80dade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7fe8effff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe8e800e400 0x7fe8e80dade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56440/0 (socket says 172.21.15.88:56440) 2023-10-16T20:39:22.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7f2ca359e640 1 -- start start 2023-10-16T20:39:22.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7fe8effff640 1 -- 172.21.15.88:0/3339947350 learned_addr learned my addr 172.21.15.88:0/3339947350 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7fe8ef7fe640 1 --2- 172.21.15.88:0/3339947350 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8e8005ff0 0x7fe8e8045280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7fe8f50dc640 1 -- 172.21.15.88:0/3339947350 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8e8007890 con 0x7fe8e800e400 2023-10-16T20:39:22.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7fe8f50dc640 1 -- 172.21.15.88:0/3339947350 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8e8007400 con 0x7fe8e80457c0 2023-10-16T20:39:22.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7fe8f50dc640 1 -- 172.21.15.88:0/3339947350 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e80067f0 con 0x7fe8e8005ff0 2023-10-16T20:39:22.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7f0bb650f640 1 Processor -- start 2023-10-16T20:39:22.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7fe8f48db640 1 --2- 172.21.15.88:0/3339947350 >> v2:172.21.15.191:3300/0 conn(0x7fe8e80457c0 0x7fe8e804b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7f01d5d9b640 1 -- 172.21.15.88:0/2752408007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01d80c7570 msgr2=0x7f01d80c7950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T20:39:22.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7f01d5d9b640 1 --2- 172.21.15.88:0/2752408007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01d80c7570 0x7f01d80c7950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7f01d5d9b640 1 -- 172.21.15.88:0/2752408007 >> v1:172.21.15.88:6789/0 conn(0x7f01d80c2050 legacy=0x7f01d8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7f01d5d9b640 1 -- 172.21.15.88:0/2752408007 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d8159710 con 0x7f01d8155c80 2023-10-16T20:39:22.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f2ca359e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2c9c04c580 0x7f2c9c0e3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f2ca359e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c9c035c10 con 0x7f2c9c0dd300 2023-10-16T20:39:22.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f2ca359e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c9c0e39e0 con 0x7f2c9c0e35b0 2023-10-16T20:39:22.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f2ca359e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c9c0e3c50 con 0x7f2c9c04c580 2023-10-16T20:39:22.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f2ca259c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2c9c0dd300 0x7f2c9c04ace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36954/0 (socket says 172.21.15.88:36954) 2023-10-16T20:39:22.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f2ca259c640 1 -- 172.21.15.88:0/1135758372 learned_addr learned my addr 172.21.15.88:0/1135758372 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:22.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7f0bb650f640 1 -- start start 2023-10-16T20:39:22.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f0bb650f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0bb0153780 0x7f0bb00c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f0bb650f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0bb0159490 0x7f0bb00c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f0bb650f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb0159d90 con 0x7f0bb00c5930 2023-10-16T20:39:22.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f0bb650f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb015a070 con 0x7f0bb0159490 2023-10-16T20:39:22.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f0bb650f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb015a350 con 0x7f0bb0153780 2023-10-16T20:39:22.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f2ca1d9b640 1 --2- 172.21.15.88:0/1135758372 >> v2:172.21.15.88:3300/0 conn(0x7f2c9c04c580 0x7f2c9c0e3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f2ca159a640 1 -- 172.21.15.88:0/1135758372 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126592469 0 0) 0x7f2c94001120 con 0x7f2c9c0dd300 2023-10-16T20:39:22.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f2ca159a640 1 -- 172.21.15.88:0/1135758372 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c8c003530 con 0x7f2c9c0dd300 2023-10-16T20:39:22.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f2ca159a640 1 -- 172.21.15.88:0/1135758372 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2531735105 0 0) 0x7f2c8c003530 con 0x7f2c9c0dd300 2023-10-16T20:39:22.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f2ca1d9b640 1 -- 172.21.15.88:0/1135758372 >> v1:172.21.15.191:6789/0 conn(0x7f2c9c0e35b0 legacy=0x7f2c9c0461b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f0bb4aac640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0bb00c5930 0x7f0bb00ced80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36964/0 (socket says 172.21.15.88:36964) 2023-10-16T20:39:22.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f0bb4aac640 1 -- 172.21.15.88:0/2019620290 learned_addr learned my addr 172.21.15.88:0/2019620290 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:22.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f0baffff640 1 --2- 172.21.15.88:0/2019620290 >> v2:172.21.15.191:3300/0 conn(0x7f0bb0153780 0x7f0bb00c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f2ca1d9b640 1 -- 172.21.15.88:0/1135758372 >> v1:172.21.15.88:6789/0 conn(0x7f2c9c0dd300 legacy=0x7f2c9c04ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f2ca1d9b640 1 -- 172.21.15.88:0/1135758372 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9c045410 con 0x7f2c9c04c580 2023-10-16T20:39:22.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f0baffff640 1 -- 172.21.15.88:0/2019620290 >> v2:172.21.15.88:3300/0 conn(0x7f0bb0159490 msgr2=0x7f0bb00c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7fe8f48db640 1 -- 172.21.15.88:0/3339947350 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8e8005ff0 msgr2=0x7fe8e8045280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7fe8f48db640 1 --2- 172.21.15.88:0/3339947350 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8e8005ff0 0x7fe8e8045280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7fe8f48db640 1 -- 172.21.15.88:0/3339947350 >> v2:172.21.15.88:3300/0 conn(0x7fe8e800e400 msgr2=0x7fe8e80dade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7fe8f48db640 1 --2- 172.21.15.88:0/3339947350 >> v2:172.21.15.88:3300/0 conn(0x7fe8e800e400 0x7fe8e80dade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.657+0000 7fe8f48db640 1 -- 172.21.15.88:0/3339947350 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e8006ad0 con 0x7fe8e80457c0 2023-10-16T20:39:22.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f0baf7fe640 1 --2- 172.21.15.88:0/2019620290 >> v2:172.21.15.88:3300/0 conn(0x7f0bb0159490 0x7f0bb00c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f0baffff640 1 --2- 172.21.15.88:0/2019620290 >> v2:172.21.15.88:3300/0 conn(0x7f0bb0159490 0x7f0bb00c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.661+0000 7f0baffff640 1 -- 172.21.15.88:0/2019620290 >> v1:172.21.15.88:6789/0 conn(0x7f0bb00c5930 legacy=0x7f0bb00ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7fe8f48db640 1 --2- 172.21.15.88:0/3339947350 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8e80457c0 0x7fe8e804b1f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe8f0063ef0 tx=0x7fe8f00972b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91389b92f1fb91ff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7f0baffff640 1 -- 172.21.15.88:0/2019620290 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb00c4910 con 0x7f0bb0153780 2023-10-16T20:39:22.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7f0baf7fe640 1 --2- 172.21.15.88:0/2019620290 >> v2:172.21.15.88:3300/0 conn(0x7f0bb0159490 0x7f0bb00c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:22.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7f25169d0640 1 Processor -- start 2023-10-16T20:39:22.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7f25169d0640 1 -- start start 2023-10-16T20:39:22.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7fe8eeffd640 1 -- 172.21.15.88:0/3339947350 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e80d1d10 con 0x7fe8e80457c0 2023-10-16T20:39:22.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7fe8eeffd640 1 -- 172.21.15.88:0/3339947350 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f0122050 con 0x7fe8e80457c0 2023-10-16T20:39:22.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7fe8eeffd640 1 -- 172.21.15.88:0/3339947350 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f0126940 con 0x7fe8e80457c0 2023-10-16T20:39:22.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7f01d5d9b640 1 --2- 172.21.15.88:0/2752408007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01d8155c80 0x7f01d81591d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f01c800ba50 tx=0x7f01c801ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89d5e868cdc500a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7fe8f50dc640 1 -- 172.21.15.88:0/3339947350 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8e80457c0 msgr2=0x7fe8e804b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7fe8f50dc640 1 --2- 172.21.15.88:0/3339947350 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8e80457c0 0x7fe8e804b1f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe8f0063ef0 tx=0x7fe8f00972b0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7f25169d0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f251006ae00 0x7f25100c20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7f25169d0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2510157f00 0x7f25100c2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7fe8f50dc640 1 -- 172.21.15.88:0/3339947350 shutdown_connections 2023-10-16T20:39:22.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7fe8f50dc640 1 --2- 172.21.15.88:0/3339947350 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8e80457c0 0x7fe8e804b1f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7fe8f50dc640 1 --2- 172.21.15.88:0/3339947350 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8e8005ff0 0x7fe8e8045280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7fe8f50dc640 1 --2- 172.21.15.88:0/3339947350 >> v2:172.21.15.88:3300/0 conn(0x7fe8e800e400 0x7fe8e80dade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7fe8f50dc640 1 -- 172.21.15.88:0/3339947350 >> 172.21.15.88:0/3339947350 conn(0x7fe8e80502e0 msgr2=0x7fe8e80506f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7f01d559a640 1 -- 172.21.15.88:0/2752408007 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c80a79f0 con 0x7f01d8155c80 2023-10-16T20:39:22.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7f01d559a640 1 -- 172.21.15.88:0/2752408007 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c80ab070 con 0x7f01d8155c80 2023-10-16T20:39:22.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7f01d559a640 1 -- 172.21.15.88:0/2752408007 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c80a7d70 con 0x7f01d8155c80 2023-10-16T20:39:22.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7f250f7fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f251006ae00 0x7f25100c20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7f250ffff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f2510157b20 0x7f2510069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:40118/0 (socket says 172.21.15.88:40118) 2023-10-16T20:39:22.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7f250f7fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f251006ae00 0x7f25100c20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:40210/0 (socket says 172.21.15.88:40210) 2023-10-16T20:39:22.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7f250ffff640 1 -- 172.21.15.88:0/2463110875 learned_addr learned my addr 172.21.15.88:0/2463110875 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:22.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7fe8f50dc640 1 -- 172.21.15.88:0/3339947350 shutdown_connections 2023-10-16T20:39:22.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.665+0000 7fe8f50dc640 1 -- 172.21.15.88:0/3339947350 wait complete. 2023-10-16T20:39:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f2514f6d640 1 --2- 172.21.15.88:0/2463110875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2510157f00 0x7f25100c2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f25169d0640 1 -- 172.21.15.88:0/2463110875 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f251015dba0 con 0x7f2510157b20 2023-10-16T20:39:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f25169d0640 1 -- 172.21.15.88:0/2463110875 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f251015d9b0 con 0x7f251006ae00 2023-10-16T20:39:22.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f0baffff640 1 --2- 172.21.15.88:0/2019620290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bb0153780 0x7f0bb00c9be0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0bb00c9900 tx=0x7f0ba801f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=818ac7dc7665aa03 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f25169d0640 1 -- 172.21.15.88:0/2463110875 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2510072a10 con 0x7f2510157f00 2023-10-16T20:39:22.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f2ca1d9b640 1 --2- 172.21.15.88:0/1135758372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c9c04c580 0x7f2c9c0e3070 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2c980097c0 tx=0x7f2c9801eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5034d8663a3b7c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f0baeffd640 1 -- 172.21.15.88:0/2019620290 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba80af600 con 0x7f0bb0153780 2023-10-16T20:39:22.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f0baeffd640 1 -- 172.21.15.88:0/2019620290 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba80ab050 con 0x7f0bb0153780 2023-10-16T20:39:22.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f0baeffd640 1 -- 172.21.15.88:0/2019620290 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba80af980 con 0x7f0bb0153780 2023-10-16T20:39:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f250f7fe640 1 -- 172.21.15.88:0/2463110875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2510157f00 msgr2=0x7f25100c2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f250f7fe640 1 --2- 172.21.15.88:0/2463110875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2510157f00 0x7f25100c2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f01d759e640 1 -- 172.21.15.88:0/2752408007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01d8155c80 msgr2=0x7f01d81591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f01d759e640 1 --2- 172.21.15.88:0/2752408007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01d8155c80 0x7f01d81591d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f01c800ba50 tx=0x7f01c801ea70 comp rx=0 tx=0).stop 2023-10-16T20:39:22.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f250f7fe640 1 -- 172.21.15.88:0/2463110875 >> v1:172.21.15.191:6789/0 conn(0x7f2510157b20 legacy=0x7f2510069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f250effd640 1 -- 172.21.15.88:0/2463110875 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427517236 0 0) 0x7f251015dba0 con 0x7f2510157b20 2023-10-16T20:39:22.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7fe8f50dc640 1 Processor -- start 2023-10-16T20:39:22.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f01d759e640 1 -- 172.21.15.88:0/2752408007 shutdown_connections 2023-10-16T20:39:22.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f01d759e640 1 --2- 172.21.15.88:0/2752408007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01d8155c80 0x7f01d81591d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f01d759e640 1 --2- 172.21.15.88:0/2752408007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01d80c7570 0x7f01d80c7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f01d759e640 1 -- 172.21.15.88:0/2752408007 >> 172.21.15.88:0/2752408007 conn(0x7f01d806d1e0 msgr2=0x7f01d806d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f01d759e640 1 -- 172.21.15.88:0/2752408007 shutdown_connections 2023-10-16T20:39:22.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f30a289e640 1 Processor -- start 2023-10-16T20:39:22.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f30a289e640 1 -- start start 2023-10-16T20:39:22.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f250f7fe640 1 -- 172.21.15.88:0/2463110875 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251015de80 con 0x7f251006ae00 2023-10-16T20:39:22.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7fe8f50dc640 1 -- start start 2023-10-16T20:39:22.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f2ca159a640 1 -- 172.21.15.88:0/1135758372 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c940017f0 con 0x7f2c9c04c580 2023-10-16T20:39:22.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f2ca159a640 1 -- 172.21.15.88:0/1135758372 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c980a8610 con 0x7f2c9c04c580 2023-10-16T20:39:22.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f2ca159a640 1 -- 172.21.15.88:0/1135758372 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c980a89a0 con 0x7f2c9c04c580 2023-10-16T20:39:22.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f2ca359e640 1 -- 172.21.15.88:0/1135758372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c9c04c580 msgr2=0x7f2c9c0e3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f2ca359e640 1 --2- 172.21.15.88:0/1135758372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c9c04c580 0x7f2c9c0e3070 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2c980097c0 tx=0x7f2c9801eb90 comp rx=0 tx=0).stop 2023-10-16T20:39:22.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f0bb650f640 1 -- 172.21.15.88:0/2019620290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bb0153780 msgr2=0x7f0bb00c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f0bb650f640 1 --2- 172.21.15.88:0/2019620290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bb0153780 0x7f0bb00c9be0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0bb00c9900 tx=0x7f0ba801f150 comp rx=0 tx=0).stop 2023-10-16T20:39:22.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f0bb650f640 1 -- 172.21.15.88:0/2019620290 shutdown_connections 2023-10-16T20:39:22.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f0bb650f640 1 --2- 172.21.15.88:0/2019620290 >> v2:172.21.15.88:3300/0 conn(0x7f0bb0159490 0x7f0bb00c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f0bb650f640 1 --2- 172.21.15.88:0/2019620290 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bb0153780 0x7f0bb00c9be0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f0bb650f640 1 -- 172.21.15.88:0/2019620290 >> 172.21.15.88:0/2019620290 conn(0x7f0bb006bcb0 msgr2=0x7f0bb006c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f0bb650f640 1 -- 172.21.15.88:0/2019620290 shutdown_connections 2023-10-16T20:39:22.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f0bb650f640 1 -- 172.21.15.88:0/2019620290 wait complete. 2023-10-16T20:39:22.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f2514f6d640 1 --2- 172.21.15.88:0/2463110875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2510157f00 0x7f25100c2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:39:22.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f2ca359e640 1 -- 172.21.15.88:0/1135758372 shutdown_connections 2023-10-16T20:39:22.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f30a289e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f309c153780 0x7f309c0c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f2ca359e640 1 --2- 172.21.15.88:0/1135758372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c9c04c580 0x7f2c9c0e3070 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7fe8f50dc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8e8005ff0 0x7fe8e8048900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7fe8f50dc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8e800e400 0x7fe8e804a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f30a289e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f309c159490 0x7f309c0c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f2ca359e640 1 -- 172.21.15.88:0/1135758372 >> 172.21.15.88:0/1135758372 conn(0x7f2c9c0501d0 msgr2=0x7f2c9c0505e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7fe8f50dc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8e80457c0 0x7fe8e80d5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f2ca359e640 1 -- 172.21.15.88:0/1135758372 shutdown_connections 2023-10-16T20:39:22.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f2ca359e640 1 -- 172.21.15.88:0/1135758372 wait complete. 2023-10-16T20:39:22.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7fe8ef7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8e800e400 0x7fe8e804a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7fe8ef7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8e800e400 0x7fe8e804a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:40226/0 (socket says 172.21.15.88:40226) 2023-10-16T20:39:22.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f30a289e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f309c159d90 con 0x7f309c0c5930 2023-10-16T20:39:22.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f0bb650f640 1 Processor -- start 2023-10-16T20:39:22.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f30a289e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309c15a070 con 0x7f309c159490 2023-10-16T20:39:22.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.669+0000 7f01d759e640 1 -- 172.21.15.88:0/2752408007 wait complete. 2023-10-16T20:39:22.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7fe8ef7fe640 1 -- 172.21.15.88:0/476655804 learned_addr learned my addr 172.21.15.88:0/476655804 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f01d759e640 1 Processor -- start 2023-10-16T20:39:22.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7fe8f50dc640 1 -- 172.21.15.88:0/476655804 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e8049f20 con 0x7fe8e80457c0 2023-10-16T20:39:22.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7fe8f50dc640 1 -- 172.21.15.88:0/476655804 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e8049070 con 0x7fe8e800e400 2023-10-16T20:39:22.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7fe8f50dc640 1 -- 172.21.15.88:0/476655804 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e80492a0 con 0x7fe8e8005ff0 2023-10-16T20:39:22.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f0bb650f640 1 -- start start 2023-10-16T20:39:22.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f30a289e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309c15a350 con 0x7f309c153780 2023-10-16T20:39:22.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f309bfff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f309c153780 0x7f309c0c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7fe8effff640 1 --2- 172.21.15.88:0/476655804 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8e8005ff0 0x7fe8e8048900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f250f7fe640 1 --2- 172.21.15.88:0/2463110875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f251006ae00 0x7f25100c20e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f25000097f0 tx=0x7f250001ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a78f4895c19234c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f01d759e640 1 -- start start 2023-10-16T20:39:22.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7fe8ef7fe640 1 -- 172.21.15.88:0/476655804 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8e8005ff0 msgr2=0x7fe8e8048900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7fe8ef7fe640 1 --2- 172.21.15.88:0/476655804 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8e8005ff0 0x7fe8e8048900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7fe8ef7fe640 1 -- 172.21.15.88:0/476655804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8e80457c0 msgr2=0x7fe8e80d5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7fe8ef7fe640 1 --2- 172.21.15.88:0/476655804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8e80457c0 0x7fe8e80d5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7fe8ef7fe640 1 -- 172.21.15.88:0/476655804 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f004f120 con 0x7fe8e800e400 2023-10-16T20:39:22.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7fe8ef7fe640 1 --2- 172.21.15.88:0/476655804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8e800e400 0x7fe8e804a340 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe8e000d620 tx=0x7fe8e000dae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f0bb650f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb00c5930 0x7f0bb0151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f250effd640 1 -- 172.21.15.88:0/2463110875 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25000a8970 con 0x7f251006ae00 2023-10-16T20:39:22.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f250effd640 1 -- 172.21.15.88:0/2463110875 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25000a8ad0 con 0x7f251006ae00 2023-10-16T20:39:22.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f250effd640 1 -- 172.21.15.88:0/2463110875 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25000a8e60 con 0x7f251006ae00 2023-10-16T20:39:22.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f0bb650f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bb0153780 0x7f0bb0158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f0bb650f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0bb0159490 0x7f0bb01589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f0bb650f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb014dc70 con 0x7f0bb00c5930 2023-10-16T20:39:22.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f0bb650f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb014f300 con 0x7f0bb0153780 2023-10-16T20:39:22.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f0bb650f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb014d9d0 con 0x7f0bb0159490 2023-10-16T20:39:22.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7fe8ecff9640 1 -- 172.21.15.88:0/476655804 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e000b620 con 0x7fe8e800e400 2023-10-16T20:39:22.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f309bfff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f309c153780 0x7f309c0c9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:40218/0 (socket says 172.21.15.88:40218) 2023-10-16T20:39:22.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f309bfff640 1 -- 172.21.15.88:0/3060293586 learned_addr learned my addr 172.21.15.88:0/3060293586 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f309bfff640 1 -- 172.21.15.88:0/3060293586 >> v2:172.21.15.88:3300/0 conn(0x7f309c159490 msgr2=0x7f309c0c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f309bfff640 1 --2- 172.21.15.88:0/3060293586 >> v2:172.21.15.88:3300/0 conn(0x7f309c159490 0x7f309c0c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f309bfff640 1 -- 172.21.15.88:0/3060293586 >> v1:172.21.15.88:6789/0 conn(0x7f309c0c5930 legacy=0x7f309c0ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.673+0000 7f309bfff640 1 -- 172.21.15.88:0/3060293586 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309c0c4910 con 0x7f309c153780 2023-10-16T20:39:22.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f0bb4aac640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0bb0159490 0x7f0bb01589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f0bb4aac640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0bb0159490 0x7f0bb01589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:37680/0 (socket says 172.21.15.88:37680) 2023-10-16T20:39:22.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f0bb4aac640 1 -- 172.21.15.88:0/770091794 learned_addr learned my addr 172.21.15.88:0/770091794 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f2ca359e640 1 Processor -- start 2023-10-16T20:39:22.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f0baf7fe640 1 --2- 172.21.15.88:0/770091794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bb0153780 0x7f0bb0158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7fe8ecff9640 1 -- 172.21.15.88:0/476655804 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e000b780 con 0x7fe8e800e400 2023-10-16T20:39:22.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7fe8f50dc640 1 -- 172.21.15.88:0/476655804 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e8049580 con 0x7fe8e800e400 2023-10-16T20:39:22.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7fe8f50dc640 1 -- 172.21.15.88:0/476655804 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e8049b40 con 0x7fe8e800e400 2023-10-16T20:39:22.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7fe8ecff9640 1 -- 172.21.15.88:0/476655804 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e000bb10 con 0x7fe8e800e400 2023-10-16T20:39:22.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f2ca359e640 1 -- start start 2023-10-16T20:39:22.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f0bb4aac640 1 -- 172.21.15.88:0/770091794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bb0153780 msgr2=0x7f0bb0158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f0bb4aac640 1 --2- 172.21.15.88:0/770091794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bb0153780 0x7f0bb0158470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f0bb4aac640 1 -- 172.21.15.88:0/770091794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb00c5930 msgr2=0x7f0bb0151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f0bb4aac640 1 --2- 172.21.15.88:0/770091794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb00c5930 0x7f0bb0151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f0bb4aac640 1 -- 172.21.15.88:0/770091794 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba8009a70 con 0x7f0bb0159490 2023-10-16T20:39:22.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f25169d0640 1 -- 172.21.15.88:0/2463110875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f251006ae00 msgr2=0x7f25100c20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f25169d0640 1 --2- 172.21.15.88:0/2463110875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f251006ae00 0x7f25100c20e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f25000097f0 tx=0x7f250001ec90 comp rx=0 tx=0).stop 2023-10-16T20:39:22.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f25169d0640 1 -- 172.21.15.88:0/2463110875 shutdown_connections 2023-10-16T20:39:22.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f25169d0640 1 --2- 172.21.15.88:0/2463110875 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2510157f00 0x7f25100c2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f25169d0640 1 --2- 172.21.15.88:0/2463110875 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f251006ae00 0x7f25100c20e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f01d759e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01d80c2050 0x7f01d8148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f01d759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01d80c7570 0x7f01d8145490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f01d759e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01d8155c80 0x7f01d81459d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f25169d0640 1 -- 172.21.15.88:0/2463110875 >> 172.21.15.88:0/2463110875 conn(0x7f25100ca9f0 msgr2=0x7f25100cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f01d759e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d814a020 con 0x7f01d80c7570 2023-10-16T20:39:22.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f01d759e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d81f1c70 con 0x7f01d8155c80 2023-10-16T20:39:22.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f01d759e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d81ede50 con 0x7f01d80c2050 2023-10-16T20:39:22.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f0bb4aac640 1 --2- 172.21.15.88:0/770091794 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0bb0159490 0x7f0bb01589b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0ba400c3c0 tx=0x7f0ba400c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f01d659c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01d80c2050 0x7f01d8148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f01d659c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01d80c2050 0x7f01d8148ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:37694/0 (socket says 172.21.15.88:37694) 2023-10-16T20:39:22.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f01d659c640 1 -- 172.21.15.88:0/3427918045 learned_addr learned my addr 172.21.15.88:0/3427918045 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f0bacff9640 1 -- 172.21.15.88:0/770091794 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4005cf0 con 0x7f0bb0159490 2023-10-16T20:39:22.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f0bacff9640 1 -- 172.21.15.88:0/770091794 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba4005e70 con 0x7f0bb0159490 2023-10-16T20:39:22.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f0bacff9640 1 -- 172.21.15.88:0/770091794 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba40072f0 con 0x7f0bb0159490 2023-10-16T20:39:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f0bb650f640 1 -- 172.21.15.88:0/770091794 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb015a860 con 0x7f0bb0159490 2023-10-16T20:39:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f0bb650f640 1 -- 172.21.15.88:0/770091794 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb014ec40 con 0x7f0bb0159490 2023-10-16T20:39:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f2ca359e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c9c04c580 0x7f2c9c0db9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f2ca359e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c9c0dd300 0x7f2c9c0ced90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f2ca359e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c9c0e35b0 0x7f2c9c0d9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f2ca359e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c0e2f90 con 0x7f2c9c04c580 2023-10-16T20:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f2ca359e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c17ba60 con 0x7f2c9c0e35b0 2023-10-16T20:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f2ca359e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c0e33b0 con 0x7f2c9c0dd300 2023-10-16T20:39:22.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f47d93a7640 1 Processor -- start 2023-10-16T20:39:22.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f01d6d9d640 1 --2- 172.21.15.88:0/3427918045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01d8155c80 0x7f01d81459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f01d659c640 1 -- 172.21.15.88:0/3427918045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01d8155c80 msgr2=0x7f01d81459d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f01d659c640 1 --2- 172.21.15.88:0/3427918045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01d8155c80 0x7f01d81459d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f01d659c640 1 -- 172.21.15.88:0/3427918045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01d80c7570 msgr2=0x7f01d8145490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f01d659c640 1 --2- 172.21.15.88:0/3427918045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01d80c7570 0x7f01d8145490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f01d659c640 1 -- 172.21.15.88:0/3427918045 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c800b6b0 con 0x7f01d80c2050 2023-10-16T20:39:22.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f47d93a7640 1 -- start start 2023-10-16T20:39:22.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7f25169d0640 1 -- 172.21.15.88:0/2463110875 shutdown_connections 2023-10-16T20:39:22.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f2ca2d9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c9c0e35b0 0x7f2c9c0d9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f2ca1d9b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c9c0dd300 0x7f2c9c0ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f2ca2d9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c9c0e35b0 0x7f2c9c0d9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:40250/0 (socket says 172.21.15.88:40250) 2023-10-16T20:39:22.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f2ca2d9d640 1 -- 172.21.15.88:0/3337933951 learned_addr learned my addr 172.21.15.88:0/3337933951 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f01d659c640 1 --2- 172.21.15.88:0/3427918045 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01d80c2050 0x7f01d8148ed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00b4a0 tx=0x7f01cc00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f0bb650f640 1 -- 172.21.15.88:0/770091794 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb0074700 con 0x7f0bb0159490 2023-10-16T20:39:22.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f25169d0640 1 -- 172.21.15.88:0/2463110875 wait complete. 2023-10-16T20:39:22.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.677+0000 7fe8f50dc640 1 -- 172.21.15.88:0/476655804 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d0005260 con 0x7fe8e800e400 2023-10-16T20:39:22.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7fe8ecff9640 1 -- 172.21.15.88:0/476655804 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe8e000bc70 con 0x7fe8e800e400 2023-10-16T20:39:22.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f2ca259c640 1 --2- 172.21.15.88:0/3337933951 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c9c04c580 0x7f2c9c0db9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f01c6ffd640 1 -- 172.21.15.88:0/3427918045 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc010610 con 0x7f01d80c2050 2023-10-16T20:39:22.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f01d759e640 1 -- 172.21.15.88:0/3427918045 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d81f1e60 con 0x7f01d80c2050 2023-10-16T20:39:22.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f2ca2d9d640 1 -- 172.21.15.88:0/3337933951 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c9c0dd300 msgr2=0x7f2c9c0ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f2ca2d9d640 1 --2- 172.21.15.88:0/3337933951 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c9c0dd300 0x7f2c9c0ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f2ca2d9d640 1 -- 172.21.15.88:0/3337933951 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c9c04c580 msgr2=0x7f2c9c0db9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f2ca2d9d640 1 --2- 172.21.15.88:0/3337933951 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c9c04c580 0x7f2c9c0db9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f2ca2d9d640 1 -- 172.21.15.88:0/3337933951 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c98009420 con 0x7f2c9c0e35b0 2023-10-16T20:39:22.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f01d759e640 1 -- 172.21.15.88:0/3427918045 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d81ee3d0 con 0x7f01d80c2050 2023-10-16T20:39:22.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f01c6ffd640 1 -- 172.21.15.88:0/3427918045 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01cc010770 con 0x7f01d80c2050 2023-10-16T20:39:22.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f01c6ffd640 1 -- 172.21.15.88:0/3427918045 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc010610 con 0x7f01d80c2050 2023-10-16T20:39:22.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f2ca259c640 1 --2- 172.21.15.88:0/3337933951 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c9c04c580 0x7f2c9c0db9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:22.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f0bacff9640 1 -- 172.21.15.88:0/770091794 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0ba4007450 con 0x7f0bb0159490 2023-10-16T20:39:22.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f47d93a7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f47d40c67a0 0x7f47d4162af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f47d93a7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d4151640 con 0x7f47d40c6310 2023-10-16T20:39:22.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f47d93a7640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d40c6bd0 con 0x7f47d4151260 2023-10-16T20:39:22.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f47d93a7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47d40c5770 con 0x7f47d40c67a0 2023-10-16T20:39:22.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7fe8ecff9640 1 --2- 172.21.15.88:0/476655804 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe8dc0664a0 0x7fe8dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f2ca2d9d640 1 --2- 172.21.15.88:0/3337933951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c9c0e35b0 0x7f2c9c0d9720 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4067e50 tx=0x7f2ca404eb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7fe8effff640 1 --2- 172.21.15.88:0/476655804 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe8dc0664a0 0x7fe8dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f47d3fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f47d4151260 0x7f47d40720e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:40132/0 (socket says 172.21.15.88:40132) 2023-10-16T20:39:22.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f47d3fff640 1 -- 172.21.15.88:0/3016054804 learned_addr learned my addr 172.21.15.88:0/3016054804 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:22.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7fe8ecff9640 1 -- 172.21.15.88:0/476655804 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe8e0084780 con 0x7fe8e800e400 2023-10-16T20:39:22.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f0bacff9640 1 --2- 172.21.15.88:0/770091794 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0b9c0665a0 0x7f0b9c068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f25169d0640 1 Processor -- start 2023-10-16T20:39:22.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f47d8ba6640 1 --2- 172.21.15.88:0/3016054804 >> v2:172.21.15.88:3300/0 conn(0x7f47d40c67a0 0x7f47d4162af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f0bacff9640 1 -- 172.21.15.88:0/770091794 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0ba4083650 con 0x7f0bb0159490 2023-10-16T20:39:22.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f25169d0640 1 -- start start 2023-10-16T20:39:22.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f0baffff640 1 --2- 172.21.15.88:0/770091794 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0b9c0665a0 0x7f0b9c068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f309bfff640 1 --2- 172.21.15.88:0/3060293586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f309c153780 0x7f309c0c9be0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f309c0cfef0 tx=0x7f308c01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5919a2badfb01d0c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7fe8effff640 1 --2- 172.21.15.88:0/476655804 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe8dc0664a0 0x7fe8dc068960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4009a30 tx=0x7fe8e4009240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.681+0000 7f01d759e640 1 -- 172.21.15.88:0/3427918045 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01bc005260 con 0x7f01d80c2050 2023-10-16T20:39:22.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f01c6ffd640 1 -- 172.21.15.88:0/3427918045 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f01cc0108e0 con 0x7f01d80c2050 2023-10-16T20:39:22.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f47d37fe640 1 -- 172.21.15.88:0/3016054804 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615156226 0 0) 0x7f47d4151640 con 0x7f47d40c6310 2023-10-16T20:39:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f47d37fe640 1 -- 172.21.15.88:0/3016054804 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c0003530 con 0x7f47d40c6310 2023-10-16T20:39:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f25169d0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f251006ae00 0x7f25101597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f25169d0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2510157b20 0x7f2510159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f0baffff640 1 --2- 172.21.15.88:0/770091794 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0b9c0665a0 0x7f0b9c068a60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0ba80b1000 tx=0x7f0ba8004ac0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f25169d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2510157f00 0x7f25101f6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f25169d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2510159390 con 0x7f2510157f00 2023-10-16T20:39:22.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f25169d0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25101f6fd0 con 0x7f2510157b20 2023-10-16T20:39:22.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f25169d0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25100c1e80 con 0x7f251006ae00 2023-10-16T20:39:22.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f250f7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2510157b20 0x7f2510159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f250f7fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2510157b20 0x7f2510159cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:40258/0 (socket says 172.21.15.88:40258) 2023-10-16T20:39:22.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f250f7fe640 1 -- 172.21.15.88:0/2136331067 learned_addr learned my addr 172.21.15.88:0/2136331067 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f2514f6d640 1 --2- 172.21.15.88:0/2136331067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2510157f00 0x7f25101f6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f47d8ba6640 1 -- 172.21.15.88:0/3016054804 >> v1:172.21.15.191:6789/0 conn(0x7f47d4151260 legacy=0x7f47d40720e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f47d8ba6640 1 -- 172.21.15.88:0/3016054804 >> v1:172.21.15.88:6789/0 conn(0x7f47d40c6310 legacy=0x7f47d4157820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f47d8ba6640 1 -- 172.21.15.88:0/3016054804 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d40c41e0 con 0x7f47d40c67a0 2023-10-16T20:39:22.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f250ffff640 1 --2- 172.21.15.88:0/2136331067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f251006ae00 0x7f25101597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f250f7fe640 1 -- 172.21.15.88:0/2136331067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f251006ae00 msgr2=0x7f25101597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f250f7fe640 1 --2- 172.21.15.88:0/2136331067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f251006ae00 0x7f25101597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f250f7fe640 1 -- 172.21.15.88:0/2136331067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2510157f00 msgr2=0x7f25101f6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f250f7fe640 1 --2- 172.21.15.88:0/2136331067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2510157f00 0x7f25101f6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f250f7fe640 1 -- 172.21.15.88:0/2136331067 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f8009420 con 0x7f2510157b20 2023-10-16T20:39:22.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f2514f6d640 1 --2- 172.21.15.88:0/2136331067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2510157f00 0x7f25101f6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:39:22.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f01c6ffd640 1 --2- 172.21.15.88:0/3427918045 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f01b8066500 0x7f01b80689c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f01c6ffd640 1 -- 172.21.15.88:0/3427918045 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f01cc083a90 con 0x7f01d80c2050 2023-10-16T20:39:22.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f250f7fe640 1 --2- 172.21.15.88:0/2136331067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2510157b20 0x7f2510159cf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f250001c8c0 tx=0x7f250001ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f01d5d9b640 1 --2- 172.21.15.88:0/3427918045 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f01b8066500 0x7f01b80689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f8ec7fff640 1 Processor -- start 2023-10-16T20:39:22.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f8ec7fff640 1 -- start start 2023-10-16T20:39:22.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f2c92ffd640 1 -- 172.21.15.88:0/3337933951 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4072150 con 0x7f2c9c0e35b0 2023-10-16T20:39:22.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f2ca359e640 1 -- 172.21.15.88:0/3337933951 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c9c0dae50 con 0x7f2c9c0e35b0 2023-10-16T20:39:22.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f250cff9640 1 -- 172.21.15.88:0/2136331067 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25000b0e10 con 0x7f2510157b20 2023-10-16T20:39:22.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f309affd640 1 -- 172.21.15.88:0/3060293586 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c0a8990 con 0x7f309c153780 2023-10-16T20:39:22.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f25169d0640 1 -- 172.21.15.88:0/2136331067 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2500009420 con 0x7f2510157b20 2023-10-16T20:39:22.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f25169d0640 1 -- 172.21.15.88:0/2136331067 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2510153620 con 0x7f2510157b20 2023-10-16T20:39:22.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7fe8ecff9640 1 -- 172.21.15.88:0/476655804 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe8e0051710 con 0x7fe8e800e400 2023-10-16T20:39:22.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f01d5d9b640 1 --2- 172.21.15.88:0/3427918045 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f01b8066500 0x7f01b80689c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f01c80b2000 tx=0x7f01c8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f2c92ffd640 1 -- 172.21.15.88:0/3337933951 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca40722b0 con 0x7f2c9c0e35b0 2023-10-16T20:39:22.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f0bacff9640 1 -- 172.21.15.88:0/770091794 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0ba4050830 con 0x7f0bb0159490 2023-10-16T20:39:22.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f2c92ffd640 1 -- 172.21.15.88:0/3337933951 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4072620 con 0x7f2c9c0e35b0 2023-10-16T20:39:22.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f2ca359e640 1 -- 172.21.15.88:0/3337933951 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c9c0e25f0 con 0x7f2c9c0e35b0 2023-10-16T20:39:22.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f309affd640 1 -- 172.21.15.88:0/3060293586 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308c0a8af0 con 0x7f309c153780 2023-10-16T20:39:22.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.685+0000 7f309affd640 1 -- 172.21.15.88:0/3060293586 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c0a8e80 con 0x7f309c153780 2023-10-16T20:39:22.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f250cff9640 1 -- 172.21.15.88:0/2136331067 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2500006e20 con 0x7f2510157b20 2023-10-16T20:39:22.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f250cff9640 1 -- 172.21.15.88:0/2136331067 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25000b9630 con 0x7f2510157b20 2023-10-16T20:39:22.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f30a289e640 1 -- 172.21.15.88:0/3060293586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f309c153780 msgr2=0x7f309c0c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f30a289e640 1 --2- 172.21.15.88:0/3060293586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f309c153780 0x7f309c0c9be0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f309c0cfef0 tx=0x7f308c01ec90 comp rx=0 tx=0).stop 2023-10-16T20:39:22.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f30a289e640 1 -- 172.21.15.88:0/3060293586 shutdown_connections 2023-10-16T20:39:22.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f30a289e640 1 --2- 172.21.15.88:0/3060293586 >> v2:172.21.15.88:3300/0 conn(0x7f309c159490 0x7f309c0c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f30a289e640 1 --2- 172.21.15.88:0/3060293586 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f309c153780 0x7f309c0c9be0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f30a289e640 1 -- 172.21.15.88:0/3060293586 >> 172.21.15.88:0/3060293586 conn(0x7f309c06bcb0 msgr2=0x7f309c06c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f30a289e640 1 -- 172.21.15.88:0/3060293586 shutdown_connections 2023-10-16T20:39:22.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f2ca359e640 1 -- 172.21.15.88:0/3337933951 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c9c005350 con 0x7f2c9c0e35b0 2023-10-16T20:39:22.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f30a289e640 1 -- 172.21.15.88:0/3060293586 wait complete. 2023-10-16T20:39:22.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f47d8ba6640 1 --2- 172.21.15.88:0/3016054804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d40c67a0 0x7f47d4162af0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009d90 tx=0x7f47cc01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9c25ff56961090 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f47d37fe640 1 -- 172.21.15.88:0/3016054804 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc0af600 con 0x7f47d40c67a0 2023-10-16T20:39:22.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f47d37fe640 1 -- 172.21.15.88:0/3016054804 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47cc0ab050 con 0x7f47d40c67a0 2023-10-16T20:39:22.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f47d37fe640 1 -- 172.21.15.88:0/3016054804 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc0af980 con 0x7f47d40c67a0 2023-10-16T20:39:22.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f25169d0640 1 -- 172.21.15.88:0/2136331067 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f4005260 con 0x7f2510157b20 2023-10-16T20:39:22.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f2c92ffd640 1 -- 172.21.15.88:0/3337933951 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2ca4091020 con 0x7f2c9c0e35b0 2023-10-16T20:39:22.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f47d93a7640 1 -- 172.21.15.88:0/3016054804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d40c67a0 msgr2=0x7f47d4162af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f47d93a7640 1 --2- 172.21.15.88:0/3016054804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d40c67a0 0x7f47d4162af0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009d90 tx=0x7f47cc01f130 comp rx=0 tx=0).stop 2023-10-16T20:39:22.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f250cff9640 1 -- 172.21.15.88:0/2136331067 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f25000b7040 con 0x7f2510157b20 2023-10-16T20:39:22.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f47d93a7640 1 -- 172.21.15.88:0/3016054804 shutdown_connections 2023-10-16T20:39:22.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f47d93a7640 1 --2- 172.21.15.88:0/3016054804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d40c67a0 0x7f47d4162af0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f47d93a7640 1 -- 172.21.15.88:0/3016054804 >> 172.21.15.88:0/3016054804 conn(0x7f47d4065a80 msgr2=0x7f47d4065e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f8ec7fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8ec80c9600 0x7f8ec80c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f30a289e640 1 Processor -- start 2023-10-16T20:39:22.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f2c92ffd640 1 --2- 172.21.15.88:0/3337933951 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2c840664a0 0x7f2c84068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f47d93a7640 1 -- 172.21.15.88:0/3016054804 shutdown_connections 2023-10-16T20:39:22.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.689+0000 7f47d93a7640 1 -- 172.21.15.88:0/3016054804 wait complete. 2023-10-16T20:39:22.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f2c92ffd640 1 -- 172.21.15.88:0/3337933951 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2ca40ce0a0 con 0x7f2c9c0e35b0 2023-10-16T20:39:22.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f8ec7fff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec80ca3e0 0x7f8ec815ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f8ec7fff640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec8159460 con 0x7f8ec81536a0 2023-10-16T20:39:22.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f8ec7fff640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ec80cb270 con 0x7f8ec80c9600 2023-10-16T20:39:22.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f8ec7fff640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec8159720 con 0x7f8ec80ca3e0 2023-10-16T20:39:22.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f2ca259c640 1 --2- 172.21.15.88:0/3337933951 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2c840664a0 0x7f2c84068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f250cff9640 1 --2- 172.21.15.88:0/2136331067 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f24f00664a0 0x7f24f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f250cff9640 1 -- 172.21.15.88:0/2136331067 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2500121d30 con 0x7f2510157b20 2023-10-16T20:39:22.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f250ffff640 1 --2- 172.21.15.88:0/2136331067 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f24f00664a0 0x7f24f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f8ec6ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8ec81536a0 0x7f8ec80ca9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36996/0 (socket says 172.21.15.88:36996) 2023-10-16T20:39:22.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f8ec6ffd640 1 -- 172.21.15.88:0/2187195956 learned_addr learned my addr 172.21.15.88:0/2187195956 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:22.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f2ca259c640 1 --2- 172.21.15.88:0/3337933951 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2c840664a0 0x7f2c84068960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2c94007c90 tx=0x7f2c94009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f01c6ffd640 1 -- 172.21.15.88:0/3427918045 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f01cc050b60 con 0x7f01d80c2050 2023-10-16T20:39:22.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f8ec77fe640 1 --2- 172.21.15.88:0/2187195956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec80ca3e0 0x7f8ec815ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f8ebffff640 1 --2- 172.21.15.88:0/2187195956 >> v2:172.21.15.88:3300/0 conn(0x7f8ec80c9600 0x7f8ec80c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f8ec67fc640 1 -- 172.21.15.88:0/2187195956 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540693183 0 0) 0x7f8ec8159460 con 0x7f8ec81536a0 2023-10-16T20:39:22.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f30a289e640 1 -- start start 2023-10-16T20:39:22.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f250ffff640 1 --2- 172.21.15.88:0/2136331067 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f24f00664a0 0x7f24f0068960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f25080012f0 tx=0x7f2508001770 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f30a289e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f309c0c5930 0x7f309c15d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f30a289e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f309c153780 0x7f309c147950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f30a289e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f309c159490 0x7f309c158570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f30a289e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c15d620 con 0x7f309c159490 2023-10-16T20:39:22.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f30a289e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c14fa70 con 0x7f309c153780 2023-10-16T20:39:22.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f30a289e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309c157b20 con 0x7f309c0c5930 2023-10-16T20:39:22.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f309bfff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f309c0c5930 0x7f309c15d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f309bfff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f309c0c5930 0x7f309c15d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:37730/0 (socket says 172.21.15.88:37730) 2023-10-16T20:39:22.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f309bfff640 1 -- 172.21.15.88:0/2133868088 learned_addr learned my addr 172.21.15.88:0/2133868088 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f47d93a7640 1 Processor -- start 2023-10-16T20:39:22.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f8ec77fe640 1 -- 172.21.15.88:0/2187195956 >> v2:172.21.15.88:3300/0 conn(0x7f8ec80c9600 msgr2=0x7f8ec80c9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:22.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f8ec77fe640 1 --2- 172.21.15.88:0/2187195956 >> v2:172.21.15.88:3300/0 conn(0x7f8ec80c9600 0x7f8ec80c9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f8ec77fe640 1 -- 172.21.15.88:0/2187195956 >> v1:172.21.15.88:6789/0 conn(0x7f8ec81536a0 legacy=0x7f8ec80ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f8ec77fe640 1 -- 172.21.15.88:0/2187195956 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec8159a00 con 0x7f8ec80ca3e0 2023-10-16T20:39:22.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f309bfff640 1 -- 172.21.15.88:0/2133868088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f309c153780 msgr2=0x7f309c147950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-16T20:39:22.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f309bfff640 1 --2- 172.21.15.88:0/2133868088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f309c153780 0x7f309c147950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f309bfff640 1 -- 172.21.15.88:0/2133868088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f309c159490 msgr2=0x7f309c158570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-16T20:39:22.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f47d93a7640 1 -- start start 2023-10-16T20:39:22.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f30a0e3b640 1 --2- 172.21.15.88:0/2133868088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f309c159490 0x7f309c158570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f2c92ffd640 1 -- 172.21.15.88:0/3337933951 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2ca4102060 con 0x7f2c9c0e35b0 2023-10-16T20:39:22.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f309bfff640 1 --2- 172.21.15.88:0/2133868088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f309c159490 0x7f309c158570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.693+0000 7f309bfff640 1 -- 172.21.15.88:0/2133868088 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c009480 con 0x7f309c0c5930 2023-10-16T20:39:22.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f30a0e3b640 1 --2- 172.21.15.88:0/2133868088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f309c159490 0x7f309c158570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:22.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f309bfff640 1 --2- 172.21.15.88:0/2133868088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f309c0c5930 0x7f309c15d0e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f308c01c8c0 tx=0x7f308c01ba30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d93a7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d40c6310 0x7f47d41558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d93a7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47d40c67a0 0x7f47d40cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d93a7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47d4151260 0x7f47d414dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d93a7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d41563a0 con 0x7f47d40c6310 2023-10-16T20:39:22.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d93a7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d4156f60 con 0x7f47d4151260 2023-10-16T20:39:22.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d93a7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d414b5f0 con 0x7f47d40c67a0 2023-10-16T20:39:22.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d40c6310 0x7f47d41558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d40c6310 0x7f47d41558c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56564/0 (socket says 172.21.15.88:56564) 2023-10-16T20:39:22.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d8ba6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47d4151260 0x7f47d414dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d3fff640 1 -- 172.21.15.88:0/1910090534 learned_addr learned my addr 172.21.15.88:0/1910090534 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47cb7fe640 1 --2- 172.21.15.88:0/1910090534 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47d40c67a0 0x7f47d40cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f3098ff9640 1 -- 172.21.15.88:0/2133868088 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c0be070 con 0x7f309c0c5930 2023-10-16T20:39:22.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f3098ff9640 1 -- 172.21.15.88:0/2133868088 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308c0a8910 con 0x7f309c0c5930 2023-10-16T20:39:22.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f3098ff9640 1 -- 172.21.15.88:0/2133868088 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c003a70 con 0x7f309c0c5930 2023-10-16T20:39:22.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f30a289e640 1 -- 172.21.15.88:0/2133868088 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309c14f230 con 0x7f309c0c5930 2023-10-16T20:39:22.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f30a289e640 1 -- 172.21.15.88:0/2133868088 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309c14f4b0 con 0x7f309c0c5930 2023-10-16T20:39:22.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d3fff640 1 -- 172.21.15.88:0/1910090534 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47d40c67a0 msgr2=0x7f47d40cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d3fff640 1 --2- 172.21.15.88:0/1910090534 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47d40c67a0 0x7f47d40cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d3fff640 1 -- 172.21.15.88:0/1910090534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47d4151260 msgr2=0x7f47d414dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d3fff640 1 --2- 172.21.15.88:0/1910090534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47d4151260 0x7f47d414dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d3fff640 1 -- 172.21.15.88:0/1910090534 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47cc009a70 con 0x7f47d40c6310 2023-10-16T20:39:22.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47cb7fe640 1 --2- 172.21.15.88:0/1910090534 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47d40c67a0 0x7f47d40cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:22.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f30a289e640 1 -- 172.21.15.88:0/2133868088 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309c14d810 con 0x7f309c0c5930 2023-10-16T20:39:22.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d3fff640 1 --2- 172.21.15.88:0/1910090534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d40c6310 0x7f47d41558c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f47c400c400 tx=0x7f47c400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d17fa640 1 -- 172.21.15.88:0/1910090534 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c4005cf0 con 0x7f47d40c6310 2023-10-16T20:39:22.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d17fa640 1 -- 172.21.15.88:0/1910090534 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c4005e50 con 0x7f47d40c6310 2023-10-16T20:39:22.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d17fa640 1 -- 172.21.15.88:0/1910090534 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c40072d0 con 0x7f47d40c6310 2023-10-16T20:39:22.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d93a7640 1 -- 172.21.15.88:0/1910090534 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d413f9b0 con 0x7f47d40c6310 2023-10-16T20:39:22.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.697+0000 7f47d93a7640 1 -- 172.21.15.88:0/1910090534 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d413ff70 con 0x7f47d40c6310 2023-10-16T20:39:22.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.701+0000 7f47d93a7640 1 -- 172.21.15.88:0/1910090534 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d414b7c0 con 0x7f47d40c6310 2023-10-16T20:39:22.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.701+0000 7f250cff9640 1 -- 172.21.15.88:0/2136331067 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f25000eed60 con 0x7f2510157b20 2023-10-16T20:39:22.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.701+0000 7f8ec77fe640 1 --2- 172.21.15.88:0/2187195956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec80ca3e0 0x7f8ec815ca50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0009d90 tx=0x7f8ec001c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6494bb75aeecffe1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.701+0000 7f3098ff9640 1 -- 172.21.15.88:0/2133868088 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f308c0b7070 con 0x7f309c0c5930 2023-10-16T20:39:22.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.701+0000 7f3098ff9640 1 --2- 172.21.15.88:0/2133868088 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f307c066520 0x7f307c0689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.701+0000 7f3098ff9640 1 -- 172.21.15.88:0/2133868088 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f308c121440 con 0x7f309c0c5930 2023-10-16T20:39:22.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.701+0000 7f309b7fe640 1 --2- 172.21.15.88:0/2133868088 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f307c066520 0x7f307c0689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.701+0000 7f47d17fa640 1 -- 172.21.15.88:0/1910090534 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f47c4007430 con 0x7f47d40c6310 2023-10-16T20:39:22.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.705+0000 7f309b7fe640 1 --2- 172.21.15.88:0/2133868088 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f307c066520 0x7f307c0689e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f309c0c4820 tx=0x7f3084006b90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.705+0000 7f47d17fa640 1 --2- 172.21.15.88:0/1910090534 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f47b4066570 0x7f47b4068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.705+0000 7f47d17fa640 1 -- 172.21.15.88:0/1910090534 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f47c4083e70 con 0x7f47d40c6310 2023-10-16T20:39:22.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.705+0000 7f47cb7fe640 1 --2- 172.21.15.88:0/1910090534 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f47b4066570 0x7f47b4068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.705+0000 7f8ec67fc640 1 -- 172.21.15.88:0/2187195956 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec00a59e0 con 0x7f8ec80ca3e0 2023-10-16T20:39:22.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.705+0000 7f8ec67fc640 1 -- 172.21.15.88:0/2187195956 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec00a5b40 con 0x7f8ec80ca3e0 2023-10-16T20:39:22.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.705+0000 7f8ec67fc640 1 -- 172.21.15.88:0/2187195956 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec00ad440 con 0x7f8ec80ca3e0 2023-10-16T20:39:22.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.705+0000 7f3098ff9640 1 -- 172.21.15.88:0/2133868088 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f308c0ee670 con 0x7f309c0c5930 2023-10-16T20:39:22.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.705+0000 7f47cb7fe640 1 --2- 172.21.15.88:0/1910090534 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f47b4066570 0x7f47b4068a30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f47bc007fd0 tx=0x7f47bc007bb0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.705+0000 7f8ec7fff640 1 -- 172.21.15.88:0/2187195956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec80ca3e0 msgr2=0x7f8ec815ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.705+0000 7f8ec7fff640 1 --2- 172.21.15.88:0/2187195956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec80ca3e0 0x7f8ec815ca50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0009d90 tx=0x7f8ec001c920 comp rx=0 tx=0).stop 2023-10-16T20:39:22.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.709+0000 7f8ec7fff640 1 -- 172.21.15.88:0/2187195956 shutdown_connections 2023-10-16T20:39:22.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.709+0000 7f8ec7fff640 1 --2- 172.21.15.88:0/2187195956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec80ca3e0 0x7f8ec815ca50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.709+0000 7f8ec7fff640 1 --2- 172.21.15.88:0/2187195956 >> v2:172.21.15.88:3300/0 conn(0x7f8ec80c9600 0x7f8ec80c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.709+0000 7f8ec7fff640 1 -- 172.21.15.88:0/2187195956 >> 172.21.15.88:0/2187195956 conn(0x7f8ec806c220 msgr2=0x7f8ec80715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:22.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.709+0000 7f8ec7fff640 1 -- 172.21.15.88:0/2187195956 shutdown_connections 2023-10-16T20:39:22.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.709+0000 7f47d17fa640 1 -- 172.21.15.88:0/1910090534 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f47c4050f30 con 0x7f47d40c6310 2023-10-16T20:39:22.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.709+0000 7f8ec7fff640 1 -- 172.21.15.88:0/2187195956 wait complete. 2023-10-16T20:39:22.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.713+0000 7f8ec7fff640 1 Processor -- start 2023-10-16T20:39:22.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.713+0000 7f8ec7fff640 1 -- start start 2023-10-16T20:39:22.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.717+0000 7f8ec7fff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec80c9600 0x7f8ec80cde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.717+0000 7f8ec7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ec80ca3e0 0x7f8ec815c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.717+0000 7f8ec7fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ec81536a0 0x7f8ec815a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.717+0000 7f8ec7fff640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec815bd70 con 0x7f8ec80ca3e0 2023-10-16T20:39:22.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.717+0000 7f8ec7fff640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec815b8e0 con 0x7f8ec81536a0 2023-10-16T20:39:22.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.717+0000 7f8ec7fff640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec80b1790 con 0x7f8ec80c9600 2023-10-16T20:39:22.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.717+0000 7f8ec6ffd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec80c9600 0x7f8ec80cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.717+0000 7f8ec6ffd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec80c9600 0x7f8ec80cde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:37748/0 (socket says 172.21.15.88:37748) 2023-10-16T20:39:22.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.717+0000 7f8ec6ffd640 1 -- 172.21.15.88:0/3228573700 learned_addr learned my addr 172.21.15.88:0/3228573700 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:22.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.717+0000 7f8ebffff640 1 --2- 172.21.15.88:0/3228573700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ec80ca3e0 0x7f8ec815c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.717+0000 7f8ebffff640 1 -- 172.21.15.88:0/3228573700 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec80c9600 msgr2=0x7f8ec80cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.717+0000 7f8ebffff640 1 --2- 172.21.15.88:0/3228573700 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec80c9600 0x7f8ec80cde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.717+0000 7f8ebffff640 1 -- 172.21.15.88:0/3228573700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ec81536a0 msgr2=0x7f8ec815a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:22.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.717+0000 7f8ebffff640 1 --2- 172.21.15.88:0/3228573700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ec81536a0 0x7f8ec815a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:22.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.717+0000 7f8ebffff640 1 -- 172.21.15.88:0/3228573700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec0009a70 con 0x7f8ec80ca3e0 2023-10-16T20:39:22.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.717+0000 7f8ebffff640 1 --2- 172.21.15.88:0/3228573700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ec80ca3e0 0x7f8ec815c6c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000d5f0 tx=0x7f8eb000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.721+0000 7f8ebf7fe640 1 -- 172.21.15.88:0/3228573700 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb00125f0 con 0x7f8ec80ca3e0 2023-10-16T20:39:22.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.721+0000 7f8ec7fff640 1 -- 172.21.15.88:0/3228573700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec80cccb0 con 0x7f8ec80ca3e0 2023-10-16T20:39:22.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.721+0000 7f8ec7fff640 1 -- 172.21.15.88:0/3228573700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec80cf960 con 0x7f8ec80ca3e0 2023-10-16T20:39:22.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.721+0000 7f8ebf7fe640 1 -- 172.21.15.88:0/3228573700 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb0012750 con 0x7f8ec80ca3e0 2023-10-16T20:39:22.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.721+0000 7f8ebf7fe640 1 -- 172.21.15.88:0/3228573700 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb00125f0 con 0x7f8ec80ca3e0 2023-10-16T20:39:22.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.721+0000 7f8ec7fff640 1 -- 172.21.15.88:0/3228573700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec804e9c0 con 0x7f8ec80ca3e0 2023-10-16T20:39:22.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.729+0000 7f8ebf7fe640 1 -- 172.21.15.88:0/3228573700 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8eb00128c0 con 0x7f8ec80ca3e0 2023-10-16T20:39:22.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.729+0000 7f8ebf7fe640 1 --2- 172.21.15.88:0/3228573700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ea8066530 0x7f8ea80689f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:22.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.729+0000 7f8ebf7fe640 1 -- 172.21.15.88:0/3228573700 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8eb0084d30 con 0x7f8ec80ca3e0 2023-10-16T20:39:22.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.729+0000 7f8ec6ffd640 1 --2- 172.21.15.88:0/3228573700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ea8066530 0x7f8ea80689f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:22.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.733+0000 7f8ec6ffd640 1 --2- 172.21.15.88:0/3228573700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ea8066530 0x7f8ea80689f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8000f70 tx=0x7f8eb8007eb0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:22.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:22.737+0000 7f8ebf7fe640 1 -- 172.21.15.88:0/3228573700 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8eb0051e60 con 0x7f8ec80ca3e0 2023-10-16T20:39:23.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.009+0000 7f0bb650f640 1 -- 172.21.15.88:0/770091794 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0bb01572a0 con 0x7f0bb0159490 2023-10-16T20:39:23.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.009+0000 7f0bacff9640 1 -- 172.21.15.88:0/770091794 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0bb01572a0 con 0x7f0bb0159490 2023-10-16T20:39:23.023 INFO:teuthology.orchestra.run.smithi088.stdout:64424509475 2023-10-16T20:39:23.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.013+0000 7f0bb650f640 1 -- 172.21.15.88:0/770091794 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0b9c0665a0 msgr2=0x7f0b9c068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:23.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.013+0000 7f0bb650f640 1 --2- 172.21.15.88:0/770091794 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0b9c0665a0 0x7f0b9c068a60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0ba80b1000 tx=0x7f0ba8004ac0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.013+0000 7f0bb650f640 1 -- 172.21.15.88:0/770091794 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0bb0159490 msgr2=0x7f0bb01589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:23.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.013+0000 7f0bb650f640 1 --2- 172.21.15.88:0/770091794 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0bb0159490 0x7f0bb01589b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0ba400c3c0 tx=0x7f0ba400c880 comp rx=0 tx=0).stop 2023-10-16T20:39:23.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.013+0000 7f0bb650f640 1 -- 172.21.15.88:0/770091794 shutdown_connections 2023-10-16T20:39:23.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.013+0000 7f0bb650f640 1 --2- 172.21.15.88:0/770091794 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0b9c0665a0 0x7f0b9c068a60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.013+0000 7f0bb650f640 1 --2- 172.21.15.88:0/770091794 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0bb0159490 0x7f0bb01589b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.013+0000 7f0bb650f640 1 --2- 172.21.15.88:0/770091794 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0bb0153780 0x7f0bb0158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.013+0000 7f0bb650f640 1 --2- 172.21.15.88:0/770091794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb00c5930 0x7f0bb0151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.013+0000 7f0bb650f640 1 -- 172.21.15.88:0/770091794 >> 172.21.15.88:0/770091794 conn(0x7f0bb006bcb0 msgr2=0x7f0bb00c9900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:23.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.013+0000 7f0bb650f640 1 -- 172.21.15.88:0/770091794 shutdown_connections 2023-10-16T20:39:23.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.013+0000 7f0bb650f640 1 -- 172.21.15.88:0/770091794 wait complete. 2023-10-16T20:39:23.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.017+0000 7fe8f50dc640 1 -- 172.21.15.88:0/476655804 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe8d0005510 con 0x7fe8e800e400 2023-10-16T20:39:23.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.021+0000 7f01d759e640 1 -- 172.21.15.88:0/3427918045 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f01bc005510 con 0x7f01d80c2050 2023-10-16T20:39:23.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.021+0000 7fe8ecff9640 1 -- 172.21.15.88:0/476655804 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe8e0054d30 con 0x7fe8e800e400 2023-10-16T20:39:23.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.021+0000 7f01c6ffd640 1 -- 172.21.15.88:0/3427918045 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f01cc014090 con 0x7f01d80c2050 2023-10-16T20:39:23.029 INFO:teuthology.orchestra.run.smithi088.stdout:55834574882 2023-10-16T20:39:23.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.021+0000 7f2ca359e640 1 -- 172.21.15.88:0/3337933951 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2c9c0d3390 con 0x7f2c9c0e35b0 2023-10-16T20:39:23.030 INFO:teuthology.orchestra.run.smithi088.stdout:64424509473 2023-10-16T20:39:23.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7fe8f50dc640 1 -- 172.21.15.88:0/476655804 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe8dc0664a0 msgr2=0x7fe8dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:23.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7f2c92ffd640 1 -- 172.21.15.88:0/3337933951 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2ca4102090 con 0x7f2c9c0e35b0 2023-10-16T20:39:23.032 INFO:teuthology.orchestra.run.smithi088.stdout:64424509473 2023-10-16T20:39:23.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7fe8f50dc640 1 --2- 172.21.15.88:0/476655804 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe8dc0664a0 0x7fe8dc068960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4009a30 tx=0x7fe8e4009240 comp rx=0 tx=0).stop 2023-10-16T20:39:23.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7fe8f50dc640 1 -- 172.21.15.88:0/476655804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8e800e400 msgr2=0x7fe8e804a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:23.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7fe8f50dc640 1 --2- 172.21.15.88:0/476655804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8e800e400 0x7fe8e804a340 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe8e000d620 tx=0x7fe8e000dae0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7fe8f50dc640 1 -- 172.21.15.88:0/476655804 shutdown_connections 2023-10-16T20:39:23.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7fe8f50dc640 1 --2- 172.21.15.88:0/476655804 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe8dc0664a0 0x7fe8dc068960 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4009a30 tx=0x7fe8e4009240 comp rx=0 tx=0).stop 2023-10-16T20:39:23.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7fe8f50dc640 1 --2- 172.21.15.88:0/476655804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8e80457c0 0x7fe8e80d5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7fe8f50dc640 1 --2- 172.21.15.88:0/476655804 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8e800e400 0x7fe8e804a340 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7fe8f50dc640 1 --2- 172.21.15.88:0/476655804 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8e8005ff0 0x7fe8e8048900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7fe8f50dc640 1 -- 172.21.15.88:0/476655804 >> 172.21.15.88:0/476655804 conn(0x7fe8e80502e0 msgr2=0x7fe8e80ce480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:23.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7f25169d0640 1 -- 172.21.15.88:0/2136331067 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f24f40057e0 con 0x7f2510157b20 2023-10-16T20:39:23.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7f250cff9640 1 -- 172.21.15.88:0/2136331067 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f25000b5070 con 0x7f2510157b20 2023-10-16T20:39:23.035 INFO:teuthology.orchestra.run.smithi088.stdout:64424509473 2023-10-16T20:39:23.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7fe8f50dc640 1 -- 172.21.15.88:0/476655804 shutdown_connections 2023-10-16T20:39:23.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7f01d759e640 1 -- 172.21.15.88:0/3427918045 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f01b8066500 msgr2=0x7f01b80689c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:23.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7f01d759e640 1 --2- 172.21.15.88:0/3427918045 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f01b8066500 0x7f01b80689c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f01c80b2000 tx=0x7f01c8009040 comp rx=0 tx=0).stop 2023-10-16T20:39:23.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7f01d759e640 1 -- 172.21.15.88:0/3427918045 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01d80c2050 msgr2=0x7f01d8148ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:23.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7f01d759e640 1 --2- 172.21.15.88:0/3427918045 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01d80c2050 0x7f01d8148ed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00b4a0 tx=0x7f01cc00b960 comp rx=0 tx=0).stop 2023-10-16T20:39:23.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7f01d759e640 1 -- 172.21.15.88:0/3427918045 shutdown_connections 2023-10-16T20:39:23.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7f01d759e640 1 --2- 172.21.15.88:0/3427918045 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f01b8066500 0x7f01b80689c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7f01d759e640 1 --2- 172.21.15.88:0/3427918045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f01d8155c80 0x7f01d81459d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7f01d759e640 1 --2- 172.21.15.88:0/3427918045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01d80c7570 0x7f01d8145490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7f01d759e640 1 --2- 172.21.15.88:0/3427918045 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f01d80c2050 0x7f01d8148ed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7f01d759e640 1 -- 172.21.15.88:0/3427918045 >> 172.21.15.88:0/3427918045 conn(0x7f01d806d1e0 msgr2=0x7f01d814cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:23.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.025+0000 7fe8f50dc640 1 -- 172.21.15.88:0/476655804 wait complete. 2023-10-16T20:39:23.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f01d759e640 1 -- 172.21.15.88:0/3427918045 shutdown_connections 2023-10-16T20:39:23.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f01d759e640 1 -- 172.21.15.88:0/3427918045 wait complete. 2023-10-16T20:39:23.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f2ca359e640 1 -- 172.21.15.88:0/3337933951 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2c840664a0 msgr2=0x7f2c84068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:23.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f2ca359e640 1 --2- 172.21.15.88:0/3337933951 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2c840664a0 0x7f2c84068960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2c94007c90 tx=0x7f2c94009040 comp rx=0 tx=0).stop 2023-10-16T20:39:23.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f2ca359e640 1 -- 172.21.15.88:0/3337933951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c9c0e35b0 msgr2=0x7f2c9c0d9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:23.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f2ca359e640 1 --2- 172.21.15.88:0/3337933951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c9c0e35b0 0x7f2c9c0d9720 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4067e50 tx=0x7f2ca404eb70 comp rx=0 tx=0).stop 2023-10-16T20:39:23.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f2ca359e640 1 -- 172.21.15.88:0/3337933951 shutdown_connections 2023-10-16T20:39:23.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f2ca359e640 1 --2- 172.21.15.88:0/3337933951 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2c840664a0 0x7f2c84068960 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f2ca359e640 1 --2- 172.21.15.88:0/3337933951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c9c0e35b0 0x7f2c9c0d9720 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f2ca359e640 1 --2- 172.21.15.88:0/3337933951 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c9c0dd300 0x7f2c9c0ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f2ca359e640 1 --2- 172.21.15.88:0/3337933951 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c9c04c580 0x7f2c9c0db9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f2ca359e640 1 -- 172.21.15.88:0/3337933951 >> 172.21.15.88:0/3337933951 conn(0x7f2c9c0501d0 msgr2=0x7f2c9c005e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:23.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f2ca359e640 1 -- 172.21.15.88:0/3337933951 shutdown_connections 2023-10-16T20:39:23.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f2ca359e640 1 -- 172.21.15.88:0/3337933951 wait complete. 2023-10-16T20:39:23.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f25169d0640 1 -- 172.21.15.88:0/2136331067 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f24f00664a0 msgr2=0x7f24f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:23.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f25169d0640 1 --2- 172.21.15.88:0/2136331067 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f24f00664a0 0x7f24f0068960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f25080012f0 tx=0x7f2508001770 comp rx=0 tx=0).stop 2023-10-16T20:39:23.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f25169d0640 1 -- 172.21.15.88:0/2136331067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2510157b20 msgr2=0x7f2510159cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:23.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f25169d0640 1 --2- 172.21.15.88:0/2136331067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2510157b20 0x7f2510159cf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f250001c8c0 tx=0x7f250001ba30 comp rx=0 tx=0).stop 2023-10-16T20:39:23.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f25169d0640 1 -- 172.21.15.88:0/2136331067 shutdown_connections 2023-10-16T20:39:23.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f25169d0640 1 --2- 172.21.15.88:0/2136331067 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f24f00664a0 0x7f24f0068960 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f25169d0640 1 --2- 172.21.15.88:0/2136331067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2510157f00 0x7f25101f6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f25169d0640 1 --2- 172.21.15.88:0/2136331067 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2510157b20 0x7f2510159cf0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f25169d0640 1 --2- 172.21.15.88:0/2136331067 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f251006ae00 0x7f25101597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f25169d0640 1 -- 172.21.15.88:0/2136331067 >> 172.21.15.88:0/2136331067 conn(0x7f25100ca9f0 msgr2=0x7f251015c170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:23.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f25169d0640 1 -- 172.21.15.88:0/2136331067 shutdown_connections 2023-10-16T20:39:23.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.029+0000 7f25169d0640 1 -- 172.21.15.88:0/2136331067 wait complete. 2023-10-16T20:39:23.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.033+0000 7f30a289e640 1 -- 172.21.15.88:0/2133868088 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f309c14d810 con 0x7f309c0c5930 2023-10-16T20:39:23.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.037+0000 7f3098ff9640 1 -- 172.21.15.88:0/2133868088 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f308c0f1c90 con 0x7f309c0c5930 2023-10-16T20:39:23.046 INFO:teuthology.orchestra.run.smithi088.stdout:64424509473 2023-10-16T20:39:23.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.037+0000 7f47d93a7640 1 -- 172.21.15.88:0/1910090534 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f47d404e9c0 con 0x7f47d40c6310 2023-10-16T20:39:23.048 INFO:teuthology.orchestra.run.smithi088.stdout:64424509473 2023-10-16T20:39:23.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.037+0000 7f47d17fa640 1 -- 172.21.15.88:0/1910090534 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f47c4021090 con 0x7f47d40c6310 2023-10-16T20:39:23.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f30a289e640 1 -- 172.21.15.88:0/2133868088 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f307c066520 msgr2=0x7f307c0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:23.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f30a289e640 1 --2- 172.21.15.88:0/2133868088 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f307c066520 0x7f307c0689e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f309c0c4820 tx=0x7f3084006b90 comp rx=0 tx=0).stop 2023-10-16T20:39:23.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f30a289e640 1 -- 172.21.15.88:0/2133868088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f309c0c5930 msgr2=0x7f309c15d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:23.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f30a289e640 1 --2- 172.21.15.88:0/2133868088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f309c0c5930 0x7f309c15d0e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f308c01c8c0 tx=0x7f308c01ba30 comp rx=0 tx=0).stop 2023-10-16T20:39:23.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f30a289e640 1 -- 172.21.15.88:0/2133868088 shutdown_connections 2023-10-16T20:39:23.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f30a289e640 1 --2- 172.21.15.88:0/2133868088 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f307c066520 0x7f307c0689e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f30a289e640 1 --2- 172.21.15.88:0/2133868088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f309c159490 0x7f309c158570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f30a289e640 1 --2- 172.21.15.88:0/2133868088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f309c153780 0x7f309c147950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f30a289e640 1 --2- 172.21.15.88:0/2133868088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f309c0c5930 0x7f309c15d0e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f30a289e640 1 -- 172.21.15.88:0/2133868088 >> 172.21.15.88:0/2133868088 conn(0x7f309c06bcb0 msgr2=0x7f309c0caf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:23.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f47d93a7640 1 -- 172.21.15.88:0/1910090534 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f47b4066570 msgr2=0x7f47b4068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:23.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f30a289e640 1 -- 172.21.15.88:0/2133868088 shutdown_connections 2023-10-16T20:39:23.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f47d93a7640 1 --2- 172.21.15.88:0/1910090534 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f47b4066570 0x7f47b4068a30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f47bc007fd0 tx=0x7f47bc007bb0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f47d93a7640 1 -- 172.21.15.88:0/1910090534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d40c6310 msgr2=0x7f47d41558c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:23.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f47d93a7640 1 --2- 172.21.15.88:0/1910090534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d40c6310 0x7f47d41558c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f47c400c400 tx=0x7f47c400c8c0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f30a289e640 1 -- 172.21.15.88:0/2133868088 wait complete. 2023-10-16T20:39:23.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f47d93a7640 1 -- 172.21.15.88:0/1910090534 shutdown_connections 2023-10-16T20:39:23.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f47d93a7640 1 --2- 172.21.15.88:0/1910090534 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f47b4066570 0x7f47b4068a30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f47d93a7640 1 --2- 172.21.15.88:0/1910090534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f47d4151260 0x7f47d414dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.041+0000 7f47d93a7640 1 --2- 172.21.15.88:0/1910090534 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f47d40c67a0 0x7f47d40cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.045+0000 7f47d93a7640 1 --2- 172.21.15.88:0/1910090534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d40c6310 0x7f47d41558c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.045+0000 7f47d93a7640 1 -- 172.21.15.88:0/1910090534 >> 172.21.15.88:0/1910090534 conn(0x7f47d4065a80 msgr2=0x7f47d414a850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:23.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.045+0000 7f47d93a7640 1 -- 172.21.15.88:0/1910090534 shutdown_connections 2023-10-16T20:39:23.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.045+0000 7f47d93a7640 1 -- 172.21.15.88:0/1910090534 wait complete. 2023-10-16T20:39:23.059 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509476 got 64424509475 for osd.5 2023-10-16T20:39:23.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.061+0000 7f8ec7fff640 1 -- 172.21.15.88:0/3228573700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f8ec815ba20 con 0x7f8ec80ca3e0 2023-10-16T20:39:23.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.061+0000 7f8ebf7fe640 1 -- 172.21.15.88:0/3228573700 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8eb0055480 con 0x7f8ec80ca3e0 2023-10-16T20:39:23.070 INFO:teuthology.orchestra.run.smithi088.stdout:64424509473 2023-10-16T20:39:23.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.069+0000 7f8ec7fff640 1 -- 172.21.15.88:0/3228573700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ea8066530 msgr2=0x7f8ea80689f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:23.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.069+0000 7f8ec7fff640 1 --2- 172.21.15.88:0/3228573700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ea8066530 0x7f8ea80689f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8000f70 tx=0x7f8eb8007eb0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.069+0000 7f8ec7fff640 1 -- 172.21.15.88:0/3228573700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ec80ca3e0 msgr2=0x7f8ec815c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:23.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.069+0000 7f8ec7fff640 1 --2- 172.21.15.88:0/3228573700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ec80ca3e0 0x7f8ec815c6c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000d5f0 tx=0x7f8eb000dab0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.069+0000 7f8ec7fff640 1 -- 172.21.15.88:0/3228573700 shutdown_connections 2023-10-16T20:39:23.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.069+0000 7f8ec7fff640 1 --2- 172.21.15.88:0/3228573700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ea8066530 0x7f8ea80689f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.069+0000 7f8ec7fff640 1 --2- 172.21.15.88:0/3228573700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ec81536a0 0x7f8ec815a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.069+0000 7f8ec7fff640 1 --2- 172.21.15.88:0/3228573700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ec80ca3e0 0x7f8ec815c6c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.069+0000 7f8ec7fff640 1 --2- 172.21.15.88:0/3228573700 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec80c9600 0x7f8ec80cde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:23.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.069+0000 7f8ec7fff640 1 -- 172.21.15.88:0/3228573700 >> 172.21.15.88:0/3228573700 conn(0x7f8ec806c220 msgr2=0x7f8ec81497b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:23.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.069+0000 7f8ec7fff640 1 -- 172.21.15.88:0/3228573700 shutdown_connections 2023-10-16T20:39:23.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:23.069+0000 7f8ec7fff640 1 -- 172.21.15.88:0/3228573700 wait complete. 2023-10-16T20:39:23.076 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509474 got 64424509473 for osd.2 2023-10-16T20:39:23.079 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509474 got 64424509473 for osd.3 2023-10-16T20:39:23.080 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574883 got 55834574882 for osd.0 2023-10-16T20:39:23.083 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509474 got 64424509473 for osd.1 2023-10-16T20:39:23.086 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509474 got 64424509473 for osd.4 2023-10-16T20:39:23.087 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509474 got 64424509473 for osd.7 2023-10-16T20:39:23.097 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509474 got 64424509473 for osd.6 2023-10-16T20:39:23.969 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:39:23.973 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:39:24.059 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-16T20:39:24.077 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-16T20:39:24.081 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-16T20:39:24.081 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-16T20:39:24.085 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-16T20:39:24.087 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-16T20:39:24.088 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-16T20:39:24.098 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-16T20:39:24.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.265+0000 7f8c7e8c9640 1 Processor -- start 2023-10-16T20:39:24.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.265+0000 7f8c7e8c9640 1 -- start start 2023-10-16T20:39:24.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.265+0000 7f8c7e8c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8c7814f540 0x7f8c780c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.265+0000 7f8c7e8c9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8c7814f920 0x7f8c781593d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.269+0000 7f8c7e8c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c780c1e20 con 0x7f8c7806a560 2023-10-16T20:39:24.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.269+0000 7f8c7e8c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c781553d0 con 0x7f8c7814f540 2023-10-16T20:39:24.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.269+0000 7f8c7e8c9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c78156190 con 0x7f8c7814f920 2023-10-16T20:39:24.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.269+0000 7f8c77fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8c7814f540 0x7f8c780c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.269+0000 7f8c77fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8c7814f540 0x7f8c780c5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56574/0 (socket says 172.21.15.88:56574) 2023-10-16T20:39:24.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.269+0000 7f8c77fff640 1 -- 172.21.15.88:0/3391808468 learned_addr learned my addr 172.21.15.88:0/3391808468 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:24.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.269+0000 7f8c77fff640 1 -- 172.21.15.88:0/3391808468 >> v2:172.21.15.191:3300/0 conn(0x7f8c7814f920 msgr2=0x7f8c781593d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T20:39:24.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.269+0000 7f8c77fff640 1 --2- 172.21.15.88:0/3391808468 >> v2:172.21.15.191:3300/0 conn(0x7f8c7814f920 0x7f8c781593d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.269+0000 7f8c77fff640 1 -- 172.21.15.88:0/3391808468 >> v1:172.21.15.88:6789/0 conn(0x7f8c7806a560 legacy=0x7f8c780c22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:24.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.269+0000 7f8c77fff640 1 -- 172.21.15.88:0/3391808468 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c780c6470 con 0x7f8c7814f540 2023-10-16T20:39:24.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.273+0000 7f8c77fff640 1 --2- 172.21.15.88:0/3391808468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c7814f540 0x7f8c780c5f30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8c780c5c50 tx=0x7f8c7001f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8896a3c29187578 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.277+0000 7f8c76ffd640 1 -- 172.21.15.88:0/3391808468 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c700af400 con 0x7f8c7814f540 2023-10-16T20:39:24.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.277+0000 7f8c76ffd640 1 -- 172.21.15.88:0/3391808468 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c700ab050 con 0x7f8c7814f540 2023-10-16T20:39:24.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.277+0000 7f8c76ffd640 1 -- 172.21.15.88:0/3391808468 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c700af780 con 0x7f8c7814f540 2023-10-16T20:39:24.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.277+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/3391808468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c7814f540 msgr2=0x7f8c780c5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.277+0000 7f8c7e8c9640 1 --2- 172.21.15.88:0/3391808468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c7814f540 0x7f8c780c5f30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8c780c5c50 tx=0x7f8c7001f1a0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.277+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/3391808468 shutdown_connections 2023-10-16T20:39:24.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.277+0000 7f8c7e8c9640 1 --2- 172.21.15.88:0/3391808468 >> v2:172.21.15.191:3300/0 conn(0x7f8c7814f920 0x7f8c781593d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.277+0000 7f8c7e8c9640 1 --2- 172.21.15.88:0/3391808468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c7814f540 0x7f8c780c5f30 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.277+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/3391808468 >> 172.21.15.88:0/3391808468 conn(0x7f8c780cac60 msgr2=0x7f8c780cb070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:24.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.277+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/3391808468 shutdown_connections 2023-10-16T20:39:24.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.277+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/3391808468 wait complete. 2023-10-16T20:39:24.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.281+0000 7f8c7e8c9640 1 Processor -- start 2023-10-16T20:39:24.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.281+0000 7f8c7e8c9640 1 -- start start 2023-10-16T20:39:24.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.281+0000 7f8c7e8c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c7806a560 0x7f8c7814d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c7e8c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c7814f920 0x7f8c781edc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c7e8c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c78071e80 0x7f8c78144090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c77fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c7806a560 0x7f8c7814d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c77fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c7806a560 0x7f8c7814d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:40288/0 (socket says 172.21.15.88:40288) 2023-10-16T20:39:24.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c7ce66640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c78071e80 0x7f8c78144090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c777fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c7814f920 0x7f8c781edc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c7ce66640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c78071e80 0x7f8c78144090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:37764/0 (socket says 172.21.15.88:37764) 2023-10-16T20:39:24.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c777fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c7814f920 0x7f8c781edc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56586/0 (socket says 172.21.15.88:56586) 2023-10-16T20:39:24.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c77fff640 1 -- 172.21.15.88:0/4241974937 learned_addr learned my addr 172.21.15.88:0/4241974937 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:24.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/4241974937 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c781510a0 con 0x7f8c7814f920 2023-10-16T20:39:24.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/4241974937 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c78152200 con 0x7f8c7806a560 2023-10-16T20:39:24.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/4241974937 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c78152c70 con 0x7f8c78071e80 2023-10-16T20:39:24.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c7ce66640 1 -- 172.21.15.88:0/4241974937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c7806a560 msgr2=0x7f8c7814d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c7ce66640 1 --2- 172.21.15.88:0/4241974937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c7806a560 0x7f8c7814d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c7ce66640 1 -- 172.21.15.88:0/4241974937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c7814f920 msgr2=0x7f8c781edc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c7ce66640 1 --2- 172.21.15.88:0/4241974937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c7814f920 0x7f8c781edc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c7ce66640 1 -- 172.21.15.88:0/4241974937 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c70009b30 con 0x7f8c78071e80 2023-10-16T20:39:24.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.285+0000 7f8c7ce66640 1 --2- 172.21.15.88:0/4241974937 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c78071e80 0x7f8c78144090 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00c410 tx=0x7f8c6c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.289+0000 7f8c74ff9640 1 -- 172.21.15.88:0/4241974937 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c003600 con 0x7f8c78071e80 2023-10-16T20:39:24.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.289+0000 7f8c74ff9640 1 -- 172.21.15.88:0/4241974937 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6c003760 con 0x7f8c78071e80 2023-10-16T20:39:24.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.289+0000 7f8c74ff9640 1 -- 172.21.15.88:0/4241974937 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c003b50 con 0x7f8c78071e80 2023-10-16T20:39:24.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.289+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/4241974937 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7806fe00 con 0x7f8c78071e80 2023-10-16T20:39:24.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.289+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/4241974937 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c780703c0 con 0x7f8c78071e80 2023-10-16T20:39:24.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.293+0000 7f8c74ff9640 1 -- 172.21.15.88:0/4241974937 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8c6c003cb0 con 0x7f8c78071e80 2023-10-16T20:39:24.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.293+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/4241974937 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7804e9c0 con 0x7f8c78071e80 2023-10-16T20:39:24.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.293+0000 7f8c74ff9640 1 --2- 172.21.15.88:0/4241974937 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8c640664f0 0x7f8c640689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.297+0000 7f8c74ff9640 1 -- 172.21.15.88:0/4241974937 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8c6c083400 con 0x7f8c78071e80 2023-10-16T20:39:24.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.297+0000 7f8c77fff640 1 --2- 172.21.15.88:0/4241974937 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8c640664f0 0x7f8c640689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.297+0000 7f8c77fff640 1 --2- 172.21.15.88:0/4241974937 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8c640664f0 0x7f8c640689b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8c70000f90 tx=0x7f8c700aa000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.301+0000 7f8c74ff9640 1 -- 172.21.15.88:0/4241974937 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8c6c0504d0 con 0x7f8c78071e80 2023-10-16T20:39:24.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.341+0000 7fd95d8c9640 1 Processor -- start 2023-10-16T20:39:24.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.341+0000 7fd95d8c9640 1 -- start start 2023-10-16T20:39:24.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.341+0000 7fd95d8c9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd9580c9eb0 0x7fd9580ca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.341+0000 7fd95d8c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd95815d260 0x7fd958163de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.341+0000 7fd95d8c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd958153b70 con 0x7fd9580cb5e0 2023-10-16T20:39:24.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.341+0000 7fd95d8c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9580c5630 con 0x7fd95815d260 2023-10-16T20:39:24.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.341+0000 7fd95d8c9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9580ca7d0 con 0x7fd9580c9eb0 2023-10-16T20:39:24.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.341+0000 7fd9567fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd9580cb5e0 0x7fd95815cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:37004/0 (socket says 172.21.15.88:37004) 2023-10-16T20:39:24.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.341+0000 7fd9567fc640 1 -- 172.21.15.88:0/2868492765 learned_addr learned my addr 172.21.15.88:0/2868492765 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:24.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.341+0000 7fd9577fe640 1 --2- 172.21.15.88:0/2868492765 >> v2:172.21.15.88:3300/0 conn(0x7fd95815d260 0x7fd958163de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.341+0000 7fd956ffd640 1 --2- 172.21.15.88:0/2868492765 >> v2:172.21.15.191:3300/0 conn(0x7fd9580c9eb0 0x7fd9580ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.345+0000 7fd955ffb640 1 -- 172.21.15.88:0/2868492765 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095028681 0 0) 0x7fd958153b70 con 0x7fd9580cb5e0 2023-10-16T20:39:24.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.345+0000 7fd9577fe640 1 -- 172.21.15.88:0/2868492765 >> v2:172.21.15.191:3300/0 conn(0x7fd9580c9eb0 msgr2=0x7fd9580ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:24.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.345+0000 7fd9577fe640 1 --2- 172.21.15.88:0/2868492765 >> v2:172.21.15.191:3300/0 conn(0x7fd9580c9eb0 0x7fd9580ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.345+0000 7fd9577fe640 1 -- 172.21.15.88:0/2868492765 >> v1:172.21.15.88:6789/0 conn(0x7fd9580cb5e0 legacy=0x7fd95815cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.345+0000 7f4500fd0640 1 Processor -- start 2023-10-16T20:39:24.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.345+0000 7fd9577fe640 1 -- 172.21.15.88:0/2868492765 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95815a180 con 0x7fd95815d260 2023-10-16T20:39:24.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.345+0000 7f4500fd0640 1 -- start start 2023-10-16T20:39:24.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.345+0000 7f4500fd0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44fc0cb5c0 0x7f44fc15a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.345+0000 7f4500fd0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f44fc0c66e0 0x7f44fc150150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.349+0000 7f4500fd0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44fc070f40 con 0x7f44fc150690 2023-10-16T20:39:24.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.349+0000 7f4500fd0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44fc15a790 con 0x7f44fc0c66e0 2023-10-16T20:39:24.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.349+0000 7f4500fd0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc15cbb0 con 0x7f44fc0cb5c0 2023-10-16T20:39:24.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.349+0000 7f44f1d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f44fc0c66e0 0x7f44fc150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.349+0000 7f44f1d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f44fc0c66e0 0x7f44fc150150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56608/0 (socket says 172.21.15.88:56608) 2023-10-16T20:39:24.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.349+0000 7f44fad9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f44fc150690 0x7f44fc071420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:37012/0 (socket says 172.21.15.88:37012) 2023-10-16T20:39:24.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.349+0000 7f44fad9d640 1 -- 172.21.15.88:0/3969819104 learned_addr learned my addr 172.21.15.88:0/3969819104 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:24.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.349+0000 7f44fa59c640 1 --2- 172.21.15.88:0/3969819104 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44fc0cb5c0 0x7f44fc15a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.349+0000 7f44f9d9b640 1 -- 172.21.15.88:0/3969819104 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248510869 0 0) 0x7f44fc070f40 con 0x7f44fc150690 2023-10-16T20:39:24.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.349+0000 7fd9577fe640 1 --2- 172.21.15.88:0/2868492765 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd95815d260 0x7fd958163de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd95806c9f0 tx=0x7fd95001c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68a007aba230ada3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.353+0000 7fd955ffb640 1 -- 172.21.15.88:0/2868492765 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9500a59a0 con 0x7fd95815d260 2023-10-16T20:39:24.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.353+0000 7fd955ffb640 1 -- 172.21.15.88:0/2868492765 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9500a5b00 con 0x7fd95815d260 2023-10-16T20:39:24.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.353+0000 7fd955ffb640 1 -- 172.21.15.88:0/2868492765 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9500a59a0 con 0x7fd95815d260 2023-10-16T20:39:24.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.349+0000 7f44f1d9b640 1 -- 172.21.15.88:0/3969819104 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44fc0cb5c0 msgr2=0x7f44fc15a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:24.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.349+0000 7f44f1d9b640 1 --2- 172.21.15.88:0/3969819104 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44fc0cb5c0 0x7f44fc15a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.349+0000 7f44f1d9b640 1 -- 172.21.15.88:0/3969819104 >> v1:172.21.15.88:6789/0 conn(0x7f44fc150690 legacy=0x7f44fc071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.349+0000 7f44f1d9b640 1 -- 172.21.15.88:0/3969819104 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc06a2d0 con 0x7f44fc0c66e0 2023-10-16T20:39:24.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.353+0000 7fd95d8c9640 1 -- 172.21.15.88:0/2868492765 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd95815d260 msgr2=0x7fd958163de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.353+0000 7fd95d8c9640 1 --2- 172.21.15.88:0/2868492765 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd95815d260 0x7fd958163de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd95806c9f0 tx=0x7fd95001c900 comp rx=0 tx=0).stop 2023-10-16T20:39:24.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f44f1d9b640 1 --2- 172.21.15.88:0/3969819104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44fc0c66e0 0x7f44fc150150 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f44e40097c0 tx=0x7f44e401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ceb0de3240fdac4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f44f9d9b640 1 -- 172.21.15.88:0/3969819104 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e40ab600 con 0x7f44fc0c66e0 2023-10-16T20:39:24.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.353+0000 7f1c71a3a640 1 Processor -- start 2023-10-16T20:39:24.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.353+0000 7f1c71a3a640 1 -- start start 2023-10-16T20:39:24.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f44f9d9b640 1 -- 172.21.15.88:0/3969819104 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e40a7050 con 0x7f44fc0c66e0 2023-10-16T20:39:24.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.353+0000 7fd95d8c9640 1 -- 172.21.15.88:0/2868492765 shutdown_connections 2023-10-16T20:39:24.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.353+0000 7fd95d8c9640 1 --2- 172.21.15.88:0/2868492765 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd95815d260 0x7fd958163de0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.353+0000 7fd95d8c9640 1 --2- 172.21.15.88:0/2868492765 >> v2:172.21.15.191:3300/0 conn(0x7fd9580c9eb0 0x7fd9580ca290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.353+0000 7fd95d8c9640 1 -- 172.21.15.88:0/2868492765 >> 172.21.15.88:0/2868492765 conn(0x7fd95806bc00 msgr2=0x7fd95806bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:24.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f1c71a3a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1c6c157ba0 0x7f1c6c161780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f1c71a3a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1c6c0c5940 0x7f1c6c15ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f1c71a3a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6c157f80 con 0x7f1c6c0c6eb0 2023-10-16T20:39:24.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7fd95d8c9640 1 -- 172.21.15.88:0/2868492765 shutdown_connections 2023-10-16T20:39:24.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f1c71a3a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c6c06ae20 con 0x7f1c6c157ba0 2023-10-16T20:39:24.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f4500fd0640 1 -- 172.21.15.88:0/3969819104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44fc0c66e0 msgr2=0x7f44fc150150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f44f9d9b640 1 -- 172.21.15.88:0/3969819104 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e40ab980 con 0x7f44fc0c66e0 2023-10-16T20:39:24.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f4500fd0640 1 --2- 172.21.15.88:0/3969819104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44fc0c66e0 0x7f44fc150150 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f44e40097c0 tx=0x7f44e401c410 comp rx=0 tx=0).stop 2023-10-16T20:39:24.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f9e7c46b640 1 Processor -- start 2023-10-16T20:39:24.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f1c71a3a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c6c161cc0 con 0x7f1c6c0c5940 2023-10-16T20:39:24.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7fd95d8c9640 1 -- 172.21.15.88:0/2868492765 wait complete. 2023-10-16T20:39:24.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7fd95d8c9640 1 Processor -- start 2023-10-16T20:39:24.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7fd95d8c9640 1 -- start start 2023-10-16T20:39:24.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7fd95d8c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9580c9eb0 0x7fd9581ff8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7fd95d8c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9580cb5e0 0x7fd9581fe750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7fd95d8c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd95815d260 0x7fd9581fec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7fd95d8c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd958158980 con 0x7fd9580c9eb0 2023-10-16T20:39:24.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7fd95d8c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9580c78c0 con 0x7fd9580cb5e0 2023-10-16T20:39:24.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7fd95d8c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9581583c0 con 0x7fd95815d260 2023-10-16T20:39:24.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7fd956ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9580c9eb0 0x7fd9581ff8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7fd956ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9580c9eb0 0x7fd9581ff8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56638/0 (socket says 172.21.15.88:56638) 2023-10-16T20:39:24.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7fd956ffd640 1 -- 172.21.15.88:0/4196904983 learned_addr learned my addr 172.21.15.88:0/4196904983 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:24.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f4500fd0640 1 -- 172.21.15.88:0/3969819104 shutdown_connections 2023-10-16T20:39:24.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f4500fd0640 1 --2- 172.21.15.88:0/3969819104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44fc0c66e0 0x7f44fc150150 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f4500fd0640 1 --2- 172.21.15.88:0/3969819104 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44fc0cb5c0 0x7f44fc15a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f4500fd0640 1 -- 172.21.15.88:0/3969819104 >> 172.21.15.88:0/3969819104 conn(0x7f44fc0ca9f0 msgr2=0x7f44fc0cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:24.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f4500fd0640 1 -- 172.21.15.88:0/3969819104 shutdown_connections 2023-10-16T20:39:24.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f4500fd0640 1 -- 172.21.15.88:0/3969819104 wait complete. 2023-10-16T20:39:24.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f4500fd0640 1 Processor -- start 2023-10-16T20:39:24.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f65f359e640 1 Processor -- start 2023-10-16T20:39:24.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f65f359e640 1 -- start start 2023-10-16T20:39:24.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f65f359e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f65f40c2050 0x7f65f4159b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f65f359e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f40c6ff0 con 0x7f65f406dd00 2023-10-16T20:39:24.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f65f359e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f40c6a30 con 0x7f65f4155c80 2023-10-16T20:39:24.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f65f359e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f40c6510 con 0x7f65f40c2050 2023-10-16T20:39:24.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f65f2d9d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f65f40c2050 0x7f65f4159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f65f2d9d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f65f40c2050 0x7f65f4159b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:40324/0 (socket says 172.21.15.88:40324) 2023-10-16T20:39:24.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f65f2d9d640 1 -- 172.21.15.88:0/374113453 learned_addr learned my addr 172.21.15.88:0/374113453 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:24.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f65f1d9b640 1 --1- 172.21.15.88:0/374113453 >> v1:172.21.15.191:6789/0 conn(0x7f65f4155c80 0x7f65f40c7540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:40144/0 (socket says 172.21.15.88:40144) 2023-10-16T20:39:24.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f1c70a38640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1c6c157ba0 0x7f1c6c161780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f1c71239640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1c6c0c5940 0x7f1c6c15ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f1c70a38640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1c6c157ba0 0x7f1c6c161780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56618/0 (socket says 172.21.15.88:56618) 2023-10-16T20:39:24.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f1c70a38640 1 -- 172.21.15.88:0/3886465792 learned_addr learned my addr 172.21.15.88:0/3886465792 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:24.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f1c71239640 1 -- 172.21.15.88:0/3886465792 >> v2:172.21.15.88:3300/0 conn(0x7f1c6c157ba0 msgr2=0x7f1c6c161780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:24.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f1c71239640 1 --2- 172.21.15.88:0/3886465792 >> v2:172.21.15.88:3300/0 conn(0x7f1c6c157ba0 0x7f1c6c161780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f1c71239640 1 -- 172.21.15.88:0/3886465792 >> v1:172.21.15.88:6789/0 conn(0x7f1c6c0c6eb0 legacy=0x7f1c6c06d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:24.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f1c71239640 1 -- 172.21.15.88:0/3886465792 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6c069f30 con 0x7f1c6c0c5940 2023-10-16T20:39:24.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.357+0000 7f9e7c46b640 1 -- start start 2023-10-16T20:39:24.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f9e7c46b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9e6c0dd300 0x7f9e6c04ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f9e7c46b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e6c04c580 0x7f9e6c04b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f9e7a207640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9e6c0dd300 0x7f9e6c04ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f9e7a207640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9e6c0dd300 0x7f9e6c04ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56630/0 (socket says 172.21.15.88:56630) 2023-10-16T20:39:24.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f9e7a207640 1 -- 172.21.15.88:0/1445124053 learned_addr learned my addr 172.21.15.88:0/1445124053 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:24.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f9e79a06640 1 --2- 172.21.15.88:0/1445124053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e6c04c580 0x7f9e6c04b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1445124053 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6c035c10 con 0x7f9e6c0e3070 2023-10-16T20:39:24.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1445124053 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e6c0e3c50 con 0x7f9e6c0dd300 2023-10-16T20:39:24.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1445124053 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c045410 con 0x7f9e6c04c580 2023-10-16T20:39:24.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7fd9567fc640 1 --2- 172.21.15.88:0/4196904983 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9580cb5e0 0x7fd9581fe750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7fd9577fe640 1 --2- 172.21.15.88:0/4196904983 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd95815d260 0x7fd9581fec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f4500fd0640 1 -- start start 2023-10-16T20:39:24.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f65f2d9d640 1 -- 172.21.15.88:0/374113453 >> v1:172.21.15.191:6789/0 conn(0x7f65f4155c80 legacy=0x7f65f40c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f65f2d9d640 1 -- 172.21.15.88:0/374113453 >> v1:172.21.15.88:6789/0 conn(0x7f65f406dd00 legacy=0x7f65f41591d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T20:39:24.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f65f2d9d640 1 -- 172.21.15.88:0/374113453 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f40c6d10 con 0x7f65f40c2050 2023-10-16T20:39:24.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f65f159a640 1 -- 172.21.15.88:0/374113453 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178219626 0 0) 0x7f65f40c6a30 con 0x7f65f4155c80 2023-10-16T20:39:24.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7fd956ffd640 1 -- 172.21.15.88:0/4196904983 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd95815d260 msgr2=0x7fd9581fec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7fd956ffd640 1 --2- 172.21.15.88:0/4196904983 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd95815d260 0x7fd9581fec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7fd956ffd640 1 -- 172.21.15.88:0/4196904983 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9580cb5e0 msgr2=0x7fd9581fe750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7fd956ffd640 1 --2- 172.21.15.88:0/4196904983 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9580cb5e0 0x7fd9581fe750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7fd956ffd640 1 -- 172.21.15.88:0/4196904983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd950009a70 con 0x7fd9580c9eb0 2023-10-16T20:39:24.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7fd956ffd640 1 --2- 172.21.15.88:0/4196904983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9580c9eb0 0x7fd9581ff8f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd94800c7f0 tx=0x7fd94800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f1c71239640 1 --2- 172.21.15.88:0/3886465792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c6c0c5940 0x7f1c6c15ef50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c06f580 tx=0x7f1c6401c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1111fe25a440fa80 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f9e79a06640 1 -- 172.21.15.88:0/1445124053 >> v2:172.21.15.88:3300/0 conn(0x7f9e6c0dd300 msgr2=0x7f9e6c04ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:24.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f9e79a06640 1 --2- 172.21.15.88:0/1445124053 >> v2:172.21.15.88:3300/0 conn(0x7f9e6c0dd300 0x7f9e6c04ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7fd93ffff640 1 -- 172.21.15.88:0/4196904983 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd948004220 con 0x7fd9580c9eb0 2023-10-16T20:39:24.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7fd95d8c9640 1 -- 172.21.15.88:0/4196904983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9581586a0 con 0x7fd9580c9eb0 2023-10-16T20:39:24.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f4500fd0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44fc0c66e0 0x7f44fc162de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f1c6b7fe640 1 -- 172.21.15.88:0/3886465792 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c640a5960 con 0x7f1c6c0c5940 2023-10-16T20:39:24.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.361+0000 7f9e79a06640 1 -- 172.21.15.88:0/1445124053 >> v1:172.21.15.88:6789/0 conn(0x7f9e6c0e3070 legacy=0x7f9e6c0461b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f9e79a06640 1 -- 172.21.15.88:0/1445124053 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6c0e3f10 con 0x7f9e6c04c580 2023-10-16T20:39:24.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f4500fd0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f44fc0cb5c0 0x7f44fc0c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f4ae6f65640 1 Processor -- start 2023-10-16T20:39:24.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f4500fd0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44fc150690 0x7f44fc0c2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f44fa59c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44fc0c66e0 0x7f44fc162de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f44fa59c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44fc0c66e0 0x7f44fc162de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:37812/0 (socket says 172.21.15.88:37812) 2023-10-16T20:39:24.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f1c6b7fe640 1 -- 172.21.15.88:0/3886465792 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c640a5ac0 con 0x7f1c6c0c5940 2023-10-16T20:39:24.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f4ae6f65640 1 -- start start 2023-10-16T20:39:24.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f44fa59c640 1 -- 172.21.15.88:0/3484737768 learned_addr learned my addr 172.21.15.88:0/3484737768 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:24.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f4500fd0640 1 -- 172.21.15.88:0/3484737768 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc164de0 con 0x7f44fc150690 2023-10-16T20:39:24.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f1c6b7fe640 1 -- 172.21.15.88:0/3886465792 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c640a5960 con 0x7f1c6c0c5940 2023-10-16T20:39:24.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f4500fd0640 1 -- 172.21.15.88:0/3484737768 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc163420 con 0x7f44fc0cb5c0 2023-10-16T20:39:24.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f4500fd0640 1 -- 172.21.15.88:0/3484737768 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc163700 con 0x7f44fc0c66e0 2023-10-16T20:39:24.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f44f1d9b640 1 --2- 172.21.15.88:0/3484737768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f44fc0cb5c0 0x7f44fc0c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7f44fad9d640 1 --2- 172.21.15.88:0/3484737768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44fc150690 0x7f44fc0c2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7fd95d8c9640 1 -- 172.21.15.88:0/4196904983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9580c7db0 con 0x7fd9580c9eb0 2023-10-16T20:39:24.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.365+0000 7fd93ffff640 1 -- 172.21.15.88:0/4196904983 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd948004380 con 0x7fd9580c9eb0 2023-10-16T20:39:24.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7fd93ffff640 1 -- 172.21.15.88:0/4196904983 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd948004220 con 0x7fd9580c9eb0 2023-10-16T20:39:24.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f1c71a3a640 1 -- 172.21.15.88:0/3886465792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c6c0c5940 msgr2=0x7f1c6c15ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f1c71a3a640 1 --2- 172.21.15.88:0/3886465792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c6c0c5940 0x7f1c6c15ef50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c06f580 tx=0x7f1c6401c8a0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f9e79a06640 1 --2- 172.21.15.88:0/1445124053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e6c04c580 0x7f9e6c04b220 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9e74063ef0 tx=0x7f9e74099be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43dc6ea2ea566ead server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f4ae6f65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ae0152bb0 0x7f4ae0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f4ae6f65640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4ae00c6fa0 0x7f4ae006a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7fd95d8c9640 1 -- 172.21.15.88:0/4196904983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd938005260 con 0x7fd9580c9eb0 2023-10-16T20:39:24.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f44f1d9b640 1 -- 172.21.15.88:0/3484737768 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44fc0c66e0 msgr2=0x7f44fc162de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f44f1d9b640 1 --2- 172.21.15.88:0/3484737768 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44fc0c66e0 0x7f44fc162de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f44f1d9b640 1 -- 172.21.15.88:0/3484737768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44fc150690 msgr2=0x7f44fc0c2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f44f1d9b640 1 --2- 172.21.15.88:0/3484737768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44fc150690 0x7f44fc0c2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f44f1d9b640 1 -- 172.21.15.88:0/3484737768 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e4009420 con 0x7f44fc0cb5c0 2023-10-16T20:39:24.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f1c71a3a640 1 -- 172.21.15.88:0/3886465792 shutdown_connections 2023-10-16T20:39:24.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f1c71a3a640 1 --2- 172.21.15.88:0/3886465792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c6c0c5940 0x7f1c6c15ef50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f1c71a3a640 1 --2- 172.21.15.88:0/3886465792 >> v2:172.21.15.88:3300/0 conn(0x7f1c6c157ba0 0x7f1c6c161780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f1c71a3a640 1 -- 172.21.15.88:0/3886465792 >> 172.21.15.88:0/3886465792 conn(0x7f1c6c0caee0 msgr2=0x7f1c6c0cb2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:24.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f1c71a3a640 1 -- 172.21.15.88:0/3886465792 shutdown_connections 2023-10-16T20:39:24.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f9e79205640 1 -- 172.21.15.88:0/1445124053 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e74072040 con 0x7f9e6c04c580 2023-10-16T20:39:24.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f9e79205640 1 -- 172.21.15.88:0/1445124053 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e741238d0 con 0x7f9e6c04c580 2023-10-16T20:39:24.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f9e79205640 1 -- 172.21.15.88:0/1445124053 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e74123d00 con 0x7f9e6c04c580 2023-10-16T20:39:24.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f4ae4d01640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ae0152bb0 0x7f4ae0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f4ae4d01640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ae0152bb0 0x7f4ae0069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:37814/0 (socket says 172.21.15.88:37814) 2023-10-16T20:39:24.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f4ae4d01640 1 -- 172.21.15.88:0/1239777421 learned_addr learned my addr 172.21.15.88:0/1239777421 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:24.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f4ae6f65640 1 --2- 172.21.15.88:0/1239777421 >> v2:172.21.15.191:3300/0 conn(0x7f4ae00cb510 0x7f4ae0158b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f4ae6f65640 1 -- 172.21.15.88:0/1239777421 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ae01565a0 con 0x7f4ae00c6fa0 2023-10-16T20:39:24.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f4ae6f65640 1 -- 172.21.15.88:0/1239777421 --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ae01556d0 con 0x7f4ae00cb510 2023-10-16T20:39:24.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f4ae6f65640 1 -- 172.21.15.88:0/1239777421 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae0155920 con 0x7f4ae0152bb0 2023-10-16T20:39:24.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f3e6613b640 1 Processor -- start 2023-10-16T20:39:24.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f65f2d9d640 1 --2- 172.21.15.88:0/374113453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65f40c2050 0x7f65f4159b90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f65f40c5580 tx=0x7f65e401c4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4b3770ad1ce74d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1445124053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e6c04c580 msgr2=0x7f9e6c04b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f9e7c46b640 1 --2- 172.21.15.88:0/1445124053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e6c04c580 0x7f9e6c04b220 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9e74063ef0 tx=0x7f9e74099be0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f4adffff640 1 --2- 172.21.15.88:0/1239777421 >> v2:172.21.15.88:3300/0 conn(0x7f4ae00c6fa0 0x7f4ae006a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f4ae5502640 1 --2- 172.21.15.88:0/1239777421 >> v2:172.21.15.191:3300/0 conn(0x7f4ae00cb510 0x7f4ae0158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f4ae4d01640 1 -- 172.21.15.88:0/1239777421 >> v2:172.21.15.191:3300/0 conn(0x7f4ae00cb510 msgr2=0x7f4ae0158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:24.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f4ae4d01640 1 --2- 172.21.15.88:0/1239777421 >> v2:172.21.15.191:3300/0 conn(0x7f4ae00cb510 0x7f4ae0158b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f4ae4d01640 1 -- 172.21.15.88:0/1239777421 >> v2:172.21.15.88:3300/0 conn(0x7f4ae00c6fa0 msgr2=0x7f4ae006a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:24.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f4ae4d01640 1 --2- 172.21.15.88:0/1239777421 >> v2:172.21.15.88:3300/0 conn(0x7f4ae00c6fa0 0x7f4ae006a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f4ae4d01640 1 -- 172.21.15.88:0/1239777421 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae0155c00 con 0x7f4ae0152bb0 2023-10-16T20:39:24.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1445124053 shutdown_connections 2023-10-16T20:39:24.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f9e7c46b640 1 --2- 172.21.15.88:0/1445124053 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e6c04c580 0x7f9e6c04b220 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f1c71a3a640 1 -- 172.21.15.88:0/3886465792 wait complete. 2023-10-16T20:39:24.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f9e7c46b640 1 --2- 172.21.15.88:0/1445124053 >> v2:172.21.15.88:3300/0 conn(0x7f9e6c0dd300 0x7f9e6c04ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1445124053 >> 172.21.15.88:0/1445124053 conn(0x7f9e6c0501d0 msgr2=0x7f9e6c0505e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:24.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f4adffff640 1 --2- 172.21.15.88:0/1239777421 >> v2:172.21.15.88:3300/0 conn(0x7f4ae00c6fa0 0x7f4ae006a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:24.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f3e6613b640 1 -- start start 2023-10-16T20:39:24.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f65f159a640 1 -- 172.21.15.88:0/374113453 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e40ac630 con 0x7f65f40c2050 2023-10-16T20:39:24.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1445124053 shutdown_connections 2023-10-16T20:39:24.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f65f159a640 1 -- 172.21.15.88:0/374113453 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e40ac790 con 0x7f65f40c2050 2023-10-16T20:39:24.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f65f159a640 1 -- 172.21.15.88:0/374113453 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e40acb20 con 0x7f65f40c2050 2023-10-16T20:39:24.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1445124053 wait complete. 2023-10-16T20:39:24.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7f44f1d9b640 1 --2- 172.21.15.88:0/3484737768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f44fc0cb5c0 0x7f44fc0c8a30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f44e40ab000 tx=0x7f44e401ac50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e6613b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3e600c7570 0x7f3e600c7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f44f37fe640 1 -- 172.21.15.88:0/3484737768 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4006e20 con 0x7f44fc0cb5c0 2023-10-16T20:39:24.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f44f37fe640 1 -- 172.21.15.88:0/3484737768 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e4004030 con 0x7f44fc0cb5c0 2023-10-16T20:39:24.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f44f37fe640 1 -- 172.21.15.88:0/3484737768 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4004400 con 0x7f44fc0cb5c0 2023-10-16T20:39:24.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f4500fd0640 1 -- 172.21.15.88:0/3484737768 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44fc1639e0 con 0x7f44fc0cb5c0 2023-10-16T20:39:24.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f4500fd0640 1 -- 172.21.15.88:0/3484737768 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44fc163fa0 con 0x7f44fc0cb5c0 2023-10-16T20:39:24.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e6613b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e600c6ff0 con 0x7f3e600c2050 2023-10-16T20:39:24.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e6613b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e600c6a30 con 0x7f3e60155c80 2023-10-16T20:39:24.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e6613b640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e600c6510 con 0x7f3e600c7570 2023-10-16T20:39:24.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.369+0000 7fd93ffff640 1 -- 172.21.15.88:0/4196904983 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd948003830 con 0x7fd9580c9eb0 2023-10-16T20:39:24.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7fd93ffff640 1 --2- 172.21.15.88:0/4196904983 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd944066530 0x7fd9440689f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e64938640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f3e60155c80 0x7f3e601591d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:40158/0 (socket says 172.21.15.88:40158) 2023-10-16T20:39:24.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e64938640 1 -- 172.21.15.88:0/3096781051 learned_addr learned my addr 172.21.15.88:0/3096781051 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:24.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f1c71a3a640 1 Processor -- start 2023-10-16T20:39:24.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7fd93ffff640 1 -- 172.21.15.88:0/4196904983 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd9480556c0 con 0x7fd9580c9eb0 2023-10-16T20:39:24.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f4500fd0640 1 -- 172.21.15.88:0/3484737768 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44fc074700 con 0x7f44fc0cb5c0 2023-10-16T20:39:24.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f9e7c46b640 1 Processor -- start 2023-10-16T20:39:24.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f65f359e640 1 -- 172.21.15.88:0/374113453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65f40c2050 msgr2=0x7f65f4159b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f65f359e640 1 --2- 172.21.15.88:0/374113453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65f40c2050 0x7f65f4159b90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f65f40c5580 tx=0x7f65e401c4e0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f65f359e640 1 -- 172.21.15.88:0/374113453 shutdown_connections 2023-10-16T20:39:24.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f65f359e640 1 --2- 172.21.15.88:0/374113453 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65f40c2050 0x7f65f4159b90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f65f359e640 1 -- 172.21.15.88:0/374113453 >> 172.21.15.88:0/374113453 conn(0x7f65f406d1e0 msgr2=0x7f65f406d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:24.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f65f359e640 1 -- 172.21.15.88:0/374113453 shutdown_connections 2023-10-16T20:39:24.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f65f359e640 1 -- 172.21.15.88:0/374113453 wait complete. 2023-10-16T20:39:24.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e65139640 1 --2- 172.21.15.88:0/3096781051 >> v2:172.21.15.191:3300/0 conn(0x7f3e600c7570 0x7f3e600c7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e57fff640 1 -- 172.21.15.88:0/3096781051 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293176029 0 0) 0x7f3e600c6a30 con 0x7f3e60155c80 2023-10-16T20:39:24.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7fd9567fc640 1 --2- 172.21.15.88:0/4196904983 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd944066530 0x7fd9440689f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f9e7c46b640 1 -- start start 2023-10-16T20:39:24.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e57fff640 1 -- 172.21.15.88:0/3096781051 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e4c003450 con 0x7f3e60155c80 2023-10-16T20:39:24.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e57fff640 1 -- 172.21.15.88:0/3096781051 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3308008300 0 0) 0x7f3e4c003450 con 0x7f3e60155c80 2023-10-16T20:39:24.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e57fff640 1 -- 172.21.15.88:0/3096781051 >> v2:172.21.15.191:3300/0 conn(0x7f3e600c7570 msgr2=0x7f3e600c7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:24.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e57fff640 1 --2- 172.21.15.88:0/3096781051 >> v2:172.21.15.191:3300/0 conn(0x7f3e600c7570 0x7f3e600c7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e57fff640 1 -- 172.21.15.88:0/3096781051 >> v1:172.21.15.88:6789/0 conn(0x7f3e600c2050 legacy=0x7f3e6015fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e57fff640 1 -- 172.21.15.88:0/3096781051 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e600c6d10 con 0x7f3e60155c80 2023-10-16T20:39:24.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e57fff640 1 -- 172.21.15.88:0/3096781051 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f3e5c004ce0 con 0x7f3e60155c80 2023-10-16T20:39:24.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e57fff640 1 -- 172.21.15.88:0/3096781051 >> v1:172.21.15.191:6789/0 conn(0x7f3e60155c80 legacy=0x7f3e601591d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e57fff640 1 --2- 172.21.15.88:0/3096781051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e4c003ac0 0x7f3e4c003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e57fff640 1 --2- 172.21.15.88:0/3096781051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e4c006650 0x7f3e4c008ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e57fff640 1 --2- 172.21.15.88:0/3096781051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e4c009330 0x7f3e4c0097e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e57fff640 1 -- 172.21.15.88:0/3096781051 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e600c6510 con 0x7f3e4c009330 2023-10-16T20:39:24.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e57fff640 1 -- 172.21.15.88:0/3096781051 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c00a0c0 con 0x7f3e4c006650 2023-10-16T20:39:24.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f3e57fff640 1 -- 172.21.15.88:0/3096781051 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c00a280 con 0x7f3e4c003ac0 2023-10-16T20:39:24.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.373+0000 7f1c71a3a640 1 -- start start 2023-10-16T20:39:24.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7c46b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e6c04c580 0x7f9e6c16fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c71a3a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c6c0c5940 0x7f1c6c15c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c71a3a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c6c0c6eb0 0x7f1c6c15b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c71a3a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c6c157ba0 0x7f1c6c1520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c71a3a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c15cd90 con 0x7f1c6c0c6eb0 2023-10-16T20:39:24.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c71a3a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c15e180 con 0x7f1c6c0c5940 2023-10-16T20:39:24.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c71a3a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c15d5b0 con 0x7f1c6c157ba0 2023-10-16T20:39:24.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7c46b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e6c0dd300 0x7f9e6c17bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7c46b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e6c0e3070 0x7f9e6c0ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7c46b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c0d5a90 con 0x7f9e6c0dd300 2023-10-16T20:39:24.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7c46b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c0d54d0 con 0x7f9e6c0e3070 2023-10-16T20:39:24.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7c46b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c0d57b0 con 0x7f9e6c04c580 2023-10-16T20:39:24.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f3e64938640 1 --2- 172.21.15.88:0/3096781051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e4c006650 0x7f3e4c008ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f3e65139640 1 --2- 172.21.15.88:0/3096781051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e4c003ac0 0x7f3e4c003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f44f37fe640 1 -- 172.21.15.88:0/3484737768 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f44e40b7660 con 0x7f44fc0cb5c0 2023-10-16T20:39:24.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7fd9567fc640 1 --2- 172.21.15.88:0/4196904983 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd944066530 0x7fd9440689f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd9581642f0 tx=0x7fd94c007b60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c71239640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c6c157ba0 0x7f1c6c1520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c71239640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c6c157ba0 0x7f1c6c1520c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:37832/0 (socket says 172.21.15.88:37832) 2023-10-16T20:39:24.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c71239640 1 -- 172.21.15.88:0/3500516803 learned_addr learned my addr 172.21.15.88:0/3500516803 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:24.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f4ae4d01640 1 --2- 172.21.15.88:0/1239777421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ae0152bb0 0x7f4ae0069e90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4ae006c800 tx=0x7f4ad401e8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=583552f5c8df3e30 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7aa08640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e6c0e3070 0x7f9e6c0ceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7a207640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e6c04c580 0x7f9e6c16fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7aa08640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e6c0e3070 0x7f9e6c0ceb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:40392/0 (socket says 172.21.15.88:40392) 2023-10-16T20:39:24.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7aa08640 1 -- 172.21.15.88:0/1796124193 learned_addr learned my addr 172.21.15.88:0/1796124193 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:24.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f3e65139640 1 -- 172.21.15.88:0/3096781051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e4c006650 msgr2=0x7f3e4c008ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:24.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f3e65139640 1 --2- 172.21.15.88:0/3096781051 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e4c006650 0x7f3e4c008ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f3e65139640 1 -- 172.21.15.88:0/3096781051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e4c009330 msgr2=0x7f3e4c0097e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-16T20:39:24.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f3e65139640 1 --2- 172.21.15.88:0/3096781051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e4c009330 0x7f3e4c0097e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f3e65139640 1 -- 172.21.15.88:0/3096781051 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e600c6d10 con 0x7f3e4c003ac0 2023-10-16T20:39:24.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c71239640 1 -- 172.21.15.88:0/3500516803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c6c0c5940 msgr2=0x7f1c6c15c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c71239640 1 --2- 172.21.15.88:0/3500516803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c6c0c5940 0x7f1c6c15c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c71239640 1 -- 172.21.15.88:0/3500516803 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c6c0c6eb0 msgr2=0x7f1c6c15b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c71239640 1 --2- 172.21.15.88:0/3500516803 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c6c0c6eb0 0x7f1c6c15b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c71239640 1 -- 172.21.15.88:0/3500516803 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c64009a70 con 0x7f1c6c157ba0 2023-10-16T20:39:24.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f44f37fe640 1 --2- 172.21.15.88:0/3484737768 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44dc0664f0 0x7f44dc0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f4adf7fe640 1 -- 172.21.15.88:0/1239777421 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad40a7950 con 0x7f4ae0152bb0 2023-10-16T20:39:24.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f4adf7fe640 1 -- 172.21.15.88:0/1239777421 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad40a7ab0 con 0x7f4ae0152bb0 2023-10-16T20:39:24.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f4adf7fe640 1 -- 172.21.15.88:0/1239777421 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad40a7e40 con 0x7f4ae0152bb0 2023-10-16T20:39:24.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7a207640 1 -- 172.21.15.88:0/1796124193 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e6c0e3070 msgr2=0x7f9e6c0ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7a207640 1 --2- 172.21.15.88:0/1796124193 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e6c0e3070 0x7f9e6c0ceb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7a207640 1 -- 172.21.15.88:0/1796124193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e6c0dd300 msgr2=0x7f9e6c17bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7a207640 1 --2- 172.21.15.88:0/1796124193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e6c0dd300 0x7f9e6c17bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7a207640 1 -- 172.21.15.88:0/1796124193 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e74067780 con 0x7f9e6c04c580 2023-10-16T20:39:24.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f3e6593a640 1 -- 172.21.15.88:0/3096781051 reap_dead start 2023-10-16T20:39:24.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f44f37fe640 1 -- 172.21.15.88:0/3484737768 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f44e411e1e0 con 0x7f44fc0cb5c0 2023-10-16T20:39:24.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7a207640 1 --2- 172.21.15.88:0/1796124193 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e6c04c580 0x7f9e6c16fb90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9e6400e870 tx=0x7f9e6400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f65f359e640 1 Processor -- start 2023-10-16T20:39:24.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c71239640 1 --2- 172.21.15.88:0/3500516803 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c6c157ba0 0x7f1c6c1520c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1c6401af50 tx=0x7f1c64019640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c697fa640 1 -- 172.21.15.88:0/3500516803 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c640a5960 con 0x7f1c6c157ba0 2023-10-16T20:39:24.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e6affd640 1 -- 172.21.15.88:0/1796124193 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6400ee60 con 0x7f9e6c04c580 2023-10-16T20:39:24.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e6affd640 1 -- 172.21.15.88:0/1796124193 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6400c940 con 0x7f9e6c04c580 2023-10-16T20:39:24.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e6affd640 1 -- 172.21.15.88:0/1796124193 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6400ccb0 con 0x7f9e6c04c580 2023-10-16T20:39:24.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f65f359e640 1 -- start start 2023-10-16T20:39:24.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1796124193 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6c0d93b0 con 0x7f9e6c04c580 2023-10-16T20:39:24.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1796124193 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6c0d9970 con 0x7f9e6c04c580 2023-10-16T20:39:24.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c697fa640 1 -- 172.21.15.88:0/3500516803 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c64009450 con 0x7f1c6c157ba0 2023-10-16T20:39:24.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.377+0000 7f1c697fa640 1 -- 172.21.15.88:0/3500516803 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c640b7410 con 0x7f1c6c157ba0 2023-10-16T20:39:24.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f44fa59c640 1 --2- 172.21.15.88:0/3484737768 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44dc0664f0 0x7f44dc0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f65f359e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65f406dd00 0x7f65f414fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f65f359e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65f40c2050 0x7f65f406a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f65f359e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65f4155c80 0x7f65f4140b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f65f359e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f414eec0 con 0x7f65f40c2050 2023-10-16T20:39:24.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f65f359e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f4152780 con 0x7f65f4155c80 2023-10-16T20:39:24.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f65f359e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f4151420 con 0x7f65f406dd00 2023-10-16T20:39:24.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f1c71a3a640 1 -- 172.21.15.88:0/3500516803 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c6c15be70 con 0x7f1c6c157ba0 2023-10-16T20:39:24.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f1c71a3a640 1 -- 172.21.15.88:0/3500516803 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c6c15db30 con 0x7f1c6c157ba0 2023-10-16T20:39:24.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1796124193 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e5c005260 con 0x7f9e6c04c580 2023-10-16T20:39:24.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f65f259c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65f406dd00 0x7f65f414fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f65f1d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65f40c2050 0x7f65f406a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f65f1d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65f40c2050 0x7f65f406a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56686/0 (socket says 172.21.15.88:56686) 2023-10-16T20:39:24.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f65f1d9b640 1 -- 172.21.15.88:0/2939244196 learned_addr learned my addr 172.21.15.88:0/2939244196 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:24.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f65f2d9d640 1 --2- 172.21.15.88:0/2939244196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65f4155c80 0x7f65f4140b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f44fa59c640 1 --2- 172.21.15.88:0/3484737768 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44dc0664f0 0x7f44dc0689b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009c20 tx=0x7f44ec009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f1c71a3a640 1 -- 172.21.15.88:0/3500516803 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c6c04e9c0 con 0x7f1c6c157ba0 2023-10-16T20:39:24.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f4ae6f65640 1 -- 172.21.15.88:0/1239777421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ae0152bb0 msgr2=0x7f4ae0069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f4ae6f65640 1 --2- 172.21.15.88:0/1239777421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ae0152bb0 0x7f4ae0069e90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4ae006c800 tx=0x7f4ad401e8e0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7fd93ffff640 1 -- 172.21.15.88:0/4196904983 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd948051a90 con 0x7fd9580c9eb0 2023-10-16T20:39:24.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f44f37fe640 1 -- 172.21.15.88:0/3484737768 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f44e40eb3a0 con 0x7f44fc0cb5c0 2023-10-16T20:39:24.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f65f2d9d640 1 -- 172.21.15.88:0/2939244196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65f406dd00 msgr2=0x7f65f414fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f65f2d9d640 1 --2- 172.21.15.88:0/2939244196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65f406dd00 0x7f65f414fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f65f2d9d640 1 -- 172.21.15.88:0/2939244196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65f40c2050 msgr2=0x7f65f406a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f65f2d9d640 1 --2- 172.21.15.88:0/2939244196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65f40c2050 0x7f65f406a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f65f2d9d640 1 -- 172.21.15.88:0/2939244196 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e40094d0 con 0x7f65f4155c80 2023-10-16T20:39:24.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f4ae6f65640 1 -- 172.21.15.88:0/1239777421 shutdown_connections 2023-10-16T20:39:24.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f4ae6f65640 1 --2- 172.21.15.88:0/1239777421 >> v2:172.21.15.191:3300/0 conn(0x7f4ae00cb510 0x7f4ae0158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f4ae6f65640 1 --2- 172.21.15.88:0/1239777421 >> v2:172.21.15.88:3300/0 conn(0x7f4ae00c6fa0 0x7f4ae006a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f4ae6f65640 1 --2- 172.21.15.88:0/1239777421 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ae0152bb0 0x7f4ae0069e90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f4ae6f65640 1 -- 172.21.15.88:0/1239777421 >> 172.21.15.88:0/1239777421 conn(0x7f4ae00ca9f0 msgr2=0x7f4ae00cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:24.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f3e65139640 1 --2- 172.21.15.88:0/3096781051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e4c003ac0 0x7f3e4c003ee0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009ae0 tx=0x7f3e5001ca50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3a27870b09bbb2e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f4ae6f65640 1 -- 172.21.15.88:0/1239777421 shutdown_connections 2023-10-16T20:39:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f4ae6f65640 1 -- 172.21.15.88:0/1239777421 wait complete. 2023-10-16T20:39:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f9e6affd640 1 -- 172.21.15.88:0/1796124193 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9e64013470 con 0x7f9e6c04c580 2023-10-16T20:39:24.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.381+0000 7f65f2d9d640 1 --2- 172.21.15.88:0/2939244196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65f4155c80 0x7f65f4140b60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f65f406af00 tx=0x7f65e4019280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.385+0000 7f65e2ffd640 1 -- 172.21.15.88:0/2939244196 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e40b6640 con 0x7f65f4155c80 2023-10-16T20:39:24.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.385+0000 7f65e2ffd640 1 -- 172.21.15.88:0/2939244196 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e40b67c0 con 0x7f65f4155c80 2023-10-16T20:39:24.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.385+0000 7f65e2ffd640 1 -- 172.21.15.88:0/2939244196 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e40b6b70 con 0x7f65f4155c80 2023-10-16T20:39:24.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.385+0000 7f65f359e640 1 -- 172.21.15.88:0/2939244196 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f41502a0 con 0x7f65f4155c80 2023-10-16T20:39:24.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.385+0000 7f65f359e640 1 -- 172.21.15.88:0/2939244196 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f4150810 con 0x7f65f4155c80 2023-10-16T20:39:24.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.385+0000 7f9e6affd640 1 --2- 172.21.15.88:0/1796124193 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9e580664a0 0x7f9e58068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.385+0000 7f9e6affd640 1 -- 172.21.15.88:0/1796124193 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f9e640841e0 con 0x7f9e6c04c580 2023-10-16T20:39:24.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.385+0000 7f1c697fa640 1 -- 172.21.15.88:0/3500516803 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1c640b4050 con 0x7f1c6c157ba0 2023-10-16T20:39:24.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.385+0000 7f65f359e640 1 -- 172.21.15.88:0/2939244196 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f404e9c0 con 0x7f65f4155c80 2023-10-16T20:39:24.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.385+0000 7f4ae6f65640 1 Processor -- start 2023-10-16T20:39:24.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.385+0000 7f9e79a06640 1 --2- 172.21.15.88:0/1796124193 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9e580664a0 0x7f9e58068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.385+0000 7f4ae6f65640 1 -- start start 2023-10-16T20:39:24.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.385+0000 7f1c697fa640 1 --2- 172.21.15.88:0/3500516803 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1c580664d0 0x7f1c58068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.385+0000 7f1c697fa640 1 -- 172.21.15.88:0/3500516803 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f1c6411f280 con 0x7f1c6c157ba0 2023-10-16T20:39:24.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.385+0000 7f1c70a38640 1 --2- 172.21.15.88:0/3500516803 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1c580664d0 0x7f1c58068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.385+0000 7f9e79a06640 1 --2- 172.21.15.88:0/1796124193 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9e580664a0 0x7f9e58068960 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c04cc20 tx=0x7f9e740753c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae6f65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ae00c6fa0 0x7f4ae0148c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae6f65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ae00cb510 0x7f4ae0147810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae6f65640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ae0152bb0 0x7f4ae0147d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae6f65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae01473f0 con 0x7f4ae00c6fa0 2023-10-16T20:39:24.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae6f65640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae0158460 con 0x7f4ae0152bb0 2023-10-16T20:39:24.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae6f65640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae01491d0 con 0x7f4ae00cb510 2023-10-16T20:39:24.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f3e57fff640 1 -- 172.21.15.88:0/3096781051 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e500a5640 con 0x7f3e4c003ac0 2023-10-16T20:39:24.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f65e2ffd640 1 -- 172.21.15.88:0/2939244196 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f65e40b6cd0 con 0x7f65f4155c80 2023-10-16T20:39:24.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f1c70a38640 1 --2- 172.21.15.88:0/3500516803 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1c580664d0 0x7f1c58068990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c0094a0 tx=0x7f1c5c0093f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f9e6affd640 1 -- 172.21.15.88:0/1796124193 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9e64051330 con 0x7f9e6c04c580 2023-10-16T20:39:24.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae4d01640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ae00c6fa0 0x7f4ae0148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae4d01640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ae00c6fa0 0x7f4ae0148c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56688/0 (socket says 172.21.15.88:56688) 2023-10-16T20:39:24.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae4d01640 1 -- 172.21.15.88:0/4057084170 learned_addr learned my addr 172.21.15.88:0/4057084170 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:24.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae5502640 1 --2- 172.21.15.88:0/4057084170 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ae0152bb0 0x7f4ae0147d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4adffff640 1 --2- 172.21.15.88:0/4057084170 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ae00cb510 0x7f4ae0147810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f65e2ffd640 1 --2- 172.21.15.88:0/2939244196 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f65d40664f0 0x7f65d40689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae4d01640 1 -- 172.21.15.88:0/4057084170 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ae00cb510 msgr2=0x7f4ae0147810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae4d01640 1 --2- 172.21.15.88:0/4057084170 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ae00cb510 0x7f4ae0147810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae4d01640 1 -- 172.21.15.88:0/4057084170 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ae0152bb0 msgr2=0x7f4ae0147d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae4d01640 1 --2- 172.21.15.88:0/4057084170 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ae0152bb0 0x7f4ae0147d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae4d01640 1 -- 172.21.15.88:0/4057084170 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad400a670 con 0x7f4ae00c6fa0 2023-10-16T20:39:24.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f65e2ffd640 1 -- 172.21.15.88:0/2939244196 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f65e411e2c0 con 0x7f65f4155c80 2023-10-16T20:39:24.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f65f259c640 1 --2- 172.21.15.88:0/2939244196 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f65d40664f0 0x7f65d40689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae4d01640 1 --2- 172.21.15.88:0/4057084170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ae00c6fa0 0x7f4ae0148c90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400a1c0 tx=0x7f4ad401b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4add7fa640 1 -- 172.21.15.88:0/4057084170 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad40b8620 con 0x7f4ae00c6fa0 2023-10-16T20:39:24.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4add7fa640 1 -- 172.21.15.88:0/4057084170 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad40b8780 con 0x7f4ae00c6fa0 2023-10-16T20:39:24.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4add7fa640 1 -- 172.21.15.88:0/4057084170 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad40b8b30 con 0x7f4ae00c6fa0 2023-10-16T20:39:24.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae6f65640 1 -- 172.21.15.88:0/4057084170 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ae0070f10 con 0x7f4ae00c6fa0 2023-10-16T20:39:24.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f4ae6f65640 1 -- 172.21.15.88:0/4057084170 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ae00714d0 con 0x7f4ae00c6fa0 2023-10-16T20:39:24.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.389+0000 7f65f259c640 1 --2- 172.21.15.88:0/2939244196 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f65d40664f0 0x7f65d40689b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f65ec001f20 tx=0x7f65ec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.393+0000 7f4ae6f65640 1 -- 172.21.15.88:0/4057084170 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ae0074700 con 0x7f4ae00c6fa0 2023-10-16T20:39:24.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.393+0000 7f3e57fff640 1 -- 172.21.15.88:0/3096781051 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e500a57a0 con 0x7f3e4c003ac0 2023-10-16T20:39:24.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.393+0000 7f3e57fff640 1 -- 172.21.15.88:0/3096781051 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e500a5b50 con 0x7f3e4c003ac0 2023-10-16T20:39:24.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.393+0000 7f1c697fa640 1 -- 172.21.15.88:0/3500516803 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1c640ec450 con 0x7f1c6c157ba0 2023-10-16T20:39:24.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.393+0000 7f3e6613b640 1 -- 172.21.15.88:0/3096781051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e4c003ac0 msgr2=0x7f3e4c003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.393+0000 7f3e6613b640 1 --2- 172.21.15.88:0/3096781051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e4c003ac0 0x7f3e4c003ee0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009ae0 tx=0x7f3e5001ca50 comp rx=0 tx=0).stop 2023-10-16T20:39:24.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.393+0000 7f65e2ffd640 1 -- 172.21.15.88:0/2939244196 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f65e40eb530 con 0x7f65f4155c80 2023-10-16T20:39:24.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.393+0000 7f4add7fa640 1 -- 172.21.15.88:0/4057084170 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4ad40b6070 con 0x7f4ae00c6fa0 2023-10-16T20:39:24.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.393+0000 7f4add7fa640 1 --2- 172.21.15.88:0/4057084170 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4ac40664a0 0x7f4ac4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.393+0000 7f4add7fa640 1 -- 172.21.15.88:0/4057084170 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f4ad411fdc0 con 0x7f4ae00c6fa0 2023-10-16T20:39:24.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.397+0000 7f3e6613b640 1 -- 172.21.15.88:0/3096781051 shutdown_connections 2023-10-16T20:39:24.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.397+0000 7f3e6613b640 1 --2- 172.21.15.88:0/3096781051 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e4c003ac0 0x7f3e4c003ee0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.397+0000 7f3e6613b640 1 -- 172.21.15.88:0/3096781051 >> 172.21.15.88:0/3096781051 conn(0x7f3e6006d1e0 msgr2=0x7f3e6006d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:24.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.397+0000 7f4adffff640 1 --2- 172.21.15.88:0/4057084170 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4ac40664a0 0x7f4ac4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.397+0000 7f4adffff640 1 --2- 172.21.15.88:0/4057084170 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4ac40664a0 0x7f4ac4068960 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0072ad0 tx=0x7f4acc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.397+0000 7f3e6613b640 1 -- 172.21.15.88:0/3096781051 shutdown_connections 2023-10-16T20:39:24.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.397+0000 7f3e6613b640 1 -- 172.21.15.88:0/3096781051 wait complete. 2023-10-16T20:39:24.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.397+0000 7f3e6613b640 1 Processor -- start 2023-10-16T20:39:24.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.401+0000 7f3e6613b640 1 -- start start 2023-10-16T20:39:24.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.401+0000 7f4add7fa640 1 -- 172.21.15.88:0/4057084170 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4ad40ece90 con 0x7f4ae00c6fa0 2023-10-16T20:39:24.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.401+0000 7f3e6613b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e4c003ac0 0x7f3e601ee3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.401+0000 7f3e6613b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e6014ab20 0x7f3e6014da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.401+0000 7f3e6613b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e60153020 0x7f3e60156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.401+0000 7f3e6613b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e600ca5c0 con 0x7f3e4c003ac0 2023-10-16T20:39:24.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.401+0000 7f3e6613b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6014b060 con 0x7f3e60153020 2023-10-16T20:39:24.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.401+0000 7f3e6613b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e600c30c0 con 0x7f3e6014ab20 2023-10-16T20:39:24.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.401+0000 7f3e6593a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e60153020 0x7f3e60156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.401+0000 7f3e64938640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e6014ab20 0x7f3e6014da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.401+0000 7f3e6593a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e60153020 0x7f3e60156400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:40424/0 (socket says 172.21.15.88:40424) 2023-10-16T20:39:24.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.401+0000 7f3e6593a640 1 -- 172.21.15.88:0/1543275112 learned_addr learned my addr 172.21.15.88:0/1543275112 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:24.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.401+0000 7f3e65139640 1 --2- 172.21.15.88:0/1543275112 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e4c003ac0 0x7f3e601ee3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.405+0000 7f3e64938640 1 -- 172.21.15.88:0/1543275112 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e60153020 msgr2=0x7f3e60156400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.405+0000 7f3e64938640 1 --2- 172.21.15.88:0/1543275112 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e60153020 0x7f3e60156400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.405+0000 7f3e64938640 1 -- 172.21.15.88:0/1543275112 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e4c003ac0 msgr2=0x7f3e601ee3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.405+0000 7f3e64938640 1 --2- 172.21.15.88:0/1543275112 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e4c003ac0 0x7f3e601ee3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.405+0000 7f3e64938640 1 -- 172.21.15.88:0/1543275112 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e600ca040 con 0x7f3e6014ab20 2023-10-16T20:39:24.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.405+0000 7f3e65139640 1 --2- 172.21.15.88:0/1543275112 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e4c003ac0 0x7f3e601ee3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:24.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.405+0000 7f3e64938640 1 --2- 172.21.15.88:0/1543275112 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e6014ab20 0x7f3e6014da00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c010d20 tx=0x7f3e5c00e6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.405+0000 7f3e55ffb640 1 -- 172.21.15.88:0/1543275112 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c004b50 con 0x7f3e6014ab20 2023-10-16T20:39:24.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.405+0000 7f3e55ffb640 1 -- 172.21.15.88:0/1543275112 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5c015070 con 0x7f3e6014ab20 2023-10-16T20:39:24.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.405+0000 7f3e55ffb640 1 -- 172.21.15.88:0/1543275112 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c011640 con 0x7f3e6014ab20 2023-10-16T20:39:24.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.405+0000 7f3e6613b640 1 -- 172.21.15.88:0/1543275112 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e601ee8e0 con 0x7f3e6014ab20 2023-10-16T20:39:24.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.405+0000 7f3e6613b640 1 -- 172.21.15.88:0/1543275112 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e601edba0 con 0x7f3e6014ab20 2023-10-16T20:39:24.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.409+0000 7f3e437fe640 1 -- 172.21.15.88:0/1543275112 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3c005260 con 0x7f3e6014ab20 2023-10-16T20:39:24.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.409+0000 7f3e55ffb640 1 -- 172.21.15.88:0/1543275112 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3e5c0117a0 con 0x7f3e6014ab20 2023-10-16T20:39:24.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.409+0000 7f3e55ffb640 1 --2- 172.21.15.88:0/1543275112 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3e44066520 0x7f3e440689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.413+0000 7f3e55ffb640 1 -- 172.21.15.88:0/1543275112 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3e5c009040 con 0x7f3e6014ab20 2023-10-16T20:39:24.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.413+0000 7f3e65139640 1 --2- 172.21.15.88:0/1543275112 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3e44066520 0x7f3e440689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.417+0000 7f3e55ffb640 1 -- 172.21.15.88:0/1543275112 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3e5c0524a0 con 0x7f3e6014ab20 2023-10-16T20:39:24.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.417+0000 7f3e65139640 1 --2- 172.21.15.88:0/1543275112 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3e44066520 0x7f3e440689e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3e500af000 tx=0x7f3e50008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.625+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/4241974937 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8c780c60a0 con 0x7f8c78071e80 2023-10-16T20:39:24.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.625+0000 7f8c74ff9640 1 -- 172.21.15.88:0/4241974937 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8c6c053af0 con 0x7f8c78071e80 2023-10-16T20:39:24.634 INFO:teuthology.orchestra.run.smithi088.stdout:64424509477 2023-10-16T20:39:24.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.629+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/4241974937 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8c640664f0 msgr2=0x7f8c640689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.629+0000 7f8c7e8c9640 1 --2- 172.21.15.88:0/4241974937 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8c640664f0 0x7f8c640689b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8c70000f90 tx=0x7f8c700aa000 comp rx=0 tx=0).stop 2023-10-16T20:39:24.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.629+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/4241974937 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c78071e80 msgr2=0x7f8c78144090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.629+0000 7f8c7e8c9640 1 --2- 172.21.15.88:0/4241974937 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c78071e80 0x7f8c78144090 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00c410 tx=0x7f8c6c00c8d0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.629+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/4241974937 shutdown_connections 2023-10-16T20:39:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.629+0000 7f8c7e8c9640 1 --2- 172.21.15.88:0/4241974937 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8c640664f0 0x7f8c640689b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.629+0000 7f8c7e8c9640 1 --2- 172.21.15.88:0/4241974937 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c78071e80 0x7f8c78144090 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.629+0000 7f8c7e8c9640 1 --2- 172.21.15.88:0/4241974937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c7814f920 0x7f8c781edc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.629+0000 7f8c7e8c9640 1 --2- 172.21.15.88:0/4241974937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c7806a560 0x7f8c7814d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.629+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/4241974937 >> 172.21.15.88:0/4241974937 conn(0x7f8c780cac60 msgr2=0x7f8c780cb040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:24.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.629+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/4241974937 shutdown_connections 2023-10-16T20:39:24.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.629+0000 7f8c7e8c9640 1 -- 172.21.15.88:0/4241974937 wait complete. 2023-10-16T20:39:24.668 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509476 got 64424509477 for osd.5 2023-10-16T20:39:24.669 DEBUG:teuthology.parallel:result is None 2023-10-16T20:39:24.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.693+0000 7f4ae6f65640 1 -- 172.21.15.88:0/4057084170 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f4ae01501c0 con 0x7f4ae00c6fa0 2023-10-16T20:39:24.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.693+0000 7f4add7fa640 1 -- 172.21.15.88:0/4057084170 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4ad40f04b0 con 0x7f4ae00c6fa0 2023-10-16T20:39:24.702 INFO:teuthology.orchestra.run.smithi088.stdout:64424509475 2023-10-16T20:39:24.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.697+0000 7f4ae6f65640 1 -- 172.21.15.88:0/4057084170 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4ac40664a0 msgr2=0x7f4ac4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.697+0000 7f4ae6f65640 1 --2- 172.21.15.88:0/4057084170 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4ac40664a0 0x7f4ac4068960 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0072ad0 tx=0x7f4acc008040 comp rx=0 tx=0).stop 2023-10-16T20:39:24.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.697+0000 7f4ae6f65640 1 -- 172.21.15.88:0/4057084170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ae00c6fa0 msgr2=0x7f4ae0148c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.697+0000 7f4ae6f65640 1 --2- 172.21.15.88:0/4057084170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ae00c6fa0 0x7f4ae0148c90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400a1c0 tx=0x7f4ad401b680 comp rx=0 tx=0).stop 2023-10-16T20:39:24.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.697+0000 7f4ae6f65640 1 -- 172.21.15.88:0/4057084170 shutdown_connections 2023-10-16T20:39:24.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.697+0000 7f4ae6f65640 1 --2- 172.21.15.88:0/4057084170 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4ac40664a0 0x7f4ac4068960 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.697+0000 7f4ae6f65640 1 --2- 172.21.15.88:0/4057084170 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ae0152bb0 0x7f4ae0147d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.697+0000 7f4ae6f65640 1 --2- 172.21.15.88:0/4057084170 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ae00cb510 0x7f4ae0147810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.697+0000 7f4ae6f65640 1 --2- 172.21.15.88:0/4057084170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ae00c6fa0 0x7f4ae0148c90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.697+0000 7f4ae6f65640 1 -- 172.21.15.88:0/4057084170 >> 172.21.15.88:0/4057084170 conn(0x7f4ae00ca9f0 msgr2=0x7f4ae01538d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:24.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.697+0000 7f4ae6f65640 1 -- 172.21.15.88:0/4057084170 shutdown_connections 2023-10-16T20:39:24.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.697+0000 7f4ae6f65640 1 -- 172.21.15.88:0/4057084170 wait complete. 2023-10-16T20:39:24.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.705+0000 7fd95d8c9640 1 -- 172.21.15.88:0/4196904983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd9380057e0 con 0x7fd9580c9eb0 2023-10-16T20:39:24.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.709+0000 7fd93ffff640 1 -- 172.21.15.88:0/4196904983 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd94801e070 con 0x7fd9580c9eb0 2023-10-16T20:39:24.715 INFO:teuthology.orchestra.run.smithi088.stdout:64424509475 2023-10-16T20:39:24.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.709+0000 7fd95d8c9640 1 -- 172.21.15.88:0/4196904983 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd944066530 msgr2=0x7fd9440689f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.709+0000 7fd95d8c9640 1 --2- 172.21.15.88:0/4196904983 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd944066530 0x7fd9440689f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd9581642f0 tx=0x7fd94c007b60 comp rx=0 tx=0).stop 2023-10-16T20:39:24.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.709+0000 7fd95d8c9640 1 -- 172.21.15.88:0/4196904983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9580c9eb0 msgr2=0x7fd9581ff8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.709+0000 7fd95d8c9640 1 --2- 172.21.15.88:0/4196904983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9580c9eb0 0x7fd9581ff8f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd94800c7f0 tx=0x7fd94800ccb0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.709+0000 7fd95d8c9640 1 -- 172.21.15.88:0/4196904983 shutdown_connections 2023-10-16T20:39:24.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.709+0000 7fd95d8c9640 1 --2- 172.21.15.88:0/4196904983 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd944066530 0x7fd9440689f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.709+0000 7fd95d8c9640 1 --2- 172.21.15.88:0/4196904983 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd95815d260 0x7fd9581fec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.709+0000 7fd95d8c9640 1 --2- 172.21.15.88:0/4196904983 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd9580cb5e0 0x7fd9581fe750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.709+0000 7fd95d8c9640 1 --2- 172.21.15.88:0/4196904983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9580c9eb0 0x7fd9581ff8f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.709+0000 7fd95d8c9640 1 -- 172.21.15.88:0/4196904983 >> 172.21.15.88:0/4196904983 conn(0x7fd95806bc00 msgr2=0x7fd9580c4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:24.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.709+0000 7fd95d8c9640 1 -- 172.21.15.88:0/4196904983 shutdown_connections 2023-10-16T20:39:24.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.713+0000 7fd95d8c9640 1 -- 172.21.15.88:0/4196904983 wait complete. 2023-10-16T20:39:24.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.713+0000 7f65f359e640 1 -- 172.21.15.88:0/2939244196 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f65f40c6ff0 con 0x7f65f4155c80 2023-10-16T20:39:24.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.713+0000 7f65e2ffd640 1 -- 172.21.15.88:0/2939244196 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f65e40b1020 con 0x7f65f4155c80 2023-10-16T20:39:24.721 INFO:teuthology.orchestra.run.smithi088.stdout:55834574883 2023-10-16T20:39:24.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.713+0000 7f4500fd0640 1 -- 172.21.15.88:0/3484737768 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f44fc0706c0 con 0x7f44fc0cb5c0 2023-10-16T20:39:24.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.713+0000 7f44f37fe640 1 -- 172.21.15.88:0/3484737768 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f44e40ee9c0 con 0x7f44fc0cb5c0 2023-10-16T20:39:24.723 INFO:teuthology.orchestra.run.smithi088.stdout:64424509475 2023-10-16T20:39:24.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.713+0000 7f65f359e640 1 -- 172.21.15.88:0/2939244196 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f65d40664f0 msgr2=0x7f65d40689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.713+0000 7f65f359e640 1 --2- 172.21.15.88:0/2939244196 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f65d40664f0 0x7f65d40689b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f65ec001f20 tx=0x7f65ec009040 comp rx=0 tx=0).stop 2023-10-16T20:39:24.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.713+0000 7f65f359e640 1 -- 172.21.15.88:0/2939244196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65f4155c80 msgr2=0x7f65f4140b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.713+0000 7f65f359e640 1 --2- 172.21.15.88:0/2939244196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65f4155c80 0x7f65f4140b60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f65f406af00 tx=0x7f65e4019280 comp rx=0 tx=0).stop 2023-10-16T20:39:24.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f65f359e640 1 -- 172.21.15.88:0/2939244196 shutdown_connections 2023-10-16T20:39:24.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f65f359e640 1 --2- 172.21.15.88:0/2939244196 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f65d40664f0 0x7f65d40689b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f65f359e640 1 --2- 172.21.15.88:0/2939244196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65f4155c80 0x7f65f4140b60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f65f359e640 1 --2- 172.21.15.88:0/2939244196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65f40c2050 0x7f65f406a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f65f359e640 1 --2- 172.21.15.88:0/2939244196 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f65f406dd00 0x7f65f414fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f65f359e640 1 -- 172.21.15.88:0/2939244196 >> 172.21.15.88:0/2939244196 conn(0x7f65f406d1e0 msgr2=0x7f65f406d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:24.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f65f359e640 1 -- 172.21.15.88:0/2939244196 shutdown_connections 2023-10-16T20:39:24.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f65f359e640 1 -- 172.21.15.88:0/2939244196 wait complete. 2023-10-16T20:39:24.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1796124193 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f9e5c0057e0 con 0x7f9e6c04c580 2023-10-16T20:39:24.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f9e6affd640 1 -- 172.21.15.88:0/1796124193 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9e64054950 con 0x7f9e6c04c580 2023-10-16T20:39:24.727 INFO:teuthology.orchestra.run.smithi088.stdout:64424509475 2023-10-16T20:39:24.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f4500fd0640 1 -- 172.21.15.88:0/3484737768 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44dc0664f0 msgr2=0x7f44dc0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f4500fd0640 1 --2- 172.21.15.88:0/3484737768 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44dc0664f0 0x7f44dc0689b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009c20 tx=0x7f44ec009190 comp rx=0 tx=0).stop 2023-10-16T20:39:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f4500fd0640 1 -- 172.21.15.88:0/3484737768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f44fc0cb5c0 msgr2=0x7f44fc0c8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f4500fd0640 1 --2- 172.21.15.88:0/3484737768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f44fc0cb5c0 0x7f44fc0c8a30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f44e40ab000 tx=0x7f44e401ac50 comp rx=0 tx=0).stop 2023-10-16T20:39:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f4500fd0640 1 -- 172.21.15.88:0/3484737768 shutdown_connections 2023-10-16T20:39:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f4500fd0640 1 --2- 172.21.15.88:0/3484737768 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44dc0664f0 0x7f44dc0689b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f4500fd0640 1 --2- 172.21.15.88:0/3484737768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44fc150690 0x7f44fc0c2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f4500fd0640 1 --2- 172.21.15.88:0/3484737768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f44fc0cb5c0 0x7f44fc0c8a30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f4500fd0640 1 --2- 172.21.15.88:0/3484737768 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44fc0c66e0 0x7f44fc162de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f4500fd0640 1 -- 172.21.15.88:0/3484737768 >> 172.21.15.88:0/3484737768 conn(0x7f44fc0ca9f0 msgr2=0x7f44fc0c14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:24.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f4500fd0640 1 -- 172.21.15.88:0/3484737768 shutdown_connections 2023-10-16T20:39:24.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.717+0000 7f4500fd0640 1 -- 172.21.15.88:0/3484737768 wait complete. 2023-10-16T20:39:24.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.721+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1796124193 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9e580664a0 msgr2=0x7f9e58068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.721+0000 7f9e7c46b640 1 --2- 172.21.15.88:0/1796124193 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9e580664a0 0x7f9e58068960 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c04cc20 tx=0x7f9e740753c0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.721+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1796124193 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e6c04c580 msgr2=0x7f9e6c16fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.721+0000 7f9e7c46b640 1 --2- 172.21.15.88:0/1796124193 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e6c04c580 0x7f9e6c16fb90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9e6400e870 tx=0x7f9e6400ed30 comp rx=0 tx=0).stop 2023-10-16T20:39:24.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.721+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1796124193 shutdown_connections 2023-10-16T20:39:24.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.721+0000 7f9e7c46b640 1 --2- 172.21.15.88:0/1796124193 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9e580664a0 0x7f9e58068960 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.721+0000 7f9e7c46b640 1 --2- 172.21.15.88:0/1796124193 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e6c0e3070 0x7f9e6c0ceb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.721+0000 7f9e7c46b640 1 --2- 172.21.15.88:0/1796124193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e6c0dd300 0x7f9e6c17bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.721+0000 7f9e7c46b640 1 --2- 172.21.15.88:0/1796124193 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e6c04c580 0x7f9e6c16fb90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.721+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1796124193 >> 172.21.15.88:0/1796124193 conn(0x7f9e6c0501d0 msgr2=0x7f9e6c0e1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:24.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.721+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1796124193 shutdown_connections 2023-10-16T20:39:24.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.721+0000 7f9e7c46b640 1 -- 172.21.15.88:0/1796124193 wait complete. 2023-10-16T20:39:24.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.725+0000 7f1c71a3a640 1 -- 172.21.15.88:0/3500516803 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1c6c0c6480 con 0x7f1c6c157ba0 2023-10-16T20:39:24.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.725+0000 7f1c697fa640 1 -- 172.21.15.88:0/3500516803 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1c640efa70 con 0x7f1c6c157ba0 2023-10-16T20:39:24.734 INFO:teuthology.orchestra.run.smithi088.stdout:64424509475 2023-10-16T20:39:24.738 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509474 got 64424509475 for osd.7 2023-10-16T20:39:24.738 DEBUG:teuthology.parallel:result is None 2023-10-16T20:39:24.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.729+0000 7f1c71a3a640 1 -- 172.21.15.88:0/3500516803 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1c580664d0 msgr2=0x7f1c58068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.729+0000 7f1c71a3a640 1 --2- 172.21.15.88:0/3500516803 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1c580664d0 0x7f1c58068990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c0094a0 tx=0x7f1c5c0093f0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.729+0000 7f1c71a3a640 1 -- 172.21.15.88:0/3500516803 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c6c157ba0 msgr2=0x7f1c6c1520c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.729+0000 7f1c71a3a640 1 --2- 172.21.15.88:0/3500516803 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c6c157ba0 0x7f1c6c1520c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1c6401af50 tx=0x7f1c64019640 comp rx=0 tx=0).stop 2023-10-16T20:39:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.729+0000 7f1c71a3a640 1 -- 172.21.15.88:0/3500516803 shutdown_connections 2023-10-16T20:39:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.729+0000 7f1c71a3a640 1 --2- 172.21.15.88:0/3500516803 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1c580664d0 0x7f1c58068990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.729+0000 7f1c71a3a640 1 --2- 172.21.15.88:0/3500516803 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c6c157ba0 0x7f1c6c1520c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.729+0000 7f1c71a3a640 1 --2- 172.21.15.88:0/3500516803 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c6c0c6eb0 0x7f1c6c15b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.729+0000 7f1c71a3a640 1 --2- 172.21.15.88:0/3500516803 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c6c0c5940 0x7f1c6c15c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.729+0000 7f1c71a3a640 1 -- 172.21.15.88:0/3500516803 >> 172.21.15.88:0/3500516803 conn(0x7f1c6c0caee0 msgr2=0x7f1c6c06d510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:24.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.729+0000 7f1c71a3a640 1 -- 172.21.15.88:0/3500516803 shutdown_connections 2023-10-16T20:39:24.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.729+0000 7f1c71a3a640 1 -- 172.21.15.88:0/3500516803 wait complete. 2023-10-16T20:39:24.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.733+0000 7f3e437fe640 1 -- 172.21.15.88:0/1543275112 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f3e3c0057e0 con 0x7f3e6014ab20 2023-10-16T20:39:24.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.733+0000 7f3e55ffb640 1 -- 172.21.15.88:0/1543275112 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3e5c020020 con 0x7f3e6014ab20 2023-10-16T20:39:24.742 INFO:teuthology.orchestra.run.smithi088.stdout:64424509475 2023-10-16T20:39:24.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.737+0000 7f3e437fe640 1 -- 172.21.15.88:0/1543275112 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3e44066520 msgr2=0x7f3e440689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.737+0000 7f3e437fe640 1 --2- 172.21.15.88:0/1543275112 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3e44066520 0x7f3e440689e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3e500af000 tx=0x7f3e50008040 comp rx=0 tx=0).stop 2023-10-16T20:39:24.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.737+0000 7f3e437fe640 1 -- 172.21.15.88:0/1543275112 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e6014ab20 msgr2=0x7f3e6014da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.737+0000 7f3e437fe640 1 --2- 172.21.15.88:0/1543275112 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e6014ab20 0x7f3e6014da00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c010d20 tx=0x7f3e5c00e6e0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.737+0000 7f3e437fe640 1 -- 172.21.15.88:0/1543275112 shutdown_connections 2023-10-16T20:39:24.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.737+0000 7f3e437fe640 1 --2- 172.21.15.88:0/1543275112 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3e44066520 0x7f3e440689e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.737+0000 7f3e437fe640 1 --2- 172.21.15.88:0/1543275112 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3e60153020 0x7f3e60156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.737+0000 7f3e437fe640 1 --2- 172.21.15.88:0/1543275112 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3e6014ab20 0x7f3e6014da00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.737+0000 7f3e437fe640 1 --2- 172.21.15.88:0/1543275112 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e4c003ac0 0x7f3e601ee3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.737+0000 7f3e437fe640 1 -- 172.21.15.88:0/1543275112 >> 172.21.15.88:0/1543275112 conn(0x7f3e6006d1e0 msgr2=0x7f3e600c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:24.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.741+0000 7f3e437fe640 1 -- 172.21.15.88:0/1543275112 shutdown_connections 2023-10-16T20:39:24.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.741+0000 7f3e437fe640 1 -- 172.21.15.88:0/1543275112 wait complete. 2023-10-16T20:39:24.756 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509474 got 64424509475 for osd.2 2023-10-16T20:39:24.756 DEBUG:teuthology.parallel:result is None 2023-10-16T20:39:24.765 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574883 got 55834574883 for osd.0 2023-10-16T20:39:24.766 DEBUG:teuthology.parallel:result is None 2023-10-16T20:39:24.770 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509474 got 64424509475 for osd.4 2023-10-16T20:39:24.770 DEBUG:teuthology.parallel:result is None 2023-10-16T20:39:24.771 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509474 got 64424509475 for osd.3 2023-10-16T20:39:24.771 DEBUG:teuthology.parallel:result is None 2023-10-16T20:39:24.773 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509474 got 64424509475 for osd.1 2023-10-16T20:39:24.773 DEBUG:teuthology.parallel:result is None 2023-10-16T20:39:24.774 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509474 got 64424509475 for osd.6 2023-10-16T20:39:24.774 DEBUG:teuthology.parallel:result is None 2023-10-16T20:39:24.774 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-16T20:39:24.775 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T20:39:24.929 INFO:tasks.ceph.mds.f.smithi191.stderr:2023-10-16T20:39:24.923+0000 7fe37f078640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-16T20:39:24.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.949+0000 7facc0a2f640 1 Processor -- start 2023-10-16T20:39:24.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.949+0000 7facc0a2f640 1 -- start start 2023-10-16T20:39:24.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.949+0000 7facc0a2f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7facbc0ca2f0 0x7facbc0c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.949+0000 7facc0a2f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facbc0cae10 con 0x7facbc0c51c0 2023-10-16T20:39:24.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.949+0000 7facc0a2f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facbc0cac30 con 0x7facbc1537c0 2023-10-16T20:39:24.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.949+0000 7facc0a2f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facbc0c55d0 con 0x7facbc0ca2f0 2023-10-16T20:39:24.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.949+0000 7facbaffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7facbc0ca2f0 0x7facbc0c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.949+0000 7facbaffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7facbc0ca2f0 0x7facbc0c4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56702/0 (socket says 172.21.15.88:56702) 2023-10-16T20:39:24.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.949+0000 7facbaffd640 1 -- 172.21.15.88:0/3216661243 learned_addr learned my addr 172.21.15.88:0/3216661243 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:24.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.949+0000 7facbbfff640 1 --1- 172.21.15.88:0/3216661243 >> v1:172.21.15.88:6789/0 conn(0x7facbc0c51c0 0x7facbc0cead0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:37042/0 (socket says 172.21.15.88:37042) 2023-10-16T20:39:24.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.949+0000 7facbaffd640 1 -- 172.21.15.88:0/3216661243 >> v1:172.21.15.191:6789/0 conn(0x7facbc1537c0 legacy=0x7facbc15c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.949+0000 7facbaffd640 1 -- 172.21.15.88:0/3216661243 >> v1:172.21.15.88:6789/0 conn(0x7facbc0c51c0 legacy=0x7facbc0cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.949+0000 7facbaffd640 1 -- 172.21.15.88:0/3216661243 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc15d2b0 con 0x7facbc0ca2f0 2023-10-16T20:39:24.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.953+0000 7facbaffd640 1 --2- 172.21.15.88:0/3216661243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0ca2f0 0x7facbc0c4c80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7facb0009870 tx=0x7facb001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b435e383ab02d48d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.953+0000 7facba7fc640 1 -- 172.21.15.88:0/3216661243 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00aa670 con 0x7facbc0ca2f0 2023-10-16T20:39:24.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.953+0000 7facba7fc640 1 -- 172.21.15.88:0/3216661243 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb00aa7d0 con 0x7facbc0ca2f0 2023-10-16T20:39:24.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.953+0000 7facba7fc640 1 -- 172.21.15.88:0/3216661243 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00aab60 con 0x7facbc0ca2f0 2023-10-16T20:39:24.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.953+0000 7facc0a2f640 1 -- 172.21.15.88:0/3216661243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0ca2f0 msgr2=0x7facbc0c4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.953+0000 7facc0a2f640 1 --2- 172.21.15.88:0/3216661243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0ca2f0 0x7facbc0c4c80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7facb0009870 tx=0x7facb001c510 comp rx=0 tx=0).stop 2023-10-16T20:39:24.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.953+0000 7facc0a2f640 1 -- 172.21.15.88:0/3216661243 shutdown_connections 2023-10-16T20:39:24.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.953+0000 7facc0a2f640 1 --2- 172.21.15.88:0/3216661243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0ca2f0 0x7facbc0c4c80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.953+0000 7facc0a2f640 1 -- 172.21.15.88:0/3216661243 >> 172.21.15.88:0/3216661243 conn(0x7facbc06c170 msgr2=0x7facbc0715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:24.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.953+0000 7facc0a2f640 1 -- 172.21.15.88:0/3216661243 shutdown_connections 2023-10-16T20:39:24.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.953+0000 7facc0a2f640 1 -- 172.21.15.88:0/3216661243 wait complete. 2023-10-16T20:39:24.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.953+0000 7facc0a2f640 1 Processor -- start 2023-10-16T20:39:24.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.953+0000 7facc0a2f640 1 -- start start 2023-10-16T20:39:24.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facc0a2f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facbc0c51c0 0x7facbc151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facc0a2f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0ca2f0 0x7facbc14d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facc0a2f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7facbc1537c0 0x7facbc1f14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facc0a2f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc0cf370 con 0x7facbc0ca2f0 2023-10-16T20:39:24.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facc0a2f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc1515c0 con 0x7facbc0c51c0 2023-10-16T20:39:24.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facc0a2f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facbc0ced40 con 0x7facbc1537c0 2023-10-16T20:39:24.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facbaffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0ca2f0 0x7facbc14d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facbaffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0ca2f0 0x7facbc14d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56708/0 (socket says 172.21.15.88:56708) 2023-10-16T20:39:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facbaffd640 1 -- 172.21.15.88:0/565786219 learned_addr learned my addr 172.21.15.88:0/565786219 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facbb7fe640 1 --2- 172.21.15.88:0/565786219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facbc0c51c0 0x7facbc151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facbbfff640 1 --2- 172.21.15.88:0/565786219 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7facbc1537c0 0x7facbc1f14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facbaffd640 1 -- 172.21.15.88:0/565786219 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7facbc1537c0 msgr2=0x7facbc1f14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facbaffd640 1 --2- 172.21.15.88:0/565786219 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7facbc1537c0 0x7facbc1f14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facbaffd640 1 -- 172.21.15.88:0/565786219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facbc0c51c0 msgr2=0x7facbc151080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:24.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facbaffd640 1 --2- 172.21.15.88:0/565786219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facbc0c51c0 0x7facbc151080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:24.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facbaffd640 1 -- 172.21.15.88:0/565786219 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb00094d0 con 0x7facbc0ca2f0 2023-10-16T20:39:24.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facbb7fe640 1 --2- 172.21.15.88:0/565786219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facbc0c51c0 0x7facbc151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:24.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facbaffd640 1 --2- 172.21.15.88:0/565786219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0ca2f0 0x7facbc14d5b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7facb0019120 tx=0x7facb001ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7faca3fff640 1 -- 172.21.15.88:0/565786219 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00b38c0 con 0x7facbc0ca2f0 2023-10-16T20:39:24.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7faca3fff640 1 -- 172.21.15.88:0/565786219 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb00b3a20 con 0x7facbc0ca2f0 2023-10-16T20:39:24.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7faca3fff640 1 -- 172.21.15.88:0/565786219 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00b3dd0 con 0x7facbc0ca2f0 2023-10-16T20:39:24.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facc0a2f640 1 -- 172.21.15.88:0/565786219 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facbc1f52b0 con 0x7facbc0ca2f0 2023-10-16T20:39:24.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facc0a2f640 1 -- 172.21.15.88:0/565786219 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facbc1f5520 con 0x7facbc0ca2f0 2023-10-16T20:39:24.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7facc0a2f640 1 -- 172.21.15.88:0/565786219 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facbc1f12c0 con 0x7facbc0ca2f0 2023-10-16T20:39:24.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.957+0000 7faca3fff640 1 -- 172.21.15.88:0/565786219 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7facb00b2380 con 0x7facbc0ca2f0 2023-10-16T20:39:24.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.961+0000 7faca3fff640 1 --2- 172.21.15.88:0/565786219 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faca80664a0 0x7faca8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:24.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.961+0000 7faca3fff640 1 -- 172.21.15.88:0/565786219 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7facb011ebe0 con 0x7facbc0ca2f0 2023-10-16T20:39:24.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.961+0000 7facbb7fe640 1 --2- 172.21.15.88:0/565786219 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faca80664a0 0x7faca8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:24.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.961+0000 7facbb7fe640 1 --2- 172.21.15.88:0/565786219 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faca80664a0 0x7faca8068960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7facac000f70 tx=0x7facac009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:24.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:24.965+0000 7faca3fff640 1 -- 172.21.15.88:0/565786219 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7facb00ebcb0 con 0x7facbc0ca2f0 2023-10-16T20:39:25.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.109+0000 7facc0a2f640 1 -- 172.21.15.88:0/565786219 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7facbc0c4480 con 0x7faca80664a0 2023-10-16T20:39:25.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.121+0000 7faca3fff640 1 -- 172.21.15.88:0/565786219 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+369837 (secure 0 0 0) 0x7facac001ca0 con 0x7faca80664a0 2023-10-16T20:39:25.130 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:25.132 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-16T20:39:25.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.125+0000 7facc0a2f640 1 -- 172.21.15.88:0/565786219 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faca80664a0 msgr2=0x7faca8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:25.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.125+0000 7facc0a2f640 1 --2- 172.21.15.88:0/565786219 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faca80664a0 0x7faca8068960 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7facac000f70 tx=0x7facac009040 comp rx=0 tx=0).stop 2023-10-16T20:39:25.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.125+0000 7facc0a2f640 1 -- 172.21.15.88:0/565786219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0ca2f0 msgr2=0x7facbc14d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:25.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.125+0000 7facc0a2f640 1 --2- 172.21.15.88:0/565786219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0ca2f0 0x7facbc14d5b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7facb0019120 tx=0x7facb001ad50 comp rx=0 tx=0).stop 2023-10-16T20:39:25.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.125+0000 7facc0a2f640 1 -- 172.21.15.88:0/565786219 shutdown_connections 2023-10-16T20:39:25.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.125+0000 7facc0a2f640 1 --2- 172.21.15.88:0/565786219 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faca80664a0 0x7faca8068960 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.125+0000 7facc0a2f640 1 --2- 172.21.15.88:0/565786219 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7facbc1537c0 0x7facbc1f14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.125+0000 7facc0a2f640 1 --2- 172.21.15.88:0/565786219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facbc0ca2f0 0x7facbc14d5b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.125+0000 7facc0a2f640 1 --2- 172.21.15.88:0/565786219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facbc0c51c0 0x7facbc151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.125+0000 7facc0a2f640 1 -- 172.21.15.88:0/565786219 >> 172.21.15.88:0/565786219 conn(0x7facbc06c170 msgr2=0x7facbc152840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:25.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.125+0000 7facc0a2f640 1 -- 172.21.15.88:0/565786219 shutdown_connections 2023-10-16T20:39:25.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.125+0000 7facc0a2f640 1 -- 172.21.15.88:0/565786219 wait complete. 2023-10-16T20:39:25.150 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2023-10-16T20:39:23.628355+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595825,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":243,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":191,"ondisk_log_size":191,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219788,"kb_used_data":3628,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754932,"statfs":{"total":773094113280,"available":772869050368,"internally_reserved":0,"allocated":3715072,"data_stored":2184008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3047,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002417"},"pg_stats":[{"pgid":"2.1f","version":"26'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435615+0000","last_change":"2023-10-16T20:36:54.763744+0000","last_active":"2023-10-16T20:37:37.435615+0000","last_peered":"2023-10-16T20:37:37.435615+0000","last_clean":"2023-10-16T20:37:37.435615+0000","last_became_active":"2023-10-16T20:36:54.763457+0000","last_became_peered":"2023-10-16T20:36:54.763457+0000","last_unstale":"2023-10-16T20:37:37.435615+0000","last_undegraded":"2023-10-16T20:37:37.435615+0000","last_fullsized":"2023-10-16T20:37:37.435615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:05.774485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465462+0000","last_change":"2023-10-16T20:36:55.302740+0000","last_active":"2023-10-16T20:37:37.465462+0000","last_peered":"2023-10-16T20:37:37.465462+0000","last_clean":"2023-10-16T20:37:37.465462+0000","last_became_active":"2023-10-16T20:36:55.302519+0000","last_became_peered":"2023-10-16T20:36:55.302519+0000","last_unstale":"2023-10-16T20:37:37.465462+0000","last_undegraded":"2023-10-16T20:37:37.465462+0000","last_fullsized":"2023-10-16T20:37:37.465462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:36.762521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404215+0000","last_change":"2023-10-16T20:36:59.466679+0000","last_active":"2023-10-16T20:37:36.404215+0000","last_peered":"2023-10-16T20:37:36.404215+0000","last_clean":"2023-10-16T20:37:36.404215+0000","last_became_active":"2023-10-16T20:36:59.466306+0000","last_became_peered":"2023-10-16T20:36:59.466306+0000","last_unstale":"2023-10-16T20:37:36.404215+0000","last_undegraded":"2023-10-16T20:37:36.404215+0000","last_fullsized":"2023-10-16T20:37:36.404215+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:14.874622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"26'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465585+0000","last_change":"2023-10-16T20:36:54.759491+0000","last_active":"2023-10-16T20:37:37.465585+0000","last_peered":"2023-10-16T20:37:37.465585+0000","last_clean":"2023-10-16T20:37:37.465585+0000","last_became_active":"2023-10-16T20:36:54.759177+0000","last_became_peered":"2023-10-16T20:36:54.759177+0000","last_unstale":"2023-10-16T20:37:37.465585+0000","last_undegraded":"2023-10-16T20:37:37.465585+0000","last_fullsized":"2023-10-16T20:37:37.465585+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:43:36.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435464+0000","last_change":"2023-10-16T20:36:55.291745+0000","last_active":"2023-10-16T20:37:37.435464+0000","last_peered":"2023-10-16T20:37:37.435464+0000","last_clean":"2023-10-16T20:37:37.435464+0000","last_became_active":"2023-10-16T20:36:55.291458+0000","last_became_peered":"2023-10-16T20:36:55.291458+0000","last_unstale":"2023-10-16T20:37:37.435464+0000","last_undegraded":"2023-10-16T20:37:37.435464+0000","last_fullsized":"2023-10-16T20:37:37.435464+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:01.291335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.401937+0000","last_change":"2023-10-16T20:36:59.303600+0000","last_active":"2023-10-16T20:37:36.401937+0000","last_peered":"2023-10-16T20:37:36.401937+0000","last_clean":"2023-10-16T20:37:36.401937+0000","last_became_active":"2023-10-16T20:36:59.302968+0000","last_became_peered":"2023-10-16T20:36:59.302968+0000","last_unstale":"2023-10-16T20:37:36.401937+0000","last_undegraded":"2023-10-16T20:37:36.401937+0000","last_fullsized":"2023-10-16T20:37:36.401937+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:11:28.322824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"27'2","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.412675+0000","last_change":"2023-10-16T20:36:54.762600+0000","last_active":"2023-10-16T20:37:36.412675+0000","last_peered":"2023-10-16T20:37:36.412675+0000","last_clean":"2023-10-16T20:37:36.412675+0000","last_became_active":"2023-10-16T20:36:54.762278+0000","last_became_peered":"2023-10-16T20:36:54.762278+0000","last_unstale":"2023-10-16T20:37:36.412675+0000","last_undegraded":"2023-10-16T20:37:36.412675+0000","last_fullsized":"2023-10-16T20:37:36.412675+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:46:03.836415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404637+0000","last_change":"2023-10-16T20:37:37.403448+0000","last_active":"2023-10-16T20:37:37.404637+0000","last_peered":"2023-10-16T20:37:37.404637+0000","last_clean":"2023-10-16T20:37:37.404637+0000","last_became_active":"2023-10-16T20:37:37.402987+0000","last_became_peered":"2023-10-16T20:37:37.402987+0000","last_unstale":"2023-10-16T20:37:37.404637+0000","last_undegraded":"2023-10-16T20:37:37.404637+0000","last_fullsized":"2023-10-16T20:37:37.404637+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:39:05.729682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.402113+0000","last_change":"2023-10-16T20:36:59.304684+0000","last_active":"2023-10-16T20:37:36.402113+0000","last_peered":"2023-10-16T20:37:36.402113+0000","last_clean":"2023-10-16T20:37:36.402113+0000","last_became_active":"2023-10-16T20:36:59.303805+0000","last_became_peered":"2023-10-16T20:36:59.303805+0000","last_unstale":"2023-10-16T20:37:36.402113+0000","last_undegraded":"2023-10-16T20:37:36.402113+0000","last_fullsized":"2023-10-16T20:37:36.402113+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:39:54.406763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"27'2","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.412121+0000","last_change":"2023-10-16T20:36:54.749849+0000","last_active":"2023-10-16T20:37:36.412121+0000","last_peered":"2023-10-16T20:37:36.412121+0000","last_clean":"2023-10-16T20:37:36.412121+0000","last_became_active":"2023-10-16T20:36:54.749316+0000","last_became_peered":"2023-10-16T20:36:54.749316+0000","last_unstale":"2023-10-16T20:37:36.412121+0000","last_undegraded":"2023-10-16T20:37:36.412121+0000","last_fullsized":"2023-10-16T20:37:36.412121+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:49:29.077126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.409911+0000","last_change":"2023-10-16T20:36:55.302838+0000","last_active":"2023-10-16T20:37:36.409911+0000","last_peered":"2023-10-16T20:37:36.409911+0000","last_clean":"2023-10-16T20:37:36.409911+0000","last_became_active":"2023-10-16T20:36:55.302387+0000","last_became_peered":"2023-10-16T20:36:55.302387+0000","last_unstale":"2023-10-16T20:37:36.409911+0000","last_undegraded":"2023-10-16T20:37:36.409911+0000","last_fullsized":"2023-10-16T20:37:36.409911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:55:56.517950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.402135+0000","last_change":"2023-10-16T20:36:59.305750+0000","last_active":"2023-10-16T20:37:36.402135+0000","last_peered":"2023-10-16T20:37:36.402135+0000","last_clean":"2023-10-16T20:37:36.402135+0000","last_became_active":"2023-10-16T20:36:59.305228+0000","last_became_peered":"2023-10-16T20:36:59.305228+0000","last_unstale":"2023-10-16T20:37:36.402135+0000","last_undegraded":"2023-10-16T20:37:36.402135+0000","last_fullsized":"2023-10-16T20:37:36.402135+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:00:18.194057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407634+0000","last_change":"2023-10-16T20:36:55.306712+0000","last_active":"2023-10-16T20:37:37.407634+0000","last_peered":"2023-10-16T20:37:37.407634+0000","last_clean":"2023-10-16T20:37:37.407634+0000","last_became_active":"2023-10-16T20:36:55.306427+0000","last_became_peered":"2023-10-16T20:36:55.306427+0000","last_unstale":"2023-10-16T20:37:37.407634+0000","last_undegraded":"2023-10-16T20:37:37.407634+0000","last_fullsized":"2023-10-16T20:37:37.407634+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:42:03.769890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.410476+0000","last_change":"2023-10-16T20:36:54.768375+0000","last_active":"2023-10-16T20:39:22.410476+0000","last_peered":"2023-10-16T20:39:22.410476+0000","last_clean":"2023-10-16T20:39:22.410476+0000","last_became_active":"2023-10-16T20:36:54.767996+0000","last_became_peered":"2023-10-16T20:36:54.767996+0000","last_unstale":"2023-10-16T20:39:22.410476+0000","last_undegraded":"2023-10-16T20:39:22.410476+0000","last_fullsized":"2023-10-16T20:39:22.410476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:55.303942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404253+0000","last_change":"2023-10-16T20:36:59.307130+0000","last_active":"2023-10-16T20:37:37.404253+0000","last_peered":"2023-10-16T20:37:37.404253+0000","last_clean":"2023-10-16T20:37:37.404253+0000","last_became_active":"2023-10-16T20:36:59.306640+0000","last_became_peered":"2023-10-16T20:36:59.306640+0000","last_unstale":"2023-10-16T20:37:37.404253+0000","last_undegraded":"2023-10-16T20:37:37.404253+0000","last_fullsized":"2023-10-16T20:37:37.404253+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:19:18.664630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465074+0000","last_change":"2023-10-16T20:36:55.275436+0000","last_active":"2023-10-16T20:37:37.465074+0000","last_peered":"2023-10-16T20:37:37.465074+0000","last_clean":"2023-10-16T20:37:37.465074+0000","last_became_active":"2023-10-16T20:36:55.275085+0000","last_became_peered":"2023-10-16T20:36:55.275085+0000","last_unstale":"2023-10-16T20:37:37.465074+0000","last_undegraded":"2023-10-16T20:37:37.465074+0000","last_fullsized":"2023-10-16T20:37:37.465074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:37:37.814511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402406+0000","last_change":"2023-10-16T20:36:54.756174+0000","last_active":"2023-10-16T20:37:37.402406+0000","last_peered":"2023-10-16T20:37:37.402406+0000","last_clean":"2023-10-16T20:37:37.402406+0000","last_became_active":"2023-10-16T20:36:54.755872+0000","last_became_peered":"2023-10-16T20:36:54.755872+0000","last_unstale":"2023-10-16T20:37:37.402406+0000","last_undegraded":"2023-10-16T20:37:37.402406+0000","last_fullsized":"2023-10-16T20:37:37.402406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:32.741671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403750+0000","last_change":"2023-10-16T20:36:59.303855+0000","last_active":"2023-10-16T20:37:37.403750+0000","last_peered":"2023-10-16T20:37:37.403750+0000","last_clean":"2023-10-16T20:37:37.403750+0000","last_became_active":"2023-10-16T20:36:59.302919+0000","last_became_peered":"2023-10-16T20:36:59.302919+0000","last_unstale":"2023-10-16T20:37:37.403750+0000","last_undegraded":"2023-10-16T20:37:37.403750+0000","last_fullsized":"2023-10-16T20:37:37.403750+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:38:02.591237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464820+0000","last_change":"2023-10-16T20:36:55.270717+0000","last_active":"2023-10-16T20:37:37.464820+0000","last_peered":"2023-10-16T20:37:37.464820+0000","last_clean":"2023-10-16T20:37:37.464820+0000","last_became_active":"2023-10-16T20:36:55.270455+0000","last_became_peered":"2023-10-16T20:36:55.270455+0000","last_unstale":"2023-10-16T20:37:37.464820+0000","last_undegraded":"2023-10-16T20:37:37.464820+0000","last_fullsized":"2023-10-16T20:37:37.464820+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:16:44.485783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.406832+0000","last_change":"2023-10-16T20:36:54.744128+0000","last_active":"2023-10-16T20:39:22.406832+0000","last_peered":"2023-10-16T20:39:22.406832+0000","last_clean":"2023-10-16T20:39:22.406832+0000","last_became_active":"2023-10-16T20:36:54.743805+0000","last_became_peered":"2023-10-16T20:36:54.743805+0000","last_unstale":"2023-10-16T20:39:22.406832+0000","last_undegraded":"2023-10-16T20:39:22.406832+0000","last_fullsized":"2023-10-16T20:39:22.406832+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:54:35.819820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402668+0000","last_change":"2023-10-16T20:36:59.465650+0000","last_active":"2023-10-16T20:37:37.402668+0000","last_peered":"2023-10-16T20:37:37.402668+0000","last_clean":"2023-10-16T20:37:37.402668+0000","last_became_active":"2023-10-16T20:36:59.465172+0000","last_became_peered":"2023-10-16T20:36:59.465172+0000","last_unstale":"2023-10-16T20:37:37.402668+0000","last_undegraded":"2023-10-16T20:37:37.402668+0000","last_fullsized":"2023-10-16T20:37:37.402668+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:12.845517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403561+0000","last_change":"2023-10-16T20:36:55.278974+0000","last_active":"2023-10-16T20:37:37.403561+0000","last_peered":"2023-10-16T20:37:37.403561+0000","last_clean":"2023-10-16T20:37:37.403561+0000","last_became_active":"2023-10-16T20:36:55.278568+0000","last_became_peered":"2023-10-16T20:36:55.278568+0000","last_unstale":"2023-10-16T20:37:37.403561+0000","last_undegraded":"2023-10-16T20:37:37.403561+0000","last_fullsized":"2023-10-16T20:37:37.403561+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:26.756114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.407659+0000","last_change":"2023-10-16T20:36:54.758778+0000","last_active":"2023-10-16T20:39:22.407659+0000","last_peered":"2023-10-16T20:39:22.407659+0000","last_clean":"2023-10-16T20:39:22.407659+0000","last_became_active":"2023-10-16T20:36:54.758496+0000","last_became_peered":"2023-10-16T20:36:54.758496+0000","last_unstale":"2023-10-16T20:39:22.407659+0000","last_undegraded":"2023-10-16T20:39:22.407659+0000","last_fullsized":"2023-10-16T20:39:22.407659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:36:00.515106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404277+0000","last_change":"2023-10-16T20:36:59.309382+0000","last_active":"2023-10-16T20:37:37.404277+0000","last_peered":"2023-10-16T20:37:37.404277+0000","last_clean":"2023-10-16T20:37:37.404277+0000","last_became_active":"2023-10-16T20:36:59.309068+0000","last_became_peered":"2023-10-16T20:36:59.309068+0000","last_unstale":"2023-10-16T20:37:37.404277+0000","last_undegraded":"2023-10-16T20:37:37.404277+0000","last_fullsized":"2023-10-16T20:37:37.404277+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:46:02.601985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435429+0000","last_change":"2023-10-16T20:36:55.273089+0000","last_active":"2023-10-16T20:37:37.435429+0000","last_peered":"2023-10-16T20:37:37.435429+0000","last_clean":"2023-10-16T20:37:37.435429+0000","last_became_active":"2023-10-16T20:36:55.272840+0000","last_became_peered":"2023-10-16T20:36:55.272840+0000","last_unstale":"2023-10-16T20:37:37.435429+0000","last_undegraded":"2023-10-16T20:37:37.435429+0000","last_fullsized":"2023-10-16T20:37:37.435429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:37.330924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.409085+0000","last_change":"2023-10-16T20:36:54.760681+0000","last_active":"2023-10-16T20:39:22.409085+0000","last_peered":"2023-10-16T20:39:22.409085+0000","last_clean":"2023-10-16T20:39:22.409085+0000","last_became_active":"2023-10-16T20:36:54.760397+0000","last_became_peered":"2023-10-16T20:36:54.760397+0000","last_unstale":"2023-10-16T20:39:22.409085+0000","last_undegraded":"2023-10-16T20:39:22.409085+0000","last_fullsized":"2023-10-16T20:39:22.409085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:47:16.510382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403318+0000","last_change":"2023-10-16T20:36:59.465626+0000","last_active":"2023-10-16T20:37:37.403318+0000","last_peered":"2023-10-16T20:37:37.403318+0000","last_clean":"2023-10-16T20:37:37.403318+0000","last_became_active":"2023-10-16T20:36:59.465152+0000","last_became_peered":"2023-10-16T20:36:59.465152+0000","last_unstale":"2023-10-16T20:37:37.403318+0000","last_undegraded":"2023-10-16T20:37:37.403318+0000","last_fullsized":"2023-10-16T20:37:37.403318+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:27:51.088333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465673+0000","last_change":"2023-10-16T20:36:55.285796+0000","last_active":"2023-10-16T20:37:37.465673+0000","last_peered":"2023-10-16T20:37:37.465673+0000","last_clean":"2023-10-16T20:37:37.465673+0000","last_became_active":"2023-10-16T20:36:55.285566+0000","last_became_peered":"2023-10-16T20:36:55.285566+0000","last_unstale":"2023-10-16T20:37:37.465673+0000","last_undegraded":"2023-10-16T20:37:37.465673+0000","last_fullsized":"2023-10-16T20:37:37.465673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:22:21.950705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"27'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402756+0000","last_change":"2023-10-16T20:36:54.758062+0000","last_active":"2023-10-16T20:37:37.402756+0000","last_peered":"2023-10-16T20:37:37.402756+0000","last_clean":"2023-10-16T20:37:37.402756+0000","last_became_active":"2023-10-16T20:36:54.757796+0000","last_became_peered":"2023-10-16T20:36:54.757796+0000","last_unstale":"2023-10-16T20:37:37.402756+0000","last_undegraded":"2023-10-16T20:37:37.402756+0000","last_fullsized":"2023-10-16T20:37:37.402756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:02:09.953795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464463+0000","last_change":"2023-10-16T20:36:59.468529+0000","last_active":"2023-10-16T20:37:37.464463+0000","last_peered":"2023-10-16T20:37:37.464463+0000","last_clean":"2023-10-16T20:37:37.464463+0000","last_became_active":"2023-10-16T20:36:59.465221+0000","last_became_peered":"2023-10-16T20:36:59.465221+0000","last_unstale":"2023-10-16T20:37:37.464463+0000","last_undegraded":"2023-10-16T20:37:37.464463+0000","last_fullsized":"2023-10-16T20:37:37.464463+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:58:08.457686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401257+0000","last_change":"2023-10-16T20:36:55.275759+0000","last_active":"2023-10-16T20:37:37.401257+0000","last_peered":"2023-10-16T20:37:37.401257+0000","last_clean":"2023-10-16T20:37:37.401257+0000","last_became_active":"2023-10-16T20:36:55.274432+0000","last_became_peered":"2023-10-16T20:36:55.274432+0000","last_unstale":"2023-10-16T20:37:37.401257+0000","last_undegraded":"2023-10-16T20:37:37.401257+0000","last_fullsized":"2023-10-16T20:37:37.401257+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:53:08.896731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:02.630911+0000","last_change":"2023-10-16T20:36:54.762356+0000","last_active":"2023-10-16T20:39:02.630911+0000","last_peered":"2023-10-16T20:39:02.630911+0000","last_clean":"2023-10-16T20:39:02.630911+0000","last_became_active":"2023-10-16T20:36:54.762167+0000","last_became_peered":"2023-10-16T20:36:54.762167+0000","last_unstale":"2023-10-16T20:39:02.630911+0000","last_undegraded":"2023-10-16T20:39:02.630911+0000","last_fullsized":"2023-10-16T20:39:02.630911+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:45.164625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434946+0000","last_change":"2023-10-16T20:36:59.467492+0000","last_active":"2023-10-16T20:37:37.434946+0000","last_peered":"2023-10-16T20:37:37.434946+0000","last_clean":"2023-10-16T20:37:37.434946+0000","last_became_active":"2023-10-16T20:36:59.466876+0000","last_became_peered":"2023-10-16T20:36:59.466876+0000","last_unstale":"2023-10-16T20:37:37.434946+0000","last_undegraded":"2023-10-16T20:37:37.434946+0000","last_fullsized":"2023-10-16T20:37:37.434946+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:29:01.292167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409570+0000","last_change":"2023-10-16T20:36:55.293537+0000","last_active":"2023-10-16T20:37:37.409570+0000","last_peered":"2023-10-16T20:37:37.409570+0000","last_clean":"2023-10-16T20:37:37.409570+0000","last_became_active":"2023-10-16T20:36:55.293228+0000","last_became_peered":"2023-10-16T20:36:55.293228+0000","last_unstale":"2023-10-16T20:37:37.409570+0000","last_undegraded":"2023-10-16T20:37:37.409570+0000","last_fullsized":"2023-10-16T20:37:37.409570+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:50:13.226645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401381+0000","last_change":"2023-10-16T20:36:54.757588+0000","last_active":"2023-10-16T20:37:37.401381+0000","last_peered":"2023-10-16T20:37:37.401381+0000","last_clean":"2023-10-16T20:37:37.401381+0000","last_became_active":"2023-10-16T20:36:54.757308+0000","last_became_peered":"2023-10-16T20:36:54.757308+0000","last_unstale":"2023-10-16T20:37:37.401381+0000","last_undegraded":"2023-10-16T20:37:37.401381+0000","last_fullsized":"2023-10-16T20:37:37.401381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:51.298239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402099+0000","last_change":"2023-10-16T20:36:59.303844+0000","last_active":"2023-10-16T20:37:37.402099+0000","last_peered":"2023-10-16T20:37:37.402099+0000","last_clean":"2023-10-16T20:37:37.402099+0000","last_became_active":"2023-10-16T20:36:59.303224+0000","last_became_peered":"2023-10-16T20:36:59.303224+0000","last_unstale":"2023-10-16T20:37:37.402099+0000","last_undegraded":"2023-10-16T20:37:37.402099+0000","last_fullsized":"2023-10-16T20:37:37.402099+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:52.447785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403798+0000","last_change":"2023-10-16T20:36:55.278310+0000","last_active":"2023-10-16T20:37:37.403798+0000","last_peered":"2023-10-16T20:37:37.403798+0000","last_clean":"2023-10-16T20:37:37.403798+0000","last_became_active":"2023-10-16T20:36:55.277872+0000","last_became_peered":"2023-10-16T20:36:55.277872+0000","last_unstale":"2023-10-16T20:37:37.403798+0000","last_undegraded":"2023-10-16T20:37:37.403798+0000","last_fullsized":"2023-10-16T20:37:37.403798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:37.320469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408898+0000","last_change":"2023-10-16T20:36:54.740644+0000","last_active":"2023-10-16T20:37:37.408898+0000","last_peered":"2023-10-16T20:37:37.408898+0000","last_clean":"2023-10-16T20:37:37.408898+0000","last_became_active":"2023-10-16T20:36:54.740340+0000","last_became_peered":"2023-10-16T20:36:54.740340+0000","last_unstale":"2023-10-16T20:37:37.408898+0000","last_undegraded":"2023-10-16T20:37:37.408898+0000","last_fullsized":"2023-10-16T20:37:37.408898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:22.221727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401155+0000","last_change":"2023-10-16T20:36:59.294914+0000","last_active":"2023-10-16T20:37:37.401155+0000","last_peered":"2023-10-16T20:37:37.401155+0000","last_clean":"2023-10-16T20:37:37.401155+0000","last_became_active":"2023-10-16T20:36:59.294548+0000","last_became_peered":"2023-10-16T20:36:59.294548+0000","last_unstale":"2023-10-16T20:37:37.401155+0000","last_undegraded":"2023-10-16T20:37:37.401155+0000","last_fullsized":"2023-10-16T20:37:37.401155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:52:15.265262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401383+0000","last_change":"2023-10-16T20:36:55.270992+0000","last_active":"2023-10-16T20:37:37.401383+0000","last_peered":"2023-10-16T20:37:37.401383+0000","last_clean":"2023-10-16T20:37:37.401383+0000","last_became_active":"2023-10-16T20:36:55.269749+0000","last_became_peered":"2023-10-16T20:36:55.269749+0000","last_unstale":"2023-10-16T20:37:37.401383+0000","last_undegraded":"2023-10-16T20:37:37.401383+0000","last_fullsized":"2023-10-16T20:37:37.401383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:30.925314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:02.622407+0000","last_change":"2023-10-16T20:36:54.757903+0000","last_active":"2023-10-16T20:39:02.622407+0000","last_peered":"2023-10-16T20:39:02.622407+0000","last_clean":"2023-10-16T20:39:02.622407+0000","last_became_active":"2023-10-16T20:36:54.757549+0000","last_became_peered":"2023-10-16T20:36:54.757549+0000","last_unstale":"2023-10-16T20:39:02.622407+0000","last_undegraded":"2023-10-16T20:39:02.622407+0000","last_fullsized":"2023-10-16T20:39:02.622407+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:02:42.090848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402786+0000","last_change":"2023-10-16T20:36:59.302063+0000","last_active":"2023-10-16T20:37:37.402786+0000","last_peered":"2023-10-16T20:37:37.402786+0000","last_clean":"2023-10-16T20:37:37.402786+0000","last_became_active":"2023-10-16T20:36:59.301693+0000","last_became_peered":"2023-10-16T20:36:59.301693+0000","last_unstale":"2023-10-16T20:37:37.402786+0000","last_undegraded":"2023-10-16T20:37:37.402786+0000","last_fullsized":"2023-10-16T20:37:37.402786+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:29:44.812282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434713+0000","last_change":"2023-10-16T20:36:55.278132+0000","last_active":"2023-10-16T20:37:37.434713+0000","last_peered":"2023-10-16T20:37:37.434713+0000","last_clean":"2023-10-16T20:37:37.434713+0000","last_became_active":"2023-10-16T20:36:55.276649+0000","last_became_peered":"2023-10-16T20:36:55.276649+0000","last_unstale":"2023-10-16T20:37:37.434713+0000","last_undegraded":"2023-10-16T20:37:37.434713+0000","last_fullsized":"2023-10-16T20:37:37.434713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:30:53.299514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:38:57.630200+0000","last_change":"2023-10-16T20:36:54.753592+0000","last_active":"2023-10-16T20:38:57.630200+0000","last_peered":"2023-10-16T20:38:57.630200+0000","last_clean":"2023-10-16T20:38:57.630200+0000","last_became_active":"2023-10-16T20:36:54.753237+0000","last_became_peered":"2023-10-16T20:36:54.753237+0000","last_unstale":"2023-10-16T20:38:57.630200+0000","last_undegraded":"2023-10-16T20:38:57.630200+0000","last_fullsized":"2023-10-16T20:38:57.630200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:53.005979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435360+0000","last_change":"2023-10-16T20:36:59.468431+0000","last_active":"2023-10-16T20:37:37.435360+0000","last_peered":"2023-10-16T20:37:37.435360+0000","last_clean":"2023-10-16T20:37:37.435360+0000","last_became_active":"2023-10-16T20:36:59.466708+0000","last_became_peered":"2023-10-16T20:36:59.466708+0000","last_unstale":"2023-10-16T20:37:37.435360+0000","last_undegraded":"2023-10-16T20:37:37.435360+0000","last_fullsized":"2023-10-16T20:37:37.435360+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:42:20.607519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464817+0000","last_change":"2023-10-16T20:36:55.294825+0000","last_active":"2023-10-16T20:37:37.464817+0000","last_peered":"2023-10-16T20:37:37.464817+0000","last_clean":"2023-10-16T20:37:37.464817+0000","last_became_active":"2023-10-16T20:36:55.294610+0000","last_became_peered":"2023-10-16T20:36:55.294610+0000","last_unstale":"2023-10-16T20:37:37.464817+0000","last_undegraded":"2023-10-16T20:37:37.464817+0000","last_fullsized":"2023-10-16T20:37:37.464817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:43.442939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.410375+0000","last_change":"2023-10-16T20:36:54.741191+0000","last_active":"2023-10-16T20:39:22.410375+0000","last_peered":"2023-10-16T20:39:22.410375+0000","last_clean":"2023-10-16T20:39:22.410375+0000","last_became_active":"2023-10-16T20:36:54.740944+0000","last_became_peered":"2023-10-16T20:36:54.740944+0000","last_unstale":"2023-10-16T20:39:22.410375+0000","last_undegraded":"2023-10-16T20:39:22.410375+0000","last_fullsized":"2023-10-16T20:39:22.410375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:05.961850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":172,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466038+0000","last_change":"2023-10-16T20:36:46.390001+0000","last_active":"2023-10-16T20:37:37.466038+0000","last_peered":"2023-10-16T20:37:37.466038+0000","last_clean":"2023-10-16T20:37:37.466038+0000","last_became_active":"2023-10-16T20:36:46.388824+0000","last_became_peered":"2023-10-16T20:36:46.388824+0000","last_unstale":"2023-10-16T20:37:37.466038+0000","last_undegraded":"2023-10-16T20:37:37.466038+0000","last_fullsized":"2023-10-16T20:37:37.466038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_clean_scrub_stamp":"2023-10-16T20:36:44.575506+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:31.004318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407653+0000","last_change":"2023-10-16T20:36:59.468528+0000","last_active":"2023-10-16T20:37:37.407653+0000","last_peered":"2023-10-16T20:37:37.407653+0000","last_clean":"2023-10-16T20:37:37.407653+0000","last_became_active":"2023-10-16T20:36:59.468082+0000","last_became_peered":"2023-10-16T20:36:59.468082+0000","last_unstale":"2023-10-16T20:37:37.407653+0000","last_undegraded":"2023-10-16T20:37:37.407653+0000","last_fullsized":"2023-10-16T20:37:37.407653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:00.920004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410690+0000","last_change":"2023-10-16T20:37:37.409578+0000","last_active":"2023-10-16T20:37:37.410690+0000","last_peered":"2023-10-16T20:37:37.410690+0000","last_clean":"2023-10-16T20:37:37.410690+0000","last_became_active":"2023-10-16T20:37:37.409004+0000","last_became_peered":"2023-10-16T20:37:37.409004+0000","last_unstale":"2023-10-16T20:37:37.410690+0000","last_undegraded":"2023-10-16T20:37:37.410690+0000","last_fullsized":"2023-10-16T20:37:37.410690+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:28:57.364794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.400188+0000","last_change":"2023-10-16T20:36:54.746600+0000","last_active":"2023-10-16T20:37:37.400188+0000","last_peered":"2023-10-16T20:37:37.400188+0000","last_clean":"2023-10-16T20:37:37.400188+0000","last_became_active":"2023-10-16T20:36:54.746247+0000","last_became_peered":"2023-10-16T20:36:54.746247+0000","last_unstale":"2023-10-16T20:37:37.400188+0000","last_undegraded":"2023-10-16T20:37:37.400188+0000","last_fullsized":"2023-10-16T20:37:37.400188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:28:57.949487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.470531+0000","last_change":"2023-10-16T20:37:37.469427+0000","last_active":"2023-10-16T20:37:37.470531+0000","last_peered":"2023-10-16T20:37:37.470531+0000","last_clean":"2023-10-16T20:37:37.470531+0000","last_became_active":"2023-10-16T20:37:37.467402+0000","last_became_peered":"2023-10-16T20:37:37.467402+0000","last_unstale":"2023-10-16T20:37:37.470531+0000","last_undegraded":"2023-10-16T20:37:37.470531+0000","last_fullsized":"2023-10-16T20:37:37.470531+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:00:19.171972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409113+0000","last_change":"2023-10-16T20:36:55.301234+0000","last_active":"2023-10-16T20:37:37.409113+0000","last_peered":"2023-10-16T20:37:37.409113+0000","last_clean":"2023-10-16T20:37:37.409113+0000","last_became_active":"2023-10-16T20:36:55.300958+0000","last_became_peered":"2023-10-16T20:36:55.300958+0000","last_unstale":"2023-10-16T20:37:37.409113+0000","last_undegraded":"2023-10-16T20:37:37.409113+0000","last_fullsized":"2023-10-16T20:37:37.409113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:16.784124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"27'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402899+0000","last_change":"2023-10-16T20:36:54.753770+0000","last_active":"2023-10-16T20:37:37.402899+0000","last_peered":"2023-10-16T20:37:37.402899+0000","last_clean":"2023-10-16T20:37:37.402899+0000","last_became_active":"2023-10-16T20:36:54.753443+0000","last_became_peered":"2023-10-16T20:36:54.753443+0000","last_unstale":"2023-10-16T20:37:37.402899+0000","last_undegraded":"2023-10-16T20:37:37.402899+0000","last_fullsized":"2023-10-16T20:37:37.402899+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:21:26.392480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435046+0000","last_change":"2023-10-16T20:36:59.303351+0000","last_active":"2023-10-16T20:37:37.435046+0000","last_peered":"2023-10-16T20:37:37.435046+0000","last_clean":"2023-10-16T20:37:37.435046+0000","last_became_active":"2023-10-16T20:36:59.302869+0000","last_became_peered":"2023-10-16T20:36:59.302869+0000","last_unstale":"2023-10-16T20:37:37.435046+0000","last_undegraded":"2023-10-16T20:37:37.435046+0000","last_fullsized":"2023-10-16T20:37:37.435046+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:26:35.886399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466830+0000","last_change":"2023-10-16T20:36:55.291947+0000","last_active":"2023-10-16T20:37:37.466830+0000","last_peered":"2023-10-16T20:37:37.466830+0000","last_clean":"2023-10-16T20:37:37.466830+0000","last_became_active":"2023-10-16T20:36:55.291633+0000","last_became_peered":"2023-10-16T20:36:55.291633+0000","last_unstale":"2023-10-16T20:37:37.466830+0000","last_undegraded":"2023-10-16T20:37:37.466830+0000","last_fullsized":"2023-10-16T20:37:37.466830+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:00:14.349339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"27'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.400583+0000","last_change":"2023-10-16T20:36:54.748196+0000","last_active":"2023-10-16T20:37:37.400583+0000","last_peered":"2023-10-16T20:37:37.400583+0000","last_clean":"2023-10-16T20:37:37.400583+0000","last_became_active":"2023-10-16T20:36:54.747704+0000","last_became_peered":"2023-10-16T20:36:54.747704+0000","last_unstale":"2023-10-16T20:37:37.400583+0000","last_undegraded":"2023-10-16T20:37:37.400583+0000","last_fullsized":"2023-10-16T20:37:37.400583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:55.471916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410236+0000","last_change":"2023-10-16T20:36:59.470368+0000","last_active":"2023-10-16T20:37:37.410236+0000","last_peered":"2023-10-16T20:37:37.410236+0000","last_clean":"2023-10-16T20:37:37.410236+0000","last_became_active":"2023-10-16T20:36:59.467291+0000","last_became_peered":"2023-10-16T20:36:59.467291+0000","last_unstale":"2023-10-16T20:37:37.410236+0000","last_undegraded":"2023-10-16T20:37:37.410236+0000","last_fullsized":"2023-10-16T20:37:37.410236+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:18.312394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466658+0000","last_change":"2023-10-16T20:36:55.291451+0000","last_active":"2023-10-16T20:37:37.466658+0000","last_peered":"2023-10-16T20:37:37.466658+0000","last_clean":"2023-10-16T20:37:37.466658+0000","last_became_active":"2023-10-16T20:36:55.290998+0000","last_became_peered":"2023-10-16T20:36:55.290998+0000","last_unstale":"2023-10-16T20:37:37.466658+0000","last_undegraded":"2023-10-16T20:37:37.466658+0000","last_fullsized":"2023-10-16T20:37:37.466658+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:05:01.634971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"26'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404148+0000","last_change":"2023-10-16T20:36:54.763818+0000","last_active":"2023-10-16T20:37:37.404148+0000","last_peered":"2023-10-16T20:37:37.404148+0000","last_clean":"2023-10-16T20:37:37.404148+0000","last_became_active":"2023-10-16T20:36:54.761337+0000","last_became_peered":"2023-10-16T20:36:54.761337+0000","last_unstale":"2023-10-16T20:37:37.404148+0000","last_undegraded":"2023-10-16T20:37:37.404148+0000","last_fullsized":"2023-10-16T20:37:37.404148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:03.093189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.470301+0000","last_change":"2023-10-16T20:37:37.468803+0000","last_active":"2023-10-16T20:37:37.470301+0000","last_peered":"2023-10-16T20:37:37.470301+0000","last_clean":"2023-10-16T20:37:37.470301+0000","last_became_active":"2023-10-16T20:37:37.468309+0000","last_became_peered":"2023-10-16T20:37:37.468309+0000","last_unstale":"2023-10-16T20:37:37.470301+0000","last_undegraded":"2023-10-16T20:37:37.470301+0000","last_fullsized":"2023-10-16T20:37:37.470301+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:11.761661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466101+0000","last_change":"2023-10-16T20:36:55.288895+0000","last_active":"2023-10-16T20:37:37.466101+0000","last_peered":"2023-10-16T20:37:37.466101+0000","last_clean":"2023-10-16T20:37:37.466101+0000","last_became_active":"2023-10-16T20:36:55.288354+0000","last_became_peered":"2023-10-16T20:36:55.288354+0000","last_unstale":"2023-10-16T20:37:37.466101+0000","last_undegraded":"2023-10-16T20:37:37.466101+0000","last_fullsized":"2023-10-16T20:37:37.466101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:53:29.192406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'10","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.402644+0000","last_change":"2023-10-16T20:36:54.738216+0000","last_active":"2023-10-16T20:39:22.402644+0000","last_peered":"2023-10-16T20:39:22.402644+0000","last_clean":"2023-10-16T20:39:22.402644+0000","last_became_active":"2023-10-16T20:36:54.737876+0000","last_became_peered":"2023-10-16T20:36:54.737876+0000","last_unstale":"2023-10-16T20:39:22.402644+0000","last_undegraded":"2023-10-16T20:39:22.402644+0000","last_fullsized":"2023-10-16T20:39:22.402644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:28.214601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435178+0000","last_change":"2023-10-16T20:36:59.307215+0000","last_active":"2023-10-16T20:37:37.435178+0000","last_peered":"2023-10-16T20:37:37.435178+0000","last_clean":"2023-10-16T20:37:37.435178+0000","last_became_active":"2023-10-16T20:36:59.306801+0000","last_became_peered":"2023-10-16T20:36:59.306801+0000","last_unstale":"2023-10-16T20:37:37.435178+0000","last_undegraded":"2023-10-16T20:37:37.435178+0000","last_fullsized":"2023-10-16T20:37:37.435178+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:41:42.656134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407054+0000","last_change":"2023-10-16T20:36:55.287714+0000","last_active":"2023-10-16T20:37:37.407054+0000","last_peered":"2023-10-16T20:37:37.407054+0000","last_clean":"2023-10-16T20:37:37.407054+0000","last_became_active":"2023-10-16T20:36:55.287066+0000","last_became_peered":"2023-10-16T20:36:55.287066+0000","last_unstale":"2023-10-16T20:37:37.407054+0000","last_undegraded":"2023-10-16T20:37:37.407054+0000","last_fullsized":"2023-10-16T20:37:37.407054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:45:21.058260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407609+0000","last_change":"2023-10-16T20:36:54.764693+0000","last_active":"2023-10-16T20:37:37.407609+0000","last_peered":"2023-10-16T20:37:37.407609+0000","last_clean":"2023-10-16T20:37:37.407609+0000","last_became_active":"2023-10-16T20:36:54.764417+0000","last_became_peered":"2023-10-16T20:36:54.764417+0000","last_unstale":"2023-10-16T20:37:37.407609+0000","last_undegraded":"2023-10-16T20:37:37.407609+0000","last_fullsized":"2023-10-16T20:37:37.407609+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:56.653149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404514+0000","last_change":"2023-10-16T20:36:59.465145+0000","last_active":"2023-10-16T20:37:36.404514+0000","last_peered":"2023-10-16T20:37:36.404514+0000","last_clean":"2023-10-16T20:37:36.404514+0000","last_became_active":"2023-10-16T20:36:59.464626+0000","last_became_peered":"2023-10-16T20:36:59.464626+0000","last_unstale":"2023-10-16T20:37:36.404514+0000","last_undegraded":"2023-10-16T20:37:36.404514+0000","last_fullsized":"2023-10-16T20:37:36.404514+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:38.928616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466290+0000","last_change":"2023-10-16T20:36:55.291718+0000","last_active":"2023-10-16T20:37:37.466290+0000","last_peered":"2023-10-16T20:37:37.466290+0000","last_clean":"2023-10-16T20:37:37.466290+0000","last_became_active":"2023-10-16T20:36:55.291396+0000","last_became_peered":"2023-10-16T20:36:55.291396+0000","last_unstale":"2023-10-16T20:37:37.466290+0000","last_undegraded":"2023-10-16T20:37:37.466290+0000","last_fullsized":"2023-10-16T20:37:37.466290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:29:49.138380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.411305+0000","last_change":"2023-10-16T20:36:54.750814+0000","last_active":"2023-10-16T20:39:22.411305+0000","last_peered":"2023-10-16T20:39:22.411305+0000","last_clean":"2023-10-16T20:39:22.411305+0000","last_became_active":"2023-10-16T20:36:54.750448+0000","last_became_peered":"2023-10-16T20:36:54.750448+0000","last_unstale":"2023-10-16T20:39:22.411305+0000","last_undegraded":"2023-10-16T20:39:22.411305+0000","last_fullsized":"2023-10-16T20:39:22.411305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:41.981408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465192+0000","last_change":"2023-10-16T20:36:59.464962+0000","last_active":"2023-10-16T20:37:37.465192+0000","last_peered":"2023-10-16T20:37:37.465192+0000","last_clean":"2023-10-16T20:37:37.465192+0000","last_became_active":"2023-10-16T20:36:59.464578+0000","last_became_peered":"2023-10-16T20:36:59.464578+0000","last_unstale":"2023-10-16T20:37:37.465192+0000","last_undegraded":"2023-10-16T20:37:37.465192+0000","last_fullsized":"2023-10-16T20:37:37.465192+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:12:27.738364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435179+0000","last_change":"2023-10-16T20:36:55.285675+0000","last_active":"2023-10-16T20:37:37.435179+0000","last_peered":"2023-10-16T20:37:37.435179+0000","last_clean":"2023-10-16T20:37:37.435179+0000","last_became_active":"2023-10-16T20:36:55.285407+0000","last_became_peered":"2023-10-16T20:36:55.285407+0000","last_unstale":"2023-10-16T20:37:37.435179+0000","last_undegraded":"2023-10-16T20:37:37.435179+0000","last_fullsized":"2023-10-16T20:37:37.435179+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:31:10.259938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"26'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435225+0000","last_change":"2023-10-16T20:36:54.757012+0000","last_active":"2023-10-16T20:37:37.435225+0000","last_peered":"2023-10-16T20:37:37.435225+0000","last_clean":"2023-10-16T20:37:37.435225+0000","last_became_active":"2023-10-16T20:36:54.756697+0000","last_became_peered":"2023-10-16T20:36:54.756697+0000","last_unstale":"2023-10-16T20:37:37.435225+0000","last_undegraded":"2023-10-16T20:37:37.435225+0000","last_fullsized":"2023-10-16T20:37:37.435225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:20.259580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.409816+0000","last_change":"2023-10-16T20:36:59.469134+0000","last_active":"2023-10-16T20:37:36.409816+0000","last_peered":"2023-10-16T20:37:36.409816+0000","last_clean":"2023-10-16T20:37:36.409816+0000","last_became_active":"2023-10-16T20:36:59.468254+0000","last_became_peered":"2023-10-16T20:36:59.468254+0000","last_unstale":"2023-10-16T20:37:36.409816+0000","last_undegraded":"2023-10-16T20:37:36.409816+0000","last_fullsized":"2023-10-16T20:37:36.409816+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:12.444815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466418+0000","last_change":"2023-10-16T20:36:55.289133+0000","last_active":"2023-10-16T20:37:37.466418+0000","last_peered":"2023-10-16T20:37:37.466418+0000","last_clean":"2023-10-16T20:37:37.466418+0000","last_became_active":"2023-10-16T20:36:55.288488+0000","last_became_peered":"2023-10-16T20:36:55.288488+0000","last_unstale":"2023-10-16T20:37:37.466418+0000","last_undegraded":"2023-10-16T20:37:37.466418+0000","last_fullsized":"2023-10-16T20:37:37.466418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:27:22.022319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'12","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.411309+0000","last_change":"2023-10-16T20:36:54.765361+0000","last_active":"2023-10-16T20:39:22.411309+0000","last_peered":"2023-10-16T20:39:22.411309+0000","last_clean":"2023-10-16T20:39:22.411309+0000","last_became_active":"2023-10-16T20:36:54.764969+0000","last_became_peered":"2023-10-16T20:36:54.764969+0000","last_unstale":"2023-10-16T20:39:22.411309+0000","last_undegraded":"2023-10-16T20:39:22.411309+0000","last_fullsized":"2023-10-16T20:39:22.411309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:21:21.183612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3841,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402193+0000","last_change":"2023-10-16T20:36:59.303588+0000","last_active":"2023-10-16T20:37:37.402193+0000","last_peered":"2023-10-16T20:37:37.402193+0000","last_clean":"2023-10-16T20:37:37.402193+0000","last_became_active":"2023-10-16T20:36:59.302698+0000","last_became_peered":"2023-10-16T20:36:59.302698+0000","last_unstale":"2023-10-16T20:37:37.402193+0000","last_undegraded":"2023-10-16T20:37:37.402193+0000","last_fullsized":"2023-10-16T20:37:37.402193+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:08:28.644565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403928+0000","last_change":"2023-10-16T20:36:55.290812+0000","last_active":"2023-10-16T20:37:37.403928+0000","last_peered":"2023-10-16T20:37:37.403928+0000","last_clean":"2023-10-16T20:37:37.403928+0000","last_became_active":"2023-10-16T20:36:55.290127+0000","last_became_peered":"2023-10-16T20:36:55.290127+0000","last_unstale":"2023-10-16T20:37:37.403928+0000","last_undegraded":"2023-10-16T20:37:37.403928+0000","last_fullsized":"2023-10-16T20:37:37.403928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:28:27.930123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.217490+0000","last_change":"2023-10-16T20:36:54.749120+0000","last_active":"2023-10-16T20:39:22.217490+0000","last_peered":"2023-10-16T20:39:22.217490+0000","last_clean":"2023-10-16T20:39:22.217490+0000","last_became_active":"2023-10-16T20:36:54.748788+0000","last_became_peered":"2023-10-16T20:36:54.748788+0000","last_unstale":"2023-10-16T20:39:22.217490+0000","last_undegraded":"2023-10-16T20:39:22.217490+0000","last_fullsized":"2023-10-16T20:39:22.217490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:11:07.257343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403049+0000","last_change":"2023-10-16T20:36:59.303546+0000","last_active":"2023-10-16T20:37:37.403049+0000","last_peered":"2023-10-16T20:37:37.403049+0000","last_clean":"2023-10-16T20:37:37.403049+0000","last_became_active":"2023-10-16T20:36:59.302916+0000","last_became_peered":"2023-10-16T20:36:59.302916+0000","last_unstale":"2023-10-16T20:37:37.403049+0000","last_undegraded":"2023-10-16T20:37:37.403049+0000","last_fullsized":"2023-10-16T20:37:37.403049+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:17:16.174476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.398104+0000","last_change":"2023-10-16T20:36:55.291739+0000","last_active":"2023-10-16T20:37:36.398104+0000","last_peered":"2023-10-16T20:37:36.398104+0000","last_clean":"2023-10-16T20:37:36.398104+0000","last_became_active":"2023-10-16T20:36:55.291448+0000","last_became_peered":"2023-10-16T20:36:55.291448+0000","last_unstale":"2023-10-16T20:37:36.398104+0000","last_undegraded":"2023-10-16T20:37:36.398104+0000","last_fullsized":"2023-10-16T20:37:36.398104+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:41:15.380678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.410923+0000","last_change":"2023-10-16T20:36:54.761824+0000","last_active":"2023-10-16T20:39:22.410923+0000","last_peered":"2023-10-16T20:39:22.410923+0000","last_clean":"2023-10-16T20:39:22.410923+0000","last_became_active":"2023-10-16T20:36:54.761541+0000","last_became_peered":"2023-10-16T20:36:54.761541+0000","last_unstale":"2023-10-16T20:39:22.410923+0000","last_undegraded":"2023-10-16T20:39:22.410923+0000","last_fullsized":"2023-10-16T20:39:22.410923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:34.512207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464594+0000","last_change":"2023-10-16T20:36:59.304165+0000","last_active":"2023-10-16T20:37:37.464594+0000","last_peered":"2023-10-16T20:37:37.464594+0000","last_clean":"2023-10-16T20:37:37.464594+0000","last_became_active":"2023-10-16T20:36:59.303773+0000","last_became_peered":"2023-10-16T20:36:59.303773+0000","last_unstale":"2023-10-16T20:37:37.464594+0000","last_undegraded":"2023-10-16T20:37:37.464594+0000","last_fullsized":"2023-10-16T20:37:37.464594+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:09.710698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409711+0000","last_change":"2023-10-16T20:36:55.300829+0000","last_active":"2023-10-16T20:37:37.409711+0000","last_peered":"2023-10-16T20:37:37.409711+0000","last_clean":"2023-10-16T20:37:37.409711+0000","last_became_active":"2023-10-16T20:36:55.300535+0000","last_became_peered":"2023-10-16T20:36:55.300535+0000","last_unstale":"2023-10-16T20:37:37.409711+0000","last_undegraded":"2023-10-16T20:37:37.409711+0000","last_fullsized":"2023-10-16T20:37:37.409711+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:34.910378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.413267+0000","last_change":"2023-10-16T20:36:54.758140+0000","last_active":"2023-10-16T20:39:22.413267+0000","last_peered":"2023-10-16T20:39:22.413267+0000","last_clean":"2023-10-16T20:39:22.413267+0000","last_became_active":"2023-10-16T20:36:54.757303+0000","last_became_peered":"2023-10-16T20:36:54.757303+0000","last_unstale":"2023-10-16T20:39:22.413267+0000","last_undegraded":"2023-10-16T20:39:22.413267+0000","last_fullsized":"2023-10-16T20:39:22.413267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:45.944191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464795+0000","last_change":"2023-10-16T20:36:59.471158+0000","last_active":"2023-10-16T20:37:37.464795+0000","last_peered":"2023-10-16T20:37:37.464795+0000","last_clean":"2023-10-16T20:37:37.464795+0000","last_became_active":"2023-10-16T20:36:59.465844+0000","last_became_peered":"2023-10-16T20:36:59.465844+0000","last_unstale":"2023-10-16T20:37:37.464795+0000","last_undegraded":"2023-10-16T20:37:37.464795+0000","last_fullsized":"2023-10-16T20:37:37.464795+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:12.133525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435182+0000","last_change":"2023-10-16T20:36:59.468843+0000","last_active":"2023-10-16T20:37:37.435182+0000","last_peered":"2023-10-16T20:37:37.435182+0000","last_clean":"2023-10-16T20:37:37.435182+0000","last_became_active":"2023-10-16T20:36:59.467356+0000","last_became_peered":"2023-10-16T20:36:59.467356+0000","last_unstale":"2023-10-16T20:37:37.435182+0000","last_undegraded":"2023-10-16T20:37:37.435182+0000","last_fullsized":"2023-10-16T20:37:37.435182+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:47:02.923540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401755+0000","last_change":"2023-10-16T20:36:55.281355+0000","last_active":"2023-10-16T20:37:37.401755+0000","last_peered":"2023-10-16T20:37:37.401755+0000","last_clean":"2023-10-16T20:37:37.401755+0000","last_became_active":"2023-10-16T20:36:55.280903+0000","last_became_peered":"2023-10-16T20:36:55.280903+0000","last_unstale":"2023-10-16T20:37:37.401755+0000","last_undegraded":"2023-10-16T20:37:37.401755+0000","last_fullsized":"2023-10-16T20:37:37.401755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:15:25.813826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.231980+0000","last_change":"2023-10-16T20:36:54.757589+0000","last_active":"2023-10-16T20:39:22.231980+0000","last_peered":"2023-10-16T20:39:22.231980+0000","last_clean":"2023-10-16T20:39:22.231980+0000","last_became_active":"2023-10-16T20:36:54.756775+0000","last_became_peered":"2023-10-16T20:36:54.756775+0000","last_unstale":"2023-10-16T20:39:22.231980+0000","last_undegraded":"2023-10-16T20:39:22.231980+0000","last_fullsized":"2023-10-16T20:39:22.231980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:40.629540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.441661+0000","last_change":"2023-10-16T20:37:37.438581+0000","last_active":"2023-10-16T20:37:37.441661+0000","last_peered":"2023-10-16T20:37:37.441661+0000","last_clean":"2023-10-16T20:37:37.441661+0000","last_became_active":"2023-10-16T20:37:37.438140+0000","last_became_peered":"2023-10-16T20:37:37.438140+0000","last_unstale":"2023-10-16T20:37:37.441661+0000","last_undegraded":"2023-10-16T20:37:37.441661+0000","last_fullsized":"2023-10-16T20:37:37.441661+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:55.110618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464854+0000","last_change":"2023-10-16T20:36:55.274207+0000","last_active":"2023-10-16T20:37:37.464854+0000","last_peered":"2023-10-16T20:37:37.464854+0000","last_clean":"2023-10-16T20:37:37.464854+0000","last_became_active":"2023-10-16T20:36:55.273959+0000","last_became_peered":"2023-10-16T20:36:55.273959+0000","last_unstale":"2023-10-16T20:37:37.464854+0000","last_undegraded":"2023-10-16T20:37:37.464854+0000","last_fullsized":"2023-10-16T20:37:37.464854+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:23:09.275091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465033+0000","last_change":"2023-10-16T20:36:54.759513+0000","last_active":"2023-10-16T20:37:37.465033+0000","last_peered":"2023-10-16T20:37:37.465033+0000","last_clean":"2023-10-16T20:37:37.465033+0000","last_became_active":"2023-10-16T20:36:54.759059+0000","last_became_peered":"2023-10-16T20:36:54.759059+0000","last_unstale":"2023-10-16T20:37:37.465033+0000","last_undegraded":"2023-10-16T20:37:37.465033+0000","last_fullsized":"2023-10-16T20:37:37.465033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:36.696053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"27'1","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404990+0000","last_change":"2023-10-16T20:36:54.764780+0000","last_active":"2023-10-16T20:37:36.404990+0000","last_peered":"2023-10-16T20:37:36.404990+0000","last_clean":"2023-10-16T20:37:36.404990+0000","last_became_active":"2023-10-16T20:36:54.764436+0000","last_became_peered":"2023-10-16T20:36:54.764436+0000","last_unstale":"2023-10-16T20:37:36.404990+0000","last_undegraded":"2023-10-16T20:37:36.404990+0000","last_fullsized":"2023-10-16T20:37:36.404990+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:55:31.816599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.398400+0000","last_change":"2023-10-16T20:36:55.278134+0000","last_active":"2023-10-16T20:37:36.398400+0000","last_peered":"2023-10-16T20:37:36.398400+0000","last_clean":"2023-10-16T20:37:36.398400+0000","last_became_active":"2023-10-16T20:36:55.277873+0000","last_became_peered":"2023-10-16T20:36:55.277873+0000","last_unstale":"2023-10-16T20:37:36.398400+0000","last_undegraded":"2023-10-16T20:37:36.398400+0000","last_fullsized":"2023-10-16T20:37:36.398400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:59:43.701800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.407258+0000","last_change":"2023-10-16T20:36:59.302774+0000","last_active":"2023-10-16T20:37:36.407258+0000","last_peered":"2023-10-16T20:37:36.407258+0000","last_clean":"2023-10-16T20:37:36.407258+0000","last_became_active":"2023-10-16T20:36:59.302435+0000","last_became_peered":"2023-10-16T20:36:59.302435+0000","last_unstale":"2023-10-16T20:37:36.407258+0000","last_undegraded":"2023-10-16T20:37:36.407258+0000","last_fullsized":"2023-10-16T20:37:36.407258+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:11:53.740982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'4","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.407881+0000","last_change":"2023-10-16T20:36:54.756190+0000","last_active":"2023-10-16T20:39:22.407881+0000","last_peered":"2023-10-16T20:39:22.407881+0000","last_clean":"2023-10-16T20:39:22.407881+0000","last_became_active":"2023-10-16T20:36:54.755834+0000","last_became_peered":"2023-10-16T20:36:54.755834+0000","last_unstale":"2023-10-16T20:39:22.407881+0000","last_undegraded":"2023-10-16T20:39:22.407881+0000","last_fullsized":"2023-10-16T20:39:22.407881+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:31.012441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.402297+0000","last_change":"2023-10-16T20:36:55.279822+0000","last_active":"2023-10-16T20:37:36.402297+0000","last_peered":"2023-10-16T20:37:36.402297+0000","last_clean":"2023-10-16T20:37:36.402297+0000","last_became_active":"2023-10-16T20:36:55.279333+0000","last_became_peered":"2023-10-16T20:36:55.279333+0000","last_unstale":"2023-10-16T20:37:36.402297+0000","last_undegraded":"2023-10-16T20:37:36.402297+0000","last_fullsized":"2023-10-16T20:37:36.402297+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:02:21.471812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.403977+0000","last_change":"2023-10-16T20:36:59.302190+0000","last_active":"2023-10-16T20:37:36.403977+0000","last_peered":"2023-10-16T20:37:36.403977+0000","last_clean":"2023-10-16T20:37:36.403977+0000","last_became_active":"2023-10-16T20:36:59.301864+0000","last_became_peered":"2023-10-16T20:36:59.301864+0000","last_unstale":"2023-10-16T20:37:36.403977+0000","last_undegraded":"2023-10-16T20:37:36.403977+0000","last_fullsized":"2023-10-16T20:37:36.403977+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:37:55.282815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.406436+0000","last_change":"2023-10-16T20:36:55.302476+0000","last_active":"2023-10-16T20:37:37.406436+0000","last_peered":"2023-10-16T20:37:37.406436+0000","last_clean":"2023-10-16T20:37:37.406436+0000","last_became_active":"2023-10-16T20:36:55.287442+0000","last_became_peered":"2023-10-16T20:36:55.287442+0000","last_unstale":"2023-10-16T20:37:37.406436+0000","last_undegraded":"2023-10-16T20:37:37.406436+0000","last_fullsized":"2023-10-16T20:37:37.406436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:19.511156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.399520+0000","last_change":"2023-10-16T20:36:55.276521+0000","last_active":"2023-10-16T20:37:37.399520+0000","last_peered":"2023-10-16T20:37:37.399520+0000","last_clean":"2023-10-16T20:37:37.399520+0000","last_became_active":"2023-10-16T20:36:55.276276+0000","last_became_peered":"2023-10-16T20:36:55.276276+0000","last_unstale":"2023-10-16T20:37:37.399520+0000","last_undegraded":"2023-10-16T20:37:37.399520+0000","last_fullsized":"2023-10-16T20:37:37.399520+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:38.218242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.415548+0000","last_change":"2023-10-16T20:37:37.413429+0000","last_active":"2023-10-16T20:37:37.415548+0000","last_peered":"2023-10-16T20:37:37.415548+0000","last_clean":"2023-10-16T20:37:37.415548+0000","last_became_active":"2023-10-16T20:37:37.413040+0000","last_became_peered":"2023-10-16T20:37:37.413040+0000","last_unstale":"2023-10-16T20:37:37.415548+0000","last_undegraded":"2023-10-16T20:37:37.415548+0000","last_fullsized":"2023-10-16T20:37:37.415548+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:39:10.897056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435108+0000","last_change":"2023-10-16T20:36:55.276258+0000","last_active":"2023-10-16T20:37:37.435108+0000","last_peered":"2023-10-16T20:37:37.435108+0000","last_clean":"2023-10-16T20:37:37.435108+0000","last_became_active":"2023-10-16T20:36:55.275349+0000","last_became_peered":"2023-10-16T20:36:55.275349+0000","last_unstale":"2023-10-16T20:37:37.435108+0000","last_undegraded":"2023-10-16T20:37:37.435108+0000","last_fullsized":"2023-10-16T20:37:37.435108+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:59.007782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465455+0000","last_change":"2023-10-16T20:36:55.277332+0000","last_active":"2023-10-16T20:37:37.465455+0000","last_peered":"2023-10-16T20:37:37.465455+0000","last_clean":"2023-10-16T20:37:37.465455+0000","last_became_active":"2023-10-16T20:36:55.277043+0000","last_became_peered":"2023-10-16T20:36:55.277043+0000","last_unstale":"2023-10-16T20:37:37.465455+0000","last_undegraded":"2023-10-16T20:37:37.465455+0000","last_fullsized":"2023-10-16T20:37:37.465455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:46:07.227813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401690+0000","last_change":"2023-10-16T20:36:55.279003+0000","last_active":"2023-10-16T20:37:37.401690+0000","last_peered":"2023-10-16T20:37:37.401690+0000","last_clean":"2023-10-16T20:37:37.401690+0000","last_became_active":"2023-10-16T20:36:55.278715+0000","last_became_peered":"2023-10-16T20:36:55.278715+0000","last_unstale":"2023-10-16T20:37:37.401690+0000","last_undegraded":"2023-10-16T20:37:37.401690+0000","last_fullsized":"2023-10-16T20:37:37.401690+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:58.274009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408078+0000","last_change":"2023-10-16T20:36:55.303936+0000","last_active":"2023-10-16T20:37:37.408078+0000","last_peered":"2023-10-16T20:37:37.408078+0000","last_clean":"2023-10-16T20:37:37.408078+0000","last_became_active":"2023-10-16T20:36:55.301604+0000","last_became_peered":"2023-10-16T20:36:55.301604+0000","last_unstale":"2023-10-16T20:37:37.408078+0000","last_undegraded":"2023-10-16T20:37:37.408078+0000","last_fullsized":"2023-10-16T20:37:37.408078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:48:14.922695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465625+0000","last_change":"2023-10-16T20:36:55.277329+0000","last_active":"2023-10-16T20:37:37.465625+0000","last_peered":"2023-10-16T20:37:37.465625+0000","last_clean":"2023-10-16T20:37:37.465625+0000","last_became_active":"2023-10-16T20:36:55.276984+0000","last_became_peered":"2023-10-16T20:36:55.276984+0000","last_unstale":"2023-10-16T20:37:37.465625+0000","last_undegraded":"2023-10-16T20:37:37.465625+0000","last_fullsized":"2023-10-16T20:37:37.465625+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:43:52.763586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408404+0000","last_change":"2023-10-16T20:36:55.290545+0000","last_active":"2023-10-16T20:37:37.408404+0000","last_peered":"2023-10-16T20:37:37.408404+0000","last_clean":"2023-10-16T20:37:37.408404+0000","last_became_active":"2023-10-16T20:36:55.290023+0000","last_became_peered":"2023-10-16T20:36:55.290023+0000","last_unstale":"2023-10-16T20:37:37.408404+0000","last_undegraded":"2023-10-16T20:37:37.408404+0000","last_fullsized":"2023-10-16T20:37:37.408404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:04:20.914854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464978+0000","last_change":"2023-10-16T20:36:55.280456+0000","last_active":"2023-10-16T20:37:37.464978+0000","last_peered":"2023-10-16T20:37:37.464978+0000","last_clean":"2023-10-16T20:37:37.464978+0000","last_became_active":"2023-10-16T20:36:55.280186+0000","last_became_peered":"2023-10-16T20:36:55.280186+0000","last_unstale":"2023-10-16T20:37:37.464978+0000","last_undegraded":"2023-10-16T20:37:37.464978+0000","last_fullsized":"2023-10-16T20:37:37.464978+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:52.469768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408252+0000","last_change":"2023-10-16T20:36:55.297917+0000","last_active":"2023-10-16T20:37:37.408252+0000","last_peered":"2023-10-16T20:37:37.408252+0000","last_clean":"2023-10-16T20:37:37.408252+0000","last_became_active":"2023-10-16T20:36:55.297253+0000","last_became_peered":"2023-10-16T20:36:55.297253+0000","last_unstale":"2023-10-16T20:37:37.408252+0000","last_undegraded":"2023-10-16T20:37:37.408252+0000","last_fullsized":"2023-10-16T20:37:37.408252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:14.977093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434979+0000","last_change":"2023-10-16T20:36:55.286327+0000","last_active":"2023-10-16T20:37:37.434979+0000","last_peered":"2023-10-16T20:37:37.434979+0000","last_clean":"2023-10-16T20:37:37.434979+0000","last_became_active":"2023-10-16T20:36:55.286044+0000","last_became_peered":"2023-10-16T20:36:55.286044+0000","last_unstale":"2023-10-16T20:37:37.434979+0000","last_undegraded":"2023-10-16T20:37:37.434979+0000","last_fullsized":"2023-10-16T20:37:37.434979+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:30:56.000502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409227+0000","last_change":"2023-10-16T20:36:55.298742+0000","last_active":"2023-10-16T20:37:37.409227+0000","last_peered":"2023-10-16T20:37:37.409227+0000","last_clean":"2023-10-16T20:37:37.409227+0000","last_became_active":"2023-10-16T20:36:55.286864+0000","last_became_peered":"2023-10-16T20:36:55.286864+0000","last_unstale":"2023-10-16T20:37:37.409227+0000","last_undegraded":"2023-10-16T20:37:37.409227+0000","last_fullsized":"2023-10-16T20:37:37.409227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:24:27.598910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466521+0000","last_change":"2023-10-16T20:36:55.292541+0000","last_active":"2023-10-16T20:37:37.466521+0000","last_peered":"2023-10-16T20:37:37.466521+0000","last_clean":"2023-10-16T20:37:37.466521+0000","last_became_active":"2023-10-16T20:36:55.292262+0000","last_became_peered":"2023-10-16T20:36:55.292262+0000","last_unstale":"2023-10-16T20:37:37.466521+0000","last_undegraded":"2023-10-16T20:37:37.466521+0000","last_fullsized":"2023-10-16T20:37:37.466521+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:10:08.644265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.400632+0000","last_change":"2023-10-16T20:36:55.283056+0000","last_active":"2023-10-16T20:37:37.400632+0000","last_peered":"2023-10-16T20:37:37.400632+0000","last_clean":"2023-10-16T20:37:37.400632+0000","last_became_active":"2023-10-16T20:36:55.282806+0000","last_became_peered":"2023-10-16T20:36:55.282806+0000","last_unstale":"2023-10-16T20:37:37.400632+0000","last_undegraded":"2023-10-16T20:37:37.400632+0000","last_fullsized":"2023-10-16T20:37:37.400632+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:21:24.681696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410649+0000","last_change":"2023-10-16T20:36:55.303385+0000","last_active":"2023-10-16T20:37:37.410649+0000","last_peered":"2023-10-16T20:37:37.410649+0000","last_clean":"2023-10-16T20:37:37.410649+0000","last_became_active":"2023-10-16T20:36:55.303095+0000","last_became_peered":"2023-10-16T20:36:55.303095+0000","last_unstale":"2023-10-16T20:37:37.410649+0000","last_undegraded":"2023-10-16T20:37:37.410649+0000","last_fullsized":"2023-10-16T20:37:37.410649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:26.368480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408312+0000","last_change":"2023-10-16T20:36:55.291852+0000","last_active":"2023-10-16T20:37:37.408312+0000","last_peered":"2023-10-16T20:37:37.408312+0000","last_clean":"2023-10-16T20:37:37.408312+0000","last_became_active":"2023-10-16T20:36:55.291447+0000","last_became_peered":"2023-10-16T20:36:55.291447+0000","last_unstale":"2023-10-16T20:37:37.408312+0000","last_undegraded":"2023-10-16T20:37:37.408312+0000","last_fullsized":"2023-10-16T20:37:37.408312+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:07.195372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401838+0000","last_change":"2023-10-16T20:36:55.281169+0000","last_active":"2023-10-16T20:37:37.401838+0000","last_peered":"2023-10-16T20:37:37.401838+0000","last_clean":"2023-10-16T20:37:37.401838+0000","last_became_active":"2023-10-16T20:36:55.280605+0000","last_became_peered":"2023-10-16T20:36:55.280605+0000","last_unstale":"2023-10-16T20:37:37.401838+0000","last_undegraded":"2023-10-16T20:37:37.401838+0000","last_fullsized":"2023-10-16T20:37:37.401838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:03.294173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435040+0000","last_change":"2023-10-16T20:36:55.277395+0000","last_active":"2023-10-16T20:37:37.435040+0000","last_peered":"2023-10-16T20:37:37.435040+0000","last_clean":"2023-10-16T20:37:37.435040+0000","last_became_active":"2023-10-16T20:36:55.277094+0000","last_became_peered":"2023-10-16T20:36:55.277094+0000","last_unstale":"2023-10-16T20:37:37.435040+0000","last_undegraded":"2023-10-16T20:37:37.435040+0000","last_fullsized":"2023-10-16T20:37:37.435040+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:23:03.400384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465006+0000","last_change":"2023-10-16T20:36:55.297429+0000","last_active":"2023-10-16T20:37:37.465006+0000","last_peered":"2023-10-16T20:37:37.465006+0000","last_clean":"2023-10-16T20:37:37.465006+0000","last_became_active":"2023-10-16T20:36:55.297199+0000","last_became_peered":"2023-10-16T20:36:55.297199+0000","last_unstale":"2023-10-16T20:37:37.465006+0000","last_undegraded":"2023-10-16T20:37:37.465006+0000","last_fullsized":"2023-10-16T20:37:37.465006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:43.860251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404029+0000","last_change":"2023-10-16T20:36:55.287421+0000","last_active":"2023-10-16T20:37:37.404029+0000","last_peered":"2023-10-16T20:37:37.404029+0000","last_clean":"2023-10-16T20:37:37.404029+0000","last_became_active":"2023-10-16T20:36:55.286964+0000","last_became_peered":"2023-10-16T20:36:55.286964+0000","last_unstale":"2023-10-16T20:37:37.404029+0000","last_undegraded":"2023-10-16T20:37:37.404029+0000","last_fullsized":"2023-10-16T20:37:37.404029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:15:54.970100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.416921+0000","last_change":"2023-10-16T20:37:37.413891+0000","last_active":"2023-10-16T20:37:37.416921+0000","last_peered":"2023-10-16T20:37:37.416921+0000","last_clean":"2023-10-16T20:37:37.416921+0000","last_became_active":"2023-10-16T20:37:37.413492+0000","last_became_peered":"2023-10-16T20:37:37.413492+0000","last_unstale":"2023-10-16T20:37:37.416921+0000","last_undegraded":"2023-10-16T20:37:37.416921+0000","last_fullsized":"2023-10-16T20:37:37.416921+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:03:46.658556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407997+0000","last_change":"2023-10-16T20:36:55.303950+0000","last_active":"2023-10-16T20:37:37.407997+0000","last_peered":"2023-10-16T20:37:37.407997+0000","last_clean":"2023-10-16T20:37:37.407997+0000","last_became_active":"2023-10-16T20:36:55.302468+0000","last_became_peered":"2023-10-16T20:36:55.302468+0000","last_unstale":"2023-10-16T20:37:37.407997+0000","last_undegraded":"2023-10-16T20:37:37.407997+0000","last_fullsized":"2023-10-16T20:37:37.407997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:02.962031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466830+0000","last_change":"2023-10-16T20:36:55.297558+0000","last_active":"2023-10-16T20:37:37.466830+0000","last_peered":"2023-10-16T20:37:37.466830+0000","last_clean":"2023-10-16T20:37:37.466830+0000","last_became_active":"2023-10-16T20:36:55.296476+0000","last_became_peered":"2023-10-16T20:36:55.296476+0000","last_unstale":"2023-10-16T20:37:37.466830+0000","last_undegraded":"2023-10-16T20:37:37.466830+0000","last_fullsized":"2023-10-16T20:37:37.466830+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:04:38.235205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.403439+0000","last_change":"2023-10-16T20:36:55.293957+0000","last_active":"2023-10-16T20:37:36.403439+0000","last_peered":"2023-10-16T20:37:36.403439+0000","last_clean":"2023-10-16T20:37:36.403439+0000","last_became_active":"2023-10-16T20:36:55.293687+0000","last_became_peered":"2023-10-16T20:36:55.293687+0000","last_unstale":"2023-10-16T20:37:36.403439+0000","last_undegraded":"2023-10-16T20:37:36.403439+0000","last_fullsized":"2023-10-16T20:37:36.403439+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:54.623691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.411550+0000","last_change":"2023-10-16T20:36:55.280728+0000","last_active":"2023-10-16T20:37:36.411550+0000","last_peered":"2023-10-16T20:37:36.411550+0000","last_clean":"2023-10-16T20:37:36.411550+0000","last_became_active":"2023-10-16T20:36:55.280444+0000","last_became_peered":"2023-10-16T20:36:55.280444+0000","last_unstale":"2023-10-16T20:37:36.411550+0000","last_undegraded":"2023-10-16T20:37:36.411550+0000","last_fullsized":"2023-10-16T20:37:36.411550+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:44.044563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.408400+0000","last_change":"2023-10-16T20:36:55.297802+0000","last_active":"2023-10-16T20:37:36.408400+0000","last_peered":"2023-10-16T20:37:36.408400+0000","last_clean":"2023-10-16T20:37:36.408400+0000","last_became_active":"2023-10-16T20:36:55.296694+0000","last_became_peered":"2023-10-16T20:36:55.296694+0000","last_unstale":"2023-10-16T20:37:36.408400+0000","last_undegraded":"2023-10-16T20:37:36.408400+0000","last_fullsized":"2023-10-16T20:37:36.408400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:00:26.797387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.411707+0000","last_change":"2023-10-16T20:36:55.291347+0000","last_active":"2023-10-16T20:37:36.411707+0000","last_peered":"2023-10-16T20:37:36.411707+0000","last_clean":"2023-10-16T20:37:36.411707+0000","last_became_active":"2023-10-16T20:36:55.291052+0000","last_became_peered":"2023-10-16T20:36:55.291052+0000","last_unstale":"2023-10-16T20:37:36.411707+0000","last_undegraded":"2023-10-16T20:37:36.411707+0000","last_fullsized":"2023-10-16T20:37:36.411707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:40:51.331446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409006+0000","last_change":"2023-10-16T20:36:55.290870+0000","last_active":"2023-10-16T20:37:37.409006+0000","last_peered":"2023-10-16T20:37:37.409006+0000","last_clean":"2023-10-16T20:37:37.409006+0000","last_became_active":"2023-10-16T20:36:55.290386+0000","last_became_peered":"2023-10-16T20:36:55.290386+0000","last_unstale":"2023-10-16T20:37:37.409006+0000","last_undegraded":"2023-10-16T20:37:37.409006+0000","last_fullsized":"2023-10-16T20:37:37.409006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:20.006864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435199+0000","last_change":"2023-10-16T20:36:55.271041+0000","last_active":"2023-10-16T20:37:37.435199+0000","last_peered":"2023-10-16T20:37:37.435199+0000","last_clean":"2023-10-16T20:37:37.435199+0000","last_became_active":"2023-10-16T20:36:55.270787+0000","last_became_peered":"2023-10-16T20:36:55.270787+0000","last_unstale":"2023-10-16T20:37:37.435199+0000","last_undegraded":"2023-10-16T20:37:37.435199+0000","last_fullsized":"2023-10-16T20:37:37.435199+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:07.944778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402678+0000","last_change":"2023-10-16T20:36:55.271632+0000","last_active":"2023-10-16T20:37:37.402678+0000","last_peered":"2023-10-16T20:37:37.402678+0000","last_clean":"2023-10-16T20:37:37.402678+0000","last_became_active":"2023-10-16T20:36:55.271373+0000","last_became_peered":"2023-10-16T20:36:55.271373+0000","last_unstale":"2023-10-16T20:37:37.402678+0000","last_undegraded":"2023-10-16T20:37:37.402678+0000","last_fullsized":"2023-10-16T20:37:37.402678+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:07.458540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466741+0000","last_change":"2023-10-16T20:36:55.291343+0000","last_active":"2023-10-16T20:37:37.466741+0000","last_peered":"2023-10-16T20:37:37.466741+0000","last_clean":"2023-10-16T20:37:37.466741+0000","last_became_active":"2023-10-16T20:36:55.291016+0000","last_became_peered":"2023-10-16T20:36:55.291016+0000","last_unstale":"2023-10-16T20:37:37.466741+0000","last_undegraded":"2023-10-16T20:37:37.466741+0000","last_fullsized":"2023-10-16T20:37:37.466741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:13:40.287782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5457,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":10892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509475,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343936,"statfs":{"total":96636764160,"available":96608190464,"internally_reserved":0,"allocated":905216,"data_stored":715195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.74299999999999999,"15min":0.748},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":5.2240000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.75700000000000001,"15min":0.753},"min":{"1min":0.58099999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94599999999999995,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":8.782}]},{"osd":1,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.74299999999999999,"15min":0.73999999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.91600000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.52100000000000002,"5min":0.497,"15min":0.497},"max":{"1min":0.91500000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":4.7300000000000004}]},{"osd":2,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.73199999999999998,"15min":0.72599999999999998},"min":{"1min":0.624,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":5.2629999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.748,"15min":0.745},"min":{"1min":0.60699999999999998,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.89700000000000002,"5min":1.083,"15min":1.083},"last":5.1589999999999998}]},{"osd":3,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.77600000000000002,"15min":0.77200000000000002},"min":{"1min":0.62,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0069999999999999,"5min":1.234,"15min":1.234},"last":4.7460000000000004},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.93200000000000005,"5min":1.024,"15min":1.024},"last":4.7770000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63100000000000001,"15min":0.63300000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.92500000000000004,"5min":1.3,"15min":1.3},"last":0.501},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60499999999999998,"15min":0.60999999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76500000000000001,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.53500000000000003}]},{"osd":5,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.47599999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.78600000000000003,"15min":0.79500000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.006,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.68000000000000005}]},{"osd":6,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.749,"15min":0.755},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.97399999999999998,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.70399999999999996,"15min":0.70599999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.88300000000000001,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.70599999999999996}]}]},{"osd":6,"up_from":15,"seq":64424509475,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":128086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.76800000000000002,"15min":0.77100000000000002},"min":{"1min":0.41899999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88,"5min":1.131,"15min":1.131},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.748,"15min":0.75800000000000001},"min":{"1min":0.48899999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.875,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.624}]},{"osd":1,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.751,"15min":0.76300000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.81799999999999995,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.77000000000000002,"15min":0.77600000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.90400000000000003,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.58899999999999997}]},{"osd":2,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.78000000000000003,"15min":0.79200000000000004},"min":{"1min":0.48599999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.90300000000000002,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.73899999999999999,"15min":0.746},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.86899999999999999,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.71799999999999997}]},{"osd":3,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.84999999999999998,"15min":0.87},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.90600000000000003,"5min":1.611,"15min":1.611},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.749,"5min":0.80900000000000005,"15min":0.81899999999999995},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.89700000000000002,"5min":1.163,"15min":1.163},"last":0.56499999999999995}]},{"osd":4,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.66700000000000004,"15min":0.67900000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.79500000000000004,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65700000000000003,"15min":0.66700000000000004},"min":{"1min":0.49299999999999999,"5min":0.435,"15min":0.435},"max":{"1min":0.69499999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.73899999999999999}]},{"osd":5,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86199999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.748},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69899999999999995,"15min":0.70599999999999996},"min":{"1min":0.54900000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.82399999999999995,"5min":1.016,"15min":1.016},"last":0.69999999999999996}]},{"osd":7,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.748,"15min":0.754},"min":{"1min":0.52200000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.93100000000000005,"5min":1.292,"15min":1.292},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.75900000000000001,"15min":0.77500000000000002},"min":{"1min":0.52800000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.92100000000000004,"5min":1.569,"15min":1.569},"last":0.77000000000000002}]}]},{"osd":5,"up_from":15,"seq":64424509477,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":311296,"data_stored":124335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.70099999999999996,"15min":0.69599999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.75800000000000001}]},{"osd":1,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70699999999999996,"15min":0.70499999999999996},"min":{"1min":0.52100000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.71499999999999997,"15min":0.70899999999999996},"min":{"1min":0.57399999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.70499999999999996}]},{"osd":2,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.72199999999999998,"15min":0.73199999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92300000000000004,"5min":1.022,"15min":1.022},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.74299999999999999,"15min":0.73699999999999999},"min":{"1min":0.60799999999999998,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.91700000000000004,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.79100000000000004}]},{"osd":3,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79200000000000004,"15min":0.79100000000000004},"min":{"1min":0.66700000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81000000000000005,"15min":0.81599999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.82599999999999996}]},{"osd":4,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65100000000000002,"15min":0.65900000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.90300000000000002,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.73399999999999999,"15min":0.75800000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.80500000000000005,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.60199999999999998}]},{"osd":6,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.70999999999999996,"15min":0.72799999999999998},"min":{"1min":0.45500000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.79700000000000004,"5min":1.069,"15min":1.069},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.495,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.038,"5min":1.264,"15min":1.264},"last":0.64000000000000001}]},{"osd":7,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.78000000000000003,"15min":0.79600000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.89200000000000002,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.79100000000000004,"15min":0.80700000000000005},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.163,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.73799999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509475,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":124371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.72199999999999998,"15min":0.72399999999999998},"min":{"1min":0.45200000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.872,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.59599999999999997,"5min":0.5,"15min":0.5},"max":{"1min":0.88200000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.55200000000000005}]},{"osd":1,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.57499999999999996,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92500000000000004,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.748,"15min":0.748},"min":{"1min":0.54500000000000004,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.66700000000000004}]},{"osd":2,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.76400000000000001,"15min":0.76200000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77700000000000002,"15min":0.77800000000000002},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.876}]},{"osd":3,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.747,"15min":0.73899999999999999},"min":{"1min":0.58899999999999997,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.78200000000000003,"15min":0.78600000000000003},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.86399999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.79000000000000004}]},{"osd":5,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.63600000000000001,"15min":0.64800000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.75600000000000001,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.44800000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95899999999999996,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.84699999999999998}]},{"osd":6,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.66500000000000004,"15min":0.66700000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73499999999999999,"15min":0.73799999999999999},"min":{"1min":0.495,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.90600000000000003}]},{"osd":7,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.747,"15min":0.751},"min":{"1min":0.52000000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.94599999999999995,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.79300000000000004,"15min":0.80300000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.96599999999999997,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.89200000000000002}]}]},{"osd":3,"up_from":15,"seq":64424509475,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":323584,"data_stored":128120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.46999999999999997,"15min":0.47199999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.69499999999999995,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.48699999999999999}]},{"osd":1,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.41499999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71699999999999997,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.52900000000000003}]},{"osd":2,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.53800000000000003,"15min":0.54400000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.67600000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.626,"15min":0.61699999999999999},"min":{"1min":0.51600000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.56899999999999995}]},{"osd":4,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.80200000000000005,"15min":0.80000000000000004},"min":{"1min":0.56200000000000006,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.054,"5min":1.181,"15min":1.181},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.77300000000000002,"15min":0.76500000000000001},"min":{"1min":0.57499999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.052,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.81999999999999995}]},{"osd":5,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.85999999999999999,"15min":0.873},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.008,"5min":1.544,"15min":1.544},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.323,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.83199999999999996}]},{"osd":6,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.85499999999999998,"15min":0.85799999999999998},"min":{"1min":0.64900000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.3480000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.72599999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.80600000000000005,"15min":0.80000000000000004},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0880000000000001,"5min":1.139,"15min":1.139},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.0029999999999999,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.79600000000000004}]}]},{"osd":2,"up_from":15,"seq":64424509475,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":124245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.47899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71099999999999997,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.434}]},{"osd":1,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77400000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.373},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55100000000000005,"15min":0.54900000000000004},"min":{"1min":0.38700000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69399999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.40799999999999997}]},{"osd":3,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54400000000000004,"15min":0.53600000000000003},"min":{"1min":0.41299999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.504},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76700000000000002,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.46899999999999997}]},{"osd":4,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.82299999999999995,"15min":0.82899999999999996},"min":{"1min":0.60799999999999998,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.97499999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.64600000000000002,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.98299999999999998,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.56200000000000006}]},{"osd":5,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.60999999999999999,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.96999999999999997,"5min":1.444,"15min":1.444},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.83899999999999997,"15min":0.84199999999999997},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":0.998,"5min":1.585,"15min":1.585},"last":0.59099999999999997}]},{"osd":6,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.84799999999999998,"15min":0.85199999999999998},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.1020000000000001,"5min":1.335,"15min":1.335},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.871,"15min":0.88200000000000001},"min":{"1min":0.68999999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":0.95099999999999996,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.61799999999999999}]},{"osd":7,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.86499999999999999,"15min":0.875},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.046,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.88300000000000001,"15min":0.89300000000000002},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.0589999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.66900000000000004}]}]},{"osd":1,"up_from":15,"seq":64424509475,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":315392,"data_stored":124405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.46400000000000002,"15min":0.46700000000000003},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.64700000000000002,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.32600000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.69899999999999995,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.52400000000000002}]},{"osd":2,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.375,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82499999999999996,"5min":1.046,"15min":1.046},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.70699999999999996,"5min":1.115,"15min":1.115},"last":0.40999999999999998}]},{"osd":3,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60599999999999998,"15min":0.60899999999999999},"min":{"1min":0.40799999999999997,"5min":0.312,"15min":0.312},"max":{"1min":0.78500000000000003,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.60799999999999998,"15min":0.61799999999999999},"min":{"1min":0.41499999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.80400000000000005,"5min":1.431,"15min":1.431},"last":0.60899999999999999}]},{"osd":4,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.83399999999999996,"15min":0.83599999999999997},"min":{"1min":0.59199999999999997,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.242,"15min":1.242},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.81299999999999994,"15min":0.81100000000000005},"min":{"1min":0.60099999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.054,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.80800000000000005,"15min":0.81999999999999995},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90800000000000003,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.871,"15min":0.872},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.1160000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.51300000000000001}]},{"osd":6,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.874,"15min":0.88100000000000001},"min":{"1min":0.65300000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.123,"5min":2.004,"15min":2.004},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.876,"15min":0.88300000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.0880000000000001,"5min":1.635,"15min":1.635},"last":0.54800000000000004}]},{"osd":7,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.878,"15min":0.89200000000000002},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.99099999999999999,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.871,"5min":0.90700000000000003,"15min":0.91300000000000003},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.0760000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.56799999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343928,"statfs":{"total":96636764160,"available":96608182272,"internally_reserved":0,"allocated":913408,"data_stored":715251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.439,"15min":0.438},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.495,"15min":0.496},"min":{"1min":0.374,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.58999999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.38600000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52300000000000002,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54100000000000004,"15min":0.54600000000000004},"min":{"1min":0.40400000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.68899999999999995,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.39700000000000002}]},{"osd":3,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54800000000000004,"15min":0.54500000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73399999999999999,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.56499999999999995,"5min":0.497,"15min":0.497},"max":{"1min":0.84999999999999998,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.55100000000000005}]},{"osd":4,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.83799999999999997,"15min":0.82999999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0600000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.85999999999999999,"15min":0.84899999999999998},"min":{"1min":0.61399999999999999,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.80800000000000005}]},{"osd":5,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.80800000000000005,"15min":0.80300000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.871},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.75700000000000001}]},{"osd":6,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.86299999999999999,"15min":0.85899999999999999},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88400000000000001,"15min":0.88300000000000001},"min":{"1min":0.60899999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.77800000000000002}]},{"osd":7,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.64900000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.83499999999999996,"15min":0.82299999999999995},"min":{"1min":0.66500000000000004,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T20:39:25.158 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T20:39:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.325+0000 7fd942a05640 1 Processor -- start 2023-10-16T20:39:25.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.325+0000 7fd942a05640 1 -- start start 2023-10-16T20:39:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.325+0000 7fd942a05640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd93c0cece0 0x7fd93c0cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.325+0000 7fd942a05640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93c153b50 con 0x7fd93c0c9bb0 2023-10-16T20:39:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.325+0000 7fd942a05640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93c0cb2f0 con 0x7fd93c153770 2023-10-16T20:39:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.325+0000 7fd942a05640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd93c0cc250 con 0x7fd93c0cece0 2023-10-16T20:39:25.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.325+0000 7fd93b7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd93c0c9bb0 0x7fd93c0c3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:37052/0 (socket says 172.21.15.88:37052) 2023-10-16T20:39:25.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.325+0000 7fd93b7fe640 1 -- 172.21.15.88:0/2520636965 learned_addr learned my addr 172.21.15.88:0/2520636965 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.325+0000 7fd940fa2640 1 --2- 172.21.15.88:0/2520636965 >> v2:172.21.15.191:3300/0 conn(0x7fd93c0cece0 0x7fd93c0cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.329+0000 7fd93affd640 1 -- 172.21.15.88:0/2520636965 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202105230 0 0) 0x7fd93c153b50 con 0x7fd93c0c9bb0 2023-10-16T20:39:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.329+0000 7fd93affd640 1 -- 172.21.15.88:0/2520636965 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd924003530 con 0x7fd93c0c9bb0 2023-10-16T20:39:25.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.329+0000 7fd93affd640 1 -- 172.21.15.88:0/2520636965 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741409331 0 0) 0x7fd93c0cb2f0 con 0x7fd93c153770 2023-10-16T20:39:25.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.329+0000 7fd93affd640 1 -- 172.21.15.88:0/2520636965 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd93c153b50 con 0x7fd93c153770 2023-10-16T20:39:25.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.329+0000 7fd940fa2640 1 -- 172.21.15.88:0/2520636965 >> v1:172.21.15.191:6789/0 conn(0x7fd93c153770 legacy=0x7fd93c159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:25.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.329+0000 7fd93affd640 1 -- 172.21.15.88:0/2520636965 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2538262901 0 0) 0x7fd924003530 con 0x7fd93c0c9bb0 2023-10-16T20:39:25.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.329+0000 7fd940fa2640 1 -- 172.21.15.88:0/2520636965 >> v1:172.21.15.88:6789/0 conn(0x7fd93c0c9bb0 legacy=0x7fd93c0c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:25.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.329+0000 7fd940fa2640 1 -- 172.21.15.88:0/2520636965 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c0cb5d0 con 0x7fd93c0cece0 2023-10-16T20:39:25.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.329+0000 7fd940fa2640 1 --2- 172.21.15.88:0/2520636965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93c0cece0 0x7fd93c0cf130 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd934009d90 tx=0x7fd93401c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a56fda1e0eff5f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:25.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.329+0000 7fd93affd640 1 -- 172.21.15.88:0/2520636965 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd930001680 con 0x7fd93c0cece0 2023-10-16T20:39:25.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.329+0000 7fd93affd640 1 -- 172.21.15.88:0/2520636965 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9340a55c0 con 0x7fd93c0cece0 2023-10-16T20:39:25.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.329+0000 7fd93affd640 1 -- 172.21.15.88:0/2520636965 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9340a5950 con 0x7fd93c0cece0 2023-10-16T20:39:25.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd942a05640 1 -- 172.21.15.88:0/2520636965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93c0cece0 msgr2=0x7fd93c0cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:25.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd942a05640 1 --2- 172.21.15.88:0/2520636965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93c0cece0 0x7fd93c0cf130 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd934009d90 tx=0x7fd93401c870 comp rx=0 tx=0).stop 2023-10-16T20:39:25.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd942a05640 1 -- 172.21.15.88:0/2520636965 shutdown_connections 2023-10-16T20:39:25.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd942a05640 1 --2- 172.21.15.88:0/2520636965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93c0cece0 0x7fd93c0cf130 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd942a05640 1 -- 172.21.15.88:0/2520636965 >> 172.21.15.88:0/2520636965 conn(0x7fd93c0715c0 msgr2=0x7fd93c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:25.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd942a05640 1 -- 172.21.15.88:0/2520636965 shutdown_connections 2023-10-16T20:39:25.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd942a05640 1 -- 172.21.15.88:0/2520636965 wait complete. 2023-10-16T20:39:25.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd942a05640 1 Processor -- start 2023-10-16T20:39:25.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd942a05640 1 -- start start 2023-10-16T20:39:25.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd942a05640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd93c0c9bb0 0x7fd93c1f2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:25.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd942a05640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93c0cece0 0x7fd93c14c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:25.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd942a05640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd93c153770 0x7fd93c143530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:25.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd942a05640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c1f2570 con 0x7fd93c0c9bb0 2023-10-16T20:39:25.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd942a05640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c14d620 con 0x7fd93c0cece0 2023-10-16T20:39:25.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd942a05640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c1f15e0 con 0x7fd93c153770 2023-10-16T20:39:25.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd93bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd93c0c9bb0 0x7fd93c1f2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:25.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd93bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd93c0c9bb0 0x7fd93c1f2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56722/0 (socket says 172.21.15.88:56722) 2023-10-16T20:39:25.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd93bfff640 1 -- 172.21.15.88:0/3311213110 learned_addr learned my addr 172.21.15.88:0/3311213110 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:25.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd93b7fe640 1 --2- 172.21.15.88:0/3311213110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93c0cece0 0x7fd93c14c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:25.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd940fa2640 1 --2- 172.21.15.88:0/3311213110 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd93c153770 0x7fd93c143530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:25.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd93bfff640 1 -- 172.21.15.88:0/3311213110 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd93c153770 msgr2=0x7fd93c143530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:25.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd93bfff640 1 --2- 172.21.15.88:0/3311213110 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd93c153770 0x7fd93c143530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd93bfff640 1 -- 172.21.15.88:0/3311213110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93c0cece0 msgr2=0x7fd93c14c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:25.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd93bfff640 1 --2- 172.21.15.88:0/3311213110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93c0cece0 0x7fd93c14c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.333+0000 7fd93bfff640 1 -- 172.21.15.88:0/3311213110 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd934009a70 con 0x7fd93c0c9bb0 2023-10-16T20:39:25.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.337+0000 7fd93bfff640 1 --2- 172.21.15.88:0/3311213110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd93c0c9bb0 0x7fd93c1f2030 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00c450 tx=0x7fd92c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:25.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.337+0000 7fd938ff9640 1 -- 172.21.15.88:0/3311213110 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c0052d0 con 0x7fd93c0c9bb0 2023-10-16T20:39:25.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.337+0000 7fd938ff9640 1 -- 172.21.15.88:0/3311213110 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92c005430 con 0x7fd93c0c9bb0 2023-10-16T20:39:25.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.337+0000 7fd942a05640 1 -- 172.21.15.88:0/3311213110 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd93c15d370 con 0x7fd93c0c9bb0 2023-10-16T20:39:25.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.337+0000 7fd938ff9640 1 -- 172.21.15.88:0/3311213110 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c003870 con 0x7fd93c0c9bb0 2023-10-16T20:39:25.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.337+0000 7fd942a05640 1 -- 172.21.15.88:0/3311213110 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd93c1f1830 con 0x7fd93c0c9bb0 2023-10-16T20:39:25.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.337+0000 7fd942a05640 1 -- 172.21.15.88:0/3311213110 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93c074700 con 0x7fd93c0c9bb0 2023-10-16T20:39:25.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.337+0000 7fd938ff9640 1 -- 172.21.15.88:0/3311213110 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd92c0039d0 con 0x7fd93c0c9bb0 2023-10-16T20:39:25.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.337+0000 7fd938ff9640 1 --2- 172.21.15.88:0/3311213110 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd9280664a0 0x7fd928068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:25.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.337+0000 7fd938ff9640 1 -- 172.21.15.88:0/3311213110 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd92c083c90 con 0x7fd93c0c9bb0 2023-10-16T20:39:25.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.337+0000 7fd93b7fe640 1 --2- 172.21.15.88:0/3311213110 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd9280664a0 0x7fd928068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:25.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.337+0000 7fd93b7fe640 1 --2- 172.21.15.88:0/3311213110 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd9280664a0 0x7fd928068960 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd93c151860 tx=0x7fd930001a60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:25.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.341+0000 7fd938ff9640 1 -- 172.21.15.88:0/3311213110 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd92c050d70 con 0x7fd93c0c9bb0 2023-10-16T20:39:25.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.489+0000 7fd942a05640 1 -- 172.21.15.88:0/3311213110 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd93c0cbe70 con 0x7fd9280664a0 2023-10-16T20:39:25.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.497+0000 7fd938ff9640 1 -- 172.21.15.88:0/3311213110 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+369837 (secure 0 0 0) 0x7fd930006e10 con 0x7fd9280664a0 2023-10-16T20:39:25.509 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:25.511 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-16T20:39:25.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.501+0000 7fd942a05640 1 -- 172.21.15.88:0/3311213110 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd9280664a0 msgr2=0x7fd928068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:25.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.501+0000 7fd942a05640 1 --2- 172.21.15.88:0/3311213110 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd9280664a0 0x7fd928068960 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd93c151860 tx=0x7fd930001a60 comp rx=0 tx=0).stop 2023-10-16T20:39:25.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.501+0000 7fd942a05640 1 -- 172.21.15.88:0/3311213110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd93c0c9bb0 msgr2=0x7fd93c1f2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:25.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.501+0000 7fd942a05640 1 --2- 172.21.15.88:0/3311213110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd93c0c9bb0 0x7fd93c1f2030 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00c450 tx=0x7fd92c00c910 comp rx=0 tx=0).stop 2023-10-16T20:39:25.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.501+0000 7fd942a05640 1 -- 172.21.15.88:0/3311213110 shutdown_connections 2023-10-16T20:39:25.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.501+0000 7fd942a05640 1 --2- 172.21.15.88:0/3311213110 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd9280664a0 0x7fd928068960 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.501+0000 7fd942a05640 1 --2- 172.21.15.88:0/3311213110 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd93c153770 0x7fd93c143530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.501+0000 7fd942a05640 1 --2- 172.21.15.88:0/3311213110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd93c0cece0 0x7fd93c14c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.501+0000 7fd942a05640 1 --2- 172.21.15.88:0/3311213110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd93c0c9bb0 0x7fd93c1f2030 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.501+0000 7fd942a05640 1 -- 172.21.15.88:0/3311213110 >> 172.21.15.88:0/3311213110 conn(0x7fd93c0715c0 msgr2=0x7fd93c0cf0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:25.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.501+0000 7fd942a05640 1 -- 172.21.15.88:0/3311213110 shutdown_connections 2023-10-16T20:39:25.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.501+0000 7fd942a05640 1 -- 172.21.15.88:0/3311213110 wait complete. 2023-10-16T20:39:25.529 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2023-10-16T20:39:23.628355+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595825,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":243,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":191,"ondisk_log_size":191,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219788,"kb_used_data":3628,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754932,"statfs":{"total":773094113280,"available":772869050368,"internally_reserved":0,"allocated":3715072,"data_stored":2184008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3047,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002417"},"pg_stats":[{"pgid":"2.1f","version":"26'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435615+0000","last_change":"2023-10-16T20:36:54.763744+0000","last_active":"2023-10-16T20:37:37.435615+0000","last_peered":"2023-10-16T20:37:37.435615+0000","last_clean":"2023-10-16T20:37:37.435615+0000","last_became_active":"2023-10-16T20:36:54.763457+0000","last_became_peered":"2023-10-16T20:36:54.763457+0000","last_unstale":"2023-10-16T20:37:37.435615+0000","last_undegraded":"2023-10-16T20:37:37.435615+0000","last_fullsized":"2023-10-16T20:37:37.435615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:05.774485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465462+0000","last_change":"2023-10-16T20:36:55.302740+0000","last_active":"2023-10-16T20:37:37.465462+0000","last_peered":"2023-10-16T20:37:37.465462+0000","last_clean":"2023-10-16T20:37:37.465462+0000","last_became_active":"2023-10-16T20:36:55.302519+0000","last_became_peered":"2023-10-16T20:36:55.302519+0000","last_unstale":"2023-10-16T20:37:37.465462+0000","last_undegraded":"2023-10-16T20:37:37.465462+0000","last_fullsized":"2023-10-16T20:37:37.465462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:36.762521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404215+0000","last_change":"2023-10-16T20:36:59.466679+0000","last_active":"2023-10-16T20:37:36.404215+0000","last_peered":"2023-10-16T20:37:36.404215+0000","last_clean":"2023-10-16T20:37:36.404215+0000","last_became_active":"2023-10-16T20:36:59.466306+0000","last_became_peered":"2023-10-16T20:36:59.466306+0000","last_unstale":"2023-10-16T20:37:36.404215+0000","last_undegraded":"2023-10-16T20:37:36.404215+0000","last_fullsized":"2023-10-16T20:37:36.404215+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:14.874622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"26'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465585+0000","last_change":"2023-10-16T20:36:54.759491+0000","last_active":"2023-10-16T20:37:37.465585+0000","last_peered":"2023-10-16T20:37:37.465585+0000","last_clean":"2023-10-16T20:37:37.465585+0000","last_became_active":"2023-10-16T20:36:54.759177+0000","last_became_peered":"2023-10-16T20:36:54.759177+0000","last_unstale":"2023-10-16T20:37:37.465585+0000","last_undegraded":"2023-10-16T20:37:37.465585+0000","last_fullsized":"2023-10-16T20:37:37.465585+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:43:36.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435464+0000","last_change":"2023-10-16T20:36:55.291745+0000","last_active":"2023-10-16T20:37:37.435464+0000","last_peered":"2023-10-16T20:37:37.435464+0000","last_clean":"2023-10-16T20:37:37.435464+0000","last_became_active":"2023-10-16T20:36:55.291458+0000","last_became_peered":"2023-10-16T20:36:55.291458+0000","last_unstale":"2023-10-16T20:37:37.435464+0000","last_undegraded":"2023-10-16T20:37:37.435464+0000","last_fullsized":"2023-10-16T20:37:37.435464+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:01.291335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.401937+0000","last_change":"2023-10-16T20:36:59.303600+0000","last_active":"2023-10-16T20:37:36.401937+0000","last_peered":"2023-10-16T20:37:36.401937+0000","last_clean":"2023-10-16T20:37:36.401937+0000","last_became_active":"2023-10-16T20:36:59.302968+0000","last_became_peered":"2023-10-16T20:36:59.302968+0000","last_unstale":"2023-10-16T20:37:36.401937+0000","last_undegraded":"2023-10-16T20:37:36.401937+0000","last_fullsized":"2023-10-16T20:37:36.401937+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:11:28.322824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"27'2","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.412675+0000","last_change":"2023-10-16T20:36:54.762600+0000","last_active":"2023-10-16T20:37:36.412675+0000","last_peered":"2023-10-16T20:37:36.412675+0000","last_clean":"2023-10-16T20:37:36.412675+0000","last_became_active":"2023-10-16T20:36:54.762278+0000","last_became_peered":"2023-10-16T20:36:54.762278+0000","last_unstale":"2023-10-16T20:37:36.412675+0000","last_undegraded":"2023-10-16T20:37:36.412675+0000","last_fullsized":"2023-10-16T20:37:36.412675+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:46:03.836415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404637+0000","last_change":"2023-10-16T20:37:37.403448+0000","last_active":"2023-10-16T20:37:37.404637+0000","last_peered":"2023-10-16T20:37:37.404637+0000","last_clean":"2023-10-16T20:37:37.404637+0000","last_became_active":"2023-10-16T20:37:37.402987+0000","last_became_peered":"2023-10-16T20:37:37.402987+0000","last_unstale":"2023-10-16T20:37:37.404637+0000","last_undegraded":"2023-10-16T20:37:37.404637+0000","last_fullsized":"2023-10-16T20:37:37.404637+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:39:05.729682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.402113+0000","last_change":"2023-10-16T20:36:59.304684+0000","last_active":"2023-10-16T20:37:36.402113+0000","last_peered":"2023-10-16T20:37:36.402113+0000","last_clean":"2023-10-16T20:37:36.402113+0000","last_became_active":"2023-10-16T20:36:59.303805+0000","last_became_peered":"2023-10-16T20:36:59.303805+0000","last_unstale":"2023-10-16T20:37:36.402113+0000","last_undegraded":"2023-10-16T20:37:36.402113+0000","last_fullsized":"2023-10-16T20:37:36.402113+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:39:54.406763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"27'2","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.412121+0000","last_change":"2023-10-16T20:36:54.749849+0000","last_active":"2023-10-16T20:37:36.412121+0000","last_peered":"2023-10-16T20:37:36.412121+0000","last_clean":"2023-10-16T20:37:36.412121+0000","last_became_active":"2023-10-16T20:36:54.749316+0000","last_became_peered":"2023-10-16T20:36:54.749316+0000","last_unstale":"2023-10-16T20:37:36.412121+0000","last_undegraded":"2023-10-16T20:37:36.412121+0000","last_fullsized":"2023-10-16T20:37:36.412121+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:49:29.077126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.409911+0000","last_change":"2023-10-16T20:36:55.302838+0000","last_active":"2023-10-16T20:37:36.409911+0000","last_peered":"2023-10-16T20:37:36.409911+0000","last_clean":"2023-10-16T20:37:36.409911+0000","last_became_active":"2023-10-16T20:36:55.302387+0000","last_became_peered":"2023-10-16T20:36:55.302387+0000","last_unstale":"2023-10-16T20:37:36.409911+0000","last_undegraded":"2023-10-16T20:37:36.409911+0000","last_fullsized":"2023-10-16T20:37:36.409911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:55:56.517950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.402135+0000","last_change":"2023-10-16T20:36:59.305750+0000","last_active":"2023-10-16T20:37:36.402135+0000","last_peered":"2023-10-16T20:37:36.402135+0000","last_clean":"2023-10-16T20:37:36.402135+0000","last_became_active":"2023-10-16T20:36:59.305228+0000","last_became_peered":"2023-10-16T20:36:59.305228+0000","last_unstale":"2023-10-16T20:37:36.402135+0000","last_undegraded":"2023-10-16T20:37:36.402135+0000","last_fullsized":"2023-10-16T20:37:36.402135+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:00:18.194057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407634+0000","last_change":"2023-10-16T20:36:55.306712+0000","last_active":"2023-10-16T20:37:37.407634+0000","last_peered":"2023-10-16T20:37:37.407634+0000","last_clean":"2023-10-16T20:37:37.407634+0000","last_became_active":"2023-10-16T20:36:55.306427+0000","last_became_peered":"2023-10-16T20:36:55.306427+0000","last_unstale":"2023-10-16T20:37:37.407634+0000","last_undegraded":"2023-10-16T20:37:37.407634+0000","last_fullsized":"2023-10-16T20:37:37.407634+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:42:03.769890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.410476+0000","last_change":"2023-10-16T20:36:54.768375+0000","last_active":"2023-10-16T20:39:22.410476+0000","last_peered":"2023-10-16T20:39:22.410476+0000","last_clean":"2023-10-16T20:39:22.410476+0000","last_became_active":"2023-10-16T20:36:54.767996+0000","last_became_peered":"2023-10-16T20:36:54.767996+0000","last_unstale":"2023-10-16T20:39:22.410476+0000","last_undegraded":"2023-10-16T20:39:22.410476+0000","last_fullsized":"2023-10-16T20:39:22.410476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:55.303942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404253+0000","last_change":"2023-10-16T20:36:59.307130+0000","last_active":"2023-10-16T20:37:37.404253+0000","last_peered":"2023-10-16T20:37:37.404253+0000","last_clean":"2023-10-16T20:37:37.404253+0000","last_became_active":"2023-10-16T20:36:59.306640+0000","last_became_peered":"2023-10-16T20:36:59.306640+0000","last_unstale":"2023-10-16T20:37:37.404253+0000","last_undegraded":"2023-10-16T20:37:37.404253+0000","last_fullsized":"2023-10-16T20:37:37.404253+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:19:18.664630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465074+0000","last_change":"2023-10-16T20:36:55.275436+0000","last_active":"2023-10-16T20:37:37.465074+0000","last_peered":"2023-10-16T20:37:37.465074+0000","last_clean":"2023-10-16T20:37:37.465074+0000","last_became_active":"2023-10-16T20:36:55.275085+0000","last_became_peered":"2023-10-16T20:36:55.275085+0000","last_unstale":"2023-10-16T20:37:37.465074+0000","last_undegraded":"2023-10-16T20:37:37.465074+0000","last_fullsized":"2023-10-16T20:37:37.465074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:37:37.814511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402406+0000","last_change":"2023-10-16T20:36:54.756174+0000","last_active":"2023-10-16T20:37:37.402406+0000","last_peered":"2023-10-16T20:37:37.402406+0000","last_clean":"2023-10-16T20:37:37.402406+0000","last_became_active":"2023-10-16T20:36:54.755872+0000","last_became_peered":"2023-10-16T20:36:54.755872+0000","last_unstale":"2023-10-16T20:37:37.402406+0000","last_undegraded":"2023-10-16T20:37:37.402406+0000","last_fullsized":"2023-10-16T20:37:37.402406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:32.741671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403750+0000","last_change":"2023-10-16T20:36:59.303855+0000","last_active":"2023-10-16T20:37:37.403750+0000","last_peered":"2023-10-16T20:37:37.403750+0000","last_clean":"2023-10-16T20:37:37.403750+0000","last_became_active":"2023-10-16T20:36:59.302919+0000","last_became_peered":"2023-10-16T20:36:59.302919+0000","last_unstale":"2023-10-16T20:37:37.403750+0000","last_undegraded":"2023-10-16T20:37:37.403750+0000","last_fullsized":"2023-10-16T20:37:37.403750+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:38:02.591237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464820+0000","last_change":"2023-10-16T20:36:55.270717+0000","last_active":"2023-10-16T20:37:37.464820+0000","last_peered":"2023-10-16T20:37:37.464820+0000","last_clean":"2023-10-16T20:37:37.464820+0000","last_became_active":"2023-10-16T20:36:55.270455+0000","last_became_peered":"2023-10-16T20:36:55.270455+0000","last_unstale":"2023-10-16T20:37:37.464820+0000","last_undegraded":"2023-10-16T20:37:37.464820+0000","last_fullsized":"2023-10-16T20:37:37.464820+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:16:44.485783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.406832+0000","last_change":"2023-10-16T20:36:54.744128+0000","last_active":"2023-10-16T20:39:22.406832+0000","last_peered":"2023-10-16T20:39:22.406832+0000","last_clean":"2023-10-16T20:39:22.406832+0000","last_became_active":"2023-10-16T20:36:54.743805+0000","last_became_peered":"2023-10-16T20:36:54.743805+0000","last_unstale":"2023-10-16T20:39:22.406832+0000","last_undegraded":"2023-10-16T20:39:22.406832+0000","last_fullsized":"2023-10-16T20:39:22.406832+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:54:35.819820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402668+0000","last_change":"2023-10-16T20:36:59.465650+0000","last_active":"2023-10-16T20:37:37.402668+0000","last_peered":"2023-10-16T20:37:37.402668+0000","last_clean":"2023-10-16T20:37:37.402668+0000","last_became_active":"2023-10-16T20:36:59.465172+0000","last_became_peered":"2023-10-16T20:36:59.465172+0000","last_unstale":"2023-10-16T20:37:37.402668+0000","last_undegraded":"2023-10-16T20:37:37.402668+0000","last_fullsized":"2023-10-16T20:37:37.402668+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:12.845517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403561+0000","last_change":"2023-10-16T20:36:55.278974+0000","last_active":"2023-10-16T20:37:37.403561+0000","last_peered":"2023-10-16T20:37:37.403561+0000","last_clean":"2023-10-16T20:37:37.403561+0000","last_became_active":"2023-10-16T20:36:55.278568+0000","last_became_peered":"2023-10-16T20:36:55.278568+0000","last_unstale":"2023-10-16T20:37:37.403561+0000","last_undegraded":"2023-10-16T20:37:37.403561+0000","last_fullsized":"2023-10-16T20:37:37.403561+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:26.756114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.407659+0000","last_change":"2023-10-16T20:36:54.758778+0000","last_active":"2023-10-16T20:39:22.407659+0000","last_peered":"2023-10-16T20:39:22.407659+0000","last_clean":"2023-10-16T20:39:22.407659+0000","last_became_active":"2023-10-16T20:36:54.758496+0000","last_became_peered":"2023-10-16T20:36:54.758496+0000","last_unstale":"2023-10-16T20:39:22.407659+0000","last_undegraded":"2023-10-16T20:39:22.407659+0000","last_fullsized":"2023-10-16T20:39:22.407659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:36:00.515106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404277+0000","last_change":"2023-10-16T20:36:59.309382+0000","last_active":"2023-10-16T20:37:37.404277+0000","last_peered":"2023-10-16T20:37:37.404277+0000","last_clean":"2023-10-16T20:37:37.404277+0000","last_became_active":"2023-10-16T20:36:59.309068+0000","last_became_peered":"2023-10-16T20:36:59.309068+0000","last_unstale":"2023-10-16T20:37:37.404277+0000","last_undegraded":"2023-10-16T20:37:37.404277+0000","last_fullsized":"2023-10-16T20:37:37.404277+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:46:02.601985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435429+0000","last_change":"2023-10-16T20:36:55.273089+0000","last_active":"2023-10-16T20:37:37.435429+0000","last_peered":"2023-10-16T20:37:37.435429+0000","last_clean":"2023-10-16T20:37:37.435429+0000","last_became_active":"2023-10-16T20:36:55.272840+0000","last_became_peered":"2023-10-16T20:36:55.272840+0000","last_unstale":"2023-10-16T20:37:37.435429+0000","last_undegraded":"2023-10-16T20:37:37.435429+0000","last_fullsized":"2023-10-16T20:37:37.435429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:37.330924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.409085+0000","last_change":"2023-10-16T20:36:54.760681+0000","last_active":"2023-10-16T20:39:22.409085+0000","last_peered":"2023-10-16T20:39:22.409085+0000","last_clean":"2023-10-16T20:39:22.409085+0000","last_became_active":"2023-10-16T20:36:54.760397+0000","last_became_peered":"2023-10-16T20:36:54.760397+0000","last_unstale":"2023-10-16T20:39:22.409085+0000","last_undegraded":"2023-10-16T20:39:22.409085+0000","last_fullsized":"2023-10-16T20:39:22.409085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:47:16.510382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403318+0000","last_change":"2023-10-16T20:36:59.465626+0000","last_active":"2023-10-16T20:37:37.403318+0000","last_peered":"2023-10-16T20:37:37.403318+0000","last_clean":"2023-10-16T20:37:37.403318+0000","last_became_active":"2023-10-16T20:36:59.465152+0000","last_became_peered":"2023-10-16T20:36:59.465152+0000","last_unstale":"2023-10-16T20:37:37.403318+0000","last_undegraded":"2023-10-16T20:37:37.403318+0000","last_fullsized":"2023-10-16T20:37:37.403318+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:27:51.088333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465673+0000","last_change":"2023-10-16T20:36:55.285796+0000","last_active":"2023-10-16T20:37:37.465673+0000","last_peered":"2023-10-16T20:37:37.465673+0000","last_clean":"2023-10-16T20:37:37.465673+0000","last_became_active":"2023-10-16T20:36:55.285566+0000","last_became_peered":"2023-10-16T20:36:55.285566+0000","last_unstale":"2023-10-16T20:37:37.465673+0000","last_undegraded":"2023-10-16T20:37:37.465673+0000","last_fullsized":"2023-10-16T20:37:37.465673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:22:21.950705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"27'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402756+0000","last_change":"2023-10-16T20:36:54.758062+0000","last_active":"2023-10-16T20:37:37.402756+0000","last_peered":"2023-10-16T20:37:37.402756+0000","last_clean":"2023-10-16T20:37:37.402756+0000","last_became_active":"2023-10-16T20:36:54.757796+0000","last_became_peered":"2023-10-16T20:36:54.757796+0000","last_unstale":"2023-10-16T20:37:37.402756+0000","last_undegraded":"2023-10-16T20:37:37.402756+0000","last_fullsized":"2023-10-16T20:37:37.402756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:02:09.953795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464463+0000","last_change":"2023-10-16T20:36:59.468529+0000","last_active":"2023-10-16T20:37:37.464463+0000","last_peered":"2023-10-16T20:37:37.464463+0000","last_clean":"2023-10-16T20:37:37.464463+0000","last_became_active":"2023-10-16T20:36:59.465221+0000","last_became_peered":"2023-10-16T20:36:59.465221+0000","last_unstale":"2023-10-16T20:37:37.464463+0000","last_undegraded":"2023-10-16T20:37:37.464463+0000","last_fullsized":"2023-10-16T20:37:37.464463+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:58:08.457686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401257+0000","last_change":"2023-10-16T20:36:55.275759+0000","last_active":"2023-10-16T20:37:37.401257+0000","last_peered":"2023-10-16T20:37:37.401257+0000","last_clean":"2023-10-16T20:37:37.401257+0000","last_became_active":"2023-10-16T20:36:55.274432+0000","last_became_peered":"2023-10-16T20:36:55.274432+0000","last_unstale":"2023-10-16T20:37:37.401257+0000","last_undegraded":"2023-10-16T20:37:37.401257+0000","last_fullsized":"2023-10-16T20:37:37.401257+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:53:08.896731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:02.630911+0000","last_change":"2023-10-16T20:36:54.762356+0000","last_active":"2023-10-16T20:39:02.630911+0000","last_peered":"2023-10-16T20:39:02.630911+0000","last_clean":"2023-10-16T20:39:02.630911+0000","last_became_active":"2023-10-16T20:36:54.762167+0000","last_became_peered":"2023-10-16T20:36:54.762167+0000","last_unstale":"2023-10-16T20:39:02.630911+0000","last_undegraded":"2023-10-16T20:39:02.630911+0000","last_fullsized":"2023-10-16T20:39:02.630911+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:45.164625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434946+0000","last_change":"2023-10-16T20:36:59.467492+0000","last_active":"2023-10-16T20:37:37.434946+0000","last_peered":"2023-10-16T20:37:37.434946+0000","last_clean":"2023-10-16T20:37:37.434946+0000","last_became_active":"2023-10-16T20:36:59.466876+0000","last_became_peered":"2023-10-16T20:36:59.466876+0000","last_unstale":"2023-10-16T20:37:37.434946+0000","last_undegraded":"2023-10-16T20:37:37.434946+0000","last_fullsized":"2023-10-16T20:37:37.434946+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:29:01.292167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409570+0000","last_change":"2023-10-16T20:36:55.293537+0000","last_active":"2023-10-16T20:37:37.409570+0000","last_peered":"2023-10-16T20:37:37.409570+0000","last_clean":"2023-10-16T20:37:37.409570+0000","last_became_active":"2023-10-16T20:36:55.293228+0000","last_became_peered":"2023-10-16T20:36:55.293228+0000","last_unstale":"2023-10-16T20:37:37.409570+0000","last_undegraded":"2023-10-16T20:37:37.409570+0000","last_fullsized":"2023-10-16T20:37:37.409570+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:50:13.226645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401381+0000","last_change":"2023-10-16T20:36:54.757588+0000","last_active":"2023-10-16T20:37:37.401381+0000","last_peered":"2023-10-16T20:37:37.401381+0000","last_clean":"2023-10-16T20:37:37.401381+0000","last_became_active":"2023-10-16T20:36:54.757308+0000","last_became_peered":"2023-10-16T20:36:54.757308+0000","last_unstale":"2023-10-16T20:37:37.401381+0000","last_undegraded":"2023-10-16T20:37:37.401381+0000","last_fullsized":"2023-10-16T20:37:37.401381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:51.298239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402099+0000","last_change":"2023-10-16T20:36:59.303844+0000","last_active":"2023-10-16T20:37:37.402099+0000","last_peered":"2023-10-16T20:37:37.402099+0000","last_clean":"2023-10-16T20:37:37.402099+0000","last_became_active":"2023-10-16T20:36:59.303224+0000","last_became_peered":"2023-10-16T20:36:59.303224+0000","last_unstale":"2023-10-16T20:37:37.402099+0000","last_undegraded":"2023-10-16T20:37:37.402099+0000","last_fullsized":"2023-10-16T20:37:37.402099+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:52.447785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403798+0000","last_change":"2023-10-16T20:36:55.278310+0000","last_active":"2023-10-16T20:37:37.403798+0000","last_peered":"2023-10-16T20:37:37.403798+0000","last_clean":"2023-10-16T20:37:37.403798+0000","last_became_active":"2023-10-16T20:36:55.277872+0000","last_became_peered":"2023-10-16T20:36:55.277872+0000","last_unstale":"2023-10-16T20:37:37.403798+0000","last_undegraded":"2023-10-16T20:37:37.403798+0000","last_fullsized":"2023-10-16T20:37:37.403798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:37.320469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408898+0000","last_change":"2023-10-16T20:36:54.740644+0000","last_active":"2023-10-16T20:37:37.408898+0000","last_peered":"2023-10-16T20:37:37.408898+0000","last_clean":"2023-10-16T20:37:37.408898+0000","last_became_active":"2023-10-16T20:36:54.740340+0000","last_became_peered":"2023-10-16T20:36:54.740340+0000","last_unstale":"2023-10-16T20:37:37.408898+0000","last_undegraded":"2023-10-16T20:37:37.408898+0000","last_fullsized":"2023-10-16T20:37:37.408898+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:22.221727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401155+0000","last_change":"2023-10-16T20:36:59.294914+0000","last_active":"2023-10-16T20:37:37.401155+0000","last_peered":"2023-10-16T20:37:37.401155+0000","last_clean":"2023-10-16T20:37:37.401155+0000","last_became_active":"2023-10-16T20:36:59.294548+0000","last_became_peered":"2023-10-16T20:36:59.294548+0000","last_unstale":"2023-10-16T20:37:37.401155+0000","last_undegraded":"2023-10-16T20:37:37.401155+0000","last_fullsized":"2023-10-16T20:37:37.401155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:52:15.265262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401383+0000","last_change":"2023-10-16T20:36:55.270992+0000","last_active":"2023-10-16T20:37:37.401383+0000","last_peered":"2023-10-16T20:37:37.401383+0000","last_clean":"2023-10-16T20:37:37.401383+0000","last_became_active":"2023-10-16T20:36:55.269749+0000","last_became_peered":"2023-10-16T20:36:55.269749+0000","last_unstale":"2023-10-16T20:37:37.401383+0000","last_undegraded":"2023-10-16T20:37:37.401383+0000","last_fullsized":"2023-10-16T20:37:37.401383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:30.925314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:02.622407+0000","last_change":"2023-10-16T20:36:54.757903+0000","last_active":"2023-10-16T20:39:02.622407+0000","last_peered":"2023-10-16T20:39:02.622407+0000","last_clean":"2023-10-16T20:39:02.622407+0000","last_became_active":"2023-10-16T20:36:54.757549+0000","last_became_peered":"2023-10-16T20:36:54.757549+0000","last_unstale":"2023-10-16T20:39:02.622407+0000","last_undegraded":"2023-10-16T20:39:02.622407+0000","last_fullsized":"2023-10-16T20:39:02.622407+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:02:42.090848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402786+0000","last_change":"2023-10-16T20:36:59.302063+0000","last_active":"2023-10-16T20:37:37.402786+0000","last_peered":"2023-10-16T20:37:37.402786+0000","last_clean":"2023-10-16T20:37:37.402786+0000","last_became_active":"2023-10-16T20:36:59.301693+0000","last_became_peered":"2023-10-16T20:36:59.301693+0000","last_unstale":"2023-10-16T20:37:37.402786+0000","last_undegraded":"2023-10-16T20:37:37.402786+0000","last_fullsized":"2023-10-16T20:37:37.402786+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:29:44.812282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434713+0000","last_change":"2023-10-16T20:36:55.278132+0000","last_active":"2023-10-16T20:37:37.434713+0000","last_peered":"2023-10-16T20:37:37.434713+0000","last_clean":"2023-10-16T20:37:37.434713+0000","last_became_active":"2023-10-16T20:36:55.276649+0000","last_became_peered":"2023-10-16T20:36:55.276649+0000","last_unstale":"2023-10-16T20:37:37.434713+0000","last_undegraded":"2023-10-16T20:37:37.434713+0000","last_fullsized":"2023-10-16T20:37:37.434713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:30:53.299514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:38:57.630200+0000","last_change":"2023-10-16T20:36:54.753592+0000","last_active":"2023-10-16T20:38:57.630200+0000","last_peered":"2023-10-16T20:38:57.630200+0000","last_clean":"2023-10-16T20:38:57.630200+0000","last_became_active":"2023-10-16T20:36:54.753237+0000","last_became_peered":"2023-10-16T20:36:54.753237+0000","last_unstale":"2023-10-16T20:38:57.630200+0000","last_undegraded":"2023-10-16T20:38:57.630200+0000","last_fullsized":"2023-10-16T20:38:57.630200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:53.005979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435360+0000","last_change":"2023-10-16T20:36:59.468431+0000","last_active":"2023-10-16T20:37:37.435360+0000","last_peered":"2023-10-16T20:37:37.435360+0000","last_clean":"2023-10-16T20:37:37.435360+0000","last_became_active":"2023-10-16T20:36:59.466708+0000","last_became_peered":"2023-10-16T20:36:59.466708+0000","last_unstale":"2023-10-16T20:37:37.435360+0000","last_undegraded":"2023-10-16T20:37:37.435360+0000","last_fullsized":"2023-10-16T20:37:37.435360+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:42:20.607519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464817+0000","last_change":"2023-10-16T20:36:55.294825+0000","last_active":"2023-10-16T20:37:37.464817+0000","last_peered":"2023-10-16T20:37:37.464817+0000","last_clean":"2023-10-16T20:37:37.464817+0000","last_became_active":"2023-10-16T20:36:55.294610+0000","last_became_peered":"2023-10-16T20:36:55.294610+0000","last_unstale":"2023-10-16T20:37:37.464817+0000","last_undegraded":"2023-10-16T20:37:37.464817+0000","last_fullsized":"2023-10-16T20:37:37.464817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:43.442939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.410375+0000","last_change":"2023-10-16T20:36:54.741191+0000","last_active":"2023-10-16T20:39:22.410375+0000","last_peered":"2023-10-16T20:39:22.410375+0000","last_clean":"2023-10-16T20:39:22.410375+0000","last_became_active":"2023-10-16T20:36:54.740944+0000","last_became_peered":"2023-10-16T20:36:54.740944+0000","last_unstale":"2023-10-16T20:39:22.410375+0000","last_undegraded":"2023-10-16T20:39:22.410375+0000","last_fullsized":"2023-10-16T20:39:22.410375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:05.961850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":172,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466038+0000","last_change":"2023-10-16T20:36:46.390001+0000","last_active":"2023-10-16T20:37:37.466038+0000","last_peered":"2023-10-16T20:37:37.466038+0000","last_clean":"2023-10-16T20:37:37.466038+0000","last_became_active":"2023-10-16T20:36:46.388824+0000","last_became_peered":"2023-10-16T20:36:46.388824+0000","last_unstale":"2023-10-16T20:37:37.466038+0000","last_undegraded":"2023-10-16T20:37:37.466038+0000","last_fullsized":"2023-10-16T20:37:37.466038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_clean_scrub_stamp":"2023-10-16T20:36:44.575506+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:31.004318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407653+0000","last_change":"2023-10-16T20:36:59.468528+0000","last_active":"2023-10-16T20:37:37.407653+0000","last_peered":"2023-10-16T20:37:37.407653+0000","last_clean":"2023-10-16T20:37:37.407653+0000","last_became_active":"2023-10-16T20:36:59.468082+0000","last_became_peered":"2023-10-16T20:36:59.468082+0000","last_unstale":"2023-10-16T20:37:37.407653+0000","last_undegraded":"2023-10-16T20:37:37.407653+0000","last_fullsized":"2023-10-16T20:37:37.407653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:00.920004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410690+0000","last_change":"2023-10-16T20:37:37.409578+0000","last_active":"2023-10-16T20:37:37.410690+0000","last_peered":"2023-10-16T20:37:37.410690+0000","last_clean":"2023-10-16T20:37:37.410690+0000","last_became_active":"2023-10-16T20:37:37.409004+0000","last_became_peered":"2023-10-16T20:37:37.409004+0000","last_unstale":"2023-10-16T20:37:37.410690+0000","last_undegraded":"2023-10-16T20:37:37.410690+0000","last_fullsized":"2023-10-16T20:37:37.410690+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:28:57.364794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.400188+0000","last_change":"2023-10-16T20:36:54.746600+0000","last_active":"2023-10-16T20:37:37.400188+0000","last_peered":"2023-10-16T20:37:37.400188+0000","last_clean":"2023-10-16T20:37:37.400188+0000","last_became_active":"2023-10-16T20:36:54.746247+0000","last_became_peered":"2023-10-16T20:36:54.746247+0000","last_unstale":"2023-10-16T20:37:37.400188+0000","last_undegraded":"2023-10-16T20:37:37.400188+0000","last_fullsized":"2023-10-16T20:37:37.400188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:28:57.949487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.470531+0000","last_change":"2023-10-16T20:37:37.469427+0000","last_active":"2023-10-16T20:37:37.470531+0000","last_peered":"2023-10-16T20:37:37.470531+0000","last_clean":"2023-10-16T20:37:37.470531+0000","last_became_active":"2023-10-16T20:37:37.467402+0000","last_became_peered":"2023-10-16T20:37:37.467402+0000","last_unstale":"2023-10-16T20:37:37.470531+0000","last_undegraded":"2023-10-16T20:37:37.470531+0000","last_fullsized":"2023-10-16T20:37:37.470531+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:00:19.171972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409113+0000","last_change":"2023-10-16T20:36:55.301234+0000","last_active":"2023-10-16T20:37:37.409113+0000","last_peered":"2023-10-16T20:37:37.409113+0000","last_clean":"2023-10-16T20:37:37.409113+0000","last_became_active":"2023-10-16T20:36:55.300958+0000","last_became_peered":"2023-10-16T20:36:55.300958+0000","last_unstale":"2023-10-16T20:37:37.409113+0000","last_undegraded":"2023-10-16T20:37:37.409113+0000","last_fullsized":"2023-10-16T20:37:37.409113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:16.784124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"27'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402899+0000","last_change":"2023-10-16T20:36:54.753770+0000","last_active":"2023-10-16T20:37:37.402899+0000","last_peered":"2023-10-16T20:37:37.402899+0000","last_clean":"2023-10-16T20:37:37.402899+0000","last_became_active":"2023-10-16T20:36:54.753443+0000","last_became_peered":"2023-10-16T20:36:54.753443+0000","last_unstale":"2023-10-16T20:37:37.402899+0000","last_undegraded":"2023-10-16T20:37:37.402899+0000","last_fullsized":"2023-10-16T20:37:37.402899+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:21:26.392480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435046+0000","last_change":"2023-10-16T20:36:59.303351+0000","last_active":"2023-10-16T20:37:37.435046+0000","last_peered":"2023-10-16T20:37:37.435046+0000","last_clean":"2023-10-16T20:37:37.435046+0000","last_became_active":"2023-10-16T20:36:59.302869+0000","last_became_peered":"2023-10-16T20:36:59.302869+0000","last_unstale":"2023-10-16T20:37:37.435046+0000","last_undegraded":"2023-10-16T20:37:37.435046+0000","last_fullsized":"2023-10-16T20:37:37.435046+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:26:35.886399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466830+0000","last_change":"2023-10-16T20:36:55.291947+0000","last_active":"2023-10-16T20:37:37.466830+0000","last_peered":"2023-10-16T20:37:37.466830+0000","last_clean":"2023-10-16T20:37:37.466830+0000","last_became_active":"2023-10-16T20:36:55.291633+0000","last_became_peered":"2023-10-16T20:36:55.291633+0000","last_unstale":"2023-10-16T20:37:37.466830+0000","last_undegraded":"2023-10-16T20:37:37.466830+0000","last_fullsized":"2023-10-16T20:37:37.466830+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:00:14.349339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"27'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.400583+0000","last_change":"2023-10-16T20:36:54.748196+0000","last_active":"2023-10-16T20:37:37.400583+0000","last_peered":"2023-10-16T20:37:37.400583+0000","last_clean":"2023-10-16T20:37:37.400583+0000","last_became_active":"2023-10-16T20:36:54.747704+0000","last_became_peered":"2023-10-16T20:36:54.747704+0000","last_unstale":"2023-10-16T20:37:37.400583+0000","last_undegraded":"2023-10-16T20:37:37.400583+0000","last_fullsized":"2023-10-16T20:37:37.400583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:55.471916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410236+0000","last_change":"2023-10-16T20:36:59.470368+0000","last_active":"2023-10-16T20:37:37.410236+0000","last_peered":"2023-10-16T20:37:37.410236+0000","last_clean":"2023-10-16T20:37:37.410236+0000","last_became_active":"2023-10-16T20:36:59.467291+0000","last_became_peered":"2023-10-16T20:36:59.467291+0000","last_unstale":"2023-10-16T20:37:37.410236+0000","last_undegraded":"2023-10-16T20:37:37.410236+0000","last_fullsized":"2023-10-16T20:37:37.410236+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:18.312394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466658+0000","last_change":"2023-10-16T20:36:55.291451+0000","last_active":"2023-10-16T20:37:37.466658+0000","last_peered":"2023-10-16T20:37:37.466658+0000","last_clean":"2023-10-16T20:37:37.466658+0000","last_became_active":"2023-10-16T20:36:55.290998+0000","last_became_peered":"2023-10-16T20:36:55.290998+0000","last_unstale":"2023-10-16T20:37:37.466658+0000","last_undegraded":"2023-10-16T20:37:37.466658+0000","last_fullsized":"2023-10-16T20:37:37.466658+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:05:01.634971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"26'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404148+0000","last_change":"2023-10-16T20:36:54.763818+0000","last_active":"2023-10-16T20:37:37.404148+0000","last_peered":"2023-10-16T20:37:37.404148+0000","last_clean":"2023-10-16T20:37:37.404148+0000","last_became_active":"2023-10-16T20:36:54.761337+0000","last_became_peered":"2023-10-16T20:36:54.761337+0000","last_unstale":"2023-10-16T20:37:37.404148+0000","last_undegraded":"2023-10-16T20:37:37.404148+0000","last_fullsized":"2023-10-16T20:37:37.404148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:03.093189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.470301+0000","last_change":"2023-10-16T20:37:37.468803+0000","last_active":"2023-10-16T20:37:37.470301+0000","last_peered":"2023-10-16T20:37:37.470301+0000","last_clean":"2023-10-16T20:37:37.470301+0000","last_became_active":"2023-10-16T20:37:37.468309+0000","last_became_peered":"2023-10-16T20:37:37.468309+0000","last_unstale":"2023-10-16T20:37:37.470301+0000","last_undegraded":"2023-10-16T20:37:37.470301+0000","last_fullsized":"2023-10-16T20:37:37.470301+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:11.761661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466101+0000","last_change":"2023-10-16T20:36:55.288895+0000","last_active":"2023-10-16T20:37:37.466101+0000","last_peered":"2023-10-16T20:37:37.466101+0000","last_clean":"2023-10-16T20:37:37.466101+0000","last_became_active":"2023-10-16T20:36:55.288354+0000","last_became_peered":"2023-10-16T20:36:55.288354+0000","last_unstale":"2023-10-16T20:37:37.466101+0000","last_undegraded":"2023-10-16T20:37:37.466101+0000","last_fullsized":"2023-10-16T20:37:37.466101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:53:29.192406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'10","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.402644+0000","last_change":"2023-10-16T20:36:54.738216+0000","last_active":"2023-10-16T20:39:22.402644+0000","last_peered":"2023-10-16T20:39:22.402644+0000","last_clean":"2023-10-16T20:39:22.402644+0000","last_became_active":"2023-10-16T20:36:54.737876+0000","last_became_peered":"2023-10-16T20:36:54.737876+0000","last_unstale":"2023-10-16T20:39:22.402644+0000","last_undegraded":"2023-10-16T20:39:22.402644+0000","last_fullsized":"2023-10-16T20:39:22.402644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:28.214601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435178+0000","last_change":"2023-10-16T20:36:59.307215+0000","last_active":"2023-10-16T20:37:37.435178+0000","last_peered":"2023-10-16T20:37:37.435178+0000","last_clean":"2023-10-16T20:37:37.435178+0000","last_became_active":"2023-10-16T20:36:59.306801+0000","last_became_peered":"2023-10-16T20:36:59.306801+0000","last_unstale":"2023-10-16T20:37:37.435178+0000","last_undegraded":"2023-10-16T20:37:37.435178+0000","last_fullsized":"2023-10-16T20:37:37.435178+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:41:42.656134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407054+0000","last_change":"2023-10-16T20:36:55.287714+0000","last_active":"2023-10-16T20:37:37.407054+0000","last_peered":"2023-10-16T20:37:37.407054+0000","last_clean":"2023-10-16T20:37:37.407054+0000","last_became_active":"2023-10-16T20:36:55.287066+0000","last_became_peered":"2023-10-16T20:36:55.287066+0000","last_unstale":"2023-10-16T20:37:37.407054+0000","last_undegraded":"2023-10-16T20:37:37.407054+0000","last_fullsized":"2023-10-16T20:37:37.407054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:45:21.058260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407609+0000","last_change":"2023-10-16T20:36:54.764693+0000","last_active":"2023-10-16T20:37:37.407609+0000","last_peered":"2023-10-16T20:37:37.407609+0000","last_clean":"2023-10-16T20:37:37.407609+0000","last_became_active":"2023-10-16T20:36:54.764417+0000","last_became_peered":"2023-10-16T20:36:54.764417+0000","last_unstale":"2023-10-16T20:37:37.407609+0000","last_undegraded":"2023-10-16T20:37:37.407609+0000","last_fullsized":"2023-10-16T20:37:37.407609+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:56.653149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404514+0000","last_change":"2023-10-16T20:36:59.465145+0000","last_active":"2023-10-16T20:37:36.404514+0000","last_peered":"2023-10-16T20:37:36.404514+0000","last_clean":"2023-10-16T20:37:36.404514+0000","last_became_active":"2023-10-16T20:36:59.464626+0000","last_became_peered":"2023-10-16T20:36:59.464626+0000","last_unstale":"2023-10-16T20:37:36.404514+0000","last_undegraded":"2023-10-16T20:37:36.404514+0000","last_fullsized":"2023-10-16T20:37:36.404514+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:38.928616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466290+0000","last_change":"2023-10-16T20:36:55.291718+0000","last_active":"2023-10-16T20:37:37.466290+0000","last_peered":"2023-10-16T20:37:37.466290+0000","last_clean":"2023-10-16T20:37:37.466290+0000","last_became_active":"2023-10-16T20:36:55.291396+0000","last_became_peered":"2023-10-16T20:36:55.291396+0000","last_unstale":"2023-10-16T20:37:37.466290+0000","last_undegraded":"2023-10-16T20:37:37.466290+0000","last_fullsized":"2023-10-16T20:37:37.466290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:29:49.138380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.411305+0000","last_change":"2023-10-16T20:36:54.750814+0000","last_active":"2023-10-16T20:39:22.411305+0000","last_peered":"2023-10-16T20:39:22.411305+0000","last_clean":"2023-10-16T20:39:22.411305+0000","last_became_active":"2023-10-16T20:36:54.750448+0000","last_became_peered":"2023-10-16T20:36:54.750448+0000","last_unstale":"2023-10-16T20:39:22.411305+0000","last_undegraded":"2023-10-16T20:39:22.411305+0000","last_fullsized":"2023-10-16T20:39:22.411305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:41.981408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465192+0000","last_change":"2023-10-16T20:36:59.464962+0000","last_active":"2023-10-16T20:37:37.465192+0000","last_peered":"2023-10-16T20:37:37.465192+0000","last_clean":"2023-10-16T20:37:37.465192+0000","last_became_active":"2023-10-16T20:36:59.464578+0000","last_became_peered":"2023-10-16T20:36:59.464578+0000","last_unstale":"2023-10-16T20:37:37.465192+0000","last_undegraded":"2023-10-16T20:37:37.465192+0000","last_fullsized":"2023-10-16T20:37:37.465192+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:12:27.738364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435179+0000","last_change":"2023-10-16T20:36:55.285675+0000","last_active":"2023-10-16T20:37:37.435179+0000","last_peered":"2023-10-16T20:37:37.435179+0000","last_clean":"2023-10-16T20:37:37.435179+0000","last_became_active":"2023-10-16T20:36:55.285407+0000","last_became_peered":"2023-10-16T20:36:55.285407+0000","last_unstale":"2023-10-16T20:37:37.435179+0000","last_undegraded":"2023-10-16T20:37:37.435179+0000","last_fullsized":"2023-10-16T20:37:37.435179+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:31:10.259938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"26'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435225+0000","last_change":"2023-10-16T20:36:54.757012+0000","last_active":"2023-10-16T20:37:37.435225+0000","last_peered":"2023-10-16T20:37:37.435225+0000","last_clean":"2023-10-16T20:37:37.435225+0000","last_became_active":"2023-10-16T20:36:54.756697+0000","last_became_peered":"2023-10-16T20:36:54.756697+0000","last_unstale":"2023-10-16T20:37:37.435225+0000","last_undegraded":"2023-10-16T20:37:37.435225+0000","last_fullsized":"2023-10-16T20:37:37.435225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:20.259580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.409816+0000","last_change":"2023-10-16T20:36:59.469134+0000","last_active":"2023-10-16T20:37:36.409816+0000","last_peered":"2023-10-16T20:37:36.409816+0000","last_clean":"2023-10-16T20:37:36.409816+0000","last_became_active":"2023-10-16T20:36:59.468254+0000","last_became_peered":"2023-10-16T20:36:59.468254+0000","last_unstale":"2023-10-16T20:37:36.409816+0000","last_undegraded":"2023-10-16T20:37:36.409816+0000","last_fullsized":"2023-10-16T20:37:36.409816+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:12.444815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466418+0000","last_change":"2023-10-16T20:36:55.289133+0000","last_active":"2023-10-16T20:37:37.466418+0000","last_peered":"2023-10-16T20:37:37.466418+0000","last_clean":"2023-10-16T20:37:37.466418+0000","last_became_active":"2023-10-16T20:36:55.288488+0000","last_became_peered":"2023-10-16T20:36:55.288488+0000","last_unstale":"2023-10-16T20:37:37.466418+0000","last_undegraded":"2023-10-16T20:37:37.466418+0000","last_fullsized":"2023-10-16T20:37:37.466418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:27:22.022319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'12","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.411309+0000","last_change":"2023-10-16T20:36:54.765361+0000","last_active":"2023-10-16T20:39:22.411309+0000","last_peered":"2023-10-16T20:39:22.411309+0000","last_clean":"2023-10-16T20:39:22.411309+0000","last_became_active":"2023-10-16T20:36:54.764969+0000","last_became_peered":"2023-10-16T20:36:54.764969+0000","last_unstale":"2023-10-16T20:39:22.411309+0000","last_undegraded":"2023-10-16T20:39:22.411309+0000","last_fullsized":"2023-10-16T20:39:22.411309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:21:21.183612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3841,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402193+0000","last_change":"2023-10-16T20:36:59.303588+0000","last_active":"2023-10-16T20:37:37.402193+0000","last_peered":"2023-10-16T20:37:37.402193+0000","last_clean":"2023-10-16T20:37:37.402193+0000","last_became_active":"2023-10-16T20:36:59.302698+0000","last_became_peered":"2023-10-16T20:36:59.302698+0000","last_unstale":"2023-10-16T20:37:37.402193+0000","last_undegraded":"2023-10-16T20:37:37.402193+0000","last_fullsized":"2023-10-16T20:37:37.402193+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:08:28.644565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403928+0000","last_change":"2023-10-16T20:36:55.290812+0000","last_active":"2023-10-16T20:37:37.403928+0000","last_peered":"2023-10-16T20:37:37.403928+0000","last_clean":"2023-10-16T20:37:37.403928+0000","last_became_active":"2023-10-16T20:36:55.290127+0000","last_became_peered":"2023-10-16T20:36:55.290127+0000","last_unstale":"2023-10-16T20:37:37.403928+0000","last_undegraded":"2023-10-16T20:37:37.403928+0000","last_fullsized":"2023-10-16T20:37:37.403928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:28:27.930123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.217490+0000","last_change":"2023-10-16T20:36:54.749120+0000","last_active":"2023-10-16T20:39:22.217490+0000","last_peered":"2023-10-16T20:39:22.217490+0000","last_clean":"2023-10-16T20:39:22.217490+0000","last_became_active":"2023-10-16T20:36:54.748788+0000","last_became_peered":"2023-10-16T20:36:54.748788+0000","last_unstale":"2023-10-16T20:39:22.217490+0000","last_undegraded":"2023-10-16T20:39:22.217490+0000","last_fullsized":"2023-10-16T20:39:22.217490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:11:07.257343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403049+0000","last_change":"2023-10-16T20:36:59.303546+0000","last_active":"2023-10-16T20:37:37.403049+0000","last_peered":"2023-10-16T20:37:37.403049+0000","last_clean":"2023-10-16T20:37:37.403049+0000","last_became_active":"2023-10-16T20:36:59.302916+0000","last_became_peered":"2023-10-16T20:36:59.302916+0000","last_unstale":"2023-10-16T20:37:37.403049+0000","last_undegraded":"2023-10-16T20:37:37.403049+0000","last_fullsized":"2023-10-16T20:37:37.403049+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:17:16.174476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.398104+0000","last_change":"2023-10-16T20:36:55.291739+0000","last_active":"2023-10-16T20:37:36.398104+0000","last_peered":"2023-10-16T20:37:36.398104+0000","last_clean":"2023-10-16T20:37:36.398104+0000","last_became_active":"2023-10-16T20:36:55.291448+0000","last_became_peered":"2023-10-16T20:36:55.291448+0000","last_unstale":"2023-10-16T20:37:36.398104+0000","last_undegraded":"2023-10-16T20:37:36.398104+0000","last_fullsized":"2023-10-16T20:37:36.398104+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:41:15.380678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.410923+0000","last_change":"2023-10-16T20:36:54.761824+0000","last_active":"2023-10-16T20:39:22.410923+0000","last_peered":"2023-10-16T20:39:22.410923+0000","last_clean":"2023-10-16T20:39:22.410923+0000","last_became_active":"2023-10-16T20:36:54.761541+0000","last_became_peered":"2023-10-16T20:36:54.761541+0000","last_unstale":"2023-10-16T20:39:22.410923+0000","last_undegraded":"2023-10-16T20:39:22.410923+0000","last_fullsized":"2023-10-16T20:39:22.410923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:34.512207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464594+0000","last_change":"2023-10-16T20:36:59.304165+0000","last_active":"2023-10-16T20:37:37.464594+0000","last_peered":"2023-10-16T20:37:37.464594+0000","last_clean":"2023-10-16T20:37:37.464594+0000","last_became_active":"2023-10-16T20:36:59.303773+0000","last_became_peered":"2023-10-16T20:36:59.303773+0000","last_unstale":"2023-10-16T20:37:37.464594+0000","last_undegraded":"2023-10-16T20:37:37.464594+0000","last_fullsized":"2023-10-16T20:37:37.464594+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:09.710698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409711+0000","last_change":"2023-10-16T20:36:55.300829+0000","last_active":"2023-10-16T20:37:37.409711+0000","last_peered":"2023-10-16T20:37:37.409711+0000","last_clean":"2023-10-16T20:37:37.409711+0000","last_became_active":"2023-10-16T20:36:55.300535+0000","last_became_peered":"2023-10-16T20:36:55.300535+0000","last_unstale":"2023-10-16T20:37:37.409711+0000","last_undegraded":"2023-10-16T20:37:37.409711+0000","last_fullsized":"2023-10-16T20:37:37.409711+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:34.910378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.413267+0000","last_change":"2023-10-16T20:36:54.758140+0000","last_active":"2023-10-16T20:39:22.413267+0000","last_peered":"2023-10-16T20:39:22.413267+0000","last_clean":"2023-10-16T20:39:22.413267+0000","last_became_active":"2023-10-16T20:36:54.757303+0000","last_became_peered":"2023-10-16T20:36:54.757303+0000","last_unstale":"2023-10-16T20:39:22.413267+0000","last_undegraded":"2023-10-16T20:39:22.413267+0000","last_fullsized":"2023-10-16T20:39:22.413267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:45.944191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464795+0000","last_change":"2023-10-16T20:36:59.471158+0000","last_active":"2023-10-16T20:37:37.464795+0000","last_peered":"2023-10-16T20:37:37.464795+0000","last_clean":"2023-10-16T20:37:37.464795+0000","last_became_active":"2023-10-16T20:36:59.465844+0000","last_became_peered":"2023-10-16T20:36:59.465844+0000","last_unstale":"2023-10-16T20:37:37.464795+0000","last_undegraded":"2023-10-16T20:37:37.464795+0000","last_fullsized":"2023-10-16T20:37:37.464795+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:12.133525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435182+0000","last_change":"2023-10-16T20:36:59.468843+0000","last_active":"2023-10-16T20:37:37.435182+0000","last_peered":"2023-10-16T20:37:37.435182+0000","last_clean":"2023-10-16T20:37:37.435182+0000","last_became_active":"2023-10-16T20:36:59.467356+0000","last_became_peered":"2023-10-16T20:36:59.467356+0000","last_unstale":"2023-10-16T20:37:37.435182+0000","last_undegraded":"2023-10-16T20:37:37.435182+0000","last_fullsized":"2023-10-16T20:37:37.435182+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:47:02.923540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401755+0000","last_change":"2023-10-16T20:36:55.281355+0000","last_active":"2023-10-16T20:37:37.401755+0000","last_peered":"2023-10-16T20:37:37.401755+0000","last_clean":"2023-10-16T20:37:37.401755+0000","last_became_active":"2023-10-16T20:36:55.280903+0000","last_became_peered":"2023-10-16T20:36:55.280903+0000","last_unstale":"2023-10-16T20:37:37.401755+0000","last_undegraded":"2023-10-16T20:37:37.401755+0000","last_fullsized":"2023-10-16T20:37:37.401755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:15:25.813826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.231980+0000","last_change":"2023-10-16T20:36:54.757589+0000","last_active":"2023-10-16T20:39:22.231980+0000","last_peered":"2023-10-16T20:39:22.231980+0000","last_clean":"2023-10-16T20:39:22.231980+0000","last_became_active":"2023-10-16T20:36:54.756775+0000","last_became_peered":"2023-10-16T20:36:54.756775+0000","last_unstale":"2023-10-16T20:39:22.231980+0000","last_undegraded":"2023-10-16T20:39:22.231980+0000","last_fullsized":"2023-10-16T20:39:22.231980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:40.629540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.441661+0000","last_change":"2023-10-16T20:37:37.438581+0000","last_active":"2023-10-16T20:37:37.441661+0000","last_peered":"2023-10-16T20:37:37.441661+0000","last_clean":"2023-10-16T20:37:37.441661+0000","last_became_active":"2023-10-16T20:37:37.438140+0000","last_became_peered":"2023-10-16T20:37:37.438140+0000","last_unstale":"2023-10-16T20:37:37.441661+0000","last_undegraded":"2023-10-16T20:37:37.441661+0000","last_fullsized":"2023-10-16T20:37:37.441661+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:55.110618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464854+0000","last_change":"2023-10-16T20:36:55.274207+0000","last_active":"2023-10-16T20:37:37.464854+0000","last_peered":"2023-10-16T20:37:37.464854+0000","last_clean":"2023-10-16T20:37:37.464854+0000","last_became_active":"2023-10-16T20:36:55.273959+0000","last_became_peered":"2023-10-16T20:36:55.273959+0000","last_unstale":"2023-10-16T20:37:37.464854+0000","last_undegraded":"2023-10-16T20:37:37.464854+0000","last_fullsized":"2023-10-16T20:37:37.464854+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:23:09.275091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465033+0000","last_change":"2023-10-16T20:36:54.759513+0000","last_active":"2023-10-16T20:37:37.465033+0000","last_peered":"2023-10-16T20:37:37.465033+0000","last_clean":"2023-10-16T20:37:37.465033+0000","last_became_active":"2023-10-16T20:36:54.759059+0000","last_became_peered":"2023-10-16T20:36:54.759059+0000","last_unstale":"2023-10-16T20:37:37.465033+0000","last_undegraded":"2023-10-16T20:37:37.465033+0000","last_fullsized":"2023-10-16T20:37:37.465033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:36.696053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"27'1","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.404990+0000","last_change":"2023-10-16T20:36:54.764780+0000","last_active":"2023-10-16T20:37:36.404990+0000","last_peered":"2023-10-16T20:37:36.404990+0000","last_clean":"2023-10-16T20:37:36.404990+0000","last_became_active":"2023-10-16T20:36:54.764436+0000","last_became_peered":"2023-10-16T20:36:54.764436+0000","last_unstale":"2023-10-16T20:37:36.404990+0000","last_undegraded":"2023-10-16T20:37:36.404990+0000","last_fullsized":"2023-10-16T20:37:36.404990+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:55:31.816599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.398400+0000","last_change":"2023-10-16T20:36:55.278134+0000","last_active":"2023-10-16T20:37:36.398400+0000","last_peered":"2023-10-16T20:37:36.398400+0000","last_clean":"2023-10-16T20:37:36.398400+0000","last_became_active":"2023-10-16T20:36:55.277873+0000","last_became_peered":"2023-10-16T20:36:55.277873+0000","last_unstale":"2023-10-16T20:37:36.398400+0000","last_undegraded":"2023-10-16T20:37:36.398400+0000","last_fullsized":"2023-10-16T20:37:36.398400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:59:43.701800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.407258+0000","last_change":"2023-10-16T20:36:59.302774+0000","last_active":"2023-10-16T20:37:36.407258+0000","last_peered":"2023-10-16T20:37:36.407258+0000","last_clean":"2023-10-16T20:37:36.407258+0000","last_became_active":"2023-10-16T20:36:59.302435+0000","last_became_peered":"2023-10-16T20:36:59.302435+0000","last_unstale":"2023-10-16T20:37:36.407258+0000","last_undegraded":"2023-10-16T20:37:36.407258+0000","last_fullsized":"2023-10-16T20:37:36.407258+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:11:53.740982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'4","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.407881+0000","last_change":"2023-10-16T20:36:54.756190+0000","last_active":"2023-10-16T20:39:22.407881+0000","last_peered":"2023-10-16T20:39:22.407881+0000","last_clean":"2023-10-16T20:39:22.407881+0000","last_became_active":"2023-10-16T20:36:54.755834+0000","last_became_peered":"2023-10-16T20:36:54.755834+0000","last_unstale":"2023-10-16T20:39:22.407881+0000","last_undegraded":"2023-10-16T20:39:22.407881+0000","last_fullsized":"2023-10-16T20:39:22.407881+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:31.012441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.402297+0000","last_change":"2023-10-16T20:36:55.279822+0000","last_active":"2023-10-16T20:37:36.402297+0000","last_peered":"2023-10-16T20:37:36.402297+0000","last_clean":"2023-10-16T20:37:36.402297+0000","last_became_active":"2023-10-16T20:36:55.279333+0000","last_became_peered":"2023-10-16T20:36:55.279333+0000","last_unstale":"2023-10-16T20:37:36.402297+0000","last_undegraded":"2023-10-16T20:37:36.402297+0000","last_fullsized":"2023-10-16T20:37:36.402297+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:02:21.471812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.403977+0000","last_change":"2023-10-16T20:36:59.302190+0000","last_active":"2023-10-16T20:37:36.403977+0000","last_peered":"2023-10-16T20:37:36.403977+0000","last_clean":"2023-10-16T20:37:36.403977+0000","last_became_active":"2023-10-16T20:36:59.301864+0000","last_became_peered":"2023-10-16T20:36:59.301864+0000","last_unstale":"2023-10-16T20:37:36.403977+0000","last_undegraded":"2023-10-16T20:37:36.403977+0000","last_fullsized":"2023-10-16T20:37:36.403977+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:37:55.282815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.406436+0000","last_change":"2023-10-16T20:36:55.302476+0000","last_active":"2023-10-16T20:37:37.406436+0000","last_peered":"2023-10-16T20:37:37.406436+0000","last_clean":"2023-10-16T20:37:37.406436+0000","last_became_active":"2023-10-16T20:36:55.287442+0000","last_became_peered":"2023-10-16T20:36:55.287442+0000","last_unstale":"2023-10-16T20:37:37.406436+0000","last_undegraded":"2023-10-16T20:37:37.406436+0000","last_fullsized":"2023-10-16T20:37:37.406436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:19.511156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.399520+0000","last_change":"2023-10-16T20:36:55.276521+0000","last_active":"2023-10-16T20:37:37.399520+0000","last_peered":"2023-10-16T20:37:37.399520+0000","last_clean":"2023-10-16T20:37:37.399520+0000","last_became_active":"2023-10-16T20:36:55.276276+0000","last_became_peered":"2023-10-16T20:36:55.276276+0000","last_unstale":"2023-10-16T20:37:37.399520+0000","last_undegraded":"2023-10-16T20:37:37.399520+0000","last_fullsized":"2023-10-16T20:37:37.399520+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:38.218242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.415548+0000","last_change":"2023-10-16T20:37:37.413429+0000","last_active":"2023-10-16T20:37:37.415548+0000","last_peered":"2023-10-16T20:37:37.415548+0000","last_clean":"2023-10-16T20:37:37.415548+0000","last_became_active":"2023-10-16T20:37:37.413040+0000","last_became_peered":"2023-10-16T20:37:37.413040+0000","last_unstale":"2023-10-16T20:37:37.415548+0000","last_undegraded":"2023-10-16T20:37:37.415548+0000","last_fullsized":"2023-10-16T20:37:37.415548+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:39:10.897056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435108+0000","last_change":"2023-10-16T20:36:55.276258+0000","last_active":"2023-10-16T20:37:37.435108+0000","last_peered":"2023-10-16T20:37:37.435108+0000","last_clean":"2023-10-16T20:37:37.435108+0000","last_became_active":"2023-10-16T20:36:55.275349+0000","last_became_peered":"2023-10-16T20:36:55.275349+0000","last_unstale":"2023-10-16T20:37:37.435108+0000","last_undegraded":"2023-10-16T20:37:37.435108+0000","last_fullsized":"2023-10-16T20:37:37.435108+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:59.007782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465455+0000","last_change":"2023-10-16T20:36:55.277332+0000","last_active":"2023-10-16T20:37:37.465455+0000","last_peered":"2023-10-16T20:37:37.465455+0000","last_clean":"2023-10-16T20:37:37.465455+0000","last_became_active":"2023-10-16T20:36:55.277043+0000","last_became_peered":"2023-10-16T20:36:55.277043+0000","last_unstale":"2023-10-16T20:37:37.465455+0000","last_undegraded":"2023-10-16T20:37:37.465455+0000","last_fullsized":"2023-10-16T20:37:37.465455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:46:07.227813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401690+0000","last_change":"2023-10-16T20:36:55.279003+0000","last_active":"2023-10-16T20:37:37.401690+0000","last_peered":"2023-10-16T20:37:37.401690+0000","last_clean":"2023-10-16T20:37:37.401690+0000","last_became_active":"2023-10-16T20:36:55.278715+0000","last_became_peered":"2023-10-16T20:36:55.278715+0000","last_unstale":"2023-10-16T20:37:37.401690+0000","last_undegraded":"2023-10-16T20:37:37.401690+0000","last_fullsized":"2023-10-16T20:37:37.401690+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:58.274009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408078+0000","last_change":"2023-10-16T20:36:55.303936+0000","last_active":"2023-10-16T20:37:37.408078+0000","last_peered":"2023-10-16T20:37:37.408078+0000","last_clean":"2023-10-16T20:37:37.408078+0000","last_became_active":"2023-10-16T20:36:55.301604+0000","last_became_peered":"2023-10-16T20:36:55.301604+0000","last_unstale":"2023-10-16T20:37:37.408078+0000","last_undegraded":"2023-10-16T20:37:37.408078+0000","last_fullsized":"2023-10-16T20:37:37.408078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:48:14.922695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465625+0000","last_change":"2023-10-16T20:36:55.277329+0000","last_active":"2023-10-16T20:37:37.465625+0000","last_peered":"2023-10-16T20:37:37.465625+0000","last_clean":"2023-10-16T20:37:37.465625+0000","last_became_active":"2023-10-16T20:36:55.276984+0000","last_became_peered":"2023-10-16T20:36:55.276984+0000","last_unstale":"2023-10-16T20:37:37.465625+0000","last_undegraded":"2023-10-16T20:37:37.465625+0000","last_fullsized":"2023-10-16T20:37:37.465625+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:43:52.763586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408404+0000","last_change":"2023-10-16T20:36:55.290545+0000","last_active":"2023-10-16T20:37:37.408404+0000","last_peered":"2023-10-16T20:37:37.408404+0000","last_clean":"2023-10-16T20:37:37.408404+0000","last_became_active":"2023-10-16T20:36:55.290023+0000","last_became_peered":"2023-10-16T20:36:55.290023+0000","last_unstale":"2023-10-16T20:37:37.408404+0000","last_undegraded":"2023-10-16T20:37:37.408404+0000","last_fullsized":"2023-10-16T20:37:37.408404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:04:20.914854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464978+0000","last_change":"2023-10-16T20:36:55.280456+0000","last_active":"2023-10-16T20:37:37.464978+0000","last_peered":"2023-10-16T20:37:37.464978+0000","last_clean":"2023-10-16T20:37:37.464978+0000","last_became_active":"2023-10-16T20:36:55.280186+0000","last_became_peered":"2023-10-16T20:36:55.280186+0000","last_unstale":"2023-10-16T20:37:37.464978+0000","last_undegraded":"2023-10-16T20:37:37.464978+0000","last_fullsized":"2023-10-16T20:37:37.464978+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:52.469768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408252+0000","last_change":"2023-10-16T20:36:55.297917+0000","last_active":"2023-10-16T20:37:37.408252+0000","last_peered":"2023-10-16T20:37:37.408252+0000","last_clean":"2023-10-16T20:37:37.408252+0000","last_became_active":"2023-10-16T20:36:55.297253+0000","last_became_peered":"2023-10-16T20:36:55.297253+0000","last_unstale":"2023-10-16T20:37:37.408252+0000","last_undegraded":"2023-10-16T20:37:37.408252+0000","last_fullsized":"2023-10-16T20:37:37.408252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:14.977093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434979+0000","last_change":"2023-10-16T20:36:55.286327+0000","last_active":"2023-10-16T20:37:37.434979+0000","last_peered":"2023-10-16T20:37:37.434979+0000","last_clean":"2023-10-16T20:37:37.434979+0000","last_became_active":"2023-10-16T20:36:55.286044+0000","last_became_peered":"2023-10-16T20:36:55.286044+0000","last_unstale":"2023-10-16T20:37:37.434979+0000","last_undegraded":"2023-10-16T20:37:37.434979+0000","last_fullsized":"2023-10-16T20:37:37.434979+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:30:56.000502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409227+0000","last_change":"2023-10-16T20:36:55.298742+0000","last_active":"2023-10-16T20:37:37.409227+0000","last_peered":"2023-10-16T20:37:37.409227+0000","last_clean":"2023-10-16T20:37:37.409227+0000","last_became_active":"2023-10-16T20:36:55.286864+0000","last_became_peered":"2023-10-16T20:36:55.286864+0000","last_unstale":"2023-10-16T20:37:37.409227+0000","last_undegraded":"2023-10-16T20:37:37.409227+0000","last_fullsized":"2023-10-16T20:37:37.409227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:24:27.598910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466521+0000","last_change":"2023-10-16T20:36:55.292541+0000","last_active":"2023-10-16T20:37:37.466521+0000","last_peered":"2023-10-16T20:37:37.466521+0000","last_clean":"2023-10-16T20:37:37.466521+0000","last_became_active":"2023-10-16T20:36:55.292262+0000","last_became_peered":"2023-10-16T20:36:55.292262+0000","last_unstale":"2023-10-16T20:37:37.466521+0000","last_undegraded":"2023-10-16T20:37:37.466521+0000","last_fullsized":"2023-10-16T20:37:37.466521+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:10:08.644265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.400632+0000","last_change":"2023-10-16T20:36:55.283056+0000","last_active":"2023-10-16T20:37:37.400632+0000","last_peered":"2023-10-16T20:37:37.400632+0000","last_clean":"2023-10-16T20:37:37.400632+0000","last_became_active":"2023-10-16T20:36:55.282806+0000","last_became_peered":"2023-10-16T20:36:55.282806+0000","last_unstale":"2023-10-16T20:37:37.400632+0000","last_undegraded":"2023-10-16T20:37:37.400632+0000","last_fullsized":"2023-10-16T20:37:37.400632+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:21:24.681696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410649+0000","last_change":"2023-10-16T20:36:55.303385+0000","last_active":"2023-10-16T20:37:37.410649+0000","last_peered":"2023-10-16T20:37:37.410649+0000","last_clean":"2023-10-16T20:37:37.410649+0000","last_became_active":"2023-10-16T20:36:55.303095+0000","last_became_peered":"2023-10-16T20:36:55.303095+0000","last_unstale":"2023-10-16T20:37:37.410649+0000","last_undegraded":"2023-10-16T20:37:37.410649+0000","last_fullsized":"2023-10-16T20:37:37.410649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:26.368480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408312+0000","last_change":"2023-10-16T20:36:55.291852+0000","last_active":"2023-10-16T20:37:37.408312+0000","last_peered":"2023-10-16T20:37:37.408312+0000","last_clean":"2023-10-16T20:37:37.408312+0000","last_became_active":"2023-10-16T20:36:55.291447+0000","last_became_peered":"2023-10-16T20:36:55.291447+0000","last_unstale":"2023-10-16T20:37:37.408312+0000","last_undegraded":"2023-10-16T20:37:37.408312+0000","last_fullsized":"2023-10-16T20:37:37.408312+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:07.195372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401838+0000","last_change":"2023-10-16T20:36:55.281169+0000","last_active":"2023-10-16T20:37:37.401838+0000","last_peered":"2023-10-16T20:37:37.401838+0000","last_clean":"2023-10-16T20:37:37.401838+0000","last_became_active":"2023-10-16T20:36:55.280605+0000","last_became_peered":"2023-10-16T20:36:55.280605+0000","last_unstale":"2023-10-16T20:37:37.401838+0000","last_undegraded":"2023-10-16T20:37:37.401838+0000","last_fullsized":"2023-10-16T20:37:37.401838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:03.294173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435040+0000","last_change":"2023-10-16T20:36:55.277395+0000","last_active":"2023-10-16T20:37:37.435040+0000","last_peered":"2023-10-16T20:37:37.435040+0000","last_clean":"2023-10-16T20:37:37.435040+0000","last_became_active":"2023-10-16T20:36:55.277094+0000","last_became_peered":"2023-10-16T20:36:55.277094+0000","last_unstale":"2023-10-16T20:37:37.435040+0000","last_undegraded":"2023-10-16T20:37:37.435040+0000","last_fullsized":"2023-10-16T20:37:37.435040+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:23:03.400384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465006+0000","last_change":"2023-10-16T20:36:55.297429+0000","last_active":"2023-10-16T20:37:37.465006+0000","last_peered":"2023-10-16T20:37:37.465006+0000","last_clean":"2023-10-16T20:37:37.465006+0000","last_became_active":"2023-10-16T20:36:55.297199+0000","last_became_peered":"2023-10-16T20:36:55.297199+0000","last_unstale":"2023-10-16T20:37:37.465006+0000","last_undegraded":"2023-10-16T20:37:37.465006+0000","last_fullsized":"2023-10-16T20:37:37.465006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:43.860251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404029+0000","last_change":"2023-10-16T20:36:55.287421+0000","last_active":"2023-10-16T20:37:37.404029+0000","last_peered":"2023-10-16T20:37:37.404029+0000","last_clean":"2023-10-16T20:37:37.404029+0000","last_became_active":"2023-10-16T20:36:55.286964+0000","last_became_peered":"2023-10-16T20:36:55.286964+0000","last_unstale":"2023-10-16T20:37:37.404029+0000","last_undegraded":"2023-10-16T20:37:37.404029+0000","last_fullsized":"2023-10-16T20:37:37.404029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:15:54.970100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.416921+0000","last_change":"2023-10-16T20:37:37.413891+0000","last_active":"2023-10-16T20:37:37.416921+0000","last_peered":"2023-10-16T20:37:37.416921+0000","last_clean":"2023-10-16T20:37:37.416921+0000","last_became_active":"2023-10-16T20:37:37.413492+0000","last_became_peered":"2023-10-16T20:37:37.413492+0000","last_unstale":"2023-10-16T20:37:37.416921+0000","last_undegraded":"2023-10-16T20:37:37.416921+0000","last_fullsized":"2023-10-16T20:37:37.416921+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:03:46.658556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407997+0000","last_change":"2023-10-16T20:36:55.303950+0000","last_active":"2023-10-16T20:37:37.407997+0000","last_peered":"2023-10-16T20:37:37.407997+0000","last_clean":"2023-10-16T20:37:37.407997+0000","last_became_active":"2023-10-16T20:36:55.302468+0000","last_became_peered":"2023-10-16T20:36:55.302468+0000","last_unstale":"2023-10-16T20:37:37.407997+0000","last_undegraded":"2023-10-16T20:37:37.407997+0000","last_fullsized":"2023-10-16T20:37:37.407997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:02.962031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466830+0000","last_change":"2023-10-16T20:36:55.297558+0000","last_active":"2023-10-16T20:37:37.466830+0000","last_peered":"2023-10-16T20:37:37.466830+0000","last_clean":"2023-10-16T20:37:37.466830+0000","last_became_active":"2023-10-16T20:36:55.296476+0000","last_became_peered":"2023-10-16T20:36:55.296476+0000","last_unstale":"2023-10-16T20:37:37.466830+0000","last_undegraded":"2023-10-16T20:37:37.466830+0000","last_fullsized":"2023-10-16T20:37:37.466830+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:04:38.235205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.403439+0000","last_change":"2023-10-16T20:36:55.293957+0000","last_active":"2023-10-16T20:37:36.403439+0000","last_peered":"2023-10-16T20:37:36.403439+0000","last_clean":"2023-10-16T20:37:36.403439+0000","last_became_active":"2023-10-16T20:36:55.293687+0000","last_became_peered":"2023-10-16T20:36:55.293687+0000","last_unstale":"2023-10-16T20:37:36.403439+0000","last_undegraded":"2023-10-16T20:37:36.403439+0000","last_fullsized":"2023-10-16T20:37:36.403439+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:54.623691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.411550+0000","last_change":"2023-10-16T20:36:55.280728+0000","last_active":"2023-10-16T20:37:36.411550+0000","last_peered":"2023-10-16T20:37:36.411550+0000","last_clean":"2023-10-16T20:37:36.411550+0000","last_became_active":"2023-10-16T20:36:55.280444+0000","last_became_peered":"2023-10-16T20:36:55.280444+0000","last_unstale":"2023-10-16T20:37:36.411550+0000","last_undegraded":"2023-10-16T20:37:36.411550+0000","last_fullsized":"2023-10-16T20:37:36.411550+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:44.044563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.408400+0000","last_change":"2023-10-16T20:36:55.297802+0000","last_active":"2023-10-16T20:37:36.408400+0000","last_peered":"2023-10-16T20:37:36.408400+0000","last_clean":"2023-10-16T20:37:36.408400+0000","last_became_active":"2023-10-16T20:36:55.296694+0000","last_became_peered":"2023-10-16T20:36:55.296694+0000","last_unstale":"2023-10-16T20:37:36.408400+0000","last_undegraded":"2023-10-16T20:37:36.408400+0000","last_fullsized":"2023-10-16T20:37:36.408400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:00:26.797387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-16T20:37:36.411707+0000","last_change":"2023-10-16T20:36:55.291347+0000","last_active":"2023-10-16T20:37:36.411707+0000","last_peered":"2023-10-16T20:37:36.411707+0000","last_clean":"2023-10-16T20:37:36.411707+0000","last_became_active":"2023-10-16T20:36:55.291052+0000","last_became_peered":"2023-10-16T20:36:55.291052+0000","last_unstale":"2023-10-16T20:37:36.411707+0000","last_undegraded":"2023-10-16T20:37:36.411707+0000","last_fullsized":"2023-10-16T20:37:36.411707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:40:51.331446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409006+0000","last_change":"2023-10-16T20:36:55.290870+0000","last_active":"2023-10-16T20:37:37.409006+0000","last_peered":"2023-10-16T20:37:37.409006+0000","last_clean":"2023-10-16T20:37:37.409006+0000","last_became_active":"2023-10-16T20:36:55.290386+0000","last_became_peered":"2023-10-16T20:36:55.290386+0000","last_unstale":"2023-10-16T20:37:37.409006+0000","last_undegraded":"2023-10-16T20:37:37.409006+0000","last_fullsized":"2023-10-16T20:37:37.409006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:20.006864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435199+0000","last_change":"2023-10-16T20:36:55.271041+0000","last_active":"2023-10-16T20:37:37.435199+0000","last_peered":"2023-10-16T20:37:37.435199+0000","last_clean":"2023-10-16T20:37:37.435199+0000","last_became_active":"2023-10-16T20:36:55.270787+0000","last_became_peered":"2023-10-16T20:36:55.270787+0000","last_unstale":"2023-10-16T20:37:37.435199+0000","last_undegraded":"2023-10-16T20:37:37.435199+0000","last_fullsized":"2023-10-16T20:37:37.435199+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:07.944778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402678+0000","last_change":"2023-10-16T20:36:55.271632+0000","last_active":"2023-10-16T20:37:37.402678+0000","last_peered":"2023-10-16T20:37:37.402678+0000","last_clean":"2023-10-16T20:37:37.402678+0000","last_became_active":"2023-10-16T20:36:55.271373+0000","last_became_peered":"2023-10-16T20:36:55.271373+0000","last_unstale":"2023-10-16T20:37:37.402678+0000","last_undegraded":"2023-10-16T20:37:37.402678+0000","last_fullsized":"2023-10-16T20:37:37.402678+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:07.458540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466741+0000","last_change":"2023-10-16T20:36:55.291343+0000","last_active":"2023-10-16T20:37:37.466741+0000","last_peered":"2023-10-16T20:37:37.466741+0000","last_clean":"2023-10-16T20:37:37.466741+0000","last_became_active":"2023-10-16T20:36:55.291016+0000","last_became_peered":"2023-10-16T20:36:55.291016+0000","last_unstale":"2023-10-16T20:37:37.466741+0000","last_undegraded":"2023-10-16T20:37:37.466741+0000","last_fullsized":"2023-10-16T20:37:37.466741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:13:40.287782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5457,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":10892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509475,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343936,"statfs":{"total":96636764160,"available":96608190464,"internally_reserved":0,"allocated":905216,"data_stored":715195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.74299999999999999,"15min":0.748},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":5.2240000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.75700000000000001,"15min":0.753},"min":{"1min":0.58099999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94599999999999995,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":8.782}]},{"osd":1,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.74299999999999999,"15min":0.73999999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.91600000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.52100000000000002,"5min":0.497,"15min":0.497},"max":{"1min":0.91500000000000004,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":4.7300000000000004}]},{"osd":2,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.73199999999999998,"15min":0.72599999999999998},"min":{"1min":0.624,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":5.2629999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.748,"15min":0.745},"min":{"1min":0.60699999999999998,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.89700000000000002,"5min":1.083,"15min":1.083},"last":5.1589999999999998}]},{"osd":3,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.77600000000000002,"15min":0.77200000000000002},"min":{"1min":0.62,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0069999999999999,"5min":1.234,"15min":1.234},"last":4.7460000000000004},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.93200000000000005,"5min":1.024,"15min":1.024},"last":4.7770000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.63100000000000001,"15min":0.63300000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.92500000000000004,"5min":1.3,"15min":1.3},"last":0.501},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60499999999999998,"15min":0.60999999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76500000000000001,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.53500000000000003}]},{"osd":5,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.69999999999999996,"15min":0.70399999999999996},"min":{"1min":0.47599999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.78600000000000003,"15min":0.79500000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.006,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.68000000000000005}]},{"osd":6,"last update":"Mon Oct 16 20:38:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.749,"15min":0.755},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.97399999999999998,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.70399999999999996,"15min":0.70599999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.88300000000000001,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.70599999999999996}]}]},{"osd":6,"up_from":15,"seq":64424509475,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":128086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.76800000000000002,"15min":0.77100000000000002},"min":{"1min":0.41899999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88,"5min":1.131,"15min":1.131},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.748,"15min":0.75800000000000001},"min":{"1min":0.48899999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.875,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.624}]},{"osd":1,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.751,"15min":0.76300000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.81799999999999995,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.77000000000000002,"15min":0.77600000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.90400000000000003,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.58899999999999997}]},{"osd":2,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.78000000000000003,"15min":0.79200000000000004},"min":{"1min":0.48599999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.90300000000000002,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.73899999999999999,"15min":0.746},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.86899999999999999,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.71799999999999997}]},{"osd":3,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.84999999999999998,"15min":0.87},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.90600000000000003,"5min":1.611,"15min":1.611},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.749,"5min":0.80900000000000005,"15min":0.81899999999999995},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.89700000000000002,"5min":1.163,"15min":1.163},"last":0.56499999999999995}]},{"osd":4,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.66700000000000004,"15min":0.67900000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.79500000000000004,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65700000000000003,"15min":0.66700000000000004},"min":{"1min":0.49299999999999999,"5min":0.435,"15min":0.435},"max":{"1min":0.69499999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.73899999999999999}]},{"osd":5,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86199999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.748},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69899999999999995,"15min":0.70599999999999996},"min":{"1min":0.54900000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.82399999999999995,"5min":1.016,"15min":1.016},"last":0.69999999999999996}]},{"osd":7,"last update":"Mon Oct 16 20:38:46 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.748,"15min":0.754},"min":{"1min":0.52200000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.93100000000000005,"5min":1.292,"15min":1.292},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.75900000000000001,"15min":0.77500000000000002},"min":{"1min":0.52800000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.92100000000000004,"5min":1.569,"15min":1.569},"last":0.77000000000000002}]}]},{"osd":5,"up_from":15,"seq":64424509477,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":311296,"data_stored":124335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.70099999999999996,"15min":0.69599999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.75800000000000001}]},{"osd":1,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70699999999999996,"15min":0.70499999999999996},"min":{"1min":0.52100000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.71499999999999997,"15min":0.70899999999999996},"min":{"1min":0.57399999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.70499999999999996}]},{"osd":2,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.72199999999999998,"15min":0.73199999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92300000000000004,"5min":1.022,"15min":1.022},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.74299999999999999,"15min":0.73699999999999999},"min":{"1min":0.60799999999999998,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.91700000000000004,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.79100000000000004}]},{"osd":3,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79200000000000004,"15min":0.79100000000000004},"min":{"1min":0.66700000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81000000000000005,"15min":0.81599999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.82599999999999996}]},{"osd":4,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65100000000000002,"15min":0.65900000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.90300000000000002,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.73399999999999999,"15min":0.75800000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.80500000000000005,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.60199999999999998}]},{"osd":6,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.70999999999999996,"15min":0.72799999999999998},"min":{"1min":0.45500000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.79700000000000004,"5min":1.069,"15min":1.069},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.495,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.038,"5min":1.264,"15min":1.264},"last":0.64000000000000001}]},{"osd":7,"last update":"Mon Oct 16 20:38:43 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.78000000000000003,"15min":0.79600000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.89200000000000002,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.79100000000000004,"15min":0.80700000000000005},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.163,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.73799999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509475,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":124371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.72199999999999998,"15min":0.72399999999999998},"min":{"1min":0.45200000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.872,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.59599999999999997,"5min":0.5,"15min":0.5},"max":{"1min":0.88200000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.55200000000000005}]},{"osd":1,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.57499999999999996,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92500000000000004,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.748,"15min":0.748},"min":{"1min":0.54500000000000004,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.66700000000000004}]},{"osd":2,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.76400000000000001,"15min":0.76200000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77700000000000002,"15min":0.77800000000000002},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.876}]},{"osd":3,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.747,"15min":0.73899999999999999},"min":{"1min":0.58899999999999997,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.78200000000000003,"15min":0.78600000000000003},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.86399999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.79000000000000004}]},{"osd":5,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.63600000000000001,"15min":0.64800000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.75600000000000001,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.44800000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95899999999999996,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.84699999999999998}]},{"osd":6,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.66500000000000004,"15min":0.66700000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.73499999999999999,"15min":0.73799999999999999},"min":{"1min":0.495,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.90600000000000003}]},{"osd":7,"last update":"Mon Oct 16 20:38:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.747,"15min":0.751},"min":{"1min":0.52000000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.94599999999999995,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.79300000000000004,"15min":0.80300000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.96599999999999997,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.89200000000000002}]}]},{"osd":3,"up_from":15,"seq":64424509475,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":323584,"data_stored":128120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.46999999999999997,"15min":0.47199999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.69499999999999995,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.48699999999999999}]},{"osd":1,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.41499999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71699999999999997,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.52900000000000003}]},{"osd":2,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.53800000000000003,"15min":0.54400000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.67600000000000005,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.626,"15min":0.61699999999999999},"min":{"1min":0.51600000000000001,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.56899999999999995}]},{"osd":4,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.80200000000000005,"15min":0.80000000000000004},"min":{"1min":0.56200000000000006,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.054,"5min":1.181,"15min":1.181},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.77300000000000002,"15min":0.76500000000000001},"min":{"1min":0.57499999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.052,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.81999999999999995}]},{"osd":5,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.85999999999999999,"15min":0.873},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.008,"5min":1.544,"15min":1.544},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.54400000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.323,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.83199999999999996}]},{"osd":6,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.85499999999999998,"15min":0.85799999999999998},"min":{"1min":0.64900000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.3480000000000001,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.72599999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.80600000000000005,"15min":0.80000000000000004},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0880000000000001,"5min":1.139,"15min":1.139},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.0029999999999999,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.79600000000000004}]}]},{"osd":2,"up_from":15,"seq":64424509475,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":124245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.47899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71099999999999997,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.434}]},{"osd":1,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77400000000000002,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.373},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55100000000000005,"15min":0.54900000000000004},"min":{"1min":0.38700000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69399999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.40799999999999997}]},{"osd":3,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54400000000000004,"15min":0.53600000000000003},"min":{"1min":0.41299999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.504},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76700000000000002,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.46899999999999997}]},{"osd":4,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.82299999999999995,"15min":0.82899999999999996},"min":{"1min":0.60799999999999998,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.97499999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.64600000000000002,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.98299999999999998,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.56200000000000006}]},{"osd":5,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.60999999999999999,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.96999999999999997,"5min":1.444,"15min":1.444},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.83899999999999997,"15min":0.84199999999999997},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":0.998,"5min":1.585,"15min":1.585},"last":0.59099999999999997}]},{"osd":6,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.84799999999999998,"15min":0.85199999999999998},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.1020000000000001,"5min":1.335,"15min":1.335},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.871,"15min":0.88200000000000001},"min":{"1min":0.68999999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":0.95099999999999996,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.61799999999999999}]},{"osd":7,"last update":"Mon Oct 16 20:38:47 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.86499999999999999,"15min":0.875},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.046,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.88300000000000001,"15min":0.89300000000000002},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.0589999999999999,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.66900000000000004}]}]},{"osd":1,"up_from":15,"seq":64424509475,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":315392,"data_stored":124405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.46400000000000002,"15min":0.46700000000000003},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.64700000000000002,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.32600000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.69899999999999995,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.52400000000000002}]},{"osd":2,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.375,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82499999999999996,"5min":1.046,"15min":1.046},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.70699999999999996,"5min":1.115,"15min":1.115},"last":0.40999999999999998}]},{"osd":3,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60599999999999998,"15min":0.60899999999999999},"min":{"1min":0.40799999999999997,"5min":0.312,"15min":0.312},"max":{"1min":0.78500000000000003,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.60799999999999998,"15min":0.61799999999999999},"min":{"1min":0.41499999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.80400000000000005,"5min":1.431,"15min":1.431},"last":0.60899999999999999}]},{"osd":4,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.83399999999999996,"15min":0.83599999999999997},"min":{"1min":0.59199999999999997,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.242,"15min":1.242},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.81299999999999994,"15min":0.81100000000000005},"min":{"1min":0.60099999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.054,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.80800000000000005,"15min":0.81999999999999995},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90800000000000003,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.871,"15min":0.872},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.1160000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.51300000000000001}]},{"osd":6,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.874,"15min":0.88100000000000001},"min":{"1min":0.65300000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.123,"5min":2.004,"15min":2.004},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.876,"15min":0.88300000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.0880000000000001,"5min":1.635,"15min":1.635},"last":0.54800000000000004}]},{"osd":7,"last update":"Mon Oct 16 20:38:48 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.878,"15min":0.89200000000000002},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.99099999999999999,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.871,"5min":0.90700000000000003,"15min":0.91300000000000003},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.0760000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.56799999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343928,"statfs":{"total":96636764160,"available":96608182272,"internally_reserved":0,"allocated":913408,"data_stored":715251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.439,"15min":0.438},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.495,"15min":0.496},"min":{"1min":0.374,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.58999999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.38600000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52300000000000002,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54100000000000004,"15min":0.54600000000000004},"min":{"1min":0.40400000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.68899999999999995,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.39700000000000002}]},{"osd":3,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54800000000000004,"15min":0.54500000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73399999999999999,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.56499999999999995,"5min":0.497,"15min":0.497},"max":{"1min":0.84999999999999998,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.55100000000000005}]},{"osd":4,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.83799999999999997,"15min":0.82999999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0600000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.85999999999999999,"15min":0.84899999999999998},"min":{"1min":0.61399999999999999,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.80800000000000005}]},{"osd":5,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.80800000000000005,"15min":0.80300000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.871},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.75700000000000001}]},{"osd":6,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.86299999999999999,"15min":0.85899999999999999},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88400000000000001,"15min":0.88300000000000001},"min":{"1min":0.60899999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.77800000000000002}]},{"osd":7,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.64900000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.83499999999999996,"15min":0.82299999999999995},"min":{"1min":0.66500000000000004,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T20:39:25.537 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-16T20:39:25.537 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-16T20:39:25.537 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-16T20:39:25.538 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-16T20:39:25.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.705+0000 7ffba27d9640 1 Processor -- start 2023-10-16T20:39:25.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.705+0000 7ffba27d9640 1 -- start start 2023-10-16T20:39:25.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.705+0000 7ffba27d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ffb9c06fe20 0x7ffb9c06a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:25.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.705+0000 7ffba27d9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb9c156430 con 0x7ffb9c069e90 2023-10-16T20:39:25.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.709+0000 7ffba27d9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb9c070230 con 0x7ffb9c14fd10 2023-10-16T20:39:25.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.709+0000 7ffba27d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb9c070500 con 0x7ffb9c06fe20 2023-10-16T20:39:25.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.709+0000 7ffba0d76640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ffb9c06fe20 0x7ffb9c06a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:25.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.709+0000 7ffba0d76640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ffb9c06fe20 0x7ffb9c06a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56728/0 (socket says 172.21.15.88:56728) 2023-10-16T20:39:25.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.709+0000 7ffb93fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ffb9c069e90 0x7ffb9c06f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:37060/0 (socket says 172.21.15.88:37060) 2023-10-16T20:39:25.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.709+0000 7ffba0d76640 1 -- 172.21.15.88:0/619823145 learned_addr learned my addr 172.21.15.88:0/619823145 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:25.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.709+0000 7ffba0d76640 1 -- 172.21.15.88:0/619823145 >> v1:172.21.15.191:6789/0 conn(0x7ffb9c14fd10 legacy=0x7ffb9c070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:25.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.709+0000 7ffba0d76640 1 -- 172.21.15.88:0/619823145 >> v1:172.21.15.88:6789/0 conn(0x7ffb9c069e90 legacy=0x7ffb9c06f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:25.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.709+0000 7ffba0d76640 1 -- 172.21.15.88:0/619823145 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c155b00 con 0x7ffb9c06fe20 2023-10-16T20:39:25.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.709+0000 7ffb9b7fe640 1 -- 172.21.15.88:0/619823145 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936781764 0 0) 0x7ffb9c156430 con 0x7ffb9c069e90 2023-10-16T20:39:25.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.709+0000 7ffba0d76640 1 --2- 172.21.15.88:0/619823145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb9c06fe20 0x7ffb9c06a2d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ffb9400af60 tx=0x7ffb94020310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa30d5ea2fceb38 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:25.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.709+0000 7ffb9b7fe640 1 -- 172.21.15.88:0/619823145 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9400a2d0 con 0x7ffb9c06fe20 2023-10-16T20:39:25.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.709+0000 7ffb9b7fe640 1 -- 172.21.15.88:0/619823145 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb940ac050 con 0x7ffb9c06fe20 2023-10-16T20:39:25.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.709+0000 7ffb9b7fe640 1 -- 172.21.15.88:0/619823145 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb94008720 con 0x7ffb9c06fe20 2023-10-16T20:39:25.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffba27d9640 1 -- 172.21.15.88:0/619823145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb9c06fe20 msgr2=0x7ffb9c06a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:25.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffba27d9640 1 --2- 172.21.15.88:0/619823145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb9c06fe20 0x7ffb9c06a2d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ffb9400af60 tx=0x7ffb94020310 comp rx=0 tx=0).stop 2023-10-16T20:39:25.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffba27d9640 1 -- 172.21.15.88:0/619823145 shutdown_connections 2023-10-16T20:39:25.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffba27d9640 1 --2- 172.21.15.88:0/619823145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb9c06fe20 0x7ffb9c06a2d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffba27d9640 1 -- 172.21.15.88:0/619823145 >> 172.21.15.88:0/619823145 conn(0x7ffb9c0cad50 msgr2=0x7ffb9c0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:25.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffba27d9640 1 -- 172.21.15.88:0/619823145 shutdown_connections 2023-10-16T20:39:25.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffba27d9640 1 -- 172.21.15.88:0/619823145 wait complete. 2023-10-16T20:39:25.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffba27d9640 1 Processor -- start 2023-10-16T20:39:25.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffba27d9640 1 -- start start 2023-10-16T20:39:25.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffba27d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb9c069e90 0x7ffb9c0c27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:25.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffba27d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb9c06fe20 0x7ffb9c1ed3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:25.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffba27d9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb9c14fd10 0x7ffb9c1dfda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:25.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffba27d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c1ecfd0 con 0x7ffb9c06fe20 2023-10-16T20:39:25.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffba27d9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c0c3960 con 0x7ffb9c14fd10 2023-10-16T20:39:25.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffba27d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c143d50 con 0x7ffb9c069e90 2023-10-16T20:39:25.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffb93fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb9c06fe20 0x7ffb9c1ed3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:25.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffb93fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb9c06fe20 0x7ffb9c1ed3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56736/0 (socket says 172.21.15.88:56736) 2023-10-16T20:39:25.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffb93fff640 1 -- 172.21.15.88:0/3014287913 learned_addr learned my addr 172.21.15.88:0/3014287913 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:25.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffba0d76640 1 --2- 172.21.15.88:0/3014287913 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb9c14fd10 0x7ffb9c1dfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:25.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffb9bfff640 1 --2- 172.21.15.88:0/3014287913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb9c069e90 0x7ffb9c0c27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffb93fff640 1 -- 172.21.15.88:0/3014287913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb9c069e90 msgr2=0x7ffb9c0c27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffb93fff640 1 --2- 172.21.15.88:0/3014287913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb9c069e90 0x7ffb9c0c27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffb93fff640 1 -- 172.21.15.88:0/3014287913 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb9c14fd10 msgr2=0x7ffb9c1dfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffb93fff640 1 --2- 172.21.15.88:0/3014287913 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb9c14fd10 0x7ffb9c1dfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffb93fff640 1 -- 172.21.15.88:0/3014287913 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9400ac40 con 0x7ffb9c06fe20 2023-10-16T20:39:25.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.713+0000 7ffb93fff640 1 --2- 172.21.15.88:0/3014287913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb9c06fe20 0x7ffb9c1ed3f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffb8400c4d0 tx=0x7ffb8400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:25.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.717+0000 7ffb997fa640 1 -- 172.21.15.88:0/3014287913 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb840074e0 con 0x7ffb9c06fe20 2023-10-16T20:39:25.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.717+0000 7ffb997fa640 1 -- 172.21.15.88:0/3014287913 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb84007640 con 0x7ffb9c06fe20 2023-10-16T20:39:25.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.717+0000 7ffb997fa640 1 -- 172.21.15.88:0/3014287913 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb84004a30 con 0x7ffb9c06fe20 2023-10-16T20:39:25.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.717+0000 7ffba27d9640 1 -- 172.21.15.88:0/3014287913 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb9c143ab0 con 0x7ffb9c06fe20 2023-10-16T20:39:25.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.717+0000 7ffba27d9640 1 -- 172.21.15.88:0/3014287913 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb9c1e3090 con 0x7ffb9c06fe20 2023-10-16T20:39:25.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.717+0000 7ffba27d9640 1 -- 172.21.15.88:0/3014287913 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb9c1ed110 con 0x7ffb9c06fe20 2023-10-16T20:39:25.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.717+0000 7ffb997fa640 1 -- 172.21.15.88:0/3014287913 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffb84004b90 con 0x7ffb9c06fe20 2023-10-16T20:39:25.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.717+0000 7ffb997fa640 1 --2- 172.21.15.88:0/3014287913 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ffb7c06a8e0 0x7ffb7c06cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:25.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.717+0000 7ffb997fa640 1 -- 172.21.15.88:0/3014287913 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ffb84083410 con 0x7ffb9c06fe20 2023-10-16T20:39:25.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.717+0000 7ffb9bfff640 1 --2- 172.21.15.88:0/3014287913 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ffb7c06a8e0 0x7ffb7c06cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:25.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.721+0000 7ffb9bfff640 1 --2- 172.21.15.88:0/3014287913 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ffb7c06a8e0 0x7ffb7c06cda0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c007fd0 tx=0x7ffb8c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:25.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.721+0000 7ffb997fa640 1 -- 172.21.15.88:0/3014287913 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffb9c1ed110 con 0x7ffb9c06fe20 2023-10-16T20:39:25.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.905+0000 7ffba27d9640 1 -- 172.21.15.88:0/3014287913 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ffb9c1ed110 con 0x7ffb9c06fe20 2023-10-16T20:39:25.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.905+0000 7ffb997fa640 1 -- 172.21.15.88:0/3014287913 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7ffb9c1ed110 con 0x7ffb9c06fe20 2023-10-16T20:39:25.914 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:25.914 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-16T20:39:25.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.909+0000 7ffba27d9640 1 -- 172.21.15.88:0/3014287913 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ffb7c06a8e0 msgr2=0x7ffb7c06cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:25.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.909+0000 7ffba27d9640 1 --2- 172.21.15.88:0/3014287913 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ffb7c06a8e0 0x7ffb7c06cda0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c007fd0 tx=0x7ffb8c009040 comp rx=0 tx=0).stop 2023-10-16T20:39:25.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.909+0000 7ffba27d9640 1 -- 172.21.15.88:0/3014287913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb9c06fe20 msgr2=0x7ffb9c1ed3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:25.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.909+0000 7ffba27d9640 1 --2- 172.21.15.88:0/3014287913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb9c06fe20 0x7ffb9c1ed3f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffb8400c4d0 tx=0x7ffb8400c990 comp rx=0 tx=0).stop 2023-10-16T20:39:25.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.909+0000 7ffba27d9640 1 -- 172.21.15.88:0/3014287913 shutdown_connections 2023-10-16T20:39:25.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.909+0000 7ffba27d9640 1 --2- 172.21.15.88:0/3014287913 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ffb7c06a8e0 0x7ffb7c06cda0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.909+0000 7ffba27d9640 1 --2- 172.21.15.88:0/3014287913 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ffb9c14fd10 0x7ffb9c1dfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.909+0000 7ffba27d9640 1 --2- 172.21.15.88:0/3014287913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb9c06fe20 0x7ffb9c1ed3f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.909+0000 7ffba27d9640 1 --2- 172.21.15.88:0/3014287913 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ffb9c069e90 0x7ffb9c0c27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:25.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.909+0000 7ffba27d9640 1 -- 172.21.15.88:0/3014287913 >> 172.21.15.88:0/3014287913 conn(0x7ffb9c0cad50 msgr2=0x7ffb9c0c5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:25.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.909+0000 7ffba27d9640 1 -- 172.21.15.88:0/3014287913 shutdown_connections 2023-10-16T20:39:25.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:25.909+0000 7ffba27d9640 1 -- 172.21.15.88:0/3014287913 wait complete. 2023-10-16T20:39:28.935 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-16T20:39:29.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.069+0000 7fbb575f1640 1 Processor -- start 2023-10-16T20:39:29.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.069+0000 7fbb575f1640 1 -- start start 2023-10-16T20:39:29.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.073+0000 7fbb575f1640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbb500c9bb0 0x7fbb500c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:29.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.073+0000 7fbb575f1640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb500c41f0 0x7fbb500cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:29.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.073+0000 7fbb575f1640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb50153b50 con 0x7fbb50153770 2023-10-16T20:39:29.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.073+0000 7fbb575f1640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb500cc250 con 0x7fbb500c9bb0 2023-10-16T20:39:29.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.073+0000 7fbb575f1640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb500cc670 con 0x7fbb500c41f0 2023-10-16T20:39:29.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.073+0000 7fbb55b8e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb500c41f0 0x7fbb500cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:29.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.073+0000 7fbb5538d640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fbb50153770 0x7fbb50159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:54620/0 (socket says 172.21.15.88:54620) 2023-10-16T20:39:29.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.073+0000 7fbb5538d640 1 -- 172.21.15.88:0/988024021 learned_addr learned my addr 172.21.15.88:0/988024021 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:29.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.073+0000 7fbb54b8c640 1 --2- 172.21.15.88:0/988024021 >> v2:172.21.15.191:3300/0 conn(0x7fbb500c9bb0 0x7fbb500c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:29.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.073+0000 7fbb47fff640 1 -- 172.21.15.88:0/988024021 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538098901 0 0) 0x7fbb50153b50 con 0x7fbb50153770 2023-10-16T20:39:29.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.073+0000 7fbb54b8c640 1 -- 172.21.15.88:0/988024021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb500c41f0 msgr2=0x7fbb500cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:29.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.073+0000 7fbb54b8c640 1 --2- 172.21.15.88:0/988024021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb500c41f0 0x7fbb500cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:29.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.073+0000 7fbb54b8c640 1 -- 172.21.15.88:0/988024021 >> v1:172.21.15.191:6789/0 conn(0x7fbb50153770 legacy=0x7fbb50159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:29.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.073+0000 7fbb54b8c640 1 -- 172.21.15.88:0/988024021 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb500cb2f0 con 0x7fbb500c9bb0 2023-10-16T20:39:29.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.073+0000 7fbb55b8e640 1 --2- 172.21.15.88:0/988024021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb500c41f0 0x7fbb500cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:39:29.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.077+0000 7fbb54b8c640 1 --2- 172.21.15.88:0/988024021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb500c9bb0 0x7fbb500c3cb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbb400097c0 tx=0x7fbb4001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43c3c28a432425ff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:29.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.077+0000 7fbb47fff640 1 -- 172.21.15.88:0/988024021 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb400ad640 con 0x7fbb500c9bb0 2023-10-16T20:39:29.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.077+0000 7fbb47fff640 1 -- 172.21.15.88:0/988024021 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb400ad7a0 con 0x7fbb500c9bb0 2023-10-16T20:39:29.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.077+0000 7fbb47fff640 1 -- 172.21.15.88:0/988024021 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb400adb30 con 0x7fbb500c9bb0 2023-10-16T20:39:29.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.077+0000 7fbb575f1640 1 -- 172.21.15.88:0/988024021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb500c9bb0 msgr2=0x7fbb500c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:29.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.077+0000 7fbb575f1640 1 --2- 172.21.15.88:0/988024021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb500c9bb0 0x7fbb500c3cb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbb400097c0 tx=0x7fbb4001c480 comp rx=0 tx=0).stop 2023-10-16T20:39:29.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.077+0000 7fbb575f1640 1 -- 172.21.15.88:0/988024021 shutdown_connections 2023-10-16T20:39:29.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.077+0000 7fbb575f1640 1 --2- 172.21.15.88:0/988024021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb500c41f0 0x7fbb500cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:29.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.077+0000 7fbb575f1640 1 --2- 172.21.15.88:0/988024021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb500c9bb0 0x7fbb500c3cb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:29.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.077+0000 7fbb575f1640 1 -- 172.21.15.88:0/988024021 >> 172.21.15.88:0/988024021 conn(0x7fbb500715c0 msgr2=0x7fbb500719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:29.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.077+0000 7fbb575f1640 1 -- 172.21.15.88:0/988024021 shutdown_connections 2023-10-16T20:39:29.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.077+0000 7fbb575f1640 1 -- 172.21.15.88:0/988024021 wait complete. 2023-10-16T20:39:29.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb575f1640 1 Processor -- start 2023-10-16T20:39:29.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb575f1640 1 -- start start 2023-10-16T20:39:29.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb575f1640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb500c41f0 0x7fbb501468f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:29.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb575f1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb500c9bb0 0x7fbb5014dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:29.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb575f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb50153770 0x7fbb5014e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:29.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb575f1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb50150140 con 0x7fbb50153770 2023-10-16T20:39:29.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb575f1640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5015b0c0 con 0x7fbb500c9bb0 2023-10-16T20:39:29.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb575f1640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb500c62c0 con 0x7fbb500c41f0 2023-10-16T20:39:29.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb55b8e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb50153770 0x7fbb5014e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:29.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb55b8e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb50153770 0x7fbb5014e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47382/0 (socket says 172.21.15.88:47382) 2023-10-16T20:39:29.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb55b8e640 1 -- 172.21.15.88:0/4227390186 learned_addr learned my addr 172.21.15.88:0/4227390186 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:29.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb5538d640 1 --2- 172.21.15.88:0/4227390186 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb500c41f0 0x7fbb501468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:29.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb54b8c640 1 --2- 172.21.15.88:0/4227390186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb500c9bb0 0x7fbb5014dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:29.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb55b8e640 1 -- 172.21.15.88:0/4227390186 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb500c41f0 msgr2=0x7fbb501468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:29.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb55b8e640 1 --2- 172.21.15.88:0/4227390186 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb500c41f0 0x7fbb501468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:29.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb55b8e640 1 -- 172.21.15.88:0/4227390186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb500c9bb0 msgr2=0x7fbb5014dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:29.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb55b8e640 1 --2- 172.21.15.88:0/4227390186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb500c9bb0 0x7fbb5014dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:29.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb55b8e640 1 -- 172.21.15.88:0/4227390186 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb48009a70 con 0x7fbb50153770 2023-10-16T20:39:29.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb55b8e640 1 --2- 172.21.15.88:0/4227390186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb50153770 0x7fbb5014e320 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbb48009e60 tx=0x7fbb4800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:29.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb45ffb640 1 -- 172.21.15.88:0/4227390186 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb48009cd0 con 0x7fbb50153770 2023-10-16T20:39:29.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb45ffb640 1 -- 172.21.15.88:0/4227390186 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4800f040 con 0x7fbb50153770 2023-10-16T20:39:29.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb45ffb640 1 -- 172.21.15.88:0/4227390186 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb480039a0 con 0x7fbb50153770 2023-10-16T20:39:29.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb575f1640 1 -- 172.21.15.88:0/4227390186 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb40009420 con 0x7fbb50153770 2023-10-16T20:39:29.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb575f1640 1 -- 172.21.15.88:0/4227390186 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb5014fb80 con 0x7fbb50153770 2023-10-16T20:39:29.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.081+0000 7fbb575f1640 1 -- 172.21.15.88:0/4227390186 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb50074700 con 0x7fbb50153770 2023-10-16T20:39:29.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.085+0000 7fbb45ffb640 1 -- 172.21.15.88:0/4227390186 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbb48014480 con 0x7fbb50153770 2023-10-16T20:39:29.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.085+0000 7fbb45ffb640 1 --2- 172.21.15.88:0/4227390186 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbb340664a0 0x7fbb34068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:29.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.085+0000 7fbb45ffb640 1 -- 172.21.15.88:0/4227390186 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fbb48084df0 con 0x7fbb50153770 2023-10-16T20:39:29.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.085+0000 7fbb5538d640 1 --2- 172.21.15.88:0/4227390186 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbb340664a0 0x7fbb34068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:29.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.085+0000 7fbb5538d640 1 --2- 172.21.15.88:0/4227390186 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbb340664a0 0x7fbb34068960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c000fc0 tx=0x7fbb4c007cf0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:29.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.089+0000 7fbb45ffb640 1 -- 172.21.15.88:0/4227390186 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbb48051e30 con 0x7fbb50153770 2023-10-16T20:39:29.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.269+0000 7fbb575f1640 1 -- 172.21.15.88:0/4227390186 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbb500c6090 con 0x7fbb50153770 2023-10-16T20:39:29.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.269+0000 7fbb45ffb640 1 -- 172.21.15.88:0/4227390186 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7fbb48018080 con 0x7fbb50153770 2023-10-16T20:39:29.277 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:29.277 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-16T20:39:29.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.269+0000 7fbb575f1640 1 -- 172.21.15.88:0/4227390186 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbb340664a0 msgr2=0x7fbb34068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:29.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.269+0000 7fbb575f1640 1 --2- 172.21.15.88:0/4227390186 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbb340664a0 0x7fbb34068960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c000fc0 tx=0x7fbb4c007cf0 comp rx=0 tx=0).stop 2023-10-16T20:39:29.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.269+0000 7fbb575f1640 1 -- 172.21.15.88:0/4227390186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb50153770 msgr2=0x7fbb5014e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:29.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.269+0000 7fbb575f1640 1 --2- 172.21.15.88:0/4227390186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb50153770 0x7fbb5014e320 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbb48009e60 tx=0x7fbb4800c490 comp rx=0 tx=0).stop 2023-10-16T20:39:29.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.269+0000 7fbb575f1640 1 -- 172.21.15.88:0/4227390186 shutdown_connections 2023-10-16T20:39:29.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.269+0000 7fbb575f1640 1 --2- 172.21.15.88:0/4227390186 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbb340664a0 0x7fbb34068960 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:29.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.269+0000 7fbb575f1640 1 --2- 172.21.15.88:0/4227390186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb50153770 0x7fbb5014e320 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:29.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.269+0000 7fbb575f1640 1 --2- 172.21.15.88:0/4227390186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb500c9bb0 0x7fbb5014dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:29.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.269+0000 7fbb575f1640 1 --2- 172.21.15.88:0/4227390186 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb500c41f0 0x7fbb501468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:29.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.269+0000 7fbb575f1640 1 -- 172.21.15.88:0/4227390186 >> 172.21.15.88:0/4227390186 conn(0x7fbb500715c0 msgr2=0x7fbb50149b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:29.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.269+0000 7fbb575f1640 1 -- 172.21.15.88:0/4227390186 shutdown_connections 2023-10-16T20:39:29.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:29.269+0000 7fbb575f1640 1 -- 172.21.15.88:0/4227390186 wait complete. 2023-10-16T20:39:29.929 INFO:tasks.ceph.mds.f.smithi191.stderr:2023-10-16T20:39:29.927+0000 7fe37f078640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-16T20:39:32.298 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-16T20:39:32.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.437+0000 7f9f25ac9640 1 Processor -- start 2023-10-16T20:39:32.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f25ac9640 1 -- start start 2023-10-16T20:39:32.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f25ac9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f200cece0 0x7f9f200cf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:32.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f25ac9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f20153b50 con 0x7f9f20153770 2023-10-16T20:39:32.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f25ac9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f200cb2f0 con 0x7f9f200c9bb0 2023-10-16T20:39:32.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f25ac9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f200cc250 con 0x7f9f200cece0 2023-10-16T20:39:32.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1f7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9f20153770 0x7f9f20159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54068/0 (socket says 172.21.15.88:54068) 2023-10-16T20:39:32.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1f7fe640 1 -- 172.21.15.88:0/2617863158 learned_addr learned my addr 172.21.15.88:0/2617863158 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:32.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1ffff640 1 --2- 172.21.15.88:0/2617863158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f200cece0 0x7f9f200cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:32.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1e7fc640 1 -- 172.21.15.88:0/2617863158 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967565984 0 0) 0x7f9f20153b50 con 0x7f9f20153770 2023-10-16T20:39:32.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1e7fc640 1 -- 172.21.15.88:0/2617863158 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f08003530 con 0x7f9f20153770 2023-10-16T20:39:32.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1e7fc640 1 -- 172.21.15.88:0/2617863158 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156120182 0 0) 0x7f9f200cb2f0 con 0x7f9f200c9bb0 2023-10-16T20:39:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1e7fc640 1 -- 172.21.15.88:0/2617863158 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f20153b50 con 0x7f9f200c9bb0 2023-10-16T20:39:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1e7fc640 1 -- 172.21.15.88:0/2617863158 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1258011609 0 0) 0x7f9f08003530 con 0x7f9f20153770 2023-10-16T20:39:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1e7fc640 1 -- 172.21.15.88:0/2617863158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f200cece0 msgr2=0x7f9f200cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1e7fc640 1 --2- 172.21.15.88:0/2617863158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f200cece0 0x7f9f200cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1e7fc640 1 -- 172.21.15.88:0/2617863158 >> v1:172.21.15.191:6789/0 conn(0x7f9f200c9bb0 legacy=0x7f9f200c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:32.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1e7fc640 1 -- 172.21.15.88:0/2617863158 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f200cb5d0 con 0x7f9f20153770 2023-10-16T20:39:32.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1e7fc640 1 -- 172.21.15.88:0/2617863158 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f9f10001800 con 0x7f9f20153770 2023-10-16T20:39:32.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1e7fc640 1 -- 172.21.15.88:0/2617863158 >> v1:172.21.15.88:6789/0 conn(0x7f9f20153770 legacy=0x7f9f20159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:32.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1e7fc640 1 --2- 172.21.15.88:0/2617863158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f08003b40 0x7f9f08003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:32.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1e7fc640 1 --2- 172.21.15.88:0/2617863158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f080047d0 0x7f9f08004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:32.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1e7fc640 1 --2- 172.21.15.88:0/2617863158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f200cece0 0x7f9f08005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:32.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1e7fc640 1 -- 172.21.15.88:0/2617863158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f200cc250 con 0x7f9f080047d0 2023-10-16T20:39:32.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1e7fc640 1 -- 172.21.15.88:0/2617863158 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0800ba40 con 0x7f9f08003b40 2023-10-16T20:39:32.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1e7fc640 1 -- 172.21.15.88:0/2617863158 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0800bbd0 con 0x7f9f200cece0 2023-10-16T20:39:32.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1effd640 1 --2- 172.21.15.88:0/2617863158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f080047d0 0x7f9f08004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:32.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1f7fe640 1 --2- 172.21.15.88:0/2617863158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f08003b40 0x7f9f08003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:32.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1ffff640 1 --2- 172.21.15.88:0/2617863158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f200cece0 0x7f9f08005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:32.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1effd640 1 -- 172.21.15.88:0/2617863158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f200cece0 msgr2=0x7f9f08005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:32.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1effd640 1 --2- 172.21.15.88:0/2617863158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f200cece0 0x7f9f08005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:32.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1effd640 1 -- 172.21.15.88:0/2617863158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f08003b40 msgr2=0x7f9f08003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:32.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1effd640 1 --2- 172.21.15.88:0/2617863158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f08003b40 0x7f9f08003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:32.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.441+0000 7f9f1effd640 1 -- 172.21.15.88:0/2617863158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f08003530 con 0x7f9f080047d0 2023-10-16T20:39:32.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.445+0000 7f9f1effd640 1 --2- 172.21.15.88:0/2617863158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f080047d0 0x7f9f08004c50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9f140018d0 tx=0x7f9f1401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eb5261532db7adf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:32.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.445+0000 7f9f1e7fc640 1 -- 172.21.15.88:0/2617863158 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f140052d0 con 0x7f9f080047d0 2023-10-16T20:39:32.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.445+0000 7f9f1e7fc640 1 -- 172.21.15.88:0/2617863158 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f14005450 con 0x7f9f080047d0 2023-10-16T20:39:32.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.445+0000 7f9f1e7fc640 1 -- 172.21.15.88:0/2617863158 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f140a88d0 con 0x7f9f080047d0 2023-10-16T20:39:32.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.445+0000 7f9f25ac9640 1 -- 172.21.15.88:0/2617863158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f080047d0 msgr2=0x7f9f08004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:32.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.445+0000 7f9f25ac9640 1 --2- 172.21.15.88:0/2617863158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f080047d0 0x7f9f08004c50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9f140018d0 tx=0x7f9f1401ec70 comp rx=0 tx=0).stop 2023-10-16T20:39:32.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.445+0000 7f9f1ffff640 1 -- 172.21.15.88:0/2617863158 reap_dead start 2023-10-16T20:39:32.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.445+0000 7f9f25ac9640 1 -- 172.21.15.88:0/2617863158 shutdown_connections 2023-10-16T20:39:32.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.445+0000 7f9f25ac9640 1 -- 172.21.15.88:0/2617863158 >> 172.21.15.88:0/2617863158 conn(0x7f9f200715c0 msgr2=0x7f9f200719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:32.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.445+0000 7f9f25ac9640 1 -- 172.21.15.88:0/2617863158 shutdown_connections 2023-10-16T20:39:32.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.445+0000 7f9f25ac9640 1 -- 172.21.15.88:0/2617863158 wait complete. 2023-10-16T20:39:32.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f25ac9640 1 Processor -- start 2023-10-16T20:39:32.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f25ac9640 1 -- start start 2023-10-16T20:39:32.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f25ac9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f080047d0 0x7f9f201f0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:32.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f25ac9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f200b1b50 0x7f9f200b4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:32.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f25ac9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f20151210 0x7f9f2015a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:32.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f25ac9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201f0cb0 con 0x7f9f080047d0 2023-10-16T20:39:32.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f25ac9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201f0f90 con 0x7f9f200b1b50 2023-10-16T20:39:32.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f25ac9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201f1270 con 0x7f9f20151210 2023-10-16T20:39:32.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f1f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f080047d0 0x7f9f201f0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:32.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f1f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f080047d0 0x7f9f201f0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47406/0 (socket says 172.21.15.88:47406) 2023-10-16T20:39:32.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f1f7fe640 1 -- 172.21.15.88:0/3069753237 learned_addr learned my addr 172.21.15.88:0/3069753237 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:32.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f1effd640 1 --2- 172.21.15.88:0/3069753237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f200b1b50 0x7f9f200b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:32.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f1ffff640 1 --2- 172.21.15.88:0/3069753237 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f20151210 0x7f9f2015a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:32.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f1f7fe640 1 -- 172.21.15.88:0/3069753237 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f20151210 msgr2=0x7f9f2015a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:32.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f1f7fe640 1 --2- 172.21.15.88:0/3069753237 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f20151210 0x7f9f2015a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:32.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f1f7fe640 1 -- 172.21.15.88:0/3069753237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f200b1b50 msgr2=0x7f9f200b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:32.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f1f7fe640 1 --2- 172.21.15.88:0/3069753237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f200b1b50 0x7f9f200b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:32.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f1f7fe640 1 -- 172.21.15.88:0/3069753237 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f201f1550 con 0x7f9f080047d0 2023-10-16T20:39:32.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f1ffff640 1 --2- 172.21.15.88:0/3069753237 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f20151210 0x7f9f2015a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:32.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f1f7fe640 1 --2- 172.21.15.88:0/3069753237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f080047d0 0x7f9f201f0670 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000f860 tx=0x7f9f1000fd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:32.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f07fff640 1 -- 172.21.15.88:0/3069753237 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f100072e0 con 0x7f9f080047d0 2023-10-16T20:39:32.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f07fff640 1 -- 172.21.15.88:0/3069753237 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f10007440 con 0x7f9f080047d0 2023-10-16T20:39:32.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f25ac9640 1 -- 172.21.15.88:0/3069753237 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f201f1830 con 0x7f9f080047d0 2023-10-16T20:39:32.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f07fff640 1 -- 172.21.15.88:0/3069753237 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f100057b0 con 0x7f9f080047d0 2023-10-16T20:39:32.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.449+0000 7f9f25ac9640 1 -- 172.21.15.88:0/3069753237 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f2015aec0 con 0x7f9f080047d0 2023-10-16T20:39:32.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.453+0000 7f9f25ac9640 1 -- 172.21.15.88:0/3069753237 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f20074700 con 0x7f9f080047d0 2023-10-16T20:39:32.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.453+0000 7f9f07fff640 1 -- 172.21.15.88:0/3069753237 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9f10005910 con 0x7f9f080047d0 2023-10-16T20:39:32.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.453+0000 7f9f07fff640 1 --2- 172.21.15.88:0/3069753237 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9f0c0664a0 0x7f9f0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:32.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.453+0000 7f9f07fff640 1 -- 172.21.15.88:0/3069753237 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f9f10084390 con 0x7f9f080047d0 2023-10-16T20:39:32.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.453+0000 7f9f1effd640 1 --2- 172.21.15.88:0/3069753237 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9f0c0664a0 0x7f9f0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:32.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.453+0000 7f9f1effd640 1 --2- 172.21.15.88:0/3069753237 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9f0c0664a0 0x7f9f0c068960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9f140b0000 tx=0x7f9f140aa070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:32.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.457+0000 7f9f07fff640 1 -- 172.21.15.88:0/3069753237 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9f100513c0 con 0x7f9f080047d0 2023-10-16T20:39:32.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.641+0000 7f9f25ac9640 1 -- 172.21.15.88:0/3069753237 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9f2015a3f0 con 0x7f9f080047d0 2023-10-16T20:39:32.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.641+0000 7f9f07fff640 1 -- 172.21.15.88:0/3069753237 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f9f100549e0 con 0x7f9f080047d0 2023-10-16T20:39:32.648 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:32.649 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-10-16T20:39:32.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.641+0000 7f9f25ac9640 1 -- 172.21.15.88:0/3069753237 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9f0c0664a0 msgr2=0x7f9f0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:32.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.641+0000 7f9f25ac9640 1 --2- 172.21.15.88:0/3069753237 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9f0c0664a0 0x7f9f0c068960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9f140b0000 tx=0x7f9f140aa070 comp rx=0 tx=0).stop 2023-10-16T20:39:32.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.641+0000 7f9f25ac9640 1 -- 172.21.15.88:0/3069753237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f080047d0 msgr2=0x7f9f201f0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:32.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.641+0000 7f9f25ac9640 1 --2- 172.21.15.88:0/3069753237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f080047d0 0x7f9f201f0670 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000f860 tx=0x7f9f1000fd20 comp rx=0 tx=0).stop 2023-10-16T20:39:32.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.641+0000 7f9f25ac9640 1 -- 172.21.15.88:0/3069753237 shutdown_connections 2023-10-16T20:39:32.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.641+0000 7f9f25ac9640 1 --2- 172.21.15.88:0/3069753237 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9f0c0664a0 0x7f9f0c068960 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:32.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.641+0000 7f9f25ac9640 1 --2- 172.21.15.88:0/3069753237 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f20151210 0x7f9f2015a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:32.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.641+0000 7f9f25ac9640 1 --2- 172.21.15.88:0/3069753237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f200b1b50 0x7f9f200b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:32.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.641+0000 7f9f25ac9640 1 --2- 172.21.15.88:0/3069753237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f080047d0 0x7f9f201f0670 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:32.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.641+0000 7f9f25ac9640 1 -- 172.21.15.88:0/3069753237 >> 172.21.15.88:0/3069753237 conn(0x7f9f200715c0 msgr2=0x7f9f20146c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:32.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.641+0000 7f9f25ac9640 1 -- 172.21.15.88:0/3069753237 shutdown_connections 2023-10-16T20:39:32.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:32.641+0000 7f9f25ac9640 1 -- 172.21.15.88:0/3069753237 wait complete. 2023-10-16T20:39:34.930 INFO:tasks.ceph.mds.f.smithi191.stderr:2023-10-16T20:39:34.927+0000 7fe37f078640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-10-16T20:39:35.669 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-16T20:39:35.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.809+0000 7fe3d3454640 1 Processor -- start 2023-10-16T20:39:35.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.809+0000 7fe3d3454640 1 -- start start 2023-10-16T20:39:35.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d3454640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3cc153670 0x7fe3cc06a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:35.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d3454640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe3cc0c58d0 0x7fe3cc06d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d3454640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3cc0c5e80 con 0x7fe3cc06d6b0 2023-10-16T20:39:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d3454640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3cc0c6860 con 0x7fe3cc0c58d0 2023-10-16T20:39:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d3454640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc15cb30 con 0x7fe3cc153670 2023-10-16T20:39:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d19f1640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe3cc06d6b0 0x7fe3cc06f810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54080/0 (socket says 172.21.15.88:54080) 2023-10-16T20:39:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d19f1640 1 -- 172.21.15.88:0/1913216684 learned_addr learned my addr 172.21.15.88:0/1913216684 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:35.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d09ef640 1 --2- 172.21.15.88:0/1913216684 >> v2:172.21.15.191:3300/0 conn(0x7fe3cc0c58d0 0x7fe3cc06d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:35.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d11f0640 1 --2- 172.21.15.88:0/1913216684 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3cc153670 0x7fe3cc06a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:35.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3c3fff640 1 -- 172.21.15.88:0/1913216684 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49754776 0 0) 0x7fe3c40014a0 con 0x7fe3cc06d6b0 2023-10-16T20:39:35.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3c3fff640 1 -- 172.21.15.88:0/1913216684 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3b8003530 con 0x7fe3cc06d6b0 2023-10-16T20:39:35.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3c3fff640 1 -- 172.21.15.88:0/1913216684 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3114666095 0 0) 0x7fe3b8003530 con 0x7fe3cc06d6b0 2023-10-16T20:39:35.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3c3fff640 1 -- 172.21.15.88:0/1913216684 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3cc153670 msgr2=0x7fe3cc06a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:35.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3c3fff640 1 --2- 172.21.15.88:0/1913216684 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3cc153670 0x7fe3cc06a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:35.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3c3fff640 1 -- 172.21.15.88:0/1913216684 >> v2:172.21.15.191:3300/0 conn(0x7fe3cc0c58d0 msgr2=0x7fe3cc06d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:35.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3c3fff640 1 --2- 172.21.15.88:0/1913216684 >> v2:172.21.15.191:3300/0 conn(0x7fe3cc0c58d0 0x7fe3cc06d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:35.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3c3fff640 1 -- 172.21.15.88:0/1913216684 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc15ce10 con 0x7fe3cc06d6b0 2023-10-16T20:39:35.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3c3fff640 1 -- 172.21.15.88:0/1913216684 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7fe3c4001b90 con 0x7fe3cc06d6b0 2023-10-16T20:39:35.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3c3fff640 1 -- 172.21.15.88:0/1913216684 >> v1:172.21.15.88:6789/0 conn(0x7fe3cc06d6b0 legacy=0x7fe3cc06f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:35.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3c3fff640 1 --2- 172.21.15.88:0/1913216684 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3b8003980 0x7fe3b8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:35.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3c3fff640 1 --2- 172.21.15.88:0/1913216684 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3b80047d0 0x7fe3b8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:35.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3c3fff640 1 --2- 172.21.15.88:0/1913216684 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3cc153670 0x7fe3b8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:35.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3c3fff640 1 -- 172.21.15.88:0/1913216684 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc0c6860 con 0x7fe3b8003980 2023-10-16T20:39:35.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3c3fff640 1 -- 172.21.15.88:0/1913216684 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc15cb30 con 0x7fe3b80047d0 2023-10-16T20:39:35.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3c3fff640 1 -- 172.21.15.88:0/1913216684 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b800b990 con 0x7fe3cc153670 2023-10-16T20:39:35.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d09ef640 1 --2- 172.21.15.88:0/1913216684 >> v2:172.21.15.191:3300/0 conn(0x7fe3cc0c58d0 0x7fe3cc06d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:39:35.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d11f0640 1 --2- 172.21.15.88:0/1913216684 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3b8003980 0x7fe3b8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:35.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d19f1640 1 --2- 172.21.15.88:0/1913216684 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3cc153670 0x7fe3b8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:35.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d09ef640 1 --2- 172.21.15.88:0/1913216684 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3b80047d0 0x7fe3b8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:35.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d11f0640 1 -- 172.21.15.88:0/1913216684 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3cc153670 msgr2=0x7fe3b8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:35.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d11f0640 1 --2- 172.21.15.88:0/1913216684 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3cc153670 0x7fe3b8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:35.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d11f0640 1 -- 172.21.15.88:0/1913216684 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3b80047d0 msgr2=0x7fe3b8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:35.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d11f0640 1 --2- 172.21.15.88:0/1913216684 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3b80047d0 0x7fe3b8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:35.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.813+0000 7fe3d11f0640 1 -- 172.21.15.88:0/1913216684 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b8003530 con 0x7fe3b8003980 2023-10-16T20:39:35.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.817+0000 7fe3d11f0640 1 --2- 172.21.15.88:0/1913216684 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3b8003980 0x7fe3b8003da0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009ae0 tx=0x7fe3c801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17a8f054c9d1517e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:35.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.817+0000 7fe3c3fff640 1 -- 172.21.15.88:0/1913216684 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c80a9450 con 0x7fe3b8003980 2023-10-16T20:39:35.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.817+0000 7fe3c3fff640 1 -- 172.21.15.88:0/1913216684 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c80a95b0 con 0x7fe3b8003980 2023-10-16T20:39:35.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.817+0000 7fe3c3fff640 1 -- 172.21.15.88:0/1913216684 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c80a9960 con 0x7fe3b8003980 2023-10-16T20:39:35.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.817+0000 7fe3d3454640 1 -- 172.21.15.88:0/1913216684 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3b8003980 msgr2=0x7fe3b8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:35.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.817+0000 7fe3d3454640 1 --2- 172.21.15.88:0/1913216684 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3b8003980 0x7fe3b8003da0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009ae0 tx=0x7fe3c801ca50 comp rx=0 tx=0).stop 2023-10-16T20:39:35.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.817+0000 7fe3d19f1640 1 -- 172.21.15.88:0/1913216684 reap_dead start 2023-10-16T20:39:35.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.817+0000 7fe3d3454640 1 -- 172.21.15.88:0/1913216684 shutdown_connections 2023-10-16T20:39:35.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.817+0000 7fe3d3454640 1 -- 172.21.15.88:0/1913216684 >> 172.21.15.88:0/1913216684 conn(0x7fe3cc0cad50 msgr2=0x7fe3cc0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:35.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.817+0000 7fe3d3454640 1 -- 172.21.15.88:0/1913216684 shutdown_connections 2023-10-16T20:39:35.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.817+0000 7fe3d3454640 1 -- 172.21.15.88:0/1913216684 wait complete. 2023-10-16T20:39:35.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d3454640 1 Processor -- start 2023-10-16T20:39:35.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d3454640 1 -- start start 2023-10-16T20:39:35.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d3454640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3b8003980 0x7fe3cc0c37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:35.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d3454640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3cc162510 0x7fe3cc1f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:35.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d3454640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3cc1586a0 0x7fe3cc1f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:35.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d3454640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc159bf0 con 0x7fe3cc1586a0 2023-10-16T20:39:35.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d3454640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc1f2e80 con 0x7fe3b8003980 2023-10-16T20:39:35.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d3454640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc1f3720 con 0x7fe3cc162510 2023-10-16T20:39:35.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d19f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3cc1586a0 0x7fe3cc1f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:35.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d19f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3cc1586a0 0x7fe3cc1f40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47416/0 (socket says 172.21.15.88:47416) 2023-10-16T20:39:35.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d19f1640 1 -- 172.21.15.88:0/3026021665 learned_addr learned my addr 172.21.15.88:0/3026021665 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:35.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d11f0640 1 --2- 172.21.15.88:0/3026021665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3b8003980 0x7fe3cc0c37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:35.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d09ef640 1 --2- 172.21.15.88:0/3026021665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3cc162510 0x7fe3cc1f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:35.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d19f1640 1 -- 172.21.15.88:0/3026021665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3cc162510 msgr2=0x7fe3cc1f3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:35.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d19f1640 1 --2- 172.21.15.88:0/3026021665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3cc162510 0x7fe3cc1f3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:35.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d19f1640 1 -- 172.21.15.88:0/3026021665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3b8003980 msgr2=0x7fe3cc0c37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:35.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d19f1640 1 --2- 172.21.15.88:0/3026021665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3b8003980 0x7fe3cc0c37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:35.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d19f1640 1 -- 172.21.15.88:0/3026021665 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc1575c0 con 0x7fe3cc1586a0 2023-10-16T20:39:35.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d19f1640 1 --2- 172.21.15.88:0/3026021665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3cc1586a0 0x7fe3cc1f40f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc06e080 tx=0x7fe3c400af10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3c1ffb640 1 -- 172.21.15.88:0/3026021665 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4007900 con 0x7fe3cc1586a0 2023-10-16T20:39:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3c1ffb640 1 -- 172.21.15.88:0/3026021665 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c4007aa0 con 0x7fe3cc1586a0 2023-10-16T20:39:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3c1ffb640 1 -- 172.21.15.88:0/3026021665 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4013630 con 0x7fe3cc1586a0 2023-10-16T20:39:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d3454640 1 -- 172.21.15.88:0/3026021665 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3cc1578a0 con 0x7fe3cc1586a0 2023-10-16T20:39:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d3454640 1 -- 172.21.15.88:0/3026021665 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3cc1f33b0 con 0x7fe3cc1586a0 2023-10-16T20:39:35.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.821+0000 7fe3d3454640 1 -- 172.21.15.88:0/3026021665 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3cc074700 con 0x7fe3cc1586a0 2023-10-16T20:39:35.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.825+0000 7fe3c1ffb640 1 -- 172.21.15.88:0/3026021665 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe3c4013790 con 0x7fe3cc1586a0 2023-10-16T20:39:35.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.825+0000 7fe3c1ffb640 1 --2- 172.21.15.88:0/3026021665 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe3b00664d0 0x7fe3b0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:35.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.825+0000 7fe3c1ffb640 1 -- 172.21.15.88:0/3026021665 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe3c4084830 con 0x7fe3cc1586a0 2023-10-16T20:39:35.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.825+0000 7fe3d11f0640 1 --2- 172.21.15.88:0/3026021665 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe3b00664d0 0x7fe3b0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:35.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.825+0000 7fe3d11f0640 1 --2- 172.21.15.88:0/3026021665 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe3b00664d0 0x7fe3b0068990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe3c80ae000 tx=0x7fe3c80046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:35.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:35.829+0000 7fe3c1ffb640 1 -- 172.21.15.88:0/3026021665 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe3c4051850 con 0x7fe3cc1586a0 2023-10-16T20:39:36.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.013+0000 7fe3d3454640 1 -- 172.21.15.88:0/3026021665 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe3cc1f3860 con 0x7fe3cc1586a0 2023-10-16T20:39:36.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.013+0000 7fe3c1ffb640 1 -- 172.21.15.88:0/3026021665 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fe3c4054e70 con 0x7fe3cc1586a0 2023-10-16T20:39:36.020 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:36.020 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-16T20:39:36.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.013+0000 7fe3d3454640 1 -- 172.21.15.88:0/3026021665 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe3b00664d0 msgr2=0x7fe3b0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.013+0000 7fe3d3454640 1 --2- 172.21.15.88:0/3026021665 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe3b00664d0 0x7fe3b0068990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe3c80ae000 tx=0x7fe3c80046d0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.013+0000 7fe3d3454640 1 -- 172.21.15.88:0/3026021665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3cc1586a0 msgr2=0x7fe3cc1f40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.013+0000 7fe3d3454640 1 --2- 172.21.15.88:0/3026021665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3cc1586a0 0x7fe3cc1f40f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc06e080 tx=0x7fe3c400af10 comp rx=0 tx=0).stop 2023-10-16T20:39:36.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.013+0000 7fe3d3454640 1 -- 172.21.15.88:0/3026021665 shutdown_connections 2023-10-16T20:39:36.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.013+0000 7fe3d3454640 1 --2- 172.21.15.88:0/3026021665 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe3b00664d0 0x7fe3b0068990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.013+0000 7fe3d3454640 1 --2- 172.21.15.88:0/3026021665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3cc1586a0 0x7fe3cc1f40f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.013+0000 7fe3d3454640 1 --2- 172.21.15.88:0/3026021665 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe3cc162510 0x7fe3cc1f3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.013+0000 7fe3d3454640 1 --2- 172.21.15.88:0/3026021665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3b8003980 0x7fe3cc0c37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.013+0000 7fe3d3454640 1 -- 172.21.15.88:0/3026021665 >> 172.21.15.88:0/3026021665 conn(0x7fe3cc0cad50 msgr2=0x7fe3cc0ca0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:36.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.013+0000 7fe3d3454640 1 -- 172.21.15.88:0/3026021665 shutdown_connections 2023-10-16T20:39:36.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.013+0000 7fe3d3454640 1 -- 172.21.15.88:0/3026021665 wait complete. 2023-10-16T20:39:36.040 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-16T20:39:36.040 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:39:36.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.169+0000 7f363899d640 1 Processor -- start 2023-10-16T20:39:36.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.169+0000 7f363899d640 1 -- start start 2023-10-16T20:39:36.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.173+0000 7f363899d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f36340c6ea0 0x7f3634156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.173+0000 7f363899d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f363406e7c0 0x7f363406ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.173+0000 7f363899d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36341599f0 con 0x7f3634150560 2023-10-16T20:39:36.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.173+0000 7f363899d640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36340c7530 con 0x7f36340c6ea0 2023-10-16T20:39:36.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.173+0000 7f363899d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36340c7330 con 0x7f363406e7c0 2023-10-16T20:39:36.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.173+0000 7f3632d9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f363406e7c0 0x7f363406ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.173+0000 7f3629d9b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f36340c6ea0 0x7f3634156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.173+0000 7f3632d9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f363406e7c0 0x7f363406ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:59040/0 (socket says 172.21.15.88:59040) 2023-10-16T20:39:36.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.173+0000 7f3632d9d640 1 -- 172.21.15.88:0/2862027865 learned_addr learned my addr 172.21.15.88:0/2862027865 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:36.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.173+0000 7f363259c640 1 --1- 172.21.15.88:0/2862027865 >> v1:172.21.15.191:6789/0 conn(0x7f3634150560 0x7f363415a1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:54648/0 (socket says 172.21.15.88:54648) 2023-10-16T20:39:36.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.173+0000 7f3629d9b640 1 -- 172.21.15.88:0/2862027865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f363406e7c0 msgr2=0x7f363406ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:36.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.173+0000 7f3629d9b640 1 --2- 172.21.15.88:0/2862027865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f363406e7c0 0x7f363406ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.173+0000 7f3629d9b640 1 -- 172.21.15.88:0/2862027865 >> v1:172.21.15.191:6789/0 conn(0x7f3634150560 legacy=0x7f363415a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.173+0000 7f3629d9b640 1 -- 172.21.15.88:0/2862027865 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36340c7df0 con 0x7f36340c6ea0 2023-10-16T20:39:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.177+0000 7f3629d9b640 1 --2- 172.21.15.88:0/2862027865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36340c6ea0 0x7f3634156ce0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f361c009dc0 tx=0x7f361c01c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea1fa90de0acca56 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.177+0000 7f3631d9b640 1 -- 172.21.15.88:0/2862027865 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c0a6700 con 0x7f36340c6ea0 2023-10-16T20:39:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.177+0000 7f3631d9b640 1 -- 172.21.15.88:0/2862027865 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361c0a6860 con 0x7f36340c6ea0 2023-10-16T20:39:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.177+0000 7f3631d9b640 1 -- 172.21.15.88:0/2862027865 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c0a6bf0 con 0x7f36340c6ea0 2023-10-16T20:39:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.177+0000 7f363899d640 1 -- 172.21.15.88:0/2862027865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36340c6ea0 msgr2=0x7f3634156ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.177+0000 7f363899d640 1 --2- 172.21.15.88:0/2862027865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36340c6ea0 0x7f3634156ce0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f361c009dc0 tx=0x7f361c01c930 comp rx=0 tx=0).stop 2023-10-16T20:39:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.177+0000 7f363899d640 1 -- 172.21.15.88:0/2862027865 shutdown_connections 2023-10-16T20:39:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.177+0000 7f363899d640 1 --2- 172.21.15.88:0/2862027865 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f363406e7c0 0x7f363406ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.177+0000 7f363899d640 1 --2- 172.21.15.88:0/2862027865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36340c6ea0 0x7f3634156ce0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.177+0000 7f363899d640 1 -- 172.21.15.88:0/2862027865 >> 172.21.15.88:0/2862027865 conn(0x7f36340cbeb0 msgr2=0x7f36340cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:36.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.177+0000 7f363899d640 1 -- 172.21.15.88:0/2862027865 shutdown_connections 2023-10-16T20:39:36.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.177+0000 7f363899d640 1 -- 172.21.15.88:0/2862027865 wait complete. 2023-10-16T20:39:36.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.177+0000 7f363899d640 1 Processor -- start 2023-10-16T20:39:36.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.177+0000 7f363899d640 1 -- start start 2023-10-16T20:39:36.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363899d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f363406e7c0 0x7f3634072520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363899d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36340c6ea0 0x7f363413a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363899d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3634150560 0x7f36341393c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363899d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3634141190 con 0x7f363406e7c0 2023-10-16T20:39:36.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363899d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36340ca990 con 0x7f36340c6ea0 2023-10-16T20:39:36.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363899d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36340cac70 con 0x7f3634150560 2023-10-16T20:39:36.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f363406e7c0 0x7f3634072520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f363406e7c0 0x7f3634072520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47428/0 (socket says 172.21.15.88:47428) 2023-10-16T20:39:36.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f3632d9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3634150560 0x7f36341393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363259c640 1 -- 172.21.15.88:0/1232564303 learned_addr learned my addr 172.21.15.88:0/1232564303 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:36.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f3629d9b640 1 --2- 172.21.15.88:0/1232564303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36340c6ea0 0x7f363413a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363259c640 1 -- 172.21.15.88:0/1232564303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3634150560 msgr2=0x7f36341393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363259c640 1 --2- 172.21.15.88:0/1232564303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3634150560 0x7f36341393c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363259c640 1 -- 172.21.15.88:0/1232564303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36340c6ea0 msgr2=0x7f363413a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363259c640 1 --2- 172.21.15.88:0/1232564303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36340c6ea0 0x7f363413a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363259c640 1 -- 172.21.15.88:0/1232564303 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c009a70 con 0x7f363406e7c0 2023-10-16T20:39:36.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363259c640 1 --2- 172.21.15.88:0/1232564303 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f363406e7c0 0x7f3634072520 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f362400c3e0 tx=0x7f362400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:36.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f362b7fe640 1 -- 172.21.15.88:0/1232564303 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3624005c80 con 0x7f363406e7c0 2023-10-16T20:39:36.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f362b7fe640 1 -- 172.21.15.88:0/1232564303 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3624005de0 con 0x7f363406e7c0 2023-10-16T20:39:36.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f362b7fe640 1 -- 172.21.15.88:0/1232564303 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36240072d0 con 0x7f363406e7c0 2023-10-16T20:39:36.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363899d640 1 -- 172.21.15.88:0/1232564303 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36340caf50 con 0x7f363406e7c0 2023-10-16T20:39:36.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363899d640 1 -- 172.21.15.88:0/1232564303 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36340cb510 con 0x7f363406e7c0 2023-10-16T20:39:36.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.181+0000 7f363899d640 1 -- 172.21.15.88:0/1232564303 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3634074700 con 0x7f363406e7c0 2023-10-16T20:39:36.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.185+0000 7f362b7fe640 1 -- 172.21.15.88:0/1232564303 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3624007430 con 0x7f363406e7c0 2023-10-16T20:39:36.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.185+0000 7f362b7fe640 1 --2- 172.21.15.88:0/1232564303 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f36140664a0 0x7f3614068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.185+0000 7f362b7fe640 1 -- 172.21.15.88:0/1232564303 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f36240837c0 con 0x7f363406e7c0 2023-10-16T20:39:36.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.185+0000 7f3629d9b640 1 --2- 172.21.15.88:0/1232564303 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f36140664a0 0x7f3614068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.185+0000 7f3629d9b640 1 --2- 172.21.15.88:0/1232564303 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f36140664a0 0x7f3614068960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f361c0af000 tx=0x7f361c0a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:36.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.189+0000 7f362b7fe640 1 -- 172.21.15.88:0/1232564303 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3624050810 con 0x7f363406e7c0 2023-10-16T20:39:36.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.337+0000 7f363899d640 1 -- 172.21.15.88:0/1232564303 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36340cb6d0 con 0x7f363406e7c0 2023-10-16T20:39:36.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.337+0000 7f362b7fe640 1 -- 172.21.15.88:0/1232564303 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18710 (secure 0 0 0) 0x7f3624053e30 con 0x7f363406e7c0 2023-10-16T20:39:36.346 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:36.346 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:37:37.392549+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:39:36.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.341+0000 7f363899d640 1 -- 172.21.15.88:0/1232564303 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f36140664a0 msgr2=0x7f3614068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.341+0000 7f363899d640 1 --2- 172.21.15.88:0/1232564303 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f36140664a0 0x7f3614068960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f361c0af000 tx=0x7f361c0a7000 comp rx=0 tx=0).stop 2023-10-16T20:39:36.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.341+0000 7f363899d640 1 -- 172.21.15.88:0/1232564303 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f363406e7c0 msgr2=0x7f3634072520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.341+0000 7f363899d640 1 --2- 172.21.15.88:0/1232564303 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f363406e7c0 0x7f3634072520 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f362400c3e0 tx=0x7f362400c8a0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.341+0000 7f363899d640 1 -- 172.21.15.88:0/1232564303 shutdown_connections 2023-10-16T20:39:36.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.341+0000 7f363899d640 1 --2- 172.21.15.88:0/1232564303 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f36140664a0 0x7f3614068960 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.341+0000 7f363899d640 1 --2- 172.21.15.88:0/1232564303 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3634150560 0x7f36341393c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.341+0000 7f363899d640 1 --2- 172.21.15.88:0/1232564303 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36340c6ea0 0x7f363413a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.341+0000 7f363899d640 1 --2- 172.21.15.88:0/1232564303 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f363406e7c0 0x7f3634072520 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.341+0000 7f363899d640 1 -- 172.21.15.88:0/1232564303 >> 172.21.15.88:0/1232564303 conn(0x7f36340cbeb0 msgr2=0x7f36340c8530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:36.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.341+0000 7f363899d640 1 -- 172.21.15.88:0/1232564303 shutdown_connections 2023-10-16T20:39:36.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.341+0000 7f363899d640 1 -- 172.21.15.88:0/1232564303 wait complete. 2023-10-16T20:39:36.367 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T20:36:43.646418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T20:36:53.546513+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T20:36:54.143703+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T20:36:57.314642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T20:39:36.368 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T20:39:36.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.501+0000 7f54e5b3e640 1 Processor -- start 2023-10-16T20:39:36.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.501+0000 7f54e5b3e640 1 -- start start 2023-10-16T20:39:36.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.501+0000 7f54e5b3e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f54e0153780 0x7f54e00c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.501+0000 7f54e5b3e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54e0159490 0x7f54e00c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.501+0000 7f54e5b3e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e0159d90 con 0x7f54e00c5930 2023-10-16T20:39:36.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.501+0000 7f54e5b3e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e015a070 con 0x7f54e0153780 2023-10-16T20:39:36.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.501+0000 7f54e5b3e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e015a350 con 0x7f54e0159490 2023-10-16T20:39:36.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.501+0000 7f54df7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f54e0153780 0x7f54e00c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.501+0000 7f54df7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f54e0153780 0x7f54e00c9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47432/0 (socket says 172.21.15.88:47432) 2023-10-16T20:39:36.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.501+0000 7f54df7fe640 1 -- 172.21.15.88:0/3380391818 learned_addr learned my addr 172.21.15.88:0/3380391818 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:36.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.501+0000 7f54deffd640 1 --2- 172.21.15.88:0/3380391818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54e0159490 0x7f54e00c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.505+0000 7f54df7fe640 1 -- 172.21.15.88:0/3380391818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54e0159490 msgr2=0x7f54e00c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:36.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.505+0000 7f54df7fe640 1 --2- 172.21.15.88:0/3380391818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54e0159490 0x7f54e00c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.505+0000 7f54df7fe640 1 -- 172.21.15.88:0/3380391818 >> v1:172.21.15.191:6789/0 conn(0x7f54e00c5930 legacy=0x7f54e00ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.505+0000 7f54df7fe640 1 -- 172.21.15.88:0/3380391818 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e00c4910 con 0x7f54e0153780 2023-10-16T20:39:36.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.505+0000 7f54deffd640 1 --2- 172.21.15.88:0/3380391818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54e0159490 0x7f54e00c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:36.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.505+0000 7f54df7fe640 1 --2- 172.21.15.88:0/3380391818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54e0153780 0x7f54e00c9be0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f54d0006fd0 tx=0x7f54d001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c1548e9627b7bc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:36.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.505+0000 7f54de7fc640 1 -- 172.21.15.88:0/3380391818 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d00a8a10 con 0x7f54e0153780 2023-10-16T20:39:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.505+0000 7f54de7fc640 1 -- 172.21.15.88:0/3380391818 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d00a8b70 con 0x7f54e0153780 2023-10-16T20:39:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.505+0000 7f54de7fc640 1 -- 172.21.15.88:0/3380391818 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d00b0440 con 0x7f54e0153780 2023-10-16T20:39:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.505+0000 7f54e5b3e640 1 -- 172.21.15.88:0/3380391818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54e0153780 msgr2=0x7f54e00c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.505+0000 7f54e5b3e640 1 --2- 172.21.15.88:0/3380391818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54e0153780 0x7f54e00c9be0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f54d0006fd0 tx=0x7f54d001ed10 comp rx=0 tx=0).stop 2023-10-16T20:39:36.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54e5b3e640 1 -- 172.21.15.88:0/3380391818 shutdown_connections 2023-10-16T20:39:36.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54e5b3e640 1 --2- 172.21.15.88:0/3380391818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54e0159490 0x7f54e00c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54e5b3e640 1 --2- 172.21.15.88:0/3380391818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54e0153780 0x7f54e00c9be0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54e5b3e640 1 -- 172.21.15.88:0/3380391818 >> 172.21.15.88:0/3380391818 conn(0x7f54e006bcb0 msgr2=0x7f54e006c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:36.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54e5b3e640 1 -- 172.21.15.88:0/3380391818 shutdown_connections 2023-10-16T20:39:36.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54e5b3e640 1 -- 172.21.15.88:0/3380391818 wait complete. 2023-10-16T20:39:36.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54e5b3e640 1 Processor -- start 2023-10-16T20:39:36.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54e5b3e640 1 -- start start 2023-10-16T20:39:36.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54e5b3e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54e00c5930 0x7f54e0142a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54e5b3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54e0153780 0x7f54e00c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54e5b3e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54e0159490 0x7f54e014f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54e5b3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e014ee80 con 0x7f54e0153780 2023-10-16T20:39:36.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54e5b3e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e00b3c30 con 0x7f54e0159490 2023-10-16T20:39:36.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54e5b3e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e00b3020 con 0x7f54e00c5930 2023-10-16T20:39:36.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54deffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54e0153780 0x7f54e00c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54deffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54e0153780 0x7f54e00c3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47444/0 (socket says 172.21.15.88:47444) 2023-10-16T20:39:36.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54deffd640 1 -- 172.21.15.88:0/1017232046 learned_addr learned my addr 172.21.15.88:0/1017232046 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:36.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54df7fe640 1 --2- 172.21.15.88:0/1017232046 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54e00c5930 0x7f54e0142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54dffff640 1 --2- 172.21.15.88:0/1017232046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54e0159490 0x7f54e014f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54deffd640 1 -- 172.21.15.88:0/1017232046 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54e00c5930 msgr2=0x7f54e0142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54deffd640 1 --2- 172.21.15.88:0/1017232046 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54e00c5930 0x7f54e0142a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54deffd640 1 -- 172.21.15.88:0/1017232046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54e0159490 msgr2=0x7f54e014f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54deffd640 1 --2- 172.21.15.88:0/1017232046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54e0159490 0x7f54e014f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54deffd640 1 -- 172.21.15.88:0/1017232046 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d0009480 con 0x7f54e0153780 2023-10-16T20:39:36.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.509+0000 7f54deffd640 1 --2- 172.21.15.88:0/1017232046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54e0153780 0x7f54e00c3d50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f54d400d800 tx=0x7f54d400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:36.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.513+0000 7f54c7fff640 1 -- 172.21.15.88:0/1017232046 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d40096c0 con 0x7f54e0153780 2023-10-16T20:39:36.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.513+0000 7f54c7fff640 1 -- 172.21.15.88:0/1017232046 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d4009820 con 0x7f54e0153780 2023-10-16T20:39:36.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.513+0000 7f54e5b3e640 1 -- 172.21.15.88:0/1017232046 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e00b3300 con 0x7f54e0153780 2023-10-16T20:39:36.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.513+0000 7f54c7fff640 1 -- 172.21.15.88:0/1017232046 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d4009bd0 con 0x7f54e0153780 2023-10-16T20:39:36.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.513+0000 7f54e5b3e640 1 -- 172.21.15.88:0/1017232046 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e00b38c0 con 0x7f54e0153780 2023-10-16T20:39:36.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.513+0000 7f54e5b3e640 1 -- 172.21.15.88:0/1017232046 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e0074700 con 0x7f54e0153780 2023-10-16T20:39:36.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.513+0000 7f54c7fff640 1 -- 172.21.15.88:0/1017232046 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f54d4009d30 con 0x7f54e0153780 2023-10-16T20:39:36.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.513+0000 7f54c7fff640 1 --2- 172.21.15.88:0/1017232046 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f54cc0664d0 0x7f54cc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.513+0000 7f54c7fff640 1 -- 172.21.15.88:0/1017232046 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f54d4084740 con 0x7f54e0153780 2023-10-16T20:39:36.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.513+0000 7f54df7fe640 1 --2- 172.21.15.88:0/1017232046 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f54cc0664d0 0x7f54cc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.513+0000 7f54df7fe640 1 --2- 172.21.15.88:0/1017232046 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f54cc0664d0 0x7f54cc068990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f54d00b2000 tx=0x7f54d0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:36.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.517+0000 7f54c7fff640 1 -- 172.21.15.88:0/1017232046 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f54d4051940 con 0x7f54e0153780 2023-10-16T20:39:36.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.669+0000 7f54e5b3e640 1 -- 172.21.15.88:0/1017232046 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f54e00c4bc0 con 0x7f54e0153780 2023-10-16T20:39:36.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.669+0000 7f54c7fff640 1 -- 172.21.15.88:0/1017232046 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f54d4054f60 con 0x7f54e0153780 2023-10-16T20:39:36.677 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-10-16T20:39:36.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.669+0000 7f54e5b3e640 1 -- 172.21.15.88:0/1017232046 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f54cc0664d0 msgr2=0x7f54cc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.669+0000 7f54e5b3e640 1 --2- 172.21.15.88:0/1017232046 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f54cc0664d0 0x7f54cc068990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f54d00b2000 tx=0x7f54d0009040 comp rx=0 tx=0).stop 2023-10-16T20:39:36.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.669+0000 7f54e5b3e640 1 -- 172.21.15.88:0/1017232046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54e0153780 msgr2=0x7f54e00c3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.669+0000 7f54e5b3e640 1 --2- 172.21.15.88:0/1017232046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54e0153780 0x7f54e00c3d50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f54d400d800 tx=0x7f54d400dcc0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.669+0000 7f54e5b3e640 1 -- 172.21.15.88:0/1017232046 shutdown_connections 2023-10-16T20:39:36.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.669+0000 7f54e5b3e640 1 --2- 172.21.15.88:0/1017232046 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f54cc0664d0 0x7f54cc068990 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.669+0000 7f54e5b3e640 1 --2- 172.21.15.88:0/1017232046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54e0159490 0x7f54e014f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.669+0000 7f54e5b3e640 1 --2- 172.21.15.88:0/1017232046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54e0153780 0x7f54e00c3d50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.669+0000 7f54e5b3e640 1 --2- 172.21.15.88:0/1017232046 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f54e00c5930 0x7f54e0142a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.669+0000 7f54e5b3e640 1 -- 172.21.15.88:0/1017232046 >> 172.21.15.88:0/1017232046 conn(0x7f54e006bcb0 msgr2=0x7f54e0148470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:36.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.673+0000 7f54e5b3e640 1 -- 172.21.15.88:0/1017232046 shutdown_connections 2023-10-16T20:39:36.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.673+0000 7f54e5b3e640 1 -- 172.21.15.88:0/1017232046 wait complete. 2023-10-16T20:39:36.698 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T20:39:36.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.829+0000 7f4cd149c640 1 Processor -- start 2023-10-16T20:39:36.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.829+0000 7f4cd149c640 1 -- start start 2023-10-16T20:39:36.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cd149c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ccc159490 0x7f4ccc0c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cd149c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4ccc0c5930 0x7f4ccc0ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cd149c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ccc159d90 con 0x7f4ccc153780 2023-10-16T20:39:36.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cd149c640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ccc15a070 con 0x7f4ccc0c5930 2023-10-16T20:39:36.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cd149c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc15a350 con 0x7f4ccc159490 2023-10-16T20:39:36.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4ccaffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4ccc153780 0x7f4ccc0c9be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54090/0 (socket says 172.21.15.88:54090) 2023-10-16T20:39:36.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4ccaffd640 1 -- 172.21.15.88:0/3588202490 learned_addr learned my addr 172.21.15.88:0/3588202490 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:36.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cca7fc640 1 -- 172.21.15.88:0/3588202490 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508996299 0 0) 0x7f4ccc159d90 con 0x7f4ccc153780 2023-10-16T20:39:36.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cca7fc640 1 -- 172.21.15.88:0/3588202490 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb8003530 con 0x7f4ccc153780 2023-10-16T20:39:36.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4ccb7fe640 1 --2- 172.21.15.88:0/3588202490 >> v2:172.21.15.191:3300/0 conn(0x7f4ccc0c5930 0x7f4ccc0ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cc27fc640 1 --2- 172.21.15.88:0/3588202490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ccc159490 0x7f4ccc0c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cca7fc640 1 -- 172.21.15.88:0/3588202490 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 612322975 0 0) 0x7f4cb8003530 con 0x7f4ccc153780 2023-10-16T20:39:36.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cca7fc640 1 -- 172.21.15.88:0/3588202490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ccc159490 msgr2=0x7f4ccc0c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:36.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cca7fc640 1 --2- 172.21.15.88:0/3588202490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ccc159490 0x7f4ccc0c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cca7fc640 1 -- 172.21.15.88:0/3588202490 >> v2:172.21.15.191:3300/0 conn(0x7f4ccc0c5930 msgr2=0x7f4ccc0ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:36.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cca7fc640 1 --2- 172.21.15.88:0/3588202490 >> v2:172.21.15.191:3300/0 conn(0x7f4ccc0c5930 0x7f4ccc0ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cca7fc640 1 -- 172.21.15.88:0/3588202490 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ccc0c4910 con 0x7f4ccc153780 2023-10-16T20:39:36.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cca7fc640 1 -- 172.21.15.88:0/3588202490 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f4cbc001b80 con 0x7f4ccc153780 2023-10-16T20:39:36.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cca7fc640 1 -- 172.21.15.88:0/3588202490 >> v1:172.21.15.88:6789/0 conn(0x7f4ccc153780 legacy=0x7f4ccc0c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cca7fc640 1 --2- 172.21.15.88:0/3588202490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4cb8003980 0x7f4cb8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cca7fc640 1 --2- 172.21.15.88:0/3588202490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cb8004370 0x7f4cb8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cca7fc640 1 --2- 172.21.15.88:0/3588202490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb8004f70 0x7f4cb800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cca7fc640 1 -- 172.21.15.88:0/3588202490 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc15a070 con 0x7f4cb8004f70 2023-10-16T20:39:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cca7fc640 1 -- 172.21.15.88:0/3588202490 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc15a350 con 0x7f4cb8004370 2023-10-16T20:39:36.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cca7fc640 1 -- 172.21.15.88:0/3588202490 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb800bf00 con 0x7f4cb8003980 2023-10-16T20:39:36.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cc27fc640 1 --2- 172.21.15.88:0/3588202490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ccc159490 0x7f4ccc0c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:36.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4ccb7fe640 1 --2- 172.21.15.88:0/3588202490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb8004f70 0x7f4cb800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4ccaffd640 1 --2- 172.21.15.88:0/3588202490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4cb8003980 0x7f4cb8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4cc27fc640 1 --2- 172.21.15.88:0/3588202490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cb8004370 0x7f4cb8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4ccb7fe640 1 -- 172.21.15.88:0/3588202490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4cb8003980 msgr2=0x7f4cb8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:36.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4ccb7fe640 1 --2- 172.21.15.88:0/3588202490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4cb8003980 0x7f4cb8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4ccb7fe640 1 -- 172.21.15.88:0/3588202490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cb8004370 msgr2=0x7f4cb8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:36.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4ccb7fe640 1 --2- 172.21.15.88:0/3588202490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cb8004370 0x7f4cb8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.833+0000 7f4ccb7fe640 1 -- 172.21.15.88:0/3588202490 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb8003530 con 0x7f4cb8004f70 2023-10-16T20:39:36.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.837+0000 7f4ccb7fe640 1 --2- 172.21.15.88:0/3588202490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb8004f70 0x7f4cb800b990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0b4fe0 tx=0x7f4cc401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58df3ff3ee44c0f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:36.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.837+0000 7f4cca7fc640 1 -- 172.21.15.88:0/3588202490 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc40a5620 con 0x7f4cb8004f70 2023-10-16T20:39:36.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.837+0000 7f4cca7fc640 1 -- 172.21.15.88:0/3588202490 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc40a5780 con 0x7f4cb8004f70 2023-10-16T20:39:36.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.837+0000 7f4cca7fc640 1 -- 172.21.15.88:0/3588202490 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc40a5b30 con 0x7f4cb8004f70 2023-10-16T20:39:36.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.837+0000 7f4cd149c640 1 -- 172.21.15.88:0/3588202490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb8004f70 msgr2=0x7f4cb800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.837+0000 7f4cd149c640 1 --2- 172.21.15.88:0/3588202490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb8004f70 0x7f4cb800b990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0b4fe0 tx=0x7f4cc401c9d0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.837+0000 7f4ccb7fe640 1 -- 172.21.15.88:0/3588202490 reap_dead start 2023-10-16T20:39:36.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.837+0000 7f4cd149c640 1 -- 172.21.15.88:0/3588202490 shutdown_connections 2023-10-16T20:39:36.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.837+0000 7f4cd149c640 1 -- 172.21.15.88:0/3588202490 >> 172.21.15.88:0/3588202490 conn(0x7f4ccc06bcb0 msgr2=0x7f4ccc06c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:36.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.837+0000 7f4cd149c640 1 -- 172.21.15.88:0/3588202490 shutdown_connections 2023-10-16T20:39:36.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.837+0000 7f4cd149c640 1 -- 172.21.15.88:0/3588202490 wait complete. 2023-10-16T20:39:36.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cd149c640 1 Processor -- start 2023-10-16T20:39:36.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cd149c640 1 -- start start 2023-10-16T20:39:36.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cd149c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cb8004f70 0x7f4ccc1f2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cd149c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ccc1eee60 0x7f4ccc1ef2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cd149c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ccc145680 0x7f4ccc145b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cd149c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc1ee9d0 con 0x7f4ccc1eee60 2023-10-16T20:39:36.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cd149c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc1592c0 con 0x7f4cb8004f70 2023-10-16T20:39:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cd149c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc158700 con 0x7f4ccc145680 2023-10-16T20:39:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cc27fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ccc1eee60 0x7f4ccc1ef2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cc27fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ccc1eee60 0x7f4ccc1ef2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47456/0 (socket says 172.21.15.88:47456) 2023-10-16T20:39:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cc27fc640 1 -- 172.21.15.88:0/624175849 learned_addr learned my addr 172.21.15.88:0/624175849 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4ccb7fe640 1 --2- 172.21.15.88:0/624175849 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ccc145680 0x7f4ccc145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4ccaffd640 1 --2- 172.21.15.88:0/624175849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cb8004f70 0x7f4ccc1f2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cc27fc640 1 -- 172.21.15.88:0/624175849 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ccc145680 msgr2=0x7f4ccc145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cc27fc640 1 --2- 172.21.15.88:0/624175849 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ccc145680 0x7f4ccc145b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cc27fc640 1 -- 172.21.15.88:0/624175849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cb8004f70 msgr2=0x7f4ccc1f2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cc27fc640 1 --2- 172.21.15.88:0/624175849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cb8004f70 0x7f4ccc1f2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cc27fc640 1 -- 172.21.15.88:0/624175849 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ccc1ef7e0 con 0x7f4ccc1eee60 2023-10-16T20:39:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cc27fc640 1 --2- 172.21.15.88:0/624175849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ccc1eee60 0x7f4ccc1ef2a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4cb40108a0 tx=0x7f4cb4010d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cc3fff640 1 -- 172.21.15.88:0/624175849 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb400c930 con 0x7f4ccc1eee60 2023-10-16T20:39:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cc3fff640 1 -- 172.21.15.88:0/624175849 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb400ca90 con 0x7f4ccc1eee60 2023-10-16T20:39:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cc3fff640 1 -- 172.21.15.88:0/624175849 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb400ce40 con 0x7f4ccc1eee60 2023-10-16T20:39:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cd149c640 1 -- 172.21.15.88:0/624175849 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ccc158d00 con 0x7f4ccc1eee60 2023-10-16T20:39:36.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cd149c640 1 -- 172.21.15.88:0/624175849 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ccc15d2b0 con 0x7f4ccc1eee60 2023-10-16T20:39:36.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.841+0000 7f4cd149c640 1 -- 172.21.15.88:0/624175849 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ccc0ca7d0 con 0x7f4ccc1eee60 2023-10-16T20:39:36.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.845+0000 7f4cc3fff640 1 -- 172.21.15.88:0/624175849 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4cb4015470 con 0x7f4ccc1eee60 2023-10-16T20:39:36.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.845+0000 7f4cc3fff640 1 --2- 172.21.15.88:0/624175849 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4cac0664d0 0x7f4cac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:36.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.845+0000 7f4cc3fff640 1 -- 172.21.15.88:0/624175849 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f4cb4084ba0 con 0x7f4ccc1eee60 2023-10-16T20:39:36.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.845+0000 7f4ccaffd640 1 --2- 172.21.15.88:0/624175849 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4cac0664d0 0x7f4cac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:36.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.845+0000 7f4ccaffd640 1 --2- 172.21.15.88:0/624175849 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4cac0664d0 0x7f4cac068990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc001470 tx=0x7f4cbc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:36.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.849+0000 7f4cc3fff640 1 -- 172.21.15.88:0/624175849 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4cb4051c80 con 0x7f4ccc1eee60 2023-10-16T20:39:37.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.997+0000 7f4cd149c640 1 -- 172.21.15.88:0/624175849 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4ccc0ce120 con 0x7f4ccc1eee60 2023-10-16T20:39:37.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:36.997+0000 7f4cc3fff640 1 -- 172.21.15.88:0/624175849 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4cb40552a0 con 0x7f4ccc1eee60 2023-10-16T20:39:37.005 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:39:37.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.001+0000 7f4cd149c640 1 -- 172.21.15.88:0/624175849 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4cac0664d0 msgr2=0x7f4cac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:37.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.001+0000 7f4cd149c640 1 --2- 172.21.15.88:0/624175849 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4cac0664d0 0x7f4cac068990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc001470 tx=0x7f4cbc009040 comp rx=0 tx=0).stop 2023-10-16T20:39:37.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.001+0000 7f4cd149c640 1 -- 172.21.15.88:0/624175849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ccc1eee60 msgr2=0x7f4ccc1ef2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:37.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.001+0000 7f4cd149c640 1 --2- 172.21.15.88:0/624175849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ccc1eee60 0x7f4ccc1ef2a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4cb40108a0 tx=0x7f4cb4010d60 comp rx=0 tx=0).stop 2023-10-16T20:39:37.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.001+0000 7f4cd149c640 1 -- 172.21.15.88:0/624175849 shutdown_connections 2023-10-16T20:39:37.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.001+0000 7f4cd149c640 1 --2- 172.21.15.88:0/624175849 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4cac0664d0 0x7f4cac068990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.001+0000 7f4cd149c640 1 --2- 172.21.15.88:0/624175849 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ccc145680 0x7f4ccc145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.001+0000 7f4cd149c640 1 --2- 172.21.15.88:0/624175849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ccc1eee60 0x7f4ccc1ef2a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.001+0000 7f4cd149c640 1 --2- 172.21.15.88:0/624175849 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4cb8004f70 0x7f4ccc1f2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.001+0000 7f4cd149c640 1 -- 172.21.15.88:0/624175849 >> 172.21.15.88:0/624175849 conn(0x7f4ccc06bcb0 msgr2=0x7f4ccc14fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:37.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.001+0000 7f4cd149c640 1 -- 172.21.15.88:0/624175849 shutdown_connections 2023-10-16T20:39:37.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.001+0000 7f4cd149c640 1 -- 172.21.15.88:0/624175849 wait complete. 2023-10-16T20:39:37.025 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T20:39:37.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.157+0000 7f20ef479640 1 Processor -- start 2023-10-16T20:39:37.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.157+0000 7f20ef479640 1 -- start start 2023-10-16T20:39:37.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.161+0000 7f20ef479640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f20e81517e0 0x7f20e81573e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.161+0000 7f20ef479640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e8069e90 con 0x7f20e80c4260 2023-10-16T20:39:37.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.161+0000 7f20ef479640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e806aea0 con 0x7f20e80c4640 2023-10-16T20:39:37.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.161+0000 7f20ef479640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20e80716a0 con 0x7f20e81517e0 2023-10-16T20:39:37.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.161+0000 7f20eda16640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f20e81517e0 0x7f20e81573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.161+0000 7f20eda16640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f20e81517e0 0x7f20e81573e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47462/0 (socket says 172.21.15.88:47462) 2023-10-16T20:39:37.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.161+0000 7f20eda16640 1 -- 172.21.15.88:0/1726096142 learned_addr learned my addr 172.21.15.88:0/1726096142 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:37.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.161+0000 7f20eda16640 1 -- 172.21.15.88:0/1726096142 >> v1:172.21.15.191:6789/0 conn(0x7f20e80c4640 legacy=0x7f20e81510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:37.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.161+0000 7f20eda16640 1 -- 172.21.15.88:0/1726096142 >> v1:172.21.15.88:6789/0 conn(0x7f20e80c4260 legacy=0x7f20e80c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:37.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.161+0000 7f20eda16640 1 -- 172.21.15.88:0/1726096142 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e81581e0 con 0x7f20e81517e0 2023-10-16T20:39:37.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.161+0000 7f20dffff640 1 -- 172.21.15.88:0/1726096142 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148552370 0 0) 0x7f20e8069e90 con 0x7f20e80c4260 2023-10-16T20:39:37.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20eda16640 1 --2- 172.21.15.88:0/1726096142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20e81517e0 0x7f20e81573e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009e50 tx=0x7f20e001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c622696ee1e9612a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:37.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20dffff640 1 -- 172.21.15.88:0/1726096142 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e00a5770 con 0x7f20e81517e0 2023-10-16T20:39:37.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20dffff640 1 -- 172.21.15.88:0/1726096142 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e00a58d0 con 0x7f20e81517e0 2023-10-16T20:39:37.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20dffff640 1 -- 172.21.15.88:0/1726096142 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e00a5c60 con 0x7f20e81517e0 2023-10-16T20:39:37.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20ef479640 1 -- 172.21.15.88:0/1726096142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20e81517e0 msgr2=0x7f20e81573e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:37.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20ef479640 1 --2- 172.21.15.88:0/1726096142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20e81517e0 0x7f20e81573e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009e50 tx=0x7f20e001c8b0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20ef479640 1 -- 172.21.15.88:0/1726096142 shutdown_connections 2023-10-16T20:39:37.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20ef479640 1 --2- 172.21.15.88:0/1726096142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20e81517e0 0x7f20e81573e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20ef479640 1 -- 172.21.15.88:0/1726096142 >> 172.21.15.88:0/1726096142 conn(0x7f20e80ca9f0 msgr2=0x7f20e80cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:37.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20ef479640 1 -- 172.21.15.88:0/1726096142 shutdown_connections 2023-10-16T20:39:37.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20ef479640 1 -- 172.21.15.88:0/1726096142 wait complete. 2023-10-16T20:39:37.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20ef479640 1 Processor -- start 2023-10-16T20:39:37.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20ef479640 1 -- start start 2023-10-16T20:39:37.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20ef479640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20e80c4260 0x7f20e81564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20ef479640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20e80c4640 0x7f20e8149cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20ef479640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f20e81517e0 0x7f20e80c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20ef479640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e8149840 con 0x7f20e80c4640 2023-10-16T20:39:37.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20ef479640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e80c8cc0 con 0x7f20e80c4260 2023-10-16T20:39:37.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20ef479640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e80c8fa0 con 0x7f20e81517e0 2023-10-16T20:39:37.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.165+0000 7f20eca14640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20e80c4640 0x7f20e8149cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20eca14640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20e80c4640 0x7f20e8149cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47474/0 (socket says 172.21.15.88:47474) 2023-10-16T20:39:37.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20eca14640 1 -- 172.21.15.88:0/1535592383 learned_addr learned my addr 172.21.15.88:0/1535592383 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:37.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20ed215640 1 --2- 172.21.15.88:0/1535592383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20e80c4260 0x7f20e81564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20eda16640 1 --2- 172.21.15.88:0/1535592383 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f20e81517e0 0x7f20e80c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20eca14640 1 -- 172.21.15.88:0/1535592383 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f20e81517e0 msgr2=0x7f20e80c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:37.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20eca14640 1 --2- 172.21.15.88:0/1535592383 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f20e81517e0 0x7f20e80c97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20eca14640 1 -- 172.21.15.88:0/1535592383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20e80c4260 msgr2=0x7f20e81564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:37.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20eca14640 1 --2- 172.21.15.88:0/1535592383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20e80c4260 0x7f20e81564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20eca14640 1 -- 172.21.15.88:0/1535592383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e0009b30 con 0x7f20e80c4640 2023-10-16T20:39:37.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20ed215640 1 --2- 172.21.15.88:0/1535592383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20e80c4260 0x7f20e81564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:37.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20eca14640 1 --2- 172.21.15.88:0/1535592383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20e80c4640 0x7f20e8149cd0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f20d800c450 tx=0x7f20d800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:37.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20ddffb640 1 -- 172.21.15.88:0/1535592383 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d8012830 con 0x7f20e80c4640 2023-10-16T20:39:37.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20ddffb640 1 -- 172.21.15.88:0/1535592383 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d80129b0 con 0x7f20e80c4640 2023-10-16T20:39:37.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20ddffb640 1 -- 172.21.15.88:0/1535592383 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d8012d60 con 0x7f20e80c4640 2023-10-16T20:39:37.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20ef479640 1 -- 172.21.15.88:0/1535592383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e80c9280 con 0x7f20e80c4640 2023-10-16T20:39:37.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20ef479640 1 -- 172.21.15.88:0/1535592383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e81494d0 con 0x7f20e80c4640 2023-10-16T20:39:37.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20ef479640 1 -- 172.21.15.88:0/1535592383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e806a680 con 0x7f20e80c4640 2023-10-16T20:39:37.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20ddffb640 1 -- 172.21.15.88:0/1535592383 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f20d801c460 con 0x7f20e80c4640 2023-10-16T20:39:37.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20ddffb640 1 --2- 172.21.15.88:0/1535592383 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f20cc0664a0 0x7f20cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.169+0000 7f20ddffb640 1 -- 172.21.15.88:0/1535592383 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f20d80841d0 con 0x7f20e80c4640 2023-10-16T20:39:37.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.173+0000 7f20ed215640 1 --2- 172.21.15.88:0/1535592383 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f20cc0664a0 0x7f20cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.173+0000 7f20ed215640 1 --2- 172.21.15.88:0/1535592383 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f20cc0664a0 0x7f20cc068960 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f20e4000f70 tx=0x7f20e4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:37.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.173+0000 7f20ddffb640 1 -- 172.21.15.88:0/1535592383 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f20e806a680 con 0x7f20e80c4640 2023-10-16T20:39:37.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.325+0000 7f20ef479640 1 -- 172.21.15.88:0/1535592383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f20e806a680 con 0x7f20e80c4640 2023-10-16T20:39:37.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.325+0000 7f20ddffb640 1 -- 172.21.15.88:0/1535592383 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f20e806a680 con 0x7f20e80c4640 2023-10-16T20:39:37.332 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2023-10-16T20:39:37.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.325+0000 7f20ef479640 1 -- 172.21.15.88:0/1535592383 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f20cc0664a0 msgr2=0x7f20cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:37.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.325+0000 7f20ef479640 1 --2- 172.21.15.88:0/1535592383 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f20cc0664a0 0x7f20cc068960 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f20e4000f70 tx=0x7f20e4009040 comp rx=0 tx=0).stop 2023-10-16T20:39:37.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.325+0000 7f20ef479640 1 -- 172.21.15.88:0/1535592383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20e80c4640 msgr2=0x7f20e8149cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:37.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.325+0000 7f20ef479640 1 --2- 172.21.15.88:0/1535592383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20e80c4640 0x7f20e8149cd0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f20d800c450 tx=0x7f20d800c910 comp rx=0 tx=0).stop 2023-10-16T20:39:37.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.325+0000 7f20ef479640 1 -- 172.21.15.88:0/1535592383 shutdown_connections 2023-10-16T20:39:37.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.325+0000 7f20ef479640 1 --2- 172.21.15.88:0/1535592383 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f20cc0664a0 0x7f20cc068960 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.325+0000 7f20ef479640 1 --2- 172.21.15.88:0/1535592383 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f20e81517e0 0x7f20e80c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.325+0000 7f20ef479640 1 --2- 172.21.15.88:0/1535592383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20e80c4640 0x7f20e8149cd0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.325+0000 7f20ef479640 1 --2- 172.21.15.88:0/1535592383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20e80c4260 0x7f20e81564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.325+0000 7f20ef479640 1 -- 172.21.15.88:0/1535592383 >> 172.21.15.88:0/1535592383 conn(0x7f20e80ca9f0 msgr2=0x7f20e8070c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:37.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.325+0000 7f20ef479640 1 -- 172.21.15.88:0/1535592383 shutdown_connections 2023-10-16T20:39:37.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.325+0000 7f20ef479640 1 -- 172.21.15.88:0/1535592383 wait complete. 2023-10-16T20:39:37.353 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T20:39:37.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.485+0000 7f5879adc640 1 Processor -- start 2023-10-16T20:39:37.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f5879adc640 1 -- start start 2023-10-16T20:39:37.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f5879adc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5874140300 0x7f58740724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f5879adc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f587406a9e0 0x7f58740c3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f5879adc640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f587406adc0 0x7f58740c4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f5879adc640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f587415b610 con 0x7f587406a9e0 2023-10-16T20:39:37.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f5879adc640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5874157060 con 0x7f587406adc0 2023-10-16T20:39:37.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f5879adc640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58740729f0 con 0x7f5874140300 2023-10-16T20:39:37.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f5873fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f587406a9e0 0x7f58740c3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f5873fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f587406a9e0 0x7f58740c3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47480/0 (socket says 172.21.15.88:47480) 2023-10-16T20:39:37.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f5873fff640 1 -- 172.21.15.88:0/2082029564 learned_addr learned my addr 172.21.15.88:0/2082029564 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:37.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f58792db640 1 --2- 172.21.15.88:0/2082029564 >> v2:172.21.15.191:3300/0 conn(0x7f587406adc0 0x7f58740c4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f5878ada640 1 --2- 172.21.15.88:0/2082029564 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5874140300 0x7f58740724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f5873fff640 1 -- 172.21.15.88:0/2082029564 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5874140300 msgr2=0x7f58740724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:37.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f5873fff640 1 --2- 172.21.15.88:0/2082029564 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5874140300 0x7f58740724b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f5873fff640 1 -- 172.21.15.88:0/2082029564 >> v2:172.21.15.191:3300/0 conn(0x7f587406adc0 msgr2=0x7f58740c4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:37.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f5873fff640 1 --2- 172.21.15.88:0/2082029564 >> v2:172.21.15.191:3300/0 conn(0x7f587406adc0 0x7f58740c4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f5873fff640 1 -- 172.21.15.88:0/2082029564 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587415a7f0 con 0x7f587406a9e0 2023-10-16T20:39:37.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.489+0000 7f5878ada640 1 --2- 172.21.15.88:0/2082029564 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5874140300 0x7f58740724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:37.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.493+0000 7f5873fff640 1 --2- 172.21.15.88:0/2082029564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587406a9e0 0x7f58740c3dc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5874071110 tx=0x7f586001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad89367360b0d35 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:37.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.493+0000 7f58737fe640 1 -- 172.21.15.88:0/2082029564 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58600a89b0 con 0x7f587406a9e0 2023-10-16T20:39:37.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.493+0000 7f58737fe640 1 -- 172.21.15.88:0/2082029564 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58600a8b10 con 0x7f587406a9e0 2023-10-16T20:39:37.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.493+0000 7f58737fe640 1 -- 172.21.15.88:0/2082029564 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58600a8ea0 con 0x7f587406a9e0 2023-10-16T20:39:37.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.493+0000 7f5879adc640 1 -- 172.21.15.88:0/2082029564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587406a9e0 msgr2=0x7f58740c3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:37.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.493+0000 7f5879adc640 1 --2- 172.21.15.88:0/2082029564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587406a9e0 0x7f58740c3dc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5874071110 tx=0x7f586001ecf0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.493+0000 7f5879adc640 1 -- 172.21.15.88:0/2082029564 shutdown_connections 2023-10-16T20:39:37.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.493+0000 7f5879adc640 1 --2- 172.21.15.88:0/2082029564 >> v2:172.21.15.191:3300/0 conn(0x7f587406adc0 0x7f58740c4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.493+0000 7f5879adc640 1 --2- 172.21.15.88:0/2082029564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587406a9e0 0x7f58740c3dc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.493+0000 7f5879adc640 1 --2- 172.21.15.88:0/2082029564 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5874140300 0x7f58740724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.493+0000 7f5879adc640 1 -- 172.21.15.88:0/2082029564 >> 172.21.15.88:0/2082029564 conn(0x7f587406d1e0 msgr2=0x7f587406d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:37.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.493+0000 7f5879adc640 1 -- 172.21.15.88:0/2082029564 shutdown_connections 2023-10-16T20:39:37.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.493+0000 7f5879adc640 1 -- 172.21.15.88:0/2082029564 wait complete. 2023-10-16T20:39:37.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.493+0000 7f5879adc640 1 Processor -- start 2023-10-16T20:39:37.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.493+0000 7f5879adc640 1 -- start start 2023-10-16T20:39:37.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5879adc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f587406a9e0 0x7f5874155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5879adc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587406adc0 0x7f58740caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5879adc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5874140300 0x7f58741f1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5879adc640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58740ca690 con 0x7f587406adc0 2023-10-16T20:39:37.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5879adc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58740c97a0 con 0x7f587406a9e0 2023-10-16T20:39:37.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5879adc640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58740c9a10 con 0x7f5874140300 2023-10-16T20:39:37.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5873fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587406adc0 0x7f58740caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5873fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587406adc0 0x7f58740caab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47494/0 (socket says 172.21.15.88:47494) 2023-10-16T20:39:37.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5873fff640 1 -- 172.21.15.88:0/508024413 learned_addr learned my addr 172.21.15.88:0/508024413 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:37.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5878ada640 1 --2- 172.21.15.88:0/508024413 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f587406a9e0 0x7f5874155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f58792db640 1 --2- 172.21.15.88:0/508024413 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5874140300 0x7f58741f1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5873fff640 1 -- 172.21.15.88:0/508024413 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5874140300 msgr2=0x7f58741f1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:37.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5873fff640 1 --2- 172.21.15.88:0/508024413 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5874140300 0x7f58741f1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5873fff640 1 -- 172.21.15.88:0/508024413 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f587406a9e0 msgr2=0x7f5874155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:37.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5873fff640 1 --2- 172.21.15.88:0/508024413 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f587406a9e0 0x7f5874155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5873fff640 1 -- 172.21.15.88:0/508024413 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58600094d0 con 0x7f587406adc0 2023-10-16T20:39:37.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5878ada640 1 --2- 172.21.15.88:0/508024413 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f587406a9e0 0x7f5874155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:37.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5873fff640 1 --2- 172.21.15.88:0/508024413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587406adc0 0x7f58740caab0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f58600b3040 tx=0x7f586001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:37.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f58717fa640 1 -- 172.21.15.88:0/508024413 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58600ba600 con 0x7f587406adc0 2023-10-16T20:39:37.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f58717fa640 1 -- 172.21.15.88:0/508024413 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58600ba760 con 0x7f587406adc0 2023-10-16T20:39:37.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f58717fa640 1 -- 172.21.15.88:0/508024413 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58600bab10 con 0x7f587406adc0 2023-10-16T20:39:37.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5879adc640 1 -- 172.21.15.88:0/508024413 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58740c9cf0 con 0x7f587406adc0 2023-10-16T20:39:37.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5879adc640 1 -- 172.21.15.88:0/508024413 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58740ca2b0 con 0x7f587406adc0 2023-10-16T20:39:37.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.497+0000 7f5879adc640 1 -- 172.21.15.88:0/508024413 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58740ca7d0 con 0x7f587406adc0 2023-10-16T20:39:37.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.501+0000 7f58717fa640 1 -- 172.21.15.88:0/508024413 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f58600bac70 con 0x7f587406adc0 2023-10-16T20:39:37.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.501+0000 7f58717fa640 1 --2- 172.21.15.88:0/508024413 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f58580664a0 0x7f5858068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.501+0000 7f58717fa640 1 -- 172.21.15.88:0/508024413 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f58601228b0 con 0x7f587406adc0 2023-10-16T20:39:37.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.501+0000 7f5878ada640 1 --2- 172.21.15.88:0/508024413 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f58580664a0 0x7f5858068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.501+0000 7f5878ada640 1 --2- 172.21.15.88:0/508024413 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f58580664a0 0x7f5858068960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f586800a9b0 tx=0x7f5868008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:37.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.505+0000 7f58717fa640 1 -- 172.21.15.88:0/508024413 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f58740ca7d0 con 0x7f587406adc0 2023-10-16T20:39:37.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.657+0000 7f5879adc640 1 -- 172.21.15.88:0/508024413 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f58740ca7d0 con 0x7f587406adc0 2023-10-16T20:39:37.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.657+0000 7f58717fa640 1 -- 172.21.15.88:0/508024413 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f58740ca7d0 con 0x7f587406adc0 2023-10-16T20:39:37.667 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:39:37.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.661+0000 7f5879adc640 1 -- 172.21.15.88:0/508024413 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f58580664a0 msgr2=0x7f5858068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:37.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.661+0000 7f5879adc640 1 --2- 172.21.15.88:0/508024413 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f58580664a0 0x7f5858068960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f586800a9b0 tx=0x7f5868008040 comp rx=0 tx=0).stop 2023-10-16T20:39:37.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.661+0000 7f5879adc640 1 -- 172.21.15.88:0/508024413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587406adc0 msgr2=0x7f58740caab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:37.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.661+0000 7f5879adc640 1 --2- 172.21.15.88:0/508024413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587406adc0 0x7f58740caab0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f58600b3040 tx=0x7f586001ba90 comp rx=0 tx=0).stop 2023-10-16T20:39:37.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.661+0000 7f5879adc640 1 -- 172.21.15.88:0/508024413 shutdown_connections 2023-10-16T20:39:37.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.661+0000 7f5879adc640 1 --2- 172.21.15.88:0/508024413 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f58580664a0 0x7f5858068960 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.661+0000 7f5879adc640 1 --2- 172.21.15.88:0/508024413 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5874140300 0x7f58741f1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.661+0000 7f5879adc640 1 --2- 172.21.15.88:0/508024413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587406adc0 0x7f58740caab0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.661+0000 7f5879adc640 1 --2- 172.21.15.88:0/508024413 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f587406a9e0 0x7f5874155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.661+0000 7f5879adc640 1 -- 172.21.15.88:0/508024413 >> 172.21.15.88:0/508024413 conn(0x7f587406d1e0 msgr2=0x7f58741573b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:37.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.661+0000 7f5879adc640 1 -- 172.21.15.88:0/508024413 shutdown_connections 2023-10-16T20:39:37.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.661+0000 7f5879adc640 1 -- 172.21.15.88:0/508024413 wait complete. 2023-10-16T20:39:37.687 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:39:37.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.817+0000 7f226399d640 1 Processor -- start 2023-10-16T20:39:37.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.817+0000 7f226399d640 1 -- start start 2023-10-16T20:39:37.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.817+0000 7f226399d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225c157b10 0x7f225c15e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.817+0000 7f226399d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f225c0c58d0 0x7f225c06d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.817+0000 7f226399d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f225c06d6b0 0x7f225c160e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.817+0000 7f226399d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f225c0c5e10 con 0x7f225c06d6b0 2023-10-16T20:39:37.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.817+0000 7f226399d640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f225c15e810 con 0x7f225c0c58d0 2023-10-16T20:39:37.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.817+0000 7f226399d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225c06f840 con 0x7f225c157b10 2023-10-16T20:39:37.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.817+0000 7f2261f3a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f225c06d6b0 0x7f225c160e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.821+0000 7f2261f3a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f225c06d6b0 0x7f225c160e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47500/0 (socket says 172.21.15.88:47500) 2023-10-16T20:39:37.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.821+0000 7f2261f3a640 1 -- 172.21.15.88:0/4195209514 learned_addr learned my addr 172.21.15.88:0/4195209514 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:37.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.821+0000 7f2261739640 1 --2- 172.21.15.88:0/4195209514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225c157b10 0x7f225c15e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.821+0000 7f2260f38640 1 --2- 172.21.15.88:0/4195209514 >> v2:172.21.15.191:3300/0 conn(0x7f225c0c58d0 0x7f225c06d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.821+0000 7f2261f3a640 1 -- 172.21.15.88:0/4195209514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225c157b10 msgr2=0x7f225c15e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:37.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.821+0000 7f2261f3a640 1 --2- 172.21.15.88:0/4195209514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225c157b10 0x7f225c15e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.821+0000 7f2261f3a640 1 -- 172.21.15.88:0/4195209514 >> v2:172.21.15.191:3300/0 conn(0x7f225c0c58d0 msgr2=0x7f225c06d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:37.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.821+0000 7f2261f3a640 1 --2- 172.21.15.88:0/4195209514 >> v2:172.21.15.191:3300/0 conn(0x7f225c0c58d0 0x7f225c06d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.821+0000 7f2261f3a640 1 -- 172.21.15.88:0/4195209514 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225c06fb20 con 0x7f225c06d6b0 2023-10-16T20:39:37.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.821+0000 7f2260f38640 1 --2- 172.21.15.88:0/4195209514 >> v2:172.21.15.191:3300/0 conn(0x7f225c0c58d0 0x7f225c06d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:37.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.821+0000 7f2261739640 1 --2- 172.21.15.88:0/4195209514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225c157b10 0x7f225c15e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:37.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.821+0000 7f2261f3a640 1 --2- 172.21.15.88:0/4195209514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225c06d6b0 0x7f225c160e70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f225c06f0b0 tx=0x7f225401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab58a035e1a6f08 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:37.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.821+0000 7f2253fff640 1 -- 172.21.15.88:0/4195209514 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22540a56d0 con 0x7f225c06d6b0 2023-10-16T20:39:37.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.821+0000 7f2253fff640 1 -- 172.21.15.88:0/4195209514 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22540a5830 con 0x7f225c06d6b0 2023-10-16T20:39:37.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.821+0000 7f2253fff640 1 -- 172.21.15.88:0/4195209514 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22540a5bc0 con 0x7f225c06d6b0 2023-10-16T20:39:37.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f226399d640 1 -- 172.21.15.88:0/4195209514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225c06d6b0 msgr2=0x7f225c160e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:37.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f226399d640 1 --2- 172.21.15.88:0/4195209514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225c06d6b0 0x7f225c160e70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f225c06f0b0 tx=0x7f225401c940 comp rx=0 tx=0).stop 2023-10-16T20:39:37.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f226399d640 1 -- 172.21.15.88:0/4195209514 shutdown_connections 2023-10-16T20:39:37.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f226399d640 1 --2- 172.21.15.88:0/4195209514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225c06d6b0 0x7f225c160e70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f226399d640 1 --2- 172.21.15.88:0/4195209514 >> v2:172.21.15.191:3300/0 conn(0x7f225c0c58d0 0x7f225c06d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f226399d640 1 --2- 172.21.15.88:0/4195209514 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225c157b10 0x7f225c15e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f226399d640 1 -- 172.21.15.88:0/4195209514 >> 172.21.15.88:0/4195209514 conn(0x7f225c0cad50 msgr2=0x7f225c0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:37.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f226399d640 1 -- 172.21.15.88:0/4195209514 shutdown_connections 2023-10-16T20:39:37.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f226399d640 1 -- 172.21.15.88:0/4195209514 wait complete. 2023-10-16T20:39:37.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f226399d640 1 Processor -- start 2023-10-16T20:39:37.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f226399d640 1 -- start start 2023-10-16T20:39:37.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f226399d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225c06d6b0 0x7f225c1536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f226399d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f225c0c58d0 0x7f225c152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f226399d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225c157b10 0x7f225c1f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f226399d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225c15fbb0 con 0x7f225c06d6b0 2023-10-16T20:39:37.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f226399d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225c0714b0 con 0x7f225c0c58d0 2023-10-16T20:39:37.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f226399d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225c071750 con 0x7f225c157b10 2023-10-16T20:39:37.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f2261739640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225c06d6b0 0x7f225c1536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f2261739640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225c06d6b0 0x7f225c1536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47512/0 (socket says 172.21.15.88:47512) 2023-10-16T20:39:37.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f2261739640 1 -- 172.21.15.88:0/2651761766 learned_addr learned my addr 172.21.15.88:0/2651761766 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:37.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f2260f38640 1 --2- 172.21.15.88:0/2651761766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f225c0c58d0 0x7f225c152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f2261f3a640 1 --2- 172.21.15.88:0/2651761766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225c157b10 0x7f225c1f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f2261739640 1 -- 172.21.15.88:0/2651761766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225c157b10 msgr2=0x7f225c1f70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:37.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f2261739640 1 --2- 172.21.15.88:0/2651761766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225c157b10 0x7f225c1f70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f2261739640 1 -- 172.21.15.88:0/2651761766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f225c0c58d0 msgr2=0x7f225c152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:37.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f2261739640 1 --2- 172.21.15.88:0/2651761766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f225c0c58d0 0x7f225c152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:37.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f2261739640 1 -- 172.21.15.88:0/2651761766 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2254009b30 con 0x7f225c06d6b0 2023-10-16T20:39:37.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f2261f3a640 1 --2- 172.21.15.88:0/2651761766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225c157b10 0x7f225c1f70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:37.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.825+0000 7f2261739640 1 --2- 172.21.15.88:0/2651761766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225c06d6b0 0x7f225c1536e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f225800c7f0 tx=0x7f225800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:37.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.829+0000 7f2251ffb640 1 -- 172.21.15.88:0/2651761766 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258007820 con 0x7f225c06d6b0 2023-10-16T20:39:37.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.829+0000 7f2251ffb640 1 -- 172.21.15.88:0/2651761766 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2258007980 con 0x7f225c06d6b0 2023-10-16T20:39:37.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.829+0000 7f2251ffb640 1 -- 172.21.15.88:0/2651761766 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258007d30 con 0x7f225c06d6b0 2023-10-16T20:39:37.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.829+0000 7f226399d640 1 -- 172.21.15.88:0/2651761766 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f225c15f5f0 con 0x7f225c06d6b0 2023-10-16T20:39:37.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.829+0000 7f226399d640 1 -- 172.21.15.88:0/2651761766 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f225c15d290 con 0x7f225c06d6b0 2023-10-16T20:39:37.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.829+0000 7f226399d640 1 -- 172.21.15.88:0/2651761766 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f225c15d0f0 con 0x7f225c06d6b0 2023-10-16T20:39:37.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.829+0000 7f2251ffb640 1 -- 172.21.15.88:0/2651761766 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f22580113f0 con 0x7f225c06d6b0 2023-10-16T20:39:37.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.829+0000 7f2251ffb640 1 --2- 172.21.15.88:0/2651761766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f22400664a0 0x7f2240068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:37.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.829+0000 7f2251ffb640 1 -- 172.21.15.88:0/2651761766 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2258084590 con 0x7f225c06d6b0 2023-10-16T20:39:37.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.829+0000 7f2260f38640 1 --2- 172.21.15.88:0/2651761766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f22400664a0 0x7f2240068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:37.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.829+0000 7f2260f38640 1 --2- 172.21.15.88:0/2651761766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f22400664a0 0x7f2240068960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f225c154480 tx=0x7f224c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:37.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:37.833+0000 7f2251ffb640 1 -- 172.21.15.88:0/2651761766 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f225c15d0f0 con 0x7f225c06d6b0 2023-10-16T20:39:38.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.013+0000 7f226399d640 1 -- 172.21.15.88:0/2651761766 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f225c15d0f0 con 0x7f225c06d6b0 2023-10-16T20:39:38.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.013+0000 7f2251ffb640 1 -- 172.21.15.88:0/2651761766 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+6560 (secure 0 0 0) 0x7f225c15d0f0 con 0x7f225c06d6b0 2023-10-16T20:39:38.022 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:38.022 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:39:34.506742+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":18,"state":"up:creating","state_seq":1,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:39:38.023 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 18 2023-10-16T20:39:38.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.017+0000 7f226399d640 1 -- 172.21.15.88:0/2651761766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f22400664a0 msgr2=0x7f2240068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.017+0000 7f226399d640 1 --2- 172.21.15.88:0/2651761766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f22400664a0 0x7f2240068960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f225c154480 tx=0x7f224c008040 comp rx=0 tx=0).stop 2023-10-16T20:39:38.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.017+0000 7f226399d640 1 -- 172.21.15.88:0/2651761766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225c06d6b0 msgr2=0x7f225c1536e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.017+0000 7f226399d640 1 --2- 172.21.15.88:0/2651761766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225c06d6b0 0x7f225c1536e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f225800c7f0 tx=0x7f225800ccb0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.017+0000 7f226399d640 1 -- 172.21.15.88:0/2651761766 shutdown_connections 2023-10-16T20:39:38.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.017+0000 7f226399d640 1 --2- 172.21.15.88:0/2651761766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f22400664a0 0x7f2240068960 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.017+0000 7f226399d640 1 --2- 172.21.15.88:0/2651761766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225c157b10 0x7f225c1f70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.017+0000 7f226399d640 1 --2- 172.21.15.88:0/2651761766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f225c0c58d0 0x7f225c152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.017+0000 7f226399d640 1 --2- 172.21.15.88:0/2651761766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225c06d6b0 0x7f225c1536e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.017+0000 7f226399d640 1 -- 172.21.15.88:0/2651761766 >> 172.21.15.88:0/2651761766 conn(0x7f225c0cad50 msgr2=0x7f225c153d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:38.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.017+0000 7f226399d640 1 -- 172.21.15.88:0/2651761766 shutdown_connections 2023-10-16T20:39:38.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.017+0000 7f226399d640 1 -- 172.21.15.88:0/2651761766 wait complete. 2023-10-16T20:39:38.043 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:39:38.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.173+0000 7f0a984c9640 1 Processor -- start 2023-10-16T20:39:38.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.173+0000 7f0a984c9640 1 -- start start 2023-10-16T20:39:38.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.177+0000 7f0a984c9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0a90157b10 0x7f0a9015e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:38.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.177+0000 7f0a984c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a900c5e10 con 0x7f0a90160e70 2023-10-16T20:39:38.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.177+0000 7f0a984c9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9015e810 con 0x7f0a900c58d0 2023-10-16T20:39:38.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.177+0000 7f0a984c9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a9006f840 con 0x7f0a90157b10 2023-10-16T20:39:38.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.177+0000 7f0a96a66640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0a90160e70 0x7f0a901612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54106/0 (socket says 172.21.15.88:54106) 2023-10-16T20:39:38.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.177+0000 7f0a96a66640 1 -- 172.21.15.88:0/869531925 learned_addr learned my addr 172.21.15.88:0/869531925 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:38.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.177+0000 7f0a96265640 1 --2- 172.21.15.88:0/869531925 >> v2:172.21.15.191:3300/0 conn(0x7f0a90157b10 0x7f0a9015e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:38.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.177+0000 7f0a95263640 1 -- 172.21.15.88:0/869531925 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442606735 0 0) 0x7f0a900c5e10 con 0x7f0a90160e70 2023-10-16T20:39:38.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.177+0000 7f0a95263640 1 -- 172.21.15.88:0/869531925 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a7c003530 con 0x7f0a90160e70 2023-10-16T20:39:38.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.177+0000 7f0a95263640 1 -- 172.21.15.88:0/869531925 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928643588 0 0) 0x7f0a9015e810 con 0x7f0a900c58d0 2023-10-16T20:39:38.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.177+0000 7f0a95263640 1 -- 172.21.15.88:0/869531925 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a900c5e10 con 0x7f0a900c58d0 2023-10-16T20:39:38.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.177+0000 7f0a96265640 1 -- 172.21.15.88:0/869531925 >> v1:172.21.15.191:6789/0 conn(0x7f0a900c58d0 legacy=0x7f0a9006d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.177+0000 7f0a96265640 1 -- 172.21.15.88:0/869531925 >> v1:172.21.15.88:6789/0 conn(0x7f0a90160e70 legacy=0x7f0a901612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.177+0000 7f0a96265640 1 -- 172.21.15.88:0/869531925 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9006fb20 con 0x7f0a90157b10 2023-10-16T20:39:38.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.181+0000 7f0a96265640 1 --2- 172.21.15.88:0/869531925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a90157b10 0x7f0a9015e2d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0097c0 tx=0x7f0a8c01c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e433dd84817636e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:38.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.181+0000 7f0a95263640 1 -- 172.21.15.88:0/869531925 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c0ac620 con 0x7f0a90157b10 2023-10-16T20:39:38.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.181+0000 7f0a95263640 1 -- 172.21.15.88:0/869531925 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8c0ac780 con 0x7f0a90157b10 2023-10-16T20:39:38.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.181+0000 7f0a95263640 1 -- 172.21.15.88:0/869531925 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c0acb10 con 0x7f0a90157b10 2023-10-16T20:39:38.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.181+0000 7f0a984c9640 1 -- 172.21.15.88:0/869531925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a90157b10 msgr2=0x7f0a9015e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.181+0000 7f0a984c9640 1 --2- 172.21.15.88:0/869531925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a90157b10 0x7f0a9015e2d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0097c0 tx=0x7f0a8c01c3c0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.181+0000 7f0a984c9640 1 -- 172.21.15.88:0/869531925 shutdown_connections 2023-10-16T20:39:38.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.181+0000 7f0a984c9640 1 --2- 172.21.15.88:0/869531925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a90157b10 0x7f0a9015e2d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.181+0000 7f0a984c9640 1 -- 172.21.15.88:0/869531925 >> 172.21.15.88:0/869531925 conn(0x7f0a900cad50 msgr2=0x7f0a900cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:38.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.181+0000 7f0a984c9640 1 -- 172.21.15.88:0/869531925 shutdown_connections 2023-10-16T20:39:38.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.181+0000 7f0a984c9640 1 -- 172.21.15.88:0/869531925 wait complete. 2023-10-16T20:39:38.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.181+0000 7f0a984c9640 1 Processor -- start 2023-10-16T20:39:38.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.181+0000 7f0a984c9640 1 -- start start 2023-10-16T20:39:38.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.181+0000 7f0a984c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a900c58d0 0x7f0a9014f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:38.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.181+0000 7f0a984c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a90157b10 0x7f0a9014c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:38.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.181+0000 7f0a984c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a90160e70 0x7f0a901f9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:38.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a984c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a900c2ce0 con 0x7f0a90160e70 2023-10-16T20:39:38.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a984c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a900c3170 con 0x7f0a90157b10 2023-10-16T20:39:38.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a984c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9014e590 con 0x7f0a900c58d0 2023-10-16T20:39:38.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a96a66640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a90160e70 0x7f0a901f9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:38.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a96a66640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a90160e70 0x7f0a901f9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47526/0 (socket says 172.21.15.88:47526) 2023-10-16T20:39:38.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a96a66640 1 -- 172.21.15.88:0/1397633306 learned_addr learned my addr 172.21.15.88:0/1397633306 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:38.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a95a64640 1 --2- 172.21.15.88:0/1397633306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a90157b10 0x7f0a9014c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:38.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a96265640 1 --2- 172.21.15.88:0/1397633306 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a900c58d0 0x7f0a9014f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:38.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a96a66640 1 -- 172.21.15.88:0/1397633306 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a900c58d0 msgr2=0x7f0a9014f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a96a66640 1 --2- 172.21.15.88:0/1397633306 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a900c58d0 0x7f0a9014f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a96a66640 1 -- 172.21.15.88:0/1397633306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a90157b10 msgr2=0x7f0a9014c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a96a66640 1 --2- 172.21.15.88:0/1397633306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a90157b10 0x7f0a9014c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a96a66640 1 -- 172.21.15.88:0/1397633306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8c009420 con 0x7f0a90160e70 2023-10-16T20:39:38.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a96a66640 1 --2- 172.21.15.88:0/1397633306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a90160e70 0x7f0a901f9a20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0a8800cba0 tx=0x7f0a8800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:38.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a86ffd640 1 -- 172.21.15.88:0/1397633306 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8800ace0 con 0x7f0a90160e70 2023-10-16T20:39:38.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a86ffd640 1 -- 172.21.15.88:0/1397633306 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8800ae40 con 0x7f0a90160e70 2023-10-16T20:39:38.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a86ffd640 1 -- 172.21.15.88:0/1397633306 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a88012620 con 0x7f0a90160e70 2023-10-16T20:39:38.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a984c9640 1 -- 172.21.15.88:0/1397633306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a900c20d0 con 0x7f0a90160e70 2023-10-16T20:39:38.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a984c9640 1 -- 172.21.15.88:0/1397633306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a900c2690 con 0x7f0a90160e70 2023-10-16T20:39:38.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a984c9640 1 -- 172.21.15.88:0/1397633306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a900c2e50 con 0x7f0a90160e70 2023-10-16T20:39:38.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.185+0000 7f0a86ffd640 1 -- 172.21.15.88:0/1397633306 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0a88007070 con 0x7f0a90160e70 2023-10-16T20:39:38.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.189+0000 7f0a86ffd640 1 --2- 172.21.15.88:0/1397633306 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0a740664a0 0x7f0a74068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:38.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.189+0000 7f0a86ffd640 1 -- 172.21.15.88:0/1397633306 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0a880843d0 con 0x7f0a90160e70 2023-10-16T20:39:38.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.189+0000 7f0a96265640 1 --2- 172.21.15.88:0/1397633306 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0a740664a0 0x7f0a74068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:38.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.189+0000 7f0a96265640 1 --2- 172.21.15.88:0/1397633306 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0a740664a0 0x7f0a74068960 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0ae000 tx=0x7f0a8c0a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:38.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.193+0000 7f0a86ffd640 1 -- 172.21.15.88:0/1397633306 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0a88054df0 con 0x7f0a90160e70 2023-10-16T20:39:38.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.345+0000 7f0a984c9640 1 -- 172.21.15.88:0/1397633306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a90074700 con 0x7f0a90160e70 2023-10-16T20:39:38.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.345+0000 7f0a86ffd640 1 -- 172.21.15.88:0/1397633306 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18710 (secure 0 0 0) 0x7f0a88017090 con 0x7f0a90160e70 2023-10-16T20:39:38.353 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:38.354 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:37:37.392549+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:39:38.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.349+0000 7f0a984c9640 1 -- 172.21.15.88:0/1397633306 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0a740664a0 msgr2=0x7f0a74068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.349+0000 7f0a984c9640 1 --2- 172.21.15.88:0/1397633306 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0a740664a0 0x7f0a74068960 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0ae000 tx=0x7f0a8c0a7000 comp rx=0 tx=0).stop 2023-10-16T20:39:38.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.349+0000 7f0a984c9640 1 -- 172.21.15.88:0/1397633306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a90160e70 msgr2=0x7f0a901f9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.349+0000 7f0a984c9640 1 --2- 172.21.15.88:0/1397633306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a90160e70 0x7f0a901f9a20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0a8800cba0 tx=0x7f0a8800a430 comp rx=0 tx=0).stop 2023-10-16T20:39:38.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.349+0000 7f0a984c9640 1 -- 172.21.15.88:0/1397633306 shutdown_connections 2023-10-16T20:39:38.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.349+0000 7f0a984c9640 1 --2- 172.21.15.88:0/1397633306 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0a740664a0 0x7f0a74068960 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.349+0000 7f0a984c9640 1 --2- 172.21.15.88:0/1397633306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a90160e70 0x7f0a901f9a20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.349+0000 7f0a984c9640 1 --2- 172.21.15.88:0/1397633306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a90157b10 0x7f0a9014c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.349+0000 7f0a984c9640 1 --2- 172.21.15.88:0/1397633306 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a900c58d0 0x7f0a9014f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.349+0000 7f0a984c9640 1 -- 172.21.15.88:0/1397633306 >> 172.21.15.88:0/1397633306 conn(0x7f0a900cad50 msgr2=0x7f0a9006d140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:38.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.349+0000 7f0a984c9640 1 -- 172.21.15.88:0/1397633306 shutdown_connections 2023-10-16T20:39:38.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.349+0000 7f0a984c9640 1 -- 172.21.15.88:0/1397633306 wait complete. 2023-10-16T20:39:38.375 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T20:36:43.646418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T20:36:53.546513+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T20:36:54.143703+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T20:36:57.314642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T20:39:38.375 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T20:39:38.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.509+0000 7f74cb59e640 1 Processor -- start 2023-10-16T20:39:38.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.509+0000 7f74cb59e640 1 -- start start 2023-10-16T20:39:38.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.509+0000 7f74cb59e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f74cc153630 0x7f74cc159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:38.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.509+0000 7f74cb59e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f74cc0c3ea0 0x7f74cc0c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:38.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.509+0000 7f74cb59e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74cc15a310 con 0x7f74cc0ca190 2023-10-16T20:39:38.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.509+0000 7f74cb59e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74cc0c4ed0 con 0x7f74cc0c3ea0 2023-10-16T20:39:38.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.509+0000 7f74cb59e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74cc0c9830 con 0x7f74cc153630 2023-10-16T20:39:38.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.509+0000 7f74c1d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f74cc0c3ea0 0x7f74cc0c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:38.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.509+0000 7f74cad9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f74cc0ca190 0x7f74cc06a1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54108/0 (socket says 172.21.15.88:54108) 2023-10-16T20:39:38.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.509+0000 7f74cad9d640 1 -- 172.21.15.88:0/1339693890 learned_addr learned my addr 172.21.15.88:0/1339693890 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:38.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.509+0000 7f74ca59c640 1 --2- 172.21.15.88:0/1339693890 >> v2:172.21.15.191:3300/0 conn(0x7f74cc153630 0x7f74cc159dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:38.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.513+0000 7f74c9d9b640 1 -- 172.21.15.88:0/1339693890 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963614219 0 0) 0x7f74cc15a310 con 0x7f74cc0ca190 2023-10-16T20:39:38.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.513+0000 7f74c9d9b640 1 -- 172.21.15.88:0/1339693890 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b8003530 con 0x7f74cc0ca190 2023-10-16T20:39:38.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.513+0000 7f74c1d9b640 1 -- 172.21.15.88:0/1339693890 >> v2:172.21.15.191:3300/0 conn(0x7f74cc153630 msgr2=0x7f74cc159dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:38.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.513+0000 7f74c1d9b640 1 --2- 172.21.15.88:0/1339693890 >> v2:172.21.15.191:3300/0 conn(0x7f74cc153630 0x7f74cc159dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.513+0000 7f74c1d9b640 1 -- 172.21.15.88:0/1339693890 >> v1:172.21.15.88:6789/0 conn(0x7f74cc0ca190 legacy=0x7f74cc06a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.513+0000 7f74c1d9b640 1 -- 172.21.15.88:0/1339693890 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc0ca5e0 con 0x7f74cc0c3ea0 2023-10-16T20:39:38.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.517+0000 7f74c1d9b640 1 --2- 172.21.15.88:0/1339693890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74cc0c3ea0 0x7f74cc0c9c50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0cc7b0 tx=0x7f74b401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2744b3c98217fd2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:38.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.517+0000 7f74c9d9b640 1 -- 172.21.15.88:0/1339693890 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b40a7070 con 0x7f74cc0c3ea0 2023-10-16T20:39:38.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.517+0000 7f74c9d9b640 1 -- 172.21.15.88:0/1339693890 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b40ab430 con 0x7f74cc0c3ea0 2023-10-16T20:39:38.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.517+0000 7f74c9d9b640 1 -- 172.21.15.88:0/1339693890 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b40ab7c0 con 0x7f74cc0c3ea0 2023-10-16T20:39:38.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.517+0000 7f74cb59e640 1 -- 172.21.15.88:0/1339693890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74cc0c3ea0 msgr2=0x7f74cc0c9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.517+0000 7f74cb59e640 1 --2- 172.21.15.88:0/1339693890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74cc0c3ea0 0x7f74cc0c9c50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0cc7b0 tx=0x7f74b401c430 comp rx=0 tx=0).stop 2023-10-16T20:39:38.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.517+0000 7f74cb59e640 1 -- 172.21.15.88:0/1339693890 shutdown_connections 2023-10-16T20:39:38.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.517+0000 7f74cb59e640 1 --2- 172.21.15.88:0/1339693890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74cc0c3ea0 0x7f74cc0c9c50 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.517+0000 7f74cb59e640 1 --2- 172.21.15.88:0/1339693890 >> v2:172.21.15.191:3300/0 conn(0x7f74cc153630 0x7f74cc159dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.517+0000 7f74cb59e640 1 -- 172.21.15.88:0/1339693890 >> 172.21.15.88:0/1339693890 conn(0x7f74cc06bc00 msgr2=0x7f74cc06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:38.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.517+0000 7f74cb59e640 1 -- 172.21.15.88:0/1339693890 shutdown_connections 2023-10-16T20:39:38.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.517+0000 7f74cb59e640 1 -- 172.21.15.88:0/1339693890 wait complete. 2023-10-16T20:39:38.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74cb59e640 1 Processor -- start 2023-10-16T20:39:38.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74cb59e640 1 -- start start 2023-10-16T20:39:38.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74cb59e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74cc0c3ea0 0x7f74cc0c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:38.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74cb59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74cc0ca190 0x7f74cc148170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:38.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74cb59e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74cc153630 0x7f74cc1486b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:38.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74cb59e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc147990 con 0x7f74cc0ca190 2023-10-16T20:39:38.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74cb59e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc14db60 con 0x7f74cc0c3ea0 2023-10-16T20:39:38.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74cb59e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc14ed10 con 0x7f74cc153630 2023-10-16T20:39:38.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74c1d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74cc0ca190 0x7f74cc148170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:38.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74c1d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74cc0ca190 0x7f74cc148170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47536/0 (socket says 172.21.15.88:47536) 2023-10-16T20:39:38.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74c1d9b640 1 -- 172.21.15.88:0/473510020 learned_addr learned my addr 172.21.15.88:0/473510020 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:38.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74ca59c640 1 --2- 172.21.15.88:0/473510020 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74cc0c3ea0 0x7f74cc0c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:38.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74cad9d640 1 --2- 172.21.15.88:0/473510020 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74cc153630 0x7f74cc1486b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:38.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74c1d9b640 1 -- 172.21.15.88:0/473510020 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74cc153630 msgr2=0x7f74cc1486b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74c1d9b640 1 --2- 172.21.15.88:0/473510020 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74cc153630 0x7f74cc1486b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74c1d9b640 1 -- 172.21.15.88:0/473510020 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74cc0c3ea0 msgr2=0x7f74cc0c76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74c1d9b640 1 --2- 172.21.15.88:0/473510020 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74cc0c3ea0 0x7f74cc0c76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74c1d9b640 1 -- 172.21.15.88:0/473510020 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b4009420 con 0x7f74cc0ca190 2023-10-16T20:39:38.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74cad9d640 1 --2- 172.21.15.88:0/473510020 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74cc153630 0x7f74cc1486b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:38.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74c1d9b640 1 --2- 172.21.15.88:0/473510020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74cc0ca190 0x7f74cc148170 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f74b401a060 tx=0x7f74b40191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:38.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74c37fe640 1 -- 172.21.15.88:0/473510020 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b40b45f0 con 0x7f74cc0ca190 2023-10-16T20:39:38.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74c37fe640 1 -- 172.21.15.88:0/473510020 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b40b4770 con 0x7f74cc0ca190 2023-10-16T20:39:38.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74c37fe640 1 -- 172.21.15.88:0/473510020 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b40b4b20 con 0x7f74cc0ca190 2023-10-16T20:39:38.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74cb59e640 1 -- 172.21.15.88:0/473510020 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74cc14efb0 con 0x7f74cc0ca190 2023-10-16T20:39:38.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.521+0000 7f74cb59e640 1 -- 172.21.15.88:0/473510020 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74cc14e090 con 0x7f74cc0ca190 2023-10-16T20:39:38.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.525+0000 7f74cb59e640 1 -- 172.21.15.88:0/473510020 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74cc04e9c0 con 0x7f74cc0ca190 2023-10-16T20:39:38.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.525+0000 7f74c37fe640 1 -- 172.21.15.88:0/473510020 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f74b40b4c80 con 0x7f74cc0ca190 2023-10-16T20:39:38.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.525+0000 7f74c37fe640 1 --2- 172.21.15.88:0/473510020 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74ac0664a0 0x7f74ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:38.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.525+0000 7f74c37fe640 1 -- 172.21.15.88:0/473510020 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f74b411e4d0 con 0x7f74cc0ca190 2023-10-16T20:39:38.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.525+0000 7f74ca59c640 1 --2- 172.21.15.88:0/473510020 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74ac0664a0 0x7f74ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:38.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.525+0000 7f74ca59c640 1 --2- 172.21.15.88:0/473510020 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74ac0664a0 0x7f74ac068960 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f74bc0113e0 tx=0x7f74bc00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:38.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.529+0000 7f74c37fe640 1 -- 172.21.15.88:0/473510020 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f74b40eb4f0 con 0x7f74cc0ca190 2023-10-16T20:39:38.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.677+0000 7f74cb59e640 1 -- 172.21.15.88:0/473510020 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f74cc0ce820 con 0x7f74cc0ca190 2023-10-16T20:39:38.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.677+0000 7f74c37fe640 1 -- 172.21.15.88:0/473510020 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f74b40eeb10 con 0x7f74cc0ca190 2023-10-16T20:39:38.684 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-10-16T20:39:38.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.677+0000 7f74cb59e640 1 -- 172.21.15.88:0/473510020 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74ac0664a0 msgr2=0x7f74ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.677+0000 7f74cb59e640 1 --2- 172.21.15.88:0/473510020 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74ac0664a0 0x7f74ac068960 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f74bc0113e0 tx=0x7f74bc00b040 comp rx=0 tx=0).stop 2023-10-16T20:39:38.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.677+0000 7f74cb59e640 1 -- 172.21.15.88:0/473510020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74cc0ca190 msgr2=0x7f74cc148170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.677+0000 7f74cb59e640 1 --2- 172.21.15.88:0/473510020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74cc0ca190 0x7f74cc148170 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f74b401a060 tx=0x7f74b40191d0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.677+0000 7f74cb59e640 1 -- 172.21.15.88:0/473510020 shutdown_connections 2023-10-16T20:39:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.677+0000 7f74cb59e640 1 --2- 172.21.15.88:0/473510020 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74ac0664a0 0x7f74ac068960 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.677+0000 7f74cb59e640 1 --2- 172.21.15.88:0/473510020 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74cc153630 0x7f74cc1486b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.677+0000 7f74cb59e640 1 --2- 172.21.15.88:0/473510020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74cc0ca190 0x7f74cc148170 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.677+0000 7f74cb59e640 1 --2- 172.21.15.88:0/473510020 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74cc0c3ea0 0x7f74cc0c76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.677+0000 7f74cb59e640 1 -- 172.21.15.88:0/473510020 >> 172.21.15.88:0/473510020 conn(0x7f74cc06bc00 msgr2=0x7f74cc0cdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:38.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.677+0000 7f74cb59e640 1 -- 172.21.15.88:0/473510020 shutdown_connections 2023-10-16T20:39:38.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.677+0000 7f74cb59e640 1 -- 172.21.15.88:0/473510020 wait complete. 2023-10-16T20:39:38.704 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T20:39:38.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.837+0000 7ff11da68640 1 Processor -- start 2023-10-16T20:39:38.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.837+0000 7ff11da68640 1 -- start start 2023-10-16T20:39:38.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.837+0000 7ff11da68640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff1180cdb20 0x7ff1180ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:38.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.837+0000 7ff11da68640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff11806dd70 0x7ff1180cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:38.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.841+0000 7ff11da68640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11806cbe0 con 0x7ff1181537d0 2023-10-16T20:39:38.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.841+0000 7ff11da68640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff11806f5f0 con 0x7ff1180cdb20 2023-10-16T20:39:38.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.841+0000 7ff11da68640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff11815d8b0 con 0x7ff11806dd70 2023-10-16T20:39:38.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.841+0000 7ff1177fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff1181537d0 0x7ff11815aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55946/0 (socket says 172.21.15.88:55946) 2023-10-16T20:39:38.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.841+0000 7ff1177fe640 1 -- 172.21.15.88:0/4039913482 learned_addr learned my addr 172.21.15.88:0/4039913482 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:38.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.841+0000 7ff116ffd640 1 --2- 172.21.15.88:0/4039913482 >> v2:172.21.15.88:3300/0 conn(0x7ff1180cdb20 0x7ff1180ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:38.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.841+0000 7ff117fff640 1 --2- 172.21.15.88:0/4039913482 >> v2:172.21.15.191:3300/0 conn(0x7ff11806dd70 0x7ff1180cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:38.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.841+0000 7ff116ffd640 1 -- 172.21.15.88:0/4039913482 >> v2:172.21.15.191:3300/0 conn(0x7ff11806dd70 msgr2=0x7ff1180cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:38.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.841+0000 7ff116ffd640 1 --2- 172.21.15.88:0/4039913482 >> v2:172.21.15.191:3300/0 conn(0x7ff11806dd70 0x7ff1180cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.841+0000 7ff116ffd640 1 -- 172.21.15.88:0/4039913482 >> v1:172.21.15.88:6789/0 conn(0x7ff1181537d0 legacy=0x7ff11815aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.841+0000 7ff116ffd640 1 -- 172.21.15.88:0/4039913482 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11806e830 con 0x7ff1180cdb20 2023-10-16T20:39:38.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.841+0000 7ff1167fc640 1 -- 172.21.15.88:0/4039913482 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485631581 0 0) 0x7ff11806cbe0 con 0x7ff1181537d0 2023-10-16T20:39:38.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.841+0000 7ff116ffd640 1 --2- 172.21.15.88:0/4039913482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1180cdb20 0x7ff1180ccdb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff1180b1ae0 tx=0x7ff10c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f781c614f3edd16 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:38.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.841+0000 7ff1167fc640 1 -- 172.21.15.88:0/4039913482 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c004230 con 0x7ff1180cdb20 2023-10-16T20:39:38.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.841+0000 7ff1167fc640 1 -- 172.21.15.88:0/4039913482 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10c0a7050 con 0x7ff1180cdb20 2023-10-16T20:39:38.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.841+0000 7ff1167fc640 1 -- 172.21.15.88:0/4039913482 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c003620 con 0x7ff1180cdb20 2023-10-16T20:39:38.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff11da68640 1 -- 172.21.15.88:0/4039913482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1180cdb20 msgr2=0x7ff1180ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff11da68640 1 --2- 172.21.15.88:0/4039913482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1180cdb20 0x7ff1180ccdb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff1180b1ae0 tx=0x7ff10c01c460 comp rx=0 tx=0).stop 2023-10-16T20:39:38.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff11da68640 1 -- 172.21.15.88:0/4039913482 shutdown_connections 2023-10-16T20:39:38.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff11da68640 1 --2- 172.21.15.88:0/4039913482 >> v2:172.21.15.191:3300/0 conn(0x7ff11806dd70 0x7ff1180cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff11da68640 1 --2- 172.21.15.88:0/4039913482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1180cdb20 0x7ff1180ccdb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff11da68640 1 -- 172.21.15.88:0/4039913482 >> 172.21.15.88:0/4039913482 conn(0x7ff11806f7d0 msgr2=0x7ff11806fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:38.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff11da68640 1 -- 172.21.15.88:0/4039913482 shutdown_connections 2023-10-16T20:39:38.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff11da68640 1 -- 172.21.15.88:0/4039913482 wait complete. 2023-10-16T20:39:38.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff11da68640 1 Processor -- start 2023-10-16T20:39:38.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff11da68640 1 -- start start 2023-10-16T20:39:38.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff11da68640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff11806dd70 0x7ff1181f4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:38.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff11da68640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1180cdb20 0x7ff1180c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:38.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff11da68640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1181537d0 0x7ff11815b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:38.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff11da68640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1180ca5f0 con 0x7ff11806dd70 2023-10-16T20:39:38.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff11da68640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1180c8ff0 con 0x7ff1180cdb20 2023-10-16T20:39:38.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff11da68640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1180c9d50 con 0x7ff1181537d0 2023-10-16T20:39:38.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff1177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff11806dd70 0x7ff1181f4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:38.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff1177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff11806dd70 0x7ff1181f4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33200/0 (socket says 172.21.15.88:33200) 2023-10-16T20:39:38.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff1177fe640 1 -- 172.21.15.88:0/251219976 learned_addr learned my addr 172.21.15.88:0/251219976 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:38.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff116ffd640 1 --2- 172.21.15.88:0/251219976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1180cdb20 0x7ff1180c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:38.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff117fff640 1 --2- 172.21.15.88:0/251219976 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1181537d0 0x7ff11815b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:38.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff1177fe640 1 -- 172.21.15.88:0/251219976 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1181537d0 msgr2=0x7ff11815b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff1177fe640 1 --2- 172.21.15.88:0/251219976 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1181537d0 0x7ff11815b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff1177fe640 1 -- 172.21.15.88:0/251219976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1180cdb20 msgr2=0x7ff1180c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:38.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff1177fe640 1 --2- 172.21.15.88:0/251219976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1180cdb20 0x7ff1180c9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:38.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.845+0000 7ff1177fe640 1 -- 172.21.15.88:0/251219976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff10c009420 con 0x7ff11806dd70 2023-10-16T20:39:38.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.849+0000 7ff1177fe640 1 --2- 172.21.15.88:0/251219976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff11806dd70 0x7ff1181f4c50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff10800c480 tx=0x7ff10800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:38.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.849+0000 7ff0fffff640 1 -- 172.21.15.88:0/251219976 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1080074e0 con 0x7ff11806dd70 2023-10-16T20:39:38.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.849+0000 7ff0fffff640 1 -- 172.21.15.88:0/251219976 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff108007660 con 0x7ff11806dd70 2023-10-16T20:39:38.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.849+0000 7ff11da68640 1 -- 172.21.15.88:0/251219976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1180ca030 con 0x7ff11806dd70 2023-10-16T20:39:38.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.849+0000 7ff0fffff640 1 -- 172.21.15.88:0/251219976 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff108004a30 con 0x7ff11806dd70 2023-10-16T20:39:38.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.849+0000 7ff11da68640 1 -- 172.21.15.88:0/251219976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff11815bc40 con 0x7ff11806dd70 2023-10-16T20:39:38.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.849+0000 7ff11da68640 1 -- 172.21.15.88:0/251219976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff118074700 con 0x7ff11806dd70 2023-10-16T20:39:38.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.849+0000 7ff0fffff640 1 -- 172.21.15.88:0/251219976 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff108004b90 con 0x7ff11806dd70 2023-10-16T20:39:38.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.849+0000 7ff0fffff640 1 --2- 172.21.15.88:0/251219976 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff1040664a0 0x7ff104068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:38.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.849+0000 7ff0fffff640 1 -- 172.21.15.88:0/251219976 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff1080836d0 con 0x7ff11806dd70 2023-10-16T20:39:38.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.849+0000 7ff116ffd640 1 --2- 172.21.15.88:0/251219976 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff1040664a0 0x7ff104068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:38.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.853+0000 7ff116ffd640 1 --2- 172.21.15.88:0/251219976 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff1040664a0 0x7ff104068960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff10c0ad000 tx=0x7ff10c0a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:38.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:38.853+0000 7ff0fffff640 1 -- 172.21.15.88:0/251219976 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff108050830 con 0x7ff11806dd70 2023-10-16T20:39:39.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.005+0000 7ff11da68640 1 -- 172.21.15.88:0/251219976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff11806e470 con 0x7ff11806dd70 2023-10-16T20:39:39.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.005+0000 7ff0fffff640 1 -- 172.21.15.88:0/251219976 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff108053e50 con 0x7ff11806dd70 2023-10-16T20:39:39.012 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:39:39.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.005+0000 7ff11da68640 1 -- 172.21.15.88:0/251219976 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff1040664a0 msgr2=0x7ff104068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:39.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.005+0000 7ff11da68640 1 --2- 172.21.15.88:0/251219976 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff1040664a0 0x7ff104068960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff10c0ad000 tx=0x7ff10c0a6000 comp rx=0 tx=0).stop 2023-10-16T20:39:39.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.005+0000 7ff11da68640 1 -- 172.21.15.88:0/251219976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff11806dd70 msgr2=0x7ff1181f4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:39.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.005+0000 7ff11da68640 1 --2- 172.21.15.88:0/251219976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff11806dd70 0x7ff1181f4c50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff10800c480 tx=0x7ff10800c940 comp rx=0 tx=0).stop 2023-10-16T20:39:39.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.005+0000 7ff11da68640 1 -- 172.21.15.88:0/251219976 shutdown_connections 2023-10-16T20:39:39.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.005+0000 7ff11da68640 1 --2- 172.21.15.88:0/251219976 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff1040664a0 0x7ff104068960 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.005+0000 7ff11da68640 1 --2- 172.21.15.88:0/251219976 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1181537d0 0x7ff11815b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.005+0000 7ff11da68640 1 --2- 172.21.15.88:0/251219976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1180cdb20 0x7ff1180c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.005+0000 7ff11da68640 1 --2- 172.21.15.88:0/251219976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff11806dd70 0x7ff1181f4c50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.005+0000 7ff11da68640 1 -- 172.21.15.88:0/251219976 >> 172.21.15.88:0/251219976 conn(0x7ff11806f7d0 msgr2=0x7ff118151ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:39.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.005+0000 7ff11da68640 1 -- 172.21.15.88:0/251219976 shutdown_connections 2023-10-16T20:39:39.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.005+0000 7ff11da68640 1 -- 172.21.15.88:0/251219976 wait complete. 2023-10-16T20:39:39.034 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T20:39:39.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.165+0000 7fbfeac6b640 1 Processor -- start 2023-10-16T20:39:39.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.169+0000 7fbfeac6b640 1 -- start start 2023-10-16T20:39:39.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.169+0000 7fbfeac6b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbfe40c6ac0 0x7fbfe4150150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.169+0000 7fbfeac6b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbfe40cb5c0 0x7fbfe4150690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.169+0000 7fbfeac6b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe4070f40 con 0x7fbfe40c66e0 2023-10-16T20:39:39.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.169+0000 7fbfeac6b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe406a2d0 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.169+0000 7fbfeac6b640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe406a5b0 con 0x7fbfe40c6ac0 2023-10-16T20:39:39.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.169+0000 7fbfe9208640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbfe40cb5c0 0x7fbfe4150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.169+0000 7fbfe8a07640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbfe40c66e0 0x7fbfe415a250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55950/0 (socket says 172.21.15.88:55950) 2023-10-16T20:39:39.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.169+0000 7fbfe8a07640 1 -- 172.21.15.88:0/1093859844 learned_addr learned my addr 172.21.15.88:0/1093859844 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:39.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.169+0000 7fbfe3fff640 1 --2- 172.21.15.88:0/1093859844 >> v2:172.21.15.191:3300/0 conn(0x7fbfe40c6ac0 0x7fbfe4150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.169+0000 7fbfe9208640 1 -- 172.21.15.88:0/1093859844 >> v2:172.21.15.191:3300/0 conn(0x7fbfe40c6ac0 msgr2=0x7fbfe4150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:39.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.169+0000 7fbfe9208640 1 --2- 172.21.15.88:0/1093859844 >> v2:172.21.15.191:3300/0 conn(0x7fbfe40c6ac0 0x7fbfe4150150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.169+0000 7fbfe9208640 1 -- 172.21.15.88:0/1093859844 >> v1:172.21.15.88:6789/0 conn(0x7fbfe40c66e0 legacy=0x7fbfe415a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:39.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.169+0000 7fbfe9208640 1 -- 172.21.15.88:0/1093859844 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe406a890 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.169+0000 7fbfe37fe640 1 -- 172.21.15.88:0/1093859844 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063645064 0 0) 0x7fbfe4070f40 con 0x7fbfe40c66e0 2023-10-16T20:39:39.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.173+0000 7fbfe9208640 1 --2- 172.21.15.88:0/1093859844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfe40cb5c0 0x7fbfe4150690 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc009d90 tx=0x7fbfdc01d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1800f8d06f9af80b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:39.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.173+0000 7fbfe37fe640 1 -- 172.21.15.88:0/1093859844 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc0a5850 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.173+0000 7fbfe37fe640 1 -- 172.21.15.88:0/1093859844 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfdc0a59b0 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.173+0000 7fbfe37fe640 1 -- 172.21.15.88:0/1093859844 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc0a5d40 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.173+0000 7fbfeac6b640 1 -- 172.21.15.88:0/1093859844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfe40cb5c0 msgr2=0x7fbfe4150690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:39.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.173+0000 7fbfeac6b640 1 --2- 172.21.15.88:0/1093859844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfe40cb5c0 0x7fbfe4150690 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc009d90 tx=0x7fbfdc01d2e0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.173+0000 7fbfeac6b640 1 -- 172.21.15.88:0/1093859844 shutdown_connections 2023-10-16T20:39:39.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.173+0000 7fbfeac6b640 1 --2- 172.21.15.88:0/1093859844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfe40cb5c0 0x7fbfe4150690 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.173+0000 7fbfeac6b640 1 --2- 172.21.15.88:0/1093859844 >> v2:172.21.15.191:3300/0 conn(0x7fbfe40c6ac0 0x7fbfe4150150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.173+0000 7fbfeac6b640 1 -- 172.21.15.88:0/1093859844 >> 172.21.15.88:0/1093859844 conn(0x7fbfe40ca9f0 msgr2=0x7fbfe40cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:39.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.173+0000 7fbfeac6b640 1 -- 172.21.15.88:0/1093859844 shutdown_connections 2023-10-16T20:39:39.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.173+0000 7fbfeac6b640 1 -- 172.21.15.88:0/1093859844 wait complete. 2023-10-16T20:39:39.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.173+0000 7fbfeac6b640 1 Processor -- start 2023-10-16T20:39:39.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.173+0000 7fbfeac6b640 1 -- start start 2023-10-16T20:39:39.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfeac6b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfe40c66e0 0x7fbfe415f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfeac6b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfe40c6ac0 0x7fbfe41f17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfeac6b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfe40cb5c0 0x7fbfe40c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfeac6b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe41f1d30 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfeac6b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe41609e0 con 0x7fbfe40c6ac0 2023-10-16T20:39:39.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfeac6b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe41f4ec0 con 0x7fbfe40c66e0 2023-10-16T20:39:39.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfe9208640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfe40cb5c0 0x7fbfe40c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfe9208640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfe40cb5c0 0x7fbfe40c1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33220/0 (socket says 172.21.15.88:33220) 2023-10-16T20:39:39.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfe9208640 1 -- 172.21.15.88:0/656648029 learned_addr learned my addr 172.21.15.88:0/656648029 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:39.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfe3fff640 1 --2- 172.21.15.88:0/656648029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfe40c6ac0 0x7fbfe41f17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfe8a07640 1 --2- 172.21.15.88:0/656648029 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfe40c66e0 0x7fbfe415f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfe9208640 1 -- 172.21.15.88:0/656648029 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfe40c66e0 msgr2=0x7fbfe415f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:39.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfe9208640 1 --2- 172.21.15.88:0/656648029 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfe40c66e0 0x7fbfe415f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfe9208640 1 -- 172.21.15.88:0/656648029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfe40c6ac0 msgr2=0x7fbfe41f17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:39.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfe9208640 1 --2- 172.21.15.88:0/656648029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfe40c6ac0 0x7fbfe41f17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfe9208640 1 -- 172.21.15.88:0/656648029 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc009a70 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfe3fff640 1 --2- 172.21.15.88:0/656648029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfe40c6ac0 0x7fbfe41f17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:39.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfe9208640 1 --2- 172.21.15.88:0/656648029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfe40cb5c0 0x7fbfe40c1f50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0b0040 tx=0x7fbfdc019620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:39.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfe17fa640 1 -- 172.21.15.88:0/656648029 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc0ad440 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfe17fa640 1 -- 172.21.15.88:0/656648029 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfdc0ad5a0 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfe17fa640 1 -- 172.21.15.88:0/656648029 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc0ad910 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfeac6b640 1 -- 172.21.15.88:0/656648029 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe4160420 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfeac6b640 1 -- 172.21.15.88:0/656648029 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe41f5160 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.177+0000 7fbfeac6b640 1 -- 172.21.15.88:0/656648029 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe4074700 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.181+0000 7fbfe17fa640 1 -- 172.21.15.88:0/656648029 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbfdc0b4040 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.181+0000 7fbfe17fa640 1 --2- 172.21.15.88:0/656648029 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbfc80664a0 0x7fbfc8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.181+0000 7fbfe17fa640 1 -- 172.21.15.88:0/656648029 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fbfdc11e850 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.181+0000 7fbfe8a07640 1 --2- 172.21.15.88:0/656648029 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbfc80664a0 0x7fbfc8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.181+0000 7fbfe8a07640 1 --2- 172.21.15.88:0/656648029 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbfc80664a0 0x7fbfc8068960 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbfd8000f70 tx=0x7fbfd8007b60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:39.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.185+0000 7fbfe17fa640 1 -- 172.21.15.88:0/656648029 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbfdc0eb990 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.333+0000 7fbfeac6b640 1 -- 172.21.15.88:0/656648029 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbfe40c8290 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.337+0000 7fbfe17fa640 1 -- 172.21.15.88:0/656648029 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbfdc0b20a0 con 0x7fbfe40cb5c0 2023-10-16T20:39:39.343 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2023-10-16T20:39:39.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.337+0000 7fbfeac6b640 1 -- 172.21.15.88:0/656648029 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbfc80664a0 msgr2=0x7fbfc8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:39.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.337+0000 7fbfeac6b640 1 --2- 172.21.15.88:0/656648029 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbfc80664a0 0x7fbfc8068960 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbfd8000f70 tx=0x7fbfd8007b60 comp rx=0 tx=0).stop 2023-10-16T20:39:39.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.337+0000 7fbfeac6b640 1 -- 172.21.15.88:0/656648029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfe40cb5c0 msgr2=0x7fbfe40c1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:39.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.337+0000 7fbfeac6b640 1 --2- 172.21.15.88:0/656648029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfe40cb5c0 0x7fbfe40c1f50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0b0040 tx=0x7fbfdc019620 comp rx=0 tx=0).stop 2023-10-16T20:39:39.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.337+0000 7fbfeac6b640 1 -- 172.21.15.88:0/656648029 shutdown_connections 2023-10-16T20:39:39.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.337+0000 7fbfeac6b640 1 --2- 172.21.15.88:0/656648029 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbfc80664a0 0x7fbfc8068960 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.337+0000 7fbfeac6b640 1 --2- 172.21.15.88:0/656648029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfe40cb5c0 0x7fbfe40c1f50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.337+0000 7fbfeac6b640 1 --2- 172.21.15.88:0/656648029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbfe40c6ac0 0x7fbfe41f17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.337+0000 7fbfeac6b640 1 --2- 172.21.15.88:0/656648029 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbfe40c66e0 0x7fbfe415f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.337+0000 7fbfeac6b640 1 -- 172.21.15.88:0/656648029 >> 172.21.15.88:0/656648029 conn(0x7fbfe40ca9f0 msgr2=0x7fbfe40c3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:39.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.337+0000 7fbfeac6b640 1 -- 172.21.15.88:0/656648029 shutdown_connections 2023-10-16T20:39:39.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.337+0000 7fbfeac6b640 1 -- 172.21.15.88:0/656648029 wait complete. 2023-10-16T20:39:39.364 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T20:39:39.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.497+0000 7f8af7b72640 1 Processor -- start 2023-10-16T20:39:39.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.501+0000 7f8af7b72640 1 -- start start 2023-10-16T20:39:39.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.501+0000 7f8af7b72640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8af00c66e0 0x7f8af015a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.501+0000 7f8af7b72640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8af00c6ac0 0x7f8af0150150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.501+0000 7f8af7b72640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af0070f40 con 0x7f8af00cb5c0 2023-10-16T20:39:39.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.501+0000 7f8af7b72640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8af015a790 con 0x7f8af00c6ac0 2023-10-16T20:39:39.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.501+0000 7f8af7b72640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af006a2d0 con 0x7f8af00c66e0 2023-10-16T20:39:39.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.501+0000 7f8af510d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8af00c6ac0 0x7f8af0150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.501+0000 7f8af610f640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8af00cb5c0 0x7f8af0071420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55960/0 (socket says 172.21.15.88:55960) 2023-10-16T20:39:39.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.501+0000 7f8af610f640 1 -- 172.21.15.88:0/2540756352 learned_addr learned my addr 172.21.15.88:0/2540756352 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:39.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.501+0000 7f8af510d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8af00c6ac0 0x7f8af0150150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33236/0 (socket says 172.21.15.88:33236) 2023-10-16T20:39:39.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.501+0000 7f8af590e640 1 --2- 172.21.15.88:0/2540756352 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8af00c66e0 0x7f8af015a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.501+0000 7f8af510d640 1 -- 172.21.15.88:0/2540756352 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8af00c66e0 msgr2=0x7f8af015a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:39.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.501+0000 7f8af510d640 1 --2- 172.21.15.88:0/2540756352 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8af00c66e0 0x7f8af015a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.501+0000 7f8af510d640 1 -- 172.21.15.88:0/2540756352 >> v1:172.21.15.88:6789/0 conn(0x7f8af00cb5c0 legacy=0x7f8af0071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:39.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.501+0000 7f8af490c640 1 -- 172.21.15.88:0/2540756352 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158880332 0 0) 0x7f8af0070f40 con 0x7f8af00cb5c0 2023-10-16T20:39:39.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.501+0000 7f8af510d640 1 -- 172.21.15.88:0/2540756352 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af006a5b0 con 0x7f8af00c6ac0 2023-10-16T20:39:39.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.505+0000 7f8af510d640 1 --2- 172.21.15.88:0/2540756352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af00c6ac0 0x7f8af0150150 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8ae000ba60 tx=0x7f8ae001eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26614699df3f9f89 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:39.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.505+0000 7f8af490c640 1 -- 172.21.15.88:0/2540756352 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae00a7980 con 0x7f8af00c6ac0 2023-10-16T20:39:39.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.505+0000 7f8af490c640 1 -- 172.21.15.88:0/2540756352 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae00a7ae0 con 0x7f8af00c6ac0 2023-10-16T20:39:39.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.505+0000 7f8af490c640 1 -- 172.21.15.88:0/2540756352 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae00a7e70 con 0x7f8af00c6ac0 2023-10-16T20:39:39.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.505+0000 7f8af7b72640 1 -- 172.21.15.88:0/2540756352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af00c6ac0 msgr2=0x7f8af0150150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:39.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.505+0000 7f8af7b72640 1 --2- 172.21.15.88:0/2540756352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af00c6ac0 0x7f8af0150150 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8ae000ba60 tx=0x7f8ae001eac0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.505+0000 7f8af7b72640 1 -- 172.21.15.88:0/2540756352 shutdown_connections 2023-10-16T20:39:39.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.505+0000 7f8af7b72640 1 --2- 172.21.15.88:0/2540756352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af00c6ac0 0x7f8af0150150 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.505+0000 7f8af7b72640 1 --2- 172.21.15.88:0/2540756352 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8af00c66e0 0x7f8af015a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.505+0000 7f8af7b72640 1 -- 172.21.15.88:0/2540756352 >> 172.21.15.88:0/2540756352 conn(0x7f8af00ca9f0 msgr2=0x7f8af00cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:39.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.505+0000 7f8af7b72640 1 -- 172.21.15.88:0/2540756352 shutdown_connections 2023-10-16T20:39:39.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.505+0000 7f8af7b72640 1 -- 172.21.15.88:0/2540756352 wait complete. 2023-10-16T20:39:39.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.505+0000 7f8af7b72640 1 Processor -- start 2023-10-16T20:39:39.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af7b72640 1 -- start start 2023-10-16T20:39:39.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af7b72640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8af00c66e0 0x7f8af014c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af7b72640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8af00c6ac0 0x7f8af01589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af7b72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af00cb5c0 0x7f8af01f5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af7b72640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af00c1fa0 con 0x7f8af00cb5c0 2023-10-16T20:39:39.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af7b72640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af01f52b0 con 0x7f8af00c66e0 2023-10-16T20:39:39.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af7b72640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af01581a0 con 0x7f8af00c6ac0 2023-10-16T20:39:39.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af610f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af00cb5c0 0x7f8af01f5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af610f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af00cb5c0 0x7f8af01f5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33240/0 (socket says 172.21.15.88:33240) 2023-10-16T20:39:39.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af610f640 1 -- 172.21.15.88:0/989617625 learned_addr learned my addr 172.21.15.88:0/989617625 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:39.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af590e640 1 --2- 172.21.15.88:0/989617625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8af00c66e0 0x7f8af014c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af510d640 1 --2- 172.21.15.88:0/989617625 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8af00c6ac0 0x7f8af01589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af610f640 1 -- 172.21.15.88:0/989617625 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8af00c6ac0 msgr2=0x7f8af01589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af610f640 1 --2- 172.21.15.88:0/989617625 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8af00c6ac0 0x7f8af01589b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af610f640 1 -- 172.21.15.88:0/989617625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8af00c66e0 msgr2=0x7f8af014c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:39.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af610f640 1 --2- 172.21.15.88:0/989617625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8af00c66e0 0x7f8af014c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af610f640 1 -- 172.21.15.88:0/989617625 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae000b6c0 con 0x7f8af00cb5c0 2023-10-16T20:39:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af610f640 1 --2- 172.21.15.88:0/989617625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af00cb5c0 0x7f8af01f5740 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8ae800c730 tx=0x7f8ae800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8ae67fc640 1 -- 172.21.15.88:0/989617625 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae800ce40 con 0x7f8af00cb5c0 2023-10-16T20:39:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8ae67fc640 1 -- 172.21.15.88:0/989617625 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae800a640 con 0x7f8af00cb5c0 2023-10-16T20:39:39.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8ae67fc640 1 -- 172.21.15.88:0/989617625 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae800aa30 con 0x7f8af00cb5c0 2023-10-16T20:39:39.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af7b72640 1 -- 172.21.15.88:0/989617625 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af014bdb0 con 0x7f8af00cb5c0 2023-10-16T20:39:39.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af7b72640 1 -- 172.21.15.88:0/989617625 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af015a3a0 con 0x7f8af00cb5c0 2023-10-16T20:39:39.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.509+0000 7f8af7b72640 1 -- 172.21.15.88:0/989617625 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af0074700 con 0x7f8af00cb5c0 2023-10-16T20:39:39.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.513+0000 7f8ae67fc640 1 -- 172.21.15.88:0/989617625 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8ae800ab90 con 0x7f8af00cb5c0 2023-10-16T20:39:39.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.513+0000 7f8ae67fc640 1 --2- 172.21.15.88:0/989617625 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ad40664a0 0x7f8ad4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.513+0000 7f8ae67fc640 1 -- 172.21.15.88:0/989617625 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8ae80837c0 con 0x7f8af00cb5c0 2023-10-16T20:39:39.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.513+0000 7f8af590e640 1 --2- 172.21.15.88:0/989617625 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ad40664a0 0x7f8ad4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.513+0000 7f8af590e640 1 --2- 172.21.15.88:0/989617625 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ad40664a0 0x7f8ad4068960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8aec009af0 tx=0x7f8aec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:39.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.521+0000 7f8ae67fc640 1 -- 172.21.15.88:0/989617625 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8ae80507e0 con 0x7f8af00cb5c0 2023-10-16T20:39:39.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.673+0000 7f8af7b72640 1 -- 172.21.15.88:0/989617625 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8af01f53f0 con 0x7f8af00cb5c0 2023-10-16T20:39:39.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.673+0000 7f8ae67fc640 1 -- 172.21.15.88:0/989617625 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8ae80180a0 con 0x7f8af00cb5c0 2023-10-16T20:39:39.680 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:39:39.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.673+0000 7f8af7b72640 1 -- 172.21.15.88:0/989617625 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ad40664a0 msgr2=0x7f8ad4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:39.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.673+0000 7f8af7b72640 1 --2- 172.21.15.88:0/989617625 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ad40664a0 0x7f8ad4068960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8aec009af0 tx=0x7f8aec008040 comp rx=0 tx=0).stop 2023-10-16T20:39:39.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.673+0000 7f8af7b72640 1 -- 172.21.15.88:0/989617625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af00cb5c0 msgr2=0x7f8af01f5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:39.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.673+0000 7f8af7b72640 1 --2- 172.21.15.88:0/989617625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af00cb5c0 0x7f8af01f5740 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8ae800c730 tx=0x7f8ae800cbf0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.673+0000 7f8af7b72640 1 -- 172.21.15.88:0/989617625 shutdown_connections 2023-10-16T20:39:39.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.673+0000 7f8af7b72640 1 --2- 172.21.15.88:0/989617625 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ad40664a0 0x7f8ad4068960 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.673+0000 7f8af7b72640 1 --2- 172.21.15.88:0/989617625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af00cb5c0 0x7f8af01f5740 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.673+0000 7f8af7b72640 1 --2- 172.21.15.88:0/989617625 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8af00c6ac0 0x7f8af01589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.673+0000 7f8af7b72640 1 --2- 172.21.15.88:0/989617625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8af00c66e0 0x7f8af014c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.673+0000 7f8af7b72640 1 -- 172.21.15.88:0/989617625 >> 172.21.15.88:0/989617625 conn(0x7f8af00ca9f0 msgr2=0x7f8af00c5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:39.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.673+0000 7f8af7b72640 1 -- 172.21.15.88:0/989617625 shutdown_connections 2023-10-16T20:39:39.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.673+0000 7f8af7b72640 1 -- 172.21.15.88:0/989617625 wait complete. 2023-10-16T20:39:39.700 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:39:39.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.833+0000 7fbf9a09a640 1 Processor -- start 2023-10-16T20:39:39.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.833+0000 7fbf9a09a640 1 -- start start 2023-10-16T20:39:39.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.833+0000 7fbf9a09a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbf940c7dc0 0x7fbf94159450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.833+0000 7fbf9a09a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbf940c81a0 0x7fbf9406b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.833+0000 7fbf9a09a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf940c8810 con 0x7fbf940c76c0 2023-10-16T20:39:39.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.833+0000 7fbf9a09a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9406b400 con 0x7fbf940c7dc0 2023-10-16T20:39:39.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.833+0000 7fbf9a09a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9406b0d0 con 0x7fbf940c81a0 2023-10-16T20:39:39.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.833+0000 7fbf937fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbf940c7dc0 0x7fbf94159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.833+0000 7fbf937fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbf940c7dc0 0x7fbf94159450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33254/0 (socket says 172.21.15.88:33254) 2023-10-16T20:39:39.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.833+0000 7fbf937fe640 1 -- 172.21.15.88:0/3205163229 learned_addr learned my addr 172.21.15.88:0/3205163229 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:39.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.833+0000 7fbf93fff640 1 --2- 172.21.15.88:0/3205163229 >> v2:172.21.15.191:3300/0 conn(0x7fbf940c81a0 0x7fbf9406b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.833+0000 7fbf937fe640 1 -- 172.21.15.88:0/3205163229 >> v2:172.21.15.191:3300/0 conn(0x7fbf940c81a0 msgr2=0x7fbf9406b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:39.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.833+0000 7fbf937fe640 1 --2- 172.21.15.88:0/3205163229 >> v2:172.21.15.191:3300/0 conn(0x7fbf940c81a0 0x7fbf9406b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.833+0000 7fbf937fe640 1 -- 172.21.15.88:0/3205163229 >> v1:172.21.15.191:6789/0 conn(0x7fbf940c76c0 legacy=0x7fbf9406d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:39.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.833+0000 7fbf937fe640 1 -- 172.21.15.88:0/3205163229 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9406e6a0 con 0x7fbf940c7dc0 2023-10-16T20:39:39.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.833+0000 7fbf93fff640 1 --2- 172.21.15.88:0/3205163229 >> v2:172.21.15.191:3300/0 conn(0x7fbf940c81a0 0x7fbf9406b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:39.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.837+0000 7fbf937fe640 1 --2- 172.21.15.88:0/3205163229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbf940c7dc0 0x7fbf94159450 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbf9406eda0 tx=0x7fbf8401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a864dc984c52e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:39.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.837+0000 7fbf927fc640 1 -- 172.21.15.88:0/3205163229 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf840ab600 con 0x7fbf940c7dc0 2023-10-16T20:39:39.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.837+0000 7fbf927fc640 1 -- 172.21.15.88:0/3205163229 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf840a7050 con 0x7fbf940c7dc0 2023-10-16T20:39:39.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.837+0000 7fbf927fc640 1 -- 172.21.15.88:0/3205163229 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf840ab980 con 0x7fbf940c7dc0 2023-10-16T20:39:39.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.837+0000 7fbf9a09a640 1 -- 172.21.15.88:0/3205163229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbf940c7dc0 msgr2=0x7fbf94159450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:39.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.837+0000 7fbf9a09a640 1 --2- 172.21.15.88:0/3205163229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbf940c7dc0 0x7fbf94159450 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbf9406eda0 tx=0x7fbf8401c430 comp rx=0 tx=0).stop 2023-10-16T20:39:39.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.837+0000 7fbf9a09a640 1 -- 172.21.15.88:0/3205163229 shutdown_connections 2023-10-16T20:39:39.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.837+0000 7fbf9a09a640 1 --2- 172.21.15.88:0/3205163229 >> v2:172.21.15.191:3300/0 conn(0x7fbf940c81a0 0x7fbf9406b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.837+0000 7fbf9a09a640 1 --2- 172.21.15.88:0/3205163229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbf940c7dc0 0x7fbf94159450 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.837+0000 7fbf9a09a640 1 -- 172.21.15.88:0/3205163229 >> 172.21.15.88:0/3205163229 conn(0x7fbf94071a60 msgr2=0x7fbf94071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:39.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.837+0000 7fbf9a09a640 1 -- 172.21.15.88:0/3205163229 shutdown_connections 2023-10-16T20:39:39.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.837+0000 7fbf9a09a640 1 -- 172.21.15.88:0/3205163229 wait complete. 2023-10-16T20:39:39.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.837+0000 7fbf9a09a640 1 Processor -- start 2023-10-16T20:39:39.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf9a09a640 1 -- start start 2023-10-16T20:39:39.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf9a09a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbf940c76c0 0x7fbf941f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf9a09a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf940c7dc0 0x7fbf94142ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf9a09a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf940c81a0 0x7fbf94141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf9a09a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9413fde0 con 0x7fbf940c76c0 2023-10-16T20:39:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf9a09a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf94142b30 con 0x7fbf940c81a0 2023-10-16T20:39:39.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf9a09a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf941f2610 con 0x7fbf940c7dc0 2023-10-16T20:39:39.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf937fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbf940c76c0 0x7fbf941f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf937fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbf940c76c0 0x7fbf941f2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33262/0 (socket says 172.21.15.88:33262) 2023-10-16T20:39:39.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf937fe640 1 -- 172.21.15.88:0/3103376098 learned_addr learned my addr 172.21.15.88:0/3103376098 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:39.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf93fff640 1 --2- 172.21.15.88:0/3103376098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf940c81a0 0x7fbf94141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf92ffd640 1 --2- 172.21.15.88:0/3103376098 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf940c7dc0 0x7fbf94142ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf937fe640 1 -- 172.21.15.88:0/3103376098 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf940c7dc0 msgr2=0x7fbf94142ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:39.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf937fe640 1 --2- 172.21.15.88:0/3103376098 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf940c7dc0 0x7fbf94142ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf937fe640 1 -- 172.21.15.88:0/3103376098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf940c81a0 msgr2=0x7fbf94141290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:39.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf937fe640 1 --2- 172.21.15.88:0/3103376098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf940c81a0 0x7fbf94141290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:39.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf937fe640 1 -- 172.21.15.88:0/3103376098 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf840094d0 con 0x7fbf940c76c0 2023-10-16T20:39:39.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf93fff640 1 --2- 172.21.15.88:0/3103376098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf940c81a0 0x7fbf94141290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf937fe640 1 --2- 172.21.15.88:0/3103376098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbf940c76c0 0x7fbf941f2aa0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbf840ae040 tx=0x7fbf8401ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf7bfff640 1 -- 172.21.15.88:0/3103376098 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf840b6620 con 0x7fbf940c76c0 2023-10-16T20:39:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf7bfff640 1 -- 172.21.15.88:0/3103376098 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf840b67a0 con 0x7fbf940c76c0 2023-10-16T20:39:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf7bfff640 1 -- 172.21.15.88:0/3103376098 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf840b6b50 con 0x7fbf940c76c0 2023-10-16T20:39:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf9a09a640 1 -- 172.21.15.88:0/3103376098 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf94143370 con 0x7fbf940c76c0 2023-10-16T20:39:39.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf9a09a640 1 -- 172.21.15.88:0/3103376098 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf941f1fc0 con 0x7fbf940c76c0 2023-10-16T20:39:39.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.841+0000 7fbf9a09a640 1 -- 172.21.15.88:0/3103376098 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf94074700 con 0x7fbf940c76c0 2023-10-16T20:39:39.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.845+0000 7fbf7bfff640 1 -- 172.21.15.88:0/3103376098 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbf840b2040 con 0x7fbf940c76c0 2023-10-16T20:39:39.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.845+0000 7fbf7bfff640 1 --2- 172.21.15.88:0/3103376098 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbf800664a0 0x7fbf80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:39.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.845+0000 7fbf7bfff640 1 -- 172.21.15.88:0/3103376098 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fbf8411e7b0 con 0x7fbf940c76c0 2023-10-16T20:39:39.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.845+0000 7fbf92ffd640 1 --2- 172.21.15.88:0/3103376098 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbf800664a0 0x7fbf80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:39.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.845+0000 7fbf92ffd640 1 --2- 172.21.15.88:0/3103376098 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbf800664a0 0x7fbf80068960 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbf940cdcf0 tx=0x7fbf88008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:39.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:39.849+0000 7fbf7bfff640 1 -- 172.21.15.88:0/3103376098 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbf840eb740 con 0x7fbf940c76c0 2023-10-16T20:39:40.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.029+0000 7fbf9a09a640 1 -- 172.21.15.88:0/3103376098 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf940c3a40 con 0x7fbf940c76c0 2023-10-16T20:39:40.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.033+0000 7fbf7bfff640 1 -- 172.21.15.88:0/3103376098 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+6560 (secure 0 0 0) 0x7fbf840eed60 con 0x7fbf940c76c0 2023-10-16T20:39:40.040 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:40.040 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:39:34.506742+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":18,"state":"up:creating","state_seq":1,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:39:40.041 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 18 2023-10-16T20:39:40.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.033+0000 7fbf9a09a640 1 -- 172.21.15.88:0/3103376098 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbf800664a0 msgr2=0x7fbf80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:40.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.033+0000 7fbf9a09a640 1 --2- 172.21.15.88:0/3103376098 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbf800664a0 0x7fbf80068960 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbf940cdcf0 tx=0x7fbf88008040 comp rx=0 tx=0).stop 2023-10-16T20:39:40.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.033+0000 7fbf9a09a640 1 -- 172.21.15.88:0/3103376098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbf940c76c0 msgr2=0x7fbf941f2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:40.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.033+0000 7fbf9a09a640 1 --2- 172.21.15.88:0/3103376098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbf940c76c0 0x7fbf941f2aa0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbf840ae040 tx=0x7fbf8401ac70 comp rx=0 tx=0).stop 2023-10-16T20:39:40.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.033+0000 7fbf9a09a640 1 -- 172.21.15.88:0/3103376098 shutdown_connections 2023-10-16T20:39:40.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.033+0000 7fbf9a09a640 1 --2- 172.21.15.88:0/3103376098 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbf800664a0 0x7fbf80068960 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.033+0000 7fbf9a09a640 1 --2- 172.21.15.88:0/3103376098 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbf940c81a0 0x7fbf94141290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.033+0000 7fbf9a09a640 1 --2- 172.21.15.88:0/3103376098 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbf940c7dc0 0x7fbf94142ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.033+0000 7fbf9a09a640 1 --2- 172.21.15.88:0/3103376098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbf940c76c0 0x7fbf941f2aa0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.033+0000 7fbf9a09a640 1 -- 172.21.15.88:0/3103376098 >> 172.21.15.88:0/3103376098 conn(0x7fbf94071a60 msgr2=0x7fbf941550b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:40.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.033+0000 7fbf9a09a640 1 -- 172.21.15.88:0/3103376098 shutdown_connections 2023-10-16T20:39:40.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.033+0000 7fbf9a09a640 1 -- 172.21.15.88:0/3103376098 wait complete. 2023-10-16T20:39:40.060 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:39:40.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.205+0000 7fa4fd2c9640 1 Processor -- start 2023-10-16T20:39:40.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.205+0000 7fa4fd2c9640 1 -- start start 2023-10-16T20:39:40.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.205+0000 7fa4fd2c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4f80c9bb0 0x7fa4f80c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:40.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.205+0000 7fa4fd2c9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa4f80c41f0 0x7fa4f80cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:40.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.205+0000 7fa4fd2c9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f8153b50 con 0x7fa4f8153770 2023-10-16T20:39:40.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.205+0000 7fa4fd2c9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4f80cc250 con 0x7fa4f80c41f0 2023-10-16T20:39:40.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.205+0000 7fa4fd2c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f80cc670 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.205+0000 7fa4f67fc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4f80c9bb0 0x7fa4f80c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:40.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.205+0000 7fa4f67fc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4f80c9bb0 0x7fa4f80c3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:45944/0 (socket says 172.21.15.88:45944) 2023-10-16T20:39:40.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.205+0000 7fa4f67fc640 1 -- 172.21.15.88:0/2094841207 learned_addr learned my addr 172.21.15.88:0/2094841207 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:40.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.205+0000 7fa4f77fe640 1 --2- 172.21.15.88:0/2094841207 >> v2:172.21.15.191:3300/0 conn(0x7fa4f80c41f0 0x7fa4f80cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:40.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.205+0000 7fa4f67fc640 1 -- 172.21.15.88:0/2094841207 >> v2:172.21.15.191:3300/0 conn(0x7fa4f80c41f0 msgr2=0x7fa4f80cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:40.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.205+0000 7fa4f67fc640 1 --2- 172.21.15.88:0/2094841207 >> v2:172.21.15.191:3300/0 conn(0x7fa4f80c41f0 0x7fa4f80cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.205+0000 7fa4f67fc640 1 -- 172.21.15.88:0/2094841207 >> v1:172.21.15.191:6789/0 conn(0x7fa4f8153770 legacy=0x7fa4f8159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:40.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.205+0000 7fa4f67fc640 1 -- 172.21.15.88:0/2094841207 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f80cb2f0 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.205+0000 7fa4f5ffb640 1 -- 172.21.15.88:0/2094841207 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429270007 0 0) 0x7fa4f8153b50 con 0x7fa4f8153770 2023-10-16T20:39:40.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.209+0000 7fa4f67fc640 1 --2- 172.21.15.88:0/2094841207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4f80c9bb0 0x7fa4f80c3cb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec009870 tx=0x7fa4ec01c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7d9fe815242f1c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:40.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.209+0000 7fa4f5ffb640 1 -- 172.21.15.88:0/2094841207 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec0ab650 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.209+0000 7fa4f5ffb640 1 -- 172.21.15.88:0/2094841207 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ec0ab7b0 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.209+0000 7fa4f5ffb640 1 -- 172.21.15.88:0/2094841207 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec0abb40 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.209+0000 7fa4fd2c9640 1 -- 172.21.15.88:0/2094841207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4f80c9bb0 msgr2=0x7fa4f80c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:40.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.209+0000 7fa4fd2c9640 1 --2- 172.21.15.88:0/2094841207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4f80c9bb0 0x7fa4f80c3cb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec009870 tx=0x7fa4ec01c540 comp rx=0 tx=0).stop 2023-10-16T20:39:40.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.209+0000 7fa4fd2c9640 1 -- 172.21.15.88:0/2094841207 shutdown_connections 2023-10-16T20:39:40.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.209+0000 7fa4fd2c9640 1 --2- 172.21.15.88:0/2094841207 >> v2:172.21.15.191:3300/0 conn(0x7fa4f80c41f0 0x7fa4f80cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.209+0000 7fa4fd2c9640 1 --2- 172.21.15.88:0/2094841207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4f80c9bb0 0x7fa4f80c3cb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.209+0000 7fa4fd2c9640 1 -- 172.21.15.88:0/2094841207 >> 172.21.15.88:0/2094841207 conn(0x7fa4f80715c0 msgr2=0x7fa4f80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:40.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.209+0000 7fa4fd2c9640 1 -- 172.21.15.88:0/2094841207 shutdown_connections 2023-10-16T20:39:40.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.209+0000 7fa4fd2c9640 1 -- 172.21.15.88:0/2094841207 wait complete. 2023-10-16T20:39:40.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4fd2c9640 1 Processor -- start 2023-10-16T20:39:40.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4fd2c9640 1 -- start start 2023-10-16T20:39:40.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4fd2c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4f80c41f0 0x7fa4f8157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:40.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4fd2c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4f80c9bb0 0x7fa4f815aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:40.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4fd2c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4f8153770 0x7fa4f81589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:40.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4fd2c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f8158ef0 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4fd2c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f8159900 con 0x7fa4f8153770 2023-10-16T20:39:40.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4fd2c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f8159380 con 0x7fa4f80c41f0 2023-10-16T20:39:40.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4f67fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4f80c9bb0 0x7fa4f815aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:40.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4f67fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4f80c9bb0 0x7fa4f815aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33266/0 (socket says 172.21.15.88:33266) 2023-10-16T20:39:40.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4f67fc640 1 -- 172.21.15.88:0/1779378039 learned_addr learned my addr 172.21.15.88:0/1779378039 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:40.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4f77fe640 1 --2- 172.21.15.88:0/1779378039 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4f8153770 0x7fa4f81589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:40.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4f6ffd640 1 --2- 172.21.15.88:0/1779378039 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4f80c41f0 0x7fa4f8157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:40.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4f67fc640 1 -- 172.21.15.88:0/1779378039 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4f80c41f0 msgr2=0x7fa4f8157f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:40.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4f67fc640 1 --2- 172.21.15.88:0/1779378039 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4f80c41f0 0x7fa4f8157f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4f67fc640 1 -- 172.21.15.88:0/1779378039 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4f8153770 msgr2=0x7fa4f81589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:40.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4f67fc640 1 --2- 172.21.15.88:0/1779378039 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4f8153770 0x7fa4f81589b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4f67fc640 1 -- 172.21.15.88:0/1779378039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ec0094d0 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4f6ffd640 1 --2- 172.21.15.88:0/1779378039 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4f80c41f0 0x7fa4f8157f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:40.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4f67fc640 1 --2- 172.21.15.88:0/1779378039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4f80c9bb0 0x7fa4f815aa30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec01abf0 tx=0x7fa4ec01ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:40.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4df7fe640 1 -- 172.21.15.88:0/1779378039 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec0b4800 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4df7fe640 1 -- 172.21.15.88:0/1779378039 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ec0b4980 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4fd2c9640 1 -- 172.21.15.88:0/1779378039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f8157460 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4df7fe640 1 -- 172.21.15.88:0/1779378039 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec0b4d30 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4fd2c9640 1 -- 172.21.15.88:0/1779378039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f8157700 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.213+0000 7fa4fd2c9640 1 -- 172.21.15.88:0/1779378039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f8074700 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.217+0000 7fa4df7fe640 1 -- 172.21.15.88:0/1779378039 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa4ec0c7390 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.217+0000 7fa4df7fe640 1 --2- 172.21.15.88:0/1779378039 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa4e40664a0 0x7fa4e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:40.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.217+0000 7fa4df7fe640 1 -- 172.21.15.88:0/1779378039 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa4ec11eb30 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.217+0000 7fa4f6ffd640 1 --2- 172.21.15.88:0/1779378039 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa4e40664a0 0x7fa4e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:40.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.217+0000 7fa4f6ffd640 1 --2- 172.21.15.88:0/1779378039 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa4e40664a0 0x7fa4e4068960 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8000fc0 tx=0x7fa4e8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:40.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.221+0000 7fa4df7fe640 1 -- 172.21.15.88:0/1779378039 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa4ec0ebd30 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.373+0000 7fa4fd2c9640 1 -- 172.21.15.88:0/1779378039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4f80cbb90 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.373+0000 7fa4df7fe640 1 -- 172.21.15.88:0/1779378039 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18710 (secure 0 0 0) 0x7fa4ec0ab610 con 0x7fa4f80c9bb0 2023-10-16T20:39:40.381 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:40.381 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:37:37.392549+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:39:40.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.373+0000 7fa4fd2c9640 1 -- 172.21.15.88:0/1779378039 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa4e40664a0 msgr2=0x7fa4e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:40.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.373+0000 7fa4fd2c9640 1 --2- 172.21.15.88:0/1779378039 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa4e40664a0 0x7fa4e4068960 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8000fc0 tx=0x7fa4e8009040 comp rx=0 tx=0).stop 2023-10-16T20:39:40.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.373+0000 7fa4fd2c9640 1 -- 172.21.15.88:0/1779378039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4f80c9bb0 msgr2=0x7fa4f815aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:40.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.373+0000 7fa4fd2c9640 1 --2- 172.21.15.88:0/1779378039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4f80c9bb0 0x7fa4f815aa30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec01abf0 tx=0x7fa4ec01ad80 comp rx=0 tx=0).stop 2023-10-16T20:39:40.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.373+0000 7fa4fd2c9640 1 -- 172.21.15.88:0/1779378039 shutdown_connections 2023-10-16T20:39:40.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.373+0000 7fa4fd2c9640 1 --2- 172.21.15.88:0/1779378039 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa4e40664a0 0x7fa4e4068960 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.373+0000 7fa4fd2c9640 1 --2- 172.21.15.88:0/1779378039 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4f8153770 0x7fa4f81589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.373+0000 7fa4fd2c9640 1 --2- 172.21.15.88:0/1779378039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4f80c9bb0 0x7fa4f815aa30 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.373+0000 7fa4fd2c9640 1 --2- 172.21.15.88:0/1779378039 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4f80c41f0 0x7fa4f8157f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.373+0000 7fa4fd2c9640 1 -- 172.21.15.88:0/1779378039 >> 172.21.15.88:0/1779378039 conn(0x7fa4f80715c0 msgr2=0x7fa4f80c4740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:40.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.377+0000 7fa4fd2c9640 1 -- 172.21.15.88:0/1779378039 shutdown_connections 2023-10-16T20:39:40.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.377+0000 7fa4fd2c9640 1 -- 172.21.15.88:0/1779378039 wait complete. 2023-10-16T20:39:40.403 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:39:40.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.537+0000 7f115b7ab640 1 Processor -- start 2023-10-16T20:39:40.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.537+0000 7f115b7ab640 1 -- start start 2023-10-16T20:39:40.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.541+0000 7f115b7ab640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1154153770 0x7f1154159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:40.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.541+0000 7f115b7ab640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f11540c9bb0 0x7f11540c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:40.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.541+0000 7f115b7ab640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115415a410 con 0x7f11540c41f0 2023-10-16T20:39:40.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.541+0000 7f115b7ab640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11540cc250 con 0x7f11540c9bb0 2023-10-16T20:39:40.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.541+0000 7f115b7ab640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11540cc670 con 0x7f1154153770 2023-10-16T20:39:40.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.541+0000 7f1159547640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1154153770 0x7f1154159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:40.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.541+0000 7f1159d48640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f11540c41f0 0x7f11540cece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:45574/0 (socket says 172.21.15.88:45574) 2023-10-16T20:39:40.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.541+0000 7f1159d48640 1 -- 172.21.15.88:0/2105787726 learned_addr learned my addr 172.21.15.88:0/2105787726 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:40.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.541+0000 7f1158d46640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f11540c9bb0 0x7f11540c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:40.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.541+0000 7f1159547640 1 -- 172.21.15.88:0/2105787726 >> v2:172.21.15.191:3300/0 conn(0x7f11540c9bb0 msgr2=0x7f11540c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:40.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.541+0000 7f1159547640 1 --2- 172.21.15.88:0/2105787726 >> v2:172.21.15.191:3300/0 conn(0x7f11540c9bb0 0x7f11540c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.541+0000 7f1159547640 1 -- 172.21.15.88:0/2105787726 >> v1:172.21.15.191:6789/0 conn(0x7f11540c41f0 legacy=0x7f11540cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:40.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.541+0000 7f1159547640 1 -- 172.21.15.88:0/2105787726 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1154153b50 con 0x7f1154153770 2023-10-16T20:39:40.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.541+0000 7f1158d46640 1 --2- 172.21.15.88:0/2105787726 >> v2:172.21.15.191:3300/0 conn(0x7f11540c9bb0 0x7f11540c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:39:40.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.545+0000 7f1159547640 1 --2- 172.21.15.88:0/2105787726 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1154153770 0x7f1154159ed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1150009de0 tx=0x7f115001c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=707fbad7618a046f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:40.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.545+0000 7f114bfff640 1 -- 172.21.15.88:0/2105787726 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11500a5960 con 0x7f1154153770 2023-10-16T20:39:40.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.545+0000 7f114bfff640 1 -- 172.21.15.88:0/2105787726 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11500a5ac0 con 0x7f1154153770 2023-10-16T20:39:40.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.545+0000 7f114bfff640 1 -- 172.21.15.88:0/2105787726 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11500a5e50 con 0x7f1154153770 2023-10-16T20:39:40.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.545+0000 7f115b7ab640 1 -- 172.21.15.88:0/2105787726 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1154153770 msgr2=0x7f1154159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:40.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.545+0000 7f115b7ab640 1 --2- 172.21.15.88:0/2105787726 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1154153770 0x7f1154159ed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1150009de0 tx=0x7f115001c990 comp rx=0 tx=0).stop 2023-10-16T20:39:40.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.545+0000 7f115b7ab640 1 -- 172.21.15.88:0/2105787726 shutdown_connections 2023-10-16T20:39:40.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.545+0000 7f115b7ab640 1 --2- 172.21.15.88:0/2105787726 >> v2:172.21.15.191:3300/0 conn(0x7f11540c9bb0 0x7f11540c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.545+0000 7f115b7ab640 1 --2- 172.21.15.88:0/2105787726 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1154153770 0x7f1154159ed0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.545+0000 7f115b7ab640 1 -- 172.21.15.88:0/2105787726 >> 172.21.15.88:0/2105787726 conn(0x7f11540715c0 msgr2=0x7f11540719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:40.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.545+0000 7f115b7ab640 1 -- 172.21.15.88:0/2105787726 shutdown_connections 2023-10-16T20:39:40.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.545+0000 7f115b7ab640 1 -- 172.21.15.88:0/2105787726 wait complete. 2023-10-16T20:39:40.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f115b7ab640 1 Processor -- start 2023-10-16T20:39:40.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f115b7ab640 1 -- start start 2023-10-16T20:39:40.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f115b7ab640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11540c41f0 0x7f1154141930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:40.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f115b7ab640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11540c9bb0 0x7f11541f4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:40.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f115b7ab640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1154153770 0x7f11541f54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:40.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f115b7ab640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11541f5ea0 con 0x7f11540c9bb0 2023-10-16T20:39:40.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f115b7ab640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115415ba80 con 0x7f11540c41f0 2023-10-16T20:39:40.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f115b7ab640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1154154820 con 0x7f1154153770 2023-10-16T20:39:40.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f1158d46640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11540c9bb0 0x7f11541f4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:40.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f1158d46640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11540c9bb0 0x7f11541f4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33276/0 (socket says 172.21.15.88:33276) 2023-10-16T20:39:40.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f1158d46640 1 -- 172.21.15.88:0/2492950468 learned_addr learned my addr 172.21.15.88:0/2492950468 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:40.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f1159547640 1 --2- 172.21.15.88:0/2492950468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11540c41f0 0x7f1154141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:40.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f1159d48640 1 --2- 172.21.15.88:0/2492950468 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1154153770 0x7f11541f54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:40.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f1158d46640 1 -- 172.21.15.88:0/2492950468 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1154153770 msgr2=0x7f11541f54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:40.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f1158d46640 1 --2- 172.21.15.88:0/2492950468 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1154153770 0x7f11541f54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f1158d46640 1 -- 172.21.15.88:0/2492950468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11540c41f0 msgr2=0x7f1154141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:40.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f1158d46640 1 --2- 172.21.15.88:0/2492950468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11540c41f0 0x7f1154141930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f1158d46640 1 -- 172.21.15.88:0/2492950468 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1144009420 con 0x7f11540c9bb0 2023-10-16T20:39:40.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f1159d48640 1 --2- 172.21.15.88:0/2492950468 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1154153770 0x7f11541f54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:40.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f1158d46640 1 --2- 172.21.15.88:0/2492950468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11540c9bb0 0x7f11541f4fb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f114400ec10 tx=0x7f114400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:40.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f1149ffb640 1 -- 172.21.15.88:0/2492950468 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114400cc30 con 0x7f11540c9bb0 2023-10-16T20:39:40.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f1149ffb640 1 -- 172.21.15.88:0/2492950468 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114400cd90 con 0x7f11540c9bb0 2023-10-16T20:39:40.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f1149ffb640 1 -- 172.21.15.88:0/2492950468 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1144013690 con 0x7f11540c9bb0 2023-10-16T20:39:40.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f115b7ab640 1 -- 172.21.15.88:0/2492950468 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1150009a70 con 0x7f11540c9bb0 2023-10-16T20:39:40.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f115b7ab640 1 -- 172.21.15.88:0/2492950468 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f115415b150 con 0x7f11540c9bb0 2023-10-16T20:39:40.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.549+0000 7f115b7ab640 1 -- 172.21.15.88:0/2492950468 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1154074700 con 0x7f11540c9bb0 2023-10-16T20:39:40.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.553+0000 7f1149ffb640 1 -- 172.21.15.88:0/2492950468 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f11440137f0 con 0x7f11540c9bb0 2023-10-16T20:39:40.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.553+0000 7f1149ffb640 1 --2- 172.21.15.88:0/2492950468 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f11380664a0 0x7f1138068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:40.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.553+0000 7f1149ffb640 1 -- 172.21.15.88:0/2492950468 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f1144084590 con 0x7f11540c9bb0 2023-10-16T20:39:40.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.553+0000 7f1159547640 1 --2- 172.21.15.88:0/2492950468 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f11380664a0 0x7f1138068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:40.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.553+0000 7f1159547640 1 --2- 172.21.15.88:0/2492950468 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f11380664a0 0x7f1138068960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f11500af000 tx=0x7f11500a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:40.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.557+0000 7f1149ffb640 1 -- 172.21.15.88:0/2492950468 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1144051660 con 0x7f11540c9bb0 2023-10-16T20:39:40.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.737+0000 7f115b7ab640 1 -- 172.21.15.88:0/2492950468 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11540c90c0 con 0x7f11540c9bb0 2023-10-16T20:39:40.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.737+0000 7f1149ffb640 1 -- 172.21.15.88:0/2492950468 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6559 (secure 0 0 0) 0x7f1144054c80 con 0x7f11540c9bb0 2023-10-16T20:39:40.746 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:40.746 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:39:39.882115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":44,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:39:40.747 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 19 2023-10-16T20:39:40.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.741+0000 7f115b7ab640 1 -- 172.21.15.88:0/2492950468 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f11380664a0 msgr2=0x7f1138068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:40.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.741+0000 7f115b7ab640 1 --2- 172.21.15.88:0/2492950468 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f11380664a0 0x7f1138068960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f11500af000 tx=0x7f11500a7070 comp rx=0 tx=0).stop 2023-10-16T20:39:40.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.741+0000 7f115b7ab640 1 -- 172.21.15.88:0/2492950468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11540c9bb0 msgr2=0x7f11541f4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:40.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.741+0000 7f115b7ab640 1 --2- 172.21.15.88:0/2492950468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11540c9bb0 0x7f11541f4fb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f114400ec10 tx=0x7f114400c520 comp rx=0 tx=0).stop 2023-10-16T20:39:40.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.741+0000 7f115b7ab640 1 -- 172.21.15.88:0/2492950468 shutdown_connections 2023-10-16T20:39:40.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.741+0000 7f115b7ab640 1 --2- 172.21.15.88:0/2492950468 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f11380664a0 0x7f1138068960 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.741+0000 7f115b7ab640 1 --2- 172.21.15.88:0/2492950468 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1154153770 0x7f11541f54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.741+0000 7f115b7ab640 1 --2- 172.21.15.88:0/2492950468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11540c9bb0 0x7f11541f4fb0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.741+0000 7f115b7ab640 1 --2- 172.21.15.88:0/2492950468 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11540c41f0 0x7f1154141930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:40.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.741+0000 7f115b7ab640 1 -- 172.21.15.88:0/2492950468 >> 172.21.15.88:0/2492950468 conn(0x7f11540715c0 msgr2=0x7f1154151e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:40.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.741+0000 7f115b7ab640 1 -- 172.21.15.88:0/2492950468 shutdown_connections 2023-10-16T20:39:40.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:40.741+0000 7f115b7ab640 1 -- 172.21.15.88:0/2492950468 wait complete. 2023-10-16T20:39:41.371 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-16T20:36:55.111692+0000', 'modified': '2023-10-16T20:39:39.882115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4288, 'mds_1': 4279, 'mds_2': 4275, 'mds_3': 4263, 'mds_4': 4250}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4250': {'gid': 4250, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 44, 'addr': '172.21.15.191:6837/1120825335', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6836', 'nonce': 1120825335}, {'type': 'v1', 'addr': '172.21.15.191:6837', 'nonce': 1120825335}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4263': {'gid': 4263, 'name': 'a', 'rank': 3, 'incarnation': 16, 'state': 'up:active', 'state_seq': 43, 'addr': '172.21.15.88:6835/518281867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 518281867}, {'type': 'v1', 'addr': '172.21.15.88:6835', 'nonce': 518281867}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4275': {'gid': 4275, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 41, 'addr': '172.21.15.88:6838/321173984', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6836', 'nonce': 321173984}, {'type': 'v1', 'addr': '172.21.15.88:6838', 'nonce': 321173984}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4279': {'gid': 4279, 'name': 'd', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 40, 'addr': '172.21.15.191:6833/2446850309', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6832', 'nonce': 2446850309}, {'type': 'v1', 'addr': '172.21.15.191:6833', 'nonce': 2446850309}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4288': {'gid': 4288, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.191:6835/3059699574', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 3059699574}, {'type': 'v1', 'addr': '172.21.15.191:6835', 'nonce': 3059699574}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-16T20:39:41.372 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2023-10-16T20:39:41.372 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-16T20:39:41.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.505+0000 7f89f6e17640 1 Processor -- start 2023-10-16T20:39:41.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.505+0000 7f89f6e17640 1 -- start start 2023-10-16T20:39:41.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.505+0000 7f89f6e17640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f89f0153a30 0x7f89f00c1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:41.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.509+0000 7f89f6e17640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f01596e0 con 0x7f89f0071f90 2023-10-16T20:39:41.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.509+0000 7f89f6e17640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f00c2da0 con 0x7f89f0071450 2023-10-16T20:39:41.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.509+0000 7f89f6e17640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f00c6cc0 con 0x7f89f0153a30 2023-10-16T20:39:41.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.509+0000 7f89f4bb3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f89f0153a30 0x7f89f00c1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:41.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.509+0000 7f89f4bb3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f89f0153a30 0x7f89f00c1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33288/0 (socket says 172.21.15.88:33288) 2023-10-16T20:39:41.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.509+0000 7f89f4bb3640 1 -- 172.21.15.88:0/1328019737 learned_addr learned my addr 172.21.15.88:0/1328019737 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:41.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.509+0000 7f89f53b4640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f89f0071f90 0x7f89f0163f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55962/0 (socket says 172.21.15.88:55962) 2023-10-16T20:39:41.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.509+0000 7f89f4bb3640 1 -- 172.21.15.88:0/1328019737 >> v1:172.21.15.191:6789/0 conn(0x7f89f0071450 legacy=0x7f89f00c65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:41.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.509+0000 7f89f4bb3640 1 -- 172.21.15.88:0/1328019737 >> v1:172.21.15.88:6789/0 conn(0x7f89f0071f90 legacy=0x7f89f0163f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:41.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.509+0000 7f89f4bb3640 1 -- 172.21.15.88:0/1328019737 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f00c1bb0 con 0x7f89f0153a30 2023-10-16T20:39:41.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.509+0000 7f89ef7fe640 1 -- 172.21.15.88:0/1328019737 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382597213 0 0) 0x7f89f01596e0 con 0x7f89f0071f90 2023-10-16T20:39:41.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.509+0000 7f89f4bb3640 1 --2- 172.21.15.88:0/1328019737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89f0153a30 0x7f89f00c1520 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009870 tx=0x7f89e401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1939c066d826ae4d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:41.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.509+0000 7f89ef7fe640 1 -- 172.21.15.88:0/1328019737 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e40aa430 con 0x7f89f0153a30 2023-10-16T20:39:41.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.509+0000 7f89ef7fe640 1 -- 172.21.15.88:0/1328019737 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e40aa590 con 0x7f89f0153a30 2023-10-16T20:39:41.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.509+0000 7f89ef7fe640 1 -- 172.21.15.88:0/1328019737 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e40aa920 con 0x7f89f0153a30 2023-10-16T20:39:41.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f6e17640 1 -- 172.21.15.88:0/1328019737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89f0153a30 msgr2=0x7f89f00c1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:41.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f6e17640 1 --2- 172.21.15.88:0/1328019737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89f0153a30 0x7f89f00c1520 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009870 tx=0x7f89e401c4d0 comp rx=0 tx=0).stop 2023-10-16T20:39:41.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f6e17640 1 -- 172.21.15.88:0/1328019737 shutdown_connections 2023-10-16T20:39:41.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f6e17640 1 --2- 172.21.15.88:0/1328019737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89f0153a30 0x7f89f00c1520 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:41.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f6e17640 1 -- 172.21.15.88:0/1328019737 >> 172.21.15.88:0/1328019737 conn(0x7f89f00c86b0 msgr2=0x7f89f00c8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:41.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f6e17640 1 -- 172.21.15.88:0/1328019737 shutdown_connections 2023-10-16T20:39:41.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f6e17640 1 -- 172.21.15.88:0/1328019737 wait complete. 2023-10-16T20:39:41.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f6e17640 1 Processor -- start 2023-10-16T20:39:41.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f6e17640 1 -- start start 2023-10-16T20:39:41.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f6e17640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89f0071450 0x7f89f0164f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:41.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f6e17640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89f0071f90 0x7f89f0150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:41.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f6e17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89f0153a30 0x7f89f0157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:41.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f6e17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f01519b0 con 0x7f89f0153a30 2023-10-16T20:39:41.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f6e17640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f01524f0 con 0x7f89f0071450 2023-10-16T20:39:41.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f6e17640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f014ddf0 con 0x7f89f0071f90 2023-10-16T20:39:41.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f53b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89f0153a30 0x7f89f0157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:41.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f53b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89f0153a30 0x7f89f0157320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33294/0 (socket says 172.21.15.88:33294) 2023-10-16T20:39:41.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f53b4640 1 -- 172.21.15.88:0/4178122177 learned_addr learned my addr 172.21.15.88:0/4178122177 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:41.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89f4bb3640 1 --2- 172.21.15.88:0/4178122177 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89f0071450 0x7f89f0164f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:41.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.513+0000 7f89effff640 1 --2- 172.21.15.88:0/4178122177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89f0071f90 0x7f89f0150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:41.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.517+0000 7f89f53b4640 1 -- 172.21.15.88:0/4178122177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89f0071f90 msgr2=0x7f89f0150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:41.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.517+0000 7f89f53b4640 1 --2- 172.21.15.88:0/4178122177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89f0071f90 0x7f89f0150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:41.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.517+0000 7f89f53b4640 1 -- 172.21.15.88:0/4178122177 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89f0071450 msgr2=0x7f89f0164f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:41.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.517+0000 7f89f53b4640 1 --2- 172.21.15.88:0/4178122177 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89f0071450 0x7f89f0164f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:41.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.517+0000 7f89f53b4640 1 -- 172.21.15.88:0/4178122177 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e40094d0 con 0x7f89f0153a30 2023-10-16T20:39:41.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.517+0000 7f89effff640 1 --2- 172.21.15.88:0/4178122177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89f0071f90 0x7f89f0150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:41.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.517+0000 7f89f53b4640 1 --2- 172.21.15.88:0/4178122177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89f0153a30 0x7f89f0157320 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f89e8001570 tx=0x7f89e800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:41.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.517+0000 7f89ed7fa640 1 -- 172.21.15.88:0/4178122177 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e800a5e0 con 0x7f89f0153a30 2023-10-16T20:39:41.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.517+0000 7f89ed7fa640 1 -- 172.21.15.88:0/4178122177 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e800a760 con 0x7f89f0153a30 2023-10-16T20:39:41.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.517+0000 7f89ed7fa640 1 -- 172.21.15.88:0/4178122177 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e800ab10 con 0x7f89f0153a30 2023-10-16T20:39:41.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.517+0000 7f89f6e17640 1 -- 172.21.15.88:0/4178122177 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f0150b00 con 0x7f89f0153a30 2023-10-16T20:39:41.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.517+0000 7f89f6e17640 1 -- 172.21.15.88:0/4178122177 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f014e040 con 0x7f89f0153a30 2023-10-16T20:39:41.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.517+0000 7f89ed7fa640 1 -- 172.21.15.88:0/4178122177 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f89e800ac70 con 0x7f89f0153a30 2023-10-16T20:39:41.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89ed7fa640 1 --2- 172.21.15.88:0/4178122177 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f89d40664a0 0x7f89d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:41.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89ed7fa640 1 -- 172.21.15.88:0/4178122177 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f89e8083660 con 0x7f89f0153a30 2023-10-16T20:39:41.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89f4bb3640 1 --2- 172.21.15.88:0/4178122177 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f89d40664a0 0x7f89d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:41.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89fa1b0000 1 Processor -- start 2023-10-16T20:39:41.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89fa1b0000 1 -- start start 2023-10-16T20:39:41.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89f4bb3640 1 --2- 172.21.15.88:0/4178122177 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f89d40664a0 0x7f89d4068960 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f89e40ae000 tx=0x7f89e40b5040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:41.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89fa1b0000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d7810 0x55d4b83d7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:41.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89fa1b0000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 0x55d4b83d8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:41.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89fa1b0000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:41.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89fa1b0000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b818bcc0 con 0x55d4b83d8a50 2023-10-16T20:39:41.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89fa1b0000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b81ee920 con 0x55d4b83d8130 2023-10-16T20:39:41.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89fa1b0000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d4b81ebe50 con 0x55d4b83d7810 2023-10-16T20:39:41.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89f53b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:41.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89f53b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33300/0 (socket says 172.21.15.88:33300) 2023-10-16T20:39:41.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89f53b4640 1 -- 172.21.15.88:0/3090495575 learned_addr learned my addr 172.21.15.88:0/3090495575 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:41.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89effff640 1 --2- 172.21.15.88:0/3090495575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 0x55d4b83d8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:41.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89f4bb3640 1 --2- 172.21.15.88:0/3090495575 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d7810 0x55d4b83d7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:41.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89f53b4640 1 -- 172.21.15.88:0/3090495575 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d7810 msgr2=0x55d4b83d7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:41.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89f53b4640 1 --2- 172.21.15.88:0/3090495575 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d7810 0x55d4b83d7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:41.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89f53b4640 1 -- 172.21.15.88:0/3090495575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 msgr2=0x55d4b83d8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:41.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89f53b4640 1 --2- 172.21.15.88:0/3090495575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 0x55d4b83d8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:41.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89f53b4640 1 -- 172.21.15.88:0/3090495575 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4b7ee0890 con 0x55d4b83d8a50 2023-10-16T20:39:41.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89effff640 1 --2- 172.21.15.88:0/3090495575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 0x55d4b83d8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:41.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89f4bb3640 1 --2- 172.21.15.88:0/3090495575 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d7810 0x55d4b83d7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:41.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89f53b4640 1 --2- 172.21.15.88:0/3090495575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e2aa0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f89e8001a30 tx=0x7f89e8001470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a8f088a9eb95f24 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:41.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89da7fc640 1 -- 172.21.15.88:0/3090495575 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e805c6d0 con 0x55d4b83d8a50 2023-10-16T20:39:41.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89da7fc640 1 -- 172.21.15.88:0/3090495575 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e8052820 con 0x55d4b83d8a50 2023-10-16T20:39:41.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89da7fc640 1 -- 172.21.15.88:0/3090495575 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e8058160 con 0x55d4b83d8a50 2023-10-16T20:39:41.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89fa1b0000 1 -- 172.21.15.88:0/3090495575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 msgr2=0x55d4b83e2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:41.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/3090495575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e2aa0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f89e8001a30 tx=0x7f89e8001470 comp rx=0 tx=0).stop 2023-10-16T20:39:41.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89fa1b0000 1 -- 172.21.15.88:0/3090495575 shutdown_connections 2023-10-16T20:39:41.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/3090495575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e2aa0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:41.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/3090495575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 0x55d4b83d8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:41.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/3090495575 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d7810 0x55d4b83d7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:41.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89fa1b0000 1 -- 172.21.15.88:0/3090495575 >> 172.21.15.88:0/3090495575 conn(0x55d4b838dfc0 msgr2=0x55d4b83d55d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:41.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89fa1b0000 1 -- 172.21.15.88:0/3090495575 shutdown_connections 2023-10-16T20:39:41.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.521+0000 7f89fa1b0000 1 -- 172.21.15.88:0/3090495575 wait complete. 2023-10-16T20:39:41.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 1 -- start start 2023-10-16T20:39:41.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 1 Processor -- start 2023-10-16T20:39:41.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d7810 0x55d4b83d1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:41.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 0x55d4b83d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:41.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:41.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b81e7740 con 0x55d4b83d8a50 2023-10-16T20:39:41.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b81a6d10 con 0x55d4b83d8130 2023-10-16T20:39:41.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d4b81a69b0 con 0x55d4b83d7810 2023-10-16T20:39:41.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 10 client.0 _ll_register_callbacks cb 0x55d4b8382390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T20:39:41.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89f53b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:41.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89f53b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33306/0 (socket says 172.21.15.88:33306) 2023-10-16T20:39:41.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89f53b4640 1 -- 172.21.15.88:0/1580064495 learned_addr learned my addr 172.21.15.88:0/1580064495 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:41.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89f4bb3640 1 --2- 172.21.15.88:0/1580064495 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d7810 0x55d4b83d1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:41.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89effff640 1 --2- 172.21.15.88:0/1580064495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 0x55d4b83d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:41.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89f53b4640 1 -- 172.21.15.88:0/1580064495 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d7810 msgr2=0x55d4b83d1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:41.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89f53b4640 1 --2- 172.21.15.88:0/1580064495 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d7810 0x55d4b83d1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:41.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89f53b4640 1 -- 172.21.15.88:0/1580064495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 msgr2=0x55d4b83d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:41.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89f53b4640 1 --2- 172.21.15.88:0/1580064495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 0x55d4b83d2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:41.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89f53b4640 1 -- 172.21.15.88:0/1580064495 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f014e040 con 0x55d4b83d8a50 2023-10-16T20:39:41.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89effff640 1 --2- 172.21.15.88:0/1580064495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 0x55d4b83d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:41.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89f53b4640 1 --2- 172.21.15.88:0/1580064495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e0a50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f89e8007fd0 tx=0x7f89e8001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23fe5d1a112bdfbc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:41.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-10-16T20:39:41.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89cf7fe640 1 -- 172.21.15.88:0/1580064495 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e8058360 con 0x55d4b83d8a50 2023-10-16T20:39:41.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89cf7fe640 1 -- 172.21.15.88:0/1580064495 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f89e805c240 con 0x55d4b83d8a50 2023-10-16T20:39:41.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 1 -- 172.21.15.88:0/1580064495 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d4b819b370 con 0x55d4b83d8a50 2023-10-16T20:39:41.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89cf7fe640 1 -- 172.21.15.88:0/1580064495 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89e805c420 con 0x55d4b83d8a50 2023-10-16T20:39:41.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 10 client.4890 fetch_fsmap learned FSMap version 19 2023-10-16T20:39:41.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 1 -- 172.21.15.88:0/1580064495 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x55d4b81a9460 con 0x55d4b83d8a50 2023-10-16T20:39:41.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89cf7fe640 1 -- 172.21.15.88:0/1580064495 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e8076020 con 0x55d4b83d8a50 2023-10-16T20:39:41.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89cf7fe640 1 -- 172.21.15.88:0/1580064495 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) v1 ==== 4500+0+0 (secure 0 0 0) 0x7f89e8072120 con 0x55d4b83d8a50 2023-10-16T20:39:41.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 10 client.4890 fetch_fsmap finished waiting for FSMap version 19 2023-10-16T20:39:41.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 10 client.4890 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T20:39:41.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 20 client.4890 populate_metadata read hostname 'smithi088' 2023-10-16T20:39:41.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/1580064495 >> [v2:172.21.15.191:6836/1120825335,v1:172.21.15.191:6837/1120825335] conn(0x55d4b83e17c0 0x55d4b83e1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:41.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 4 client.4890 mds_command: new command op to 4250 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T20:39:41.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89fa1b0000 1 -- 172.21.15.88:0/1580064495 --> [v2:172.21.15.191:6836/1120825335,v1:172.21.15.191:6837/1120825335] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d4b81812d0 con 0x55d4b83e17c0 2023-10-16T20:39:41.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.525+0000 7f89f4bb3640 1 --2- 172.21.15.88:0/1580064495 >> [v2:172.21.15.191:6836/1120825335,v1:172.21.15.191:6837/1120825335] conn(0x55d4b83e17c0 0x55d4b83e1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:41.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.529+0000 7f89f4bb3640 1 --2- 172.21.15.88:0/1580064495 >> [v2:172.21.15.191:6836/1120825335,v1:172.21.15.191:6837/1120825335] conn(0x55d4b83e17c0 0x55d4b83e1ba0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=70ac2a8a2b8aa5b5 server_cookie=4ab47e1ef4b7a530 in_seq=0 out_seq=0 2023-10-16T20:39:41.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:41.529+0000 7f89cf7fe640 10 client.4890 ms_handle_connect on v2:172.21.15.191:6836/1120825335 2023-10-16T20:39:45.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.537+0000 7f89cf7fe640 1 -- 172.21.15.88:0/1580064495 <== mds.4 v2:172.21.15.191:6836/1120825335 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d4b81812d0 con 0x55d4b83e17c0 2023-10-16T20:39:45.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.537+0000 7f89cf7fe640 10 client.4890 handle_command_reply: tid=0 2023-10-16T20:39:45.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.537+0000 7f89fa1b0000 1 client.4890 shutdown 2023-10-16T20:39:45.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.537+0000 7f89fa1b0000 1 -- 172.21.15.88:0/1580064495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 msgr2=0x55d4b83e0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.537+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/1580064495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e0a50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f89e8007fd0 tx=0x7f89e8001260 comp rx=0 tx=0).stop 2023-10-16T20:39:45.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.541+0000 7f89fa1b0000 1 -- 172.21.15.88:0/1580064495 shutdown_connections 2023-10-16T20:39:45.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.541+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/1580064495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e0a50 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.541+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/1580064495 >> [v2:172.21.15.191:6836/1120825335,v1:172.21.15.191:6837/1120825335] conn(0x55d4b83e17c0 0x55d4b83e1ba0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.541+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/1580064495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 0x55d4b83d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.541+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/1580064495 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d7810 0x55d4b83d1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.541+0000 7f89fa1b0000 1 -- 172.21.15.88:0/1580064495 >> 172.21.15.88:0/1580064495 conn(0x55d4b838dfc0 msgr2=0x55d4b83d7090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:45.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.541+0000 7f89cf7fe640 0 client.4890 ms_handle_reset on v2:172.21.15.191:6836/1120825335 2023-10-16T20:39:45.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.541+0000 7f89fa1b0000 1 -- 172.21.15.88:0/1580064495 shutdown_connections 2023-10-16T20:39:45.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.541+0000 7f89fa1b0000 1 -- 172.21.15.88:0/1580064495 wait complete. 2023-10-16T20:39:45.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.541+0000 7f89fa1b0000 20 client.4890 trim_cache size 0 max 16384 2023-10-16T20:39:45.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89fa1b0000 1 Processor -- start 2023-10-16T20:39:45.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89fa1b0000 1 -- start start 2023-10-16T20:39:45.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89fa1b0000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d7810 0x55d4b83e5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89fa1b0000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 0x55d4b83e5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89fa1b0000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89fa1b0000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b7ea1940 con 0x55d4b83d8a50 2023-10-16T20:39:45.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89fa1b0000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b81ec8b0 con 0x55d4b83d8130 2023-10-16T20:39:45.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89fa1b0000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d4b81e4b40 con 0x55d4b83d7810 2023-10-16T20:39:45.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89f53b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89f4bb3640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d7810 0x55d4b83e5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89effff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 0x55d4b83e5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89f4bb3640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d7810 0x55d4b83e5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:46012/0 (socket says 172.21.15.88:46012) 2023-10-16T20:39:45.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89f53b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33322/0 (socket says 172.21.15.88:33322) 2023-10-16T20:39:45.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89effff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 0x55d4b83e5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:53524/0 (socket says 172.21.15.88:53524) 2023-10-16T20:39:45.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89f53b4640 1 -- 172.21.15.88:0/2839396059 learned_addr learned my addr 172.21.15.88:0/2839396059 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:45.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89f53b4640 1 -- 172.21.15.88:0/2839396059 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d7810 msgr2=0x55d4b83e5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89f53b4640 1 --2- 172.21.15.88:0/2839396059 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d7810 0x55d4b83e5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89f53b4640 1 -- 172.21.15.88:0/2839396059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 msgr2=0x55d4b83e5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89f53b4640 1 --2- 172.21.15.88:0/2839396059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 0x55d4b83e5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.557+0000 7f89f53b4640 1 -- 172.21.15.88:0/2839396059 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4b818f760 con 0x55d4b83d8a50 2023-10-16T20:39:45.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89f53b4640 1 --2- 172.21.15.88:0/2839396059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e64b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f89e805c1a0 tx=0x7f89e801e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4854ce0c4e2d6ef server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:45.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89daffd640 1 -- 172.21.15.88:0/2839396059 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e805c6d0 con 0x55d4b83d8a50 2023-10-16T20:39:45.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89daffd640 1 -- 172.21.15.88:0/2839396059 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e8076040 con 0x55d4b83d8a50 2023-10-16T20:39:45.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89daffd640 1 -- 172.21.15.88:0/2839396059 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e80830a0 con 0x55d4b83d8a50 2023-10-16T20:39:45.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 1 -- 172.21.15.88:0/2839396059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 msgr2=0x55d4b83e64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/2839396059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e64b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f89e805c1a0 tx=0x7f89e801e040 comp rx=0 tx=0).stop 2023-10-16T20:39:45.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 1 -- 172.21.15.88:0/2839396059 shutdown_connections 2023-10-16T20:39:45.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/2839396059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b83e64b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/2839396059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d8130 0x55d4b83e5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/2839396059 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d7810 0x55d4b83e5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 1 -- 172.21.15.88:0/2839396059 >> 172.21.15.88:0/2839396059 conn(0x55d4b838dfc0 msgr2=0x55d4b83e2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:45.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 1 -- 172.21.15.88:0/2839396059 shutdown_connections 2023-10-16T20:39:45.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 1 -- 172.21.15.88:0/2839396059 wait complete. 2023-10-16T20:39:45.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 1 -- start start 2023-10-16T20:39:45.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 1 Processor -- start 2023-10-16T20:39:45.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d7810 0x55d4b83e8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d8130 0x55d4b8435630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b8435b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b81e4f50 con 0x55d4b83d8a50 2023-10-16T20:39:45.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b81814c0 con 0x55d4b83d7810 2023-10-16T20:39:45.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d4b81b3250 con 0x55d4b83d8130 2023-10-16T20:39:45.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89fa1b0000 10 client.0 _ll_register_callbacks cb 0x7f89e80018e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T20:39:45.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89f53b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b8435b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89f53b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b8435b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33336/0 (socket says 172.21.15.88:33336) 2023-10-16T20:39:45.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.561+0000 7f89f53b4640 1 -- 172.21.15.88:0/1447680859 learned_addr learned my addr 172.21.15.88:0/1447680859 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:45.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89f4bb3640 1 --2- 172.21.15.88:0/1447680859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d7810 0x55d4b83e8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89effff640 1 --2- 172.21.15.88:0/1447680859 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d8130 0x55d4b8435630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89f53b4640 1 -- 172.21.15.88:0/1447680859 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d8130 msgr2=0x55d4b8435630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89f53b4640 1 --2- 172.21.15.88:0/1447680859 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d8130 0x55d4b8435630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89f53b4640 1 -- 172.21.15.88:0/1447680859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d7810 msgr2=0x55d4b83e8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89f53b4640 1 --2- 172.21.15.88:0/1447680859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d7810 0x55d4b83e8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89f53b4640 1 -- 172.21.15.88:0/1447680859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d4b81a9460 con 0x55d4b83d8a50 2023-10-16T20:39:45.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89f4bb3640 1 --2- 172.21.15.88:0/1447680859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d7810 0x55d4b83e8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:45.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89f53b4640 1 --2- 172.21.15.88:0/1447680859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b8435b70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f89e8054410 tx=0x7f89e8009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b30d55551487ee6c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:45.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-10-16T20:39:45.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89cf7fe640 1 -- 172.21.15.88:0/1447680859 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e80720f0 con 0x55d4b83d8a50 2023-10-16T20:39:45.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89cf7fe640 1 -- 172.21.15.88:0/1447680859 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f89e8077080 con 0x55d4b83d8a50 2023-10-16T20:39:45.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 1 -- 172.21.15.88:0/1447680859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d4b817f6f0 con 0x55d4b83d8a50 2023-10-16T20:39:45.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89cf7fe640 1 -- 172.21.15.88:0/1447680859 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89e8072250 con 0x55d4b83d8a50 2023-10-16T20:39:45.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 10 client.4896 fetch_fsmap learned FSMap version 19 2023-10-16T20:39:45.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 1 -- 172.21.15.88:0/1447680859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x55d4b7ee0cc0 con 0x55d4b83d8a50 2023-10-16T20:39:45.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89cf7fe640 1 -- 172.21.15.88:0/1447680859 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d4b817f6f0 con 0x55d4b83d8a50 2023-10-16T20:39:45.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89cf7fe640 1 -- 172.21.15.88:0/1447680859 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) v1 ==== 4500+0+0 (secure 0 0 0) 0x7f89e8058160 con 0x55d4b83d8a50 2023-10-16T20:39:45.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 10 client.4896 fetch_fsmap finished waiting for FSMap version 19 2023-10-16T20:39:45.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 10 client.4896 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-16T20:39:45.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 20 client.4896 populate_metadata read hostname 'smithi088' 2023-10-16T20:39:45.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/1447680859 >> [v2:172.21.15.191:6836/1120825335,v1:172.21.15.191:6837/1120825335] conn(0x55d4b8434200 0x55d4b84345e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 4 client.4896 mds_command: new command op to 4250 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T20:39:45.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 1 -- 172.21.15.88:0/1447680859 --> [v2:172.21.15.191:6836/1120825335,v1:172.21.15.191:6837/1120825335] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d4b81e0720 con 0x55d4b8434200 2023-10-16T20:39:45.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89f4bb3640 1 --2- 172.21.15.88:0/1447680859 >> [v2:172.21.15.191:6836/1120825335,v1:172.21.15.191:6837/1120825335] conn(0x55d4b8434200 0x55d4b84345e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89f4bb3640 1 --2- 172.21.15.88:0/1447680859 >> [v2:172.21.15.191:6836/1120825335,v1:172.21.15.191:6837/1120825335] conn(0x55d4b8434200 0x55d4b84345e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=52b73079a6c38b10 server_cookie=19c6a77e9b22c3ce in_seq=0 out_seq=0 2023-10-16T20:39:45.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89cf7fe640 10 client.4896 ms_handle_connect on v2:172.21.15.191:6836/1120825335 2023-10-16T20:39:45.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89cf7fe640 1 -- 172.21.15.88:0/1447680859 <== mds.4 v2:172.21.15.191:6836/1120825335 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55d4b81e0720 con 0x55d4b8434200 2023-10-16T20:39:45.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89cf7fe640 10 client.4896 handle_command_reply: tid=0 2023-10-16T20:39:45.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 1 client.4896 shutdown 2023-10-16T20:39:45.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 1 -- 172.21.15.88:0/1447680859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 msgr2=0x55d4b8435b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/1447680859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b8435b70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f89e8054410 tx=0x7f89e8009040 comp rx=0 tx=0).stop 2023-10-16T20:39:45.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 1 -- 172.21.15.88:0/1447680859 shutdown_connections 2023-10-16T20:39:45.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/1447680859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4b83d8a50 0x55d4b8435b70 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/1447680859 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55d4b83d8130 0x55d4b8435630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/1447680859 >> [v2:172.21.15.191:6836/1120825335,v1:172.21.15.191:6837/1120825335] conn(0x55d4b8434200 0x55d4b84345e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 1 --2- 172.21.15.88:0/1447680859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d4b83d7810 0x55d4b83e8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 1 -- 172.21.15.88:0/1447680859 >> 172.21.15.88:0/1447680859 conn(0x55d4b838dfc0 msgr2=0x55d4b83e9130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:45.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89cf7fe640 0 client.4896 ms_handle_reset on v2:172.21.15.191:6836/1120825335 2023-10-16T20:39:45.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 1 -- 172.21.15.88:0/1447680859 shutdown_connections 2023-10-16T20:39:45.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 1 -- 172.21.15.88:0/1447680859 wait complete. 2023-10-16T20:39:45.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.565+0000 7f89fa1b0000 20 client.4896 trim_cache size 0 max 16384 2023-10-16T20:39:45.584 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-10-16T20:39:45.584 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "8cf7342f-a394-4b4c-ac35-9931ac9d9e57", 2023-10-16T20:39:45.584 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 4, 2023-10-16T20:39:45.584 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 4250, 2023-10-16T20:39:45.584 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2023-10-16T20:39:45.585 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2023-10-16T20:39:45.585 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2023-10-16T20:39:45.585 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 11.056957142, 2023-10-16T20:39:45.585 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 19, 2023-10-16T20:39:45.585 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 29, 2023-10-16T20:39:45.586 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 29, 2023-10-16T20:39:45.586 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 176.64726895499999 2023-10-16T20:39:45.586 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-10-16T20:39:45.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.569+0000 7f89f6e17640 1 -- 172.21.15.88:0/4178122177 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f89d40664a0 msgr2=0x7f89d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.569+0000 7f89f6e17640 1 --2- 172.21.15.88:0/4178122177 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f89d40664a0 0x7f89d4068960 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f89e40ae000 tx=0x7f89e40b5040 comp rx=0 tx=0).stop 2023-10-16T20:39:45.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.569+0000 7f89f6e17640 1 -- 172.21.15.88:0/4178122177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89f0153a30 msgr2=0x7f89f0157320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.569+0000 7f89f6e17640 1 --2- 172.21.15.88:0/4178122177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89f0153a30 0x7f89f0157320 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f89e8001570 tx=0x7f89e800cca0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.569+0000 7f89f6e17640 1 -- 172.21.15.88:0/4178122177 shutdown_connections 2023-10-16T20:39:45.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.569+0000 7f89f6e17640 1 --2- 172.21.15.88:0/4178122177 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f89d40664a0 0x7f89d4068960 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.569+0000 7f89f6e17640 1 --2- 172.21.15.88:0/4178122177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89f0153a30 0x7f89f0157320 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.569+0000 7f89f6e17640 1 --2- 172.21.15.88:0/4178122177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f89f0071f90 0x7f89f0150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.569+0000 7f89f6e17640 1 --2- 172.21.15.88:0/4178122177 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89f0071450 0x7f89f0164f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.569+0000 7f89f6e17640 1 -- 172.21.15.88:0/4178122177 >> 172.21.15.88:0/4178122177 conn(0x7f89f00c86b0 msgr2=0x7f89f00c9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:45.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.569+0000 7f89f6e17640 1 -- 172.21.15.88:0/4178122177 shutdown_connections 2023-10-16T20:39:45.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.569+0000 7f89f6e17640 1 -- 172.21.15.88:0/4178122177 wait complete. 2023-10-16T20:39:45.593 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-16T20:39:45.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.733+0000 7f138df72640 1 Processor -- start 2023-10-16T20:39:45.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.733+0000 7f138df72640 1 -- start start 2023-10-16T20:39:45.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.733+0000 7f138df72640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13880c1520 0x7f1388164f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.733+0000 7f138df72640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13880c1930 con 0x7f1388153a30 2023-10-16T20:39:45.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.733+0000 7f138df72640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13881596e0 con 0x7f1388071450 2023-10-16T20:39:45.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.733+0000 7f138df72640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13880c6cc0 con 0x7f13880c1520 2023-10-16T20:39:45.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.733+0000 7f13877fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1388153a30 0x7f13880c65b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55978/0 (socket says 172.21.15.88:55978) 2023-10-16T20:39:45.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.733+0000 7f13877fe640 1 -- 172.21.15.88:0/1962205499 learned_addr learned my addr 172.21.15.88:0/1962205499 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:45.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f1387fff640 1 --2- 172.21.15.88:0/1962205499 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13880c1520 0x7f1388164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13867fc640 1 -- 172.21.15.88:0/1962205499 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712173341 0 0) 0x7f13880c1930 con 0x7f1388153a30 2023-10-16T20:39:45.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13867fc640 1 -- 172.21.15.88:0/1962205499 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1370003530 con 0x7f1388153a30 2023-10-16T20:39:45.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13867fc640 1 -- 172.21.15.88:0/1962205499 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247182276 0 0) 0x7f13881596e0 con 0x7f1388071450 2023-10-16T20:39:45.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13867fc640 1 -- 172.21.15.88:0/1962205499 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13880c1930 con 0x7f1388071450 2023-10-16T20:39:45.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13867fc640 1 -- 172.21.15.88:0/1962205499 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 984011714 0 0) 0x7f1370003530 con 0x7f1388153a30 2023-10-16T20:39:45.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13867fc640 1 -- 172.21.15.88:0/1962205499 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13880c1520 msgr2=0x7f1388164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13867fc640 1 --2- 172.21.15.88:0/1962205499 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13880c1520 0x7f1388164f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13867fc640 1 -- 172.21.15.88:0/1962205499 >> v1:172.21.15.191:6789/0 conn(0x7f1388071450 legacy=0x7f1388071f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13867fc640 1 -- 172.21.15.88:0/1962205499 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13880c2da0 con 0x7f1388153a30 2023-10-16T20:39:45.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13867fc640 1 -- 172.21.15.88:0/1962205499 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f1378001680 con 0x7f1388153a30 2023-10-16T20:39:45.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13867fc640 1 -- 172.21.15.88:0/1962205499 >> v1:172.21.15.88:6789/0 conn(0x7f1388153a30 legacy=0x7f13880c65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13867fc640 1 --2- 172.21.15.88:0/1962205499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1370003b40 0x7f1370003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13867fc640 1 --2- 172.21.15.88:0/1962205499 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13880c1520 0x7f13700047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13867fc640 1 --2- 172.21.15.88:0/1962205499 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1370004d10 0x7f1370005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13867fc640 1 -- 172.21.15.88:0/1962205499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13880c6cc0 con 0x7f1370003b40 2023-10-16T20:39:45.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13867fc640 1 -- 172.21.15.88:0/1962205499 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137000ba40 con 0x7f1370004d10 2023-10-16T20:39:45.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13867fc640 1 -- 172.21.15.88:0/1962205499 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137000bc00 con 0x7f13880c1520 2023-10-16T20:39:45.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13877fe640 1 --2- 172.21.15.88:0/1962205499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1370003b40 0x7f1370003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f1387fff640 1 --2- 172.21.15.88:0/1962205499 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1370004d10 0x7f1370005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f1386ffd640 1 --2- 172.21.15.88:0/1962205499 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13880c1520 0x7f13700047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13877fe640 1 -- 172.21.15.88:0/1962205499 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13880c1520 msgr2=0x7f13700047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13877fe640 1 --2- 172.21.15.88:0/1962205499 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13880c1520 0x7f13700047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13877fe640 1 -- 172.21.15.88:0/1962205499 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1370004d10 msgr2=0x7f1370005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13877fe640 1 --2- 172.21.15.88:0/1962205499 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1370004d10 0x7f1370005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.737+0000 7f13877fe640 1 -- 172.21.15.88:0/1962205499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1370003530 con 0x7f1370003b40 2023-10-16T20:39:45.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.741+0000 7f13877fe640 1 --2- 172.21.15.88:0/1962205499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1370003b40 0x7f1370003f40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1378007fa0 tx=0x7f137801cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f02bf239e9ab7b42 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:45.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.741+0000 7f13867fc640 1 -- 172.21.15.88:0/1962205499 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1378005ae0 con 0x7f1370003b40 2023-10-16T20:39:45.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.741+0000 7f13867fc640 1 -- 172.21.15.88:0/1962205499 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1378005c40 con 0x7f1370003b40 2023-10-16T20:39:45.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.741+0000 7f13867fc640 1 -- 172.21.15.88:0/1962205499 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13780a57a0 con 0x7f1370003b40 2023-10-16T20:39:45.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.741+0000 7f138df72640 1 -- 172.21.15.88:0/1962205499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1370003b40 msgr2=0x7f1370003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.741+0000 7f138df72640 1 --2- 172.21.15.88:0/1962205499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1370003b40 0x7f1370003f40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1378007fa0 tx=0x7f137801cad0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.741+0000 7f1387fff640 1 -- 172.21.15.88:0/1962205499 reap_dead start 2023-10-16T20:39:45.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.741+0000 7f138df72640 1 -- 172.21.15.88:0/1962205499 shutdown_connections 2023-10-16T20:39:45.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.741+0000 7f138df72640 1 -- 172.21.15.88:0/1962205499 >> 172.21.15.88:0/1962205499 conn(0x7f13880c86b0 msgr2=0x7f13880c8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:45.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.741+0000 7f138df72640 1 -- 172.21.15.88:0/1962205499 shutdown_connections 2023-10-16T20:39:45.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.741+0000 7f138df72640 1 -- 172.21.15.88:0/1962205499 wait complete. 2023-10-16T20:39:45.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.741+0000 7f138df72640 1 Processor -- start 2023-10-16T20:39:45.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.741+0000 7f138df72640 1 -- start start 2023-10-16T20:39:45.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.741+0000 7f138df72640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1370003b40 0x7f138814c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.741+0000 7f138df72640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f138814a310 0x7f1388144840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f138df72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f138814bc30 0x7f138814e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f138df72640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138814c470 con 0x7f138814bc30 2023-10-16T20:39:45.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f138df72640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138814cd30 con 0x7f1370003b40 2023-10-16T20:39:45.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f138df72640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f138814dab0 con 0x7f138814a310 2023-10-16T20:39:45.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f1387fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f138814bc30 0x7f138814e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f1387fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f138814bc30 0x7f138814e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33356/0 (socket says 172.21.15.88:33356) 2023-10-16T20:39:45.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f1387fff640 1 -- 172.21.15.88:0/1661186631 learned_addr learned my addr 172.21.15.88:0/1661186631 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:45.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f13877fe640 1 --2- 172.21.15.88:0/1661186631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1370003b40 0x7f138814c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f1386ffd640 1 --2- 172.21.15.88:0/1661186631 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f138814a310 0x7f1388144840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f1387fff640 1 -- 172.21.15.88:0/1661186631 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f138814a310 msgr2=0x7f1388144840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f1387fff640 1 --2- 172.21.15.88:0/1661186631 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f138814a310 0x7f1388144840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f1387fff640 1 -- 172.21.15.88:0/1661186631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1370003b40 msgr2=0x7f138814c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f1387fff640 1 --2- 172.21.15.88:0/1661186631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1370003b40 0x7f138814c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f1387fff640 1 -- 172.21.15.88:0/1661186631 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138814c190 con 0x7f138814bc30 2023-10-16T20:39:45.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f1387fff640 1 --2- 172.21.15.88:0/1661186631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f138814bc30 0x7f138814e3f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1388072b60 tx=0x7f13800115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:45.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f136ffff640 1 -- 172.21.15.88:0/1661186631 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138000e070 con 0x7f138814bc30 2023-10-16T20:39:45.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f136ffff640 1 -- 172.21.15.88:0/1661186631 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1380011e10 con 0x7f138814bc30 2023-10-16T20:39:45.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f136ffff640 1 -- 172.21.15.88:0/1661186631 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13800039a0 con 0x7f138814bc30 2023-10-16T20:39:45.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f138df72640 1 -- 172.21.15.88:0/1661186631 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f138814d180 con 0x7f138814bc30 2023-10-16T20:39:45.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f138df72640 1 -- 172.21.15.88:0/1661186631 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f138814d740 con 0x7f138814bc30 2023-10-16T20:39:45.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f136ffff640 1 -- 172.21.15.88:0/1661186631 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1380003b00 con 0x7f138814bc30 2023-10-16T20:39:45.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.745+0000 7f136ffff640 1 --2- 172.21.15.88:0/1661186631 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f13740664a0 0x7f1374068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f136ffff640 1 -- 172.21.15.88:0/1661186631 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f1380084840 con 0x7f138814bc30 2023-10-16T20:39:45.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f13877fe640 1 --2- 172.21.15.88:0/1661186631 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f13740664a0 0x7f1374068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f13877fe640 1 --2- 172.21.15.88:0/1661186631 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f13740664a0 0x7f1374068960 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f13780ab000 tx=0x7f13780a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:45.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1391301000 1 Processor -- start 2023-10-16T20:39:45.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1391301000 1 -- start start 2023-10-16T20:39:45.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1391301000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a27810 0x559ba6a27bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1391301000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a28510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1391301000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a28a50 0x559ba6a32aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1391301000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ba67dbcc0 con 0x559ba6a28130 2023-10-16T20:39:45.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1391301000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ba683e920 con 0x559ba6a27810 2023-10-16T20:39:45.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1391301000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ba683be50 con 0x559ba6a28a50 2023-10-16T20:39:45.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1386ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a28510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1386ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a28510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33362/0 (socket says 172.21.15.88:33362) 2023-10-16T20:39:45.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1386ffd640 1 -- 172.21.15.88:0/1802706551 learned_addr learned my addr 172.21.15.88:0/1802706551 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:45.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f13877fe640 1 --2- 172.21.15.88:0/1802706551 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a27810 0x559ba6a27bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1387fff640 1 --2- 172.21.15.88:0/1802706551 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a28a50 0x559ba6a32aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1386ffd640 1 -- 172.21.15.88:0/1802706551 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a28a50 msgr2=0x559ba6a32aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1386ffd640 1 --2- 172.21.15.88:0/1802706551 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a28a50 0x559ba6a32aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1386ffd640 1 -- 172.21.15.88:0/1802706551 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a27810 msgr2=0x559ba6a27bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1386ffd640 1 --2- 172.21.15.88:0/1802706551 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a27810 0x559ba6a27bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1386ffd640 1 -- 172.21.15.88:0/1802706551 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ba6530890 con 0x559ba6a28130 2023-10-16T20:39:45.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1387fff640 1 --2- 172.21.15.88:0/1802706551 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a28a50 0x559ba6a32aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:45.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1386ffd640 1 --2- 172.21.15.88:0/1802706551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a28510 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f137c001720 tx=0x7f137c001be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f70f1e2426d2477 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:45.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f136d7fa640 1 -- 172.21.15.88:0/1802706551 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c01a070 con 0x559ba6a28130 2023-10-16T20:39:45.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f136d7fa640 1 -- 172.21.15.88:0/1802706551 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f137c005320 con 0x559ba6a28130 2023-10-16T20:39:45.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f136d7fa640 1 -- 172.21.15.88:0/1802706551 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c001400 con 0x559ba6a28130 2023-10-16T20:39:45.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1391301000 1 -- 172.21.15.88:0/1802706551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 msgr2=0x559ba6a28510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.749+0000 7f1391301000 1 --2- 172.21.15.88:0/1802706551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a28510 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f137c001720 tx=0x7f137c001be0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 -- 172.21.15.88:0/1802706551 shutdown_connections 2023-10-16T20:39:45.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 --2- 172.21.15.88:0/1802706551 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a28a50 0x559ba6a32aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 --2- 172.21.15.88:0/1802706551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a28510 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 --2- 172.21.15.88:0/1802706551 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a27810 0x559ba6a27bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 -- 172.21.15.88:0/1802706551 >> 172.21.15.88:0/1802706551 conn(0x559ba69ddfc0 msgr2=0x559ba6a255d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:45.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 -- 172.21.15.88:0/1802706551 shutdown_connections 2023-10-16T20:39:45.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 -- 172.21.15.88:0/1802706551 wait complete. 2023-10-16T20:39:45.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 -- start start 2023-10-16T20:39:45.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 Processor -- start 2023-10-16T20:39:45.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a27810 0x559ba6a21b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a220a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a28a50 0x559ba6a225e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ba6837740 con 0x559ba6a28130 2023-10-16T20:39:45.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ba67f6d10 con 0x559ba6a27810 2023-10-16T20:39:45.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ba67f69b0 con 0x559ba6a28a50 2023-10-16T20:39:45.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 10 client.0 _ll_register_callbacks cb 0x559ba69d2390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T20:39:45.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1386ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1386ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a220a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33378/0 (socket says 172.21.15.88:33378) 2023-10-16T20:39:45.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1386ffd640 1 -- 172.21.15.88:0/52905703 learned_addr learned my addr 172.21.15.88:0/52905703 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:45.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f13877fe640 1 --2- 172.21.15.88:0/52905703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a27810 0x559ba6a21b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1387fff640 1 --2- 172.21.15.88:0/52905703 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a28a50 0x559ba6a225e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1386ffd640 1 -- 172.21.15.88:0/52905703 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a28a50 msgr2=0x559ba6a225e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1386ffd640 1 --2- 172.21.15.88:0/52905703 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a28a50 0x559ba6a225e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1386ffd640 1 -- 172.21.15.88:0/52905703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a27810 msgr2=0x559ba6a21b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1386ffd640 1 --2- 172.21.15.88:0/52905703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a27810 0x559ba6a21b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1386ffd640 1 -- 172.21.15.88:0/52905703 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137c00f040 con 0x559ba6a28130 2023-10-16T20:39:45.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1386ffd640 1 --2- 172.21.15.88:0/52905703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a220a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f137c009010 tx=0x7f137c0012e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f19b175bb3cc5ce9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:45.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f13667fc640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-10-16T20:39:45.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f13667fc640 1 -- 172.21.15.88:0/52905703 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c01a070 con 0x559ba6a28130 2023-10-16T20:39:45.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f13667fc640 1 -- 172.21.15.88:0/52905703 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f137c011df0 con 0x559ba6a28130 2023-10-16T20:39:45.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 -- 172.21.15.88:0/52905703 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559ba67eb370 con 0x559ba6a28130 2023-10-16T20:39:45.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f13667fc640 1 -- 172.21.15.88:0/52905703 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f137c0067e0 con 0x559ba6a28130 2023-10-16T20:39:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 10 client.4908 fetch_fsmap learned FSMap version 19 2023-10-16T20:39:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 -- 172.21.15.88:0/52905703 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x559ba67f9460 con 0x559ba6a28130 2023-10-16T20:39:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f13667fc640 1 -- 172.21.15.88:0/52905703 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559ba67eb370 con 0x559ba6a28130 2023-10-16T20:39:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f13667fc640 1 -- 172.21.15.88:0/52905703 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) v1 ==== 4500+0+0 (secure 0 0 0) 0x7f137c006940 con 0x559ba6a28130 2023-10-16T20:39:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 10 client.4908 fetch_fsmap finished waiting for FSMap version 19 2023-10-16T20:39:45.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 10 client.4908 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T20:39:45.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 20 client.4908 populate_metadata read hostname 'smithi088' 2023-10-16T20:39:45.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 --2- 172.21.15.88:0/52905703 >> [v2:172.21.15.88:6834/518281867,v1:172.21.15.88:6835/518281867] conn(0x559ba6a3b260 0x559ba6a3d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 4 client.4908 mds_command: new command op to 4263 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T20:39:45.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.753+0000 7f1391301000 1 -- 172.21.15.88:0/52905703 --> [v2:172.21.15.88:6834/518281867,v1:172.21.15.88:6835/518281867] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559ba67d12d0 con 0x559ba6a3b260 2023-10-16T20:39:45.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f13877fe640 1 --2- 172.21.15.88:0/52905703 >> [v2:172.21.15.88:6834/518281867,v1:172.21.15.88:6835/518281867] conn(0x559ba6a3b260 0x559ba6a3d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f13877fe640 1 --2- 172.21.15.88:0/52905703 >> [v2:172.21.15.88:6834/518281867,v1:172.21.15.88:6835/518281867] conn(0x559ba6a3b260 0x559ba6a3d650 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3de015c40dea9b12 server_cookie=93abfa6930dc805e in_seq=0 out_seq=0 2023-10-16T20:39:45.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f13667fc640 10 client.4908 ms_handle_connect on v2:172.21.15.88:6834/518281867 2023-10-16T20:39:45.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f13667fc640 1 -- 172.21.15.88:0/52905703 <== mds.3 v2:172.21.15.88:6834/518281867 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559ba67d12d0 con 0x559ba6a3b260 2023-10-16T20:39:45.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f13667fc640 10 client.4908 handle_command_reply: tid=0 2023-10-16T20:39:45.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f1391301000 1 client.4908 shutdown 2023-10-16T20:39:45.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f1391301000 1 -- 172.21.15.88:0/52905703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 msgr2=0x559ba6a220a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f1391301000 1 --2- 172.21.15.88:0/52905703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a220a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f137c009010 tx=0x7f137c0012e0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f1391301000 1 -- 172.21.15.88:0/52905703 shutdown_connections 2023-10-16T20:39:45.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f1391301000 1 --2- 172.21.15.88:0/52905703 >> [v2:172.21.15.88:6834/518281867,v1:172.21.15.88:6835/518281867] conn(0x559ba6a3b260 0x559ba6a3d650 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f1391301000 1 --2- 172.21.15.88:0/52905703 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a28a50 0x559ba6a225e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f1391301000 1 --2- 172.21.15.88:0/52905703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a220a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f1391301000 1 --2- 172.21.15.88:0/52905703 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a27810 0x559ba6a21b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f1391301000 1 -- 172.21.15.88:0/52905703 >> 172.21.15.88:0/52905703 conn(0x559ba69ddfc0 msgr2=0x559ba6a27090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:45.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f13667fc640 0 client.4908 ms_handle_reset on v2:172.21.15.88:6834/518281867 2023-10-16T20:39:45.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f1391301000 1 -- 172.21.15.88:0/52905703 shutdown_connections 2023-10-16T20:39:45.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f1391301000 1 -- 172.21.15.88:0/52905703 wait complete. 2023-10-16T20:39:45.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.757+0000 7f1391301000 20 client.4908 trim_cache size 0 max 16384 2023-10-16T20:39:45.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1391301000 1 Processor -- start 2023-10-16T20:39:45.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1391301000 1 -- start start 2023-10-16T20:39:45.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1391301000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a27810 0x559ba6a86e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1391301000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a87340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1391301000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a28a50 0x559ba6a87880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1391301000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ba64f1940 con 0x559ba6a28130 2023-10-16T20:39:45.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1391301000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ba6831a40 con 0x559ba6a28a50 2023-10-16T20:39:45.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1391301000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ba6834f50 con 0x559ba6a27810 2023-10-16T20:39:45.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1386ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a87340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1386ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a87340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33380/0 (socket says 172.21.15.88:33380) 2023-10-16T20:39:45.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f13877fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a27810 0x559ba6a86e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1386ffd640 1 -- 172.21.15.88:0/4162683492 learned_addr learned my addr 172.21.15.88:0/4162683492 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:45.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1387fff640 1 --2- 172.21.15.88:0/4162683492 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a28a50 0x559ba6a87880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1386ffd640 1 -- 172.21.15.88:0/4162683492 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a27810 msgr2=0x559ba6a86e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1386ffd640 1 --2- 172.21.15.88:0/4162683492 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a27810 0x559ba6a86e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1386ffd640 1 -- 172.21.15.88:0/4162683492 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a28a50 msgr2=0x559ba6a87880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1386ffd640 1 --2- 172.21.15.88:0/4162683492 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a28a50 0x559ba6a87880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1386ffd640 1 -- 172.21.15.88:0/4162683492 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ba6a39f50 con 0x559ba6a28130 2023-10-16T20:39:45.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1386ffd640 1 --2- 172.21.15.88:0/4162683492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a87340 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f137c001210 tx=0x7f137c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303fbb1f0f2da8c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:45.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f136dffb640 1 -- 172.21.15.88:0/4162683492 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c011df0 con 0x559ba6a28130 2023-10-16T20:39:45.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f136dffb640 1 -- 172.21.15.88:0/4162683492 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f137c01b040 con 0x559ba6a28130 2023-10-16T20:39:45.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f136dffb640 1 -- 172.21.15.88:0/4162683492 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c006a20 con 0x559ba6a28130 2023-10-16T20:39:45.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1391301000 1 -- 172.21.15.88:0/4162683492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 msgr2=0x559ba6a87340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1391301000 1 --2- 172.21.15.88:0/4162683492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a87340 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f137c001210 tx=0x7f137c020000 comp rx=0 tx=0).stop 2023-10-16T20:39:45.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1391301000 1 -- 172.21.15.88:0/4162683492 shutdown_connections 2023-10-16T20:39:45.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1391301000 1 --2- 172.21.15.88:0/4162683492 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a28a50 0x559ba6a87880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1391301000 1 --2- 172.21.15.88:0/4162683492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28130 0x559ba6a87340 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1391301000 1 --2- 172.21.15.88:0/4162683492 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a27810 0x559ba6a86e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.773+0000 7f1391301000 1 -- 172.21.15.88:0/4162683492 >> 172.21.15.88:0/4162683492 conn(0x559ba69ddfc0 msgr2=0x559ba6a2d080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:45.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 1 -- 172.21.15.88:0/4162683492 shutdown_connections 2023-10-16T20:39:45.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 1 -- 172.21.15.88:0/4162683492 wait complete. 2023-10-16T20:39:45.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 1 -- start start 2023-10-16T20:39:45.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 1 Processor -- start 2023-10-16T20:39:45.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a27810 0x559ba6a84db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a28130 0x559ba6a852f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28a50 0x559ba6a85a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ba68330b0 con 0x559ba6a28a50 2023-10-16T20:39:45.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ba6878ba0 con 0x559ba6a28130 2023-10-16T20:39:45.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ba68365d0 con 0x559ba6a27810 2023-10-16T20:39:45.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 10 client.0 _ll_register_callbacks cb 0x559ba6837140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T20:39:45.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1387fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28a50 0x559ba6a85a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1387fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28a50 0x559ba6a85a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33388/0 (socket says 172.21.15.88:33388) 2023-10-16T20:39:45.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1387fff640 1 -- 172.21.15.88:0/1792131353 learned_addr learned my addr 172.21.15.88:0/1792131353 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:45.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f13877fe640 1 --2- 172.21.15.88:0/1792131353 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a27810 0x559ba6a84db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1386ffd640 1 --2- 172.21.15.88:0/1792131353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a28130 0x559ba6a852f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1387fff640 1 -- 172.21.15.88:0/1792131353 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a27810 msgr2=0x559ba6a84db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1387fff640 1 --2- 172.21.15.88:0/1792131353 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a27810 0x559ba6a84db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1387fff640 1 -- 172.21.15.88:0/1792131353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a28130 msgr2=0x559ba6a852f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1387fff640 1 --2- 172.21.15.88:0/1792131353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a28130 0x559ba6a852f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1387fff640 1 -- 172.21.15.88:0/1792131353 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559ba67f9460 con 0x559ba6a28a50 2023-10-16T20:39:45.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1386ffd640 1 --2- 172.21.15.88:0/1792131353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a28130 0x559ba6a852f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:45.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f13877fe640 1 --2- 172.21.15.88:0/1792131353 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a27810 0x559ba6a84db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:45.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1387fff640 1 --2- 172.21.15.88:0/1792131353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28a50 0x559ba6a85a50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f13800552d0 tx=0x7f138005d820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30739ac4299dbda7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:45.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f13667fc640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-10-16T20:39:45.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f13667fc640 1 -- 172.21.15.88:0/1792131353 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1380053a30 con 0x559ba6a28a50 2023-10-16T20:39:45.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f13667fc640 1 -- 172.21.15.88:0/1792131353 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f138005d390 con 0x559ba6a28a50 2023-10-16T20:39:45.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 1 -- 172.21.15.88:0/1792131353 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559ba6a39950 con 0x559ba6a28a50 2023-10-16T20:39:45.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f13667fc640 1 -- 172.21.15.88:0/1792131353 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f138005d550 con 0x559ba6a28a50 2023-10-16T20:39:45.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 10 client.4914 fetch_fsmap learned FSMap version 19 2023-10-16T20:39:45.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 1 -- 172.21.15.88:0/1792131353 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x559ba67df760 con 0x559ba6a28a50 2023-10-16T20:39:45.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f13667fc640 1 -- 172.21.15.88:0/1792131353 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1380078020 con 0x559ba6a28a50 2023-10-16T20:39:45.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f13667fc640 1 -- 172.21.15.88:0/1792131353 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) v1 ==== 4500+0+0 (secure 0 0 0) 0x7f1380005d20 con 0x559ba6a28a50 2023-10-16T20:39:45.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 10 client.4914 fetch_fsmap finished waiting for FSMap version 19 2023-10-16T20:39:45.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 10 client.4914 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-16T20:39:45.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 20 client.4914 populate_metadata read hostname 'smithi088' 2023-10-16T20:39:45.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 1 --2- 172.21.15.88:0/1792131353 >> [v2:172.21.15.88:6834/518281867,v1:172.21.15.88:6835/518281867] conn(0x559ba6a91f00 0x559ba6a868b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 4 client.4914 mds_command: new command op to 4263 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T20:39:45.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f1391301000 1 -- 172.21.15.88:0/1792131353 --> [v2:172.21.15.88:6834/518281867,v1:172.21.15.88:6835/518281867] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559ba67cf6f0 con 0x559ba6a91f00 2023-10-16T20:39:45.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.777+0000 7f13877fe640 1 --2- 172.21.15.88:0/1792131353 >> [v2:172.21.15.88:6834/518281867,v1:172.21.15.88:6835/518281867] conn(0x559ba6a91f00 0x559ba6a868b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f13877fe640 1 --2- 172.21.15.88:0/1792131353 >> [v2:172.21.15.88:6834/518281867,v1:172.21.15.88:6835/518281867] conn(0x559ba6a91f00 0x559ba6a868b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=93c8864dd4b31fae server_cookie=dc2c8f0d7218b4bc in_seq=0 out_seq=0 2023-10-16T20:39:45.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f13667fc640 10 client.4914 ms_handle_connect on v2:172.21.15.88:6834/518281867 2023-10-16T20:39:45.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f13667fc640 1 -- 172.21.15.88:0/1792131353 <== mds.3 v2:172.21.15.88:6834/518281867 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x559ba67cf6f0 con 0x559ba6a91f00 2023-10-16T20:39:45.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f13667fc640 10 client.4914 handle_command_reply: tid=0 2023-10-16T20:39:45.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f1391301000 1 client.4914 shutdown 2023-10-16T20:39:45.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f1391301000 1 -- 172.21.15.88:0/1792131353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28a50 msgr2=0x559ba6a85a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f1391301000 1 --2- 172.21.15.88:0/1792131353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28a50 0x559ba6a85a50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f13800552d0 tx=0x7f138005d820 comp rx=0 tx=0).stop 2023-10-16T20:39:45.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f1391301000 1 -- 172.21.15.88:0/1792131353 shutdown_connections 2023-10-16T20:39:45.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f1391301000 1 --2- 172.21.15.88:0/1792131353 >> [v2:172.21.15.88:6834/518281867,v1:172.21.15.88:6835/518281867] conn(0x559ba6a91f00 0x559ba6a868b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f1391301000 1 --2- 172.21.15.88:0/1792131353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ba6a28a50 0x559ba6a85a50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f1391301000 1 --2- 172.21.15.88:0/1792131353 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x559ba6a28130 0x559ba6a852f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f1391301000 1 --2- 172.21.15.88:0/1792131353 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x559ba6a27810 0x559ba6a84db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f13667fc640 0 client.4914 ms_handle_reset on v2:172.21.15.88:6834/518281867 2023-10-16T20:39:45.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f1391301000 1 -- 172.21.15.88:0/1792131353 >> 172.21.15.88:0/1792131353 conn(0x559ba69ddfc0 msgr2=0x559ba6a2ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:45.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f1391301000 1 -- 172.21.15.88:0/1792131353 shutdown_connections 2023-10-16T20:39:45.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f1391301000 1 -- 172.21.15.88:0/1792131353 wait complete. 2023-10-16T20:39:45.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f1391301000 20 client.4914 trim_cache size 0 max 16384 2023-10-16T20:39:45.801 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-10-16T20:39:45.802 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "8cf7342f-a394-4b4c-ac35-9931ac9d9e57", 2023-10-16T20:39:45.802 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 3, 2023-10-16T20:39:45.802 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 4263, 2023-10-16T20:39:45.802 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2023-10-16T20:39:45.803 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2023-10-16T20:39:45.803 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2023-10-16T20:39:45.803 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 17.289591726000001, 2023-10-16T20:39:45.803 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 19, 2023-10-16T20:39:45.803 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 29, 2023-10-16T20:39:45.804 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 29, 2023-10-16T20:39:45.804 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 176.88570469999999 2023-10-16T20:39:45.804 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-10-16T20:39:45.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f138df72640 1 -- 172.21.15.88:0/1661186631 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f13740664a0 msgr2=0x7f1374068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f138df72640 1 --2- 172.21.15.88:0/1661186631 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f13740664a0 0x7f1374068960 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f13780ab000 tx=0x7f13780a7070 comp rx=0 tx=0).stop 2023-10-16T20:39:45.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f138df72640 1 -- 172.21.15.88:0/1661186631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f138814bc30 msgr2=0x7f138814e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f138df72640 1 --2- 172.21.15.88:0/1661186631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f138814bc30 0x7f138814e3f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1388072b60 tx=0x7f13800115a0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f138df72640 1 -- 172.21.15.88:0/1661186631 shutdown_connections 2023-10-16T20:39:45.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f138df72640 1 --2- 172.21.15.88:0/1661186631 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f13740664a0 0x7f1374068960 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f138df72640 1 --2- 172.21.15.88:0/1661186631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f138814bc30 0x7f138814e3f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f138df72640 1 --2- 172.21.15.88:0/1661186631 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f138814a310 0x7f1388144840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f138df72640 1 --2- 172.21.15.88:0/1661186631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1370003b40 0x7f138814c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f138df72640 1 -- 172.21.15.88:0/1661186631 >> 172.21.15.88:0/1661186631 conn(0x7f13880c86b0 msgr2=0x7f138814b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:45.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f138df72640 1 -- 172.21.15.88:0/1661186631 shutdown_connections 2023-10-16T20:39:45.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.781+0000 7f138df72640 1 -- 172.21.15.88:0/1661186631 wait complete. 2023-10-16T20:39:45.808 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-10-16T20:39:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.941+0000 7f04b3d94640 1 Processor -- start 2023-10-16T20:39:45.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.945+0000 7f04b3d94640 1 -- start start 2023-10-16T20:39:45.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.945+0000 7f04b3d94640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f04ac0cdb20 0x7f04ac0ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.945+0000 7f04b3d94640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f04ac06dd70 0x7f04ac0cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.945+0000 7f04b3d94640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ac06cbe0 con 0x7f04ac1537d0 2023-10-16T20:39:45.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.945+0000 7f04b3d94640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ac06f5f0 con 0x7f04ac0cdb20 2023-10-16T20:39:45.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.945+0000 7f04b3d94640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ac15d8b0 con 0x7f04ac06dd70 2023-10-16T20:39:45.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.945+0000 7f04b132f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f04ac0cdb20 0x7f04ac0ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.945+0000 7f04b132f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f04ac0cdb20 0x7f04ac0ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33396/0 (socket says 172.21.15.88:33396) 2023-10-16T20:39:45.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.945+0000 7f04b132f640 1 -- 172.21.15.88:0/506104268 learned_addr learned my addr 172.21.15.88:0/506104268 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:45.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.945+0000 7f04b2331640 1 --2- 172.21.15.88:0/506104268 >> v2:172.21.15.191:3300/0 conn(0x7f04ac06dd70 0x7f04ac0cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.945+0000 7f04b132f640 1 -- 172.21.15.88:0/506104268 >> v2:172.21.15.191:3300/0 conn(0x7f04ac06dd70 msgr2=0x7f04ac0cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.945+0000 7f04b132f640 1 --2- 172.21.15.88:0/506104268 >> v2:172.21.15.191:3300/0 conn(0x7f04ac06dd70 0x7f04ac0cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.945+0000 7f04b132f640 1 -- 172.21.15.88:0/506104268 >> v1:172.21.15.191:6789/0 conn(0x7f04ac1537d0 legacy=0x7f04ac15aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.945+0000 7f04b132f640 1 -- 172.21.15.88:0/506104268 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac06e830 con 0x7f04ac0cdb20 2023-10-16T20:39:45.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.945+0000 7f04b2331640 1 --2- 172.21.15.88:0/506104268 >> v2:172.21.15.191:3300/0 conn(0x7f04ac06dd70 0x7f04ac0cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:45.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.949+0000 7f04b132f640 1 --2- 172.21.15.88:0/506104268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04ac0cdb20 0x7f04ac0ccdb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0b1ae0 tx=0x7f049c01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aefbc5b96af7d4cd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:45.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.949+0000 7f04b0b2e640 1 -- 172.21.15.88:0/506104268 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049c0ac410 con 0x7f04ac0cdb20 2023-10-16T20:39:45.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.949+0000 7f04b0b2e640 1 -- 172.21.15.88:0/506104268 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049c0ac570 con 0x7f04ac0cdb20 2023-10-16T20:39:45.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.949+0000 7f04b0b2e640 1 -- 172.21.15.88:0/506104268 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049c0ac900 con 0x7f04ac0cdb20 2023-10-16T20:39:45.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.949+0000 7f04b3d94640 1 -- 172.21.15.88:0/506104268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04ac0cdb20 msgr2=0x7f04ac0ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.949+0000 7f04b3d94640 1 --2- 172.21.15.88:0/506104268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04ac0cdb20 0x7f04ac0ccdb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0b1ae0 tx=0x7f049c01c4c0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.949+0000 7f04b3d94640 1 -- 172.21.15.88:0/506104268 shutdown_connections 2023-10-16T20:39:45.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.949+0000 7f04b3d94640 1 --2- 172.21.15.88:0/506104268 >> v2:172.21.15.191:3300/0 conn(0x7f04ac06dd70 0x7f04ac0cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.949+0000 7f04b3d94640 1 --2- 172.21.15.88:0/506104268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04ac0cdb20 0x7f04ac0ccdb0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.949+0000 7f04b3d94640 1 -- 172.21.15.88:0/506104268 >> 172.21.15.88:0/506104268 conn(0x7f04ac06f7d0 msgr2=0x7f04ac06fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:45.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.949+0000 7f04b3d94640 1 -- 172.21.15.88:0/506104268 shutdown_connections 2023-10-16T20:39:45.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.949+0000 7f04b3d94640 1 -- 172.21.15.88:0/506104268 wait complete. 2023-10-16T20:39:45.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.949+0000 7f04b3d94640 1 Processor -- start 2023-10-16T20:39:45.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.949+0000 7f04b3d94640 1 -- start start 2023-10-16T20:39:45.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b3d94640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04ac06dd70 0x7f04ac141060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b3d94640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04ac0cdb20 0x7f04ac1f43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b3d94640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04ac1537d0 0x7f04ac1f4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b3d94640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac1f5db0 con 0x7f04ac06dd70 2023-10-16T20:39:45.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b3d94640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac13b630 con 0x7f04ac1537d0 2023-10-16T20:39:45.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b3d94640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac1f3f30 con 0x7f04ac0cdb20 2023-10-16T20:39:45.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b1b30640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04ac06dd70 0x7f04ac141060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b1b30640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04ac06dd70 0x7f04ac141060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33412/0 (socket says 172.21.15.88:33412) 2023-10-16T20:39:45.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b1b30640 1 -- 172.21.15.88:0/1372958545 learned_addr learned my addr 172.21.15.88:0/1372958545 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:45.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b132f640 1 --2- 172.21.15.88:0/1372958545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04ac0cdb20 0x7f04ac1f43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b2331640 1 --2- 172.21.15.88:0/1372958545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04ac1537d0 0x7f04ac1f4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b1b30640 1 -- 172.21.15.88:0/1372958545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04ac0cdb20 msgr2=0x7f04ac1f43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b1b30640 1 --2- 172.21.15.88:0/1372958545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04ac0cdb20 0x7f04ac1f43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b1b30640 1 -- 172.21.15.88:0/1372958545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04ac1537d0 msgr2=0x7f04ac1f4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b1b30640 1 --2- 172.21.15.88:0/1372958545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04ac1537d0 0x7f04ac1f4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b1b30640 1 -- 172.21.15.88:0/1372958545 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c0094d0 con 0x7f04ac06dd70 2023-10-16T20:39:45.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b2331640 1 --2- 172.21.15.88:0/1372958545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04ac1537d0 0x7f04ac1f4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:45.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b1b30640 1 --2- 172.21.15.88:0/1372958545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04ac06dd70 0x7f04ac141060 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f04a800b440 tx=0x7f04a800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:45.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04a27fc640 1 -- 172.21.15.88:0/1372958545 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8011890 con 0x7f04ac06dd70 2023-10-16T20:39:45.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04a27fc640 1 -- 172.21.15.88:0/1372958545 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a80119f0 con 0x7f04ac06dd70 2023-10-16T20:39:45.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b3d94640 1 -- 172.21.15.88:0/1372958545 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ac0c9310 con 0x7f04ac06dd70 2023-10-16T20:39:45.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04a27fc640 1 -- 172.21.15.88:0/1372958545 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8011da0 con 0x7f04ac06dd70 2023-10-16T20:39:45.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04b3d94640 1 -- 172.21.15.88:0/1372958545 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ac1f5ad0 con 0x7f04ac06dd70 2023-10-16T20:39:45.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.953+0000 7f04a27fc640 1 -- 172.21.15.88:0/1372958545 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f04a801a400 con 0x7f04ac06dd70 2023-10-16T20:39:45.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04a27fc640 1 --2- 172.21.15.88:0/1372958545 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f04900664d0 0x7f0490068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04a27fc640 1 -- 172.21.15.88:0/1372958545 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f04a8084230 con 0x7f04ac06dd70 2023-10-16T20:39:45.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b132f640 1 --2- 172.21.15.88:0/1372958545 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f04900664d0 0x7f0490068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b711a000 1 Processor -- start 2023-10-16T20:39:45.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b711a000 1 -- start start 2023-10-16T20:39:45.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b132f640 1 --2- 172.21.15.88:0/1372958545 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f04900664d0 0x7f0490068990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f049c0ae000 tx=0x7f049c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:45.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b711a000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c0810 0x55ebe93c0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b711a000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c1130 0x55ebe93c1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b711a000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c1a50 0x55ebe93cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b711a000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebe9174cc0 con 0x55ebe93c0810 2023-10-16T20:39:45.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b711a000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebe91d7920 con 0x55ebe93c1130 2023-10-16T20:39:45.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b711a000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ebe91d4e50 con 0x55ebe93c1a50 2023-10-16T20:39:45.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b1b30640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c0810 0x55ebe93c0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b1b30640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c0810 0x55ebe93c0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33418/0 (socket says 172.21.15.88:33418) 2023-10-16T20:39:45.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b1b30640 1 -- 172.21.15.88:0/1927383102 learned_addr learned my addr 172.21.15.88:0/1927383102 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:45.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b132f640 1 --2- 172.21.15.88:0/1927383102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c1130 0x55ebe93c1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b2331640 1 --2- 172.21.15.88:0/1927383102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c1a50 0x55ebe93cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b1b30640 1 -- 172.21.15.88:0/1927383102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c1a50 msgr2=0x55ebe93cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b1b30640 1 --2- 172.21.15.88:0/1927383102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c1a50 0x55ebe93cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b1b30640 1 -- 172.21.15.88:0/1927383102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c1130 msgr2=0x55ebe93c1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b1b30640 1 --2- 172.21.15.88:0/1927383102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c1130 0x55ebe93c1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b1b30640 1 -- 172.21.15.88:0/1927383102 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebe8ec9890 con 0x55ebe93c0810 2023-10-16T20:39:45.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b132f640 1 --2- 172.21.15.88:0/1927383102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c1130 0x55ebe93c1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:45.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f04b1b30640 1 --2- 172.21.15.88:0/1927383102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c0810 0x55ebe93c0bf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f04a8000c10 tx=0x7f04a80556d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e406a5af5154dd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:45.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f048f7fe640 1 -- 172.21.15.88:0/1927383102 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8059060 con 0x55ebe93c0810 2023-10-16T20:39:45.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f048f7fe640 1 -- 172.21.15.88:0/1927383102 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a805d210 con 0x55ebe93c0810 2023-10-16T20:39:45.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.957+0000 7f048f7fe640 1 -- 172.21.15.88:0/1927383102 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a80532b0 con 0x55ebe93c0810 2023-10-16T20:39:45.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 1 -- 172.21.15.88:0/1927383102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c0810 msgr2=0x55ebe93c0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 1 --2- 172.21.15.88:0/1927383102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c0810 0x55ebe93c0bf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f04a8000c10 tx=0x7f04a80556d0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 1 -- 172.21.15.88:0/1927383102 shutdown_connections 2023-10-16T20:39:45.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 1 --2- 172.21.15.88:0/1927383102 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c1a50 0x55ebe93cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 1 --2- 172.21.15.88:0/1927383102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c1130 0x55ebe93c1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 1 --2- 172.21.15.88:0/1927383102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c0810 0x55ebe93c0bf0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 1 -- 172.21.15.88:0/1927383102 >> 172.21.15.88:0/1927383102 conn(0x55ebe9376fc0 msgr2=0x55ebe93be5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:45.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 1 -- 172.21.15.88:0/1927383102 shutdown_connections 2023-10-16T20:39:45.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 1 -- 172.21.15.88:0/1927383102 wait complete. 2023-10-16T20:39:45.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 1 -- start start 2023-10-16T20:39:45.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 1 Processor -- start 2023-10-16T20:39:45.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c0810 0x55ebe93baf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c1130 0x55ebe93bb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c1a50 0x55ebe93c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebe91d0740 con 0x55ebe93c1130 2023-10-16T20:39:45.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebe918fd10 con 0x55ebe93c0810 2023-10-16T20:39:45.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ebe918f9b0 con 0x55ebe93c1a50 2023-10-16T20:39:45.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b711a000 10 client.0 _ll_register_callbacks cb 0x55ebe936b390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T20:39:45.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b132f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c1130 0x55ebe93bb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b132f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c1130 0x55ebe93bb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33434/0 (socket says 172.21.15.88:33434) 2023-10-16T20:39:45.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b132f640 1 -- 172.21.15.88:0/2580197486 learned_addr learned my addr 172.21.15.88:0/2580197486 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:45.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b1b30640 1 --2- 172.21.15.88:0/2580197486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c0810 0x55ebe93baf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b2331640 1 --2- 172.21.15.88:0/2580197486 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c1a50 0x55ebe93c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b132f640 1 -- 172.21.15.88:0/2580197486 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c1a50 msgr2=0x55ebe93c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b132f640 1 --2- 172.21.15.88:0/2580197486 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c1a50 0x55ebe93c6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b132f640 1 -- 172.21.15.88:0/2580197486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c0810 msgr2=0x55ebe93baf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b132f640 1 --2- 172.21.15.88:0/2580197486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c0810 0x55ebe93baf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b132f640 1 -- 172.21.15.88:0/2580197486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ac1f5ad0 con 0x55ebe93c1130 2023-10-16T20:39:45.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b1b30640 1 --2- 172.21.15.88:0/2580197486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c0810 0x55ebe93baf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:39:45.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f04b132f640 1 --2- 172.21.15.88:0/2580197486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c1130 0x55ebe93bb490 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f049c01c9a0 tx=0x7f049c01ad00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d6b2a7f97def57 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:45.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.961+0000 7f048cff9640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-10-16T20:39:45.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f048cff9640 1 -- 172.21.15.88:0/2580197486 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049c0bc050 con 0x55ebe93c1130 2023-10-16T20:39:45.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f048cff9640 1 -- 172.21.15.88:0/2580197486 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f049c0acd40 con 0x55ebe93c1130 2023-10-16T20:39:45.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 1 -- 172.21.15.88:0/2580197486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ebe9184370 con 0x55ebe93c1130 2023-10-16T20:39:45.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f048cff9640 1 -- 172.21.15.88:0/2580197486 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f049c0c3410 con 0x55ebe93c1130 2023-10-16T20:39:45.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 10 client.4926 fetch_fsmap learned FSMap version 19 2023-10-16T20:39:45.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 1 -- 172.21.15.88:0/2580197486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x55ebe9192460 con 0x55ebe93c1130 2023-10-16T20:39:45.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f048cff9640 1 -- 172.21.15.88:0/2580197486 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ebe9184370 con 0x55ebe93c1130 2023-10-16T20:39:45.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f048cff9640 1 -- 172.21.15.88:0/2580197486 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) v1 ==== 4500+0+0 (secure 0 0 0) 0x7f049c0cb350 con 0x55ebe93c1130 2023-10-16T20:39:45.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 10 client.4926 fetch_fsmap finished waiting for FSMap version 19 2023-10-16T20:39:45.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 10 client.4926 resolve_mds: resolved name 'e' to daemon mds.e 2023-10-16T20:39:45.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 20 client.4926 populate_metadata read hostname 'smithi088' 2023-10-16T20:39:45.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 1 --2- 172.21.15.88:0/2580197486 >> [v2:172.21.15.88:6836/321173984,v1:172.21.15.88:6838/321173984] conn(0x55ebe93c7970 0x55ebe93c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 4 client.4926 mds_command: new command op to 4275 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T20:39:45.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 1 -- 172.21.15.88:0/2580197486 --> [v2:172.21.15.88:6836/321173984,v1:172.21.15.88:6838/321173984] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ebe916a2d0 con 0x55ebe93c7970 2023-10-16T20:39:45.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b1b30640 1 --2- 172.21.15.88:0/2580197486 >> [v2:172.21.15.88:6836/321173984,v1:172.21.15.88:6838/321173984] conn(0x55ebe93c7970 0x55ebe93c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b1b30640 1 --2- 172.21.15.88:0/2580197486 >> [v2:172.21.15.88:6836/321173984,v1:172.21.15.88:6838/321173984] conn(0x55ebe93c7970 0x55ebe93c7d50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=45a5022991cf2a88 server_cookie=f6ba29cb9537068a in_seq=0 out_seq=0 2023-10-16T20:39:45.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f048cff9640 10 client.4926 ms_handle_connect on v2:172.21.15.88:6836/321173984 2023-10-16T20:39:45.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f048cff9640 1 -- 172.21.15.88:0/2580197486 <== mds.2 v2:172.21.15.88:6836/321173984 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ebe916a2d0 con 0x55ebe93c7970 2023-10-16T20:39:45.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f048cff9640 10 client.4926 handle_command_reply: tid=0 2023-10-16T20:39:45.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 1 client.4926 shutdown 2023-10-16T20:39:45.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 1 -- 172.21.15.88:0/2580197486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c1130 msgr2=0x55ebe93bb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 1 --2- 172.21.15.88:0/2580197486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c1130 0x55ebe93bb490 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f049c01c9a0 tx=0x7f049c01ad00 comp rx=0 tx=0).stop 2023-10-16T20:39:45.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 1 -- 172.21.15.88:0/2580197486 shutdown_connections 2023-10-16T20:39:45.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 1 --2- 172.21.15.88:0/2580197486 >> [v2:172.21.15.88:6836/321173984,v1:172.21.15.88:6838/321173984] conn(0x55ebe93c7970 0x55ebe93c7d50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 1 --2- 172.21.15.88:0/2580197486 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c1a50 0x55ebe93c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 1 --2- 172.21.15.88:0/2580197486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c1130 0x55ebe93bb490 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f048cff9640 0 client.4926 ms_handle_reset on v2:172.21.15.88:6836/321173984 2023-10-16T20:39:45.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 1 --2- 172.21.15.88:0/2580197486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c0810 0x55ebe93baf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 1 -- 172.21.15.88:0/2580197486 >> 172.21.15.88:0/2580197486 conn(0x55ebe9376fc0 msgr2=0x55ebe93c0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:45.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 1 -- 172.21.15.88:0/2580197486 shutdown_connections 2023-10-16T20:39:45.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 1 -- 172.21.15.88:0/2580197486 wait complete. 2023-10-16T20:39:45.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.965+0000 7f04b711a000 20 client.4926 trim_cache size 0 max 16384 2023-10-16T20:39:45.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b711a000 1 Processor -- start 2023-10-16T20:39:45.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b711a000 1 -- start start 2023-10-16T20:39:45.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b711a000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c0810 0x55ebe93c4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b711a000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c1130 0x55ebe93c4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b711a000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c1a50 0x55ebe93c4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b711a000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebe93c7030 con 0x55ebe93c1a50 2023-10-16T20:39:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b711a000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebe8e8a940 con 0x55ebe93c1130 2023-10-16T20:39:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b711a000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ebe9211ba0 con 0x55ebe93c0810 2023-10-16T20:39:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b2331640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c1a50 0x55ebe93c4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b2331640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c1a50 0x55ebe93c4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33446/0 (socket says 172.21.15.88:33446) 2023-10-16T20:39:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b2331640 1 -- 172.21.15.88:0/712763299 learned_addr learned my addr 172.21.15.88:0/712763299 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:45.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b1b30640 1 --2- 172.21.15.88:0/712763299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c0810 0x55ebe93c4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b2331640 1 -- 172.21.15.88:0/712763299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c0810 msgr2=0x55ebe93c4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b2331640 1 --2- 172.21.15.88:0/712763299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c0810 0x55ebe93c4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b2331640 1 -- 172.21.15.88:0/712763299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c1130 msgr2=0x55ebe93c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b2331640 1 --2- 172.21.15.88:0/712763299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c1130 0x55ebe93c4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b2331640 1 -- 172.21.15.88:0/712763299 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebe93d2f50 con 0x55ebe93c1a50 2023-10-16T20:39:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f04b2331640 1 --2- 172.21.15.88:0/712763299 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c1a50 0x55ebe93c4ff0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f04a400d780 tx=0x7f04a400dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff6c8dc26205eaf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f048ffff640 1 -- 172.21.15.88:0/712763299 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4013070 con 0x55ebe93c1a50 2023-10-16T20:39:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f048ffff640 1 -- 172.21.15.88:0/712763299 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a4014040 con 0x55ebe93c1a50 2023-10-16T20:39:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.981+0000 7f048ffff640 1 -- 172.21.15.88:0/712763299 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a400de30 con 0x55ebe93c1a50 2023-10-16T20:39:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 -- 172.21.15.88:0/712763299 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c1a50 msgr2=0x55ebe93c4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 --2- 172.21.15.88:0/712763299 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c1a50 0x55ebe93c4ff0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f04a400d780 tx=0x7f04a400dc40 comp rx=0 tx=0).stop 2023-10-16T20:39:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 -- 172.21.15.88:0/712763299 shutdown_connections 2023-10-16T20:39:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 --2- 172.21.15.88:0/712763299 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c1a50 0x55ebe93c4ff0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 --2- 172.21.15.88:0/712763299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c1130 0x55ebe93c4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 --2- 172.21.15.88:0/712763299 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c0810 0x55ebe93c4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:45.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 -- 172.21.15.88:0/712763299 >> 172.21.15.88:0/712763299 conn(0x55ebe9376fc0 msgr2=0x55ebe93c7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:45.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 -- 172.21.15.88:0/712763299 shutdown_connections 2023-10-16T20:39:45.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 -- 172.21.15.88:0/712763299 wait complete. 2023-10-16T20:39:45.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 -- start start 2023-10-16T20:39:45.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 Processor -- start 2023-10-16T20:39:45.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c0810 0x55ebe93e4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c1130 0x55ebe93e4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c1a50 0x55ebe93e5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:45.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebe917e6c0 con 0x55ebe93c0810 2023-10-16T20:39:45.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebe93c02e0 con 0x55ebe93c1130 2023-10-16T20:39:45.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ebe93d1e70 con 0x55ebe93c1a50 2023-10-16T20:39:45.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 10 client.0 _ll_register_callbacks cb 0x7f04a8051250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T20:39:45.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b1b30640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c0810 0x55ebe93e4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b1b30640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c0810 0x55ebe93e4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33450/0 (socket says 172.21.15.88:33450) 2023-10-16T20:39:45.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b1b30640 1 -- 172.21.15.88:0/787741283 learned_addr learned my addr 172.21.15.88:0/787741283 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:45.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b132f640 1 --2- 172.21.15.88:0/787741283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c1130 0x55ebe93e4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:45.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b2331640 1 --2- 172.21.15.88:0/787741283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c1a50 0x55ebe93e5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b1b30640 1 -- 172.21.15.88:0/787741283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c1a50 msgr2=0x55ebe93e5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b1b30640 1 --2- 172.21.15.88:0/787741283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c1a50 0x55ebe93e5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b1b30640 1 -- 172.21.15.88:0/787741283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c1130 msgr2=0x55ebe93e4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b1b30640 1 --2- 172.21.15.88:0/787741283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c1130 0x55ebe93e4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b1b30640 1 -- 172.21.15.88:0/787741283 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a400d430 con 0x55ebe93c0810 2023-10-16T20:39:46.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b1b30640 1 --2- 172.21.15.88:0/787741283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c0810 0x55ebe93e4990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f04a80550d0 tx=0x7f04a8004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb0790350efe2f11 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f048cff9640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-10-16T20:39:46.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f048cff9640 1 -- 172.21.15.88:0/787741283 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8051620 con 0x55ebe93c0810 2023-10-16T20:39:46.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f048cff9640 1 -- 172.21.15.88:0/787741283 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f04a8051780 con 0x55ebe93c0810 2023-10-16T20:39:46.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f04b711a000 1 -- 172.21.15.88:0/787741283 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ebe93d2950 con 0x55ebe93c0810 2023-10-16T20:39:46.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.985+0000 7f048cff9640 1 -- 172.21.15.88:0/787741283 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04a805cd80 con 0x55ebe93c0810 2023-10-16T20:39:46.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 10 client.4932 fetch_fsmap learned FSMap version 19 2023-10-16T20:39:46.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 1 -- 172.21.15.88:0/787741283 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x55ebe9178760 con 0x55ebe93c0810 2023-10-16T20:39:46.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f048cff9640 1 -- 172.21.15.88:0/787741283 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ebe93d2950 con 0x55ebe93c0810 2023-10-16T20:39:46.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f048cff9640 1 -- 172.21.15.88:0/787741283 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) v1 ==== 4500+0+0 (secure 0 0 0) 0x7f04a8021020 con 0x55ebe93c0810 2023-10-16T20:39:46.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 10 client.4932 fetch_fsmap finished waiting for FSMap version 19 2023-10-16T20:39:46.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 10 client.4932 resolve_mds: resolved name 'e' to daemon mds.e 2023-10-16T20:39:46.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 20 client.4932 populate_metadata read hostname 'smithi088' 2023-10-16T20:39:46.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 1 --2- 172.21.15.88:0/787741283 >> [v2:172.21.15.88:6836/321173984,v1:172.21.15.88:6838/321173984] conn(0x55ebe942a450 0x55ebe942c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 4 client.4932 mds_command: new command op to 4275 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T20:39:46.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 1 -- 172.21.15.88:0/787741283 --> [v2:172.21.15.88:6836/321173984,v1:172.21.15.88:6838/321173984] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ebe91686f0 con 0x55ebe942a450 2023-10-16T20:39:46.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b132f640 1 --2- 172.21.15.88:0/787741283 >> [v2:172.21.15.88:6836/321173984,v1:172.21.15.88:6838/321173984] conn(0x55ebe942a450 0x55ebe942c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f048cff9640 10 client.4932 ms_handle_connect on v2:172.21.15.88:6836/321173984 2023-10-16T20:39:46.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b132f640 1 --2- 172.21.15.88:0/787741283 >> [v2:172.21.15.88:6836/321173984,v1:172.21.15.88:6838/321173984] conn(0x55ebe942a450 0x55ebe942c840 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a536f61661d31b04 server_cookie=76ba88d7c5f3d878 in_seq=0 out_seq=0 2023-10-16T20:39:46.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f048cff9640 1 -- 172.21.15.88:0/787741283 <== mds.2 v2:172.21.15.88:6836/321173984 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55ebe91686f0 con 0x55ebe942a450 2023-10-16T20:39:46.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f048cff9640 10 client.4932 handle_command_reply: tid=0 2023-10-16T20:39:46.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 1 client.4932 shutdown 2023-10-16T20:39:46.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 1 -- 172.21.15.88:0/787741283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c0810 msgr2=0x55ebe93e4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 1 --2- 172.21.15.88:0/787741283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c0810 0x55ebe93e4990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f04a80550d0 tx=0x7f04a8004690 comp rx=0 tx=0).stop 2023-10-16T20:39:46.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 1 -- 172.21.15.88:0/787741283 shutdown_connections 2023-10-16T20:39:46.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 1 --2- 172.21.15.88:0/787741283 >> [v2:172.21.15.88:6836/321173984,v1:172.21.15.88:6838/321173984] conn(0x55ebe942a450 0x55ebe942c840 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 1 --2- 172.21.15.88:0/787741283 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55ebe93c1a50 0x55ebe93e5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 1 --2- 172.21.15.88:0/787741283 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55ebe93c1130 0x55ebe93e4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f048cff9640 0 client.4932 ms_handle_reset on v2:172.21.15.88:6836/321173984 2023-10-16T20:39:46.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 1 --2- 172.21.15.88:0/787741283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ebe93c0810 0x55ebe93e4990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 1 -- 172.21.15.88:0/787741283 >> 172.21.15.88:0/787741283 conn(0x55ebe9376fc0 msgr2=0x55ebe93c5010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:46.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 1 -- 172.21.15.88:0/787741283 shutdown_connections 2023-10-16T20:39:46.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 1 -- 172.21.15.88:0/787741283 wait complete. 2023-10-16T20:39:46.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b711a000 20 client.4932 trim_cache size 0 max 16384 2023-10-16T20:39:46.007 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-10-16T20:39:46.008 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "8cf7342f-a394-4b4c-ac35-9931ac9d9e57", 2023-10-16T20:39:46.008 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 2, 2023-10-16T20:39:46.008 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 4275, 2023-10-16T20:39:46.008 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2023-10-16T20:39:46.008 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2023-10-16T20:39:46.009 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2023-10-16T20:39:46.009 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 22.558888208999999, 2023-10-16T20:39:46.009 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 19, 2023-10-16T20:39:46.009 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 29, 2023-10-16T20:39:46.009 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 29, 2023-10-16T20:39:46.010 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 177.07816536300001 2023-10-16T20:39:46.010 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-10-16T20:39:46.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b3d94640 1 -- 172.21.15.88:0/1372958545 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f04900664d0 msgr2=0x7f0490068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b3d94640 1 --2- 172.21.15.88:0/1372958545 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f04900664d0 0x7f0490068990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f049c0ae000 tx=0x7f049c008040 comp rx=0 tx=0).stop 2023-10-16T20:39:46.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b3d94640 1 -- 172.21.15.88:0/1372958545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04ac06dd70 msgr2=0x7f04ac141060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b3d94640 1 --2- 172.21.15.88:0/1372958545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04ac06dd70 0x7f04ac141060 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f04a800b440 tx=0x7f04a800b900 comp rx=0 tx=0).stop 2023-10-16T20:39:46.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b3d94640 1 -- 172.21.15.88:0/1372958545 shutdown_connections 2023-10-16T20:39:46.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b3d94640 1 --2- 172.21.15.88:0/1372958545 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f04900664d0 0x7f0490068990 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b3d94640 1 --2- 172.21.15.88:0/1372958545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f04ac1537d0 0x7f04ac1f4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b3d94640 1 --2- 172.21.15.88:0/1372958545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04ac0cdb20 0x7f04ac1f43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b3d94640 1 --2- 172.21.15.88:0/1372958545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04ac06dd70 0x7f04ac141060 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b3d94640 1 -- 172.21.15.88:0/1372958545 >> 172.21.15.88:0/1372958545 conn(0x7f04ac06f7d0 msgr2=0x7f04ac06fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:46.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b3d94640 1 -- 172.21.15.88:0/1372958545 shutdown_connections 2023-10-16T20:39:46.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:45.989+0000 7f04b3d94640 1 -- 172.21.15.88:0/1372958545 wait complete. 2023-10-16T20:39:46.014 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-16T20:39:46.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08fc17640 1 Processor -- start 2023-10-16T20:39:46.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08fc17640 1 -- start start 2023-10-16T20:39:46.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08fc17640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff0880cb5c0 0x7ff088071420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08fc17640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff088070f40 con 0x7ff0880c6ac0 2023-10-16T20:39:46.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08fc17640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08806a2d0 con 0x7ff0880c66e0 2023-10-16T20:39:46.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08fc17640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08806a5b0 con 0x7ff0880cb5c0 2023-10-16T20:39:46.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08d1b2640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff0880c6ac0 0x7ff088150150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55992/0 (socket says 172.21.15.88:55992) 2023-10-16T20:39:46.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08d1b2640 1 -- 172.21.15.88:0/294816321 learned_addr learned my addr 172.21.15.88:0/294816321 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:46.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08e1b4640 1 --2- 172.21.15.88:0/294816321 >> v2:172.21.15.191:3300/0 conn(0x7ff0880cb5c0 0x7ff088071420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08c9b1640 1 -- 172.21.15.88:0/294816321 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067954767 0 0) 0x7ff088070f40 con 0x7ff0880c6ac0 2023-10-16T20:39:46.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08c9b1640 1 -- 172.21.15.88:0/294816321 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff074005740 con 0x7ff0880c6ac0 2023-10-16T20:39:46.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08c9b1640 1 -- 172.21.15.88:0/294816321 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 890557852 0 0) 0x7ff074005740 con 0x7ff0880c6ac0 2023-10-16T20:39:46.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08c9b1640 1 -- 172.21.15.88:0/294816321 >> v2:172.21.15.191:3300/0 conn(0x7ff0880cb5c0 msgr2=0x7ff088071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08c9b1640 1 --2- 172.21.15.88:0/294816321 >> v2:172.21.15.191:3300/0 conn(0x7ff0880cb5c0 0x7ff088071420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08c9b1640 1 -- 172.21.15.88:0/294816321 >> v1:172.21.15.191:6789/0 conn(0x7ff0880c66e0 legacy=0x7ff08815a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08c9b1640 1 -- 172.21.15.88:0/294816321 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08806a890 con 0x7ff0880c6ac0 2023-10-16T20:39:46.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08c9b1640 1 -- 172.21.15.88:0/294816321 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7ff078001680 con 0x7ff0880c6ac0 2023-10-16T20:39:46.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08c9b1640 1 -- 172.21.15.88:0/294816321 >> v1:172.21.15.88:6789/0 conn(0x7ff0880c6ac0 legacy=0x7ff088150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08c9b1640 1 --2- 172.21.15.88:0/294816321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff074005b90 0x7ff074005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08c9b1640 1 --2- 172.21.15.88:0/294816321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0740069e0 0x7ff074006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08c9b1640 1 --2- 172.21.15.88:0/294816321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0740073a0 0x7ff07400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08c9b1640 1 -- 172.21.15.88:0/294816321 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08806a5b0 con 0x7ff0740069e0 2023-10-16T20:39:46.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08c9b1640 1 -- 172.21.15.88:0/294816321 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07400e340 con 0x7ff0740073a0 2023-10-16T20:39:46.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.149+0000 7ff08c9b1640 1 -- 172.21.15.88:0/294816321 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff07400e500 con 0x7ff074005b90 2023-10-16T20:39:46.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.153+0000 7ff08d1b2640 1 --2- 172.21.15.88:0/294816321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0740069e0 0x7ff074006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.153+0000 7ff08d9b3640 1 --2- 172.21.15.88:0/294816321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff074005b90 0x7ff074005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.153+0000 7ff08e1b4640 1 --2- 172.21.15.88:0/294816321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0740073a0 0x7ff07400dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.153+0000 7ff08d1b2640 1 -- 172.21.15.88:0/294816321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff074005b90 msgr2=0x7ff074005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.153+0000 7ff08d1b2640 1 --2- 172.21.15.88:0/294816321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff074005b90 0x7ff074005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.153+0000 7ff08d1b2640 1 -- 172.21.15.88:0/294816321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0740073a0 msgr2=0x7ff07400dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.153+0000 7ff08d1b2640 1 --2- 172.21.15.88:0/294816321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0740073a0 0x7ff07400dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.153+0000 7ff08d1b2640 1 -- 172.21.15.88:0/294816321 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff074005740 con 0x7ff0740069e0 2023-10-16T20:39:46.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.153+0000 7ff08e1b4640 1 -- 172.21.15.88:0/294816321 reap_dead start 2023-10-16T20:39:46.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.153+0000 7ff08d1b2640 1 --2- 172.21.15.88:0/294816321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0740069e0 0x7ff074006e60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff07800c190 tx=0x7ff07801d5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda67efc251f0c47 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.153+0000 7ff08c9b1640 1 -- 172.21.15.88:0/294816321 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0780058e0 con 0x7ff0740069e0 2023-10-16T20:39:46.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.153+0000 7ff08c9b1640 1 -- 172.21.15.88:0/294816321 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff078005a40 con 0x7ff0740069e0 2023-10-16T20:39:46.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.153+0000 7ff08c9b1640 1 -- 172.21.15.88:0/294816321 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078005e70 con 0x7ff0740069e0 2023-10-16T20:39:46.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08fc17640 1 -- 172.21.15.88:0/294816321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0740069e0 msgr2=0x7ff074006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08fc17640 1 --2- 172.21.15.88:0/294816321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0740069e0 0x7ff074006e60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff07800c190 tx=0x7ff07801d5e0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08fc17640 1 -- 172.21.15.88:0/294816321 shutdown_connections 2023-10-16T20:39:46.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08fc17640 1 --2- 172.21.15.88:0/294816321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0740069e0 0x7ff074006e60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08fc17640 1 -- 172.21.15.88:0/294816321 >> 172.21.15.88:0/294816321 conn(0x7ff0880ca9f0 msgr2=0x7ff0880cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:46.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08fc17640 1 -- 172.21.15.88:0/294816321 shutdown_connections 2023-10-16T20:39:46.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08fc17640 1 -- 172.21.15.88:0/294816321 wait complete. 2023-10-16T20:39:46.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08fc17640 1 Processor -- start 2023-10-16T20:39:46.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08fc17640 1 -- start start 2023-10-16T20:39:46.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08fc17640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0740069e0 0x7ff0880bfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08fc17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0881f2df0 0x7ff088153560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08fc17640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0881f50e0 0x7ff0881f5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08fc17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff088156390 con 0x7ff0881f2df0 2023-10-16T20:39:46.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08fc17640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0881566e0 con 0x7ff0881f50e0 2023-10-16T20:39:46.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08fc17640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0880c0330 con 0x7ff0740069e0 2023-10-16T20:39:46.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08d1b2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0881f2df0 0x7ff088153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08d1b2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0881f2df0 0x7ff088153560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33462/0 (socket says 172.21.15.88:33462) 2023-10-16T20:39:46.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08d1b2640 1 -- 172.21.15.88:0/1372424230 learned_addr learned my addr 172.21.15.88:0/1372424230 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:46.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08e1b4640 1 --2- 172.21.15.88:0/1372424230 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0881f50e0 0x7ff0881f5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08d9b3640 1 --2- 172.21.15.88:0/1372424230 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0740069e0 0x7ff0880bfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08d1b2640 1 -- 172.21.15.88:0/1372424230 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0740069e0 msgr2=0x7ff0880bfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08d1b2640 1 --2- 172.21.15.88:0/1372424230 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0740069e0 0x7ff0880bfdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08d1b2640 1 -- 172.21.15.88:0/1372424230 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0881f50e0 msgr2=0x7ff0881f5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08d1b2640 1 --2- 172.21.15.88:0/1372424230 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0881f50e0 0x7ff0881f5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08d1b2640 1 -- 172.21.15.88:0/1372424230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff088155dd0 con 0x7ff0881f2df0 2023-10-16T20:39:46.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08e1b4640 1 --2- 172.21.15.88:0/1372424230 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0881f50e0 0x7ff0881f5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:46.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.157+0000 7ff08d1b2640 1 --2- 172.21.15.88:0/1372424230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0881f2df0 0x7ff088153560 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff078001940 tx=0x7ff074007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.161+0000 7ff07e7fc640 1 -- 172.21.15.88:0/1372424230 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0780047f0 con 0x7ff0881f2df0 2023-10-16T20:39:46.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.161+0000 7ff07e7fc640 1 -- 172.21.15.88:0/1372424230 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff078004950 con 0x7ff0881f2df0 2023-10-16T20:39:46.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.161+0000 7ff07e7fc640 1 -- 172.21.15.88:0/1372424230 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078004cc0 con 0x7ff0881f2df0 2023-10-16T20:39:46.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.161+0000 7ff08fc17640 1 -- 172.21.15.88:0/1372424230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0881560b0 con 0x7ff0881f2df0 2023-10-16T20:39:46.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.161+0000 7ff08fc17640 1 -- 172.21.15.88:0/1372424230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff088153aa0 con 0x7ff0881f2df0 2023-10-16T20:39:46.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.161+0000 7ff07e7fc640 1 -- 172.21.15.88:0/1372424230 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff078007110 con 0x7ff0881f2df0 2023-10-16T20:39:46.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.161+0000 7ff07e7fc640 1 --2- 172.21.15.88:0/1372424230 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff06c0664a0 0x7ff06c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff07e7fc640 1 -- 172.21.15.88:0/1372424230 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff07811d480 con 0x7ff0881f2df0 2023-10-16T20:39:46.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff08d9b3640 1 --2- 172.21.15.88:0/1372424230 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff06c0664a0 0x7ff06c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 Processor -- start 2023-10-16T20:39:46.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 -- start start 2023-10-16T20:39:46.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff08d9b3640 1 --2- 172.21.15.88:0/1372424230 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff06c0664a0 0x7ff06c068960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff088069ae0 tx=0x7ff084009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1581810 0x55c2e1581bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1582130 0x55c2e1582510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1582a50 0x55c2e158caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e1335cc0 con 0x55c2e1582130 2023-10-16T20:39:46.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e1398920 con 0x55c2e1582a50 2023-10-16T20:39:46.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2e1395e50 con 0x55c2e1581810 2023-10-16T20:39:46.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff08d1b2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1582130 0x55c2e1582510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff08d1b2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1582130 0x55c2e1582510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33476/0 (socket says 172.21.15.88:33476) 2023-10-16T20:39:46.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff08d1b2640 1 -- 172.21.15.88:0/4087725904 learned_addr learned my addr 172.21.15.88:0/4087725904 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:46.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff08d1b2640 1 -- 172.21.15.88:0/4087725904 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1581810 msgr2=0x55c2e1581bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T20:39:46.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff08d1b2640 1 --2- 172.21.15.88:0/4087725904 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1581810 0x55c2e1581bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff08d1b2640 1 -- 172.21.15.88:0/4087725904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1582a50 msgr2=0x55c2e158caa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T20:39:46.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff08d1b2640 1 --2- 172.21.15.88:0/4087725904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1582a50 0x55c2e158caa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff08d1b2640 1 -- 172.21.15.88:0/4087725904 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2e108a890 con 0x55c2e1582130 2023-10-16T20:39:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff08d1b2640 1 --2- 172.21.15.88:0/4087725904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1582130 0x55c2e1582510 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff0780f6060 tx=0x7ff078004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20380afdfdd38a1f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff06b7fe640 1 -- 172.21.15.88:0/4087725904 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0780f6590 con 0x55c2e1582130 2023-10-16T20:39:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff06b7fe640 1 -- 172.21.15.88:0/4087725904 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0780ec630 con 0x55c2e1582130 2023-10-16T20:39:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff06b7fe640 1 -- 172.21.15.88:0/4087725904 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0780ea9a0 con 0x55c2e1582130 2023-10-16T20:39:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 -- 172.21.15.88:0/4087725904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1582130 msgr2=0x55c2e1582510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 --2- 172.21.15.88:0/4087725904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1582130 0x55c2e1582510 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff0780f6060 tx=0x7ff078004f80 comp rx=0 tx=0).stop 2023-10-16T20:39:46.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 -- 172.21.15.88:0/4087725904 shutdown_connections 2023-10-16T20:39:46.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 --2- 172.21.15.88:0/4087725904 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1582a50 0x55c2e158caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 --2- 172.21.15.88:0/4087725904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1582130 0x55c2e1582510 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 --2- 172.21.15.88:0/4087725904 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1581810 0x55c2e1581bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 -- 172.21.15.88:0/4087725904 >> 172.21.15.88:0/4087725904 conn(0x55c2e1537fc0 msgr2=0x55c2e157f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:46.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 -- 172.21.15.88:0/4087725904 shutdown_connections 2023-10-16T20:39:46.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 -- 172.21.15.88:0/4087725904 wait complete. 2023-10-16T20:39:46.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 -- start start 2023-10-16T20:39:46.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.165+0000 7ff092fa8000 1 Processor -- start 2023-10-16T20:39:46.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff092fa8000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1581810 0x55c2e1543240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff092fa8000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1582130 0x55c2e1543780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff092fa8000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1582a50 0x55c2e1543cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff092fa8000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e1391740 con 0x55c2e1581810 2023-10-16T20:39:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff092fa8000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e1350d10 con 0x55c2e1582a50 2023-10-16T20:39:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff092fa8000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2e13509b0 con 0x55c2e1582130 2023-10-16T20:39:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff092fa8000 10 client.0 _ll_register_callbacks cb 0x55c2e152c390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T20:39:46.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff08d9b3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1581810 0x55c2e1543240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff08d9b3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1581810 0x55c2e1543240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33478/0 (socket says 172.21.15.88:33478) 2023-10-16T20:39:46.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff08d9b3640 1 -- 172.21.15.88:0/3108839522 learned_addr learned my addr 172.21.15.88:0/3108839522 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:46.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff08e1b4640 1 --2- 172.21.15.88:0/3108839522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1582a50 0x55c2e1543cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff08d1b2640 1 --2- 172.21.15.88:0/3108839522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1582130 0x55c2e1543780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff08d9b3640 1 -- 172.21.15.88:0/3108839522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1582130 msgr2=0x55c2e1543780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff08d9b3640 1 --2- 172.21.15.88:0/3108839522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1582130 0x55c2e1543780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff08d9b3640 1 -- 172.21.15.88:0/3108839522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1582a50 msgr2=0x55c2e1543cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff08d9b3640 1 --2- 172.21.15.88:0/3108839522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1582a50 0x55c2e1543cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff08d9b3640 1 -- 172.21.15.88:0/3108839522 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff088153aa0 con 0x55c2e1581810 2023-10-16T20:39:46.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff08e1b4640 1 --2- 172.21.15.88:0/3108839522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1582a50 0x55c2e1543cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:46.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff08d9b3640 1 --2- 172.21.15.88:0/3108839522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1581810 0x55c2e1543240 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff084001e50 tx=0x7ff084001130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa7cfc405d60b9d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff068ff9640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-10-16T20:39:46.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff068ff9640 1 -- 172.21.15.88:0/3108839522 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084015e80 con 0x55c2e1581810 2023-10-16T20:39:46.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff068ff9640 1 -- 172.21.15.88:0/3108839522 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff08401d050 con 0x55c2e1581810 2023-10-16T20:39:46.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff092fa8000 1 -- 172.21.15.88:0/3108839522 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c2e1345370 con 0x55c2e1581810 2023-10-16T20:39:46.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff068ff9640 1 -- 172.21.15.88:0/3108839522 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0840145e0 con 0x55c2e1581810 2023-10-16T20:39:46.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff092fa8000 10 client.4944 fetch_fsmap learned FSMap version 19 2023-10-16T20:39:46.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff092fa8000 1 -- 172.21.15.88:0/3108839522 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x55c2e1353460 con 0x55c2e1581810 2023-10-16T20:39:46.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff068ff9640 1 -- 172.21.15.88:0/3108839522 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c2e1345370 con 0x55c2e1581810 2023-10-16T20:39:46.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff068ff9640 1 -- 172.21.15.88:0/3108839522 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) v1 ==== 4500+0+0 (secure 0 0 0) 0x7ff084014740 con 0x55c2e1581810 2023-10-16T20:39:46.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff092fa8000 10 client.4944 fetch_fsmap finished waiting for FSMap version 19 2023-10-16T20:39:46.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff092fa8000 10 client.4944 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T20:39:46.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff092fa8000 20 client.4944 populate_metadata read hostname 'smithi088' 2023-10-16T20:39:46.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff092fa8000 1 --2- 172.21.15.88:0/3108839522 >> [v2:172.21.15.191:6832/2446850309,v1:172.21.15.191:6833/2446850309] conn(0x55c2e15446d0 0x55c2e1596900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff092fa8000 4 client.4944 mds_command: new command op to 4279 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T20:39:46.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff092fa8000 1 -- 172.21.15.88:0/3108839522 --> [v2:172.21.15.191:6832/2446850309,v1:172.21.15.191:6833/2446850309] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2e132b2d0 con 0x55c2e15446d0 2023-10-16T20:39:46.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.169+0000 7ff08d1b2640 1 --2- 172.21.15.88:0/3108839522 >> [v2:172.21.15.191:6832/2446850309,v1:172.21.15.191:6833/2446850309] conn(0x55c2e15446d0 0x55c2e1596900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.173+0000 7ff068ff9640 10 client.4944 ms_handle_connect on v2:172.21.15.191:6832/2446850309 2023-10-16T20:39:46.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.173+0000 7ff08d1b2640 1 --2- 172.21.15.88:0/3108839522 >> [v2:172.21.15.191:6832/2446850309,v1:172.21.15.191:6833/2446850309] conn(0x55c2e15446d0 0x55c2e1596900 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9b2a1396623217b5 server_cookie=8189e46c81c1dd48 in_seq=0 out_seq=0 2023-10-16T20:39:46.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.173+0000 7ff068ff9640 1 -- 172.21.15.88:0/3108839522 <== mds.1 v2:172.21.15.191:6832/2446850309 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c2e132b2d0 con 0x55c2e15446d0 2023-10-16T20:39:46.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.173+0000 7ff068ff9640 10 client.4944 handle_command_reply: tid=0 2023-10-16T20:39:46.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.173+0000 7ff092fa8000 1 client.4944 shutdown 2023-10-16T20:39:46.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.173+0000 7ff092fa8000 1 -- 172.21.15.88:0/3108839522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1581810 msgr2=0x55c2e1543240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.173+0000 7ff092fa8000 1 --2- 172.21.15.88:0/3108839522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1581810 0x55c2e1543240 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff084001e50 tx=0x7ff084001130 comp rx=0 tx=0).stop 2023-10-16T20:39:46.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.173+0000 7ff092fa8000 1 -- 172.21.15.88:0/3108839522 shutdown_connections 2023-10-16T20:39:46.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.173+0000 7ff092fa8000 1 --2- 172.21.15.88:0/3108839522 >> [v2:172.21.15.191:6832/2446850309,v1:172.21.15.191:6833/2446850309] conn(0x55c2e15446d0 0x55c2e1596900 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.173+0000 7ff092fa8000 1 --2- 172.21.15.88:0/3108839522 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1582a50 0x55c2e1543cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.173+0000 7ff068ff9640 0 client.4944 ms_handle_reset on v2:172.21.15.191:6832/2446850309 2023-10-16T20:39:46.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.173+0000 7ff092fa8000 1 --2- 172.21.15.88:0/3108839522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1582130 0x55c2e1543780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.173+0000 7ff092fa8000 1 --2- 172.21.15.88:0/3108839522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1581810 0x55c2e1543240 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.173+0000 7ff092fa8000 1 -- 172.21.15.88:0/3108839522 >> 172.21.15.88:0/3108839522 conn(0x55c2e1537fc0 msgr2=0x55c2e1581090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:46.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.173+0000 7ff092fa8000 1 -- 172.21.15.88:0/3108839522 shutdown_connections 2023-10-16T20:39:46.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.173+0000 7ff092fa8000 1 -- 172.21.15.88:0/3108839522 wait complete. 2023-10-16T20:39:46.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.173+0000 7ff092fa8000 20 client.4944 trim_cache size 0 max 16384 2023-10-16T20:39:46.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff092fa8000 1 Processor -- start 2023-10-16T20:39:46.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff092fa8000 1 -- start start 2023-10-16T20:39:46.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff092fa8000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1581810 0x55c2e153f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff092fa8000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1582130 0x55c2e153fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff092fa8000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1582a50 0x55c2e15401f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff092fa8000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e104b940 con 0x55c2e1582a50 2023-10-16T20:39:46.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff092fa8000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e13905d0 con 0x55c2e1581810 2023-10-16T20:39:46.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff092fa8000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2e139b200 con 0x55c2e1582130 2023-10-16T20:39:46.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff08e1b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1582a50 0x55c2e15401f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff08e1b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1582a50 0x55c2e15401f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33480/0 (socket says 172.21.15.88:33480) 2023-10-16T20:39:46.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff08e1b4640 1 -- 172.21.15.88:0/3655446923 learned_addr learned my addr 172.21.15.88:0/3655446923 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:46.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff08d9b3640 1 --2- 172.21.15.88:0/3655446923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1581810 0x55c2e153f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff08d1b2640 1 --2- 172.21.15.88:0/3655446923 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1582130 0x55c2e153fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff08e1b4640 1 -- 172.21.15.88:0/3655446923 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1582130 msgr2=0x55c2e153fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff08e1b4640 1 --2- 172.21.15.88:0/3655446923 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1582130 0x55c2e153fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff08e1b4640 1 -- 172.21.15.88:0/3655446923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1581810 msgr2=0x55c2e153f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff08e1b4640 1 --2- 172.21.15.88:0/3655446923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1581810 0x55c2e153f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff08e1b4640 1 -- 172.21.15.88:0/3655446923 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2e1339760 con 0x55c2e1582a50 2023-10-16T20:39:46.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff08d9b3640 1 --2- 172.21.15.88:0/3655446923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1581810 0x55c2e153f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:46.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff08e1b4640 1 --2- 172.21.15.88:0/3655446923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1582a50 0x55c2e15401f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff0880c1cc0 tx=0x7ff080009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c98afbdc632ed8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff06bfff640 1 -- 172.21.15.88:0/3655446923 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08000e040 con 0x55c2e1582a50 2023-10-16T20:39:46.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff06bfff640 1 -- 172.21.15.88:0/3655446923 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08000f040 con 0x55c2e1582a50 2023-10-16T20:39:46.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff06bfff640 1 -- 172.21.15.88:0/3655446923 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff080017830 con 0x55c2e1582a50 2023-10-16T20:39:46.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff092fa8000 1 -- 172.21.15.88:0/3655446923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1582a50 msgr2=0x55c2e15401f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff092fa8000 1 --2- 172.21.15.88:0/3655446923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1582a50 0x55c2e15401f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff0880c1cc0 tx=0x7ff080009750 comp rx=0 tx=0).stop 2023-10-16T20:39:46.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff092fa8000 1 -- 172.21.15.88:0/3655446923 shutdown_connections 2023-10-16T20:39:46.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff092fa8000 1 --2- 172.21.15.88:0/3655446923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1582a50 0x55c2e15401f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff092fa8000 1 --2- 172.21.15.88:0/3655446923 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1582130 0x55c2e153fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff092fa8000 1 --2- 172.21.15.88:0/3655446923 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1581810 0x55c2e153f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff092fa8000 1 -- 172.21.15.88:0/3655446923 >> 172.21.15.88:0/3655446923 conn(0x55c2e1537fc0 msgr2=0x55c2e157e280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:46.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff092fa8000 1 -- 172.21.15.88:0/3655446923 shutdown_connections 2023-10-16T20:39:46.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.189+0000 7ff092fa8000 1 -- 172.21.15.88:0/3655446923 wait complete. 2023-10-16T20:39:46.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 1 -- start start 2023-10-16T20:39:46.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 1 Processor -- start 2023-10-16T20:39:46.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1581810 0x55c2e15e4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1582130 0x55c2e15ddbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1582a50 0x55c2e15de130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e138eb40 con 0x55c2e1581810 2023-10-16T20:39:46.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e138ba40 con 0x55c2e1582a50 2023-10-16T20:39:46.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2e13942a0 con 0x55c2e1582130 2023-10-16T20:39:46.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 10 client.0 _ll_register_callbacks cb 0x7ff0780ee060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T20:39:46.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff08d9b3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1581810 0x55c2e15e4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff08d9b3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1581810 0x55c2e15e4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33484/0 (socket says 172.21.15.88:33484) 2023-10-16T20:39:46.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff08d9b3640 1 -- 172.21.15.88:0/3669188999 learned_addr learned my addr 172.21.15.88:0/3669188999 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:46.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff08d1b2640 1 --2- 172.21.15.88:0/3669188999 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1582130 0x55c2e15ddbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff08e1b4640 1 --2- 172.21.15.88:0/3669188999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1582a50 0x55c2e15de130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff08d9b3640 1 -- 172.21.15.88:0/3669188999 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1582130 msgr2=0x55c2e15ddbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff08d9b3640 1 --2- 172.21.15.88:0/3669188999 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1582130 0x55c2e15ddbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff08d9b3640 1 -- 172.21.15.88:0/3669188999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1582a50 msgr2=0x55c2e15de130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff08d9b3640 1 --2- 172.21.15.88:0/3669188999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1582a50 0x55c2e15de130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff08d9b3640 1 -- 172.21.15.88:0/3669188999 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0800039a0 con 0x55c2e1581810 2023-10-16T20:39:46.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff08d9b3640 1 --2- 172.21.15.88:0/3669188999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1581810 0x55c2e15e4970 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff084001e50 tx=0x7ff084016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3afe67493c6809c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff068ff9640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-10-16T20:39:46.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff068ff9640 1 -- 172.21.15.88:0/3669188999 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084014e00 con 0x55c2e1581810 2023-10-16T20:39:46.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 1 -- 172.21.15.88:0/3669188999 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c2e1544090 con 0x55c2e1581810 2023-10-16T20:39:46.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff068ff9640 1 -- 172.21.15.88:0/3669188999 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff08401d040 con 0x55c2e1581810 2023-10-16T20:39:46.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff068ff9640 1 -- 172.21.15.88:0/3669188999 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0840145e0 con 0x55c2e1581810 2023-10-16T20:39:46.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 10 client.4950 fetch_fsmap learned FSMap version 19 2023-10-16T20:39:46.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 1 -- 172.21.15.88:0/3669188999 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x55c2e108acc0 con 0x55c2e1581810 2023-10-16T20:39:46.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff068ff9640 1 -- 172.21.15.88:0/3669188999 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c2e1544090 con 0x55c2e1581810 2023-10-16T20:39:46.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff068ff9640 1 -- 172.21.15.88:0/3669188999 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) v1 ==== 4500+0+0 (secure 0 0 0) 0x7ff084014740 con 0x55c2e1581810 2023-10-16T20:39:46.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 10 client.4950 fetch_fsmap finished waiting for FSMap version 19 2023-10-16T20:39:46.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 10 client.4950 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-16T20:39:46.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 20 client.4950 populate_metadata read hostname 'smithi088' 2023-10-16T20:39:46.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 1 --2- 172.21.15.88:0/3669188999 >> [v2:172.21.15.191:6832/2446850309,v1:172.21.15.191:6833/2446850309] conn(0x55c2e15e1010 0x55c2e15e13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 4 client.4950 mds_command: new command op to 4279 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T20:39:46.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff092fa8000 1 -- 172.21.15.88:0/3669188999 --> [v2:172.21.15.191:6832/2446850309,v1:172.21.15.191:6833/2446850309] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c2e13296f0 con 0x55c2e15e1010 2023-10-16T20:39:46.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.193+0000 7ff08d1b2640 1 --2- 172.21.15.88:0/3669188999 >> [v2:172.21.15.191:6832/2446850309,v1:172.21.15.191:6833/2446850309] conn(0x55c2e15e1010 0x55c2e15e13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff08d1b2640 1 --2- 172.21.15.88:0/3669188999 >> [v2:172.21.15.191:6832/2446850309,v1:172.21.15.191:6833/2446850309] conn(0x55c2e15e1010 0x55c2e15e13f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bd25f7c31ac57839 server_cookie=58c5a2b94b6757c7 in_seq=0 out_seq=0 2023-10-16T20:39:46.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff068ff9640 10 client.4950 ms_handle_connect on v2:172.21.15.191:6832/2446850309 2023-10-16T20:39:46.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff068ff9640 1 -- 172.21.15.88:0/3669188999 <== mds.1 v2:172.21.15.191:6832/2446850309 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55c2e13296f0 con 0x55c2e15e1010 2023-10-16T20:39:46.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff068ff9640 10 client.4950 handle_command_reply: tid=0 2023-10-16T20:39:46.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff092fa8000 1 client.4950 shutdown 2023-10-16T20:39:46.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff092fa8000 1 -- 172.21.15.88:0/3669188999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1581810 msgr2=0x55c2e15e4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff092fa8000 1 --2- 172.21.15.88:0/3669188999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1581810 0x55c2e15e4970 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff084001e50 tx=0x7ff084016000 comp rx=0 tx=0).stop 2023-10-16T20:39:46.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff092fa8000 1 -- 172.21.15.88:0/3669188999 shutdown_connections 2023-10-16T20:39:46.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff092fa8000 1 --2- 172.21.15.88:0/3669188999 >> [v2:172.21.15.191:6832/2446850309,v1:172.21.15.191:6833/2446850309] conn(0x55c2e15e1010 0x55c2e15e13f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff092fa8000 1 --2- 172.21.15.88:0/3669188999 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55c2e1582a50 0x55c2e15de130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff092fa8000 1 --2- 172.21.15.88:0/3669188999 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55c2e1582130 0x55c2e15ddbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff068ff9640 0 client.4950 ms_handle_reset on v2:172.21.15.191:6832/2446850309 2023-10-16T20:39:46.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff092fa8000 1 --2- 172.21.15.88:0/3669188999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c2e1581810 0x55c2e15e4970 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff092fa8000 1 -- 172.21.15.88:0/3669188999 >> 172.21.15.88:0/3669188999 conn(0x55c2e1537fc0 msgr2=0x55c2e157eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:46.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff092fa8000 1 -- 172.21.15.88:0/3669188999 shutdown_connections 2023-10-16T20:39:46.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff092fa8000 1 -- 172.21.15.88:0/3669188999 wait complete. 2023-10-16T20:39:46.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff092fa8000 20 client.4950 trim_cache size 0 max 16384 2023-10-16T20:39:46.214 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-10-16T20:39:46.214 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "8cf7342f-a394-4b4c-ac35-9931ac9d9e57", 2023-10-16T20:39:46.214 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 1, 2023-10-16T20:39:46.214 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 4279, 2023-10-16T20:39:46.215 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2023-10-16T20:39:46.215 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2023-10-16T20:39:46.215 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2023-10-16T20:39:46.215 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 28.804126693000001, 2023-10-16T20:39:46.215 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 19, 2023-10-16T20:39:46.216 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 29, 2023-10-16T20:39:46.216 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 29, 2023-10-16T20:39:46.216 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 177.28824754799999 2023-10-16T20:39:46.216 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-10-16T20:39:46.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff08fc17640 1 -- 172.21.15.88:0/1372424230 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff06c0664a0 msgr2=0x7ff06c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff08fc17640 1 --2- 172.21.15.88:0/1372424230 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff06c0664a0 0x7ff06c068960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff088069ae0 tx=0x7ff084009040 comp rx=0 tx=0).stop 2023-10-16T20:39:46.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff08fc17640 1 -- 172.21.15.88:0/1372424230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0881f2df0 msgr2=0x7ff088153560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff08fc17640 1 --2- 172.21.15.88:0/1372424230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0881f2df0 0x7ff088153560 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff078001940 tx=0x7ff074007a10 comp rx=0 tx=0).stop 2023-10-16T20:39:46.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff08fc17640 1 -- 172.21.15.88:0/1372424230 shutdown_connections 2023-10-16T20:39:46.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff08fc17640 1 --2- 172.21.15.88:0/1372424230 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff06c0664a0 0x7ff06c068960 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff08fc17640 1 --2- 172.21.15.88:0/1372424230 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff0881f50e0 0x7ff0881f5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff08fc17640 1 --2- 172.21.15.88:0/1372424230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0881f2df0 0x7ff088153560 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff08fc17640 1 --2- 172.21.15.88:0/1372424230 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff0740069e0 0x7ff0880bfdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff08fc17640 1 -- 172.21.15.88:0/1372424230 >> 172.21.15.88:0/1372424230 conn(0x7ff0880ca9f0 msgr2=0x7ff0880c2180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:46.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff08fc17640 1 -- 172.21.15.88:0/1372424230 shutdown_connections 2023-10-16T20:39:46.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.197+0000 7ff08fc17640 1 -- 172.21.15.88:0/1372424230 wait complete. 2023-10-16T20:39:46.220 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-10-16T20:39:46.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf8c432640 1 Processor -- start 2023-10-16T20:39:46.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf8c432640 1 -- start start 2023-10-16T20:39:46.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf8c432640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7faf840c4260 0x7faf840c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf8c432640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf840c5a60 0x7faf841510d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf8c432640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf84069e90 con 0x7faf840c4640 2023-10-16T20:39:46.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf8c432640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf840716a0 con 0x7faf840c4260 2023-10-16T20:39:46.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf8c432640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8406aea0 con 0x7faf840c5a60 2023-10-16T20:39:46.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf899cd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf840c5a60 0x7faf841510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf8a9cf640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7faf840c4640 0x7faf841573e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:45636/0 (socket says 172.21.15.88:45636) 2023-10-16T20:39:46.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf8a9cf640 1 -- 172.21.15.88:0/2267551134 learned_addr learned my addr 172.21.15.88:0/2267551134 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:46.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf8a1ce640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7faf840c4260 0x7faf840c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf891cc640 1 -- 172.21.15.88:0/2267551134 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115711996 0 0) 0x7faf84069e90 con 0x7faf840c4640 2023-10-16T20:39:46.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf891cc640 1 -- 172.21.15.88:0/2267551134 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf70003530 con 0x7faf840c4640 2023-10-16T20:39:46.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf8a1ce640 1 -- 172.21.15.88:0/2267551134 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf840c5a60 msgr2=0x7faf841510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf8a1ce640 1 --2- 172.21.15.88:0/2267551134 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf840c5a60 0x7faf841510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf8a1ce640 1 -- 172.21.15.88:0/2267551134 >> v1:172.21.15.191:6789/0 conn(0x7faf840c4640 legacy=0x7faf841573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf8a1ce640 1 -- 172.21.15.88:0/2267551134 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf841581e0 con 0x7faf840c4260 2023-10-16T20:39:46.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.357+0000 7faf899cd640 1 --2- 172.21.15.88:0/2267551134 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf840c5a60 0x7faf841510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:39:46.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.361+0000 7faf8a1ce640 1 --2- 172.21.15.88:0/2267551134 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf840c4260 0x7faf840c5520 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faf800097c0 tx=0x7faf8001c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d83822d53a356c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.361+0000 7faf891cc640 1 -- 172.21.15.88:0/2267551134 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf800ab600 con 0x7faf840c4260 2023-10-16T20:39:46.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.361+0000 7faf891cc640 1 -- 172.21.15.88:0/2267551134 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf800a7050 con 0x7faf840c4260 2023-10-16T20:39:46.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.361+0000 7faf891cc640 1 -- 172.21.15.88:0/2267551134 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf800ab980 con 0x7faf840c4260 2023-10-16T20:39:46.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.361+0000 7faf8c432640 1 -- 172.21.15.88:0/2267551134 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf840c4260 msgr2=0x7faf840c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.361+0000 7faf8c432640 1 --2- 172.21.15.88:0/2267551134 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf840c4260 0x7faf840c5520 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faf800097c0 tx=0x7faf8001c330 comp rx=0 tx=0).stop 2023-10-16T20:39:46.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.361+0000 7faf8c432640 1 -- 172.21.15.88:0/2267551134 shutdown_connections 2023-10-16T20:39:46.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.361+0000 7faf8c432640 1 --2- 172.21.15.88:0/2267551134 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf840c5a60 0x7faf841510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.361+0000 7faf8c432640 1 --2- 172.21.15.88:0/2267551134 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf840c4260 0x7faf840c5520 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.361+0000 7faf8c432640 1 -- 172.21.15.88:0/2267551134 >> 172.21.15.88:0/2267551134 conn(0x7faf840ca9f0 msgr2=0x7faf840cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:46.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.361+0000 7faf8c432640 1 -- 172.21.15.88:0/2267551134 shutdown_connections 2023-10-16T20:39:46.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.361+0000 7faf8c432640 1 -- 172.21.15.88:0/2267551134 wait complete. 2023-10-16T20:39:46.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf8c432640 1 Processor -- start 2023-10-16T20:39:46.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf8c432640 1 -- start start 2023-10-16T20:39:46.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf8c432640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf840c4260 0x7faf8414d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf8c432640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf840c4640 0x7faf841469f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf8c432640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf840c5a60 0x7faf841ef9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf8c432640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf841f3870 con 0x7faf840c4640 2023-10-16T20:39:46.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf8c432640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8414dd60 con 0x7faf840c4260 2023-10-16T20:39:46.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf8c432640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8414e040 con 0x7faf840c5a60 2023-10-16T20:39:46.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf899cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf840c4640 0x7faf841469f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf899cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf840c4640 0x7faf841469f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33490/0 (socket says 172.21.15.88:33490) 2023-10-16T20:39:46.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf899cd640 1 -- 172.21.15.88:0/2003531383 learned_addr learned my addr 172.21.15.88:0/2003531383 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:46.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf8a1ce640 1 --2- 172.21.15.88:0/2003531383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf840c4260 0x7faf8414d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf8a9cf640 1 --2- 172.21.15.88:0/2003531383 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf840c5a60 0x7faf841ef9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf899cd640 1 -- 172.21.15.88:0/2003531383 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf840c5a60 msgr2=0x7faf841ef9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf899cd640 1 --2- 172.21.15.88:0/2003531383 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf840c5a60 0x7faf841ef9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf899cd640 1 -- 172.21.15.88:0/2003531383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf840c4260 msgr2=0x7faf8414d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf899cd640 1 --2- 172.21.15.88:0/2003531383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf840c4260 0x7faf8414d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf899cd640 1 -- 172.21.15.88:0/2003531383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf74009420 con 0x7faf840c4640 2023-10-16T20:39:46.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf8a1ce640 1 --2- 172.21.15.88:0/2003531383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf840c4260 0x7faf8414d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:39:46.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf899cd640 1 --2- 172.21.15.88:0/2003531383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf840c4640 0x7faf841469f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faf7400e9b0 tx=0x7faf7400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf7affd640 1 -- 172.21.15.88:0/2003531383 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7400c9a0 con 0x7faf840c4640 2023-10-16T20:39:46.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf7affd640 1 -- 172.21.15.88:0/2003531383 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf7400cb00 con 0x7faf840c4640 2023-10-16T20:39:46.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf7affd640 1 -- 172.21.15.88:0/2003531383 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf74013470 con 0x7faf840c4640 2023-10-16T20:39:46.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf8c432640 1 -- 172.21.15.88:0/2003531383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf80009420 con 0x7faf840c4640 2023-10-16T20:39:46.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.365+0000 7faf8c432640 1 -- 172.21.15.88:0/2003531383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf841f3220 con 0x7faf840c4640 2023-10-16T20:39:46.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.369+0000 7faf7affd640 1 -- 172.21.15.88:0/2003531383 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faf740135d0 con 0x7faf840c4640 2023-10-16T20:39:46.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.369+0000 7faf7affd640 1 --2- 172.21.15.88:0/2003531383 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faf680664a0 0x7faf68068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.369+0000 7faf7affd640 1 -- 172.21.15.88:0/2003531383 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7faf74084590 con 0x7faf840c4640 2023-10-16T20:39:46.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.369+0000 7faf8a1ce640 1 --2- 172.21.15.88:0/2003531383 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faf680664a0 0x7faf68068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.369+0000 7faf8f7b5000 1 Processor -- start 2023-10-16T20:39:46.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.369+0000 7faf8a1ce640 1 --2- 172.21.15.88:0/2003531383 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faf680664a0 0x7faf68068960 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7faf800ad000 tx=0x7faf80008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.369+0000 7faf8f7b5000 1 -- start start 2023-10-16T20:39:46.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8f7b5000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f6810 0x55a6e74f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8f7b5000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f7130 0x55a6e74f7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8f7b5000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f7a50 0x55a6e7501aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8f7b5000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6e72aacc0 con 0x55a6e74f7130 2023-10-16T20:39:46.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8f7b5000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6e730d920 con 0x55a6e74f6810 2023-10-16T20:39:46.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8f7b5000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a6e730ae50 con 0x55a6e74f7a50 2023-10-16T20:39:46.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf899cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f7130 0x55a6e74f7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf899cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f7130 0x55a6e74f7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33496/0 (socket says 172.21.15.88:33496) 2023-10-16T20:39:46.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf899cd640 1 -- 172.21.15.88:0/1707213037 learned_addr learned my addr 172.21.15.88:0/1707213037 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:46.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8a1ce640 1 --2- 172.21.15.88:0/1707213037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f6810 0x55a6e74f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8a9cf640 1 --2- 172.21.15.88:0/1707213037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f7a50 0x55a6e7501aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf899cd640 1 -- 172.21.15.88:0/1707213037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f7a50 msgr2=0x55a6e7501aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf899cd640 1 --2- 172.21.15.88:0/1707213037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f7a50 0x55a6e7501aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf899cd640 1 -- 172.21.15.88:0/1707213037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f6810 msgr2=0x55a6e74f6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf899cd640 1 --2- 172.21.15.88:0/1707213037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f6810 0x55a6e74f6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf899cd640 1 -- 172.21.15.88:0/1707213037 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6e6fff890 con 0x55a6e74f7130 2023-10-16T20:39:46.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf899cd640 1 --2- 172.21.15.88:0/1707213037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f7130 0x55a6e74f7510 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faf74009fd0 tx=0x7faf74055190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcca079e744f1e28 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf67fff640 1 -- 172.21.15.88:0/1707213037 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf74059170 con 0x55a6e74f7130 2023-10-16T20:39:46.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf67fff640 1 -- 172.21.15.88:0/1707213037 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf7405d570 con 0x55a6e74f7130 2023-10-16T20:39:46.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf67fff640 1 -- 172.21.15.88:0/1707213037 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf740519a0 con 0x55a6e74f7130 2023-10-16T20:39:46.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8f7b5000 1 -- 172.21.15.88:0/1707213037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f7130 msgr2=0x55a6e74f7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/1707213037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f7130 0x55a6e74f7510 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faf74009fd0 tx=0x7faf74055190 comp rx=0 tx=0).stop 2023-10-16T20:39:46.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8f7b5000 1 -- 172.21.15.88:0/1707213037 shutdown_connections 2023-10-16T20:39:46.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/1707213037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f7a50 0x55a6e7501aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/1707213037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f7130 0x55a6e74f7510 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/1707213037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f6810 0x55a6e74f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8f7b5000 1 -- 172.21.15.88:0/1707213037 >> 172.21.15.88:0/1707213037 conn(0x55a6e74acfc0 msgr2=0x55a6e74f45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:46.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8f7b5000 1 -- 172.21.15.88:0/1707213037 shutdown_connections 2023-10-16T20:39:46.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8f7b5000 1 -- 172.21.15.88:0/1707213037 wait complete. 2023-10-16T20:39:46.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8f7b5000 1 -- start start 2023-10-16T20:39:46.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.373+0000 7faf8f7b5000 1 Processor -- start 2023-10-16T20:39:46.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8f7b5000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f6810 0x55a6e74b8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8f7b5000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f7130 0x55a6e74b8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8f7b5000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f7a50 0x55a6e74b8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8f7b5000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6e7306740 con 0x55a6e74f7a50 2023-10-16T20:39:46.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8f7b5000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6e72c5d10 con 0x55a6e74f7130 2023-10-16T20:39:46.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8f7b5000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a6e72c59b0 con 0x55a6e74f6810 2023-10-16T20:39:46.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8f7b5000 10 client.0 _ll_register_callbacks cb 0x55a6e74a1390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T20:39:46.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8a9cf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f7a50 0x55a6e74b8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8a9cf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f7a50 0x55a6e74b8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33500/0 (socket says 172.21.15.88:33500) 2023-10-16T20:39:46.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8a9cf640 1 -- 172.21.15.88:0/3315427749 learned_addr learned my addr 172.21.15.88:0/3315427749 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:46.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8a1ce640 1 --2- 172.21.15.88:0/3315427749 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f6810 0x55a6e74b8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf899cd640 1 --2- 172.21.15.88:0/3315427749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f7130 0x55a6e74b8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8a9cf640 1 -- 172.21.15.88:0/3315427749 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f6810 msgr2=0x55a6e74b8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8a9cf640 1 --2- 172.21.15.88:0/3315427749 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f6810 0x55a6e74b8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8a9cf640 1 -- 172.21.15.88:0/3315427749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f7130 msgr2=0x55a6e74b8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8a9cf640 1 --2- 172.21.15.88:0/3315427749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f7130 0x55a6e74b8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8a9cf640 1 -- 172.21.15.88:0/3315427749 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf841f3220 con 0x55a6e74f7a50 2023-10-16T20:39:46.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8a9cf640 1 --2- 172.21.15.88:0/3315427749 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f7a50 0x55a6e74b8cc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faf7c007a60 tx=0x7faf7c011620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e322a67663363b3a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf657fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-10-16T20:39:46.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf657fa640 1 -- 172.21.15.88:0/3315427749 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7c0020e0 con 0x55a6e74f7a50 2023-10-16T20:39:46.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8f7b5000 1 -- 172.21.15.88:0/3315427749 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a6e72ba370 con 0x55a6e74f7a50 2023-10-16T20:39:46.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf657fa640 1 -- 172.21.15.88:0/3315427749 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7faf7c015470 con 0x55a6e74f7a50 2023-10-16T20:39:46.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf657fa640 1 -- 172.21.15.88:0/3315427749 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf7c015630 con 0x55a6e74f7a50 2023-10-16T20:39:46.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8f7b5000 10 client.4959 fetch_fsmap learned FSMap version 19 2023-10-16T20:39:46.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8f7b5000 1 -- 172.21.15.88:0/3315427749 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x55a6e72c8460 con 0x55a6e74f7a50 2023-10-16T20:39:46.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf657fa640 1 -- 172.21.15.88:0/3315427749 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a6e72ba370 con 0x55a6e74f7a50 2023-10-16T20:39:46.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf657fa640 1 -- 172.21.15.88:0/3315427749 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) v1 ==== 4500+0+0 (secure 0 0 0) 0x7faf7c015790 con 0x55a6e74f7a50 2023-10-16T20:39:46.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8f7b5000 10 client.4959 fetch_fsmap finished waiting for FSMap version 19 2023-10-16T20:39:46.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8f7b5000 10 client.4959 resolve_mds: resolved name 'f' to daemon mds.f 2023-10-16T20:39:46.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8f7b5000 20 client.4959 populate_metadata read hostname 'smithi088' 2023-10-16T20:39:46.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/3315427749 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x55a6e74b96d0 0x55a6e750b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8f7b5000 4 client.4959 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-16T20:39:46.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8f7b5000 1 -- 172.21.15.88:0/3315427749 --> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a6e72a02d0 con 0x55a6e74b96d0 2023-10-16T20:39:46.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8a1ce640 1 --2- 172.21.15.88:0/3315427749 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x55a6e74b96d0 0x55a6e750b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf657fa640 10 client.4959 ms_handle_connect on v2:172.21.15.191:6834/3059699574 2023-10-16T20:39:46.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf8a1ce640 1 --2- 172.21.15.88:0/3315427749 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x55a6e74b96d0 0x55a6e750b900 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=581e015ed3616372 server_cookie=1f02baff5c7b2dd1 in_seq=0 out_seq=0 2023-10-16T20:39:46.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf657fa640 1 -- 172.21.15.88:0/3315427749 <== mds.0 v2:172.21.15.191:6834/3059699574 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a6e72a02d0 con 0x55a6e74b96d0 2023-10-16T20:39:46.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.377+0000 7faf657fa640 10 client.4959 handle_command_reply: tid=0 2023-10-16T20:39:46.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.381+0000 7faf8f7b5000 1 client.4959 shutdown 2023-10-16T20:39:46.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.381+0000 7faf8f7b5000 1 -- 172.21.15.88:0/3315427749 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f7a50 msgr2=0x55a6e74b8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.381+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/3315427749 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f7a50 0x55a6e74b8cc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faf7c007a60 tx=0x7faf7c011620 comp rx=0 tx=0).stop 2023-10-16T20:39:46.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.381+0000 7faf8f7b5000 1 -- 172.21.15.88:0/3315427749 shutdown_connections 2023-10-16T20:39:46.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.381+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/3315427749 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f7a50 0x55a6e74b8cc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.381+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/3315427749 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f7130 0x55a6e74b8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.381+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/3315427749 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x55a6e74b96d0 0x55a6e750b900 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.381+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/3315427749 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f6810 0x55a6e74b8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.381+0000 7faf657fa640 0 client.4959 ms_handle_reset on v2:172.21.15.191:6834/3059699574 2023-10-16T20:39:46.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.381+0000 7faf8f7b5000 1 -- 172.21.15.88:0/3315427749 >> 172.21.15.88:0/3315427749 conn(0x55a6e74acfc0 msgr2=0x55a6e74f6090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:46.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.381+0000 7faf8f7b5000 1 -- 172.21.15.88:0/3315427749 shutdown_connections 2023-10-16T20:39:46.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.381+0000 7faf8f7b5000 1 -- 172.21.15.88:0/3315427749 wait complete. 2023-10-16T20:39:46.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.381+0000 7faf8f7b5000 20 client.4959 trim_cache size 0 max 16384 2023-10-16T20:39:46.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.393+0000 7faf8f7b5000 1 Processor -- start 2023-10-16T20:39:46.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.393+0000 7faf8f7b5000 1 -- start start 2023-10-16T20:39:46.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8f7b5000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f6810 0x55a6e7551440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8f7b5000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f7130 0x55a6e7551980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8f7b5000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f7a50 0x55a6e74b4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8f7b5000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6e6fc0940 con 0x55a6e74f6810 2023-10-16T20:39:46.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8f7b5000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6e750fb50 con 0x55a6e74f7a50 2023-10-16T20:39:46.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8f7b5000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a6e730a740 con 0x55a6e74f7130 2023-10-16T20:39:46.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8a1ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f6810 0x55a6e7551440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8a1ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f6810 0x55a6e7551440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33514/0 (socket says 172.21.15.88:33514) 2023-10-16T20:39:46.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8a1ce640 1 -- 172.21.15.88:0/399088816 learned_addr learned my addr 172.21.15.88:0/399088816 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:46.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf899cd640 1 --2- 172.21.15.88:0/399088816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f7130 0x55a6e7551980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8a9cf640 1 --2- 172.21.15.88:0/399088816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f7a50 0x55a6e74b4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8a1ce640 1 -- 172.21.15.88:0/399088816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f7130 msgr2=0x55a6e7551980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8a1ce640 1 --2- 172.21.15.88:0/399088816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f7130 0x55a6e7551980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8a1ce640 1 -- 172.21.15.88:0/399088816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f7a50 msgr2=0x55a6e74b4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8a1ce640 1 --2- 172.21.15.88:0/399088816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f7a50 0x55a6e74b4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8a1ce640 1 -- 172.21.15.88:0/399088816 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6e72ae760 con 0x55a6e74f6810 2023-10-16T20:39:46.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8a1ce640 1 --2- 172.21.15.88:0/399088816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f6810 0x55a6e7551440 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7faf80008010 tx=0x7faf800a6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2ef6132d828e7b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf78ff9640 1 -- 172.21.15.88:0/399088816 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf800c9b50 con 0x55a6e74f6810 2023-10-16T20:39:46.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf78ff9640 1 -- 172.21.15.88:0/399088816 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf800bd040 con 0x55a6e74f6810 2023-10-16T20:39:46.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf78ff9640 1 -- 172.21.15.88:0/399088816 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf800c9cb0 con 0x55a6e74f6810 2023-10-16T20:39:46.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8f7b5000 1 -- 172.21.15.88:0/399088816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f6810 msgr2=0x55a6e7551440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/399088816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f6810 0x55a6e7551440 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7faf80008010 tx=0x7faf800a6000 comp rx=0 tx=0).stop 2023-10-16T20:39:46.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8f7b5000 1 -- 172.21.15.88:0/399088816 shutdown_connections 2023-10-16T20:39:46.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/399088816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f7a50 0x55a6e74b4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/399088816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f7130 0x55a6e7551980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/399088816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f6810 0x55a6e7551440 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8f7b5000 1 -- 172.21.15.88:0/399088816 >> 172.21.15.88:0/399088816 conn(0x55a6e74acfc0 msgr2=0x55a6e74f3280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:46.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8f7b5000 1 -- 172.21.15.88:0/399088816 shutdown_connections 2023-10-16T20:39:46.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8f7b5000 1 -- 172.21.15.88:0/399088816 wait complete. 2023-10-16T20:39:46.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8f7b5000 1 -- start start 2023-10-16T20:39:46.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.397+0000 7faf8f7b5000 1 Processor -- start 2023-10-16T20:39:46.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f6810 0x55a6e755dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f7130 0x55a6e7554300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f7a50 0x55a6e7554840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6e7300c20 con 0x55a6e74f6810 2023-10-16T20:39:46.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6e755e500 con 0x55a6e74f7a50 2023-10-16T20:39:46.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a6e72e25a0 con 0x55a6e74f7130 2023-10-16T20:39:46.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 10 client.0 _ll_register_callbacks cb 0x7faf7c0018c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-16T20:39:46.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8a1ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f6810 0x55a6e755dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8a1ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f6810 0x55a6e755dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33526/0 (socket says 172.21.15.88:33526) 2023-10-16T20:39:46.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8a1ce640 1 -- 172.21.15.88:0/1808312477 learned_addr learned my addr 172.21.15.88:0/1808312477 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:46.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf899cd640 1 --2- 172.21.15.88:0/1808312477 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f7130 0x55a6e7554300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8a9cf640 1 --2- 172.21.15.88:0/1808312477 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f7a50 0x55a6e7554840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8a1ce640 1 -- 172.21.15.88:0/1808312477 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f7130 msgr2=0x55a6e7554300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8a1ce640 1 --2- 172.21.15.88:0/1808312477 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f7130 0x55a6e7554300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8a1ce640 1 -- 172.21.15.88:0/1808312477 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f7a50 msgr2=0x55a6e7554840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8a1ce640 1 --2- 172.21.15.88:0/1808312477 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f7a50 0x55a6e7554840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8a1ce640 1 -- 172.21.15.88:0/1808312477 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf841581e0 con 0x55a6e74f6810 2023-10-16T20:39:46.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8a9cf640 1 --2- 172.21.15.88:0/1808312477 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f7a50 0x55a6e7554840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:46.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf899cd640 1 --2- 172.21.15.88:0/1808312477 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f7130 0x55a6e7554300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:46.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8a1ce640 1 --2- 172.21.15.88:0/1808312477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f6810 0x55a6e755dfc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7faf80008010 tx=0x7faf8001ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef833b5563837c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf657fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-10-16T20:39:46.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf657fa640 1 -- 172.21.15.88:0/1808312477 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf800c9d90 con 0x55a6e74f6810 2023-10-16T20:39:46.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf657fa640 1 -- 172.21.15.88:0/1808312477 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7faf800b6a50 con 0x55a6e74f6810 2023-10-16T20:39:46.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 -- 172.21.15.88:0/1808312477 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a6e74b9090 con 0x55a6e74f6810 2023-10-16T20:39:46.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf657fa640 1 -- 172.21.15.88:0/1808312477 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf800b6c10 con 0x55a6e74f6810 2023-10-16T20:39:46.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 10 client.4965 fetch_fsmap learned FSMap version 19 2023-10-16T20:39:46.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 -- 172.21.15.88:0/1808312477 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x55a6e6fffcc0 con 0x55a6e74f6810 2023-10-16T20:39:46.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf657fa640 1 -- 172.21.15.88:0/1808312477 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a6e74b9090 con 0x55a6e74f6810 2023-10-16T20:39:46.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf657fa640 1 -- 172.21.15.88:0/1808312477 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) v1 ==== 4500+0+0 (secure 0 0 0) 0x7faf800c2040 con 0x55a6e74f6810 2023-10-16T20:39:46.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 10 client.4965 fetch_fsmap finished waiting for FSMap version 19 2023-10-16T20:39:46.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 10 client.4965 resolve_mds: resolved name 'f' to daemon mds.f 2023-10-16T20:39:46.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 20 client.4965 populate_metadata read hostname 'smithi088' 2023-10-16T20:39:46.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/1808312477 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x55a6e755bad0 0x55a6e755beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 4 client.4965 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-16T20:39:46.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 -- 172.21.15.88:0/1808312477 --> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a6e729e6f0 con 0x55a6e755bad0 2023-10-16T20:39:46.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf899cd640 1 --2- 172.21.15.88:0/1808312477 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x55a6e755bad0 0x55a6e755beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf657fa640 10 client.4965 ms_handle_connect on v2:172.21.15.191:6834/3059699574 2023-10-16T20:39:46.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf899cd640 1 --2- 172.21.15.88:0/1808312477 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x55a6e755bad0 0x55a6e755beb0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=657ba5c41b92e6ff server_cookie=a3630f02eb535301 in_seq=0 out_seq=0 2023-10-16T20:39:46.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf657fa640 1 -- 172.21.15.88:0/1808312477 <== mds.0 v2:172.21.15.191:6834/3059699574 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x55a6e729e6f0 con 0x55a6e755bad0 2023-10-16T20:39:46.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf657fa640 10 client.4965 handle_command_reply: tid=0 2023-10-16T20:39:46.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 client.4965 shutdown 2023-10-16T20:39:46.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 -- 172.21.15.88:0/1808312477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f6810 msgr2=0x55a6e755dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/1808312477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f6810 0x55a6e755dfc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7faf80008010 tx=0x7faf8001ab70 comp rx=0 tx=0).stop 2023-10-16T20:39:46.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 -- 172.21.15.88:0/1808312477 shutdown_connections 2023-10-16T20:39:46.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/1808312477 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55a6e74f7a50 0x55a6e7554840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/1808312477 >> [v2:172.21.15.191:6834/3059699574,v1:172.21.15.191:6835/3059699574] conn(0x55a6e755bad0 0x55a6e755beb0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/1808312477 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x55a6e74f7130 0x55a6e7554300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 --2- 172.21.15.88:0/1808312477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6e74f6810 0x55a6e755dfc0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf657fa640 0 client.4965 ms_handle_reset on v2:172.21.15.191:6834/3059699574 2023-10-16T20:39:46.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 -- 172.21.15.88:0/1808312477 >> 172.21.15.88:0/1808312477 conn(0x55a6e74acfc0 msgr2=0x55a6e74b3760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:46.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 -- 172.21.15.88:0/1808312477 shutdown_connections 2023-10-16T20:39:46.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 1 -- 172.21.15.88:0/1808312477 wait complete. 2023-10-16T20:39:46.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.401+0000 7faf8f7b5000 20 client.4965 trim_cache size 0 max 16384 2023-10-16T20:39:46.419 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-10-16T20:39:46.419 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "8cf7342f-a394-4b4c-ac35-9931ac9d9e57", 2023-10-16T20:39:46.419 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2023-10-16T20:39:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 4288, 2023-10-16T20:39:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2023-10-16T20:39:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2023-10-16T20:39:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2023-10-16T20:39:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 170.66117401299999, 2023-10-16T20:39:46.420 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 19, 2023-10-16T20:39:46.421 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 29, 2023-10-16T20:39:46.421 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 29, 2023-10-16T20:39:46.421 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 177.487427279 2023-10-16T20:39:46.421 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-10-16T20:39:46.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.405+0000 7faf8c432640 1 -- 172.21.15.88:0/2003531383 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faf680664a0 msgr2=0x7faf68068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.405+0000 7faf8c432640 1 --2- 172.21.15.88:0/2003531383 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faf680664a0 0x7faf68068960 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7faf800ad000 tx=0x7faf80008040 comp rx=0 tx=0).stop 2023-10-16T20:39:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.405+0000 7faf8c432640 1 -- 172.21.15.88:0/2003531383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf840c4640 msgr2=0x7faf841469f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.405+0000 7faf8c432640 1 --2- 172.21.15.88:0/2003531383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf840c4640 0x7faf841469f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faf7400e9b0 tx=0x7faf7400ee70 comp rx=0 tx=0).stop 2023-10-16T20:39:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.405+0000 7faf8c432640 1 -- 172.21.15.88:0/2003531383 shutdown_connections 2023-10-16T20:39:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.405+0000 7faf8c432640 1 --2- 172.21.15.88:0/2003531383 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faf680664a0 0x7faf68068960 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.405+0000 7faf8c432640 1 --2- 172.21.15.88:0/2003531383 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf840c5a60 0x7faf841ef9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.405+0000 7faf8c432640 1 --2- 172.21.15.88:0/2003531383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf840c4640 0x7faf841469f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.405+0000 7faf8c432640 1 --2- 172.21.15.88:0/2003531383 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf840c4260 0x7faf8414d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.405+0000 7faf8c432640 1 -- 172.21.15.88:0/2003531383 >> 172.21.15.88:0/2003531383 conn(0x7faf840ca9f0 msgr2=0x7faf840700a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:46.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.405+0000 7faf8c432640 1 -- 172.21.15.88:0/2003531383 shutdown_connections 2023-10-16T20:39:46.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.405+0000 7faf8c432640 1 -- 172.21.15.88:0/2003531383 wait complete. 2023-10-16T20:39:46.426 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-10-16T20:39:46.437 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-10-16T20:39:46.437 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2023-10-16T20:39:46.438 INFO:tasks.ceph_fuse:client.0 config is {} 2023-10-16T20:39:46.438 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-16T20:39:46.438 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T20:39:46.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:46.439 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2023-10-16T20:39:46.449 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-10-16T20:39:46.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:46.450 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-16T20:39:46.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:46.515 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link delete ceph-brx 2023-10-16T20:39:46.690 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-10-16T20:39:46.690 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-16T20:39:46.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99bb6d6640 1 Processor -- start 2023-10-16T20:39:46.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99bb6d6640 1 -- start start 2023-10-16T20:39:46.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99bb6d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99b4153770 0x7f99b4159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99bb6d6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f99b40c9bb0 0x7f99b40c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99bb6d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f99b40c41f0 0x7f99b40cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99bb6d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b40cc250 con 0x7f99b40c41f0 2023-10-16T20:39:46.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99bb6d6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b40cc670 con 0x7f99b40c9bb0 2023-10-16T20:39:46.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99bb6d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b415a410 con 0x7f99b4153770 2023-10-16T20:39:46.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99b9c73640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f99b40c41f0 0x7f99b40cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99b9c73640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f99b40c41f0 0x7f99b40cece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33534/0 (socket says 172.21.15.88:33534) 2023-10-16T20:39:46.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99b9c73640 1 -- 172.21.15.88:0/3160436105 learned_addr learned my addr 172.21.15.88:0/3160436105 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:46.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99b8c71640 1 --2- 172.21.15.88:0/3160436105 >> v2:172.21.15.191:3300/0 conn(0x7f99b40c9bb0 0x7f99b40c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99b9472640 1 --2- 172.21.15.88:0/3160436105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99b4153770 0x7f99b4159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99b9c73640 1 -- 172.21.15.88:0/3160436105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99b4153770 msgr2=0x7f99b4159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99b9c73640 1 --2- 172.21.15.88:0/3160436105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99b4153770 0x7f99b4159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99b9c73640 1 -- 172.21.15.88:0/3160436105 >> v2:172.21.15.191:3300/0 conn(0x7f99b40c9bb0 msgr2=0x7f99b40c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:46.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99b9c73640 1 --2- 172.21.15.88:0/3160436105 >> v2:172.21.15.191:3300/0 conn(0x7f99b40c9bb0 0x7f99b40c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.825+0000 7f99b9c73640 1 -- 172.21.15.88:0/3160436105 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b4153b50 con 0x7f99b40c41f0 2023-10-16T20:39:46.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.829+0000 7f99b9c73640 1 --2- 172.21.15.88:0/3160436105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40c41f0 0x7f99b40cece0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f99b40cc810 tx=0x7f99ac01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e906fa01f148b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.829+0000 7f99abfff640 1 -- 172.21.15.88:0/3160436105 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ac0a5760 con 0x7f99b40c41f0 2023-10-16T20:39:46.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.829+0000 7f99abfff640 1 -- 172.21.15.88:0/3160436105 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ac0a58c0 con 0x7f99b40c41f0 2023-10-16T20:39:46.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.829+0000 7f99abfff640 1 -- 172.21.15.88:0/3160436105 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ac0a5c50 con 0x7f99b40c41f0 2023-10-16T20:39:46.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.829+0000 7f99bb6d6640 1 -- 172.21.15.88:0/3160436105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40c41f0 msgr2=0x7f99b40cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.829+0000 7f99bb6d6640 1 --2- 172.21.15.88:0/3160436105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40c41f0 0x7f99b40cece0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f99b40cc810 tx=0x7f99ac01c9d0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.829+0000 7f99bb6d6640 1 -- 172.21.15.88:0/3160436105 shutdown_connections 2023-10-16T20:39:46.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.829+0000 7f99bb6d6640 1 --2- 172.21.15.88:0/3160436105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40c41f0 0x7f99b40cece0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.829+0000 7f99bb6d6640 1 --2- 172.21.15.88:0/3160436105 >> v2:172.21.15.191:3300/0 conn(0x7f99b40c9bb0 0x7f99b40c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.829+0000 7f99bb6d6640 1 --2- 172.21.15.88:0/3160436105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99b4153770 0x7f99b4159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.829+0000 7f99bb6d6640 1 -- 172.21.15.88:0/3160436105 >> 172.21.15.88:0/3160436105 conn(0x7f99b40715c0 msgr2=0x7f99b40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:46.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.829+0000 7f99bb6d6640 1 -- 172.21.15.88:0/3160436105 shutdown_connections 2023-10-16T20:39:46.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.829+0000 7f99bb6d6640 1 -- 172.21.15.88:0/3160436105 wait complete. 2023-10-16T20:39:46.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99bb6d6640 1 Processor -- start 2023-10-16T20:39:46.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99bb6d6640 1 -- start start 2023-10-16T20:39:46.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99bb6d6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99b40c41f0 0x7f99b41511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99bb6d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40c9bb0 0x7f99b40b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99bb6d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99b4153770 0x7f99b40c6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99bb6d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b41503a0 con 0x7f99b40c9bb0 2023-10-16T20:39:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99bb6d6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b4150680 con 0x7f99b40c41f0 2023-10-16T20:39:46.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99bb6d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b4150960 con 0x7f99b4153770 2023-10-16T20:39:46.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99b8c71640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40c9bb0 0x7f99b40b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99b8c71640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40c9bb0 0x7f99b40b3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33542/0 (socket says 172.21.15.88:33542) 2023-10-16T20:39:46.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99b8c71640 1 -- 172.21.15.88:0/179706425 learned_addr learned my addr 172.21.15.88:0/179706425 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:46.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99b9472640 1 --2- 172.21.15.88:0/179706425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99b40c41f0 0x7f99b41511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99b9c73640 1 --2- 172.21.15.88:0/179706425 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99b4153770 0x7f99b40c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99b8c71640 1 -- 172.21.15.88:0/179706425 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99b4153770 msgr2=0x7f99b40c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99b8c71640 1 --2- 172.21.15.88:0/179706425 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99b4153770 0x7f99b40c6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99b8c71640 1 -- 172.21.15.88:0/179706425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99b40c41f0 msgr2=0x7f99b41511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:46.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99b8c71640 1 --2- 172.21.15.88:0/179706425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99b40c41f0 0x7f99b41511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:46.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99b8c71640 1 -- 172.21.15.88:0/179706425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ac009b30 con 0x7f99b40c9bb0 2023-10-16T20:39:46.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99b8c71640 1 --2- 172.21.15.88:0/179706425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40c9bb0 0x7f99b40b3560 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f99a400d800 tx=0x7f99a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99a9ffb640 1 -- 172.21.15.88:0/179706425 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a40096a0 con 0x7f99b40c9bb0 2023-10-16T20:39:46.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99a9ffb640 1 -- 172.21.15.88:0/179706425 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a4009800 con 0x7f99b40c9bb0 2023-10-16T20:39:46.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99a9ffb640 1 -- 172.21.15.88:0/179706425 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4009990 con 0x7f99b40c9bb0 2023-10-16T20:39:46.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99bb6d6640 1 -- 172.21.15.88:0/179706425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b414f480 con 0x7f99b40c9bb0 2023-10-16T20:39:46.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.833+0000 7f99bb6d6640 1 -- 172.21.15.88:0/179706425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b414fa40 con 0x7f99b40c9bb0 2023-10-16T20:39:46.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.837+0000 7f99bb6d6640 1 -- 172.21.15.88:0/179706425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b4074700 con 0x7f99b40c9bb0 2023-10-16T20:39:46.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.837+0000 7f99a9ffb640 1 -- 172.21.15.88:0/179706425 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f99a4009af0 con 0x7f99b40c9bb0 2023-10-16T20:39:46.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.837+0000 7f99a9ffb640 1 --2- 172.21.15.88:0/179706425 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f99980664a0 0x7f9998068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:46.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.837+0000 7f99a9ffb640 1 -- 172.21.15.88:0/179706425 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f99a4084550 con 0x7f99b40c9bb0 2023-10-16T20:39:46.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.837+0000 7f99b9472640 1 --2- 172.21.15.88:0/179706425 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f99980664a0 0x7f9998068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:46.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.837+0000 7f99b9472640 1 --2- 172.21.15.88:0/179706425 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f99980664a0 0x7f9998068960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f99b0006fd0 tx=0x7f99b0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:46.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:46.841+0000 7f99a9ffb640 1 -- 172.21.15.88:0/179706425 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f99a40516c0 con 0x7f99b40c9bb0 2023-10-16T20:39:47.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:47.021+0000 7f99bb6d6640 1 -- 172.21.15.88:0/179706425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f99b40cf900 con 0x7f99b40c9bb0 2023-10-16T20:39:47.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:47.021+0000 7f99a9ffb640 1 -- 172.21.15.88:0/179706425 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+88 (secure 0 0 0) 0x7f99a4054ce0 con 0x7f99b40c9bb0 2023-10-16T20:39:47.027 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-16T20:39:47.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:47.025+0000 7f99bb6d6640 1 -- 172.21.15.88:0/179706425 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f99980664a0 msgr2=0x7f9998068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:47.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:47.025+0000 7f99bb6d6640 1 --2- 172.21.15.88:0/179706425 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f99980664a0 0x7f9998068960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f99b0006fd0 tx=0x7f99b0008040 comp rx=0 tx=0).stop 2023-10-16T20:39:47.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:47.025+0000 7f99bb6d6640 1 -- 172.21.15.88:0/179706425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40c9bb0 msgr2=0x7f99b40b3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:47.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:47.025+0000 7f99bb6d6640 1 --2- 172.21.15.88:0/179706425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40c9bb0 0x7f99b40b3560 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f99a400d800 tx=0x7f99a400dcc0 comp rx=0 tx=0).stop 2023-10-16T20:39:47.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:47.025+0000 7f99bb6d6640 1 -- 172.21.15.88:0/179706425 shutdown_connections 2023-10-16T20:39:47.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:47.025+0000 7f99bb6d6640 1 --2- 172.21.15.88:0/179706425 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f99980664a0 0x7f9998068960 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:47.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:47.025+0000 7f99bb6d6640 1 --2- 172.21.15.88:0/179706425 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f99b4153770 0x7f99b40c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:47.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:47.025+0000 7f99bb6d6640 1 --2- 172.21.15.88:0/179706425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b40c9bb0 0x7f99b40b3560 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:47.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:47.025+0000 7f99bb6d6640 1 --2- 172.21.15.88:0/179706425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f99b40c41f0 0x7f99b41511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:47.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:47.025+0000 7f99bb6d6640 1 -- 172.21.15.88:0/179706425 >> 172.21.15.88:0/179706425 conn(0x7f99b40715c0 msgr2=0x7f99b4159aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:47.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:47.025+0000 7f99bb6d6640 1 -- 172.21.15.88:0/179706425 shutdown_connections 2023-10-16T20:39:47.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:47.025+0000 7f99bb6d6640 1 -- 172.21.15.88:0/179706425 wait complete. 2023-10-16T20:39:47.046 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-16T20:39:47.047 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-16T20:39:47.047 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2023-10-16T20:39:47.047 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-16T20:39:47.047 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-16T20:39:47.048 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-16T20:39:47.048 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-16T20:39:47.048 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-16T20:39:47.048 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-16T20:39:47.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:47.049 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2023-10-16T20:39:47.059 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-16T20:39:47.059 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-16T20:39:47.059 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2023-10-16T20:39:47.059 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2023-10-16T20:39:47.060 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2023-10-16T20:39:47.060 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2023-10-16T20:39:47.060 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-16T20:39:47.060 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2023-10-16T20:39:47.060 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2023-10-16T20:39:47.061 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-16T20:39:47.061 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2023-10-16T20:39:47.061 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2023-10-16T20:39:47.061 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-16T20:39:47.061 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2023-10-16T20:39:47.061 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-16T20:39:47.062 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2023-10-16T20:39:47.062 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-10-16T20:39:47.062 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42269sec preferred_lft 42269sec 2023-10-16T20:39:47.062 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link 2023-10-16T20:39:47.062 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2023-10-16T20:39:47.062 INFO:teuthology.orchestra.run.smithi088.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-10-16T20:39:47.063 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 02:42:34:f9:43:24 brd ff:ff:ff:ff:ff:ff 2023-10-16T20:39:47.063 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-10-16T20:39:47.063 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2023-10-16T20:39:47.063 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-16T20:39:47.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:47.064 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-10-16T20:39:47.064 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-10-16T20:39:47.065 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link add name ceph-brx type bridge 2023-10-16T20:39:47.065 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip addr flush dev ceph-brx 2023-10-16T20:39:47.065 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set ceph-brx up 2023-10-16T20:39:47.065 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-16T20:39:47.066 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-10-16T20:39:47.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:47.178 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-16T20:39:47.197 INFO:teuthology.orchestra.run.smithi088.stdout:1 2023-10-16T20:39:47.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:47.200 DEBUG:teuthology.orchestra.run.smithi088:> route 2023-10-16T20:39:47.253 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2023-10-16T20:39:47.253 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-16T20:39:47.253 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-10-16T20:39:47.254 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-16T20:39:47.254 INFO:teuthology.orchestra.run.smithi088.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-10-16T20:39:47.254 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-16T20:39:47.254 INFO:teuthology.orchestra.run.smithi088.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-16T20:39:47.254 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-16T20:39:47.255 INFO:teuthology.orchestra.run.smithi088.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-16T20:39:47.255 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-16T20:39:47.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:47.256 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-10-16T20:39:47.256 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-10-16T20:39:47.256 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-16T20:39:47.256 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-16T20:39:47.257 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-16T20:39:47.257 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-10-16T20:39:47.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:47.347 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2023-10-16T20:39:47.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:47.358 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list-id 2023-10-16T20:39:47.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:47.407 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-10-16T20:39:47.408 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-10-16T20:39:47.408 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T20:39:47.408 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-16T20:39:47.408 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-10-16T20:39:47.492 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-16T20:39:47.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:47.493 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-10-16T20:39:47.493 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-10-16T20:39:47.493 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-16T20:39:47.494 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-16T20:39:47.494 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-16T20:39:47.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-16T20:39:47.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-16T20:39:47.495 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-10-16T20:39:47.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:47.630 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-10-16T20:39:47.631 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-10-16T20:39:47.631 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2023-10-16T20:39:47.631 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set dev brx.0 master ceph-brx 2023-10-16T20:39:47.632 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-10-16T20:39:47.672 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-10-16T20:39:47.672 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T20:39:47.672 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-16T20:39:47.727 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-16T20:39:47.727 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-16T20:39:47.728 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-16T20:39:47.782 DEBUG:teuthology.orchestra.run.smithi088:> sudo modprobe fuse 2023-10-16T20:39:47.847 DEBUG:teuthology.orchestra.run.smithi088:> cat /proc/self/mounts | awk '{print $2}' 2023-10-16T20:39:47.900 INFO:teuthology.orchestra.run.smithi088.stdout:/sys 2023-10-16T20:39:47.900 INFO:teuthology.orchestra.run.smithi088.stdout:/proc 2023-10-16T20:39:47.900 INFO:teuthology.orchestra.run.smithi088.stdout:/dev 2023-10-16T20:39:47.901 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/pts 2023-10-16T20:39:47.901 INFO:teuthology.orchestra.run.smithi088.stdout:/run 2023-10-16T20:39:47.901 INFO:teuthology.orchestra.run.smithi088.stdout:/ 2023-10-16T20:39:47.901 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/security 2023-10-16T20:39:47.901 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/shm 2023-10-16T20:39:47.902 INFO:teuthology.orchestra.run.smithi088.stdout:/run/lock 2023-10-16T20:39:47.902 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/cgroup 2023-10-16T20:39:47.902 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/pstore 2023-10-16T20:39:47.902 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/bpf 2023-10-16T20:39:47.902 INFO:teuthology.orchestra.run.smithi088.stdout:/proc/sys/fs/binfmt_misc 2023-10-16T20:39:47.903 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/hugepages 2023-10-16T20:39:47.903 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/mqueue 2023-10-16T20:39:47.903 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/debug 2023-10-16T20:39:47.903 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/tracing 2023-10-16T20:39:47.903 INFO:teuthology.orchestra.run.smithi088.stdout:/proc/fs/nfsd 2023-10-16T20:39:47.904 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/fuse/connections 2023-10-16T20:39:47.904 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/config 2023-10-16T20:39:47.904 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-sysusers.service 2023-10-16T20:39:47.904 INFO:teuthology.orchestra.run.smithi088.stdout:/run/rpc_pipefs 2023-10-16T20:39:47.904 INFO:teuthology.orchestra.run.smithi088.stdout:/run/user/1000 2023-10-16T20:39:47.905 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph 2023-10-16T20:39:47.905 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-0 2023-10-16T20:39:47.905 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-1 2023-10-16T20:39:47.905 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-2 2023-10-16T20:39:47.905 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-3 2023-10-16T20:39:47.906 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns 2023-10-16T20:39:47.906 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T20:39:47.906 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T20:39:47.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:47.907 DEBUG:teuthology.orchestra.run.smithi088:> ls /sys/fs/fuse/connections 2023-10-16T20:39:47.958 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-16T20:39:47.959 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-10-16T20:39:48.005 DEBUG:teuthology.orchestra.run.smithi088:> sudo modprobe fuse 2023-10-16T20:39:48.026 DEBUG:teuthology.orchestra.run.smithi088:> cat /proc/self/mounts | awk '{print $2}' 2023-10-16T20:39:48.081 INFO:teuthology.orchestra.run.smithi088.stdout:/sys 2023-10-16T20:39:48.081 INFO:teuthology.orchestra.run.smithi088.stdout:/proc 2023-10-16T20:39:48.081 INFO:teuthology.orchestra.run.smithi088.stdout:/dev 2023-10-16T20:39:48.082 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/pts 2023-10-16T20:39:48.082 INFO:teuthology.orchestra.run.smithi088.stdout:/run 2023-10-16T20:39:48.082 INFO:teuthology.orchestra.run.smithi088.stdout:/ 2023-10-16T20:39:48.082 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/security 2023-10-16T20:39:48.082 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/shm 2023-10-16T20:39:48.083 INFO:teuthology.orchestra.run.smithi088.stdout:/run/lock 2023-10-16T20:39:48.083 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/cgroup 2023-10-16T20:39:48.083 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/pstore 2023-10-16T20:39:48.083 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/bpf 2023-10-16T20:39:48.083 INFO:teuthology.orchestra.run.smithi088.stdout:/proc/sys/fs/binfmt_misc 2023-10-16T20:39:48.083 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/hugepages 2023-10-16T20:39:48.084 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/mqueue 2023-10-16T20:39:48.084 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/debug 2023-10-16T20:39:48.084 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/tracing 2023-10-16T20:39:48.084 INFO:teuthology.orchestra.run.smithi088.stdout:/proc/fs/nfsd 2023-10-16T20:39:48.084 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/fuse/connections 2023-10-16T20:39:48.085 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/config 2023-10-16T20:39:48.085 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-sysusers.service 2023-10-16T20:39:48.085 INFO:teuthology.orchestra.run.smithi088.stdout:/run/rpc_pipefs 2023-10-16T20:39:48.085 INFO:teuthology.orchestra.run.smithi088.stdout:/run/user/1000 2023-10-16T20:39:48.085 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph 2023-10-16T20:39:48.085 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-0 2023-10-16T20:39:48.086 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-1 2023-10-16T20:39:48.086 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-2 2023-10-16T20:39:48.086 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-3 2023-10-16T20:39:48.086 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns 2023-10-16T20:39:48.087 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T20:39:48.087 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T20:39:48.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:48.088 DEBUG:teuthology.orchestra.run.smithi088:> ls /sys/fs/fuse/connections 2023-10-16T20:39:48.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi088.stderr:2023-10-16T20:39:48.097+0000 7f695be44580 -1 init, newargv = 0x55ba2e778cc0 newargc=15 2023-10-16T20:39:48.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi088.stderr:ceph-fuse[46445]: starting ceph client 2023-10-16T20:39:48.656 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi088.stderr:ceph-fuse[46445]: starting fuse 2023-10-16T20:39:49.210 DEBUG:teuthology.orchestra.run.smithi088:> sudo modprobe fuse 2023-10-16T20:39:49.234 DEBUG:teuthology.orchestra.run.smithi088:> cat /proc/self/mounts | awk '{print $2}' 2023-10-16T20:39:49.289 INFO:teuthology.orchestra.run.smithi088.stdout:/sys 2023-10-16T20:39:49.289 INFO:teuthology.orchestra.run.smithi088.stdout:/proc 2023-10-16T20:39:49.289 INFO:teuthology.orchestra.run.smithi088.stdout:/dev 2023-10-16T20:39:49.290 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/pts 2023-10-16T20:39:49.290 INFO:teuthology.orchestra.run.smithi088.stdout:/run 2023-10-16T20:39:49.290 INFO:teuthology.orchestra.run.smithi088.stdout:/ 2023-10-16T20:39:49.290 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/security 2023-10-16T20:39:49.290 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/shm 2023-10-16T20:39:49.291 INFO:teuthology.orchestra.run.smithi088.stdout:/run/lock 2023-10-16T20:39:49.291 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/cgroup 2023-10-16T20:39:49.291 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/pstore 2023-10-16T20:39:49.291 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/bpf 2023-10-16T20:39:49.291 INFO:teuthology.orchestra.run.smithi088.stdout:/proc/sys/fs/binfmt_misc 2023-10-16T20:39:49.292 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/hugepages 2023-10-16T20:39:49.292 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/mqueue 2023-10-16T20:39:49.292 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/debug 2023-10-16T20:39:49.292 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/tracing 2023-10-16T20:39:49.292 INFO:teuthology.orchestra.run.smithi088.stdout:/proc/fs/nfsd 2023-10-16T20:39:49.292 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/fuse/connections 2023-10-16T20:39:49.293 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/config 2023-10-16T20:39:49.293 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-sysusers.service 2023-10-16T20:39:49.293 INFO:teuthology.orchestra.run.smithi088.stdout:/run/rpc_pipefs 2023-10-16T20:39:49.293 INFO:teuthology.orchestra.run.smithi088.stdout:/run/user/1000 2023-10-16T20:39:49.293 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph 2023-10-16T20:39:49.294 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-0 2023-10-16T20:39:49.294 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-1 2023-10-16T20:39:49.294 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-2 2023-10-16T20:39:49.294 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-3 2023-10-16T20:39:49.294 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns 2023-10-16T20:39:49.295 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T20:39:49.295 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T20:39:49.295 INFO:teuthology.orchestra.run.smithi088.stdout:/home/ubuntu/cephtest/mnt.0 2023-10-16T20:39:49.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:49.296 DEBUG:teuthology.orchestra.run.smithi088:> ls /sys/fs/fuse/connections 2023-10-16T20:39:49.347 INFO:teuthology.orchestra.run.smithi088.stdout:43 2023-10-16T20:39:49.347 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-10-16T20:39:49.348 DEBUG:teuthology.orchestra.run.smithi088:> sudo stdin-killer -- python3 -c ' 2023-10-16T20:39:49.348 DEBUG:teuthology.orchestra.run.smithi088:> import glob 2023-10-16T20:39:49.348 DEBUG:teuthology.orchestra.run.smithi088:> import re 2023-10-16T20:39:49.349 DEBUG:teuthology.orchestra.run.smithi088:> import os 2023-10-16T20:39:49.349 DEBUG:teuthology.orchestra.run.smithi088:> import subprocess 2023-10-16T20:39:49.349 DEBUG:teuthology.orchestra.run.smithi088:> 2023-10-16T20:39:49.350 DEBUG:teuthology.orchestra.run.smithi088:> def _find_admin_socket(client_name): 2023-10-16T20:39:49.350 DEBUG:teuthology.orchestra.run.smithi088:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-16T20:39:49.350 DEBUG:teuthology.orchestra.run.smithi088:> files = glob.glob(asok_path) 2023-10-16T20:39:49.351 DEBUG:teuthology.orchestra.run.smithi088:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-16T20:39:49.351 DEBUG:teuthology.orchestra.run.smithi088:> 2023-10-16T20:39:49.351 DEBUG:teuthology.orchestra.run.smithi088:> # Given a non-glob path, it better be there 2023-10-16T20:39:49.351 DEBUG:teuthology.orchestra.run.smithi088:> if "*" not in asok_path: 2023-10-16T20:39:49.352 DEBUG:teuthology.orchestra.run.smithi088:> assert(len(files) == 1) 2023-10-16T20:39:49.352 DEBUG:teuthology.orchestra.run.smithi088:> return files[0] 2023-10-16T20:39:49.352 DEBUG:teuthology.orchestra.run.smithi088:> 2023-10-16T20:39:49.353 DEBUG:teuthology.orchestra.run.smithi088:> for f in files: 2023-10-16T20:39:49.353 DEBUG:teuthology.orchestra.run.smithi088:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-16T20:39:49.353 DEBUG:teuthology.orchestra.run.smithi088:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-16T20:39:49.354 DEBUG:teuthology.orchestra.run.smithi088:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-16T20:39:49.354 DEBUG:teuthology.orchestra.run.smithi088:> contents = proc_f.read() 2023-10-16T20:39:49.354 DEBUG:teuthology.orchestra.run.smithi088:> if mountpoint in contents: 2023-10-16T20:39:49.354 DEBUG:teuthology.orchestra.run.smithi088:> return f 2023-10-16T20:39:49.355 DEBUG:teuthology.orchestra.run.smithi088:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-16T20:39:49.355 DEBUG:teuthology.orchestra.run.smithi088:> 2023-10-16T20:39:49.355 DEBUG:teuthology.orchestra.run.smithi088:> print(_find_admin_socket("client.0")) 2023-10-16T20:39:49.356 DEBUG:teuthology.orchestra.run.smithi088:> ' 2023-10-16T20:39:49.482 INFO:teuthology.orchestra.run.smithi088.stdout:/var/run/ceph/ceph-client.0.46445.asok 2023-10-16T20:39:49.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-10-16T20:39:49.494 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.46445.asok 2023-10-16T20:39:49.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:49.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.46445.asok status 2023-10-16T20:39:49.668 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-10-16T20:39:49.669 INFO:teuthology.orchestra.run.smithi088.stdout: "metadata": { 2023-10-16T20:39:49.669 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_sha1": "4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0", 2023-10-16T20:39:49.669 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_version": "ceph version 18.0.0-6730-g4ba8a190 (4ba8a190d7130cb1f1e7ceedfc0d129df5dd29b0) reef (dev)", 2023-10-16T20:39:49.669 INFO:teuthology.orchestra.run.smithi088.stdout: "entity_id": "0", 2023-10-16T20:39:49.669 INFO:teuthology.orchestra.run.smithi088.stdout: "hostname": "smithi088", 2023-10-16T20:39:49.670 INFO:teuthology.orchestra.run.smithi088.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-16T20:39:49.670 INFO:teuthology.orchestra.run.smithi088.stdout: "pid": "46445", 2023-10-16T20:39:49.670 INFO:teuthology.orchestra.run.smithi088.stdout: "root": "/" 2023-10-16T20:39:49.670 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-10-16T20:39:49.671 INFO:teuthology.orchestra.run.smithi088.stdout: "dentry_count": 0, 2023-10-16T20:39:49.671 INFO:teuthology.orchestra.run.smithi088.stdout: "dentry_pinned_count": 0, 2023-10-16T20:39:49.671 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 4977, 2023-10-16T20:39:49.671 INFO:teuthology.orchestra.run.smithi088.stdout: "inst": { 2023-10-16T20:39:49.671 INFO:teuthology.orchestra.run.smithi088.stdout: "name": { 2023-10-16T20:39:49.672 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "client", 2023-10-16T20:39:49.672 INFO:teuthology.orchestra.run.smithi088.stdout: "num": 4977 2023-10-16T20:39:49.672 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-10-16T20:39:49.672 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": { 2023-10-16T20:39:49.673 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2023-10-16T20:39:49.673 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "192.168.0.1:0", 2023-10-16T20:39:49.673 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3410099399 2023-10-16T20:39:49.673 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-10-16T20:39:49.674 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-10-16T20:39:49.674 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": { 2023-10-16T20:39:49.674 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2023-10-16T20:39:49.674 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "192.168.0.1:0", 2023-10-16T20:39:49.675 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3410099399 2023-10-16T20:39:49.675 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-10-16T20:39:49.675 INFO:teuthology.orchestra.run.smithi088.stdout: "inst_str": "client.4977 192.168.0.1:0/3410099399", 2023-10-16T20:39:49.676 INFO:teuthology.orchestra.run.smithi088.stdout: "addr_str": "192.168.0.1:0/3410099399", 2023-10-16T20:39:49.677 INFO:teuthology.orchestra.run.smithi088.stdout: "inode_count": 1, 2023-10-16T20:39:49.677 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_epoch": 19, 2023-10-16T20:39:49.677 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_epoch": 29, 2023-10-16T20:39:49.678 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_epoch_barrier": 0, 2023-10-16T20:39:49.678 INFO:teuthology.orchestra.run.smithi088.stdout: "blocklisted": false, 2023-10-16T20:39:49.678 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs" 2023-10-16T20:39:49.678 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-10-16T20:39:49.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:49.683 DEBUG:teuthology.orchestra.run.smithi088:> stat --file-system '--printf=%T 2023-10-16T20:39:49.684 DEBUG:teuthology.orchestra.run.smithi088:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-16T20:39:49.693 INFO:teuthology.orchestra.run.smithi088.stdout:fuseblk 2023-10-16T20:39:49.694 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-16T20:39:49.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:39:49.694 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-16T20:39:49.754 INFO:teuthology.run_tasks:Running task mds_thrash... 2023-10-16T20:39:49.768 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:39:49.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.936+0000 7f56e3aea640 1 Processor -- start 2023-10-16T20:39:49.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.936+0000 7f56e3aea640 1 -- start start 2023-10-16T20:39:49.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.936+0000 7f56e3aea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f56dc1538e0 0x7f56dc0ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:49.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.936+0000 7f56e3aea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56dc0ca610 con 0x7f56dc15a2a0 2023-10-16T20:39:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.936+0000 7f56e3aea640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56dc15d540 con 0x7f56dc15cb20 2023-10-16T20:39:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.936+0000 7f56e3aea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56dc1595b0 con 0x7f56dc1538e0 2023-10-16T20:39:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.936+0000 7f56e1886640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f56dc1538e0 0x7f56dc0ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.936+0000 7f56e1886640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f56dc1538e0 0x7f56dc0ca0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47596/0 (socket says 172.21.15.88:47596) 2023-10-16T20:39:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.936+0000 7f56e1886640 1 -- 172.21.15.88:0/585675557 learned_addr learned my addr 172.21.15.88:0/585675557 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:49.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.936+0000 7f56e1085640 1 --1- 172.21.15.88:0/585675557 >> v1:172.21.15.88:6789/0 conn(0x7f56dc15a2a0 0x7f56dc0c3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58354/0 (socket says 172.21.15.88:58354) 2023-10-16T20:39:49.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.936+0000 7f56e1886640 1 -- 172.21.15.88:0/585675557 >> v1:172.21.15.191:6789/0 conn(0x7f56dc15cb20 legacy=0x7f56dc0cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:49.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.936+0000 7f56e1886640 1 -- 172.21.15.88:0/585675557 >> v1:172.21.15.88:6789/0 conn(0x7f56dc15a2a0 legacy=0x7f56dc0c3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:49.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.936+0000 7f56e0884640 1 -- 172.21.15.88:0/585675557 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164229900 0 0) 0x7f56dc0ca610 con 0x7f56dc15a2a0 2023-10-16T20:39:49.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.936+0000 7f56e1886640 1 -- 172.21.15.88:0/585675557 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56dc15d820 con 0x7f56dc1538e0 2023-10-16T20:39:49.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.940+0000 7f56e1886640 1 --2- 172.21.15.88:0/585675557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56dc1538e0 0x7f56dc0ca0d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009870 tx=0x7f56d801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306aee1b36c159c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:49.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.940+0000 7f56e0884640 1 -- 172.21.15.88:0/585675557 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d80ab600 con 0x7f56dc1538e0 2023-10-16T20:39:49.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.940+0000 7f56e0884640 1 -- 172.21.15.88:0/585675557 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d80a7050 con 0x7f56dc1538e0 2023-10-16T20:39:49.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.940+0000 7f56e0884640 1 -- 172.21.15.88:0/585675557 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d80ab980 con 0x7f56dc1538e0 2023-10-16T20:39:49.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.940+0000 7f56e3aea640 1 -- 172.21.15.88:0/585675557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56dc1538e0 msgr2=0x7f56dc0ca0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:49.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.940+0000 7f56e3aea640 1 --2- 172.21.15.88:0/585675557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56dc1538e0 0x7f56dc0ca0d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009870 tx=0x7f56d801c540 comp rx=0 tx=0).stop 2023-10-16T20:39:49.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.940+0000 7f56e3aea640 1 -- 172.21.15.88:0/585675557 shutdown_connections 2023-10-16T20:39:49.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.940+0000 7f56e3aea640 1 --2- 172.21.15.88:0/585675557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56dc1538e0 0x7f56dc0ca0d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:49.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.940+0000 7f56e3aea640 1 -- 172.21.15.88:0/585675557 >> 172.21.15.88:0/585675557 conn(0x7f56dc06bd50 msgr2=0x7f56dc06c160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:49.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.940+0000 7f56e3aea640 1 -- 172.21.15.88:0/585675557 shutdown_connections 2023-10-16T20:39:49.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.940+0000 7f56e3aea640 1 -- 172.21.15.88:0/585675557 wait complete. 2023-10-16T20:39:49.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e3aea640 1 Processor -- start 2023-10-16T20:39:49.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e3aea640 1 -- start start 2023-10-16T20:39:49.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e3aea640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56dc1538e0 0x7f56dc0cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:49.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e3aea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56dc15a2a0 0x7f56dc156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:49.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e3aea640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56dc15cb20 0x7f56dc156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:49.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e3aea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc155d10 con 0x7f56dc15a2a0 2023-10-16T20:39:49.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e3aea640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc1561b0 con 0x7f56dc15cb20 2023-10-16T20:39:49.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e3aea640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc1f60f0 con 0x7f56dc1538e0 2023-10-16T20:39:49.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e1085640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56dc15a2a0 0x7f56dc156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:49.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e1085640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56dc15a2a0 0x7f56dc156640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47604/0 (socket says 172.21.15.88:47604) 2023-10-16T20:39:49.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e1085640 1 -- 172.21.15.88:0/795140275 learned_addr learned my addr 172.21.15.88:0/795140275 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:49.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e1085640 1 -- 172.21.15.88:0/795140275 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56dc1538e0 msgr2=0x7f56dc0cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:49.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e1085640 1 --2- 172.21.15.88:0/795140275 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56dc1538e0 0x7f56dc0cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:49.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e1085640 1 -- 172.21.15.88:0/795140275 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56dc15cb20 msgr2=0x7f56dc156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:49.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e1085640 1 --2- 172.21.15.88:0/795140275 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56dc15cb20 0x7f56dc156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:49.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e1085640 1 -- 172.21.15.88:0/795140275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d80094d0 con 0x7f56dc15a2a0 2023-10-16T20:39:49.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e1085640 1 --2- 172.21.15.88:0/795140275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56dc15a2a0 0x7f56dc156640 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00c4a0 tx=0x7f56cc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:49.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56d27fc640 1 -- 172.21.15.88:0/795140275 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc0074e0 con 0x7f56dc15a2a0 2023-10-16T20:39:49.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56d27fc640 1 -- 172.21.15.88:0/795140275 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56cc007640 con 0x7f56dc15a2a0 2023-10-16T20:39:49.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56d27fc640 1 -- 172.21.15.88:0/795140275 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc004a30 con 0x7f56dc15a2a0 2023-10-16T20:39:49.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e3aea640 1 -- 172.21.15.88:0/795140275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56dc1f5570 con 0x7f56dc15a2a0 2023-10-16T20:39:49.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e3aea640 1 -- 172.21.15.88:0/795140275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56dc1f5b30 con 0x7f56dc15a2a0 2023-10-16T20:39:49.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.944+0000 7f56e3aea640 1 -- 172.21.15.88:0/795140275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56dc074700 con 0x7f56dc15a2a0 2023-10-16T20:39:49.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.948+0000 7f56d27fc640 1 -- 172.21.15.88:0/795140275 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f56cc004b90 con 0x7f56dc15a2a0 2023-10-16T20:39:49.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.948+0000 7f56d27fc640 1 --2- 172.21.15.88:0/795140275 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56c00664a0 0x7f56c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:49.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.948+0000 7f56d27fc640 1 -- 172.21.15.88:0/795140275 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f56cc083550 con 0x7f56dc15a2a0 2023-10-16T20:39:49.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.948+0000 7f56e1886640 1 --2- 172.21.15.88:0/795140275 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56c00664a0 0x7f56c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:49.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.948+0000 7f56e1886640 1 --2- 172.21.15.88:0/795140275 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56c00664a0 0x7f56c0068960 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f56dc068400 tx=0x7f56d8004740 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:49.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:49.952+0000 7f56d27fc640 1 -- 172.21.15.88:0/795140275 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f56cc050570 con 0x7f56dc15a2a0 2023-10-16T20:39:50.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.132+0000 7f56e3aea640 1 -- 172.21.15.88:0/795140275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56dc0ca7a0 con 0x7f56dc15a2a0 2023-10-16T20:39:50.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.132+0000 7f56d27fc640 1 -- 172.21.15.88:0/795140275 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18710 (secure 0 0 0) 0x7f56cc018090 con 0x7f56dc15a2a0 2023-10-16T20:39:50.138 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:50.138 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:37:37.392549+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:39:50.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.136+0000 7f56e3aea640 1 -- 172.21.15.88:0/795140275 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56c00664a0 msgr2=0x7f56c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:50.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.136+0000 7f56e3aea640 1 --2- 172.21.15.88:0/795140275 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56c00664a0 0x7f56c0068960 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f56dc068400 tx=0x7f56d8004740 comp rx=0 tx=0).stop 2023-10-16T20:39:50.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.136+0000 7f56e3aea640 1 -- 172.21.15.88:0/795140275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56dc15a2a0 msgr2=0x7f56dc156640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:50.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.136+0000 7f56e3aea640 1 --2- 172.21.15.88:0/795140275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56dc15a2a0 0x7f56dc156640 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00c4a0 tx=0x7f56cc00c960 comp rx=0 tx=0).stop 2023-10-16T20:39:50.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.136+0000 7f56e3aea640 1 -- 172.21.15.88:0/795140275 shutdown_connections 2023-10-16T20:39:50.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.136+0000 7f56e3aea640 1 --2- 172.21.15.88:0/795140275 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56c00664a0 0x7f56c0068960 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.136+0000 7f56e3aea640 1 --2- 172.21.15.88:0/795140275 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56dc15cb20 0x7f56dc156b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.136+0000 7f56e3aea640 1 --2- 172.21.15.88:0/795140275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56dc15a2a0 0x7f56dc156640 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.136+0000 7f56e3aea640 1 --2- 172.21.15.88:0/795140275 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56dc1538e0 0x7f56dc0cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.136+0000 7f56e3aea640 1 -- 172.21.15.88:0/795140275 >> 172.21.15.88:0/795140275 conn(0x7f56dc06bd50 msgr2=0x7f56dc0c9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:50.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.136+0000 7f56e3aea640 1 -- 172.21.15.88:0/795140275 shutdown_connections 2023-10-16T20:39:50.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.136+0000 7f56e3aea640 1 -- 172.21.15.88:0/795140275 wait complete. 2023-10-16T20:39:50.159 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T20:36:43.646418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T20:36:53.546513+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T20:36:54.143703+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T20:36:57.314642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T20:39:50.160 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T20:39:50.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.296+0000 7f7ff330f640 1 Processor -- start 2023-10-16T20:39:50.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.296+0000 7f7ff330f640 1 -- start start 2023-10-16T20:39:50.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.300+0000 7f7ff330f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7fec159490 0x7f7fec0c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.300+0000 7f7ff330f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7fec0c5930 0x7f7fec0ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.300+0000 7f7ff330f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fec159d90 con 0x7f7fec153780 2023-10-16T20:39:50.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.300+0000 7f7ff330f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fec15a070 con 0x7f7fec0c5930 2023-10-16T20:39:50.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.300+0000 7f7ff330f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fec15a350 con 0x7f7fec159490 2023-10-16T20:39:50.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.300+0000 7f7ff18ac640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7fec0c5930 0x7f7fec0ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.300+0000 7f7ff18ac640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7fec0c5930 0x7f7fec0ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47614/0 (socket says 172.21.15.88:47614) 2023-10-16T20:39:50.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.300+0000 7f7ff18ac640 1 -- 172.21.15.88:0/747480150 learned_addr learned my addr 172.21.15.88:0/747480150 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:50.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.300+0000 7f7ff10ab640 1 --1- 172.21.15.88:0/747480150 >> v1:172.21.15.191:6789/0 conn(0x7f7fec153780 0x7f7fec0c9be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:57180/0 (socket says 172.21.15.88:57180) 2023-10-16T20:39:50.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.300+0000 7f7ff08aa640 1 --2- 172.21.15.88:0/747480150 >> v2:172.21.15.191:3300/0 conn(0x7f7fec159490 0x7f7fec0c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.300+0000 7f7ff18ac640 1 -- 172.21.15.88:0/747480150 >> v2:172.21.15.191:3300/0 conn(0x7f7fec159490 msgr2=0x7f7fec0c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:50.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.300+0000 7f7ff18ac640 1 --2- 172.21.15.88:0/747480150 >> v2:172.21.15.191:3300/0 conn(0x7f7fec159490 0x7f7fec0c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.300+0000 7f7ff18ac640 1 -- 172.21.15.88:0/747480150 >> v1:172.21.15.191:6789/0 conn(0x7f7fec153780 legacy=0x7f7fec0c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:50.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.300+0000 7f7ff18ac640 1 -- 172.21.15.88:0/747480150 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec0c4910 con 0x7f7fec0c5930 2023-10-16T20:39:50.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.304+0000 7f7ff18ac640 1 --2- 172.21.15.88:0/747480150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fec0c5930 0x7f7fec0ced80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0cfef0 tx=0x7f7fe401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa09edecbba18778 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:50.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.304+0000 7f7fe3fff640 1 -- 172.21.15.88:0/747480150 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe40af400 con 0x7f7fec0c5930 2023-10-16T20:39:50.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.304+0000 7f7fe3fff640 1 -- 172.21.15.88:0/747480150 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe40ab050 con 0x7f7fec0c5930 2023-10-16T20:39:50.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.304+0000 7f7fe3fff640 1 -- 172.21.15.88:0/747480150 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe40af780 con 0x7f7fec0c5930 2023-10-16T20:39:50.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.304+0000 7f7ff330f640 1 -- 172.21.15.88:0/747480150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fec0c5930 msgr2=0x7f7fec0ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:50.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.304+0000 7f7ff330f640 1 --2- 172.21.15.88:0/747480150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fec0c5930 0x7f7fec0ced80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0cfef0 tx=0x7f7fe401f1c0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.304+0000 7f7ff330f640 1 -- 172.21.15.88:0/747480150 shutdown_connections 2023-10-16T20:39:50.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.304+0000 7f7ff330f640 1 --2- 172.21.15.88:0/747480150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fec0c5930 0x7f7fec0ced80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.304+0000 7f7ff330f640 1 --2- 172.21.15.88:0/747480150 >> v2:172.21.15.191:3300/0 conn(0x7f7fec159490 0x7f7fec0c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.304+0000 7f7ff330f640 1 -- 172.21.15.88:0/747480150 >> 172.21.15.88:0/747480150 conn(0x7f7fec06bcb0 msgr2=0x7f7fec06c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:50.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.304+0000 7f7ff330f640 1 -- 172.21.15.88:0/747480150 shutdown_connections 2023-10-16T20:39:50.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.304+0000 7f7ff330f640 1 -- 172.21.15.88:0/747480150 wait complete. 2023-10-16T20:39:50.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.304+0000 7f7ff330f640 1 Processor -- start 2023-10-16T20:39:50.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.304+0000 7f7ff330f640 1 -- start start 2023-10-16T20:39:50.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff330f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fec0c5930 0x7f7fec0b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff330f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fec153780 0x7f7fec14ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff330f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fec159490 0x7f7fec152800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff330f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec0c67b0 con 0x7f7fec153780 2023-10-16T20:39:50.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff330f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec0b48a0 con 0x7f7fec159490 2023-10-16T20:39:50.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff330f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fec0b4e40 con 0x7f7fec0c5930 2023-10-16T20:39:50.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff08aa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fec153780 0x7f7fec14ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff08aa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fec153780 0x7f7fec14ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47618/0 (socket says 172.21.15.88:47618) 2023-10-16T20:39:50.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff08aa640 1 -- 172.21.15.88:0/40992342 learned_addr learned my addr 172.21.15.88:0/40992342 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:50.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff18ac640 1 --2- 172.21.15.88:0/40992342 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fec159490 0x7f7fec152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff10ab640 1 --2- 172.21.15.88:0/40992342 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fec0c5930 0x7f7fec0b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff08aa640 1 -- 172.21.15.88:0/40992342 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fec0c5930 msgr2=0x7f7fec0b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:50.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff08aa640 1 --2- 172.21.15.88:0/40992342 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fec0c5930 0x7f7fec0b3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff08aa640 1 -- 172.21.15.88:0/40992342 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fec159490 msgr2=0x7f7fec152800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:50.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff08aa640 1 --2- 172.21.15.88:0/40992342 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fec159490 0x7f7fec152800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff08aa640 1 -- 172.21.15.88:0/40992342 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe4009b30 con 0x7f7fec153780 2023-10-16T20:39:50.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff08aa640 1 --2- 172.21.15.88:0/40992342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fec153780 0x7f7fec14ef10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc00d5f0 tx=0x7f7fdc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:50.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7fe1ffb640 1 -- 172.21.15.88:0/40992342 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc0125f0 con 0x7f7fec153780 2023-10-16T20:39:50.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7fe1ffb640 1 -- 172.21.15.88:0/40992342 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fdc012750 con 0x7f7fec153780 2023-10-16T20:39:50.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff330f640 1 -- 172.21.15.88:0/40992342 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fec0b4b60 con 0x7f7fec153780 2023-10-16T20:39:50.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7fe1ffb640 1 -- 172.21.15.88:0/40992342 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc012ae0 con 0x7f7fec153780 2023-10-16T20:39:50.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff330f640 1 -- 172.21.15.88:0/40992342 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fec0b3200 con 0x7f7fec153780 2023-10-16T20:39:50.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7ff330f640 1 -- 172.21.15.88:0/40992342 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fec074700 con 0x7f7fec153780 2023-10-16T20:39:50.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.308+0000 7f7fe1ffb640 1 -- 172.21.15.88:0/40992342 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7fdc012c40 con 0x7f7fec153780 2023-10-16T20:39:50.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.312+0000 7f7fe1ffb640 1 --2- 172.21.15.88:0/40992342 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7fd00664a0 0x7f7fd0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.312+0000 7f7fe1ffb640 1 -- 172.21.15.88:0/40992342 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7fdc084660 con 0x7f7fec153780 2023-10-16T20:39:50.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.312+0000 7f7ff10ab640 1 --2- 172.21.15.88:0/40992342 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7fd00664a0 0x7f7fd0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.312+0000 7f7ff10ab640 1 --2- 172.21.15.88:0/40992342 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7fd00664a0 0x7f7fd0068960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8000fc0 tx=0x7f7fe8000f30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:50.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.316+0000 7f7fe1ffb640 1 -- 172.21.15.88:0/40992342 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7fdc051730 con 0x7f7fec153780 2023-10-16T20:39:50.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.468+0000 7f7ff330f640 1 -- 172.21.15.88:0/40992342 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7fec0ca7d0 con 0x7f7fec153780 2023-10-16T20:39:50.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.468+0000 7f7fe1ffb640 1 -- 172.21.15.88:0/40992342 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7fdc054d50 con 0x7f7fec153780 2023-10-16T20:39:50.473 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-10-16T20:39:50.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.468+0000 7f7ff330f640 1 -- 172.21.15.88:0/40992342 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7fd00664a0 msgr2=0x7f7fd0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:50.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.468+0000 7f7ff330f640 1 --2- 172.21.15.88:0/40992342 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7fd00664a0 0x7f7fd0068960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8000fc0 tx=0x7f7fe8000f30 comp rx=0 tx=0).stop 2023-10-16T20:39:50.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.468+0000 7f7ff330f640 1 -- 172.21.15.88:0/40992342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fec153780 msgr2=0x7f7fec14ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:50.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.468+0000 7f7ff330f640 1 --2- 172.21.15.88:0/40992342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fec153780 0x7f7fec14ef10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc00d5f0 tx=0x7f7fdc00dab0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.468+0000 7f7ff330f640 1 -- 172.21.15.88:0/40992342 shutdown_connections 2023-10-16T20:39:50.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.468+0000 7f7ff330f640 1 --2- 172.21.15.88:0/40992342 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7fd00664a0 0x7f7fd0068960 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.468+0000 7f7ff330f640 1 --2- 172.21.15.88:0/40992342 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7fec159490 0x7f7fec152800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.468+0000 7f7ff330f640 1 --2- 172.21.15.88:0/40992342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7fec153780 0x7f7fec14ef10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.468+0000 7f7ff330f640 1 --2- 172.21.15.88:0/40992342 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7fec0c5930 0x7f7fec0b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.468+0000 7f7ff330f640 1 -- 172.21.15.88:0/40992342 >> 172.21.15.88:0/40992342 conn(0x7f7fec06bcb0 msgr2=0x7f7fec0715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:50.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.468+0000 7f7ff330f640 1 -- 172.21.15.88:0/40992342 shutdown_connections 2023-10-16T20:39:50.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.468+0000 7f7ff330f640 1 -- 172.21.15.88:0/40992342 wait complete. 2023-10-16T20:39:50.493 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T20:39:50.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.632+0000 7fd61a0d6640 1 Processor -- start 2023-10-16T20:39:50.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.632+0000 7fd61a0d6640 1 -- start start 2023-10-16T20:39:50.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.632+0000 7fd61a0d6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd614153780 0x7fd6140c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.632+0000 7fd61a0d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd6141598c0 0x7fd6140ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.632+0000 7fd61a0d6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd614159d90 con 0x7fd614159490 2023-10-16T20:39:50.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.632+0000 7fd61a0d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd61415a070 con 0x7fd6141598c0 2023-10-16T20:39:50.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.632+0000 7fd61a0d6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd61415a350 con 0x7fd614153780 2023-10-16T20:39:50.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.632+0000 7fd613fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd6141598c0 0x7fd6140ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.632+0000 7fd613fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd6141598c0 0x7fd6140ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47624/0 (socket says 172.21.15.88:47624) 2023-10-16T20:39:50.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.632+0000 7fd613fff640 1 -- 172.21.15.88:0/3110620524 learned_addr learned my addr 172.21.15.88:0/3110620524 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:50.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.632+0000 7fd6137fe640 1 --2- 172.21.15.88:0/3110620524 >> v2:172.21.15.191:3300/0 conn(0x7fd614153780 0x7fd6140c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.632+0000 7fd613fff640 1 -- 172.21.15.88:0/3110620524 >> v2:172.21.15.191:3300/0 conn(0x7fd614153780 msgr2=0x7fd6140c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:50.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.632+0000 7fd613fff640 1 --2- 172.21.15.88:0/3110620524 >> v2:172.21.15.191:3300/0 conn(0x7fd614153780 0x7fd6140c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.632+0000 7fd613fff640 1 -- 172.21.15.88:0/3110620524 >> v1:172.21.15.191:6789/0 conn(0x7fd614159490 legacy=0x7fd6140c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:50.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.632+0000 7fd613fff640 1 -- 172.21.15.88:0/3110620524 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6140c4910 con 0x7fd6141598c0 2023-10-16T20:39:50.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.636+0000 7fd613fff640 1 --2- 172.21.15.88:0/3110620524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6141598c0 0x7fd6140ced80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd6140c9900 tx=0x7fd60c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784164483939da0e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:50.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.636+0000 7fd6127fc640 1 -- 172.21.15.88:0/3110620524 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60c0af400 con 0x7fd6141598c0 2023-10-16T20:39:50.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.636+0000 7fd6127fc640 1 -- 172.21.15.88:0/3110620524 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd60c0ab050 con 0x7fd6141598c0 2023-10-16T20:39:50.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.636+0000 7fd6127fc640 1 -- 172.21.15.88:0/3110620524 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60c0af780 con 0x7fd6141598c0 2023-10-16T20:39:50.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.636+0000 7fd61a0d6640 1 -- 172.21.15.88:0/3110620524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6141598c0 msgr2=0x7fd6140ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:50.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.636+0000 7fd61a0d6640 1 --2- 172.21.15.88:0/3110620524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6141598c0 0x7fd6140ced80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd6140c9900 tx=0x7fd60c01f1c0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.636+0000 7fd61a0d6640 1 -- 172.21.15.88:0/3110620524 shutdown_connections 2023-10-16T20:39:50.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.636+0000 7fd61a0d6640 1 --2- 172.21.15.88:0/3110620524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6141598c0 0x7fd6140ced80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.636+0000 7fd61a0d6640 1 --2- 172.21.15.88:0/3110620524 >> v2:172.21.15.191:3300/0 conn(0x7fd614153780 0x7fd6140c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.636+0000 7fd61a0d6640 1 -- 172.21.15.88:0/3110620524 >> 172.21.15.88:0/3110620524 conn(0x7fd61406bcb0 msgr2=0x7fd61406c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:50.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.636+0000 7fd61a0d6640 1 -- 172.21.15.88:0/3110620524 shutdown_connections 2023-10-16T20:39:50.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.636+0000 7fd61a0d6640 1 -- 172.21.15.88:0/3110620524 wait complete. 2023-10-16T20:39:50.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd61a0d6640 1 Processor -- start 2023-10-16T20:39:50.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd61a0d6640 1 -- start start 2023-10-16T20:39:50.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd61a0d6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd614153780 0x7fd61414ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd61a0d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd614159490 0x7fd614155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd61a0d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6141598c0 0x7fd614155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd61a0d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd614156c10 con 0x7fd6141598c0 2023-10-16T20:39:50.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd61a0d6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd614153dc0 con 0x7fd614153780 2023-10-16T20:39:50.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd61a0d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd614146780 con 0x7fd614159490 2023-10-16T20:39:50.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd613fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6141598c0 0x7fd614155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd6137fe640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd614153780 0x7fd61414ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd613fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6141598c0 0x7fd614155960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47638/0 (socket says 172.21.15.88:47638) 2023-10-16T20:39:50.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd613fff640 1 -- 172.21.15.88:0/373271177 learned_addr learned my addr 172.21.15.88:0/373271177 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:50.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd612ffd640 1 --2- 172.21.15.88:0/373271177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd614159490 0x7fd614155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd613fff640 1 -- 172.21.15.88:0/373271177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd614159490 msgr2=0x7fd614155420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:50.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd613fff640 1 --2- 172.21.15.88:0/373271177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd614159490 0x7fd614155420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd613fff640 1 -- 172.21.15.88:0/373271177 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd614153780 msgr2=0x7fd61414ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:50.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd613fff640 1 --2- 172.21.15.88:0/373271177 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd614153780 0x7fd61414ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd613fff640 1 -- 172.21.15.88:0/373271177 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60c009b30 con 0x7fd6141598c0 2023-10-16T20:39:50.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd613fff640 1 --2- 172.21.15.88:0/373271177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6141598c0 0x7fd614155960 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0b2040 tx=0x7fd60c01bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:50.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd5fbfff640 1 -- 172.21.15.88:0/373271177 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60c0afdc0 con 0x7fd6141598c0 2023-10-16T20:39:50.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd5fbfff640 1 -- 172.21.15.88:0/373271177 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd60c009510 con 0x7fd6141598c0 2023-10-16T20:39:50.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd5fbfff640 1 -- 172.21.15.88:0/373271177 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60c0ba660 con 0x7fd6141598c0 2023-10-16T20:39:50.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd61a0d6640 1 -- 172.21.15.88:0/373271177 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd614154f90 con 0x7fd6141598c0 2023-10-16T20:39:50.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd61a0d6640 1 -- 172.21.15.88:0/373271177 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6141542b0 con 0x7fd6141598c0 2023-10-16T20:39:50.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.640+0000 7fd61a0d6640 1 -- 172.21.15.88:0/373271177 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd614155120 con 0x7fd6141598c0 2023-10-16T20:39:50.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.644+0000 7fd5fbfff640 1 -- 172.21.15.88:0/373271177 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd60c0b6040 con 0x7fd6141598c0 2023-10-16T20:39:50.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.644+0000 7fd5fbfff640 1 --2- 172.21.15.88:0/373271177 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd6000664d0 0x7fd600068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.644+0000 7fd5fbfff640 1 -- 172.21.15.88:0/373271177 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd60c1221c0 con 0x7fd6141598c0 2023-10-16T20:39:50.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.644+0000 7fd6137fe640 1 --2- 172.21.15.88:0/373271177 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd6000664d0 0x7fd600068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.644+0000 7fd6137fe640 1 --2- 172.21.15.88:0/373271177 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd6000664d0 0x7fd600068990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd604006fd0 tx=0x7fd604008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:50.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.648+0000 7fd5fbfff640 1 -- 172.21.15.88:0/373271177 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd60c0ef2c0 con 0x7fd6141598c0 2023-10-16T20:39:50.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.804+0000 7fd61a0d6640 1 -- 172.21.15.88:0/373271177 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd6140c4ea0 con 0x7fd6141598c0 2023-10-16T20:39:50.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.804+0000 7fd5fbfff640 1 -- 172.21.15.88:0/373271177 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd60c0f28e0 con 0x7fd6141598c0 2023-10-16T20:39:50.809 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:39:50.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.804+0000 7fd61a0d6640 1 -- 172.21.15.88:0/373271177 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd6000664d0 msgr2=0x7fd600068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:50.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.804+0000 7fd61a0d6640 1 --2- 172.21.15.88:0/373271177 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd6000664d0 0x7fd600068990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd604006fd0 tx=0x7fd604008040 comp rx=0 tx=0).stop 2023-10-16T20:39:50.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.804+0000 7fd61a0d6640 1 -- 172.21.15.88:0/373271177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6141598c0 msgr2=0x7fd614155960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:50.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.804+0000 7fd61a0d6640 1 --2- 172.21.15.88:0/373271177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6141598c0 0x7fd614155960 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0b2040 tx=0x7fd60c01bf60 comp rx=0 tx=0).stop 2023-10-16T20:39:50.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.804+0000 7fd61a0d6640 1 -- 172.21.15.88:0/373271177 shutdown_connections 2023-10-16T20:39:50.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.804+0000 7fd61a0d6640 1 --2- 172.21.15.88:0/373271177 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd6000664d0 0x7fd600068990 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.804+0000 7fd61a0d6640 1 --2- 172.21.15.88:0/373271177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6141598c0 0x7fd614155960 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.804+0000 7fd61a0d6640 1 --2- 172.21.15.88:0/373271177 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd614159490 0x7fd614155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.804+0000 7fd61a0d6640 1 --2- 172.21.15.88:0/373271177 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd614153780 0x7fd61414ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.804+0000 7fd61a0d6640 1 -- 172.21.15.88:0/373271177 >> 172.21.15.88:0/373271177 conn(0x7fd61406bcb0 msgr2=0x7fd6140cbfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:50.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.804+0000 7fd61a0d6640 1 -- 172.21.15.88:0/373271177 shutdown_connections 2023-10-16T20:39:50.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.804+0000 7fd61a0d6640 1 -- 172.21.15.88:0/373271177 wait complete. 2023-10-16T20:39:50.829 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T20:39:50.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.964+0000 7fc76c69c640 1 Processor -- start 2023-10-16T20:39:50.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.964+0000 7fc76c69c640 1 -- start start 2023-10-16T20:39:50.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.968+0000 7fc76c69c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc7640c4640 0x7fc7641510d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.968+0000 7fc76c69c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc764151610 0x7fc7641573e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.968+0000 7fc76c69c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc764069e90 con 0x7fc7640c4260 2023-10-16T20:39:50.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.968+0000 7fc76c69c640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7640716a0 con 0x7fc7640c4640 2023-10-16T20:39:50.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.968+0000 7fc76c69c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc76406aea0 con 0x7fc764151610 2023-10-16T20:39:50.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.968+0000 7fc76a438640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc7640c4260 0x7fc7640c5520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58360/0 (socket says 172.21.15.88:58360) 2023-10-16T20:39:50.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.968+0000 7fc76a438640 1 -- 172.21.15.88:0/146700890 learned_addr learned my addr 172.21.15.88:0/146700890 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:50.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.968+0000 7fc76ac39640 1 --2- 172.21.15.88:0/146700890 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc764151610 0x7fc7641573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.968+0000 7fc769c37640 1 --2- 172.21.15.88:0/146700890 >> v2:172.21.15.191:3300/0 conn(0x7fc7640c4640 0x7fc7641510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.968+0000 7fc769436640 1 -- 172.21.15.88:0/146700890 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716379926 0 0) 0x7fc764069e90 con 0x7fc7640c4260 2023-10-16T20:39:50.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.968+0000 7fc769436640 1 -- 172.21.15.88:0/146700890 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc750003530 con 0x7fc7640c4260 2023-10-16T20:39:50.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.968+0000 7fc76ac39640 1 -- 172.21.15.88:0/146700890 >> v2:172.21.15.191:3300/0 conn(0x7fc7640c4640 msgr2=0x7fc7641510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:50.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.968+0000 7fc76ac39640 1 --2- 172.21.15.88:0/146700890 >> v2:172.21.15.191:3300/0 conn(0x7fc7640c4640 0x7fc7641510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.968+0000 7fc76ac39640 1 -- 172.21.15.88:0/146700890 >> v1:172.21.15.88:6789/0 conn(0x7fc7640c4260 legacy=0x7fc7640c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:50.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.968+0000 7fc769436640 1 -- 172.21.15.88:0/146700890 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4234511387 0 0) 0x7fc750003530 con 0x7fc7640c4260 2023-10-16T20:39:50.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.968+0000 7fc76ac39640 1 -- 172.21.15.88:0/146700890 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7641581e0 con 0x7fc764151610 2023-10-16T20:39:50.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.972+0000 7fc76ac39640 1 --2- 172.21.15.88:0/146700890 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc764151610 0x7fc7641573e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009d90 tx=0x7fc75c01c720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a8c62cea324ea22 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:50.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.972+0000 7fc769436640 1 -- 172.21.15.88:0/146700890 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760001680 con 0x7fc764151610 2023-10-16T20:39:50.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.972+0000 7fc769436640 1 -- 172.21.15.88:0/146700890 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75c0a5520 con 0x7fc764151610 2023-10-16T20:39:50.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.972+0000 7fc769436640 1 -- 172.21.15.88:0/146700890 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c0a58b0 con 0x7fc764151610 2023-10-16T20:39:50.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.972+0000 7fc76c69c640 1 -- 172.21.15.88:0/146700890 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc764151610 msgr2=0x7fc7641573e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:50.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.972+0000 7fc76c69c640 1 --2- 172.21.15.88:0/146700890 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc764151610 0x7fc7641573e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009d90 tx=0x7fc75c01c720 comp rx=0 tx=0).stop 2023-10-16T20:39:50.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.972+0000 7fc76c69c640 1 -- 172.21.15.88:0/146700890 shutdown_connections 2023-10-16T20:39:50.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.972+0000 7fc76c69c640 1 --2- 172.21.15.88:0/146700890 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc764151610 0x7fc7641573e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.972+0000 7fc76c69c640 1 --2- 172.21.15.88:0/146700890 >> v2:172.21.15.191:3300/0 conn(0x7fc7640c4640 0x7fc7641510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.972+0000 7fc76c69c640 1 -- 172.21.15.88:0/146700890 >> 172.21.15.88:0/146700890 conn(0x7fc7640ca9f0 msgr2=0x7fc7640cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:50.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.972+0000 7fc76c69c640 1 -- 172.21.15.88:0/146700890 shutdown_connections 2023-10-16T20:39:50.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.972+0000 7fc76c69c640 1 -- 172.21.15.88:0/146700890 wait complete. 2023-10-16T20:39:50.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.972+0000 7fc76c69c640 1 Processor -- start 2023-10-16T20:39:50.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76c69c640 1 -- start start 2023-10-16T20:39:50.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76c69c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7640c4260 0x7fc76414d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76c69c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7640c4640 0x7fc764146aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76c69c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc764151610 0x7fc764146fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76c69c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc764147630 con 0x7fc764151610 2023-10-16T20:39:50.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76c69c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc764144760 con 0x7fc7640c4640 2023-10-16T20:39:50.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76c69c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc764144a40 con 0x7fc7640c4260 2023-10-16T20:39:50.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76ac39640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc764151610 0x7fc764146fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76ac39640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc764151610 0x7fc764146fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47648/0 (socket says 172.21.15.88:47648) 2023-10-16T20:39:50.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76ac39640 1 -- 172.21.15.88:0/3171235319 learned_addr learned my addr 172.21.15.88:0/3171235319 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76a438640 1 --2- 172.21.15.88:0/3171235319 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7640c4260 0x7fc76414d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc769c37640 1 --2- 172.21.15.88:0/3171235319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7640c4640 0x7fc764146aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76ac39640 1 -- 172.21.15.88:0/3171235319 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7640c4260 msgr2=0x7fc76414d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76ac39640 1 --2- 172.21.15.88:0/3171235319 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7640c4260 0x7fc76414d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76ac39640 1 -- 172.21.15.88:0/3171235319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7640c4640 msgr2=0x7fc764146aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:50.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76ac39640 1 --2- 172.21.15.88:0/3171235319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7640c4640 0x7fc764146aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:50.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76ac39640 1 -- 172.21.15.88:0/3171235319 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75c009a70 con 0x7fc764151610 2023-10-16T20:39:50.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76ac39640 1 --2- 172.21.15.88:0/3171235319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc764151610 0x7fc764146fe0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc75c0b0040 tx=0x7fc75c0194c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:50.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc75affd640 1 -- 172.21.15.88:0/3171235319 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c0ad640 con 0x7fc764151610 2023-10-16T20:39:50.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc75affd640 1 -- 172.21.15.88:0/3171235319 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75c0ad7a0 con 0x7fc764151610 2023-10-16T20:39:50.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc75affd640 1 -- 172.21.15.88:0/3171235319 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c0adb90 con 0x7fc764151610 2023-10-16T20:39:50.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76c69c640 1 -- 172.21.15.88:0/3171235319 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc764144d20 con 0x7fc764151610 2023-10-16T20:39:50.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76c69c640 1 -- 172.21.15.88:0/3171235319 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7641452e0 con 0x7fc764151610 2023-10-16T20:39:50.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.976+0000 7fc76c69c640 1 -- 172.21.15.88:0/3171235319 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc764074700 con 0x7fc764151610 2023-10-16T20:39:50.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.980+0000 7fc75affd640 1 -- 172.21.15.88:0/3171235319 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc75c0b4050 con 0x7fc764151610 2023-10-16T20:39:50.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.980+0000 7fc75affd640 1 --2- 172.21.15.88:0/3171235319 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc7480664d0 0x7fc748068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:50.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.980+0000 7fc75affd640 1 -- 172.21.15.88:0/3171235319 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc75c11edb0 con 0x7fc764151610 2023-10-16T20:39:50.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.980+0000 7fc76a438640 1 --2- 172.21.15.88:0/3171235319 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc7480664d0 0x7fc748068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:50.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.980+0000 7fc76a438640 1 --2- 172.21.15.88:0/3171235319 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc7480664d0 0x7fc748068990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc760001ec0 tx=0x7fc760001a60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:50.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:50.984+0000 7fc75affd640 1 -- 172.21.15.88:0/3171235319 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc75c0ec0d0 con 0x7fc764151610 2023-10-16T20:39:51.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.136+0000 7fc76c69c640 1 -- 172.21.15.88:0/3171235319 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc7640c6e80 con 0x7fc764151610 2023-10-16T20:39:51.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.136+0000 7fc75affd640 1 -- 172.21.15.88:0/3171235319 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc75c0c7020 con 0x7fc764151610 2023-10-16T20:39:51.139 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2023-10-16T20:39:51.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.136+0000 7fc76c69c640 1 -- 172.21.15.88:0/3171235319 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc7480664d0 msgr2=0x7fc748068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.136+0000 7fc76c69c640 1 --2- 172.21.15.88:0/3171235319 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc7480664d0 0x7fc748068990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc760001ec0 tx=0x7fc760001a60 comp rx=0 tx=0).stop 2023-10-16T20:39:51.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.136+0000 7fc76c69c640 1 -- 172.21.15.88:0/3171235319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc764151610 msgr2=0x7fc764146fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.136+0000 7fc76c69c640 1 --2- 172.21.15.88:0/3171235319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc764151610 0x7fc764146fe0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc75c0b0040 tx=0x7fc75c0194c0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.136+0000 7fc76c69c640 1 -- 172.21.15.88:0/3171235319 shutdown_connections 2023-10-16T20:39:51.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.136+0000 7fc76c69c640 1 --2- 172.21.15.88:0/3171235319 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc7480664d0 0x7fc748068990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.136+0000 7fc76c69c640 1 --2- 172.21.15.88:0/3171235319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc764151610 0x7fc764146fe0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.136+0000 7fc76c69c640 1 --2- 172.21.15.88:0/3171235319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc7640c4640 0x7fc764146aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.136+0000 7fc76c69c640 1 --2- 172.21.15.88:0/3171235319 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc7640c4260 0x7fc76414d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.136+0000 7fc76c69c640 1 -- 172.21.15.88:0/3171235319 >> 172.21.15.88:0/3171235319 conn(0x7fc7640ca9f0 msgr2=0x7fc7640c9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:51.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.136+0000 7fc76c69c640 1 -- 172.21.15.88:0/3171235319 shutdown_connections 2023-10-16T20:39:51.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.136+0000 7fc76c69c640 1 -- 172.21.15.88:0/3171235319 wait complete. 2023-10-16T20:39:51.160 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T20:39:51.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.296+0000 7f72ef4bb640 1 Processor -- start 2023-10-16T20:39:51.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.296+0000 7f72ef4bb640 1 -- start start 2023-10-16T20:39:51.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.300+0000 7f72ef4bb640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72e81501e0 0x7f72e80c6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.300+0000 7f72ef4bb640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f72e80c34e0 0x7f72e80c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.300+0000 7f72ef4bb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e8156e40 con 0x7f72e80c3000 2023-10-16T20:39:51.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.300+0000 7f72ef4bb640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e80c6150 con 0x7f72e80c34e0 2023-10-16T20:39:51.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.300+0000 7f72ef4bb640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e80c29a0 con 0x7f72e81501e0 2023-10-16T20:39:51.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.300+0000 7f72edcb8640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f72e80c3000 0x7f72e80c71c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58368/0 (socket says 172.21.15.88:58368) 2023-10-16T20:39:51.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.300+0000 7f72edcb8640 1 -- 172.21.15.88:0/2413438350 learned_addr learned my addr 172.21.15.88:0/2413438350 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:51.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.300+0000 7f72ee4b9640 1 --2- 172.21.15.88:0/2413438350 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72e81501e0 0x7f72e80c6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.300+0000 7f72eecba640 1 --2- 172.21.15.88:0/2413438350 >> v2:172.21.15.191:3300/0 conn(0x7f72e80c34e0 0x7f72e80c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.300+0000 7f72ed4b7640 1 -- 172.21.15.88:0/2413438350 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057126249 0 0) 0x7f72e8156e40 con 0x7f72e80c3000 2023-10-16T20:39:51.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.300+0000 7f72ed4b7640 1 -- 172.21.15.88:0/2413438350 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72d4003530 con 0x7f72e80c3000 2023-10-16T20:39:51.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.300+0000 7f72ee4b9640 1 -- 172.21.15.88:0/2413438350 >> v2:172.21.15.191:3300/0 conn(0x7f72e80c34e0 msgr2=0x7f72e80c5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:51.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.300+0000 7f72ee4b9640 1 --2- 172.21.15.88:0/2413438350 >> v2:172.21.15.191:3300/0 conn(0x7f72e80c34e0 0x7f72e80c5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.300+0000 7f72ee4b9640 1 -- 172.21.15.88:0/2413438350 >> v1:172.21.15.88:6789/0 conn(0x7f72e80c3000 legacy=0x7f72e80c71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.300+0000 7f72ee4b9640 1 -- 172.21.15.88:0/2413438350 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e80c2c80 con 0x7f72e81501e0 2023-10-16T20:39:51.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ee4b9640 1 --2- 172.21.15.88:0/2413438350 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72e81501e0 0x7f72e80c6820 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f72e40097c0 tx=0x7f72e401c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ffbb80c2df20bfc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:51.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ed4b7640 1 -- 172.21.15.88:0/2413438350 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e40ad640 con 0x7f72e81501e0 2023-10-16T20:39:51.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ed4b7640 1 -- 172.21.15.88:0/2413438350 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e40ad7a0 con 0x7f72e81501e0 2023-10-16T20:39:51.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ed4b7640 1 -- 172.21.15.88:0/2413438350 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e40adb30 con 0x7f72e81501e0 2023-10-16T20:39:51.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ef4bb640 1 -- 172.21.15.88:0/2413438350 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72e81501e0 msgr2=0x7f72e80c6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ef4bb640 1 --2- 172.21.15.88:0/2413438350 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72e81501e0 0x7f72e80c6820 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f72e40097c0 tx=0x7f72e401c480 comp rx=0 tx=0).stop 2023-10-16T20:39:51.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ef4bb640 1 -- 172.21.15.88:0/2413438350 shutdown_connections 2023-10-16T20:39:51.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ef4bb640 1 --2- 172.21.15.88:0/2413438350 >> v2:172.21.15.191:3300/0 conn(0x7f72e80c34e0 0x7f72e80c5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ef4bb640 1 --2- 172.21.15.88:0/2413438350 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72e81501e0 0x7f72e80c6820 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ef4bb640 1 -- 172.21.15.88:0/2413438350 >> 172.21.15.88:0/2413438350 conn(0x7f72e806d1e0 msgr2=0x7f72e806d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:51.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ef4bb640 1 -- 172.21.15.88:0/2413438350 shutdown_connections 2023-10-16T20:39:51.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ef4bb640 1 -- 172.21.15.88:0/2413438350 wait complete. 2023-10-16T20:39:51.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ef4bb640 1 Processor -- start 2023-10-16T20:39:51.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ef4bb640 1 -- start start 2023-10-16T20:39:51.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ef4bb640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e80c3000 0x7f72e8146690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ef4bb640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72e80c34e0 0x7f72e81583d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ef4bb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72e81501e0 0x7f72e8155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ef4bb640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e814f700 con 0x7f72e81501e0 2023-10-16T20:39:51.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ef4bb640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e8157c10 con 0x7f72e80c3000 2023-10-16T20:39:51.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.304+0000 7f72ef4bb640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e814f420 con 0x7f72e80c34e0 2023-10-16T20:39:51.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72eecba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72e81501e0 0x7f72e8155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72eecba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72e81501e0 0x7f72e8155160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47650/0 (socket says 172.21.15.88:47650) 2023-10-16T20:39:51.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72eecba640 1 -- 172.21.15.88:0/892455257 learned_addr learned my addr 172.21.15.88:0/892455257 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:51.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72ee4b9640 1 --2- 172.21.15.88:0/892455257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e80c3000 0x7f72e8146690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72edcb8640 1 --2- 172.21.15.88:0/892455257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72e80c34e0 0x7f72e81583d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72eecba640 1 -- 172.21.15.88:0/892455257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72e80c34e0 msgr2=0x7f72e81583d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72eecba640 1 --2- 172.21.15.88:0/892455257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72e80c34e0 0x7f72e81583d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72eecba640 1 -- 172.21.15.88:0/892455257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e80c3000 msgr2=0x7f72e8146690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72eecba640 1 --2- 172.21.15.88:0/892455257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e80c3000 0x7f72e8146690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72eecba640 1 -- 172.21.15.88:0/892455257 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e4009420 con 0x7f72e81501e0 2023-10-16T20:39:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72eecba640 1 --2- 172.21.15.88:0/892455257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72e81501e0 0x7f72e8155160 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f72e000efc0 tx=0x7f72e000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:51.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72deffd640 1 -- 172.21.15.88:0/892455257 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0014480 con 0x7f72e81501e0 2023-10-16T20:39:51.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72deffd640 1 -- 172.21.15.88:0/892455257 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e000f040 con 0x7f72e81501e0 2023-10-16T20:39:51.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72deffd640 1 -- 172.21.15.88:0/892455257 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0014820 con 0x7f72e81501e0 2023-10-16T20:39:51.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72ef4bb640 1 -- 172.21.15.88:0/892455257 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e815a040 con 0x7f72e81501e0 2023-10-16T20:39:51.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72ef4bb640 1 -- 172.21.15.88:0/892455257 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e8157eb0 con 0x7f72e81501e0 2023-10-16T20:39:51.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72ef4bb640 1 -- 172.21.15.88:0/892455257 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e804e9c0 con 0x7f72e81501e0 2023-10-16T20:39:51.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72deffd640 1 -- 172.21.15.88:0/892455257 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f72e0014980 con 0x7f72e81501e0 2023-10-16T20:39:51.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.308+0000 7f72deffd640 1 --2- 172.21.15.88:0/892455257 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f72cc0664d0 0x7f72cc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.312+0000 7f72deffd640 1 -- 172.21.15.88:0/892455257 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f72e0084c20 con 0x7f72e81501e0 2023-10-16T20:39:51.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.312+0000 7f72ee4b9640 1 --2- 172.21.15.88:0/892455257 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f72cc0664d0 0x7f72cc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.312+0000 7f72ee4b9640 1 --2- 172.21.15.88:0/892455257 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f72cc0664d0 0x7f72cc068990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f72e40af000 tx=0x7f72e40a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:51.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.312+0000 7f72deffd640 1 -- 172.21.15.88:0/892455257 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f72e0051d90 con 0x7f72e81501e0 2023-10-16T20:39:51.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.460+0000 7f72ef4bb640 1 -- 172.21.15.88:0/892455257 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f72e80c6e70 con 0x7f72e81501e0 2023-10-16T20:39:51.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.464+0000 7f72deffd640 1 -- 172.21.15.88:0/892455257 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f72e0018020 con 0x7f72e81501e0 2023-10-16T20:39:51.468 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:39:51.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.464+0000 7f72ef4bb640 1 -- 172.21.15.88:0/892455257 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f72cc0664d0 msgr2=0x7f72cc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.464+0000 7f72ef4bb640 1 --2- 172.21.15.88:0/892455257 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f72cc0664d0 0x7f72cc068990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f72e40af000 tx=0x7f72e40a7070 comp rx=0 tx=0).stop 2023-10-16T20:39:51.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.464+0000 7f72ef4bb640 1 -- 172.21.15.88:0/892455257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72e81501e0 msgr2=0x7f72e8155160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.464+0000 7f72ef4bb640 1 --2- 172.21.15.88:0/892455257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72e81501e0 0x7f72e8155160 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f72e000efc0 tx=0x7f72e000c490 comp rx=0 tx=0).stop 2023-10-16T20:39:51.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.464+0000 7f72ef4bb640 1 -- 172.21.15.88:0/892455257 shutdown_connections 2023-10-16T20:39:51.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.464+0000 7f72ef4bb640 1 --2- 172.21.15.88:0/892455257 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f72cc0664d0 0x7f72cc068990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.464+0000 7f72ef4bb640 1 --2- 172.21.15.88:0/892455257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72e81501e0 0x7f72e8155160 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.464+0000 7f72ef4bb640 1 --2- 172.21.15.88:0/892455257 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f72e80c34e0 0x7f72e81583d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.464+0000 7f72ef4bb640 1 --2- 172.21.15.88:0/892455257 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f72e80c3000 0x7f72e8146690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.464+0000 7f72ef4bb640 1 -- 172.21.15.88:0/892455257 >> 172.21.15.88:0/892455257 conn(0x7f72e806d1e0 msgr2=0x7f72e814d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:51.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.464+0000 7f72ef4bb640 1 -- 172.21.15.88:0/892455257 shutdown_connections 2023-10-16T20:39:51.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.464+0000 7f72ef4bb640 1 -- 172.21.15.88:0/892455257 wait complete. 2023-10-16T20:39:51.488 INFO:tasks.mds_thrash:mds thrasher using random seed: 1697488791 2023-10-16T20:39:51.488 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:39:51.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde56417640 1 Processor -- start 2023-10-16T20:39:51.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde56417640 1 -- start start 2023-10-16T20:39:51.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde56417640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fde500cece0 0x7fde500cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde56417640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde50153b50 con 0x7fde500c9bb0 2023-10-16T20:39:51.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde56417640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde500cb2f0 con 0x7fde50153770 2023-10-16T20:39:51.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde56417640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde500cc250 con 0x7fde500cece0 2023-10-16T20:39:51.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4f7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fde500c9bb0 0x7fde500c3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58374/0 (socket says 172.21.15.88:58374) 2023-10-16T20:39:51.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4f7fe640 1 -- 172.21.15.88:0/187404095 learned_addr learned my addr 172.21.15.88:0/187404095 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:51.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde549b4640 1 --2- 172.21.15.88:0/187404095 >> v2:172.21.15.191:3300/0 conn(0x7fde500cece0 0x7fde500cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4effd640 1 -- 172.21.15.88:0/187404095 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676427886 0 0) 0x7fde50153b50 con 0x7fde500c9bb0 2023-10-16T20:39:51.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4effd640 1 -- 172.21.15.88:0/187404095 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde38003530 con 0x7fde500c9bb0 2023-10-16T20:39:51.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4effd640 1 -- 172.21.15.88:0/187404095 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4278478045 0 0) 0x7fde38003530 con 0x7fde500c9bb0 2023-10-16T20:39:51.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4effd640 1 -- 172.21.15.88:0/187404095 >> v2:172.21.15.191:3300/0 conn(0x7fde500cece0 msgr2=0x7fde500cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:51.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4effd640 1 --2- 172.21.15.88:0/187404095 >> v2:172.21.15.191:3300/0 conn(0x7fde500cece0 0x7fde500cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4effd640 1 -- 172.21.15.88:0/187404095 >> v1:172.21.15.191:6789/0 conn(0x7fde50153770 legacy=0x7fde50159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4effd640 1 -- 172.21.15.88:0/187404095 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde500cb5d0 con 0x7fde500c9bb0 2023-10-16T20:39:51.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4effd640 1 -- 172.21.15.88:0/187404095 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7fde44001680 con 0x7fde500c9bb0 2023-10-16T20:39:51.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4effd640 1 -- 172.21.15.88:0/187404095 >> v1:172.21.15.88:6789/0 conn(0x7fde500c9bb0 legacy=0x7fde500c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4effd640 1 --2- 172.21.15.88:0/187404095 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde38003980 0x7fde38003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4effd640 1 --2- 172.21.15.88:0/187404095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde380047d0 0x7fde38004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4effd640 1 --2- 172.21.15.88:0/187404095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde38005190 0x7fde3800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4effd640 1 -- 172.21.15.88:0/187404095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde500cc250 con 0x7fde380047d0 2023-10-16T20:39:51.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4effd640 1 -- 172.21.15.88:0/187404095 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3800c030 con 0x7fde38005190 2023-10-16T20:39:51.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4effd640 1 -- 172.21.15.88:0/187404095 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde3800c1f0 con 0x7fde38003980 2023-10-16T20:39:51.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4f7fe640 1 --2- 172.21.15.88:0/187404095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde380047d0 0x7fde38004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde549b4640 1 --2- 172.21.15.88:0/187404095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde38005190 0x7fde3800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4ffff640 1 --2- 172.21.15.88:0/187404095 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde38003980 0x7fde38003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4f7fe640 1 -- 172.21.15.88:0/187404095 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde38003980 msgr2=0x7fde38003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:51.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4f7fe640 1 --2- 172.21.15.88:0/187404095 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde38003980 0x7fde38003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4f7fe640 1 -- 172.21.15.88:0/187404095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde38005190 msgr2=0x7fde3800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:51.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4f7fe640 1 --2- 172.21.15.88:0/187404095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde38005190 0x7fde3800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde4f7fe640 1 -- 172.21.15.88:0/187404095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde38003530 con 0x7fde380047d0 2023-10-16T20:39:51.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.624+0000 7fde549b4640 1 -- 172.21.15.88:0/187404095 reap_dead start 2023-10-16T20:39:51.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.628+0000 7fde4f7fe640 1 --2- 172.21.15.88:0/187404095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde380047d0 0x7fde38004c50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fde4400c240 tx=0x7fde4401f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9311b5f9933fa905 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:51.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.628+0000 7fde4effd640 1 -- 172.21.15.88:0/187404095 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde44005ae0 con 0x7fde380047d0 2023-10-16T20:39:51.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.628+0000 7fde4effd640 1 -- 172.21.15.88:0/187404095 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde440ab070 con 0x7fde380047d0 2023-10-16T20:39:51.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.628+0000 7fde4effd640 1 -- 172.21.15.88:0/187404095 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde44005ea0 con 0x7fde380047d0 2023-10-16T20:39:51.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.628+0000 7fde56417640 1 -- 172.21.15.88:0/187404095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde380047d0 msgr2=0x7fde38004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.628+0000 7fde56417640 1 --2- 172.21.15.88:0/187404095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde380047d0 0x7fde38004c50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fde4400c240 tx=0x7fde4401f330 comp rx=0 tx=0).stop 2023-10-16T20:39:51.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.628+0000 7fde56417640 1 -- 172.21.15.88:0/187404095 shutdown_connections 2023-10-16T20:39:51.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.628+0000 7fde56417640 1 --2- 172.21.15.88:0/187404095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde380047d0 0x7fde38004c50 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.628+0000 7fde56417640 1 -- 172.21.15.88:0/187404095 >> 172.21.15.88:0/187404095 conn(0x7fde500715c0 msgr2=0x7fde500719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:51.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.628+0000 7fde56417640 1 -- 172.21.15.88:0/187404095 shutdown_connections 2023-10-16T20:39:51.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.628+0000 7fde56417640 1 -- 172.21.15.88:0/187404095 wait complete. 2023-10-16T20:39:51.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde56417640 1 Processor -- start 2023-10-16T20:39:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde56417640 1 -- start start 2023-10-16T20:39:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde56417640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde380047d0 0x7fde5015a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde56417640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde500c8e10 0x7fde50143d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde56417640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde50145f40 0x7fde50145690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde56417640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde50145bd0 con 0x7fde50145f40 2023-10-16T20:39:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde56417640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde50144680 con 0x7fde380047d0 2023-10-16T20:39:51.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde56417640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde50145200 con 0x7fde500c8e10 2023-10-16T20:39:51.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde549b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde50145f40 0x7fde50145690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde4ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde380047d0 0x7fde5015a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde4ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde380047d0 0x7fde5015a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:56828/0 (socket says 172.21.15.88:56828) 2023-10-16T20:39:51.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde4ffff640 1 -- 172.21.15.88:0/58176187 learned_addr learned my addr 172.21.15.88:0/58176187 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:51.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde549b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde50145f40 0x7fde50145690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47660/0 (socket says 172.21.15.88:47660) 2023-10-16T20:39:51.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde4f7fe640 1 --2- 172.21.15.88:0/58176187 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde500c8e10 0x7fde50143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde549b4640 1 -- 172.21.15.88:0/58176187 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde500c8e10 msgr2=0x7fde50143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde549b4640 1 --2- 172.21.15.88:0/58176187 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde500c8e10 0x7fde50143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde549b4640 1 -- 172.21.15.88:0/58176187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde380047d0 msgr2=0x7fde5015a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde549b4640 1 --2- 172.21.15.88:0/58176187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde380047d0 0x7fde5015a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde549b4640 1 -- 172.21.15.88:0/58176187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5015aca0 con 0x7fde50145f40 2023-10-16T20:39:51.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde549b4640 1 --2- 172.21.15.88:0/58176187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde50145f40 0x7fde50145690 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fde500b19e0 tx=0x7fde4800f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:51.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde4cff9640 1 -- 172.21.15.88:0/58176187 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4800e040 con 0x7fde50145f40 2023-10-16T20:39:51.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde4cff9640 1 -- 172.21.15.88:0/58176187 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4800f040 con 0x7fde50145f40 2023-10-16T20:39:51.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde56417640 1 -- 172.21.15.88:0/58176187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde501448d0 con 0x7fde50145f40 2023-10-16T20:39:51.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.632+0000 7fde4cff9640 1 -- 172.21.15.88:0/58176187 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde48015430 con 0x7fde50145f40 2023-10-16T20:39:51.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.636+0000 7fde56417640 1 -- 172.21.15.88:0/58176187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde50144e90 con 0x7fde50145f40 2023-10-16T20:39:51.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.636+0000 7fde56417640 1 -- 172.21.15.88:0/58176187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde50074700 con 0x7fde50145f40 2023-10-16T20:39:51.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.636+0000 7fde4cff9640 1 -- 172.21.15.88:0/58176187 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fde48015590 con 0x7fde50145f40 2023-10-16T20:39:51.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.636+0000 7fde4cff9640 1 --2- 172.21.15.88:0/58176187 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fde3c0664a0 0x7fde3c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.636+0000 7fde4cff9640 1 -- 172.21.15.88:0/58176187 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fde480849b0 con 0x7fde50145f40 2023-10-16T20:39:51.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.636+0000 7fde4ffff640 1 --2- 172.21.15.88:0/58176187 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fde3c0664a0 0x7fde3c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.636+0000 7fde4ffff640 1 --2- 172.21.15.88:0/58176187 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fde3c0664a0 0x7fde3c068960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fde40001880 tx=0x7fde40009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:51.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.640+0000 7fde4cff9640 1 -- 172.21.15.88:0/58176187 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fde480519f0 con 0x7fde50145f40 2023-10-16T20:39:51.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.796+0000 7fde56417640 1 -- 172.21.15.88:0/58176187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde50145d20 con 0x7fde50145f40 2023-10-16T20:39:51.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.800+0000 7fde4cff9640 1 -- 172.21.15.88:0/58176187 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18710 (secure 0 0 0) 0x7fde48055010 con 0x7fde50145f40 2023-10-16T20:39:51.803 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:51.804 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:37:37.392549+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:39:51.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.800+0000 7fde56417640 1 -- 172.21.15.88:0/58176187 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fde3c0664a0 msgr2=0x7fde3c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.800+0000 7fde56417640 1 --2- 172.21.15.88:0/58176187 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fde3c0664a0 0x7fde3c068960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fde40001880 tx=0x7fde40009040 comp rx=0 tx=0).stop 2023-10-16T20:39:51.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.800+0000 7fde56417640 1 -- 172.21.15.88:0/58176187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde50145f40 msgr2=0x7fde50145690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.800+0000 7fde56417640 1 --2- 172.21.15.88:0/58176187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde50145f40 0x7fde50145690 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fde500b19e0 tx=0x7fde4800f1b0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.800+0000 7fde56417640 1 -- 172.21.15.88:0/58176187 shutdown_connections 2023-10-16T20:39:51.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.800+0000 7fde56417640 1 --2- 172.21.15.88:0/58176187 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fde3c0664a0 0x7fde3c068960 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.800+0000 7fde56417640 1 --2- 172.21.15.88:0/58176187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde50145f40 0x7fde50145690 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.800+0000 7fde56417640 1 --2- 172.21.15.88:0/58176187 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fde500c8e10 0x7fde50143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.800+0000 7fde56417640 1 --2- 172.21.15.88:0/58176187 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde380047d0 0x7fde5015a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.800+0000 7fde56417640 1 -- 172.21.15.88:0/58176187 >> 172.21.15.88:0/58176187 conn(0x7fde500715c0 msgr2=0x7fde500cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:51.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.800+0000 7fde56417640 1 -- 172.21.15.88:0/58176187 shutdown_connections 2023-10-16T20:39:51.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.800+0000 7fde56417640 1 -- 172.21.15.88:0/58176187 wait complete. 2023-10-16T20:39:51.824 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T20:36:43.646418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T20:36:53.546513+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T20:36:54.143703+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T20:36:57.314642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T20:39:51.825 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T20:39:51.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.960+0000 7f677f4c9640 1 Processor -- start 2023-10-16T20:39:51.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.960+0000 7f677f4c9640 1 -- start start 2023-10-16T20:39:51.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.960+0000 7f677f4c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f677806ae00 0x7f67780c20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.960+0000 7f677f4c9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6778157f00 0x7f67780c2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.960+0000 7f677f4c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f677815dba0 con 0x7f6778157b20 2023-10-16T20:39:51.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.960+0000 7f677f4c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f677815d9b0 con 0x7f677806ae00 2023-10-16T20:39:51.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.960+0000 7f677f4c9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6778072a10 con 0x7f6778157f00 2023-10-16T20:39:51.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.960+0000 7f677ca64640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f677806ae00 0x7f67780c20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.960+0000 7f677ca64640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f677806ae00 0x7f67780c20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47662/0 (socket says 172.21.15.88:47662) 2023-10-16T20:39:51.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.960+0000 7f677ca64640 1 -- 172.21.15.88:0/4272306295 learned_addr learned my addr 172.21.15.88:0/4272306295 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:51.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.960+0000 7f677d265640 1 --1- 172.21.15.88:0/4272306295 >> v1:172.21.15.88:6789/0 conn(0x7f6778157b20 0x7f6778069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58382/0 (socket says 172.21.15.88:58382) 2023-10-16T20:39:51.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.964+0000 7f677da66640 1 --2- 172.21.15.88:0/4272306295 >> v2:172.21.15.191:3300/0 conn(0x7f6778157f00 0x7f67780c2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.964+0000 7f677ca64640 1 -- 172.21.15.88:0/4272306295 >> v2:172.21.15.191:3300/0 conn(0x7f6778157f00 msgr2=0x7f67780c2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:51.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.964+0000 7f677ca64640 1 --2- 172.21.15.88:0/4272306295 >> v2:172.21.15.191:3300/0 conn(0x7f6778157f00 0x7f67780c2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.964+0000 7f677ca64640 1 -- 172.21.15.88:0/4272306295 >> v1:172.21.15.88:6789/0 conn(0x7f6778157b20 legacy=0x7f6778069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.964+0000 7f677ca64640 1 -- 172.21.15.88:0/4272306295 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677815de80 con 0x7f677806ae00 2023-10-16T20:39:51.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.964+0000 7f676ffff640 1 -- 172.21.15.88:0/4272306295 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524952016 0 0) 0x7f677815dba0 con 0x7f6778157b20 2023-10-16T20:39:51.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.964+0000 7f677ca64640 1 --2- 172.21.15.88:0/4272306295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f677806ae00 0x7f67780c20e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f677806c820 tx=0x7f676801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9677310e7c90ff93 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:51.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.964+0000 7f676ffff640 1 -- 172.21.15.88:0/4272306295 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67680a89f0 con 0x7f677806ae00 2023-10-16T20:39:51.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.964+0000 7f676ffff640 1 -- 172.21.15.88:0/4272306295 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67680a8b50 con 0x7f677806ae00 2023-10-16T20:39:51.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.964+0000 7f676ffff640 1 -- 172.21.15.88:0/4272306295 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67680b0440 con 0x7f677806ae00 2023-10-16T20:39:51.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.964+0000 7f677f4c9640 1 -- 172.21.15.88:0/4272306295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f677806ae00 msgr2=0x7f67780c20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.964+0000 7f677f4c9640 1 --2- 172.21.15.88:0/4272306295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f677806ae00 0x7f67780c20e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f677806c820 tx=0x7f676801ecf0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.964+0000 7f677f4c9640 1 -- 172.21.15.88:0/4272306295 shutdown_connections 2023-10-16T20:39:51.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.964+0000 7f677f4c9640 1 --2- 172.21.15.88:0/4272306295 >> v2:172.21.15.191:3300/0 conn(0x7f6778157f00 0x7f67780c2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.964+0000 7f677f4c9640 1 --2- 172.21.15.88:0/4272306295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f677806ae00 0x7f67780c20e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.964+0000 7f677f4c9640 1 -- 172.21.15.88:0/4272306295 >> 172.21.15.88:0/4272306295 conn(0x7f67780ca9f0 msgr2=0x7f67780cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:51.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677f4c9640 1 -- 172.21.15.88:0/4272306295 shutdown_connections 2023-10-16T20:39:51.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677f4c9640 1 -- 172.21.15.88:0/4272306295 wait complete. 2023-10-16T20:39:51.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677f4c9640 1 Processor -- start 2023-10-16T20:39:51.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677f4c9640 1 -- start start 2023-10-16T20:39:51.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677f4c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f677806ae00 0x7f67780cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677f4c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6778157b20 0x7f67781f3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677f4c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6778157f00 0x7f67781f37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677f4c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677815b070 con 0x7f677806ae00 2023-10-16T20:39:51.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677f4c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67781f6290 con 0x7f6778157b20 2023-10-16T20:39:51.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677f4c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67781f64e0 con 0x7f6778157f00 2023-10-16T20:39:51.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677d265640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f677806ae00 0x7f67780cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677d265640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f677806ae00 0x7f67780cb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47678/0 (socket says 172.21.15.88:47678) 2023-10-16T20:39:51.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677d265640 1 -- 172.21.15.88:0/1083407861 learned_addr learned my addr 172.21.15.88:0/1083407861 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:51.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677ca64640 1 --2- 172.21.15.88:0/1083407861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6778157b20 0x7f67781f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677d265640 1 -- 172.21.15.88:0/1083407861 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6778157f00 msgr2=0x7f67781f37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677d265640 1 --2- 172.21.15.88:0/1083407861 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6778157f00 0x7f67781f37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677d265640 1 -- 172.21.15.88:0/1083407861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6778157b20 msgr2=0x7f67781f3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:51.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677d265640 1 --2- 172.21.15.88:0/1083407861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6778157b20 0x7f67781f3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:51.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677d265640 1 -- 172.21.15.88:0/1083407861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67680094d0 con 0x7f677806ae00 2023-10-16T20:39:51.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677ca64640 1 --2- 172.21.15.88:0/1083407861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6778157b20 0x7f67781f3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:51.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677d265640 1 --2- 172.21.15.88:0/1083407861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f677806ae00 0x7f67780cb030 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f67740011f0 tx=0x7f677400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:51.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f676dffb640 1 -- 172.21.15.88:0/1083407861 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67740074e0 con 0x7f677806ae00 2023-10-16T20:39:51.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f676dffb640 1 -- 172.21.15.88:0/1083407861 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6774007640 con 0x7f677806ae00 2023-10-16T20:39:51.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f676dffb640 1 -- 172.21.15.88:0/1083407861 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6774004a10 con 0x7f677806ae00 2023-10-16T20:39:51.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.968+0000 7f677f4c9640 1 -- 172.21.15.88:0/1083407861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67781f67c0 con 0x7f677806ae00 2023-10-16T20:39:51.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.972+0000 7f677f4c9640 1 -- 172.21.15.88:0/1083407861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67781f6d80 con 0x7f677806ae00 2023-10-16T20:39:51.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.972+0000 7f677f4c9640 1 -- 172.21.15.88:0/1083407861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6778074700 con 0x7f677806ae00 2023-10-16T20:39:51.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.972+0000 7f676dffb640 1 -- 172.21.15.88:0/1083407861 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6774004b70 con 0x7f677806ae00 2023-10-16T20:39:51.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.972+0000 7f676dffb640 1 --2- 172.21.15.88:0/1083407861 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f675c0664a0 0x7f675c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:51.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.972+0000 7f676dffb640 1 -- 172.21.15.88:0/1083407861 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f67740835e0 con 0x7f677806ae00 2023-10-16T20:39:51.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.972+0000 7f677ca64640 1 --2- 172.21.15.88:0/1083407861 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f675c0664a0 0x7f675c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:51.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.972+0000 7f677ca64640 1 --2- 172.21.15.88:0/1083407861 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f675c0664a0 0x7f675c068960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f67680b2000 tx=0x7f67680aa070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:51.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:51.976+0000 7f676dffb640 1 -- 172.21.15.88:0/1083407861 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6774050600 con 0x7f677806ae00 2023-10-16T20:39:52.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.124+0000 7f677f4c9640 1 -- 172.21.15.88:0/1083407861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6778156420 con 0x7f677806ae00 2023-10-16T20:39:52.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.124+0000 7f676dffb640 1 -- 172.21.15.88:0/1083407861 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f67740180a0 con 0x7f677806ae00 2023-10-16T20:39:52.130 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-10-16T20:39:52.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.128+0000 7f677f4c9640 1 -- 172.21.15.88:0/1083407861 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f675c0664a0 msgr2=0x7f675c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.128+0000 7f677f4c9640 1 --2- 172.21.15.88:0/1083407861 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f675c0664a0 0x7f675c068960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f67680b2000 tx=0x7f67680aa070 comp rx=0 tx=0).stop 2023-10-16T20:39:52.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.128+0000 7f677f4c9640 1 -- 172.21.15.88:0/1083407861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f677806ae00 msgr2=0x7f67780cb030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.128+0000 7f677f4c9640 1 --2- 172.21.15.88:0/1083407861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f677806ae00 0x7f67780cb030 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f67740011f0 tx=0x7f677400c9b0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.128+0000 7f677f4c9640 1 -- 172.21.15.88:0/1083407861 shutdown_connections 2023-10-16T20:39:52.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.128+0000 7f677f4c9640 1 --2- 172.21.15.88:0/1083407861 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f675c0664a0 0x7f675c068960 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.128+0000 7f677f4c9640 1 --2- 172.21.15.88:0/1083407861 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6778157f00 0x7f67781f37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.128+0000 7f677f4c9640 1 --2- 172.21.15.88:0/1083407861 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6778157b20 0x7f67781f3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.128+0000 7f677f4c9640 1 --2- 172.21.15.88:0/1083407861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f677806ae00 0x7f67780cb030 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.128+0000 7f677f4c9640 1 -- 172.21.15.88:0/1083407861 >> 172.21.15.88:0/1083407861 conn(0x7f67780ca9f0 msgr2=0x7f67780c5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:52.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.128+0000 7f677f4c9640 1 -- 172.21.15.88:0/1083407861 shutdown_connections 2023-10-16T20:39:52.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.128+0000 7f677f4c9640 1 -- 172.21.15.88:0/1083407861 wait complete. 2023-10-16T20:39:52.151 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T20:39:52.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.284+0000 7f61c809e640 1 Processor -- start 2023-10-16T20:39:52.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.288+0000 7f61c809e640 1 -- start start 2023-10-16T20:39:52.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.288+0000 7f61c809e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f61c00c4260 0x7f61c00c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:52.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.288+0000 7f61c809e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f61c00c4640 0x7f61c01573e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:52.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.288+0000 7f61c809e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c0069e90 con 0x7f61c00c5a60 2023-10-16T20:39:52.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.288+0000 7f61c809e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c00716a0 con 0x7f61c00c4260 2023-10-16T20:39:52.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.288+0000 7f61c809e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c006aea0 con 0x7f61c00c4640 2023-10-16T20:39:52.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.288+0000 7f61c5e3a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f61c00c4260 0x7f61c00c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:52.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.288+0000 7f61c5639640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f61c00c5a60 0x7f61c01510d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58390/0 (socket says 172.21.15.88:58390) 2023-10-16T20:39:52.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.288+0000 7f61c5639640 1 -- 172.21.15.88:0/2943627978 learned_addr learned my addr 172.21.15.88:0/2943627978 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:52.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.288+0000 7f61c663b640 1 --2- 172.21.15.88:0/2943627978 >> v2:172.21.15.191:3300/0 conn(0x7f61c00c4640 0x7f61c01573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:52.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.288+0000 7f61c5e3a640 1 -- 172.21.15.88:0/2943627978 >> v2:172.21.15.191:3300/0 conn(0x7f61c00c4640 msgr2=0x7f61c01573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:52.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.288+0000 7f61c5e3a640 1 --2- 172.21.15.88:0/2943627978 >> v2:172.21.15.191:3300/0 conn(0x7f61c00c4640 0x7f61c01573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.288+0000 7f61c4e38640 1 -- 172.21.15.88:0/2943627978 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578359706 0 0) 0x7f61c0069e90 con 0x7f61c00c5a60 2023-10-16T20:39:52.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.288+0000 7f61c5e3a640 1 -- 172.21.15.88:0/2943627978 >> v1:172.21.15.88:6789/0 conn(0x7f61c00c5a60 legacy=0x7f61c01510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.288+0000 7f61c5e3a640 1 -- 172.21.15.88:0/2943627978 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c01581e0 con 0x7f61c00c4260 2023-10-16T20:39:52.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.292+0000 7f61c5e3a640 1 --2- 172.21.15.88:0/2943627978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61c00c4260 0x7f61c00c5520 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f61c00c1cc0 tx=0x7f61bc01c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2744312f412685 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:52.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.292+0000 7f61c4e38640 1 -- 172.21.15.88:0/2943627978 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc0ab600 con 0x7f61c00c4260 2023-10-16T20:39:52.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.292+0000 7f61c4e38640 1 -- 172.21.15.88:0/2943627978 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61bc0a7050 con 0x7f61c00c4260 2023-10-16T20:39:52.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.292+0000 7f61c4e38640 1 -- 172.21.15.88:0/2943627978 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc0ab980 con 0x7f61c00c4260 2023-10-16T20:39:52.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.292+0000 7f61c809e640 1 -- 172.21.15.88:0/2943627978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61c00c4260 msgr2=0x7f61c00c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.292+0000 7f61c809e640 1 --2- 172.21.15.88:0/2943627978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61c00c4260 0x7f61c00c5520 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f61c00c1cc0 tx=0x7f61bc01c310 comp rx=0 tx=0).stop 2023-10-16T20:39:52.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.292+0000 7f61c809e640 1 -- 172.21.15.88:0/2943627978 shutdown_connections 2023-10-16T20:39:52.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.292+0000 7f61c809e640 1 --2- 172.21.15.88:0/2943627978 >> v2:172.21.15.191:3300/0 conn(0x7f61c00c4640 0x7f61c01573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.292+0000 7f61c809e640 1 --2- 172.21.15.88:0/2943627978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61c00c4260 0x7f61c00c5520 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.292+0000 7f61c809e640 1 -- 172.21.15.88:0/2943627978 >> 172.21.15.88:0/2943627978 conn(0x7f61c00ca9f0 msgr2=0x7f61c00cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:52.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.292+0000 7f61c809e640 1 -- 172.21.15.88:0/2943627978 shutdown_connections 2023-10-16T20:39:52.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.292+0000 7f61c809e640 1 -- 172.21.15.88:0/2943627978 wait complete. 2023-10-16T20:39:52.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.292+0000 7f61c809e640 1 Processor -- start 2023-10-16T20:39:52.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.292+0000 7f61c809e640 1 -- start start 2023-10-16T20:39:52.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c809e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61c00c4260 0x7f61c01483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:52.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c809e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61c00c4640 0x7f61c014f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:52.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c809e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61c00c5a60 0x7f61c01551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:52.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c809e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c0149a30 con 0x7f61c00c5a60 2023-10-16T20:39:52.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c809e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c014e5e0 con 0x7f61c00c4640 2023-10-16T20:39:52.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c809e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c00701c0 con 0x7f61c00c4260 2023-10-16T20:39:52.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c663b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61c00c5a60 0x7f61c01551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:52.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c663b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61c00c5a60 0x7f61c01551f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47692/0 (socket says 172.21.15.88:47692) 2023-10-16T20:39:52.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c663b640 1 -- 172.21.15.88:0/3113350766 learned_addr learned my addr 172.21.15.88:0/3113350766 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:52.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c5639640 1 --2- 172.21.15.88:0/3113350766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61c00c4640 0x7f61c014f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:52.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c5e3a640 1 --2- 172.21.15.88:0/3113350766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61c00c4260 0x7f61c01483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:52.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c663b640 1 -- 172.21.15.88:0/3113350766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61c00c4260 msgr2=0x7f61c01483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c663b640 1 --2- 172.21.15.88:0/3113350766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61c00c4260 0x7f61c01483d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c663b640 1 -- 172.21.15.88:0/3113350766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61c00c4640 msgr2=0x7f61c014f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c663b640 1 --2- 172.21.15.88:0/3113350766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61c00c4640 0x7f61c014f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c663b640 1 -- 172.21.15.88:0/3113350766 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61bc009420 con 0x7f61c00c5a60 2023-10-16T20:39:52.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c663b640 1 --2- 172.21.15.88:0/3113350766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61c00c5a60 0x7f61c01551f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f61b8009dd0 tx=0x7f61b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:52.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61b67fc640 1 -- 172.21.15.88:0/3113350766 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b8014480 con 0x7f61c00c5a60 2023-10-16T20:39:52.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61b67fc640 1 -- 172.21.15.88:0/3113350766 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b800f040 con 0x7f61c00c5a60 2023-10-16T20:39:52.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61b67fc640 1 -- 172.21.15.88:0/3113350766 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b8014840 con 0x7f61c00c5a60 2023-10-16T20:39:52.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c809e640 1 -- 172.21.15.88:0/3113350766 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c0148910 con 0x7f61c00c5a60 2023-10-16T20:39:52.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c809e640 1 -- 172.21.15.88:0/3113350766 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c014ebe0 con 0x7f61c00c5a60 2023-10-16T20:39:52.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.296+0000 7f61c809e640 1 -- 172.21.15.88:0/3113350766 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c0074700 con 0x7f61c00c5a60 2023-10-16T20:39:52.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.300+0000 7f61b67fc640 1 -- 172.21.15.88:0/3113350766 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f61b80149a0 con 0x7f61c00c5a60 2023-10-16T20:39:52.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.300+0000 7f61b67fc640 1 --2- 172.21.15.88:0/3113350766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f61a40664d0 0x7f61a4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:52.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.300+0000 7f61b67fc640 1 -- 172.21.15.88:0/3113350766 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f61b8084d00 con 0x7f61c00c5a60 2023-10-16T20:39:52.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.300+0000 7f61c5e3a640 1 --2- 172.21.15.88:0/3113350766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f61a40664d0 0x7f61a4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:52.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.300+0000 7f61c5e3a640 1 --2- 172.21.15.88:0/3113350766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f61a40664d0 0x7f61a4068990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f61bc0ad000 tx=0x7f61bc0a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:52.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.304+0000 7f61b67fc640 1 -- 172.21.15.88:0/3113350766 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f61b8051e70 con 0x7f61c00c5a60 2023-10-16T20:39:52.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.456+0000 7f61c809e640 1 -- 172.21.15.88:0/3113350766 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f61c014ebe0 con 0x7f61c00c5a60 2023-10-16T20:39:52.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.456+0000 7f61b67fc640 1 -- 172.21.15.88:0/3113350766 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f61b8018020 con 0x7f61c00c5a60 2023-10-16T20:39:52.461 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:39:52.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.456+0000 7f61c809e640 1 -- 172.21.15.88:0/3113350766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f61a40664d0 msgr2=0x7f61a4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.456+0000 7f61c809e640 1 --2- 172.21.15.88:0/3113350766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f61a40664d0 0x7f61a4068990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f61bc0ad000 tx=0x7f61bc0a6000 comp rx=0 tx=0).stop 2023-10-16T20:39:52.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.456+0000 7f61c809e640 1 -- 172.21.15.88:0/3113350766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61c00c5a60 msgr2=0x7f61c01551f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.456+0000 7f61c809e640 1 --2- 172.21.15.88:0/3113350766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61c00c5a60 0x7f61c01551f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f61b8009dd0 tx=0x7f61b800c490 comp rx=0 tx=0).stop 2023-10-16T20:39:52.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.456+0000 7f61c809e640 1 -- 172.21.15.88:0/3113350766 shutdown_connections 2023-10-16T20:39:52.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.456+0000 7f61c809e640 1 --2- 172.21.15.88:0/3113350766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f61a40664d0 0x7f61a4068990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.456+0000 7f61c809e640 1 --2- 172.21.15.88:0/3113350766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61c00c5a60 0x7f61c01551f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.456+0000 7f61c809e640 1 --2- 172.21.15.88:0/3113350766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f61c00c4640 0x7f61c014f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.456+0000 7f61c809e640 1 --2- 172.21.15.88:0/3113350766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f61c00c4260 0x7f61c01483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.456+0000 7f61c809e640 1 -- 172.21.15.88:0/3113350766 >> 172.21.15.88:0/3113350766 conn(0x7f61c00ca9f0 msgr2=0x7f61c00c6020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:52.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.460+0000 7f61c809e640 1 -- 172.21.15.88:0/3113350766 shutdown_connections 2023-10-16T20:39:52.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.460+0000 7f61c809e640 1 -- 172.21.15.88:0/3113350766 wait complete. 2023-10-16T20:39:52.481 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T20:39:52.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.616+0000 7f3a81b65640 1 Processor -- start 2023-10-16T20:39:52.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.616+0000 7f3a81b65640 1 -- start start 2023-10-16T20:39:52.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.616+0000 7f3a81b65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a7c0c66e0 0x7f3a7c15a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:52.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a81b65640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3a7c0c6ac0 0x7f3a7c150150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:52.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a81b65640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7c070f40 con 0x7f3a7c0cb5c0 2023-10-16T20:39:52.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a81b65640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a7c15a790 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a81b65640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c06a2d0 con 0x7f3a7c0c66e0 2023-10-16T20:39:52.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a73fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3a7c0c6ac0 0x7f3a7c150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:52.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a73fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3a7c0c6ac0 0x7f3a7c150150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47694/0 (socket says 172.21.15.88:47694) 2023-10-16T20:39:52.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a73fff640 1 -- 172.21.15.88:0/757151210 learned_addr learned my addr 172.21.15.88:0/757151210 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:52.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a7b7fe640 1 --2- 172.21.15.88:0/757151210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a7c0c66e0 0x7f3a7c15a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:52.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a73fff640 1 -- 172.21.15.88:0/757151210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a7c0c66e0 msgr2=0x7f3a7c15a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:52.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a73fff640 1 --2- 172.21.15.88:0/757151210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a7c0c66e0 0x7f3a7c15a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a73fff640 1 -- 172.21.15.88:0/757151210 >> v1:172.21.15.88:6789/0 conn(0x7f3a7c0cb5c0 legacy=0x7f3a7c071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a73fff640 1 -- 172.21.15.88:0/757151210 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c06a5b0 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a7b7fe640 1 --2- 172.21.15.88:0/757151210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a7c0c66e0 0x7f3a7c15a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:52.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a7affd640 1 -- 172.21.15.88:0/757151210 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360252394 0 0) 0x7f3a7c070f40 con 0x7f3a7c0cb5c0 2023-10-16T20:39:52.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a73fff640 1 --2- 172.21.15.88:0/757151210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7c0c6ac0 0x7f3a7c150150 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400bb10 tx=0x7f3a6401eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e8327006f03946 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:52.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a7affd640 1 -- 172.21.15.88:0/757151210 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a640a7a40 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a7affd640 1 -- 172.21.15.88:0/757151210 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a640a7ba0 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.620+0000 7f3a7affd640 1 -- 172.21.15.88:0/757151210 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a640af440 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a81b65640 1 -- 172.21.15.88:0/757151210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7c0c6ac0 msgr2=0x7f3a7c150150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a81b65640 1 --2- 172.21.15.88:0/757151210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7c0c6ac0 0x7f3a7c150150 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400bb10 tx=0x7f3a6401eb80 comp rx=0 tx=0).stop 2023-10-16T20:39:52.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a81b65640 1 -- 172.21.15.88:0/757151210 shutdown_connections 2023-10-16T20:39:52.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a81b65640 1 --2- 172.21.15.88:0/757151210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7c0c6ac0 0x7f3a7c150150 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a81b65640 1 --2- 172.21.15.88:0/757151210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a7c0c66e0 0x7f3a7c15a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a81b65640 1 -- 172.21.15.88:0/757151210 >> 172.21.15.88:0/757151210 conn(0x7f3a7c0ca9f0 msgr2=0x7f3a7c0cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:52.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a81b65640 1 -- 172.21.15.88:0/757151210 shutdown_connections 2023-10-16T20:39:52.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a81b65640 1 -- 172.21.15.88:0/757151210 wait complete. 2023-10-16T20:39:52.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a81b65640 1 Processor -- start 2023-10-16T20:39:52.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a81b65640 1 -- start start 2023-10-16T20:39:52.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a81b65640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a7c0c66e0 0x7f3a7c14aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:52.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a81b65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7c0c6ac0 0x7f3a7c1f55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:52.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a81b65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a7c0cb5c0 0x7f3a7c1f5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:52.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a81b65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c148c90 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a81b65640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c15fcf0 con 0x7f3a7c0c66e0 2023-10-16T20:39:52.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a81b65640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c15ffd0 con 0x7f3a7c0cb5c0 2023-10-16T20:39:52.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a73fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7c0c6ac0 0x7f3a7c1f55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:52.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a73fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7c0c6ac0 0x7f3a7c1f55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47698/0 (socket says 172.21.15.88:47698) 2023-10-16T20:39:52.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a73fff640 1 -- 172.21.15.88:0/2139408836 learned_addr learned my addr 172.21.15.88:0/2139408836 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:52.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a7b7fe640 1 --2- 172.21.15.88:0/2139408836 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a7c0c66e0 0x7f3a7c14aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:52.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a7bfff640 1 --2- 172.21.15.88:0/2139408836 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a7c0cb5c0 0x7f3a7c1f5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:52.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a73fff640 1 -- 172.21.15.88:0/2139408836 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a7c0cb5c0 msgr2=0x7f3a7c1f5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.624+0000 7f3a73fff640 1 --2- 172.21.15.88:0/2139408836 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a7c0cb5c0 0x7f3a7c1f5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.628+0000 7f3a73fff640 1 -- 172.21.15.88:0/2139408836 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a7c0c66e0 msgr2=0x7f3a7c14aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.628+0000 7f3a73fff640 1 --2- 172.21.15.88:0/2139408836 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a7c0c66e0 0x7f3a7c14aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.628+0000 7f3a73fff640 1 -- 172.21.15.88:0/2139408836 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6400b770 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.628+0000 7f3a7b7fe640 1 --2- 172.21.15.88:0/2139408836 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a7c0c66e0 0x7f3a7c14aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:52.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.628+0000 7f3a73fff640 1 --2- 172.21.15.88:0/2139408836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7c0c6ac0 0x7f3a7c1f55f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400b2c0 tx=0x7f3a6401b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:52.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.628+0000 7f3a78ff9640 1 -- 172.21.15.88:0/2139408836 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a640a7a40 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.628+0000 7f3a78ff9640 1 -- 172.21.15.88:0/2139408836 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a640ba440 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.628+0000 7f3a78ff9640 1 -- 172.21.15.88:0/2139408836 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a640ba810 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.628+0000 7f3a81b65640 1 -- 172.21.15.88:0/2139408836 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a7c1488b0 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.628+0000 7f3a81b65640 1 -- 172.21.15.88:0/2139408836 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7c14af90 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.628+0000 7f3a81b65640 1 -- 172.21.15.88:0/2139408836 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7c160190 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.628+0000 7f3a78ff9640 1 -- 172.21.15.88:0/2139408836 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3a640b93e0 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.628+0000 7f3a78ff9640 1 --2- 172.21.15.88:0/2139408836 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3a5c0664a0 0x7f3a5c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:52.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.628+0000 7f3a78ff9640 1 -- 172.21.15.88:0/2139408836 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3a6411eb20 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.628+0000 7f3a7b7fe640 1 --2- 172.21.15.88:0/2139408836 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3a5c0664a0 0x7f3a5c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:52.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.632+0000 7f3a7b7fe640 1 --2- 172.21.15.88:0/2139408836 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3a5c0664a0 0x7f3a5c068960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009970 tx=0x7f3a6c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:52.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.632+0000 7f3a78ff9640 1 -- 172.21.15.88:0/2139408836 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3a7c160190 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.784+0000 7f3a81b65640 1 -- 172.21.15.88:0/2139408836 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3a7c160190 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.784+0000 7f3a78ff9640 1 -- 172.21.15.88:0/2139408836 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3a7c160190 con 0x7f3a7c0c6ac0 2023-10-16T20:39:52.789 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2023-10-16T20:39:52.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.784+0000 7f3a81b65640 1 -- 172.21.15.88:0/2139408836 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3a5c0664a0 msgr2=0x7f3a5c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.784+0000 7f3a81b65640 1 --2- 172.21.15.88:0/2139408836 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3a5c0664a0 0x7f3a5c068960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009970 tx=0x7f3a6c008040 comp rx=0 tx=0).stop 2023-10-16T20:39:52.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.784+0000 7f3a81b65640 1 -- 172.21.15.88:0/2139408836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7c0c6ac0 msgr2=0x7f3a7c1f55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.784+0000 7f3a81b65640 1 --2- 172.21.15.88:0/2139408836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7c0c6ac0 0x7f3a7c1f55f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400b2c0 tx=0x7f3a6401b920 comp rx=0 tx=0).stop 2023-10-16T20:39:52.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.784+0000 7f3a81b65640 1 -- 172.21.15.88:0/2139408836 shutdown_connections 2023-10-16T20:39:52.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.788+0000 7f3a81b65640 1 --2- 172.21.15.88:0/2139408836 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3a5c0664a0 0x7f3a5c068960 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.788+0000 7f3a81b65640 1 --2- 172.21.15.88:0/2139408836 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a7c0cb5c0 0x7f3a7c1f5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.788+0000 7f3a81b65640 1 --2- 172.21.15.88:0/2139408836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7c0c6ac0 0x7f3a7c1f55f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.788+0000 7f3a81b65640 1 --2- 172.21.15.88:0/2139408836 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a7c0c66e0 0x7f3a7c14aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.788+0000 7f3a81b65640 1 -- 172.21.15.88:0/2139408836 >> 172.21.15.88:0/2139408836 conn(0x7f3a7c0ca9f0 msgr2=0x7f3a7c0c1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:52.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.788+0000 7f3a81b65640 1 -- 172.21.15.88:0/2139408836 shutdown_connections 2023-10-16T20:39:52.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.788+0000 7f3a81b65640 1 -- 172.21.15.88:0/2139408836 wait complete. 2023-10-16T20:39:52.809 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T20:39:52.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.944+0000 7fc037046640 1 Processor -- start 2023-10-16T20:39:52.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.948+0000 7fc037046640 1 -- start start 2023-10-16T20:39:52.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.948+0000 7fc037046640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0301517e0 0x7fc0301573e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:52.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.948+0000 7fc037046640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc030069e90 con 0x7fc0300c4640 2023-10-16T20:39:52.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.948+0000 7fc037046640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03006aea0 con 0x7fc0300c4260 2023-10-16T20:39:52.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.948+0000 7fc037046640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0300716a0 con 0x7fc0301517e0 2023-10-16T20:39:52.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.948+0000 7fc02ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc0300c4640 0x7fc0301510d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58402/0 (socket says 172.21.15.88:58402) 2023-10-16T20:39:52.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.948+0000 7fc02ffff640 1 -- 172.21.15.88:0/1424608843 learned_addr learned my addr 172.21.15.88:0/1424608843 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:52.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.948+0000 7fc0355e3640 1 --2- 172.21.15.88:0/1424608843 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0301517e0 0x7fc0301573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:52.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.948+0000 7fc02f7fe640 1 -- 172.21.15.88:0/1424608843 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867627460 0 0) 0x7fc030069e90 con 0x7fc0300c4640 2023-10-16T20:39:52.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.948+0000 7fc02f7fe640 1 -- 172.21.15.88:0/1424608843 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc020003530 con 0x7fc0300c4640 2023-10-16T20:39:52.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.948+0000 7fc02f7fe640 1 -- 172.21.15.88:0/1424608843 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888264295 0 0) 0x7fc03006aea0 con 0x7fc0300c4260 2023-10-16T20:39:52.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.948+0000 7fc02f7fe640 1 -- 172.21.15.88:0/1424608843 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc030069e90 con 0x7fc0300c4260 2023-10-16T20:39:52.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.948+0000 7fc0355e3640 1 -- 172.21.15.88:0/1424608843 >> v1:172.21.15.191:6789/0 conn(0x7fc0300c4260 legacy=0x7fc0300c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.948+0000 7fc0355e3640 1 -- 172.21.15.88:0/1424608843 >> v1:172.21.15.88:6789/0 conn(0x7fc0300c4640 legacy=0x7fc0301510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.948+0000 7fc02f7fe640 1 -- 172.21.15.88:0/1424608843 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1990569895 0 0) 0x7fc020003530 con 0x7fc0300c4640 2023-10-16T20:39:52.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.948+0000 7fc0355e3640 1 -- 172.21.15.88:0/1424608843 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0301581e0 con 0x7fc0301517e0 2023-10-16T20:39:52.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.952+0000 7fc0355e3640 1 --2- 172.21.15.88:0/1424608843 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0301517e0 0x7fc0301573e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc028009dc0 tx=0x7fc02801c770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9ca15d0bd975f3e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:52.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.952+0000 7fc02f7fe640 1 -- 172.21.15.88:0/1424608843 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01c001680 con 0x7fc0301517e0 2023-10-16T20:39:52.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.952+0000 7fc02f7fe640 1 -- 172.21.15.88:0/1424608843 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0280a55b0 con 0x7fc0301517e0 2023-10-16T20:39:52.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.952+0000 7fc02f7fe640 1 -- 172.21.15.88:0/1424608843 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0280a5940 con 0x7fc0301517e0 2023-10-16T20:39:52.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.952+0000 7fc037046640 1 -- 172.21.15.88:0/1424608843 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0301517e0 msgr2=0x7fc0301573e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.952+0000 7fc037046640 1 --2- 172.21.15.88:0/1424608843 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0301517e0 0x7fc0301573e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc028009dc0 tx=0x7fc02801c770 comp rx=0 tx=0).stop 2023-10-16T20:39:52.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.952+0000 7fc037046640 1 -- 172.21.15.88:0/1424608843 shutdown_connections 2023-10-16T20:39:52.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.952+0000 7fc037046640 1 --2- 172.21.15.88:0/1424608843 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0301517e0 0x7fc0301573e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.952+0000 7fc037046640 1 -- 172.21.15.88:0/1424608843 >> 172.21.15.88:0/1424608843 conn(0x7fc0300ca9f0 msgr2=0x7fc0300cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:52.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.952+0000 7fc037046640 1 -- 172.21.15.88:0/1424608843 shutdown_connections 2023-10-16T20:39:52.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.952+0000 7fc037046640 1 -- 172.21.15.88:0/1424608843 wait complete. 2023-10-16T20:39:52.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc037046640 1 Processor -- start 2023-10-16T20:39:52.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc037046640 1 -- start start 2023-10-16T20:39:52.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc037046640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0300c4260 0x7fc03014c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:52.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc037046640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0300c4640 0x7fc030145940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:52.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc037046640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc0301517e0 0x7fc030142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:52.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc037046640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc030142930 con 0x7fc0300c4260 2023-10-16T20:39:52.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc037046640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03014b8f0 con 0x7fc0301517e0 2023-10-16T20:39:52.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc037046640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc030146230 con 0x7fc0300c4640 2023-10-16T20:39:52.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc034de2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0300c4260 0x7fc03014c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:52.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc034de2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0300c4260 0x7fc03014c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47710/0 (socket says 172.21.15.88:47710) 2023-10-16T20:39:52.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc034de2640 1 -- 172.21.15.88:0/2803964816 learned_addr learned my addr 172.21.15.88:0/2803964816 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:52.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc034de2640 1 -- 172.21.15.88:0/2803964816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0300c4640 msgr2=0x7fc030145940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc034de2640 1 --2- 172.21.15.88:0/2803964816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0300c4640 0x7fc030145940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc034de2640 1 -- 172.21.15.88:0/2803964816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc0301517e0 msgr2=0x7fc030142dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:52.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc034de2640 1 --2- 172.21.15.88:0/2803964816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc0301517e0 0x7fc030142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:52.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc034de2640 1 -- 172.21.15.88:0/2803964816 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc028009a70 con 0x7fc0300c4260 2023-10-16T20:39:52.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc034de2640 1 --2- 172.21.15.88:0/2803964816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0300c4260 0x7fc03014c030 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc02400c490 tx=0x7fc02400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:52.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc02d7fa640 1 -- 172.21.15.88:0/2803964816 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0240052d0 con 0x7fc0300c4260 2023-10-16T20:39:52.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc02d7fa640 1 -- 172.21.15.88:0/2803964816 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc024005430 con 0x7fc0300c4260 2023-10-16T20:39:52.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc037046640 1 -- 172.21.15.88:0/2803964816 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc030141a40 con 0x7fc0300c4260 2023-10-16T20:39:52.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc02d7fa640 1 -- 172.21.15.88:0/2803964816 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc024012870 con 0x7fc0300c4260 2023-10-16T20:39:52.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc037046640 1 -- 172.21.15.88:0/2803964816 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc030142000 con 0x7fc0300c4260 2023-10-16T20:39:52.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.956+0000 7fc037046640 1 -- 172.21.15.88:0/2803964816 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc030146d50 con 0x7fc0300c4260 2023-10-16T20:39:52.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.960+0000 7fc02d7fa640 1 -- 172.21.15.88:0/2803964816 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc0240129d0 con 0x7fc0300c4260 2023-10-16T20:39:52.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.960+0000 7fc02d7fa640 1 --2- 172.21.15.88:0/2803964816 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0140664a0 0x7fc014068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:52.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.960+0000 7fc02d7fa640 1 -- 172.21.15.88:0/2803964816 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc024083e00 con 0x7fc0300c4260 2023-10-16T20:39:52.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.960+0000 7fc02ffff640 1 --2- 172.21.15.88:0/2803964816 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0140664a0 0x7fc014068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:52.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.960+0000 7fc02ffff640 1 --2- 172.21.15.88:0/2803964816 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0140664a0 0x7fc014068960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc01c001350 tx=0x7fc01c001280 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:52.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:52.964+0000 7fc02d7fa640 1 -- 172.21.15.88:0/2803964816 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc024050ec0 con 0x7fc0300c4260 2023-10-16T20:39:53.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.116+0000 7fc037046640 1 -- 172.21.15.88:0/2803964816 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc0300c6e80 con 0x7fc0300c4260 2023-10-16T20:39:53.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.116+0000 7fc02d7fa640 1 -- 172.21.15.88:0/2803964816 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc0240544e0 con 0x7fc0300c4260 2023-10-16T20:39:53.120 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:39:53.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.116+0000 7fc037046640 1 -- 172.21.15.88:0/2803964816 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0140664a0 msgr2=0x7fc014068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:53.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.116+0000 7fc037046640 1 --2- 172.21.15.88:0/2803964816 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0140664a0 0x7fc014068960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc01c001350 tx=0x7fc01c001280 comp rx=0 tx=0).stop 2023-10-16T20:39:53.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.116+0000 7fc037046640 1 -- 172.21.15.88:0/2803964816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0300c4260 msgr2=0x7fc03014c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:53.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.116+0000 7fc037046640 1 --2- 172.21.15.88:0/2803964816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0300c4260 0x7fc03014c030 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc02400c490 tx=0x7fc02400c950 comp rx=0 tx=0).stop 2023-10-16T20:39:53.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.116+0000 7fc037046640 1 -- 172.21.15.88:0/2803964816 shutdown_connections 2023-10-16T20:39:53.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.116+0000 7fc037046640 1 --2- 172.21.15.88:0/2803964816 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0140664a0 0x7fc014068960 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.116+0000 7fc037046640 1 --2- 172.21.15.88:0/2803964816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc0301517e0 0x7fc030142dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.116+0000 7fc037046640 1 --2- 172.21.15.88:0/2803964816 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc0300c4640 0x7fc030145940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.116+0000 7fc037046640 1 --2- 172.21.15.88:0/2803964816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0300c4260 0x7fc03014c030 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.116+0000 7fc037046640 1 -- 172.21.15.88:0/2803964816 >> 172.21.15.88:0/2803964816 conn(0x7fc0300ca9f0 msgr2=0x7fc0300c5e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:53.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.116+0000 7fc037046640 1 -- 172.21.15.88:0/2803964816 shutdown_connections 2023-10-16T20:39:53.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.116+0000 7fc037046640 1 -- 172.21.15.88:0/2803964816 wait complete. 2023-10-16T20:39:53.141 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2023-10-16T20:39:53.141 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:39:53.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.272+0000 7f5662217640 1 Processor -- start 2023-10-16T20:39:53.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.276+0000 7f5662217640 1 -- start start 2023-10-16T20:39:53.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.276+0000 7f5662217640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f565c1537c0 0x7f565c15c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:53.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.276+0000 7f5662217640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565c0cae10 con 0x7f565c0ca2f0 2023-10-16T20:39:53.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.276+0000 7f5662217640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565c0cac30 con 0x7f565c15cdb0 2023-10-16T20:39:53.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.276+0000 7f5662217640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f565c15d2b0 con 0x7f565c1537c0 2023-10-16T20:39:53.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.276+0000 7f5661215640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f565c1537c0 0x7f565c15c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:53.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.276+0000 7f5661a16640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f565c0ca2f0 0x7f565c0cead0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58414/0 (socket says 172.21.15.88:58414) 2023-10-16T20:39:53.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.276+0000 7f5661215640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f565c1537c0 0x7f565c15c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47712/0 (socket says 172.21.15.88:47712) 2023-10-16T20:39:53.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.276+0000 7f5661a16640 1 -- 172.21.15.88:0/1185462738 learned_addr learned my addr 172.21.15.88:0/1185462738 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:53.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.276+0000 7f5661215640 1 -- 172.21.15.88:0/1185462738 >> v1:172.21.15.191:6789/0 conn(0x7f565c15cdb0 legacy=0x7f565c0c4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:53.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.276+0000 7f5661215640 1 -- 172.21.15.88:0/1185462738 >> v1:172.21.15.88:6789/0 conn(0x7f565c0ca2f0 legacy=0x7f565c0cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:53.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.276+0000 7f5661215640 1 -- 172.21.15.88:0/1185462738 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565c15d590 con 0x7f565c1537c0 2023-10-16T20:39:53.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.276+0000 7f5653fff640 1 -- 172.21.15.88:0/1185462738 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485558491 0 0) 0x7f565c0cae10 con 0x7f565c0ca2f0 2023-10-16T20:39:53.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.280+0000 7f5661215640 1 --2- 172.21.15.88:0/1185462738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f565c1537c0 0x7f565c15c870 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f56580097c0 tx=0x7f565801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b0e8089bdf8234c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:53.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.280+0000 7f5653fff640 1 -- 172.21.15.88:0/1185462738 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56580a8970 con 0x7f565c1537c0 2023-10-16T20:39:53.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.280+0000 7f5653fff640 1 -- 172.21.15.88:0/1185462738 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56580a8ad0 con 0x7f565c1537c0 2023-10-16T20:39:53.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.280+0000 7f5653fff640 1 -- 172.21.15.88:0/1185462738 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56580a8e60 con 0x7f565c1537c0 2023-10-16T20:39:53.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.280+0000 7f5662217640 1 -- 172.21.15.88:0/1185462738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f565c1537c0 msgr2=0x7f565c15c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:53.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.280+0000 7f5662217640 1 --2- 172.21.15.88:0/1185462738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f565c1537c0 0x7f565c15c870 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f56580097c0 tx=0x7f565801ec70 comp rx=0 tx=0).stop 2023-10-16T20:39:53.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.280+0000 7f5662217640 1 -- 172.21.15.88:0/1185462738 shutdown_connections 2023-10-16T20:39:53.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.280+0000 7f5662217640 1 --2- 172.21.15.88:0/1185462738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f565c1537c0 0x7f565c15c870 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.280+0000 7f5662217640 1 -- 172.21.15.88:0/1185462738 >> 172.21.15.88:0/1185462738 conn(0x7f565c06c170 msgr2=0x7f565c0715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:53.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.280+0000 7f5662217640 1 -- 172.21.15.88:0/1185462738 shutdown_connections 2023-10-16T20:39:53.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.280+0000 7f5662217640 1 -- 172.21.15.88:0/1185462738 wait complete. 2023-10-16T20:39:53.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.280+0000 7f5662217640 1 Processor -- start 2023-10-16T20:39:53.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.280+0000 7f5662217640 1 -- start start 2023-10-16T20:39:53.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5662217640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f565c0ca2f0 0x7f565c0cece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:53.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5662217640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f565c1537c0 0x7f565c149b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:53.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5662217640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f565c15cdb0 0x7f565c1483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:53.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5662217640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c156110 con 0x7f565c1537c0 2023-10-16T20:39:53.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5662217640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0c8fa0 con 0x7f565c15cdb0 2023-10-16T20:39:53.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5662217640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565c155770 con 0x7f565c0ca2f0 2023-10-16T20:39:53.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5660a14640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f565c1537c0 0x7f565c149b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:53.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5660a14640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f565c1537c0 0x7f565c149b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47718/0 (socket says 172.21.15.88:47718) 2023-10-16T20:39:53.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5660a14640 1 -- 172.21.15.88:0/684337870 learned_addr learned my addr 172.21.15.88:0/684337870 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:53.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5661a16640 1 --2- 172.21.15.88:0/684337870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f565c15cdb0 0x7f565c1483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:53.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5661215640 1 --2- 172.21.15.88:0/684337870 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f565c0ca2f0 0x7f565c0cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:53.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5660a14640 1 -- 172.21.15.88:0/684337870 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f565c0ca2f0 msgr2=0x7f565c0cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:53.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5660a14640 1 --2- 172.21.15.88:0/684337870 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f565c0ca2f0 0x7f565c0cece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5660a14640 1 -- 172.21.15.88:0/684337870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f565c15cdb0 msgr2=0x7f565c1483f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:53.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5660a14640 1 --2- 172.21.15.88:0/684337870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f565c15cdb0 0x7f565c1483f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5660a14640 1 -- 172.21.15.88:0/684337870 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5658009420 con 0x7f565c1537c0 2023-10-16T20:39:53.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5661a16640 1 --2- 172.21.15.88:0/684337870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f565c15cdb0 0x7f565c1483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:53.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5660a14640 1 --2- 172.21.15.88:0/684337870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f565c1537c0 0x7f565c149b70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f564c00c390 tx=0x7f564c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:53.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5651ffb640 1 -- 172.21.15.88:0/684337870 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c0052d0 con 0x7f565c1537c0 2023-10-16T20:39:53.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5651ffb640 1 -- 172.21.15.88:0/684337870 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564c005430 con 0x7f565c1537c0 2023-10-16T20:39:53.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5651ffb640 1 -- 172.21.15.88:0/684337870 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c003820 con 0x7f565c1537c0 2023-10-16T20:39:53.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5662217640 1 -- 172.21.15.88:0/684337870 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565c155a50 con 0x7f565c1537c0 2023-10-16T20:39:53.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5662217640 1 -- 172.21.15.88:0/684337870 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565c152f70 con 0x7f565c1537c0 2023-10-16T20:39:53.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.284+0000 7f5662217640 1 -- 172.21.15.88:0/684337870 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565c156250 con 0x7f565c1537c0 2023-10-16T20:39:53.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.288+0000 7f5651ffb640 1 -- 172.21.15.88:0/684337870 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f564c003980 con 0x7f565c1537c0 2023-10-16T20:39:53.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.288+0000 7f5651ffb640 1 --2- 172.21.15.88:0/684337870 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56400664a0 0x7f5640068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:53.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.288+0000 7f5651ffb640 1 -- 172.21.15.88:0/684337870 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f564c083810 con 0x7f565c1537c0 2023-10-16T20:39:53.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.288+0000 7f5661215640 1 --2- 172.21.15.88:0/684337870 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56400664a0 0x7f5640068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:53.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.288+0000 7f5661215640 1 --2- 172.21.15.88:0/684337870 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56400664a0 0x7f5640068960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f56580b2000 tx=0x7f56580aa070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:53.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.292+0000 7f5651ffb640 1 -- 172.21.15.88:0/684337870 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f565c156250 con 0x7f565c1537c0 2023-10-16T20:39:53.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.472+0000 7f5662217640 1 -- 172.21.15.88:0/684337870 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f565c156250 con 0x7f565c1537c0 2023-10-16T20:39:53.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.472+0000 7f5651ffb640 1 -- 172.21.15.88:0/684337870 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6559 (secure 0 0 0) 0x7f565c156250 con 0x7f565c1537c0 2023-10-16T20:39:53.478 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:53.479 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:39:39.882115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":44,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:39:53.479 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 19 2023-10-16T20:39:53.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.476+0000 7f5662217640 1 -- 172.21.15.88:0/684337870 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56400664a0 msgr2=0x7f5640068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:53.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.476+0000 7f5662217640 1 --2- 172.21.15.88:0/684337870 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56400664a0 0x7f5640068960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f56580b2000 tx=0x7f56580aa070 comp rx=0 tx=0).stop 2023-10-16T20:39:53.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.476+0000 7f5662217640 1 -- 172.21.15.88:0/684337870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f565c1537c0 msgr2=0x7f565c149b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:53.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.476+0000 7f5662217640 1 --2- 172.21.15.88:0/684337870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f565c1537c0 0x7f565c149b70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f564c00c390 tx=0x7f564c00c850 comp rx=0 tx=0).stop 2023-10-16T20:39:53.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.476+0000 7f5662217640 1 -- 172.21.15.88:0/684337870 shutdown_connections 2023-10-16T20:39:53.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.476+0000 7f5662217640 1 --2- 172.21.15.88:0/684337870 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56400664a0 0x7f5640068960 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.476+0000 7f5662217640 1 --2- 172.21.15.88:0/684337870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f565c15cdb0 0x7f565c1483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.476+0000 7f5662217640 1 --2- 172.21.15.88:0/684337870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f565c1537c0 0x7f565c149b70 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.476+0000 7f5662217640 1 --2- 172.21.15.88:0/684337870 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f565c0ca2f0 0x7f565c0cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.476+0000 7f5662217640 1 -- 172.21.15.88:0/684337870 >> 172.21.15.88:0/684337870 conn(0x7f565c06c170 msgr2=0x7f565c0c4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:53.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.476+0000 7f5662217640 1 -- 172.21.15.88:0/684337870 shutdown_connections 2023-10-16T20:39:53.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.476+0000 7f5662217640 1 -- 172.21.15.88:0/684337870 wait complete. 2023-10-16T20:39:53.499 INFO:tasks.mds_thrash:Ready to start thrashing 2023-10-16T20:39:53.500 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2023-10-16T20:39:53.500 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T20:39:53.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b35646640 1 Processor -- start 2023-10-16T20:39:53.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b35646640 1 -- start start 2023-10-16T20:39:53.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b35646640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b300cece0 0x7f6b300cf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:53.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b35646640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b30153b50 con 0x7f6b30153770 2023-10-16T20:39:53.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b35646640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b300cb2f0 con 0x7f6b300c9bb0 2023-10-16T20:39:53.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b35646640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b300cc250 con 0x7f6b300cece0 2023-10-16T20:39:53.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2effd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6b30153770 0x7f6b30159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58430/0 (socket says 172.21.15.88:58430) 2023-10-16T20:39:53.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2effd640 1 -- 172.21.15.88:0/2479895916 learned_addr learned my addr 172.21.15.88:0/2479895916 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:53.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2f7fe640 1 --2- 172.21.15.88:0/2479895916 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b300cece0 0x7f6b300cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:53.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2e7fc640 1 -- 172.21.15.88:0/2479895916 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284658818 0 0) 0x7f6b30153b50 con 0x7f6b30153770 2023-10-16T20:39:53.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2e7fc640 1 -- 172.21.15.88:0/2479895916 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b1c003530 con 0x7f6b30153770 2023-10-16T20:39:53.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2e7fc640 1 -- 172.21.15.88:0/2479895916 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611643704 0 0) 0x7f6b300cb2f0 con 0x7f6b300c9bb0 2023-10-16T20:39:53.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2e7fc640 1 -- 172.21.15.88:0/2479895916 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b30153b50 con 0x7f6b300c9bb0 2023-10-16T20:39:53.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2e7fc640 1 -- 172.21.15.88:0/2479895916 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3058566153 0 0) 0x7f6b1c003530 con 0x7f6b30153770 2023-10-16T20:39:53.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2e7fc640 1 -- 172.21.15.88:0/2479895916 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b300cece0 msgr2=0x7f6b300cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:53.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2e7fc640 1 --2- 172.21.15.88:0/2479895916 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b300cece0 0x7f6b300cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2e7fc640 1 -- 172.21.15.88:0/2479895916 >> v1:172.21.15.191:6789/0 conn(0x7f6b300c9bb0 legacy=0x7f6b300c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:53.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2e7fc640 1 -- 172.21.15.88:0/2479895916 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b300cb5d0 con 0x7f6b30153770 2023-10-16T20:39:53.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2e7fc640 1 -- 172.21.15.88:0/2479895916 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f6b20001800 con 0x7f6b30153770 2023-10-16T20:39:53.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2e7fc640 1 -- 172.21.15.88:0/2479895916 >> v1:172.21.15.88:6789/0 conn(0x7f6b30153770 legacy=0x7f6b30159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:53.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2e7fc640 1 --2- 172.21.15.88:0/2479895916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b1c003b40 0x7f6b1c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:53.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2e7fc640 1 --2- 172.21.15.88:0/2479895916 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b300cece0 0x7f6b1c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:53.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2e7fc640 1 --2- 172.21.15.88:0/2479895916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b1c004d10 0x7f6b1c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:53.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2e7fc640 1 -- 172.21.15.88:0/2479895916 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b300cc250 con 0x7f6b1c003b40 2023-10-16T20:39:53.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2e7fc640 1 -- 172.21.15.88:0/2479895916 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c00ba40 con 0x7f6b1c004d10 2023-10-16T20:39:53.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2e7fc640 1 -- 172.21.15.88:0/2479895916 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c00bbb0 con 0x7f6b300cece0 2023-10-16T20:39:53.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2effd640 1 --2- 172.21.15.88:0/2479895916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b1c003b40 0x7f6b1c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:53.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2f7fe640 1 --2- 172.21.15.88:0/2479895916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b1c004d10 0x7f6b1c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:53.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b27fff640 1 --2- 172.21.15.88:0/2479895916 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b300cece0 0x7f6b1c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:53.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2effd640 1 -- 172.21.15.88:0/2479895916 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b300cece0 msgr2=0x7f6b1c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:53.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2effd640 1 --2- 172.21.15.88:0/2479895916 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b300cece0 0x7f6b1c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2effd640 1 -- 172.21.15.88:0/2479895916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b1c004d10 msgr2=0x7f6b1c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:53.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2effd640 1 --2- 172.21.15.88:0/2479895916 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b1c004d10 0x7f6b1c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b2effd640 1 -- 172.21.15.88:0/2479895916 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c003530 con 0x7f6b1c003b40 2023-10-16T20:39:53.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.636+0000 7f6b27fff640 1 --2- 172.21.15.88:0/2479895916 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b300cece0 0x7f6b1c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:53.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.640+0000 7f6b2effd640 1 --2- 172.21.15.88:0/2479895916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b1c003b40 0x7f6b1c003f40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6b2000c190 tx=0x7f6b2001cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ea4af1e214e2c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:53.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.640+0000 7f6b2e7fc640 1 -- 172.21.15.88:0/2479895916 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b200070d0 con 0x7f6b1c003b40 2023-10-16T20:39:53.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.640+0000 7f6b2e7fc640 1 -- 172.21.15.88:0/2479895916 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b20007230 con 0x7f6b1c003b40 2023-10-16T20:39:53.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.640+0000 7f6b2e7fc640 1 -- 172.21.15.88:0/2479895916 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b20007600 con 0x7f6b1c003b40 2023-10-16T20:39:53.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.640+0000 7f6b35646640 1 -- 172.21.15.88:0/2479895916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b1c003b40 msgr2=0x7f6b1c003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:53.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.640+0000 7f6b35646640 1 --2- 172.21.15.88:0/2479895916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b1c003b40 0x7f6b1c003f40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6b2000c190 tx=0x7f6b2001cbc0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.640+0000 7f6b2f7fe640 1 -- 172.21.15.88:0/2479895916 reap_dead start 2023-10-16T20:39:53.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.640+0000 7f6b35646640 1 -- 172.21.15.88:0/2479895916 shutdown_connections 2023-10-16T20:39:53.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.640+0000 7f6b35646640 1 -- 172.21.15.88:0/2479895916 >> 172.21.15.88:0/2479895916 conn(0x7f6b300715c0 msgr2=0x7f6b300719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:53.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.640+0000 7f6b35646640 1 -- 172.21.15.88:0/2479895916 shutdown_connections 2023-10-16T20:39:53.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.640+0000 7f6b35646640 1 -- 172.21.15.88:0/2479895916 wait complete. 2023-10-16T20:39:53.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b35646640 1 Processor -- start 2023-10-16T20:39:53.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b35646640 1 -- start start 2023-10-16T20:39:53.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b35646640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b1c003b40 0x7f6b3015d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:53.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b35646640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b300cddd0 0x7f6b300ce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:53.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b35646640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b30154820 0x7f6b30154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:53.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b35646640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3015cd20 con 0x7f6b300cddd0 2023-10-16T20:39:53.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b35646640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b30154080 con 0x7f6b30154820 2023-10-16T20:39:53.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b35646640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3015a720 con 0x7f6b1c003b40 2023-10-16T20:39:53.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b27fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b300cddd0 0x7f6b300ce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:53.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b27fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b300cddd0 0x7f6b300ce1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47748/0 (socket says 172.21.15.88:47748) 2023-10-16T20:39:53.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b27fff640 1 -- 172.21.15.88:0/3333030333 learned_addr learned my addr 172.21.15.88:0/3333030333 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:53.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b2f7fe640 1 --2- 172.21.15.88:0/3333030333 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b30154820 0x7f6b30154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:53.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b2effd640 1 --2- 172.21.15.88:0/3333030333 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b1c003b40 0x7f6b3015d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:53.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b27fff640 1 -- 172.21.15.88:0/3333030333 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b1c003b40 msgr2=0x7f6b3015d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:53.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b27fff640 1 --2- 172.21.15.88:0/3333030333 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b1c003b40 0x7f6b3015d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b27fff640 1 -- 172.21.15.88:0/3333030333 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b30154820 msgr2=0x7f6b30154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:53.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b27fff640 1 --2- 172.21.15.88:0/3333030333 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b30154820 0x7f6b30154c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b27fff640 1 -- 172.21.15.88:0/3333030333 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b30153de0 con 0x7f6b300cddd0 2023-10-16T20:39:53.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b2effd640 1 --2- 172.21.15.88:0/3333030333 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b1c003b40 0x7f6b3015d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:53.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b27fff640 1 --2- 172.21.15.88:0/3333030333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b300cddd0 0x7f6b300ce1f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6b1800e7e0 tx=0x7f6b1800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:53.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b277fe640 1 -- 172.21.15.88:0/3333030333 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18017070 con 0x7f6b300cddd0 2023-10-16T20:39:53.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b277fe640 1 -- 172.21.15.88:0/3333030333 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b180050d0 con 0x7f6b300cddd0 2023-10-16T20:39:53.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b277fe640 1 -- 172.21.15.88:0/3333030333 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18005240 con 0x7f6b300cddd0 2023-10-16T20:39:53.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b35646640 1 -- 172.21.15.88:0/3333030333 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3015a9a0 con 0x7f6b300cddd0 2023-10-16T20:39:53.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.644+0000 7f6b35646640 1 -- 172.21.15.88:0/3333030333 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3015af60 con 0x7f6b300cddd0 2023-10-16T20:39:53.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.648+0000 7f6b35646640 1 -- 172.21.15.88:0/3333030333 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3015b120 con 0x7f6b300cddd0 2023-10-16T20:39:53.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.648+0000 7f6b277fe640 1 -- 172.21.15.88:0/3333030333 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6b18007020 con 0x7f6b300cddd0 2023-10-16T20:39:53.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.648+0000 7f6b277fe640 1 --2- 172.21.15.88:0/3333030333 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6b100664a0 0x7f6b10068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:53.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.648+0000 7f6b277fe640 1 -- 172.21.15.88:0/3333030333 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f6b18083a20 con 0x7f6b300cddd0 2023-10-16T20:39:53.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.648+0000 7f6b2effd640 1 --2- 172.21.15.88:0/3333030333 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6b100664a0 0x7f6b10068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:53.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.648+0000 7f6b2effd640 1 --2- 172.21.15.88:0/3333030333 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6b100664a0 0x7f6b10068960 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6b200aa000 tx=0x7f6b200b0040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:53.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.652+0000 7f6b277fe640 1 -- 172.21.15.88:0/3333030333 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6b18050ad0 con 0x7f6b300cddd0 2023-10-16T20:39:53.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.804+0000 7f6b35646640 1 -- 172.21.15.88:0/3333030333 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6b300cf500 con 0x7f6b100664a0 2023-10-16T20:39:53.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.812+0000 7f6b277fe640 1 -- 172.21.15.88:0/3333030333 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+369924 (secure 0 0 0) 0x7f6b20001600 con 0x7f6b100664a0 2023-10-16T20:39:53.821 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:53.824 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-16T20:39:53.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.816+0000 7f6b35646640 1 -- 172.21.15.88:0/3333030333 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6b100664a0 msgr2=0x7f6b10068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:53.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.816+0000 7f6b35646640 1 --2- 172.21.15.88:0/3333030333 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6b100664a0 0x7f6b10068960 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6b200aa000 tx=0x7f6b200b0040 comp rx=0 tx=0).stop 2023-10-16T20:39:53.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.816+0000 7f6b35646640 1 -- 172.21.15.88:0/3333030333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b300cddd0 msgr2=0x7f6b300ce1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:53.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.816+0000 7f6b35646640 1 --2- 172.21.15.88:0/3333030333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b300cddd0 0x7f6b300ce1f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6b1800e7e0 tx=0x7f6b1800eca0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.816+0000 7f6b35646640 1 -- 172.21.15.88:0/3333030333 shutdown_connections 2023-10-16T20:39:53.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.816+0000 7f6b35646640 1 --2- 172.21.15.88:0/3333030333 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6b100664a0 0x7f6b10068960 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.816+0000 7f6b35646640 1 --2- 172.21.15.88:0/3333030333 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6b30154820 0x7f6b30154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.816+0000 7f6b35646640 1 --2- 172.21.15.88:0/3333030333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b300cddd0 0x7f6b300ce1f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.816+0000 7f6b35646640 1 --2- 172.21.15.88:0/3333030333 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6b1c003b40 0x7f6b3015d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:53.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.816+0000 7f6b35646640 1 -- 172.21.15.88:0/3333030333 >> 172.21.15.88:0/3333030333 conn(0x7f6b300715c0 msgr2=0x7f6b300c46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:53.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.816+0000 7f6b35646640 1 -- 172.21.15.88:0/3333030333 shutdown_connections 2023-10-16T20:39:53.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:53.816+0000 7f6b35646640 1 -- 172.21.15.88:0/3333030333 wait complete. 2023-10-16T20:39:53.841 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2023-10-16T20:39:53.635556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":602391,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":375,"num_write_kb":2490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307,"ondisk_log_size":307,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219944,"kb_used_data":3816,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754776,"statfs":{"total":773094113280,"available":772868890624,"internally_reserved":0,"allocated":3907584,"data_stored":2198742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3076,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002564"},"pg_stats":[{"pgid":"2.1f","version":"29'13","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.526948+0000","last_change":"2023-10-16T20:36:54.763744+0000","last_active":"2023-10-16T20:39:39.526948+0000","last_peered":"2023-10-16T20:39:39.526948+0000","last_clean":"2023-10-16T20:39:39.526948+0000","last_became_active":"2023-10-16T20:36:54.763457+0000","last_became_peered":"2023-10-16T20:36:54.763457+0000","last_unstale":"2023-10-16T20:39:39.526948+0000","last_undegraded":"2023-10-16T20:39:39.526948+0000","last_fullsized":"2023-10-16T20:39:39.526948+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:05.774485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465462+0000","last_change":"2023-10-16T20:36:55.302740+0000","last_active":"2023-10-16T20:37:37.465462+0000","last_peered":"2023-10-16T20:37:37.465462+0000","last_clean":"2023-10-16T20:37:37.465462+0000","last_became_active":"2023-10-16T20:36:55.302519+0000","last_became_peered":"2023-10-16T20:36:55.302519+0000","last_unstale":"2023-10-16T20:37:37.465462+0000","last_undegraded":"2023-10-16T20:37:37.465462+0000","last_fullsized":"2023-10-16T20:37:37.465462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:36.762521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464759+0000","last_change":"2023-10-16T20:36:59.466679+0000","last_active":"2023-10-16T20:37:37.464759+0000","last_peered":"2023-10-16T20:37:37.464759+0000","last_clean":"2023-10-16T20:37:37.464759+0000","last_became_active":"2023-10-16T20:36:59.466306+0000","last_became_peered":"2023-10-16T20:36:59.466306+0000","last_unstale":"2023-10-16T20:37:37.464759+0000","last_undegraded":"2023-10-16T20:37:37.464759+0000","last_fullsized":"2023-10-16T20:37:37.464759+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:14.874622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:38:57.624590+0000","last_change":"2023-10-16T20:36:54.759491+0000","last_active":"2023-10-16T20:38:57.624590+0000","last_peered":"2023-10-16T20:38:57.624590+0000","last_clean":"2023-10-16T20:38:57.624590+0000","last_became_active":"2023-10-16T20:36:54.759177+0000","last_became_peered":"2023-10-16T20:36:54.759177+0000","last_unstale":"2023-10-16T20:38:57.624590+0000","last_undegraded":"2023-10-16T20:38:57.624590+0000","last_fullsized":"2023-10-16T20:38:57.624590+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:43:36.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435464+0000","last_change":"2023-10-16T20:36:55.291745+0000","last_active":"2023-10-16T20:37:37.435464+0000","last_peered":"2023-10-16T20:37:37.435464+0000","last_clean":"2023-10-16T20:37:37.435464+0000","last_became_active":"2023-10-16T20:36:55.291458+0000","last_became_peered":"2023-10-16T20:36:55.291458+0000","last_unstale":"2023-10-16T20:37:37.435464+0000","last_undegraded":"2023-10-16T20:37:37.435464+0000","last_fullsized":"2023-10-16T20:37:37.435464+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:01.291335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402322+0000","last_change":"2023-10-16T20:36:59.303600+0000","last_active":"2023-10-16T20:37:37.402322+0000","last_peered":"2023-10-16T20:37:37.402322+0000","last_clean":"2023-10-16T20:37:37.402322+0000","last_became_active":"2023-10-16T20:36:59.302968+0000","last_became_peered":"2023-10-16T20:36:59.302968+0000","last_unstale":"2023-10-16T20:37:37.402322+0000","last_undegraded":"2023-10-16T20:37:37.402322+0000","last_fullsized":"2023-10-16T20:37:37.402322+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:11:28.322824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"29'12","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.529183+0000","last_change":"2023-10-16T20:36:54.762600+0000","last_active":"2023-10-16T20:39:39.529183+0000","last_peered":"2023-10-16T20:39:39.529183+0000","last_clean":"2023-10-16T20:39:39.529183+0000","last_became_active":"2023-10-16T20:36:54.762278+0000","last_became_peered":"2023-10-16T20:36:54.762278+0000","last_unstale":"2023-10-16T20:39:39.529183+0000","last_undegraded":"2023-10-16T20:39:39.529183+0000","last_fullsized":"2023-10-16T20:39:39.529183+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:46:03.836415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404637+0000","last_change":"2023-10-16T20:37:37.403448+0000","last_active":"2023-10-16T20:37:37.404637+0000","last_peered":"2023-10-16T20:37:37.404637+0000","last_clean":"2023-10-16T20:37:37.404637+0000","last_became_active":"2023-10-16T20:37:37.402987+0000","last_became_peered":"2023-10-16T20:37:37.402987+0000","last_unstale":"2023-10-16T20:37:37.404637+0000","last_undegraded":"2023-10-16T20:37:37.404637+0000","last_fullsized":"2023-10-16T20:37:37.404637+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:39:05.729682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402868+0000","last_change":"2023-10-16T20:36:59.304684+0000","last_active":"2023-10-16T20:37:37.402868+0000","last_peered":"2023-10-16T20:37:37.402868+0000","last_clean":"2023-10-16T20:37:37.402868+0000","last_became_active":"2023-10-16T20:36:59.303805+0000","last_became_peered":"2023-10-16T20:36:59.303805+0000","last_unstale":"2023-10-16T20:37:37.402868+0000","last_undegraded":"2023-10-16T20:37:37.402868+0000","last_fullsized":"2023-10-16T20:37:37.402868+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:39:54.406763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.526474+0000","last_change":"2023-10-16T20:36:54.749849+0000","last_active":"2023-10-16T20:39:39.526474+0000","last_peered":"2023-10-16T20:39:39.526474+0000","last_clean":"2023-10-16T20:39:39.526474+0000","last_became_active":"2023-10-16T20:36:54.749316+0000","last_became_peered":"2023-10-16T20:36:54.749316+0000","last_unstale":"2023-10-16T20:39:39.526474+0000","last_undegraded":"2023-10-16T20:39:39.526474+0000","last_fullsized":"2023-10-16T20:39:39.526474+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:49:29.077126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409527+0000","last_change":"2023-10-16T20:36:55.302838+0000","last_active":"2023-10-16T20:37:37.409527+0000","last_peered":"2023-10-16T20:37:37.409527+0000","last_clean":"2023-10-16T20:37:37.409527+0000","last_became_active":"2023-10-16T20:36:55.302387+0000","last_became_peered":"2023-10-16T20:36:55.302387+0000","last_unstale":"2023-10-16T20:37:37.409527+0000","last_undegraded":"2023-10-16T20:37:37.409527+0000","last_fullsized":"2023-10-16T20:37:37.409527+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:55:56.517950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403226+0000","last_change":"2023-10-16T20:36:59.305750+0000","last_active":"2023-10-16T20:37:37.403226+0000","last_peered":"2023-10-16T20:37:37.403226+0000","last_clean":"2023-10-16T20:37:37.403226+0000","last_became_active":"2023-10-16T20:36:59.305228+0000","last_became_peered":"2023-10-16T20:36:59.305228+0000","last_unstale":"2023-10-16T20:37:37.403226+0000","last_undegraded":"2023-10-16T20:37:37.403226+0000","last_fullsized":"2023-10-16T20:37:37.403226+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:00:18.194057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407634+0000","last_change":"2023-10-16T20:36:55.306712+0000","last_active":"2023-10-16T20:37:37.407634+0000","last_peered":"2023-10-16T20:37:37.407634+0000","last_clean":"2023-10-16T20:37:37.407634+0000","last_became_active":"2023-10-16T20:36:55.306427+0000","last_became_peered":"2023-10-16T20:36:55.306427+0000","last_unstale":"2023-10-16T20:37:37.407634+0000","last_undegraded":"2023-10-16T20:37:37.407634+0000","last_fullsized":"2023-10-16T20:37:37.407634+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:42:03.769890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.410476+0000","last_change":"2023-10-16T20:36:54.768375+0000","last_active":"2023-10-16T20:39:22.410476+0000","last_peered":"2023-10-16T20:39:22.410476+0000","last_clean":"2023-10-16T20:39:22.410476+0000","last_became_active":"2023-10-16T20:36:54.767996+0000","last_became_peered":"2023-10-16T20:36:54.767996+0000","last_unstale":"2023-10-16T20:39:22.410476+0000","last_undegraded":"2023-10-16T20:39:22.410476+0000","last_fullsized":"2023-10-16T20:39:22.410476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:55.303942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404253+0000","last_change":"2023-10-16T20:36:59.307130+0000","last_active":"2023-10-16T20:37:37.404253+0000","last_peered":"2023-10-16T20:37:37.404253+0000","last_clean":"2023-10-16T20:37:37.404253+0000","last_became_active":"2023-10-16T20:36:59.306640+0000","last_became_peered":"2023-10-16T20:36:59.306640+0000","last_unstale":"2023-10-16T20:37:37.404253+0000","last_undegraded":"2023-10-16T20:37:37.404253+0000","last_fullsized":"2023-10-16T20:37:37.404253+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:19:18.664630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465074+0000","last_change":"2023-10-16T20:36:55.275436+0000","last_active":"2023-10-16T20:37:37.465074+0000","last_peered":"2023-10-16T20:37:37.465074+0000","last_clean":"2023-10-16T20:37:37.465074+0000","last_became_active":"2023-10-16T20:36:55.275085+0000","last_became_peered":"2023-10-16T20:36:55.275085+0000","last_unstale":"2023-10-16T20:37:37.465074+0000","last_undegraded":"2023-10-16T20:37:37.465074+0000","last_fullsized":"2023-10-16T20:37:37.465074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:37:37.814511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.502271+0000","last_change":"2023-10-16T20:36:54.756174+0000","last_active":"2023-10-16T20:39:33.502271+0000","last_peered":"2023-10-16T20:39:33.502271+0000","last_clean":"2023-10-16T20:39:33.502271+0000","last_became_active":"2023-10-16T20:36:54.755872+0000","last_became_peered":"2023-10-16T20:36:54.755872+0000","last_unstale":"2023-10-16T20:39:33.502271+0000","last_undegraded":"2023-10-16T20:39:33.502271+0000","last_fullsized":"2023-10-16T20:39:33.502271+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:32.741671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403750+0000","last_change":"2023-10-16T20:36:59.303855+0000","last_active":"2023-10-16T20:37:37.403750+0000","last_peered":"2023-10-16T20:37:37.403750+0000","last_clean":"2023-10-16T20:37:37.403750+0000","last_became_active":"2023-10-16T20:36:59.302919+0000","last_became_peered":"2023-10-16T20:36:59.302919+0000","last_unstale":"2023-10-16T20:37:37.403750+0000","last_undegraded":"2023-10-16T20:37:37.403750+0000","last_fullsized":"2023-10-16T20:37:37.403750+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:38:02.591237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464820+0000","last_change":"2023-10-16T20:36:55.270717+0000","last_active":"2023-10-16T20:37:37.464820+0000","last_peered":"2023-10-16T20:37:37.464820+0000","last_clean":"2023-10-16T20:37:37.464820+0000","last_became_active":"2023-10-16T20:36:55.270455+0000","last_became_peered":"2023-10-16T20:36:55.270455+0000","last_unstale":"2023-10-16T20:37:37.464820+0000","last_undegraded":"2023-10-16T20:37:37.464820+0000","last_fullsized":"2023-10-16T20:37:37.464820+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:16:44.485783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'7","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.524872+0000","last_change":"2023-10-16T20:36:54.744128+0000","last_active":"2023-10-16T20:39:39.524872+0000","last_peered":"2023-10-16T20:39:39.524872+0000","last_clean":"2023-10-16T20:39:39.524872+0000","last_became_active":"2023-10-16T20:36:54.743805+0000","last_became_peered":"2023-10-16T20:36:54.743805+0000","last_unstale":"2023-10-16T20:39:39.524872+0000","last_undegraded":"2023-10-16T20:39:39.524872+0000","last_fullsized":"2023-10-16T20:39:39.524872+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:54:35.819820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402668+0000","last_change":"2023-10-16T20:36:59.465650+0000","last_active":"2023-10-16T20:37:37.402668+0000","last_peered":"2023-10-16T20:37:37.402668+0000","last_clean":"2023-10-16T20:37:37.402668+0000","last_became_active":"2023-10-16T20:36:59.465172+0000","last_became_peered":"2023-10-16T20:36:59.465172+0000","last_unstale":"2023-10-16T20:37:37.402668+0000","last_undegraded":"2023-10-16T20:37:37.402668+0000","last_fullsized":"2023-10-16T20:37:37.402668+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:12.845517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403561+0000","last_change":"2023-10-16T20:36:55.278974+0000","last_active":"2023-10-16T20:37:37.403561+0000","last_peered":"2023-10-16T20:37:37.403561+0000","last_clean":"2023-10-16T20:37:37.403561+0000","last_became_active":"2023-10-16T20:36:55.278568+0000","last_became_peered":"2023-10-16T20:36:55.278568+0000","last_unstale":"2023-10-16T20:37:37.403561+0000","last_undegraded":"2023-10-16T20:37:37.403561+0000","last_fullsized":"2023-10-16T20:37:37.403561+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:26.756114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.407659+0000","last_change":"2023-10-16T20:36:54.758778+0000","last_active":"2023-10-16T20:39:22.407659+0000","last_peered":"2023-10-16T20:39:22.407659+0000","last_clean":"2023-10-16T20:39:22.407659+0000","last_became_active":"2023-10-16T20:36:54.758496+0000","last_became_peered":"2023-10-16T20:36:54.758496+0000","last_unstale":"2023-10-16T20:39:22.407659+0000","last_undegraded":"2023-10-16T20:39:22.407659+0000","last_fullsized":"2023-10-16T20:39:22.407659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:36:00.515106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404277+0000","last_change":"2023-10-16T20:36:59.309382+0000","last_active":"2023-10-16T20:37:37.404277+0000","last_peered":"2023-10-16T20:37:37.404277+0000","last_clean":"2023-10-16T20:37:37.404277+0000","last_became_active":"2023-10-16T20:36:59.309068+0000","last_became_peered":"2023-10-16T20:36:59.309068+0000","last_unstale":"2023-10-16T20:37:37.404277+0000","last_undegraded":"2023-10-16T20:37:37.404277+0000","last_fullsized":"2023-10-16T20:37:37.404277+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:46:02.601985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435429+0000","last_change":"2023-10-16T20:36:55.273089+0000","last_active":"2023-10-16T20:37:37.435429+0000","last_peered":"2023-10-16T20:37:37.435429+0000","last_clean":"2023-10-16T20:37:37.435429+0000","last_became_active":"2023-10-16T20:36:55.272840+0000","last_became_peered":"2023-10-16T20:36:55.272840+0000","last_unstale":"2023-10-16T20:37:37.435429+0000","last_undegraded":"2023-10-16T20:37:37.435429+0000","last_fullsized":"2023-10-16T20:37:37.435429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:37.330924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.530212+0000","last_change":"2023-10-16T20:36:54.760681+0000","last_active":"2023-10-16T20:39:39.530212+0000","last_peered":"2023-10-16T20:39:39.530212+0000","last_clean":"2023-10-16T20:39:39.530212+0000","last_became_active":"2023-10-16T20:36:54.760397+0000","last_became_peered":"2023-10-16T20:36:54.760397+0000","last_unstale":"2023-10-16T20:39:39.530212+0000","last_undegraded":"2023-10-16T20:39:39.530212+0000","last_fullsized":"2023-10-16T20:39:39.530212+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:47:16.510382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403318+0000","last_change":"2023-10-16T20:36:59.465626+0000","last_active":"2023-10-16T20:37:37.403318+0000","last_peered":"2023-10-16T20:37:37.403318+0000","last_clean":"2023-10-16T20:37:37.403318+0000","last_became_active":"2023-10-16T20:36:59.465152+0000","last_became_peered":"2023-10-16T20:36:59.465152+0000","last_unstale":"2023-10-16T20:37:37.403318+0000","last_undegraded":"2023-10-16T20:37:37.403318+0000","last_fullsized":"2023-10-16T20:37:37.403318+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:27:51.088333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465673+0000","last_change":"2023-10-16T20:36:55.285796+0000","last_active":"2023-10-16T20:37:37.465673+0000","last_peered":"2023-10-16T20:37:37.465673+0000","last_clean":"2023-10-16T20:37:37.465673+0000","last_became_active":"2023-10-16T20:36:55.285566+0000","last_became_peered":"2023-10-16T20:36:55.285566+0000","last_unstale":"2023-10-16T20:37:37.465673+0000","last_undegraded":"2023-10-16T20:37:37.465673+0000","last_fullsized":"2023-10-16T20:37:37.465673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:22:21.950705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.503684+0000","last_change":"2023-10-16T20:36:54.758062+0000","last_active":"2023-10-16T20:39:33.503684+0000","last_peered":"2023-10-16T20:39:33.503684+0000","last_clean":"2023-10-16T20:39:33.503684+0000","last_became_active":"2023-10-16T20:36:54.757796+0000","last_became_peered":"2023-10-16T20:36:54.757796+0000","last_unstale":"2023-10-16T20:39:33.503684+0000","last_undegraded":"2023-10-16T20:39:33.503684+0000","last_fullsized":"2023-10-16T20:39:33.503684+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:02:09.953795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464463+0000","last_change":"2023-10-16T20:36:59.468529+0000","last_active":"2023-10-16T20:37:37.464463+0000","last_peered":"2023-10-16T20:37:37.464463+0000","last_clean":"2023-10-16T20:37:37.464463+0000","last_became_active":"2023-10-16T20:36:59.465221+0000","last_became_peered":"2023-10-16T20:36:59.465221+0000","last_unstale":"2023-10-16T20:37:37.464463+0000","last_undegraded":"2023-10-16T20:37:37.464463+0000","last_fullsized":"2023-10-16T20:37:37.464463+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:58:08.457686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401257+0000","last_change":"2023-10-16T20:36:55.275759+0000","last_active":"2023-10-16T20:37:37.401257+0000","last_peered":"2023-10-16T20:37:37.401257+0000","last_clean":"2023-10-16T20:37:37.401257+0000","last_became_active":"2023-10-16T20:36:55.274432+0000","last_became_peered":"2023-10-16T20:36:55.274432+0000","last_unstale":"2023-10-16T20:37:37.401257+0000","last_undegraded":"2023-10-16T20:37:37.401257+0000","last_fullsized":"2023-10-16T20:37:37.401257+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:53:08.896731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.508458+0000","last_change":"2023-10-16T20:36:54.762356+0000","last_active":"2023-10-16T20:39:33.508458+0000","last_peered":"2023-10-16T20:39:33.508458+0000","last_clean":"2023-10-16T20:39:33.508458+0000","last_became_active":"2023-10-16T20:36:54.762167+0000","last_became_peered":"2023-10-16T20:36:54.762167+0000","last_unstale":"2023-10-16T20:39:33.508458+0000","last_undegraded":"2023-10-16T20:39:33.508458+0000","last_fullsized":"2023-10-16T20:39:33.508458+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:45.164625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434946+0000","last_change":"2023-10-16T20:36:59.467492+0000","last_active":"2023-10-16T20:37:37.434946+0000","last_peered":"2023-10-16T20:37:37.434946+0000","last_clean":"2023-10-16T20:37:37.434946+0000","last_became_active":"2023-10-16T20:36:59.466876+0000","last_became_peered":"2023-10-16T20:36:59.466876+0000","last_unstale":"2023-10-16T20:37:37.434946+0000","last_undegraded":"2023-10-16T20:37:37.434946+0000","last_fullsized":"2023-10-16T20:37:37.434946+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:29:01.292167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409570+0000","last_change":"2023-10-16T20:36:55.293537+0000","last_active":"2023-10-16T20:37:37.409570+0000","last_peered":"2023-10-16T20:37:37.409570+0000","last_clean":"2023-10-16T20:37:37.409570+0000","last_became_active":"2023-10-16T20:36:55.293228+0000","last_became_peered":"2023-10-16T20:36:55.293228+0000","last_unstale":"2023-10-16T20:37:37.409570+0000","last_undegraded":"2023-10-16T20:37:37.409570+0000","last_fullsized":"2023-10-16T20:37:37.409570+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:50:13.226645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:02.624662+0000","last_change":"2023-10-16T20:36:54.757588+0000","last_active":"2023-10-16T20:39:02.624662+0000","last_peered":"2023-10-16T20:39:02.624662+0000","last_clean":"2023-10-16T20:39:02.624662+0000","last_became_active":"2023-10-16T20:36:54.757308+0000","last_became_peered":"2023-10-16T20:36:54.757308+0000","last_unstale":"2023-10-16T20:39:02.624662+0000","last_undegraded":"2023-10-16T20:39:02.624662+0000","last_fullsized":"2023-10-16T20:39:02.624662+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:51.298239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402099+0000","last_change":"2023-10-16T20:36:59.303844+0000","last_active":"2023-10-16T20:37:37.402099+0000","last_peered":"2023-10-16T20:37:37.402099+0000","last_clean":"2023-10-16T20:37:37.402099+0000","last_became_active":"2023-10-16T20:36:59.303224+0000","last_became_peered":"2023-10-16T20:36:59.303224+0000","last_unstale":"2023-10-16T20:37:37.402099+0000","last_undegraded":"2023-10-16T20:37:37.402099+0000","last_fullsized":"2023-10-16T20:37:37.402099+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:52.447785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403798+0000","last_change":"2023-10-16T20:36:55.278310+0000","last_active":"2023-10-16T20:37:37.403798+0000","last_peered":"2023-10-16T20:37:37.403798+0000","last_clean":"2023-10-16T20:37:37.403798+0000","last_became_active":"2023-10-16T20:36:55.277872+0000","last_became_peered":"2023-10-16T20:36:55.277872+0000","last_unstale":"2023-10-16T20:37:37.403798+0000","last_undegraded":"2023-10-16T20:37:37.403798+0000","last_fullsized":"2023-10-16T20:37:37.403798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:37.320469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.527106+0000","last_change":"2023-10-16T20:36:54.740644+0000","last_active":"2023-10-16T20:39:39.527106+0000","last_peered":"2023-10-16T20:39:39.527106+0000","last_clean":"2023-10-16T20:39:39.527106+0000","last_became_active":"2023-10-16T20:36:54.740340+0000","last_became_peered":"2023-10-16T20:36:54.740340+0000","last_unstale":"2023-10-16T20:39:39.527106+0000","last_undegraded":"2023-10-16T20:39:39.527106+0000","last_fullsized":"2023-10-16T20:39:39.527106+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:22.221727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401155+0000","last_change":"2023-10-16T20:36:59.294914+0000","last_active":"2023-10-16T20:37:37.401155+0000","last_peered":"2023-10-16T20:37:37.401155+0000","last_clean":"2023-10-16T20:37:37.401155+0000","last_became_active":"2023-10-16T20:36:59.294548+0000","last_became_peered":"2023-10-16T20:36:59.294548+0000","last_unstale":"2023-10-16T20:37:37.401155+0000","last_undegraded":"2023-10-16T20:37:37.401155+0000","last_fullsized":"2023-10-16T20:37:37.401155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:52:15.265262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401383+0000","last_change":"2023-10-16T20:36:55.270992+0000","last_active":"2023-10-16T20:37:37.401383+0000","last_peered":"2023-10-16T20:37:37.401383+0000","last_clean":"2023-10-16T20:37:37.401383+0000","last_became_active":"2023-10-16T20:36:55.269749+0000","last_became_peered":"2023-10-16T20:36:55.269749+0000","last_unstale":"2023-10-16T20:37:37.401383+0000","last_undegraded":"2023-10-16T20:37:37.401383+0000","last_fullsized":"2023-10-16T20:37:37.401383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:30.925314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'6","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.522848+0000","last_change":"2023-10-16T20:36:54.757903+0000","last_active":"2023-10-16T20:39:39.522848+0000","last_peered":"2023-10-16T20:39:39.522848+0000","last_clean":"2023-10-16T20:39:39.522848+0000","last_became_active":"2023-10-16T20:36:54.757549+0000","last_became_peered":"2023-10-16T20:36:54.757549+0000","last_unstale":"2023-10-16T20:39:39.522848+0000","last_undegraded":"2023-10-16T20:39:39.522848+0000","last_fullsized":"2023-10-16T20:39:39.522848+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:02:42.090848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402786+0000","last_change":"2023-10-16T20:36:59.302063+0000","last_active":"2023-10-16T20:37:37.402786+0000","last_peered":"2023-10-16T20:37:37.402786+0000","last_clean":"2023-10-16T20:37:37.402786+0000","last_became_active":"2023-10-16T20:36:59.301693+0000","last_became_peered":"2023-10-16T20:36:59.301693+0000","last_unstale":"2023-10-16T20:37:37.402786+0000","last_undegraded":"2023-10-16T20:37:37.402786+0000","last_fullsized":"2023-10-16T20:37:37.402786+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:29:44.812282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434713+0000","last_change":"2023-10-16T20:36:55.278132+0000","last_active":"2023-10-16T20:37:37.434713+0000","last_peered":"2023-10-16T20:37:37.434713+0000","last_clean":"2023-10-16T20:37:37.434713+0000","last_became_active":"2023-10-16T20:36:55.276649+0000","last_became_peered":"2023-10-16T20:36:55.276649+0000","last_unstale":"2023-10-16T20:37:37.434713+0000","last_undegraded":"2023-10-16T20:37:37.434713+0000","last_fullsized":"2023-10-16T20:37:37.434713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:30:53.299514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:28.448344+0000","last_change":"2023-10-16T20:36:54.753592+0000","last_active":"2023-10-16T20:39:28.448344+0000","last_peered":"2023-10-16T20:39:28.448344+0000","last_clean":"2023-10-16T20:39:28.448344+0000","last_became_active":"2023-10-16T20:36:54.753237+0000","last_became_peered":"2023-10-16T20:36:54.753237+0000","last_unstale":"2023-10-16T20:39:28.448344+0000","last_undegraded":"2023-10-16T20:39:28.448344+0000","last_fullsized":"2023-10-16T20:39:28.448344+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:53.005979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435360+0000","last_change":"2023-10-16T20:36:59.468431+0000","last_active":"2023-10-16T20:37:37.435360+0000","last_peered":"2023-10-16T20:37:37.435360+0000","last_clean":"2023-10-16T20:37:37.435360+0000","last_became_active":"2023-10-16T20:36:59.466708+0000","last_became_peered":"2023-10-16T20:36:59.466708+0000","last_unstale":"2023-10-16T20:37:37.435360+0000","last_undegraded":"2023-10-16T20:37:37.435360+0000","last_fullsized":"2023-10-16T20:37:37.435360+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:42:20.607519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464817+0000","last_change":"2023-10-16T20:36:55.294825+0000","last_active":"2023-10-16T20:37:37.464817+0000","last_peered":"2023-10-16T20:37:37.464817+0000","last_clean":"2023-10-16T20:37:37.464817+0000","last_became_active":"2023-10-16T20:36:55.294610+0000","last_became_peered":"2023-10-16T20:36:55.294610+0000","last_unstale":"2023-10-16T20:37:37.464817+0000","last_undegraded":"2023-10-16T20:37:37.464817+0000","last_fullsized":"2023-10-16T20:37:37.464817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:43.442939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.507413+0000","last_change":"2023-10-16T20:36:54.741191+0000","last_active":"2023-10-16T20:39:33.507413+0000","last_peered":"2023-10-16T20:39:33.507413+0000","last_clean":"2023-10-16T20:39:33.507413+0000","last_became_active":"2023-10-16T20:36:54.740944+0000","last_became_peered":"2023-10-16T20:36:54.740944+0000","last_unstale":"2023-10-16T20:39:33.507413+0000","last_undegraded":"2023-10-16T20:39:33.507413+0000","last_fullsized":"2023-10-16T20:39:33.507413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:05.961850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":172,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466038+0000","last_change":"2023-10-16T20:36:46.390001+0000","last_active":"2023-10-16T20:37:37.466038+0000","last_peered":"2023-10-16T20:37:37.466038+0000","last_clean":"2023-10-16T20:37:37.466038+0000","last_became_active":"2023-10-16T20:36:46.388824+0000","last_became_peered":"2023-10-16T20:36:46.388824+0000","last_unstale":"2023-10-16T20:37:37.466038+0000","last_undegraded":"2023-10-16T20:37:37.466038+0000","last_fullsized":"2023-10-16T20:37:37.466038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_clean_scrub_stamp":"2023-10-16T20:36:44.575506+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:31.004318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407653+0000","last_change":"2023-10-16T20:36:59.468528+0000","last_active":"2023-10-16T20:37:37.407653+0000","last_peered":"2023-10-16T20:37:37.407653+0000","last_clean":"2023-10-16T20:37:37.407653+0000","last_became_active":"2023-10-16T20:36:59.468082+0000","last_became_peered":"2023-10-16T20:36:59.468082+0000","last_unstale":"2023-10-16T20:37:37.407653+0000","last_undegraded":"2023-10-16T20:37:37.407653+0000","last_fullsized":"2023-10-16T20:37:37.407653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:00.920004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.613641+0000","last_change":"2023-10-16T20:37:37.409578+0000","last_active":"2023-10-16T20:37:37.613641+0000","last_peered":"2023-10-16T20:37:37.613641+0000","last_clean":"2023-10-16T20:37:37.613641+0000","last_became_active":"2023-10-16T20:37:37.409004+0000","last_became_peered":"2023-10-16T20:37:37.409004+0000","last_unstale":"2023-10-16T20:37:37.613641+0000","last_undegraded":"2023-10-16T20:37:37.613641+0000","last_fullsized":"2023-10-16T20:37:37.613641+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:28:57.364794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"29'6","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.507117+0000","last_change":"2023-10-16T20:36:54.746600+0000","last_active":"2023-10-16T20:39:33.507117+0000","last_peered":"2023-10-16T20:39:33.507117+0000","last_clean":"2023-10-16T20:39:33.507117+0000","last_became_active":"2023-10-16T20:36:54.746247+0000","last_became_peered":"2023-10-16T20:36:54.746247+0000","last_unstale":"2023-10-16T20:39:33.507117+0000","last_undegraded":"2023-10-16T20:39:33.507117+0000","last_fullsized":"2023-10-16T20:39:33.507117+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:28:57.949487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1610,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.470531+0000","last_change":"2023-10-16T20:37:37.469427+0000","last_active":"2023-10-16T20:37:37.470531+0000","last_peered":"2023-10-16T20:37:37.470531+0000","last_clean":"2023-10-16T20:37:37.470531+0000","last_became_active":"2023-10-16T20:37:37.467402+0000","last_became_peered":"2023-10-16T20:37:37.467402+0000","last_unstale":"2023-10-16T20:37:37.470531+0000","last_undegraded":"2023-10-16T20:37:37.470531+0000","last_fullsized":"2023-10-16T20:37:37.470531+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:00:19.171972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409113+0000","last_change":"2023-10-16T20:36:55.301234+0000","last_active":"2023-10-16T20:37:37.409113+0000","last_peered":"2023-10-16T20:37:37.409113+0000","last_clean":"2023-10-16T20:37:37.409113+0000","last_became_active":"2023-10-16T20:36:55.300958+0000","last_became_peered":"2023-10-16T20:36:55.300958+0000","last_unstale":"2023-10-16T20:37:37.409113+0000","last_undegraded":"2023-10-16T20:37:37.409113+0000","last_fullsized":"2023-10-16T20:37:37.409113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:16.784124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.506025+0000","last_change":"2023-10-16T20:36:54.753770+0000","last_active":"2023-10-16T20:39:33.506025+0000","last_peered":"2023-10-16T20:39:33.506025+0000","last_clean":"2023-10-16T20:39:33.506025+0000","last_became_active":"2023-10-16T20:36:54.753443+0000","last_became_peered":"2023-10-16T20:36:54.753443+0000","last_unstale":"2023-10-16T20:39:33.506025+0000","last_undegraded":"2023-10-16T20:39:33.506025+0000","last_fullsized":"2023-10-16T20:39:33.506025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:21:26.392480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435046+0000","last_change":"2023-10-16T20:36:59.303351+0000","last_active":"2023-10-16T20:37:37.435046+0000","last_peered":"2023-10-16T20:37:37.435046+0000","last_clean":"2023-10-16T20:37:37.435046+0000","last_became_active":"2023-10-16T20:36:59.302869+0000","last_became_peered":"2023-10-16T20:36:59.302869+0000","last_unstale":"2023-10-16T20:37:37.435046+0000","last_undegraded":"2023-10-16T20:37:37.435046+0000","last_fullsized":"2023-10-16T20:37:37.435046+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:26:35.886399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466830+0000","last_change":"2023-10-16T20:36:55.291947+0000","last_active":"2023-10-16T20:37:37.466830+0000","last_peered":"2023-10-16T20:37:37.466830+0000","last_clean":"2023-10-16T20:37:37.466830+0000","last_became_active":"2023-10-16T20:36:55.291633+0000","last_became_peered":"2023-10-16T20:36:55.291633+0000","last_unstale":"2023-10-16T20:37:37.466830+0000","last_undegraded":"2023-10-16T20:37:37.466830+0000","last_fullsized":"2023-10-16T20:37:37.466830+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:00:14.349339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.519506+0000","last_change":"2023-10-16T20:36:54.748196+0000","last_active":"2023-10-16T20:39:39.519506+0000","last_peered":"2023-10-16T20:39:39.519506+0000","last_clean":"2023-10-16T20:39:39.519506+0000","last_became_active":"2023-10-16T20:36:54.747704+0000","last_became_peered":"2023-10-16T20:36:54.747704+0000","last_unstale":"2023-10-16T20:39:39.519506+0000","last_undegraded":"2023-10-16T20:39:39.519506+0000","last_fullsized":"2023-10-16T20:39:39.519506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:55.471916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410236+0000","last_change":"2023-10-16T20:36:59.470368+0000","last_active":"2023-10-16T20:37:37.410236+0000","last_peered":"2023-10-16T20:37:37.410236+0000","last_clean":"2023-10-16T20:37:37.410236+0000","last_became_active":"2023-10-16T20:36:59.467291+0000","last_became_peered":"2023-10-16T20:36:59.467291+0000","last_unstale":"2023-10-16T20:37:37.410236+0000","last_undegraded":"2023-10-16T20:37:37.410236+0000","last_fullsized":"2023-10-16T20:37:37.410236+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:18.312394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466658+0000","last_change":"2023-10-16T20:36:55.291451+0000","last_active":"2023-10-16T20:37:37.466658+0000","last_peered":"2023-10-16T20:37:37.466658+0000","last_clean":"2023-10-16T20:37:37.466658+0000","last_became_active":"2023-10-16T20:36:55.290998+0000","last_became_peered":"2023-10-16T20:36:55.290998+0000","last_unstale":"2023-10-16T20:37:37.466658+0000","last_undegraded":"2023-10-16T20:37:37.466658+0000","last_fullsized":"2023-10-16T20:37:37.466658+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:05:01.634971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"29'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:12.629039+0000","last_change":"2023-10-16T20:36:54.763818+0000","last_active":"2023-10-16T20:39:12.629039+0000","last_peered":"2023-10-16T20:39:12.629039+0000","last_clean":"2023-10-16T20:39:12.629039+0000","last_became_active":"2023-10-16T20:36:54.761337+0000","last_became_peered":"2023-10-16T20:36:54.761337+0000","last_unstale":"2023-10-16T20:39:12.629039+0000","last_undegraded":"2023-10-16T20:39:12.629039+0000","last_fullsized":"2023-10-16T20:39:12.629039+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:03.093189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.470301+0000","last_change":"2023-10-16T20:37:37.468803+0000","last_active":"2023-10-16T20:37:37.470301+0000","last_peered":"2023-10-16T20:37:37.470301+0000","last_clean":"2023-10-16T20:37:37.470301+0000","last_became_active":"2023-10-16T20:37:37.468309+0000","last_became_peered":"2023-10-16T20:37:37.468309+0000","last_unstale":"2023-10-16T20:37:37.470301+0000","last_undegraded":"2023-10-16T20:37:37.470301+0000","last_fullsized":"2023-10-16T20:37:37.470301+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:11.761661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466101+0000","last_change":"2023-10-16T20:36:55.288895+0000","last_active":"2023-10-16T20:37:37.466101+0000","last_peered":"2023-10-16T20:37:37.466101+0000","last_clean":"2023-10-16T20:37:37.466101+0000","last_became_active":"2023-10-16T20:36:55.288354+0000","last_became_peered":"2023-10-16T20:36:55.288354+0000","last_unstale":"2023-10-16T20:37:37.466101+0000","last_undegraded":"2023-10-16T20:37:37.466101+0000","last_fullsized":"2023-10-16T20:37:37.466101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:53:29.192406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'17","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.501703+0000","last_change":"2023-10-16T20:36:54.738216+0000","last_active":"2023-10-16T20:39:33.501703+0000","last_peered":"2023-10-16T20:39:33.501703+0000","last_clean":"2023-10-16T20:39:33.501703+0000","last_became_active":"2023-10-16T20:36:54.737876+0000","last_became_peered":"2023-10-16T20:36:54.737876+0000","last_unstale":"2023-10-16T20:39:33.501703+0000","last_undegraded":"2023-10-16T20:39:33.501703+0000","last_fullsized":"2023-10-16T20:39:33.501703+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:28.214601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435178+0000","last_change":"2023-10-16T20:36:59.307215+0000","last_active":"2023-10-16T20:37:37.435178+0000","last_peered":"2023-10-16T20:37:37.435178+0000","last_clean":"2023-10-16T20:37:37.435178+0000","last_became_active":"2023-10-16T20:36:59.306801+0000","last_became_peered":"2023-10-16T20:36:59.306801+0000","last_unstale":"2023-10-16T20:37:37.435178+0000","last_undegraded":"2023-10-16T20:37:37.435178+0000","last_fullsized":"2023-10-16T20:37:37.435178+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:41:42.656134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407054+0000","last_change":"2023-10-16T20:36:55.287714+0000","last_active":"2023-10-16T20:37:37.407054+0000","last_peered":"2023-10-16T20:37:37.407054+0000","last_clean":"2023-10-16T20:37:37.407054+0000","last_became_active":"2023-10-16T20:36:55.287066+0000","last_became_peered":"2023-10-16T20:36:55.287066+0000","last_unstale":"2023-10-16T20:37:37.407054+0000","last_undegraded":"2023-10-16T20:37:37.407054+0000","last_fullsized":"2023-10-16T20:37:37.407054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:45:21.058260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"29'6","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.509257+0000","last_change":"2023-10-16T20:36:54.764693+0000","last_active":"2023-10-16T20:39:33.509257+0000","last_peered":"2023-10-16T20:39:33.509257+0000","last_clean":"2023-10-16T20:39:33.509257+0000","last_became_active":"2023-10-16T20:36:54.764417+0000","last_became_peered":"2023-10-16T20:36:54.764417+0000","last_unstale":"2023-10-16T20:39:33.509257+0000","last_undegraded":"2023-10-16T20:39:33.509257+0000","last_fullsized":"2023-10-16T20:39:33.509257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:56.653149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1028,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465588+0000","last_change":"2023-10-16T20:36:59.465145+0000","last_active":"2023-10-16T20:37:37.465588+0000","last_peered":"2023-10-16T20:37:37.465588+0000","last_clean":"2023-10-16T20:37:37.465588+0000","last_became_active":"2023-10-16T20:36:59.464626+0000","last_became_peered":"2023-10-16T20:36:59.464626+0000","last_unstale":"2023-10-16T20:37:37.465588+0000","last_undegraded":"2023-10-16T20:37:37.465588+0000","last_fullsized":"2023-10-16T20:37:37.465588+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:38.928616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466290+0000","last_change":"2023-10-16T20:36:55.291718+0000","last_active":"2023-10-16T20:37:37.466290+0000","last_peered":"2023-10-16T20:37:37.466290+0000","last_clean":"2023-10-16T20:37:37.466290+0000","last_became_active":"2023-10-16T20:36:55.291396+0000","last_became_peered":"2023-10-16T20:36:55.291396+0000","last_unstale":"2023-10-16T20:37:37.466290+0000","last_undegraded":"2023-10-16T20:37:37.466290+0000","last_fullsized":"2023-10-16T20:37:37.466290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:29:49.138380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.507860+0000","last_change":"2023-10-16T20:36:54.750814+0000","last_active":"2023-10-16T20:39:33.507860+0000","last_peered":"2023-10-16T20:39:33.507860+0000","last_clean":"2023-10-16T20:39:33.507860+0000","last_became_active":"2023-10-16T20:36:54.750448+0000","last_became_peered":"2023-10-16T20:36:54.750448+0000","last_unstale":"2023-10-16T20:39:33.507860+0000","last_undegraded":"2023-10-16T20:39:33.507860+0000","last_fullsized":"2023-10-16T20:39:33.507860+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:41.981408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465192+0000","last_change":"2023-10-16T20:36:59.464962+0000","last_active":"2023-10-16T20:37:37.465192+0000","last_peered":"2023-10-16T20:37:37.465192+0000","last_clean":"2023-10-16T20:37:37.465192+0000","last_became_active":"2023-10-16T20:36:59.464578+0000","last_became_peered":"2023-10-16T20:36:59.464578+0000","last_unstale":"2023-10-16T20:37:37.465192+0000","last_undegraded":"2023-10-16T20:37:37.465192+0000","last_fullsized":"2023-10-16T20:37:37.465192+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:12:27.738364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435179+0000","last_change":"2023-10-16T20:36:55.285675+0000","last_active":"2023-10-16T20:37:37.435179+0000","last_peered":"2023-10-16T20:37:37.435179+0000","last_clean":"2023-10-16T20:37:37.435179+0000","last_became_active":"2023-10-16T20:36:55.285407+0000","last_became_peered":"2023-10-16T20:36:55.285407+0000","last_unstale":"2023-10-16T20:37:37.435179+0000","last_undegraded":"2023-10-16T20:37:37.435179+0000","last_fullsized":"2023-10-16T20:37:37.435179+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:31:10.259938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'9","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.524706+0000","last_change":"2023-10-16T20:36:54.757012+0000","last_active":"2023-10-16T20:39:39.524706+0000","last_peered":"2023-10-16T20:39:39.524706+0000","last_clean":"2023-10-16T20:39:39.524706+0000","last_became_active":"2023-10-16T20:36:54.756697+0000","last_became_peered":"2023-10-16T20:36:54.756697+0000","last_unstale":"2023-10-16T20:39:39.524706+0000","last_undegraded":"2023-10-16T20:39:39.524706+0000","last_fullsized":"2023-10-16T20:39:39.524706+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:20.259580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409477+0000","last_change":"2023-10-16T20:36:59.469134+0000","last_active":"2023-10-16T20:37:37.409477+0000","last_peered":"2023-10-16T20:37:37.409477+0000","last_clean":"2023-10-16T20:37:37.409477+0000","last_became_active":"2023-10-16T20:36:59.468254+0000","last_became_peered":"2023-10-16T20:36:59.468254+0000","last_unstale":"2023-10-16T20:37:37.409477+0000","last_undegraded":"2023-10-16T20:37:37.409477+0000","last_fullsized":"2023-10-16T20:37:37.409477+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:12.444815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466418+0000","last_change":"2023-10-16T20:36:55.289133+0000","last_active":"2023-10-16T20:37:37.466418+0000","last_peered":"2023-10-16T20:37:37.466418+0000","last_clean":"2023-10-16T20:37:37.466418+0000","last_became_active":"2023-10-16T20:36:55.288488+0000","last_became_peered":"2023-10-16T20:36:55.288488+0000","last_unstale":"2023-10-16T20:37:37.466418+0000","last_undegraded":"2023-10-16T20:37:37.466418+0000","last_fullsized":"2023-10-16T20:37:37.466418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:27:22.022319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'16","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:48.656539+0000","last_change":"2023-10-16T20:36:54.765361+0000","last_active":"2023-10-16T20:39:48.656539+0000","last_peered":"2023-10-16T20:39:48.656539+0000","last_clean":"2023-10-16T20:39:48.656539+0000","last_became_active":"2023-10-16T20:36:54.764969+0000","last_became_peered":"2023-10-16T20:36:54.764969+0000","last_unstale":"2023-10-16T20:39:48.656539+0000","last_undegraded":"2023-10-16T20:39:48.656539+0000","last_fullsized":"2023-10-16T20:39:48.656539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:21:21.183612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4281,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402193+0000","last_change":"2023-10-16T20:36:59.303588+0000","last_active":"2023-10-16T20:37:37.402193+0000","last_peered":"2023-10-16T20:37:37.402193+0000","last_clean":"2023-10-16T20:37:37.402193+0000","last_became_active":"2023-10-16T20:36:59.302698+0000","last_became_peered":"2023-10-16T20:36:59.302698+0000","last_unstale":"2023-10-16T20:37:37.402193+0000","last_undegraded":"2023-10-16T20:37:37.402193+0000","last_fullsized":"2023-10-16T20:37:37.402193+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:08:28.644565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403928+0000","last_change":"2023-10-16T20:36:55.290812+0000","last_active":"2023-10-16T20:37:37.403928+0000","last_peered":"2023-10-16T20:37:37.403928+0000","last_clean":"2023-10-16T20:37:37.403928+0000","last_became_active":"2023-10-16T20:36:55.290127+0000","last_became_peered":"2023-10-16T20:36:55.290127+0000","last_unstale":"2023-10-16T20:37:37.403928+0000","last_undegraded":"2023-10-16T20:37:37.403928+0000","last_fullsized":"2023-10-16T20:37:37.403928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:28:27.930123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.406599+0000","last_change":"2023-10-16T20:36:54.749120+0000","last_active":"2023-10-16T20:39:22.406599+0000","last_peered":"2023-10-16T20:39:22.406599+0000","last_clean":"2023-10-16T20:39:22.406599+0000","last_became_active":"2023-10-16T20:36:54.748788+0000","last_became_peered":"2023-10-16T20:36:54.748788+0000","last_unstale":"2023-10-16T20:39:22.406599+0000","last_undegraded":"2023-10-16T20:39:22.406599+0000","last_fullsized":"2023-10-16T20:39:22.406599+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:11:07.257343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403049+0000","last_change":"2023-10-16T20:36:59.303546+0000","last_active":"2023-10-16T20:37:37.403049+0000","last_peered":"2023-10-16T20:37:37.403049+0000","last_clean":"2023-10-16T20:37:37.403049+0000","last_became_active":"2023-10-16T20:36:59.302916+0000","last_became_peered":"2023-10-16T20:36:59.302916+0000","last_unstale":"2023-10-16T20:37:37.403049+0000","last_undegraded":"2023-10-16T20:37:37.403049+0000","last_fullsized":"2023-10-16T20:37:37.403049+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:17:16.174476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435547+0000","last_change":"2023-10-16T20:36:55.291739+0000","last_active":"2023-10-16T20:37:37.435547+0000","last_peered":"2023-10-16T20:37:37.435547+0000","last_clean":"2023-10-16T20:37:37.435547+0000","last_became_active":"2023-10-16T20:36:55.291448+0000","last_became_peered":"2023-10-16T20:36:55.291448+0000","last_unstale":"2023-10-16T20:37:37.435547+0000","last_undegraded":"2023-10-16T20:37:37.435547+0000","last_fullsized":"2023-10-16T20:37:37.435547+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:41:15.380678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.526865+0000","last_change":"2023-10-16T20:36:54.761824+0000","last_active":"2023-10-16T20:39:39.526865+0000","last_peered":"2023-10-16T20:39:39.526865+0000","last_clean":"2023-10-16T20:39:39.526865+0000","last_became_active":"2023-10-16T20:36:54.761541+0000","last_became_peered":"2023-10-16T20:36:54.761541+0000","last_unstale":"2023-10-16T20:39:39.526865+0000","last_undegraded":"2023-10-16T20:39:39.526865+0000","last_fullsized":"2023-10-16T20:39:39.526865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:34.512207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464594+0000","last_change":"2023-10-16T20:36:59.304165+0000","last_active":"2023-10-16T20:37:37.464594+0000","last_peered":"2023-10-16T20:37:37.464594+0000","last_clean":"2023-10-16T20:37:37.464594+0000","last_became_active":"2023-10-16T20:36:59.303773+0000","last_became_peered":"2023-10-16T20:36:59.303773+0000","last_unstale":"2023-10-16T20:37:37.464594+0000","last_undegraded":"2023-10-16T20:37:37.464594+0000","last_fullsized":"2023-10-16T20:37:37.464594+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:09.710698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409711+0000","last_change":"2023-10-16T20:36:55.300829+0000","last_active":"2023-10-16T20:37:37.409711+0000","last_peered":"2023-10-16T20:37:37.409711+0000","last_clean":"2023-10-16T20:37:37.409711+0000","last_became_active":"2023-10-16T20:36:55.300535+0000","last_became_peered":"2023-10-16T20:36:55.300535+0000","last_unstale":"2023-10-16T20:37:37.409711+0000","last_undegraded":"2023-10-16T20:37:37.409711+0000","last_fullsized":"2023-10-16T20:37:37.409711+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:34.910378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.508094+0000","last_change":"2023-10-16T20:36:54.758140+0000","last_active":"2023-10-16T20:39:33.508094+0000","last_peered":"2023-10-16T20:39:33.508094+0000","last_clean":"2023-10-16T20:39:33.508094+0000","last_became_active":"2023-10-16T20:36:54.757303+0000","last_became_peered":"2023-10-16T20:36:54.757303+0000","last_unstale":"2023-10-16T20:39:33.508094+0000","last_undegraded":"2023-10-16T20:39:33.508094+0000","last_fullsized":"2023-10-16T20:39:33.508094+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:45.944191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464795+0000","last_change":"2023-10-16T20:36:59.471158+0000","last_active":"2023-10-16T20:37:37.464795+0000","last_peered":"2023-10-16T20:37:37.464795+0000","last_clean":"2023-10-16T20:37:37.464795+0000","last_became_active":"2023-10-16T20:36:59.465844+0000","last_became_peered":"2023-10-16T20:36:59.465844+0000","last_unstale":"2023-10-16T20:37:37.464795+0000","last_undegraded":"2023-10-16T20:37:37.464795+0000","last_fullsized":"2023-10-16T20:37:37.464795+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:12.133525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435182+0000","last_change":"2023-10-16T20:36:59.468843+0000","last_active":"2023-10-16T20:37:37.435182+0000","last_peered":"2023-10-16T20:37:37.435182+0000","last_clean":"2023-10-16T20:37:37.435182+0000","last_became_active":"2023-10-16T20:36:59.467356+0000","last_became_peered":"2023-10-16T20:36:59.467356+0000","last_unstale":"2023-10-16T20:37:37.435182+0000","last_undegraded":"2023-10-16T20:37:37.435182+0000","last_fullsized":"2023-10-16T20:37:37.435182+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:47:02.923540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401755+0000","last_change":"2023-10-16T20:36:55.281355+0000","last_active":"2023-10-16T20:37:37.401755+0000","last_peered":"2023-10-16T20:37:37.401755+0000","last_clean":"2023-10-16T20:37:37.401755+0000","last_became_active":"2023-10-16T20:36:55.280903+0000","last_became_peered":"2023-10-16T20:36:55.280903+0000","last_unstale":"2023-10-16T20:37:37.401755+0000","last_undegraded":"2023-10-16T20:37:37.401755+0000","last_fullsized":"2023-10-16T20:37:37.401755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:15:25.813826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.507692+0000","last_change":"2023-10-16T20:36:54.757589+0000","last_active":"2023-10-16T20:39:33.507692+0000","last_peered":"2023-10-16T20:39:33.507692+0000","last_clean":"2023-10-16T20:39:33.507692+0000","last_became_active":"2023-10-16T20:36:54.756775+0000","last_became_peered":"2023-10-16T20:36:54.756775+0000","last_unstale":"2023-10-16T20:39:33.507692+0000","last_undegraded":"2023-10-16T20:39:33.507692+0000","last_fullsized":"2023-10-16T20:39:33.507692+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:40.629540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.441661+0000","last_change":"2023-10-16T20:37:37.438581+0000","last_active":"2023-10-16T20:37:37.441661+0000","last_peered":"2023-10-16T20:37:37.441661+0000","last_clean":"2023-10-16T20:37:37.441661+0000","last_became_active":"2023-10-16T20:37:37.438140+0000","last_became_peered":"2023-10-16T20:37:37.438140+0000","last_unstale":"2023-10-16T20:37:37.441661+0000","last_undegraded":"2023-10-16T20:37:37.441661+0000","last_fullsized":"2023-10-16T20:37:37.441661+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:55.110618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464854+0000","last_change":"2023-10-16T20:36:55.274207+0000","last_active":"2023-10-16T20:37:37.464854+0000","last_peered":"2023-10-16T20:37:37.464854+0000","last_clean":"2023-10-16T20:37:37.464854+0000","last_became_active":"2023-10-16T20:36:55.273959+0000","last_became_peered":"2023-10-16T20:36:55.273959+0000","last_unstale":"2023-10-16T20:37:37.464854+0000","last_undegraded":"2023-10-16T20:37:37.464854+0000","last_fullsized":"2023-10-16T20:37:37.464854+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:23:09.275091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:28.444227+0000","last_change":"2023-10-16T20:36:54.759513+0000","last_active":"2023-10-16T20:39:28.444227+0000","last_peered":"2023-10-16T20:39:28.444227+0000","last_clean":"2023-10-16T20:39:28.444227+0000","last_became_active":"2023-10-16T20:36:54.759059+0000","last_became_peered":"2023-10-16T20:36:54.759059+0000","last_unstale":"2023-10-16T20:39:28.444227+0000","last_undegraded":"2023-10-16T20:39:28.444227+0000","last_fullsized":"2023-10-16T20:39:28.444227+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:36.696053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.509348+0000","last_change":"2023-10-16T20:36:54.764780+0000","last_active":"2023-10-16T20:39:33.509348+0000","last_peered":"2023-10-16T20:39:33.509348+0000","last_clean":"2023-10-16T20:39:33.509348+0000","last_became_active":"2023-10-16T20:36:54.764436+0000","last_became_peered":"2023-10-16T20:36:54.764436+0000","last_unstale":"2023-10-16T20:39:33.509348+0000","last_undegraded":"2023-10-16T20:39:33.509348+0000","last_fullsized":"2023-10-16T20:39:33.509348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:55:31.816599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435152+0000","last_change":"2023-10-16T20:36:55.278134+0000","last_active":"2023-10-16T20:37:37.435152+0000","last_peered":"2023-10-16T20:37:37.435152+0000","last_clean":"2023-10-16T20:37:37.435152+0000","last_became_active":"2023-10-16T20:36:55.277873+0000","last_became_peered":"2023-10-16T20:36:55.277873+0000","last_unstale":"2023-10-16T20:37:37.435152+0000","last_undegraded":"2023-10-16T20:37:37.435152+0000","last_fullsized":"2023-10-16T20:37:37.435152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:59:43.701800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.399947+0000","last_change":"2023-10-16T20:36:59.302774+0000","last_active":"2023-10-16T20:37:37.399947+0000","last_peered":"2023-10-16T20:37:37.399947+0000","last_clean":"2023-10-16T20:37:37.399947+0000","last_became_active":"2023-10-16T20:36:59.302435+0000","last_became_peered":"2023-10-16T20:36:59.302435+0000","last_unstale":"2023-10-16T20:37:37.399947+0000","last_undegraded":"2023-10-16T20:37:37.399947+0000","last_fullsized":"2023-10-16T20:37:37.399947+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:11:53.740982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'6","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:27.631103+0000","last_change":"2023-10-16T20:36:54.756190+0000","last_active":"2023-10-16T20:39:27.631103+0000","last_peered":"2023-10-16T20:39:27.631103+0000","last_clean":"2023-10-16T20:39:27.631103+0000","last_became_active":"2023-10-16T20:36:54.755834+0000","last_became_peered":"2023-10-16T20:36:54.755834+0000","last_unstale":"2023-10-16T20:39:27.631103+0000","last_undegraded":"2023-10-16T20:39:27.631103+0000","last_fullsized":"2023-10-16T20:39:27.631103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:31.012441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1062,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403395+0000","last_change":"2023-10-16T20:36:55.279822+0000","last_active":"2023-10-16T20:37:37.403395+0000","last_peered":"2023-10-16T20:37:37.403395+0000","last_clean":"2023-10-16T20:37:37.403395+0000","last_became_active":"2023-10-16T20:36:55.279333+0000","last_became_peered":"2023-10-16T20:36:55.279333+0000","last_unstale":"2023-10-16T20:37:37.403395+0000","last_undegraded":"2023-10-16T20:37:37.403395+0000","last_fullsized":"2023-10-16T20:37:37.403395+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:02:21.471812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465391+0000","last_change":"2023-10-16T20:36:59.302190+0000","last_active":"2023-10-16T20:37:37.465391+0000","last_peered":"2023-10-16T20:37:37.465391+0000","last_clean":"2023-10-16T20:37:37.465391+0000","last_became_active":"2023-10-16T20:36:59.301864+0000","last_became_peered":"2023-10-16T20:36:59.301864+0000","last_unstale":"2023-10-16T20:37:37.465391+0000","last_undegraded":"2023-10-16T20:37:37.465391+0000","last_fullsized":"2023-10-16T20:37:37.465391+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:37:55.282815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.406436+0000","last_change":"2023-10-16T20:36:55.302476+0000","last_active":"2023-10-16T20:37:37.406436+0000","last_peered":"2023-10-16T20:37:37.406436+0000","last_clean":"2023-10-16T20:37:37.406436+0000","last_became_active":"2023-10-16T20:36:55.287442+0000","last_became_peered":"2023-10-16T20:36:55.287442+0000","last_unstale":"2023-10-16T20:37:37.406436+0000","last_undegraded":"2023-10-16T20:37:37.406436+0000","last_fullsized":"2023-10-16T20:37:37.406436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:19.511156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.399520+0000","last_change":"2023-10-16T20:36:55.276521+0000","last_active":"2023-10-16T20:37:37.399520+0000","last_peered":"2023-10-16T20:37:37.399520+0000","last_clean":"2023-10-16T20:37:37.399520+0000","last_became_active":"2023-10-16T20:36:55.276276+0000","last_became_peered":"2023-10-16T20:36:55.276276+0000","last_unstale":"2023-10-16T20:37:37.399520+0000","last_undegraded":"2023-10-16T20:37:37.399520+0000","last_fullsized":"2023-10-16T20:37:37.399520+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:38.218242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.415548+0000","last_change":"2023-10-16T20:37:37.413429+0000","last_active":"2023-10-16T20:37:37.415548+0000","last_peered":"2023-10-16T20:37:37.415548+0000","last_clean":"2023-10-16T20:37:37.415548+0000","last_became_active":"2023-10-16T20:37:37.413040+0000","last_became_peered":"2023-10-16T20:37:37.413040+0000","last_unstale":"2023-10-16T20:37:37.415548+0000","last_undegraded":"2023-10-16T20:37:37.415548+0000","last_fullsized":"2023-10-16T20:37:37.415548+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:39:10.897056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435108+0000","last_change":"2023-10-16T20:36:55.276258+0000","last_active":"2023-10-16T20:37:37.435108+0000","last_peered":"2023-10-16T20:37:37.435108+0000","last_clean":"2023-10-16T20:37:37.435108+0000","last_became_active":"2023-10-16T20:36:55.275349+0000","last_became_peered":"2023-10-16T20:36:55.275349+0000","last_unstale":"2023-10-16T20:37:37.435108+0000","last_undegraded":"2023-10-16T20:37:37.435108+0000","last_fullsized":"2023-10-16T20:37:37.435108+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:59.007782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465455+0000","last_change":"2023-10-16T20:36:55.277332+0000","last_active":"2023-10-16T20:37:37.465455+0000","last_peered":"2023-10-16T20:37:37.465455+0000","last_clean":"2023-10-16T20:37:37.465455+0000","last_became_active":"2023-10-16T20:36:55.277043+0000","last_became_peered":"2023-10-16T20:36:55.277043+0000","last_unstale":"2023-10-16T20:37:37.465455+0000","last_undegraded":"2023-10-16T20:37:37.465455+0000","last_fullsized":"2023-10-16T20:37:37.465455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:46:07.227813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401690+0000","last_change":"2023-10-16T20:36:55.279003+0000","last_active":"2023-10-16T20:37:37.401690+0000","last_peered":"2023-10-16T20:37:37.401690+0000","last_clean":"2023-10-16T20:37:37.401690+0000","last_became_active":"2023-10-16T20:36:55.278715+0000","last_became_peered":"2023-10-16T20:36:55.278715+0000","last_unstale":"2023-10-16T20:37:37.401690+0000","last_undegraded":"2023-10-16T20:37:37.401690+0000","last_fullsized":"2023-10-16T20:37:37.401690+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:58.274009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408078+0000","last_change":"2023-10-16T20:36:55.303936+0000","last_active":"2023-10-16T20:37:37.408078+0000","last_peered":"2023-10-16T20:37:37.408078+0000","last_clean":"2023-10-16T20:37:37.408078+0000","last_became_active":"2023-10-16T20:36:55.301604+0000","last_became_peered":"2023-10-16T20:36:55.301604+0000","last_unstale":"2023-10-16T20:37:37.408078+0000","last_undegraded":"2023-10-16T20:37:37.408078+0000","last_fullsized":"2023-10-16T20:37:37.408078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:48:14.922695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465625+0000","last_change":"2023-10-16T20:36:55.277329+0000","last_active":"2023-10-16T20:37:37.465625+0000","last_peered":"2023-10-16T20:37:37.465625+0000","last_clean":"2023-10-16T20:37:37.465625+0000","last_became_active":"2023-10-16T20:36:55.276984+0000","last_became_peered":"2023-10-16T20:36:55.276984+0000","last_unstale":"2023-10-16T20:37:37.465625+0000","last_undegraded":"2023-10-16T20:37:37.465625+0000","last_fullsized":"2023-10-16T20:37:37.465625+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:43:52.763586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408404+0000","last_change":"2023-10-16T20:36:55.290545+0000","last_active":"2023-10-16T20:37:37.408404+0000","last_peered":"2023-10-16T20:37:37.408404+0000","last_clean":"2023-10-16T20:37:37.408404+0000","last_became_active":"2023-10-16T20:36:55.290023+0000","last_became_peered":"2023-10-16T20:36:55.290023+0000","last_unstale":"2023-10-16T20:37:37.408404+0000","last_undegraded":"2023-10-16T20:37:37.408404+0000","last_fullsized":"2023-10-16T20:37:37.408404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:04:20.914854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464978+0000","last_change":"2023-10-16T20:36:55.280456+0000","last_active":"2023-10-16T20:37:37.464978+0000","last_peered":"2023-10-16T20:37:37.464978+0000","last_clean":"2023-10-16T20:37:37.464978+0000","last_became_active":"2023-10-16T20:36:55.280186+0000","last_became_peered":"2023-10-16T20:36:55.280186+0000","last_unstale":"2023-10-16T20:37:37.464978+0000","last_undegraded":"2023-10-16T20:37:37.464978+0000","last_fullsized":"2023-10-16T20:37:37.464978+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:52.469768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408252+0000","last_change":"2023-10-16T20:36:55.297917+0000","last_active":"2023-10-16T20:37:37.408252+0000","last_peered":"2023-10-16T20:37:37.408252+0000","last_clean":"2023-10-16T20:37:37.408252+0000","last_became_active":"2023-10-16T20:36:55.297253+0000","last_became_peered":"2023-10-16T20:36:55.297253+0000","last_unstale":"2023-10-16T20:37:37.408252+0000","last_undegraded":"2023-10-16T20:37:37.408252+0000","last_fullsized":"2023-10-16T20:37:37.408252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:14.977093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434979+0000","last_change":"2023-10-16T20:36:55.286327+0000","last_active":"2023-10-16T20:37:37.434979+0000","last_peered":"2023-10-16T20:37:37.434979+0000","last_clean":"2023-10-16T20:37:37.434979+0000","last_became_active":"2023-10-16T20:36:55.286044+0000","last_became_peered":"2023-10-16T20:36:55.286044+0000","last_unstale":"2023-10-16T20:37:37.434979+0000","last_undegraded":"2023-10-16T20:37:37.434979+0000","last_fullsized":"2023-10-16T20:37:37.434979+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:30:56.000502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409227+0000","last_change":"2023-10-16T20:36:55.298742+0000","last_active":"2023-10-16T20:37:37.409227+0000","last_peered":"2023-10-16T20:37:37.409227+0000","last_clean":"2023-10-16T20:37:37.409227+0000","last_became_active":"2023-10-16T20:36:55.286864+0000","last_became_peered":"2023-10-16T20:36:55.286864+0000","last_unstale":"2023-10-16T20:37:37.409227+0000","last_undegraded":"2023-10-16T20:37:37.409227+0000","last_fullsized":"2023-10-16T20:37:37.409227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:24:27.598910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466521+0000","last_change":"2023-10-16T20:36:55.292541+0000","last_active":"2023-10-16T20:37:37.466521+0000","last_peered":"2023-10-16T20:37:37.466521+0000","last_clean":"2023-10-16T20:37:37.466521+0000","last_became_active":"2023-10-16T20:36:55.292262+0000","last_became_peered":"2023-10-16T20:36:55.292262+0000","last_unstale":"2023-10-16T20:37:37.466521+0000","last_undegraded":"2023-10-16T20:37:37.466521+0000","last_fullsized":"2023-10-16T20:37:37.466521+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:10:08.644265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.400632+0000","last_change":"2023-10-16T20:36:55.283056+0000","last_active":"2023-10-16T20:37:37.400632+0000","last_peered":"2023-10-16T20:37:37.400632+0000","last_clean":"2023-10-16T20:37:37.400632+0000","last_became_active":"2023-10-16T20:36:55.282806+0000","last_became_peered":"2023-10-16T20:36:55.282806+0000","last_unstale":"2023-10-16T20:37:37.400632+0000","last_undegraded":"2023-10-16T20:37:37.400632+0000","last_fullsized":"2023-10-16T20:37:37.400632+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:21:24.681696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410649+0000","last_change":"2023-10-16T20:36:55.303385+0000","last_active":"2023-10-16T20:37:37.410649+0000","last_peered":"2023-10-16T20:37:37.410649+0000","last_clean":"2023-10-16T20:37:37.410649+0000","last_became_active":"2023-10-16T20:36:55.303095+0000","last_became_peered":"2023-10-16T20:36:55.303095+0000","last_unstale":"2023-10-16T20:37:37.410649+0000","last_undegraded":"2023-10-16T20:37:37.410649+0000","last_fullsized":"2023-10-16T20:37:37.410649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:26.368480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408312+0000","last_change":"2023-10-16T20:36:55.291852+0000","last_active":"2023-10-16T20:37:37.408312+0000","last_peered":"2023-10-16T20:37:37.408312+0000","last_clean":"2023-10-16T20:37:37.408312+0000","last_became_active":"2023-10-16T20:36:55.291447+0000","last_became_peered":"2023-10-16T20:36:55.291447+0000","last_unstale":"2023-10-16T20:37:37.408312+0000","last_undegraded":"2023-10-16T20:37:37.408312+0000","last_fullsized":"2023-10-16T20:37:37.408312+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:07.195372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401838+0000","last_change":"2023-10-16T20:36:55.281169+0000","last_active":"2023-10-16T20:37:37.401838+0000","last_peered":"2023-10-16T20:37:37.401838+0000","last_clean":"2023-10-16T20:37:37.401838+0000","last_became_active":"2023-10-16T20:36:55.280605+0000","last_became_peered":"2023-10-16T20:36:55.280605+0000","last_unstale":"2023-10-16T20:37:37.401838+0000","last_undegraded":"2023-10-16T20:37:37.401838+0000","last_fullsized":"2023-10-16T20:37:37.401838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:03.294173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435040+0000","last_change":"2023-10-16T20:36:55.277395+0000","last_active":"2023-10-16T20:37:37.435040+0000","last_peered":"2023-10-16T20:37:37.435040+0000","last_clean":"2023-10-16T20:37:37.435040+0000","last_became_active":"2023-10-16T20:36:55.277094+0000","last_became_peered":"2023-10-16T20:36:55.277094+0000","last_unstale":"2023-10-16T20:37:37.435040+0000","last_undegraded":"2023-10-16T20:37:37.435040+0000","last_fullsized":"2023-10-16T20:37:37.435040+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:23:03.400384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465006+0000","last_change":"2023-10-16T20:36:55.297429+0000","last_active":"2023-10-16T20:37:37.465006+0000","last_peered":"2023-10-16T20:37:37.465006+0000","last_clean":"2023-10-16T20:37:37.465006+0000","last_became_active":"2023-10-16T20:36:55.297199+0000","last_became_peered":"2023-10-16T20:36:55.297199+0000","last_unstale":"2023-10-16T20:37:37.465006+0000","last_undegraded":"2023-10-16T20:37:37.465006+0000","last_fullsized":"2023-10-16T20:37:37.465006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:43.860251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404029+0000","last_change":"2023-10-16T20:36:55.287421+0000","last_active":"2023-10-16T20:37:37.404029+0000","last_peered":"2023-10-16T20:37:37.404029+0000","last_clean":"2023-10-16T20:37:37.404029+0000","last_became_active":"2023-10-16T20:36:55.286964+0000","last_became_peered":"2023-10-16T20:36:55.286964+0000","last_unstale":"2023-10-16T20:37:37.404029+0000","last_undegraded":"2023-10-16T20:37:37.404029+0000","last_fullsized":"2023-10-16T20:37:37.404029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:15:54.970100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.416921+0000","last_change":"2023-10-16T20:37:37.413891+0000","last_active":"2023-10-16T20:37:37.416921+0000","last_peered":"2023-10-16T20:37:37.416921+0000","last_clean":"2023-10-16T20:37:37.416921+0000","last_became_active":"2023-10-16T20:37:37.413492+0000","last_became_peered":"2023-10-16T20:37:37.413492+0000","last_unstale":"2023-10-16T20:37:37.416921+0000","last_undegraded":"2023-10-16T20:37:37.416921+0000","last_fullsized":"2023-10-16T20:37:37.416921+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:03:46.658556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407997+0000","last_change":"2023-10-16T20:36:55.303950+0000","last_active":"2023-10-16T20:37:37.407997+0000","last_peered":"2023-10-16T20:37:37.407997+0000","last_clean":"2023-10-16T20:37:37.407997+0000","last_became_active":"2023-10-16T20:36:55.302468+0000","last_became_peered":"2023-10-16T20:36:55.302468+0000","last_unstale":"2023-10-16T20:37:37.407997+0000","last_undegraded":"2023-10-16T20:37:37.407997+0000","last_fullsized":"2023-10-16T20:37:37.407997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:02.962031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466830+0000","last_change":"2023-10-16T20:36:55.297558+0000","last_active":"2023-10-16T20:37:37.466830+0000","last_peered":"2023-10-16T20:37:37.466830+0000","last_clean":"2023-10-16T20:37:37.466830+0000","last_became_active":"2023-10-16T20:36:55.296476+0000","last_became_peered":"2023-10-16T20:36:55.296476+0000","last_unstale":"2023-10-16T20:37:37.466830+0000","last_undegraded":"2023-10-16T20:37:37.466830+0000","last_fullsized":"2023-10-16T20:37:37.466830+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:04:38.235205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464724+0000","last_change":"2023-10-16T20:36:55.293957+0000","last_active":"2023-10-16T20:37:37.464724+0000","last_peered":"2023-10-16T20:37:37.464724+0000","last_clean":"2023-10-16T20:37:37.464724+0000","last_became_active":"2023-10-16T20:36:55.293687+0000","last_became_peered":"2023-10-16T20:36:55.293687+0000","last_unstale":"2023-10-16T20:37:37.464724+0000","last_undegraded":"2023-10-16T20:37:37.464724+0000","last_fullsized":"2023-10-16T20:37:37.464724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:54.623691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466003+0000","last_change":"2023-10-16T20:36:55.280728+0000","last_active":"2023-10-16T20:37:37.466003+0000","last_peered":"2023-10-16T20:37:37.466003+0000","last_clean":"2023-10-16T20:37:37.466003+0000","last_became_active":"2023-10-16T20:36:55.280444+0000","last_became_peered":"2023-10-16T20:36:55.280444+0000","last_unstale":"2023-10-16T20:37:37.466003+0000","last_undegraded":"2023-10-16T20:37:37.466003+0000","last_fullsized":"2023-10-16T20:37:37.466003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:44.044563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408158+0000","last_change":"2023-10-16T20:36:55.297802+0000","last_active":"2023-10-16T20:37:37.408158+0000","last_peered":"2023-10-16T20:37:37.408158+0000","last_clean":"2023-10-16T20:37:37.408158+0000","last_became_active":"2023-10-16T20:36:55.296694+0000","last_became_peered":"2023-10-16T20:36:55.296694+0000","last_unstale":"2023-10-16T20:37:37.408158+0000","last_undegraded":"2023-10-16T20:37:37.408158+0000","last_fullsized":"2023-10-16T20:37:37.408158+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:00:26.797387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466169+0000","last_change":"2023-10-16T20:36:55.291347+0000","last_active":"2023-10-16T20:37:37.466169+0000","last_peered":"2023-10-16T20:37:37.466169+0000","last_clean":"2023-10-16T20:37:37.466169+0000","last_became_active":"2023-10-16T20:36:55.291052+0000","last_became_peered":"2023-10-16T20:36:55.291052+0000","last_unstale":"2023-10-16T20:37:37.466169+0000","last_undegraded":"2023-10-16T20:37:37.466169+0000","last_fullsized":"2023-10-16T20:37:37.466169+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:40:51.331446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409006+0000","last_change":"2023-10-16T20:36:55.290870+0000","last_active":"2023-10-16T20:37:37.409006+0000","last_peered":"2023-10-16T20:37:37.409006+0000","last_clean":"2023-10-16T20:37:37.409006+0000","last_became_active":"2023-10-16T20:36:55.290386+0000","last_became_peered":"2023-10-16T20:36:55.290386+0000","last_unstale":"2023-10-16T20:37:37.409006+0000","last_undegraded":"2023-10-16T20:37:37.409006+0000","last_fullsized":"2023-10-16T20:37:37.409006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:20.006864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435199+0000","last_change":"2023-10-16T20:36:55.271041+0000","last_active":"2023-10-16T20:37:37.435199+0000","last_peered":"2023-10-16T20:37:37.435199+0000","last_clean":"2023-10-16T20:37:37.435199+0000","last_became_active":"2023-10-16T20:36:55.270787+0000","last_became_peered":"2023-10-16T20:36:55.270787+0000","last_unstale":"2023-10-16T20:37:37.435199+0000","last_undegraded":"2023-10-16T20:37:37.435199+0000","last_fullsized":"2023-10-16T20:37:37.435199+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:07.944778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402678+0000","last_change":"2023-10-16T20:36:55.271632+0000","last_active":"2023-10-16T20:37:37.402678+0000","last_peered":"2023-10-16T20:37:37.402678+0000","last_clean":"2023-10-16T20:37:37.402678+0000","last_became_active":"2023-10-16T20:36:55.271373+0000","last_became_peered":"2023-10-16T20:36:55.271373+0000","last_unstale":"2023-10-16T20:37:37.402678+0000","last_undegraded":"2023-10-16T20:37:37.402678+0000","last_fullsized":"2023-10-16T20:37:37.402678+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:07.458540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466741+0000","last_change":"2023-10-16T20:36:55.291343+0000","last_active":"2023-10-16T20:37:37.466741+0000","last_peered":"2023-10-16T20:37:37.466741+0000","last_clean":"2023-10-16T20:37:37.466741+0000","last_became_active":"2023-10-16T20:36:55.291016+0000","last_became_peered":"2023-10-16T20:36:55.291016+0000","last_unstale":"2023-10-16T20:37:37.466741+0000","last_undegraded":"2023-10-16T20:37:37.466741+0000","last_fullsized":"2023-10-16T20:37:37.466741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:13:40.287782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":12023,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":24046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":190,"ondisk_log_size":190,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509481,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":937984,"data_stored":717153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.75600000000000001,"15min":0.752},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":5.2240000000000002,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.046,"5min":0.81599999999999995,"15min":0.77300000000000002},"min":{"1min":0.47399999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":8.782,"5min":8.782,"15min":8.782},"last":0.64100000000000001}]},{"osd":1,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.73699999999999999,"15min":0.73799999999999999},"min":{"1min":0.49199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.91800000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.76300000000000001,"15min":0.74399999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":4.7300000000000004,"5min":4.7300000000000004,"15min":4.7300000000000004},"last":0.58999999999999997}]},{"osd":2,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.76900000000000002,"15min":0.73899999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":5.2629999999999999,"5min":5.2629999999999999,"15min":5.2629999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.77100000000000002,"15min":0.753},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.1589999999999998,"5min":5.1589999999999998,"15min":5.1589999999999998},"last":0.66900000000000004}]},{"osd":3,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.79900000000000004,"15min":0.77900000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":4.7460000000000004,"5min":4.7460000000000004,"15min":4.7460000000000004},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.78400000000000003,"15min":0.77000000000000002},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":4.7770000000000001,"5min":4.7770000000000001,"15min":4.7770000000000001},"last":0.80600000000000005}]},{"osd":4,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.61599999999999999,"15min":0.628},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.82099999999999995,"5min":1.3,"15min":1.3},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.58899999999999997,"15min":0.60499999999999998},"min":{"1min":0.375,"5min":0.373,"15min":0.373},"max":{"1min":0.77600000000000002,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.51200000000000001}]},{"osd":5,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68700000000000006,"15min":0.69899999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83799999999999997,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.76600000000000001,"15min":0.78800000000000003},"min":{"1min":0.57899999999999996,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.88600000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.746}]},{"osd":6,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.73499999999999999,"15min":0.75},"min":{"1min":0.51600000000000001,"5min":0.499,"15min":0.499},"max":{"1min":0.877,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.68799999999999994,"15min":0.70099999999999996},"min":{"1min":0.44500000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.85899999999999999,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.83699999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509481,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":129756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.747,"15min":0.76400000000000001},"min":{"1min":0.47399999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.74299999999999999,"15min":0.75600000000000001},"min":{"1min":0.51300000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.73499999999999999,"15min":0.75700000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.85399999999999998,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77000000000000002,"15min":0.77600000000000002},"min":{"1min":0.46300000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":3.387,"5min":3.387,"15min":3.387},"last":0.46500000000000002}]},{"osd":2,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.79000000000000004,"15min":0.79500000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":3.9489999999999998,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.755,"15min":0.751},"min":{"1min":0.53200000000000003,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":3.6179999999999999,"5min":3.6179999999999999,"15min":3.6179999999999999},"last":0.78100000000000003}]},{"osd":3,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.81899999999999995,"15min":0.85899999999999999},"min":{"1min":0.53700000000000003,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0009999999999999,"5min":1.611,"15min":1.611},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.79400000000000004,"15min":0.81399999999999995},"min":{"1min":0.56499999999999995,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.052,"5min":1.163,"15min":1.163},"last":0.75700000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.66000000000000003,"15min":0.67700000000000005},"min":{"1min":0.47399999999999998,"5min":0.443,"15min":0.443},"max":{"1min":0.93899999999999995,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.65600000000000003,"15min":0.66600000000000004},"min":{"1min":0.497,"5min":0.435,"15min":0.435},"max":{"1min":0.877,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.622}]},{"osd":5,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.46500000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.83699999999999997,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.69999999999999996,"15min":0.70599999999999996},"min":{"1min":0.52600000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.91300000000000003,"5min":1.016,"15min":1.016},"last":0.59899999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.73899999999999999,"15min":0.751},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.92000000000000004,"5min":1.292,"15min":1.292},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.748,"15min":0.77100000000000002},"min":{"1min":0.51900000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.90600000000000003,"5min":1.569,"15min":1.569},"last":0.69199999999999995}]}]},{"osd":5,"up_from":15,"seq":64424509483,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":356352,"data_stored":128617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70199999999999996,"15min":0.69599999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.88800000000000001,"5min":1.006,"15min":1.006},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.56899999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.93899999999999995,"5min":1.083,"15min":1.083},"last":0.64500000000000002}]},{"osd":1,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.71099999999999997,"15min":0.70599999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.96099999999999997,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71799999999999997,"15min":0.70999999999999996},"min":{"1min":0.53300000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.97799999999999998,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.73299999999999998}]},{"osd":2,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.90300000000000002,"5min":1.022,"15min":1.022},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.74199999999999999,"15min":0.73699999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.80300000000000005}]},{"osd":3,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78600000000000003,"15min":0.78900000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.97499999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.79500000000000004,"15min":0.81100000000000005},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.88800000000000001,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.66300000000000003}]},{"osd":4,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64400000000000002,"15min":0.65700000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.71099999999999997,"15min":0.751},"min":{"1min":0.496,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84099999999999997,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.77400000000000002}]},{"osd":6,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.69399999999999995,"15min":0.72299999999999998},"min":{"1min":0.46899999999999997,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.95099999999999996,"5min":1.069,"15min":1.069},"last":0.622},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.70999999999999996,"15min":0.72299999999999998},"min":{"1min":0.49199999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.84599999999999997,"5min":1.264,"15min":1.264},"last":0.83899999999999997}]},{"osd":7,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76200000000000001,"15min":0.79000000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.026,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.77200000000000002,"15min":0.80000000000000004},"min":{"1min":0.50900000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.96499999999999997,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.85599999999999998}]}]},{"osd":4,"up_from":15,"seq":64424509481,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":331776,"data_stored":126015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.72199999999999998,"15min":0.72399999999999998},"min":{"1min":0.45200000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.872,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.59599999999999997,"5min":0.5,"15min":0.5},"max":{"1min":0.88200000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.82099999999999995}]},{"osd":1,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.48999999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.74199999999999999,"15min":0.746},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.59499999999999997}]},{"osd":2,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.752,"15min":0.75800000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.66200000000000003}]},{"osd":3,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.75800000000000001,"15min":0.74199999999999999},"min":{"1min":0.54700000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.77000000000000002,"15min":0.78200000000000003},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":5,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.627,"15min":0.64500000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.82199999999999995,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.64700000000000002,"15min":0.65100000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.91700000000000004,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.73299999999999998}]},{"osd":6,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.71899999999999997,"15min":0.73299999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.90600000000000003,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.69099999999999995}]},{"osd":7,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.72799999999999998,"15min":0.745},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.94099999999999995,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.621},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.77800000000000002,"15min":0.79800000000000004},"min":{"1min":0.52800000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1080000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.85699999999999998}]}]},{"osd":3,"up_from":15,"seq":64424509481,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":129790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.503,"15min":0.48299999999999998},"min":{"1min":0.30099999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.3479999999999999,"5min":4.3479999999999999,"15min":4.3479999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.53700000000000003,"15min":0.51900000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":4.29,"5min":4.29,"15min":4.29},"last":0.40300000000000002}]},{"osd":1,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.52200000000000002,"15min":0.48799999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":4.5800000000000001,"5min":4.5800000000000001,"15min":4.5800000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.59699999999999998,"15min":0.56799999999999995},"min":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":4.7190000000000003,"5min":4.7190000000000003,"15min":4.7190000000000003},"last":0.48199999999999998}]},{"osd":2,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.56299999999999994,"15min":0.55200000000000005},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.3899999999999997,"5min":4.3899999999999997,"15min":4.3899999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.65900000000000003,"15min":0.628},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":0.44900000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.82799999999999996,"15min":0.80800000000000005},"min":{"1min":0.56799999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":4.0629999999999997,"5min":4.0629999999999997,"15min":4.0629999999999997},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77700000000000002,"15min":0.76600000000000001},"min":{"1min":0.56399999999999995,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.026,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.752}]},{"osd":5,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.86799999999999999,"15min":0.876},"min":{"1min":0.57699999999999996,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":3.9529999999999998,"5min":3.9529999999999998,"15min":3.9529999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.85399999999999998,"15min":0.83899999999999997},"min":{"1min":0.60599999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":4.4960000000000004,"5min":4.4960000000000004,"15min":4.4960000000000004},"last":0.76900000000000002}]},{"osd":6,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.76000000000000001,"15min":0.76800000000000002},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.86699999999999999,"15min":0.86199999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.81899999999999995}]},{"osd":7,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.83299999999999996,"15min":0.80900000000000005},"min":{"1min":0.56699999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":4.0049999999999999,"5min":4.0049999999999999,"15min":4.0049999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.749,"5min":0.79000000000000004,"15min":0.79800000000000004},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":0.93799999999999994,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.64600000000000002}]}]},{"osd":2,"up_from":15,"seq":64424509481,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":126147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.46700000000000003,"15min":0.47499999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70399999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.51900000000000002,"15min":0.52500000000000002},"min":{"1min":0.35899999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.68700000000000006,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.29699999999999999}]},{"osd":1,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.51500000000000001,"15min":0.52600000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.74299999999999999,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.36499999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.64700000000000002,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.47799999999999998}]},{"osd":3,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.80400000000000005,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57799999999999996,"15min":0.59299999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.74399999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.40400000000000003}]},{"osd":4,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.80900000000000005,"15min":0.82399999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.02,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.626},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.75900000000000001,"15min":0.76100000000000001},"min":{"1min":0.56200000000000006,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.92600000000000005,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.51600000000000001}]},{"osd":5,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.77200000000000002,"15min":0.78000000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.04,"5min":1.444,"15min":1.444},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.81799999999999995,"15min":0.83499999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.052,"5min":1.585,"15min":1.585},"last":0.52700000000000002}]},{"osd":6,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.82399999999999995,"15min":0.84399999999999997},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":0.96099999999999997,"5min":1.335,"15min":1.335},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.84099999999999997,"15min":0.872},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.036,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.63900000000000001}]},{"osd":7,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.83299999999999996,"15min":0.86399999999999999},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.93500000000000005,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.86099999999999999,"15min":0.88500000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.0820000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.59299999999999997}]}]},{"osd":1,"up_from":15,"seq":64424509481,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":125777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.45100000000000001,"15min":0.46300000000000002},"min":{"1min":0.29199999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69199999999999995,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.51500000000000001,"15min":0.52800000000000002},"min":{"1min":0.318,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.97899999999999998,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.52400000000000002,"15min":0.54400000000000004},"min":{"1min":0.32600000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66900000000000004,"5min":1.046,"15min":1.046},"last":0.377},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.55800000000000005,"15min":0.58199999999999996},"min":{"1min":0.33700000000000002,"5min":0.314,"15min":0.314},"max":{"1min":1.0740000000000001,"5min":1.115,"15min":1.115},"last":0.33700000000000002}]},{"osd":3,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.40600000000000003,"5min":0.312,"15min":0.312},"max":{"1min":1.155,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72699999999999998,"5min":1.431,"15min":1.431},"last":0.497}]},{"osd":4,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.81899999999999995,"15min":0.83099999999999996},"min":{"1min":0.56100000000000005,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.80700000000000005,"15min":0.80900000000000005},"min":{"1min":0.61199999999999999,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.115,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.628}]},{"osd":5,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.78900000000000003,"15min":0.81299999999999994},"min":{"1min":0.58399999999999996,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90200000000000002,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.84699999999999998,"15min":0.86399999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2490000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.59799999999999998}]},{"osd":6,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.84999999999999998,"15min":0.873},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.016,"5min":2.004,"15min":2.004},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.85399999999999998,"15min":0.876},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.3300000000000001,"5min":1.635,"15min":1.635},"last":0.67100000000000004}]},{"osd":7,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.85299999999999998,"15min":0.88300000000000001},"min":{"1min":0.58999999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0249999999999999,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.878,"15min":0.90300000000000002},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0489999999999999,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.70399999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343916,"statfs":{"total":96636764160,"available":96608169984,"internally_reserved":0,"allocated":929792,"data_stored":715487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.439,"15min":0.438},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.495,"15min":0.496},"min":{"1min":0.374,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.58999999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.36499999999999999}]},{"osd":2,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52300000000000002,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.433},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54100000000000004,"15min":0.54600000000000004},"min":{"1min":0.40400000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.68899999999999995,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.47499999999999998}]},{"osd":3,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54800000000000004,"15min":0.54500000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73399999999999999,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.56499999999999995,"5min":0.497,"15min":0.497},"max":{"1min":0.84999999999999998,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.49199999999999999}]},{"osd":4,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.83799999999999997,"15min":0.82999999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0600000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.85999999999999999,"15min":0.84899999999999998},"min":{"1min":0.61399999999999999,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.80800000000000005}]},{"osd":5,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.80800000000000005,"15min":0.80300000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.66800000000000004}]},{"osd":6,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.86299999999999999,"15min":0.85899999999999999},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.755},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88400000000000001,"15min":0.88300000000000001},"min":{"1min":0.60899999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.79400000000000004}]},{"osd":7,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.64900000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.83499999999999996,"15min":0.82299999999999995},"min":{"1min":0.66500000000000004,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T20:39:53.849 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T20:39:54.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.012+0000 7fd31ff9d640 1 Processor -- start 2023-10-16T20:39:54.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.016+0000 7fd31ff9d640 1 -- start start 2023-10-16T20:39:54.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.016+0000 7fd31ff9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd318150280 0x7fd3180c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.016+0000 7fd31ff9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd3180c6bc0 0x7fd3180c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.016+0000 7fd31ff9d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3180c5c60 con 0x7fd318150660 2023-10-16T20:39:54.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.016+0000 7fd31ff9d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3180c5370 con 0x7fd3180c6bc0 2023-10-16T20:39:54.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.016+0000 7fd31ff9d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd318156d10 con 0x7fd318150280 2023-10-16T20:39:54.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.016+0000 7fd31d538640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd318150660 0x7fd31806d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58442/0 (socket says 172.21.15.88:58442) 2023-10-16T20:39:54.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.016+0000 7fd31d538640 1 -- 172.21.15.88:0/1117969192 learned_addr learned my addr 172.21.15.88:0/1117969192 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:54.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.016+0000 7fd31e53a640 1 --2- 172.21.15.88:0/1117969192 >> v2:172.21.15.88:3300/0 conn(0x7fd3180c6bc0 0x7fd3180c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:54.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.016+0000 7fd31dd39640 1 --2- 172.21.15.88:0/1117969192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd318150280 0x7fd3180c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:54.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.016+0000 7fd31cd37640 1 -- 172.21.15.88:0/1117969192 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527228631 0 0) 0x7fd3180c5c60 con 0x7fd318150660 2023-10-16T20:39:54.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.016+0000 7fd31cd37640 1 -- 172.21.15.88:0/1117969192 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd304003530 con 0x7fd318150660 2023-10-16T20:39:54.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.016+0000 7fd31e53a640 1 -- 172.21.15.88:0/1117969192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd318150280 msgr2=0x7fd3180c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:54.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.016+0000 7fd31e53a640 1 --2- 172.21.15.88:0/1117969192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd318150280 0x7fd3180c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.016+0000 7fd31e53a640 1 -- 172.21.15.88:0/1117969192 >> v1:172.21.15.88:6789/0 conn(0x7fd318150660 legacy=0x7fd31806d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.016+0000 7fd31e53a640 1 -- 172.21.15.88:0/1117969192 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3180c5610 con 0x7fd3180c6bc0 2023-10-16T20:39:54.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.020+0000 7fd31e53a640 1 --2- 172.21.15.88:0/1117969192 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3180c6bc0 0x7fd3180c3110 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd310009d90 tx=0x7fd31001d2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c058d1630cf1377 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:54.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.020+0000 7fd31cd37640 1 -- 172.21.15.88:0/1117969192 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3100a5970 con 0x7fd3180c6bc0 2023-10-16T20:39:54.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.020+0000 7fd31cd37640 1 -- 172.21.15.88:0/1117969192 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3100a5ad0 con 0x7fd3180c6bc0 2023-10-16T20:39:54.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.020+0000 7fd31cd37640 1 -- 172.21.15.88:0/1117969192 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3100a5e60 con 0x7fd3180c6bc0 2023-10-16T20:39:54.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.020+0000 7fd31ff9d640 1 -- 172.21.15.88:0/1117969192 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3180c6bc0 msgr2=0x7fd3180c3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.020+0000 7fd31ff9d640 1 --2- 172.21.15.88:0/1117969192 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3180c6bc0 0x7fd3180c3110 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd310009d90 tx=0x7fd31001d2f0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.020+0000 7fd31ff9d640 1 -- 172.21.15.88:0/1117969192 shutdown_connections 2023-10-16T20:39:54.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.020+0000 7fd31ff9d640 1 --2- 172.21.15.88:0/1117969192 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3180c6bc0 0x7fd3180c3110 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.020+0000 7fd31ff9d640 1 --2- 172.21.15.88:0/1117969192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd318150280 0x7fd3180c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.020+0000 7fd31ff9d640 1 -- 172.21.15.88:0/1117969192 >> 172.21.15.88:0/1117969192 conn(0x7fd3180cad50 msgr2=0x7fd3180cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:54.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.020+0000 7fd31ff9d640 1 -- 172.21.15.88:0/1117969192 shutdown_connections 2023-10-16T20:39:54.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.020+0000 7fd31ff9d640 1 -- 172.21.15.88:0/1117969192 wait complete. 2023-10-16T20:39:54.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.020+0000 7fd31ff9d640 1 Processor -- start 2023-10-16T20:39:54.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31ff9d640 1 -- start start 2023-10-16T20:39:54.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31ff9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3180c6bc0 0x7fd3180c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31ff9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd318150280 0x7fd318152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31ff9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd318150660 0x7fd318153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31ff9d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd318150df0 con 0x7fd318150660 2023-10-16T20:39:54.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31ff9d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3181f27f0 con 0x7fd3180c6bc0 2023-10-16T20:39:54.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31ff9d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3181f5e80 con 0x7fd318150280 2023-10-16T20:39:54.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31e53a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd318150660 0x7fd318153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:54.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31e53a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd318150660 0x7fd318153120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47760/0 (socket says 172.21.15.88:47760) 2023-10-16T20:39:54.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31e53a640 1 -- 172.21.15.88:0/4180735028 learned_addr learned my addr 172.21.15.88:0/4180735028 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:54.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31e53a640 1 -- 172.21.15.88:0/4180735028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd318150280 msgr2=0x7fd318152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31e53a640 1 --2- 172.21.15.88:0/4180735028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd318150280 0x7fd318152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31e53a640 1 -- 172.21.15.88:0/4180735028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3180c6bc0 msgr2=0x7fd3180c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31e53a640 1 --2- 172.21.15.88:0/4180735028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3180c6bc0 0x7fd3180c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31e53a640 1 -- 172.21.15.88:0/4180735028 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd310009a70 con 0x7fd318150660 2023-10-16T20:39:54.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31e53a640 1 --2- 172.21.15.88:0/4180735028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd318150660 0x7fd318153120 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd3100b0040 tx=0x7fd310019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:54.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd30e7fc640 1 -- 172.21.15.88:0/4180735028 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3100add70 con 0x7fd318150660 2023-10-16T20:39:54.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd30e7fc640 1 -- 172.21.15.88:0/4180735028 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd310009450 con 0x7fd318150660 2023-10-16T20:39:54.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd30e7fc640 1 -- 172.21.15.88:0/4180735028 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3100b8640 con 0x7fd318150660 2023-10-16T20:39:54.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31ff9d640 1 -- 172.21.15.88:0/4180735028 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3181f1bb0 con 0x7fd318150660 2023-10-16T20:39:54.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31ff9d640 1 -- 172.21.15.88:0/4180735028 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3181f2130 con 0x7fd318150660 2023-10-16T20:39:54.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.024+0000 7fd31ff9d640 1 -- 172.21.15.88:0/4180735028 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd318074700 con 0x7fd318150660 2023-10-16T20:39:54.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.028+0000 7fd30e7fc640 1 -- 172.21.15.88:0/4180735028 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd3100b4040 con 0x7fd318150660 2023-10-16T20:39:54.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.028+0000 7fd30e7fc640 1 --2- 172.21.15.88:0/4180735028 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2fc0664a0 0x7fd2fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.028+0000 7fd30e7fc640 1 -- 172.21.15.88:0/4180735028 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd31011f5d0 con 0x7fd318150660 2023-10-16T20:39:54.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.028+0000 7fd31dd39640 1 --2- 172.21.15.88:0/4180735028 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2fc0664a0 0x7fd2fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:54.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.028+0000 7fd31dd39640 1 --2- 172.21.15.88:0/4180735028 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2fc0664a0 0x7fd2fc068960 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd3140098f0 tx=0x7fd314008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:54.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.032+0000 7fd30e7fc640 1 -- 172.21.15.88:0/4180735028 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd3100ec870 con 0x7fd318150660 2023-10-16T20:39:54.048 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:39:54.052 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:39:54.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.180+0000 7fd31ff9d640 1 -- 172.21.15.88:0/4180735028 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd3181629e0 con 0x7fd2fc0664a0 2023-10-16T20:39:54.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.188+0000 7fd30e7fc640 1 -- 172.21.15.88:0/4180735028 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+369924 (secure 0 0 0) 0x7fd314009e70 con 0x7fd2fc0664a0 2023-10-16T20:39:54.196 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:54.198 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-16T20:39:54.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.192+0000 7fd31ff9d640 1 -- 172.21.15.88:0/4180735028 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2fc0664a0 msgr2=0x7fd2fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.192+0000 7fd31ff9d640 1 --2- 172.21.15.88:0/4180735028 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2fc0664a0 0x7fd2fc068960 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd3140098f0 tx=0x7fd314008040 comp rx=0 tx=0).stop 2023-10-16T20:39:54.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.192+0000 7fd31ff9d640 1 -- 172.21.15.88:0/4180735028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd318150660 msgr2=0x7fd318153120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.192+0000 7fd31ff9d640 1 --2- 172.21.15.88:0/4180735028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd318150660 0x7fd318153120 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd3100b0040 tx=0x7fd310019630 comp rx=0 tx=0).stop 2023-10-16T20:39:54.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.192+0000 7fd31ff9d640 1 -- 172.21.15.88:0/4180735028 shutdown_connections 2023-10-16T20:39:54.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.192+0000 7fd31ff9d640 1 --2- 172.21.15.88:0/4180735028 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2fc0664a0 0x7fd2fc068960 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.192+0000 7fd31ff9d640 1 --2- 172.21.15.88:0/4180735028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd318150660 0x7fd318153120 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.192+0000 7fd31ff9d640 1 --2- 172.21.15.88:0/4180735028 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd318150280 0x7fd318152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.192+0000 7fd31ff9d640 1 --2- 172.21.15.88:0/4180735028 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3180c6bc0 0x7fd3180c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.192+0000 7fd31ff9d640 1 -- 172.21.15.88:0/4180735028 >> 172.21.15.88:0/4180735028 conn(0x7fd3180cad50 msgr2=0x7fd3180c4510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:54.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.192+0000 7fd31ff9d640 1 -- 172.21.15.88:0/4180735028 shutdown_connections 2023-10-16T20:39:54.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.192+0000 7fd31ff9d640 1 -- 172.21.15.88:0/4180735028 wait complete. 2023-10-16T20:39:54.215 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2023-10-16T20:39:53.635556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":602391,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":375,"num_write_kb":2490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307,"ondisk_log_size":307,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219944,"kb_used_data":3816,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754776,"statfs":{"total":773094113280,"available":772868890624,"internally_reserved":0,"allocated":3907584,"data_stored":2198742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3076,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002564"},"pg_stats":[{"pgid":"2.1f","version":"29'13","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.526948+0000","last_change":"2023-10-16T20:36:54.763744+0000","last_active":"2023-10-16T20:39:39.526948+0000","last_peered":"2023-10-16T20:39:39.526948+0000","last_clean":"2023-10-16T20:39:39.526948+0000","last_became_active":"2023-10-16T20:36:54.763457+0000","last_became_peered":"2023-10-16T20:36:54.763457+0000","last_unstale":"2023-10-16T20:39:39.526948+0000","last_undegraded":"2023-10-16T20:39:39.526948+0000","last_fullsized":"2023-10-16T20:39:39.526948+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:05.774485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465462+0000","last_change":"2023-10-16T20:36:55.302740+0000","last_active":"2023-10-16T20:37:37.465462+0000","last_peered":"2023-10-16T20:37:37.465462+0000","last_clean":"2023-10-16T20:37:37.465462+0000","last_became_active":"2023-10-16T20:36:55.302519+0000","last_became_peered":"2023-10-16T20:36:55.302519+0000","last_unstale":"2023-10-16T20:37:37.465462+0000","last_undegraded":"2023-10-16T20:37:37.465462+0000","last_fullsized":"2023-10-16T20:37:37.465462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:36.762521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464759+0000","last_change":"2023-10-16T20:36:59.466679+0000","last_active":"2023-10-16T20:37:37.464759+0000","last_peered":"2023-10-16T20:37:37.464759+0000","last_clean":"2023-10-16T20:37:37.464759+0000","last_became_active":"2023-10-16T20:36:59.466306+0000","last_became_peered":"2023-10-16T20:36:59.466306+0000","last_unstale":"2023-10-16T20:37:37.464759+0000","last_undegraded":"2023-10-16T20:37:37.464759+0000","last_fullsized":"2023-10-16T20:37:37.464759+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:14.874622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:38:57.624590+0000","last_change":"2023-10-16T20:36:54.759491+0000","last_active":"2023-10-16T20:38:57.624590+0000","last_peered":"2023-10-16T20:38:57.624590+0000","last_clean":"2023-10-16T20:38:57.624590+0000","last_became_active":"2023-10-16T20:36:54.759177+0000","last_became_peered":"2023-10-16T20:36:54.759177+0000","last_unstale":"2023-10-16T20:38:57.624590+0000","last_undegraded":"2023-10-16T20:38:57.624590+0000","last_fullsized":"2023-10-16T20:38:57.624590+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:43:36.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435464+0000","last_change":"2023-10-16T20:36:55.291745+0000","last_active":"2023-10-16T20:37:37.435464+0000","last_peered":"2023-10-16T20:37:37.435464+0000","last_clean":"2023-10-16T20:37:37.435464+0000","last_became_active":"2023-10-16T20:36:55.291458+0000","last_became_peered":"2023-10-16T20:36:55.291458+0000","last_unstale":"2023-10-16T20:37:37.435464+0000","last_undegraded":"2023-10-16T20:37:37.435464+0000","last_fullsized":"2023-10-16T20:37:37.435464+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:01.291335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402322+0000","last_change":"2023-10-16T20:36:59.303600+0000","last_active":"2023-10-16T20:37:37.402322+0000","last_peered":"2023-10-16T20:37:37.402322+0000","last_clean":"2023-10-16T20:37:37.402322+0000","last_became_active":"2023-10-16T20:36:59.302968+0000","last_became_peered":"2023-10-16T20:36:59.302968+0000","last_unstale":"2023-10-16T20:37:37.402322+0000","last_undegraded":"2023-10-16T20:37:37.402322+0000","last_fullsized":"2023-10-16T20:37:37.402322+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:11:28.322824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"29'12","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.529183+0000","last_change":"2023-10-16T20:36:54.762600+0000","last_active":"2023-10-16T20:39:39.529183+0000","last_peered":"2023-10-16T20:39:39.529183+0000","last_clean":"2023-10-16T20:39:39.529183+0000","last_became_active":"2023-10-16T20:36:54.762278+0000","last_became_peered":"2023-10-16T20:36:54.762278+0000","last_unstale":"2023-10-16T20:39:39.529183+0000","last_undegraded":"2023-10-16T20:39:39.529183+0000","last_fullsized":"2023-10-16T20:39:39.529183+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:46:03.836415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404637+0000","last_change":"2023-10-16T20:37:37.403448+0000","last_active":"2023-10-16T20:37:37.404637+0000","last_peered":"2023-10-16T20:37:37.404637+0000","last_clean":"2023-10-16T20:37:37.404637+0000","last_became_active":"2023-10-16T20:37:37.402987+0000","last_became_peered":"2023-10-16T20:37:37.402987+0000","last_unstale":"2023-10-16T20:37:37.404637+0000","last_undegraded":"2023-10-16T20:37:37.404637+0000","last_fullsized":"2023-10-16T20:37:37.404637+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:39:05.729682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402868+0000","last_change":"2023-10-16T20:36:59.304684+0000","last_active":"2023-10-16T20:37:37.402868+0000","last_peered":"2023-10-16T20:37:37.402868+0000","last_clean":"2023-10-16T20:37:37.402868+0000","last_became_active":"2023-10-16T20:36:59.303805+0000","last_became_peered":"2023-10-16T20:36:59.303805+0000","last_unstale":"2023-10-16T20:37:37.402868+0000","last_undegraded":"2023-10-16T20:37:37.402868+0000","last_fullsized":"2023-10-16T20:37:37.402868+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:39:54.406763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.526474+0000","last_change":"2023-10-16T20:36:54.749849+0000","last_active":"2023-10-16T20:39:39.526474+0000","last_peered":"2023-10-16T20:39:39.526474+0000","last_clean":"2023-10-16T20:39:39.526474+0000","last_became_active":"2023-10-16T20:36:54.749316+0000","last_became_peered":"2023-10-16T20:36:54.749316+0000","last_unstale":"2023-10-16T20:39:39.526474+0000","last_undegraded":"2023-10-16T20:39:39.526474+0000","last_fullsized":"2023-10-16T20:39:39.526474+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:49:29.077126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409527+0000","last_change":"2023-10-16T20:36:55.302838+0000","last_active":"2023-10-16T20:37:37.409527+0000","last_peered":"2023-10-16T20:37:37.409527+0000","last_clean":"2023-10-16T20:37:37.409527+0000","last_became_active":"2023-10-16T20:36:55.302387+0000","last_became_peered":"2023-10-16T20:36:55.302387+0000","last_unstale":"2023-10-16T20:37:37.409527+0000","last_undegraded":"2023-10-16T20:37:37.409527+0000","last_fullsized":"2023-10-16T20:37:37.409527+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:55:56.517950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403226+0000","last_change":"2023-10-16T20:36:59.305750+0000","last_active":"2023-10-16T20:37:37.403226+0000","last_peered":"2023-10-16T20:37:37.403226+0000","last_clean":"2023-10-16T20:37:37.403226+0000","last_became_active":"2023-10-16T20:36:59.305228+0000","last_became_peered":"2023-10-16T20:36:59.305228+0000","last_unstale":"2023-10-16T20:37:37.403226+0000","last_undegraded":"2023-10-16T20:37:37.403226+0000","last_fullsized":"2023-10-16T20:37:37.403226+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:00:18.194057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407634+0000","last_change":"2023-10-16T20:36:55.306712+0000","last_active":"2023-10-16T20:37:37.407634+0000","last_peered":"2023-10-16T20:37:37.407634+0000","last_clean":"2023-10-16T20:37:37.407634+0000","last_became_active":"2023-10-16T20:36:55.306427+0000","last_became_peered":"2023-10-16T20:36:55.306427+0000","last_unstale":"2023-10-16T20:37:37.407634+0000","last_undegraded":"2023-10-16T20:37:37.407634+0000","last_fullsized":"2023-10-16T20:37:37.407634+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:42:03.769890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.410476+0000","last_change":"2023-10-16T20:36:54.768375+0000","last_active":"2023-10-16T20:39:22.410476+0000","last_peered":"2023-10-16T20:39:22.410476+0000","last_clean":"2023-10-16T20:39:22.410476+0000","last_became_active":"2023-10-16T20:36:54.767996+0000","last_became_peered":"2023-10-16T20:36:54.767996+0000","last_unstale":"2023-10-16T20:39:22.410476+0000","last_undegraded":"2023-10-16T20:39:22.410476+0000","last_fullsized":"2023-10-16T20:39:22.410476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:55.303942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404253+0000","last_change":"2023-10-16T20:36:59.307130+0000","last_active":"2023-10-16T20:37:37.404253+0000","last_peered":"2023-10-16T20:37:37.404253+0000","last_clean":"2023-10-16T20:37:37.404253+0000","last_became_active":"2023-10-16T20:36:59.306640+0000","last_became_peered":"2023-10-16T20:36:59.306640+0000","last_unstale":"2023-10-16T20:37:37.404253+0000","last_undegraded":"2023-10-16T20:37:37.404253+0000","last_fullsized":"2023-10-16T20:37:37.404253+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:19:18.664630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465074+0000","last_change":"2023-10-16T20:36:55.275436+0000","last_active":"2023-10-16T20:37:37.465074+0000","last_peered":"2023-10-16T20:37:37.465074+0000","last_clean":"2023-10-16T20:37:37.465074+0000","last_became_active":"2023-10-16T20:36:55.275085+0000","last_became_peered":"2023-10-16T20:36:55.275085+0000","last_unstale":"2023-10-16T20:37:37.465074+0000","last_undegraded":"2023-10-16T20:37:37.465074+0000","last_fullsized":"2023-10-16T20:37:37.465074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:37:37.814511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.502271+0000","last_change":"2023-10-16T20:36:54.756174+0000","last_active":"2023-10-16T20:39:33.502271+0000","last_peered":"2023-10-16T20:39:33.502271+0000","last_clean":"2023-10-16T20:39:33.502271+0000","last_became_active":"2023-10-16T20:36:54.755872+0000","last_became_peered":"2023-10-16T20:36:54.755872+0000","last_unstale":"2023-10-16T20:39:33.502271+0000","last_undegraded":"2023-10-16T20:39:33.502271+0000","last_fullsized":"2023-10-16T20:39:33.502271+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:32.741671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403750+0000","last_change":"2023-10-16T20:36:59.303855+0000","last_active":"2023-10-16T20:37:37.403750+0000","last_peered":"2023-10-16T20:37:37.403750+0000","last_clean":"2023-10-16T20:37:37.403750+0000","last_became_active":"2023-10-16T20:36:59.302919+0000","last_became_peered":"2023-10-16T20:36:59.302919+0000","last_unstale":"2023-10-16T20:37:37.403750+0000","last_undegraded":"2023-10-16T20:37:37.403750+0000","last_fullsized":"2023-10-16T20:37:37.403750+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:38:02.591237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464820+0000","last_change":"2023-10-16T20:36:55.270717+0000","last_active":"2023-10-16T20:37:37.464820+0000","last_peered":"2023-10-16T20:37:37.464820+0000","last_clean":"2023-10-16T20:37:37.464820+0000","last_became_active":"2023-10-16T20:36:55.270455+0000","last_became_peered":"2023-10-16T20:36:55.270455+0000","last_unstale":"2023-10-16T20:37:37.464820+0000","last_undegraded":"2023-10-16T20:37:37.464820+0000","last_fullsized":"2023-10-16T20:37:37.464820+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:16:44.485783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'7","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.524872+0000","last_change":"2023-10-16T20:36:54.744128+0000","last_active":"2023-10-16T20:39:39.524872+0000","last_peered":"2023-10-16T20:39:39.524872+0000","last_clean":"2023-10-16T20:39:39.524872+0000","last_became_active":"2023-10-16T20:36:54.743805+0000","last_became_peered":"2023-10-16T20:36:54.743805+0000","last_unstale":"2023-10-16T20:39:39.524872+0000","last_undegraded":"2023-10-16T20:39:39.524872+0000","last_fullsized":"2023-10-16T20:39:39.524872+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:54:35.819820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402668+0000","last_change":"2023-10-16T20:36:59.465650+0000","last_active":"2023-10-16T20:37:37.402668+0000","last_peered":"2023-10-16T20:37:37.402668+0000","last_clean":"2023-10-16T20:37:37.402668+0000","last_became_active":"2023-10-16T20:36:59.465172+0000","last_became_peered":"2023-10-16T20:36:59.465172+0000","last_unstale":"2023-10-16T20:37:37.402668+0000","last_undegraded":"2023-10-16T20:37:37.402668+0000","last_fullsized":"2023-10-16T20:37:37.402668+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:12.845517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403561+0000","last_change":"2023-10-16T20:36:55.278974+0000","last_active":"2023-10-16T20:37:37.403561+0000","last_peered":"2023-10-16T20:37:37.403561+0000","last_clean":"2023-10-16T20:37:37.403561+0000","last_became_active":"2023-10-16T20:36:55.278568+0000","last_became_peered":"2023-10-16T20:36:55.278568+0000","last_unstale":"2023-10-16T20:37:37.403561+0000","last_undegraded":"2023-10-16T20:37:37.403561+0000","last_fullsized":"2023-10-16T20:37:37.403561+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:26.756114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.407659+0000","last_change":"2023-10-16T20:36:54.758778+0000","last_active":"2023-10-16T20:39:22.407659+0000","last_peered":"2023-10-16T20:39:22.407659+0000","last_clean":"2023-10-16T20:39:22.407659+0000","last_became_active":"2023-10-16T20:36:54.758496+0000","last_became_peered":"2023-10-16T20:36:54.758496+0000","last_unstale":"2023-10-16T20:39:22.407659+0000","last_undegraded":"2023-10-16T20:39:22.407659+0000","last_fullsized":"2023-10-16T20:39:22.407659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:36:00.515106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404277+0000","last_change":"2023-10-16T20:36:59.309382+0000","last_active":"2023-10-16T20:37:37.404277+0000","last_peered":"2023-10-16T20:37:37.404277+0000","last_clean":"2023-10-16T20:37:37.404277+0000","last_became_active":"2023-10-16T20:36:59.309068+0000","last_became_peered":"2023-10-16T20:36:59.309068+0000","last_unstale":"2023-10-16T20:37:37.404277+0000","last_undegraded":"2023-10-16T20:37:37.404277+0000","last_fullsized":"2023-10-16T20:37:37.404277+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:46:02.601985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435429+0000","last_change":"2023-10-16T20:36:55.273089+0000","last_active":"2023-10-16T20:37:37.435429+0000","last_peered":"2023-10-16T20:37:37.435429+0000","last_clean":"2023-10-16T20:37:37.435429+0000","last_became_active":"2023-10-16T20:36:55.272840+0000","last_became_peered":"2023-10-16T20:36:55.272840+0000","last_unstale":"2023-10-16T20:37:37.435429+0000","last_undegraded":"2023-10-16T20:37:37.435429+0000","last_fullsized":"2023-10-16T20:37:37.435429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:37.330924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.530212+0000","last_change":"2023-10-16T20:36:54.760681+0000","last_active":"2023-10-16T20:39:39.530212+0000","last_peered":"2023-10-16T20:39:39.530212+0000","last_clean":"2023-10-16T20:39:39.530212+0000","last_became_active":"2023-10-16T20:36:54.760397+0000","last_became_peered":"2023-10-16T20:36:54.760397+0000","last_unstale":"2023-10-16T20:39:39.530212+0000","last_undegraded":"2023-10-16T20:39:39.530212+0000","last_fullsized":"2023-10-16T20:39:39.530212+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:47:16.510382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403318+0000","last_change":"2023-10-16T20:36:59.465626+0000","last_active":"2023-10-16T20:37:37.403318+0000","last_peered":"2023-10-16T20:37:37.403318+0000","last_clean":"2023-10-16T20:37:37.403318+0000","last_became_active":"2023-10-16T20:36:59.465152+0000","last_became_peered":"2023-10-16T20:36:59.465152+0000","last_unstale":"2023-10-16T20:37:37.403318+0000","last_undegraded":"2023-10-16T20:37:37.403318+0000","last_fullsized":"2023-10-16T20:37:37.403318+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:27:51.088333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465673+0000","last_change":"2023-10-16T20:36:55.285796+0000","last_active":"2023-10-16T20:37:37.465673+0000","last_peered":"2023-10-16T20:37:37.465673+0000","last_clean":"2023-10-16T20:37:37.465673+0000","last_became_active":"2023-10-16T20:36:55.285566+0000","last_became_peered":"2023-10-16T20:36:55.285566+0000","last_unstale":"2023-10-16T20:37:37.465673+0000","last_undegraded":"2023-10-16T20:37:37.465673+0000","last_fullsized":"2023-10-16T20:37:37.465673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:22:21.950705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.503684+0000","last_change":"2023-10-16T20:36:54.758062+0000","last_active":"2023-10-16T20:39:33.503684+0000","last_peered":"2023-10-16T20:39:33.503684+0000","last_clean":"2023-10-16T20:39:33.503684+0000","last_became_active":"2023-10-16T20:36:54.757796+0000","last_became_peered":"2023-10-16T20:36:54.757796+0000","last_unstale":"2023-10-16T20:39:33.503684+0000","last_undegraded":"2023-10-16T20:39:33.503684+0000","last_fullsized":"2023-10-16T20:39:33.503684+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:02:09.953795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464463+0000","last_change":"2023-10-16T20:36:59.468529+0000","last_active":"2023-10-16T20:37:37.464463+0000","last_peered":"2023-10-16T20:37:37.464463+0000","last_clean":"2023-10-16T20:37:37.464463+0000","last_became_active":"2023-10-16T20:36:59.465221+0000","last_became_peered":"2023-10-16T20:36:59.465221+0000","last_unstale":"2023-10-16T20:37:37.464463+0000","last_undegraded":"2023-10-16T20:37:37.464463+0000","last_fullsized":"2023-10-16T20:37:37.464463+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:58:08.457686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401257+0000","last_change":"2023-10-16T20:36:55.275759+0000","last_active":"2023-10-16T20:37:37.401257+0000","last_peered":"2023-10-16T20:37:37.401257+0000","last_clean":"2023-10-16T20:37:37.401257+0000","last_became_active":"2023-10-16T20:36:55.274432+0000","last_became_peered":"2023-10-16T20:36:55.274432+0000","last_unstale":"2023-10-16T20:37:37.401257+0000","last_undegraded":"2023-10-16T20:37:37.401257+0000","last_fullsized":"2023-10-16T20:37:37.401257+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:53:08.896731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.508458+0000","last_change":"2023-10-16T20:36:54.762356+0000","last_active":"2023-10-16T20:39:33.508458+0000","last_peered":"2023-10-16T20:39:33.508458+0000","last_clean":"2023-10-16T20:39:33.508458+0000","last_became_active":"2023-10-16T20:36:54.762167+0000","last_became_peered":"2023-10-16T20:36:54.762167+0000","last_unstale":"2023-10-16T20:39:33.508458+0000","last_undegraded":"2023-10-16T20:39:33.508458+0000","last_fullsized":"2023-10-16T20:39:33.508458+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:45.164625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434946+0000","last_change":"2023-10-16T20:36:59.467492+0000","last_active":"2023-10-16T20:37:37.434946+0000","last_peered":"2023-10-16T20:37:37.434946+0000","last_clean":"2023-10-16T20:37:37.434946+0000","last_became_active":"2023-10-16T20:36:59.466876+0000","last_became_peered":"2023-10-16T20:36:59.466876+0000","last_unstale":"2023-10-16T20:37:37.434946+0000","last_undegraded":"2023-10-16T20:37:37.434946+0000","last_fullsized":"2023-10-16T20:37:37.434946+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:29:01.292167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409570+0000","last_change":"2023-10-16T20:36:55.293537+0000","last_active":"2023-10-16T20:37:37.409570+0000","last_peered":"2023-10-16T20:37:37.409570+0000","last_clean":"2023-10-16T20:37:37.409570+0000","last_became_active":"2023-10-16T20:36:55.293228+0000","last_became_peered":"2023-10-16T20:36:55.293228+0000","last_unstale":"2023-10-16T20:37:37.409570+0000","last_undegraded":"2023-10-16T20:37:37.409570+0000","last_fullsized":"2023-10-16T20:37:37.409570+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:50:13.226645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:02.624662+0000","last_change":"2023-10-16T20:36:54.757588+0000","last_active":"2023-10-16T20:39:02.624662+0000","last_peered":"2023-10-16T20:39:02.624662+0000","last_clean":"2023-10-16T20:39:02.624662+0000","last_became_active":"2023-10-16T20:36:54.757308+0000","last_became_peered":"2023-10-16T20:36:54.757308+0000","last_unstale":"2023-10-16T20:39:02.624662+0000","last_undegraded":"2023-10-16T20:39:02.624662+0000","last_fullsized":"2023-10-16T20:39:02.624662+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:51.298239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402099+0000","last_change":"2023-10-16T20:36:59.303844+0000","last_active":"2023-10-16T20:37:37.402099+0000","last_peered":"2023-10-16T20:37:37.402099+0000","last_clean":"2023-10-16T20:37:37.402099+0000","last_became_active":"2023-10-16T20:36:59.303224+0000","last_became_peered":"2023-10-16T20:36:59.303224+0000","last_unstale":"2023-10-16T20:37:37.402099+0000","last_undegraded":"2023-10-16T20:37:37.402099+0000","last_fullsized":"2023-10-16T20:37:37.402099+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:52.447785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403798+0000","last_change":"2023-10-16T20:36:55.278310+0000","last_active":"2023-10-16T20:37:37.403798+0000","last_peered":"2023-10-16T20:37:37.403798+0000","last_clean":"2023-10-16T20:37:37.403798+0000","last_became_active":"2023-10-16T20:36:55.277872+0000","last_became_peered":"2023-10-16T20:36:55.277872+0000","last_unstale":"2023-10-16T20:37:37.403798+0000","last_undegraded":"2023-10-16T20:37:37.403798+0000","last_fullsized":"2023-10-16T20:37:37.403798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:37.320469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.527106+0000","last_change":"2023-10-16T20:36:54.740644+0000","last_active":"2023-10-16T20:39:39.527106+0000","last_peered":"2023-10-16T20:39:39.527106+0000","last_clean":"2023-10-16T20:39:39.527106+0000","last_became_active":"2023-10-16T20:36:54.740340+0000","last_became_peered":"2023-10-16T20:36:54.740340+0000","last_unstale":"2023-10-16T20:39:39.527106+0000","last_undegraded":"2023-10-16T20:39:39.527106+0000","last_fullsized":"2023-10-16T20:39:39.527106+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:22.221727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401155+0000","last_change":"2023-10-16T20:36:59.294914+0000","last_active":"2023-10-16T20:37:37.401155+0000","last_peered":"2023-10-16T20:37:37.401155+0000","last_clean":"2023-10-16T20:37:37.401155+0000","last_became_active":"2023-10-16T20:36:59.294548+0000","last_became_peered":"2023-10-16T20:36:59.294548+0000","last_unstale":"2023-10-16T20:37:37.401155+0000","last_undegraded":"2023-10-16T20:37:37.401155+0000","last_fullsized":"2023-10-16T20:37:37.401155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:52:15.265262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401383+0000","last_change":"2023-10-16T20:36:55.270992+0000","last_active":"2023-10-16T20:37:37.401383+0000","last_peered":"2023-10-16T20:37:37.401383+0000","last_clean":"2023-10-16T20:37:37.401383+0000","last_became_active":"2023-10-16T20:36:55.269749+0000","last_became_peered":"2023-10-16T20:36:55.269749+0000","last_unstale":"2023-10-16T20:37:37.401383+0000","last_undegraded":"2023-10-16T20:37:37.401383+0000","last_fullsized":"2023-10-16T20:37:37.401383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:30.925314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'6","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.522848+0000","last_change":"2023-10-16T20:36:54.757903+0000","last_active":"2023-10-16T20:39:39.522848+0000","last_peered":"2023-10-16T20:39:39.522848+0000","last_clean":"2023-10-16T20:39:39.522848+0000","last_became_active":"2023-10-16T20:36:54.757549+0000","last_became_peered":"2023-10-16T20:36:54.757549+0000","last_unstale":"2023-10-16T20:39:39.522848+0000","last_undegraded":"2023-10-16T20:39:39.522848+0000","last_fullsized":"2023-10-16T20:39:39.522848+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:02:42.090848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402786+0000","last_change":"2023-10-16T20:36:59.302063+0000","last_active":"2023-10-16T20:37:37.402786+0000","last_peered":"2023-10-16T20:37:37.402786+0000","last_clean":"2023-10-16T20:37:37.402786+0000","last_became_active":"2023-10-16T20:36:59.301693+0000","last_became_peered":"2023-10-16T20:36:59.301693+0000","last_unstale":"2023-10-16T20:37:37.402786+0000","last_undegraded":"2023-10-16T20:37:37.402786+0000","last_fullsized":"2023-10-16T20:37:37.402786+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:29:44.812282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434713+0000","last_change":"2023-10-16T20:36:55.278132+0000","last_active":"2023-10-16T20:37:37.434713+0000","last_peered":"2023-10-16T20:37:37.434713+0000","last_clean":"2023-10-16T20:37:37.434713+0000","last_became_active":"2023-10-16T20:36:55.276649+0000","last_became_peered":"2023-10-16T20:36:55.276649+0000","last_unstale":"2023-10-16T20:37:37.434713+0000","last_undegraded":"2023-10-16T20:37:37.434713+0000","last_fullsized":"2023-10-16T20:37:37.434713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:30:53.299514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:28.448344+0000","last_change":"2023-10-16T20:36:54.753592+0000","last_active":"2023-10-16T20:39:28.448344+0000","last_peered":"2023-10-16T20:39:28.448344+0000","last_clean":"2023-10-16T20:39:28.448344+0000","last_became_active":"2023-10-16T20:36:54.753237+0000","last_became_peered":"2023-10-16T20:36:54.753237+0000","last_unstale":"2023-10-16T20:39:28.448344+0000","last_undegraded":"2023-10-16T20:39:28.448344+0000","last_fullsized":"2023-10-16T20:39:28.448344+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:53.005979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435360+0000","last_change":"2023-10-16T20:36:59.468431+0000","last_active":"2023-10-16T20:37:37.435360+0000","last_peered":"2023-10-16T20:37:37.435360+0000","last_clean":"2023-10-16T20:37:37.435360+0000","last_became_active":"2023-10-16T20:36:59.466708+0000","last_became_peered":"2023-10-16T20:36:59.466708+0000","last_unstale":"2023-10-16T20:37:37.435360+0000","last_undegraded":"2023-10-16T20:37:37.435360+0000","last_fullsized":"2023-10-16T20:37:37.435360+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:42:20.607519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464817+0000","last_change":"2023-10-16T20:36:55.294825+0000","last_active":"2023-10-16T20:37:37.464817+0000","last_peered":"2023-10-16T20:37:37.464817+0000","last_clean":"2023-10-16T20:37:37.464817+0000","last_became_active":"2023-10-16T20:36:55.294610+0000","last_became_peered":"2023-10-16T20:36:55.294610+0000","last_unstale":"2023-10-16T20:37:37.464817+0000","last_undegraded":"2023-10-16T20:37:37.464817+0000","last_fullsized":"2023-10-16T20:37:37.464817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:43.442939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.507413+0000","last_change":"2023-10-16T20:36:54.741191+0000","last_active":"2023-10-16T20:39:33.507413+0000","last_peered":"2023-10-16T20:39:33.507413+0000","last_clean":"2023-10-16T20:39:33.507413+0000","last_became_active":"2023-10-16T20:36:54.740944+0000","last_became_peered":"2023-10-16T20:36:54.740944+0000","last_unstale":"2023-10-16T20:39:33.507413+0000","last_undegraded":"2023-10-16T20:39:33.507413+0000","last_fullsized":"2023-10-16T20:39:33.507413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:05.961850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":172,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466038+0000","last_change":"2023-10-16T20:36:46.390001+0000","last_active":"2023-10-16T20:37:37.466038+0000","last_peered":"2023-10-16T20:37:37.466038+0000","last_clean":"2023-10-16T20:37:37.466038+0000","last_became_active":"2023-10-16T20:36:46.388824+0000","last_became_peered":"2023-10-16T20:36:46.388824+0000","last_unstale":"2023-10-16T20:37:37.466038+0000","last_undegraded":"2023-10-16T20:37:37.466038+0000","last_fullsized":"2023-10-16T20:37:37.466038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_clean_scrub_stamp":"2023-10-16T20:36:44.575506+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:31.004318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407653+0000","last_change":"2023-10-16T20:36:59.468528+0000","last_active":"2023-10-16T20:37:37.407653+0000","last_peered":"2023-10-16T20:37:37.407653+0000","last_clean":"2023-10-16T20:37:37.407653+0000","last_became_active":"2023-10-16T20:36:59.468082+0000","last_became_peered":"2023-10-16T20:36:59.468082+0000","last_unstale":"2023-10-16T20:37:37.407653+0000","last_undegraded":"2023-10-16T20:37:37.407653+0000","last_fullsized":"2023-10-16T20:37:37.407653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:00.920004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.613641+0000","last_change":"2023-10-16T20:37:37.409578+0000","last_active":"2023-10-16T20:37:37.613641+0000","last_peered":"2023-10-16T20:37:37.613641+0000","last_clean":"2023-10-16T20:37:37.613641+0000","last_became_active":"2023-10-16T20:37:37.409004+0000","last_became_peered":"2023-10-16T20:37:37.409004+0000","last_unstale":"2023-10-16T20:37:37.613641+0000","last_undegraded":"2023-10-16T20:37:37.613641+0000","last_fullsized":"2023-10-16T20:37:37.613641+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:28:57.364794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"29'6","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.507117+0000","last_change":"2023-10-16T20:36:54.746600+0000","last_active":"2023-10-16T20:39:33.507117+0000","last_peered":"2023-10-16T20:39:33.507117+0000","last_clean":"2023-10-16T20:39:33.507117+0000","last_became_active":"2023-10-16T20:36:54.746247+0000","last_became_peered":"2023-10-16T20:36:54.746247+0000","last_unstale":"2023-10-16T20:39:33.507117+0000","last_undegraded":"2023-10-16T20:39:33.507117+0000","last_fullsized":"2023-10-16T20:39:33.507117+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:28:57.949487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1610,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.470531+0000","last_change":"2023-10-16T20:37:37.469427+0000","last_active":"2023-10-16T20:37:37.470531+0000","last_peered":"2023-10-16T20:37:37.470531+0000","last_clean":"2023-10-16T20:37:37.470531+0000","last_became_active":"2023-10-16T20:37:37.467402+0000","last_became_peered":"2023-10-16T20:37:37.467402+0000","last_unstale":"2023-10-16T20:37:37.470531+0000","last_undegraded":"2023-10-16T20:37:37.470531+0000","last_fullsized":"2023-10-16T20:37:37.470531+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:00:19.171972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409113+0000","last_change":"2023-10-16T20:36:55.301234+0000","last_active":"2023-10-16T20:37:37.409113+0000","last_peered":"2023-10-16T20:37:37.409113+0000","last_clean":"2023-10-16T20:37:37.409113+0000","last_became_active":"2023-10-16T20:36:55.300958+0000","last_became_peered":"2023-10-16T20:36:55.300958+0000","last_unstale":"2023-10-16T20:37:37.409113+0000","last_undegraded":"2023-10-16T20:37:37.409113+0000","last_fullsized":"2023-10-16T20:37:37.409113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:16.784124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.506025+0000","last_change":"2023-10-16T20:36:54.753770+0000","last_active":"2023-10-16T20:39:33.506025+0000","last_peered":"2023-10-16T20:39:33.506025+0000","last_clean":"2023-10-16T20:39:33.506025+0000","last_became_active":"2023-10-16T20:36:54.753443+0000","last_became_peered":"2023-10-16T20:36:54.753443+0000","last_unstale":"2023-10-16T20:39:33.506025+0000","last_undegraded":"2023-10-16T20:39:33.506025+0000","last_fullsized":"2023-10-16T20:39:33.506025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:21:26.392480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435046+0000","last_change":"2023-10-16T20:36:59.303351+0000","last_active":"2023-10-16T20:37:37.435046+0000","last_peered":"2023-10-16T20:37:37.435046+0000","last_clean":"2023-10-16T20:37:37.435046+0000","last_became_active":"2023-10-16T20:36:59.302869+0000","last_became_peered":"2023-10-16T20:36:59.302869+0000","last_unstale":"2023-10-16T20:37:37.435046+0000","last_undegraded":"2023-10-16T20:37:37.435046+0000","last_fullsized":"2023-10-16T20:37:37.435046+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:26:35.886399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466830+0000","last_change":"2023-10-16T20:36:55.291947+0000","last_active":"2023-10-16T20:37:37.466830+0000","last_peered":"2023-10-16T20:37:37.466830+0000","last_clean":"2023-10-16T20:37:37.466830+0000","last_became_active":"2023-10-16T20:36:55.291633+0000","last_became_peered":"2023-10-16T20:36:55.291633+0000","last_unstale":"2023-10-16T20:37:37.466830+0000","last_undegraded":"2023-10-16T20:37:37.466830+0000","last_fullsized":"2023-10-16T20:37:37.466830+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:00:14.349339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.519506+0000","last_change":"2023-10-16T20:36:54.748196+0000","last_active":"2023-10-16T20:39:39.519506+0000","last_peered":"2023-10-16T20:39:39.519506+0000","last_clean":"2023-10-16T20:39:39.519506+0000","last_became_active":"2023-10-16T20:36:54.747704+0000","last_became_peered":"2023-10-16T20:36:54.747704+0000","last_unstale":"2023-10-16T20:39:39.519506+0000","last_undegraded":"2023-10-16T20:39:39.519506+0000","last_fullsized":"2023-10-16T20:39:39.519506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:55.471916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410236+0000","last_change":"2023-10-16T20:36:59.470368+0000","last_active":"2023-10-16T20:37:37.410236+0000","last_peered":"2023-10-16T20:37:37.410236+0000","last_clean":"2023-10-16T20:37:37.410236+0000","last_became_active":"2023-10-16T20:36:59.467291+0000","last_became_peered":"2023-10-16T20:36:59.467291+0000","last_unstale":"2023-10-16T20:37:37.410236+0000","last_undegraded":"2023-10-16T20:37:37.410236+0000","last_fullsized":"2023-10-16T20:37:37.410236+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:18.312394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466658+0000","last_change":"2023-10-16T20:36:55.291451+0000","last_active":"2023-10-16T20:37:37.466658+0000","last_peered":"2023-10-16T20:37:37.466658+0000","last_clean":"2023-10-16T20:37:37.466658+0000","last_became_active":"2023-10-16T20:36:55.290998+0000","last_became_peered":"2023-10-16T20:36:55.290998+0000","last_unstale":"2023-10-16T20:37:37.466658+0000","last_undegraded":"2023-10-16T20:37:37.466658+0000","last_fullsized":"2023-10-16T20:37:37.466658+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:05:01.634971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"29'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:12.629039+0000","last_change":"2023-10-16T20:36:54.763818+0000","last_active":"2023-10-16T20:39:12.629039+0000","last_peered":"2023-10-16T20:39:12.629039+0000","last_clean":"2023-10-16T20:39:12.629039+0000","last_became_active":"2023-10-16T20:36:54.761337+0000","last_became_peered":"2023-10-16T20:36:54.761337+0000","last_unstale":"2023-10-16T20:39:12.629039+0000","last_undegraded":"2023-10-16T20:39:12.629039+0000","last_fullsized":"2023-10-16T20:39:12.629039+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:03.093189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.470301+0000","last_change":"2023-10-16T20:37:37.468803+0000","last_active":"2023-10-16T20:37:37.470301+0000","last_peered":"2023-10-16T20:37:37.470301+0000","last_clean":"2023-10-16T20:37:37.470301+0000","last_became_active":"2023-10-16T20:37:37.468309+0000","last_became_peered":"2023-10-16T20:37:37.468309+0000","last_unstale":"2023-10-16T20:37:37.470301+0000","last_undegraded":"2023-10-16T20:37:37.470301+0000","last_fullsized":"2023-10-16T20:37:37.470301+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:11.761661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466101+0000","last_change":"2023-10-16T20:36:55.288895+0000","last_active":"2023-10-16T20:37:37.466101+0000","last_peered":"2023-10-16T20:37:37.466101+0000","last_clean":"2023-10-16T20:37:37.466101+0000","last_became_active":"2023-10-16T20:36:55.288354+0000","last_became_peered":"2023-10-16T20:36:55.288354+0000","last_unstale":"2023-10-16T20:37:37.466101+0000","last_undegraded":"2023-10-16T20:37:37.466101+0000","last_fullsized":"2023-10-16T20:37:37.466101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:53:29.192406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'17","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.501703+0000","last_change":"2023-10-16T20:36:54.738216+0000","last_active":"2023-10-16T20:39:33.501703+0000","last_peered":"2023-10-16T20:39:33.501703+0000","last_clean":"2023-10-16T20:39:33.501703+0000","last_became_active":"2023-10-16T20:36:54.737876+0000","last_became_peered":"2023-10-16T20:36:54.737876+0000","last_unstale":"2023-10-16T20:39:33.501703+0000","last_undegraded":"2023-10-16T20:39:33.501703+0000","last_fullsized":"2023-10-16T20:39:33.501703+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:28.214601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435178+0000","last_change":"2023-10-16T20:36:59.307215+0000","last_active":"2023-10-16T20:37:37.435178+0000","last_peered":"2023-10-16T20:37:37.435178+0000","last_clean":"2023-10-16T20:37:37.435178+0000","last_became_active":"2023-10-16T20:36:59.306801+0000","last_became_peered":"2023-10-16T20:36:59.306801+0000","last_unstale":"2023-10-16T20:37:37.435178+0000","last_undegraded":"2023-10-16T20:37:37.435178+0000","last_fullsized":"2023-10-16T20:37:37.435178+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:41:42.656134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407054+0000","last_change":"2023-10-16T20:36:55.287714+0000","last_active":"2023-10-16T20:37:37.407054+0000","last_peered":"2023-10-16T20:37:37.407054+0000","last_clean":"2023-10-16T20:37:37.407054+0000","last_became_active":"2023-10-16T20:36:55.287066+0000","last_became_peered":"2023-10-16T20:36:55.287066+0000","last_unstale":"2023-10-16T20:37:37.407054+0000","last_undegraded":"2023-10-16T20:37:37.407054+0000","last_fullsized":"2023-10-16T20:37:37.407054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:45:21.058260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"29'6","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.509257+0000","last_change":"2023-10-16T20:36:54.764693+0000","last_active":"2023-10-16T20:39:33.509257+0000","last_peered":"2023-10-16T20:39:33.509257+0000","last_clean":"2023-10-16T20:39:33.509257+0000","last_became_active":"2023-10-16T20:36:54.764417+0000","last_became_peered":"2023-10-16T20:36:54.764417+0000","last_unstale":"2023-10-16T20:39:33.509257+0000","last_undegraded":"2023-10-16T20:39:33.509257+0000","last_fullsized":"2023-10-16T20:39:33.509257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:56.653149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1028,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465588+0000","last_change":"2023-10-16T20:36:59.465145+0000","last_active":"2023-10-16T20:37:37.465588+0000","last_peered":"2023-10-16T20:37:37.465588+0000","last_clean":"2023-10-16T20:37:37.465588+0000","last_became_active":"2023-10-16T20:36:59.464626+0000","last_became_peered":"2023-10-16T20:36:59.464626+0000","last_unstale":"2023-10-16T20:37:37.465588+0000","last_undegraded":"2023-10-16T20:37:37.465588+0000","last_fullsized":"2023-10-16T20:37:37.465588+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:38.928616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466290+0000","last_change":"2023-10-16T20:36:55.291718+0000","last_active":"2023-10-16T20:37:37.466290+0000","last_peered":"2023-10-16T20:37:37.466290+0000","last_clean":"2023-10-16T20:37:37.466290+0000","last_became_active":"2023-10-16T20:36:55.291396+0000","last_became_peered":"2023-10-16T20:36:55.291396+0000","last_unstale":"2023-10-16T20:37:37.466290+0000","last_undegraded":"2023-10-16T20:37:37.466290+0000","last_fullsized":"2023-10-16T20:37:37.466290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:29:49.138380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.507860+0000","last_change":"2023-10-16T20:36:54.750814+0000","last_active":"2023-10-16T20:39:33.507860+0000","last_peered":"2023-10-16T20:39:33.507860+0000","last_clean":"2023-10-16T20:39:33.507860+0000","last_became_active":"2023-10-16T20:36:54.750448+0000","last_became_peered":"2023-10-16T20:36:54.750448+0000","last_unstale":"2023-10-16T20:39:33.507860+0000","last_undegraded":"2023-10-16T20:39:33.507860+0000","last_fullsized":"2023-10-16T20:39:33.507860+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:41.981408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465192+0000","last_change":"2023-10-16T20:36:59.464962+0000","last_active":"2023-10-16T20:37:37.465192+0000","last_peered":"2023-10-16T20:37:37.465192+0000","last_clean":"2023-10-16T20:37:37.465192+0000","last_became_active":"2023-10-16T20:36:59.464578+0000","last_became_peered":"2023-10-16T20:36:59.464578+0000","last_unstale":"2023-10-16T20:37:37.465192+0000","last_undegraded":"2023-10-16T20:37:37.465192+0000","last_fullsized":"2023-10-16T20:37:37.465192+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:12:27.738364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435179+0000","last_change":"2023-10-16T20:36:55.285675+0000","last_active":"2023-10-16T20:37:37.435179+0000","last_peered":"2023-10-16T20:37:37.435179+0000","last_clean":"2023-10-16T20:37:37.435179+0000","last_became_active":"2023-10-16T20:36:55.285407+0000","last_became_peered":"2023-10-16T20:36:55.285407+0000","last_unstale":"2023-10-16T20:37:37.435179+0000","last_undegraded":"2023-10-16T20:37:37.435179+0000","last_fullsized":"2023-10-16T20:37:37.435179+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:31:10.259938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'9","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.524706+0000","last_change":"2023-10-16T20:36:54.757012+0000","last_active":"2023-10-16T20:39:39.524706+0000","last_peered":"2023-10-16T20:39:39.524706+0000","last_clean":"2023-10-16T20:39:39.524706+0000","last_became_active":"2023-10-16T20:36:54.756697+0000","last_became_peered":"2023-10-16T20:36:54.756697+0000","last_unstale":"2023-10-16T20:39:39.524706+0000","last_undegraded":"2023-10-16T20:39:39.524706+0000","last_fullsized":"2023-10-16T20:39:39.524706+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:20.259580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409477+0000","last_change":"2023-10-16T20:36:59.469134+0000","last_active":"2023-10-16T20:37:37.409477+0000","last_peered":"2023-10-16T20:37:37.409477+0000","last_clean":"2023-10-16T20:37:37.409477+0000","last_became_active":"2023-10-16T20:36:59.468254+0000","last_became_peered":"2023-10-16T20:36:59.468254+0000","last_unstale":"2023-10-16T20:37:37.409477+0000","last_undegraded":"2023-10-16T20:37:37.409477+0000","last_fullsized":"2023-10-16T20:37:37.409477+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:12.444815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466418+0000","last_change":"2023-10-16T20:36:55.289133+0000","last_active":"2023-10-16T20:37:37.466418+0000","last_peered":"2023-10-16T20:37:37.466418+0000","last_clean":"2023-10-16T20:37:37.466418+0000","last_became_active":"2023-10-16T20:36:55.288488+0000","last_became_peered":"2023-10-16T20:36:55.288488+0000","last_unstale":"2023-10-16T20:37:37.466418+0000","last_undegraded":"2023-10-16T20:37:37.466418+0000","last_fullsized":"2023-10-16T20:37:37.466418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:27:22.022319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'16","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:48.656539+0000","last_change":"2023-10-16T20:36:54.765361+0000","last_active":"2023-10-16T20:39:48.656539+0000","last_peered":"2023-10-16T20:39:48.656539+0000","last_clean":"2023-10-16T20:39:48.656539+0000","last_became_active":"2023-10-16T20:36:54.764969+0000","last_became_peered":"2023-10-16T20:36:54.764969+0000","last_unstale":"2023-10-16T20:39:48.656539+0000","last_undegraded":"2023-10-16T20:39:48.656539+0000","last_fullsized":"2023-10-16T20:39:48.656539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:21:21.183612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4281,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402193+0000","last_change":"2023-10-16T20:36:59.303588+0000","last_active":"2023-10-16T20:37:37.402193+0000","last_peered":"2023-10-16T20:37:37.402193+0000","last_clean":"2023-10-16T20:37:37.402193+0000","last_became_active":"2023-10-16T20:36:59.302698+0000","last_became_peered":"2023-10-16T20:36:59.302698+0000","last_unstale":"2023-10-16T20:37:37.402193+0000","last_undegraded":"2023-10-16T20:37:37.402193+0000","last_fullsized":"2023-10-16T20:37:37.402193+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:08:28.644565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403928+0000","last_change":"2023-10-16T20:36:55.290812+0000","last_active":"2023-10-16T20:37:37.403928+0000","last_peered":"2023-10-16T20:37:37.403928+0000","last_clean":"2023-10-16T20:37:37.403928+0000","last_became_active":"2023-10-16T20:36:55.290127+0000","last_became_peered":"2023-10-16T20:36:55.290127+0000","last_unstale":"2023-10-16T20:37:37.403928+0000","last_undegraded":"2023-10-16T20:37:37.403928+0000","last_fullsized":"2023-10-16T20:37:37.403928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:28:27.930123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.406599+0000","last_change":"2023-10-16T20:36:54.749120+0000","last_active":"2023-10-16T20:39:22.406599+0000","last_peered":"2023-10-16T20:39:22.406599+0000","last_clean":"2023-10-16T20:39:22.406599+0000","last_became_active":"2023-10-16T20:36:54.748788+0000","last_became_peered":"2023-10-16T20:36:54.748788+0000","last_unstale":"2023-10-16T20:39:22.406599+0000","last_undegraded":"2023-10-16T20:39:22.406599+0000","last_fullsized":"2023-10-16T20:39:22.406599+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:11:07.257343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403049+0000","last_change":"2023-10-16T20:36:59.303546+0000","last_active":"2023-10-16T20:37:37.403049+0000","last_peered":"2023-10-16T20:37:37.403049+0000","last_clean":"2023-10-16T20:37:37.403049+0000","last_became_active":"2023-10-16T20:36:59.302916+0000","last_became_peered":"2023-10-16T20:36:59.302916+0000","last_unstale":"2023-10-16T20:37:37.403049+0000","last_undegraded":"2023-10-16T20:37:37.403049+0000","last_fullsized":"2023-10-16T20:37:37.403049+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:17:16.174476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435547+0000","last_change":"2023-10-16T20:36:55.291739+0000","last_active":"2023-10-16T20:37:37.435547+0000","last_peered":"2023-10-16T20:37:37.435547+0000","last_clean":"2023-10-16T20:37:37.435547+0000","last_became_active":"2023-10-16T20:36:55.291448+0000","last_became_peered":"2023-10-16T20:36:55.291448+0000","last_unstale":"2023-10-16T20:37:37.435547+0000","last_undegraded":"2023-10-16T20:37:37.435547+0000","last_fullsized":"2023-10-16T20:37:37.435547+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:41:15.380678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.526865+0000","last_change":"2023-10-16T20:36:54.761824+0000","last_active":"2023-10-16T20:39:39.526865+0000","last_peered":"2023-10-16T20:39:39.526865+0000","last_clean":"2023-10-16T20:39:39.526865+0000","last_became_active":"2023-10-16T20:36:54.761541+0000","last_became_peered":"2023-10-16T20:36:54.761541+0000","last_unstale":"2023-10-16T20:39:39.526865+0000","last_undegraded":"2023-10-16T20:39:39.526865+0000","last_fullsized":"2023-10-16T20:39:39.526865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:34.512207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464594+0000","last_change":"2023-10-16T20:36:59.304165+0000","last_active":"2023-10-16T20:37:37.464594+0000","last_peered":"2023-10-16T20:37:37.464594+0000","last_clean":"2023-10-16T20:37:37.464594+0000","last_became_active":"2023-10-16T20:36:59.303773+0000","last_became_peered":"2023-10-16T20:36:59.303773+0000","last_unstale":"2023-10-16T20:37:37.464594+0000","last_undegraded":"2023-10-16T20:37:37.464594+0000","last_fullsized":"2023-10-16T20:37:37.464594+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:09.710698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409711+0000","last_change":"2023-10-16T20:36:55.300829+0000","last_active":"2023-10-16T20:37:37.409711+0000","last_peered":"2023-10-16T20:37:37.409711+0000","last_clean":"2023-10-16T20:37:37.409711+0000","last_became_active":"2023-10-16T20:36:55.300535+0000","last_became_peered":"2023-10-16T20:36:55.300535+0000","last_unstale":"2023-10-16T20:37:37.409711+0000","last_undegraded":"2023-10-16T20:37:37.409711+0000","last_fullsized":"2023-10-16T20:37:37.409711+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:34.910378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.508094+0000","last_change":"2023-10-16T20:36:54.758140+0000","last_active":"2023-10-16T20:39:33.508094+0000","last_peered":"2023-10-16T20:39:33.508094+0000","last_clean":"2023-10-16T20:39:33.508094+0000","last_became_active":"2023-10-16T20:36:54.757303+0000","last_became_peered":"2023-10-16T20:36:54.757303+0000","last_unstale":"2023-10-16T20:39:33.508094+0000","last_undegraded":"2023-10-16T20:39:33.508094+0000","last_fullsized":"2023-10-16T20:39:33.508094+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:45.944191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464795+0000","last_change":"2023-10-16T20:36:59.471158+0000","last_active":"2023-10-16T20:37:37.464795+0000","last_peered":"2023-10-16T20:37:37.464795+0000","last_clean":"2023-10-16T20:37:37.464795+0000","last_became_active":"2023-10-16T20:36:59.465844+0000","last_became_peered":"2023-10-16T20:36:59.465844+0000","last_unstale":"2023-10-16T20:37:37.464795+0000","last_undegraded":"2023-10-16T20:37:37.464795+0000","last_fullsized":"2023-10-16T20:37:37.464795+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:12.133525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435182+0000","last_change":"2023-10-16T20:36:59.468843+0000","last_active":"2023-10-16T20:37:37.435182+0000","last_peered":"2023-10-16T20:37:37.435182+0000","last_clean":"2023-10-16T20:37:37.435182+0000","last_became_active":"2023-10-16T20:36:59.467356+0000","last_became_peered":"2023-10-16T20:36:59.467356+0000","last_unstale":"2023-10-16T20:37:37.435182+0000","last_undegraded":"2023-10-16T20:37:37.435182+0000","last_fullsized":"2023-10-16T20:37:37.435182+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:47:02.923540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401755+0000","last_change":"2023-10-16T20:36:55.281355+0000","last_active":"2023-10-16T20:37:37.401755+0000","last_peered":"2023-10-16T20:37:37.401755+0000","last_clean":"2023-10-16T20:37:37.401755+0000","last_became_active":"2023-10-16T20:36:55.280903+0000","last_became_peered":"2023-10-16T20:36:55.280903+0000","last_unstale":"2023-10-16T20:37:37.401755+0000","last_undegraded":"2023-10-16T20:37:37.401755+0000","last_fullsized":"2023-10-16T20:37:37.401755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:15:25.813826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.507692+0000","last_change":"2023-10-16T20:36:54.757589+0000","last_active":"2023-10-16T20:39:33.507692+0000","last_peered":"2023-10-16T20:39:33.507692+0000","last_clean":"2023-10-16T20:39:33.507692+0000","last_became_active":"2023-10-16T20:36:54.756775+0000","last_became_peered":"2023-10-16T20:36:54.756775+0000","last_unstale":"2023-10-16T20:39:33.507692+0000","last_undegraded":"2023-10-16T20:39:33.507692+0000","last_fullsized":"2023-10-16T20:39:33.507692+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:40.629540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.441661+0000","last_change":"2023-10-16T20:37:37.438581+0000","last_active":"2023-10-16T20:37:37.441661+0000","last_peered":"2023-10-16T20:37:37.441661+0000","last_clean":"2023-10-16T20:37:37.441661+0000","last_became_active":"2023-10-16T20:37:37.438140+0000","last_became_peered":"2023-10-16T20:37:37.438140+0000","last_unstale":"2023-10-16T20:37:37.441661+0000","last_undegraded":"2023-10-16T20:37:37.441661+0000","last_fullsized":"2023-10-16T20:37:37.441661+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:55.110618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464854+0000","last_change":"2023-10-16T20:36:55.274207+0000","last_active":"2023-10-16T20:37:37.464854+0000","last_peered":"2023-10-16T20:37:37.464854+0000","last_clean":"2023-10-16T20:37:37.464854+0000","last_became_active":"2023-10-16T20:36:55.273959+0000","last_became_peered":"2023-10-16T20:36:55.273959+0000","last_unstale":"2023-10-16T20:37:37.464854+0000","last_undegraded":"2023-10-16T20:37:37.464854+0000","last_fullsized":"2023-10-16T20:37:37.464854+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:23:09.275091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:28.444227+0000","last_change":"2023-10-16T20:36:54.759513+0000","last_active":"2023-10-16T20:39:28.444227+0000","last_peered":"2023-10-16T20:39:28.444227+0000","last_clean":"2023-10-16T20:39:28.444227+0000","last_became_active":"2023-10-16T20:36:54.759059+0000","last_became_peered":"2023-10-16T20:36:54.759059+0000","last_unstale":"2023-10-16T20:39:28.444227+0000","last_undegraded":"2023-10-16T20:39:28.444227+0000","last_fullsized":"2023-10-16T20:39:28.444227+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:36.696053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.509348+0000","last_change":"2023-10-16T20:36:54.764780+0000","last_active":"2023-10-16T20:39:33.509348+0000","last_peered":"2023-10-16T20:39:33.509348+0000","last_clean":"2023-10-16T20:39:33.509348+0000","last_became_active":"2023-10-16T20:36:54.764436+0000","last_became_peered":"2023-10-16T20:36:54.764436+0000","last_unstale":"2023-10-16T20:39:33.509348+0000","last_undegraded":"2023-10-16T20:39:33.509348+0000","last_fullsized":"2023-10-16T20:39:33.509348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:55:31.816599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435152+0000","last_change":"2023-10-16T20:36:55.278134+0000","last_active":"2023-10-16T20:37:37.435152+0000","last_peered":"2023-10-16T20:37:37.435152+0000","last_clean":"2023-10-16T20:37:37.435152+0000","last_became_active":"2023-10-16T20:36:55.277873+0000","last_became_peered":"2023-10-16T20:36:55.277873+0000","last_unstale":"2023-10-16T20:37:37.435152+0000","last_undegraded":"2023-10-16T20:37:37.435152+0000","last_fullsized":"2023-10-16T20:37:37.435152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:59:43.701800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.399947+0000","last_change":"2023-10-16T20:36:59.302774+0000","last_active":"2023-10-16T20:37:37.399947+0000","last_peered":"2023-10-16T20:37:37.399947+0000","last_clean":"2023-10-16T20:37:37.399947+0000","last_became_active":"2023-10-16T20:36:59.302435+0000","last_became_peered":"2023-10-16T20:36:59.302435+0000","last_unstale":"2023-10-16T20:37:37.399947+0000","last_undegraded":"2023-10-16T20:37:37.399947+0000","last_fullsized":"2023-10-16T20:37:37.399947+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:11:53.740982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'6","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:27.631103+0000","last_change":"2023-10-16T20:36:54.756190+0000","last_active":"2023-10-16T20:39:27.631103+0000","last_peered":"2023-10-16T20:39:27.631103+0000","last_clean":"2023-10-16T20:39:27.631103+0000","last_became_active":"2023-10-16T20:36:54.755834+0000","last_became_peered":"2023-10-16T20:36:54.755834+0000","last_unstale":"2023-10-16T20:39:27.631103+0000","last_undegraded":"2023-10-16T20:39:27.631103+0000","last_fullsized":"2023-10-16T20:39:27.631103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:31.012441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1062,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403395+0000","last_change":"2023-10-16T20:36:55.279822+0000","last_active":"2023-10-16T20:37:37.403395+0000","last_peered":"2023-10-16T20:37:37.403395+0000","last_clean":"2023-10-16T20:37:37.403395+0000","last_became_active":"2023-10-16T20:36:55.279333+0000","last_became_peered":"2023-10-16T20:36:55.279333+0000","last_unstale":"2023-10-16T20:37:37.403395+0000","last_undegraded":"2023-10-16T20:37:37.403395+0000","last_fullsized":"2023-10-16T20:37:37.403395+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:02:21.471812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465391+0000","last_change":"2023-10-16T20:36:59.302190+0000","last_active":"2023-10-16T20:37:37.465391+0000","last_peered":"2023-10-16T20:37:37.465391+0000","last_clean":"2023-10-16T20:37:37.465391+0000","last_became_active":"2023-10-16T20:36:59.301864+0000","last_became_peered":"2023-10-16T20:36:59.301864+0000","last_unstale":"2023-10-16T20:37:37.465391+0000","last_undegraded":"2023-10-16T20:37:37.465391+0000","last_fullsized":"2023-10-16T20:37:37.465391+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:37:55.282815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.406436+0000","last_change":"2023-10-16T20:36:55.302476+0000","last_active":"2023-10-16T20:37:37.406436+0000","last_peered":"2023-10-16T20:37:37.406436+0000","last_clean":"2023-10-16T20:37:37.406436+0000","last_became_active":"2023-10-16T20:36:55.287442+0000","last_became_peered":"2023-10-16T20:36:55.287442+0000","last_unstale":"2023-10-16T20:37:37.406436+0000","last_undegraded":"2023-10-16T20:37:37.406436+0000","last_fullsized":"2023-10-16T20:37:37.406436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:19.511156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.399520+0000","last_change":"2023-10-16T20:36:55.276521+0000","last_active":"2023-10-16T20:37:37.399520+0000","last_peered":"2023-10-16T20:37:37.399520+0000","last_clean":"2023-10-16T20:37:37.399520+0000","last_became_active":"2023-10-16T20:36:55.276276+0000","last_became_peered":"2023-10-16T20:36:55.276276+0000","last_unstale":"2023-10-16T20:37:37.399520+0000","last_undegraded":"2023-10-16T20:37:37.399520+0000","last_fullsized":"2023-10-16T20:37:37.399520+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:38.218242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.415548+0000","last_change":"2023-10-16T20:37:37.413429+0000","last_active":"2023-10-16T20:37:37.415548+0000","last_peered":"2023-10-16T20:37:37.415548+0000","last_clean":"2023-10-16T20:37:37.415548+0000","last_became_active":"2023-10-16T20:37:37.413040+0000","last_became_peered":"2023-10-16T20:37:37.413040+0000","last_unstale":"2023-10-16T20:37:37.415548+0000","last_undegraded":"2023-10-16T20:37:37.415548+0000","last_fullsized":"2023-10-16T20:37:37.415548+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:39:10.897056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435108+0000","last_change":"2023-10-16T20:36:55.276258+0000","last_active":"2023-10-16T20:37:37.435108+0000","last_peered":"2023-10-16T20:37:37.435108+0000","last_clean":"2023-10-16T20:37:37.435108+0000","last_became_active":"2023-10-16T20:36:55.275349+0000","last_became_peered":"2023-10-16T20:36:55.275349+0000","last_unstale":"2023-10-16T20:37:37.435108+0000","last_undegraded":"2023-10-16T20:37:37.435108+0000","last_fullsized":"2023-10-16T20:37:37.435108+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:59.007782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465455+0000","last_change":"2023-10-16T20:36:55.277332+0000","last_active":"2023-10-16T20:37:37.465455+0000","last_peered":"2023-10-16T20:37:37.465455+0000","last_clean":"2023-10-16T20:37:37.465455+0000","last_became_active":"2023-10-16T20:36:55.277043+0000","last_became_peered":"2023-10-16T20:36:55.277043+0000","last_unstale":"2023-10-16T20:37:37.465455+0000","last_undegraded":"2023-10-16T20:37:37.465455+0000","last_fullsized":"2023-10-16T20:37:37.465455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:46:07.227813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401690+0000","last_change":"2023-10-16T20:36:55.279003+0000","last_active":"2023-10-16T20:37:37.401690+0000","last_peered":"2023-10-16T20:37:37.401690+0000","last_clean":"2023-10-16T20:37:37.401690+0000","last_became_active":"2023-10-16T20:36:55.278715+0000","last_became_peered":"2023-10-16T20:36:55.278715+0000","last_unstale":"2023-10-16T20:37:37.401690+0000","last_undegraded":"2023-10-16T20:37:37.401690+0000","last_fullsized":"2023-10-16T20:37:37.401690+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:58.274009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408078+0000","last_change":"2023-10-16T20:36:55.303936+0000","last_active":"2023-10-16T20:37:37.408078+0000","last_peered":"2023-10-16T20:37:37.408078+0000","last_clean":"2023-10-16T20:37:37.408078+0000","last_became_active":"2023-10-16T20:36:55.301604+0000","last_became_peered":"2023-10-16T20:36:55.301604+0000","last_unstale":"2023-10-16T20:37:37.408078+0000","last_undegraded":"2023-10-16T20:37:37.408078+0000","last_fullsized":"2023-10-16T20:37:37.408078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:48:14.922695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465625+0000","last_change":"2023-10-16T20:36:55.277329+0000","last_active":"2023-10-16T20:37:37.465625+0000","last_peered":"2023-10-16T20:37:37.465625+0000","last_clean":"2023-10-16T20:37:37.465625+0000","last_became_active":"2023-10-16T20:36:55.276984+0000","last_became_peered":"2023-10-16T20:36:55.276984+0000","last_unstale":"2023-10-16T20:37:37.465625+0000","last_undegraded":"2023-10-16T20:37:37.465625+0000","last_fullsized":"2023-10-16T20:37:37.465625+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:43:52.763586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408404+0000","last_change":"2023-10-16T20:36:55.290545+0000","last_active":"2023-10-16T20:37:37.408404+0000","last_peered":"2023-10-16T20:37:37.408404+0000","last_clean":"2023-10-16T20:37:37.408404+0000","last_became_active":"2023-10-16T20:36:55.290023+0000","last_became_peered":"2023-10-16T20:36:55.290023+0000","last_unstale":"2023-10-16T20:37:37.408404+0000","last_undegraded":"2023-10-16T20:37:37.408404+0000","last_fullsized":"2023-10-16T20:37:37.408404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:04:20.914854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464978+0000","last_change":"2023-10-16T20:36:55.280456+0000","last_active":"2023-10-16T20:37:37.464978+0000","last_peered":"2023-10-16T20:37:37.464978+0000","last_clean":"2023-10-16T20:37:37.464978+0000","last_became_active":"2023-10-16T20:36:55.280186+0000","last_became_peered":"2023-10-16T20:36:55.280186+0000","last_unstale":"2023-10-16T20:37:37.464978+0000","last_undegraded":"2023-10-16T20:37:37.464978+0000","last_fullsized":"2023-10-16T20:37:37.464978+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:52.469768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408252+0000","last_change":"2023-10-16T20:36:55.297917+0000","last_active":"2023-10-16T20:37:37.408252+0000","last_peered":"2023-10-16T20:37:37.408252+0000","last_clean":"2023-10-16T20:37:37.408252+0000","last_became_active":"2023-10-16T20:36:55.297253+0000","last_became_peered":"2023-10-16T20:36:55.297253+0000","last_unstale":"2023-10-16T20:37:37.408252+0000","last_undegraded":"2023-10-16T20:37:37.408252+0000","last_fullsized":"2023-10-16T20:37:37.408252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:14.977093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434979+0000","last_change":"2023-10-16T20:36:55.286327+0000","last_active":"2023-10-16T20:37:37.434979+0000","last_peered":"2023-10-16T20:37:37.434979+0000","last_clean":"2023-10-16T20:37:37.434979+0000","last_became_active":"2023-10-16T20:36:55.286044+0000","last_became_peered":"2023-10-16T20:36:55.286044+0000","last_unstale":"2023-10-16T20:37:37.434979+0000","last_undegraded":"2023-10-16T20:37:37.434979+0000","last_fullsized":"2023-10-16T20:37:37.434979+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:30:56.000502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409227+0000","last_change":"2023-10-16T20:36:55.298742+0000","last_active":"2023-10-16T20:37:37.409227+0000","last_peered":"2023-10-16T20:37:37.409227+0000","last_clean":"2023-10-16T20:37:37.409227+0000","last_became_active":"2023-10-16T20:36:55.286864+0000","last_became_peered":"2023-10-16T20:36:55.286864+0000","last_unstale":"2023-10-16T20:37:37.409227+0000","last_undegraded":"2023-10-16T20:37:37.409227+0000","last_fullsized":"2023-10-16T20:37:37.409227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:24:27.598910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466521+0000","last_change":"2023-10-16T20:36:55.292541+0000","last_active":"2023-10-16T20:37:37.466521+0000","last_peered":"2023-10-16T20:37:37.466521+0000","last_clean":"2023-10-16T20:37:37.466521+0000","last_became_active":"2023-10-16T20:36:55.292262+0000","last_became_peered":"2023-10-16T20:36:55.292262+0000","last_unstale":"2023-10-16T20:37:37.466521+0000","last_undegraded":"2023-10-16T20:37:37.466521+0000","last_fullsized":"2023-10-16T20:37:37.466521+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:10:08.644265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.400632+0000","last_change":"2023-10-16T20:36:55.283056+0000","last_active":"2023-10-16T20:37:37.400632+0000","last_peered":"2023-10-16T20:37:37.400632+0000","last_clean":"2023-10-16T20:37:37.400632+0000","last_became_active":"2023-10-16T20:36:55.282806+0000","last_became_peered":"2023-10-16T20:36:55.282806+0000","last_unstale":"2023-10-16T20:37:37.400632+0000","last_undegraded":"2023-10-16T20:37:37.400632+0000","last_fullsized":"2023-10-16T20:37:37.400632+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:21:24.681696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410649+0000","last_change":"2023-10-16T20:36:55.303385+0000","last_active":"2023-10-16T20:37:37.410649+0000","last_peered":"2023-10-16T20:37:37.410649+0000","last_clean":"2023-10-16T20:37:37.410649+0000","last_became_active":"2023-10-16T20:36:55.303095+0000","last_became_peered":"2023-10-16T20:36:55.303095+0000","last_unstale":"2023-10-16T20:37:37.410649+0000","last_undegraded":"2023-10-16T20:37:37.410649+0000","last_fullsized":"2023-10-16T20:37:37.410649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:26.368480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408312+0000","last_change":"2023-10-16T20:36:55.291852+0000","last_active":"2023-10-16T20:37:37.408312+0000","last_peered":"2023-10-16T20:37:37.408312+0000","last_clean":"2023-10-16T20:37:37.408312+0000","last_became_active":"2023-10-16T20:36:55.291447+0000","last_became_peered":"2023-10-16T20:36:55.291447+0000","last_unstale":"2023-10-16T20:37:37.408312+0000","last_undegraded":"2023-10-16T20:37:37.408312+0000","last_fullsized":"2023-10-16T20:37:37.408312+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:07.195372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401838+0000","last_change":"2023-10-16T20:36:55.281169+0000","last_active":"2023-10-16T20:37:37.401838+0000","last_peered":"2023-10-16T20:37:37.401838+0000","last_clean":"2023-10-16T20:37:37.401838+0000","last_became_active":"2023-10-16T20:36:55.280605+0000","last_became_peered":"2023-10-16T20:36:55.280605+0000","last_unstale":"2023-10-16T20:37:37.401838+0000","last_undegraded":"2023-10-16T20:37:37.401838+0000","last_fullsized":"2023-10-16T20:37:37.401838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:03.294173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435040+0000","last_change":"2023-10-16T20:36:55.277395+0000","last_active":"2023-10-16T20:37:37.435040+0000","last_peered":"2023-10-16T20:37:37.435040+0000","last_clean":"2023-10-16T20:37:37.435040+0000","last_became_active":"2023-10-16T20:36:55.277094+0000","last_became_peered":"2023-10-16T20:36:55.277094+0000","last_unstale":"2023-10-16T20:37:37.435040+0000","last_undegraded":"2023-10-16T20:37:37.435040+0000","last_fullsized":"2023-10-16T20:37:37.435040+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:23:03.400384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465006+0000","last_change":"2023-10-16T20:36:55.297429+0000","last_active":"2023-10-16T20:37:37.465006+0000","last_peered":"2023-10-16T20:37:37.465006+0000","last_clean":"2023-10-16T20:37:37.465006+0000","last_became_active":"2023-10-16T20:36:55.297199+0000","last_became_peered":"2023-10-16T20:36:55.297199+0000","last_unstale":"2023-10-16T20:37:37.465006+0000","last_undegraded":"2023-10-16T20:37:37.465006+0000","last_fullsized":"2023-10-16T20:37:37.465006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:43.860251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404029+0000","last_change":"2023-10-16T20:36:55.287421+0000","last_active":"2023-10-16T20:37:37.404029+0000","last_peered":"2023-10-16T20:37:37.404029+0000","last_clean":"2023-10-16T20:37:37.404029+0000","last_became_active":"2023-10-16T20:36:55.286964+0000","last_became_peered":"2023-10-16T20:36:55.286964+0000","last_unstale":"2023-10-16T20:37:37.404029+0000","last_undegraded":"2023-10-16T20:37:37.404029+0000","last_fullsized":"2023-10-16T20:37:37.404029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:15:54.970100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.416921+0000","last_change":"2023-10-16T20:37:37.413891+0000","last_active":"2023-10-16T20:37:37.416921+0000","last_peered":"2023-10-16T20:37:37.416921+0000","last_clean":"2023-10-16T20:37:37.416921+0000","last_became_active":"2023-10-16T20:37:37.413492+0000","last_became_peered":"2023-10-16T20:37:37.413492+0000","last_unstale":"2023-10-16T20:37:37.416921+0000","last_undegraded":"2023-10-16T20:37:37.416921+0000","last_fullsized":"2023-10-16T20:37:37.416921+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:03:46.658556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407997+0000","last_change":"2023-10-16T20:36:55.303950+0000","last_active":"2023-10-16T20:37:37.407997+0000","last_peered":"2023-10-16T20:37:37.407997+0000","last_clean":"2023-10-16T20:37:37.407997+0000","last_became_active":"2023-10-16T20:36:55.302468+0000","last_became_peered":"2023-10-16T20:36:55.302468+0000","last_unstale":"2023-10-16T20:37:37.407997+0000","last_undegraded":"2023-10-16T20:37:37.407997+0000","last_fullsized":"2023-10-16T20:37:37.407997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:02.962031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466830+0000","last_change":"2023-10-16T20:36:55.297558+0000","last_active":"2023-10-16T20:37:37.466830+0000","last_peered":"2023-10-16T20:37:37.466830+0000","last_clean":"2023-10-16T20:37:37.466830+0000","last_became_active":"2023-10-16T20:36:55.296476+0000","last_became_peered":"2023-10-16T20:36:55.296476+0000","last_unstale":"2023-10-16T20:37:37.466830+0000","last_undegraded":"2023-10-16T20:37:37.466830+0000","last_fullsized":"2023-10-16T20:37:37.466830+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:04:38.235205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464724+0000","last_change":"2023-10-16T20:36:55.293957+0000","last_active":"2023-10-16T20:37:37.464724+0000","last_peered":"2023-10-16T20:37:37.464724+0000","last_clean":"2023-10-16T20:37:37.464724+0000","last_became_active":"2023-10-16T20:36:55.293687+0000","last_became_peered":"2023-10-16T20:36:55.293687+0000","last_unstale":"2023-10-16T20:37:37.464724+0000","last_undegraded":"2023-10-16T20:37:37.464724+0000","last_fullsized":"2023-10-16T20:37:37.464724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:54.623691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466003+0000","last_change":"2023-10-16T20:36:55.280728+0000","last_active":"2023-10-16T20:37:37.466003+0000","last_peered":"2023-10-16T20:37:37.466003+0000","last_clean":"2023-10-16T20:37:37.466003+0000","last_became_active":"2023-10-16T20:36:55.280444+0000","last_became_peered":"2023-10-16T20:36:55.280444+0000","last_unstale":"2023-10-16T20:37:37.466003+0000","last_undegraded":"2023-10-16T20:37:37.466003+0000","last_fullsized":"2023-10-16T20:37:37.466003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:44.044563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408158+0000","last_change":"2023-10-16T20:36:55.297802+0000","last_active":"2023-10-16T20:37:37.408158+0000","last_peered":"2023-10-16T20:37:37.408158+0000","last_clean":"2023-10-16T20:37:37.408158+0000","last_became_active":"2023-10-16T20:36:55.296694+0000","last_became_peered":"2023-10-16T20:36:55.296694+0000","last_unstale":"2023-10-16T20:37:37.408158+0000","last_undegraded":"2023-10-16T20:37:37.408158+0000","last_fullsized":"2023-10-16T20:37:37.408158+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:00:26.797387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466169+0000","last_change":"2023-10-16T20:36:55.291347+0000","last_active":"2023-10-16T20:37:37.466169+0000","last_peered":"2023-10-16T20:37:37.466169+0000","last_clean":"2023-10-16T20:37:37.466169+0000","last_became_active":"2023-10-16T20:36:55.291052+0000","last_became_peered":"2023-10-16T20:36:55.291052+0000","last_unstale":"2023-10-16T20:37:37.466169+0000","last_undegraded":"2023-10-16T20:37:37.466169+0000","last_fullsized":"2023-10-16T20:37:37.466169+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:40:51.331446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409006+0000","last_change":"2023-10-16T20:36:55.290870+0000","last_active":"2023-10-16T20:37:37.409006+0000","last_peered":"2023-10-16T20:37:37.409006+0000","last_clean":"2023-10-16T20:37:37.409006+0000","last_became_active":"2023-10-16T20:36:55.290386+0000","last_became_peered":"2023-10-16T20:36:55.290386+0000","last_unstale":"2023-10-16T20:37:37.409006+0000","last_undegraded":"2023-10-16T20:37:37.409006+0000","last_fullsized":"2023-10-16T20:37:37.409006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:20.006864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435199+0000","last_change":"2023-10-16T20:36:55.271041+0000","last_active":"2023-10-16T20:37:37.435199+0000","last_peered":"2023-10-16T20:37:37.435199+0000","last_clean":"2023-10-16T20:37:37.435199+0000","last_became_active":"2023-10-16T20:36:55.270787+0000","last_became_peered":"2023-10-16T20:36:55.270787+0000","last_unstale":"2023-10-16T20:37:37.435199+0000","last_undegraded":"2023-10-16T20:37:37.435199+0000","last_fullsized":"2023-10-16T20:37:37.435199+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:07.944778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402678+0000","last_change":"2023-10-16T20:36:55.271632+0000","last_active":"2023-10-16T20:37:37.402678+0000","last_peered":"2023-10-16T20:37:37.402678+0000","last_clean":"2023-10-16T20:37:37.402678+0000","last_became_active":"2023-10-16T20:36:55.271373+0000","last_became_peered":"2023-10-16T20:36:55.271373+0000","last_unstale":"2023-10-16T20:37:37.402678+0000","last_undegraded":"2023-10-16T20:37:37.402678+0000","last_fullsized":"2023-10-16T20:37:37.402678+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:07.458540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466741+0000","last_change":"2023-10-16T20:36:55.291343+0000","last_active":"2023-10-16T20:37:37.466741+0000","last_peered":"2023-10-16T20:37:37.466741+0000","last_clean":"2023-10-16T20:37:37.466741+0000","last_became_active":"2023-10-16T20:36:55.291016+0000","last_became_peered":"2023-10-16T20:36:55.291016+0000","last_unstale":"2023-10-16T20:37:37.466741+0000","last_undegraded":"2023-10-16T20:37:37.466741+0000","last_fullsized":"2023-10-16T20:37:37.466741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:13:40.287782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":12023,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":24046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":190,"ondisk_log_size":190,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509481,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":937984,"data_stored":717153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.75600000000000001,"15min":0.752},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":5.2240000000000002,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.046,"5min":0.81599999999999995,"15min":0.77300000000000002},"min":{"1min":0.47399999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":8.782,"5min":8.782,"15min":8.782},"last":0.64100000000000001}]},{"osd":1,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.73699999999999999,"15min":0.73799999999999999},"min":{"1min":0.49199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.91800000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.76300000000000001,"15min":0.74399999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":4.7300000000000004,"5min":4.7300000000000004,"15min":4.7300000000000004},"last":0.58999999999999997}]},{"osd":2,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.76900000000000002,"15min":0.73899999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":5.2629999999999999,"5min":5.2629999999999999,"15min":5.2629999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.77100000000000002,"15min":0.753},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.1589999999999998,"5min":5.1589999999999998,"15min":5.1589999999999998},"last":0.66900000000000004}]},{"osd":3,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.79900000000000004,"15min":0.77900000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":4.7460000000000004,"5min":4.7460000000000004,"15min":4.7460000000000004},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.78400000000000003,"15min":0.77000000000000002},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":4.7770000000000001,"5min":4.7770000000000001,"15min":4.7770000000000001},"last":0.80600000000000005}]},{"osd":4,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.61599999999999999,"15min":0.628},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.82099999999999995,"5min":1.3,"15min":1.3},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.58899999999999997,"15min":0.60499999999999998},"min":{"1min":0.375,"5min":0.373,"15min":0.373},"max":{"1min":0.77600000000000002,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.51200000000000001}]},{"osd":5,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68700000000000006,"15min":0.69899999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83799999999999997,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.76600000000000001,"15min":0.78800000000000003},"min":{"1min":0.57899999999999996,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.88600000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.746}]},{"osd":6,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.73499999999999999,"15min":0.75},"min":{"1min":0.51600000000000001,"5min":0.499,"15min":0.499},"max":{"1min":0.877,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.68799999999999994,"15min":0.70099999999999996},"min":{"1min":0.44500000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.85899999999999999,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.83699999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509481,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":129756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.747,"15min":0.76400000000000001},"min":{"1min":0.47399999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.74299999999999999,"15min":0.75600000000000001},"min":{"1min":0.51300000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.73499999999999999,"15min":0.75700000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.85399999999999998,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77000000000000002,"15min":0.77600000000000002},"min":{"1min":0.46300000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":3.387,"5min":3.387,"15min":3.387},"last":0.46500000000000002}]},{"osd":2,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.79000000000000004,"15min":0.79500000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":3.9489999999999998,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.755,"15min":0.751},"min":{"1min":0.53200000000000003,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":3.6179999999999999,"5min":3.6179999999999999,"15min":3.6179999999999999},"last":0.78100000000000003}]},{"osd":3,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.81899999999999995,"15min":0.85899999999999999},"min":{"1min":0.53700000000000003,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0009999999999999,"5min":1.611,"15min":1.611},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.79400000000000004,"15min":0.81399999999999995},"min":{"1min":0.56499999999999995,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.052,"5min":1.163,"15min":1.163},"last":0.75700000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.66000000000000003,"15min":0.67700000000000005},"min":{"1min":0.47399999999999998,"5min":0.443,"15min":0.443},"max":{"1min":0.93899999999999995,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.65600000000000003,"15min":0.66600000000000004},"min":{"1min":0.497,"5min":0.435,"15min":0.435},"max":{"1min":0.877,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.622}]},{"osd":5,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.46500000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.83699999999999997,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.69999999999999996,"15min":0.70599999999999996},"min":{"1min":0.52600000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.91300000000000003,"5min":1.016,"15min":1.016},"last":0.59899999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.73899999999999999,"15min":0.751},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.92000000000000004,"5min":1.292,"15min":1.292},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.748,"15min":0.77100000000000002},"min":{"1min":0.51900000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.90600000000000003,"5min":1.569,"15min":1.569},"last":0.69199999999999995}]}]},{"osd":5,"up_from":15,"seq":64424509483,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":356352,"data_stored":128617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70199999999999996,"15min":0.69599999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.88800000000000001,"5min":1.006,"15min":1.006},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.56899999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.93899999999999995,"5min":1.083,"15min":1.083},"last":0.64500000000000002}]},{"osd":1,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.71099999999999997,"15min":0.70599999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.96099999999999997,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71799999999999997,"15min":0.70999999999999996},"min":{"1min":0.53300000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.97799999999999998,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.73299999999999998}]},{"osd":2,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.90300000000000002,"5min":1.022,"15min":1.022},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.74199999999999999,"15min":0.73699999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.80300000000000005}]},{"osd":3,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78600000000000003,"15min":0.78900000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.97499999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.79500000000000004,"15min":0.81100000000000005},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.88800000000000001,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.66300000000000003}]},{"osd":4,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64400000000000002,"15min":0.65700000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.71099999999999997,"15min":0.751},"min":{"1min":0.496,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84099999999999997,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.77400000000000002}]},{"osd":6,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.69399999999999995,"15min":0.72299999999999998},"min":{"1min":0.46899999999999997,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.95099999999999996,"5min":1.069,"15min":1.069},"last":0.622},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.70999999999999996,"15min":0.72299999999999998},"min":{"1min":0.49199999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.84599999999999997,"5min":1.264,"15min":1.264},"last":0.83899999999999997}]},{"osd":7,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76200000000000001,"15min":0.79000000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.026,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.77200000000000002,"15min":0.80000000000000004},"min":{"1min":0.50900000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.96499999999999997,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.85599999999999998}]}]},{"osd":4,"up_from":15,"seq":64424509481,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":331776,"data_stored":126015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.72199999999999998,"15min":0.72399999999999998},"min":{"1min":0.45200000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.872,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.59599999999999997,"5min":0.5,"15min":0.5},"max":{"1min":0.88200000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.82099999999999995}]},{"osd":1,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.48999999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.74199999999999999,"15min":0.746},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.59499999999999997}]},{"osd":2,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.752,"15min":0.75800000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.66200000000000003}]},{"osd":3,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.75800000000000001,"15min":0.74199999999999999},"min":{"1min":0.54700000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.77000000000000002,"15min":0.78200000000000003},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":5,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.627,"15min":0.64500000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.82199999999999995,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.64700000000000002,"15min":0.65100000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.91700000000000004,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.73299999999999998}]},{"osd":6,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.71899999999999997,"15min":0.73299999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.90600000000000003,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.69099999999999995}]},{"osd":7,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.72799999999999998,"15min":0.745},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.94099999999999995,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.621},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.77800000000000002,"15min":0.79800000000000004},"min":{"1min":0.52800000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1080000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.85699999999999998}]}]},{"osd":3,"up_from":15,"seq":64424509481,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":129790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.503,"15min":0.48299999999999998},"min":{"1min":0.30099999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.3479999999999999,"5min":4.3479999999999999,"15min":4.3479999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.53700000000000003,"15min":0.51900000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":4.29,"5min":4.29,"15min":4.29},"last":0.40300000000000002}]},{"osd":1,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.52200000000000002,"15min":0.48799999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":4.5800000000000001,"5min":4.5800000000000001,"15min":4.5800000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.59699999999999998,"15min":0.56799999999999995},"min":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":4.7190000000000003,"5min":4.7190000000000003,"15min":4.7190000000000003},"last":0.48199999999999998}]},{"osd":2,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.56299999999999994,"15min":0.55200000000000005},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.3899999999999997,"5min":4.3899999999999997,"15min":4.3899999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.65900000000000003,"15min":0.628},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":0.44900000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.82799999999999996,"15min":0.80800000000000005},"min":{"1min":0.56799999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":4.0629999999999997,"5min":4.0629999999999997,"15min":4.0629999999999997},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77700000000000002,"15min":0.76600000000000001},"min":{"1min":0.56399999999999995,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.026,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.752}]},{"osd":5,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.86799999999999999,"15min":0.876},"min":{"1min":0.57699999999999996,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":3.9529999999999998,"5min":3.9529999999999998,"15min":3.9529999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.85399999999999998,"15min":0.83899999999999997},"min":{"1min":0.60599999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":4.4960000000000004,"5min":4.4960000000000004,"15min":4.4960000000000004},"last":0.76900000000000002}]},{"osd":6,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.76000000000000001,"15min":0.76800000000000002},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.86699999999999999,"15min":0.86199999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.81899999999999995}]},{"osd":7,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.83299999999999996,"15min":0.80900000000000005},"min":{"1min":0.56699999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":4.0049999999999999,"5min":4.0049999999999999,"15min":4.0049999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.749,"5min":0.79000000000000004,"15min":0.79800000000000004},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":0.93799999999999994,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.64600000000000002}]}]},{"osd":2,"up_from":15,"seq":64424509481,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":126147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.46700000000000003,"15min":0.47499999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70399999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.51900000000000002,"15min":0.52500000000000002},"min":{"1min":0.35899999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.68700000000000006,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.29699999999999999}]},{"osd":1,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.51500000000000001,"15min":0.52600000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.74299999999999999,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.36499999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.64700000000000002,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.47799999999999998}]},{"osd":3,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.80400000000000005,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57799999999999996,"15min":0.59299999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.74399999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.40400000000000003}]},{"osd":4,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.80900000000000005,"15min":0.82399999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.02,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.626},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.75900000000000001,"15min":0.76100000000000001},"min":{"1min":0.56200000000000006,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.92600000000000005,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.51600000000000001}]},{"osd":5,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.77200000000000002,"15min":0.78000000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.04,"5min":1.444,"15min":1.444},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.81799999999999995,"15min":0.83499999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.052,"5min":1.585,"15min":1.585},"last":0.52700000000000002}]},{"osd":6,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.82399999999999995,"15min":0.84399999999999997},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":0.96099999999999997,"5min":1.335,"15min":1.335},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.84099999999999997,"15min":0.872},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.036,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.63900000000000001}]},{"osd":7,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.83299999999999996,"15min":0.86399999999999999},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.93500000000000005,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.86099999999999999,"15min":0.88500000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.0820000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.59299999999999997}]}]},{"osd":1,"up_from":15,"seq":64424509481,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":125777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.45100000000000001,"15min":0.46300000000000002},"min":{"1min":0.29199999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69199999999999995,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.51500000000000001,"15min":0.52800000000000002},"min":{"1min":0.318,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.97899999999999998,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.52400000000000002,"15min":0.54400000000000004},"min":{"1min":0.32600000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66900000000000004,"5min":1.046,"15min":1.046},"last":0.377},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.55800000000000005,"15min":0.58199999999999996},"min":{"1min":0.33700000000000002,"5min":0.314,"15min":0.314},"max":{"1min":1.0740000000000001,"5min":1.115,"15min":1.115},"last":0.33700000000000002}]},{"osd":3,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.40600000000000003,"5min":0.312,"15min":0.312},"max":{"1min":1.155,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72699999999999998,"5min":1.431,"15min":1.431},"last":0.497}]},{"osd":4,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.81899999999999995,"15min":0.83099999999999996},"min":{"1min":0.56100000000000005,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.80700000000000005,"15min":0.80900000000000005},"min":{"1min":0.61199999999999999,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.115,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.628}]},{"osd":5,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.78900000000000003,"15min":0.81299999999999994},"min":{"1min":0.58399999999999996,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90200000000000002,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.84699999999999998,"15min":0.86399999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2490000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.59799999999999998}]},{"osd":6,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.84999999999999998,"15min":0.873},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.016,"5min":2.004,"15min":2.004},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.85399999999999998,"15min":0.876},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.3300000000000001,"5min":1.635,"15min":1.635},"last":0.67100000000000004}]},{"osd":7,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.85299999999999998,"15min":0.88300000000000001},"min":{"1min":0.58999999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0249999999999999,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.878,"15min":0.90300000000000002},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0489999999999999,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.70399999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343916,"statfs":{"total":96636764160,"available":96608169984,"internally_reserved":0,"allocated":929792,"data_stored":715487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.439,"15min":0.438},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.495,"15min":0.496},"min":{"1min":0.374,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.58999999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.36499999999999999}]},{"osd":2,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52300000000000002,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.433},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54100000000000004,"15min":0.54600000000000004},"min":{"1min":0.40400000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.68899999999999995,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.47499999999999998}]},{"osd":3,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54800000000000004,"15min":0.54500000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73399999999999999,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.56499999999999995,"5min":0.497,"15min":0.497},"max":{"1min":0.84999999999999998,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.49199999999999999}]},{"osd":4,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.83799999999999997,"15min":0.82999999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0600000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.85999999999999999,"15min":0.84899999999999998},"min":{"1min":0.61399999999999999,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.80800000000000005}]},{"osd":5,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.80800000000000005,"15min":0.80300000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.66800000000000004}]},{"osd":6,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.86299999999999999,"15min":0.85899999999999999},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.755},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88400000000000001,"15min":0.88300000000000001},"min":{"1min":0.60899999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.79400000000000004}]},{"osd":7,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.64900000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.83499999999999996,"15min":0.82299999999999995},"min":{"1min":0.66500000000000004,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T20:39:54.223 INFO:tasks.mds_thrash.ceph_manager:clean! 2023-10-16T20:39:54.223 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T20:39:54.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.388+0000 7f5542017640 1 Processor -- start 2023-10-16T20:39:54.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.392+0000 7f5542017640 1 -- start start 2023-10-16T20:39:54.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.392+0000 7f5542017640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f553c0c58d0 0x7f553c06a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.392+0000 7f5542017640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f553c06f810 0x7f553c06fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.392+0000 7f5542017640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c0c5e80 con 0x7f553c153670 2023-10-16T20:39:54.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.392+0000 7f5542017640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f553c0c6860 con 0x7f553c06f810 2023-10-16T20:39:54.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.392+0000 7f5542017640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553c0701a0 con 0x7f553c0c58d0 2023-10-16T20:39:54.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.392+0000 7f553bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f553c06f810 0x7f553c06fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:54.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.392+0000 7f553b7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f553c153670 0x7f553c06d170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58452/0 (socket says 172.21.15.88:58452) 2023-10-16T20:39:54.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.392+0000 7f553b7fe640 1 -- 172.21.15.88:0/401605745 learned_addr learned my addr 172.21.15.88:0/401605745 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:54.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.392+0000 7f553bfff640 1 --2- 172.21.15.88:0/401605745 >> v2:172.21.15.88:3300/0 conn(0x7f553c06f810 0x7f553c06fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47770/0 (socket says 172.21.15.88:47770) 2023-10-16T20:39:54.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.392+0000 7f553affd640 1 --2- 172.21.15.88:0/401605745 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f553c0c58d0 0x7f553c06a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:54.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.392+0000 7f553bfff640 1 -- 172.21.15.88:0/401605745 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f553c0c58d0 msgr2=0x7f553c06a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:54.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.392+0000 7f553bfff640 1 --2- 172.21.15.88:0/401605745 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f553c0c58d0 0x7f553c06a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.392+0000 7f553bfff640 1 -- 172.21.15.88:0/401605745 >> v1:172.21.15.88:6789/0 conn(0x7f553c153670 legacy=0x7f553c06d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.392+0000 7f553bfff640 1 -- 172.21.15.88:0/401605745 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553c15cb30 con 0x7f553c06f810 2023-10-16T20:39:54.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.392+0000 7f553a7fc640 1 -- 172.21.15.88:0/401605745 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173027178 0 0) 0x7f552c001120 con 0x7f553c153670 2023-10-16T20:39:54.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.396+0000 7f553bfff640 1 --2- 172.21.15.88:0/401605745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f553c06f810 0x7f553c06fc60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5534009d90 tx=0x7f553401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea1865b1a71afc11 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:54.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.396+0000 7f553a7fc640 1 -- 172.21.15.88:0/401605745 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55340a5810 con 0x7f553c06f810 2023-10-16T20:39:54.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.396+0000 7f553a7fc640 1 -- 172.21.15.88:0/401605745 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55340a5970 con 0x7f553c06f810 2023-10-16T20:39:54.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.396+0000 7f553a7fc640 1 -- 172.21.15.88:0/401605745 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55340a5d00 con 0x7f553c06f810 2023-10-16T20:39:54.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.396+0000 7f5542017640 1 -- 172.21.15.88:0/401605745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f553c06f810 msgr2=0x7f553c06fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.396+0000 7f5542017640 1 --2- 172.21.15.88:0/401605745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f553c06f810 0x7f553c06fc60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5534009d90 tx=0x7f553401c950 comp rx=0 tx=0).stop 2023-10-16T20:39:54.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.396+0000 7f5542017640 1 -- 172.21.15.88:0/401605745 shutdown_connections 2023-10-16T20:39:54.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.396+0000 7f5542017640 1 --2- 172.21.15.88:0/401605745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f553c06f810 0x7f553c06fc60 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.396+0000 7f5542017640 1 --2- 172.21.15.88:0/401605745 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f553c0c58d0 0x7f553c06a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.396+0000 7f5542017640 1 -- 172.21.15.88:0/401605745 >> 172.21.15.88:0/401605745 conn(0x7f553c0cad50 msgr2=0x7f553c0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:54.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.396+0000 7f5542017640 1 -- 172.21.15.88:0/401605745 shutdown_connections 2023-10-16T20:39:54.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.396+0000 7f5542017640 1 -- 172.21.15.88:0/401605745 wait complete. 2023-10-16T20:39:54.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.396+0000 7f5542017640 1 Processor -- start 2023-10-16T20:39:54.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.396+0000 7f5542017640 1 -- start start 2023-10-16T20:39:54.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f5542017640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f553c06f810 0x7f553c1581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f5542017640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f553c0c58d0 0x7f553c1f1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f5542017640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f553c153670 0x7f553c14ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f5542017640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c157800 con 0x7f553c0c58d0 2023-10-16T20:39:54.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f5542017640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c0c4090 con 0x7f553c06f810 2023-10-16T20:39:54.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f5542017640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553c0c3ad0 con 0x7f553c153670 2023-10-16T20:39:54.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f553affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f553c0c58d0 0x7f553c1f1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:54.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f553affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f553c0c58d0 0x7f553c1f1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47776/0 (socket says 172.21.15.88:47776) 2023-10-16T20:39:54.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f553affd640 1 -- 172.21.15.88:0/2934605150 learned_addr learned my addr 172.21.15.88:0/2934605150 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:54.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f553b7fe640 1 --2- 172.21.15.88:0/2934605150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f553c06f810 0x7f553c1581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:54.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f553bfff640 1 --2- 172.21.15.88:0/2934605150 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f553c153670 0x7f553c14ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:54.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f553affd640 1 -- 172.21.15.88:0/2934605150 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f553c153670 msgr2=0x7f553c14ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f553affd640 1 --2- 172.21.15.88:0/2934605150 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f553c153670 0x7f553c14ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f553affd640 1 -- 172.21.15.88:0/2934605150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f553c06f810 msgr2=0x7f553c1581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f553affd640 1 --2- 172.21.15.88:0/2934605150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f553c06f810 0x7f553c1581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f553affd640 1 -- 172.21.15.88:0/2934605150 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5534009a70 con 0x7f553c0c58d0 2023-10-16T20:39:54.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f553affd640 1 --2- 172.21.15.88:0/2934605150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f553c0c58d0 0x7f553c1f1860 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f553000d800 tx=0x7f553000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:54.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f5523fff640 1 -- 172.21.15.88:0/2934605150 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553000b6c0 con 0x7f553c0c58d0 2023-10-16T20:39:54.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f5523fff640 1 -- 172.21.15.88:0/2934605150 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553000b820 con 0x7f553c0c58d0 2023-10-16T20:39:54.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f5542017640 1 -- 172.21.15.88:0/2934605150 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f553c0c3db0 con 0x7f553c0c58d0 2023-10-16T20:39:54.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f5523fff640 1 -- 172.21.15.88:0/2934605150 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553000bb90 con 0x7f553c0c58d0 2023-10-16T20:39:54.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f5542017640 1 -- 172.21.15.88:0/2934605150 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f553c150d20 con 0x7f553c0c58d0 2023-10-16T20:39:54.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f5542017640 1 -- 172.21.15.88:0/2934605150 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f553c074700 con 0x7f553c0c58d0 2023-10-16T20:39:54.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.400+0000 7f5523fff640 1 -- 172.21.15.88:0/2934605150 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f553000bcf0 con 0x7f553c0c58d0 2023-10-16T20:39:54.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.404+0000 7f5523fff640 1 --2- 172.21.15.88:0/2934605150 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55280664a0 0x7f5528068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.404+0000 7f5523fff640 1 -- 172.21.15.88:0/2934605150 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5530084740 con 0x7f553c0c58d0 2023-10-16T20:39:54.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.404+0000 7f553b7fe640 1 --2- 172.21.15.88:0/2934605150 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55280664a0 0x7f5528068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:54.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.404+0000 7f553b7fe640 1 --2- 172.21.15.88:0/2934605150 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55280664a0 0x7f5528068960 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f552c007fd0 tx=0x7f552c0079c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:54.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.408+0000 7f5523fff640 1 -- 172.21.15.88:0/2934605150 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5530051890 con 0x7f553c0c58d0 2023-10-16T20:39:54.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.556+0000 7f5542017640 1 -- 172.21.15.88:0/2934605150 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f553c15d6b0 con 0x7f55280664a0 2023-10-16T20:39:54.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.564+0000 7f5523fff640 1 -- 172.21.15.88:0/2934605150 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+369924 (secure 0 0 0) 0x7f552c001450 con 0x7f55280664a0 2023-10-16T20:39:54.572 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:54.574 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-16T20:39:54.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.568+0000 7f5542017640 1 -- 172.21.15.88:0/2934605150 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55280664a0 msgr2=0x7f5528068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.568+0000 7f5542017640 1 --2- 172.21.15.88:0/2934605150 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55280664a0 0x7f5528068960 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f552c007fd0 tx=0x7f552c0079c0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.568+0000 7f5542017640 1 -- 172.21.15.88:0/2934605150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f553c0c58d0 msgr2=0x7f553c1f1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.568+0000 7f5542017640 1 --2- 172.21.15.88:0/2934605150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f553c0c58d0 0x7f553c1f1860 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f553000d800 tx=0x7f553000dcc0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.568+0000 7f5542017640 1 -- 172.21.15.88:0/2934605150 shutdown_connections 2023-10-16T20:39:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.568+0000 7f5542017640 1 --2- 172.21.15.88:0/2934605150 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55280664a0 0x7f5528068960 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.568+0000 7f5542017640 1 --2- 172.21.15.88:0/2934605150 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f553c153670 0x7f553c14ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.568+0000 7f5542017640 1 --2- 172.21.15.88:0/2934605150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f553c0c58d0 0x7f553c1f1860 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.568+0000 7f5542017640 1 --2- 172.21.15.88:0/2934605150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f553c06f810 0x7f553c1581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.568+0000 7f5542017640 1 -- 172.21.15.88:0/2934605150 >> 172.21.15.88:0/2934605150 conn(0x7f553c0cad50 msgr2=0x7f553c06d140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:54.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.568+0000 7f5542017640 1 -- 172.21.15.88:0/2934605150 shutdown_connections 2023-10-16T20:39:54.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.568+0000 7f5542017640 1 -- 172.21.15.88:0/2934605150 wait complete. 2023-10-16T20:39:54.591 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2023-10-16T20:39:53.635556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":602391,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":375,"num_write_kb":2490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307,"ondisk_log_size":307,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219944,"kb_used_data":3816,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754776,"statfs":{"total":773094113280,"available":772868890624,"internally_reserved":0,"allocated":3907584,"data_stored":2198742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3076,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002564"},"pg_stats":[{"pgid":"2.1f","version":"29'13","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.526948+0000","last_change":"2023-10-16T20:36:54.763744+0000","last_active":"2023-10-16T20:39:39.526948+0000","last_peered":"2023-10-16T20:39:39.526948+0000","last_clean":"2023-10-16T20:39:39.526948+0000","last_became_active":"2023-10-16T20:36:54.763457+0000","last_became_peered":"2023-10-16T20:36:54.763457+0000","last_unstale":"2023-10-16T20:39:39.526948+0000","last_undegraded":"2023-10-16T20:39:39.526948+0000","last_fullsized":"2023-10-16T20:39:39.526948+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:05.774485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465462+0000","last_change":"2023-10-16T20:36:55.302740+0000","last_active":"2023-10-16T20:37:37.465462+0000","last_peered":"2023-10-16T20:37:37.465462+0000","last_clean":"2023-10-16T20:37:37.465462+0000","last_became_active":"2023-10-16T20:36:55.302519+0000","last_became_peered":"2023-10-16T20:36:55.302519+0000","last_unstale":"2023-10-16T20:37:37.465462+0000","last_undegraded":"2023-10-16T20:37:37.465462+0000","last_fullsized":"2023-10-16T20:37:37.465462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:36.762521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464759+0000","last_change":"2023-10-16T20:36:59.466679+0000","last_active":"2023-10-16T20:37:37.464759+0000","last_peered":"2023-10-16T20:37:37.464759+0000","last_clean":"2023-10-16T20:37:37.464759+0000","last_became_active":"2023-10-16T20:36:59.466306+0000","last_became_peered":"2023-10-16T20:36:59.466306+0000","last_unstale":"2023-10-16T20:37:37.464759+0000","last_undegraded":"2023-10-16T20:37:37.464759+0000","last_fullsized":"2023-10-16T20:37:37.464759+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:14.874622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:38:57.624590+0000","last_change":"2023-10-16T20:36:54.759491+0000","last_active":"2023-10-16T20:38:57.624590+0000","last_peered":"2023-10-16T20:38:57.624590+0000","last_clean":"2023-10-16T20:38:57.624590+0000","last_became_active":"2023-10-16T20:36:54.759177+0000","last_became_peered":"2023-10-16T20:36:54.759177+0000","last_unstale":"2023-10-16T20:38:57.624590+0000","last_undegraded":"2023-10-16T20:38:57.624590+0000","last_fullsized":"2023-10-16T20:38:57.624590+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:43:36.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435464+0000","last_change":"2023-10-16T20:36:55.291745+0000","last_active":"2023-10-16T20:37:37.435464+0000","last_peered":"2023-10-16T20:37:37.435464+0000","last_clean":"2023-10-16T20:37:37.435464+0000","last_became_active":"2023-10-16T20:36:55.291458+0000","last_became_peered":"2023-10-16T20:36:55.291458+0000","last_unstale":"2023-10-16T20:37:37.435464+0000","last_undegraded":"2023-10-16T20:37:37.435464+0000","last_fullsized":"2023-10-16T20:37:37.435464+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:58:01.291335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402322+0000","last_change":"2023-10-16T20:36:59.303600+0000","last_active":"2023-10-16T20:37:37.402322+0000","last_peered":"2023-10-16T20:37:37.402322+0000","last_clean":"2023-10-16T20:37:37.402322+0000","last_became_active":"2023-10-16T20:36:59.302968+0000","last_became_peered":"2023-10-16T20:36:59.302968+0000","last_unstale":"2023-10-16T20:37:37.402322+0000","last_undegraded":"2023-10-16T20:37:37.402322+0000","last_fullsized":"2023-10-16T20:37:37.402322+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:11:28.322824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"29'12","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.529183+0000","last_change":"2023-10-16T20:36:54.762600+0000","last_active":"2023-10-16T20:39:39.529183+0000","last_peered":"2023-10-16T20:39:39.529183+0000","last_clean":"2023-10-16T20:39:39.529183+0000","last_became_active":"2023-10-16T20:36:54.762278+0000","last_became_peered":"2023-10-16T20:36:54.762278+0000","last_unstale":"2023-10-16T20:39:39.529183+0000","last_undegraded":"2023-10-16T20:39:39.529183+0000","last_fullsized":"2023-10-16T20:39:39.529183+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:46:03.836415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404637+0000","last_change":"2023-10-16T20:37:37.403448+0000","last_active":"2023-10-16T20:37:37.404637+0000","last_peered":"2023-10-16T20:37:37.404637+0000","last_clean":"2023-10-16T20:37:37.404637+0000","last_became_active":"2023-10-16T20:37:37.402987+0000","last_became_peered":"2023-10-16T20:37:37.402987+0000","last_unstale":"2023-10-16T20:37:37.404637+0000","last_undegraded":"2023-10-16T20:37:37.404637+0000","last_fullsized":"2023-10-16T20:37:37.404637+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:39:05.729682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402868+0000","last_change":"2023-10-16T20:36:59.304684+0000","last_active":"2023-10-16T20:37:37.402868+0000","last_peered":"2023-10-16T20:37:37.402868+0000","last_clean":"2023-10-16T20:37:37.402868+0000","last_became_active":"2023-10-16T20:36:59.303805+0000","last_became_peered":"2023-10-16T20:36:59.303805+0000","last_unstale":"2023-10-16T20:37:37.402868+0000","last_undegraded":"2023-10-16T20:37:37.402868+0000","last_fullsized":"2023-10-16T20:37:37.402868+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:39:54.406763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.526474+0000","last_change":"2023-10-16T20:36:54.749849+0000","last_active":"2023-10-16T20:39:39.526474+0000","last_peered":"2023-10-16T20:39:39.526474+0000","last_clean":"2023-10-16T20:39:39.526474+0000","last_became_active":"2023-10-16T20:36:54.749316+0000","last_became_peered":"2023-10-16T20:36:54.749316+0000","last_unstale":"2023-10-16T20:39:39.526474+0000","last_undegraded":"2023-10-16T20:39:39.526474+0000","last_fullsized":"2023-10-16T20:39:39.526474+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:49:29.077126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409527+0000","last_change":"2023-10-16T20:36:55.302838+0000","last_active":"2023-10-16T20:37:37.409527+0000","last_peered":"2023-10-16T20:37:37.409527+0000","last_clean":"2023-10-16T20:37:37.409527+0000","last_became_active":"2023-10-16T20:36:55.302387+0000","last_became_peered":"2023-10-16T20:36:55.302387+0000","last_unstale":"2023-10-16T20:37:37.409527+0000","last_undegraded":"2023-10-16T20:37:37.409527+0000","last_fullsized":"2023-10-16T20:37:37.409527+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:55:56.517950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403226+0000","last_change":"2023-10-16T20:36:59.305750+0000","last_active":"2023-10-16T20:37:37.403226+0000","last_peered":"2023-10-16T20:37:37.403226+0000","last_clean":"2023-10-16T20:37:37.403226+0000","last_became_active":"2023-10-16T20:36:59.305228+0000","last_became_peered":"2023-10-16T20:36:59.305228+0000","last_unstale":"2023-10-16T20:37:37.403226+0000","last_undegraded":"2023-10-16T20:37:37.403226+0000","last_fullsized":"2023-10-16T20:37:37.403226+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:00:18.194057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407634+0000","last_change":"2023-10-16T20:36:55.306712+0000","last_active":"2023-10-16T20:37:37.407634+0000","last_peered":"2023-10-16T20:37:37.407634+0000","last_clean":"2023-10-16T20:37:37.407634+0000","last_became_active":"2023-10-16T20:36:55.306427+0000","last_became_peered":"2023-10-16T20:36:55.306427+0000","last_unstale":"2023-10-16T20:37:37.407634+0000","last_undegraded":"2023-10-16T20:37:37.407634+0000","last_fullsized":"2023-10-16T20:37:37.407634+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:42:03.769890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.410476+0000","last_change":"2023-10-16T20:36:54.768375+0000","last_active":"2023-10-16T20:39:22.410476+0000","last_peered":"2023-10-16T20:39:22.410476+0000","last_clean":"2023-10-16T20:39:22.410476+0000","last_became_active":"2023-10-16T20:36:54.767996+0000","last_became_peered":"2023-10-16T20:36:54.767996+0000","last_unstale":"2023-10-16T20:39:22.410476+0000","last_undegraded":"2023-10-16T20:39:22.410476+0000","last_fullsized":"2023-10-16T20:39:22.410476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:55.303942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404253+0000","last_change":"2023-10-16T20:36:59.307130+0000","last_active":"2023-10-16T20:37:37.404253+0000","last_peered":"2023-10-16T20:37:37.404253+0000","last_clean":"2023-10-16T20:37:37.404253+0000","last_became_active":"2023-10-16T20:36:59.306640+0000","last_became_peered":"2023-10-16T20:36:59.306640+0000","last_unstale":"2023-10-16T20:37:37.404253+0000","last_undegraded":"2023-10-16T20:37:37.404253+0000","last_fullsized":"2023-10-16T20:37:37.404253+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:19:18.664630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465074+0000","last_change":"2023-10-16T20:36:55.275436+0000","last_active":"2023-10-16T20:37:37.465074+0000","last_peered":"2023-10-16T20:37:37.465074+0000","last_clean":"2023-10-16T20:37:37.465074+0000","last_became_active":"2023-10-16T20:36:55.275085+0000","last_became_peered":"2023-10-16T20:36:55.275085+0000","last_unstale":"2023-10-16T20:37:37.465074+0000","last_undegraded":"2023-10-16T20:37:37.465074+0000","last_fullsized":"2023-10-16T20:37:37.465074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:37:37.814511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.502271+0000","last_change":"2023-10-16T20:36:54.756174+0000","last_active":"2023-10-16T20:39:33.502271+0000","last_peered":"2023-10-16T20:39:33.502271+0000","last_clean":"2023-10-16T20:39:33.502271+0000","last_became_active":"2023-10-16T20:36:54.755872+0000","last_became_peered":"2023-10-16T20:36:54.755872+0000","last_unstale":"2023-10-16T20:39:33.502271+0000","last_undegraded":"2023-10-16T20:39:33.502271+0000","last_fullsized":"2023-10-16T20:39:33.502271+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:32.741671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403750+0000","last_change":"2023-10-16T20:36:59.303855+0000","last_active":"2023-10-16T20:37:37.403750+0000","last_peered":"2023-10-16T20:37:37.403750+0000","last_clean":"2023-10-16T20:37:37.403750+0000","last_became_active":"2023-10-16T20:36:59.302919+0000","last_became_peered":"2023-10-16T20:36:59.302919+0000","last_unstale":"2023-10-16T20:37:37.403750+0000","last_undegraded":"2023-10-16T20:37:37.403750+0000","last_fullsized":"2023-10-16T20:37:37.403750+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:38:02.591237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464820+0000","last_change":"2023-10-16T20:36:55.270717+0000","last_active":"2023-10-16T20:37:37.464820+0000","last_peered":"2023-10-16T20:37:37.464820+0000","last_clean":"2023-10-16T20:37:37.464820+0000","last_became_active":"2023-10-16T20:36:55.270455+0000","last_became_peered":"2023-10-16T20:36:55.270455+0000","last_unstale":"2023-10-16T20:37:37.464820+0000","last_undegraded":"2023-10-16T20:37:37.464820+0000","last_fullsized":"2023-10-16T20:37:37.464820+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:16:44.485783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"29'7","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.524872+0000","last_change":"2023-10-16T20:36:54.744128+0000","last_active":"2023-10-16T20:39:39.524872+0000","last_peered":"2023-10-16T20:39:39.524872+0000","last_clean":"2023-10-16T20:39:39.524872+0000","last_became_active":"2023-10-16T20:36:54.743805+0000","last_became_peered":"2023-10-16T20:36:54.743805+0000","last_unstale":"2023-10-16T20:39:39.524872+0000","last_undegraded":"2023-10-16T20:39:39.524872+0000","last_fullsized":"2023-10-16T20:39:39.524872+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:54:35.819820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402668+0000","last_change":"2023-10-16T20:36:59.465650+0000","last_active":"2023-10-16T20:37:37.402668+0000","last_peered":"2023-10-16T20:37:37.402668+0000","last_clean":"2023-10-16T20:37:37.402668+0000","last_became_active":"2023-10-16T20:36:59.465172+0000","last_became_peered":"2023-10-16T20:36:59.465172+0000","last_unstale":"2023-10-16T20:37:37.402668+0000","last_undegraded":"2023-10-16T20:37:37.402668+0000","last_fullsized":"2023-10-16T20:37:37.402668+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:12.845517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403561+0000","last_change":"2023-10-16T20:36:55.278974+0000","last_active":"2023-10-16T20:37:37.403561+0000","last_peered":"2023-10-16T20:37:37.403561+0000","last_clean":"2023-10-16T20:37:37.403561+0000","last_became_active":"2023-10-16T20:36:55.278568+0000","last_became_peered":"2023-10-16T20:36:55.278568+0000","last_unstale":"2023-10-16T20:37:37.403561+0000","last_undegraded":"2023-10-16T20:37:37.403561+0000","last_fullsized":"2023-10-16T20:37:37.403561+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:26.756114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.407659+0000","last_change":"2023-10-16T20:36:54.758778+0000","last_active":"2023-10-16T20:39:22.407659+0000","last_peered":"2023-10-16T20:39:22.407659+0000","last_clean":"2023-10-16T20:39:22.407659+0000","last_became_active":"2023-10-16T20:36:54.758496+0000","last_became_peered":"2023-10-16T20:36:54.758496+0000","last_unstale":"2023-10-16T20:39:22.407659+0000","last_undegraded":"2023-10-16T20:39:22.407659+0000","last_fullsized":"2023-10-16T20:39:22.407659+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:36:00.515106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404277+0000","last_change":"2023-10-16T20:36:59.309382+0000","last_active":"2023-10-16T20:37:37.404277+0000","last_peered":"2023-10-16T20:37:37.404277+0000","last_clean":"2023-10-16T20:37:37.404277+0000","last_became_active":"2023-10-16T20:36:59.309068+0000","last_became_peered":"2023-10-16T20:36:59.309068+0000","last_unstale":"2023-10-16T20:37:37.404277+0000","last_undegraded":"2023-10-16T20:37:37.404277+0000","last_fullsized":"2023-10-16T20:37:37.404277+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:46:02.601985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435429+0000","last_change":"2023-10-16T20:36:55.273089+0000","last_active":"2023-10-16T20:37:37.435429+0000","last_peered":"2023-10-16T20:37:37.435429+0000","last_clean":"2023-10-16T20:37:37.435429+0000","last_became_active":"2023-10-16T20:36:55.272840+0000","last_became_peered":"2023-10-16T20:36:55.272840+0000","last_unstale":"2023-10-16T20:37:37.435429+0000","last_undegraded":"2023-10-16T20:37:37.435429+0000","last_fullsized":"2023-10-16T20:37:37.435429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:37.330924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.530212+0000","last_change":"2023-10-16T20:36:54.760681+0000","last_active":"2023-10-16T20:39:39.530212+0000","last_peered":"2023-10-16T20:39:39.530212+0000","last_clean":"2023-10-16T20:39:39.530212+0000","last_became_active":"2023-10-16T20:36:54.760397+0000","last_became_peered":"2023-10-16T20:36:54.760397+0000","last_unstale":"2023-10-16T20:39:39.530212+0000","last_undegraded":"2023-10-16T20:39:39.530212+0000","last_fullsized":"2023-10-16T20:39:39.530212+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:47:16.510382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403318+0000","last_change":"2023-10-16T20:36:59.465626+0000","last_active":"2023-10-16T20:37:37.403318+0000","last_peered":"2023-10-16T20:37:37.403318+0000","last_clean":"2023-10-16T20:37:37.403318+0000","last_became_active":"2023-10-16T20:36:59.465152+0000","last_became_peered":"2023-10-16T20:36:59.465152+0000","last_unstale":"2023-10-16T20:37:37.403318+0000","last_undegraded":"2023-10-16T20:37:37.403318+0000","last_fullsized":"2023-10-16T20:37:37.403318+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:27:51.088333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465673+0000","last_change":"2023-10-16T20:36:55.285796+0000","last_active":"2023-10-16T20:37:37.465673+0000","last_peered":"2023-10-16T20:37:37.465673+0000","last_clean":"2023-10-16T20:37:37.465673+0000","last_became_active":"2023-10-16T20:36:55.285566+0000","last_became_peered":"2023-10-16T20:36:55.285566+0000","last_unstale":"2023-10-16T20:37:37.465673+0000","last_undegraded":"2023-10-16T20:37:37.465673+0000","last_fullsized":"2023-10-16T20:37:37.465673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:22:21.950705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.503684+0000","last_change":"2023-10-16T20:36:54.758062+0000","last_active":"2023-10-16T20:39:33.503684+0000","last_peered":"2023-10-16T20:39:33.503684+0000","last_clean":"2023-10-16T20:39:33.503684+0000","last_became_active":"2023-10-16T20:36:54.757796+0000","last_became_peered":"2023-10-16T20:36:54.757796+0000","last_unstale":"2023-10-16T20:39:33.503684+0000","last_undegraded":"2023-10-16T20:39:33.503684+0000","last_fullsized":"2023-10-16T20:39:33.503684+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:02:09.953795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464463+0000","last_change":"2023-10-16T20:36:59.468529+0000","last_active":"2023-10-16T20:37:37.464463+0000","last_peered":"2023-10-16T20:37:37.464463+0000","last_clean":"2023-10-16T20:37:37.464463+0000","last_became_active":"2023-10-16T20:36:59.465221+0000","last_became_peered":"2023-10-16T20:36:59.465221+0000","last_unstale":"2023-10-16T20:37:37.464463+0000","last_undegraded":"2023-10-16T20:37:37.464463+0000","last_fullsized":"2023-10-16T20:37:37.464463+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:58:08.457686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401257+0000","last_change":"2023-10-16T20:36:55.275759+0000","last_active":"2023-10-16T20:37:37.401257+0000","last_peered":"2023-10-16T20:37:37.401257+0000","last_clean":"2023-10-16T20:37:37.401257+0000","last_became_active":"2023-10-16T20:36:55.274432+0000","last_became_peered":"2023-10-16T20:36:55.274432+0000","last_unstale":"2023-10-16T20:37:37.401257+0000","last_undegraded":"2023-10-16T20:37:37.401257+0000","last_fullsized":"2023-10-16T20:37:37.401257+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:53:08.896731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.508458+0000","last_change":"2023-10-16T20:36:54.762356+0000","last_active":"2023-10-16T20:39:33.508458+0000","last_peered":"2023-10-16T20:39:33.508458+0000","last_clean":"2023-10-16T20:39:33.508458+0000","last_became_active":"2023-10-16T20:36:54.762167+0000","last_became_peered":"2023-10-16T20:36:54.762167+0000","last_unstale":"2023-10-16T20:39:33.508458+0000","last_undegraded":"2023-10-16T20:39:33.508458+0000","last_fullsized":"2023-10-16T20:39:33.508458+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:45.164625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434946+0000","last_change":"2023-10-16T20:36:59.467492+0000","last_active":"2023-10-16T20:37:37.434946+0000","last_peered":"2023-10-16T20:37:37.434946+0000","last_clean":"2023-10-16T20:37:37.434946+0000","last_became_active":"2023-10-16T20:36:59.466876+0000","last_became_peered":"2023-10-16T20:36:59.466876+0000","last_unstale":"2023-10-16T20:37:37.434946+0000","last_undegraded":"2023-10-16T20:37:37.434946+0000","last_fullsized":"2023-10-16T20:37:37.434946+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:29:01.292167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409570+0000","last_change":"2023-10-16T20:36:55.293537+0000","last_active":"2023-10-16T20:37:37.409570+0000","last_peered":"2023-10-16T20:37:37.409570+0000","last_clean":"2023-10-16T20:37:37.409570+0000","last_became_active":"2023-10-16T20:36:55.293228+0000","last_became_peered":"2023-10-16T20:36:55.293228+0000","last_unstale":"2023-10-16T20:37:37.409570+0000","last_undegraded":"2023-10-16T20:37:37.409570+0000","last_fullsized":"2023-10-16T20:37:37.409570+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:50:13.226645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:02.624662+0000","last_change":"2023-10-16T20:36:54.757588+0000","last_active":"2023-10-16T20:39:02.624662+0000","last_peered":"2023-10-16T20:39:02.624662+0000","last_clean":"2023-10-16T20:39:02.624662+0000","last_became_active":"2023-10-16T20:36:54.757308+0000","last_became_peered":"2023-10-16T20:36:54.757308+0000","last_unstale":"2023-10-16T20:39:02.624662+0000","last_undegraded":"2023-10-16T20:39:02.624662+0000","last_fullsized":"2023-10-16T20:39:02.624662+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:51.298239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402099+0000","last_change":"2023-10-16T20:36:59.303844+0000","last_active":"2023-10-16T20:37:37.402099+0000","last_peered":"2023-10-16T20:37:37.402099+0000","last_clean":"2023-10-16T20:37:37.402099+0000","last_became_active":"2023-10-16T20:36:59.303224+0000","last_became_peered":"2023-10-16T20:36:59.303224+0000","last_unstale":"2023-10-16T20:37:37.402099+0000","last_undegraded":"2023-10-16T20:37:37.402099+0000","last_fullsized":"2023-10-16T20:37:37.402099+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:52.447785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403798+0000","last_change":"2023-10-16T20:36:55.278310+0000","last_active":"2023-10-16T20:37:37.403798+0000","last_peered":"2023-10-16T20:37:37.403798+0000","last_clean":"2023-10-16T20:37:37.403798+0000","last_became_active":"2023-10-16T20:36:55.277872+0000","last_became_peered":"2023-10-16T20:36:55.277872+0000","last_unstale":"2023-10-16T20:37:37.403798+0000","last_undegraded":"2023-10-16T20:37:37.403798+0000","last_fullsized":"2023-10-16T20:37:37.403798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:08:37.320469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.527106+0000","last_change":"2023-10-16T20:36:54.740644+0000","last_active":"2023-10-16T20:39:39.527106+0000","last_peered":"2023-10-16T20:39:39.527106+0000","last_clean":"2023-10-16T20:39:39.527106+0000","last_became_active":"2023-10-16T20:36:54.740340+0000","last_became_peered":"2023-10-16T20:36:54.740340+0000","last_unstale":"2023-10-16T20:39:39.527106+0000","last_undegraded":"2023-10-16T20:39:39.527106+0000","last_fullsized":"2023-10-16T20:39:39.527106+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:22.221727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401155+0000","last_change":"2023-10-16T20:36:59.294914+0000","last_active":"2023-10-16T20:37:37.401155+0000","last_peered":"2023-10-16T20:37:37.401155+0000","last_clean":"2023-10-16T20:37:37.401155+0000","last_became_active":"2023-10-16T20:36:59.294548+0000","last_became_peered":"2023-10-16T20:36:59.294548+0000","last_unstale":"2023-10-16T20:37:37.401155+0000","last_undegraded":"2023-10-16T20:37:37.401155+0000","last_fullsized":"2023-10-16T20:37:37.401155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:52:15.265262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401383+0000","last_change":"2023-10-16T20:36:55.270992+0000","last_active":"2023-10-16T20:37:37.401383+0000","last_peered":"2023-10-16T20:37:37.401383+0000","last_clean":"2023-10-16T20:37:37.401383+0000","last_became_active":"2023-10-16T20:36:55.269749+0000","last_became_peered":"2023-10-16T20:36:55.269749+0000","last_unstale":"2023-10-16T20:37:37.401383+0000","last_undegraded":"2023-10-16T20:37:37.401383+0000","last_fullsized":"2023-10-16T20:37:37.401383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:47:30.925314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'6","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.522848+0000","last_change":"2023-10-16T20:36:54.757903+0000","last_active":"2023-10-16T20:39:39.522848+0000","last_peered":"2023-10-16T20:39:39.522848+0000","last_clean":"2023-10-16T20:39:39.522848+0000","last_became_active":"2023-10-16T20:36:54.757549+0000","last_became_peered":"2023-10-16T20:36:54.757549+0000","last_unstale":"2023-10-16T20:39:39.522848+0000","last_undegraded":"2023-10-16T20:39:39.522848+0000","last_fullsized":"2023-10-16T20:39:39.522848+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:02:42.090848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402786+0000","last_change":"2023-10-16T20:36:59.302063+0000","last_active":"2023-10-16T20:37:37.402786+0000","last_peered":"2023-10-16T20:37:37.402786+0000","last_clean":"2023-10-16T20:37:37.402786+0000","last_became_active":"2023-10-16T20:36:59.301693+0000","last_became_peered":"2023-10-16T20:36:59.301693+0000","last_unstale":"2023-10-16T20:37:37.402786+0000","last_undegraded":"2023-10-16T20:37:37.402786+0000","last_fullsized":"2023-10-16T20:37:37.402786+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:29:44.812282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434713+0000","last_change":"2023-10-16T20:36:55.278132+0000","last_active":"2023-10-16T20:37:37.434713+0000","last_peered":"2023-10-16T20:37:37.434713+0000","last_clean":"2023-10-16T20:37:37.434713+0000","last_became_active":"2023-10-16T20:36:55.276649+0000","last_became_peered":"2023-10-16T20:36:55.276649+0000","last_unstale":"2023-10-16T20:37:37.434713+0000","last_undegraded":"2023-10-16T20:37:37.434713+0000","last_fullsized":"2023-10-16T20:37:37.434713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:30:53.299514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:28.448344+0000","last_change":"2023-10-16T20:36:54.753592+0000","last_active":"2023-10-16T20:39:28.448344+0000","last_peered":"2023-10-16T20:39:28.448344+0000","last_clean":"2023-10-16T20:39:28.448344+0000","last_became_active":"2023-10-16T20:36:54.753237+0000","last_became_peered":"2023-10-16T20:36:54.753237+0000","last_unstale":"2023-10-16T20:39:28.448344+0000","last_undegraded":"2023-10-16T20:39:28.448344+0000","last_fullsized":"2023-10-16T20:39:28.448344+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:53.005979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435360+0000","last_change":"2023-10-16T20:36:59.468431+0000","last_active":"2023-10-16T20:37:37.435360+0000","last_peered":"2023-10-16T20:37:37.435360+0000","last_clean":"2023-10-16T20:37:37.435360+0000","last_became_active":"2023-10-16T20:36:59.466708+0000","last_became_peered":"2023-10-16T20:36:59.466708+0000","last_unstale":"2023-10-16T20:37:37.435360+0000","last_undegraded":"2023-10-16T20:37:37.435360+0000","last_fullsized":"2023-10-16T20:37:37.435360+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:42:20.607519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464817+0000","last_change":"2023-10-16T20:36:55.294825+0000","last_active":"2023-10-16T20:37:37.464817+0000","last_peered":"2023-10-16T20:37:37.464817+0000","last_clean":"2023-10-16T20:37:37.464817+0000","last_became_active":"2023-10-16T20:36:55.294610+0000","last_became_peered":"2023-10-16T20:36:55.294610+0000","last_unstale":"2023-10-16T20:37:37.464817+0000","last_undegraded":"2023-10-16T20:37:37.464817+0000","last_fullsized":"2023-10-16T20:37:37.464817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:43.442939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.507413+0000","last_change":"2023-10-16T20:36:54.741191+0000","last_active":"2023-10-16T20:39:33.507413+0000","last_peered":"2023-10-16T20:39:33.507413+0000","last_clean":"2023-10-16T20:39:33.507413+0000","last_became_active":"2023-10-16T20:36:54.740944+0000","last_became_peered":"2023-10-16T20:36:54.740944+0000","last_unstale":"2023-10-16T20:39:33.507413+0000","last_undegraded":"2023-10-16T20:39:33.507413+0000","last_fullsized":"2023-10-16T20:39:33.507413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:05.961850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":172,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466038+0000","last_change":"2023-10-16T20:36:46.390001+0000","last_active":"2023-10-16T20:37:37.466038+0000","last_peered":"2023-10-16T20:37:37.466038+0000","last_clean":"2023-10-16T20:37:37.466038+0000","last_became_active":"2023-10-16T20:36:46.388824+0000","last_became_peered":"2023-10-16T20:36:46.388824+0000","last_unstale":"2023-10-16T20:37:37.466038+0000","last_undegraded":"2023-10-16T20:37:37.466038+0000","last_fullsized":"2023-10-16T20:37:37.466038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_clean_scrub_stamp":"2023-10-16T20:36:44.575506+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:31.004318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407653+0000","last_change":"2023-10-16T20:36:59.468528+0000","last_active":"2023-10-16T20:37:37.407653+0000","last_peered":"2023-10-16T20:37:37.407653+0000","last_clean":"2023-10-16T20:37:37.407653+0000","last_became_active":"2023-10-16T20:36:59.468082+0000","last_became_peered":"2023-10-16T20:36:59.468082+0000","last_unstale":"2023-10-16T20:37:37.407653+0000","last_undegraded":"2023-10-16T20:37:37.407653+0000","last_fullsized":"2023-10-16T20:37:37.407653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:00.920004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.613641+0000","last_change":"2023-10-16T20:37:37.409578+0000","last_active":"2023-10-16T20:37:37.613641+0000","last_peered":"2023-10-16T20:37:37.613641+0000","last_clean":"2023-10-16T20:37:37.613641+0000","last_became_active":"2023-10-16T20:37:37.409004+0000","last_became_peered":"2023-10-16T20:37:37.409004+0000","last_unstale":"2023-10-16T20:37:37.613641+0000","last_undegraded":"2023-10-16T20:37:37.613641+0000","last_fullsized":"2023-10-16T20:37:37.613641+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:28:57.364794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"29'6","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.507117+0000","last_change":"2023-10-16T20:36:54.746600+0000","last_active":"2023-10-16T20:39:33.507117+0000","last_peered":"2023-10-16T20:39:33.507117+0000","last_clean":"2023-10-16T20:39:33.507117+0000","last_became_active":"2023-10-16T20:36:54.746247+0000","last_became_peered":"2023-10-16T20:36:54.746247+0000","last_unstale":"2023-10-16T20:39:33.507117+0000","last_undegraded":"2023-10-16T20:39:33.507117+0000","last_fullsized":"2023-10-16T20:39:33.507117+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:28:57.949487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1610,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.470531+0000","last_change":"2023-10-16T20:37:37.469427+0000","last_active":"2023-10-16T20:37:37.470531+0000","last_peered":"2023-10-16T20:37:37.470531+0000","last_clean":"2023-10-16T20:37:37.470531+0000","last_became_active":"2023-10-16T20:37:37.467402+0000","last_became_peered":"2023-10-16T20:37:37.467402+0000","last_unstale":"2023-10-16T20:37:37.470531+0000","last_undegraded":"2023-10-16T20:37:37.470531+0000","last_fullsized":"2023-10-16T20:37:37.470531+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:00:19.171972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409113+0000","last_change":"2023-10-16T20:36:55.301234+0000","last_active":"2023-10-16T20:37:37.409113+0000","last_peered":"2023-10-16T20:37:37.409113+0000","last_clean":"2023-10-16T20:37:37.409113+0000","last_became_active":"2023-10-16T20:36:55.300958+0000","last_became_peered":"2023-10-16T20:36:55.300958+0000","last_unstale":"2023-10-16T20:37:37.409113+0000","last_undegraded":"2023-10-16T20:37:37.409113+0000","last_fullsized":"2023-10-16T20:37:37.409113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:16.784124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.506025+0000","last_change":"2023-10-16T20:36:54.753770+0000","last_active":"2023-10-16T20:39:33.506025+0000","last_peered":"2023-10-16T20:39:33.506025+0000","last_clean":"2023-10-16T20:39:33.506025+0000","last_became_active":"2023-10-16T20:36:54.753443+0000","last_became_peered":"2023-10-16T20:36:54.753443+0000","last_unstale":"2023-10-16T20:39:33.506025+0000","last_undegraded":"2023-10-16T20:39:33.506025+0000","last_fullsized":"2023-10-16T20:39:33.506025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:21:26.392480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435046+0000","last_change":"2023-10-16T20:36:59.303351+0000","last_active":"2023-10-16T20:37:37.435046+0000","last_peered":"2023-10-16T20:37:37.435046+0000","last_clean":"2023-10-16T20:37:37.435046+0000","last_became_active":"2023-10-16T20:36:59.302869+0000","last_became_peered":"2023-10-16T20:36:59.302869+0000","last_unstale":"2023-10-16T20:37:37.435046+0000","last_undegraded":"2023-10-16T20:37:37.435046+0000","last_fullsized":"2023-10-16T20:37:37.435046+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:26:35.886399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466830+0000","last_change":"2023-10-16T20:36:55.291947+0000","last_active":"2023-10-16T20:37:37.466830+0000","last_peered":"2023-10-16T20:37:37.466830+0000","last_clean":"2023-10-16T20:37:37.466830+0000","last_became_active":"2023-10-16T20:36:55.291633+0000","last_became_peered":"2023-10-16T20:36:55.291633+0000","last_unstale":"2023-10-16T20:37:37.466830+0000","last_undegraded":"2023-10-16T20:37:37.466830+0000","last_fullsized":"2023-10-16T20:37:37.466830+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:00:14.349339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.519506+0000","last_change":"2023-10-16T20:36:54.748196+0000","last_active":"2023-10-16T20:39:39.519506+0000","last_peered":"2023-10-16T20:39:39.519506+0000","last_clean":"2023-10-16T20:39:39.519506+0000","last_became_active":"2023-10-16T20:36:54.747704+0000","last_became_peered":"2023-10-16T20:36:54.747704+0000","last_unstale":"2023-10-16T20:39:39.519506+0000","last_undegraded":"2023-10-16T20:39:39.519506+0000","last_fullsized":"2023-10-16T20:39:39.519506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:55.471916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410236+0000","last_change":"2023-10-16T20:36:59.470368+0000","last_active":"2023-10-16T20:37:37.410236+0000","last_peered":"2023-10-16T20:37:37.410236+0000","last_clean":"2023-10-16T20:37:37.410236+0000","last_became_active":"2023-10-16T20:36:59.467291+0000","last_became_peered":"2023-10-16T20:36:59.467291+0000","last_unstale":"2023-10-16T20:37:37.410236+0000","last_undegraded":"2023-10-16T20:37:37.410236+0000","last_fullsized":"2023-10-16T20:37:37.410236+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:44:18.312394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466658+0000","last_change":"2023-10-16T20:36:55.291451+0000","last_active":"2023-10-16T20:37:37.466658+0000","last_peered":"2023-10-16T20:37:37.466658+0000","last_clean":"2023-10-16T20:37:37.466658+0000","last_became_active":"2023-10-16T20:36:55.290998+0000","last_became_peered":"2023-10-16T20:36:55.290998+0000","last_unstale":"2023-10-16T20:37:37.466658+0000","last_undegraded":"2023-10-16T20:37:37.466658+0000","last_fullsized":"2023-10-16T20:37:37.466658+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:05:01.634971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"29'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:12.629039+0000","last_change":"2023-10-16T20:36:54.763818+0000","last_active":"2023-10-16T20:39:12.629039+0000","last_peered":"2023-10-16T20:39:12.629039+0000","last_clean":"2023-10-16T20:39:12.629039+0000","last_became_active":"2023-10-16T20:36:54.761337+0000","last_became_peered":"2023-10-16T20:36:54.761337+0000","last_unstale":"2023-10-16T20:39:12.629039+0000","last_undegraded":"2023-10-16T20:39:12.629039+0000","last_fullsized":"2023-10-16T20:39:12.629039+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:03.093189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.470301+0000","last_change":"2023-10-16T20:37:37.468803+0000","last_active":"2023-10-16T20:37:37.470301+0000","last_peered":"2023-10-16T20:37:37.470301+0000","last_clean":"2023-10-16T20:37:37.470301+0000","last_became_active":"2023-10-16T20:37:37.468309+0000","last_became_peered":"2023-10-16T20:37:37.468309+0000","last_unstale":"2023-10-16T20:37:37.470301+0000","last_undegraded":"2023-10-16T20:37:37.470301+0000","last_fullsized":"2023-10-16T20:37:37.470301+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:23:11.761661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466101+0000","last_change":"2023-10-16T20:36:55.288895+0000","last_active":"2023-10-16T20:37:37.466101+0000","last_peered":"2023-10-16T20:37:37.466101+0000","last_clean":"2023-10-16T20:37:37.466101+0000","last_became_active":"2023-10-16T20:36:55.288354+0000","last_became_peered":"2023-10-16T20:36:55.288354+0000","last_unstale":"2023-10-16T20:37:37.466101+0000","last_undegraded":"2023-10-16T20:37:37.466101+0000","last_fullsized":"2023-10-16T20:37:37.466101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:53:29.192406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'17","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.501703+0000","last_change":"2023-10-16T20:36:54.738216+0000","last_active":"2023-10-16T20:39:33.501703+0000","last_peered":"2023-10-16T20:39:33.501703+0000","last_clean":"2023-10-16T20:39:33.501703+0000","last_became_active":"2023-10-16T20:36:54.737876+0000","last_became_peered":"2023-10-16T20:36:54.737876+0000","last_unstale":"2023-10-16T20:39:33.501703+0000","last_undegraded":"2023-10-16T20:39:33.501703+0000","last_fullsized":"2023-10-16T20:39:33.501703+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:28.214601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435178+0000","last_change":"2023-10-16T20:36:59.307215+0000","last_active":"2023-10-16T20:37:37.435178+0000","last_peered":"2023-10-16T20:37:37.435178+0000","last_clean":"2023-10-16T20:37:37.435178+0000","last_became_active":"2023-10-16T20:36:59.306801+0000","last_became_peered":"2023-10-16T20:36:59.306801+0000","last_unstale":"2023-10-16T20:37:37.435178+0000","last_undegraded":"2023-10-16T20:37:37.435178+0000","last_fullsized":"2023-10-16T20:37:37.435178+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:41:42.656134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407054+0000","last_change":"2023-10-16T20:36:55.287714+0000","last_active":"2023-10-16T20:37:37.407054+0000","last_peered":"2023-10-16T20:37:37.407054+0000","last_clean":"2023-10-16T20:37:37.407054+0000","last_became_active":"2023-10-16T20:36:55.287066+0000","last_became_peered":"2023-10-16T20:36:55.287066+0000","last_unstale":"2023-10-16T20:37:37.407054+0000","last_undegraded":"2023-10-16T20:37:37.407054+0000","last_fullsized":"2023-10-16T20:37:37.407054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:45:21.058260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"29'6","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.509257+0000","last_change":"2023-10-16T20:36:54.764693+0000","last_active":"2023-10-16T20:39:33.509257+0000","last_peered":"2023-10-16T20:39:33.509257+0000","last_clean":"2023-10-16T20:39:33.509257+0000","last_became_active":"2023-10-16T20:36:54.764417+0000","last_became_peered":"2023-10-16T20:36:54.764417+0000","last_unstale":"2023-10-16T20:39:33.509257+0000","last_undegraded":"2023-10-16T20:39:33.509257+0000","last_fullsized":"2023-10-16T20:39:33.509257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:56.653149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1028,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465588+0000","last_change":"2023-10-16T20:36:59.465145+0000","last_active":"2023-10-16T20:37:37.465588+0000","last_peered":"2023-10-16T20:37:37.465588+0000","last_clean":"2023-10-16T20:37:37.465588+0000","last_became_active":"2023-10-16T20:36:59.464626+0000","last_became_peered":"2023-10-16T20:36:59.464626+0000","last_unstale":"2023-10-16T20:37:37.465588+0000","last_undegraded":"2023-10-16T20:37:37.465588+0000","last_fullsized":"2023-10-16T20:37:37.465588+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:51:38.928616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466290+0000","last_change":"2023-10-16T20:36:55.291718+0000","last_active":"2023-10-16T20:37:37.466290+0000","last_peered":"2023-10-16T20:37:37.466290+0000","last_clean":"2023-10-16T20:37:37.466290+0000","last_became_active":"2023-10-16T20:36:55.291396+0000","last_became_peered":"2023-10-16T20:36:55.291396+0000","last_unstale":"2023-10-16T20:37:37.466290+0000","last_undegraded":"2023-10-16T20:37:37.466290+0000","last_fullsized":"2023-10-16T20:37:37.466290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:29:49.138380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.507860+0000","last_change":"2023-10-16T20:36:54.750814+0000","last_active":"2023-10-16T20:39:33.507860+0000","last_peered":"2023-10-16T20:39:33.507860+0000","last_clean":"2023-10-16T20:39:33.507860+0000","last_became_active":"2023-10-16T20:36:54.750448+0000","last_became_peered":"2023-10-16T20:36:54.750448+0000","last_unstale":"2023-10-16T20:39:33.507860+0000","last_undegraded":"2023-10-16T20:39:33.507860+0000","last_fullsized":"2023-10-16T20:39:33.507860+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:41.981408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465192+0000","last_change":"2023-10-16T20:36:59.464962+0000","last_active":"2023-10-16T20:37:37.465192+0000","last_peered":"2023-10-16T20:37:37.465192+0000","last_clean":"2023-10-16T20:37:37.465192+0000","last_became_active":"2023-10-16T20:36:59.464578+0000","last_became_peered":"2023-10-16T20:36:59.464578+0000","last_unstale":"2023-10-16T20:37:37.465192+0000","last_undegraded":"2023-10-16T20:37:37.465192+0000","last_fullsized":"2023-10-16T20:37:37.465192+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:12:27.738364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435179+0000","last_change":"2023-10-16T20:36:55.285675+0000","last_active":"2023-10-16T20:37:37.435179+0000","last_peered":"2023-10-16T20:37:37.435179+0000","last_clean":"2023-10-16T20:37:37.435179+0000","last_became_active":"2023-10-16T20:36:55.285407+0000","last_became_peered":"2023-10-16T20:36:55.285407+0000","last_unstale":"2023-10-16T20:37:37.435179+0000","last_undegraded":"2023-10-16T20:37:37.435179+0000","last_fullsized":"2023-10-16T20:37:37.435179+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:31:10.259938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'9","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.524706+0000","last_change":"2023-10-16T20:36:54.757012+0000","last_active":"2023-10-16T20:39:39.524706+0000","last_peered":"2023-10-16T20:39:39.524706+0000","last_clean":"2023-10-16T20:39:39.524706+0000","last_became_active":"2023-10-16T20:36:54.756697+0000","last_became_peered":"2023-10-16T20:36:54.756697+0000","last_unstale":"2023-10-16T20:39:39.524706+0000","last_undegraded":"2023-10-16T20:39:39.524706+0000","last_fullsized":"2023-10-16T20:39:39.524706+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:20.259580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409477+0000","last_change":"2023-10-16T20:36:59.469134+0000","last_active":"2023-10-16T20:37:37.409477+0000","last_peered":"2023-10-16T20:37:37.409477+0000","last_clean":"2023-10-16T20:37:37.409477+0000","last_became_active":"2023-10-16T20:36:59.468254+0000","last_became_peered":"2023-10-16T20:36:59.468254+0000","last_unstale":"2023-10-16T20:37:37.409477+0000","last_undegraded":"2023-10-16T20:37:37.409477+0000","last_fullsized":"2023-10-16T20:37:37.409477+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:46:12.444815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466418+0000","last_change":"2023-10-16T20:36:55.289133+0000","last_active":"2023-10-16T20:37:37.466418+0000","last_peered":"2023-10-16T20:37:37.466418+0000","last_clean":"2023-10-16T20:37:37.466418+0000","last_became_active":"2023-10-16T20:36:55.288488+0000","last_became_peered":"2023-10-16T20:36:55.288488+0000","last_unstale":"2023-10-16T20:37:37.466418+0000","last_undegraded":"2023-10-16T20:37:37.466418+0000","last_fullsized":"2023-10-16T20:37:37.466418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:27:22.022319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'16","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:48.656539+0000","last_change":"2023-10-16T20:36:54.765361+0000","last_active":"2023-10-16T20:39:48.656539+0000","last_peered":"2023-10-16T20:39:48.656539+0000","last_clean":"2023-10-16T20:39:48.656539+0000","last_became_active":"2023-10-16T20:36:54.764969+0000","last_became_peered":"2023-10-16T20:36:54.764969+0000","last_unstale":"2023-10-16T20:39:48.656539+0000","last_undegraded":"2023-10-16T20:39:48.656539+0000","last_fullsized":"2023-10-16T20:39:48.656539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:21:21.183612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4281,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402193+0000","last_change":"2023-10-16T20:36:59.303588+0000","last_active":"2023-10-16T20:37:37.402193+0000","last_peered":"2023-10-16T20:37:37.402193+0000","last_clean":"2023-10-16T20:37:37.402193+0000","last_became_active":"2023-10-16T20:36:59.302698+0000","last_became_peered":"2023-10-16T20:36:59.302698+0000","last_unstale":"2023-10-16T20:37:37.402193+0000","last_undegraded":"2023-10-16T20:37:37.402193+0000","last_fullsized":"2023-10-16T20:37:37.402193+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:08:28.644565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403928+0000","last_change":"2023-10-16T20:36:55.290812+0000","last_active":"2023-10-16T20:37:37.403928+0000","last_peered":"2023-10-16T20:37:37.403928+0000","last_clean":"2023-10-16T20:37:37.403928+0000","last_became_active":"2023-10-16T20:36:55.290127+0000","last_became_peered":"2023-10-16T20:36:55.290127+0000","last_unstale":"2023-10-16T20:37:37.403928+0000","last_undegraded":"2023-10-16T20:37:37.403928+0000","last_fullsized":"2023-10-16T20:37:37.403928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:28:27.930123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:22.406599+0000","last_change":"2023-10-16T20:36:54.749120+0000","last_active":"2023-10-16T20:39:22.406599+0000","last_peered":"2023-10-16T20:39:22.406599+0000","last_clean":"2023-10-16T20:39:22.406599+0000","last_became_active":"2023-10-16T20:36:54.748788+0000","last_became_peered":"2023-10-16T20:36:54.748788+0000","last_unstale":"2023-10-16T20:39:22.406599+0000","last_undegraded":"2023-10-16T20:39:22.406599+0000","last_fullsized":"2023-10-16T20:39:22.406599+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:11:07.257343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403049+0000","last_change":"2023-10-16T20:36:59.303546+0000","last_active":"2023-10-16T20:37:37.403049+0000","last_peered":"2023-10-16T20:37:37.403049+0000","last_clean":"2023-10-16T20:37:37.403049+0000","last_became_active":"2023-10-16T20:36:59.302916+0000","last_became_peered":"2023-10-16T20:36:59.302916+0000","last_unstale":"2023-10-16T20:37:37.403049+0000","last_undegraded":"2023-10-16T20:37:37.403049+0000","last_fullsized":"2023-10-16T20:37:37.403049+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:17:16.174476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435547+0000","last_change":"2023-10-16T20:36:55.291739+0000","last_active":"2023-10-16T20:37:37.435547+0000","last_peered":"2023-10-16T20:37:37.435547+0000","last_clean":"2023-10-16T20:37:37.435547+0000","last_became_active":"2023-10-16T20:36:55.291448+0000","last_became_peered":"2023-10-16T20:36:55.291448+0000","last_unstale":"2023-10-16T20:37:37.435547+0000","last_undegraded":"2023-10-16T20:37:37.435547+0000","last_fullsized":"2023-10-16T20:37:37.435547+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:41:15.380678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:39.526865+0000","last_change":"2023-10-16T20:36:54.761824+0000","last_active":"2023-10-16T20:39:39.526865+0000","last_peered":"2023-10-16T20:39:39.526865+0000","last_clean":"2023-10-16T20:39:39.526865+0000","last_became_active":"2023-10-16T20:36:54.761541+0000","last_became_peered":"2023-10-16T20:36:54.761541+0000","last_unstale":"2023-10-16T20:39:39.526865+0000","last_undegraded":"2023-10-16T20:39:39.526865+0000","last_fullsized":"2023-10-16T20:39:39.526865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:34.512207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464594+0000","last_change":"2023-10-16T20:36:59.304165+0000","last_active":"2023-10-16T20:37:37.464594+0000","last_peered":"2023-10-16T20:37:37.464594+0000","last_clean":"2023-10-16T20:37:37.464594+0000","last_became_active":"2023-10-16T20:36:59.303773+0000","last_became_peered":"2023-10-16T20:36:59.303773+0000","last_unstale":"2023-10-16T20:37:37.464594+0000","last_undegraded":"2023-10-16T20:37:37.464594+0000","last_fullsized":"2023-10-16T20:37:37.464594+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:24:09.710698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409711+0000","last_change":"2023-10-16T20:36:55.300829+0000","last_active":"2023-10-16T20:37:37.409711+0000","last_peered":"2023-10-16T20:37:37.409711+0000","last_clean":"2023-10-16T20:37:37.409711+0000","last_became_active":"2023-10-16T20:36:55.300535+0000","last_became_peered":"2023-10-16T20:36:55.300535+0000","last_unstale":"2023-10-16T20:37:37.409711+0000","last_undegraded":"2023-10-16T20:37:37.409711+0000","last_fullsized":"2023-10-16T20:37:37.409711+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:29:34.910378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'5","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.508094+0000","last_change":"2023-10-16T20:36:54.758140+0000","last_active":"2023-10-16T20:39:33.508094+0000","last_peered":"2023-10-16T20:39:33.508094+0000","last_clean":"2023-10-16T20:39:33.508094+0000","last_became_active":"2023-10-16T20:36:54.757303+0000","last_became_peered":"2023-10-16T20:36:54.757303+0000","last_unstale":"2023-10-16T20:39:33.508094+0000","last_undegraded":"2023-10-16T20:39:33.508094+0000","last_fullsized":"2023-10-16T20:39:33.508094+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:45.944191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464795+0000","last_change":"2023-10-16T20:36:59.471158+0000","last_active":"2023-10-16T20:37:37.464795+0000","last_peered":"2023-10-16T20:37:37.464795+0000","last_clean":"2023-10-16T20:37:37.464795+0000","last_became_active":"2023-10-16T20:36:59.465844+0000","last_became_peered":"2023-10-16T20:36:59.465844+0000","last_unstale":"2023-10-16T20:37:37.464795+0000","last_undegraded":"2023-10-16T20:37:37.464795+0000","last_fullsized":"2023-10-16T20:37:37.464795+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:12.133525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435182+0000","last_change":"2023-10-16T20:36:59.468843+0000","last_active":"2023-10-16T20:37:37.435182+0000","last_peered":"2023-10-16T20:37:37.435182+0000","last_clean":"2023-10-16T20:37:37.435182+0000","last_became_active":"2023-10-16T20:36:59.467356+0000","last_became_peered":"2023-10-16T20:36:59.467356+0000","last_unstale":"2023-10-16T20:37:37.435182+0000","last_undegraded":"2023-10-16T20:37:37.435182+0000","last_fullsized":"2023-10-16T20:37:37.435182+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:47:02.923540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401755+0000","last_change":"2023-10-16T20:36:55.281355+0000","last_active":"2023-10-16T20:37:37.401755+0000","last_peered":"2023-10-16T20:37:37.401755+0000","last_clean":"2023-10-16T20:37:37.401755+0000","last_became_active":"2023-10-16T20:36:55.280903+0000","last_became_peered":"2023-10-16T20:36:55.280903+0000","last_unstale":"2023-10-16T20:37:37.401755+0000","last_undegraded":"2023-10-16T20:37:37.401755+0000","last_fullsized":"2023-10-16T20:37:37.401755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:15:25.813826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.507692+0000","last_change":"2023-10-16T20:36:54.757589+0000","last_active":"2023-10-16T20:39:33.507692+0000","last_peered":"2023-10-16T20:39:33.507692+0000","last_clean":"2023-10-16T20:39:33.507692+0000","last_became_active":"2023-10-16T20:36:54.756775+0000","last_became_peered":"2023-10-16T20:36:54.756775+0000","last_unstale":"2023-10-16T20:39:33.507692+0000","last_undegraded":"2023-10-16T20:39:33.507692+0000","last_fullsized":"2023-10-16T20:39:33.507692+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:40.629540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.441661+0000","last_change":"2023-10-16T20:37:37.438581+0000","last_active":"2023-10-16T20:37:37.441661+0000","last_peered":"2023-10-16T20:37:37.441661+0000","last_clean":"2023-10-16T20:37:37.441661+0000","last_became_active":"2023-10-16T20:37:37.438140+0000","last_became_peered":"2023-10-16T20:37:37.438140+0000","last_unstale":"2023-10-16T20:37:37.441661+0000","last_undegraded":"2023-10-16T20:37:37.441661+0000","last_fullsized":"2023-10-16T20:37:37.441661+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:55.110618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464854+0000","last_change":"2023-10-16T20:36:55.274207+0000","last_active":"2023-10-16T20:37:37.464854+0000","last_peered":"2023-10-16T20:37:37.464854+0000","last_clean":"2023-10-16T20:37:37.464854+0000","last_became_active":"2023-10-16T20:36:55.273959+0000","last_became_peered":"2023-10-16T20:36:55.273959+0000","last_unstale":"2023-10-16T20:37:37.464854+0000","last_undegraded":"2023-10-16T20:37:37.464854+0000","last_fullsized":"2023-10-16T20:37:37.464854+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:23:09.275091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:28.444227+0000","last_change":"2023-10-16T20:36:54.759513+0000","last_active":"2023-10-16T20:39:28.444227+0000","last_peered":"2023-10-16T20:39:28.444227+0000","last_clean":"2023-10-16T20:39:28.444227+0000","last_became_active":"2023-10-16T20:36:54.759059+0000","last_became_peered":"2023-10-16T20:36:54.759059+0000","last_unstale":"2023-10-16T20:39:28.444227+0000","last_undegraded":"2023-10-16T20:39:28.444227+0000","last_fullsized":"2023-10-16T20:39:28.444227+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:36.696053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:33.509348+0000","last_change":"2023-10-16T20:36:54.764780+0000","last_active":"2023-10-16T20:39:33.509348+0000","last_peered":"2023-10-16T20:39:33.509348+0000","last_clean":"2023-10-16T20:39:33.509348+0000","last_became_active":"2023-10-16T20:36:54.764436+0000","last_became_peered":"2023-10-16T20:36:54.764436+0000","last_unstale":"2023-10-16T20:39:33.509348+0000","last_undegraded":"2023-10-16T20:39:33.509348+0000","last_fullsized":"2023-10-16T20:39:33.509348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:55:31.816599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435152+0000","last_change":"2023-10-16T20:36:55.278134+0000","last_active":"2023-10-16T20:37:37.435152+0000","last_peered":"2023-10-16T20:37:37.435152+0000","last_clean":"2023-10-16T20:37:37.435152+0000","last_became_active":"2023-10-16T20:36:55.277873+0000","last_became_peered":"2023-10-16T20:36:55.277873+0000","last_unstale":"2023-10-16T20:37:37.435152+0000","last_undegraded":"2023-10-16T20:37:37.435152+0000","last_fullsized":"2023-10-16T20:37:37.435152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:59:43.701800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.399947+0000","last_change":"2023-10-16T20:36:59.302774+0000","last_active":"2023-10-16T20:37:37.399947+0000","last_peered":"2023-10-16T20:37:37.399947+0000","last_clean":"2023-10-16T20:37:37.399947+0000","last_became_active":"2023-10-16T20:36:59.302435+0000","last_became_peered":"2023-10-16T20:36:59.302435+0000","last_unstale":"2023-10-16T20:37:37.399947+0000","last_undegraded":"2023-10-16T20:37:37.399947+0000","last_fullsized":"2023-10-16T20:37:37.399947+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:11:53.740982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'6","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:39:27.631103+0000","last_change":"2023-10-16T20:36:54.756190+0000","last_active":"2023-10-16T20:39:27.631103+0000","last_peered":"2023-10-16T20:39:27.631103+0000","last_clean":"2023-10-16T20:39:27.631103+0000","last_became_active":"2023-10-16T20:36:54.755834+0000","last_became_peered":"2023-10-16T20:36:54.755834+0000","last_unstale":"2023-10-16T20:39:27.631103+0000","last_undegraded":"2023-10-16T20:39:27.631103+0000","last_fullsized":"2023-10-16T20:39:27.631103+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:31.012441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1062,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.403395+0000","last_change":"2023-10-16T20:36:55.279822+0000","last_active":"2023-10-16T20:37:37.403395+0000","last_peered":"2023-10-16T20:37:37.403395+0000","last_clean":"2023-10-16T20:37:37.403395+0000","last_became_active":"2023-10-16T20:36:55.279333+0000","last_became_peered":"2023-10-16T20:36:55.279333+0000","last_unstale":"2023-10-16T20:37:37.403395+0000","last_undegraded":"2023-10-16T20:37:37.403395+0000","last_fullsized":"2023-10-16T20:37:37.403395+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:02:21.471812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465391+0000","last_change":"2023-10-16T20:36:59.302190+0000","last_active":"2023-10-16T20:37:37.465391+0000","last_peered":"2023-10-16T20:37:37.465391+0000","last_clean":"2023-10-16T20:37:37.465391+0000","last_became_active":"2023-10-16T20:36:59.301864+0000","last_became_peered":"2023-10-16T20:36:59.301864+0000","last_unstale":"2023-10-16T20:37:37.465391+0000","last_undegraded":"2023-10-16T20:37:37.465391+0000","last_fullsized":"2023-10-16T20:37:37.465391+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:37:55.282815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.406436+0000","last_change":"2023-10-16T20:36:55.302476+0000","last_active":"2023-10-16T20:37:37.406436+0000","last_peered":"2023-10-16T20:37:37.406436+0000","last_clean":"2023-10-16T20:37:37.406436+0000","last_became_active":"2023-10-16T20:36:55.287442+0000","last_became_peered":"2023-10-16T20:36:55.287442+0000","last_unstale":"2023-10-16T20:37:37.406436+0000","last_undegraded":"2023-10-16T20:37:37.406436+0000","last_fullsized":"2023-10-16T20:37:37.406436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:09:19.511156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.399520+0000","last_change":"2023-10-16T20:36:55.276521+0000","last_active":"2023-10-16T20:37:37.399520+0000","last_peered":"2023-10-16T20:37:37.399520+0000","last_clean":"2023-10-16T20:37:37.399520+0000","last_became_active":"2023-10-16T20:36:55.276276+0000","last_became_peered":"2023-10-16T20:36:55.276276+0000","last_unstale":"2023-10-16T20:37:37.399520+0000","last_undegraded":"2023-10-16T20:37:37.399520+0000","last_fullsized":"2023-10-16T20:37:37.399520+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:38.218242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.415548+0000","last_change":"2023-10-16T20:37:37.413429+0000","last_active":"2023-10-16T20:37:37.415548+0000","last_peered":"2023-10-16T20:37:37.415548+0000","last_clean":"2023-10-16T20:37:37.415548+0000","last_became_active":"2023-10-16T20:37:37.413040+0000","last_became_peered":"2023-10-16T20:37:37.413040+0000","last_unstale":"2023-10-16T20:37:37.415548+0000","last_undegraded":"2023-10-16T20:37:37.415548+0000","last_fullsized":"2023-10-16T20:37:37.415548+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:39:10.897056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435108+0000","last_change":"2023-10-16T20:36:55.276258+0000","last_active":"2023-10-16T20:37:37.435108+0000","last_peered":"2023-10-16T20:37:37.435108+0000","last_clean":"2023-10-16T20:37:37.435108+0000","last_became_active":"2023-10-16T20:36:55.275349+0000","last_became_peered":"2023-10-16T20:36:55.275349+0000","last_unstale":"2023-10-16T20:37:37.435108+0000","last_undegraded":"2023-10-16T20:37:37.435108+0000","last_fullsized":"2023-10-16T20:37:37.435108+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:59.007782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465455+0000","last_change":"2023-10-16T20:36:55.277332+0000","last_active":"2023-10-16T20:37:37.465455+0000","last_peered":"2023-10-16T20:37:37.465455+0000","last_clean":"2023-10-16T20:37:37.465455+0000","last_became_active":"2023-10-16T20:36:55.277043+0000","last_became_peered":"2023-10-16T20:36:55.277043+0000","last_unstale":"2023-10-16T20:37:37.465455+0000","last_undegraded":"2023-10-16T20:37:37.465455+0000","last_fullsized":"2023-10-16T20:37:37.465455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:46:07.227813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401690+0000","last_change":"2023-10-16T20:36:55.279003+0000","last_active":"2023-10-16T20:37:37.401690+0000","last_peered":"2023-10-16T20:37:37.401690+0000","last_clean":"2023-10-16T20:37:37.401690+0000","last_became_active":"2023-10-16T20:36:55.278715+0000","last_became_peered":"2023-10-16T20:36:55.278715+0000","last_unstale":"2023-10-16T20:37:37.401690+0000","last_undegraded":"2023-10-16T20:37:37.401690+0000","last_fullsized":"2023-10-16T20:37:37.401690+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:03:58.274009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408078+0000","last_change":"2023-10-16T20:36:55.303936+0000","last_active":"2023-10-16T20:37:37.408078+0000","last_peered":"2023-10-16T20:37:37.408078+0000","last_clean":"2023-10-16T20:37:37.408078+0000","last_became_active":"2023-10-16T20:36:55.301604+0000","last_became_peered":"2023-10-16T20:36:55.301604+0000","last_unstale":"2023-10-16T20:37:37.408078+0000","last_undegraded":"2023-10-16T20:37:37.408078+0000","last_fullsized":"2023-10-16T20:37:37.408078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:48:14.922695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465625+0000","last_change":"2023-10-16T20:36:55.277329+0000","last_active":"2023-10-16T20:37:37.465625+0000","last_peered":"2023-10-16T20:37:37.465625+0000","last_clean":"2023-10-16T20:37:37.465625+0000","last_became_active":"2023-10-16T20:36:55.276984+0000","last_became_peered":"2023-10-16T20:36:55.276984+0000","last_unstale":"2023-10-16T20:37:37.465625+0000","last_undegraded":"2023-10-16T20:37:37.465625+0000","last_fullsized":"2023-10-16T20:37:37.465625+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:43:52.763586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408404+0000","last_change":"2023-10-16T20:36:55.290545+0000","last_active":"2023-10-16T20:37:37.408404+0000","last_peered":"2023-10-16T20:37:37.408404+0000","last_clean":"2023-10-16T20:37:37.408404+0000","last_became_active":"2023-10-16T20:36:55.290023+0000","last_became_peered":"2023-10-16T20:36:55.290023+0000","last_unstale":"2023-10-16T20:37:37.408404+0000","last_undegraded":"2023-10-16T20:37:37.408404+0000","last_fullsized":"2023-10-16T20:37:37.408404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:04:20.914854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464978+0000","last_change":"2023-10-16T20:36:55.280456+0000","last_active":"2023-10-16T20:37:37.464978+0000","last_peered":"2023-10-16T20:37:37.464978+0000","last_clean":"2023-10-16T20:37:37.464978+0000","last_became_active":"2023-10-16T20:36:55.280186+0000","last_became_peered":"2023-10-16T20:36:55.280186+0000","last_unstale":"2023-10-16T20:37:37.464978+0000","last_undegraded":"2023-10-16T20:37:37.464978+0000","last_fullsized":"2023-10-16T20:37:37.464978+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:52.469768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408252+0000","last_change":"2023-10-16T20:36:55.297917+0000","last_active":"2023-10-16T20:37:37.408252+0000","last_peered":"2023-10-16T20:37:37.408252+0000","last_clean":"2023-10-16T20:37:37.408252+0000","last_became_active":"2023-10-16T20:36:55.297253+0000","last_became_peered":"2023-10-16T20:36:55.297253+0000","last_unstale":"2023-10-16T20:37:37.408252+0000","last_undegraded":"2023-10-16T20:37:37.408252+0000","last_fullsized":"2023-10-16T20:37:37.408252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:23:14.977093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.434979+0000","last_change":"2023-10-16T20:36:55.286327+0000","last_active":"2023-10-16T20:37:37.434979+0000","last_peered":"2023-10-16T20:37:37.434979+0000","last_clean":"2023-10-16T20:37:37.434979+0000","last_became_active":"2023-10-16T20:36:55.286044+0000","last_became_peered":"2023-10-16T20:36:55.286044+0000","last_unstale":"2023-10-16T20:37:37.434979+0000","last_undegraded":"2023-10-16T20:37:37.434979+0000","last_fullsized":"2023-10-16T20:37:37.434979+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:30:56.000502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409227+0000","last_change":"2023-10-16T20:36:55.298742+0000","last_active":"2023-10-16T20:37:37.409227+0000","last_peered":"2023-10-16T20:37:37.409227+0000","last_clean":"2023-10-16T20:37:37.409227+0000","last_became_active":"2023-10-16T20:36:55.286864+0000","last_became_peered":"2023-10-16T20:36:55.286864+0000","last_unstale":"2023-10-16T20:37:37.409227+0000","last_undegraded":"2023-10-16T20:37:37.409227+0000","last_fullsized":"2023-10-16T20:37:37.409227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:24:27.598910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466521+0000","last_change":"2023-10-16T20:36:55.292541+0000","last_active":"2023-10-16T20:37:37.466521+0000","last_peered":"2023-10-16T20:37:37.466521+0000","last_clean":"2023-10-16T20:37:37.466521+0000","last_became_active":"2023-10-16T20:36:55.292262+0000","last_became_peered":"2023-10-16T20:36:55.292262+0000","last_unstale":"2023-10-16T20:37:37.466521+0000","last_undegraded":"2023-10-16T20:37:37.466521+0000","last_fullsized":"2023-10-16T20:37:37.466521+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:10:08.644265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.400632+0000","last_change":"2023-10-16T20:36:55.283056+0000","last_active":"2023-10-16T20:37:37.400632+0000","last_peered":"2023-10-16T20:37:37.400632+0000","last_clean":"2023-10-16T20:37:37.400632+0000","last_became_active":"2023-10-16T20:36:55.282806+0000","last_became_peered":"2023-10-16T20:36:55.282806+0000","last_unstale":"2023-10-16T20:37:37.400632+0000","last_undegraded":"2023-10-16T20:37:37.400632+0000","last_fullsized":"2023-10-16T20:37:37.400632+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:21:24.681696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.410649+0000","last_change":"2023-10-16T20:36:55.303385+0000","last_active":"2023-10-16T20:37:37.410649+0000","last_peered":"2023-10-16T20:37:37.410649+0000","last_clean":"2023-10-16T20:37:37.410649+0000","last_became_active":"2023-10-16T20:36:55.303095+0000","last_became_peered":"2023-10-16T20:36:55.303095+0000","last_unstale":"2023-10-16T20:37:37.410649+0000","last_undegraded":"2023-10-16T20:37:37.410649+0000","last_fullsized":"2023-10-16T20:37:37.410649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:26.368480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408312+0000","last_change":"2023-10-16T20:36:55.291852+0000","last_active":"2023-10-16T20:37:37.408312+0000","last_peered":"2023-10-16T20:37:37.408312+0000","last_clean":"2023-10-16T20:37:37.408312+0000","last_became_active":"2023-10-16T20:36:55.291447+0000","last_became_peered":"2023-10-16T20:36:55.291447+0000","last_unstale":"2023-10-16T20:37:37.408312+0000","last_undegraded":"2023-10-16T20:37:37.408312+0000","last_fullsized":"2023-10-16T20:37:37.408312+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:07.195372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.401838+0000","last_change":"2023-10-16T20:36:55.281169+0000","last_active":"2023-10-16T20:37:37.401838+0000","last_peered":"2023-10-16T20:37:37.401838+0000","last_clean":"2023-10-16T20:37:37.401838+0000","last_became_active":"2023-10-16T20:36:55.280605+0000","last_became_peered":"2023-10-16T20:36:55.280605+0000","last_unstale":"2023-10-16T20:37:37.401838+0000","last_undegraded":"2023-10-16T20:37:37.401838+0000","last_fullsized":"2023-10-16T20:37:37.401838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:03.294173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435040+0000","last_change":"2023-10-16T20:36:55.277395+0000","last_active":"2023-10-16T20:37:37.435040+0000","last_peered":"2023-10-16T20:37:37.435040+0000","last_clean":"2023-10-16T20:37:37.435040+0000","last_became_active":"2023-10-16T20:36:55.277094+0000","last_became_peered":"2023-10-16T20:36:55.277094+0000","last_unstale":"2023-10-16T20:37:37.435040+0000","last_undegraded":"2023-10-16T20:37:37.435040+0000","last_fullsized":"2023-10-16T20:37:37.435040+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:23:03.400384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.465006+0000","last_change":"2023-10-16T20:36:55.297429+0000","last_active":"2023-10-16T20:37:37.465006+0000","last_peered":"2023-10-16T20:37:37.465006+0000","last_clean":"2023-10-16T20:37:37.465006+0000","last_became_active":"2023-10-16T20:36:55.297199+0000","last_became_peered":"2023-10-16T20:36:55.297199+0000","last_unstale":"2023-10-16T20:37:37.465006+0000","last_undegraded":"2023-10-16T20:37:37.465006+0000","last_fullsized":"2023-10-16T20:37:37.465006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:37:43.860251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.404029+0000","last_change":"2023-10-16T20:36:55.287421+0000","last_active":"2023-10-16T20:37:37.404029+0000","last_peered":"2023-10-16T20:37:37.404029+0000","last_clean":"2023-10-16T20:37:37.404029+0000","last_became_active":"2023-10-16T20:36:55.286964+0000","last_became_peered":"2023-10-16T20:36:55.286964+0000","last_unstale":"2023-10-16T20:37:37.404029+0000","last_undegraded":"2023-10-16T20:37:37.404029+0000","last_fullsized":"2023-10-16T20:37:37.404029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:15:54.970100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.416921+0000","last_change":"2023-10-16T20:37:37.413891+0000","last_active":"2023-10-16T20:37:37.416921+0000","last_peered":"2023-10-16T20:37:37.416921+0000","last_clean":"2023-10-16T20:37:37.416921+0000","last_became_active":"2023-10-16T20:37:37.413492+0000","last_became_peered":"2023-10-16T20:37:37.413492+0000","last_unstale":"2023-10-16T20:37:37.416921+0000","last_undegraded":"2023-10-16T20:37:37.416921+0000","last_fullsized":"2023-10-16T20:37:37.416921+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:03:46.658556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.407997+0000","last_change":"2023-10-16T20:36:55.303950+0000","last_active":"2023-10-16T20:37:37.407997+0000","last_peered":"2023-10-16T20:37:37.407997+0000","last_clean":"2023-10-16T20:37:37.407997+0000","last_became_active":"2023-10-16T20:36:55.302468+0000","last_became_peered":"2023-10-16T20:36:55.302468+0000","last_unstale":"2023-10-16T20:37:37.407997+0000","last_undegraded":"2023-10-16T20:37:37.407997+0000","last_fullsized":"2023-10-16T20:37:37.407997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:02.962031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466830+0000","last_change":"2023-10-16T20:36:55.297558+0000","last_active":"2023-10-16T20:37:37.466830+0000","last_peered":"2023-10-16T20:37:37.466830+0000","last_clean":"2023-10-16T20:37:37.466830+0000","last_became_active":"2023-10-16T20:36:55.296476+0000","last_became_peered":"2023-10-16T20:36:55.296476+0000","last_unstale":"2023-10-16T20:37:37.466830+0000","last_undegraded":"2023-10-16T20:37:37.466830+0000","last_fullsized":"2023-10-16T20:37:37.466830+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:04:38.235205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.464724+0000","last_change":"2023-10-16T20:36:55.293957+0000","last_active":"2023-10-16T20:37:37.464724+0000","last_peered":"2023-10-16T20:37:37.464724+0000","last_clean":"2023-10-16T20:37:37.464724+0000","last_became_active":"2023-10-16T20:36:55.293687+0000","last_became_peered":"2023-10-16T20:36:55.293687+0000","last_unstale":"2023-10-16T20:37:37.464724+0000","last_undegraded":"2023-10-16T20:37:37.464724+0000","last_fullsized":"2023-10-16T20:37:37.464724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:34:54.623691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466003+0000","last_change":"2023-10-16T20:36:55.280728+0000","last_active":"2023-10-16T20:37:37.466003+0000","last_peered":"2023-10-16T20:37:37.466003+0000","last_clean":"2023-10-16T20:37:37.466003+0000","last_became_active":"2023-10-16T20:36:55.280444+0000","last_became_peered":"2023-10-16T20:36:55.280444+0000","last_unstale":"2023-10-16T20:37:37.466003+0000","last_undegraded":"2023-10-16T20:37:37.466003+0000","last_fullsized":"2023-10-16T20:37:37.466003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:44.044563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.408158+0000","last_change":"2023-10-16T20:36:55.297802+0000","last_active":"2023-10-16T20:37:37.408158+0000","last_peered":"2023-10-16T20:37:37.408158+0000","last_clean":"2023-10-16T20:37:37.408158+0000","last_became_active":"2023-10-16T20:36:55.296694+0000","last_became_peered":"2023-10-16T20:36:55.296694+0000","last_unstale":"2023-10-16T20:37:37.408158+0000","last_undegraded":"2023-10-16T20:37:37.408158+0000","last_fullsized":"2023-10-16T20:37:37.408158+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:00:26.797387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466169+0000","last_change":"2023-10-16T20:36:55.291347+0000","last_active":"2023-10-16T20:37:37.466169+0000","last_peered":"2023-10-16T20:37:37.466169+0000","last_clean":"2023-10-16T20:37:37.466169+0000","last_became_active":"2023-10-16T20:36:55.291052+0000","last_became_peered":"2023-10-16T20:36:55.291052+0000","last_unstale":"2023-10-16T20:37:37.466169+0000","last_undegraded":"2023-10-16T20:37:37.466169+0000","last_fullsized":"2023-10-16T20:37:37.466169+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:40:51.331446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.409006+0000","last_change":"2023-10-16T20:36:55.290870+0000","last_active":"2023-10-16T20:37:37.409006+0000","last_peered":"2023-10-16T20:37:37.409006+0000","last_clean":"2023-10-16T20:37:37.409006+0000","last_became_active":"2023-10-16T20:36:55.290386+0000","last_became_peered":"2023-10-16T20:36:55.290386+0000","last_unstale":"2023-10-16T20:37:37.409006+0000","last_undegraded":"2023-10-16T20:37:37.409006+0000","last_fullsized":"2023-10-16T20:37:37.409006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:20.006864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.435199+0000","last_change":"2023-10-16T20:36:55.271041+0000","last_active":"2023-10-16T20:37:37.435199+0000","last_peered":"2023-10-16T20:37:37.435199+0000","last_clean":"2023-10-16T20:37:37.435199+0000","last_became_active":"2023-10-16T20:36:55.270787+0000","last_became_peered":"2023-10-16T20:36:55.270787+0000","last_unstale":"2023-10-16T20:37:37.435199+0000","last_undegraded":"2023-10-16T20:37:37.435199+0000","last_fullsized":"2023-10-16T20:37:37.435199+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:07.944778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.402678+0000","last_change":"2023-10-16T20:36:55.271632+0000","last_active":"2023-10-16T20:37:37.402678+0000","last_peered":"2023-10-16T20:37:37.402678+0000","last_clean":"2023-10-16T20:37:37.402678+0000","last_became_active":"2023-10-16T20:36:55.271373+0000","last_became_peered":"2023-10-16T20:36:55.271373+0000","last_unstale":"2023-10-16T20:37:37.402678+0000","last_undegraded":"2023-10-16T20:37:37.402678+0000","last_fullsized":"2023-10-16T20:37:37.402678+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:47:07.458540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-16T20:37:37.466741+0000","last_change":"2023-10-16T20:36:55.291343+0000","last_active":"2023-10-16T20:37:37.466741+0000","last_peered":"2023-10-16T20:37:37.466741+0000","last_clean":"2023-10-16T20:37:37.466741+0000","last_became_active":"2023-10-16T20:36:55.291016+0000","last_became_peered":"2023-10-16T20:36:55.291016+0000","last_unstale":"2023-10-16T20:37:37.466741+0000","last_undegraded":"2023-10-16T20:37:37.466741+0000","last_fullsized":"2023-10-16T20:37:37.466741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:13:40.287782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":12023,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":24046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":190,"ondisk_log_size":190,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509481,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":937984,"data_stored":717153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.75600000000000001,"15min":0.752},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":5.2240000000000002,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.046,"5min":0.81599999999999995,"15min":0.77300000000000002},"min":{"1min":0.47399999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":8.782,"5min":8.782,"15min":8.782},"last":0.64100000000000001}]},{"osd":1,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.73699999999999999,"15min":0.73799999999999999},"min":{"1min":0.49199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.91800000000000004,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.76300000000000001,"15min":0.74399999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":4.7300000000000004,"5min":4.7300000000000004,"15min":4.7300000000000004},"last":0.58999999999999997}]},{"osd":2,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.76900000000000002,"15min":0.73899999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":5.2629999999999999,"5min":5.2629999999999999,"15min":5.2629999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.77100000000000002,"15min":0.753},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.1589999999999998,"5min":5.1589999999999998,"15min":5.1589999999999998},"last":0.66900000000000004}]},{"osd":3,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.79900000000000004,"15min":0.77900000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":4.7460000000000004,"5min":4.7460000000000004,"15min":4.7460000000000004},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.78400000000000003,"15min":0.77000000000000002},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":4.7770000000000001,"5min":4.7770000000000001,"15min":4.7770000000000001},"last":0.80600000000000005}]},{"osd":4,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.61599999999999999,"15min":0.628},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.82099999999999995,"5min":1.3,"15min":1.3},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.58899999999999997,"15min":0.60499999999999998},"min":{"1min":0.375,"5min":0.373,"15min":0.373},"max":{"1min":0.77600000000000002,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.51200000000000001}]},{"osd":5,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68700000000000006,"15min":0.69899999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83799999999999997,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.76600000000000001,"15min":0.78800000000000003},"min":{"1min":0.57899999999999996,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.88600000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.746}]},{"osd":6,"last update":"Mon Oct 16 20:39:50 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.73499999999999999,"15min":0.75},"min":{"1min":0.51600000000000001,"5min":0.499,"15min":0.499},"max":{"1min":0.877,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.68799999999999994,"15min":0.70099999999999996},"min":{"1min":0.44500000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.85899999999999999,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.83699999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509481,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":129756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.747,"15min":0.76400000000000001},"min":{"1min":0.47399999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.74299999999999999,"15min":0.75600000000000001},"min":{"1min":0.51300000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.73499999999999999,"15min":0.75700000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.85399999999999998,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77000000000000002,"15min":0.77600000000000002},"min":{"1min":0.46300000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":3.387,"5min":3.387,"15min":3.387},"last":0.46500000000000002}]},{"osd":2,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.79000000000000004,"15min":0.79500000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":3.9489999999999998,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.755,"15min":0.751},"min":{"1min":0.53200000000000003,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":3.6179999999999999,"5min":3.6179999999999999,"15min":3.6179999999999999},"last":0.78100000000000003}]},{"osd":3,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.81899999999999995,"15min":0.85899999999999999},"min":{"1min":0.53700000000000003,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0009999999999999,"5min":1.611,"15min":1.611},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.79400000000000004,"15min":0.81399999999999995},"min":{"1min":0.56499999999999995,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.052,"5min":1.163,"15min":1.163},"last":0.75700000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.66000000000000003,"15min":0.67700000000000005},"min":{"1min":0.47399999999999998,"5min":0.443,"15min":0.443},"max":{"1min":0.93899999999999995,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.65600000000000003,"15min":0.66600000000000004},"min":{"1min":0.497,"5min":0.435,"15min":0.435},"max":{"1min":0.877,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.622}]},{"osd":5,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.46500000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.83699999999999997,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.69999999999999996,"15min":0.70599999999999996},"min":{"1min":0.52600000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.91300000000000003,"5min":1.016,"15min":1.016},"last":0.59899999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.73899999999999999,"15min":0.751},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.92000000000000004,"5min":1.292,"15min":1.292},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.748,"15min":0.77100000000000002},"min":{"1min":0.51900000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.90600000000000003,"5min":1.569,"15min":1.569},"last":0.69199999999999995}]}]},{"osd":5,"up_from":15,"seq":64424509483,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":356352,"data_stored":128617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70199999999999996,"15min":0.69599999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.88800000000000001,"5min":1.006,"15min":1.006},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.56899999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.93899999999999995,"5min":1.083,"15min":1.083},"last":0.64500000000000002}]},{"osd":1,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.71099999999999997,"15min":0.70599999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.96099999999999997,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71799999999999997,"15min":0.70999999999999996},"min":{"1min":0.53300000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.97799999999999998,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.73299999999999998}]},{"osd":2,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.71399999999999997,"15min":0.72999999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.90300000000000002,"5min":1.022,"15min":1.022},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.74199999999999999,"15min":0.73699999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.80300000000000005}]},{"osd":3,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78600000000000003,"15min":0.78900000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.97499999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.79500000000000004,"15min":0.81100000000000005},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.88800000000000001,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.66300000000000003}]},{"osd":4,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.64400000000000002,"15min":0.65700000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.71099999999999997,"15min":0.751},"min":{"1min":0.496,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84099999999999997,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.77400000000000002}]},{"osd":6,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.69399999999999995,"15min":0.72299999999999998},"min":{"1min":0.46899999999999997,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.95099999999999996,"5min":1.069,"15min":1.069},"last":0.622},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.70999999999999996,"15min":0.72299999999999998},"min":{"1min":0.49199999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.84599999999999997,"5min":1.264,"15min":1.264},"last":0.83899999999999997}]},{"osd":7,"last update":"Mon Oct 16 20:39:46 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76200000000000001,"15min":0.79000000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.026,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.77200000000000002,"15min":0.80000000000000004},"min":{"1min":0.50900000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.96499999999999997,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.85599999999999998}]}]},{"osd":4,"up_from":15,"seq":64424509481,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":331776,"data_stored":126015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.72199999999999998,"15min":0.72399999999999998},"min":{"1min":0.45200000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.872,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.59599999999999997,"5min":0.5,"15min":0.5},"max":{"1min":0.88200000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.82099999999999995}]},{"osd":1,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.48999999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.74199999999999999,"15min":0.746},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.59499999999999997}]},{"osd":2,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.752,"15min":0.75800000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.66200000000000003}]},{"osd":3,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.75800000000000001,"15min":0.74199999999999999},"min":{"1min":0.54700000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.77000000000000002,"15min":0.78200000000000003},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":5,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.627,"15min":0.64500000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.82199999999999995,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.64700000000000002,"15min":0.65100000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.91700000000000004,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.73299999999999998}]},{"osd":6,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.71899999999999997,"15min":0.73299999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.90600000000000003,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.69099999999999995}]},{"osd":7,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.72799999999999998,"15min":0.745},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.94099999999999995,"5min":1.5860000000000001,"15min":1.5860000000000001},"last":0.621},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.77800000000000002,"15min":0.79800000000000004},"min":{"1min":0.52800000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1080000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.85699999999999998}]}]},{"osd":3,"up_from":15,"seq":64424509481,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":129790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.503,"15min":0.48299999999999998},"min":{"1min":0.30099999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.3479999999999999,"5min":4.3479999999999999,"15min":4.3479999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.53700000000000003,"15min":0.51900000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":4.29,"5min":4.29,"15min":4.29},"last":0.40300000000000002}]},{"osd":1,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.52200000000000002,"15min":0.48799999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":4.5800000000000001,"5min":4.5800000000000001,"15min":4.5800000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.59699999999999998,"15min":0.56799999999999995},"min":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":4.7190000000000003,"5min":4.7190000000000003,"15min":4.7190000000000003},"last":0.48199999999999998}]},{"osd":2,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.56299999999999994,"15min":0.55200000000000005},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.3899999999999997,"5min":4.3899999999999997,"15min":4.3899999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.65900000000000003,"15min":0.628},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":0.44900000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.82799999999999996,"15min":0.80800000000000005},"min":{"1min":0.56799999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":4.0629999999999997,"5min":4.0629999999999997,"15min":4.0629999999999997},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77700000000000002,"15min":0.76600000000000001},"min":{"1min":0.56399999999999995,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.026,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.752}]},{"osd":5,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.86799999999999999,"15min":0.876},"min":{"1min":0.57699999999999996,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":3.9529999999999998,"5min":3.9529999999999998,"15min":3.9529999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.85399999999999998,"15min":0.83899999999999997},"min":{"1min":0.60599999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":4.4960000000000004,"5min":4.4960000000000004,"15min":4.4960000000000004},"last":0.76900000000000002}]},{"osd":6,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.76000000000000001,"15min":0.76800000000000002},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.86699999999999999,"15min":0.86199999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.81899999999999995}]},{"osd":7,"last update":"Mon Oct 16 20:39:48 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.83299999999999996,"15min":0.80900000000000005},"min":{"1min":0.56699999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":4.0049999999999999,"5min":4.0049999999999999,"15min":4.0049999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.749,"5min":0.79000000000000004,"15min":0.79800000000000004},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":0.93799999999999994,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.64600000000000002}]}]},{"osd":2,"up_from":15,"seq":64424509481,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":126147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.46700000000000003,"15min":0.47499999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70399999999999996,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.51900000000000002,"15min":0.52500000000000002},"min":{"1min":0.35899999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.68700000000000006,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.29699999999999999}]},{"osd":1,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.51500000000000001,"15min":0.52600000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.74299999999999999,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.36499999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.64700000000000002,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.47799999999999998}]},{"osd":3,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.80400000000000005,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57799999999999996,"15min":0.59299999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.74399999999999999,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.40400000000000003}]},{"osd":4,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.80900000000000005,"15min":0.82399999999999995},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.02,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.626},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.75900000000000001,"15min":0.76100000000000001},"min":{"1min":0.56200000000000006,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.92600000000000005,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.51600000000000001}]},{"osd":5,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.77200000000000002,"15min":0.78000000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.04,"5min":1.444,"15min":1.444},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.81799999999999995,"15min":0.83499999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.052,"5min":1.585,"15min":1.585},"last":0.52700000000000002}]},{"osd":6,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.82399999999999995,"15min":0.84399999999999997},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":0.96099999999999997,"5min":1.335,"15min":1.335},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.84099999999999997,"15min":0.872},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.036,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.63900000000000001}]},{"osd":7,"last update":"Mon Oct 16 20:39:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.83299999999999996,"15min":0.86399999999999999},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.93500000000000005,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.86099999999999999,"15min":0.88500000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.0820000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.59299999999999997}]}]},{"osd":1,"up_from":15,"seq":64424509481,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":125777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.45100000000000001,"15min":0.46300000000000002},"min":{"1min":0.29199999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69199999999999995,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.51500000000000001,"15min":0.52800000000000002},"min":{"1min":0.318,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.97899999999999998,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.52400000000000002,"15min":0.54400000000000004},"min":{"1min":0.32600000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66900000000000004,"5min":1.046,"15min":1.046},"last":0.377},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.55800000000000005,"15min":0.58199999999999996},"min":{"1min":0.33700000000000002,"5min":0.314,"15min":0.314},"max":{"1min":1.0740000000000001,"5min":1.115,"15min":1.115},"last":0.33700000000000002}]},{"osd":3,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.40600000000000003,"5min":0.312,"15min":0.312},"max":{"1min":1.155,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72699999999999998,"5min":1.431,"15min":1.431},"last":0.497}]},{"osd":4,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.81899999999999995,"15min":0.83099999999999996},"min":{"1min":0.56100000000000005,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.80700000000000005,"15min":0.80900000000000005},"min":{"1min":0.61199999999999999,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.115,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.628}]},{"osd":5,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.78900000000000003,"15min":0.81299999999999994},"min":{"1min":0.58399999999999996,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90200000000000002,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.84699999999999998,"15min":0.86399999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2490000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.59799999999999998}]},{"osd":6,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.84999999999999998,"15min":0.873},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.016,"5min":2.004,"15min":2.004},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.85399999999999998,"15min":0.876},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.3300000000000001,"5min":1.635,"15min":1.635},"last":0.67100000000000004}]},{"osd":7,"last update":"Mon Oct 16 20:39:51 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.85299999999999998,"15min":0.88300000000000001},"min":{"1min":0.58999999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0249999999999999,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.878,"15min":0.90300000000000002},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0489999999999999,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.70399999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343916,"statfs":{"total":96636764160,"available":96608169984,"internally_reserved":0,"allocated":929792,"data_stored":715487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.439,"15min":0.438},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.495,"15min":0.496},"min":{"1min":0.374,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.58999999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.36499999999999999}]},{"osd":2,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52300000000000002,"15min":0.52200000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.433},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54100000000000004,"15min":0.54600000000000004},"min":{"1min":0.40400000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.68899999999999995,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.47499999999999998}]},{"osd":3,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54800000000000004,"15min":0.54500000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.73399999999999999,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.56499999999999995,"5min":0.497,"15min":0.497},"max":{"1min":0.84999999999999998,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.49199999999999999}]},{"osd":4,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.83799999999999997,"15min":0.82999999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0600000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.85999999999999999,"15min":0.84899999999999998},"min":{"1min":0.61399999999999999,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.80800000000000005}]},{"osd":5,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.80800000000000005,"15min":0.80300000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.66800000000000004}]},{"osd":6,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.86299999999999999,"15min":0.85899999999999999},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.755},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88400000000000001,"15min":0.88300000000000001},"min":{"1min":0.60899999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.79400000000000004}]},{"osd":7,"last update":"Mon Oct 16 20:38:52 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.64900000000000002,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.83499999999999996,"15min":0.82299999999999995},"min":{"1min":0.66500000000000004,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T20:39:54.599 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:39:54.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.768+0000 7f8be3068640 1 Processor -- start 2023-10-16T20:39:54.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.768+0000 7f8be3068640 1 -- start start 2023-10-16T20:39:54.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.768+0000 7f8be3068640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8bdc0c9bb0 0x7f8bdc0c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.768+0000 7f8be3068640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8bdc0c41f0 0x7f8bdc0cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.768+0000 7f8be3068640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bdc153b50 con 0x7f8bdc153770 2023-10-16T20:39:54.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.768+0000 7f8be3068640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bdc0cc250 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.768+0000 7f8be3068640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bdc0cc670 con 0x7f8bdc0c41f0 2023-10-16T20:39:54.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.768+0000 7f8bdbfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8bdc0c9bb0 0x7f8bdc0c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:54.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.768+0000 7f8bdbfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8bdc0c9bb0 0x7f8bdc0c3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47782/0 (socket says 172.21.15.88:47782) 2023-10-16T20:39:54.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.768+0000 7f8bdbfff640 1 -- 172.21.15.88:0/2807096240 learned_addr learned my addr 172.21.15.88:0/2807096240 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:54.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.768+0000 7f8be1605640 1 --2- 172.21.15.88:0/2807096240 >> v2:172.21.15.191:3300/0 conn(0x7f8bdc0c41f0 0x7f8bdc0cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:54.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.768+0000 7f8bdbfff640 1 -- 172.21.15.88:0/2807096240 >> v2:172.21.15.191:3300/0 conn(0x7f8bdc0c41f0 msgr2=0x7f8bdc0cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:54.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.768+0000 7f8bdbfff640 1 --2- 172.21.15.88:0/2807096240 >> v2:172.21.15.191:3300/0 conn(0x7f8bdc0c41f0 0x7f8bdc0cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.768+0000 7f8bdbfff640 1 -- 172.21.15.88:0/2807096240 >> v1:172.21.15.191:6789/0 conn(0x7f8bdc153770 legacy=0x7f8bdc159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.768+0000 7f8bdbfff640 1 -- 172.21.15.88:0/2807096240 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bdc0cb2f0 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.772+0000 7f8bdbfff640 1 --2- 172.21.15.88:0/2807096240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bdc0c9bb0 0x7f8bdc0c3cb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc0b19e0 tx=0x7f8bc801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9d049ea8c4893e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:54.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.772+0000 7f8bdb7fe640 1 -- 172.21.15.88:0/2807096240 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc80ab610 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.772+0000 7f8bdb7fe640 1 -- 172.21.15.88:0/2807096240 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc80ab770 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.772+0000 7f8bdb7fe640 1 -- 172.21.15.88:0/2807096240 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc80abb00 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.772+0000 7f8be3068640 1 -- 172.21.15.88:0/2807096240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bdc0c9bb0 msgr2=0x7f8bdc0c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.772+0000 7f8be3068640 1 --2- 172.21.15.88:0/2807096240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bdc0c9bb0 0x7f8bdc0c3cb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc0b19e0 tx=0x7f8bc801c500 comp rx=0 tx=0).stop 2023-10-16T20:39:54.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.772+0000 7f8be3068640 1 -- 172.21.15.88:0/2807096240 shutdown_connections 2023-10-16T20:39:54.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.772+0000 7f8be3068640 1 --2- 172.21.15.88:0/2807096240 >> v2:172.21.15.191:3300/0 conn(0x7f8bdc0c41f0 0x7f8bdc0cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.772+0000 7f8be3068640 1 --2- 172.21.15.88:0/2807096240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bdc0c9bb0 0x7f8bdc0c3cb0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.772+0000 7f8be3068640 1 -- 172.21.15.88:0/2807096240 >> 172.21.15.88:0/2807096240 conn(0x7f8bdc0715c0 msgr2=0x7f8bdc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:54.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.772+0000 7f8be3068640 1 -- 172.21.15.88:0/2807096240 shutdown_connections 2023-10-16T20:39:54.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.772+0000 7f8be3068640 1 -- 172.21.15.88:0/2807096240 wait complete. 2023-10-16T20:39:54.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8be3068640 1 Processor -- start 2023-10-16T20:39:54.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8be3068640 1 -- start start 2023-10-16T20:39:54.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8be3068640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bdc0c41f0 0x7f8bdc1469f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8be3068640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bdc0c9bb0 0x7f8bdc14c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8be3068640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bdc153770 0x7f8bdc14c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8be3068640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc1ebd10 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8be3068640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc1ebf60 con 0x7f8bdc153770 2023-10-16T20:39:54.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8be3068640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc1ec240 con 0x7f8bdc0c41f0 2023-10-16T20:39:54.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8bdbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bdc0c9bb0 0x7f8bdc14c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:54.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8bdbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bdc0c9bb0 0x7f8bdc14c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47784/0 (socket says 172.21.15.88:47784) 2023-10-16T20:39:54.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8bdbfff640 1 -- 172.21.15.88:0/3271877079 learned_addr learned my addr 172.21.15.88:0/3271877079 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:54.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8be1605640 1 --2- 172.21.15.88:0/3271877079 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bdc153770 0x7f8bdc14c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:54.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8be0e04640 1 --2- 172.21.15.88:0/3271877079 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bdc0c41f0 0x7f8bdc1469f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:54.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8bdbfff640 1 -- 172.21.15.88:0/3271877079 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bdc0c41f0 msgr2=0x7f8bdc1469f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8bdbfff640 1 --2- 172.21.15.88:0/3271877079 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bdc0c41f0 0x7f8bdc1469f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8bdbfff640 1 -- 172.21.15.88:0/3271877079 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bdc153770 msgr2=0x7f8bdc14c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8bdbfff640 1 --2- 172.21.15.88:0/3271877079 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bdc153770 0x7f8bdc14c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8bdbfff640 1 -- 172.21.15.88:0/3271877079 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc80094d0 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8bdbfff640 1 --2- 172.21.15.88:0/3271877079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bdc0c9bb0 0x7f8bdc14c1c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8019110 tx=0x7f8bc801ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:54.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8bd97fa640 1 -- 172.21.15.88:0/3271877079 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc80ab610 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8bd97fa640 1 -- 172.21.15.88:0/3271877079 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc80b4800 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8bd97fa640 1 -- 172.21.15.88:0/3271877079 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc80b4bb0 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8be3068640 1 -- 172.21.15.88:0/3271877079 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bdc14b9c0 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8be3068640 1 -- 172.21.15.88:0/3271877079 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bdc14b6c0 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.776+0000 7f8be3068640 1 -- 172.21.15.88:0/3271877079 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bdc074700 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.780+0000 7f8bd97fa640 1 -- 172.21.15.88:0/3271877079 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8bc80b4d10 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.780+0000 7f8bd97fa640 1 --2- 172.21.15.88:0/3271877079 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8bc00664a0 0x7f8bc0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:54.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.780+0000 7f8bd97fa640 1 -- 172.21.15.88:0/3271877079 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8bc811e7e0 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.780+0000 7f8be0e04640 1 --2- 172.21.15.88:0/3271877079 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8bc00664a0 0x7f8bc0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:54.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.780+0000 7f8be0e04640 1 --2- 172.21.15.88:0/3271877079 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8bc00664a0 0x7f8bc0068960 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0000fc0 tx=0x7f8bd0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:54.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.784+0000 7f8bd97fa640 1 -- 172.21.15.88:0/3271877079 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8bc80eba60 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.936+0000 7f8be3068640 1 -- 172.21.15.88:0/3271877079 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bdc1510a0 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.936+0000 7f8bd97fa640 1 -- 172.21.15.88:0/3271877079 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18710 (secure 0 0 0) 0x7f8bc80ef080 con 0x7f8bdc0c9bb0 2023-10-16T20:39:54.941 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:54.942 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:37:37.392549+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:39:54.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.940+0000 7f8be3068640 1 -- 172.21.15.88:0/3271877079 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8bc00664a0 msgr2=0x7f8bc0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.940+0000 7f8be3068640 1 --2- 172.21.15.88:0/3271877079 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8bc00664a0 0x7f8bc0068960 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0000fc0 tx=0x7f8bd0009040 comp rx=0 tx=0).stop 2023-10-16T20:39:54.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.940+0000 7f8be3068640 1 -- 172.21.15.88:0/3271877079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bdc0c9bb0 msgr2=0x7f8bdc14c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:54.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.940+0000 7f8be3068640 1 --2- 172.21.15.88:0/3271877079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bdc0c9bb0 0x7f8bdc14c1c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8019110 tx=0x7f8bc801ad40 comp rx=0 tx=0).stop 2023-10-16T20:39:54.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.940+0000 7f8be3068640 1 -- 172.21.15.88:0/3271877079 shutdown_connections 2023-10-16T20:39:54.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.940+0000 7f8be3068640 1 --2- 172.21.15.88:0/3271877079 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8bc00664a0 0x7f8bc0068960 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.940+0000 7f8be3068640 1 --2- 172.21.15.88:0/3271877079 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8bdc153770 0x7f8bdc14c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.940+0000 7f8be3068640 1 --2- 172.21.15.88:0/3271877079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bdc0c9bb0 0x7f8bdc14c1c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.940+0000 7f8be3068640 1 --2- 172.21.15.88:0/3271877079 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8bdc0c41f0 0x7f8bdc1469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:54.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.940+0000 7f8be3068640 1 -- 172.21.15.88:0/3271877079 >> 172.21.15.88:0/3271877079 conn(0x7f8bdc0715c0 msgr2=0x7f8bdc0cde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:54.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.940+0000 7f8be3068640 1 -- 172.21.15.88:0/3271877079 shutdown_connections 2023-10-16T20:39:54.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:54.940+0000 7f8be3068640 1 -- 172.21.15.88:0/3271877079 wait complete. 2023-10-16T20:39:54.963 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T20:36:43.646418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T20:36:53.546513+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T20:36:54.143703+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T20:36:57.314642+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T20:39:54.963 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T20:39:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.100+0000 7fee781d0640 1 Processor -- start 2023-10-16T20:39:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.100+0000 7fee781d0640 1 -- start start 2023-10-16T20:39:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.100+0000 7fee781d0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fee700c4640 0x7fee701510d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.100+0000 7fee781d0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee70069e90 con 0x7fee70151610 2023-10-16T20:39:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.100+0000 7fee781d0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee7006aea0 con 0x7fee700c4260 2023-10-16T20:39:55.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.100+0000 7fee781d0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee700716a0 con 0x7fee700c4640 2023-10-16T20:39:55.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.100+0000 7fee7576b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fee700c4640 0x7fee701510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.100+0000 7fee7676d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fee70151610 0x7fee701573e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58460/0 (socket says 172.21.15.88:58460) 2023-10-16T20:39:55.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.100+0000 7fee7676d640 1 -- 172.21.15.88:0/690841778 learned_addr learned my addr 172.21.15.88:0/690841778 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:55.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.100+0000 7fee7576b640 1 -- 172.21.15.88:0/690841778 >> v1:172.21.15.191:6789/0 conn(0x7fee700c4260 legacy=0x7fee700c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.100+0000 7fee7576b640 1 -- 172.21.15.88:0/690841778 >> v1:172.21.15.88:6789/0 conn(0x7fee70151610 legacy=0x7fee701573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.104+0000 7fee7576b640 1 -- 172.21.15.88:0/690841778 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee701581e0 con 0x7fee700c4640 2023-10-16T20:39:55.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.104+0000 7fee7576b640 1 --2- 172.21.15.88:0/690841778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee700c4640 0x7fee701510d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fee600097c0 tx=0x7fee6001c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c8d0e76ef32101 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:55.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.104+0000 7fee74f6a640 1 -- 172.21.15.88:0/690841778 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee600ab600 con 0x7fee700c4640 2023-10-16T20:39:55.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.104+0000 7fee74f6a640 1 -- 172.21.15.88:0/690841778 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee600a7050 con 0x7fee700c4640 2023-10-16T20:39:55.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.104+0000 7fee74f6a640 1 -- 172.21.15.88:0/690841778 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee600ab980 con 0x7fee700c4640 2023-10-16T20:39:55.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.104+0000 7fee781d0640 1 -- 172.21.15.88:0/690841778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee700c4640 msgr2=0x7fee701510d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.104+0000 7fee781d0640 1 --2- 172.21.15.88:0/690841778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee700c4640 0x7fee701510d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fee600097c0 tx=0x7fee6001c330 comp rx=0 tx=0).stop 2023-10-16T20:39:55.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.104+0000 7fee781d0640 1 -- 172.21.15.88:0/690841778 shutdown_connections 2023-10-16T20:39:55.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.104+0000 7fee781d0640 1 --2- 172.21.15.88:0/690841778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee700c4640 0x7fee701510d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.104+0000 7fee781d0640 1 -- 172.21.15.88:0/690841778 >> 172.21.15.88:0/690841778 conn(0x7fee700ca9f0 msgr2=0x7fee700cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:55.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.104+0000 7fee781d0640 1 -- 172.21.15.88:0/690841778 shutdown_connections 2023-10-16T20:39:55.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.104+0000 7fee781d0640 1 -- 172.21.15.88:0/690841778 wait complete. 2023-10-16T20:39:55.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee781d0640 1 Processor -- start 2023-10-16T20:39:55.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee781d0640 1 -- start start 2023-10-16T20:39:55.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee781d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee700c4260 0x7fee701491f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee781d0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee700c4640 0x7fee701f5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee781d0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee70151610 0x7fee701f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee781d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee70143460 con 0x7fee700c4260 2023-10-16T20:39:55.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee781d0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee70162bb0 con 0x7fee700c4640 2023-10-16T20:39:55.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee781d0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee701f5920 con 0x7fee70151610 2023-10-16T20:39:55.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee75f6c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee700c4260 0x7fee701491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee75f6c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee700c4260 0x7fee701491f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47806/0 (socket says 172.21.15.88:47806) 2023-10-16T20:39:55.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee75f6c640 1 -- 172.21.15.88:0/968583248 learned_addr learned my addr 172.21.15.88:0/968583248 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:55.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee7576b640 1 --2- 172.21.15.88:0/968583248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee700c4640 0x7fee701f5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee7676d640 1 --2- 172.21.15.88:0/968583248 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee70151610 0x7fee701f62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee75f6c640 1 -- 172.21.15.88:0/968583248 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee70151610 msgr2=0x7fee701f62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee75f6c640 1 --2- 172.21.15.88:0/968583248 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee70151610 0x7fee701f62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee75f6c640 1 -- 172.21.15.88:0/968583248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee700c4640 msgr2=0x7fee701f5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee75f6c640 1 --2- 172.21.15.88:0/968583248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee700c4640 0x7fee701f5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee75f6c640 1 -- 172.21.15.88:0/968583248 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee60009420 con 0x7fee700c4260 2023-10-16T20:39:55.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee75f6c640 1 --2- 172.21.15.88:0/968583248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee700c4260 0x7fee701491f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fee6c00c500 tx=0x7fee6c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:55.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee667fc640 1 -- 172.21.15.88:0/968583248 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6c005220 con 0x7fee700c4260 2023-10-16T20:39:55.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee667fc640 1 -- 172.21.15.88:0/968583248 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6c005380 con 0x7fee700c4260 2023-10-16T20:39:55.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee781d0640 1 -- 172.21.15.88:0/968583248 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee70161250 con 0x7fee700c4260 2023-10-16T20:39:55.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee667fc640 1 -- 172.21.15.88:0/968583248 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6c007270 con 0x7fee700c4260 2023-10-16T20:39:55.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.108+0000 7fee781d0640 1 -- 172.21.15.88:0/968583248 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee700c3a40 con 0x7fee700c4260 2023-10-16T20:39:55.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.112+0000 7fee781d0640 1 -- 172.21.15.88:0/968583248 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee700c38a0 con 0x7fee700c4260 2023-10-16T20:39:55.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.112+0000 7fee667fc640 1 -- 172.21.15.88:0/968583248 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fee6c0073d0 con 0x7fee700c4260 2023-10-16T20:39:55.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.112+0000 7fee667fc640 1 --2- 172.21.15.88:0/968583248 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee540664a0 0x7fee54068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.112+0000 7fee667fc640 1 -- 172.21.15.88:0/968583248 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fee6c083ab0 con 0x7fee700c4260 2023-10-16T20:39:55.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.112+0000 7fee7576b640 1 --2- 172.21.15.88:0/968583248 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee540664a0 0x7fee54068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.112+0000 7fee7576b640 1 --2- 172.21.15.88:0/968583248 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee540664a0 0x7fee54068960 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fee600ad000 tx=0x7fee60008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:55.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.116+0000 7fee667fc640 1 -- 172.21.15.88:0/968583248 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fee700c38a0 con 0x7fee700c4260 2023-10-16T20:39:55.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.268+0000 7fee781d0640 1 -- 172.21.15.88:0/968583248 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fee700c38a0 con 0x7fee700c4260 2023-10-16T20:39:55.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.268+0000 7fee667fc640 1 -- 172.21.15.88:0/968583248 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fee700c38a0 con 0x7fee700c4260 2023-10-16T20:39:55.273 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-10-16T20:39:55.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.268+0000 7fee781d0640 1 -- 172.21.15.88:0/968583248 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee540664a0 msgr2=0x7fee54068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.268+0000 7fee781d0640 1 --2- 172.21.15.88:0/968583248 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee540664a0 0x7fee54068960 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fee600ad000 tx=0x7fee60008040 comp rx=0 tx=0).stop 2023-10-16T20:39:55.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.268+0000 7fee781d0640 1 -- 172.21.15.88:0/968583248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee700c4260 msgr2=0x7fee701491f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.268+0000 7fee781d0640 1 --2- 172.21.15.88:0/968583248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee700c4260 0x7fee701491f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fee6c00c500 tx=0x7fee6c00c9c0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.268+0000 7fee781d0640 1 -- 172.21.15.88:0/968583248 shutdown_connections 2023-10-16T20:39:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.268+0000 7fee781d0640 1 --2- 172.21.15.88:0/968583248 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fee540664a0 0x7fee54068960 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.268+0000 7fee781d0640 1 --2- 172.21.15.88:0/968583248 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fee70151610 0x7fee701f62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.268+0000 7fee781d0640 1 --2- 172.21.15.88:0/968583248 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee700c4640 0x7fee701f5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.268+0000 7fee781d0640 1 --2- 172.21.15.88:0/968583248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee700c4260 0x7fee701491f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.268+0000 7fee781d0640 1 -- 172.21.15.88:0/968583248 >> 172.21.15.88:0/968583248 conn(0x7fee700ca9f0 msgr2=0x7fee700c88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:55.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.268+0000 7fee781d0640 1 -- 172.21.15.88:0/968583248 shutdown_connections 2023-10-16T20:39:55.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.268+0000 7fee781d0640 1 -- 172.21.15.88:0/968583248 wait complete. 2023-10-16T20:39:55.294 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T20:39:55.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.432+0000 7f14c4965640 1 Processor -- start 2023-10-16T20:39:55.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.432+0000 7f14c4965640 1 -- start start 2023-10-16T20:39:55.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.432+0000 7f14c4965640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14c00c66e0 0x7f14c015a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.432+0000 7f14c4965640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f14c00c6ac0 0x7f14c0150150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.432+0000 7f14c4965640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c0070f40 con 0x7f14c00cb5c0 2023-10-16T20:39:55.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.432+0000 7f14c4965640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c015a790 con 0x7f14c00c6ac0 2023-10-16T20:39:55.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.432+0000 7f14c4965640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c006a2d0 con 0x7f14c00c66e0 2023-10-16T20:39:55.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bed9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f14c00cb5c0 0x7f14c0071420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58470/0 (socket says 172.21.15.88:58470) 2023-10-16T20:39:55.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bed9d640 1 -- 172.21.15.88:0/1128311021 learned_addr learned my addr 172.21.15.88:0/1128311021 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:55.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14b5d9b640 1 --2- 172.21.15.88:0/1128311021 >> v2:172.21.15.191:3300/0 conn(0x7f14c00c6ac0 0x7f14c0150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14be59c640 1 --2- 172.21.15.88:0/1128311021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14c00c66e0 0x7f14c015a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bdd9b640 1 -- 172.21.15.88:0/1128311021 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461548917 0 0) 0x7f14c0070f40 con 0x7f14c00cb5c0 2023-10-16T20:39:55.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bdd9b640 1 -- 172.21.15.88:0/1128311021 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14ac003530 con 0x7f14c00cb5c0 2023-10-16T20:39:55.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bdd9b640 1 -- 172.21.15.88:0/1128311021 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3172108090 0 0) 0x7f14ac003530 con 0x7f14c00cb5c0 2023-10-16T20:39:55.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bdd9b640 1 -- 172.21.15.88:0/1128311021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14c00c66e0 msgr2=0x7f14c015a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:55.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bdd9b640 1 --2- 172.21.15.88:0/1128311021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14c00c66e0 0x7f14c015a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bdd9b640 1 -- 172.21.15.88:0/1128311021 >> v2:172.21.15.191:3300/0 conn(0x7f14c00c6ac0 msgr2=0x7f14c0150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:55.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bdd9b640 1 --2- 172.21.15.88:0/1128311021 >> v2:172.21.15.191:3300/0 conn(0x7f14c00c6ac0 0x7f14c0150150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bdd9b640 1 -- 172.21.15.88:0/1128311021 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c006a5b0 con 0x7f14c00cb5c0 2023-10-16T20:39:55.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bdd9b640 1 -- 172.21.15.88:0/1128311021 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f14b8001a00 con 0x7f14c00cb5c0 2023-10-16T20:39:55.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14b5d9b640 1 --2- 172.21.15.88:0/1128311021 >> v2:172.21.15.191:3300/0 conn(0x7f14c00c6ac0 0x7f14c0150150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:39:55.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bdd9b640 1 -- 172.21.15.88:0/1128311021 >> v1:172.21.15.88:6789/0 conn(0x7f14c00cb5c0 legacy=0x7f14c0071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bdd9b640 1 --2- 172.21.15.88:0/1128311021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14c00c66e0 0x7f14ac003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bdd9b640 1 --2- 172.21.15.88:0/1128311021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14ac003f00 0x7f14ac0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bdd9b640 1 --2- 172.21.15.88:0/1128311021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14ac004d10 0x7f14ac005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bdd9b640 1 -- 172.21.15.88:0/1128311021 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c015a790 con 0x7f14ac004d10 2023-10-16T20:39:55.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bdd9b640 1 -- 172.21.15.88:0/1128311021 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c006a2d0 con 0x7f14ac003f00 2023-10-16T20:39:55.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bdd9b640 1 -- 172.21.15.88:0/1128311021 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac00b9e0 con 0x7f14c00c66e0 2023-10-16T20:39:55.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bed9d640 1 --2- 172.21.15.88:0/1128311021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14ac004d10 0x7f14ac005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14b5d9b640 1 --2- 172.21.15.88:0/1128311021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14ac003f00 0x7f14ac0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14be59c640 1 --2- 172.21.15.88:0/1128311021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14c00c66e0 0x7f14ac003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bed9d640 1 -- 172.21.15.88:0/1128311021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14c00c66e0 msgr2=0x7f14ac003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:55.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bed9d640 1 --2- 172.21.15.88:0/1128311021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14c00c66e0 0x7f14ac003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bed9d640 1 -- 172.21.15.88:0/1128311021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14ac003f00 msgr2=0x7f14ac0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:55.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bed9d640 1 --2- 172.21.15.88:0/1128311021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14ac003f00 0x7f14ac0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.436+0000 7f14bed9d640 1 -- 172.21.15.88:0/1128311021 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac003530 con 0x7f14ac004d10 2023-10-16T20:39:55.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.440+0000 7f14bed9d640 1 --2- 172.21.15.88:0/1128311021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14ac004d10 0x7f14ac005190 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f14b800a890 tx=0x7f14b801fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f733124d172480f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:55.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.440+0000 7f14bdd9b640 1 -- 172.21.15.88:0/1128311021 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b80aa6f0 con 0x7f14ac004d10 2023-10-16T20:39:55.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.440+0000 7f14bdd9b640 1 -- 172.21.15.88:0/1128311021 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b80aa850 con 0x7f14ac004d10 2023-10-16T20:39:55.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.440+0000 7f14bdd9b640 1 -- 172.21.15.88:0/1128311021 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b80aac60 con 0x7f14ac004d10 2023-10-16T20:39:55.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.440+0000 7f14c4965640 1 -- 172.21.15.88:0/1128311021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14ac004d10 msgr2=0x7f14ac005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.440+0000 7f14c4965640 1 --2- 172.21.15.88:0/1128311021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14ac004d10 0x7f14ac005190 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f14b800a890 tx=0x7f14b801fac0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.440+0000 7f14bed9d640 1 -- 172.21.15.88:0/1128311021 reap_dead start 2023-10-16T20:39:55.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.440+0000 7f14c4965640 1 -- 172.21.15.88:0/1128311021 shutdown_connections 2023-10-16T20:39:55.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.440+0000 7f14c4965640 1 -- 172.21.15.88:0/1128311021 >> 172.21.15.88:0/1128311021 conn(0x7f14c00ca9f0 msgr2=0x7f14c00cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:55.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.440+0000 7f14c4965640 1 -- 172.21.15.88:0/1128311021 shutdown_connections 2023-10-16T20:39:55.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.440+0000 7f14c4965640 1 -- 172.21.15.88:0/1128311021 wait complete. 2023-10-16T20:39:55.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.440+0000 7f14c4965640 1 Processor -- start 2023-10-16T20:39:55.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.440+0000 7f14c4965640 1 -- start start 2023-10-16T20:39:55.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14c4965640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14ac004d10 0x7f14c01ed660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14c4965640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14c00c3890 0x7f14c01edba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14c4965640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14c0149da0 0x7f14c0147690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14c4965640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c01ee260 con 0x7f14ac004d10 2023-10-16T20:39:55.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14c4965640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c00c3600 con 0x7f14c00c3890 2023-10-16T20:39:55.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14c4965640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c00c2f60 con 0x7f14c0149da0 2023-10-16T20:39:55.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14be59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14ac004d10 0x7f14c01ed660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14be59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14ac004d10 0x7f14c01ed660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47824/0 (socket says 172.21.15.88:47824) 2023-10-16T20:39:55.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14be59c640 1 -- 172.21.15.88:0/2153622617 learned_addr learned my addr 172.21.15.88:0/2153622617 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14b5d9b640 1 --2- 172.21.15.88:0/2153622617 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14c00c3890 0x7f14c01edba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14bed9d640 1 --2- 172.21.15.88:0/2153622617 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14c0149da0 0x7f14c0147690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14be59c640 1 -- 172.21.15.88:0/2153622617 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14c0149da0 msgr2=0x7f14c0147690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14be59c640 1 --2- 172.21.15.88:0/2153622617 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14c0149da0 0x7f14c0147690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14be59c640 1 -- 172.21.15.88:0/2153622617 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14c00c3890 msgr2=0x7f14c01edba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14be59c640 1 --2- 172.21.15.88:0/2153622617 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14c00c3890 0x7f14c01edba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14be59c640 1 -- 172.21.15.88:0/2153622617 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c00c3f90 con 0x7f14ac004d10 2023-10-16T20:39:55.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14be59c640 1 --2- 172.21.15.88:0/2153622617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14ac004d10 0x7f14c01ed660 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f14a80119d0 tx=0x7f14a8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:55.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14b77fe640 1 -- 172.21.15.88:0/2153622617 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a800fb50 con 0x7f14ac004d10 2023-10-16T20:39:55.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14b77fe640 1 -- 172.21.15.88:0/2153622617 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a800fcb0 con 0x7f14ac004d10 2023-10-16T20:39:55.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14b77fe640 1 -- 172.21.15.88:0/2153622617 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a8014440 con 0x7f14ac004d10 2023-10-16T20:39:55.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14c4965640 1 -- 172.21.15.88:0/2153622617 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c00c4270 con 0x7f14ac004d10 2023-10-16T20:39:55.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14c4965640 1 -- 172.21.15.88:0/2153622617 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c0156f20 con 0x7f14ac004d10 2023-10-16T20:39:55.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14c4965640 1 -- 172.21.15.88:0/2153622617 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c00c3ce0 con 0x7f14ac004d10 2023-10-16T20:39:55.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.444+0000 7f14b77fe640 1 -- 172.21.15.88:0/2153622617 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f14a80145a0 con 0x7f14ac004d10 2023-10-16T20:39:55.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.448+0000 7f14b77fe640 1 --2- 172.21.15.88:0/2153622617 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f14a00664c0 0x7f14a0068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.448+0000 7f14b77fe640 1 -- 172.21.15.88:0/2153622617 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f14a8084a10 con 0x7f14ac004d10 2023-10-16T20:39:55.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.448+0000 7f14b5d9b640 1 --2- 172.21.15.88:0/2153622617 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f14a00664c0 0x7f14a0068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.448+0000 7f14b5d9b640 1 --2- 172.21.15.88:0/2153622617 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f14a00664c0 0x7f14a0068980 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f14c00c9f60 tx=0x7f14b0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:55.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.452+0000 7f14b77fe640 1 -- 172.21.15.88:0/2153622617 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f14a8051b90 con 0x7f14ac004d10 2023-10-16T20:39:55.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.600+0000 7f14c4965640 1 -- 172.21.15.88:0/2153622617 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f14c014cd10 con 0x7f14ac004d10 2023-10-16T20:39:55.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.600+0000 7f14b77fe640 1 -- 172.21.15.88:0/2153622617 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f14a8022020 con 0x7f14ac004d10 2023-10-16T20:39:55.606 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:39:55.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.604+0000 7f14c4965640 1 -- 172.21.15.88:0/2153622617 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f14a00664c0 msgr2=0x7f14a0068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.604+0000 7f14c4965640 1 --2- 172.21.15.88:0/2153622617 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f14a00664c0 0x7f14a0068980 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f14c00c9f60 tx=0x7f14b0009190 comp rx=0 tx=0).stop 2023-10-16T20:39:55.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.604+0000 7f14c4965640 1 -- 172.21.15.88:0/2153622617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14ac004d10 msgr2=0x7f14c01ed660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.604+0000 7f14c4965640 1 --2- 172.21.15.88:0/2153622617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14ac004d10 0x7f14c01ed660 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f14a80119d0 tx=0x7f14a8011e90 comp rx=0 tx=0).stop 2023-10-16T20:39:55.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.604+0000 7f14c4965640 1 -- 172.21.15.88:0/2153622617 shutdown_connections 2023-10-16T20:39:55.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.604+0000 7f14c4965640 1 --2- 172.21.15.88:0/2153622617 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f14a00664c0 0x7f14a0068980 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.604+0000 7f14c4965640 1 --2- 172.21.15.88:0/2153622617 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f14c0149da0 0x7f14c0147690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.604+0000 7f14c4965640 1 --2- 172.21.15.88:0/2153622617 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14c00c3890 0x7f14c01edba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.604+0000 7f14c4965640 1 --2- 172.21.15.88:0/2153622617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14ac004d10 0x7f14c01ed660 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.604+0000 7f14c4965640 1 -- 172.21.15.88:0/2153622617 >> 172.21.15.88:0/2153622617 conn(0x7f14c00ca9f0 msgr2=0x7f14c00c9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:55.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.604+0000 7f14c4965640 1 -- 172.21.15.88:0/2153622617 shutdown_connections 2023-10-16T20:39:55.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.604+0000 7f14c4965640 1 -- 172.21.15.88:0/2153622617 wait complete. 2023-10-16T20:39:55.627 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T20:39:55.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.764+0000 7fa726f0f640 1 Processor -- start 2023-10-16T20:39:55.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.764+0000 7fa726f0f640 1 -- start start 2023-10-16T20:39:55.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.764+0000 7fa726f0f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa7201537d0 0x7fa72015aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.764+0000 7fa726f0f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72006cbe0 con 0x7fa7200cdb20 2023-10-16T20:39:55.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.764+0000 7fa726f0f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72015d8b0 con 0x7fa72006dd70 2023-10-16T20:39:55.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.764+0000 7fa726f0f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa72006f5f0 con 0x7fa7201537d0 2023-10-16T20:39:55.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.764+0000 7fa717fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa7200cdb20 0x7fa7200ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58476/0 (socket says 172.21.15.88:58476) 2023-10-16T20:39:55.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.764+0000 7fa717fff640 1 -- 172.21.15.88:0/128303628 learned_addr learned my addr 172.21.15.88:0/128303628 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:55.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.764+0000 7fa724cab640 1 --2- 172.21.15.88:0/128303628 >> v2:172.21.15.191:3300/0 conn(0x7fa7201537d0 0x7fa72015aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.764+0000 7fa7177fe640 1 -- 172.21.15.88:0/128303628 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535602785 0 0) 0x7fa72006cbe0 con 0x7fa7200cdb20 2023-10-16T20:39:55.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.764+0000 7fa7177fe640 1 -- 172.21.15.88:0/128303628 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa708003530 con 0x7fa7200cdb20 2023-10-16T20:39:55.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa7177fe640 1 -- 172.21.15.88:0/128303628 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 292849958 0 0) 0x7fa708003530 con 0x7fa7200cdb20 2023-10-16T20:39:55.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa7177fe640 1 -- 172.21.15.88:0/128303628 >> v2:172.21.15.191:3300/0 conn(0x7fa7201537d0 msgr2=0x7fa72015aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:55.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa7177fe640 1 --2- 172.21.15.88:0/128303628 >> v2:172.21.15.191:3300/0 conn(0x7fa7201537d0 0x7fa72015aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa7177fe640 1 -- 172.21.15.88:0/128303628 >> v1:172.21.15.191:6789/0 conn(0x7fa72006dd70 legacy=0x7fa720163ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa7177fe640 1 -- 172.21.15.88:0/128303628 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72006e830 con 0x7fa7200cdb20 2023-10-16T20:39:55.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa7177fe640 1 -- 172.21.15.88:0/128303628 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7fa710001680 con 0x7fa7200cdb20 2023-10-16T20:39:55.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa7177fe640 1 -- 172.21.15.88:0/128303628 >> v1:172.21.15.88:6789/0 conn(0x7fa7200cdb20 legacy=0x7fa7200ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa7177fe640 1 --2- 172.21.15.88:0/128303628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa708003980 0x7fa708003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa7177fe640 1 --2- 172.21.15.88:0/128303628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa708004380 0x7fa708004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa7177fe640 1 --2- 172.21.15.88:0/128303628 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa708005050 0x7fa70800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa7177fe640 1 -- 172.21.15.88:0/128303628 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72006f5f0 con 0x7fa708003980 2023-10-16T20:39:55.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa7177fe640 1 -- 172.21.15.88:0/128303628 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70800bfb0 con 0x7fa708004380 2023-10-16T20:39:55.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa7177fe640 1 -- 172.21.15.88:0/128303628 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa70800c170 con 0x7fa708005050 2023-10-16T20:39:55.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa724cab640 1 --2- 172.21.15.88:0/128303628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa708003980 0x7fa708003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa717fff640 1 --2- 172.21.15.88:0/128303628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa708004380 0x7fa708004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa7254ac640 1 --2- 172.21.15.88:0/128303628 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa708005050 0x7fa70800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa724cab640 1 -- 172.21.15.88:0/128303628 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa708005050 msgr2=0x7fa70800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:55.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa724cab640 1 --2- 172.21.15.88:0/128303628 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa708005050 0x7fa70800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa724cab640 1 -- 172.21.15.88:0/128303628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa708004380 msgr2=0x7fa708004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:55.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa724cab640 1 --2- 172.21.15.88:0/128303628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa708004380 0x7fa708004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa724cab640 1 -- 172.21.15.88:0/128303628 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa708003530 con 0x7fa708003980 2023-10-16T20:39:55.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa7254ac640 1 --2- 172.21.15.88:0/128303628 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa708005050 0x7fa70800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:55.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa7254ac640 1 -- 172.21.15.88:0/128303628 reap_dead start 2023-10-16T20:39:55.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.768+0000 7fa724cab640 1 --2- 172.21.15.88:0/128303628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa708003980 0x7fa708003d60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009ae0 tx=0x7fa71c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cad75f8ad82f475 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:55.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa7177fe640 1 -- 172.21.15.88:0/128303628 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71c0a5510 con 0x7fa708003980 2023-10-16T20:39:55.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa7177fe640 1 -- 172.21.15.88:0/128303628 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa71c0a5670 con 0x7fa708003980 2023-10-16T20:39:55.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa7177fe640 1 -- 172.21.15.88:0/128303628 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71c0a5a20 con 0x7fa708003980 2023-10-16T20:39:55.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa726f0f640 1 -- 172.21.15.88:0/128303628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa708003980 msgr2=0x7fa708003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa726f0f640 1 --2- 172.21.15.88:0/128303628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa708003980 0x7fa708003d60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009ae0 tx=0x7fa71c01ca50 comp rx=0 tx=0).stop 2023-10-16T20:39:55.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa726f0f640 1 -- 172.21.15.88:0/128303628 shutdown_connections 2023-10-16T20:39:55.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa726f0f640 1 --2- 172.21.15.88:0/128303628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa708003980 0x7fa708003d60 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa726f0f640 1 -- 172.21.15.88:0/128303628 >> 172.21.15.88:0/128303628 conn(0x7fa72006f7d0 msgr2=0x7fa72006fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:55.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa726f0f640 1 -- 172.21.15.88:0/128303628 shutdown_connections 2023-10-16T20:39:55.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa726f0f640 1 -- 172.21.15.88:0/128303628 wait complete. 2023-10-16T20:39:55.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa726f0f640 1 Processor -- start 2023-10-16T20:39:55.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa726f0f640 1 -- start start 2023-10-16T20:39:55.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa726f0f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa708003980 0x7fa7200c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa726f0f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa7200c76a0 0x7fa7201f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa726f0f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa720150620 0x7fa7201e5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa726f0f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72015c850 con 0x7fa708003980 2023-10-16T20:39:55.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa726f0f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7200ca4e0 con 0x7fa720150620 2023-10-16T20:39:55.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa726f0f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa72015c2d0 con 0x7fa7200c76a0 2023-10-16T20:39:55.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa724cab640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa708003980 0x7fa7200c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa724cab640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa708003980 0x7fa7200c9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47844/0 (socket says 172.21.15.88:47844) 2023-10-16T20:39:55.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa724cab640 1 -- 172.21.15.88:0/3531780624 learned_addr learned my addr 172.21.15.88:0/3531780624 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:55.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa724cab640 1 -- 172.21.15.88:0/3531780624 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa7200c76a0 msgr2=0x7fa7201f1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa724cab640 1 --2- 172.21.15.88:0/3531780624 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa7200c76a0 0x7fa7201f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa724cab640 1 -- 172.21.15.88:0/3531780624 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa720150620 msgr2=0x7fa7201e5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa724cab640 1 --2- 172.21.15.88:0/3531780624 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa720150620 0x7fa7201e5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.772+0000 7fa724cab640 1 -- 172.21.15.88:0/3531780624 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72015c570 con 0x7fa708003980 2023-10-16T20:39:55.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.776+0000 7fa724cab640 1 --2- 172.21.15.88:0/3531780624 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa708003980 0x7fa7200c9fa0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa71c01c880 tx=0x7fa71c0197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:55.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.776+0000 7fa7157fa640 1 -- 172.21.15.88:0/3531780624 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71c0a5510 con 0x7fa708003980 2023-10-16T20:39:55.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.776+0000 7fa7157fa640 1 -- 172.21.15.88:0/3531780624 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa71c00f000 con 0x7fa708003980 2023-10-16T20:39:55.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.776+0000 7fa726f0f640 1 -- 172.21.15.88:0/3531780624 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7201f16e0 con 0x7fa708003980 2023-10-16T20:39:55.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.776+0000 7fa7157fa640 1 -- 172.21.15.88:0/3531780624 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71c004260 con 0x7fa708003980 2023-10-16T20:39:55.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.776+0000 7fa726f0f640 1 -- 172.21.15.88:0/3531780624 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7200c7c50 con 0x7fa708003980 2023-10-16T20:39:55.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.776+0000 7fa726f0f640 1 -- 172.21.15.88:0/3531780624 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7201f1540 con 0x7fa708003980 2023-10-16T20:39:55.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.776+0000 7fa7157fa640 1 -- 172.21.15.88:0/3531780624 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa71c0b4070 con 0x7fa708003980 2023-10-16T20:39:55.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.776+0000 7fa7157fa640 1 --2- 172.21.15.88:0/3531780624 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa70c0664a0 0x7fa70c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:55.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.776+0000 7fa7157fa640 1 -- 172.21.15.88:0/3531780624 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa71c11e340 con 0x7fa708003980 2023-10-16T20:39:55.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.776+0000 7fa717fff640 1 --2- 172.21.15.88:0/3531780624 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa70c0664a0 0x7fa70c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:55.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.780+0000 7fa717fff640 1 --2- 172.21.15.88:0/3531780624 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa70c0664a0 0x7fa70c068960 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa710000f70 tx=0x7fa710000ee0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:55.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.780+0000 7fa7157fa640 1 -- 172.21.15.88:0/3531780624 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa7201f1540 con 0x7fa708003980 2023-10-16T20:39:55.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.928+0000 7fa726f0f640 1 -- 172.21.15.88:0/3531780624 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa7200c9c50 con 0x7fa708003980 2023-10-16T20:39:55.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.928+0000 7fa7157fa640 1 -- 172.21.15.88:0/3531780624 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa7200c9c50 con 0x7fa708003980 2023-10-16T20:39:55.935 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2023-10-16T20:39:55.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.932+0000 7fa726f0f640 1 -- 172.21.15.88:0/3531780624 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa70c0664a0 msgr2=0x7fa70c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.932+0000 7fa726f0f640 1 --2- 172.21.15.88:0/3531780624 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa70c0664a0 0x7fa70c068960 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa710000f70 tx=0x7fa710000ee0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.932+0000 7fa726f0f640 1 -- 172.21.15.88:0/3531780624 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa708003980 msgr2=0x7fa7200c9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:55.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.932+0000 7fa726f0f640 1 --2- 172.21.15.88:0/3531780624 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa708003980 0x7fa7200c9fa0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa71c01c880 tx=0x7fa71c0197f0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.932+0000 7fa726f0f640 1 -- 172.21.15.88:0/3531780624 shutdown_connections 2023-10-16T20:39:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.932+0000 7fa726f0f640 1 --2- 172.21.15.88:0/3531780624 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa70c0664a0 0x7fa70c068960 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.932+0000 7fa726f0f640 1 --2- 172.21.15.88:0/3531780624 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa720150620 0x7fa7201e5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.932+0000 7fa726f0f640 1 --2- 172.21.15.88:0/3531780624 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa7200c76a0 0x7fa7201f1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.932+0000 7fa726f0f640 1 --2- 172.21.15.88:0/3531780624 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa708003980 0x7fa7200c9fa0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.932+0000 7fa726f0f640 1 -- 172.21.15.88:0/3531780624 >> 172.21.15.88:0/3531780624 conn(0x7fa72006f7d0 msgr2=0x7fa720149870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:55.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.932+0000 7fa726f0f640 1 -- 172.21.15.88:0/3531780624 shutdown_connections 2023-10-16T20:39:55.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:55.932+0000 7fa726f0f640 1 -- 172.21.15.88:0/3531780624 wait complete. 2023-10-16T20:39:55.956 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T20:39:56.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.092+0000 7fc5987d9640 1 Processor -- start 2023-10-16T20:39:56.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.092+0000 7fc5987d9640 1 -- start start 2023-10-16T20:39:56.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.092+0000 7fc5987d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5900c4260 0x7fc5900c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.092+0000 7fc5987d9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc5900c4640 0x7fc5901573e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.092+0000 7fc5987d9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc590069e90 con 0x7fc5900c5a60 2023-10-16T20:39:56.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.092+0000 7fc5987d9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5900716a0 con 0x7fc5900c4640 2023-10-16T20:39:56.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.092+0000 7fc5987d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc59006aea0 con 0x7fc5900c4260 2023-10-16T20:39:56.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.092+0000 7fc595d74640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc5900c5a60 0x7fc5901510d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58482/0 (socket says 172.21.15.88:58482) 2023-10-16T20:39:56.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.092+0000 7fc595d74640 1 -- 172.21.15.88:0/1411618264 learned_addr learned my addr 172.21.15.88:0/1411618264 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:56.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.092+0000 7fc596d76640 1 --2- 172.21.15.88:0/1411618264 >> v2:172.21.15.191:3300/0 conn(0x7fc5900c4640 0x7fc5901573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.092+0000 7fc596575640 1 --2- 172.21.15.88:0/1411618264 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5900c4260 0x7fc5900c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.092+0000 7fc595573640 1 -- 172.21.15.88:0/1411618264 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904050996 0 0) 0x7fc590069e90 con 0x7fc5900c5a60 2023-10-16T20:39:56.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.092+0000 7fc595573640 1 -- 172.21.15.88:0/1411618264 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc57c003530 con 0x7fc5900c5a60 2023-10-16T20:39:56.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.092+0000 7fc596d76640 1 -- 172.21.15.88:0/1411618264 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5900c4260 msgr2=0x7fc5900c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:56.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.092+0000 7fc596d76640 1 --2- 172.21.15.88:0/1411618264 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5900c4260 0x7fc5900c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.092+0000 7fc596d76640 1 -- 172.21.15.88:0/1411618264 >> v1:172.21.15.88:6789/0 conn(0x7fc5900c5a60 legacy=0x7fc5901510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.092+0000 7fc596d76640 1 -- 172.21.15.88:0/1411618264 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5901581e0 con 0x7fc5900c4640 2023-10-16T20:39:56.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.096+0000 7fc596d76640 1 --2- 172.21.15.88:0/1411618264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5900c4640 0x7fc5901573e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc588009d90 tx=0x7fc58801c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=924159c337d55e94 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:56.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.096+0000 7fc595573640 1 -- 172.21.15.88:0/1411618264 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5880a58e0 con 0x7fc5900c4640 2023-10-16T20:39:56.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.096+0000 7fc595573640 1 -- 172.21.15.88:0/1411618264 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5880a5a40 con 0x7fc5900c4640 2023-10-16T20:39:56.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.096+0000 7fc595573640 1 -- 172.21.15.88:0/1411618264 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5880a5dd0 con 0x7fc5900c4640 2023-10-16T20:39:56.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.096+0000 7fc5987d9640 1 -- 172.21.15.88:0/1411618264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5900c4640 msgr2=0x7fc5901573e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.096+0000 7fc5987d9640 1 --2- 172.21.15.88:0/1411618264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5900c4640 0x7fc5901573e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc588009d90 tx=0x7fc58801c800 comp rx=0 tx=0).stop 2023-10-16T20:39:56.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.096+0000 7fc5987d9640 1 -- 172.21.15.88:0/1411618264 shutdown_connections 2023-10-16T20:39:56.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.096+0000 7fc5987d9640 1 --2- 172.21.15.88:0/1411618264 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5900c4640 0x7fc5901573e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.096+0000 7fc5987d9640 1 --2- 172.21.15.88:0/1411618264 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5900c4260 0x7fc5900c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.096+0000 7fc5987d9640 1 -- 172.21.15.88:0/1411618264 >> 172.21.15.88:0/1411618264 conn(0x7fc5900ca9f0 msgr2=0x7fc5900cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:56.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.096+0000 7fc5987d9640 1 -- 172.21.15.88:0/1411618264 shutdown_connections 2023-10-16T20:39:56.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.096+0000 7fc5987d9640 1 -- 172.21.15.88:0/1411618264 wait complete. 2023-10-16T20:39:56.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc5987d9640 1 Processor -- start 2023-10-16T20:39:56.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc5987d9640 1 -- start start 2023-10-16T20:39:56.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc5987d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5900c4260 0x7fc5901612f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc5987d9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5900c4640 0x7fc590156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc5987d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5900c5a60 0x7fc590144c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc5987d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5901587c0 con 0x7fc5900c4260 2023-10-16T20:39:56.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc5987d9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5900ca3d0 con 0x7fc5900c4640 2023-10-16T20:39:56.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc5987d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc590158ae0 con 0x7fc5900c5a60 2023-10-16T20:39:56.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc596575640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5900c4260 0x7fc5901612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc596575640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5900c4260 0x7fc5901612f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47852/0 (socket says 172.21.15.88:47852) 2023-10-16T20:39:56.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc596575640 1 -- 172.21.15.88:0/828210526 learned_addr learned my addr 172.21.15.88:0/828210526 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:56.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc596d76640 1 --2- 172.21.15.88:0/828210526 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5900c5a60 0x7fc590144c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc595d74640 1 --2- 172.21.15.88:0/828210526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5900c4640 0x7fc590156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc596575640 1 -- 172.21.15.88:0/828210526 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5900c5a60 msgr2=0x7fc590144c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc596575640 1 --2- 172.21.15.88:0/828210526 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5900c5a60 0x7fc590144c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc596575640 1 -- 172.21.15.88:0/828210526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5900c4640 msgr2=0x7fc590156f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc596575640 1 --2- 172.21.15.88:0/828210526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5900c4640 0x7fc590156f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc596575640 1 -- 172.21.15.88:0/828210526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc588009a70 con 0x7fc5900c4260 2023-10-16T20:39:56.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc595d74640 1 --2- 172.21.15.88:0/828210526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5900c4640 0x7fc590156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:56.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc596575640 1 --2- 172.21.15.88:0/828210526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5900c4260 0x7fc5901612f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc58c00e960 tx=0x7fc58c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:56.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc586ffd640 1 -- 172.21.15.88:0/828210526 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c009690 con 0x7fc5900c4260 2023-10-16T20:39:56.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc586ffd640 1 -- 172.21.15.88:0/828210526 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58c00ca00 con 0x7fc5900c4260 2023-10-16T20:39:56.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc586ffd640 1 -- 172.21.15.88:0/828210526 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c00cd70 con 0x7fc5900c4260 2023-10-16T20:39:56.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc5987d9640 1 -- 172.21.15.88:0/828210526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5901f5c20 con 0x7fc5900c4260 2023-10-16T20:39:56.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc5987d9640 1 -- 172.21.15.88:0/828210526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5901f50a0 con 0x7fc5900c4260 2023-10-16T20:39:56.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.100+0000 7fc5987d9640 1 -- 172.21.15.88:0/828210526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc590074700 con 0x7fc5900c4260 2023-10-16T20:39:56.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.104+0000 7fc586ffd640 1 -- 172.21.15.88:0/828210526 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc58c013470 con 0x7fc5900c4260 2023-10-16T20:39:56.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.104+0000 7fc586ffd640 1 --2- 172.21.15.88:0/828210526 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc5740664a0 0x7fc574068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.104+0000 7fc586ffd640 1 -- 172.21.15.88:0/828210526 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc58c084460 con 0x7fc5900c4260 2023-10-16T20:39:56.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.104+0000 7fc595d74640 1 --2- 172.21.15.88:0/828210526 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc5740664a0 0x7fc574068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.104+0000 7fc595d74640 1 --2- 172.21.15.88:0/828210526 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc5740664a0 0x7fc574068960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc5900c9f80 tx=0x7fc580009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:56.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.108+0000 7fc586ffd640 1 -- 172.21.15.88:0/828210526 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc58c051530 con 0x7fc5900c4260 2023-10-16T20:39:56.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.256+0000 7fc5987d9640 1 -- 172.21.15.88:0/828210526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc59014d6c0 con 0x7fc5900c4260 2023-10-16T20:39:56.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.256+0000 7fc586ffd640 1 -- 172.21.15.88:0/828210526 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc58c054b50 con 0x7fc5900c4260 2023-10-16T20:39:56.261 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:39:56.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.256+0000 7fc5987d9640 1 -- 172.21.15.88:0/828210526 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc5740664a0 msgr2=0x7fc574068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.256+0000 7fc5987d9640 1 --2- 172.21.15.88:0/828210526 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc5740664a0 0x7fc574068960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc5900c9f80 tx=0x7fc580009040 comp rx=0 tx=0).stop 2023-10-16T20:39:56.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.256+0000 7fc5987d9640 1 -- 172.21.15.88:0/828210526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5900c4260 msgr2=0x7fc5901612f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.256+0000 7fc5987d9640 1 --2- 172.21.15.88:0/828210526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5900c4260 0x7fc5901612f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc58c00e960 tx=0x7fc58c00ee20 comp rx=0 tx=0).stop 2023-10-16T20:39:56.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.256+0000 7fc5987d9640 1 -- 172.21.15.88:0/828210526 shutdown_connections 2023-10-16T20:39:56.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.256+0000 7fc5987d9640 1 --2- 172.21.15.88:0/828210526 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc5740664a0 0x7fc574068960 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.256+0000 7fc5987d9640 1 --2- 172.21.15.88:0/828210526 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc5900c5a60 0x7fc590144c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.256+0000 7fc5987d9640 1 --2- 172.21.15.88:0/828210526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc5900c4640 0x7fc590156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.256+0000 7fc5987d9640 1 --2- 172.21.15.88:0/828210526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5900c4260 0x7fc5901612f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.256+0000 7fc5987d9640 1 -- 172.21.15.88:0/828210526 >> 172.21.15.88:0/828210526 conn(0x7fc5900ca9f0 msgr2=0x7fc59006f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:56.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.256+0000 7fc5987d9640 1 -- 172.21.15.88:0/828210526 shutdown_connections 2023-10-16T20:39:56.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.256+0000 7fc5987d9640 1 -- 172.21.15.88:0/828210526 wait complete. 2023-10-16T20:39:56.281 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:39:56.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.416+0000 7f1aeded6640 1 Processor -- start 2023-10-16T20:39:56.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.416+0000 7f1aeded6640 1 -- start start 2023-10-16T20:39:56.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.416+0000 7f1aeded6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ae806a970 0x7f1ae8071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.416+0000 7f1aeded6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae815b3c0 con 0x7f1ae81406d0 2023-10-16T20:39:56.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.416+0000 7f1aeded6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae815b680 con 0x7f1ae8071e60 2023-10-16T20:39:56.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.416+0000 7f1aeded6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80c2460 con 0x7f1ae806a970 2023-10-16T20:39:56.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.416+0000 7f1ae77fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1ae81406d0 0x7f1ae80c1d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58494/0 (socket says 172.21.15.88:58494) 2023-10-16T20:39:56.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.416+0000 7f1ae77fe640 1 -- 172.21.15.88:0/4125292761 learned_addr learned my addr 172.21.15.88:0/4125292761 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:39:56.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.416+0000 7f1adffff640 1 --2- 172.21.15.88:0/4125292761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ae806a970 0x7f1ae8071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.416+0000 7f1ae6ffd640 1 -- 172.21.15.88:0/4125292761 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596789360 0 0) 0x7f1ae815b3c0 con 0x7f1ae81406d0 2023-10-16T20:39:56.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae6ffd640 1 -- 172.21.15.88:0/4125292761 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad4003530 con 0x7f1ae81406d0 2023-10-16T20:39:56.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae6ffd640 1 -- 172.21.15.88:0/4125292761 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2887718766 0 0) 0x7f1ad4003530 con 0x7f1ae81406d0 2023-10-16T20:39:56.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae6ffd640 1 -- 172.21.15.88:0/4125292761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ae806a970 msgr2=0x7f1ae8071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:56.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae6ffd640 1 --2- 172.21.15.88:0/4125292761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ae806a970 0x7f1ae8071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae6ffd640 1 -- 172.21.15.88:0/4125292761 >> v1:172.21.15.191:6789/0 conn(0x7f1ae8071e60 legacy=0x7f1ae815a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae6ffd640 1 -- 172.21.15.88:0/4125292761 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae8157570 con 0x7f1ae81406d0 2023-10-16T20:39:56.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae6ffd640 1 -- 172.21.15.88:0/4125292761 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f1ad8001680 con 0x7f1ae81406d0 2023-10-16T20:39:56.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae6ffd640 1 -- 172.21.15.88:0/4125292761 >> v1:172.21.15.88:6789/0 conn(0x7f1ae81406d0 legacy=0x7f1ae80c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae6ffd640 1 --2- 172.21.15.88:0/4125292761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ae806a970 0x7f1ad40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae6ffd640 1 --2- 172.21.15.88:0/4125292761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ad4003f30 0x7f1ad40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae6ffd640 1 --2- 172.21.15.88:0/4125292761 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ad4004d10 0x7f1ad4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae6ffd640 1 -- 172.21.15.88:0/4125292761 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80c2460 con 0x7f1ad4004d10 2023-10-16T20:39:56.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae6ffd640 1 -- 172.21.15.88:0/4125292761 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad400ba70 con 0x7f1ad4003f30 2023-10-16T20:39:56.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae6ffd640 1 -- 172.21.15.88:0/4125292761 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad400bc30 con 0x7f1ae806a970 2023-10-16T20:39:56.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae7fff640 1 --2- 172.21.15.88:0/4125292761 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ad4004d10 0x7f1ad4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae77fe640 1 --2- 172.21.15.88:0/4125292761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ae806a970 0x7f1ad40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1adffff640 1 --2- 172.21.15.88:0/4125292761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ad4003f30 0x7f1ad40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae7fff640 1 -- 172.21.15.88:0/4125292761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ae806a970 msgr2=0x7f1ad40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:56.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae7fff640 1 --2- 172.21.15.88:0/4125292761 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ae806a970 0x7f1ad40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae7fff640 1 -- 172.21.15.88:0/4125292761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ad4003f30 msgr2=0x7f1ad40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:56.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae7fff640 1 --2- 172.21.15.88:0/4125292761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ad4003f30 0x7f1ad40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.420+0000 7f1ae7fff640 1 -- 172.21.15.88:0/4125292761 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad4003530 con 0x7f1ad4004d10 2023-10-16T20:39:56.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1ae7fff640 1 --2- 172.21.15.88:0/4125292761 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ad4004d10 0x7f1ad4005190 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0001e70 tx=0x7f1ae001c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=914cdd67d58698a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:56.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1ae6ffd640 1 -- 172.21.15.88:0/4125292761 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0003b80 con 0x7f1ad4004d10 2023-10-16T20:39:56.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1ae6ffd640 1 -- 172.21.15.88:0/4125292761 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae0003ce0 con 0x7f1ad4004d10 2023-10-16T20:39:56.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1ae6ffd640 1 -- 172.21.15.88:0/4125292761 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae00052c0 con 0x7f1ad4004d10 2023-10-16T20:39:56.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1aeded6640 1 -- 172.21.15.88:0/4125292761 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ad4004d10 msgr2=0x7f1ad4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1aeded6640 1 --2- 172.21.15.88:0/4125292761 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ad4004d10 0x7f1ad4005190 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0001e70 tx=0x7f1ae001c760 comp rx=0 tx=0).stop 2023-10-16T20:39:56.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1ae7fff640 1 -- 172.21.15.88:0/4125292761 reap_dead start 2023-10-16T20:39:56.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1aeded6640 1 -- 172.21.15.88:0/4125292761 shutdown_connections 2023-10-16T20:39:56.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1aeded6640 1 -- 172.21.15.88:0/4125292761 >> 172.21.15.88:0/4125292761 conn(0x7f1ae80cad50 msgr2=0x7f1ae80cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:56.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1aeded6640 1 -- 172.21.15.88:0/4125292761 shutdown_connections 2023-10-16T20:39:56.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1aeded6640 1 -- 172.21.15.88:0/4125292761 wait complete. 2023-10-16T20:39:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1aeded6640 1 Processor -- start 2023-10-16T20:39:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1aeded6640 1 -- start start 2023-10-16T20:39:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1aeded6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ad4004d10 0x7f1ae80c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1aeded6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ae813ee60 0x7f1ae81598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1aeded6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae8151b10 0x7f1ae81efcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1aeded6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80cb690 con 0x7f1ae8151b10 2023-10-16T20:39:56.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1aeded6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae8139e20 con 0x7f1ad4004d10 2023-10-16T20:39:56.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1aeded6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80c7dc0 con 0x7f1ae813ee60 2023-10-16T20:39:56.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1ae7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae8151b10 0x7f1ae81efcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1ae7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae8151b10 0x7f1ae81efcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47866/0 (socket says 172.21.15.88:47866) 2023-10-16T20:39:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.424+0000 7f1ae7fff640 1 -- 172.21.15.88:0/3248250820 learned_addr learned my addr 172.21.15.88:0/3248250820 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1ae77fe640 1 --2- 172.21.15.88:0/3248250820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ad4004d10 0x7f1ae80c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1adffff640 1 --2- 172.21.15.88:0/3248250820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ae813ee60 0x7f1ae81598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1ae7fff640 1 -- 172.21.15.88:0/3248250820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ae813ee60 msgr2=0x7f1ae81598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1ae7fff640 1 --2- 172.21.15.88:0/3248250820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ae813ee60 0x7f1ae81598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1ae7fff640 1 -- 172.21.15.88:0/3248250820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ad4004d10 msgr2=0x7f1ae80c5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1ae7fff640 1 --2- 172.21.15.88:0/3248250820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ad4004d10 0x7f1ae80c5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1ae7fff640 1 -- 172.21.15.88:0/3248250820 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae80cb160 con 0x7f1ae8151b10 2023-10-16T20:39:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1ae77fe640 1 --2- 172.21.15.88:0/3248250820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ad4004d10 0x7f1ae80c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1adffff640 1 --2- 172.21.15.88:0/3248250820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ae813ee60 0x7f1ae81598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:39:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1ae7fff640 1 --2- 172.21.15.88:0/3248250820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae8151b10 0x7f1ae81efcf0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1ae001ccd0 tx=0x7f1ad4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1ae4ff9640 1 -- 172.21.15.88:0/3248250820 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0006d60 con 0x7f1ae8151b10 2023-10-16T20:39:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1ae4ff9640 1 -- 172.21.15.88:0/3248250820 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae0005830 con 0x7f1ae8151b10 2023-10-16T20:39:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1ae4ff9640 1 -- 172.21.15.88:0/3248250820 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0007230 con 0x7f1ae8151b10 2023-10-16T20:39:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1aeded6640 1 -- 172.21.15.88:0/3248250820 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae80c6940 con 0x7f1ae8151b10 2023-10-16T20:39:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1aeded6640 1 -- 172.21.15.88:0/3248250820 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae80c7ff0 con 0x7f1ae8151b10 2023-10-16T20:39:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1aeded6640 1 -- 172.21.15.88:0/3248250820 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae80c4fa0 con 0x7f1ae8151b10 2023-10-16T20:39:56.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1ae4ff9640 1 -- 172.21.15.88:0/3248250820 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1ae0007390 con 0x7f1ae8151b10 2023-10-16T20:39:56.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1ae4ff9640 1 --2- 172.21.15.88:0/3248250820 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1ac80664a0 0x7f1ac8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1ae4ff9640 1 -- 172.21.15.88:0/3248250820 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f1ae011e040 con 0x7f1ae8151b10 2023-10-16T20:39:56.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.428+0000 7f1ae77fe640 1 --2- 172.21.15.88:0/3248250820 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1ac80664a0 0x7f1ac8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.432+0000 7f1ae77fe640 1 --2- 172.21.15.88:0/3248250820 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1ac80664a0 0x7f1ac8068960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8007d90 tx=0x7f1ad8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:56.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.432+0000 7f1ae4ff9640 1 -- 172.21.15.88:0/3248250820 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1ae80c4fa0 con 0x7f1ae8151b10 2023-10-16T20:39:56.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.616+0000 7f1aeded6640 1 -- 172.21.15.88:0/3248250820 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ae80c4fa0 con 0x7f1ae8151b10 2023-10-16T20:39:56.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.616+0000 7f1ae4ff9640 1 -- 172.21.15.88:0/3248250820 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6559 (secure 0 0 0) 0x7f1ae80c4fa0 con 0x7f1ae8151b10 2023-10-16T20:39:56.620 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:56.621 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:39:39.882115+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":44,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:39:56.621 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 19 2023-10-16T20:39:56.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.616+0000 7f1aeded6640 1 -- 172.21.15.88:0/3248250820 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1ac80664a0 msgr2=0x7f1ac8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.616+0000 7f1aeded6640 1 --2- 172.21.15.88:0/3248250820 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1ac80664a0 0x7f1ac8068960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8007d90 tx=0x7f1ad8009040 comp rx=0 tx=0).stop 2023-10-16T20:39:56.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.616+0000 7f1aeded6640 1 -- 172.21.15.88:0/3248250820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae8151b10 msgr2=0x7f1ae81efcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.616+0000 7f1aeded6640 1 --2- 172.21.15.88:0/3248250820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae8151b10 0x7f1ae81efcf0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1ae001ccd0 tx=0x7f1ad4005800 comp rx=0 tx=0).stop 2023-10-16T20:39:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.616+0000 7f1aeded6640 1 -- 172.21.15.88:0/3248250820 shutdown_connections 2023-10-16T20:39:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.616+0000 7f1aeded6640 1 --2- 172.21.15.88:0/3248250820 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1ac80664a0 0x7f1ac8068960 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.616+0000 7f1aeded6640 1 --2- 172.21.15.88:0/3248250820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae8151b10 0x7f1ae81efcf0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.616+0000 7f1aeded6640 1 --2- 172.21.15.88:0/3248250820 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1ae813ee60 0x7f1ae81598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.616+0000 7f1aeded6640 1 --2- 172.21.15.88:0/3248250820 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ad4004d10 0x7f1ae80c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.616+0000 7f1aeded6640 1 -- 172.21.15.88:0/3248250820 >> 172.21.15.88:0/3248250820 conn(0x7f1ae80cad50 msgr2=0x7f1ae813dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:56.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.616+0000 7f1aeded6640 1 -- 172.21.15.88:0/3248250820 shutdown_connections 2023-10-16T20:39:56.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.616+0000 7f1aeded6640 1 -- 172.21.15.88:0/3248250820 wait complete. 2023-10-16T20:39:56.641 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:39:56.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82fafe4640 1 Processor -- start 2023-10-16T20:39:56.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82fafe4640 1 -- start start 2023-10-16T20:39:56.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82fafe4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82f41537c0 0x7f82f415c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82fafe4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f82f415cdb0 0x7f82f40c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82fafe4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f82f40ca2f0 0x7f82f40c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82fafe4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f40cae10 con 0x7f82f40ca2f0 2023-10-16T20:39:56.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82fafe4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f40cac30 con 0x7f82f415cdb0 2023-10-16T20:39:56.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82fafe4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f415d2b0 con 0x7f82f41537c0 2023-10-16T20:39:56.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82fa7e3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f82f40ca2f0 0x7f82f40c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82fa7e3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f82f40ca2f0 0x7f82f40c51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47878/0 (socket says 172.21.15.88:47878) 2023-10-16T20:39:56.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82fa7e3640 1 -- 172.21.15.88:0/937291186 learned_addr learned my addr 172.21.15.88:0/937291186 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82f97e1640 1 --2- 172.21.15.88:0/937291186 >> v2:172.21.15.191:3300/0 conn(0x7f82f415cdb0 0x7f82f40c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82fa7e3640 1 -- 172.21.15.88:0/937291186 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82f41537c0 msgr2=0x7f82f415c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:56.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82f9fe2640 1 --2- 172.21.15.88:0/937291186 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82f41537c0 0x7f82f415c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82fa7e3640 1 --2- 172.21.15.88:0/937291186 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82f41537c0 0x7f82f415c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82fa7e3640 1 -- 172.21.15.88:0/937291186 >> v2:172.21.15.191:3300/0 conn(0x7f82f415cdb0 msgr2=0x7f82f40c4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:39:56.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82fa7e3640 1 --2- 172.21.15.88:0/937291186 >> v2:172.21.15.191:3300/0 conn(0x7f82f415cdb0 0x7f82f40c4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82fa7e3640 1 -- 172.21.15.88:0/937291186 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f415d590 con 0x7f82f40ca2f0 2023-10-16T20:39:56.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82f97e1640 1 --2- 172.21.15.88:0/937291186 >> v2:172.21.15.191:3300/0 conn(0x7f82f415cdb0 0x7f82f40c4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:56.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.776+0000 7f82f9fe2640 1 --2- 172.21.15.88:0/937291186 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82f41537c0 0x7f82f415c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:56.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.780+0000 7f82fa7e3640 1 --2- 172.21.15.88:0/937291186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f40ca2f0 0x7f82f40c51c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f82f40cc650 tx=0x7f82ec01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e630622fcc3e6e7b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:56.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.780+0000 7f82f8fe0640 1 -- 172.21.15.88:0/937291186 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec0af400 con 0x7f82f40ca2f0 2023-10-16T20:39:56.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.780+0000 7f82f8fe0640 1 -- 172.21.15.88:0/937291186 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec0ab050 con 0x7f82f40ca2f0 2023-10-16T20:39:56.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.780+0000 7f82f8fe0640 1 -- 172.21.15.88:0/937291186 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec0af780 con 0x7f82f40ca2f0 2023-10-16T20:39:56.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.780+0000 7f82fafe4640 1 -- 172.21.15.88:0/937291186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f40ca2f0 msgr2=0x7f82f40c51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.780+0000 7f82fafe4640 1 --2- 172.21.15.88:0/937291186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f40ca2f0 0x7f82f40c51c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f82f40cc650 tx=0x7f82ec01f1c0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.780+0000 7f82fafe4640 1 -- 172.21.15.88:0/937291186 shutdown_connections 2023-10-16T20:39:56.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.780+0000 7f82fafe4640 1 --2- 172.21.15.88:0/937291186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f40ca2f0 0x7f82f40c51c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.780+0000 7f82fafe4640 1 --2- 172.21.15.88:0/937291186 >> v2:172.21.15.191:3300/0 conn(0x7f82f415cdb0 0x7f82f40c4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.780+0000 7f82fafe4640 1 --2- 172.21.15.88:0/937291186 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82f41537c0 0x7f82f415c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.780+0000 7f82fafe4640 1 -- 172.21.15.88:0/937291186 >> 172.21.15.88:0/937291186 conn(0x7f82f406c170 msgr2=0x7f82f40715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:56.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.780+0000 7f82fafe4640 1 -- 172.21.15.88:0/937291186 shutdown_connections 2023-10-16T20:39:56.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.780+0000 7f82fafe4640 1 -- 172.21.15.88:0/937291186 wait complete. 2023-10-16T20:39:56.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.780+0000 7f82fafe4640 1 Processor -- start 2023-10-16T20:39:56.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fafe4640 1 -- start start 2023-10-16T20:39:56.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fafe4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82f40ca2f0 0x7f82f40ce810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fafe4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82f41537c0 0x7f82f4154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fafe4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f415cdb0 0x7f82f4159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fafe4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4153d30 con 0x7f82f415cdb0 2023-10-16T20:39:56.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fafe4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4156c50 con 0x7f82f41537c0 2023-10-16T20:39:56.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fafe4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4153f10 con 0x7f82f40ca2f0 2023-10-16T20:39:56.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fa7e3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f415cdb0 0x7f82f4159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fa7e3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f415cdb0 0x7f82f4159290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47882/0 (socket says 172.21.15.88:47882) 2023-10-16T20:39:56.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fa7e3640 1 -- 172.21.15.88:0/641791739 learned_addr learned my addr 172.21.15.88:0/641791739 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:39:56.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82f9fe2640 1 --2- 172.21.15.88:0/641791739 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82f40ca2f0 0x7f82f40ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82f97e1640 1 --2- 172.21.15.88:0/641791739 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82f41537c0 0x7f82f4154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fa7e3640 1 -- 172.21.15.88:0/641791739 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82f40ca2f0 msgr2=0x7f82f40ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fa7e3640 1 --2- 172.21.15.88:0/641791739 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82f40ca2f0 0x7f82f40ce810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fa7e3640 1 -- 172.21.15.88:0/641791739 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82f41537c0 msgr2=0x7f82f4154f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fa7e3640 1 --2- 172.21.15.88:0/641791739 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82f41537c0 0x7f82f4154f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fa7e3640 1 -- 172.21.15.88:0/641791739 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec009b30 con 0x7f82f415cdb0 2023-10-16T20:39:56.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82f9fe2640 1 --2- 172.21.15.88:0/641791739 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82f40ca2f0 0x7f82f40ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82f97e1640 1 --2- 172.21.15.88:0/641791739 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82f41537c0 0x7f82f4154f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:39:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fa7e3640 1 --2- 172.21.15.88:0/641791739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f415cdb0 0x7f82f4159290 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0b2040 tx=0x7f82ec01bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82ea7fc640 1 -- 172.21.15.88:0/641791739 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec0afdc0 con 0x7f82f415cdb0 2023-10-16T20:39:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82ea7fc640 1 -- 172.21.15.88:0/641791739 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec009510 con 0x7f82f415cdb0 2023-10-16T20:39:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82ea7fc640 1 -- 172.21.15.88:0/641791739 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec0ba660 con 0x7f82f415cdb0 2023-10-16T20:39:56.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fafe4640 1 -- 172.21.15.88:0/641791739 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f4157b20 con 0x7f82f415cdb0 2023-10-16T20:39:56.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fafe4640 1 -- 172.21.15.88:0/641791739 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f41580e0 con 0x7f82f415cdb0 2023-10-16T20:39:56.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.784+0000 7f82fafe4640 1 -- 172.21.15.88:0/641791739 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f40c4740 con 0x7f82f415cdb0 2023-10-16T20:39:56.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.788+0000 7f82ea7fc640 1 -- 172.21.15.88:0/641791739 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f82ec0b6040 con 0x7f82f415cdb0 2023-10-16T20:39:56.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.788+0000 7f82ea7fc640 1 --2- 172.21.15.88:0/641791739 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f82d80664a0 0x7f82d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:39:56.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.788+0000 7f82ea7fc640 1 -- 172.21.15.88:0/641791739 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f82ec122570 con 0x7f82f415cdb0 2023-10-16T20:39:56.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.788+0000 7f82f9fe2640 1 --2- 172.21.15.88:0/641791739 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f82d80664a0 0x7f82d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:39:56.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.788+0000 7f82f9fe2640 1 --2- 172.21.15.88:0/641791739 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f82d80664a0 0x7f82d8068960 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f82f0008930 tx=0x7f82f0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:39:56.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.792+0000 7f82ea7fc640 1 -- 172.21.15.88:0/641791739 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f82f40c4740 con 0x7f82f415cdb0 2023-10-16T20:39:56.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.944+0000 7f82fafe4640 1 -- 172.21.15.88:0/641791739 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82f40c4740 con 0x7f82f415cdb0 2023-10-16T20:39:56.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.944+0000 7f82ea7fc640 1 -- 172.21.15.88:0/641791739 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18710 (secure 0 0 0) 0x7f82f40c4740 con 0x7f82f415cdb0 2023-10-16T20:39:56.948 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:39:56.949 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:37:37.392549+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:39:56.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.944+0000 7f82fafe4640 1 -- 172.21.15.88:0/641791739 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f82d80664a0 msgr2=0x7f82d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.944+0000 7f82fafe4640 1 --2- 172.21.15.88:0/641791739 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f82d80664a0 0x7f82d8068960 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f82f0008930 tx=0x7f82f0008040 comp rx=0 tx=0).stop 2023-10-16T20:39:56.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.944+0000 7f82fafe4640 1 -- 172.21.15.88:0/641791739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f415cdb0 msgr2=0x7f82f4159290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:39:56.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.944+0000 7f82fafe4640 1 --2- 172.21.15.88:0/641791739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f415cdb0 0x7f82f4159290 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0b2040 tx=0x7f82ec01bf60 comp rx=0 tx=0).stop 2023-10-16T20:39:56.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.944+0000 7f82fafe4640 1 -- 172.21.15.88:0/641791739 shutdown_connections 2023-10-16T20:39:56.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.944+0000 7f82fafe4640 1 --2- 172.21.15.88:0/641791739 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f82d80664a0 0x7f82d8068960 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.944+0000 7f82fafe4640 1 --2- 172.21.15.88:0/641791739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f415cdb0 0x7f82f4159290 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.944+0000 7f82fafe4640 1 --2- 172.21.15.88:0/641791739 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82f41537c0 0x7f82f4154f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.944+0000 7f82fafe4640 1 --2- 172.21.15.88:0/641791739 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f82f40ca2f0 0x7f82f40ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:39:56.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.944+0000 7f82fafe4640 1 -- 172.21.15.88:0/641791739 >> 172.21.15.88:0/641791739 conn(0x7f82f406c170 msgr2=0x7f82f40c4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:39:56.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.944+0000 7f82fafe4640 1 -- 172.21.15.88:0/641791739 shutdown_connections 2023-10-16T20:39:56.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:39:56.944+0000 7f82fafe4640 1 -- 172.21.15.88:0/641791739 wait complete. 2023-10-16T20:39:56.969 DEBUG:tasks.mds_thrash:Yielding 2023-10-16T20:39:56.970 INFO:teuthology.run_tasks:Running task workunit... 2023-10-16T20:39:56.983 INFO:tasks.workunit:Pulling workunits from ref 19fd96fe3581520b6d9f50ffdabf29051aee5d4d 2023-10-16T20:39:56.983 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-16T20:39:56.983 INFO:tasks.workunit:timeout=6h 2023-10-16T20:39:56.984 INFO:tasks.workunit:cleanup=True 2023-10-16T20:39:56.984 DEBUG:teuthology.orchestra.run.smithi088:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-16T20:39:56.985 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2023-10-16T20:39:56.985 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 116 secs before thrashing 2023-10-16T20:39:56.994 INFO:teuthology.orchestra.run.smithi088.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-10-16T20:39:56.995 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2023-10-16T20:39:56.995 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 2bh/43d Inode: 1 Links: 2 2023-10-16T20:39:56.995 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T20:39:56.995 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-10-16T20:39:56.995 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-10-16 20:37:00.747821839 +0000 2023-10-16T20:39:56.996 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-10-16 20:39:10.034794399 +0000 2023-10-16T20:39:56.996 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-10-16T20:39:56.996 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-10-16T20:39:56.996 DEBUG:teuthology.orchestra.run.smithi088:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-10-16T20:39:57.069 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/rishabh-d-dave/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 19fd96fe3581520b6d9f50ffdabf29051aee5d4d 2023-10-16T20:39:57.121 INFO:tasks.workunit.client.0.smithi088.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-16T20:40:24.106 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:40:24.112 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:40:38.900 INFO:tasks.workunit.client.0.smithi088.stderr:Note: switching to '19fd96fe3581520b6d9f50ffdabf29051aee5d4d'. 2023-10-16T20:40:38.900 INFO:tasks.workunit.client.0.smithi088.stderr: 2023-10-16T20:40:38.901 INFO:tasks.workunit.client.0.smithi088.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-16T20:40:38.901 INFO:tasks.workunit.client.0.smithi088.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-16T20:40:38.901 INFO:tasks.workunit.client.0.smithi088.stderr:state without impacting any branches by switching back to a branch. 2023-10-16T20:40:38.901 INFO:tasks.workunit.client.0.smithi088.stderr: 2023-10-16T20:40:38.902 INFO:tasks.workunit.client.0.smithi088.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-16T20:40:38.902 INFO:tasks.workunit.client.0.smithi088.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-16T20:40:38.902 INFO:tasks.workunit.client.0.smithi088.stderr: 2023-10-16T20:40:38.902 INFO:tasks.workunit.client.0.smithi088.stderr: git switch -c 2023-10-16T20:40:38.903 INFO:tasks.workunit.client.0.smithi088.stderr: 2023-10-16T20:40:38.903 INFO:tasks.workunit.client.0.smithi088.stderr:Or undo this operation with: 2023-10-16T20:40:38.903 INFO:tasks.workunit.client.0.smithi088.stderr: 2023-10-16T20:40:38.903 INFO:tasks.workunit.client.0.smithi088.stderr: git switch - 2023-10-16T20:40:38.903 INFO:tasks.workunit.client.0.smithi088.stderr: 2023-10-16T20:40:38.903 INFO:tasks.workunit.client.0.smithi088.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-16T20:40:38.904 INFO:tasks.workunit.client.0.smithi088.stderr: 2023-10-16T20:40:38.904 INFO:tasks.workunit.client.0.smithi088.stderr:HEAD is now at 19fd96fe358 Merge remote-tracking branch 'rishabh/unique-cephfs-name' into rishabh-2023oct13-qa2 2023-10-16T20:40:38.914 DEBUG:teuthology.orchestra.run.smithi088:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-16T20:40:38.923 INFO:tasks.workunit.client.0.smithi088.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-16T20:40:38.926 INFO:tasks.workunit.client.0.smithi088.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-16T20:40:38.927 INFO:tasks.workunit.client.0.smithi088.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-16T20:40:39.675 INFO:tasks.workunit.client.0.smithi088.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-16T20:40:39.758 INFO:tasks.workunit.client.0.smithi088.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-16T20:40:39.814 INFO:tasks.workunit.client.0.smithi088.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-16T20:40:39.816 INFO:tasks.workunit.client.0.smithi088.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-16T20:40:39.816 INFO:tasks.workunit.client.0.smithi088.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-16T20:40:39.869 INFO:tasks.workunit.client.0.smithi088.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-16T20:40:39.875 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:40:39.875 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-16T20:40:39.931 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2023-10-16T20:40:39.932 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2023-10-16T20:40:39.933 DEBUG:teuthology.orchestra.run.smithi088:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=19fd96fe3581520b6d9f50ffdabf29051aee5d4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2023-10-16T20:40:39.998 INFO:tasks.workunit.client.0.smithi088.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2023-10-16T20:40:43.809 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir foo 2023-10-16T20:40:43.822 INFO:tasks.workunit.client.0.smithi088.stderr:+ cp linux-2.6.33.tar.bz2 foo 2023-10-16T20:40:44.419 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir foo/.snap/barsnap 2023-10-16T20:40:44.439 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm foo/linux-2.6.33.tar.bz2 2023-10-16T20:40:44.462 INFO:tasks.workunit.client.0.smithi088.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2023-10-16T20:40:44.663 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo passed: files are identical 2023-10-16T20:40:44.664 INFO:tasks.workunit.client.0.smithi088.stdout:passed: files are identical 2023-10-16T20:40:44.664 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir foo/.snap/barsnap 2023-10-16T20:40:45.194 INFO:tasks.workunit.client.0.smithi088.stdout:OK 2023-10-16T20:40:45.195 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo OK 2023-10-16T20:40:45.196 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-16T20:40:45.196 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-16T20:40:45.290 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2023-10-16T20:40:45.291 DEBUG:teuthology.orchestra.run.smithi088:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=19fd96fe3581520b6d9f50ffdabf29051aee5d4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2023-10-16T20:40:45.317 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 1 2023-10-16T20:40:45.319 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 2 2023-10-16T20:40:45.322 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 3 2023-10-16T20:40:45.324 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -e file4 ']' 2023-10-16T20:40:45.324 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir .snap/snap1 2023-10-16T20:40:45.338 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 4 2023-10-16T20:40:45.342 INFO:tasks.workunit.client.0.smithi088.stderr:++ ls 2023-10-16T20:40:45.348 INFO:tasks.workunit.client.0.smithi088.stderr:+ now='file1 2023-10-16T20:40:45.349 INFO:tasks.workunit.client.0.smithi088.stderr:file2 2023-10-16T20:40:45.349 INFO:tasks.workunit.client.0.smithi088.stderr:file3 2023-10-16T20:40:45.349 INFO:tasks.workunit.client.0.smithi088.stderr:file4' 2023-10-16T20:40:45.349 INFO:tasks.workunit.client.0.smithi088.stderr:++ ls .snap/snap1 2023-10-16T20:40:45.357 INFO:tasks.workunit.client.0.smithi088.stderr:+ then='file1 2023-10-16T20:40:45.358 INFO:tasks.workunit.client.0.smithi088.stderr:file2 2023-10-16T20:40:45.358 INFO:tasks.workunit.client.0.smithi088.stderr:file3' 2023-10-16T20:40:45.358 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir .snap/snap1 2023-10-16T20:40:45.839 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 'file1 2023-10-16T20:40:45.839 INFO:tasks.workunit.client.0.smithi088.stderr:file2 2023-10-16T20:40:45.839 INFO:tasks.workunit.client.0.smithi088.stderr:file3 2023-10-16T20:40:45.840 INFO:tasks.workunit.client.0.smithi088.stderr:file4' = 'file1 2023-10-16T20:40:45.840 INFO:tasks.workunit.client.0.smithi088.stderr:file2 2023-10-16T20:40:45.840 INFO:tasks.workunit.client.0.smithi088.stderr:file3' ']' 2023-10-16T20:40:45.840 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 1 2023-10-16T20:40:45.845 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 2 2023-10-16T20:40:45.855 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 3 2023-10-16T20:40:45.870 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir .snap/snap1 2023-10-16T20:40:45.884 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 4 2023-10-16T20:40:45.893 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir .snap/snap1 2023-10-16T20:40:45.909 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm file1 file2 file3 file4 2023-10-16T20:40:45.919 INFO:tasks.workunit.client.0.smithi088.stdout:OK 2023-10-16T20:40:45.920 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo OK 2023-10-16T20:40:45.920 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-16T20:40:45.920 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-16T20:40:45.996 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2023-10-16T20:40:45.996 DEBUG:teuthology.orchestra.run.smithi088:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=19fd96fe3581520b6d9f50ffdabf29051aee5d4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2023-10-16T20:40:46.067 INFO:tasks.workunit.client.0.smithi088.stdout:Create dir 100 to 199 ... 2023-10-16T20:40:46.069 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 100 2023-10-16T20:40:46.106 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 101 2023-10-16T20:40:46.153 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 102 2023-10-16T20:40:46.185 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 103 2023-10-16T20:40:46.218 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 104 2023-10-16T20:40:46.251 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 105 2023-10-16T20:40:46.282 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 106 2023-10-16T20:40:46.313 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 107 2023-10-16T20:40:46.347 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 108 2023-10-16T20:40:46.379 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 109 2023-10-16T20:40:46.684 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 110 2023-10-16T20:40:46.715 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 111 2023-10-16T20:40:46.748 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 112 2023-10-16T20:40:46.781 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 113 2023-10-16T20:40:46.813 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 114 2023-10-16T20:40:46.846 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 115 2023-10-16T20:40:46.876 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 116 2023-10-16T20:40:46.909 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 117 2023-10-16T20:40:46.943 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 118 2023-10-16T20:40:46.974 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 119 2023-10-16T20:40:47.007 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 120 2023-10-16T20:40:47.039 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 121 2023-10-16T20:40:47.071 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 122 2023-10-16T20:40:47.103 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 123 2023-10-16T20:40:47.137 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 124 2023-10-16T20:40:47.169 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 125 2023-10-16T20:40:47.202 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 126 2023-10-16T20:40:47.235 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 127 2023-10-16T20:40:47.267 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 128 2023-10-16T20:40:47.300 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 129 2023-10-16T20:40:47.337 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 130 2023-10-16T20:40:47.375 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 131 2023-10-16T20:40:47.443 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 132 2023-10-16T20:40:47.536 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 133 2023-10-16T20:40:47.569 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 134 2023-10-16T20:40:47.603 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 135 2023-10-16T20:40:47.635 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 136 2023-10-16T20:40:47.667 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 137 2023-10-16T20:40:47.698 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 138 2023-10-16T20:40:48.336 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 139 2023-10-16T20:40:48.370 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 140 2023-10-16T20:40:48.403 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 141 2023-10-16T20:40:48.436 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 142 2023-10-16T20:40:48.468 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 143 2023-10-16T20:40:48.501 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 144 2023-10-16T20:40:48.533 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 145 2023-10-16T20:40:48.567 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 146 2023-10-16T20:40:48.599 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 147 2023-10-16T20:40:48.631 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 148 2023-10-16T20:40:48.663 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 149 2023-10-16T20:40:48.696 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 150 2023-10-16T20:40:48.727 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 151 2023-10-16T20:40:48.759 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 152 2023-10-16T20:40:48.790 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 153 2023-10-16T20:40:48.823 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 154 2023-10-16T20:40:48.855 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 155 2023-10-16T20:40:48.887 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 156 2023-10-16T20:40:48.920 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 157 2023-10-16T20:40:48.952 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 158 2023-10-16T20:40:49.104 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 159 2023-10-16T20:40:49.135 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 160 2023-10-16T20:40:49.169 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 161 2023-10-16T20:40:49.202 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 162 2023-10-16T20:40:49.234 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 163 2023-10-16T20:40:49.268 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 164 2023-10-16T20:40:49.300 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 165 2023-10-16T20:40:49.335 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 166 2023-10-16T20:40:49.367 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 167 2023-10-16T20:40:50.281 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 168 2023-10-16T20:40:50.314 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 169 2023-10-16T20:40:50.349 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 170 2023-10-16T20:40:50.384 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 171 2023-10-16T20:40:50.418 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 172 2023-10-16T20:40:50.452 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 173 2023-10-16T20:40:50.486 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 174 2023-10-16T20:40:50.520 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 175 2023-10-16T20:40:50.553 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 176 2023-10-16T20:40:50.586 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 177 2023-10-16T20:40:50.619 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 178 2023-10-16T20:40:50.651 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 179 2023-10-16T20:40:50.686 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 180 2023-10-16T20:40:50.717 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 181 2023-10-16T20:40:50.749 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 182 2023-10-16T20:40:50.906 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 183 2023-10-16T20:40:50.938 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 184 2023-10-16T20:40:50.970 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 185 2023-10-16T20:40:51.002 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 186 2023-10-16T20:40:51.035 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 187 2023-10-16T20:40:51.067 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 188 2023-10-16T20:40:51.099 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 189 2023-10-16T20:40:51.132 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 190 2023-10-16T20:40:51.169 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 191 2023-10-16T20:40:51.202 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 192 2023-10-16T20:40:51.232 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 193 2023-10-16T20:40:51.266 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 194 2023-10-16T20:40:51.298 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 195 2023-10-16T20:40:51.613 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 196 2023-10-16T20:40:51.644 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 197 2023-10-16T20:40:51.677 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 198 2023-10-16T20:40:51.709 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 199 2023-10-16T20:40:51.741 INFO:tasks.workunit.client.0.smithi088.stdout:Take first snapshot .snap/test1 2023-10-16T20:40:51.840 INFO:tasks.workunit.client.0.smithi088.stdout:Create dir 200 to 299 ... 2023-10-16T20:40:51.842 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 200 2023-10-16T20:40:52.410 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 201 2023-10-16T20:40:52.481 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 202 2023-10-16T20:40:52.511 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 203 2023-10-16T20:40:52.541 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 204 2023-10-16T20:40:52.654 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 205 2023-10-16T20:40:52.685 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 206 2023-10-16T20:40:52.715 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 207 2023-10-16T20:40:52.745 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 208 2023-10-16T20:40:52.774 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 209 2023-10-16T20:40:52.804 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 210 2023-10-16T20:40:52.835 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 211 2023-10-16T20:40:52.865 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 212 2023-10-16T20:40:52.894 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 213 2023-10-16T20:40:52.924 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 214 2023-10-16T20:40:52.955 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 215 2023-10-16T20:40:52.983 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 216 2023-10-16T20:40:53.014 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 217 2023-10-16T20:40:53.388 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 218 2023-10-16T20:40:53.418 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 219 2023-10-16T20:40:53.448 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 220 2023-10-16T20:40:53.867 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 221 2023-10-16T20:40:53.899 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 222 2023-10-16T20:40:53.930 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 223 2023-10-16T20:40:53.960 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 224 2023-10-16T20:40:53.991 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 225 2023-10-16T20:40:54.021 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 226 2023-10-16T20:40:54.052 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 227 2023-10-16T20:40:54.082 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 228 2023-10-16T20:40:54.112 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 229 2023-10-16T20:40:54.142 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 230 2023-10-16T20:40:54.178 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:40:54.182 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:40:54.252 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 231 2023-10-16T20:40:54.281 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 232 2023-10-16T20:40:54.311 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 233 2023-10-16T20:40:54.341 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 234 2023-10-16T20:40:54.372 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 235 2023-10-16T20:40:54.404 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 236 2023-10-16T20:40:54.434 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 237 2023-10-16T20:40:54.466 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 238 2023-10-16T20:40:54.495 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 239 2023-10-16T20:40:54.525 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 240 2023-10-16T20:40:54.556 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 241 2023-10-16T20:40:54.585 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 242 2023-10-16T20:40:55.678 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 243 2023-10-16T20:40:55.706 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 244 2023-10-16T20:40:55.737 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 245 2023-10-16T20:40:55.766 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 246 2023-10-16T20:40:55.796 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 247 2023-10-16T20:40:55.827 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 248 2023-10-16T20:40:55.857 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 249 2023-10-16T20:40:55.887 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 250 2023-10-16T20:40:55.926 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 251 2023-10-16T20:40:57.402 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 252 2023-10-16T20:40:57.434 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 253 2023-10-16T20:40:57.465 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 254 2023-10-16T20:40:57.494 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 255 2023-10-16T20:40:57.524 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 256 2023-10-16T20:40:57.554 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 257 2023-10-16T20:40:57.584 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 258 2023-10-16T20:40:57.614 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 259 2023-10-16T20:40:57.646 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 260 2023-10-16T20:40:57.677 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 261 2023-10-16T20:40:57.708 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 262 2023-10-16T20:40:57.738 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 263 2023-10-16T20:40:57.767 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 264 2023-10-16T20:40:57.799 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 265 2023-10-16T20:40:57.828 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 266 2023-10-16T20:40:57.858 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 267 2023-10-16T20:40:57.888 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 268 2023-10-16T20:40:57.918 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 269 2023-10-16T20:40:57.947 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 270 2023-10-16T20:40:57.976 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 271 2023-10-16T20:40:58.006 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 272 2023-10-16T20:40:58.036 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 273 2023-10-16T20:40:58.066 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 274 2023-10-16T20:40:58.095 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 275 2023-10-16T20:40:58.124 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 276 2023-10-16T20:40:58.154 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 277 2023-10-16T20:40:58.184 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 278 2023-10-16T20:40:58.265 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 279 2023-10-16T20:40:58.294 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 280 2023-10-16T20:40:58.323 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 281 2023-10-16T20:40:58.354 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 282 2023-10-16T20:40:58.384 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 283 2023-10-16T20:40:58.413 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 284 2023-10-16T20:40:58.442 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 285 2023-10-16T20:40:58.473 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 286 2023-10-16T20:40:58.504 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 287 2023-10-16T20:40:58.532 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 288 2023-10-16T20:40:58.561 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 289 2023-10-16T20:40:58.590 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 290 2023-10-16T20:40:58.620 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 291 2023-10-16T20:40:58.650 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 292 2023-10-16T20:40:58.681 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 293 2023-10-16T20:40:58.712 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 294 2023-10-16T20:40:58.742 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 295 2023-10-16T20:40:58.772 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 296 2023-10-16T20:40:58.801 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 297 2023-10-16T20:40:58.832 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 298 2023-10-16T20:40:58.862 INFO:tasks.workunit.client.0.smithi088.stdout: create dir 299 2023-10-16T20:40:58.891 INFO:tasks.workunit.client.0.smithi088.stdout:Create a snapshot in every first level dir ... 2023-10-16T20:40:59.122 INFO:tasks.workunit.client.0.smithi088.stdout: create 100/.snap/snap-subdir-test 2023-10-16T20:40:59.890 INFO:tasks.workunit.client.0.smithi088.stdout: create 100/30 file after the snapshot 2023-10-16T20:40:59.892 INFO:tasks.workunit.client.0.smithi088.stdout: create 100/31 file after the snapshot 2023-10-16T20:40:59.895 INFO:tasks.workunit.client.0.smithi088.stdout: create 100/32 file after the snapshot 2023-10-16T20:40:59.897 INFO:tasks.workunit.client.0.smithi088.stdout: create 100/33 file after the snapshot 2023-10-16T20:40:59.900 INFO:tasks.workunit.client.0.smithi088.stdout: create 100/34 file after the snapshot 2023-10-16T20:40:59.903 INFO:tasks.workunit.client.0.smithi088.stdout: create 100/35 file after the snapshot 2023-10-16T20:40:59.905 INFO:tasks.workunit.client.0.smithi088.stdout: create 100/36 file after the snapshot 2023-10-16T20:40:59.907 INFO:tasks.workunit.client.0.smithi088.stdout: create 100/37 file after the snapshot 2023-10-16T20:40:59.910 INFO:tasks.workunit.client.0.smithi088.stdout: create 100/38 file after the snapshot 2023-10-16T20:40:59.912 INFO:tasks.workunit.client.0.smithi088.stdout: create 100/39 file after the snapshot 2023-10-16T20:40:59.915 INFO:tasks.workunit.client.0.smithi088.stdout: create 101/.snap/snap-subdir-test 2023-10-16T20:40:59.939 INFO:tasks.workunit.client.0.smithi088.stdout: create 101/30 file after the snapshot 2023-10-16T20:40:59.941 INFO:tasks.workunit.client.0.smithi088.stdout: create 101/31 file after the snapshot 2023-10-16T20:40:59.943 INFO:tasks.workunit.client.0.smithi088.stdout: create 101/32 file after the snapshot 2023-10-16T20:40:59.946 INFO:tasks.workunit.client.0.smithi088.stdout: create 101/33 file after the snapshot 2023-10-16T20:40:59.948 INFO:tasks.workunit.client.0.smithi088.stdout: create 101/34 file after the snapshot 2023-10-16T20:40:59.950 INFO:tasks.workunit.client.0.smithi088.stdout: create 101/35 file after the snapshot 2023-10-16T20:40:59.953 INFO:tasks.workunit.client.0.smithi088.stdout: create 101/36 file after the snapshot 2023-10-16T20:40:59.955 INFO:tasks.workunit.client.0.smithi088.stdout: create 101/37 file after the snapshot 2023-10-16T20:40:59.957 INFO:tasks.workunit.client.0.smithi088.stdout: create 101/38 file after the snapshot 2023-10-16T20:40:59.960 INFO:tasks.workunit.client.0.smithi088.stdout: create 101/39 file after the snapshot 2023-10-16T20:40:59.962 INFO:tasks.workunit.client.0.smithi088.stdout: create 102/.snap/snap-subdir-test 2023-10-16T20:40:59.986 INFO:tasks.workunit.client.0.smithi088.stdout: create 102/30 file after the snapshot 2023-10-16T20:40:59.988 INFO:tasks.workunit.client.0.smithi088.stdout: create 102/31 file after the snapshot 2023-10-16T20:40:59.990 INFO:tasks.workunit.client.0.smithi088.stdout: create 102/32 file after the snapshot 2023-10-16T20:40:59.993 INFO:tasks.workunit.client.0.smithi088.stdout: create 102/33 file after the snapshot 2023-10-16T20:40:59.995 INFO:tasks.workunit.client.0.smithi088.stdout: create 102/34 file after the snapshot 2023-10-16T20:40:59.998 INFO:tasks.workunit.client.0.smithi088.stdout: create 102/35 file after the snapshot 2023-10-16T20:41:00.000 INFO:tasks.workunit.client.0.smithi088.stdout: create 102/36 file after the snapshot 2023-10-16T20:41:00.002 INFO:tasks.workunit.client.0.smithi088.stdout: create 102/37 file after the snapshot 2023-10-16T20:41:00.005 INFO:tasks.workunit.client.0.smithi088.stdout: create 102/38 file after the snapshot 2023-10-16T20:41:00.008 INFO:tasks.workunit.client.0.smithi088.stdout: create 102/39 file after the snapshot 2023-10-16T20:41:00.010 INFO:tasks.workunit.client.0.smithi088.stdout: create 103/.snap/snap-subdir-test 2023-10-16T20:41:00.045 INFO:tasks.workunit.client.0.smithi088.stdout: create 103/30 file after the snapshot 2023-10-16T20:41:00.047 INFO:tasks.workunit.client.0.smithi088.stdout: create 103/31 file after the snapshot 2023-10-16T20:41:00.050 INFO:tasks.workunit.client.0.smithi088.stdout: create 103/32 file after the snapshot 2023-10-16T20:41:00.052 INFO:tasks.workunit.client.0.smithi088.stdout: create 103/33 file after the snapshot 2023-10-16T20:41:00.055 INFO:tasks.workunit.client.0.smithi088.stdout: create 103/34 file after the snapshot 2023-10-16T20:41:00.057 INFO:tasks.workunit.client.0.smithi088.stdout: create 103/35 file after the snapshot 2023-10-16T20:41:00.060 INFO:tasks.workunit.client.0.smithi088.stdout: create 103/36 file after the snapshot 2023-10-16T20:41:00.063 INFO:tasks.workunit.client.0.smithi088.stdout: create 103/37 file after the snapshot 2023-10-16T20:41:00.065 INFO:tasks.workunit.client.0.smithi088.stdout: create 103/38 file after the snapshot 2023-10-16T20:41:00.067 INFO:tasks.workunit.client.0.smithi088.stdout: create 103/39 file after the snapshot 2023-10-16T20:41:00.070 INFO:tasks.workunit.client.0.smithi088.stdout: create 104/.snap/snap-subdir-test 2023-10-16T20:41:00.094 INFO:tasks.workunit.client.0.smithi088.stdout: create 104/30 file after the snapshot 2023-10-16T20:41:00.096 INFO:tasks.workunit.client.0.smithi088.stdout: create 104/31 file after the snapshot 2023-10-16T20:41:00.098 INFO:tasks.workunit.client.0.smithi088.stdout: create 104/32 file after the snapshot 2023-10-16T20:41:00.101 INFO:tasks.workunit.client.0.smithi088.stdout: create 104/33 file after the snapshot 2023-10-16T20:41:00.103 INFO:tasks.workunit.client.0.smithi088.stdout: create 104/34 file after the snapshot 2023-10-16T20:41:00.106 INFO:tasks.workunit.client.0.smithi088.stdout: create 104/35 file after the snapshot 2023-10-16T20:41:00.108 INFO:tasks.workunit.client.0.smithi088.stdout: create 104/36 file after the snapshot 2023-10-16T20:41:00.111 INFO:tasks.workunit.client.0.smithi088.stdout: create 104/37 file after the snapshot 2023-10-16T20:41:00.113 INFO:tasks.workunit.client.0.smithi088.stdout: create 104/38 file after the snapshot 2023-10-16T20:41:00.222 INFO:tasks.workunit.client.0.smithi088.stdout: create 104/39 file after the snapshot 2023-10-16T20:41:00.266 INFO:tasks.workunit.client.0.smithi088.stdout: create 105/.snap/snap-subdir-test 2023-10-16T20:41:00.308 INFO:tasks.workunit.client.0.smithi088.stdout: create 105/30 file after the snapshot 2023-10-16T20:41:00.310 INFO:tasks.workunit.client.0.smithi088.stdout: create 105/31 file after the snapshot 2023-10-16T20:41:00.312 INFO:tasks.workunit.client.0.smithi088.stdout: create 105/32 file after the snapshot 2023-10-16T20:41:00.315 INFO:tasks.workunit.client.0.smithi088.stdout: create 105/33 file after the snapshot 2023-10-16T20:41:00.317 INFO:tasks.workunit.client.0.smithi088.stdout: create 105/34 file after the snapshot 2023-10-16T20:41:00.320 INFO:tasks.workunit.client.0.smithi088.stdout: create 105/35 file after the snapshot 2023-10-16T20:41:00.323 INFO:tasks.workunit.client.0.smithi088.stdout: create 105/36 file after the snapshot 2023-10-16T20:41:00.326 INFO:tasks.workunit.client.0.smithi088.stdout: create 105/37 file after the snapshot 2023-10-16T20:41:00.328 INFO:tasks.workunit.client.0.smithi088.stdout: create 105/38 file after the snapshot 2023-10-16T20:41:00.332 INFO:tasks.workunit.client.0.smithi088.stdout: create 105/39 file after the snapshot 2023-10-16T20:41:00.334 INFO:tasks.workunit.client.0.smithi088.stdout: create 106/.snap/snap-subdir-test 2023-10-16T20:41:00.366 INFO:tasks.workunit.client.0.smithi088.stdout: create 106/30 file after the snapshot 2023-10-16T20:41:00.368 INFO:tasks.workunit.client.0.smithi088.stdout: create 106/31 file after the snapshot 2023-10-16T20:41:00.370 INFO:tasks.workunit.client.0.smithi088.stdout: create 106/32 file after the snapshot 2023-10-16T20:41:00.373 INFO:tasks.workunit.client.0.smithi088.stdout: create 106/33 file after the snapshot 2023-10-16T20:41:00.375 INFO:tasks.workunit.client.0.smithi088.stdout: create 106/34 file after the snapshot 2023-10-16T20:41:00.377 INFO:tasks.workunit.client.0.smithi088.stdout: create 106/35 file after the snapshot 2023-10-16T20:41:00.380 INFO:tasks.workunit.client.0.smithi088.stdout: create 106/36 file after the snapshot 2023-10-16T20:41:00.382 INFO:tasks.workunit.client.0.smithi088.stdout: create 106/37 file after the snapshot 2023-10-16T20:41:00.385 INFO:tasks.workunit.client.0.smithi088.stdout: create 106/38 file after the snapshot 2023-10-16T20:41:00.387 INFO:tasks.workunit.client.0.smithi088.stdout: create 106/39 file after the snapshot 2023-10-16T20:41:00.390 INFO:tasks.workunit.client.0.smithi088.stdout: create 107/.snap/snap-subdir-test 2023-10-16T20:41:00.413 INFO:tasks.workunit.client.0.smithi088.stdout: create 107/30 file after the snapshot 2023-10-16T20:41:00.415 INFO:tasks.workunit.client.0.smithi088.stdout: create 107/31 file after the snapshot 2023-10-16T20:41:00.418 INFO:tasks.workunit.client.0.smithi088.stdout: create 107/32 file after the snapshot 2023-10-16T20:41:00.420 INFO:tasks.workunit.client.0.smithi088.stdout: create 107/33 file after the snapshot 2023-10-16T20:41:00.423 INFO:tasks.workunit.client.0.smithi088.stdout: create 107/34 file after the snapshot 2023-10-16T20:41:00.425 INFO:tasks.workunit.client.0.smithi088.stdout: create 107/35 file after the snapshot 2023-10-16T20:41:00.428 INFO:tasks.workunit.client.0.smithi088.stdout: create 107/36 file after the snapshot 2023-10-16T20:41:00.431 INFO:tasks.workunit.client.0.smithi088.stdout: create 107/37 file after the snapshot 2023-10-16T20:41:00.434 INFO:tasks.workunit.client.0.smithi088.stdout: create 107/38 file after the snapshot 2023-10-16T20:41:00.437 INFO:tasks.workunit.client.0.smithi088.stdout: create 107/39 file after the snapshot 2023-10-16T20:41:00.439 INFO:tasks.workunit.client.0.smithi088.stdout: create 108/.snap/snap-subdir-test 2023-10-16T20:41:00.462 INFO:tasks.workunit.client.0.smithi088.stdout: create 108/30 file after the snapshot 2023-10-16T20:41:00.464 INFO:tasks.workunit.client.0.smithi088.stdout: create 108/31 file after the snapshot 2023-10-16T20:41:00.467 INFO:tasks.workunit.client.0.smithi088.stdout: create 108/32 file after the snapshot 2023-10-16T20:41:00.469 INFO:tasks.workunit.client.0.smithi088.stdout: create 108/33 file after the snapshot 2023-10-16T20:41:00.473 INFO:tasks.workunit.client.0.smithi088.stdout: create 108/34 file after the snapshot 2023-10-16T20:41:00.475 INFO:tasks.workunit.client.0.smithi088.stdout: create 108/35 file after the snapshot 2023-10-16T20:41:00.477 INFO:tasks.workunit.client.0.smithi088.stdout: create 108/36 file after the snapshot 2023-10-16T20:41:00.480 INFO:tasks.workunit.client.0.smithi088.stdout: create 108/37 file after the snapshot 2023-10-16T20:41:00.482 INFO:tasks.workunit.client.0.smithi088.stdout: create 108/38 file after the snapshot 2023-10-16T20:41:00.485 INFO:tasks.workunit.client.0.smithi088.stdout: create 108/39 file after the snapshot 2023-10-16T20:41:00.489 INFO:tasks.workunit.client.0.smithi088.stdout: create 109/.snap/snap-subdir-test 2023-10-16T20:41:00.512 INFO:tasks.workunit.client.0.smithi088.stdout: create 109/30 file after the snapshot 2023-10-16T20:41:00.514 INFO:tasks.workunit.client.0.smithi088.stdout: create 109/31 file after the snapshot 2023-10-16T20:41:00.517 INFO:tasks.workunit.client.0.smithi088.stdout: create 109/32 file after the snapshot 2023-10-16T20:41:00.519 INFO:tasks.workunit.client.0.smithi088.stdout: create 109/33 file after the snapshot 2023-10-16T20:41:00.522 INFO:tasks.workunit.client.0.smithi088.stdout: create 109/34 file after the snapshot 2023-10-16T20:41:00.524 INFO:tasks.workunit.client.0.smithi088.stdout: create 109/35 file after the snapshot 2023-10-16T20:41:00.526 INFO:tasks.workunit.client.0.smithi088.stdout: create 109/36 file after the snapshot 2023-10-16T20:41:00.528 INFO:tasks.workunit.client.0.smithi088.stdout: create 109/37 file after the snapshot 2023-10-16T20:41:00.531 INFO:tasks.workunit.client.0.smithi088.stdout: create 109/38 file after the snapshot 2023-10-16T20:41:00.534 INFO:tasks.workunit.client.0.smithi088.stdout: create 109/39 file after the snapshot 2023-10-16T20:41:00.536 INFO:tasks.workunit.client.0.smithi088.stdout: create 110/.snap/snap-subdir-test 2023-10-16T20:41:01.367 INFO:tasks.workunit.client.0.smithi088.stdout: create 110/30 file after the snapshot 2023-10-16T20:41:01.369 INFO:tasks.workunit.client.0.smithi088.stdout: create 110/31 file after the snapshot 2023-10-16T20:41:01.372 INFO:tasks.workunit.client.0.smithi088.stdout: create 110/32 file after the snapshot 2023-10-16T20:41:01.374 INFO:tasks.workunit.client.0.smithi088.stdout: create 110/33 file after the snapshot 2023-10-16T20:41:01.377 INFO:tasks.workunit.client.0.smithi088.stdout: create 110/34 file after the snapshot 2023-10-16T20:41:01.380 INFO:tasks.workunit.client.0.smithi088.stdout: create 110/35 file after the snapshot 2023-10-16T20:41:01.382 INFO:tasks.workunit.client.0.smithi088.stdout: create 110/36 file after the snapshot 2023-10-16T20:41:01.385 INFO:tasks.workunit.client.0.smithi088.stdout: create 110/37 file after the snapshot 2023-10-16T20:41:01.388 INFO:tasks.workunit.client.0.smithi088.stdout: create 110/38 file after the snapshot 2023-10-16T20:41:01.391 INFO:tasks.workunit.client.0.smithi088.stdout: create 110/39 file after the snapshot 2023-10-16T20:41:01.394 INFO:tasks.workunit.client.0.smithi088.stdout: create 111/.snap/snap-subdir-test 2023-10-16T20:41:01.428 INFO:tasks.workunit.client.0.smithi088.stdout: create 111/30 file after the snapshot 2023-10-16T20:41:01.430 INFO:tasks.workunit.client.0.smithi088.stdout: create 111/31 file after the snapshot 2023-10-16T20:41:01.433 INFO:tasks.workunit.client.0.smithi088.stdout: create 111/32 file after the snapshot 2023-10-16T20:41:01.436 INFO:tasks.workunit.client.0.smithi088.stdout: create 111/33 file after the snapshot 2023-10-16T20:41:01.438 INFO:tasks.workunit.client.0.smithi088.stdout: create 111/34 file after the snapshot 2023-10-16T20:41:01.441 INFO:tasks.workunit.client.0.smithi088.stdout: create 111/35 file after the snapshot 2023-10-16T20:41:01.443 INFO:tasks.workunit.client.0.smithi088.stdout: create 111/36 file after the snapshot 2023-10-16T20:41:01.446 INFO:tasks.workunit.client.0.smithi088.stdout: create 111/37 file after the snapshot 2023-10-16T20:41:01.449 INFO:tasks.workunit.client.0.smithi088.stdout: create 111/38 file after the snapshot 2023-10-16T20:41:01.451 INFO:tasks.workunit.client.0.smithi088.stdout: create 111/39 file after the snapshot 2023-10-16T20:41:01.453 INFO:tasks.workunit.client.0.smithi088.stdout: create 112/.snap/snap-subdir-test 2023-10-16T20:41:01.477 INFO:tasks.workunit.client.0.smithi088.stdout: create 112/30 file after the snapshot 2023-10-16T20:41:01.479 INFO:tasks.workunit.client.0.smithi088.stdout: create 112/31 file after the snapshot 2023-10-16T20:41:01.482 INFO:tasks.workunit.client.0.smithi088.stdout: create 112/32 file after the snapshot 2023-10-16T20:41:01.484 INFO:tasks.workunit.client.0.smithi088.stdout: create 112/33 file after the snapshot 2023-10-16T20:41:01.487 INFO:tasks.workunit.client.0.smithi088.stdout: create 112/34 file after the snapshot 2023-10-16T20:41:01.490 INFO:tasks.workunit.client.0.smithi088.stdout: create 112/35 file after the snapshot 2023-10-16T20:41:01.492 INFO:tasks.workunit.client.0.smithi088.stdout: create 112/36 file after the snapshot 2023-10-16T20:41:01.495 INFO:tasks.workunit.client.0.smithi088.stdout: create 112/37 file after the snapshot 2023-10-16T20:41:01.497 INFO:tasks.workunit.client.0.smithi088.stdout: create 112/38 file after the snapshot 2023-10-16T20:41:01.500 INFO:tasks.workunit.client.0.smithi088.stdout: create 112/39 file after the snapshot 2023-10-16T20:41:01.502 INFO:tasks.workunit.client.0.smithi088.stdout: create 113/.snap/snap-subdir-test 2023-10-16T20:41:01.526 INFO:tasks.workunit.client.0.smithi088.stdout: create 113/30 file after the snapshot 2023-10-16T20:41:01.528 INFO:tasks.workunit.client.0.smithi088.stdout: create 113/31 file after the snapshot 2023-10-16T20:41:01.531 INFO:tasks.workunit.client.0.smithi088.stdout: create 113/32 file after the snapshot 2023-10-16T20:41:01.533 INFO:tasks.workunit.client.0.smithi088.stdout: create 113/33 file after the snapshot 2023-10-16T20:41:01.536 INFO:tasks.workunit.client.0.smithi088.stdout: create 113/34 file after the snapshot 2023-10-16T20:41:01.538 INFO:tasks.workunit.client.0.smithi088.stdout: create 113/35 file after the snapshot 2023-10-16T20:41:01.541 INFO:tasks.workunit.client.0.smithi088.stdout: create 113/36 file after the snapshot 2023-10-16T20:41:01.543 INFO:tasks.workunit.client.0.smithi088.stdout: create 113/37 file after the snapshot 2023-10-16T20:41:01.546 INFO:tasks.workunit.client.0.smithi088.stdout: create 113/38 file after the snapshot 2023-10-16T20:41:01.548 INFO:tasks.workunit.client.0.smithi088.stdout: create 113/39 file after the snapshot 2023-10-16T20:41:01.551 INFO:tasks.workunit.client.0.smithi088.stdout: create 114/.snap/snap-subdir-test 2023-10-16T20:41:01.574 INFO:tasks.workunit.client.0.smithi088.stdout: create 114/30 file after the snapshot 2023-10-16T20:41:01.577 INFO:tasks.workunit.client.0.smithi088.stdout: create 114/31 file after the snapshot 2023-10-16T20:41:01.579 INFO:tasks.workunit.client.0.smithi088.stdout: create 114/32 file after the snapshot 2023-10-16T20:41:01.581 INFO:tasks.workunit.client.0.smithi088.stdout: create 114/33 file after the snapshot 2023-10-16T20:41:01.584 INFO:tasks.workunit.client.0.smithi088.stdout: create 114/34 file after the snapshot 2023-10-16T20:41:01.586 INFO:tasks.workunit.client.0.smithi088.stdout: create 114/35 file after the snapshot 2023-10-16T20:41:01.589 INFO:tasks.workunit.client.0.smithi088.stdout: create 114/36 file after the snapshot 2023-10-16T20:41:01.592 INFO:tasks.workunit.client.0.smithi088.stdout: create 114/37 file after the snapshot 2023-10-16T20:41:01.594 INFO:tasks.workunit.client.0.smithi088.stdout: create 114/38 file after the snapshot 2023-10-16T20:41:01.596 INFO:tasks.workunit.client.0.smithi088.stdout: create 114/39 file after the snapshot 2023-10-16T20:41:01.598 INFO:tasks.workunit.client.0.smithi088.stdout: create 115/.snap/snap-subdir-test 2023-10-16T20:41:01.621 INFO:tasks.workunit.client.0.smithi088.stdout: create 115/30 file after the snapshot 2023-10-16T20:41:01.623 INFO:tasks.workunit.client.0.smithi088.stdout: create 115/31 file after the snapshot 2023-10-16T20:41:01.625 INFO:tasks.workunit.client.0.smithi088.stdout: create 115/32 file after the snapshot 2023-10-16T20:41:01.628 INFO:tasks.workunit.client.0.smithi088.stdout: create 115/33 file after the snapshot 2023-10-16T20:41:01.631 INFO:tasks.workunit.client.0.smithi088.stdout: create 115/34 file after the snapshot 2023-10-16T20:41:01.634 INFO:tasks.workunit.client.0.smithi088.stdout: create 115/35 file after the snapshot 2023-10-16T20:41:01.636 INFO:tasks.workunit.client.0.smithi088.stdout: create 115/36 file after the snapshot 2023-10-16T20:41:01.639 INFO:tasks.workunit.client.0.smithi088.stdout: create 115/37 file after the snapshot 2023-10-16T20:41:01.641 INFO:tasks.workunit.client.0.smithi088.stdout: create 115/38 file after the snapshot 2023-10-16T20:41:01.644 INFO:tasks.workunit.client.0.smithi088.stdout: create 115/39 file after the snapshot 2023-10-16T20:41:01.646 INFO:tasks.workunit.client.0.smithi088.stdout: create 116/.snap/snap-subdir-test 2023-10-16T20:41:01.670 INFO:tasks.workunit.client.0.smithi088.stdout: create 116/30 file after the snapshot 2023-10-16T20:41:01.672 INFO:tasks.workunit.client.0.smithi088.stdout: create 116/31 file after the snapshot 2023-10-16T20:41:01.674 INFO:tasks.workunit.client.0.smithi088.stdout: create 116/32 file after the snapshot 2023-10-16T20:41:01.677 INFO:tasks.workunit.client.0.smithi088.stdout: create 116/33 file after the snapshot 2023-10-16T20:41:01.679 INFO:tasks.workunit.client.0.smithi088.stdout: create 116/34 file after the snapshot 2023-10-16T20:41:01.682 INFO:tasks.workunit.client.0.smithi088.stdout: create 116/35 file after the snapshot 2023-10-16T20:41:01.685 INFO:tasks.workunit.client.0.smithi088.stdout: create 116/36 file after the snapshot 2023-10-16T20:41:01.687 INFO:tasks.workunit.client.0.smithi088.stdout: create 116/37 file after the snapshot 2023-10-16T20:41:01.690 INFO:tasks.workunit.client.0.smithi088.stdout: create 116/38 file after the snapshot 2023-10-16T20:41:01.692 INFO:tasks.workunit.client.0.smithi088.stdout: create 116/39 file after the snapshot 2023-10-16T20:41:01.696 INFO:tasks.workunit.client.0.smithi088.stdout: create 117/.snap/snap-subdir-test 2023-10-16T20:41:01.719 INFO:tasks.workunit.client.0.smithi088.stdout: create 117/30 file after the snapshot 2023-10-16T20:41:01.720 INFO:tasks.workunit.client.0.smithi088.stdout: create 117/31 file after the snapshot 2023-10-16T20:41:01.722 INFO:tasks.workunit.client.0.smithi088.stdout: create 117/32 file after the snapshot 2023-10-16T20:41:01.725 INFO:tasks.workunit.client.0.smithi088.stdout: create 117/33 file after the snapshot 2023-10-16T20:41:01.727 INFO:tasks.workunit.client.0.smithi088.stdout: create 117/34 file after the snapshot 2023-10-16T20:41:01.730 INFO:tasks.workunit.client.0.smithi088.stdout: create 117/35 file after the snapshot 2023-10-16T20:41:01.732 INFO:tasks.workunit.client.0.smithi088.stdout: create 117/36 file after the snapshot 2023-10-16T20:41:01.735 INFO:tasks.workunit.client.0.smithi088.stdout: create 117/37 file after the snapshot 2023-10-16T20:41:01.737 INFO:tasks.workunit.client.0.smithi088.stdout: create 117/38 file after the snapshot 2023-10-16T20:41:01.739 INFO:tasks.workunit.client.0.smithi088.stdout: create 117/39 file after the snapshot 2023-10-16T20:41:01.742 INFO:tasks.workunit.client.0.smithi088.stdout: create 118/.snap/snap-subdir-test 2023-10-16T20:41:01.774 INFO:tasks.workunit.client.0.smithi088.stdout: create 118/30 file after the snapshot 2023-10-16T20:41:01.776 INFO:tasks.workunit.client.0.smithi088.stdout: create 118/31 file after the snapshot 2023-10-16T20:41:01.778 INFO:tasks.workunit.client.0.smithi088.stdout: create 118/32 file after the snapshot 2023-10-16T20:41:01.780 INFO:tasks.workunit.client.0.smithi088.stdout: create 118/33 file after the snapshot 2023-10-16T20:41:01.783 INFO:tasks.workunit.client.0.smithi088.stdout: create 118/34 file after the snapshot 2023-10-16T20:41:01.785 INFO:tasks.workunit.client.0.smithi088.stdout: create 118/35 file after the snapshot 2023-10-16T20:41:01.787 INFO:tasks.workunit.client.0.smithi088.stdout: create 118/36 file after the snapshot 2023-10-16T20:41:01.790 INFO:tasks.workunit.client.0.smithi088.stdout: create 118/37 file after the snapshot 2023-10-16T20:41:01.793 INFO:tasks.workunit.client.0.smithi088.stdout: create 118/38 file after the snapshot 2023-10-16T20:41:01.795 INFO:tasks.workunit.client.0.smithi088.stdout: create 118/39 file after the snapshot 2023-10-16T20:41:01.798 INFO:tasks.workunit.client.0.smithi088.stdout: create 119/.snap/snap-subdir-test 2023-10-16T20:41:01.823 INFO:tasks.workunit.client.0.smithi088.stdout: create 119/30 file after the snapshot 2023-10-16T20:41:01.825 INFO:tasks.workunit.client.0.smithi088.stdout: create 119/31 file after the snapshot 2023-10-16T20:41:01.827 INFO:tasks.workunit.client.0.smithi088.stdout: create 119/32 file after the snapshot 2023-10-16T20:41:01.830 INFO:tasks.workunit.client.0.smithi088.stdout: create 119/33 file after the snapshot 2023-10-16T20:41:01.832 INFO:tasks.workunit.client.0.smithi088.stdout: create 119/34 file after the snapshot 2023-10-16T20:41:01.835 INFO:tasks.workunit.client.0.smithi088.stdout: create 119/35 file after the snapshot 2023-10-16T20:41:01.838 INFO:tasks.workunit.client.0.smithi088.stdout: create 119/36 file after the snapshot 2023-10-16T20:41:01.840 INFO:tasks.workunit.client.0.smithi088.stdout: create 119/37 file after the snapshot 2023-10-16T20:41:01.843 INFO:tasks.workunit.client.0.smithi088.stdout: create 119/38 file after the snapshot 2023-10-16T20:41:01.845 INFO:tasks.workunit.client.0.smithi088.stdout: create 119/39 file after the snapshot 2023-10-16T20:41:01.848 INFO:tasks.workunit.client.0.smithi088.stdout: create 120/.snap/snap-subdir-test 2023-10-16T20:41:01.870 INFO:tasks.workunit.client.0.smithi088.stdout: create 120/30 file after the snapshot 2023-10-16T20:41:01.872 INFO:tasks.workunit.client.0.smithi088.stdout: create 120/31 file after the snapshot 2023-10-16T20:41:01.875 INFO:tasks.workunit.client.0.smithi088.stdout: create 120/32 file after the snapshot 2023-10-16T20:41:01.878 INFO:tasks.workunit.client.0.smithi088.stdout: create 120/33 file after the snapshot 2023-10-16T20:41:01.880 INFO:tasks.workunit.client.0.smithi088.stdout: create 120/34 file after the snapshot 2023-10-16T20:41:01.883 INFO:tasks.workunit.client.0.smithi088.stdout: create 120/35 file after the snapshot 2023-10-16T20:41:01.886 INFO:tasks.workunit.client.0.smithi088.stdout: create 120/36 file after the snapshot 2023-10-16T20:41:01.889 INFO:tasks.workunit.client.0.smithi088.stdout: create 120/37 file after the snapshot 2023-10-16T20:41:01.891 INFO:tasks.workunit.client.0.smithi088.stdout: create 120/38 file after the snapshot 2023-10-16T20:41:01.893 INFO:tasks.workunit.client.0.smithi088.stdout: create 120/39 file after the snapshot 2023-10-16T20:41:01.896 INFO:tasks.workunit.client.0.smithi088.stdout: create 121/.snap/snap-subdir-test 2023-10-16T20:41:01.918 INFO:tasks.workunit.client.0.smithi088.stdout: create 121/30 file after the snapshot 2023-10-16T20:41:01.922 INFO:tasks.workunit.client.0.smithi088.stdout: create 121/31 file after the snapshot 2023-10-16T20:41:01.923 INFO:tasks.workunit.client.0.smithi088.stdout: create 121/32 file after the snapshot 2023-10-16T20:41:01.925 INFO:tasks.workunit.client.0.smithi088.stdout: create 121/33 file after the snapshot 2023-10-16T20:41:01.928 INFO:tasks.workunit.client.0.smithi088.stdout: create 121/34 file after the snapshot 2023-10-16T20:41:01.930 INFO:tasks.workunit.client.0.smithi088.stdout: create 121/35 file after the snapshot 2023-10-16T20:41:01.933 INFO:tasks.workunit.client.0.smithi088.stdout: create 121/36 file after the snapshot 2023-10-16T20:41:01.935 INFO:tasks.workunit.client.0.smithi088.stdout: create 121/37 file after the snapshot 2023-10-16T20:41:01.937 INFO:tasks.workunit.client.0.smithi088.stdout: create 121/38 file after the snapshot 2023-10-16T20:41:01.939 INFO:tasks.workunit.client.0.smithi088.stdout: create 121/39 file after the snapshot 2023-10-16T20:41:01.942 INFO:tasks.workunit.client.0.smithi088.stdout: create 122/.snap/snap-subdir-test 2023-10-16T20:41:01.964 INFO:tasks.workunit.client.0.smithi088.stdout: create 122/30 file after the snapshot 2023-10-16T20:41:01.966 INFO:tasks.workunit.client.0.smithi088.stdout: create 122/31 file after the snapshot 2023-10-16T20:41:01.968 INFO:tasks.workunit.client.0.smithi088.stdout: create 122/32 file after the snapshot 2023-10-16T20:41:01.971 INFO:tasks.workunit.client.0.smithi088.stdout: create 122/33 file after the snapshot 2023-10-16T20:41:01.973 INFO:tasks.workunit.client.0.smithi088.stdout: create 122/34 file after the snapshot 2023-10-16T20:41:01.975 INFO:tasks.workunit.client.0.smithi088.stdout: create 122/35 file after the snapshot 2023-10-16T20:41:01.978 INFO:tasks.workunit.client.0.smithi088.stdout: create 122/36 file after the snapshot 2023-10-16T20:41:01.980 INFO:tasks.workunit.client.0.smithi088.stdout: create 122/37 file after the snapshot 2023-10-16T20:41:01.982 INFO:tasks.workunit.client.0.smithi088.stdout: create 122/38 file after the snapshot 2023-10-16T20:41:01.985 INFO:tasks.workunit.client.0.smithi088.stdout: create 122/39 file after the snapshot 2023-10-16T20:41:01.987 INFO:tasks.workunit.client.0.smithi088.stdout: create 123/.snap/snap-subdir-test 2023-10-16T20:41:02.010 INFO:tasks.workunit.client.0.smithi088.stdout: create 123/30 file after the snapshot 2023-10-16T20:41:02.012 INFO:tasks.workunit.client.0.smithi088.stdout: create 123/31 file after the snapshot 2023-10-16T20:41:02.015 INFO:tasks.workunit.client.0.smithi088.stdout: create 123/32 file after the snapshot 2023-10-16T20:41:02.017 INFO:tasks.workunit.client.0.smithi088.stdout: create 123/33 file after the snapshot 2023-10-16T20:41:02.020 INFO:tasks.workunit.client.0.smithi088.stdout: create 123/34 file after the snapshot 2023-10-16T20:41:02.022 INFO:tasks.workunit.client.0.smithi088.stdout: create 123/35 file after the snapshot 2023-10-16T20:41:02.025 INFO:tasks.workunit.client.0.smithi088.stdout: create 123/36 file after the snapshot 2023-10-16T20:41:02.028 INFO:tasks.workunit.client.0.smithi088.stdout: create 123/37 file after the snapshot 2023-10-16T20:41:02.030 INFO:tasks.workunit.client.0.smithi088.stdout: create 123/38 file after the snapshot 2023-10-16T20:41:02.033 INFO:tasks.workunit.client.0.smithi088.stdout: create 123/39 file after the snapshot 2023-10-16T20:41:02.035 INFO:tasks.workunit.client.0.smithi088.stdout: create 124/.snap/snap-subdir-test 2023-10-16T20:41:02.266 INFO:tasks.workunit.client.0.smithi088.stdout: create 124/30 file after the snapshot 2023-10-16T20:41:02.268 INFO:tasks.workunit.client.0.smithi088.stdout: create 124/31 file after the snapshot 2023-10-16T20:41:02.271 INFO:tasks.workunit.client.0.smithi088.stdout: create 124/32 file after the snapshot 2023-10-16T20:41:02.274 INFO:tasks.workunit.client.0.smithi088.stdout: create 124/33 file after the snapshot 2023-10-16T20:41:02.276 INFO:tasks.workunit.client.0.smithi088.stdout: create 124/34 file after the snapshot 2023-10-16T20:41:02.279 INFO:tasks.workunit.client.0.smithi088.stdout: create 124/35 file after the snapshot 2023-10-16T20:41:02.281 INFO:tasks.workunit.client.0.smithi088.stdout: create 124/36 file after the snapshot 2023-10-16T20:41:02.284 INFO:tasks.workunit.client.0.smithi088.stdout: create 124/37 file after the snapshot 2023-10-16T20:41:02.286 INFO:tasks.workunit.client.0.smithi088.stdout: create 124/38 file after the snapshot 2023-10-16T20:41:02.289 INFO:tasks.workunit.client.0.smithi088.stdout: create 124/39 file after the snapshot 2023-10-16T20:41:02.292 INFO:tasks.workunit.client.0.smithi088.stdout: create 125/.snap/snap-subdir-test 2023-10-16T20:41:02.314 INFO:tasks.workunit.client.0.smithi088.stdout: create 125/30 file after the snapshot 2023-10-16T20:41:02.316 INFO:tasks.workunit.client.0.smithi088.stdout: create 125/31 file after the snapshot 2023-10-16T20:41:02.333 INFO:tasks.workunit.client.0.smithi088.stdout: create 125/32 file after the snapshot 2023-10-16T20:41:02.342 INFO:tasks.workunit.client.0.smithi088.stdout: create 125/33 file after the snapshot 2023-10-16T20:41:02.345 INFO:tasks.workunit.client.0.smithi088.stdout: create 125/34 file after the snapshot 2023-10-16T20:41:02.348 INFO:tasks.workunit.client.0.smithi088.stdout: create 125/35 file after the snapshot 2023-10-16T20:41:02.354 INFO:tasks.workunit.client.0.smithi088.stdout: create 125/36 file after the snapshot 2023-10-16T20:41:02.356 INFO:tasks.workunit.client.0.smithi088.stdout: create 125/37 file after the snapshot 2023-10-16T20:41:02.359 INFO:tasks.workunit.client.0.smithi088.stdout: create 125/38 file after the snapshot 2023-10-16T20:41:02.361 INFO:tasks.workunit.client.0.smithi088.stdout: create 125/39 file after the snapshot 2023-10-16T20:41:02.364 INFO:tasks.workunit.client.0.smithi088.stdout: create 126/.snap/snap-subdir-test 2023-10-16T20:41:02.384 INFO:tasks.workunit.client.0.smithi088.stdout: create 126/30 file after the snapshot 2023-10-16T20:41:02.387 INFO:tasks.workunit.client.0.smithi088.stdout: create 126/31 file after the snapshot 2023-10-16T20:41:02.389 INFO:tasks.workunit.client.0.smithi088.stdout: create 126/32 file after the snapshot 2023-10-16T20:41:02.391 INFO:tasks.workunit.client.0.smithi088.stdout: create 126/33 file after the snapshot 2023-10-16T20:41:02.394 INFO:tasks.workunit.client.0.smithi088.stdout: create 126/34 file after the snapshot 2023-10-16T20:41:02.397 INFO:tasks.workunit.client.0.smithi088.stdout: create 126/35 file after the snapshot 2023-10-16T20:41:02.399 INFO:tasks.workunit.client.0.smithi088.stdout: create 126/36 file after the snapshot 2023-10-16T20:41:02.402 INFO:tasks.workunit.client.0.smithi088.stdout: create 126/37 file after the snapshot 2023-10-16T20:41:02.404 INFO:tasks.workunit.client.0.smithi088.stdout: create 126/38 file after the snapshot 2023-10-16T20:41:02.407 INFO:tasks.workunit.client.0.smithi088.stdout: create 126/39 file after the snapshot 2023-10-16T20:41:02.409 INFO:tasks.workunit.client.0.smithi088.stdout: create 127/.snap/snap-subdir-test 2023-10-16T20:41:02.433 INFO:tasks.workunit.client.0.smithi088.stdout: create 127/30 file after the snapshot 2023-10-16T20:41:02.435 INFO:tasks.workunit.client.0.smithi088.stdout: create 127/31 file after the snapshot 2023-10-16T20:41:02.438 INFO:tasks.workunit.client.0.smithi088.stdout: create 127/32 file after the snapshot 2023-10-16T20:41:02.440 INFO:tasks.workunit.client.0.smithi088.stdout: create 127/33 file after the snapshot 2023-10-16T20:41:02.443 INFO:tasks.workunit.client.0.smithi088.stdout: create 127/34 file after the snapshot 2023-10-16T20:41:02.445 INFO:tasks.workunit.client.0.smithi088.stdout: create 127/35 file after the snapshot 2023-10-16T20:41:02.447 INFO:tasks.workunit.client.0.smithi088.stdout: create 127/36 file after the snapshot 2023-10-16T20:41:02.450 INFO:tasks.workunit.client.0.smithi088.stdout: create 127/37 file after the snapshot 2023-10-16T20:41:02.452 INFO:tasks.workunit.client.0.smithi088.stdout: create 127/38 file after the snapshot 2023-10-16T20:41:02.455 INFO:tasks.workunit.client.0.smithi088.stdout: create 127/39 file after the snapshot 2023-10-16T20:41:02.458 INFO:tasks.workunit.client.0.smithi088.stdout: create 128/.snap/snap-subdir-test 2023-10-16T20:41:02.488 INFO:tasks.workunit.client.0.smithi088.stdout: create 128/30 file after the snapshot 2023-10-16T20:41:02.490 INFO:tasks.workunit.client.0.smithi088.stdout: create 128/31 file after the snapshot 2023-10-16T20:41:02.492 INFO:tasks.workunit.client.0.smithi088.stdout: create 128/32 file after the snapshot 2023-10-16T20:41:02.495 INFO:tasks.workunit.client.0.smithi088.stdout: create 128/33 file after the snapshot 2023-10-16T20:41:02.498 INFO:tasks.workunit.client.0.smithi088.stdout: create 128/34 file after the snapshot 2023-10-16T20:41:02.500 INFO:tasks.workunit.client.0.smithi088.stdout: create 128/35 file after the snapshot 2023-10-16T20:41:02.502 INFO:tasks.workunit.client.0.smithi088.stdout: create 128/36 file after the snapshot 2023-10-16T20:41:02.505 INFO:tasks.workunit.client.0.smithi088.stdout: create 128/37 file after the snapshot 2023-10-16T20:41:02.507 INFO:tasks.workunit.client.0.smithi088.stdout: create 128/38 file after the snapshot 2023-10-16T20:41:02.509 INFO:tasks.workunit.client.0.smithi088.stdout: create 128/39 file after the snapshot 2023-10-16T20:41:02.512 INFO:tasks.workunit.client.0.smithi088.stdout: create 129/.snap/snap-subdir-test 2023-10-16T20:41:02.534 INFO:tasks.workunit.client.0.smithi088.stdout: create 129/30 file after the snapshot 2023-10-16T20:41:02.536 INFO:tasks.workunit.client.0.smithi088.stdout: create 129/31 file after the snapshot 2023-10-16T20:41:02.538 INFO:tasks.workunit.client.0.smithi088.stdout: create 129/32 file after the snapshot 2023-10-16T20:41:02.541 INFO:tasks.workunit.client.0.smithi088.stdout: create 129/33 file after the snapshot 2023-10-16T20:41:02.543 INFO:tasks.workunit.client.0.smithi088.stdout: create 129/34 file after the snapshot 2023-10-16T20:41:02.546 INFO:tasks.workunit.client.0.smithi088.stdout: create 129/35 file after the snapshot 2023-10-16T20:41:02.548 INFO:tasks.workunit.client.0.smithi088.stdout: create 129/36 file after the snapshot 2023-10-16T20:41:02.551 INFO:tasks.workunit.client.0.smithi088.stdout: create 129/37 file after the snapshot 2023-10-16T20:41:02.553 INFO:tasks.workunit.client.0.smithi088.stdout: create 129/38 file after the snapshot 2023-10-16T20:41:02.555 INFO:tasks.workunit.client.0.smithi088.stdout: create 129/39 file after the snapshot 2023-10-16T20:41:02.558 INFO:tasks.workunit.client.0.smithi088.stdout: create 130/.snap/snap-subdir-test 2023-10-16T20:41:02.581 INFO:tasks.workunit.client.0.smithi088.stdout: create 130/30 file after the snapshot 2023-10-16T20:41:02.583 INFO:tasks.workunit.client.0.smithi088.stdout: create 130/31 file after the snapshot 2023-10-16T20:41:02.585 INFO:tasks.workunit.client.0.smithi088.stdout: create 130/32 file after the snapshot 2023-10-16T20:41:02.587 INFO:tasks.workunit.client.0.smithi088.stdout: create 130/33 file after the snapshot 2023-10-16T20:41:02.590 INFO:tasks.workunit.client.0.smithi088.stdout: create 130/34 file after the snapshot 2023-10-16T20:41:02.592 INFO:tasks.workunit.client.0.smithi088.stdout: create 130/35 file after the snapshot 2023-10-16T20:41:02.594 INFO:tasks.workunit.client.0.smithi088.stdout: create 130/36 file after the snapshot 2023-10-16T20:41:02.597 INFO:tasks.workunit.client.0.smithi088.stdout: create 130/37 file after the snapshot 2023-10-16T20:41:02.599 INFO:tasks.workunit.client.0.smithi088.stdout: create 130/38 file after the snapshot 2023-10-16T20:41:02.602 INFO:tasks.workunit.client.0.smithi088.stdout: create 130/39 file after the snapshot 2023-10-16T20:41:02.604 INFO:tasks.workunit.client.0.smithi088.stdout: create 131/.snap/snap-subdir-test 2023-10-16T20:41:02.628 INFO:tasks.workunit.client.0.smithi088.stdout: create 131/30 file after the snapshot 2023-10-16T20:41:02.630 INFO:tasks.workunit.client.0.smithi088.stdout: create 131/31 file after the snapshot 2023-10-16T20:41:02.632 INFO:tasks.workunit.client.0.smithi088.stdout: create 131/32 file after the snapshot 2023-10-16T20:41:02.634 INFO:tasks.workunit.client.0.smithi088.stdout: create 131/33 file after the snapshot 2023-10-16T20:41:02.637 INFO:tasks.workunit.client.0.smithi088.stdout: create 131/34 file after the snapshot 2023-10-16T20:41:02.639 INFO:tasks.workunit.client.0.smithi088.stdout: create 131/35 file after the snapshot 2023-10-16T20:41:02.641 INFO:tasks.workunit.client.0.smithi088.stdout: create 131/36 file after the snapshot 2023-10-16T20:41:02.644 INFO:tasks.workunit.client.0.smithi088.stdout: create 131/37 file after the snapshot 2023-10-16T20:41:02.647 INFO:tasks.workunit.client.0.smithi088.stdout: create 131/38 file after the snapshot 2023-10-16T20:41:02.649 INFO:tasks.workunit.client.0.smithi088.stdout: create 131/39 file after the snapshot 2023-10-16T20:41:02.651 INFO:tasks.workunit.client.0.smithi088.stdout: create 132/.snap/snap-subdir-test 2023-10-16T20:41:02.675 INFO:tasks.workunit.client.0.smithi088.stdout: create 132/30 file after the snapshot 2023-10-16T20:41:02.677 INFO:tasks.workunit.client.0.smithi088.stdout: create 132/31 file after the snapshot 2023-10-16T20:41:02.680 INFO:tasks.workunit.client.0.smithi088.stdout: create 132/32 file after the snapshot 2023-10-16T20:41:02.683 INFO:tasks.workunit.client.0.smithi088.stdout: create 132/33 file after the snapshot 2023-10-16T20:41:02.685 INFO:tasks.workunit.client.0.smithi088.stdout: create 132/34 file after the snapshot 2023-10-16T20:41:02.688 INFO:tasks.workunit.client.0.smithi088.stdout: create 132/35 file after the snapshot 2023-10-16T20:41:02.691 INFO:tasks.workunit.client.0.smithi088.stdout: create 132/36 file after the snapshot 2023-10-16T20:41:02.693 INFO:tasks.workunit.client.0.smithi088.stdout: create 132/37 file after the snapshot 2023-10-16T20:41:02.696 INFO:tasks.workunit.client.0.smithi088.stdout: create 132/38 file after the snapshot 2023-10-16T20:41:02.699 INFO:tasks.workunit.client.0.smithi088.stdout: create 132/39 file after the snapshot 2023-10-16T20:41:02.701 INFO:tasks.workunit.client.0.smithi088.stdout: create 133/.snap/snap-subdir-test 2023-10-16T20:41:02.723 INFO:tasks.workunit.client.0.smithi088.stdout: create 133/30 file after the snapshot 2023-10-16T20:41:02.725 INFO:tasks.workunit.client.0.smithi088.stdout: create 133/31 file after the snapshot 2023-10-16T20:41:02.728 INFO:tasks.workunit.client.0.smithi088.stdout: create 133/32 file after the snapshot 2023-10-16T20:41:02.730 INFO:tasks.workunit.client.0.smithi088.stdout: create 133/33 file after the snapshot 2023-10-16T20:41:02.733 INFO:tasks.workunit.client.0.smithi088.stdout: create 133/34 file after the snapshot 2023-10-16T20:41:02.735 INFO:tasks.workunit.client.0.smithi088.stdout: create 133/35 file after the snapshot 2023-10-16T20:41:02.738 INFO:tasks.workunit.client.0.smithi088.stdout: create 133/36 file after the snapshot 2023-10-16T20:41:02.741 INFO:tasks.workunit.client.0.smithi088.stdout: create 133/37 file after the snapshot 2023-10-16T20:41:02.743 INFO:tasks.workunit.client.0.smithi088.stdout: create 133/38 file after the snapshot 2023-10-16T20:41:02.745 INFO:tasks.workunit.client.0.smithi088.stdout: create 133/39 file after the snapshot 2023-10-16T20:41:02.748 INFO:tasks.workunit.client.0.smithi088.stdout: create 134/.snap/snap-subdir-test 2023-10-16T20:41:02.771 INFO:tasks.workunit.client.0.smithi088.stdout: create 134/30 file after the snapshot 2023-10-16T20:41:02.773 INFO:tasks.workunit.client.0.smithi088.stdout: create 134/31 file after the snapshot 2023-10-16T20:41:02.775 INFO:tasks.workunit.client.0.smithi088.stdout: create 134/32 file after the snapshot 2023-10-16T20:41:02.778 INFO:tasks.workunit.client.0.smithi088.stdout: create 134/33 file after the snapshot 2023-10-16T20:41:02.780 INFO:tasks.workunit.client.0.smithi088.stdout: create 134/34 file after the snapshot 2023-10-16T20:41:02.783 INFO:tasks.workunit.client.0.smithi088.stdout: create 134/35 file after the snapshot 2023-10-16T20:41:02.786 INFO:tasks.workunit.client.0.smithi088.stdout: create 134/36 file after the snapshot 2023-10-16T20:41:02.788 INFO:tasks.workunit.client.0.smithi088.stdout: create 134/37 file after the snapshot 2023-10-16T20:41:02.791 INFO:tasks.workunit.client.0.smithi088.stdout: create 134/38 file after the snapshot 2023-10-16T20:41:02.793 INFO:tasks.workunit.client.0.smithi088.stdout: create 134/39 file after the snapshot 2023-10-16T20:41:02.796 INFO:tasks.workunit.client.0.smithi088.stdout: create 135/.snap/snap-subdir-test 2023-10-16T20:41:02.819 INFO:tasks.workunit.client.0.smithi088.stdout: create 135/30 file after the snapshot 2023-10-16T20:41:02.821 INFO:tasks.workunit.client.0.smithi088.stdout: create 135/31 file after the snapshot 2023-10-16T20:41:02.824 INFO:tasks.workunit.client.0.smithi088.stdout: create 135/32 file after the snapshot 2023-10-16T20:41:02.826 INFO:tasks.workunit.client.0.smithi088.stdout: create 135/33 file after the snapshot 2023-10-16T20:41:02.829 INFO:tasks.workunit.client.0.smithi088.stdout: create 135/34 file after the snapshot 2023-10-16T20:41:02.831 INFO:tasks.workunit.client.0.smithi088.stdout: create 135/35 file after the snapshot 2023-10-16T20:41:02.834 INFO:tasks.workunit.client.0.smithi088.stdout: create 135/36 file after the snapshot 2023-10-16T20:41:02.836 INFO:tasks.workunit.client.0.smithi088.stdout: create 135/37 file after the snapshot 2023-10-16T20:41:02.839 INFO:tasks.workunit.client.0.smithi088.stdout: create 135/38 file after the snapshot 2023-10-16T20:41:02.842 INFO:tasks.workunit.client.0.smithi088.stdout: create 135/39 file after the snapshot 2023-10-16T20:41:02.844 INFO:tasks.workunit.client.0.smithi088.stdout: create 136/.snap/snap-subdir-test 2023-10-16T20:41:02.867 INFO:tasks.workunit.client.0.smithi088.stdout: create 136/30 file after the snapshot 2023-10-16T20:41:02.870 INFO:tasks.workunit.client.0.smithi088.stdout: create 136/31 file after the snapshot 2023-10-16T20:41:02.872 INFO:tasks.workunit.client.0.smithi088.stdout: create 136/32 file after the snapshot 2023-10-16T20:41:02.874 INFO:tasks.workunit.client.0.smithi088.stdout: create 136/33 file after the snapshot 2023-10-16T20:41:02.876 INFO:tasks.workunit.client.0.smithi088.stdout: create 136/34 file after the snapshot 2023-10-16T20:41:02.879 INFO:tasks.workunit.client.0.smithi088.stdout: create 136/35 file after the snapshot 2023-10-16T20:41:02.881 INFO:tasks.workunit.client.0.smithi088.stdout: create 136/36 file after the snapshot 2023-10-16T20:41:02.884 INFO:tasks.workunit.client.0.smithi088.stdout: create 136/37 file after the snapshot 2023-10-16T20:41:02.887 INFO:tasks.workunit.client.0.smithi088.stdout: create 136/38 file after the snapshot 2023-10-16T20:41:02.889 INFO:tasks.workunit.client.0.smithi088.stdout: create 136/39 file after the snapshot 2023-10-16T20:41:02.891 INFO:tasks.workunit.client.0.smithi088.stdout: create 137/.snap/snap-subdir-test 2023-10-16T20:41:02.915 INFO:tasks.workunit.client.0.smithi088.stdout: create 137/30 file after the snapshot 2023-10-16T20:41:02.917 INFO:tasks.workunit.client.0.smithi088.stdout: create 137/31 file after the snapshot 2023-10-16T20:41:02.919 INFO:tasks.workunit.client.0.smithi088.stdout: create 137/32 file after the snapshot 2023-10-16T20:41:02.921 INFO:tasks.workunit.client.0.smithi088.stdout: create 137/33 file after the snapshot 2023-10-16T20:41:02.924 INFO:tasks.workunit.client.0.smithi088.stdout: create 137/34 file after the snapshot 2023-10-16T20:41:02.926 INFO:tasks.workunit.client.0.smithi088.stdout: create 137/35 file after the snapshot 2023-10-16T20:41:02.929 INFO:tasks.workunit.client.0.smithi088.stdout: create 137/36 file after the snapshot 2023-10-16T20:41:02.932 INFO:tasks.workunit.client.0.smithi088.stdout: create 137/37 file after the snapshot 2023-10-16T20:41:02.934 INFO:tasks.workunit.client.0.smithi088.stdout: create 137/38 file after the snapshot 2023-10-16T20:41:02.936 INFO:tasks.workunit.client.0.smithi088.stdout: create 137/39 file after the snapshot 2023-10-16T20:41:02.939 INFO:tasks.workunit.client.0.smithi088.stdout: create 138/.snap/snap-subdir-test 2023-10-16T20:41:02.961 INFO:tasks.workunit.client.0.smithi088.stdout: create 138/30 file after the snapshot 2023-10-16T20:41:02.963 INFO:tasks.workunit.client.0.smithi088.stdout: create 138/31 file after the snapshot 2023-10-16T20:41:02.966 INFO:tasks.workunit.client.0.smithi088.stdout: create 138/32 file after the snapshot 2023-10-16T20:41:02.968 INFO:tasks.workunit.client.0.smithi088.stdout: create 138/33 file after the snapshot 2023-10-16T20:41:02.971 INFO:tasks.workunit.client.0.smithi088.stdout: create 138/34 file after the snapshot 2023-10-16T20:41:02.974 INFO:tasks.workunit.client.0.smithi088.stdout: create 138/35 file after the snapshot 2023-10-16T20:41:02.976 INFO:tasks.workunit.client.0.smithi088.stdout: create 138/36 file after the snapshot 2023-10-16T20:41:02.979 INFO:tasks.workunit.client.0.smithi088.stdout: create 138/37 file after the snapshot 2023-10-16T20:41:02.981 INFO:tasks.workunit.client.0.smithi088.stdout: create 138/38 file after the snapshot 2023-10-16T20:41:02.983 INFO:tasks.workunit.client.0.smithi088.stdout: create 138/39 file after the snapshot 2023-10-16T20:41:02.986 INFO:tasks.workunit.client.0.smithi088.stdout: create 139/.snap/snap-subdir-test 2023-10-16T20:41:03.010 INFO:tasks.workunit.client.0.smithi088.stdout: create 139/30 file after the snapshot 2023-10-16T20:41:03.013 INFO:tasks.workunit.client.0.smithi088.stdout: create 139/31 file after the snapshot 2023-10-16T20:41:03.015 INFO:tasks.workunit.client.0.smithi088.stdout: create 139/32 file after the snapshot 2023-10-16T20:41:03.018 INFO:tasks.workunit.client.0.smithi088.stdout: create 139/33 file after the snapshot 2023-10-16T20:41:03.021 INFO:tasks.workunit.client.0.smithi088.stdout: create 139/34 file after the snapshot 2023-10-16T20:41:03.022 INFO:tasks.workunit.client.0.smithi088.stdout: create 139/35 file after the snapshot 2023-10-16T20:41:03.025 INFO:tasks.workunit.client.0.smithi088.stdout: create 139/36 file after the snapshot 2023-10-16T20:41:03.027 INFO:tasks.workunit.client.0.smithi088.stdout: create 139/37 file after the snapshot 2023-10-16T20:41:03.030 INFO:tasks.workunit.client.0.smithi088.stdout: create 139/38 file after the snapshot 2023-10-16T20:41:03.032 INFO:tasks.workunit.client.0.smithi088.stdout: create 139/39 file after the snapshot 2023-10-16T20:41:03.035 INFO:tasks.workunit.client.0.smithi088.stdout: create 140/.snap/snap-subdir-test 2023-10-16T20:41:03.057 INFO:tasks.workunit.client.0.smithi088.stdout: create 140/30 file after the snapshot 2023-10-16T20:41:03.060 INFO:tasks.workunit.client.0.smithi088.stdout: create 140/31 file after the snapshot 2023-10-16T20:41:03.062 INFO:tasks.workunit.client.0.smithi088.stdout: create 140/32 file after the snapshot 2023-10-16T20:41:03.066 INFO:tasks.workunit.client.0.smithi088.stdout: create 140/33 file after the snapshot 2023-10-16T20:41:03.067 INFO:tasks.workunit.client.0.smithi088.stdout: create 140/34 file after the snapshot 2023-10-16T20:41:03.069 INFO:tasks.workunit.client.0.smithi088.stdout: create 140/35 file after the snapshot 2023-10-16T20:41:03.072 INFO:tasks.workunit.client.0.smithi088.stdout: create 140/36 file after the snapshot 2023-10-16T20:41:03.074 INFO:tasks.workunit.client.0.smithi088.stdout: create 140/37 file after the snapshot 2023-10-16T20:41:03.077 INFO:tasks.workunit.client.0.smithi088.stdout: create 140/38 file after the snapshot 2023-10-16T20:41:03.079 INFO:tasks.workunit.client.0.smithi088.stdout: create 140/39 file after the snapshot 2023-10-16T20:41:03.082 INFO:tasks.workunit.client.0.smithi088.stdout: create 141/.snap/snap-subdir-test 2023-10-16T20:41:03.104 INFO:tasks.workunit.client.0.smithi088.stdout: create 141/30 file after the snapshot 2023-10-16T20:41:03.106 INFO:tasks.workunit.client.0.smithi088.stdout: create 141/31 file after the snapshot 2023-10-16T20:41:03.108 INFO:tasks.workunit.client.0.smithi088.stdout: create 141/32 file after the snapshot 2023-10-16T20:41:03.111 INFO:tasks.workunit.client.0.smithi088.stdout: create 141/33 file after the snapshot 2023-10-16T20:41:03.113 INFO:tasks.workunit.client.0.smithi088.stdout: create 141/34 file after the snapshot 2023-10-16T20:41:03.116 INFO:tasks.workunit.client.0.smithi088.stdout: create 141/35 file after the snapshot 2023-10-16T20:41:03.118 INFO:tasks.workunit.client.0.smithi088.stdout: create 141/36 file after the snapshot 2023-10-16T20:41:03.121 INFO:tasks.workunit.client.0.smithi088.stdout: create 141/37 file after the snapshot 2023-10-16T20:41:03.123 INFO:tasks.workunit.client.0.smithi088.stdout: create 141/38 file after the snapshot 2023-10-16T20:41:03.126 INFO:tasks.workunit.client.0.smithi088.stdout: create 141/39 file after the snapshot 2023-10-16T20:41:03.128 INFO:tasks.workunit.client.0.smithi088.stdout: create 142/.snap/snap-subdir-test 2023-10-16T20:41:03.158 INFO:tasks.workunit.client.0.smithi088.stdout: create 142/30 file after the snapshot 2023-10-16T20:41:03.161 INFO:tasks.workunit.client.0.smithi088.stdout: create 142/31 file after the snapshot 2023-10-16T20:41:03.163 INFO:tasks.workunit.client.0.smithi088.stdout: create 142/32 file after the snapshot 2023-10-16T20:41:03.166 INFO:tasks.workunit.client.0.smithi088.stdout: create 142/33 file after the snapshot 2023-10-16T20:41:03.168 INFO:tasks.workunit.client.0.smithi088.stdout: create 142/34 file after the snapshot 2023-10-16T20:41:03.171 INFO:tasks.workunit.client.0.smithi088.stdout: create 142/35 file after the snapshot 2023-10-16T20:41:03.173 INFO:tasks.workunit.client.0.smithi088.stdout: create 142/36 file after the snapshot 2023-10-16T20:41:03.175 INFO:tasks.workunit.client.0.smithi088.stdout: create 142/37 file after the snapshot 2023-10-16T20:41:03.178 INFO:tasks.workunit.client.0.smithi088.stdout: create 142/38 file after the snapshot 2023-10-16T20:41:03.180 INFO:tasks.workunit.client.0.smithi088.stdout: create 142/39 file after the snapshot 2023-10-16T20:41:03.183 INFO:tasks.workunit.client.0.smithi088.stdout: create 143/.snap/snap-subdir-test 2023-10-16T20:41:03.205 INFO:tasks.workunit.client.0.smithi088.stdout: create 143/30 file after the snapshot 2023-10-16T20:41:03.206 INFO:tasks.workunit.client.0.smithi088.stdout: create 143/31 file after the snapshot 2023-10-16T20:41:03.208 INFO:tasks.workunit.client.0.smithi088.stdout: create 143/32 file after the snapshot 2023-10-16T20:41:03.212 INFO:tasks.workunit.client.0.smithi088.stdout: create 143/33 file after the snapshot 2023-10-16T20:41:03.214 INFO:tasks.workunit.client.0.smithi088.stdout: create 143/34 file after the snapshot 2023-10-16T20:41:03.216 INFO:tasks.workunit.client.0.smithi088.stdout: create 143/35 file after the snapshot 2023-10-16T20:41:03.219 INFO:tasks.workunit.client.0.smithi088.stdout: create 143/36 file after the snapshot 2023-10-16T20:41:03.221 INFO:tasks.workunit.client.0.smithi088.stdout: create 143/37 file after the snapshot 2023-10-16T20:41:03.224 INFO:tasks.workunit.client.0.smithi088.stdout: create 143/38 file after the snapshot 2023-10-16T20:41:03.226 INFO:tasks.workunit.client.0.smithi088.stdout: create 143/39 file after the snapshot 2023-10-16T20:41:03.880 INFO:tasks.workunit.client.0.smithi088.stdout: create 144/.snap/snap-subdir-test 2023-10-16T20:41:03.958 INFO:tasks.workunit.client.0.smithi088.stdout: create 144/30 file after the snapshot 2023-10-16T20:41:03.960 INFO:tasks.workunit.client.0.smithi088.stdout: create 144/31 file after the snapshot 2023-10-16T20:41:03.963 INFO:tasks.workunit.client.0.smithi088.stdout: create 144/32 file after the snapshot 2023-10-16T20:41:03.965 INFO:tasks.workunit.client.0.smithi088.stdout: create 144/33 file after the snapshot 2023-10-16T20:41:03.968 INFO:tasks.workunit.client.0.smithi088.stdout: create 144/34 file after the snapshot 2023-10-16T20:41:03.970 INFO:tasks.workunit.client.0.smithi088.stdout: create 144/35 file after the snapshot 2023-10-16T20:41:03.973 INFO:tasks.workunit.client.0.smithi088.stdout: create 144/36 file after the snapshot 2023-10-16T20:41:03.975 INFO:tasks.workunit.client.0.smithi088.stdout: create 144/37 file after the snapshot 2023-10-16T20:41:03.978 INFO:tasks.workunit.client.0.smithi088.stdout: create 144/38 file after the snapshot 2023-10-16T20:41:03.980 INFO:tasks.workunit.client.0.smithi088.stdout: create 144/39 file after the snapshot 2023-10-16T20:41:03.983 INFO:tasks.workunit.client.0.smithi088.stdout: create 145/.snap/snap-subdir-test 2023-10-16T20:41:04.991 INFO:tasks.workunit.client.0.smithi088.stdout: create 145/30 file after the snapshot 2023-10-16T20:41:04.993 INFO:tasks.workunit.client.0.smithi088.stdout: create 145/31 file after the snapshot 2023-10-16T20:41:04.998 INFO:tasks.workunit.client.0.smithi088.stdout: create 145/32 file after the snapshot 2023-10-16T20:41:04.999 INFO:tasks.workunit.client.0.smithi088.stdout: create 145/33 file after the snapshot 2023-10-16T20:41:05.001 INFO:tasks.workunit.client.0.smithi088.stdout: create 145/34 file after the snapshot 2023-10-16T20:41:05.003 INFO:tasks.workunit.client.0.smithi088.stdout: create 145/35 file after the snapshot 2023-10-16T20:41:05.006 INFO:tasks.workunit.client.0.smithi088.stdout: create 145/36 file after the snapshot 2023-10-16T20:41:05.009 INFO:tasks.workunit.client.0.smithi088.stdout: create 145/37 file after the snapshot 2023-10-16T20:41:05.012 INFO:tasks.workunit.client.0.smithi088.stdout: create 145/38 file after the snapshot 2023-10-16T20:41:05.015 INFO:tasks.workunit.client.0.smithi088.stdout: create 145/39 file after the snapshot 2023-10-16T20:41:05.018 INFO:tasks.workunit.client.0.smithi088.stdout: create 146/.snap/snap-subdir-test 2023-10-16T20:41:05.040 INFO:tasks.workunit.client.0.smithi088.stdout: create 146/30 file after the snapshot 2023-10-16T20:41:05.042 INFO:tasks.workunit.client.0.smithi088.stdout: create 146/31 file after the snapshot 2023-10-16T20:41:05.045 INFO:tasks.workunit.client.0.smithi088.stdout: create 146/32 file after the snapshot 2023-10-16T20:41:05.047 INFO:tasks.workunit.client.0.smithi088.stdout: create 146/33 file after the snapshot 2023-10-16T20:41:05.050 INFO:tasks.workunit.client.0.smithi088.stdout: create 146/34 file after the snapshot 2023-10-16T20:41:05.052 INFO:tasks.workunit.client.0.smithi088.stdout: create 146/35 file after the snapshot 2023-10-16T20:41:05.055 INFO:tasks.workunit.client.0.smithi088.stdout: create 146/36 file after the snapshot 2023-10-16T20:41:05.057 INFO:tasks.workunit.client.0.smithi088.stdout: create 146/37 file after the snapshot 2023-10-16T20:41:05.060 INFO:tasks.workunit.client.0.smithi088.stdout: create 146/38 file after the snapshot 2023-10-16T20:41:05.062 INFO:tasks.workunit.client.0.smithi088.stdout: create 146/39 file after the snapshot 2023-10-16T20:41:05.066 INFO:tasks.workunit.client.0.smithi088.stdout: create 147/.snap/snap-subdir-test 2023-10-16T20:41:05.097 INFO:tasks.workunit.client.0.smithi088.stdout: create 147/30 file after the snapshot 2023-10-16T20:41:05.099 INFO:tasks.workunit.client.0.smithi088.stdout: create 147/31 file after the snapshot 2023-10-16T20:41:05.102 INFO:tasks.workunit.client.0.smithi088.stdout: create 147/32 file after the snapshot 2023-10-16T20:41:05.104 INFO:tasks.workunit.client.0.smithi088.stdout: create 147/33 file after the snapshot 2023-10-16T20:41:05.106 INFO:tasks.workunit.client.0.smithi088.stdout: create 147/34 file after the snapshot 2023-10-16T20:41:05.109 INFO:tasks.workunit.client.0.smithi088.stdout: create 147/35 file after the snapshot 2023-10-16T20:41:05.111 INFO:tasks.workunit.client.0.smithi088.stdout: create 147/36 file after the snapshot 2023-10-16T20:41:05.113 INFO:tasks.workunit.client.0.smithi088.stdout: create 147/37 file after the snapshot 2023-10-16T20:41:05.116 INFO:tasks.workunit.client.0.smithi088.stdout: create 147/38 file after the snapshot 2023-10-16T20:41:05.118 INFO:tasks.workunit.client.0.smithi088.stdout: create 147/39 file after the snapshot 2023-10-16T20:41:05.121 INFO:tasks.workunit.client.0.smithi088.stdout: create 148/.snap/snap-subdir-test 2023-10-16T20:41:05.144 INFO:tasks.workunit.client.0.smithi088.stdout: create 148/30 file after the snapshot 2023-10-16T20:41:05.146 INFO:tasks.workunit.client.0.smithi088.stdout: create 148/31 file after the snapshot 2023-10-16T20:41:05.148 INFO:tasks.workunit.client.0.smithi088.stdout: create 148/32 file after the snapshot 2023-10-16T20:41:05.150 INFO:tasks.workunit.client.0.smithi088.stdout: create 148/33 file after the snapshot 2023-10-16T20:41:05.153 INFO:tasks.workunit.client.0.smithi088.stdout: create 148/34 file after the snapshot 2023-10-16T20:41:05.156 INFO:tasks.workunit.client.0.smithi088.stdout: create 148/35 file after the snapshot 2023-10-16T20:41:05.158 INFO:tasks.workunit.client.0.smithi088.stdout: create 148/36 file after the snapshot 2023-10-16T20:41:05.161 INFO:tasks.workunit.client.0.smithi088.stdout: create 148/37 file after the snapshot 2023-10-16T20:41:05.163 INFO:tasks.workunit.client.0.smithi088.stdout: create 148/38 file after the snapshot 2023-10-16T20:41:05.165 INFO:tasks.workunit.client.0.smithi088.stdout: create 148/39 file after the snapshot 2023-10-16T20:41:05.168 INFO:tasks.workunit.client.0.smithi088.stdout: create 149/.snap/snap-subdir-test 2023-10-16T20:41:05.191 INFO:tasks.workunit.client.0.smithi088.stdout: create 149/30 file after the snapshot 2023-10-16T20:41:05.193 INFO:tasks.workunit.client.0.smithi088.stdout: create 149/31 file after the snapshot 2023-10-16T20:41:05.195 INFO:tasks.workunit.client.0.smithi088.stdout: create 149/32 file after the snapshot 2023-10-16T20:41:05.198 INFO:tasks.workunit.client.0.smithi088.stdout: create 149/33 file after the snapshot 2023-10-16T20:41:05.201 INFO:tasks.workunit.client.0.smithi088.stdout: create 149/34 file after the snapshot 2023-10-16T20:41:05.203 INFO:tasks.workunit.client.0.smithi088.stdout: create 149/35 file after the snapshot 2023-10-16T20:41:05.205 INFO:tasks.workunit.client.0.smithi088.stdout: create 149/36 file after the snapshot 2023-10-16T20:41:05.208 INFO:tasks.workunit.client.0.smithi088.stdout: create 149/37 file after the snapshot 2023-10-16T20:41:05.210 INFO:tasks.workunit.client.0.smithi088.stdout: create 149/38 file after the snapshot 2023-10-16T20:41:05.213 INFO:tasks.workunit.client.0.smithi088.stdout: create 149/39 file after the snapshot 2023-10-16T20:41:05.216 INFO:tasks.workunit.client.0.smithi088.stdout: create 150/.snap/snap-subdir-test 2023-10-16T20:41:05.237 INFO:tasks.workunit.client.0.smithi088.stdout: create 150/30 file after the snapshot 2023-10-16T20:41:05.239 INFO:tasks.workunit.client.0.smithi088.stdout: create 150/31 file after the snapshot 2023-10-16T20:41:05.241 INFO:tasks.workunit.client.0.smithi088.stdout: create 150/32 file after the snapshot 2023-10-16T20:41:05.244 INFO:tasks.workunit.client.0.smithi088.stdout: create 150/33 file after the snapshot 2023-10-16T20:41:05.247 INFO:tasks.workunit.client.0.smithi088.stdout: create 150/34 file after the snapshot 2023-10-16T20:41:05.249 INFO:tasks.workunit.client.0.smithi088.stdout: create 150/35 file after the snapshot 2023-10-16T20:41:05.251 INFO:tasks.workunit.client.0.smithi088.stdout: create 150/36 file after the snapshot 2023-10-16T20:41:05.254 INFO:tasks.workunit.client.0.smithi088.stdout: create 150/37 file after the snapshot 2023-10-16T20:41:05.260 INFO:tasks.workunit.client.0.smithi088.stdout: create 150/38 file after the snapshot 2023-10-16T20:41:05.275 INFO:tasks.workunit.client.0.smithi088.stdout: create 150/39 file after the snapshot 2023-10-16T20:41:05.277 INFO:tasks.workunit.client.0.smithi088.stdout: create 151/.snap/snap-subdir-test 2023-10-16T20:41:05.307 INFO:tasks.workunit.client.0.smithi088.stdout: create 151/30 file after the snapshot 2023-10-16T20:41:05.310 INFO:tasks.workunit.client.0.smithi088.stdout: create 151/31 file after the snapshot 2023-10-16T20:41:05.312 INFO:tasks.workunit.client.0.smithi088.stdout: create 151/32 file after the snapshot 2023-10-16T20:41:05.315 INFO:tasks.workunit.client.0.smithi088.stdout: create 151/33 file after the snapshot 2023-10-16T20:41:05.317 INFO:tasks.workunit.client.0.smithi088.stdout: create 151/34 file after the snapshot 2023-10-16T20:41:05.320 INFO:tasks.workunit.client.0.smithi088.stdout: create 151/35 file after the snapshot 2023-10-16T20:41:05.322 INFO:tasks.workunit.client.0.smithi088.stdout: create 151/36 file after the snapshot 2023-10-16T20:41:05.325 INFO:tasks.workunit.client.0.smithi088.stdout: create 151/37 file after the snapshot 2023-10-16T20:41:05.328 INFO:tasks.workunit.client.0.smithi088.stdout: create 151/38 file after the snapshot 2023-10-16T20:41:05.330 INFO:tasks.workunit.client.0.smithi088.stdout: create 151/39 file after the snapshot 2023-10-16T20:41:05.333 INFO:tasks.workunit.client.0.smithi088.stdout: create 152/.snap/snap-subdir-test 2023-10-16T20:41:05.361 INFO:tasks.workunit.client.0.smithi088.stdout: create 152/30 file after the snapshot 2023-10-16T20:41:05.363 INFO:tasks.workunit.client.0.smithi088.stdout: create 152/31 file after the snapshot 2023-10-16T20:41:05.366 INFO:tasks.workunit.client.0.smithi088.stdout: create 152/32 file after the snapshot 2023-10-16T20:41:05.368 INFO:tasks.workunit.client.0.smithi088.stdout: create 152/33 file after the snapshot 2023-10-16T20:41:05.371 INFO:tasks.workunit.client.0.smithi088.stdout: create 152/34 file after the snapshot 2023-10-16T20:41:05.373 INFO:tasks.workunit.client.0.smithi088.stdout: create 152/35 file after the snapshot 2023-10-16T20:41:05.376 INFO:tasks.workunit.client.0.smithi088.stdout: create 152/36 file after the snapshot 2023-10-16T20:41:05.378 INFO:tasks.workunit.client.0.smithi088.stdout: create 152/37 file after the snapshot 2023-10-16T20:41:05.380 INFO:tasks.workunit.client.0.smithi088.stdout: create 152/38 file after the snapshot 2023-10-16T20:41:05.383 INFO:tasks.workunit.client.0.smithi088.stdout: create 152/39 file after the snapshot 2023-10-16T20:41:05.385 INFO:tasks.workunit.client.0.smithi088.stdout: create 153/.snap/snap-subdir-test 2023-10-16T20:41:05.408 INFO:tasks.workunit.client.0.smithi088.stdout: create 153/30 file after the snapshot 2023-10-16T20:41:05.410 INFO:tasks.workunit.client.0.smithi088.stdout: create 153/31 file after the snapshot 2023-10-16T20:41:05.412 INFO:tasks.workunit.client.0.smithi088.stdout: create 153/32 file after the snapshot 2023-10-16T20:41:05.414 INFO:tasks.workunit.client.0.smithi088.stdout: create 153/33 file after the snapshot 2023-10-16T20:41:05.417 INFO:tasks.workunit.client.0.smithi088.stdout: create 153/34 file after the snapshot 2023-10-16T20:41:05.420 INFO:tasks.workunit.client.0.smithi088.stdout: create 153/35 file after the snapshot 2023-10-16T20:41:05.422 INFO:tasks.workunit.client.0.smithi088.stdout: create 153/36 file after the snapshot 2023-10-16T20:41:05.425 INFO:tasks.workunit.client.0.smithi088.stdout: create 153/37 file after the snapshot 2023-10-16T20:41:05.427 INFO:tasks.workunit.client.0.smithi088.stdout: create 153/38 file after the snapshot 2023-10-16T20:41:05.430 INFO:tasks.workunit.client.0.smithi088.stdout: create 153/39 file after the snapshot 2023-10-16T20:41:05.432 INFO:tasks.workunit.client.0.smithi088.stdout: create 154/.snap/snap-subdir-test 2023-10-16T20:41:05.463 INFO:tasks.workunit.client.0.smithi088.stdout: create 154/30 file after the snapshot 2023-10-16T20:41:05.465 INFO:tasks.workunit.client.0.smithi088.stdout: create 154/31 file after the snapshot 2023-10-16T20:41:05.468 INFO:tasks.workunit.client.0.smithi088.stdout: create 154/32 file after the snapshot 2023-10-16T20:41:05.470 INFO:tasks.workunit.client.0.smithi088.stdout: create 154/33 file after the snapshot 2023-10-16T20:41:05.473 INFO:tasks.workunit.client.0.smithi088.stdout: create 154/34 file after the snapshot 2023-10-16T20:41:05.476 INFO:tasks.workunit.client.0.smithi088.stdout: create 154/35 file after the snapshot 2023-10-16T20:41:05.478 INFO:tasks.workunit.client.0.smithi088.stdout: create 154/36 file after the snapshot 2023-10-16T20:41:05.480 INFO:tasks.workunit.client.0.smithi088.stdout: create 154/37 file after the snapshot 2023-10-16T20:41:05.483 INFO:tasks.workunit.client.0.smithi088.stdout: create 154/38 file after the snapshot 2023-10-16T20:41:05.486 INFO:tasks.workunit.client.0.smithi088.stdout: create 154/39 file after the snapshot 2023-10-16T20:41:05.488 INFO:tasks.workunit.client.0.smithi088.stdout: create 155/.snap/snap-subdir-test 2023-10-16T20:41:05.897 INFO:tasks.workunit.client.0.smithi088.stdout: create 155/30 file after the snapshot 2023-10-16T20:41:05.899 INFO:tasks.workunit.client.0.smithi088.stdout: create 155/31 file after the snapshot 2023-10-16T20:41:05.901 INFO:tasks.workunit.client.0.smithi088.stdout: create 155/32 file after the snapshot 2023-10-16T20:41:05.904 INFO:tasks.workunit.client.0.smithi088.stdout: create 155/33 file after the snapshot 2023-10-16T20:41:05.906 INFO:tasks.workunit.client.0.smithi088.stdout: create 155/34 file after the snapshot 2023-10-16T20:41:05.909 INFO:tasks.workunit.client.0.smithi088.stdout: create 155/35 file after the snapshot 2023-10-16T20:41:05.911 INFO:tasks.workunit.client.0.smithi088.stdout: create 155/36 file after the snapshot 2023-10-16T20:41:05.913 INFO:tasks.workunit.client.0.smithi088.stdout: create 155/37 file after the snapshot 2023-10-16T20:41:05.916 INFO:tasks.workunit.client.0.smithi088.stdout: create 155/38 file after the snapshot 2023-10-16T20:41:05.919 INFO:tasks.workunit.client.0.smithi088.stdout: create 155/39 file after the snapshot 2023-10-16T20:41:05.921 INFO:tasks.workunit.client.0.smithi088.stdout: create 156/.snap/snap-subdir-test 2023-10-16T20:41:06.838 INFO:tasks.workunit.client.0.smithi088.stdout: create 156/30 file after the snapshot 2023-10-16T20:41:06.840 INFO:tasks.workunit.client.0.smithi088.stdout: create 156/31 file after the snapshot 2023-10-16T20:41:06.843 INFO:tasks.workunit.client.0.smithi088.stdout: create 156/32 file after the snapshot 2023-10-16T20:41:06.846 INFO:tasks.workunit.client.0.smithi088.stdout: create 156/33 file after the snapshot 2023-10-16T20:41:06.849 INFO:tasks.workunit.client.0.smithi088.stdout: create 156/34 file after the snapshot 2023-10-16T20:41:06.850 INFO:tasks.workunit.client.0.smithi088.stdout: create 156/35 file after the snapshot 2023-10-16T20:41:06.854 INFO:tasks.workunit.client.0.smithi088.stdout: create 156/36 file after the snapshot 2023-10-16T20:41:06.856 INFO:tasks.workunit.client.0.smithi088.stdout: create 156/37 file after the snapshot 2023-10-16T20:41:06.859 INFO:tasks.workunit.client.0.smithi088.stdout: create 156/38 file after the snapshot 2023-10-16T20:41:06.861 INFO:tasks.workunit.client.0.smithi088.stdout: create 156/39 file after the snapshot 2023-10-16T20:41:06.864 INFO:tasks.workunit.client.0.smithi088.stdout: create 157/.snap/snap-subdir-test 2023-10-16T20:41:06.888 INFO:tasks.workunit.client.0.smithi088.stdout: create 157/30 file after the snapshot 2023-10-16T20:41:06.890 INFO:tasks.workunit.client.0.smithi088.stdout: create 157/31 file after the snapshot 2023-10-16T20:41:06.892 INFO:tasks.workunit.client.0.smithi088.stdout: create 157/32 file after the snapshot 2023-10-16T20:41:06.894 INFO:tasks.workunit.client.0.smithi088.stdout: create 157/33 file after the snapshot 2023-10-16T20:41:06.897 INFO:tasks.workunit.client.0.smithi088.stdout: create 157/34 file after the snapshot 2023-10-16T20:41:07.272 INFO:tasks.workunit.client.0.smithi088.stdout: create 157/35 file after the snapshot 2023-10-16T20:41:08.131 INFO:tasks.workunit.client.0.smithi088.stdout: create 157/36 file after the snapshot 2023-10-16T20:41:08.134 INFO:tasks.workunit.client.0.smithi088.stdout: create 157/37 file after the snapshot 2023-10-16T20:41:08.136 INFO:tasks.workunit.client.0.smithi088.stdout: create 157/38 file after the snapshot 2023-10-16T20:41:08.139 INFO:tasks.workunit.client.0.smithi088.stdout: create 157/39 file after the snapshot 2023-10-16T20:41:08.143 INFO:tasks.workunit.client.0.smithi088.stdout: create 158/.snap/snap-subdir-test 2023-10-16T20:41:08.167 INFO:tasks.workunit.client.0.smithi088.stdout: create 158/30 file after the snapshot 2023-10-16T20:41:08.169 INFO:tasks.workunit.client.0.smithi088.stdout: create 158/31 file after the snapshot 2023-10-16T20:41:08.171 INFO:tasks.workunit.client.0.smithi088.stdout: create 158/32 file after the snapshot 2023-10-16T20:41:08.173 INFO:tasks.workunit.client.0.smithi088.stdout: create 158/33 file after the snapshot 2023-10-16T20:41:08.176 INFO:tasks.workunit.client.0.smithi088.stdout: create 158/34 file after the snapshot 2023-10-16T20:41:08.179 INFO:tasks.workunit.client.0.smithi088.stdout: create 158/35 file after the snapshot 2023-10-16T20:41:08.181 INFO:tasks.workunit.client.0.smithi088.stdout: create 158/36 file after the snapshot 2023-10-16T20:41:08.184 INFO:tasks.workunit.client.0.smithi088.stdout: create 158/37 file after the snapshot 2023-10-16T20:41:08.186 INFO:tasks.workunit.client.0.smithi088.stdout: create 158/38 file after the snapshot 2023-10-16T20:41:08.189 INFO:tasks.workunit.client.0.smithi088.stdout: create 158/39 file after the snapshot 2023-10-16T20:41:08.191 INFO:tasks.workunit.client.0.smithi088.stdout: create 159/.snap/snap-subdir-test 2023-10-16T20:41:08.215 INFO:tasks.workunit.client.0.smithi088.stdout: create 159/30 file after the snapshot 2023-10-16T20:41:08.217 INFO:tasks.workunit.client.0.smithi088.stdout: create 159/31 file after the snapshot 2023-10-16T20:41:08.219 INFO:tasks.workunit.client.0.smithi088.stdout: create 159/32 file after the snapshot 2023-10-16T20:41:08.222 INFO:tasks.workunit.client.0.smithi088.stdout: create 159/33 file after the snapshot 2023-10-16T20:41:08.225 INFO:tasks.workunit.client.0.smithi088.stdout: create 159/34 file after the snapshot 2023-10-16T20:41:08.227 INFO:tasks.workunit.client.0.smithi088.stdout: create 159/35 file after the snapshot 2023-10-16T20:41:08.229 INFO:tasks.workunit.client.0.smithi088.stdout: create 159/36 file after the snapshot 2023-10-16T20:41:08.232 INFO:tasks.workunit.client.0.smithi088.stdout: create 159/37 file after the snapshot 2023-10-16T20:41:08.234 INFO:tasks.workunit.client.0.smithi088.stdout: create 159/38 file after the snapshot 2023-10-16T20:41:08.237 INFO:tasks.workunit.client.0.smithi088.stdout: create 159/39 file after the snapshot 2023-10-16T20:41:08.239 INFO:tasks.workunit.client.0.smithi088.stdout: create 160/.snap/snap-subdir-test 2023-10-16T20:41:08.261 INFO:tasks.workunit.client.0.smithi088.stdout: create 160/30 file after the snapshot 2023-10-16T20:41:08.263 INFO:tasks.workunit.client.0.smithi088.stdout: create 160/31 file after the snapshot 2023-10-16T20:41:08.265 INFO:tasks.workunit.client.0.smithi088.stdout: create 160/32 file after the snapshot 2023-10-16T20:41:08.268 INFO:tasks.workunit.client.0.smithi088.stdout: create 160/33 file after the snapshot 2023-10-16T20:41:08.271 INFO:tasks.workunit.client.0.smithi088.stdout: create 160/34 file after the snapshot 2023-10-16T20:41:08.279 INFO:tasks.workunit.client.0.smithi088.stdout: create 160/35 file after the snapshot 2023-10-16T20:41:09.128 INFO:tasks.workunit.client.0.smithi088.stdout: create 160/36 file after the snapshot 2023-10-16T20:41:09.130 INFO:tasks.workunit.client.0.smithi088.stdout: create 160/37 file after the snapshot 2023-10-16T20:41:09.133 INFO:tasks.workunit.client.0.smithi088.stdout: create 160/38 file after the snapshot 2023-10-16T20:41:09.135 INFO:tasks.workunit.client.0.smithi088.stdout: create 160/39 file after the snapshot 2023-10-16T20:41:09.138 INFO:tasks.workunit.client.0.smithi088.stdout: create 161/.snap/snap-subdir-test 2023-10-16T20:41:09.186 INFO:tasks.workunit.client.0.smithi088.stdout: create 161/30 file after the snapshot 2023-10-16T20:41:09.188 INFO:tasks.workunit.client.0.smithi088.stdout: create 161/31 file after the snapshot 2023-10-16T20:41:09.190 INFO:tasks.workunit.client.0.smithi088.stdout: create 161/32 file after the snapshot 2023-10-16T20:41:09.193 INFO:tasks.workunit.client.0.smithi088.stdout: create 161/33 file after the snapshot 2023-10-16T20:41:09.195 INFO:tasks.workunit.client.0.smithi088.stdout: create 161/34 file after the snapshot 2023-10-16T20:41:09.197 INFO:tasks.workunit.client.0.smithi088.stdout: create 161/35 file after the snapshot 2023-10-16T20:41:09.199 INFO:tasks.workunit.client.0.smithi088.stdout: create 161/36 file after the snapshot 2023-10-16T20:41:09.202 INFO:tasks.workunit.client.0.smithi088.stdout: create 161/37 file after the snapshot 2023-10-16T20:41:09.281 INFO:tasks.workunit.client.0.smithi088.stdout: create 161/38 file after the snapshot 2023-10-16T20:41:10.200 INFO:tasks.workunit.client.0.smithi088.stdout: create 161/39 file after the snapshot 2023-10-16T20:41:10.202 INFO:tasks.workunit.client.0.smithi088.stdout: create 162/.snap/snap-subdir-test 2023-10-16T20:41:10.230 INFO:tasks.workunit.client.0.smithi088.stdout: create 162/30 file after the snapshot 2023-10-16T20:41:10.233 INFO:tasks.workunit.client.0.smithi088.stdout: create 162/31 file after the snapshot 2023-10-16T20:41:10.235 INFO:tasks.workunit.client.0.smithi088.stdout: create 162/32 file after the snapshot 2023-10-16T20:41:10.238 INFO:tasks.workunit.client.0.smithi088.stdout: create 162/33 file after the snapshot 2023-10-16T20:41:10.241 INFO:tasks.workunit.client.0.smithi088.stdout: create 162/34 file after the snapshot 2023-10-16T20:41:10.243 INFO:tasks.workunit.client.0.smithi088.stdout: create 162/35 file after the snapshot 2023-10-16T20:41:10.246 INFO:tasks.workunit.client.0.smithi088.stdout: create 162/36 file after the snapshot 2023-10-16T20:41:10.248 INFO:tasks.workunit.client.0.smithi088.stdout: create 162/37 file after the snapshot 2023-10-16T20:41:10.251 INFO:tasks.workunit.client.0.smithi088.stdout: create 162/38 file after the snapshot 2023-10-16T20:41:10.254 INFO:tasks.workunit.client.0.smithi088.stdout: create 162/39 file after the snapshot 2023-10-16T20:41:10.257 INFO:tasks.workunit.client.0.smithi088.stdout: create 163/.snap/snap-subdir-test 2023-10-16T20:41:10.282 INFO:tasks.workunit.client.0.smithi088.stdout: create 163/30 file after the snapshot 2023-10-16T20:41:11.239 INFO:tasks.workunit.client.0.smithi088.stdout: create 163/31 file after the snapshot 2023-10-16T20:41:11.241 INFO:tasks.workunit.client.0.smithi088.stdout: create 163/32 file after the snapshot 2023-10-16T20:41:11.244 INFO:tasks.workunit.client.0.smithi088.stdout: create 163/33 file after the snapshot 2023-10-16T20:41:11.247 INFO:tasks.workunit.client.0.smithi088.stdout: create 163/34 file after the snapshot 2023-10-16T20:41:11.250 INFO:tasks.workunit.client.0.smithi088.stdout: create 163/35 file after the snapshot 2023-10-16T20:41:11.253 INFO:tasks.workunit.client.0.smithi088.stdout: create 163/36 file after the snapshot 2023-10-16T20:41:11.256 INFO:tasks.workunit.client.0.smithi088.stdout: create 163/37 file after the snapshot 2023-10-16T20:41:11.259 INFO:tasks.workunit.client.0.smithi088.stdout: create 163/38 file after the snapshot 2023-10-16T20:41:11.261 INFO:tasks.workunit.client.0.smithi088.stdout: create 163/39 file after the snapshot 2023-10-16T20:41:11.264 INFO:tasks.workunit.client.0.smithi088.stdout: create 164/.snap/snap-subdir-test 2023-10-16T20:41:11.313 INFO:tasks.workunit.client.0.smithi088.stdout: create 164/30 file after the snapshot 2023-10-16T20:41:11.330 INFO:tasks.workunit.client.0.smithi088.stdout: create 164/31 file after the snapshot 2023-10-16T20:41:11.332 INFO:tasks.workunit.client.0.smithi088.stdout: create 164/32 file after the snapshot 2023-10-16T20:41:11.335 INFO:tasks.workunit.client.0.smithi088.stdout: create 164/33 file after the snapshot 2023-10-16T20:41:11.338 INFO:tasks.workunit.client.0.smithi088.stdout: create 164/34 file after the snapshot 2023-10-16T20:41:11.340 INFO:tasks.workunit.client.0.smithi088.stdout: create 164/35 file after the snapshot 2023-10-16T20:41:11.343 INFO:tasks.workunit.client.0.smithi088.stdout: create 164/36 file after the snapshot 2023-10-16T20:41:11.346 INFO:tasks.workunit.client.0.smithi088.stdout: create 164/37 file after the snapshot 2023-10-16T20:41:11.348 INFO:tasks.workunit.client.0.smithi088.stdout: create 164/38 file after the snapshot 2023-10-16T20:41:11.351 INFO:tasks.workunit.client.0.smithi088.stdout: create 164/39 file after the snapshot 2023-10-16T20:41:11.357 INFO:tasks.workunit.client.0.smithi088.stdout: create 165/.snap/snap-subdir-test 2023-10-16T20:41:11.385 INFO:tasks.workunit.client.0.smithi088.stdout: create 165/30 file after the snapshot 2023-10-16T20:41:11.387 INFO:tasks.workunit.client.0.smithi088.stdout: create 165/31 file after the snapshot 2023-10-16T20:41:11.390 INFO:tasks.workunit.client.0.smithi088.stdout: create 165/32 file after the snapshot 2023-10-16T20:41:11.392 INFO:tasks.workunit.client.0.smithi088.stdout: create 165/33 file after the snapshot 2023-10-16T20:41:11.395 INFO:tasks.workunit.client.0.smithi088.stdout: create 165/34 file after the snapshot 2023-10-16T20:41:12.290 INFO:tasks.workunit.client.0.smithi088.stdout: create 165/35 file after the snapshot 2023-10-16T20:41:12.391 INFO:tasks.workunit.client.0.smithi088.stdout: create 165/36 file after the snapshot 2023-10-16T20:41:12.394 INFO:tasks.workunit.client.0.smithi088.stdout: create 165/37 file after the snapshot 2023-10-16T20:41:12.397 INFO:tasks.workunit.client.0.smithi088.stdout: create 165/38 file after the snapshot 2023-10-16T20:41:12.399 INFO:tasks.workunit.client.0.smithi088.stdout: create 165/39 file after the snapshot 2023-10-16T20:41:12.402 INFO:tasks.workunit.client.0.smithi088.stdout: create 166/.snap/snap-subdir-test 2023-10-16T20:41:12.424 INFO:tasks.workunit.client.0.smithi088.stdout: create 166/30 file after the snapshot 2023-10-16T20:41:12.426 INFO:tasks.workunit.client.0.smithi088.stdout: create 166/31 file after the snapshot 2023-10-16T20:41:12.429 INFO:tasks.workunit.client.0.smithi088.stdout: create 166/32 file after the snapshot 2023-10-16T20:41:12.431 INFO:tasks.workunit.client.0.smithi088.stdout: create 166/33 file after the snapshot 2023-10-16T20:41:12.434 INFO:tasks.workunit.client.0.smithi088.stdout: create 166/34 file after the snapshot 2023-10-16T20:41:12.436 INFO:tasks.workunit.client.0.smithi088.stdout: create 166/35 file after the snapshot 2023-10-16T20:41:12.439 INFO:tasks.workunit.client.0.smithi088.stdout: create 166/36 file after the snapshot 2023-10-16T20:41:12.442 INFO:tasks.workunit.client.0.smithi088.stdout: create 166/37 file after the snapshot 2023-10-16T20:41:12.444 INFO:tasks.workunit.client.0.smithi088.stdout: create 166/38 file after the snapshot 2023-10-16T20:41:12.447 INFO:tasks.workunit.client.0.smithi088.stdout: create 166/39 file after the snapshot 2023-10-16T20:41:12.450 INFO:tasks.workunit.client.0.smithi088.stdout: create 167/.snap/snap-subdir-test 2023-10-16T20:41:12.620 INFO:tasks.workunit.client.0.smithi088.stdout: create 167/30 file after the snapshot 2023-10-16T20:41:12.620 INFO:tasks.workunit.client.0.smithi088.stdout: create 167/31 file after the snapshot 2023-10-16T20:41:12.621 INFO:tasks.workunit.client.0.smithi088.stdout: create 167/32 file after the snapshot 2023-10-16T20:41:12.621 INFO:tasks.workunit.client.0.smithi088.stdout: create 167/33 file after the snapshot 2023-10-16T20:41:12.621 INFO:tasks.workunit.client.0.smithi088.stdout: create 167/34 file after the snapshot 2023-10-16T20:41:12.621 INFO:tasks.workunit.client.0.smithi088.stdout: create 167/35 file after the snapshot 2023-10-16T20:41:12.622 INFO:tasks.workunit.client.0.smithi088.stdout: create 167/36 file after the snapshot 2023-10-16T20:41:12.622 INFO:tasks.workunit.client.0.smithi088.stdout: create 167/37 file after the snapshot 2023-10-16T20:41:12.622 INFO:tasks.workunit.client.0.smithi088.stdout: create 167/38 file after the snapshot 2023-10-16T20:41:12.622 INFO:tasks.workunit.client.0.smithi088.stdout: create 167/39 file after the snapshot 2023-10-16T20:41:12.623 INFO:tasks.workunit.client.0.smithi088.stdout: create 168/.snap/snap-subdir-test 2023-10-16T20:41:12.623 INFO:tasks.workunit.client.0.smithi088.stdout: create 168/30 file after the snapshot 2023-10-16T20:41:12.623 INFO:tasks.workunit.client.0.smithi088.stdout: create 168/31 file after the snapshot 2023-10-16T20:41:12.623 INFO:tasks.workunit.client.0.smithi088.stdout: create 168/32 file after the snapshot 2023-10-16T20:41:12.624 INFO:tasks.workunit.client.0.smithi088.stdout: create 168/33 file after the snapshot 2023-10-16T20:41:12.624 INFO:tasks.workunit.client.0.smithi088.stdout: create 168/34 file after the snapshot 2023-10-16T20:41:12.624 INFO:tasks.workunit.client.0.smithi088.stdout: create 168/35 file after the snapshot 2023-10-16T20:41:12.624 INFO:tasks.workunit.client.0.smithi088.stdout: create 168/36 file after the snapshot 2023-10-16T20:41:12.624 INFO:tasks.workunit.client.0.smithi088.stdout: create 168/37 file after the snapshot 2023-10-16T20:41:12.625 INFO:tasks.workunit.client.0.smithi088.stdout: create 168/38 file after the snapshot 2023-10-16T20:41:12.625 INFO:tasks.workunit.client.0.smithi088.stdout: create 168/39 file after the snapshot 2023-10-16T20:41:12.625 INFO:tasks.workunit.client.0.smithi088.stdout: create 169/.snap/snap-subdir-test 2023-10-16T20:41:12.625 INFO:tasks.workunit.client.0.smithi088.stdout: create 169/30 file after the snapshot 2023-10-16T20:41:12.626 INFO:tasks.workunit.client.0.smithi088.stdout: create 169/31 file after the snapshot 2023-10-16T20:41:12.626 INFO:tasks.workunit.client.0.smithi088.stdout: create 169/32 file after the snapshot 2023-10-16T20:41:12.626 INFO:tasks.workunit.client.0.smithi088.stdout: create 169/33 file after the snapshot 2023-10-16T20:41:12.626 INFO:tasks.workunit.client.0.smithi088.stdout: create 169/34 file after the snapshot 2023-10-16T20:41:12.627 INFO:tasks.workunit.client.0.smithi088.stdout: create 169/35 file after the snapshot 2023-10-16T20:41:12.627 INFO:tasks.workunit.client.0.smithi088.stdout: create 169/36 file after the snapshot 2023-10-16T20:41:12.627 INFO:tasks.workunit.client.0.smithi088.stdout: create 169/37 file after the snapshot 2023-10-16T20:41:12.628 INFO:tasks.workunit.client.0.smithi088.stdout: create 169/38 file after the snapshot 2023-10-16T20:41:12.628 INFO:tasks.workunit.client.0.smithi088.stdout: create 169/39 file after the snapshot 2023-10-16T20:41:12.628 INFO:tasks.workunit.client.0.smithi088.stdout: create 170/.snap/snap-subdir-test 2023-10-16T20:41:12.805 INFO:tasks.workunit.client.0.smithi088.stdout: create 170/30 file after the snapshot 2023-10-16T20:41:12.807 INFO:tasks.workunit.client.0.smithi088.stdout: create 170/31 file after the snapshot 2023-10-16T20:41:12.809 INFO:tasks.workunit.client.0.smithi088.stdout: create 170/32 file after the snapshot 2023-10-16T20:41:12.812 INFO:tasks.workunit.client.0.smithi088.stdout: create 170/33 file after the snapshot 2023-10-16T20:41:12.814 INFO:tasks.workunit.client.0.smithi088.stdout: create 170/34 file after the snapshot 2023-10-16T20:41:12.817 INFO:tasks.workunit.client.0.smithi088.stdout: create 170/35 file after the snapshot 2023-10-16T20:41:12.820 INFO:tasks.workunit.client.0.smithi088.stdout: create 170/36 file after the snapshot 2023-10-16T20:41:12.822 INFO:tasks.workunit.client.0.smithi088.stdout: create 170/37 file after the snapshot 2023-10-16T20:41:12.825 INFO:tasks.workunit.client.0.smithi088.stdout: create 170/38 file after the snapshot 2023-10-16T20:41:12.828 INFO:tasks.workunit.client.0.smithi088.stdout: create 170/39 file after the snapshot 2023-10-16T20:41:12.831 INFO:tasks.workunit.client.0.smithi088.stdout: create 171/.snap/snap-subdir-test 2023-10-16T20:41:12.853 INFO:tasks.workunit.client.0.smithi088.stdout: create 171/30 file after the snapshot 2023-10-16T20:41:12.855 INFO:tasks.workunit.client.0.smithi088.stdout: create 171/31 file after the snapshot 2023-10-16T20:41:12.858 INFO:tasks.workunit.client.0.smithi088.stdout: create 171/32 file after the snapshot 2023-10-16T20:41:12.861 INFO:tasks.workunit.client.0.smithi088.stdout: create 171/33 file after the snapshot 2023-10-16T20:41:12.863 INFO:tasks.workunit.client.0.smithi088.stdout: create 171/34 file after the snapshot 2023-10-16T20:41:12.865 INFO:tasks.workunit.client.0.smithi088.stdout: create 171/35 file after the snapshot 2023-10-16T20:41:12.868 INFO:tasks.workunit.client.0.smithi088.stdout: create 171/36 file after the snapshot 2023-10-16T20:41:12.870 INFO:tasks.workunit.client.0.smithi088.stdout: create 171/37 file after the snapshot 2023-10-16T20:41:12.872 INFO:tasks.workunit.client.0.smithi088.stdout: create 171/38 file after the snapshot 2023-10-16T20:41:12.875 INFO:tasks.workunit.client.0.smithi088.stdout: create 171/39 file after the snapshot 2023-10-16T20:41:12.877 INFO:tasks.workunit.client.0.smithi088.stdout: create 172/.snap/snap-subdir-test 2023-10-16T20:41:12.899 INFO:tasks.workunit.client.0.smithi088.stdout: create 172/30 file after the snapshot 2023-10-16T20:41:12.901 INFO:tasks.workunit.client.0.smithi088.stdout: create 172/31 file after the snapshot 2023-10-16T20:41:12.903 INFO:tasks.workunit.client.0.smithi088.stdout: create 172/32 file after the snapshot 2023-10-16T20:41:12.906 INFO:tasks.workunit.client.0.smithi088.stdout: create 172/33 file after the snapshot 2023-10-16T20:41:12.908 INFO:tasks.workunit.client.0.smithi088.stdout: create 172/34 file after the snapshot 2023-10-16T20:41:12.911 INFO:tasks.workunit.client.0.smithi088.stdout: create 172/35 file after the snapshot 2023-10-16T20:41:12.914 INFO:tasks.workunit.client.0.smithi088.stdout: create 172/36 file after the snapshot 2023-10-16T20:41:12.916 INFO:tasks.workunit.client.0.smithi088.stdout: create 172/37 file after the snapshot 2023-10-16T20:41:12.919 INFO:tasks.workunit.client.0.smithi088.stdout: create 172/38 file after the snapshot 2023-10-16T20:41:12.921 INFO:tasks.workunit.client.0.smithi088.stdout: create 172/39 file after the snapshot 2023-10-16T20:41:12.924 INFO:tasks.workunit.client.0.smithi088.stdout: create 173/.snap/snap-subdir-test 2023-10-16T20:41:12.945 INFO:tasks.workunit.client.0.smithi088.stdout: create 173/30 file after the snapshot 2023-10-16T20:41:12.947 INFO:tasks.workunit.client.0.smithi088.stdout: create 173/31 file after the snapshot 2023-10-16T20:41:12.949 INFO:tasks.workunit.client.0.smithi088.stdout: create 173/32 file after the snapshot 2023-10-16T20:41:12.952 INFO:tasks.workunit.client.0.smithi088.stdout: create 173/33 file after the snapshot 2023-10-16T20:41:12.955 INFO:tasks.workunit.client.0.smithi088.stdout: create 173/34 file after the snapshot 2023-10-16T20:41:12.957 INFO:tasks.workunit.client.0.smithi088.stdout: create 173/35 file after the snapshot 2023-10-16T20:41:12.960 INFO:tasks.workunit.client.0.smithi088.stdout: create 173/36 file after the snapshot 2023-10-16T20:41:12.962 INFO:tasks.workunit.client.0.smithi088.stdout: create 173/37 file after the snapshot 2023-10-16T20:41:12.964 INFO:tasks.workunit.client.0.smithi088.stdout: create 173/38 file after the snapshot 2023-10-16T20:41:12.968 INFO:tasks.workunit.client.0.smithi088.stdout: create 173/39 file after the snapshot 2023-10-16T20:41:12.969 INFO:tasks.workunit.client.0.smithi088.stdout: create 174/.snap/snap-subdir-test 2023-10-16T20:41:12.991 INFO:tasks.workunit.client.0.smithi088.stdout: create 174/30 file after the snapshot 2023-10-16T20:41:12.992 INFO:tasks.workunit.client.0.smithi088.stdout: create 174/31 file after the snapshot 2023-10-16T20:41:12.994 INFO:tasks.workunit.client.0.smithi088.stdout: create 174/32 file after the snapshot 2023-10-16T20:41:12.997 INFO:tasks.workunit.client.0.smithi088.stdout: create 174/33 file after the snapshot 2023-10-16T20:41:12.999 INFO:tasks.workunit.client.0.smithi088.stdout: create 174/34 file after the snapshot 2023-10-16T20:41:13.001 INFO:tasks.workunit.client.0.smithi088.stdout: create 174/35 file after the snapshot 2023-10-16T20:41:13.003 INFO:tasks.workunit.client.0.smithi088.stdout: create 174/36 file after the snapshot 2023-10-16T20:41:13.006 INFO:tasks.workunit.client.0.smithi088.stdout: create 174/37 file after the snapshot 2023-10-16T20:41:13.008 INFO:tasks.workunit.client.0.smithi088.stdout: create 174/38 file after the snapshot 2023-10-16T20:41:13.011 INFO:tasks.workunit.client.0.smithi088.stdout: create 174/39 file after the snapshot 2023-10-16T20:41:13.014 INFO:tasks.workunit.client.0.smithi088.stdout: create 175/.snap/snap-subdir-test 2023-10-16T20:41:13.385 INFO:tasks.workunit.client.0.smithi088.stdout: create 175/30 file after the snapshot 2023-10-16T20:41:13.532 INFO:tasks.workunit.client.0.smithi088.stdout: create 175/31 file after the snapshot 2023-10-16T20:41:13.534 INFO:tasks.workunit.client.0.smithi088.stdout: create 175/32 file after the snapshot 2023-10-16T20:41:13.537 INFO:tasks.workunit.client.0.smithi088.stdout: create 175/33 file after the snapshot 2023-10-16T20:41:13.539 INFO:tasks.workunit.client.0.smithi088.stdout: create 175/34 file after the snapshot 2023-10-16T20:41:13.542 INFO:tasks.workunit.client.0.smithi088.stdout: create 175/35 file after the snapshot 2023-10-16T20:41:13.544 INFO:tasks.workunit.client.0.smithi088.stdout: create 175/36 file after the snapshot 2023-10-16T20:41:13.547 INFO:tasks.workunit.client.0.smithi088.stdout: create 175/37 file after the snapshot 2023-10-16T20:41:13.549 INFO:tasks.workunit.client.0.smithi088.stdout: create 175/38 file after the snapshot 2023-10-16T20:41:13.552 INFO:tasks.workunit.client.0.smithi088.stdout: create 175/39 file after the snapshot 2023-10-16T20:41:13.555 INFO:tasks.workunit.client.0.smithi088.stdout: create 176/.snap/snap-subdir-test 2023-10-16T20:41:13.584 INFO:tasks.workunit.client.0.smithi088.stdout: create 176/30 file after the snapshot 2023-10-16T20:41:13.586 INFO:tasks.workunit.client.0.smithi088.stdout: create 176/31 file after the snapshot 2023-10-16T20:41:13.589 INFO:tasks.workunit.client.0.smithi088.stdout: create 176/32 file after the snapshot 2023-10-16T20:41:13.591 INFO:tasks.workunit.client.0.smithi088.stdout: create 176/33 file after the snapshot 2023-10-16T20:41:13.593 INFO:tasks.workunit.client.0.smithi088.stdout: create 176/34 file after the snapshot 2023-10-16T20:41:13.596 INFO:tasks.workunit.client.0.smithi088.stdout: create 176/35 file after the snapshot 2023-10-16T20:41:13.599 INFO:tasks.workunit.client.0.smithi088.stdout: create 176/36 file after the snapshot 2023-10-16T20:41:13.601 INFO:tasks.workunit.client.0.smithi088.stdout: create 176/37 file after the snapshot 2023-10-16T20:41:13.604 INFO:tasks.workunit.client.0.smithi088.stdout: create 176/38 file after the snapshot 2023-10-16T20:41:13.606 INFO:tasks.workunit.client.0.smithi088.stdout: create 176/39 file after the snapshot 2023-10-16T20:41:13.609 INFO:tasks.workunit.client.0.smithi088.stdout: create 177/.snap/snap-subdir-test 2023-10-16T20:41:13.630 INFO:tasks.workunit.client.0.smithi088.stdout: create 177/30 file after the snapshot 2023-10-16T20:41:13.633 INFO:tasks.workunit.client.0.smithi088.stdout: create 177/31 file after the snapshot 2023-10-16T20:41:13.635 INFO:tasks.workunit.client.0.smithi088.stdout: create 177/32 file after the snapshot 2023-10-16T20:41:13.638 INFO:tasks.workunit.client.0.smithi088.stdout: create 177/33 file after the snapshot 2023-10-16T20:41:13.641 INFO:tasks.workunit.client.0.smithi088.stdout: create 177/34 file after the snapshot 2023-10-16T20:41:13.643 INFO:tasks.workunit.client.0.smithi088.stdout: create 177/35 file after the snapshot 2023-10-16T20:41:13.646 INFO:tasks.workunit.client.0.smithi088.stdout: create 177/36 file after the snapshot 2023-10-16T20:41:13.648 INFO:tasks.workunit.client.0.smithi088.stdout: create 177/37 file after the snapshot 2023-10-16T20:41:13.651 INFO:tasks.workunit.client.0.smithi088.stdout: create 177/38 file after the snapshot 2023-10-16T20:41:13.653 INFO:tasks.workunit.client.0.smithi088.stdout: create 177/39 file after the snapshot 2023-10-16T20:41:13.656 INFO:tasks.workunit.client.0.smithi088.stdout: create 178/.snap/snap-subdir-test 2023-10-16T20:41:13.678 INFO:tasks.workunit.client.0.smithi088.stdout: create 178/30 file after the snapshot 2023-10-16T20:41:13.680 INFO:tasks.workunit.client.0.smithi088.stdout: create 178/31 file after the snapshot 2023-10-16T20:41:13.682 INFO:tasks.workunit.client.0.smithi088.stdout: create 178/32 file after the snapshot 2023-10-16T20:41:13.685 INFO:tasks.workunit.client.0.smithi088.stdout: create 178/33 file after the snapshot 2023-10-16T20:41:13.687 INFO:tasks.workunit.client.0.smithi088.stdout: create 178/34 file after the snapshot 2023-10-16T20:41:13.690 INFO:tasks.workunit.client.0.smithi088.stdout: create 178/35 file after the snapshot 2023-10-16T20:41:13.692 INFO:tasks.workunit.client.0.smithi088.stdout: create 178/36 file after the snapshot 2023-10-16T20:41:13.695 INFO:tasks.workunit.client.0.smithi088.stdout: create 178/37 file after the snapshot 2023-10-16T20:41:13.697 INFO:tasks.workunit.client.0.smithi088.stdout: create 178/38 file after the snapshot 2023-10-16T20:41:13.700 INFO:tasks.workunit.client.0.smithi088.stdout: create 178/39 file after the snapshot 2023-10-16T20:41:13.703 INFO:tasks.workunit.client.0.smithi088.stdout: create 179/.snap/snap-subdir-test 2023-10-16T20:41:13.730 INFO:tasks.workunit.client.0.smithi088.stdout: create 179/30 file after the snapshot 2023-10-16T20:41:13.732 INFO:tasks.workunit.client.0.smithi088.stdout: create 179/31 file after the snapshot 2023-10-16T20:41:13.735 INFO:tasks.workunit.client.0.smithi088.stdout: create 179/32 file after the snapshot 2023-10-16T20:41:13.738 INFO:tasks.workunit.client.0.smithi088.stdout: create 179/33 file after the snapshot 2023-10-16T20:41:13.740 INFO:tasks.workunit.client.0.smithi088.stdout: create 179/34 file after the snapshot 2023-10-16T20:41:13.743 INFO:tasks.workunit.client.0.smithi088.stdout: create 179/35 file after the snapshot 2023-10-16T20:41:13.745 INFO:tasks.workunit.client.0.smithi088.stdout: create 179/36 file after the snapshot 2023-10-16T20:41:13.748 INFO:tasks.workunit.client.0.smithi088.stdout: create 179/37 file after the snapshot 2023-10-16T20:41:13.750 INFO:tasks.workunit.client.0.smithi088.stdout: create 179/38 file after the snapshot 2023-10-16T20:41:13.753 INFO:tasks.workunit.client.0.smithi088.stdout: create 179/39 file after the snapshot 2023-10-16T20:41:13.756 INFO:tasks.workunit.client.0.smithi088.stdout: create 180/.snap/snap-subdir-test 2023-10-16T20:41:13.778 INFO:tasks.workunit.client.0.smithi088.stdout: create 180/30 file after the snapshot 2023-10-16T20:41:13.780 INFO:tasks.workunit.client.0.smithi088.stdout: create 180/31 file after the snapshot 2023-10-16T20:41:13.782 INFO:tasks.workunit.client.0.smithi088.stdout: create 180/32 file after the snapshot 2023-10-16T20:41:13.784 INFO:tasks.workunit.client.0.smithi088.stdout: create 180/33 file after the snapshot 2023-10-16T20:41:13.787 INFO:tasks.workunit.client.0.smithi088.stdout: create 180/34 file after the snapshot 2023-10-16T20:41:13.789 INFO:tasks.workunit.client.0.smithi088.stdout: create 180/35 file after the snapshot 2023-10-16T20:41:13.791 INFO:tasks.workunit.client.0.smithi088.stdout: create 180/36 file after the snapshot 2023-10-16T20:41:13.794 INFO:tasks.workunit.client.0.smithi088.stdout: create 180/37 file after the snapshot 2023-10-16T20:41:13.796 INFO:tasks.workunit.client.0.smithi088.stdout: create 180/38 file after the snapshot 2023-10-16T20:41:13.799 INFO:tasks.workunit.client.0.smithi088.stdout: create 180/39 file after the snapshot 2023-10-16T20:41:13.801 INFO:tasks.workunit.client.0.smithi088.stdout: create 181/.snap/snap-subdir-test 2023-10-16T20:41:13.822 INFO:tasks.workunit.client.0.smithi088.stdout: create 181/30 file after the snapshot 2023-10-16T20:41:13.823 INFO:tasks.workunit.client.0.smithi088.stdout: create 181/31 file after the snapshot 2023-10-16T20:41:13.826 INFO:tasks.workunit.client.0.smithi088.stdout: create 181/32 file after the snapshot 2023-10-16T20:41:13.828 INFO:tasks.workunit.client.0.smithi088.stdout: create 181/33 file after the snapshot 2023-10-16T20:41:13.830 INFO:tasks.workunit.client.0.smithi088.stdout: create 181/34 file after the snapshot 2023-10-16T20:41:13.832 INFO:tasks.workunit.client.0.smithi088.stdout: create 181/35 file after the snapshot 2023-10-16T20:41:13.835 INFO:tasks.workunit.client.0.smithi088.stdout: create 181/36 file after the snapshot 2023-10-16T20:41:13.838 INFO:tasks.workunit.client.0.smithi088.stdout: create 181/37 file after the snapshot 2023-10-16T20:41:13.840 INFO:tasks.workunit.client.0.smithi088.stdout: create 181/38 file after the snapshot 2023-10-16T20:41:13.843 INFO:tasks.workunit.client.0.smithi088.stdout: create 181/39 file after the snapshot 2023-10-16T20:41:13.845 INFO:tasks.workunit.client.0.smithi088.stdout: create 182/.snap/snap-subdir-test 2023-10-16T20:41:13.874 INFO:tasks.workunit.client.0.smithi088.stdout: create 182/30 file after the snapshot 2023-10-16T20:41:13.876 INFO:tasks.workunit.client.0.smithi088.stdout: create 182/31 file after the snapshot 2023-10-16T20:41:13.878 INFO:tasks.workunit.client.0.smithi088.stdout: create 182/32 file after the snapshot 2023-10-16T20:41:13.881 INFO:tasks.workunit.client.0.smithi088.stdout: create 182/33 file after the snapshot 2023-10-16T20:41:13.883 INFO:tasks.workunit.client.0.smithi088.stdout: create 182/34 file after the snapshot 2023-10-16T20:41:13.885 INFO:tasks.workunit.client.0.smithi088.stdout: create 182/35 file after the snapshot 2023-10-16T20:41:13.888 INFO:tasks.workunit.client.0.smithi088.stdout: create 182/36 file after the snapshot 2023-10-16T20:41:13.890 INFO:tasks.workunit.client.0.smithi088.stdout: create 182/37 file after the snapshot 2023-10-16T20:41:13.893 INFO:tasks.workunit.client.0.smithi088.stdout: create 182/38 file after the snapshot 2023-10-16T20:41:13.895 INFO:tasks.workunit.client.0.smithi088.stdout: create 182/39 file after the snapshot 2023-10-16T20:41:13.898 INFO:tasks.workunit.client.0.smithi088.stdout: create 183/.snap/snap-subdir-test 2023-10-16T20:41:13.918 INFO:tasks.workunit.client.0.smithi088.stdout: create 183/30 file after the snapshot 2023-10-16T20:41:13.920 INFO:tasks.workunit.client.0.smithi088.stdout: create 183/31 file after the snapshot 2023-10-16T20:41:13.923 INFO:tasks.workunit.client.0.smithi088.stdout: create 183/32 file after the snapshot 2023-10-16T20:41:13.925 INFO:tasks.workunit.client.0.smithi088.stdout: create 183/33 file after the snapshot 2023-10-16T20:41:13.928 INFO:tasks.workunit.client.0.smithi088.stdout: create 183/34 file after the snapshot 2023-10-16T20:41:13.930 INFO:tasks.workunit.client.0.smithi088.stdout: create 183/35 file after the snapshot 2023-10-16T20:41:13.933 INFO:tasks.workunit.client.0.smithi088.stdout: create 183/36 file after the snapshot 2023-10-16T20:41:13.935 INFO:tasks.workunit.client.0.smithi088.stdout: create 183/37 file after the snapshot 2023-10-16T20:41:13.937 INFO:tasks.workunit.client.0.smithi088.stdout: create 183/38 file after the snapshot 2023-10-16T20:41:13.940 INFO:tasks.workunit.client.0.smithi088.stdout: create 183/39 file after the snapshot 2023-10-16T20:41:13.942 INFO:tasks.workunit.client.0.smithi088.stdout: create 184/.snap/snap-subdir-test 2023-10-16T20:41:13.965 INFO:tasks.workunit.client.0.smithi088.stdout: create 184/30 file after the snapshot 2023-10-16T20:41:13.967 INFO:tasks.workunit.client.0.smithi088.stdout: create 184/31 file after the snapshot 2023-10-16T20:41:13.970 INFO:tasks.workunit.client.0.smithi088.stdout: create 184/32 file after the snapshot 2023-10-16T20:41:13.972 INFO:tasks.workunit.client.0.smithi088.stdout: create 184/33 file after the snapshot 2023-10-16T20:41:13.975 INFO:tasks.workunit.client.0.smithi088.stdout: create 184/34 file after the snapshot 2023-10-16T20:41:13.977 INFO:tasks.workunit.client.0.smithi088.stdout: create 184/35 file after the snapshot 2023-10-16T20:41:13.980 INFO:tasks.workunit.client.0.smithi088.stdout: create 184/36 file after the snapshot 2023-10-16T20:41:13.982 INFO:tasks.workunit.client.0.smithi088.stdout: create 184/37 file after the snapshot 2023-10-16T20:41:13.985 INFO:tasks.workunit.client.0.smithi088.stdout: create 184/38 file after the snapshot 2023-10-16T20:41:13.987 INFO:tasks.workunit.client.0.smithi088.stdout: create 184/39 file after the snapshot 2023-10-16T20:41:13.990 INFO:tasks.workunit.client.0.smithi088.stdout: create 185/.snap/snap-subdir-test 2023-10-16T20:41:14.012 INFO:tasks.workunit.client.0.smithi088.stdout: create 185/30 file after the snapshot 2023-10-16T20:41:14.014 INFO:tasks.workunit.client.0.smithi088.stdout: create 185/31 file after the snapshot 2023-10-16T20:41:14.017 INFO:tasks.workunit.client.0.smithi088.stdout: create 185/32 file after the snapshot 2023-10-16T20:41:14.019 INFO:tasks.workunit.client.0.smithi088.stdout: create 185/33 file after the snapshot 2023-10-16T20:41:14.021 INFO:tasks.workunit.client.0.smithi088.stdout: create 185/34 file after the snapshot 2023-10-16T20:41:14.024 INFO:tasks.workunit.client.0.smithi088.stdout: create 185/35 file after the snapshot 2023-10-16T20:41:14.026 INFO:tasks.workunit.client.0.smithi088.stdout: create 185/36 file after the snapshot 2023-10-16T20:41:14.029 INFO:tasks.workunit.client.0.smithi088.stdout: create 185/37 file after the snapshot 2023-10-16T20:41:14.031 INFO:tasks.workunit.client.0.smithi088.stdout: create 185/38 file after the snapshot 2023-10-16T20:41:14.292 INFO:tasks.workunit.client.0.smithi088.stdout: create 185/39 file after the snapshot 2023-10-16T20:41:14.634 INFO:tasks.workunit.client.0.smithi088.stdout: create 186/.snap/snap-subdir-test 2023-10-16T20:41:14.667 INFO:tasks.workunit.client.0.smithi088.stdout: create 186/30 file after the snapshot 2023-10-16T20:41:14.669 INFO:tasks.workunit.client.0.smithi088.stdout: create 186/31 file after the snapshot 2023-10-16T20:41:14.671 INFO:tasks.workunit.client.0.smithi088.stdout: create 186/32 file after the snapshot 2023-10-16T20:41:14.674 INFO:tasks.workunit.client.0.smithi088.stdout: create 186/33 file after the snapshot 2023-10-16T20:41:14.677 INFO:tasks.workunit.client.0.smithi088.stdout: create 186/34 file after the snapshot 2023-10-16T20:41:14.679 INFO:tasks.workunit.client.0.smithi088.stdout: create 186/35 file after the snapshot 2023-10-16T20:41:14.681 INFO:tasks.workunit.client.0.smithi088.stdout: create 186/36 file after the snapshot 2023-10-16T20:41:14.684 INFO:tasks.workunit.client.0.smithi088.stdout: create 186/37 file after the snapshot 2023-10-16T20:41:14.687 INFO:tasks.workunit.client.0.smithi088.stdout: create 186/38 file after the snapshot 2023-10-16T20:41:14.689 INFO:tasks.workunit.client.0.smithi088.stdout: create 186/39 file after the snapshot 2023-10-16T20:41:14.692 INFO:tasks.workunit.client.0.smithi088.stdout: create 187/.snap/snap-subdir-test 2023-10-16T20:41:14.718 INFO:tasks.workunit.client.0.smithi088.stdout: create 187/30 file after the snapshot 2023-10-16T20:41:14.720 INFO:tasks.workunit.client.0.smithi088.stdout: create 187/31 file after the snapshot 2023-10-16T20:41:14.723 INFO:tasks.workunit.client.0.smithi088.stdout: create 187/32 file after the snapshot 2023-10-16T20:41:14.725 INFO:tasks.workunit.client.0.smithi088.stdout: create 187/33 file after the snapshot 2023-10-16T20:41:14.728 INFO:tasks.workunit.client.0.smithi088.stdout: create 187/34 file after the snapshot 2023-10-16T20:41:14.730 INFO:tasks.workunit.client.0.smithi088.stdout: create 187/35 file after the snapshot 2023-10-16T20:41:14.733 INFO:tasks.workunit.client.0.smithi088.stdout: create 187/36 file after the snapshot 2023-10-16T20:41:14.735 INFO:tasks.workunit.client.0.smithi088.stdout: create 187/37 file after the snapshot 2023-10-16T20:41:14.738 INFO:tasks.workunit.client.0.smithi088.stdout: create 187/38 file after the snapshot 2023-10-16T20:41:14.741 INFO:tasks.workunit.client.0.smithi088.stdout: create 187/39 file after the snapshot 2023-10-16T20:41:14.743 INFO:tasks.workunit.client.0.smithi088.stdout: create 188/.snap/snap-subdir-test 2023-10-16T20:41:14.764 INFO:tasks.workunit.client.0.smithi088.stdout: create 188/30 file after the snapshot 2023-10-16T20:41:14.767 INFO:tasks.workunit.client.0.smithi088.stdout: create 188/31 file after the snapshot 2023-10-16T20:41:14.769 INFO:tasks.workunit.client.0.smithi088.stdout: create 188/32 file after the snapshot 2023-10-16T20:41:14.772 INFO:tasks.workunit.client.0.smithi088.stdout: create 188/33 file after the snapshot 2023-10-16T20:41:14.774 INFO:tasks.workunit.client.0.smithi088.stdout: create 188/34 file after the snapshot 2023-10-16T20:41:14.777 INFO:tasks.workunit.client.0.smithi088.stdout: create 188/35 file after the snapshot 2023-10-16T20:41:14.780 INFO:tasks.workunit.client.0.smithi088.stdout: create 188/36 file after the snapshot 2023-10-16T20:41:14.782 INFO:tasks.workunit.client.0.smithi088.stdout: create 188/37 file after the snapshot 2023-10-16T20:41:14.785 INFO:tasks.workunit.client.0.smithi088.stdout: create 188/38 file after the snapshot 2023-10-16T20:41:14.788 INFO:tasks.workunit.client.0.smithi088.stdout: create 188/39 file after the snapshot 2023-10-16T20:41:14.790 INFO:tasks.workunit.client.0.smithi088.stdout: create 189/.snap/snap-subdir-test 2023-10-16T20:41:14.813 INFO:tasks.workunit.client.0.smithi088.stdout: create 189/30 file after the snapshot 2023-10-16T20:41:14.815 INFO:tasks.workunit.client.0.smithi088.stdout: create 189/31 file after the snapshot 2023-10-16T20:41:14.818 INFO:tasks.workunit.client.0.smithi088.stdout: create 189/32 file after the snapshot 2023-10-16T20:41:14.820 INFO:tasks.workunit.client.0.smithi088.stdout: create 189/33 file after the snapshot 2023-10-16T20:41:14.823 INFO:tasks.workunit.client.0.smithi088.stdout: create 189/34 file after the snapshot 2023-10-16T20:41:14.825 INFO:tasks.workunit.client.0.smithi088.stdout: create 189/35 file after the snapshot 2023-10-16T20:41:14.828 INFO:tasks.workunit.client.0.smithi088.stdout: create 189/36 file after the snapshot 2023-10-16T20:41:14.830 INFO:tasks.workunit.client.0.smithi088.stdout: create 189/37 file after the snapshot 2023-10-16T20:41:14.833 INFO:tasks.workunit.client.0.smithi088.stdout: create 189/38 file after the snapshot 2023-10-16T20:41:14.835 INFO:tasks.workunit.client.0.smithi088.stdout: create 189/39 file after the snapshot 2023-10-16T20:41:14.838 INFO:tasks.workunit.client.0.smithi088.stdout: create 190/.snap/snap-subdir-test 2023-10-16T20:41:14.859 INFO:tasks.workunit.client.0.smithi088.stdout: create 190/30 file after the snapshot 2023-10-16T20:41:14.861 INFO:tasks.workunit.client.0.smithi088.stdout: create 190/31 file after the snapshot 2023-10-16T20:41:14.863 INFO:tasks.workunit.client.0.smithi088.stdout: create 190/32 file after the snapshot 2023-10-16T20:41:14.866 INFO:tasks.workunit.client.0.smithi088.stdout: create 190/33 file after the snapshot 2023-10-16T20:41:14.868 INFO:tasks.workunit.client.0.smithi088.stdout: create 190/34 file after the snapshot 2023-10-16T20:41:14.870 INFO:tasks.workunit.client.0.smithi088.stdout: create 190/35 file after the snapshot 2023-10-16T20:41:14.873 INFO:tasks.workunit.client.0.smithi088.stdout: create 190/36 file after the snapshot 2023-10-16T20:41:14.875 INFO:tasks.workunit.client.0.smithi088.stdout: create 190/37 file after the snapshot 2023-10-16T20:41:14.878 INFO:tasks.workunit.client.0.smithi088.stdout: create 190/38 file after the snapshot 2023-10-16T20:41:14.880 INFO:tasks.workunit.client.0.smithi088.stdout: create 190/39 file after the snapshot 2023-10-16T20:41:14.882 INFO:tasks.workunit.client.0.smithi088.stdout: create 191/.snap/snap-subdir-test 2023-10-16T20:41:14.909 INFO:tasks.workunit.client.0.smithi088.stdout: create 191/30 file after the snapshot 2023-10-16T20:41:14.911 INFO:tasks.workunit.client.0.smithi088.stdout: create 191/31 file after the snapshot 2023-10-16T20:41:14.914 INFO:tasks.workunit.client.0.smithi088.stdout: create 191/32 file after the snapshot 2023-10-16T20:41:14.916 INFO:tasks.workunit.client.0.smithi088.stdout: create 191/33 file after the snapshot 2023-10-16T20:41:14.919 INFO:tasks.workunit.client.0.smithi088.stdout: create 191/34 file after the snapshot 2023-10-16T20:41:14.921 INFO:tasks.workunit.client.0.smithi088.stdout: create 191/35 file after the snapshot 2023-10-16T20:41:14.924 INFO:tasks.workunit.client.0.smithi088.stdout: create 191/36 file after the snapshot 2023-10-16T20:41:14.926 INFO:tasks.workunit.client.0.smithi088.stdout: create 191/37 file after the snapshot 2023-10-16T20:41:14.929 INFO:tasks.workunit.client.0.smithi088.stdout: create 191/38 file after the snapshot 2023-10-16T20:41:14.931 INFO:tasks.workunit.client.0.smithi088.stdout: create 191/39 file after the snapshot 2023-10-16T20:41:14.934 INFO:tasks.workunit.client.0.smithi088.stdout: create 192/.snap/snap-subdir-test 2023-10-16T20:41:14.955 INFO:tasks.workunit.client.0.smithi088.stdout: create 192/30 file after the snapshot 2023-10-16T20:41:14.957 INFO:tasks.workunit.client.0.smithi088.stdout: create 192/31 file after the snapshot 2023-10-16T20:41:14.959 INFO:tasks.workunit.client.0.smithi088.stdout: create 192/32 file after the snapshot 2023-10-16T20:41:14.962 INFO:tasks.workunit.client.0.smithi088.stdout: create 192/33 file after the snapshot 2023-10-16T20:41:14.964 INFO:tasks.workunit.client.0.smithi088.stdout: create 192/34 file after the snapshot 2023-10-16T20:41:14.967 INFO:tasks.workunit.client.0.smithi088.stdout: create 192/35 file after the snapshot 2023-10-16T20:41:14.969 INFO:tasks.workunit.client.0.smithi088.stdout: create 192/36 file after the snapshot 2023-10-16T20:41:14.971 INFO:tasks.workunit.client.0.smithi088.stdout: create 192/37 file after the snapshot 2023-10-16T20:41:14.974 INFO:tasks.workunit.client.0.smithi088.stdout: create 192/38 file after the snapshot 2023-10-16T20:41:15.294 INFO:tasks.workunit.client.0.smithi088.stdout: create 192/39 file after the snapshot 2023-10-16T20:41:15.826 INFO:tasks.workunit.client.0.smithi088.stdout: create 193/.snap/snap-subdir-test 2023-10-16T20:41:16.751 INFO:tasks.workunit.client.0.smithi088.stdout: create 193/30 file after the snapshot 2023-10-16T20:41:16.753 INFO:tasks.workunit.client.0.smithi088.stdout: create 193/31 file after the snapshot 2023-10-16T20:41:16.756 INFO:tasks.workunit.client.0.smithi088.stdout: create 193/32 file after the snapshot 2023-10-16T20:41:16.760 INFO:tasks.workunit.client.0.smithi088.stdout: create 193/33 file after the snapshot 2023-10-16T20:41:16.763 INFO:tasks.workunit.client.0.smithi088.stdout: create 193/34 file after the snapshot 2023-10-16T20:41:16.765 INFO:tasks.workunit.client.0.smithi088.stdout: create 193/35 file after the snapshot 2023-10-16T20:41:16.768 INFO:tasks.workunit.client.0.smithi088.stdout: create 193/36 file after the snapshot 2023-10-16T20:41:16.771 INFO:tasks.workunit.client.0.smithi088.stdout: create 193/37 file after the snapshot 2023-10-16T20:41:16.774 INFO:tasks.workunit.client.0.smithi088.stdout: create 193/38 file after the snapshot 2023-10-16T20:41:16.777 INFO:tasks.workunit.client.0.smithi088.stdout: create 193/39 file after the snapshot 2023-10-16T20:41:16.779 INFO:tasks.workunit.client.0.smithi088.stdout: create 194/.snap/snap-subdir-test 2023-10-16T20:41:16.807 INFO:tasks.workunit.client.0.smithi088.stdout: create 194/30 file after the snapshot 2023-10-16T20:41:16.809 INFO:tasks.workunit.client.0.smithi088.stdout: create 194/31 file after the snapshot 2023-10-16T20:41:16.812 INFO:tasks.workunit.client.0.smithi088.stdout: create 194/32 file after the snapshot 2023-10-16T20:41:16.815 INFO:tasks.workunit.client.0.smithi088.stdout: create 194/33 file after the snapshot 2023-10-16T20:41:16.818 INFO:tasks.workunit.client.0.smithi088.stdout: create 194/34 file after the snapshot 2023-10-16T20:41:16.820 INFO:tasks.workunit.client.0.smithi088.stdout: create 194/35 file after the snapshot 2023-10-16T20:41:16.823 INFO:tasks.workunit.client.0.smithi088.stdout: create 194/36 file after the snapshot 2023-10-16T20:41:16.826 INFO:tasks.workunit.client.0.smithi088.stdout: create 194/37 file after the snapshot 2023-10-16T20:41:16.828 INFO:tasks.workunit.client.0.smithi088.stdout: create 194/38 file after the snapshot 2023-10-16T20:41:16.831 INFO:tasks.workunit.client.0.smithi088.stdout: create 194/39 file after the snapshot 2023-10-16T20:41:16.834 INFO:tasks.workunit.client.0.smithi088.stdout: create 195/.snap/snap-subdir-test 2023-10-16T20:41:16.857 INFO:tasks.workunit.client.0.smithi088.stdout: create 195/30 file after the snapshot 2023-10-16T20:41:17.296 INFO:tasks.workunit.client.0.smithi088.stdout: create 195/31 file after the snapshot 2023-10-16T20:41:17.472 INFO:tasks.workunit.client.0.smithi088.stdout: create 195/32 file after the snapshot 2023-10-16T20:41:17.492 INFO:tasks.workunit.client.0.smithi088.stdout: create 195/33 file after the snapshot 2023-10-16T20:41:17.514 INFO:tasks.workunit.client.0.smithi088.stdout: create 195/34 file after the snapshot 2023-10-16T20:41:17.517 INFO:tasks.workunit.client.0.smithi088.stdout: create 195/35 file after the snapshot 2023-10-16T20:41:17.520 INFO:tasks.workunit.client.0.smithi088.stdout: create 195/36 file after the snapshot 2023-10-16T20:41:17.524 INFO:tasks.workunit.client.0.smithi088.stdout: create 195/37 file after the snapshot 2023-10-16T20:41:17.527 INFO:tasks.workunit.client.0.smithi088.stdout: create 195/38 file after the snapshot 2023-10-16T20:41:17.530 INFO:tasks.workunit.client.0.smithi088.stdout: create 195/39 file after the snapshot 2023-10-16T20:41:17.534 INFO:tasks.workunit.client.0.smithi088.stdout: create 196/.snap/snap-subdir-test 2023-10-16T20:41:18.317 INFO:tasks.workunit.client.0.smithi088.stdout: create 196/30 file after the snapshot 2023-10-16T20:41:18.320 INFO:tasks.workunit.client.0.smithi088.stdout: create 196/31 file after the snapshot 2023-10-16T20:41:18.323 INFO:tasks.workunit.client.0.smithi088.stdout: create 196/32 file after the snapshot 2023-10-16T20:41:18.326 INFO:tasks.workunit.client.0.smithi088.stdout: create 196/33 file after the snapshot 2023-10-16T20:41:18.329 INFO:tasks.workunit.client.0.smithi088.stdout: create 196/34 file after the snapshot 2023-10-16T20:41:18.332 INFO:tasks.workunit.client.0.smithi088.stdout: create 196/35 file after the snapshot 2023-10-16T20:41:18.335 INFO:tasks.workunit.client.0.smithi088.stdout: create 196/36 file after the snapshot 2023-10-16T20:41:18.338 INFO:tasks.workunit.client.0.smithi088.stdout: create 196/37 file after the snapshot 2023-10-16T20:41:18.340 INFO:tasks.workunit.client.0.smithi088.stdout: create 196/38 file after the snapshot 2023-10-16T20:41:18.343 INFO:tasks.workunit.client.0.smithi088.stdout: create 196/39 file after the snapshot 2023-10-16T20:41:18.346 INFO:tasks.workunit.client.0.smithi088.stdout: create 197/.snap/snap-subdir-test 2023-10-16T20:41:18.382 INFO:tasks.workunit.client.0.smithi088.stdout: create 197/30 file after the snapshot 2023-10-16T20:41:18.384 INFO:tasks.workunit.client.0.smithi088.stdout: create 197/31 file after the snapshot 2023-10-16T20:41:18.387 INFO:tasks.workunit.client.0.smithi088.stdout: create 197/32 file after the snapshot 2023-10-16T20:41:18.389 INFO:tasks.workunit.client.0.smithi088.stdout: create 197/33 file after the snapshot 2023-10-16T20:41:18.392 INFO:tasks.workunit.client.0.smithi088.stdout: create 197/34 file after the snapshot 2023-10-16T20:41:18.395 INFO:tasks.workunit.client.0.smithi088.stdout: create 197/35 file after the snapshot 2023-10-16T20:41:18.398 INFO:tasks.workunit.client.0.smithi088.stdout: create 197/36 file after the snapshot 2023-10-16T20:41:18.400 INFO:tasks.workunit.client.0.smithi088.stdout: create 197/37 file after the snapshot 2023-10-16T20:41:18.404 INFO:tasks.workunit.client.0.smithi088.stdout: create 197/38 file after the snapshot 2023-10-16T20:41:18.405 INFO:tasks.workunit.client.0.smithi088.stdout: create 197/39 file after the snapshot 2023-10-16T20:41:18.409 INFO:tasks.workunit.client.0.smithi088.stdout: create 198/.snap/snap-subdir-test 2023-10-16T20:41:18.434 INFO:tasks.workunit.client.0.smithi088.stdout: create 198/30 file after the snapshot 2023-10-16T20:41:18.436 INFO:tasks.workunit.client.0.smithi088.stdout: create 198/31 file after the snapshot 2023-10-16T20:41:18.439 INFO:tasks.workunit.client.0.smithi088.stdout: create 198/32 file after the snapshot 2023-10-16T20:41:18.442 INFO:tasks.workunit.client.0.smithi088.stdout: create 198/33 file after the snapshot 2023-10-16T20:41:18.444 INFO:tasks.workunit.client.0.smithi088.stdout: create 198/34 file after the snapshot 2023-10-16T20:41:18.447 INFO:tasks.workunit.client.0.smithi088.stdout: create 198/35 file after the snapshot 2023-10-16T20:41:18.450 INFO:tasks.workunit.client.0.smithi088.stdout: create 198/36 file after the snapshot 2023-10-16T20:41:18.452 INFO:tasks.workunit.client.0.smithi088.stdout: create 198/37 file after the snapshot 2023-10-16T20:41:18.455 INFO:tasks.workunit.client.0.smithi088.stdout: create 198/38 file after the snapshot 2023-10-16T20:41:18.457 INFO:tasks.workunit.client.0.smithi088.stdout: create 198/39 file after the snapshot 2023-10-16T20:41:18.460 INFO:tasks.workunit.client.0.smithi088.stdout: create 199/.snap/snap-subdir-test 2023-10-16T20:41:18.481 INFO:tasks.workunit.client.0.smithi088.stdout: create 199/30 file after the snapshot 2023-10-16T20:41:18.483 INFO:tasks.workunit.client.0.smithi088.stdout: create 199/31 file after the snapshot 2023-10-16T20:41:18.485 INFO:tasks.workunit.client.0.smithi088.stdout: create 199/32 file after the snapshot 2023-10-16T20:41:18.488 INFO:tasks.workunit.client.0.smithi088.stdout: create 199/33 file after the snapshot 2023-10-16T20:41:18.491 INFO:tasks.workunit.client.0.smithi088.stdout: create 199/34 file after the snapshot 2023-10-16T20:41:18.493 INFO:tasks.workunit.client.0.smithi088.stdout: create 199/35 file after the snapshot 2023-10-16T20:41:18.497 INFO:tasks.workunit.client.0.smithi088.stdout: create 199/36 file after the snapshot 2023-10-16T20:41:18.499 INFO:tasks.workunit.client.0.smithi088.stdout: create 199/37 file after the snapshot 2023-10-16T20:41:18.502 INFO:tasks.workunit.client.0.smithi088.stdout: create 199/38 file after the snapshot 2023-10-16T20:41:18.504 INFO:tasks.workunit.client.0.smithi088.stdout: create 199/39 file after the snapshot 2023-10-16T20:41:18.506 INFO:tasks.workunit.client.0.smithi088.stdout: create 200/.snap/snap-subdir-test 2023-10-16T20:41:18.527 INFO:tasks.workunit.client.0.smithi088.stdout: create 200/30 file after the snapshot 2023-10-16T20:41:18.529 INFO:tasks.workunit.client.0.smithi088.stdout: create 200/31 file after the snapshot 2023-10-16T20:41:18.531 INFO:tasks.workunit.client.0.smithi088.stdout: create 200/32 file after the snapshot 2023-10-16T20:41:18.534 INFO:tasks.workunit.client.0.smithi088.stdout: create 200/33 file after the snapshot 2023-10-16T20:41:18.537 INFO:tasks.workunit.client.0.smithi088.stdout: create 200/34 file after the snapshot 2023-10-16T20:41:18.539 INFO:tasks.workunit.client.0.smithi088.stdout: create 200/35 file after the snapshot 2023-10-16T20:41:18.541 INFO:tasks.workunit.client.0.smithi088.stdout: create 200/36 file after the snapshot 2023-10-16T20:41:18.544 INFO:tasks.workunit.client.0.smithi088.stdout: create 200/37 file after the snapshot 2023-10-16T20:41:18.547 INFO:tasks.workunit.client.0.smithi088.stdout: create 200/38 file after the snapshot 2023-10-16T20:41:18.549 INFO:tasks.workunit.client.0.smithi088.stdout: create 200/39 file after the snapshot 2023-10-16T20:41:18.552 INFO:tasks.workunit.client.0.smithi088.stdout: create 201/.snap/snap-subdir-test 2023-10-16T20:41:18.572 INFO:tasks.workunit.client.0.smithi088.stdout: create 201/30 file after the snapshot 2023-10-16T20:41:18.574 INFO:tasks.workunit.client.0.smithi088.stdout: create 201/31 file after the snapshot 2023-10-16T20:41:18.577 INFO:tasks.workunit.client.0.smithi088.stdout: create 201/32 file after the snapshot 2023-10-16T20:41:18.579 INFO:tasks.workunit.client.0.smithi088.stdout: create 201/33 file after the snapshot 2023-10-16T20:41:18.582 INFO:tasks.workunit.client.0.smithi088.stdout: create 201/34 file after the snapshot 2023-10-16T20:41:18.585 INFO:tasks.workunit.client.0.smithi088.stdout: create 201/35 file after the snapshot 2023-10-16T20:41:18.588 INFO:tasks.workunit.client.0.smithi088.stdout: create 201/36 file after the snapshot 2023-10-16T20:41:18.591 INFO:tasks.workunit.client.0.smithi088.stdout: create 201/37 file after the snapshot 2023-10-16T20:41:18.593 INFO:tasks.workunit.client.0.smithi088.stdout: create 201/38 file after the snapshot 2023-10-16T20:41:18.595 INFO:tasks.workunit.client.0.smithi088.stdout: create 201/39 file after the snapshot 2023-10-16T20:41:18.598 INFO:tasks.workunit.client.0.smithi088.stdout: create 202/.snap/snap-subdir-test 2023-10-16T20:41:19.086 INFO:tasks.workunit.client.0.smithi088.stdout: create 202/30 file after the snapshot 2023-10-16T20:41:19.088 INFO:tasks.workunit.client.0.smithi088.stdout: create 202/31 file after the snapshot 2023-10-16T20:41:19.091 INFO:tasks.workunit.client.0.smithi088.stdout: create 202/32 file after the snapshot 2023-10-16T20:41:19.093 INFO:tasks.workunit.client.0.smithi088.stdout: create 202/33 file after the snapshot 2023-10-16T20:41:19.097 INFO:tasks.workunit.client.0.smithi088.stdout: create 202/34 file after the snapshot 2023-10-16T20:41:19.099 INFO:tasks.workunit.client.0.smithi088.stdout: create 202/35 file after the snapshot 2023-10-16T20:41:19.102 INFO:tasks.workunit.client.0.smithi088.stdout: create 202/36 file after the snapshot 2023-10-16T20:41:19.105 INFO:tasks.workunit.client.0.smithi088.stdout: create 202/37 file after the snapshot 2023-10-16T20:41:19.107 INFO:tasks.workunit.client.0.smithi088.stdout: create 202/38 file after the snapshot 2023-10-16T20:41:19.110 INFO:tasks.workunit.client.0.smithi088.stdout: create 202/39 file after the snapshot 2023-10-16T20:41:19.113 INFO:tasks.workunit.client.0.smithi088.stdout: create 203/.snap/snap-subdir-test 2023-10-16T20:41:19.135 INFO:tasks.workunit.client.0.smithi088.stdout: create 203/30 file after the snapshot 2023-10-16T20:41:19.137 INFO:tasks.workunit.client.0.smithi088.stdout: create 203/31 file after the snapshot 2023-10-16T20:41:19.140 INFO:tasks.workunit.client.0.smithi088.stdout: create 203/32 file after the snapshot 2023-10-16T20:41:19.143 INFO:tasks.workunit.client.0.smithi088.stdout: create 203/33 file after the snapshot 2023-10-16T20:41:19.145 INFO:tasks.workunit.client.0.smithi088.stdout: create 203/34 file after the snapshot 2023-10-16T20:41:19.148 INFO:tasks.workunit.client.0.smithi088.stdout: create 203/35 file after the snapshot 2023-10-16T20:41:19.149 INFO:tasks.workunit.client.0.smithi088.stdout: create 203/36 file after the snapshot 2023-10-16T20:41:19.152 INFO:tasks.workunit.client.0.smithi088.stdout: create 203/37 file after the snapshot 2023-10-16T20:41:19.154 INFO:tasks.workunit.client.0.smithi088.stdout: create 203/38 file after the snapshot 2023-10-16T20:41:19.156 INFO:tasks.workunit.client.0.smithi088.stdout: create 203/39 file after the snapshot 2023-10-16T20:41:19.159 INFO:tasks.workunit.client.0.smithi088.stdout: create 204/.snap/snap-subdir-test 2023-10-16T20:41:19.179 INFO:tasks.workunit.client.0.smithi088.stdout: create 204/30 file after the snapshot 2023-10-16T20:41:19.183 INFO:tasks.workunit.client.0.smithi088.stdout: create 204/31 file after the snapshot 2023-10-16T20:41:19.185 INFO:tasks.workunit.client.0.smithi088.stdout: create 204/32 file after the snapshot 2023-10-16T20:41:19.187 INFO:tasks.workunit.client.0.smithi088.stdout: create 204/33 file after the snapshot 2023-10-16T20:41:19.190 INFO:tasks.workunit.client.0.smithi088.stdout: create 204/34 file after the snapshot 2023-10-16T20:41:19.193 INFO:tasks.workunit.client.0.smithi088.stdout: create 204/35 file after the snapshot 2023-10-16T20:41:19.195 INFO:tasks.workunit.client.0.smithi088.stdout: create 204/36 file after the snapshot 2023-10-16T20:41:19.198 INFO:tasks.workunit.client.0.smithi088.stdout: create 204/37 file after the snapshot 2023-10-16T20:41:19.200 INFO:tasks.workunit.client.0.smithi088.stdout: create 204/38 file after the snapshot 2023-10-16T20:41:19.203 INFO:tasks.workunit.client.0.smithi088.stdout: create 204/39 file after the snapshot 2023-10-16T20:41:19.206 INFO:tasks.workunit.client.0.smithi088.stdout: create 205/.snap/snap-subdir-test 2023-10-16T20:41:19.224 INFO:tasks.workunit.client.0.smithi088.stdout: create 205/30 file after the snapshot 2023-10-16T20:41:19.226 INFO:tasks.workunit.client.0.smithi088.stdout: create 205/31 file after the snapshot 2023-10-16T20:41:19.228 INFO:tasks.workunit.client.0.smithi088.stdout: create 205/32 file after the snapshot 2023-10-16T20:41:19.230 INFO:tasks.workunit.client.0.smithi088.stdout: create 205/33 file after the snapshot 2023-10-16T20:41:19.233 INFO:tasks.workunit.client.0.smithi088.stdout: create 205/34 file after the snapshot 2023-10-16T20:41:19.235 INFO:tasks.workunit.client.0.smithi088.stdout: create 205/35 file after the snapshot 2023-10-16T20:41:19.238 INFO:tasks.workunit.client.0.smithi088.stdout: create 205/36 file after the snapshot 2023-10-16T20:41:19.240 INFO:tasks.workunit.client.0.smithi088.stdout: create 205/37 file after the snapshot 2023-10-16T20:41:19.243 INFO:tasks.workunit.client.0.smithi088.stdout: create 205/38 file after the snapshot 2023-10-16T20:41:19.245 INFO:tasks.workunit.client.0.smithi088.stdout: create 205/39 file after the snapshot 2023-10-16T20:41:19.248 INFO:tasks.workunit.client.0.smithi088.stdout: create 206/.snap/snap-subdir-test 2023-10-16T20:41:19.267 INFO:tasks.workunit.client.0.smithi088.stdout: create 206/30 file after the snapshot 2023-10-16T20:41:19.269 INFO:tasks.workunit.client.0.smithi088.stdout: create 206/31 file after the snapshot 2023-10-16T20:41:19.272 INFO:tasks.workunit.client.0.smithi088.stdout: create 206/32 file after the snapshot 2023-10-16T20:41:19.274 INFO:tasks.workunit.client.0.smithi088.stdout: create 206/33 file after the snapshot 2023-10-16T20:41:19.276 INFO:tasks.workunit.client.0.smithi088.stdout: create 206/34 file after the snapshot 2023-10-16T20:41:19.279 INFO:tasks.workunit.client.0.smithi088.stdout: create 206/35 file after the snapshot 2023-10-16T20:41:19.281 INFO:tasks.workunit.client.0.smithi088.stdout: create 206/36 file after the snapshot 2023-10-16T20:41:19.283 INFO:tasks.workunit.client.0.smithi088.stdout: create 206/37 file after the snapshot 2023-10-16T20:41:19.286 INFO:tasks.workunit.client.0.smithi088.stdout: create 206/38 file after the snapshot 2023-10-16T20:41:19.289 INFO:tasks.workunit.client.0.smithi088.stdout: create 206/39 file after the snapshot 2023-10-16T20:41:19.291 INFO:tasks.workunit.client.0.smithi088.stdout: create 207/.snap/snap-subdir-test 2023-10-16T20:41:19.340 INFO:tasks.workunit.client.0.smithi088.stdout: create 207/30 file after the snapshot 2023-10-16T20:41:19.344 INFO:tasks.workunit.client.0.smithi088.stdout: create 207/31 file after the snapshot 2023-10-16T20:41:19.346 INFO:tasks.workunit.client.0.smithi088.stdout: create 207/32 file after the snapshot 2023-10-16T20:41:19.350 INFO:tasks.workunit.client.0.smithi088.stdout: create 207/33 file after the snapshot 2023-10-16T20:41:19.352 INFO:tasks.workunit.client.0.smithi088.stdout: create 207/34 file after the snapshot 2023-10-16T20:41:19.354 INFO:tasks.workunit.client.0.smithi088.stdout: create 207/35 file after the snapshot 2023-10-16T20:41:19.356 INFO:tasks.workunit.client.0.smithi088.stdout: create 207/36 file after the snapshot 2023-10-16T20:41:19.358 INFO:tasks.workunit.client.0.smithi088.stdout: create 207/37 file after the snapshot 2023-10-16T20:41:19.466 INFO:tasks.workunit.client.0.smithi088.stdout: create 207/38 file after the snapshot 2023-10-16T20:41:19.468 INFO:tasks.workunit.client.0.smithi088.stdout: create 207/39 file after the snapshot 2023-10-16T20:41:19.471 INFO:tasks.workunit.client.0.smithi088.stdout: create 208/.snap/snap-subdir-test 2023-10-16T20:41:19.533 INFO:tasks.workunit.client.0.smithi088.stdout: create 208/30 file after the snapshot 2023-10-16T20:41:19.535 INFO:tasks.workunit.client.0.smithi088.stdout: create 208/31 file after the snapshot 2023-10-16T20:41:19.537 INFO:tasks.workunit.client.0.smithi088.stdout: create 208/32 file after the snapshot 2023-10-16T20:41:19.542 INFO:tasks.workunit.client.0.smithi088.stdout: create 208/33 file after the snapshot 2023-10-16T20:41:19.543 INFO:tasks.workunit.client.0.smithi088.stdout: create 208/34 file after the snapshot 2023-10-16T20:41:19.546 INFO:tasks.workunit.client.0.smithi088.stdout: create 208/35 file after the snapshot 2023-10-16T20:41:19.548 INFO:tasks.workunit.client.0.smithi088.stdout: create 208/36 file after the snapshot 2023-10-16T20:41:19.550 INFO:tasks.workunit.client.0.smithi088.stdout: create 208/37 file after the snapshot 2023-10-16T20:41:19.553 INFO:tasks.workunit.client.0.smithi088.stdout: create 208/38 file after the snapshot 2023-10-16T20:41:19.556 INFO:tasks.workunit.client.0.smithi088.stdout: create 208/39 file after the snapshot 2023-10-16T20:41:19.558 INFO:tasks.workunit.client.0.smithi088.stdout: create 209/.snap/snap-subdir-test 2023-10-16T20:41:19.582 INFO:tasks.workunit.client.0.smithi088.stdout: create 209/30 file after the snapshot 2023-10-16T20:41:19.584 INFO:tasks.workunit.client.0.smithi088.stdout: create 209/31 file after the snapshot 2023-10-16T20:41:19.587 INFO:tasks.workunit.client.0.smithi088.stdout: create 209/32 file after the snapshot 2023-10-16T20:41:19.589 INFO:tasks.workunit.client.0.smithi088.stdout: create 209/33 file after the snapshot 2023-10-16T20:41:19.592 INFO:tasks.workunit.client.0.smithi088.stdout: create 209/34 file after the snapshot 2023-10-16T20:41:19.594 INFO:tasks.workunit.client.0.smithi088.stdout: create 209/35 file after the snapshot 2023-10-16T20:41:19.597 INFO:tasks.workunit.client.0.smithi088.stdout: create 209/36 file after the snapshot 2023-10-16T20:41:19.600 INFO:tasks.workunit.client.0.smithi088.stdout: create 209/37 file after the snapshot 2023-10-16T20:41:19.602 INFO:tasks.workunit.client.0.smithi088.stdout: create 209/38 file after the snapshot 2023-10-16T20:41:19.605 INFO:tasks.workunit.client.0.smithi088.stdout: create 209/39 file after the snapshot 2023-10-16T20:41:19.608 INFO:tasks.workunit.client.0.smithi088.stdout: create 210/.snap/snap-subdir-test 2023-10-16T20:41:19.633 INFO:tasks.workunit.client.0.smithi088.stdout: create 210/30 file after the snapshot 2023-10-16T20:41:19.636 INFO:tasks.workunit.client.0.smithi088.stdout: create 210/31 file after the snapshot 2023-10-16T20:41:19.638 INFO:tasks.workunit.client.0.smithi088.stdout: create 210/32 file after the snapshot 2023-10-16T20:41:19.641 INFO:tasks.workunit.client.0.smithi088.stdout: create 210/33 file after the snapshot 2023-10-16T20:41:19.643 INFO:tasks.workunit.client.0.smithi088.stdout: create 210/34 file after the snapshot 2023-10-16T20:41:19.646 INFO:tasks.workunit.client.0.smithi088.stdout: create 210/35 file after the snapshot 2023-10-16T20:41:19.648 INFO:tasks.workunit.client.0.smithi088.stdout: create 210/36 file after the snapshot 2023-10-16T20:41:19.651 INFO:tasks.workunit.client.0.smithi088.stdout: create 210/37 file after the snapshot 2023-10-16T20:41:19.653 INFO:tasks.workunit.client.0.smithi088.stdout: create 210/38 file after the snapshot 2023-10-16T20:41:19.655 INFO:tasks.workunit.client.0.smithi088.stdout: create 210/39 file after the snapshot 2023-10-16T20:41:19.658 INFO:tasks.workunit.client.0.smithi088.stdout: create 211/.snap/snap-subdir-test 2023-10-16T20:41:19.676 INFO:tasks.workunit.client.0.smithi088.stdout: create 211/30 file after the snapshot 2023-10-16T20:41:19.679 INFO:tasks.workunit.client.0.smithi088.stdout: create 211/31 file after the snapshot 2023-10-16T20:41:19.681 INFO:tasks.workunit.client.0.smithi088.stdout: create 211/32 file after the snapshot 2023-10-16T20:41:19.683 INFO:tasks.workunit.client.0.smithi088.stdout: create 211/33 file after the snapshot 2023-10-16T20:41:19.686 INFO:tasks.workunit.client.0.smithi088.stdout: create 211/34 file after the snapshot 2023-10-16T20:41:19.688 INFO:tasks.workunit.client.0.smithi088.stdout: create 211/35 file after the snapshot 2023-10-16T20:41:19.690 INFO:tasks.workunit.client.0.smithi088.stdout: create 211/36 file after the snapshot 2023-10-16T20:41:19.693 INFO:tasks.workunit.client.0.smithi088.stdout: create 211/37 file after the snapshot 2023-10-16T20:41:19.695 INFO:tasks.workunit.client.0.smithi088.stdout: create 211/38 file after the snapshot 2023-10-16T20:41:19.698 INFO:tasks.workunit.client.0.smithi088.stdout: create 211/39 file after the snapshot 2023-10-16T20:41:19.700 INFO:tasks.workunit.client.0.smithi088.stdout: create 212/.snap/snap-subdir-test 2023-10-16T20:41:19.720 INFO:tasks.workunit.client.0.smithi088.stdout: create 212/30 file after the snapshot 2023-10-16T20:41:19.722 INFO:tasks.workunit.client.0.smithi088.stdout: create 212/31 file after the snapshot 2023-10-16T20:41:19.724 INFO:tasks.workunit.client.0.smithi088.stdout: create 212/32 file after the snapshot 2023-10-16T20:41:19.727 INFO:tasks.workunit.client.0.smithi088.stdout: create 212/33 file after the snapshot 2023-10-16T20:41:19.729 INFO:tasks.workunit.client.0.smithi088.stdout: create 212/34 file after the snapshot 2023-10-16T20:41:19.731 INFO:tasks.workunit.client.0.smithi088.stdout: create 212/35 file after the snapshot 2023-10-16T20:41:19.733 INFO:tasks.workunit.client.0.smithi088.stdout: create 212/36 file after the snapshot 2023-10-16T20:41:19.736 INFO:tasks.workunit.client.0.smithi088.stdout: create 212/37 file after the snapshot 2023-10-16T20:41:19.738 INFO:tasks.workunit.client.0.smithi088.stdout: create 212/38 file after the snapshot 2023-10-16T20:41:19.741 INFO:tasks.workunit.client.0.smithi088.stdout: create 212/39 file after the snapshot 2023-10-16T20:41:19.745 INFO:tasks.workunit.client.0.smithi088.stdout: create 213/.snap/snap-subdir-test 2023-10-16T20:41:19.763 INFO:tasks.workunit.client.0.smithi088.stdout: create 213/30 file after the snapshot 2023-10-16T20:41:19.765 INFO:tasks.workunit.client.0.smithi088.stdout: create 213/31 file after the snapshot 2023-10-16T20:41:19.767 INFO:tasks.workunit.client.0.smithi088.stdout: create 213/32 file after the snapshot 2023-10-16T20:41:19.770 INFO:tasks.workunit.client.0.smithi088.stdout: create 213/33 file after the snapshot 2023-10-16T20:41:19.773 INFO:tasks.workunit.client.0.smithi088.stdout: create 213/34 file after the snapshot 2023-10-16T20:41:19.776 INFO:tasks.workunit.client.0.smithi088.stdout: create 213/35 file after the snapshot 2023-10-16T20:41:19.779 INFO:tasks.workunit.client.0.smithi088.stdout: create 213/36 file after the snapshot 2023-10-16T20:41:19.781 INFO:tasks.workunit.client.0.smithi088.stdout: create 213/37 file after the snapshot 2023-10-16T20:41:19.783 INFO:tasks.workunit.client.0.smithi088.stdout: create 213/38 file after the snapshot 2023-10-16T20:41:19.786 INFO:tasks.workunit.client.0.smithi088.stdout: create 213/39 file after the snapshot 2023-10-16T20:41:19.790 INFO:tasks.workunit.client.0.smithi088.stdout: create 214/.snap/snap-subdir-test 2023-10-16T20:41:19.815 INFO:tasks.workunit.client.0.smithi088.stdout: create 214/30 file after the snapshot 2023-10-16T20:41:19.817 INFO:tasks.workunit.client.0.smithi088.stdout: create 214/31 file after the snapshot 2023-10-16T20:41:19.819 INFO:tasks.workunit.client.0.smithi088.stdout: create 214/32 file after the snapshot 2023-10-16T20:41:19.822 INFO:tasks.workunit.client.0.smithi088.stdout: create 214/33 file after the snapshot 2023-10-16T20:41:19.824 INFO:tasks.workunit.client.0.smithi088.stdout: create 214/34 file after the snapshot 2023-10-16T20:41:19.827 INFO:tasks.workunit.client.0.smithi088.stdout: create 214/35 file after the snapshot 2023-10-16T20:41:19.829 INFO:tasks.workunit.client.0.smithi088.stdout: create 214/36 file after the snapshot 2023-10-16T20:41:19.832 INFO:tasks.workunit.client.0.smithi088.stdout: create 214/37 file after the snapshot 2023-10-16T20:41:19.834 INFO:tasks.workunit.client.0.smithi088.stdout: create 214/38 file after the snapshot 2023-10-16T20:41:19.836 INFO:tasks.workunit.client.0.smithi088.stdout: create 214/39 file after the snapshot 2023-10-16T20:41:19.839 INFO:tasks.workunit.client.0.smithi088.stdout: create 215/.snap/snap-subdir-test 2023-10-16T20:41:19.858 INFO:tasks.workunit.client.0.smithi088.stdout: create 215/30 file after the snapshot 2023-10-16T20:41:19.860 INFO:tasks.workunit.client.0.smithi088.stdout: create 215/31 file after the snapshot 2023-10-16T20:41:19.862 INFO:tasks.workunit.client.0.smithi088.stdout: create 215/32 file after the snapshot 2023-10-16T20:41:19.865 INFO:tasks.workunit.client.0.smithi088.stdout: create 215/33 file after the snapshot 2023-10-16T20:41:19.868 INFO:tasks.workunit.client.0.smithi088.stdout: create 215/34 file after the snapshot 2023-10-16T20:41:19.870 INFO:tasks.workunit.client.0.smithi088.stdout: create 215/35 file after the snapshot 2023-10-16T20:41:19.873 INFO:tasks.workunit.client.0.smithi088.stdout: create 215/36 file after the snapshot 2023-10-16T20:41:19.875 INFO:tasks.workunit.client.0.smithi088.stdout: create 215/37 file after the snapshot 2023-10-16T20:41:19.878 INFO:tasks.workunit.client.0.smithi088.stdout: create 215/38 file after the snapshot 2023-10-16T20:41:19.881 INFO:tasks.workunit.client.0.smithi088.stdout: create 215/39 file after the snapshot 2023-10-16T20:41:19.883 INFO:tasks.workunit.client.0.smithi088.stdout: create 216/.snap/snap-subdir-test 2023-10-16T20:41:19.900 INFO:tasks.workunit.client.0.smithi088.stdout: create 216/30 file after the snapshot 2023-10-16T20:41:19.902 INFO:tasks.workunit.client.0.smithi088.stdout: create 216/31 file after the snapshot 2023-10-16T20:41:19.904 INFO:tasks.workunit.client.0.smithi088.stdout: create 216/32 file after the snapshot 2023-10-16T20:41:19.907 INFO:tasks.workunit.client.0.smithi088.stdout: create 216/33 file after the snapshot 2023-10-16T20:41:19.910 INFO:tasks.workunit.client.0.smithi088.stdout: create 216/34 file after the snapshot 2023-10-16T20:41:19.912 INFO:tasks.workunit.client.0.smithi088.stdout: create 216/35 file after the snapshot 2023-10-16T20:41:19.914 INFO:tasks.workunit.client.0.smithi088.stdout: create 216/36 file after the snapshot 2023-10-16T20:41:19.917 INFO:tasks.workunit.client.0.smithi088.stdout: create 216/37 file after the snapshot 2023-10-16T20:41:19.920 INFO:tasks.workunit.client.0.smithi088.stdout: create 216/38 file after the snapshot 2023-10-16T20:41:19.922 INFO:tasks.workunit.client.0.smithi088.stdout: create 216/39 file after the snapshot 2023-10-16T20:41:19.927 INFO:tasks.workunit.client.0.smithi088.stdout: create 217/.snap/snap-subdir-test 2023-10-16T20:41:19.944 INFO:tasks.workunit.client.0.smithi088.stdout: create 217/30 file after the snapshot 2023-10-16T20:41:19.946 INFO:tasks.workunit.client.0.smithi088.stdout: create 217/31 file after the snapshot 2023-10-16T20:41:19.949 INFO:tasks.workunit.client.0.smithi088.stdout: create 217/32 file after the snapshot 2023-10-16T20:41:19.951 INFO:tasks.workunit.client.0.smithi088.stdout: create 217/33 file after the snapshot 2023-10-16T20:41:19.953 INFO:tasks.workunit.client.0.smithi088.stdout: create 217/34 file after the snapshot 2023-10-16T20:41:19.956 INFO:tasks.workunit.client.0.smithi088.stdout: create 217/35 file after the snapshot 2023-10-16T20:41:19.958 INFO:tasks.workunit.client.0.smithi088.stdout: create 217/36 file after the snapshot 2023-10-16T20:41:19.960 INFO:tasks.workunit.client.0.smithi088.stdout: create 217/37 file after the snapshot 2023-10-16T20:41:19.962 INFO:tasks.workunit.client.0.smithi088.stdout: create 217/38 file after the snapshot 2023-10-16T20:41:19.965 INFO:tasks.workunit.client.0.smithi088.stdout: create 217/39 file after the snapshot 2023-10-16T20:41:19.967 INFO:tasks.workunit.client.0.smithi088.stdout: create 218/.snap/snap-subdir-test 2023-10-16T20:41:19.987 INFO:tasks.workunit.client.0.smithi088.stdout: create 218/30 file after the snapshot 2023-10-16T20:41:19.989 INFO:tasks.workunit.client.0.smithi088.stdout: create 218/31 file after the snapshot 2023-10-16T20:41:19.991 INFO:tasks.workunit.client.0.smithi088.stdout: create 218/32 file after the snapshot 2023-10-16T20:41:19.994 INFO:tasks.workunit.client.0.smithi088.stdout: create 218/33 file after the snapshot 2023-10-16T20:41:19.996 INFO:tasks.workunit.client.0.smithi088.stdout: create 218/34 file after the snapshot 2023-10-16T20:41:19.998 INFO:tasks.workunit.client.0.smithi088.stdout: create 218/35 file after the snapshot 2023-10-16T20:41:20.001 INFO:tasks.workunit.client.0.smithi088.stdout: create 218/36 file after the snapshot 2023-10-16T20:41:20.004 INFO:tasks.workunit.client.0.smithi088.stdout: create 218/37 file after the snapshot 2023-10-16T20:41:20.007 INFO:tasks.workunit.client.0.smithi088.stdout: create 218/38 file after the snapshot 2023-10-16T20:41:20.009 INFO:tasks.workunit.client.0.smithi088.stdout: create 218/39 file after the snapshot 2023-10-16T20:41:20.012 INFO:tasks.workunit.client.0.smithi088.stdout: create 219/.snap/snap-subdir-test 2023-10-16T20:41:20.030 INFO:tasks.workunit.client.0.smithi088.stdout: create 219/30 file after the snapshot 2023-10-16T20:41:20.032 INFO:tasks.workunit.client.0.smithi088.stdout: create 219/31 file after the snapshot 2023-10-16T20:41:20.034 INFO:tasks.workunit.client.0.smithi088.stdout: create 219/32 file after the snapshot 2023-10-16T20:41:20.037 INFO:tasks.workunit.client.0.smithi088.stdout: create 219/33 file after the snapshot 2023-10-16T20:41:20.040 INFO:tasks.workunit.client.0.smithi088.stdout: create 219/34 file after the snapshot 2023-10-16T20:41:20.042 INFO:tasks.workunit.client.0.smithi088.stdout: create 219/35 file after the snapshot 2023-10-16T20:41:20.044 INFO:tasks.workunit.client.0.smithi088.stdout: create 219/36 file after the snapshot 2023-10-16T20:41:20.047 INFO:tasks.workunit.client.0.smithi088.stdout: create 219/37 file after the snapshot 2023-10-16T20:41:20.049 INFO:tasks.workunit.client.0.smithi088.stdout: create 219/38 file after the snapshot 2023-10-16T20:41:20.052 INFO:tasks.workunit.client.0.smithi088.stdout: create 219/39 file after the snapshot 2023-10-16T20:41:20.054 INFO:tasks.workunit.client.0.smithi088.stdout: create 220/.snap/snap-subdir-test 2023-10-16T20:41:20.075 INFO:tasks.workunit.client.0.smithi088.stdout: create 220/30 file after the snapshot 2023-10-16T20:41:20.077 INFO:tasks.workunit.client.0.smithi088.stdout: create 220/31 file after the snapshot 2023-10-16T20:41:20.079 INFO:tasks.workunit.client.0.smithi088.stdout: create 220/32 file after the snapshot 2023-10-16T20:41:20.081 INFO:tasks.workunit.client.0.smithi088.stdout: create 220/33 file after the snapshot 2023-10-16T20:41:20.084 INFO:tasks.workunit.client.0.smithi088.stdout: create 220/34 file after the snapshot 2023-10-16T20:41:20.086 INFO:tasks.workunit.client.0.smithi088.stdout: create 220/35 file after the snapshot 2023-10-16T20:41:20.089 INFO:tasks.workunit.client.0.smithi088.stdout: create 220/36 file after the snapshot 2023-10-16T20:41:20.091 INFO:tasks.workunit.client.0.smithi088.stdout: create 220/37 file after the snapshot 2023-10-16T20:41:20.094 INFO:tasks.workunit.client.0.smithi088.stdout: create 220/38 file after the snapshot 2023-10-16T20:41:20.096 INFO:tasks.workunit.client.0.smithi088.stdout: create 220/39 file after the snapshot 2023-10-16T20:41:20.099 INFO:tasks.workunit.client.0.smithi088.stdout: create 221/.snap/snap-subdir-test 2023-10-16T20:41:20.118 INFO:tasks.workunit.client.0.smithi088.stdout: create 221/30 file after the snapshot 2023-10-16T20:41:20.120 INFO:tasks.workunit.client.0.smithi088.stdout: create 221/31 file after the snapshot 2023-10-16T20:41:20.123 INFO:tasks.workunit.client.0.smithi088.stdout: create 221/32 file after the snapshot 2023-10-16T20:41:20.125 INFO:tasks.workunit.client.0.smithi088.stdout: create 221/33 file after the snapshot 2023-10-16T20:41:20.128 INFO:tasks.workunit.client.0.smithi088.stdout: create 221/34 file after the snapshot 2023-10-16T20:41:20.130 INFO:tasks.workunit.client.0.smithi088.stdout: create 221/35 file after the snapshot 2023-10-16T20:41:20.132 INFO:tasks.workunit.client.0.smithi088.stdout: create 221/36 file after the snapshot 2023-10-16T20:41:20.135 INFO:tasks.workunit.client.0.smithi088.stdout: create 221/37 file after the snapshot 2023-10-16T20:41:20.138 INFO:tasks.workunit.client.0.smithi088.stdout: create 221/38 file after the snapshot 2023-10-16T20:41:20.140 INFO:tasks.workunit.client.0.smithi088.stdout: create 221/39 file after the snapshot 2023-10-16T20:41:20.142 INFO:tasks.workunit.client.0.smithi088.stdout: create 222/.snap/snap-subdir-test 2023-10-16T20:41:20.166 INFO:tasks.workunit.client.0.smithi088.stdout: create 222/30 file after the snapshot 2023-10-16T20:41:20.169 INFO:tasks.workunit.client.0.smithi088.stdout: create 222/31 file after the snapshot 2023-10-16T20:41:20.171 INFO:tasks.workunit.client.0.smithi088.stdout: create 222/32 file after the snapshot 2023-10-16T20:41:20.173 INFO:tasks.workunit.client.0.smithi088.stdout: create 222/33 file after the snapshot 2023-10-16T20:41:20.176 INFO:tasks.workunit.client.0.smithi088.stdout: create 222/34 file after the snapshot 2023-10-16T20:41:20.180 INFO:tasks.workunit.client.0.smithi088.stdout: create 222/35 file after the snapshot 2023-10-16T20:41:20.183 INFO:tasks.workunit.client.0.smithi088.stdout: create 222/36 file after the snapshot 2023-10-16T20:41:20.186 INFO:tasks.workunit.client.0.smithi088.stdout: create 222/37 file after the snapshot 2023-10-16T20:41:20.188 INFO:tasks.workunit.client.0.smithi088.stdout: create 222/38 file after the snapshot 2023-10-16T20:41:20.191 INFO:tasks.workunit.client.0.smithi088.stdout: create 222/39 file after the snapshot 2023-10-16T20:41:20.193 INFO:tasks.workunit.client.0.smithi088.stdout: create 223/.snap/snap-subdir-test 2023-10-16T20:41:20.213 INFO:tasks.workunit.client.0.smithi088.stdout: create 223/30 file after the snapshot 2023-10-16T20:41:20.216 INFO:tasks.workunit.client.0.smithi088.stdout: create 223/31 file after the snapshot 2023-10-16T20:41:20.218 INFO:tasks.workunit.client.0.smithi088.stdout: create 223/32 file after the snapshot 2023-10-16T20:41:20.220 INFO:tasks.workunit.client.0.smithi088.stdout: create 223/33 file after the snapshot 2023-10-16T20:41:20.223 INFO:tasks.workunit.client.0.smithi088.stdout: create 223/34 file after the snapshot 2023-10-16T20:41:20.225 INFO:tasks.workunit.client.0.smithi088.stdout: create 223/35 file after the snapshot 2023-10-16T20:41:20.228 INFO:tasks.workunit.client.0.smithi088.stdout: create 223/36 file after the snapshot 2023-10-16T20:41:20.230 INFO:tasks.workunit.client.0.smithi088.stdout: create 223/37 file after the snapshot 2023-10-16T20:41:20.233 INFO:tasks.workunit.client.0.smithi088.stdout: create 223/38 file after the snapshot 2023-10-16T20:41:20.235 INFO:tasks.workunit.client.0.smithi088.stdout: create 223/39 file after the snapshot 2023-10-16T20:41:20.237 INFO:tasks.workunit.client.0.smithi088.stdout: create 224/.snap/snap-subdir-test 2023-10-16T20:41:20.257 INFO:tasks.workunit.client.0.smithi088.stdout: create 224/30 file after the snapshot 2023-10-16T20:41:20.259 INFO:tasks.workunit.client.0.smithi088.stdout: create 224/31 file after the snapshot 2023-10-16T20:41:20.262 INFO:tasks.workunit.client.0.smithi088.stdout: create 224/32 file after the snapshot 2023-10-16T20:41:20.264 INFO:tasks.workunit.client.0.smithi088.stdout: create 224/33 file after the snapshot 2023-10-16T20:41:20.267 INFO:tasks.workunit.client.0.smithi088.stdout: create 224/34 file after the snapshot 2023-10-16T20:41:20.269 INFO:tasks.workunit.client.0.smithi088.stdout: create 224/35 file after the snapshot 2023-10-16T20:41:20.272 INFO:tasks.workunit.client.0.smithi088.stdout: create 224/36 file after the snapshot 2023-10-16T20:41:20.275 INFO:tasks.workunit.client.0.smithi088.stdout: create 224/37 file after the snapshot 2023-10-16T20:41:20.278 INFO:tasks.workunit.client.0.smithi088.stdout: create 224/38 file after the snapshot 2023-10-16T20:41:20.280 INFO:tasks.workunit.client.0.smithi088.stdout: create 224/39 file after the snapshot 2023-10-16T20:41:20.283 INFO:tasks.workunit.client.0.smithi088.stdout: create 225/.snap/snap-subdir-test 2023-10-16T20:41:20.302 INFO:tasks.workunit.client.0.smithi088.stdout: create 225/30 file after the snapshot 2023-10-16T20:41:20.304 INFO:tasks.workunit.client.0.smithi088.stdout: create 225/31 file after the snapshot 2023-10-16T20:41:20.306 INFO:tasks.workunit.client.0.smithi088.stdout: create 225/32 file after the snapshot 2023-10-16T20:41:20.314 INFO:tasks.workunit.client.0.smithi088.stdout: create 225/33 file after the snapshot 2023-10-16T20:41:20.321 INFO:tasks.workunit.client.0.smithi088.stdout: create 225/34 file after the snapshot 2023-10-16T20:41:20.323 INFO:tasks.workunit.client.0.smithi088.stdout: create 225/35 file after the snapshot 2023-10-16T20:41:20.327 INFO:tasks.workunit.client.0.smithi088.stdout: create 225/36 file after the snapshot 2023-10-16T20:41:20.332 INFO:tasks.workunit.client.0.smithi088.stdout: create 225/37 file after the snapshot 2023-10-16T20:41:20.334 INFO:tasks.workunit.client.0.smithi088.stdout: create 225/38 file after the snapshot 2023-10-16T20:41:20.337 INFO:tasks.workunit.client.0.smithi088.stdout: create 225/39 file after the snapshot 2023-10-16T20:41:20.340 INFO:tasks.workunit.client.0.smithi088.stdout: create 226/.snap/snap-subdir-test 2023-10-16T20:41:20.726 INFO:tasks.workunit.client.0.smithi088.stdout: create 226/30 file after the snapshot 2023-10-16T20:41:20.728 INFO:tasks.workunit.client.0.smithi088.stdout: create 226/31 file after the snapshot 2023-10-16T20:41:20.730 INFO:tasks.workunit.client.0.smithi088.stdout: create 226/32 file after the snapshot 2023-10-16T20:41:20.733 INFO:tasks.workunit.client.0.smithi088.stdout: create 226/33 file after the snapshot 2023-10-16T20:41:20.736 INFO:tasks.workunit.client.0.smithi088.stdout: create 226/34 file after the snapshot 2023-10-16T20:41:20.739 INFO:tasks.workunit.client.0.smithi088.stdout: create 226/35 file after the snapshot 2023-10-16T20:41:20.742 INFO:tasks.workunit.client.0.smithi088.stdout: create 226/36 file after the snapshot 2023-10-16T20:41:20.744 INFO:tasks.workunit.client.0.smithi088.stdout: create 226/37 file after the snapshot 2023-10-16T20:41:20.747 INFO:tasks.workunit.client.0.smithi088.stdout: create 226/38 file after the snapshot 2023-10-16T20:41:20.749 INFO:tasks.workunit.client.0.smithi088.stdout: create 226/39 file after the snapshot 2023-10-16T20:41:20.752 INFO:tasks.workunit.client.0.smithi088.stdout: create 227/.snap/snap-subdir-test 2023-10-16T20:41:20.775 INFO:tasks.workunit.client.0.smithi088.stdout: create 227/30 file after the snapshot 2023-10-16T20:41:20.777 INFO:tasks.workunit.client.0.smithi088.stdout: create 227/31 file after the snapshot 2023-10-16T20:41:20.779 INFO:tasks.workunit.client.0.smithi088.stdout: create 227/32 file after the snapshot 2023-10-16T20:41:20.782 INFO:tasks.workunit.client.0.smithi088.stdout: create 227/33 file after the snapshot 2023-10-16T20:41:20.784 INFO:tasks.workunit.client.0.smithi088.stdout: create 227/34 file after the snapshot 2023-10-16T20:41:20.787 INFO:tasks.workunit.client.0.smithi088.stdout: create 227/35 file after the snapshot 2023-10-16T20:41:20.789 INFO:tasks.workunit.client.0.smithi088.stdout: create 227/36 file after the snapshot 2023-10-16T20:41:20.791 INFO:tasks.workunit.client.0.smithi088.stdout: create 227/37 file after the snapshot 2023-10-16T20:41:20.794 INFO:tasks.workunit.client.0.smithi088.stdout: create 227/38 file after the snapshot 2023-10-16T20:41:20.797 INFO:tasks.workunit.client.0.smithi088.stdout: create 227/39 file after the snapshot 2023-10-16T20:41:20.799 INFO:tasks.workunit.client.0.smithi088.stdout: create 228/.snap/snap-subdir-test 2023-10-16T20:41:20.816 INFO:tasks.workunit.client.0.smithi088.stdout: create 228/30 file after the snapshot 2023-10-16T20:41:20.818 INFO:tasks.workunit.client.0.smithi088.stdout: create 228/31 file after the snapshot 2023-10-16T20:41:20.820 INFO:tasks.workunit.client.0.smithi088.stdout: create 228/32 file after the snapshot 2023-10-16T20:41:20.823 INFO:tasks.workunit.client.0.smithi088.stdout: create 228/33 file after the snapshot 2023-10-16T20:41:20.826 INFO:tasks.workunit.client.0.smithi088.stdout: create 228/34 file after the snapshot 2023-10-16T20:41:20.828 INFO:tasks.workunit.client.0.smithi088.stdout: create 228/35 file after the snapshot 2023-10-16T20:41:20.830 INFO:tasks.workunit.client.0.smithi088.stdout: create 228/36 file after the snapshot 2023-10-16T20:41:20.833 INFO:tasks.workunit.client.0.smithi088.stdout: create 228/37 file after the snapshot 2023-10-16T20:41:20.836 INFO:tasks.workunit.client.0.smithi088.stdout: create 228/38 file after the snapshot 2023-10-16T20:41:20.838 INFO:tasks.workunit.client.0.smithi088.stdout: create 228/39 file after the snapshot 2023-10-16T20:41:20.840 INFO:tasks.workunit.client.0.smithi088.stdout: create 229/.snap/snap-subdir-test 2023-10-16T20:41:20.860 INFO:tasks.workunit.client.0.smithi088.stdout: create 229/30 file after the snapshot 2023-10-16T20:41:20.862 INFO:tasks.workunit.client.0.smithi088.stdout: create 229/31 file after the snapshot 2023-10-16T20:41:20.864 INFO:tasks.workunit.client.0.smithi088.stdout: create 229/32 file after the snapshot 2023-10-16T20:41:20.877 INFO:tasks.workunit.client.0.smithi088.stdout: create 229/33 file after the snapshot 2023-10-16T20:41:20.879 INFO:tasks.workunit.client.0.smithi088.stdout: create 229/34 file after the snapshot 2023-10-16T20:41:20.882 INFO:tasks.workunit.client.0.smithi088.stdout: create 229/35 file after the snapshot 2023-10-16T20:41:20.885 INFO:tasks.workunit.client.0.smithi088.stdout: create 229/36 file after the snapshot 2023-10-16T20:41:20.887 INFO:tasks.workunit.client.0.smithi088.stdout: create 229/37 file after the snapshot 2023-10-16T20:41:20.890 INFO:tasks.workunit.client.0.smithi088.stdout: create 229/38 file after the snapshot 2023-10-16T20:41:20.892 INFO:tasks.workunit.client.0.smithi088.stdout: create 229/39 file after the snapshot 2023-10-16T20:41:20.895 INFO:tasks.workunit.client.0.smithi088.stdout: create 230/.snap/snap-subdir-test 2023-10-16T20:41:20.916 INFO:tasks.workunit.client.0.smithi088.stdout: create 230/30 file after the snapshot 2023-10-16T20:41:20.918 INFO:tasks.workunit.client.0.smithi088.stdout: create 230/31 file after the snapshot 2023-10-16T20:41:20.920 INFO:tasks.workunit.client.0.smithi088.stdout: create 230/32 file after the snapshot 2023-10-16T20:41:20.923 INFO:tasks.workunit.client.0.smithi088.stdout: create 230/33 file after the snapshot 2023-10-16T20:41:20.926 INFO:tasks.workunit.client.0.smithi088.stdout: create 230/34 file after the snapshot 2023-10-16T20:41:20.928 INFO:tasks.workunit.client.0.smithi088.stdout: create 230/35 file after the snapshot 2023-10-16T20:41:20.931 INFO:tasks.workunit.client.0.smithi088.stdout: create 230/36 file after the snapshot 2023-10-16T20:41:20.933 INFO:tasks.workunit.client.0.smithi088.stdout: create 230/37 file after the snapshot 2023-10-16T20:41:20.936 INFO:tasks.workunit.client.0.smithi088.stdout: create 230/38 file after the snapshot 2023-10-16T20:41:20.938 INFO:tasks.workunit.client.0.smithi088.stdout: create 230/39 file after the snapshot 2023-10-16T20:41:20.940 INFO:tasks.workunit.client.0.smithi088.stdout: create 231/.snap/snap-subdir-test 2023-10-16T20:41:20.961 INFO:tasks.workunit.client.0.smithi088.stdout: create 231/30 file after the snapshot 2023-10-16T20:41:20.963 INFO:tasks.workunit.client.0.smithi088.stdout: create 231/31 file after the snapshot 2023-10-16T20:41:20.965 INFO:tasks.workunit.client.0.smithi088.stdout: create 231/32 file after the snapshot 2023-10-16T20:41:20.967 INFO:tasks.workunit.client.0.smithi088.stdout: create 231/33 file after the snapshot 2023-10-16T20:41:20.970 INFO:tasks.workunit.client.0.smithi088.stdout: create 231/34 file after the snapshot 2023-10-16T20:41:20.973 INFO:tasks.workunit.client.0.smithi088.stdout: create 231/35 file after the snapshot 2023-10-16T20:41:20.975 INFO:tasks.workunit.client.0.smithi088.stdout: create 231/36 file after the snapshot 2023-10-16T20:41:20.978 INFO:tasks.workunit.client.0.smithi088.stdout: create 231/37 file after the snapshot 2023-10-16T20:41:20.980 INFO:tasks.workunit.client.0.smithi088.stdout: create 231/38 file after the snapshot 2023-10-16T20:41:20.983 INFO:tasks.workunit.client.0.smithi088.stdout: create 231/39 file after the snapshot 2023-10-16T20:41:20.986 INFO:tasks.workunit.client.0.smithi088.stdout: create 232/.snap/snap-subdir-test 2023-10-16T20:41:21.005 INFO:tasks.workunit.client.0.smithi088.stdout: create 232/30 file after the snapshot 2023-10-16T20:41:21.007 INFO:tasks.workunit.client.0.smithi088.stdout: create 232/31 file after the snapshot 2023-10-16T20:41:21.010 INFO:tasks.workunit.client.0.smithi088.stdout: create 232/32 file after the snapshot 2023-10-16T20:41:21.012 INFO:tasks.workunit.client.0.smithi088.stdout: create 232/33 file after the snapshot 2023-10-16T20:41:21.015 INFO:tasks.workunit.client.0.smithi088.stdout: create 232/34 file after the snapshot 2023-10-16T20:41:21.017 INFO:tasks.workunit.client.0.smithi088.stdout: create 232/35 file after the snapshot 2023-10-16T20:41:21.020 INFO:tasks.workunit.client.0.smithi088.stdout: create 232/36 file after the snapshot 2023-10-16T20:41:21.023 INFO:tasks.workunit.client.0.smithi088.stdout: create 232/37 file after the snapshot 2023-10-16T20:41:21.025 INFO:tasks.workunit.client.0.smithi088.stdout: create 232/38 file after the snapshot 2023-10-16T20:41:21.028 INFO:tasks.workunit.client.0.smithi088.stdout: create 232/39 file after the snapshot 2023-10-16T20:41:21.030 INFO:tasks.workunit.client.0.smithi088.stdout: create 233/.snap/snap-subdir-test 2023-10-16T20:41:21.050 INFO:tasks.workunit.client.0.smithi088.stdout: create 233/30 file after the snapshot 2023-10-16T20:41:21.052 INFO:tasks.workunit.client.0.smithi088.stdout: create 233/31 file after the snapshot 2023-10-16T20:41:21.054 INFO:tasks.workunit.client.0.smithi088.stdout: create 233/32 file after the snapshot 2023-10-16T20:41:21.057 INFO:tasks.workunit.client.0.smithi088.stdout: create 233/33 file after the snapshot 2023-10-16T20:41:21.059 INFO:tasks.workunit.client.0.smithi088.stdout: create 233/34 file after the snapshot 2023-10-16T20:41:21.062 INFO:tasks.workunit.client.0.smithi088.stdout: create 233/35 file after the snapshot 2023-10-16T20:41:21.064 INFO:tasks.workunit.client.0.smithi088.stdout: create 233/36 file after the snapshot 2023-10-16T20:41:21.066 INFO:tasks.workunit.client.0.smithi088.stdout: create 233/37 file after the snapshot 2023-10-16T20:41:21.069 INFO:tasks.workunit.client.0.smithi088.stdout: create 233/38 file after the snapshot 2023-10-16T20:41:21.071 INFO:tasks.workunit.client.0.smithi088.stdout: create 233/39 file after the snapshot 2023-10-16T20:41:21.074 INFO:tasks.workunit.client.0.smithi088.stdout: create 234/.snap/snap-subdir-test 2023-10-16T20:41:21.093 INFO:tasks.workunit.client.0.smithi088.stdout: create 234/30 file after the snapshot 2023-10-16T20:41:21.095 INFO:tasks.workunit.client.0.smithi088.stdout: create 234/31 file after the snapshot 2023-10-16T20:41:21.097 INFO:tasks.workunit.client.0.smithi088.stdout: create 234/32 file after the snapshot 2023-10-16T20:41:21.099 INFO:tasks.workunit.client.0.smithi088.stdout: create 234/33 file after the snapshot 2023-10-16T20:41:21.102 INFO:tasks.workunit.client.0.smithi088.stdout: create 234/34 file after the snapshot 2023-10-16T20:41:21.104 INFO:tasks.workunit.client.0.smithi088.stdout: create 234/35 file after the snapshot 2023-10-16T20:41:21.107 INFO:tasks.workunit.client.0.smithi088.stdout: create 234/36 file after the snapshot 2023-10-16T20:41:21.109 INFO:tasks.workunit.client.0.smithi088.stdout: create 234/37 file after the snapshot 2023-10-16T20:41:21.112 INFO:tasks.workunit.client.0.smithi088.stdout: create 234/38 file after the snapshot 2023-10-16T20:41:21.114 INFO:tasks.workunit.client.0.smithi088.stdout: create 234/39 file after the snapshot 2023-10-16T20:41:21.118 INFO:tasks.workunit.client.0.smithi088.stdout: create 235/.snap/snap-subdir-test 2023-10-16T20:41:21.135 INFO:tasks.workunit.client.0.smithi088.stdout: create 235/30 file after the snapshot 2023-10-16T20:41:21.137 INFO:tasks.workunit.client.0.smithi088.stdout: create 235/31 file after the snapshot 2023-10-16T20:41:21.139 INFO:tasks.workunit.client.0.smithi088.stdout: create 235/32 file after the snapshot 2023-10-16T20:41:21.142 INFO:tasks.workunit.client.0.smithi088.stdout: create 235/33 file after the snapshot 2023-10-16T20:41:21.144 INFO:tasks.workunit.client.0.smithi088.stdout: create 235/34 file after the snapshot 2023-10-16T20:41:21.146 INFO:tasks.workunit.client.0.smithi088.stdout: create 235/35 file after the snapshot 2023-10-16T20:41:21.149 INFO:tasks.workunit.client.0.smithi088.stdout: create 235/36 file after the snapshot 2023-10-16T20:41:21.151 INFO:tasks.workunit.client.0.smithi088.stdout: create 235/37 file after the snapshot 2023-10-16T20:41:21.154 INFO:tasks.workunit.client.0.smithi088.stdout: create 235/38 file after the snapshot 2023-10-16T20:41:21.156 INFO:tasks.workunit.client.0.smithi088.stdout: create 235/39 file after the snapshot 2023-10-16T20:41:21.159 INFO:tasks.workunit.client.0.smithi088.stdout: create 236/.snap/snap-subdir-test 2023-10-16T20:41:21.178 INFO:tasks.workunit.client.0.smithi088.stdout: create 236/30 file after the snapshot 2023-10-16T20:41:21.179 INFO:tasks.workunit.client.0.smithi088.stdout: create 236/31 file after the snapshot 2023-10-16T20:41:21.182 INFO:tasks.workunit.client.0.smithi088.stdout: create 236/32 file after the snapshot 2023-10-16T20:41:21.184 INFO:tasks.workunit.client.0.smithi088.stdout: create 236/33 file after the snapshot 2023-10-16T20:41:21.274 INFO:tasks.workunit.client.0.smithi088.stdout: create 236/34 file after the snapshot 2023-10-16T20:41:21.276 INFO:tasks.workunit.client.0.smithi088.stdout: create 236/35 file after the snapshot 2023-10-16T20:41:21.279 INFO:tasks.workunit.client.0.smithi088.stdout: create 236/36 file after the snapshot 2023-10-16T20:41:21.282 INFO:tasks.workunit.client.0.smithi088.stdout: create 236/37 file after the snapshot 2023-10-16T20:41:21.284 INFO:tasks.workunit.client.0.smithi088.stdout: create 236/38 file after the snapshot 2023-10-16T20:41:21.287 INFO:tasks.workunit.client.0.smithi088.stdout: create 236/39 file after the snapshot 2023-10-16T20:41:21.290 INFO:tasks.workunit.client.0.smithi088.stdout: create 237/.snap/snap-subdir-test 2023-10-16T20:41:21.523 INFO:tasks.workunit.client.0.smithi088.stdout: create 237/30 file after the snapshot 2023-10-16T20:41:21.525 INFO:tasks.workunit.client.0.smithi088.stdout: create 237/31 file after the snapshot 2023-10-16T20:41:21.528 INFO:tasks.workunit.client.0.smithi088.stdout: create 237/32 file after the snapshot 2023-10-16T20:41:21.530 INFO:tasks.workunit.client.0.smithi088.stdout: create 237/33 file after the snapshot 2023-10-16T20:41:21.533 INFO:tasks.workunit.client.0.smithi088.stdout: create 237/34 file after the snapshot 2023-10-16T20:41:21.536 INFO:tasks.workunit.client.0.smithi088.stdout: create 237/35 file after the snapshot 2023-10-16T20:41:21.538 INFO:tasks.workunit.client.0.smithi088.stdout: create 237/36 file after the snapshot 2023-10-16T20:41:21.541 INFO:tasks.workunit.client.0.smithi088.stdout: create 237/37 file after the snapshot 2023-10-16T20:41:21.543 INFO:tasks.workunit.client.0.smithi088.stdout: create 237/38 file after the snapshot 2023-10-16T20:41:21.546 INFO:tasks.workunit.client.0.smithi088.stdout: create 237/39 file after the snapshot 2023-10-16T20:41:21.548 INFO:tasks.workunit.client.0.smithi088.stdout: create 238/.snap/snap-subdir-test 2023-10-16T20:41:21.768 INFO:tasks.workunit.client.0.smithi088.stdout: create 238/30 file after the snapshot 2023-10-16T20:41:21.770 INFO:tasks.workunit.client.0.smithi088.stdout: create 238/31 file after the snapshot 2023-10-16T20:41:21.772 INFO:tasks.workunit.client.0.smithi088.stdout: create 238/32 file after the snapshot 2023-10-16T20:41:21.775 INFO:tasks.workunit.client.0.smithi088.stdout: create 238/33 file after the snapshot 2023-10-16T20:41:21.777 INFO:tasks.workunit.client.0.smithi088.stdout: create 238/34 file after the snapshot 2023-10-16T20:41:21.780 INFO:tasks.workunit.client.0.smithi088.stdout: create 238/35 file after the snapshot 2023-10-16T20:41:21.783 INFO:tasks.workunit.client.0.smithi088.stdout: create 238/36 file after the snapshot 2023-10-16T20:41:21.785 INFO:tasks.workunit.client.0.smithi088.stdout: create 238/37 file after the snapshot 2023-10-16T20:41:21.788 INFO:tasks.workunit.client.0.smithi088.stdout: create 238/38 file after the snapshot 2023-10-16T20:41:21.790 INFO:tasks.workunit.client.0.smithi088.stdout: create 238/39 file after the snapshot 2023-10-16T20:41:21.793 INFO:tasks.workunit.client.0.smithi088.stdout: create 239/.snap/snap-subdir-test 2023-10-16T20:41:21.811 INFO:tasks.workunit.client.0.smithi088.stdout: create 239/30 file after the snapshot 2023-10-16T20:41:21.813 INFO:tasks.workunit.client.0.smithi088.stdout: create 239/31 file after the snapshot 2023-10-16T20:41:21.815 INFO:tasks.workunit.client.0.smithi088.stdout: create 239/32 file after the snapshot 2023-10-16T20:41:21.817 INFO:tasks.workunit.client.0.smithi088.stdout: create 239/33 file after the snapshot 2023-10-16T20:41:21.820 INFO:tasks.workunit.client.0.smithi088.stdout: create 239/34 file after the snapshot 2023-10-16T20:41:21.822 INFO:tasks.workunit.client.0.smithi088.stdout: create 239/35 file after the snapshot 2023-10-16T20:41:21.825 INFO:tasks.workunit.client.0.smithi088.stdout: create 239/36 file after the snapshot 2023-10-16T20:41:21.828 INFO:tasks.workunit.client.0.smithi088.stdout: create 239/37 file after the snapshot 2023-10-16T20:41:21.830 INFO:tasks.workunit.client.0.smithi088.stdout: create 239/38 file after the snapshot 2023-10-16T20:41:21.832 INFO:tasks.workunit.client.0.smithi088.stdout: create 239/39 file after the snapshot 2023-10-16T20:41:21.835 INFO:tasks.workunit.client.0.smithi088.stdout: create 240/.snap/snap-subdir-test 2023-10-16T20:41:21.853 INFO:tasks.workunit.client.0.smithi088.stdout: create 240/30 file after the snapshot 2023-10-16T20:41:21.855 INFO:tasks.workunit.client.0.smithi088.stdout: create 240/31 file after the snapshot 2023-10-16T20:41:21.857 INFO:tasks.workunit.client.0.smithi088.stdout: create 240/32 file after the snapshot 2023-10-16T20:41:21.860 INFO:tasks.workunit.client.0.smithi088.stdout: create 240/33 file after the snapshot 2023-10-16T20:41:21.862 INFO:tasks.workunit.client.0.smithi088.stdout: create 240/34 file after the snapshot 2023-10-16T20:41:21.865 INFO:tasks.workunit.client.0.smithi088.stdout: create 240/35 file after the snapshot 2023-10-16T20:41:21.867 INFO:tasks.workunit.client.0.smithi088.stdout: create 240/36 file after the snapshot 2023-10-16T20:41:21.870 INFO:tasks.workunit.client.0.smithi088.stdout: create 240/37 file after the snapshot 2023-10-16T20:41:21.873 INFO:tasks.workunit.client.0.smithi088.stdout: create 240/38 file after the snapshot 2023-10-16T20:41:21.875 INFO:tasks.workunit.client.0.smithi088.stdout: create 240/39 file after the snapshot 2023-10-16T20:41:21.877 INFO:tasks.workunit.client.0.smithi088.stdout: create 241/.snap/snap-subdir-test 2023-10-16T20:41:21.898 INFO:tasks.workunit.client.0.smithi088.stdout: create 241/30 file after the snapshot 2023-10-16T20:41:21.899 INFO:tasks.workunit.client.0.smithi088.stdout: create 241/31 file after the snapshot 2023-10-16T20:41:21.902 INFO:tasks.workunit.client.0.smithi088.stdout: create 241/32 file after the snapshot 2023-10-16T20:41:21.904 INFO:tasks.workunit.client.0.smithi088.stdout: create 241/33 file after the snapshot 2023-10-16T20:41:21.906 INFO:tasks.workunit.client.0.smithi088.stdout: create 241/34 file after the snapshot 2023-10-16T20:41:21.909 INFO:tasks.workunit.client.0.smithi088.stdout: create 241/35 file after the snapshot 2023-10-16T20:41:21.911 INFO:tasks.workunit.client.0.smithi088.stdout: create 241/36 file after the snapshot 2023-10-16T20:41:21.914 INFO:tasks.workunit.client.0.smithi088.stdout: create 241/37 file after the snapshot 2023-10-16T20:41:21.917 INFO:tasks.workunit.client.0.smithi088.stdout: create 241/38 file after the snapshot 2023-10-16T20:41:21.919 INFO:tasks.workunit.client.0.smithi088.stdout: create 241/39 file after the snapshot 2023-10-16T20:41:21.921 INFO:tasks.workunit.client.0.smithi088.stdout: create 242/.snap/snap-subdir-test 2023-10-16T20:41:21.939 INFO:tasks.workunit.client.0.smithi088.stdout: create 242/30 file after the snapshot 2023-10-16T20:41:21.942 INFO:tasks.workunit.client.0.smithi088.stdout: create 242/31 file after the snapshot 2023-10-16T20:41:21.944 INFO:tasks.workunit.client.0.smithi088.stdout: create 242/32 file after the snapshot 2023-10-16T20:41:21.946 INFO:tasks.workunit.client.0.smithi088.stdout: create 242/33 file after the snapshot 2023-10-16T20:41:21.949 INFO:tasks.workunit.client.0.smithi088.stdout: create 242/34 file after the snapshot 2023-10-16T20:41:21.951 INFO:tasks.workunit.client.0.smithi088.stdout: create 242/35 file after the snapshot 2023-10-16T20:41:21.954 INFO:tasks.workunit.client.0.smithi088.stdout: create 242/36 file after the snapshot 2023-10-16T20:41:21.957 INFO:tasks.workunit.client.0.smithi088.stdout: create 242/37 file after the snapshot 2023-10-16T20:41:21.959 INFO:tasks.workunit.client.0.smithi088.stdout: create 242/38 file after the snapshot 2023-10-16T20:41:21.962 INFO:tasks.workunit.client.0.smithi088.stdout: create 242/39 file after the snapshot 2023-10-16T20:41:21.964 INFO:tasks.workunit.client.0.smithi088.stdout: create 243/.snap/snap-subdir-test 2023-10-16T20:41:21.981 INFO:tasks.workunit.client.0.smithi088.stdout: create 243/30 file after the snapshot 2023-10-16T20:41:21.983 INFO:tasks.workunit.client.0.smithi088.stdout: create 243/31 file after the snapshot 2023-10-16T20:41:21.985 INFO:tasks.workunit.client.0.smithi088.stdout: create 243/32 file after the snapshot 2023-10-16T20:41:21.988 INFO:tasks.workunit.client.0.smithi088.stdout: create 243/33 file after the snapshot 2023-10-16T20:41:21.990 INFO:tasks.workunit.client.0.smithi088.stdout: create 243/34 file after the snapshot 2023-10-16T20:41:21.993 INFO:tasks.workunit.client.0.smithi088.stdout: create 243/35 file after the snapshot 2023-10-16T20:41:21.995 INFO:tasks.workunit.client.0.smithi088.stdout: create 243/36 file after the snapshot 2023-10-16T20:41:21.997 INFO:tasks.workunit.client.0.smithi088.stdout: create 243/37 file after the snapshot 2023-10-16T20:41:22.001 INFO:tasks.workunit.client.0.smithi088.stdout: create 243/38 file after the snapshot 2023-10-16T20:41:22.002 INFO:tasks.workunit.client.0.smithi088.stdout: create 243/39 file after the snapshot 2023-10-16T20:41:22.005 INFO:tasks.workunit.client.0.smithi088.stdout: create 244/.snap/snap-subdir-test 2023-10-16T20:41:22.024 INFO:tasks.workunit.client.0.smithi088.stdout: create 244/30 file after the snapshot 2023-10-16T20:41:22.025 INFO:tasks.workunit.client.0.smithi088.stdout: create 244/31 file after the snapshot 2023-10-16T20:41:22.028 INFO:tasks.workunit.client.0.smithi088.stdout: create 244/32 file after the snapshot 2023-10-16T20:41:22.031 INFO:tasks.workunit.client.0.smithi088.stdout: create 244/33 file after the snapshot 2023-10-16T20:41:22.033 INFO:tasks.workunit.client.0.smithi088.stdout: create 244/34 file after the snapshot 2023-10-16T20:41:22.035 INFO:tasks.workunit.client.0.smithi088.stdout: create 244/35 file after the snapshot 2023-10-16T20:41:22.038 INFO:tasks.workunit.client.0.smithi088.stdout: create 244/36 file after the snapshot 2023-10-16T20:41:22.041 INFO:tasks.workunit.client.0.smithi088.stdout: create 244/37 file after the snapshot 2023-10-16T20:41:22.043 INFO:tasks.workunit.client.0.smithi088.stdout: create 244/38 file after the snapshot 2023-10-16T20:41:22.046 INFO:tasks.workunit.client.0.smithi088.stdout: create 244/39 file after the snapshot 2023-10-16T20:41:22.048 INFO:tasks.workunit.client.0.smithi088.stdout: create 245/.snap/snap-subdir-test 2023-10-16T20:41:22.068 INFO:tasks.workunit.client.0.smithi088.stdout: create 245/30 file after the snapshot 2023-10-16T20:41:22.070 INFO:tasks.workunit.client.0.smithi088.stdout: create 245/31 file after the snapshot 2023-10-16T20:41:22.072 INFO:tasks.workunit.client.0.smithi088.stdout: create 245/32 file after the snapshot 2023-10-16T20:41:22.075 INFO:tasks.workunit.client.0.smithi088.stdout: create 245/33 file after the snapshot 2023-10-16T20:41:22.078 INFO:tasks.workunit.client.0.smithi088.stdout: create 245/34 file after the snapshot 2023-10-16T20:41:22.080 INFO:tasks.workunit.client.0.smithi088.stdout: create 245/35 file after the snapshot 2023-10-16T20:41:22.083 INFO:tasks.workunit.client.0.smithi088.stdout: create 245/36 file after the snapshot 2023-10-16T20:41:22.086 INFO:tasks.workunit.client.0.smithi088.stdout: create 245/37 file after the snapshot 2023-10-16T20:41:22.088 INFO:tasks.workunit.client.0.smithi088.stdout: create 245/38 file after the snapshot 2023-10-16T20:41:22.092 INFO:tasks.workunit.client.0.smithi088.stdout: create 245/39 file after the snapshot 2023-10-16T20:41:22.094 INFO:tasks.workunit.client.0.smithi088.stdout: create 246/.snap/snap-subdir-test 2023-10-16T20:41:22.112 INFO:tasks.workunit.client.0.smithi088.stdout: create 246/30 file after the snapshot 2023-10-16T20:41:22.114 INFO:tasks.workunit.client.0.smithi088.stdout: create 246/31 file after the snapshot 2023-10-16T20:41:22.116 INFO:tasks.workunit.client.0.smithi088.stdout: create 246/32 file after the snapshot 2023-10-16T20:41:22.119 INFO:tasks.workunit.client.0.smithi088.stdout: create 246/33 file after the snapshot 2023-10-16T20:41:22.121 INFO:tasks.workunit.client.0.smithi088.stdout: create 246/34 file after the snapshot 2023-10-16T20:41:22.124 INFO:tasks.workunit.client.0.smithi088.stdout: create 246/35 file after the snapshot 2023-10-16T20:41:22.126 INFO:tasks.workunit.client.0.smithi088.stdout: create 246/36 file after the snapshot 2023-10-16T20:41:22.128 INFO:tasks.workunit.client.0.smithi088.stdout: create 246/37 file after the snapshot 2023-10-16T20:41:22.131 INFO:tasks.workunit.client.0.smithi088.stdout: create 246/38 file after the snapshot 2023-10-16T20:41:22.133 INFO:tasks.workunit.client.0.smithi088.stdout: create 246/39 file after the snapshot 2023-10-16T20:41:22.135 INFO:tasks.workunit.client.0.smithi088.stdout: create 247/.snap/snap-subdir-test 2023-10-16T20:41:22.155 INFO:tasks.workunit.client.0.smithi088.stdout: create 247/30 file after the snapshot 2023-10-16T20:41:22.157 INFO:tasks.workunit.client.0.smithi088.stdout: create 247/31 file after the snapshot 2023-10-16T20:41:22.159 INFO:tasks.workunit.client.0.smithi088.stdout: create 247/32 file after the snapshot 2023-10-16T20:41:22.162 INFO:tasks.workunit.client.0.smithi088.stdout: create 247/33 file after the snapshot 2023-10-16T20:41:22.164 INFO:tasks.workunit.client.0.smithi088.stdout: create 247/34 file after the snapshot 2023-10-16T20:41:22.166 INFO:tasks.workunit.client.0.smithi088.stdout: create 247/35 file after the snapshot 2023-10-16T20:41:22.168 INFO:tasks.workunit.client.0.smithi088.stdout: create 247/36 file after the snapshot 2023-10-16T20:41:22.171 INFO:tasks.workunit.client.0.smithi088.stdout: create 247/37 file after the snapshot 2023-10-16T20:41:22.174 INFO:tasks.workunit.client.0.smithi088.stdout: create 247/38 file after the snapshot 2023-10-16T20:41:22.176 INFO:tasks.workunit.client.0.smithi088.stdout: create 247/39 file after the snapshot 2023-10-16T20:41:22.179 INFO:tasks.workunit.client.0.smithi088.stdout: create 248/.snap/snap-subdir-test 2023-10-16T20:41:22.314 INFO:tasks.workunit.client.0.smithi088.stdout: create 248/30 file after the snapshot 2023-10-16T20:41:22.317 INFO:tasks.workunit.client.0.smithi088.stdout: create 248/31 file after the snapshot 2023-10-16T20:41:22.320 INFO:tasks.workunit.client.0.smithi088.stdout: create 248/32 file after the snapshot 2023-10-16T20:41:22.324 INFO:tasks.workunit.client.0.smithi088.stdout: create 248/33 file after the snapshot 2023-10-16T20:41:22.327 INFO:tasks.workunit.client.0.smithi088.stdout: create 248/34 file after the snapshot 2023-10-16T20:41:22.330 INFO:tasks.workunit.client.0.smithi088.stdout: create 248/35 file after the snapshot 2023-10-16T20:41:22.332 INFO:tasks.workunit.client.0.smithi088.stdout: create 248/36 file after the snapshot 2023-10-16T20:41:22.336 INFO:tasks.workunit.client.0.smithi088.stdout: create 248/37 file after the snapshot 2023-10-16T20:41:22.339 INFO:tasks.workunit.client.0.smithi088.stdout: create 248/38 file after the snapshot 2023-10-16T20:41:22.341 INFO:tasks.workunit.client.0.smithi088.stdout: create 248/39 file after the snapshot 2023-10-16T20:41:22.343 INFO:tasks.workunit.client.0.smithi088.stdout: create 249/.snap/snap-subdir-test 2023-10-16T20:41:23.095 INFO:tasks.workunit.client.0.smithi088.stdout: create 249/30 file after the snapshot 2023-10-16T20:41:23.099 INFO:tasks.workunit.client.0.smithi088.stdout: create 249/31 file after the snapshot 2023-10-16T20:41:23.102 INFO:tasks.workunit.client.0.smithi088.stdout: create 249/32 file after the snapshot 2023-10-16T20:41:23.105 INFO:tasks.workunit.client.0.smithi088.stdout: create 249/33 file after the snapshot 2023-10-16T20:41:23.108 INFO:tasks.workunit.client.0.smithi088.stdout: create 249/34 file after the snapshot 2023-10-16T20:41:23.110 INFO:tasks.workunit.client.0.smithi088.stdout: create 249/35 file after the snapshot 2023-10-16T20:41:23.113 INFO:tasks.workunit.client.0.smithi088.stdout: create 249/36 file after the snapshot 2023-10-16T20:41:23.115 INFO:tasks.workunit.client.0.smithi088.stdout: create 249/37 file after the snapshot 2023-10-16T20:41:23.118 INFO:tasks.workunit.client.0.smithi088.stdout: create 249/38 file after the snapshot 2023-10-16T20:41:23.121 INFO:tasks.workunit.client.0.smithi088.stdout: create 249/39 file after the snapshot 2023-10-16T20:41:23.124 INFO:tasks.workunit.client.0.smithi088.stdout: create 250/.snap/snap-subdir-test 2023-10-16T20:41:23.145 INFO:tasks.workunit.client.0.smithi088.stdout: create 250/30 file after the snapshot 2023-10-16T20:41:23.148 INFO:tasks.workunit.client.0.smithi088.stdout: create 250/31 file after the snapshot 2023-10-16T20:41:23.151 INFO:tasks.workunit.client.0.smithi088.stdout: create 250/32 file after the snapshot 2023-10-16T20:41:23.154 INFO:tasks.workunit.client.0.smithi088.stdout: create 250/33 file after the snapshot 2023-10-16T20:41:23.157 INFO:tasks.workunit.client.0.smithi088.stdout: create 250/34 file after the snapshot 2023-10-16T20:41:23.159 INFO:tasks.workunit.client.0.smithi088.stdout: create 250/35 file after the snapshot 2023-10-16T20:41:23.162 INFO:tasks.workunit.client.0.smithi088.stdout: create 250/36 file after the snapshot 2023-10-16T20:41:23.165 INFO:tasks.workunit.client.0.smithi088.stdout: create 250/37 file after the snapshot 2023-10-16T20:41:23.167 INFO:tasks.workunit.client.0.smithi088.stdout: create 250/38 file after the snapshot 2023-10-16T20:41:23.170 INFO:tasks.workunit.client.0.smithi088.stdout: create 250/39 file after the snapshot 2023-10-16T20:41:23.173 INFO:tasks.workunit.client.0.smithi088.stdout: create 251/.snap/snap-subdir-test 2023-10-16T20:41:23.194 INFO:tasks.workunit.client.0.smithi088.stdout: create 251/30 file after the snapshot 2023-10-16T20:41:23.197 INFO:tasks.workunit.client.0.smithi088.stdout: create 251/31 file after the snapshot 2023-10-16T20:41:23.199 INFO:tasks.workunit.client.0.smithi088.stdout: create 251/32 file after the snapshot 2023-10-16T20:41:23.202 INFO:tasks.workunit.client.0.smithi088.stdout: create 251/33 file after the snapshot 2023-10-16T20:41:23.204 INFO:tasks.workunit.client.0.smithi088.stdout: create 251/34 file after the snapshot 2023-10-16T20:41:23.207 INFO:tasks.workunit.client.0.smithi088.stdout: create 251/35 file after the snapshot 2023-10-16T20:41:23.209 INFO:tasks.workunit.client.0.smithi088.stdout: create 251/36 file after the snapshot 2023-10-16T20:41:23.212 INFO:tasks.workunit.client.0.smithi088.stdout: create 251/37 file after the snapshot 2023-10-16T20:41:23.214 INFO:tasks.workunit.client.0.smithi088.stdout: create 251/38 file after the snapshot 2023-10-16T20:41:23.216 INFO:tasks.workunit.client.0.smithi088.stdout: create 251/39 file after the snapshot 2023-10-16T20:41:23.218 INFO:tasks.workunit.client.0.smithi088.stdout: create 252/.snap/snap-subdir-test 2023-10-16T20:41:23.238 INFO:tasks.workunit.client.0.smithi088.stdout: create 252/30 file after the snapshot 2023-10-16T20:41:23.241 INFO:tasks.workunit.client.0.smithi088.stdout: create 252/31 file after the snapshot 2023-10-16T20:41:23.243 INFO:tasks.workunit.client.0.smithi088.stdout: create 252/32 file after the snapshot 2023-10-16T20:41:23.245 INFO:tasks.workunit.client.0.smithi088.stdout: create 252/33 file after the snapshot 2023-10-16T20:41:23.248 INFO:tasks.workunit.client.0.smithi088.stdout: create 252/34 file after the snapshot 2023-10-16T20:41:23.250 INFO:tasks.workunit.client.0.smithi088.stdout: create 252/35 file after the snapshot 2023-10-16T20:41:23.253 INFO:tasks.workunit.client.0.smithi088.stdout: create 252/36 file after the snapshot 2023-10-16T20:41:23.255 INFO:tasks.workunit.client.0.smithi088.stdout: create 252/37 file after the snapshot 2023-10-16T20:41:23.258 INFO:tasks.workunit.client.0.smithi088.stdout: create 252/38 file after the snapshot 2023-10-16T20:41:23.261 INFO:tasks.workunit.client.0.smithi088.stdout: create 252/39 file after the snapshot 2023-10-16T20:41:23.264 INFO:tasks.workunit.client.0.smithi088.stdout: create 253/.snap/snap-subdir-test 2023-10-16T20:41:23.285 INFO:tasks.workunit.client.0.smithi088.stdout: create 253/30 file after the snapshot 2023-10-16T20:41:23.287 INFO:tasks.workunit.client.0.smithi088.stdout: create 253/31 file after the snapshot 2023-10-16T20:41:23.289 INFO:tasks.workunit.client.0.smithi088.stdout: create 253/32 file after the snapshot 2023-10-16T20:41:23.291 INFO:tasks.workunit.client.0.smithi088.stdout: create 253/33 file after the snapshot 2023-10-16T20:41:23.294 INFO:tasks.workunit.client.0.smithi088.stdout: create 253/34 file after the snapshot 2023-10-16T20:41:23.296 INFO:tasks.workunit.client.0.smithi088.stdout: create 253/35 file after the snapshot 2023-10-16T20:41:23.299 INFO:tasks.workunit.client.0.smithi088.stdout: create 253/36 file after the snapshot 2023-10-16T20:41:23.301 INFO:tasks.workunit.client.0.smithi088.stdout: create 253/37 file after the snapshot 2023-10-16T20:41:23.303 INFO:tasks.workunit.client.0.smithi088.stdout: create 253/38 file after the snapshot 2023-10-16T20:41:23.306 INFO:tasks.workunit.client.0.smithi088.stdout: create 253/39 file after the snapshot 2023-10-16T20:41:23.309 INFO:tasks.workunit.client.0.smithi088.stdout: create 254/.snap/snap-subdir-test 2023-10-16T20:41:23.338 INFO:tasks.workunit.client.0.smithi088.stdout: create 254/30 file after the snapshot 2023-10-16T20:41:23.340 INFO:tasks.workunit.client.0.smithi088.stdout: create 254/31 file after the snapshot 2023-10-16T20:41:23.342 INFO:tasks.workunit.client.0.smithi088.stdout: create 254/32 file after the snapshot 2023-10-16T20:41:23.344 INFO:tasks.workunit.client.0.smithi088.stdout: create 254/33 file after the snapshot 2023-10-16T20:41:23.347 INFO:tasks.workunit.client.0.smithi088.stdout: create 254/34 file after the snapshot 2023-10-16T20:41:23.350 INFO:tasks.workunit.client.0.smithi088.stdout: create 254/35 file after the snapshot 2023-10-16T20:41:23.352 INFO:tasks.workunit.client.0.smithi088.stdout: create 254/36 file after the snapshot 2023-10-16T20:41:23.354 INFO:tasks.workunit.client.0.smithi088.stdout: create 254/37 file after the snapshot 2023-10-16T20:41:23.357 INFO:tasks.workunit.client.0.smithi088.stdout: create 254/38 file after the snapshot 2023-10-16T20:41:23.361 INFO:tasks.workunit.client.0.smithi088.stdout: create 254/39 file after the snapshot 2023-10-16T20:41:23.362 INFO:tasks.workunit.client.0.smithi088.stdout: create 255/.snap/snap-subdir-test 2023-10-16T20:41:23.383 INFO:tasks.workunit.client.0.smithi088.stdout: create 255/30 file after the snapshot 2023-10-16T20:41:23.385 INFO:tasks.workunit.client.0.smithi088.stdout: create 255/31 file after the snapshot 2023-10-16T20:41:23.388 INFO:tasks.workunit.client.0.smithi088.stdout: create 255/32 file after the snapshot 2023-10-16T20:41:23.391 INFO:tasks.workunit.client.0.smithi088.stdout: create 255/33 file after the snapshot 2023-10-16T20:41:23.394 INFO:tasks.workunit.client.0.smithi088.stdout: create 255/34 file after the snapshot 2023-10-16T20:41:23.397 INFO:tasks.workunit.client.0.smithi088.stdout: create 255/35 file after the snapshot 2023-10-16T20:41:23.399 INFO:tasks.workunit.client.0.smithi088.stdout: create 255/36 file after the snapshot 2023-10-16T20:41:23.402 INFO:tasks.workunit.client.0.smithi088.stdout: create 255/37 file after the snapshot 2023-10-16T20:41:23.404 INFO:tasks.workunit.client.0.smithi088.stdout: create 255/38 file after the snapshot 2023-10-16T20:41:23.408 INFO:tasks.workunit.client.0.smithi088.stdout: create 255/39 file after the snapshot 2023-10-16T20:41:23.410 INFO:tasks.workunit.client.0.smithi088.stdout: create 256/.snap/snap-subdir-test 2023-10-16T20:41:23.429 INFO:tasks.workunit.client.0.smithi088.stdout: create 256/30 file after the snapshot 2023-10-16T20:41:23.429 INFO:tasks.workunit.client.0.smithi088.stdout: create 256/31 file after the snapshot 2023-10-16T20:41:23.432 INFO:tasks.workunit.client.0.smithi088.stdout: create 256/32 file after the snapshot 2023-10-16T20:41:23.434 INFO:tasks.workunit.client.0.smithi088.stdout: create 256/33 file after the snapshot 2023-10-16T20:41:23.437 INFO:tasks.workunit.client.0.smithi088.stdout: create 256/34 file after the snapshot 2023-10-16T20:41:23.440 INFO:tasks.workunit.client.0.smithi088.stdout: create 256/35 file after the snapshot 2023-10-16T20:41:23.442 INFO:tasks.workunit.client.0.smithi088.stdout: create 256/36 file after the snapshot 2023-10-16T20:41:23.445 INFO:tasks.workunit.client.0.smithi088.stdout: create 256/37 file after the snapshot 2023-10-16T20:41:23.447 INFO:tasks.workunit.client.0.smithi088.stdout: create 256/38 file after the snapshot 2023-10-16T20:41:23.450 INFO:tasks.workunit.client.0.smithi088.stdout: create 256/39 file after the snapshot 2023-10-16T20:41:23.452 INFO:tasks.workunit.client.0.smithi088.stdout: create 257/.snap/snap-subdir-test 2023-10-16T20:41:23.471 INFO:tasks.workunit.client.0.smithi088.stdout: create 257/30 file after the snapshot 2023-10-16T20:41:23.473 INFO:tasks.workunit.client.0.smithi088.stdout: create 257/31 file after the snapshot 2023-10-16T20:41:23.475 INFO:tasks.workunit.client.0.smithi088.stdout: create 257/32 file after the snapshot 2023-10-16T20:41:23.478 INFO:tasks.workunit.client.0.smithi088.stdout: create 257/33 file after the snapshot 2023-10-16T20:41:23.481 INFO:tasks.workunit.client.0.smithi088.stdout: create 257/34 file after the snapshot 2023-10-16T20:41:23.484 INFO:tasks.workunit.client.0.smithi088.stdout: create 257/35 file after the snapshot 2023-10-16T20:41:23.486 INFO:tasks.workunit.client.0.smithi088.stdout: create 257/36 file after the snapshot 2023-10-16T20:41:23.489 INFO:tasks.workunit.client.0.smithi088.stdout: create 257/37 file after the snapshot 2023-10-16T20:41:23.492 INFO:tasks.workunit.client.0.smithi088.stdout: create 257/38 file after the snapshot 2023-10-16T20:41:23.495 INFO:tasks.workunit.client.0.smithi088.stdout: create 257/39 file after the snapshot 2023-10-16T20:41:23.498 INFO:tasks.workunit.client.0.smithi088.stdout: create 258/.snap/snap-subdir-test 2023-10-16T20:41:23.518 INFO:tasks.workunit.client.0.smithi088.stdout: create 258/30 file after the snapshot 2023-10-16T20:41:23.520 INFO:tasks.workunit.client.0.smithi088.stdout: create 258/31 file after the snapshot 2023-10-16T20:41:23.523 INFO:tasks.workunit.client.0.smithi088.stdout: create 258/32 file after the snapshot 2023-10-16T20:41:23.525 INFO:tasks.workunit.client.0.smithi088.stdout: create 258/33 file after the snapshot 2023-10-16T20:41:23.527 INFO:tasks.workunit.client.0.smithi088.stdout: create 258/34 file after the snapshot 2023-10-16T20:41:23.530 INFO:tasks.workunit.client.0.smithi088.stdout: create 258/35 file after the snapshot 2023-10-16T20:41:23.532 INFO:tasks.workunit.client.0.smithi088.stdout: create 258/36 file after the snapshot 2023-10-16T20:41:23.535 INFO:tasks.workunit.client.0.smithi088.stdout: create 258/37 file after the snapshot 2023-10-16T20:41:23.537 INFO:tasks.workunit.client.0.smithi088.stdout: create 258/38 file after the snapshot 2023-10-16T20:41:23.540 INFO:tasks.workunit.client.0.smithi088.stdout: create 258/39 file after the snapshot 2023-10-16T20:41:23.543 INFO:tasks.workunit.client.0.smithi088.stdout: create 259/.snap/snap-subdir-test 2023-10-16T20:41:23.562 INFO:tasks.workunit.client.0.smithi088.stdout: create 259/30 file after the snapshot 2023-10-16T20:41:23.564 INFO:tasks.workunit.client.0.smithi088.stdout: create 259/31 file after the snapshot 2023-10-16T20:41:23.566 INFO:tasks.workunit.client.0.smithi088.stdout: create 259/32 file after the snapshot 2023-10-16T20:41:23.568 INFO:tasks.workunit.client.0.smithi088.stdout: create 259/33 file after the snapshot 2023-10-16T20:41:23.571 INFO:tasks.workunit.client.0.smithi088.stdout: create 259/34 file after the snapshot 2023-10-16T20:41:23.574 INFO:tasks.workunit.client.0.smithi088.stdout: create 259/35 file after the snapshot 2023-10-16T20:41:23.576 INFO:tasks.workunit.client.0.smithi088.stdout: create 259/36 file after the snapshot 2023-10-16T20:41:23.579 INFO:tasks.workunit.client.0.smithi088.stdout: create 259/37 file after the snapshot 2023-10-16T20:41:23.582 INFO:tasks.workunit.client.0.smithi088.stdout: create 259/38 file after the snapshot 2023-10-16T20:41:23.584 INFO:tasks.workunit.client.0.smithi088.stdout: create 259/39 file after the snapshot 2023-10-16T20:41:23.586 INFO:tasks.workunit.client.0.smithi088.stdout: create 260/.snap/snap-subdir-test 2023-10-16T20:41:24.135 INFO:tasks.workunit.client.0.smithi088.stdout: create 260/30 file after the snapshot 2023-10-16T20:41:24.137 INFO:tasks.workunit.client.0.smithi088.stdout: create 260/31 file after the snapshot 2023-10-16T20:41:24.139 INFO:tasks.workunit.client.0.smithi088.stdout: create 260/32 file after the snapshot 2023-10-16T20:41:24.142 INFO:tasks.workunit.client.0.smithi088.stdout: create 260/33 file after the snapshot 2023-10-16T20:41:24.144 INFO:tasks.workunit.client.0.smithi088.stdout: create 260/34 file after the snapshot 2023-10-16T20:41:24.147 INFO:tasks.workunit.client.0.smithi088.stdout: create 260/35 file after the snapshot 2023-10-16T20:41:24.150 INFO:tasks.workunit.client.0.smithi088.stdout: create 260/36 file after the snapshot 2023-10-16T20:41:24.152 INFO:tasks.workunit.client.0.smithi088.stdout: create 260/37 file after the snapshot 2023-10-16T20:41:24.155 INFO:tasks.workunit.client.0.smithi088.stdout: create 260/38 file after the snapshot 2023-10-16T20:41:24.158 INFO:tasks.workunit.client.0.smithi088.stdout: create 260/39 file after the snapshot 2023-10-16T20:41:24.160 INFO:tasks.workunit.client.0.smithi088.stdout: create 261/.snap/snap-subdir-test 2023-10-16T20:41:24.180 INFO:tasks.workunit.client.0.smithi088.stdout: create 261/30 file after the snapshot 2023-10-16T20:41:24.182 INFO:tasks.workunit.client.0.smithi088.stdout: create 261/31 file after the snapshot 2023-10-16T20:41:24.185 INFO:tasks.workunit.client.0.smithi088.stdout: create 261/32 file after the snapshot 2023-10-16T20:41:24.187 INFO:tasks.workunit.client.0.smithi088.stdout: create 261/33 file after the snapshot 2023-10-16T20:41:24.190 INFO:tasks.workunit.client.0.smithi088.stdout: create 261/34 file after the snapshot 2023-10-16T20:41:24.193 INFO:tasks.workunit.client.0.smithi088.stdout: create 261/35 file after the snapshot 2023-10-16T20:41:24.195 INFO:tasks.workunit.client.0.smithi088.stdout: create 261/36 file after the snapshot 2023-10-16T20:41:24.199 INFO:tasks.workunit.client.0.smithi088.stdout: create 261/37 file after the snapshot 2023-10-16T20:41:24.202 INFO:tasks.workunit.client.0.smithi088.stdout: create 261/38 file after the snapshot 2023-10-16T20:41:24.204 INFO:tasks.workunit.client.0.smithi088.stdout: create 261/39 file after the snapshot 2023-10-16T20:41:24.207 INFO:tasks.workunit.client.0.smithi088.stdout: create 262/.snap/snap-subdir-test 2023-10-16T20:41:24.227 INFO:tasks.workunit.client.0.smithi088.stdout: create 262/30 file after the snapshot 2023-10-16T20:41:24.230 INFO:tasks.workunit.client.0.smithi088.stdout: create 262/31 file after the snapshot 2023-10-16T20:41:24.231 INFO:tasks.workunit.client.0.smithi088.stdout: create 262/32 file after the snapshot 2023-10-16T20:41:24.233 INFO:tasks.workunit.client.0.smithi088.stdout: create 262/33 file after the snapshot 2023-10-16T20:41:24.236 INFO:tasks.workunit.client.0.smithi088.stdout: create 262/34 file after the snapshot 2023-10-16T20:41:24.238 INFO:tasks.workunit.client.0.smithi088.stdout: create 262/35 file after the snapshot 2023-10-16T20:41:24.240 INFO:tasks.workunit.client.0.smithi088.stdout: create 262/36 file after the snapshot 2023-10-16T20:41:24.243 INFO:tasks.workunit.client.0.smithi088.stdout: create 262/37 file after the snapshot 2023-10-16T20:41:24.246 INFO:tasks.workunit.client.0.smithi088.stdout: create 262/38 file after the snapshot 2023-10-16T20:41:24.249 INFO:tasks.workunit.client.0.smithi088.stdout: create 262/39 file after the snapshot 2023-10-16T20:41:24.251 INFO:tasks.workunit.client.0.smithi088.stdout: create 263/.snap/snap-subdir-test 2023-10-16T20:41:24.272 INFO:tasks.workunit.client.0.smithi088.stdout: create 263/30 file after the snapshot 2023-10-16T20:41:24.274 INFO:tasks.workunit.client.0.smithi088.stdout: create 263/31 file after the snapshot 2023-10-16T20:41:24.276 INFO:tasks.workunit.client.0.smithi088.stdout: create 263/32 file after the snapshot 2023-10-16T20:41:24.278 INFO:tasks.workunit.client.0.smithi088.stdout: create 263/33 file after the snapshot 2023-10-16T20:41:24.281 INFO:tasks.workunit.client.0.smithi088.stdout: create 263/34 file after the snapshot 2023-10-16T20:41:24.283 INFO:tasks.workunit.client.0.smithi088.stdout: create 263/35 file after the snapshot 2023-10-16T20:41:24.286 INFO:tasks.workunit.client.0.smithi088.stdout: create 263/36 file after the snapshot 2023-10-16T20:41:24.288 INFO:tasks.workunit.client.0.smithi088.stdout: create 263/37 file after the snapshot 2023-10-16T20:41:24.290 INFO:tasks.workunit.client.0.smithi088.stdout: create 263/38 file after the snapshot 2023-10-16T20:41:24.293 INFO:tasks.workunit.client.0.smithi088.stdout: create 263/39 file after the snapshot 2023-10-16T20:41:24.295 INFO:tasks.workunit.client.0.smithi088.stdout: create 264/.snap/snap-subdir-test 2023-10-16T20:41:24.314 INFO:tasks.workunit.client.0.smithi088.stdout: create 264/30 file after the snapshot 2023-10-16T20:41:24.316 INFO:tasks.workunit.client.0.smithi088.stdout: create 264/31 file after the snapshot 2023-10-16T20:41:24.322 INFO:tasks.workunit.client.0.smithi088.stdout: create 264/32 file after the snapshot 2023-10-16T20:41:24.606 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:41:24.610 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:41:25.312 INFO:tasks.workunit.client.0.smithi088.stdout: create 264/33 file after the snapshot 2023-10-16T20:41:25.315 INFO:tasks.workunit.client.0.smithi088.stdout: create 264/34 file after the snapshot 2023-10-16T20:41:25.317 INFO:tasks.workunit.client.0.smithi088.stdout: create 264/35 file after the snapshot 2023-10-16T20:41:25.321 INFO:tasks.workunit.client.0.smithi088.stdout: create 264/36 file after the snapshot 2023-10-16T20:41:25.328 INFO:tasks.workunit.client.0.smithi088.stdout: create 264/37 file after the snapshot 2023-10-16T20:41:25.384 INFO:tasks.workunit.client.0.smithi088.stdout: create 264/38 file after the snapshot 2023-10-16T20:41:25.386 INFO:tasks.workunit.client.0.smithi088.stdout: create 264/39 file after the snapshot 2023-10-16T20:41:25.389 INFO:tasks.workunit.client.0.smithi088.stdout: create 265/.snap/snap-subdir-test 2023-10-16T20:41:25.455 INFO:tasks.workunit.client.0.smithi088.stdout: create 265/30 file after the snapshot 2023-10-16T20:41:25.457 INFO:tasks.workunit.client.0.smithi088.stdout: create 265/31 file after the snapshot 2023-10-16T20:41:25.459 INFO:tasks.workunit.client.0.smithi088.stdout: create 265/32 file after the snapshot 2023-10-16T20:41:25.462 INFO:tasks.workunit.client.0.smithi088.stdout: create 265/33 file after the snapshot 2023-10-16T20:41:25.464 INFO:tasks.workunit.client.0.smithi088.stdout: create 265/34 file after the snapshot 2023-10-16T20:41:25.467 INFO:tasks.workunit.client.0.smithi088.stdout: create 265/35 file after the snapshot 2023-10-16T20:41:25.469 INFO:tasks.workunit.client.0.smithi088.stdout: create 265/36 file after the snapshot 2023-10-16T20:41:25.471 INFO:tasks.workunit.client.0.smithi088.stdout: create 265/37 file after the snapshot 2023-10-16T20:41:25.475 INFO:tasks.workunit.client.0.smithi088.stdout: create 265/38 file after the snapshot 2023-10-16T20:41:25.476 INFO:tasks.workunit.client.0.smithi088.stdout: create 265/39 file after the snapshot 2023-10-16T20:41:25.481 INFO:tasks.workunit.client.0.smithi088.stdout: create 266/.snap/snap-subdir-test 2023-10-16T20:41:25.497 INFO:tasks.workunit.client.0.smithi088.stdout: create 266/30 file after the snapshot 2023-10-16T20:41:25.499 INFO:tasks.workunit.client.0.smithi088.stdout: create 266/31 file after the snapshot 2023-10-16T20:41:25.501 INFO:tasks.workunit.client.0.smithi088.stdout: create 266/32 file after the snapshot 2023-10-16T20:41:25.504 INFO:tasks.workunit.client.0.smithi088.stdout: create 266/33 file after the snapshot 2023-10-16T20:41:25.507 INFO:tasks.workunit.client.0.smithi088.stdout: create 266/34 file after the snapshot 2023-10-16T20:41:25.509 INFO:tasks.workunit.client.0.smithi088.stdout: create 266/35 file after the snapshot 2023-10-16T20:41:25.512 INFO:tasks.workunit.client.0.smithi088.stdout: create 266/36 file after the snapshot 2023-10-16T20:41:25.514 INFO:tasks.workunit.client.0.smithi088.stdout: create 266/37 file after the snapshot 2023-10-16T20:41:25.516 INFO:tasks.workunit.client.0.smithi088.stdout: create 266/38 file after the snapshot 2023-10-16T20:41:25.519 INFO:tasks.workunit.client.0.smithi088.stdout: create 266/39 file after the snapshot 2023-10-16T20:41:25.521 INFO:tasks.workunit.client.0.smithi088.stdout: create 267/.snap/snap-subdir-test 2023-10-16T20:41:25.541 INFO:tasks.workunit.client.0.smithi088.stdout: create 267/30 file after the snapshot 2023-10-16T20:41:25.543 INFO:tasks.workunit.client.0.smithi088.stdout: create 267/31 file after the snapshot 2023-10-16T20:41:25.545 INFO:tasks.workunit.client.0.smithi088.stdout: create 267/32 file after the snapshot 2023-10-16T20:41:25.547 INFO:tasks.workunit.client.0.smithi088.stdout: create 267/33 file after the snapshot 2023-10-16T20:41:25.550 INFO:tasks.workunit.client.0.smithi088.stdout: create 267/34 file after the snapshot 2023-10-16T20:41:25.553 INFO:tasks.workunit.client.0.smithi088.stdout: create 267/35 file after the snapshot 2023-10-16T20:41:25.555 INFO:tasks.workunit.client.0.smithi088.stdout: create 267/36 file after the snapshot 2023-10-16T20:41:25.557 INFO:tasks.workunit.client.0.smithi088.stdout: create 267/37 file after the snapshot 2023-10-16T20:41:25.560 INFO:tasks.workunit.client.0.smithi088.stdout: create 267/38 file after the snapshot 2023-10-16T20:41:25.562 INFO:tasks.workunit.client.0.smithi088.stdout: create 267/39 file after the snapshot 2023-10-16T20:41:25.565 INFO:tasks.workunit.client.0.smithi088.stdout: create 268/.snap/snap-subdir-test 2023-10-16T20:41:25.585 INFO:tasks.workunit.client.0.smithi088.stdout: create 268/30 file after the snapshot 2023-10-16T20:41:25.587 INFO:tasks.workunit.client.0.smithi088.stdout: create 268/31 file after the snapshot 2023-10-16T20:41:25.589 INFO:tasks.workunit.client.0.smithi088.stdout: create 268/32 file after the snapshot 2023-10-16T20:41:25.591 INFO:tasks.workunit.client.0.smithi088.stdout: create 268/33 file after the snapshot 2023-10-16T20:41:25.594 INFO:tasks.workunit.client.0.smithi088.stdout: create 268/34 file after the snapshot 2023-10-16T20:41:25.596 INFO:tasks.workunit.client.0.smithi088.stdout: create 268/35 file after the snapshot 2023-10-16T20:41:25.599 INFO:tasks.workunit.client.0.smithi088.stdout: create 268/36 file after the snapshot 2023-10-16T20:41:25.601 INFO:tasks.workunit.client.0.smithi088.stdout: create 268/37 file after the snapshot 2023-10-16T20:41:25.604 INFO:tasks.workunit.client.0.smithi088.stdout: create 268/38 file after the snapshot 2023-10-16T20:41:25.606 INFO:tasks.workunit.client.0.smithi088.stdout: create 268/39 file after the snapshot 2023-10-16T20:41:25.609 INFO:tasks.workunit.client.0.smithi088.stdout: create 269/.snap/snap-subdir-test 2023-10-16T20:41:25.629 INFO:tasks.workunit.client.0.smithi088.stdout: create 269/30 file after the snapshot 2023-10-16T20:41:25.631 INFO:tasks.workunit.client.0.smithi088.stdout: create 269/31 file after the snapshot 2023-10-16T20:41:25.634 INFO:tasks.workunit.client.0.smithi088.stdout: create 269/32 file after the snapshot 2023-10-16T20:41:25.636 INFO:tasks.workunit.client.0.smithi088.stdout: create 269/33 file after the snapshot 2023-10-16T20:41:25.639 INFO:tasks.workunit.client.0.smithi088.stdout: create 269/34 file after the snapshot 2023-10-16T20:41:25.642 INFO:tasks.workunit.client.0.smithi088.stdout: create 269/35 file after the snapshot 2023-10-16T20:41:26.334 INFO:tasks.workunit.client.0.smithi088.stdout: create 269/36 file after the snapshot 2023-10-16T20:41:26.422 INFO:tasks.workunit.client.0.smithi088.stdout: create 269/37 file after the snapshot 2023-10-16T20:41:26.426 INFO:tasks.workunit.client.0.smithi088.stdout: create 269/38 file after the snapshot 2023-10-16T20:41:26.428 INFO:tasks.workunit.client.0.smithi088.stdout: create 269/39 file after the snapshot 2023-10-16T20:41:26.432 INFO:tasks.workunit.client.0.smithi088.stdout: create 270/.snap/snap-subdir-test 2023-10-16T20:41:27.219 INFO:tasks.workunit.client.0.smithi088.stdout: create 270/30 file after the snapshot 2023-10-16T20:41:27.221 INFO:tasks.workunit.client.0.smithi088.stdout: create 270/31 file after the snapshot 2023-10-16T20:41:27.224 INFO:tasks.workunit.client.0.smithi088.stdout: create 270/32 file after the snapshot 2023-10-16T20:41:27.227 INFO:tasks.workunit.client.0.smithi088.stdout: create 270/33 file after the snapshot 2023-10-16T20:41:27.230 INFO:tasks.workunit.client.0.smithi088.stdout: create 270/34 file after the snapshot 2023-10-16T20:41:27.233 INFO:tasks.workunit.client.0.smithi088.stdout: create 270/35 file after the snapshot 2023-10-16T20:41:27.236 INFO:tasks.workunit.client.0.smithi088.stdout: create 270/36 file after the snapshot 2023-10-16T20:41:27.239 INFO:tasks.workunit.client.0.smithi088.stdout: create 270/37 file after the snapshot 2023-10-16T20:41:27.242 INFO:tasks.workunit.client.0.smithi088.stdout: create 270/38 file after the snapshot 2023-10-16T20:41:27.245 INFO:tasks.workunit.client.0.smithi088.stdout: create 270/39 file after the snapshot 2023-10-16T20:41:27.247 INFO:tasks.workunit.client.0.smithi088.stdout: create 271/.snap/snap-subdir-test 2023-10-16T20:41:27.379 INFO:tasks.workunit.client.0.smithi088.stdout: create 271/30 file after the snapshot 2023-10-16T20:41:27.381 INFO:tasks.workunit.client.0.smithi088.stdout: create 271/31 file after the snapshot 2023-10-16T20:41:27.384 INFO:tasks.workunit.client.0.smithi088.stdout: create 271/32 file after the snapshot 2023-10-16T20:41:27.386 INFO:tasks.workunit.client.0.smithi088.stdout: create 271/33 file after the snapshot 2023-10-16T20:41:27.389 INFO:tasks.workunit.client.0.smithi088.stdout: create 271/34 file after the snapshot 2023-10-16T20:41:27.391 INFO:tasks.workunit.client.0.smithi088.stdout: create 271/35 file after the snapshot 2023-10-16T20:41:27.394 INFO:tasks.workunit.client.0.smithi088.stdout: create 271/36 file after the snapshot 2023-10-16T20:41:27.397 INFO:tasks.workunit.client.0.smithi088.stdout: create 271/37 file after the snapshot 2023-10-16T20:41:27.399 INFO:tasks.workunit.client.0.smithi088.stdout: create 271/38 file after the snapshot 2023-10-16T20:41:27.401 INFO:tasks.workunit.client.0.smithi088.stdout: create 271/39 file after the snapshot 2023-10-16T20:41:27.404 INFO:tasks.workunit.client.0.smithi088.stdout: create 272/.snap/snap-subdir-test 2023-10-16T20:41:27.423 INFO:tasks.workunit.client.0.smithi088.stdout: create 272/30 file after the snapshot 2023-10-16T20:41:27.426 INFO:tasks.workunit.client.0.smithi088.stdout: create 272/31 file after the snapshot 2023-10-16T20:41:27.428 INFO:tasks.workunit.client.0.smithi088.stdout: create 272/32 file after the snapshot 2023-10-16T20:41:27.431 INFO:tasks.workunit.client.0.smithi088.stdout: create 272/33 file after the snapshot 2023-10-16T20:41:27.433 INFO:tasks.workunit.client.0.smithi088.stdout: create 272/34 file after the snapshot 2023-10-16T20:41:27.436 INFO:tasks.workunit.client.0.smithi088.stdout: create 272/35 file after the snapshot 2023-10-16T20:41:27.439 INFO:tasks.workunit.client.0.smithi088.stdout: create 272/36 file after the snapshot 2023-10-16T20:41:27.441 INFO:tasks.workunit.client.0.smithi088.stdout: create 272/37 file after the snapshot 2023-10-16T20:41:27.444 INFO:tasks.workunit.client.0.smithi088.stdout: create 272/38 file after the snapshot 2023-10-16T20:41:27.446 INFO:tasks.workunit.client.0.smithi088.stdout: create 272/39 file after the snapshot 2023-10-16T20:41:27.449 INFO:tasks.workunit.client.0.smithi088.stdout: create 273/.snap/snap-subdir-test 2023-10-16T20:41:27.482 INFO:tasks.workunit.client.0.smithi088.stdout: create 273/30 file after the snapshot 2023-10-16T20:41:27.484 INFO:tasks.workunit.client.0.smithi088.stdout: create 273/31 file after the snapshot 2023-10-16T20:41:27.487 INFO:tasks.workunit.client.0.smithi088.stdout: create 273/32 file after the snapshot 2023-10-16T20:41:27.490 INFO:tasks.workunit.client.0.smithi088.stdout: create 273/33 file after the snapshot 2023-10-16T20:41:27.492 INFO:tasks.workunit.client.0.smithi088.stdout: create 273/34 file after the snapshot 2023-10-16T20:41:27.494 INFO:tasks.workunit.client.0.smithi088.stdout: create 273/35 file after the snapshot 2023-10-16T20:41:27.497 INFO:tasks.workunit.client.0.smithi088.stdout: create 273/36 file after the snapshot 2023-10-16T20:41:27.500 INFO:tasks.workunit.client.0.smithi088.stdout: create 273/37 file after the snapshot 2023-10-16T20:41:27.504 INFO:tasks.workunit.client.0.smithi088.stdout: create 273/38 file after the snapshot 2023-10-16T20:41:27.506 INFO:tasks.workunit.client.0.smithi088.stdout: create 273/39 file after the snapshot 2023-10-16T20:41:27.509 INFO:tasks.workunit.client.0.smithi088.stdout: create 274/.snap/snap-subdir-test 2023-10-16T20:41:28.672 INFO:tasks.workunit.client.0.smithi088.stdout: create 274/30 file after the snapshot 2023-10-16T20:41:28.674 INFO:tasks.workunit.client.0.smithi088.stdout: create 274/31 file after the snapshot 2023-10-16T20:41:28.677 INFO:tasks.workunit.client.0.smithi088.stdout: create 274/32 file after the snapshot 2023-10-16T20:41:28.680 INFO:tasks.workunit.client.0.smithi088.stdout: create 274/33 file after the snapshot 2023-10-16T20:41:28.683 INFO:tasks.workunit.client.0.smithi088.stdout: create 274/34 file after the snapshot 2023-10-16T20:41:28.686 INFO:tasks.workunit.client.0.smithi088.stdout: create 274/35 file after the snapshot 2023-10-16T20:41:28.689 INFO:tasks.workunit.client.0.smithi088.stdout: create 274/36 file after the snapshot 2023-10-16T20:41:28.692 INFO:tasks.workunit.client.0.smithi088.stdout: create 274/37 file after the snapshot 2023-10-16T20:41:28.695 INFO:tasks.workunit.client.0.smithi088.stdout: create 274/38 file after the snapshot 2023-10-16T20:41:28.698 INFO:tasks.workunit.client.0.smithi088.stdout: create 274/39 file after the snapshot 2023-10-16T20:41:28.701 INFO:tasks.workunit.client.0.smithi088.stdout: create 275/.snap/snap-subdir-test 2023-10-16T20:41:28.717 INFO:tasks.workunit.client.0.smithi088.stdout: create 275/30 file after the snapshot 2023-10-16T20:41:28.719 INFO:tasks.workunit.client.0.smithi088.stdout: create 275/31 file after the snapshot 2023-10-16T20:41:28.722 INFO:tasks.workunit.client.0.smithi088.stdout: create 275/32 file after the snapshot 2023-10-16T20:41:28.724 INFO:tasks.workunit.client.0.smithi088.stdout: create 275/33 file after the snapshot 2023-10-16T20:41:28.726 INFO:tasks.workunit.client.0.smithi088.stdout: create 275/34 file after the snapshot 2023-10-16T20:41:28.729 INFO:tasks.workunit.client.0.smithi088.stdout: create 275/35 file after the snapshot 2023-10-16T20:41:28.734 INFO:tasks.workunit.client.0.smithi088.stdout: create 275/36 file after the snapshot 2023-10-16T20:41:28.735 INFO:tasks.workunit.client.0.smithi088.stdout: create 275/37 file after the snapshot 2023-10-16T20:41:28.736 INFO:tasks.workunit.client.0.smithi088.stdout: create 275/38 file after the snapshot 2023-10-16T20:41:28.739 INFO:tasks.workunit.client.0.smithi088.stdout: create 275/39 file after the snapshot 2023-10-16T20:41:28.742 INFO:tasks.workunit.client.0.smithi088.stdout: create 276/.snap/snap-subdir-test 2023-10-16T20:41:28.764 INFO:tasks.workunit.client.0.smithi088.stdout: create 276/30 file after the snapshot 2023-10-16T20:41:28.768 INFO:tasks.workunit.client.0.smithi088.stdout: create 276/31 file after the snapshot 2023-10-16T20:41:28.773 INFO:tasks.workunit.client.0.smithi088.stdout: create 276/32 file after the snapshot 2023-10-16T20:41:28.777 INFO:tasks.workunit.client.0.smithi088.stdout: create 276/33 file after the snapshot 2023-10-16T20:41:28.781 INFO:tasks.workunit.client.0.smithi088.stdout: create 276/34 file after the snapshot 2023-10-16T20:41:28.785 INFO:tasks.workunit.client.0.smithi088.stdout: create 276/35 file after the snapshot 2023-10-16T20:41:28.789 INFO:tasks.workunit.client.0.smithi088.stdout: create 276/36 file after the snapshot 2023-10-16T20:41:28.794 INFO:tasks.workunit.client.0.smithi088.stdout: create 276/37 file after the snapshot 2023-10-16T20:41:28.798 INFO:tasks.workunit.client.0.smithi088.stdout: create 276/38 file after the snapshot 2023-10-16T20:41:28.802 INFO:tasks.workunit.client.0.smithi088.stdout: create 276/39 file after the snapshot 2023-10-16T20:41:28.806 INFO:tasks.workunit.client.0.smithi088.stdout: create 277/.snap/snap-subdir-test 2023-10-16T20:41:28.822 INFO:tasks.workunit.client.0.smithi088.stdout: create 277/30 file after the snapshot 2023-10-16T20:41:28.824 INFO:tasks.workunit.client.0.smithi088.stdout: create 277/31 file after the snapshot 2023-10-16T20:41:28.827 INFO:tasks.workunit.client.0.smithi088.stdout: create 277/32 file after the snapshot 2023-10-16T20:41:28.830 INFO:tasks.workunit.client.0.smithi088.stdout: create 277/33 file after the snapshot 2023-10-16T20:41:28.832 INFO:tasks.workunit.client.0.smithi088.stdout: create 277/34 file after the snapshot 2023-10-16T20:41:28.834 INFO:tasks.workunit.client.0.smithi088.stdout: create 277/35 file after the snapshot 2023-10-16T20:41:28.837 INFO:tasks.workunit.client.0.smithi088.stdout: create 277/36 file after the snapshot 2023-10-16T20:41:28.840 INFO:tasks.workunit.client.0.smithi088.stdout: create 277/37 file after the snapshot 2023-10-16T20:41:28.842 INFO:tasks.workunit.client.0.smithi088.stdout: create 277/38 file after the snapshot 2023-10-16T20:41:28.845 INFO:tasks.workunit.client.0.smithi088.stdout: create 277/39 file after the snapshot 2023-10-16T20:41:28.847 INFO:tasks.workunit.client.0.smithi088.stdout: create 278/.snap/snap-subdir-test 2023-10-16T20:41:28.911 INFO:tasks.workunit.client.0.smithi088.stdout: create 278/30 file after the snapshot 2023-10-16T20:41:28.913 INFO:tasks.workunit.client.0.smithi088.stdout: create 278/31 file after the snapshot 2023-10-16T20:41:28.916 INFO:tasks.workunit.client.0.smithi088.stdout: create 278/32 file after the snapshot 2023-10-16T20:41:28.918 INFO:tasks.workunit.client.0.smithi088.stdout: create 278/33 file after the snapshot 2023-10-16T20:41:28.921 INFO:tasks.workunit.client.0.smithi088.stdout: create 278/34 file after the snapshot 2023-10-16T20:41:28.923 INFO:tasks.workunit.client.0.smithi088.stdout: create 278/35 file after the snapshot 2023-10-16T20:41:28.926 INFO:tasks.workunit.client.0.smithi088.stdout: create 278/36 file after the snapshot 2023-10-16T20:41:28.929 INFO:tasks.workunit.client.0.smithi088.stdout: create 278/37 file after the snapshot 2023-10-16T20:41:28.931 INFO:tasks.workunit.client.0.smithi088.stdout: create 278/38 file after the snapshot 2023-10-16T20:41:28.934 INFO:tasks.workunit.client.0.smithi088.stdout: create 278/39 file after the snapshot 2023-10-16T20:41:28.936 INFO:tasks.workunit.client.0.smithi088.stdout: create 279/.snap/snap-subdir-test 2023-10-16T20:41:29.173 INFO:tasks.workunit.client.0.smithi088.stdout: create 279/30 file after the snapshot 2023-10-16T20:41:29.175 INFO:tasks.workunit.client.0.smithi088.stdout: create 279/31 file after the snapshot 2023-10-16T20:41:29.178 INFO:tasks.workunit.client.0.smithi088.stdout: create 279/32 file after the snapshot 2023-10-16T20:41:29.181 INFO:tasks.workunit.client.0.smithi088.stdout: create 279/33 file after the snapshot 2023-10-16T20:41:29.184 INFO:tasks.workunit.client.0.smithi088.stdout: create 279/34 file after the snapshot 2023-10-16T20:41:29.186 INFO:tasks.workunit.client.0.smithi088.stdout: create 279/35 file after the snapshot 2023-10-16T20:41:29.189 INFO:tasks.workunit.client.0.smithi088.stdout: create 279/36 file after the snapshot 2023-10-16T20:41:29.191 INFO:tasks.workunit.client.0.smithi088.stdout: create 279/37 file after the snapshot 2023-10-16T20:41:29.194 INFO:tasks.workunit.client.0.smithi088.stdout: create 279/38 file after the snapshot 2023-10-16T20:41:29.197 INFO:tasks.workunit.client.0.smithi088.stdout: create 279/39 file after the snapshot 2023-10-16T20:41:29.199 INFO:tasks.workunit.client.0.smithi088.stdout: create 280/.snap/snap-subdir-test 2023-10-16T20:41:29.218 INFO:tasks.workunit.client.0.smithi088.stdout: create 280/30 file after the snapshot 2023-10-16T20:41:29.220 INFO:tasks.workunit.client.0.smithi088.stdout: create 280/31 file after the snapshot 2023-10-16T20:41:29.223 INFO:tasks.workunit.client.0.smithi088.stdout: create 280/32 file after the snapshot 2023-10-16T20:41:29.225 INFO:tasks.workunit.client.0.smithi088.stdout: create 280/33 file after the snapshot 2023-10-16T20:41:29.228 INFO:tasks.workunit.client.0.smithi088.stdout: create 280/34 file after the snapshot 2023-10-16T20:41:29.230 INFO:tasks.workunit.client.0.smithi088.stdout: create 280/35 file after the snapshot 2023-10-16T20:41:29.233 INFO:tasks.workunit.client.0.smithi088.stdout: create 280/36 file after the snapshot 2023-10-16T20:41:29.236 INFO:tasks.workunit.client.0.smithi088.stdout: create 280/37 file after the snapshot 2023-10-16T20:41:29.238 INFO:tasks.workunit.client.0.smithi088.stdout: create 280/38 file after the snapshot 2023-10-16T20:41:29.241 INFO:tasks.workunit.client.0.smithi088.stdout: create 280/39 file after the snapshot 2023-10-16T20:41:29.244 INFO:tasks.workunit.client.0.smithi088.stdout: create 281/.snap/snap-subdir-test 2023-10-16T20:41:29.528 INFO:tasks.workunit.client.0.smithi088.stdout: create 281/30 file after the snapshot 2023-10-16T20:41:29.530 INFO:tasks.workunit.client.0.smithi088.stdout: create 281/31 file after the snapshot 2023-10-16T20:41:29.532 INFO:tasks.workunit.client.0.smithi088.stdout: create 281/32 file after the snapshot 2023-10-16T20:41:29.535 INFO:tasks.workunit.client.0.smithi088.stdout: create 281/33 file after the snapshot 2023-10-16T20:41:29.537 INFO:tasks.workunit.client.0.smithi088.stdout: create 281/34 file after the snapshot 2023-10-16T20:41:29.540 INFO:tasks.workunit.client.0.smithi088.stdout: create 281/35 file after the snapshot 2023-10-16T20:41:29.543 INFO:tasks.workunit.client.0.smithi088.stdout: create 281/36 file after the snapshot 2023-10-16T20:41:29.545 INFO:tasks.workunit.client.0.smithi088.stdout: create 281/37 file after the snapshot 2023-10-16T20:41:29.548 INFO:tasks.workunit.client.0.smithi088.stdout: create 281/38 file after the snapshot 2023-10-16T20:41:29.550 INFO:tasks.workunit.client.0.smithi088.stdout: create 281/39 file after the snapshot 2023-10-16T20:41:29.553 INFO:tasks.workunit.client.0.smithi088.stdout: create 282/.snap/snap-subdir-test 2023-10-16T20:41:29.761 INFO:tasks.workunit.client.0.smithi088.stdout: create 282/30 file after the snapshot 2023-10-16T20:41:29.765 INFO:tasks.workunit.client.0.smithi088.stdout: create 282/31 file after the snapshot 2023-10-16T20:41:29.769 INFO:tasks.workunit.client.0.smithi088.stdout: create 282/32 file after the snapshot 2023-10-16T20:41:29.774 INFO:tasks.workunit.client.0.smithi088.stdout: create 282/33 file after the snapshot 2023-10-16T20:41:29.778 INFO:tasks.workunit.client.0.smithi088.stdout: create 282/34 file after the snapshot 2023-10-16T20:41:29.782 INFO:tasks.workunit.client.0.smithi088.stdout: create 282/35 file after the snapshot 2023-10-16T20:41:29.786 INFO:tasks.workunit.client.0.smithi088.stdout: create 282/36 file after the snapshot 2023-10-16T20:41:29.790 INFO:tasks.workunit.client.0.smithi088.stdout: create 282/37 file after the snapshot 2023-10-16T20:41:29.794 INFO:tasks.workunit.client.0.smithi088.stdout: create 282/38 file after the snapshot 2023-10-16T20:41:29.798 INFO:tasks.workunit.client.0.smithi088.stdout: create 282/39 file after the snapshot 2023-10-16T20:41:29.802 INFO:tasks.workunit.client.0.smithi088.stdout: create 283/.snap/snap-subdir-test 2023-10-16T20:41:29.819 INFO:tasks.workunit.client.0.smithi088.stdout: create 283/30 file after the snapshot 2023-10-16T20:41:29.824 INFO:tasks.workunit.client.0.smithi088.stdout: create 283/31 file after the snapshot 2023-10-16T20:41:29.825 INFO:tasks.workunit.client.0.smithi088.stdout: create 283/32 file after the snapshot 2023-10-16T20:41:29.828 INFO:tasks.workunit.client.0.smithi088.stdout: create 283/33 file after the snapshot 2023-10-16T20:41:29.830 INFO:tasks.workunit.client.0.smithi088.stdout: create 283/34 file after the snapshot 2023-10-16T20:41:29.832 INFO:tasks.workunit.client.0.smithi088.stdout: create 283/35 file after the snapshot 2023-10-16T20:41:29.835 INFO:tasks.workunit.client.0.smithi088.stdout: create 283/36 file after the snapshot 2023-10-16T20:41:29.837 INFO:tasks.workunit.client.0.smithi088.stdout: create 283/37 file after the snapshot 2023-10-16T20:41:29.840 INFO:tasks.workunit.client.0.smithi088.stdout: create 283/38 file after the snapshot 2023-10-16T20:41:29.842 INFO:tasks.workunit.client.0.smithi088.stdout: create 283/39 file after the snapshot 2023-10-16T20:41:29.845 INFO:tasks.workunit.client.0.smithi088.stdout: create 284/.snap/snap-subdir-test 2023-10-16T20:41:30.553 INFO:tasks.workunit.client.0.smithi088.stdout: create 284/30 file after the snapshot 2023-10-16T20:41:30.556 INFO:tasks.workunit.client.0.smithi088.stdout: create 284/31 file after the snapshot 2023-10-16T20:41:30.559 INFO:tasks.workunit.client.0.smithi088.stdout: create 284/32 file after the snapshot 2023-10-16T20:41:30.561 INFO:tasks.workunit.client.0.smithi088.stdout: create 284/33 file after the snapshot 2023-10-16T20:41:30.564 INFO:tasks.workunit.client.0.smithi088.stdout: create 284/34 file after the snapshot 2023-10-16T20:41:30.567 INFO:tasks.workunit.client.0.smithi088.stdout: create 284/35 file after the snapshot 2023-10-16T20:41:30.570 INFO:tasks.workunit.client.0.smithi088.stdout: create 284/36 file after the snapshot 2023-10-16T20:41:30.572 INFO:tasks.workunit.client.0.smithi088.stdout: create 284/37 file after the snapshot 2023-10-16T20:41:30.575 INFO:tasks.workunit.client.0.smithi088.stdout: create 284/38 file after the snapshot 2023-10-16T20:41:30.578 INFO:tasks.workunit.client.0.smithi088.stdout: create 284/39 file after the snapshot 2023-10-16T20:41:30.581 INFO:tasks.workunit.client.0.smithi088.stdout: create 285/.snap/snap-subdir-test 2023-10-16T20:41:30.599 INFO:tasks.workunit.client.0.smithi088.stdout: create 285/30 file after the snapshot 2023-10-16T20:41:30.601 INFO:tasks.workunit.client.0.smithi088.stdout: create 285/31 file after the snapshot 2023-10-16T20:41:30.603 INFO:tasks.workunit.client.0.smithi088.stdout: create 285/32 file after the snapshot 2023-10-16T20:41:30.606 INFO:tasks.workunit.client.0.smithi088.stdout: create 285/33 file after the snapshot 2023-10-16T20:41:30.608 INFO:tasks.workunit.client.0.smithi088.stdout: create 285/34 file after the snapshot 2023-10-16T20:41:30.611 INFO:tasks.workunit.client.0.smithi088.stdout: create 285/35 file after the snapshot 2023-10-16T20:41:30.614 INFO:tasks.workunit.client.0.smithi088.stdout: create 285/36 file after the snapshot 2023-10-16T20:41:30.616 INFO:tasks.workunit.client.0.smithi088.stdout: create 285/37 file after the snapshot 2023-10-16T20:41:30.619 INFO:tasks.workunit.client.0.smithi088.stdout: create 285/38 file after the snapshot 2023-10-16T20:41:30.621 INFO:tasks.workunit.client.0.smithi088.stdout: create 285/39 file after the snapshot 2023-10-16T20:41:30.624 INFO:tasks.workunit.client.0.smithi088.stdout: create 286/.snap/snap-subdir-test 2023-10-16T20:41:30.645 INFO:tasks.workunit.client.0.smithi088.stdout: create 286/30 file after the snapshot 2023-10-16T20:41:30.648 INFO:tasks.workunit.client.0.smithi088.stdout: create 286/31 file after the snapshot 2023-10-16T20:41:30.653 INFO:tasks.workunit.client.0.smithi088.stdout: create 286/32 file after the snapshot 2023-10-16T20:41:30.658 INFO:tasks.workunit.client.0.smithi088.stdout: create 286/33 file after the snapshot 2023-10-16T20:41:30.662 INFO:tasks.workunit.client.0.smithi088.stdout: create 286/34 file after the snapshot 2023-10-16T20:41:30.666 INFO:tasks.workunit.client.0.smithi088.stdout: create 286/35 file after the snapshot 2023-10-16T20:41:30.671 INFO:tasks.workunit.client.0.smithi088.stdout: create 286/36 file after the snapshot 2023-10-16T20:41:30.675 INFO:tasks.workunit.client.0.smithi088.stdout: create 286/37 file after the snapshot 2023-10-16T20:41:30.679 INFO:tasks.workunit.client.0.smithi088.stdout: create 286/38 file after the snapshot 2023-10-16T20:41:30.683 INFO:tasks.workunit.client.0.smithi088.stdout: create 286/39 file after the snapshot 2023-10-16T20:41:30.687 INFO:tasks.workunit.client.0.smithi088.stdout: create 287/.snap/snap-subdir-test 2023-10-16T20:41:30.703 INFO:tasks.workunit.client.0.smithi088.stdout: create 287/30 file after the snapshot 2023-10-16T20:41:30.705 INFO:tasks.workunit.client.0.smithi088.stdout: create 287/31 file after the snapshot 2023-10-16T20:41:30.707 INFO:tasks.workunit.client.0.smithi088.stdout: create 287/32 file after the snapshot 2023-10-16T20:41:30.710 INFO:tasks.workunit.client.0.smithi088.stdout: create 287/33 file after the snapshot 2023-10-16T20:41:30.712 INFO:tasks.workunit.client.0.smithi088.stdout: create 287/34 file after the snapshot 2023-10-16T20:41:30.715 INFO:tasks.workunit.client.0.smithi088.stdout: create 287/35 file after the snapshot 2023-10-16T20:41:30.718 INFO:tasks.workunit.client.0.smithi088.stdout: create 287/36 file after the snapshot 2023-10-16T20:41:30.720 INFO:tasks.workunit.client.0.smithi088.stdout: create 287/37 file after the snapshot 2023-10-16T20:41:30.722 INFO:tasks.workunit.client.0.smithi088.stdout: create 287/38 file after the snapshot 2023-10-16T20:41:30.725 INFO:tasks.workunit.client.0.smithi088.stdout: create 287/39 file after the snapshot 2023-10-16T20:41:30.728 INFO:tasks.workunit.client.0.smithi088.stdout: create 288/.snap/snap-subdir-test 2023-10-16T20:41:30.744 INFO:tasks.workunit.client.0.smithi088.stdout: create 288/30 file after the snapshot 2023-10-16T20:41:30.746 INFO:tasks.workunit.client.0.smithi088.stdout: create 288/31 file after the snapshot 2023-10-16T20:41:30.748 INFO:tasks.workunit.client.0.smithi088.stdout: create 288/32 file after the snapshot 2023-10-16T20:41:30.754 INFO:tasks.workunit.client.0.smithi088.stdout: create 288/33 file after the snapshot 2023-10-16T20:41:30.755 INFO:tasks.workunit.client.0.smithi088.stdout: create 288/34 file after the snapshot 2023-10-16T20:41:30.756 INFO:tasks.workunit.client.0.smithi088.stdout: create 288/35 file after the snapshot 2023-10-16T20:41:30.758 INFO:tasks.workunit.client.0.smithi088.stdout: create 288/36 file after the snapshot 2023-10-16T20:41:30.761 INFO:tasks.workunit.client.0.smithi088.stdout: create 288/37 file after the snapshot 2023-10-16T20:41:30.764 INFO:tasks.workunit.client.0.smithi088.stdout: create 288/38 file after the snapshot 2023-10-16T20:41:30.766 INFO:tasks.workunit.client.0.smithi088.stdout: create 288/39 file after the snapshot 2023-10-16T20:41:30.769 INFO:tasks.workunit.client.0.smithi088.stdout: create 289/.snap/snap-subdir-test 2023-10-16T20:41:30.787 INFO:tasks.workunit.client.0.smithi088.stdout: create 289/30 file after the snapshot 2023-10-16T20:41:30.789 INFO:tasks.workunit.client.0.smithi088.stdout: create 289/31 file after the snapshot 2023-10-16T20:41:30.791 INFO:tasks.workunit.client.0.smithi088.stdout: create 289/32 file after the snapshot 2023-10-16T20:41:30.794 INFO:tasks.workunit.client.0.smithi088.stdout: create 289/33 file after the snapshot 2023-10-16T20:41:30.796 INFO:tasks.workunit.client.0.smithi088.stdout: create 289/34 file after the snapshot 2023-10-16T20:41:30.798 INFO:tasks.workunit.client.0.smithi088.stdout: create 289/35 file after the snapshot 2023-10-16T20:41:30.801 INFO:tasks.workunit.client.0.smithi088.stdout: create 289/36 file after the snapshot 2023-10-16T20:41:30.803 INFO:tasks.workunit.client.0.smithi088.stdout: create 289/37 file after the snapshot 2023-10-16T20:41:30.806 INFO:tasks.workunit.client.0.smithi088.stdout: create 289/38 file after the snapshot 2023-10-16T20:41:30.808 INFO:tasks.workunit.client.0.smithi088.stdout: create 289/39 file after the snapshot 2023-10-16T20:41:30.811 INFO:tasks.workunit.client.0.smithi088.stdout: create 290/.snap/snap-subdir-test 2023-10-16T20:41:30.837 INFO:tasks.workunit.client.0.smithi088.stdout: create 290/30 file after the snapshot 2023-10-16T20:41:30.839 INFO:tasks.workunit.client.0.smithi088.stdout: create 290/31 file after the snapshot 2023-10-16T20:41:30.841 INFO:tasks.workunit.client.0.smithi088.stdout: create 290/32 file after the snapshot 2023-10-16T20:41:30.844 INFO:tasks.workunit.client.0.smithi088.stdout: create 290/33 file after the snapshot 2023-10-16T20:41:30.847 INFO:tasks.workunit.client.0.smithi088.stdout: create 290/34 file after the snapshot 2023-10-16T20:41:30.849 INFO:tasks.workunit.client.0.smithi088.stdout: create 290/35 file after the snapshot 2023-10-16T20:41:30.852 INFO:tasks.workunit.client.0.smithi088.stdout: create 290/36 file after the snapshot 2023-10-16T20:41:30.854 INFO:tasks.workunit.client.0.smithi088.stdout: create 290/37 file after the snapshot 2023-10-16T20:41:30.858 INFO:tasks.workunit.client.0.smithi088.stdout: create 290/38 file after the snapshot 2023-10-16T20:41:30.859 INFO:tasks.workunit.client.0.smithi088.stdout: create 290/39 file after the snapshot 2023-10-16T20:41:30.861 INFO:tasks.workunit.client.0.smithi088.stdout: create 291/.snap/snap-subdir-test 2023-10-16T20:41:30.877 INFO:tasks.workunit.client.0.smithi088.stdout: create 291/30 file after the snapshot 2023-10-16T20:41:30.880 INFO:tasks.workunit.client.0.smithi088.stdout: create 291/31 file after the snapshot 2023-10-16T20:41:30.882 INFO:tasks.workunit.client.0.smithi088.stdout: create 291/32 file after the snapshot 2023-10-16T20:41:30.885 INFO:tasks.workunit.client.0.smithi088.stdout: create 291/33 file after the snapshot 2023-10-16T20:41:30.888 INFO:tasks.workunit.client.0.smithi088.stdout: create 291/34 file after the snapshot 2023-10-16T20:41:30.890 INFO:tasks.workunit.client.0.smithi088.stdout: create 291/35 file after the snapshot 2023-10-16T20:41:30.893 INFO:tasks.workunit.client.0.smithi088.stdout: create 291/36 file after the snapshot 2023-10-16T20:41:30.895 INFO:tasks.workunit.client.0.smithi088.stdout: create 291/37 file after the snapshot 2023-10-16T20:41:30.898 INFO:tasks.workunit.client.0.smithi088.stdout: create 291/38 file after the snapshot 2023-10-16T20:41:30.900 INFO:tasks.workunit.client.0.smithi088.stdout: create 291/39 file after the snapshot 2023-10-16T20:41:30.903 INFO:tasks.workunit.client.0.smithi088.stdout: create 292/.snap/snap-subdir-test 2023-10-16T20:41:30.921 INFO:tasks.workunit.client.0.smithi088.stdout: create 292/30 file after the snapshot 2023-10-16T20:41:30.923 INFO:tasks.workunit.client.0.smithi088.stdout: create 292/31 file after the snapshot 2023-10-16T20:41:30.926 INFO:tasks.workunit.client.0.smithi088.stdout: create 292/32 file after the snapshot 2023-10-16T20:41:30.928 INFO:tasks.workunit.client.0.smithi088.stdout: create 292/33 file after the snapshot 2023-10-16T20:41:30.931 INFO:tasks.workunit.client.0.smithi088.stdout: create 292/34 file after the snapshot 2023-10-16T20:41:30.933 INFO:tasks.workunit.client.0.smithi088.stdout: create 292/35 file after the snapshot 2023-10-16T20:41:30.935 INFO:tasks.workunit.client.0.smithi088.stdout: create 292/36 file after the snapshot 2023-10-16T20:41:30.938 INFO:tasks.workunit.client.0.smithi088.stdout: create 292/37 file after the snapshot 2023-10-16T20:41:30.940 INFO:tasks.workunit.client.0.smithi088.stdout: create 292/38 file after the snapshot 2023-10-16T20:41:30.943 INFO:tasks.workunit.client.0.smithi088.stdout: create 292/39 file after the snapshot 2023-10-16T20:41:30.946 INFO:tasks.workunit.client.0.smithi088.stdout: create 293/.snap/snap-subdir-test 2023-10-16T20:41:30.961 INFO:tasks.workunit.client.0.smithi088.stdout: create 293/30 file after the snapshot 2023-10-16T20:41:30.963 INFO:tasks.workunit.client.0.smithi088.stdout: create 293/31 file after the snapshot 2023-10-16T20:41:30.965 INFO:tasks.workunit.client.0.smithi088.stdout: create 293/32 file after the snapshot 2023-10-16T20:41:30.967 INFO:tasks.workunit.client.0.smithi088.stdout: create 293/33 file after the snapshot 2023-10-16T20:41:30.969 INFO:tasks.workunit.client.0.smithi088.stdout: create 293/34 file after the snapshot 2023-10-16T20:41:30.972 INFO:tasks.workunit.client.0.smithi088.stdout: create 293/35 file after the snapshot 2023-10-16T20:41:30.974 INFO:tasks.workunit.client.0.smithi088.stdout: create 293/36 file after the snapshot 2023-10-16T20:41:30.976 INFO:tasks.workunit.client.0.smithi088.stdout: create 293/37 file after the snapshot 2023-10-16T20:41:30.979 INFO:tasks.workunit.client.0.smithi088.stdout: create 293/38 file after the snapshot 2023-10-16T20:41:30.981 INFO:tasks.workunit.client.0.smithi088.stdout: create 293/39 file after the snapshot 2023-10-16T20:41:30.983 INFO:tasks.workunit.client.0.smithi088.stdout: create 294/.snap/snap-subdir-test 2023-10-16T20:41:31.002 INFO:tasks.workunit.client.0.smithi088.stdout: create 294/30 file after the snapshot 2023-10-16T20:41:31.006 INFO:tasks.workunit.client.0.smithi088.stdout: create 294/31 file after the snapshot 2023-10-16T20:41:31.010 INFO:tasks.workunit.client.0.smithi088.stdout: create 294/32 file after the snapshot 2023-10-16T20:41:31.015 INFO:tasks.workunit.client.0.smithi088.stdout: create 294/33 file after the snapshot 2023-10-16T20:41:31.019 INFO:tasks.workunit.client.0.smithi088.stdout: create 294/34 file after the snapshot 2023-10-16T20:41:31.023 INFO:tasks.workunit.client.0.smithi088.stdout: create 294/35 file after the snapshot 2023-10-16T20:41:31.027 INFO:tasks.workunit.client.0.smithi088.stdout: create 294/36 file after the snapshot 2023-10-16T20:41:31.031 INFO:tasks.workunit.client.0.smithi088.stdout: create 294/37 file after the snapshot 2023-10-16T20:41:31.035 INFO:tasks.workunit.client.0.smithi088.stdout: create 294/38 file after the snapshot 2023-10-16T20:41:31.040 INFO:tasks.workunit.client.0.smithi088.stdout: create 294/39 file after the snapshot 2023-10-16T20:41:31.044 INFO:tasks.workunit.client.0.smithi088.stdout: create 295/.snap/snap-subdir-test 2023-10-16T20:41:31.059 INFO:tasks.workunit.client.0.smithi088.stdout: create 295/30 file after the snapshot 2023-10-16T20:41:31.061 INFO:tasks.workunit.client.0.smithi088.stdout: create 295/31 file after the snapshot 2023-10-16T20:41:31.063 INFO:tasks.workunit.client.0.smithi088.stdout: create 295/32 file after the snapshot 2023-10-16T20:41:31.066 INFO:tasks.workunit.client.0.smithi088.stdout: create 295/33 file after the snapshot 2023-10-16T20:41:31.068 INFO:tasks.workunit.client.0.smithi088.stdout: create 295/34 file after the snapshot 2023-10-16T20:41:31.070 INFO:tasks.workunit.client.0.smithi088.stdout: create 295/35 file after the snapshot 2023-10-16T20:41:31.073 INFO:tasks.workunit.client.0.smithi088.stdout: create 295/36 file after the snapshot 2023-10-16T20:41:31.076 INFO:tasks.workunit.client.0.smithi088.stdout: create 295/37 file after the snapshot 2023-10-16T20:41:31.079 INFO:tasks.workunit.client.0.smithi088.stdout: create 295/38 file after the snapshot 2023-10-16T20:41:31.081 INFO:tasks.workunit.client.0.smithi088.stdout: create 295/39 file after the snapshot 2023-10-16T20:41:31.083 INFO:tasks.workunit.client.0.smithi088.stdout: create 296/.snap/snap-subdir-test 2023-10-16T20:41:31.100 INFO:tasks.workunit.client.0.smithi088.stdout: create 296/30 file after the snapshot 2023-10-16T20:41:31.102 INFO:tasks.workunit.client.0.smithi088.stdout: create 296/31 file after the snapshot 2023-10-16T20:41:31.105 INFO:tasks.workunit.client.0.smithi088.stdout: create 296/32 file after the snapshot 2023-10-16T20:41:31.107 INFO:tasks.workunit.client.0.smithi088.stdout: create 296/33 file after the snapshot 2023-10-16T20:41:31.110 INFO:tasks.workunit.client.0.smithi088.stdout: create 296/34 file after the snapshot 2023-10-16T20:41:31.112 INFO:tasks.workunit.client.0.smithi088.stdout: create 296/35 file after the snapshot 2023-10-16T20:41:31.114 INFO:tasks.workunit.client.0.smithi088.stdout: create 296/36 file after the snapshot 2023-10-16T20:41:31.117 INFO:tasks.workunit.client.0.smithi088.stdout: create 296/37 file after the snapshot 2023-10-16T20:41:31.119 INFO:tasks.workunit.client.0.smithi088.stdout: create 296/38 file after the snapshot 2023-10-16T20:41:31.121 INFO:tasks.workunit.client.0.smithi088.stdout: create 296/39 file after the snapshot 2023-10-16T20:41:31.124 INFO:tasks.workunit.client.0.smithi088.stdout: create 297/.snap/snap-subdir-test 2023-10-16T20:41:31.143 INFO:tasks.workunit.client.0.smithi088.stdout: create 297/30 file after the snapshot 2023-10-16T20:41:31.146 INFO:tasks.workunit.client.0.smithi088.stdout: create 297/31 file after the snapshot 2023-10-16T20:41:31.148 INFO:tasks.workunit.client.0.smithi088.stdout: create 297/32 file after the snapshot 2023-10-16T20:41:31.154 INFO:tasks.workunit.client.0.smithi088.stdout: create 297/33 file after the snapshot 2023-10-16T20:41:31.156 INFO:tasks.workunit.client.0.smithi088.stdout: create 297/34 file after the snapshot 2023-10-16T20:41:31.159 INFO:tasks.workunit.client.0.smithi088.stdout: create 297/35 file after the snapshot 2023-10-16T20:41:31.161 INFO:tasks.workunit.client.0.smithi088.stdout: create 297/36 file after the snapshot 2023-10-16T20:41:31.164 INFO:tasks.workunit.client.0.smithi088.stdout: create 297/37 file after the snapshot 2023-10-16T20:41:31.167 INFO:tasks.workunit.client.0.smithi088.stdout: create 297/38 file after the snapshot 2023-10-16T20:41:31.169 INFO:tasks.workunit.client.0.smithi088.stdout: create 297/39 file after the snapshot 2023-10-16T20:41:31.172 INFO:tasks.workunit.client.0.smithi088.stdout: create 298/.snap/snap-subdir-test 2023-10-16T20:41:31.188 INFO:tasks.workunit.client.0.smithi088.stdout: create 298/30 file after the snapshot 2023-10-16T20:41:31.189 INFO:tasks.workunit.client.0.smithi088.stdout: create 298/31 file after the snapshot 2023-10-16T20:41:31.191 INFO:tasks.workunit.client.0.smithi088.stdout: create 298/32 file after the snapshot 2023-10-16T20:41:31.194 INFO:tasks.workunit.client.0.smithi088.stdout: create 298/33 file after the snapshot 2023-10-16T20:41:31.197 INFO:tasks.workunit.client.0.smithi088.stdout: create 298/34 file after the snapshot 2023-10-16T20:41:31.199 INFO:tasks.workunit.client.0.smithi088.stdout: create 298/35 file after the snapshot 2023-10-16T20:41:31.202 INFO:tasks.workunit.client.0.smithi088.stdout: create 298/36 file after the snapshot 2023-10-16T20:41:31.205 INFO:tasks.workunit.client.0.smithi088.stdout: create 298/37 file after the snapshot 2023-10-16T20:41:31.209 INFO:tasks.workunit.client.0.smithi088.stdout: create 298/38 file after the snapshot 2023-10-16T20:41:31.210 INFO:tasks.workunit.client.0.smithi088.stdout: create 298/39 file after the snapshot 2023-10-16T20:41:31.213 INFO:tasks.workunit.client.0.smithi088.stdout: create 299/.snap/snap-subdir-test 2023-10-16T20:41:31.228 INFO:tasks.workunit.client.0.smithi088.stdout: create 299/30 file after the snapshot 2023-10-16T20:41:31.230 INFO:tasks.workunit.client.0.smithi088.stdout: create 299/31 file after the snapshot 2023-10-16T20:41:31.232 INFO:tasks.workunit.client.0.smithi088.stdout: create 299/32 file after the snapshot 2023-10-16T20:41:31.235 INFO:tasks.workunit.client.0.smithi088.stdout: create 299/33 file after the snapshot 2023-10-16T20:41:31.238 INFO:tasks.workunit.client.0.smithi088.stdout: create 299/34 file after the snapshot 2023-10-16T20:41:31.240 INFO:tasks.workunit.client.0.smithi088.stdout: create 299/35 file after the snapshot 2023-10-16T20:41:31.243 INFO:tasks.workunit.client.0.smithi088.stdout: create 299/36 file after the snapshot 2023-10-16T20:41:31.245 INFO:tasks.workunit.client.0.smithi088.stdout: create 299/37 file after the snapshot 2023-10-16T20:41:31.248 INFO:tasks.workunit.client.0.smithi088.stdout: create 299/38 file after the snapshot 2023-10-16T20:41:31.250 INFO:tasks.workunit.client.0.smithi088.stdout: create 299/39 file after the snapshot 2023-10-16T20:41:31.253 INFO:tasks.workunit.client.0.smithi088.stdout:Take second snapshot .snap/test2 2023-10-16T20:41:32.348 INFO:tasks.workunit.client.0.smithi088.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2023-10-16T20:41:36.404 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2023-10-16T20:41:36.404 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2023-10-16T20:41:36.404 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2023-10-16T20:41:36.404 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2023-10-16T20:41:36.405 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2023-10-16T20:41:36.405 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2023-10-16T20:41:36.405 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2023-10-16T20:41:36.405 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2023-10-16T20:41:36.405 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2023-10-16T20:41:36.406 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2023-10-16T20:41:36.406 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2023-10-16T20:41:36.406 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2023-10-16T20:41:36.406 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2023-10-16T20:41:36.407 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2023-10-16T20:41:36.407 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2023-10-16T20:41:36.407 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2023-10-16T20:41:36.407 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2023-10-16T20:41:36.407 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2023-10-16T20:41:36.407 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2023-10-16T20:41:36.408 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2023-10-16T20:41:36.408 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2023-10-16T20:41:36.408 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2023-10-16T20:41:36.408 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2023-10-16T20:41:36.408 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2023-10-16T20:41:36.409 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2023-10-16T20:41:36.409 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2023-10-16T20:41:36.409 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2023-10-16T20:41:36.409 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2023-10-16T20:41:36.409 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2023-10-16T20:41:36.410 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2023-10-16T20:41:36.410 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2023-10-16T20:41:36.410 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2023-10-16T20:41:36.410 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2023-10-16T20:41:36.410 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2023-10-16T20:41:36.411 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2023-10-16T20:41:36.411 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2023-10-16T20:41:36.411 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2023-10-16T20:41:36.411 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2023-10-16T20:41:36.411 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2023-10-16T20:41:36.412 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2023-10-16T20:41:36.412 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2023-10-16T20:41:36.412 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2023-10-16T20:41:36.412 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2023-10-16T20:41:36.412 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2023-10-16T20:41:36.413 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2023-10-16T20:41:36.413 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2023-10-16T20:41:36.413 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2023-10-16T20:41:36.413 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2023-10-16T20:41:36.413 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2023-10-16T20:41:36.413 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2023-10-16T20:41:36.414 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2023-10-16T20:41:36.414 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2023-10-16T20:41:36.414 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2023-10-16T20:41:36.414 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2023-10-16T20:41:36.415 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2023-10-16T20:41:36.415 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2023-10-16T20:41:36.415 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2023-10-16T20:41:36.415 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2023-10-16T20:41:36.415 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2023-10-16T20:41:36.416 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2023-10-16T20:41:36.416 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2023-10-16T20:41:36.416 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2023-10-16T20:41:36.416 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2023-10-16T20:41:36.416 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2023-10-16T20:41:36.417 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2023-10-16T20:41:36.417 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2023-10-16T20:41:36.417 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2023-10-16T20:41:36.417 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2023-10-16T20:41:36.418 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2023-10-16T20:41:36.418 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2023-10-16T20:41:36.418 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2023-10-16T20:41:36.418 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2023-10-16T20:41:36.418 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2023-10-16T20:41:36.419 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2023-10-16T20:41:36.419 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2023-10-16T20:41:36.419 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2023-10-16T20:41:36.419 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2023-10-16T20:41:36.419 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2023-10-16T20:41:36.420 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2023-10-16T20:41:36.420 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2023-10-16T20:41:36.420 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2023-10-16T20:41:40.309 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2023-10-16T20:41:40.309 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2023-10-16T20:41:40.310 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2023-10-16T20:41:40.310 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2023-10-16T20:41:40.310 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2023-10-16T20:41:40.311 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2023-10-16T20:41:40.311 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2023-10-16T20:41:40.311 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2023-10-16T20:41:40.311 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2023-10-16T20:41:40.311 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2023-10-16T20:41:40.312 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2023-10-16T20:41:40.312 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2023-10-16T20:41:40.312 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2023-10-16T20:41:40.312 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2023-10-16T20:41:40.312 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2023-10-16T20:41:40.313 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2023-10-16T20:41:40.313 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2023-10-16T20:41:40.313 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2023-10-16T20:41:40.313 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2023-10-16T20:41:40.313 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2023-10-16T20:41:40.314 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2023-10-16T20:41:40.314 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2023-10-16T20:41:40.314 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2023-10-16T20:41:40.314 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2023-10-16T20:41:40.314 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2023-10-16T20:41:40.315 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2023-10-16T20:41:40.315 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2023-10-16T20:41:40.315 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2023-10-16T20:41:40.315 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2023-10-16T20:41:40.315 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2023-10-16T20:41:40.316 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2023-10-16T20:41:40.316 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2023-10-16T20:41:40.316 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2023-10-16T20:41:40.316 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2023-10-16T20:41:40.316 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2023-10-16T20:41:40.316 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2023-10-16T20:41:40.317 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2023-10-16T20:41:40.317 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2023-10-16T20:41:40.317 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2023-10-16T20:41:40.317 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2023-10-16T20:41:40.317 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2023-10-16T20:41:40.317 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2023-10-16T20:41:40.318 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2023-10-16T20:41:40.318 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2023-10-16T20:41:40.318 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2023-10-16T20:41:40.318 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2023-10-16T20:41:40.319 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2023-10-16T20:41:40.319 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2023-10-16T20:41:40.319 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2023-10-16T20:41:40.319 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2023-10-16T20:41:40.319 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2023-10-16T20:41:40.320 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2023-10-16T20:41:40.320 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2023-10-16T20:41:40.320 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2023-10-16T20:41:40.320 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2023-10-16T20:41:40.320 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2023-10-16T20:41:40.321 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2023-10-16T20:41:40.321 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2023-10-16T20:41:40.321 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2023-10-16T20:41:40.321 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2023-10-16T20:41:40.321 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2023-10-16T20:41:40.322 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2023-10-16T20:41:40.322 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2023-10-16T20:41:40.322 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2023-10-16T20:41:40.322 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2023-10-16T20:41:40.322 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2023-10-16T20:41:40.323 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2023-10-16T20:41:40.323 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2023-10-16T20:41:40.323 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2023-10-16T20:41:40.323 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2023-10-16T20:41:40.323 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2023-10-16T20:41:40.323 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2023-10-16T20:41:40.324 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2023-10-16T20:41:40.324 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2023-10-16T20:41:40.324 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2023-10-16T20:41:40.324 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2023-10-16T20:41:40.324 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2023-10-16T20:41:40.324 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2023-10-16T20:41:40.325 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2023-10-16T20:41:40.325 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2023-10-16T20:41:40.325 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2023-10-16T20:41:47.381 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2023-10-16T20:41:47.381 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2023-10-16T20:41:47.381 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2023-10-16T20:41:47.382 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2023-10-16T20:41:47.382 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2023-10-16T20:41:47.382 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2023-10-16T20:41:47.382 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2023-10-16T20:41:47.382 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2023-10-16T20:41:47.383 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2023-10-16T20:41:47.383 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2023-10-16T20:41:47.383 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2023-10-16T20:41:47.383 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2023-10-16T20:41:47.383 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2023-10-16T20:41:47.384 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2023-10-16T20:41:47.384 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2023-10-16T20:41:47.384 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2023-10-16T20:41:47.384 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2023-10-16T20:41:47.384 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2023-10-16T20:41:47.384 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2023-10-16T20:41:47.385 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2023-10-16T20:41:47.385 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2023-10-16T20:41:47.385 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2023-10-16T20:41:47.385 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2023-10-16T20:41:47.385 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2023-10-16T20:41:47.386 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2023-10-16T20:41:47.386 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2023-10-16T20:41:47.386 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2023-10-16T20:41:47.386 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2023-10-16T20:41:47.386 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2023-10-16T20:41:47.387 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2023-10-16T20:41:47.387 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2023-10-16T20:41:47.387 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2023-10-16T20:41:47.387 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2023-10-16T20:41:47.387 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2023-10-16T20:41:47.388 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2023-10-16T20:41:47.388 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2023-10-16T20:41:47.388 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2023-10-16T20:41:47.388 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2023-10-16T20:41:47.388 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2023-10-16T20:41:47.388 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2023-10-16T20:41:47.389 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2023-10-16T20:41:47.389 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2023-10-16T20:41:47.389 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2023-10-16T20:41:47.389 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2023-10-16T20:41:47.389 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2023-10-16T20:41:47.389 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2023-10-16T20:41:47.390 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2023-10-16T20:41:47.390 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2023-10-16T20:41:47.390 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2023-10-16T20:41:47.390 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2023-10-16T20:41:47.391 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2023-10-16T20:41:47.391 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2023-10-16T20:41:47.391 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2023-10-16T20:41:47.391 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2023-10-16T20:41:47.391 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2023-10-16T20:41:47.391 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2023-10-16T20:41:47.392 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2023-10-16T20:41:47.392 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2023-10-16T20:41:47.392 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2023-10-16T20:41:47.392 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2023-10-16T20:41:47.392 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2023-10-16T20:41:47.392 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2023-10-16T20:41:47.393 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2023-10-16T20:41:47.393 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2023-10-16T20:41:47.393 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2023-10-16T20:41:47.393 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2023-10-16T20:41:47.393 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2023-10-16T20:41:47.393 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2023-10-16T20:41:47.394 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2023-10-16T20:41:47.394 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2023-10-16T20:41:47.394 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2023-10-16T20:41:47.394 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2023-10-16T20:41:47.394 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2023-10-16T20:41:47.395 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2023-10-16T20:41:47.395 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2023-10-16T20:41:47.395 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2023-10-16T20:41:47.395 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2023-10-16T20:41:47.395 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2023-10-16T20:41:47.395 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2023-10-16T20:41:47.396 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2023-10-16T20:41:47.396 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2023-10-16T20:41:49.198 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2023-10-16T20:41:49.199 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2023-10-16T20:41:49.199 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2023-10-16T20:41:49.199 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2023-10-16T20:41:49.200 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2023-10-16T20:41:49.200 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2023-10-16T20:41:49.200 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2023-10-16T20:41:49.200 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2023-10-16T20:41:49.200 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2023-10-16T20:41:49.200 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2023-10-16T20:41:49.201 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2023-10-16T20:41:49.201 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2023-10-16T20:41:49.201 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2023-10-16T20:41:49.201 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2023-10-16T20:41:49.202 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2023-10-16T20:41:49.202 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2023-10-16T20:41:49.202 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2023-10-16T20:41:49.202 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2023-10-16T20:41:49.202 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2023-10-16T20:41:49.203 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2023-10-16T20:41:49.203 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2023-10-16T20:41:49.203 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2023-10-16T20:41:49.203 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2023-10-16T20:41:49.203 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2023-10-16T20:41:49.204 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2023-10-16T20:41:49.204 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2023-10-16T20:41:49.204 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2023-10-16T20:41:49.204 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2023-10-16T20:41:49.205 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2023-10-16T20:41:49.205 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2023-10-16T20:41:49.205 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2023-10-16T20:41:49.205 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2023-10-16T20:41:49.205 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2023-10-16T20:41:49.206 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2023-10-16T20:41:49.206 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2023-10-16T20:41:49.206 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2023-10-16T20:41:49.206 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2023-10-16T20:41:49.207 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2023-10-16T20:41:49.207 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2023-10-16T20:41:49.207 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2023-10-16T20:41:49.207 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2023-10-16T20:41:49.207 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2023-10-16T20:41:49.208 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2023-10-16T20:41:49.208 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2023-10-16T20:41:49.208 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2023-10-16T20:41:49.208 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2023-10-16T20:41:49.208 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2023-10-16T20:41:49.209 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2023-10-16T20:41:49.209 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2023-10-16T20:41:49.209 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2023-10-16T20:41:49.209 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2023-10-16T20:41:49.209 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2023-10-16T20:41:49.210 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2023-10-16T20:41:49.210 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2023-10-16T20:41:49.210 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2023-10-16T20:41:49.210 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2023-10-16T20:41:49.211 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2023-10-16T20:41:49.211 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2023-10-16T20:41:49.211 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2023-10-16T20:41:49.211 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2023-10-16T20:41:49.212 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2023-10-16T20:41:49.212 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2023-10-16T20:41:49.212 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2023-10-16T20:41:49.212 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2023-10-16T20:41:49.212 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2023-10-16T20:41:49.213 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2023-10-16T20:41:49.213 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2023-10-16T20:41:49.213 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2023-10-16T20:41:49.213 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2023-10-16T20:41:49.213 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2023-10-16T20:41:49.214 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2023-10-16T20:41:49.214 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2023-10-16T20:41:49.214 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2023-10-16T20:41:49.214 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2023-10-16T20:41:49.215 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2023-10-16T20:41:49.215 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2023-10-16T20:41:49.215 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2023-10-16T20:41:49.215 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2023-10-16T20:41:49.215 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2023-10-16T20:41:49.216 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2023-10-16T20:41:49.216 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2023-10-16T20:41:52.986 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:41:53.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.125+0000 7f7a73ac9640 1 Processor -- start 2023-10-16T20:41:53.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.129+0000 7f7a73ac9640 1 -- start start 2023-10-16T20:41:53.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.129+0000 7f7a73ac9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7a6c06a970 0x7f7a6c071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:53.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.129+0000 7f7a73ac9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a6c071e60 0x7f7a6c15a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:53.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.129+0000 7f7a73ac9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6c15b3c0 con 0x7f7a6c1406d0 2023-10-16T20:41:53.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.129+0000 7f7a73ac9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a6c0c2460 con 0x7f7a6c06a970 2023-10-16T20:41:53.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.129+0000 7f7a73ac9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c15b680 con 0x7f7a6c071e60 2023-10-16T20:41:53.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.129+0000 7f7a71865640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7a6c1406d0 0x7f7a6c0c1d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40732/0 (socket says 172.21.15.88:40732) 2023-10-16T20:41:53.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.129+0000 7f7a71865640 1 -- 172.21.15.88:0/1858907737 learned_addr learned my addr 172.21.15.88:0/1858907737 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:41:53.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.129+0000 7f7a71064640 1 --2- 172.21.15.88:0/1858907737 >> v2:172.21.15.191:3300/0 conn(0x7f7a6c06a970 0x7f7a6c071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:53.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.129+0000 7f7a72066640 1 --2- 172.21.15.88:0/1858907737 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a6c071e60 0x7f7a6c15a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:53.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.129+0000 7f7a70863640 1 -- 172.21.15.88:0/1858907737 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205780929 0 0) 0x7f7a6c15b3c0 con 0x7f7a6c1406d0 2023-10-16T20:41:53.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.129+0000 7f7a70863640 1 -- 172.21.15.88:0/1858907737 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a58003530 con 0x7f7a6c1406d0 2023-10-16T20:41:53.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.129+0000 7f7a71064640 1 -- 172.21.15.88:0/1858907737 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a6c071e60 msgr2=0x7f7a6c15a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:41:53.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.129+0000 7f7a71064640 1 --2- 172.21.15.88:0/1858907737 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a6c071e60 0x7f7a6c15a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.129+0000 7f7a71064640 1 -- 172.21.15.88:0/1858907737 >> v1:172.21.15.88:6789/0 conn(0x7f7a6c1406d0 legacy=0x7f7a6c0c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:53.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.129+0000 7f7a70863640 1 -- 172.21.15.88:0/1858907737 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 342515014 0 0) 0x7f7a58003530 con 0x7f7a6c1406d0 2023-10-16T20:41:53.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.129+0000 7f7a71064640 1 -- 172.21.15.88:0/1858907737 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c157570 con 0x7f7a6c06a970 2023-10-16T20:41:53.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.133+0000 7f7a71064640 1 --2- 172.21.15.88:0/1858907737 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a6c06a970 0x7f7a6c071920 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c0097c0 tx=0x7f7a5c01c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f430488b4b280db server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:41:53.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.133+0000 7f7a70863640 1 -- 172.21.15.88:0/1858907737 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a68001680 con 0x7f7a6c06a970 2023-10-16T20:41:53.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.133+0000 7f7a70863640 1 -- 172.21.15.88:0/1858907737 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a5c0a7050 con 0x7f7a6c06a970 2023-10-16T20:41:53.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.133+0000 7f7a70863640 1 -- 172.21.15.88:0/1858907737 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c0ab600 con 0x7f7a6c06a970 2023-10-16T20:41:53.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.133+0000 7f7a73ac9640 1 -- 172.21.15.88:0/1858907737 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a6c06a970 msgr2=0x7f7a6c071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:53.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.133+0000 7f7a73ac9640 1 --2- 172.21.15.88:0/1858907737 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a6c06a970 0x7f7a6c071920 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c0097c0 tx=0x7f7a5c01c3a0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.133+0000 7f7a73ac9640 1 -- 172.21.15.88:0/1858907737 shutdown_connections 2023-10-16T20:41:53.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.133+0000 7f7a73ac9640 1 --2- 172.21.15.88:0/1858907737 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a6c071e60 0x7f7a6c15a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.133+0000 7f7a73ac9640 1 --2- 172.21.15.88:0/1858907737 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a6c06a970 0x7f7a6c071920 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.133+0000 7f7a73ac9640 1 -- 172.21.15.88:0/1858907737 >> 172.21.15.88:0/1858907737 conn(0x7f7a6c0cad50 msgr2=0x7f7a6c0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:41:53.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.133+0000 7f7a73ac9640 1 -- 172.21.15.88:0/1858907737 shutdown_connections 2023-10-16T20:41:53.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.133+0000 7f7a73ac9640 1 -- 172.21.15.88:0/1858907737 wait complete. 2023-10-16T20:41:53.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a73ac9640 1 Processor -- start 2023-10-16T20:41:53.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a73ac9640 1 -- start start 2023-10-16T20:41:53.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a73ac9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a6c06a970 0x7f7a6c1529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:53.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a73ac9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a6c071e60 0x7f7a6c0ca090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:53.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a73ac9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a6c1406d0 0x7f7a6c139130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:53.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a73ac9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c0ca5d0 con 0x7f7a6c06a970 2023-10-16T20:41:53.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a73ac9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c153730 con 0x7f7a6c071e60 2023-10-16T20:41:53.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a73ac9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c15a520 con 0x7f7a6c1406d0 2023-10-16T20:41:53.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a71865640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a6c06a970 0x7f7a6c1529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:53.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a71865640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a6c06a970 0x7f7a6c1529b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36204/0 (socket says 172.21.15.88:36204) 2023-10-16T20:41:53.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a71865640 1 -- 172.21.15.88:0/2175600987 learned_addr learned my addr 172.21.15.88:0/2175600987 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:41:53.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a71064640 1 --2- 172.21.15.88:0/2175600987 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a6c071e60 0x7f7a6c0ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:53.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a72066640 1 --2- 172.21.15.88:0/2175600987 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a6c1406d0 0x7f7a6c139130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:53.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a71865640 1 -- 172.21.15.88:0/2175600987 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a6c1406d0 msgr2=0x7f7a6c139130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:53.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a71865640 1 --2- 172.21.15.88:0/2175600987 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a6c1406d0 0x7f7a6c139130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a71865640 1 -- 172.21.15.88:0/2175600987 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a6c071e60 msgr2=0x7f7a6c0ca090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:53.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a71865640 1 --2- 172.21.15.88:0/2175600987 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a6c071e60 0x7f7a6c0ca090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a71865640 1 -- 172.21.15.88:0/2175600987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c009420 con 0x7f7a6c06a970 2023-10-16T20:41:53.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a72066640 1 --2- 172.21.15.88:0/2175600987 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a6c1406d0 0x7f7a6c139130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:41:53.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a71865640 1 --2- 172.21.15.88:0/2175600987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a6c06a970 0x7f7a6c1529b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800c750 tx=0x7f7a6800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:41:53.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a627fc640 1 -- 172.21.15.88:0/2175600987 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a68005ae0 con 0x7f7a6c06a970 2023-10-16T20:41:53.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a627fc640 1 -- 172.21.15.88:0/2175600987 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a68005c40 con 0x7f7a6c06a970 2023-10-16T20:41:53.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a627fc640 1 -- 172.21.15.88:0/2175600987 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a68007070 con 0x7f7a6c06a970 2023-10-16T20:41:53.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a73ac9640 1 -- 172.21.15.88:0/2175600987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6c15a7e0 con 0x7f7a6c06a970 2023-10-16T20:41:53.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a73ac9640 1 -- 172.21.15.88:0/2175600987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6c154390 con 0x7f7a6c06a970 2023-10-16T20:41:53.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.137+0000 7f7a73ac9640 1 -- 172.21.15.88:0/2175600987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6c1f2c70 con 0x7f7a6c06a970 2023-10-16T20:41:53.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.141+0000 7f7a627fc640 1 -- 172.21.15.88:0/2175600987 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7a680071d0 con 0x7f7a6c06a970 2023-10-16T20:41:53.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.141+0000 7f7a627fc640 1 --2- 172.21.15.88:0/2175600987 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7a500664a0 0x7f7a50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:53.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.141+0000 7f7a627fc640 1 -- 172.21.15.88:0/2175600987 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f7a68083b10 con 0x7f7a6c06a970 2023-10-16T20:41:53.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.141+0000 7f7a71064640 1 --2- 172.21.15.88:0/2175600987 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7a500664a0 0x7f7a50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:53.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.141+0000 7f7a71064640 1 --2- 172.21.15.88:0/2175600987 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7a500664a0 0x7f7a50068960 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c0ad000 tx=0x7f7a5c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:41:53.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.145+0000 7f7a627fc640 1 -- 172.21.15.88:0/2175600987 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7a68050a70 con 0x7f7a6c06a970 2023-10-16T20:41:53.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.373+0000 7f7a73ac9640 1 -- 172.21.15.88:0/2175600987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a6c074700 con 0x7f7a6c06a970 2023-10-16T20:41:53.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.373+0000 7f7a627fc640 1 -- 172.21.15.88:0/2175600987 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6566 (secure 0 0 0) 0x7f7a68054090 con 0x7f7a6c06a970 2023-10-16T20:41:53.381 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:41:53.382 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:41:32.316596+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":44,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:41:53.382 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 23 2023-10-16T20:41:53.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.377+0000 7f7a73ac9640 1 -- 172.21.15.88:0/2175600987 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7a500664a0 msgr2=0x7f7a50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:53.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.377+0000 7f7a73ac9640 1 --2- 172.21.15.88:0/2175600987 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7a500664a0 0x7f7a50068960 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c0ad000 tx=0x7f7a5c008040 comp rx=0 tx=0).stop 2023-10-16T20:41:53.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.377+0000 7f7a73ac9640 1 -- 172.21.15.88:0/2175600987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a6c06a970 msgr2=0x7f7a6c1529b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:53.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.377+0000 7f7a73ac9640 1 --2- 172.21.15.88:0/2175600987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a6c06a970 0x7f7a6c1529b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800c750 tx=0x7f7a6800cc10 comp rx=0 tx=0).stop 2023-10-16T20:41:53.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.377+0000 7f7a73ac9640 1 -- 172.21.15.88:0/2175600987 shutdown_connections 2023-10-16T20:41:53.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.377+0000 7f7a73ac9640 1 --2- 172.21.15.88:0/2175600987 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7a500664a0 0x7f7a50068960 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.377+0000 7f7a73ac9640 1 --2- 172.21.15.88:0/2175600987 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a6c1406d0 0x7f7a6c139130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.377+0000 7f7a73ac9640 1 --2- 172.21.15.88:0/2175600987 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a6c071e60 0x7f7a6c0ca090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.377+0000 7f7a73ac9640 1 --2- 172.21.15.88:0/2175600987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a6c06a970 0x7f7a6c1529b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.377+0000 7f7a73ac9640 1 -- 172.21.15.88:0/2175600987 >> 172.21.15.88:0/2175600987 conn(0x7f7a6c0cad50 msgr2=0x7f7a6c139f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:41:53.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.377+0000 7f7a73ac9640 1 -- 172.21.15.88:0/2175600987 shutdown_connections 2023-10-16T20:41:53.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.377+0000 7f7a73ac9640 1 -- 172.21.15.88:0/2175600987 wait complete. 2023-10-16T20:41:53.408 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=4) 2023-10-16T20:41:53.409 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T20:41:53.409 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T20:41:53.424 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T20:41:53.424 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:41:53.442 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2023-10-16T20:41:53.443 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2023-10-16T20:41:53.443 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2023-10-16T20:41:53.444 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2023-10-16T20:41:53.444 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2023-10-16T20:41:53.444 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2023-10-16T20:41:53.444 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2023-10-16T20:41:53.445 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2023-10-16T20:41:53.445 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2023-10-16T20:41:53.445 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2023-10-16T20:41:53.445 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2023-10-16T20:41:53.445 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2023-10-16T20:41:53.446 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2023-10-16T20:41:53.446 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2023-10-16T20:41:53.446 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2023-10-16T20:41:53.446 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2023-10-16T20:41:53.447 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2023-10-16T20:41:53.447 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2023-10-16T20:41:53.447 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2023-10-16T20:41:53.447 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2023-10-16T20:41:53.447 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2023-10-16T20:41:53.448 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2023-10-16T20:41:53.448 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2023-10-16T20:41:53.448 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2023-10-16T20:41:53.448 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2023-10-16T20:41:53.448 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2023-10-16T20:41:53.449 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2023-10-16T20:41:53.449 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2023-10-16T20:41:53.449 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2023-10-16T20:41:53.449 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2023-10-16T20:41:53.449 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2023-10-16T20:41:53.450 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2023-10-16T20:41:53.450 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2023-10-16T20:41:53.450 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2023-10-16T20:41:53.450 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2023-10-16T20:41:53.450 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2023-10-16T20:41:53.451 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2023-10-16T20:41:53.451 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2023-10-16T20:41:53.451 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2023-10-16T20:41:53.451 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2023-10-16T20:41:53.451 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2023-10-16T20:41:53.452 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2023-10-16T20:41:53.452 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2023-10-16T20:41:53.452 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2023-10-16T20:41:53.452 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2023-10-16T20:41:53.453 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2023-10-16T20:41:53.453 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2023-10-16T20:41:53.453 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2023-10-16T20:41:53.453 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2023-10-16T20:41:53.454 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2023-10-16T20:41:53.454 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2023-10-16T20:41:53.454 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2023-10-16T20:41:53.454 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2023-10-16T20:41:53.454 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2023-10-16T20:41:53.455 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2023-10-16T20:41:53.455 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2023-10-16T20:41:53.455 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2023-10-16T20:41:53.455 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2023-10-16T20:41:53.455 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2023-10-16T20:41:53.456 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2023-10-16T20:41:53.456 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2023-10-16T20:41:53.456 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2023-10-16T20:41:53.456 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2023-10-16T20:41:53.456 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2023-10-16T20:41:53.457 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2023-10-16T20:41:53.457 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2023-10-16T20:41:53.457 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2023-10-16T20:41:53.457 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2023-10-16T20:41:53.457 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2023-10-16T20:41:53.458 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2023-10-16T20:41:53.458 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2023-10-16T20:41:53.458 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2023-10-16T20:41:53.458 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2023-10-16T20:41:53.459 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2023-10-16T20:41:53.459 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2023-10-16T20:41:53.459 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2023-10-16T20:41:53.459 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2023-10-16T20:41:53.459 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2023-10-16T20:41:53.460 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2023-10-16T20:41:53.460 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2023-10-16T20:41:53.460 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2023-10-16T20:41:53.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.569+0000 7f1f865a7640 1 Processor -- start 2023-10-16T20:41:53.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.569+0000 7f1f865a7640 1 -- start start 2023-10-16T20:41:53.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.569+0000 7f1f865a7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1f801537c0 0x7f1f8015c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:53.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.569+0000 7f1f865a7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f8015cdb0 0x7f1f800c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:53.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.569+0000 7f1f865a7640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f800cae10 con 0x7f1f800ca2f0 2023-10-16T20:41:53.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.569+0000 7f1f865a7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f800cac30 con 0x7f1f801537c0 2023-10-16T20:41:53.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.569+0000 7f1f865a7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8015d2b0 con 0x7f1f8015cdb0 2023-10-16T20:41:53.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.569+0000 7f1f855a5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1f801537c0 0x7f1f8015c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:53.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.569+0000 7f1f855a5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1f801537c0 0x7f1f8015c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36218/0 (socket says 172.21.15.88:36218) 2023-10-16T20:41:53.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.569+0000 7f1f855a5640 1 -- 172.21.15.88:0/4064002396 learned_addr learned my addr 172.21.15.88:0/4064002396 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:41:53.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.569+0000 7f1f84da4640 1 --2- 172.21.15.88:0/4064002396 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f8015cdb0 0x7f1f800c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:53.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.569+0000 7f1f855a5640 1 -- 172.21.15.88:0/4064002396 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f8015cdb0 msgr2=0x7f1f800c4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:41:53.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.569+0000 7f1f855a5640 1 --2- 172.21.15.88:0/4064002396 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f8015cdb0 0x7f1f800c4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.569+0000 7f1f855a5640 1 -- 172.21.15.88:0/4064002396 >> v1:172.21.15.191:6789/0 conn(0x7f1f800ca2f0 legacy=0x7f1f800cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:53.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.569+0000 7f1f855a5640 1 -- 172.21.15.88:0/4064002396 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8015d590 con 0x7f1f801537c0 2023-10-16T20:41:53.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.569+0000 7f1f84da4640 1 --2- 172.21.15.88:0/4064002396 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f8015cdb0 0x7f1f800c4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:41:53.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.573+0000 7f1f855a5640 1 --2- 172.21.15.88:0/4064002396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f801537c0 0x7f1f8015c870 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c009870 tx=0x7f1f7c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a37150023ad6b51 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:41:53.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.573+0000 7f1f77fff640 1 -- 172.21.15.88:0/4064002396 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c0a8a30 con 0x7f1f801537c0 2023-10-16T20:41:53.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.573+0000 7f1f77fff640 1 -- 172.21.15.88:0/4064002396 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7c0a8b90 con 0x7f1f801537c0 2023-10-16T20:41:53.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.573+0000 7f1f77fff640 1 -- 172.21.15.88:0/4064002396 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c0b0440 con 0x7f1f801537c0 2023-10-16T20:41:53.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.573+0000 7f1f865a7640 1 -- 172.21.15.88:0/4064002396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f801537c0 msgr2=0x7f1f8015c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:53.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.573+0000 7f1f865a7640 1 --2- 172.21.15.88:0/4064002396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f801537c0 0x7f1f8015c870 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c009870 tx=0x7f1f7c01ed30 comp rx=0 tx=0).stop 2023-10-16T20:41:53.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.573+0000 7f1f865a7640 1 -- 172.21.15.88:0/4064002396 shutdown_connections 2023-10-16T20:41:53.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.573+0000 7f1f865a7640 1 --2- 172.21.15.88:0/4064002396 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f8015cdb0 0x7f1f800c4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.573+0000 7f1f865a7640 1 --2- 172.21.15.88:0/4064002396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f801537c0 0x7f1f8015c870 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.573+0000 7f1f865a7640 1 -- 172.21.15.88:0/4064002396 >> 172.21.15.88:0/4064002396 conn(0x7f1f8006c170 msgr2=0x7f1f800715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:41:53.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.573+0000 7f1f865a7640 1 -- 172.21.15.88:0/4064002396 shutdown_connections 2023-10-16T20:41:53.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.573+0000 7f1f865a7640 1 -- 172.21.15.88:0/4064002396 wait complete. 2023-10-16T20:41:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f865a7640 1 Processor -- start 2023-10-16T20:41:53.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f865a7640 1 -- start start 2023-10-16T20:41:53.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f865a7640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f800ca2f0 0x7f1f801e5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:53.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f865a7640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f801537c0 0x7f1f801e24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:53.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f865a7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f8015cdb0 0x7f1f8014da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:53.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f865a7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8015b610 con 0x7f1f8015cdb0 2023-10-16T20:41:53.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f865a7640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f800c5250 con 0x7f1f800ca2f0 2023-10-16T20:41:53.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f865a7640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f80157330 con 0x7f1f801537c0 2023-10-16T20:41:53.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f85da6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f8015cdb0 0x7f1f8014da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:53.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f85da6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f8015cdb0 0x7f1f8014da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36222/0 (socket says 172.21.15.88:36222) 2023-10-16T20:41:53.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f85da6640 1 -- 172.21.15.88:0/1115206458 learned_addr learned my addr 172.21.15.88:0/1115206458 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:41:53.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f855a5640 1 --2- 172.21.15.88:0/1115206458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f800ca2f0 0x7f1f801e5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:53.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f84da4640 1 --2- 172.21.15.88:0/1115206458 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f801537c0 0x7f1f801e24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:53.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f85da6640 1 -- 172.21.15.88:0/1115206458 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f801537c0 msgr2=0x7f1f801e24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:53.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f85da6640 1 --2- 172.21.15.88:0/1115206458 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f801537c0 0x7f1f801e24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f85da6640 1 -- 172.21.15.88:0/1115206458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f800ca2f0 msgr2=0x7f1f801e5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:53.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f85da6640 1 --2- 172.21.15.88:0/1115206458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f800ca2f0 0x7f1f801e5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f85da6640 1 -- 172.21.15.88:0/1115206458 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7c0094d0 con 0x7f1f8015cdb0 2023-10-16T20:41:53.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f85da6640 1 --2- 172.21.15.88:0/1115206458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f8015cdb0 0x7f1f8014da50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800c770 tx=0x7f1f7800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:41:53.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f75ffb640 1 -- 172.21.15.88:0/1115206458 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f78003b80 con 0x7f1f8015cdb0 2023-10-16T20:41:53.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f75ffb640 1 -- 172.21.15.88:0/1115206458 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f78003ce0 con 0x7f1f8015cdb0 2023-10-16T20:41:53.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f75ffb640 1 -- 172.21.15.88:0/1115206458 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7800aa90 con 0x7f1f8015cdb0 2023-10-16T20:41:53.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.577+0000 7f1f865a7640 1 -- 172.21.15.88:0/1115206458 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f80157610 con 0x7f1f8015cdb0 2023-10-16T20:41:53.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.581+0000 7f1f865a7640 1 -- 172.21.15.88:0/1115206458 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f80157bd0 con 0x7f1f8015cdb0 2023-10-16T20:41:53.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.581+0000 7f1f865a7640 1 -- 172.21.15.88:0/1115206458 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8004e9c0 con 0x7f1f8015cdb0 2023-10-16T20:41:53.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.581+0000 7f1f75ffb640 1 -- 172.21.15.88:0/1115206458 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1f7800abf0 con 0x7f1f8015cdb0 2023-10-16T20:41:53.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.581+0000 7f1f75ffb640 1 --2- 172.21.15.88:0/1115206458 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1f640664a0 0x7f1f64068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:53.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.581+0000 7f1f75ffb640 1 -- 172.21.15.88:0/1115206458 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f1f78083fe0 con 0x7f1f8015cdb0 2023-10-16T20:41:53.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.581+0000 7f1f855a5640 1 --2- 172.21.15.88:0/1115206458 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1f640664a0 0x7f1f64068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:53.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.581+0000 7f1f855a5640 1 --2- 172.21.15.88:0/1115206458 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1f640664a0 0x7f1f64068960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0b2000 tx=0x7f1f7c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:41:53.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.585+0000 7f1f75ffb640 1 -- 172.21.15.88:0/1115206458 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1f780510a0 con 0x7f1f8015cdb0 2023-10-16T20:41:53.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.769+0000 7f1f865a7640 1 -- 172.21.15.88:0/1115206458 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f800ca0d0 con 0x7f1f8015cdb0 2023-10-16T20:41:53.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.769+0000 7f1f75ffb640 1 -- 172.21.15.88:0/1115206458 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6566 (secure 0 0 0) 0x7f1f780546c0 con 0x7f1f8015cdb0 2023-10-16T20:41:53.778 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:41:53.778 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:41:32.316596+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":44,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:41:53.778 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 23 2023-10-16T20:41:53.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.769+0000 7f1f865a7640 1 -- 172.21.15.88:0/1115206458 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1f640664a0 msgr2=0x7f1f64068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:53.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.769+0000 7f1f865a7640 1 --2- 172.21.15.88:0/1115206458 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1f640664a0 0x7f1f64068960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0b2000 tx=0x7f1f7c009040 comp rx=0 tx=0).stop 2023-10-16T20:41:53.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.769+0000 7f1f865a7640 1 -- 172.21.15.88:0/1115206458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f8015cdb0 msgr2=0x7f1f8014da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:53.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.769+0000 7f1f865a7640 1 --2- 172.21.15.88:0/1115206458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f8015cdb0 0x7f1f8014da50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800c770 tx=0x7f1f7800cc30 comp rx=0 tx=0).stop 2023-10-16T20:41:53.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.773+0000 7f1f865a7640 1 -- 172.21.15.88:0/1115206458 shutdown_connections 2023-10-16T20:41:53.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.773+0000 7f1f865a7640 1 --2- 172.21.15.88:0/1115206458 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1f640664a0 0x7f1f64068960 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.773+0000 7f1f865a7640 1 --2- 172.21.15.88:0/1115206458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f8015cdb0 0x7f1f8014da50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.773+0000 7f1f865a7640 1 --2- 172.21.15.88:0/1115206458 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f801537c0 0x7f1f801e24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.773+0000 7f1f865a7640 1 --2- 172.21.15.88:0/1115206458 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f800ca2f0 0x7f1f801e5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:53.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.773+0000 7f1f865a7640 1 -- 172.21.15.88:0/1115206458 >> 172.21.15.88:0/1115206458 conn(0x7f1f8006c170 msgr2=0x7f1f800c9900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:41:53.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.773+0000 7f1f865a7640 1 -- 172.21.15.88:0/1115206458 shutdown_connections 2023-10-16T20:41:53.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:53.773+0000 7f1f865a7640 1 -- 172.21.15.88:0/1115206458 wait complete. 2023-10-16T20:41:53.798 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-16T20:41:54.663 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:41:54.667 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:41:55.800 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:41:55.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.937+0000 7f323633e640 1 Processor -- start 2023-10-16T20:41:55.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.937+0000 7f323633e640 1 -- start start 2023-10-16T20:41:55.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.941+0000 7f323633e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3230150280 0x7f32300c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:55.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.941+0000 7f323633e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32300c6bc0 0x7f32300c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:55.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.941+0000 7f323633e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32300c5c60 con 0x7f3230150660 2023-10-16T20:41:55.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.941+0000 7f323633e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32300c5370 con 0x7f3230150280 2023-10-16T20:41:55.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.941+0000 7f323633e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3230156d10 con 0x7f32300c6bc0 2023-10-16T20:41:55.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.941+0000 7f322ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3230150280 0x7f32300c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:55.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.941+0000 7f322ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3230150280 0x7f32300c60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36226/0 (socket says 172.21.15.88:36226) 2023-10-16T20:41:55.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.941+0000 7f322ffff640 1 -- 172.21.15.88:0/3429206202 learned_addr learned my addr 172.21.15.88:0/3429206202 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:41:55.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.941+0000 7f322ffff640 1 -- 172.21.15.88:0/3429206202 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32300c6bc0 msgr2=0x7f32300c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:41:55.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.941+0000 7f32348db640 1 --2- 172.21.15.88:0/3429206202 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32300c6bc0 0x7f32300c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:55.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.941+0000 7f322ffff640 1 --2- 172.21.15.88:0/3429206202 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32300c6bc0 0x7f32300c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:55.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.941+0000 7f322ffff640 1 -- 172.21.15.88:0/3429206202 >> v1:172.21.15.88:6789/0 conn(0x7f3230150660 legacy=0x7f323006d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:55.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.941+0000 7f322effd640 1 -- 172.21.15.88:0/3429206202 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434818085 0 0) 0x7f32300c5c60 con 0x7f3230150660 2023-10-16T20:41:55.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.941+0000 7f322ffff640 1 -- 172.21.15.88:0/3429206202 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32300c5610 con 0x7f3230150280 2023-10-16T20:41:55.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.941+0000 7f32348db640 1 --2- 172.21.15.88:0/3429206202 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32300c6bc0 0x7f32300c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:41:55.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.945+0000 7f322ffff640 1 --2- 172.21.15.88:0/3429206202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3230150280 0x7f32300c60f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f322000bb10 tx=0x7f322001f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe6b4fd274e1a18 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:41:55.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.945+0000 7f322effd640 1 -- 172.21.15.88:0/3429206202 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32200a7cd0 con 0x7f3230150280 2023-10-16T20:41:55.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.945+0000 7f322effd640 1 -- 172.21.15.88:0/3429206202 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32200a7e30 con 0x7f3230150280 2023-10-16T20:41:55.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.945+0000 7f322effd640 1 -- 172.21.15.88:0/3429206202 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32200af660 con 0x7f3230150280 2023-10-16T20:41:55.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.945+0000 7f323633e640 1 -- 172.21.15.88:0/3429206202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3230150280 msgr2=0x7f32300c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:55.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.945+0000 7f323633e640 1 --2- 172.21.15.88:0/3429206202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3230150280 0x7f32300c60f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f322000bb10 tx=0x7f322001f640 comp rx=0 tx=0).stop 2023-10-16T20:41:55.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.945+0000 7f323633e640 1 -- 172.21.15.88:0/3429206202 shutdown_connections 2023-10-16T20:41:55.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.945+0000 7f323633e640 1 --2- 172.21.15.88:0/3429206202 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32300c6bc0 0x7f32300c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:55.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.945+0000 7f323633e640 1 --2- 172.21.15.88:0/3429206202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3230150280 0x7f32300c60f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:55.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.945+0000 7f323633e640 1 -- 172.21.15.88:0/3429206202 >> 172.21.15.88:0/3429206202 conn(0x7f32300cad50 msgr2=0x7f32300cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:41:55.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.945+0000 7f323633e640 1 -- 172.21.15.88:0/3429206202 shutdown_connections 2023-10-16T20:41:55.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.945+0000 7f323633e640 1 -- 172.21.15.88:0/3429206202 wait complete. 2023-10-16T20:41:55.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.945+0000 7f323633e640 1 Processor -- start 2023-10-16T20:41:55.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.945+0000 7f323633e640 1 -- start start 2023-10-16T20:41:55.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f323633e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32300c6bc0 0x7f3230140de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:55.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f323633e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3230150280 0x7f323014f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:55.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f323633e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3230150660 0x7f323014dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:55.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f323633e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32300c9b60 con 0x7f3230150660 2023-10-16T20:41:55.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f323633e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3230141dd0 con 0x7f3230150280 2023-10-16T20:41:55.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f323633e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32300c9df0 con 0x7f32300c6bc0 2023-10-16T20:41:55.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f32348db640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3230150660 0x7f323014dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:55.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f32348db640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3230150660 0x7f323014dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36232/0 (socket says 172.21.15.88:36232) 2023-10-16T20:41:55.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f32348db640 1 -- 172.21.15.88:0/3868340385 learned_addr learned my addr 172.21.15.88:0/3868340385 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:41:55.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f322ffff640 1 --2- 172.21.15.88:0/3868340385 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32300c6bc0 0x7f3230140de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:55.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f322f7fe640 1 --2- 172.21.15.88:0/3868340385 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3230150280 0x7f323014f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:55.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f32348db640 1 -- 172.21.15.88:0/3868340385 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32300c6bc0 msgr2=0x7f3230140de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:55.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f32348db640 1 --2- 172.21.15.88:0/3868340385 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32300c6bc0 0x7f3230140de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:55.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f32348db640 1 -- 172.21.15.88:0/3868340385 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3230150280 msgr2=0x7f323014f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:55.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f32348db640 1 --2- 172.21.15.88:0/3868340385 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3230150280 0x7f323014f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:55.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f32348db640 1 -- 172.21.15.88:0/3868340385 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322000b770 con 0x7f3230150660 2023-10-16T20:41:55.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f32348db640 1 --2- 172.21.15.88:0/3868340385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3230150660 0x7f323014dab0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f322800cef0 tx=0x7f322800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:41:55.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f322cff9640 1 -- 172.21.15.88:0/3868340385 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322800d840 con 0x7f3230150660 2023-10-16T20:41:55.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f322cff9640 1 -- 172.21.15.88:0/3868340385 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322800d9c0 con 0x7f3230150660 2023-10-16T20:41:55.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f322cff9640 1 -- 172.21.15.88:0/3868340385 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322800dd70 con 0x7f3230150660 2023-10-16T20:41:55.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f323633e640 1 -- 172.21.15.88:0/3868340385 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32300c9910 con 0x7f3230150660 2023-10-16T20:41:55.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f323633e640 1 -- 172.21.15.88:0/3868340385 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32301400d0 con 0x7f3230150660 2023-10-16T20:41:55.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f323633e640 1 -- 172.21.15.88:0/3868340385 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3230141f10 con 0x7f3230150660 2023-10-16T20:41:55.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.949+0000 7f322cff9640 1 -- 172.21.15.88:0/3868340385 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f322801e3b0 con 0x7f3230150660 2023-10-16T20:41:55.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.953+0000 7f322cff9640 1 --2- 172.21.15.88:0/3868340385 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f321c0664a0 0x7f321c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:55.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.953+0000 7f322cff9640 1 -- 172.21.15.88:0/3868340385 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f3228084c40 con 0x7f3230150660 2023-10-16T20:41:55.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.953+0000 7f322ffff640 1 --2- 172.21.15.88:0/3868340385 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f321c0664a0 0x7f321c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:55.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.953+0000 7f322ffff640 1 --2- 172.21.15.88:0/3868340385 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f321c0664a0 0x7f321c068960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f32200b1000 tx=0x7f3220008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:41:55.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:55.957+0000 7f322cff9640 1 -- 172.21.15.88:0/3868340385 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3230141f10 con 0x7f3230150660 2023-10-16T20:41:56.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:56.137+0000 7f323633e640 1 -- 172.21.15.88:0/3868340385 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3230141f10 con 0x7f3230150660 2023-10-16T20:41:56.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:56.137+0000 7f322cff9640 1 -- 172.21.15.88:0/3868340385 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6566 (secure 0 0 0) 0x7f3230141f10 con 0x7f3230150660 2023-10-16T20:41:56.145 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:41:56.145 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:41:32.316596+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":44,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:41:56.145 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 23 2023-10-16T20:41:56.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:56.137+0000 7f323633e640 1 -- 172.21.15.88:0/3868340385 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f321c0664a0 msgr2=0x7f321c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:56.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:56.137+0000 7f323633e640 1 --2- 172.21.15.88:0/3868340385 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f321c0664a0 0x7f321c068960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f32200b1000 tx=0x7f3220008040 comp rx=0 tx=0).stop 2023-10-16T20:41:56.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:56.137+0000 7f323633e640 1 -- 172.21.15.88:0/3868340385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3230150660 msgr2=0x7f323014dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:56.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:56.137+0000 7f323633e640 1 --2- 172.21.15.88:0/3868340385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3230150660 0x7f323014dab0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f322800cef0 tx=0x7f322800a710 comp rx=0 tx=0).stop 2023-10-16T20:41:56.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:56.137+0000 7f323633e640 1 -- 172.21.15.88:0/3868340385 shutdown_connections 2023-10-16T20:41:56.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:56.137+0000 7f323633e640 1 --2- 172.21.15.88:0/3868340385 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f321c0664a0 0x7f321c068960 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:56.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:56.137+0000 7f323633e640 1 --2- 172.21.15.88:0/3868340385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3230150660 0x7f323014dab0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:56.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:56.137+0000 7f323633e640 1 --2- 172.21.15.88:0/3868340385 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3230150280 0x7f323014f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:56.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:56.137+0000 7f323633e640 1 --2- 172.21.15.88:0/3868340385 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f32300c6bc0 0x7f3230140de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:56.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:56.137+0000 7f323633e640 1 -- 172.21.15.88:0/3868340385 >> 172.21.15.88:0/3868340385 conn(0x7f32300cad50 msgr2=0x7f3230153f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:41:56.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:56.137+0000 7f323633e640 1 -- 172.21.15.88:0/3868340385 shutdown_connections 2023-10-16T20:41:56.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:56.137+0000 7f323633e640 1 -- 172.21.15.88:0/3868340385 wait complete. 2023-10-16T20:41:56.165 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-16T20:41:58.166 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:41:58.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.305+0000 7f7776101640 1 Processor -- start 2023-10-16T20:41:58.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.305+0000 7f7776101640 1 -- start start 2023-10-16T20:41:58.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.309+0000 7f7776101640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f77701406d0 0x7f77700c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:58.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.309+0000 7f7776101640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f777006a970 0x7f7770071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:58.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.309+0000 7f7776101640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7770071e60 0x7f777015a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:58.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.309+0000 7f7776101640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f777015b3c0 con 0x7f77701406d0 2023-10-16T20:41:58.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.309+0000 7f7776101640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f777015b680 con 0x7f7770071e60 2023-10-16T20:41:58.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.309+0000 7f7776101640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7770157570 con 0x7f777006a970 2023-10-16T20:41:58.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.309+0000 7f776f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f77701406d0 0x7f77700c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:58.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.309+0000 7f776f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f77701406d0 0x7f77700c1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36244/0 (socket says 172.21.15.88:36244) 2023-10-16T20:41:58.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.309+0000 7f776f7fe640 1 -- 172.21.15.88:0/2723460620 learned_addr learned my addr 172.21.15.88:0/2723460620 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:41:58.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.309+0000 7f776ffff640 1 --2- 172.21.15.88:0/2723460620 >> v2:172.21.15.191:3300/0 conn(0x7f7770071e60 0x7f777015a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:58.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.309+0000 7f7767fff640 1 --2- 172.21.15.88:0/2723460620 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f777006a970 0x7f7770071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:58.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.309+0000 7f776f7fe640 1 -- 172.21.15.88:0/2723460620 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f777006a970 msgr2=0x7f7770071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:41:58.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.309+0000 7f776f7fe640 1 --2- 172.21.15.88:0/2723460620 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f777006a970 0x7f7770071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:58.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.309+0000 7f776f7fe640 1 -- 172.21.15.88:0/2723460620 >> v2:172.21.15.191:3300/0 conn(0x7f7770071e60 msgr2=0x7f777015a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:41:58.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.309+0000 7f776f7fe640 1 --2- 172.21.15.88:0/2723460620 >> v2:172.21.15.191:3300/0 conn(0x7f7770071e60 0x7f777015a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:58.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.309+0000 7f776f7fe640 1 -- 172.21.15.88:0/2723460620 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7770157850 con 0x7f77701406d0 2023-10-16T20:41:58.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.309+0000 7f776ffff640 1 --2- 172.21.15.88:0/2723460620 >> v2:172.21.15.191:3300/0 conn(0x7f7770071e60 0x7f777015a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:41:58.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f776f7fe640 1 --2- 172.21.15.88:0/2723460620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77701406d0 0x7f77700c1d50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f777006caf0 tx=0x7f776001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe38f550a21461d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:41:58.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f776effd640 1 -- 172.21.15.88:0/2723460620 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77600ac600 con 0x7f77701406d0 2023-10-16T20:41:58.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f776effd640 1 -- 172.21.15.88:0/2723460620 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77600a8050 con 0x7f77701406d0 2023-10-16T20:41:58.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f776effd640 1 -- 172.21.15.88:0/2723460620 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77600ac980 con 0x7f77701406d0 2023-10-16T20:41:58.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f7776101640 1 -- 172.21.15.88:0/2723460620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77701406d0 msgr2=0x7f77700c1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:58.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f7776101640 1 --2- 172.21.15.88:0/2723460620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77701406d0 0x7f77700c1d50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f777006caf0 tx=0x7f776001c4d0 comp rx=0 tx=0).stop 2023-10-16T20:41:58.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f7776101640 1 -- 172.21.15.88:0/2723460620 shutdown_connections 2023-10-16T20:41:58.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f7776101640 1 --2- 172.21.15.88:0/2723460620 >> v2:172.21.15.191:3300/0 conn(0x7f7770071e60 0x7f777015a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:58.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f7776101640 1 --2- 172.21.15.88:0/2723460620 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f777006a970 0x7f7770071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:58.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f7776101640 1 --2- 172.21.15.88:0/2723460620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77701406d0 0x7f77700c1d50 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:58.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f7776101640 1 -- 172.21.15.88:0/2723460620 >> 172.21.15.88:0/2723460620 conn(0x7f77700cad50 msgr2=0x7f77700cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:41:58.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f7776101640 1 -- 172.21.15.88:0/2723460620 shutdown_connections 2023-10-16T20:41:58.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f7776101640 1 -- 172.21.15.88:0/2723460620 wait complete. 2023-10-16T20:41:58.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f7776101640 1 Processor -- start 2023-10-16T20:41:58.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f7776101640 1 -- start start 2023-10-16T20:41:58.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f7776101640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f777006a970 0x7f777013c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:58.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f7776101640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7770071e60 0x7f777013c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:58.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.313+0000 7f7776101640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77701406d0 0x7f777013aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:58.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f7776101640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777006f710 con 0x7f77701406d0 2023-10-16T20:41:58.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f7776101640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77701f2fe0 con 0x7f777006a970 2023-10-16T20:41:58.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f7776101640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77700708b0 con 0x7f7770071e60 2023-10-16T20:41:58.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f776ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77701406d0 0x7f777013aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:58.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f776ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77701406d0 0x7f777013aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36258/0 (socket says 172.21.15.88:36258) 2023-10-16T20:41:58.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f776ffff640 1 -- 172.21.15.88:0/1989654610 learned_addr learned my addr 172.21.15.88:0/1989654610 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:41:58.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f776f7fe640 1 --2- 172.21.15.88:0/1989654610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f777006a970 0x7f777013c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:58.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f7767fff640 1 --2- 172.21.15.88:0/1989654610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7770071e60 0x7f777013c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:58.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f776ffff640 1 -- 172.21.15.88:0/1989654610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7770071e60 msgr2=0x7f777013c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:58.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f776ffff640 1 --2- 172.21.15.88:0/1989654610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7770071e60 0x7f777013c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:58.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f776ffff640 1 -- 172.21.15.88:0/1989654610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f777006a970 msgr2=0x7f777013c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:58.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f776ffff640 1 --2- 172.21.15.88:0/1989654610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f777006a970 0x7f777013c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:58.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f776ffff640 1 -- 172.21.15.88:0/1989654610 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77600094d0 con 0x7f77701406d0 2023-10-16T20:41:58.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f776ffff640 1 --2- 172.21.15.88:0/1989654610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77701406d0 0x7f777013aca0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7768009fa0 tx=0x7f776800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:41:58.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f776cff9640 1 -- 172.21.15.88:0/1989654610 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768003ba0 con 0x7f77701406d0 2023-10-16T20:41:58.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f776cff9640 1 -- 172.21.15.88:0/1989654610 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776800f040 con 0x7f77701406d0 2023-10-16T20:41:58.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f776cff9640 1 -- 172.21.15.88:0/1989654610 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768004030 con 0x7f77701406d0 2023-10-16T20:41:58.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f7776101640 1 -- 172.21.15.88:0/1989654610 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7770070b90 con 0x7f77701406d0 2023-10-16T20:41:58.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f7776101640 1 -- 172.21.15.88:0/1989654610 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77700714a0 con 0x7f77701406d0 2023-10-16T20:41:58.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f7776101640 1 -- 172.21.15.88:0/1989654610 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777006f850 con 0x7f77701406d0 2023-10-16T20:41:58.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f776cff9640 1 -- 172.21.15.88:0/1989654610 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7768004190 con 0x7f77701406d0 2023-10-16T20:41:58.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.317+0000 7f776cff9640 1 --2- 172.21.15.88:0/1989654610 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f77500664a0 0x7f7750068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:41:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.321+0000 7f776cff9640 1 -- 172.21.15.88:0/1989654610 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f7768084670 con 0x7f77701406d0 2023-10-16T20:41:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.321+0000 7f776f7fe640 1 --2- 172.21.15.88:0/1989654610 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f77500664a0 0x7f7750068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:41:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.321+0000 7f776f7fe640 1 --2- 172.21.15.88:0/1989654610 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f77500664a0 0x7f7750068960 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f77600ae000 tx=0x7f7760008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:41:58.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.325+0000 7f776cff9640 1 -- 172.21.15.88:0/1989654610 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f777006f850 con 0x7f77701406d0 2023-10-16T20:41:58.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.505+0000 7f7776101640 1 -- 172.21.15.88:0/1989654610 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f777006f850 con 0x7f77701406d0 2023-10-16T20:41:58.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.505+0000 7f776cff9640 1 -- 172.21.15.88:0/1989654610 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6566 (secure 0 0 0) 0x7f777006f850 con 0x7f77701406d0 2023-10-16T20:41:58.515 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:41:58.515 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:41:32.316596+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":44,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:41:58.515 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 23 2023-10-16T20:41:58.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.509+0000 7f7776101640 1 -- 172.21.15.88:0/1989654610 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f77500664a0 msgr2=0x7f7750068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:58.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.509+0000 7f7776101640 1 --2- 172.21.15.88:0/1989654610 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f77500664a0 0x7f7750068960 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f77600ae000 tx=0x7f7760008040 comp rx=0 tx=0).stop 2023-10-16T20:41:58.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.509+0000 7f7776101640 1 -- 172.21.15.88:0/1989654610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77701406d0 msgr2=0x7f777013aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:41:58.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.509+0000 7f7776101640 1 --2- 172.21.15.88:0/1989654610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77701406d0 0x7f777013aca0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7768009fa0 tx=0x7f776800d560 comp rx=0 tx=0).stop 2023-10-16T20:41:58.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.509+0000 7f7776101640 1 -- 172.21.15.88:0/1989654610 shutdown_connections 2023-10-16T20:41:58.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.509+0000 7f7776101640 1 --2- 172.21.15.88:0/1989654610 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f77500664a0 0x7f7750068960 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:58.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.509+0000 7f7776101640 1 --2- 172.21.15.88:0/1989654610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77701406d0 0x7f777013aca0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:58.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.509+0000 7f7776101640 1 --2- 172.21.15.88:0/1989654610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7770071e60 0x7f777013c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:58.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.509+0000 7f7776101640 1 --2- 172.21.15.88:0/1989654610 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f777006a970 0x7f777013c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:41:58.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.509+0000 7f7776101640 1 -- 172.21.15.88:0/1989654610 >> 172.21.15.88:0/1989654610 conn(0x7f77700cad50 msgr2=0x7f77700732e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:41:58.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.509+0000 7f7776101640 1 -- 172.21.15.88:0/1989654610 shutdown_connections 2023-10-16T20:41:58.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:41:58.509+0000 7f7776101640 1 -- 172.21.15.88:0/1989654610 wait complete. 2023-10-16T20:41:58.535 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-16T20:42:00.536 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:00.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.677+0000 7f4f26446640 1 Processor -- start 2023-10-16T20:42:00.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.677+0000 7f4f26446640 1 -- start start 2023-10-16T20:42:00.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.677+0000 7f4f26446640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f201406d0 0x7f4f200c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:00.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.681+0000 7f4f26446640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f2006a970 0x7f4f20071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:00.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.681+0000 7f4f26446640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2015b3c0 con 0x7f4f20071e60 2023-10-16T20:42:00.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.681+0000 7f4f26446640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f2015b680 con 0x7f4f2006a970 2023-10-16T20:42:00.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.681+0000 7f4f26446640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f20157570 con 0x7f4f201406d0 2023-10-16T20:42:00.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.681+0000 7f4f1f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f2006a970 0x7f4f20071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:00.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.681+0000 7f4f1f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f2006a970 0x7f4f20071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38308/0 (socket says 172.21.15.88:38308) 2023-10-16T20:42:00.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.681+0000 7f4f1f7fe640 1 -- 172.21.15.88:0/3727616645 learned_addr learned my addr 172.21.15.88:0/3727616645 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:00.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.681+0000 7f4f1ffff640 1 --2- 172.21.15.88:0/3727616645 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f201406d0 0x7f4f200c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:00.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.681+0000 7f4f1f7fe640 1 -- 172.21.15.88:0/3727616645 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f201406d0 msgr2=0x7f4f200c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:00.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.681+0000 7f4f1f7fe640 1 --2- 172.21.15.88:0/3727616645 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f201406d0 0x7f4f200c1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:00.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.681+0000 7f4f1f7fe640 1 -- 172.21.15.88:0/3727616645 >> v1:172.21.15.191:6789/0 conn(0x7f4f20071e60 legacy=0x7f4f2015a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:00.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.681+0000 7f4f1f7fe640 1 -- 172.21.15.88:0/3727616645 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f20157850 con 0x7f4f2006a970 2023-10-16T20:42:00.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.681+0000 7f4f1f7fe640 1 --2- 172.21.15.88:0/3727616645 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f2006a970 0x7f4f20071920 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4f14009870 tx=0x7f4f1401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0be8f949ab1d25 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:00.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.681+0000 7f4f1effd640 1 -- 172.21.15.88:0/3727616645 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f140ab620 con 0x7f4f2006a970 2023-10-16T20:42:00.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f1effd640 1 -- 172.21.15.88:0/3727616645 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f140ab780 con 0x7f4f2006a970 2023-10-16T20:42:00.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f1effd640 1 -- 172.21.15.88:0/3727616645 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f140abb10 con 0x7f4f2006a970 2023-10-16T20:42:00.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f26446640 1 -- 172.21.15.88:0/3727616645 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f2006a970 msgr2=0x7f4f20071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:00.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f26446640 1 --2- 172.21.15.88:0/3727616645 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f2006a970 0x7f4f20071920 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4f14009870 tx=0x7f4f1401c510 comp rx=0 tx=0).stop 2023-10-16T20:42:00.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f26446640 1 -- 172.21.15.88:0/3727616645 shutdown_connections 2023-10-16T20:42:00.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f26446640 1 --2- 172.21.15.88:0/3727616645 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f2006a970 0x7f4f20071920 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:00.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f26446640 1 --2- 172.21.15.88:0/3727616645 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f201406d0 0x7f4f200c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:00.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f26446640 1 -- 172.21.15.88:0/3727616645 >> 172.21.15.88:0/3727616645 conn(0x7f4f200cad50 msgr2=0x7f4f200cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:00.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f26446640 1 -- 172.21.15.88:0/3727616645 shutdown_connections 2023-10-16T20:42:00.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f26446640 1 -- 172.21.15.88:0/3727616645 wait complete. 2023-10-16T20:42:00.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f26446640 1 Processor -- start 2023-10-16T20:42:00.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f26446640 1 -- start start 2023-10-16T20:42:00.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f26446640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f2006a970 0x7f4f200709c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:00.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f26446640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f20071e60 0x7f4f200c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:00.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f26446640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f201406d0 0x7f4f200c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:00.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f26446640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f200c7520 con 0x7f4f2006a970 2023-10-16T20:42:00.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f26446640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f200c62a0 con 0x7f4f201406d0 2023-10-16T20:42:00.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f26446640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f201f23b0 con 0x7f4f20071e60 2023-10-16T20:42:00.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f1ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f2006a970 0x7f4f200709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:00.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f1ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f2006a970 0x7f4f200709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38324/0 (socket says 172.21.15.88:38324) 2023-10-16T20:42:00.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f1ffff640 1 -- 172.21.15.88:0/90994263 learned_addr learned my addr 172.21.15.88:0/90994263 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:00.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.685+0000 7f4f1f7fe640 1 --2- 172.21.15.88:0/90994263 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f20071e60 0x7f4f200c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:00.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.689+0000 7f4f249e3640 1 --2- 172.21.15.88:0/90994263 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f201406d0 0x7f4f200c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:00.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.689+0000 7f4f1ffff640 1 -- 172.21.15.88:0/90994263 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f20071e60 msgr2=0x7f4f200c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:00.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.689+0000 7f4f1ffff640 1 --2- 172.21.15.88:0/90994263 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f20071e60 0x7f4f200c7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:00.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.689+0000 7f4f1ffff640 1 -- 172.21.15.88:0/90994263 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f201406d0 msgr2=0x7f4f200c7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:00.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.689+0000 7f4f1ffff640 1 --2- 172.21.15.88:0/90994263 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f201406d0 0x7f4f200c7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:00.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.689+0000 7f4f1ffff640 1 -- 172.21.15.88:0/90994263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f140094d0 con 0x7f4f2006a970 2023-10-16T20:42:00.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.689+0000 7f4f1ffff640 1 --2- 172.21.15.88:0/90994263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f2006a970 0x7f4f200709c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4f1000d800 tx=0x7f4f1000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:00.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.689+0000 7f4f1cff9640 1 -- 172.21.15.88:0/90994263 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1000b6c0 con 0x7f4f2006a970 2023-10-16T20:42:00.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.689+0000 7f4f1cff9640 1 -- 172.21.15.88:0/90994263 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1000b820 con 0x7f4f2006a970 2023-10-16T20:42:00.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.689+0000 7f4f26446640 1 -- 172.21.15.88:0/90994263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f201581d0 con 0x7f4f2006a970 2023-10-16T20:42:00.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.689+0000 7f4f1cff9640 1 -- 172.21.15.88:0/90994263 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1000bb90 con 0x7f4f2006a970 2023-10-16T20:42:00.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.689+0000 7f4f26446640 1 -- 172.21.15.88:0/90994263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f201f2970 con 0x7f4f2006a970 2023-10-16T20:42:00.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.689+0000 7f4f26446640 1 -- 172.21.15.88:0/90994263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f20074700 con 0x7f4f2006a970 2023-10-16T20:42:00.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.689+0000 7f4f1cff9640 1 -- 172.21.15.88:0/90994263 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4f1000bcf0 con 0x7f4f2006a970 2023-10-16T20:42:00.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.689+0000 7f4f1cff9640 1 --2- 172.21.15.88:0/90994263 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4f0c0664a0 0x7f4f0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:00.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.689+0000 7f4f1cff9640 1 -- 172.21.15.88:0/90994263 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f4f100847c0 con 0x7f4f2006a970 2023-10-16T20:42:00.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.693+0000 7f4f1f7fe640 1 --2- 172.21.15.88:0/90994263 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4f0c0664a0 0x7f4f0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:00.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.693+0000 7f4f1f7fe640 1 --2- 172.21.15.88:0/90994263 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4f0c0664a0 0x7f4f0c068960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4f140af000 tx=0x7f4f140b6040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:00.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.697+0000 7f4f1cff9640 1 -- 172.21.15.88:0/90994263 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4f100517b0 con 0x7f4f2006a970 2023-10-16T20:42:00.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.877+0000 7f4f26446640 1 -- 172.21.15.88:0/90994263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f200c3dc0 con 0x7f4f2006a970 2023-10-16T20:42:00.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.877+0000 7f4f1cff9640 1 -- 172.21.15.88:0/90994263 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6566 (secure 0 0 0) 0x7f4f10054dd0 con 0x7f4f2006a970 2023-10-16T20:42:00.884 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:00.884 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:41:32.316596+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":44,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:00.885 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 23 2023-10-16T20:42:00.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.877+0000 7f4f26446640 1 -- 172.21.15.88:0/90994263 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4f0c0664a0 msgr2=0x7f4f0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:00.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.877+0000 7f4f26446640 1 --2- 172.21.15.88:0/90994263 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4f0c0664a0 0x7f4f0c068960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4f140af000 tx=0x7f4f140b6040 comp rx=0 tx=0).stop 2023-10-16T20:42:00.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.877+0000 7f4f26446640 1 -- 172.21.15.88:0/90994263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f2006a970 msgr2=0x7f4f200709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:00.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.877+0000 7f4f26446640 1 --2- 172.21.15.88:0/90994263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f2006a970 0x7f4f200709c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4f1000d800 tx=0x7f4f1000dcc0 comp rx=0 tx=0).stop 2023-10-16T20:42:00.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.877+0000 7f4f26446640 1 -- 172.21.15.88:0/90994263 shutdown_connections 2023-10-16T20:42:00.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.877+0000 7f4f26446640 1 --2- 172.21.15.88:0/90994263 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4f0c0664a0 0x7f4f0c068960 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:00.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.877+0000 7f4f26446640 1 --2- 172.21.15.88:0/90994263 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f201406d0 0x7f4f200c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:00.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.877+0000 7f4f26446640 1 --2- 172.21.15.88:0/90994263 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f20071e60 0x7f4f200c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:00.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.877+0000 7f4f26446640 1 --2- 172.21.15.88:0/90994263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f2006a970 0x7f4f200709c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:00.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.877+0000 7f4f26446640 1 -- 172.21.15.88:0/90994263 >> 172.21.15.88:0/90994263 conn(0x7f4f200cad50 msgr2=0x7f4f2013e520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:00.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.877+0000 7f4f26446640 1 -- 172.21.15.88:0/90994263 shutdown_connections 2023-10-16T20:42:00.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:00.877+0000 7f4f26446640 1 -- 172.21.15.88:0/90994263 wait complete. 2023-10-16T20:42:00.905 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-16T20:42:02.906 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:03.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa67d394640 1 Processor -- start 2023-10-16T20:42:03.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa67d394640 1 -- start start 2023-10-16T20:42:03.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa67d394640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa6780c5120 0x7fa6780c1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:03.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa67d394640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa678140990 con 0x7fa6781405b0 2023-10-16T20:42:03.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa67d394640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6780c7df0 con 0x7fa6780be6d0 2023-10-16T20:42:03.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa67d394640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa678157170 con 0x7fa6780c5120 2023-10-16T20:42:03.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa676ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa6781405b0 0x7fa6780c3db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59556/0 (socket says 172.21.15.88:59556) 2023-10-16T20:42:03.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa676ffd640 1 -- 172.21.15.88:0/1022950081 learned_addr learned my addr 172.21.15.88:0/1022950081 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:42:03.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa6767fc640 1 --2- 172.21.15.88:0/1022950081 >> v2:172.21.15.191:3300/0 conn(0x7fa6780c5120 0x7fa6780c1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:03.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa675ffb640 1 -- 172.21.15.88:0/1022950081 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511252762 0 0) 0x7fa678140990 con 0x7fa6781405b0 2023-10-16T20:42:03.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa675ffb640 1 -- 172.21.15.88:0/1022950081 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa660003530 con 0x7fa6781405b0 2023-10-16T20:42:03.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa675ffb640 1 -- 172.21.15.88:0/1022950081 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455020436 0 0) 0x7fa6780c7df0 con 0x7fa6780be6d0 2023-10-16T20:42:03.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa675ffb640 1 -- 172.21.15.88:0/1022950081 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa678140990 con 0x7fa6780be6d0 2023-10-16T20:42:03.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa675ffb640 1 -- 172.21.15.88:0/1022950081 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1626642565 0 0) 0x7fa660003530 con 0x7fa6781405b0 2023-10-16T20:42:03.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa675ffb640 1 -- 172.21.15.88:0/1022950081 >> v2:172.21.15.191:3300/0 conn(0x7fa6780c5120 msgr2=0x7fa6780c1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:03.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa675ffb640 1 --2- 172.21.15.88:0/1022950081 >> v2:172.21.15.191:3300/0 conn(0x7fa6780c5120 0x7fa6780c1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:03.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa675ffb640 1 -- 172.21.15.88:0/1022950081 >> v1:172.21.15.191:6789/0 conn(0x7fa6780be6d0 legacy=0x7fa6780beb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:03.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa675ffb640 1 -- 172.21.15.88:0/1022950081 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa678157450 con 0x7fa6781405b0 2023-10-16T20:42:03.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa675ffb640 1 -- 172.21.15.88:0/1022950081 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7fa668001680 con 0x7fa6781405b0 2023-10-16T20:42:03.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa675ffb640 1 -- 172.21.15.88:0/1022950081 >> v1:172.21.15.88:6789/0 conn(0x7fa6781405b0 legacy=0x7fa6780c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:03.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa675ffb640 1 --2- 172.21.15.88:0/1022950081 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa660003b90 0x7fa660003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:03.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa675ffb640 1 --2- 172.21.15.88:0/1022950081 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6600047d0 0x7fa660004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:03.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa675ffb640 1 --2- 172.21.15.88:0/1022950081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa660005190 0x7fa66000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:03.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa675ffb640 1 -- 172.21.15.88:0/1022950081 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa678157170 con 0x7fa660005190 2023-10-16T20:42:03.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa675ffb640 1 -- 172.21.15.88:0/1022950081 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66000c090 con 0x7fa6600047d0 2023-10-16T20:42:03.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa675ffb640 1 -- 172.21.15.88:0/1022950081 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66000c250 con 0x7fa660003b90 2023-10-16T20:42:03.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa6777fe640 1 --2- 172.21.15.88:0/1022950081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa660005190 0x7fa66000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:03.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa676ffd640 1 --2- 172.21.15.88:0/1022950081 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa660003b90 0x7fa660003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:03.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa6767fc640 1 --2- 172.21.15.88:0/1022950081 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6600047d0 0x7fa660004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:03.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa6777fe640 1 -- 172.21.15.88:0/1022950081 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa660003b90 msgr2=0x7fa660003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:03.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa6777fe640 1 --2- 172.21.15.88:0/1022950081 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa660003b90 0x7fa660003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:03.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa6777fe640 1 -- 172.21.15.88:0/1022950081 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6600047d0 msgr2=0x7fa660004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:03.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa6777fe640 1 --2- 172.21.15.88:0/1022950081 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6600047d0 0x7fa660004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:03.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.049+0000 7fa6777fe640 1 -- 172.21.15.88:0/1022950081 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa660003530 con 0x7fa660005190 2023-10-16T20:42:03.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.053+0000 7fa6777fe640 1 -- 172.21.15.88:0/1022950081 reap_dead start 2023-10-16T20:42:03.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.053+0000 7fa6777fe640 1 --2- 172.21.15.88:0/1022950081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa660005190 0x7fa66000b990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa670001fd0 tx=0x7fa67001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68575baadd2a720e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:03.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.053+0000 7fa675ffb640 1 -- 172.21.15.88:0/1022950081 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa670003b80 con 0x7fa660005190 2023-10-16T20:42:03.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.053+0000 7fa675ffb640 1 -- 172.21.15.88:0/1022950081 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa670003d00 con 0x7fa660005190 2023-10-16T20:42:03.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.053+0000 7fa675ffb640 1 -- 172.21.15.88:0/1022950081 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa670005310 con 0x7fa660005190 2023-10-16T20:42:03.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.053+0000 7fa67d394640 1 -- 172.21.15.88:0/1022950081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa660005190 msgr2=0x7fa66000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:03.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.053+0000 7fa67d394640 1 --2- 172.21.15.88:0/1022950081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa660005190 0x7fa66000b990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa670001fd0 tx=0x7fa67001c8e0 comp rx=0 tx=0).stop 2023-10-16T20:42:03.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.053+0000 7fa67d394640 1 -- 172.21.15.88:0/1022950081 shutdown_connections 2023-10-16T20:42:03.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.053+0000 7fa67d394640 1 --2- 172.21.15.88:0/1022950081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa660005190 0x7fa66000b990 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:03.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.053+0000 7fa67d394640 1 -- 172.21.15.88:0/1022950081 >> 172.21.15.88:0/1022950081 conn(0x7fa6780c7220 msgr2=0x7fa6780c7630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:03.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.053+0000 7fa67d394640 1 -- 172.21.15.88:0/1022950081 shutdown_connections 2023-10-16T20:42:03.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.053+0000 7fa67d394640 1 -- 172.21.15.88:0/1022950081 wait complete. 2023-10-16T20:42:03.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa67d394640 1 Processor -- start 2023-10-16T20:42:03.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa67d394640 1 -- start start 2023-10-16T20:42:03.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa67d394640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa660005190 0x7fa67813d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:03.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa67d394640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa678158220 0x7fa678153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:03.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa67d394640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa678154110 0x7fa6781f13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:03.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa67d394640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6781534a0 con 0x7fa678154110 2023-10-16T20:42:03.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa67d394640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6781523e0 con 0x7fa660005190 2023-10-16T20:42:03.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa67d394640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6780c8580 con 0x7fa678158220 2023-10-16T20:42:03.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa6777fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa678154110 0x7fa6781f13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:03.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa6777fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa678154110 0x7fa6781f13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38346/0 (socket says 172.21.15.88:38346) 2023-10-16T20:42:03.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa6777fe640 1 -- 172.21.15.88:0/2168547045 learned_addr learned my addr 172.21.15.88:0/2168547045 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:03.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa6767fc640 1 --2- 172.21.15.88:0/2168547045 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa678158220 0x7fa678153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:03.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa676ffd640 1 --2- 172.21.15.88:0/2168547045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa660005190 0x7fa67813d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:03.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa6777fe640 1 -- 172.21.15.88:0/2168547045 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa678158220 msgr2=0x7fa678153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:03.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa6777fe640 1 --2- 172.21.15.88:0/2168547045 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa678158220 0x7fa678153bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:03.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa6777fe640 1 -- 172.21.15.88:0/2168547045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa660005190 msgr2=0x7fa67813d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:03.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa6777fe640 1 --2- 172.21.15.88:0/2168547045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa660005190 0x7fa67813d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:03.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa6777fe640 1 -- 172.21.15.88:0/2168547045 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6781587d0 con 0x7fa678154110 2023-10-16T20:42:03.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa6777fe640 1 --2- 172.21.15.88:0/2168547045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa678154110 0x7fa6781f13e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa67001d180 tx=0x7fa660005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:03.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa65f7fe640 1 -- 172.21.15.88:0/2168547045 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6700b4070 con 0x7fa678154110 2023-10-16T20:42:03.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa65f7fe640 1 -- 172.21.15.88:0/2168547045 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa670003e70 con 0x7fa678154110 2023-10-16T20:42:03.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa67d394640 1 -- 172.21.15.88:0/2168547045 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa678155e00 con 0x7fa678154110 2023-10-16T20:42:03.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa65f7fe640 1 -- 172.21.15.88:0/2168547045 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa670005880 con 0x7fa678154110 2023-10-16T20:42:03.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.057+0000 7fa67d394640 1 -- 172.21.15.88:0/2168547045 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6781563c0 con 0x7fa678154110 2023-10-16T20:42:03.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.061+0000 7fa67d394640 1 -- 172.21.15.88:0/2168547045 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6781565b0 con 0x7fa678154110 2023-10-16T20:42:03.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.061+0000 7fa65f7fe640 1 -- 172.21.15.88:0/2168547045 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa670006b60 con 0x7fa678154110 2023-10-16T20:42:03.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.061+0000 7fa65f7fe640 1 --2- 172.21.15.88:0/2168547045 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6640664a0 0x7fa664068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:03.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.061+0000 7fa65f7fe640 1 -- 172.21.15.88:0/2168547045 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fa6700aa070 con 0x7fa678154110 2023-10-16T20:42:03.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.061+0000 7fa676ffd640 1 --2- 172.21.15.88:0/2168547045 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6640664a0 0x7fa664068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:03.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.061+0000 7fa676ffd640 1 --2- 172.21.15.88:0/2168547045 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6640664a0 0x7fa664068960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa668001060 tx=0x7fa668001120 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:03.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.065+0000 7fa65f7fe640 1 -- 172.21.15.88:0/2168547045 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa6700ea960 con 0x7fa678154110 2023-10-16T20:42:03.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.245+0000 7fa67d394640 1 -- 172.21.15.88:0/2168547045 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa67804e9c0 con 0x7fa678154110 2023-10-16T20:42:03.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.245+0000 7fa65f7fe640 1 -- 172.21.15.88:0/2168547045 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6566 (secure 0 0 0) 0x7fa6700ec890 con 0x7fa678154110 2023-10-16T20:42:03.253 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:03.254 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:41:32.316596+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":44,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:03.255 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 23 2023-10-16T20:42:03.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.245+0000 7fa67d394640 1 -- 172.21.15.88:0/2168547045 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6640664a0 msgr2=0x7fa664068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:03.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.245+0000 7fa67d394640 1 --2- 172.21.15.88:0/2168547045 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6640664a0 0x7fa664068960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa668001060 tx=0x7fa668001120 comp rx=0 tx=0).stop 2023-10-16T20:42:03.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.245+0000 7fa67d394640 1 -- 172.21.15.88:0/2168547045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa678154110 msgr2=0x7fa6781f13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:03.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.245+0000 7fa67d394640 1 --2- 172.21.15.88:0/2168547045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa678154110 0x7fa6781f13e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa67001d180 tx=0x7fa660005800 comp rx=0 tx=0).stop 2023-10-16T20:42:03.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.245+0000 7fa67d394640 1 -- 172.21.15.88:0/2168547045 shutdown_connections 2023-10-16T20:42:03.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.245+0000 7fa67d394640 1 --2- 172.21.15.88:0/2168547045 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6640664a0 0x7fa664068960 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:03.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.245+0000 7fa67d394640 1 --2- 172.21.15.88:0/2168547045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa678154110 0x7fa6781f13e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:03.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.245+0000 7fa67d394640 1 --2- 172.21.15.88:0/2168547045 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa678158220 0x7fa678153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:03.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.245+0000 7fa67d394640 1 --2- 172.21.15.88:0/2168547045 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa660005190 0x7fa67813d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:03.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.245+0000 7fa67d394640 1 -- 172.21.15.88:0/2168547045 >> 172.21.15.88:0/2168547045 conn(0x7fa6780c7220 msgr2=0x7fa67806d790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:03.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.245+0000 7fa67d394640 1 -- 172.21.15.88:0/2168547045 shutdown_connections 2023-10-16T20:42:03.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:03.245+0000 7fa67d394640 1 -- 172.21.15.88:0/2168547045 wait complete. 2023-10-16T20:42:03.274 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-16T20:42:05.275 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:05.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f220b29c640 1 Processor -- start 2023-10-16T20:42:05.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f220b29c640 1 -- start start 2023-10-16T20:42:05.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f220b29c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22040cece0 0x7f22040cf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:05.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f220b29c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2204153b50 con 0x7f2204153770 2023-10-16T20:42:05.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f220b29c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22040cb2f0 con 0x7f22040c9bb0 2023-10-16T20:42:05.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f220b29c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22040cc250 con 0x7f22040cece0 2023-10-16T20:42:05.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f2209038640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2204153770 0x7f2204159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59558/0 (socket says 172.21.15.88:59558) 2023-10-16T20:42:05.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f2209038640 1 -- 172.21.15.88:0/2640242852 learned_addr learned my addr 172.21.15.88:0/2640242852 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:42:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f2209839640 1 --2- 172.21.15.88:0/2640242852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22040cece0 0x7f22040cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f21fbfff640 1 -- 172.21.15.88:0/2640242852 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486628249 0 0) 0x7f2204153b50 con 0x7f2204153770 2023-10-16T20:42:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f21fbfff640 1 -- 172.21.15.88:0/2640242852 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21f0003530 con 0x7f2204153770 2023-10-16T20:42:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f21fbfff640 1 -- 172.21.15.88:0/2640242852 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927052101 0 0) 0x7f22040cb2f0 con 0x7f22040c9bb0 2023-10-16T20:42:05.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f21fbfff640 1 -- 172.21.15.88:0/2640242852 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2204153b50 con 0x7f22040c9bb0 2023-10-16T20:42:05.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f21fbfff640 1 -- 172.21.15.88:0/2640242852 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1956792294 0 0) 0x7f21f0003530 con 0x7f2204153770 2023-10-16T20:42:05.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f21fbfff640 1 -- 172.21.15.88:0/2640242852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22040cece0 msgr2=0x7f22040cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:05.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f21fbfff640 1 --2- 172.21.15.88:0/2640242852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22040cece0 0x7f22040cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:05.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f21fbfff640 1 -- 172.21.15.88:0/2640242852 >> v1:172.21.15.191:6789/0 conn(0x7f22040c9bb0 legacy=0x7f22040c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:05.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f21fbfff640 1 -- 172.21.15.88:0/2640242852 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22040cb5d0 con 0x7f2204153770 2023-10-16T20:42:05.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f21fbfff640 1 -- 172.21.15.88:0/2640242852 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f2200001800 con 0x7f2204153770 2023-10-16T20:42:05.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f21fbfff640 1 -- 172.21.15.88:0/2640242852 >> v1:172.21.15.88:6789/0 conn(0x7f2204153770 legacy=0x7f2204159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:05.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f21fbfff640 1 --2- 172.21.15.88:0/2640242852 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21f0003b40 0x7f21f0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:05.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f21fbfff640 1 --2- 172.21.15.88:0/2640242852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21f00047d0 0x7f21f0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:05.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.417+0000 7f21fbfff640 1 --2- 172.21.15.88:0/2640242852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22040cece0 0x7f21f0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:05.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.421+0000 7f21fbfff640 1 -- 172.21.15.88:0/2640242852 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22040cc250 con 0x7f21f0003b40 2023-10-16T20:42:05.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.421+0000 7f21fbfff640 1 -- 172.21.15.88:0/2640242852 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f000ba40 con 0x7f21f00047d0 2023-10-16T20:42:05.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.421+0000 7f21fbfff640 1 -- 172.21.15.88:0/2640242852 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f000bbd0 con 0x7f22040cece0 2023-10-16T20:42:05.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.421+0000 7f2209038640 1 --2- 172.21.15.88:0/2640242852 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21f0003b40 0x7f21f0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:05.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.421+0000 7f2209839640 1 --2- 172.21.15.88:0/2640242852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22040cece0 0x7f21f0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:05.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.421+0000 7f2208837640 1 --2- 172.21.15.88:0/2640242852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21f00047d0 0x7f21f0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:05.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.421+0000 7f2209038640 1 -- 172.21.15.88:0/2640242852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22040cece0 msgr2=0x7f21f0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:05.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.421+0000 7f2209038640 1 --2- 172.21.15.88:0/2640242852 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22040cece0 0x7f21f0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:05.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.421+0000 7f2209038640 1 -- 172.21.15.88:0/2640242852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21f00047d0 msgr2=0x7f21f0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:05.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.421+0000 7f2209038640 1 --2- 172.21.15.88:0/2640242852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21f00047d0 0x7f21f0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:05.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.421+0000 7f2209038640 1 -- 172.21.15.88:0/2640242852 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f0003530 con 0x7f21f0003b40 2023-10-16T20:42:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.421+0000 7f2209038640 1 --2- 172.21.15.88:0/2640242852 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21f0003b40 0x7f21f0003f40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f220000c190 tx=0x7f220001cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c5da5849ee2822 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:05.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.421+0000 7f21fbfff640 1 -- 172.21.15.88:0/2640242852 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22000070d0 con 0x7f21f0003b40 2023-10-16T20:42:05.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.421+0000 7f21fbfff640 1 -- 172.21.15.88:0/2640242852 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2200007230 con 0x7f21f0003b40 2023-10-16T20:42:05.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.421+0000 7f21fbfff640 1 -- 172.21.15.88:0/2640242852 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2200007600 con 0x7f21f0003b40 2023-10-16T20:42:05.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f220b29c640 1 -- 172.21.15.88:0/2640242852 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21f0003b40 msgr2=0x7f21f0003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:05.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f220b29c640 1 --2- 172.21.15.88:0/2640242852 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21f0003b40 0x7f21f0003f40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f220000c190 tx=0x7f220001cb60 comp rx=0 tx=0).stop 2023-10-16T20:42:05.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f2209839640 1 -- 172.21.15.88:0/2640242852 reap_dead start 2023-10-16T20:42:05.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f220b29c640 1 -- 172.21.15.88:0/2640242852 shutdown_connections 2023-10-16T20:42:05.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f220b29c640 1 -- 172.21.15.88:0/2640242852 >> 172.21.15.88:0/2640242852 conn(0x7f22040715c0 msgr2=0x7f22040719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:05.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f220b29c640 1 -- 172.21.15.88:0/2640242852 shutdown_connections 2023-10-16T20:42:05.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f220b29c640 1 -- 172.21.15.88:0/2640242852 wait complete. 2023-10-16T20:42:05.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f220b29c640 1 Processor -- start 2023-10-16T20:42:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f220b29c640 1 -- start start 2023-10-16T20:42:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f220b29c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21f0003b40 0x7f22040c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f220b29c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22040c5cc0 0x7f22041ea300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f220b29c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2204146560 0x7f22041469e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:05.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f220b29c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2204154b80 con 0x7f21f0003b40 2023-10-16T20:42:05.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f220b29c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22041578d0 con 0x7f2204146560 2023-10-16T20:42:05.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f220b29c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2204146f20 con 0x7f22040c5cc0 2023-10-16T20:42:05.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f2209038640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21f0003b40 0x7f22040c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:05.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f2209038640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21f0003b40 0x7f22040c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38360/0 (socket says 172.21.15.88:38360) 2023-10-16T20:42:05.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f2209038640 1 -- 172.21.15.88:0/3273868030 learned_addr learned my addr 172.21.15.88:0/3273868030 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:05.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f2209038640 1 -- 172.21.15.88:0/3273868030 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22040c5cc0 msgr2=0x7f22041ea300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:05.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f2209038640 1 --2- 172.21.15.88:0/3273868030 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22040c5cc0 0x7f22041ea300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:05.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f2209038640 1 -- 172.21.15.88:0/3273868030 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2204146560 msgr2=0x7f22041469e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:05.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f2209038640 1 --2- 172.21.15.88:0/3273868030 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2204146560 0x7f22041469e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:05.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f2209038640 1 -- 172.21.15.88:0/3273868030 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22041f5df0 con 0x7f21f0003b40 2023-10-16T20:42:05.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.425+0000 7f2209038640 1 --2- 172.21.15.88:0/3273868030 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21f0003b40 0x7f22040c73a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f220000c190 tx=0x7f21f0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:05.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.429+0000 7f21f9ffb640 1 -- 172.21.15.88:0/3273868030 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2200001800 con 0x7f21f0003b40 2023-10-16T20:42:05.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.429+0000 7f21f9ffb640 1 -- 172.21.15.88:0/3273868030 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2200000f80 con 0x7f21f0003b40 2023-10-16T20:42:05.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.429+0000 7f220b29c640 1 -- 172.21.15.88:0/3273868030 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22040c7a50 con 0x7f21f0003b40 2023-10-16T20:42:05.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.429+0000 7f21f9ffb640 1 -- 172.21.15.88:0/3273868030 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22000059b0 con 0x7f21f0003b40 2023-10-16T20:42:05.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.429+0000 7f220b29c640 1 -- 172.21.15.88:0/3273868030 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22040c8010 con 0x7f21f0003b40 2023-10-16T20:42:05.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.429+0000 7f220b29c640 1 -- 172.21.15.88:0/3273868030 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2204074700 con 0x7f21f0003b40 2023-10-16T20:42:05.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.429+0000 7f21f9ffb640 1 -- 172.21.15.88:0/3273868030 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f22000b6070 con 0x7f21f0003b40 2023-10-16T20:42:05.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.429+0000 7f21f9ffb640 1 --2- 172.21.15.88:0/3273868030 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21e80664d0 0x7f21e8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:05.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.429+0000 7f21f9ffb640 1 -- 172.21.15.88:0/3273868030 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f220011d680 con 0x7f21f0003b40 2023-10-16T20:42:05.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.429+0000 7f2208837640 1 --2- 172.21.15.88:0/3273868030 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21e80664d0 0x7f21e8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:05.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.433+0000 7f2208837640 1 --2- 172.21.15.88:0/3273868030 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21e80664d0 0x7f21e8068990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f22040ccf50 tx=0x7f21f4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:05.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.433+0000 7f21f9ffb640 1 -- 172.21.15.88:0/3273868030 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f22000ea5e0 con 0x7f21f0003b40 2023-10-16T20:42:05.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.613+0000 7f220b29c640 1 -- 172.21.15.88:0/3273868030 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22040cd110 con 0x7f21f0003b40 2023-10-16T20:42:05.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.617+0000 7f21f9ffb640 1 -- 172.21.15.88:0/3273868030 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6566 (secure 0 0 0) 0x7f22000edc00 con 0x7f21f0003b40 2023-10-16T20:42:05.623 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:05.624 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:41:32.316596+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":44,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:05.624 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 23 2023-10-16T20:42:05.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.617+0000 7f220b29c640 1 -- 172.21.15.88:0/3273868030 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21e80664d0 msgr2=0x7f21e8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:05.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.617+0000 7f220b29c640 1 --2- 172.21.15.88:0/3273868030 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21e80664d0 0x7f21e8068990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f22040ccf50 tx=0x7f21f4009040 comp rx=0 tx=0).stop 2023-10-16T20:42:05.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.617+0000 7f220b29c640 1 -- 172.21.15.88:0/3273868030 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21f0003b40 msgr2=0x7f22040c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:05.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.617+0000 7f220b29c640 1 --2- 172.21.15.88:0/3273868030 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21f0003b40 0x7f22040c73a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f220000c190 tx=0x7f21f0005800 comp rx=0 tx=0).stop 2023-10-16T20:42:05.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.617+0000 7f220b29c640 1 -- 172.21.15.88:0/3273868030 shutdown_connections 2023-10-16T20:42:05.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.617+0000 7f220b29c640 1 --2- 172.21.15.88:0/3273868030 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21e80664d0 0x7f21e8068990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:05.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.617+0000 7f220b29c640 1 --2- 172.21.15.88:0/3273868030 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2204146560 0x7f22041469e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:05.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.617+0000 7f220b29c640 1 --2- 172.21.15.88:0/3273868030 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f22040c5cc0 0x7f22041ea300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:05.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.617+0000 7f220b29c640 1 --2- 172.21.15.88:0/3273868030 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21f0003b40 0x7f22040c73a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:05.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.617+0000 7f220b29c640 1 -- 172.21.15.88:0/3273868030 >> 172.21.15.88:0/3273868030 conn(0x7f22040715c0 msgr2=0x7f2204149a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:05.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.617+0000 7f220b29c640 1 -- 172.21.15.88:0/3273868030 shutdown_connections 2023-10-16T20:42:05.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:05.617+0000 7f220b29c640 1 -- 172.21.15.88:0/3273868030 wait complete. 2023-10-16T20:42:05.644 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-16T20:42:07.646 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:07.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.785+0000 7f50a7e6b640 1 Processor -- start 2023-10-16T20:42:07.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.789+0000 7f50a7e6b640 1 -- start start 2023-10-16T20:42:07.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.789+0000 7f50a7e6b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50a00c9bb0 0x7f50a00c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:07.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.789+0000 7f50a7e6b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a0153b50 con 0x7f50a0153770 2023-10-16T20:42:07.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.789+0000 7f50a7e6b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a00cb2f0 con 0x7f50a00c41f0 2023-10-16T20:42:07.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.789+0000 7f50a7e6b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a00cc250 con 0x7f50a00c9bb0 2023-10-16T20:42:07.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.789+0000 7f50a5c07640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f50a0153770 0x7f50a0159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59560/0 (socket says 172.21.15.88:59560) 2023-10-16T20:42:07.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.789+0000 7f50a5c07640 1 -- 172.21.15.88:0/3451833800 learned_addr learned my addr 172.21.15.88:0/3451833800 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:42:07.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.789+0000 7f50a5406640 1 --2- 172.21.15.88:0/3451833800 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50a00c9bb0 0x7f50a00c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:07.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.789+0000 7f50a4c05640 1 -- 172.21.15.88:0/3451833800 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966180713 0 0) 0x7f50a0153b50 con 0x7f50a0153770 2023-10-16T20:42:07.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.789+0000 7f50a4c05640 1 -- 172.21.15.88:0/3451833800 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f508c003530 con 0x7f50a0153770 2023-10-16T20:42:07.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.789+0000 7f50a4c05640 1 -- 172.21.15.88:0/3451833800 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343110642 0 0) 0x7f50a00cb2f0 con 0x7f50a00c41f0 2023-10-16T20:42:07.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.789+0000 7f50a4c05640 1 -- 172.21.15.88:0/3451833800 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50a0153b50 con 0x7f50a00c41f0 2023-10-16T20:42:07.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.789+0000 7f50a5406640 1 -- 172.21.15.88:0/3451833800 >> v1:172.21.15.191:6789/0 conn(0x7f50a00c41f0 legacy=0x7f50a00cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:07.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.789+0000 7f50a5406640 1 -- 172.21.15.88:0/3451833800 >> v1:172.21.15.88:6789/0 conn(0x7f50a0153770 legacy=0x7f50a0159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:07.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.789+0000 7f50a5406640 1 -- 172.21.15.88:0/3451833800 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a00cb5d0 con 0x7f50a00c9bb0 2023-10-16T20:42:07.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.793+0000 7f50a5406640 1 --2- 172.21.15.88:0/3451833800 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50a00c9bb0 0x7f50a00c3cb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f50900097f0 tx=0x7f509001c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49567d0274f9bc0c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:07.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.793+0000 7f50a4c05640 1 -- 172.21.15.88:0/3451833800 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50900ac630 con 0x7f50a00c9bb0 2023-10-16T20:42:07.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.793+0000 7f50a4c05640 1 -- 172.21.15.88:0/3451833800 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50900ac790 con 0x7f50a00c9bb0 2023-10-16T20:42:07.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.793+0000 7f50a4c05640 1 -- 172.21.15.88:0/3451833800 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50900acb20 con 0x7f50a00c9bb0 2023-10-16T20:42:07.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.793+0000 7f50a7e6b640 1 -- 172.21.15.88:0/3451833800 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50a00c9bb0 msgr2=0x7f50a00c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:07.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.793+0000 7f50a7e6b640 1 --2- 172.21.15.88:0/3451833800 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50a00c9bb0 0x7f50a00c3cb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f50900097f0 tx=0x7f509001c4c0 comp rx=0 tx=0).stop 2023-10-16T20:42:07.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.793+0000 7f50a7e6b640 1 -- 172.21.15.88:0/3451833800 shutdown_connections 2023-10-16T20:42:07.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.793+0000 7f50a7e6b640 1 --2- 172.21.15.88:0/3451833800 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50a00c9bb0 0x7f50a00c3cb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:07.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.793+0000 7f50a7e6b640 1 -- 172.21.15.88:0/3451833800 >> 172.21.15.88:0/3451833800 conn(0x7f50a00715c0 msgr2=0x7f50a00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:07.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.793+0000 7f50a7e6b640 1 -- 172.21.15.88:0/3451833800 shutdown_connections 2023-10-16T20:42:07.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.793+0000 7f50a7e6b640 1 -- 172.21.15.88:0/3451833800 wait complete. 2023-10-16T20:42:07.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a7e6b640 1 Processor -- start 2023-10-16T20:42:07.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a7e6b640 1 -- start start 2023-10-16T20:42:07.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a7e6b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50a00c41f0 0x7f50a01f2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:07.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a7e6b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50a00c9bb0 0x7f50a0149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:07.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a7e6b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50a0153770 0x7f50a01455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:07.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a7e6b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a01520b0 con 0x7f50a0153770 2023-10-16T20:42:07.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a7e6b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a01f15e0 con 0x7f50a00c41f0 2023-10-16T20:42:07.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a7e6b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a01f5600 con 0x7f50a00c9bb0 2023-10-16T20:42:07.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a6408640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50a0153770 0x7f50a01455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:07.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a6408640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50a0153770 0x7f50a01455e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38364/0 (socket says 172.21.15.88:38364) 2023-10-16T20:42:07.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a6408640 1 -- 172.21.15.88:0/198365490 learned_addr learned my addr 172.21.15.88:0/198365490 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:07.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a5c07640 1 --2- 172.21.15.88:0/198365490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50a00c41f0 0x7f50a01f2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:07.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a5406640 1 --2- 172.21.15.88:0/198365490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50a00c9bb0 0x7f50a0149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:07.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a6408640 1 -- 172.21.15.88:0/198365490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50a00c9bb0 msgr2=0x7f50a0149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:07.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a6408640 1 --2- 172.21.15.88:0/198365490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50a00c9bb0 0x7f50a0149970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:07.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a6408640 1 -- 172.21.15.88:0/198365490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50a00c41f0 msgr2=0x7f50a01f2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:07.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a6408640 1 --2- 172.21.15.88:0/198365490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50a00c41f0 0x7f50a01f2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:07.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a6408640 1 -- 172.21.15.88:0/198365490 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5090009420 con 0x7f50a0153770 2023-10-16T20:42:07.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a6408640 1 --2- 172.21.15.88:0/198365490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50a0153770 0x7f50a01455e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f509800c730 tx=0x7f509800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:07.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50967fc640 1 -- 172.21.15.88:0/198365490 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5098003b80 con 0x7f50a0153770 2023-10-16T20:42:07.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50967fc640 1 -- 172.21.15.88:0/198365490 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5098003ce0 con 0x7f50a0153770 2023-10-16T20:42:07.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50967fc640 1 -- 172.21.15.88:0/198365490 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509800aa90 con 0x7f50a0153770 2023-10-16T20:42:07.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a7e6b640 1 -- 172.21.15.88:0/198365490 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a01f58e0 con 0x7f50a0153770 2023-10-16T20:42:07.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a7e6b640 1 -- 172.21.15.88:0/198365490 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a01f1b80 con 0x7f50a0153770 2023-10-16T20:42:07.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.797+0000 7f50a7e6b640 1 -- 172.21.15.88:0/198365490 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a00c91f0 con 0x7f50a0153770 2023-10-16T20:42:07.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.801+0000 7f50967fc640 1 -- 172.21.15.88:0/198365490 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f509800abf0 con 0x7f50a0153770 2023-10-16T20:42:07.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.801+0000 7f50967fc640 1 --2- 172.21.15.88:0/198365490 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f50840664a0 0x7f5084068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:07.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.801+0000 7f50967fc640 1 -- 172.21.15.88:0/198365490 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f5098083fe0 con 0x7f50a0153770 2023-10-16T20:42:07.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.801+0000 7f50a5c07640 1 --2- 172.21.15.88:0/198365490 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f50840664a0 0x7f5084068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:07.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.801+0000 7f50a5c07640 1 --2- 172.21.15.88:0/198365490 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f50840664a0 0x7f5084068960 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f509c0010f0 tx=0x7f509c007bf0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:07.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.805+0000 7f50967fc640 1 -- 172.21.15.88:0/198365490 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5098051080 con 0x7f50a0153770 2023-10-16T20:42:07.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.985+0000 7f50a7e6b640 1 -- 172.21.15.88:0/198365490 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50a0074700 con 0x7f50a0153770 2023-10-16T20:42:07.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.985+0000 7f50967fc640 1 -- 172.21.15.88:0/198365490 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+6566 (secure 0 0 0) 0x7f50980546a0 con 0x7f50a0153770 2023-10-16T20:42:07.993 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:07.993 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:41:32.316596+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4250":{"gid":4250,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":44,"addr":"172.21.15.191:6837/1120825335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":1120825335},{"type":"v1","addr":"172.21.15.191:6837","nonce":1120825335}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:07.993 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 23 2023-10-16T20:42:07.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.985+0000 7f50a7e6b640 1 -- 172.21.15.88:0/198365490 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f50840664a0 msgr2=0x7f5084068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:07.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.985+0000 7f50a7e6b640 1 --2- 172.21.15.88:0/198365490 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f50840664a0 0x7f5084068960 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f509c0010f0 tx=0x7f509c007bf0 comp rx=0 tx=0).stop 2023-10-16T20:42:07.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.985+0000 7f50a7e6b640 1 -- 172.21.15.88:0/198365490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50a0153770 msgr2=0x7f50a01455e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:07.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.985+0000 7f50a7e6b640 1 --2- 172.21.15.88:0/198365490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50a0153770 0x7f50a01455e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f509800c730 tx=0x7f509800cbf0 comp rx=0 tx=0).stop 2023-10-16T20:42:07.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.985+0000 7f50a7e6b640 1 -- 172.21.15.88:0/198365490 shutdown_connections 2023-10-16T20:42:07.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.985+0000 7f50a7e6b640 1 --2- 172.21.15.88:0/198365490 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f50840664a0 0x7f5084068960 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:07.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.985+0000 7f50a7e6b640 1 --2- 172.21.15.88:0/198365490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50a0153770 0x7f50a01455e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:07.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.985+0000 7f50a7e6b640 1 --2- 172.21.15.88:0/198365490 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f50a00c9bb0 0x7f50a0149970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:07.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.985+0000 7f50a7e6b640 1 --2- 172.21.15.88:0/198365490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50a00c41f0 0x7f50a01f2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:07.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.985+0000 7f50a7e6b640 1 -- 172.21.15.88:0/198365490 >> 172.21.15.88:0/198365490 conn(0x7f50a00715c0 msgr2=0x7f50a00cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:07.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.985+0000 7f50a7e6b640 1 -- 172.21.15.88:0/198365490 shutdown_connections 2023-10-16T20:42:07.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:07.985+0000 7f50a7e6b640 1 -- 172.21.15.88:0/198365490 wait complete. 2023-10-16T20:42:08.013 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-16T20:42:10.014 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:10.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.153+0000 7f216ff80640 1 Processor -- start 2023-10-16T20:42:10.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.153+0000 7f216ff80640 1 -- start start 2023-10-16T20:42:10.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.157+0000 7f216ff80640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21680c9bb0 0x7f21680c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:10.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.157+0000 7f216ff80640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f21680c41f0 0x7f21680cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:10.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.157+0000 7f216ff80640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2168153b50 con 0x7f2168153770 2023-10-16T20:42:10.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.157+0000 7f216ff80640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21680cc250 con 0x7f21680c41f0 2023-10-16T20:42:10.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.157+0000 7f216ff80640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21680cc670 con 0x7f21680c9bb0 2023-10-16T20:42:10.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.157+0000 7f216dd1c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2168153770 0x7f2168159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36958/0 (socket says 172.21.15.88:36958) 2023-10-16T20:42:10.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.157+0000 7f216dd1c640 1 -- 172.21.15.88:0/2797328222 learned_addr learned my addr 172.21.15.88:0/2797328222 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:42:10.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.157+0000 7f216e51d640 1 --2- 172.21.15.88:0/2797328222 >> v2:172.21.15.88:3300/0 conn(0x7f21680c41f0 0x7f21680cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:10.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.157+0000 7f216d51b640 1 --2- 172.21.15.88:0/2797328222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21680c9bb0 0x7f21680c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:10.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.157+0000 7f216e51d640 1 -- 172.21.15.88:0/2797328222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21680c9bb0 msgr2=0x7f21680c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:10.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.157+0000 7f216cd1a640 1 -- 172.21.15.88:0/2797328222 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314924971 0 0) 0x7f2168153b50 con 0x7f2168153770 2023-10-16T20:42:10.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.157+0000 7f216e51d640 1 --2- 172.21.15.88:0/2797328222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21680c9bb0 0x7f21680c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.157+0000 7f216e51d640 1 -- 172.21.15.88:0/2797328222 >> v1:172.21.15.88:6789/0 conn(0x7f2168153770 legacy=0x7f2168159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:10.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.157+0000 7f216e51d640 1 -- 172.21.15.88:0/2797328222 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21680cb2f0 con 0x7f21680c41f0 2023-10-16T20:42:10.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.157+0000 7f216e51d640 1 --2- 172.21.15.88:0/2797328222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21680c41f0 0x7f21680cece0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2160009d90 tx=0x7f216001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a2c89f43b7f36d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:10.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216cd1a640 1 -- 172.21.15.88:0/2797328222 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21600a5920 con 0x7f21680c41f0 2023-10-16T20:42:10.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216cd1a640 1 -- 172.21.15.88:0/2797328222 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21600a5a80 con 0x7f21680c41f0 2023-10-16T20:42:10.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216cd1a640 1 -- 172.21.15.88:0/2797328222 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21600a5e10 con 0x7f21680c41f0 2023-10-16T20:42:10.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216ff80640 1 -- 172.21.15.88:0/2797328222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21680c41f0 msgr2=0x7f21680cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:10.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216ff80640 1 --2- 172.21.15.88:0/2797328222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21680c41f0 0x7f21680cece0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2160009d90 tx=0x7f216001c950 comp rx=0 tx=0).stop 2023-10-16T20:42:10.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216ff80640 1 -- 172.21.15.88:0/2797328222 shutdown_connections 2023-10-16T20:42:10.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216ff80640 1 --2- 172.21.15.88:0/2797328222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21680c41f0 0x7f21680cece0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216ff80640 1 --2- 172.21.15.88:0/2797328222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21680c9bb0 0x7f21680c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216ff80640 1 -- 172.21.15.88:0/2797328222 >> 172.21.15.88:0/2797328222 conn(0x7f21680715c0 msgr2=0x7f21680719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:10.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216ff80640 1 -- 172.21.15.88:0/2797328222 shutdown_connections 2023-10-16T20:42:10.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216ff80640 1 -- 172.21.15.88:0/2797328222 wait complete. 2023-10-16T20:42:10.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216ff80640 1 Processor -- start 2023-10-16T20:42:10.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216ff80640 1 -- start start 2023-10-16T20:42:10.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216ff80640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21680c41f0 0x7f216815d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:10.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216ff80640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21680c9bb0 0x7f216814bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:10.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216ff80640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2168153770 0x7f216815c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:10.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216ff80640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21680c8960 con 0x7f2168153770 2023-10-16T20:42:10.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216ff80640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21680c9120 con 0x7f21680c41f0 2023-10-16T20:42:10.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216ff80640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21681f2450 con 0x7f21680c9bb0 2023-10-16T20:42:10.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216e51d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2168153770 0x7f216815c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:10.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.161+0000 7f216e51d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2168153770 0x7f216815c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57092/0 (socket says 172.21.15.88:57092) 2023-10-16T20:42:10.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f216e51d640 1 -- 172.21.15.88:0/435409763 learned_addr learned my addr 172.21.15.88:0/435409763 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:10.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f216dd1c640 1 --2- 172.21.15.88:0/435409763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21680c41f0 0x7f216815d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:10.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f216d51b640 1 --2- 172.21.15.88:0/435409763 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21680c9bb0 0x7f216814bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:10.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f216e51d640 1 -- 172.21.15.88:0/435409763 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21680c9bb0 msgr2=0x7f216814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:10.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f216e51d640 1 --2- 172.21.15.88:0/435409763 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21680c9bb0 0x7f216814bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f216e51d640 1 -- 172.21.15.88:0/435409763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21680c41f0 msgr2=0x7f216815d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:10.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f216e51d640 1 --2- 172.21.15.88:0/435409763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21680c41f0 0x7f216815d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f216e51d640 1 -- 172.21.15.88:0/435409763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2160009a70 con 0x7f2168153770 2023-10-16T20:42:10.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f216e51d640 1 --2- 172.21.15.88:0/435409763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2168153770 0x7f216815c3f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f216001b000 tx=0x7f21600196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:10.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f215e7fc640 1 -- 172.21.15.88:0/435409763 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21600ad850 con 0x7f2168153770 2023-10-16T20:42:10.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f215e7fc640 1 -- 172.21.15.88:0/435409763 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21600ad9b0 con 0x7f2168153770 2023-10-16T20:42:10.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f215e7fc640 1 -- 172.21.15.88:0/435409763 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21600add20 con 0x7f2168153770 2023-10-16T20:42:10.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f216ff80640 1 -- 172.21.15.88:0/435409763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21681f2730 con 0x7f2168153770 2023-10-16T20:42:10.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f216ff80640 1 -- 172.21.15.88:0/435409763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216815bbf0 con 0x7f2168153770 2023-10-16T20:42:10.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f216ff80640 1 -- 172.21.15.88:0/435409763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2168074700 con 0x7f2168153770 2023-10-16T20:42:10.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f215e7fc640 1 -- 172.21.15.88:0/435409763 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f21600b4050 con 0x7f2168153770 2023-10-16T20:42:10.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f215e7fc640 1 --2- 172.21.15.88:0/435409763 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f214c0664d0 0x7f214c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:10.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.165+0000 7f215e7fc640 1 -- 172.21.15.88:0/435409763 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f216011ed80 con 0x7f2168153770 2023-10-16T20:42:10.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.169+0000 7f216dd1c640 1 --2- 172.21.15.88:0/435409763 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f214c0664d0 0x7f214c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:10.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.169+0000 7f216dd1c640 1 --2- 172.21.15.88:0/435409763 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f214c0664d0 0x7f214c068990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f21640010f0 tx=0x7f2164009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:10.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.169+0000 7f215e7fc640 1 -- 172.21.15.88:0/435409763 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f21600ebec0 con 0x7f2168153770 2023-10-16T20:42:10.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.361+0000 7f216ff80640 1 -- 172.21.15.88:0/435409763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21680ca740 con 0x7f2168153770 2023-10-16T20:42:10.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.361+0000 7f215e7fc640 1 -- 172.21.15.88:0/435409763 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f21600ef4e0 con 0x7f2168153770 2023-10-16T20:42:10.371 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:10.371 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:10.335736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:10.371 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 24 2023-10-16T20:42:10.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.365+0000 7f216ff80640 1 -- 172.21.15.88:0/435409763 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f214c0664d0 msgr2=0x7f214c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:10.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.365+0000 7f216ff80640 1 --2- 172.21.15.88:0/435409763 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f214c0664d0 0x7f214c068990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f21640010f0 tx=0x7f2164009040 comp rx=0 tx=0).stop 2023-10-16T20:42:10.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.365+0000 7f216ff80640 1 -- 172.21.15.88:0/435409763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2168153770 msgr2=0x7f216815c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:10.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.365+0000 7f216ff80640 1 --2- 172.21.15.88:0/435409763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2168153770 0x7f216815c3f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f216001b000 tx=0x7f21600196f0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.365+0000 7f216ff80640 1 -- 172.21.15.88:0/435409763 shutdown_connections 2023-10-16T20:42:10.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.365+0000 7f216ff80640 1 --2- 172.21.15.88:0/435409763 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f214c0664d0 0x7f214c068990 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.365+0000 7f216ff80640 1 --2- 172.21.15.88:0/435409763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2168153770 0x7f216815c3f0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.365+0000 7f216ff80640 1 --2- 172.21.15.88:0/435409763 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f21680c9bb0 0x7f216814bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.365+0000 7f216ff80640 1 --2- 172.21.15.88:0/435409763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f21680c41f0 0x7f216815d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.365+0000 7f216ff80640 1 -- 172.21.15.88:0/435409763 >> 172.21.15.88:0/435409763 conn(0x7f21680715c0 msgr2=0x7f21680719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:10.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.365+0000 7f216ff80640 1 -- 172.21.15.88:0/435409763 shutdown_connections 2023-10-16T20:42:10.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.365+0000 7f216ff80640 1 -- 172.21.15.88:0/435409763 wait complete. 2023-10-16T20:42:10.396 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-10-16T20:42:10.396 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-16T20:42:10.397 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:10.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.533+0000 7f394f980640 1 Processor -- start 2023-10-16T20:42:10.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.533+0000 7f394f980640 1 -- start start 2023-10-16T20:42:10.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.533+0000 7f394f980640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f39480c5300 0x7f394813e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:10.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.533+0000 7f394f980640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f39480cb890 0x7f394806ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:10.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.533+0000 7f394f980640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394815a590 con 0x7f394815d840 2023-10-16T20:42:10.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.533+0000 7f394f980640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f394815d420 con 0x7f39480cb890 2023-10-16T20:42:10.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.533+0000 7f394f980640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39480c7830 con 0x7f39480c5300 2023-10-16T20:42:10.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.533+0000 7f394df1d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f39480cb890 0x7f394806ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:10.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.533+0000 7f394df1d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f39480cb890 0x7f394806ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57096/0 (socket says 172.21.15.88:57096) 2023-10-16T20:42:10.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.533+0000 7f394df1d640 1 -- 172.21.15.88:0/543679012 learned_addr learned my addr 172.21.15.88:0/543679012 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.537+0000 7f394df1d640 1 -- 172.21.15.88:0/543679012 >> v2:172.21.15.191:3300/0 conn(0x7f39480c5300 msgr2=0x7f394813e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.537+0000 7f394d71c640 1 --2- 172.21.15.88:0/543679012 >> v2:172.21.15.191:3300/0 conn(0x7f39480c5300 0x7f394813e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.537+0000 7f394df1d640 1 --2- 172.21.15.88:0/543679012 >> v2:172.21.15.191:3300/0 conn(0x7f39480c5300 0x7f394813e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.537+0000 7f394df1d640 1 -- 172.21.15.88:0/543679012 >> v1:172.21.15.191:6789/0 conn(0x7f394815d840 legacy=0x7f3948159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.537+0000 7f394df1d640 1 -- 172.21.15.88:0/543679012 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3948159d50 con 0x7f39480cb890 2023-10-16T20:42:10.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.537+0000 7f394d71c640 1 --2- 172.21.15.88:0/543679012 >> v2:172.21.15.191:3300/0 conn(0x7f39480c5300 0x7f394813e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:42:10.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.537+0000 7f394df1d640 1 --2- 172.21.15.88:0/543679012 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39480cb890 0x7f394806ffe0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f39480c8d20 tx=0x7f394001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f29ccc565de71f80 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:10.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.537+0000 7f393ffff640 1 -- 172.21.15.88:0/543679012 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39400a5710 con 0x7f39480cb890 2023-10-16T20:42:10.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.537+0000 7f393ffff640 1 -- 172.21.15.88:0/543679012 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39400a5870 con 0x7f39480cb890 2023-10-16T20:42:10.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.537+0000 7f393ffff640 1 -- 172.21.15.88:0/543679012 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39400a5c00 con 0x7f39480cb890 2023-10-16T20:42:10.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.537+0000 7f394f980640 1 -- 172.21.15.88:0/543679012 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39480cb890 msgr2=0x7f394806ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:10.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.537+0000 7f394f980640 1 --2- 172.21.15.88:0/543679012 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39480cb890 0x7f394806ffe0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f39480c8d20 tx=0x7f394001c940 comp rx=0 tx=0).stop 2023-10-16T20:42:10.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394f980640 1 -- 172.21.15.88:0/543679012 shutdown_connections 2023-10-16T20:42:10.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394f980640 1 --2- 172.21.15.88:0/543679012 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39480cb890 0x7f394806ffe0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394f980640 1 --2- 172.21.15.88:0/543679012 >> v2:172.21.15.191:3300/0 conn(0x7f39480c5300 0x7f394813e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394f980640 1 -- 172.21.15.88:0/543679012 >> 172.21.15.88:0/543679012 conn(0x7f394806e0f0 msgr2=0x7f39480cb180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394f980640 1 -- 172.21.15.88:0/543679012 shutdown_connections 2023-10-16T20:42:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394f980640 1 -- 172.21.15.88:0/543679012 wait complete. 2023-10-16T20:42:10.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394f980640 1 Processor -- start 2023-10-16T20:42:10.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394f980640 1 -- start start 2023-10-16T20:42:10.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394f980640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39480c5300 0x7f39480c8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394f980640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39480cb890 0x7f39480c30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394f980640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f394815d840 0x7f39480c3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394f980640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39480c4970 con 0x7f39480cb890 2023-10-16T20:42:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394f980640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394813e730 con 0x7f394815d840 2023-10-16T20:42:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394f980640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39480c4160 con 0x7f39480c5300 2023-10-16T20:42:10.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394cf1b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39480cb890 0x7f39480c30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:10.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394cf1b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39480cb890 0x7f39480c30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57104/0 (socket says 172.21.15.88:57104) 2023-10-16T20:42:10.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394cf1b640 1 -- 172.21.15.88:0/3123435147 learned_addr learned my addr 172.21.15.88:0/3123435147 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:10.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394df1d640 1 --2- 172.21.15.88:0/3123435147 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f394815d840 0x7f39480c3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:10.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394d71c640 1 --2- 172.21.15.88:0/3123435147 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39480c5300 0x7f39480c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:10.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394cf1b640 1 -- 172.21.15.88:0/3123435147 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39480c5300 msgr2=0x7f39480c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:10.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394cf1b640 1 --2- 172.21.15.88:0/3123435147 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39480c5300 0x7f39480c8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394cf1b640 1 -- 172.21.15.88:0/3123435147 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f394815d840 msgr2=0x7f39480c3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:10.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394cf1b640 1 --2- 172.21.15.88:0/3123435147 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f394815d840 0x7f39480c3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394cf1b640 1 -- 172.21.15.88:0/3123435147 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3940009b30 con 0x7f39480cb890 2023-10-16T20:42:10.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394d71c640 1 --2- 172.21.15.88:0/3123435147 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39480c5300 0x7f39480c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:42:10.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394cf1b640 1 --2- 172.21.15.88:0/3123435147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39480cb890 0x7f39480c30c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f393800c3c0 tx=0x7f393800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:10.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f393dffb640 1 -- 172.21.15.88:0/3123435147 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39380052d0 con 0x7f39480cb890 2023-10-16T20:42:10.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f393dffb640 1 -- 172.21.15.88:0/3123435147 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3938005430 con 0x7f39480cb890 2023-10-16T20:42:10.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f394f980640 1 -- 172.21.15.88:0/3123435147 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394813e960 con 0x7f39480cb890 2023-10-16T20:42:10.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.541+0000 7f393dffb640 1 -- 172.21.15.88:0/3123435147 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3938003840 con 0x7f39480cb890 2023-10-16T20:42:10.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.545+0000 7f394f980640 1 -- 172.21.15.88:0/3123435147 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39480c4690 con 0x7f39480cb890 2023-10-16T20:42:10.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.545+0000 7f394f980640 1 -- 172.21.15.88:0/3123435147 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3948074700 con 0x7f39480cb890 2023-10-16T20:42:10.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.545+0000 7f393dffb640 1 -- 172.21.15.88:0/3123435147 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f39380039a0 con 0x7f39480cb890 2023-10-16T20:42:10.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.545+0000 7f393dffb640 1 --2- 172.21.15.88:0/3123435147 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f392c0664a0 0x7f392c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:10.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.545+0000 7f393dffb640 1 -- 172.21.15.88:0/3123435147 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f39380837c0 con 0x7f39480cb890 2023-10-16T20:42:10.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.545+0000 7f394d71c640 1 --2- 172.21.15.88:0/3123435147 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f392c0664a0 0x7f392c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:10.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.545+0000 7f394d71c640 1 --2- 172.21.15.88:0/3123435147 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f392c0664a0 0x7f392c068960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f394400a9b0 tx=0x7f3944008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:10.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.549+0000 7f393dffb640 1 -- 172.21.15.88:0/3123435147 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3938050840 con 0x7f39480cb890 2023-10-16T20:42:10.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.729+0000 7f394f980640 1 -- 172.21.15.88:0/3123435147 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3948163da0 con 0x7f39480cb890 2023-10-16T20:42:10.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.729+0000 7f393dffb640 1 -- 172.21.15.88:0/3123435147 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f3938053e60 con 0x7f39480cb890 2023-10-16T20:42:10.739 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:10.740 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:10.335736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:10.741 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 24 2023-10-16T20:42:10.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.733+0000 7f394f980640 1 -- 172.21.15.88:0/3123435147 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f392c0664a0 msgr2=0x7f392c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:10.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.733+0000 7f394f980640 1 --2- 172.21.15.88:0/3123435147 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f392c0664a0 0x7f392c068960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f394400a9b0 tx=0x7f3944008040 comp rx=0 tx=0).stop 2023-10-16T20:42:10.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.733+0000 7f394f980640 1 -- 172.21.15.88:0/3123435147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39480cb890 msgr2=0x7f39480c30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:10.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.733+0000 7f394f980640 1 --2- 172.21.15.88:0/3123435147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39480cb890 0x7f39480c30c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f393800c3c0 tx=0x7f393800c880 comp rx=0 tx=0).stop 2023-10-16T20:42:10.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.733+0000 7f394f980640 1 -- 172.21.15.88:0/3123435147 shutdown_connections 2023-10-16T20:42:10.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.733+0000 7f394f980640 1 --2- 172.21.15.88:0/3123435147 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f392c0664a0 0x7f392c068960 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.733+0000 7f394f980640 1 --2- 172.21.15.88:0/3123435147 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f394815d840 0x7f39480c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.733+0000 7f394f980640 1 --2- 172.21.15.88:0/3123435147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39480cb890 0x7f39480c30c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.733+0000 7f394f980640 1 --2- 172.21.15.88:0/3123435147 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39480c5300 0x7f39480c8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:10.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.733+0000 7f394f980640 1 -- 172.21.15.88:0/3123435147 >> 172.21.15.88:0/3123435147 conn(0x7f394806e0f0 msgr2=0x7f3948138740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:10.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.733+0000 7f394f980640 1 -- 172.21.15.88:0/3123435147 shutdown_connections 2023-10-16T20:42:10.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:10.733+0000 7f394f980640 1 -- 172.21.15.88:0/3123435147 wait complete. 2023-10-16T20:42:10.762 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 24, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 24, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4244": { "addr": "172.21.15.88:6839/1987579860", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6837", "nonce": 1987579860, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1987579860, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 24, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:replay", "state_seq": 1 }, "gid_4263": { "addr": "172.21.15.88:6835/518281867", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 518281867, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 518281867, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4263, "incarnation": 16, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 43 }, "gid_4275": { "addr": "172.21.15.88:6838/321173984", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 321173984, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 321173984, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4275, "incarnation": 14, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 41 }, "gid_4279": { "addr": "172.21.15.191:6833/2446850309", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 2446850309, "type": "v2" }, { "addr": "172.21.15.191:6833", "nonce": 2446850309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 40 }, "gid_4288": { "addr": "172.21.15.191:6835/3059699574", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3059699574, "type": "v2" }, { "addr": "172.21.15.191:6835", "nonce": 3059699574, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:42:10.335736+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 4279, "mds_2": 4275, "mds_3": 4263, "mds_4": 4244 } } } ], "standbys": [] } 2023-10-16T20:42:12.763 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:12.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.901+0000 7fd7cf5b8640 1 Processor -- start 2023-10-16T20:42:12.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.901+0000 7fd7cf5b8640 1 -- start start 2023-10-16T20:42:12.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.905+0000 7fd7cf5b8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7c81406d0 0x7fd7c80c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:12.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.905+0000 7fd7cf5b8640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd7c806a970 0x7fd7c8071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:12.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.905+0000 7fd7cf5b8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c815b3c0 con 0x7fd7c8071e60 2023-10-16T20:42:12.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.905+0000 7fd7cf5b8640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c815b680 con 0x7fd7c806a970 2023-10-16T20:42:12.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.905+0000 7fd7cf5b8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c8157570 con 0x7fd7c81406d0 2023-10-16T20:42:12.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.905+0000 7fd7cdb55640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd7c8071e60 0x7fd7c815a810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36968/0 (socket says 172.21.15.88:36968) 2023-10-16T20:42:12.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.905+0000 7fd7cdb55640 1 -- 172.21.15.88:0/1203503988 learned_addr learned my addr 172.21.15.88:0/1203503988 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:42:12.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.905+0000 7fd7cd354640 1 --2- 172.21.15.88:0/1203503988 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7c81406d0 0x7fd7c80c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:12.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.905+0000 7fd7ccb53640 1 --2- 172.21.15.88:0/1203503988 >> v2:172.21.15.191:3300/0 conn(0x7fd7c806a970 0x7fd7c8071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:12.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.905+0000 7fd7bffff640 1 -- 172.21.15.88:0/1203503988 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187355862 0 0) 0x7fd7c815b3c0 con 0x7fd7c8071e60 2023-10-16T20:42:12.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.905+0000 7fd7bffff640 1 -- 172.21.15.88:0/1203503988 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7b4003530 con 0x7fd7c8071e60 2023-10-16T20:42:12.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.905+0000 7fd7ccb53640 1 -- 172.21.15.88:0/1203503988 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7c81406d0 msgr2=0x7fd7c80c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:12.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.905+0000 7fd7ccb53640 1 --2- 172.21.15.88:0/1203503988 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7c81406d0 0x7fd7c80c1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:12.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.905+0000 7fd7ccb53640 1 -- 172.21.15.88:0/1203503988 >> v1:172.21.15.88:6789/0 conn(0x7fd7c8071e60 legacy=0x7fd7c815a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:12.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.905+0000 7fd7ccb53640 1 -- 172.21.15.88:0/1203503988 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c8157850 con 0x7fd7c806a970 2023-10-16T20:42:12.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.905+0000 7fd7cd354640 1 --2- 172.21.15.88:0/1203503988 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7c81406d0 0x7fd7c80c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:42:12.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.909+0000 7fd7ccb53640 1 --2- 172.21.15.88:0/1203503988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7c806a970 0x7fd7c8071920 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd7b80097c0 tx=0x7fd7b801c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f90ac630809d5d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:12.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.909+0000 7fd7bffff640 1 -- 172.21.15.88:0/1203503988 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b80ad660 con 0x7fd7c806a970 2023-10-16T20:42:12.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.909+0000 7fd7bffff640 1 -- 172.21.15.88:0/1203503988 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b80ad7c0 con 0x7fd7c806a970 2023-10-16T20:42:12.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.909+0000 7fd7bffff640 1 -- 172.21.15.88:0/1203503988 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b80adb50 con 0x7fd7c806a970 2023-10-16T20:42:12.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.909+0000 7fd7cf5b8640 1 -- 172.21.15.88:0/1203503988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7c806a970 msgr2=0x7fd7c8071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:12.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.909+0000 7fd7cf5b8640 1 --2- 172.21.15.88:0/1203503988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7c806a970 0x7fd7c8071920 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd7b80097c0 tx=0x7fd7b801c450 comp rx=0 tx=0).stop 2023-10-16T20:42:12.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.909+0000 7fd7cf5b8640 1 -- 172.21.15.88:0/1203503988 shutdown_connections 2023-10-16T20:42:12.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.909+0000 7fd7cf5b8640 1 --2- 172.21.15.88:0/1203503988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7c806a970 0x7fd7c8071920 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:12.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.909+0000 7fd7cf5b8640 1 --2- 172.21.15.88:0/1203503988 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7c81406d0 0x7fd7c80c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:12.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.909+0000 7fd7cf5b8640 1 -- 172.21.15.88:0/1203503988 >> 172.21.15.88:0/1203503988 conn(0x7fd7c80cad50 msgr2=0x7fd7c80cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:12.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.909+0000 7fd7cf5b8640 1 -- 172.21.15.88:0/1203503988 shutdown_connections 2023-10-16T20:42:12.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.909+0000 7fd7cf5b8640 1 -- 172.21.15.88:0/1203503988 wait complete. 2023-10-16T20:42:12.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.909+0000 7fd7cf5b8640 1 Processor -- start 2023-10-16T20:42:12.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cf5b8640 1 -- start start 2023-10-16T20:42:12.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cf5b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7c806a970 0x7fd7c8154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:12.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cf5b8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7c8071e60 0x7fd7c8151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:12.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cf5b8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7c81406d0 0x7fd7c813b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:12.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cf5b8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c81f0530 con 0x7fd7c806a970 2023-10-16T20:42:12.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cf5b8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c8154860 con 0x7fd7c8071e60 2023-10-16T20:42:12.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cf5b8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c8139a00 con 0x7fd7c81406d0 2023-10-16T20:42:12.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cd354640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7c806a970 0x7fd7c8154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:12.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cd354640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7c806a970 0x7fd7c8154320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57112/0 (socket says 172.21.15.88:57112) 2023-10-16T20:42:12.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cd354640 1 -- 172.21.15.88:0/701175207 learned_addr learned my addr 172.21.15.88:0/701175207 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:12.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cdb55640 1 --2- 172.21.15.88:0/701175207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7c81406d0 0x7fd7c813b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:12.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7ccb53640 1 --2- 172.21.15.88:0/701175207 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7c8071e60 0x7fd7c8151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:12.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cd354640 1 -- 172.21.15.88:0/701175207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7c81406d0 msgr2=0x7fd7c813b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:12.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cd354640 1 --2- 172.21.15.88:0/701175207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7c81406d0 0x7fd7c813b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:12.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cd354640 1 -- 172.21.15.88:0/701175207 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7c8071e60 msgr2=0x7fd7c8151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:12.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cd354640 1 --2- 172.21.15.88:0/701175207 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7c8071e60 0x7fd7c8151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:12.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cd354640 1 -- 172.21.15.88:0/701175207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c4009420 con 0x7fd7c806a970 2023-10-16T20:42:12.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cdb55640 1 --2- 172.21.15.88:0/701175207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7c81406d0 0x7fd7c813b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:42:12.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cd354640 1 --2- 172.21.15.88:0/701175207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7c806a970 0x7fd7c8154320 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd7c400ebe0 tx=0x7fd7c400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:12.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7bdffb640 1 -- 172.21.15.88:0/701175207 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c400eee0 con 0x7fd7c806a970 2023-10-16T20:42:12.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7bdffb640 1 -- 172.21.15.88:0/701175207 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c400cc30 con 0x7fd7c806a970 2023-10-16T20:42:12.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cf5b8640 1 -- 172.21.15.88:0/701175207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b8009420 con 0x7fd7c806a970 2023-10-16T20:42:12.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7bdffb640 1 -- 172.21.15.88:0/701175207 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c4013470 con 0x7fd7c806a970 2023-10-16T20:42:12.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cf5b8640 1 -- 172.21.15.88:0/701175207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c81581d0 con 0x7fd7c806a970 2023-10-16T20:42:12.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.913+0000 7fd7cf5b8640 1 -- 172.21.15.88:0/701175207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c8074700 con 0x7fd7c806a970 2023-10-16T20:42:12.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.917+0000 7fd7bdffb640 1 -- 172.21.15.88:0/701175207 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd7c40135d0 con 0x7fd7c806a970 2023-10-16T20:42:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.917+0000 7fd7bdffb640 1 --2- 172.21.15.88:0/701175207 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd7ac0664a0 0x7fd7ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.917+0000 7fd7bdffb640 1 -- 172.21.15.88:0/701175207 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7fd7c40848f0 con 0x7fd7c806a970 2023-10-16T20:42:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.917+0000 7fd7ccb53640 1 --2- 172.21.15.88:0/701175207 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd7ac0664a0 0x7fd7ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.917+0000 7fd7ccb53640 1 --2- 172.21.15.88:0/701175207 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd7ac0664a0 0x7fd7ac068960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd7b80af000 tx=0x7fd7b8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:12.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:12.921+0000 7fd7bdffb640 1 -- 172.21.15.88:0/701175207 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd7c4051970 con 0x7fd7c806a970 2023-10-16T20:42:13.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:13.101+0000 7fd7cf5b8640 1 -- 172.21.15.88:0/701175207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7c815ad90 con 0x7fd7c806a970 2023-10-16T20:42:13.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:13.101+0000 7fd7bdffb640 1 -- 172.21.15.88:0/701175207 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fd7c4054f90 con 0x7fd7c806a970 2023-10-16T20:42:13.110 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:13.110 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:10.335736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:13.111 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 24 2023-10-16T20:42:13.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:13.105+0000 7fd7cf5b8640 1 -- 172.21.15.88:0/701175207 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd7ac0664a0 msgr2=0x7fd7ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:13.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:13.105+0000 7fd7cf5b8640 1 --2- 172.21.15.88:0/701175207 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd7ac0664a0 0x7fd7ac068960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd7b80af000 tx=0x7fd7b8008040 comp rx=0 tx=0).stop 2023-10-16T20:42:13.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:13.105+0000 7fd7cf5b8640 1 -- 172.21.15.88:0/701175207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7c806a970 msgr2=0x7fd7c8154320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:13.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:13.105+0000 7fd7cf5b8640 1 --2- 172.21.15.88:0/701175207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7c806a970 0x7fd7c8154320 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd7c400ebe0 tx=0x7fd7c400c520 comp rx=0 tx=0).stop 2023-10-16T20:42:13.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:13.105+0000 7fd7cf5b8640 1 -- 172.21.15.88:0/701175207 shutdown_connections 2023-10-16T20:42:13.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:13.105+0000 7fd7cf5b8640 1 --2- 172.21.15.88:0/701175207 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd7ac0664a0 0x7fd7ac068960 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:13.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:13.105+0000 7fd7cf5b8640 1 --2- 172.21.15.88:0/701175207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7c81406d0 0x7fd7c813b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:13.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:13.105+0000 7fd7cf5b8640 1 --2- 172.21.15.88:0/701175207 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7c8071e60 0x7fd7c8151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:13.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:13.105+0000 7fd7cf5b8640 1 --2- 172.21.15.88:0/701175207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7c806a970 0x7fd7c8154320 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:13.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:13.105+0000 7fd7cf5b8640 1 -- 172.21.15.88:0/701175207 >> 172.21.15.88:0/701175207 conn(0x7fd7c80cad50 msgr2=0x7fd7c813fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:13.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:13.105+0000 7fd7cf5b8640 1 -- 172.21.15.88:0/701175207 shutdown_connections 2023-10-16T20:42:13.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:13.105+0000 7fd7cf5b8640 1 -- 172.21.15.88:0/701175207 wait complete. 2023-10-16T20:42:13.131 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:42:15.132 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:15.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.277+0000 7f040f1c4640 1 Processor -- start 2023-10-16T20:42:15.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.277+0000 7f040f1c4640 1 -- start start 2023-10-16T20:42:15.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.277+0000 7f040f1c4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0408150280 0x7f04080c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:15.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.277+0000 7f040f1c4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0408150660 0x7f04080c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:15.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.277+0000 7f040f1c4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04080c5c60 con 0x7f04080c6bc0 2023-10-16T20:42:15.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.277+0000 7f040f1c4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04080c5370 con 0x7f0408150660 2023-10-16T20:42:15.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.277+0000 7f040f1c4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0408156d10 con 0x7f0408150280 2023-10-16T20:42:15.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.277+0000 7f040d761640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f04080c6bc0 0x7f040806d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36970/0 (socket says 172.21.15.88:36970) 2023-10-16T20:42:15.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.277+0000 7f040d761640 1 -- 172.21.15.88:0/2606937284 learned_addr learned my addr 172.21.15.88:0/2606937284 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:42:15.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.277+0000 7f040cf60640 1 --2- 172.21.15.88:0/2606937284 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0408150280 0x7f04080c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:15.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.277+0000 7f0407fff640 1 --2- 172.21.15.88:0/2606937284 >> v2:172.21.15.191:3300/0 conn(0x7f0408150660 0x7f04080c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:15.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.277+0000 7f04077fe640 1 -- 172.21.15.88:0/2606937284 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788861976 0 0) 0x7f04080c5c60 con 0x7f04080c6bc0 2023-10-16T20:42:15.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.277+0000 7f04077fe640 1 -- 172.21.15.88:0/2606937284 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03f8003530 con 0x7f04080c6bc0 2023-10-16T20:42:15.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.277+0000 7f0407fff640 1 -- 172.21.15.88:0/2606937284 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0408150280 msgr2=0x7f04080c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:15.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.277+0000 7f04077fe640 1 -- 172.21.15.88:0/2606937284 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 456622297 0 0) 0x7f03f8003530 con 0x7f04080c6bc0 2023-10-16T20:42:15.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.277+0000 7f0407fff640 1 --2- 172.21.15.88:0/2606937284 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0408150280 0x7f04080c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:15.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.281+0000 7f0407fff640 1 -- 172.21.15.88:0/2606937284 >> v1:172.21.15.88:6789/0 conn(0x7f04080c6bc0 legacy=0x7f040806d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:15.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.281+0000 7f0407fff640 1 -- 172.21.15.88:0/2606937284 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04080c5610 con 0x7f0408150660 2023-10-16T20:42:15.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.281+0000 7f040cf60640 1 --2- 172.21.15.88:0/2606937284 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0408150280 0x7f04080c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:42:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.281+0000 7f0407fff640 1 --2- 172.21.15.88:0/2606937284 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0408150660 0x7f04080c3110 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f03f40097c0 tx=0x7f03f401cd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa8671d9b7234a21 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.281+0000 7f04077fe640 1 -- 172.21.15.88:0/2606937284 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0400001a00 con 0x7f0408150660 2023-10-16T20:42:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.281+0000 7f04077fe640 1 -- 172.21.15.88:0/2606937284 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f40a7050 con 0x7f0408150660 2023-10-16T20:42:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.281+0000 7f04077fe640 1 -- 172.21.15.88:0/2606937284 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f40ab600 con 0x7f0408150660 2023-10-16T20:42:15.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040f1c4640 1 -- 172.21.15.88:0/2606937284 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0408150660 msgr2=0x7f04080c3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:15.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040f1c4640 1 --2- 172.21.15.88:0/2606937284 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0408150660 0x7f04080c3110 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f03f40097c0 tx=0x7f03f401cd40 comp rx=0 tx=0).stop 2023-10-16T20:42:15.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040f1c4640 1 -- 172.21.15.88:0/2606937284 shutdown_connections 2023-10-16T20:42:15.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040f1c4640 1 --2- 172.21.15.88:0/2606937284 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0408150660 0x7f04080c3110 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:15.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040f1c4640 1 --2- 172.21.15.88:0/2606937284 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0408150280 0x7f04080c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:15.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040f1c4640 1 -- 172.21.15.88:0/2606937284 >> 172.21.15.88:0/2606937284 conn(0x7f04080cad50 msgr2=0x7f04080cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:15.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040f1c4640 1 -- 172.21.15.88:0/2606937284 shutdown_connections 2023-10-16T20:42:15.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040f1c4640 1 -- 172.21.15.88:0/2606937284 wait complete. 2023-10-16T20:42:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040f1c4640 1 Processor -- start 2023-10-16T20:42:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040f1c4640 1 -- start start 2023-10-16T20:42:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040f1c4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04080c6bc0 0x7f0408151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040f1c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0408150280 0x7f04080c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040f1c4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0408150660 0x7f04081f4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040f1c4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04081f2300 con 0x7f0408150280 2023-10-16T20:42:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040f1c4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0408151770 con 0x7f0408150660 2023-10-16T20:42:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040f1c4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04081f1780 con 0x7f04080c6bc0 2023-10-16T20:42:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f0407fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0408150280 0x7f04080c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f0407fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0408150280 0x7f04080c9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57126/0 (socket says 172.21.15.88:57126) 2023-10-16T20:42:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f0407fff640 1 -- 172.21.15.88:0/4097085700 learned_addr learned my addr 172.21.15.88:0/4097085700 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040cf60640 1 --2- 172.21.15.88:0/4097085700 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04080c6bc0 0x7f0408151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.285+0000 7f040d761640 1 --2- 172.21.15.88:0/4097085700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0408150660 0x7f04081f4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.289+0000 7f0407fff640 1 -- 172.21.15.88:0/4097085700 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04080c6bc0 msgr2=0x7f0408151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.289+0000 7f0407fff640 1 --2- 172.21.15.88:0/4097085700 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04080c6bc0 0x7f0408151230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.289+0000 7f0407fff640 1 -- 172.21.15.88:0/4097085700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0408150660 msgr2=0x7f04081f4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.289+0000 7f0407fff640 1 --2- 172.21.15.88:0/4097085700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0408150660 0x7f04081f4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.289+0000 7f0407fff640 1 -- 172.21.15.88:0/4097085700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03fc00b6c0 con 0x7f0408150280 2023-10-16T20:42:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.289+0000 7f040d761640 1 --2- 172.21.15.88:0/4097085700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0408150660 0x7f04081f4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:42:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.289+0000 7f0407fff640 1 --2- 172.21.15.88:0/4097085700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0408150280 0x7f04080c9f40 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f03f4000c10 tx=0x7f03f4019080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.289+0000 7f04057fa640 1 -- 172.21.15.88:0/4097085700 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f4003a30 con 0x7f0408150280 2023-10-16T20:42:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.289+0000 7f040f1c4640 1 -- 172.21.15.88:0/4097085700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f4009420 con 0x7f0408150280 2023-10-16T20:42:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.289+0000 7f04057fa640 1 -- 172.21.15.88:0/4097085700 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f4003b90 con 0x7f0408150280 2023-10-16T20:42:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.289+0000 7f04057fa640 1 -- 172.21.15.88:0/4097085700 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f40b5410 con 0x7f0408150280 2023-10-16T20:42:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.289+0000 7f040f1c4640 1 -- 172.21.15.88:0/4097085700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04081f1d40 con 0x7f0408150280 2023-10-16T20:42:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.289+0000 7f040f1c4640 1 -- 172.21.15.88:0/4097085700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0408074700 con 0x7f0408150280 2023-10-16T20:42:15.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.289+0000 7f04057fa640 1 -- 172.21.15.88:0/4097085700 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f03f40b2070 con 0x7f0408150280 2023-10-16T20:42:15.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.289+0000 7f04057fa640 1 --2- 172.21.15.88:0/4097085700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f03ec0664a0 0x7f03ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:15.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.289+0000 7f04057fa640 1 -- 172.21.15.88:0/4097085700 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f03f411e550 con 0x7f0408150280 2023-10-16T20:42:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.289+0000 7f040cf60640 1 --2- 172.21.15.88:0/4097085700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f03ec0664a0 0x7f03ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.293+0000 7f040cf60640 1 --2- 172.21.15.88:0/4097085700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f03ec0664a0 0x7f03ec068960 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00be70 tx=0x7f03fc00b3a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.293+0000 7f04057fa640 1 -- 172.21.15.88:0/4097085700 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f03f40eb670 con 0x7f0408150280 2023-10-16T20:42:15.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.481+0000 7f040f1c4640 1 -- 172.21.15.88:0/4097085700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04080c0f50 con 0x7f0408150280 2023-10-16T20:42:15.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.481+0000 7f04057fa640 1 -- 172.21.15.88:0/4097085700 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f03f40eec90 con 0x7f0408150280 2023-10-16T20:42:15.491 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:15.491 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:10.335736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:15.493 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 24 2023-10-16T20:42:15.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.485+0000 7f040f1c4640 1 -- 172.21.15.88:0/4097085700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f03ec0664a0 msgr2=0x7f03ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:15.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.485+0000 7f040f1c4640 1 --2- 172.21.15.88:0/4097085700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f03ec0664a0 0x7f03ec068960 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00be70 tx=0x7f03fc00b3a0 comp rx=0 tx=0).stop 2023-10-16T20:42:15.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.485+0000 7f040f1c4640 1 -- 172.21.15.88:0/4097085700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0408150280 msgr2=0x7f04080c9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:15.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.485+0000 7f040f1c4640 1 --2- 172.21.15.88:0/4097085700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0408150280 0x7f04080c9f40 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f03f4000c10 tx=0x7f03f4019080 comp rx=0 tx=0).stop 2023-10-16T20:42:15.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.485+0000 7f040f1c4640 1 -- 172.21.15.88:0/4097085700 shutdown_connections 2023-10-16T20:42:15.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.485+0000 7f040f1c4640 1 --2- 172.21.15.88:0/4097085700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f03ec0664a0 0x7f03ec068960 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:15.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.485+0000 7f040f1c4640 1 --2- 172.21.15.88:0/4097085700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0408150660 0x7f04081f4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:15.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.485+0000 7f040f1c4640 1 --2- 172.21.15.88:0/4097085700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0408150280 0x7f04080c9f40 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:15.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.485+0000 7f040f1c4640 1 --2- 172.21.15.88:0/4097085700 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f04080c6bc0 0x7f0408151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:15.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.485+0000 7f040f1c4640 1 -- 172.21.15.88:0/4097085700 >> 172.21.15.88:0/4097085700 conn(0x7f04080cad50 msgr2=0x7f040806a220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:15.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.485+0000 7f040f1c4640 1 -- 172.21.15.88:0/4097085700 shutdown_connections 2023-10-16T20:42:15.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:15.485+0000 7f040f1c4640 1 -- 172.21.15.88:0/4097085700 wait complete. 2023-10-16T20:42:15.512 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:42:17.513 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:17.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.653+0000 7f3a66d2a640 1 Processor -- start 2023-10-16T20:42:17.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.653+0000 7f3a66d2a640 1 -- start start 2023-10-16T20:42:17.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.653+0000 7f3a66d2a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3a60153770 0x7f3a60159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:17.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.653+0000 7f3a66d2a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a6015a410 con 0x7f3a600cece0 2023-10-16T20:42:17.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.653+0000 7f3a66d2a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a60153b50 con 0x7f3a600c9bb0 2023-10-16T20:42:17.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.653+0000 7f3a66d2a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a600cc250 con 0x7f3a60153770 2023-10-16T20:42:17.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.653+0000 7f3a652c7640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3a600cece0 0x7f3a60163dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36976/0 (socket says 172.21.15.88:36976) 2023-10-16T20:42:17.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.653+0000 7f3a652c7640 1 -- 172.21.15.88:0/2655304902 learned_addr learned my addr 172.21.15.88:0/2655304902 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:42:17.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.653+0000 7f3a64ac6640 1 --2- 172.21.15.88:0/2655304902 >> v2:172.21.15.191:3300/0 conn(0x7f3a60153770 0x7f3a60159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:17.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.653+0000 7f3a5f7fe640 1 -- 172.21.15.88:0/2655304902 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940290595 0 0) 0x7f3a6015a410 con 0x7f3a600cece0 2023-10-16T20:42:17.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.653+0000 7f3a5f7fe640 1 -- 172.21.15.88:0/2655304902 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a50003530 con 0x7f3a600cece0 2023-10-16T20:42:17.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.653+0000 7f3a5f7fe640 1 -- 172.21.15.88:0/2655304902 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640928065 0 0) 0x7f3a60153b50 con 0x7f3a600c9bb0 2023-10-16T20:42:17.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.653+0000 7f3a5f7fe640 1 -- 172.21.15.88:0/2655304902 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a6015a410 con 0x7f3a600c9bb0 2023-10-16T20:42:17.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.653+0000 7f3a64ac6640 1 -- 172.21.15.88:0/2655304902 >> v1:172.21.15.191:6789/0 conn(0x7f3a600c9bb0 legacy=0x7f3a600c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:17.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.653+0000 7f3a64ac6640 1 -- 172.21.15.88:0/2655304902 >> v1:172.21.15.88:6789/0 conn(0x7f3a600cece0 legacy=0x7f3a60163dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:17.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.653+0000 7f3a64ac6640 1 -- 172.21.15.88:0/2655304902 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a600cb2f0 con 0x7f3a60153770 2023-10-16T20:42:17.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.657+0000 7f3a64ac6640 1 --2- 172.21.15.88:0/2655304902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a60153770 0x7f3a60159ed0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3a540097a0 tx=0x7f3a5401c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78c26fb5675fef14 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:17.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.657+0000 7f3a5f7fe640 1 -- 172.21.15.88:0/2655304902 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a540ab600 con 0x7f3a60153770 2023-10-16T20:42:17.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.657+0000 7f3a5f7fe640 1 -- 172.21.15.88:0/2655304902 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a540a7050 con 0x7f3a60153770 2023-10-16T20:42:17.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.657+0000 7f3a5f7fe640 1 -- 172.21.15.88:0/2655304902 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a540ab980 con 0x7f3a60153770 2023-10-16T20:42:17.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.657+0000 7f3a66d2a640 1 -- 172.21.15.88:0/2655304902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a60153770 msgr2=0x7f3a60159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:17.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.657+0000 7f3a66d2a640 1 --2- 172.21.15.88:0/2655304902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a60153770 0x7f3a60159ed0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3a540097a0 tx=0x7f3a5401c460 comp rx=0 tx=0).stop 2023-10-16T20:42:17.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.657+0000 7f3a66d2a640 1 -- 172.21.15.88:0/2655304902 shutdown_connections 2023-10-16T20:42:17.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.657+0000 7f3a66d2a640 1 --2- 172.21.15.88:0/2655304902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a60153770 0x7f3a60159ed0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:17.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.657+0000 7f3a66d2a640 1 -- 172.21.15.88:0/2655304902 >> 172.21.15.88:0/2655304902 conn(0x7f3a600715c0 msgr2=0x7f3a600719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:17.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.657+0000 7f3a66d2a640 1 -- 172.21.15.88:0/2655304902 shutdown_connections 2023-10-16T20:42:17.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.657+0000 7f3a66d2a640 1 -- 172.21.15.88:0/2655304902 wait complete. 2023-10-16T20:42:17.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a66d2a640 1 Processor -- start 2023-10-16T20:42:17.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a66d2a640 1 -- start start 2023-10-16T20:42:17.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a66d2a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a600c9bb0 0x7f3a601482f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:17.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a66d2a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a600cece0 0x7f3a601494e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:17.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a66d2a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a60153770 0x7f3a6013fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:17.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a66d2a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6013d3b0 con 0x7f3a600c9bb0 2023-10-16T20:42:17.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a66d2a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6013e1d0 con 0x7f3a60153770 2023-10-16T20:42:17.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a66d2a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6013d5c0 con 0x7f3a600cece0 2023-10-16T20:42:17.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a64ac6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a600c9bb0 0x7f3a601482f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:17.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a64ac6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a600c9bb0 0x7f3a601482f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57136/0 (socket says 172.21.15.88:57136) 2023-10-16T20:42:17.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a64ac6640 1 -- 172.21.15.88:0/3629467677 learned_addr learned my addr 172.21.15.88:0/3629467677 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:17.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a5ffff640 1 --2- 172.21.15.88:0/3629467677 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a600cece0 0x7f3a601494e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:17.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a652c7640 1 --2- 172.21.15.88:0/3629467677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a60153770 0x7f3a6013fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:17.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a64ac6640 1 -- 172.21.15.88:0/3629467677 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a600cece0 msgr2=0x7f3a601494e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:17.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a64ac6640 1 --2- 172.21.15.88:0/3629467677 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a600cece0 0x7f3a601494e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:17.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a64ac6640 1 -- 172.21.15.88:0/3629467677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a60153770 msgr2=0x7f3a6013fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:17.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a64ac6640 1 --2- 172.21.15.88:0/3629467677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a60153770 0x7f3a6013fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:17.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a64ac6640 1 -- 172.21.15.88:0/3629467677 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a540093b0 con 0x7f3a600c9bb0 2023-10-16T20:42:17.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a64ac6640 1 --2- 172.21.15.88:0/3629467677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a600c9bb0 0x7f3a601482f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3a540ae040 tx=0x7f3a54019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:17.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a5d7fa640 1 -- 172.21.15.88:0/3629467677 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a540059a0 con 0x7f3a600c9bb0 2023-10-16T20:42:17.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a5d7fa640 1 -- 172.21.15.88:0/3629467677 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a54005b20 con 0x7f3a600c9bb0 2023-10-16T20:42:17.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a5d7fa640 1 -- 172.21.15.88:0/3629467677 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a54006020 con 0x7f3a600c9bb0 2023-10-16T20:42:17.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a66d2a640 1 -- 172.21.15.88:0/3629467677 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6013d8a0 con 0x7f3a600c9bb0 2023-10-16T20:42:17.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.661+0000 7f3a66d2a640 1 -- 172.21.15.88:0/3629467677 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6013de60 con 0x7f3a600c9bb0 2023-10-16T20:42:17.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.665+0000 7f3a66d2a640 1 -- 172.21.15.88:0/3629467677 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a600c5470 con 0x7f3a600c9bb0 2023-10-16T20:42:17.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.665+0000 7f3a5d7fa640 1 -- 172.21.15.88:0/3629467677 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3a540b2040 con 0x7f3a600c9bb0 2023-10-16T20:42:17.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.665+0000 7f3a5d7fa640 1 --2- 172.21.15.88:0/3629467677 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3a440664a0 0x7f3a44068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:17.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.665+0000 7f3a5d7fa640 1 -- 172.21.15.88:0/3629467677 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f3a5411d440 con 0x7f3a600c9bb0 2023-10-16T20:42:17.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.665+0000 7f3a5ffff640 1 --2- 172.21.15.88:0/3629467677 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3a440664a0 0x7f3a44068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:17.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.665+0000 7f3a5ffff640 1 --2- 172.21.15.88:0/3629467677 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3a440664a0 0x7f3a44068960 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3a600b4360 tx=0x7f3a4c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:17.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.669+0000 7f3a5d7fa640 1 -- 172.21.15.88:0/3629467677 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3a540ea4c0 con 0x7f3a600c9bb0 2023-10-16T20:42:17.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.853+0000 7f3a66d2a640 1 -- 172.21.15.88:0/3629467677 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a60074700 con 0x7f3a600c9bb0 2023-10-16T20:42:17.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.853+0000 7f3a5d7fa640 1 -- 172.21.15.88:0/3629467677 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5776 (secure 0 0 0) 0x7f3a540edae0 con 0x7f3a600c9bb0 2023-10-16T20:42:17.862 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:17.862 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:16.016135+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:resolve","state_seq":83,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:17.864 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 25 2023-10-16T20:42:17.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.857+0000 7f3a66d2a640 1 -- 172.21.15.88:0/3629467677 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3a440664a0 msgr2=0x7f3a44068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:17.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.857+0000 7f3a66d2a640 1 --2- 172.21.15.88:0/3629467677 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3a440664a0 0x7f3a44068960 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3a600b4360 tx=0x7f3a4c009040 comp rx=0 tx=0).stop 2023-10-16T20:42:17.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.857+0000 7f3a66d2a640 1 -- 172.21.15.88:0/3629467677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a600c9bb0 msgr2=0x7f3a601482f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:17.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.857+0000 7f3a66d2a640 1 --2- 172.21.15.88:0/3629467677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a600c9bb0 0x7f3a601482f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3a540ae040 tx=0x7f3a54019200 comp rx=0 tx=0).stop 2023-10-16T20:42:17.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.857+0000 7f3a66d2a640 1 -- 172.21.15.88:0/3629467677 shutdown_connections 2023-10-16T20:42:17.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.857+0000 7f3a66d2a640 1 --2- 172.21.15.88:0/3629467677 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3a440664a0 0x7f3a44068960 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:17.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.857+0000 7f3a66d2a640 1 --2- 172.21.15.88:0/3629467677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a60153770 0x7f3a6013fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:17.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.857+0000 7f3a66d2a640 1 --2- 172.21.15.88:0/3629467677 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a600cece0 0x7f3a601494e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:17.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.857+0000 7f3a66d2a640 1 --2- 172.21.15.88:0/3629467677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a600c9bb0 0x7f3a601482f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:17.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.857+0000 7f3a66d2a640 1 -- 172.21.15.88:0/3629467677 >> 172.21.15.88:0/3629467677 conn(0x7f3a600715c0 msgr2=0x7f3a600b4900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:17.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.857+0000 7f3a66d2a640 1 -- 172.21.15.88:0/3629467677 shutdown_connections 2023-10-16T20:42:17.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:17.857+0000 7f3a66d2a640 1 -- 172.21.15.88:0/3629467677 wait complete. 2023-10-16T20:42:17.882 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:42:19.884 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:20.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.025+0000 7f166bed6640 1 Processor -- start 2023-10-16T20:42:20.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.025+0000 7f166bed6640 1 -- start start 2023-10-16T20:42:20.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.029+0000 7f166bed6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1664157b10 0x7f166415e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:20.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.029+0000 7f166bed6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f16640c58d0 0x7f166406d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.029+0000 7f166bed6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16640c5e10 con 0x7f166406d6b0 2023-10-16T20:42:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.029+0000 7f166bed6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f166415e810 con 0x7f16640c58d0 2023-10-16T20:42:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.029+0000 7f166bed6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166406f840 con 0x7f1664157b10 2023-10-16T20:42:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.029+0000 7f1669471640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f16640c58d0 0x7f166406d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.029+0000 7f1669471640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f16640c58d0 0x7f166406d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47754/0 (socket says 172.21.15.88:47754) 2023-10-16T20:42:20.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.029+0000 7f1669471640 1 -- 172.21.15.88:0/2122864716 learned_addr learned my addr 172.21.15.88:0/2122864716 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:20.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.029+0000 7f1669c72640 1 --2- 172.21.15.88:0/2122864716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1664157b10 0x7f166415e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:20.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.029+0000 7f1669471640 1 -- 172.21.15.88:0/2122864716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1664157b10 msgr2=0x7f166415e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:20.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.029+0000 7f1669471640 1 --2- 172.21.15.88:0/2122864716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1664157b10 0x7f166415e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:20.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.029+0000 7f1669471640 1 -- 172.21.15.88:0/2122864716 >> v1:172.21.15.191:6789/0 conn(0x7f166406d6b0 legacy=0x7f1664160e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:20.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.029+0000 7f1669471640 1 -- 172.21.15.88:0/2122864716 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166406fb20 con 0x7f16640c58d0 2023-10-16T20:42:20.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.029+0000 7f1669471640 1 --2- 172.21.15.88:0/2122864716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16640c58d0 0x7f166406d170 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1654009870 tx=0x7f165401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c6726b19ea7ae6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:20.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f1668c70640 1 -- 172.21.15.88:0/2122864716 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16540ac600 con 0x7f16640c58d0 2023-10-16T20:42:20.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f1668c70640 1 -- 172.21.15.88:0/2122864716 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16540a8050 con 0x7f16640c58d0 2023-10-16T20:42:20.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f1668c70640 1 -- 172.21.15.88:0/2122864716 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16540ac980 con 0x7f16640c58d0 2023-10-16T20:42:20.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f166bed6640 1 -- 172.21.15.88:0/2122864716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16640c58d0 msgr2=0x7f166406d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f166bed6640 1 --2- 172.21.15.88:0/2122864716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16640c58d0 0x7f166406d170 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1654009870 tx=0x7f165401c4b0 comp rx=0 tx=0).stop 2023-10-16T20:42:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f166bed6640 1 -- 172.21.15.88:0/2122864716 shutdown_connections 2023-10-16T20:42:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f166bed6640 1 --2- 172.21.15.88:0/2122864716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16640c58d0 0x7f166406d170 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f166bed6640 1 --2- 172.21.15.88:0/2122864716 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1664157b10 0x7f166415e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f166bed6640 1 -- 172.21.15.88:0/2122864716 >> 172.21.15.88:0/2122864716 conn(0x7f16640cad50 msgr2=0x7f16640cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:20.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f166bed6640 1 -- 172.21.15.88:0/2122864716 shutdown_connections 2023-10-16T20:42:20.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f166bed6640 1 -- 172.21.15.88:0/2122864716 wait complete. 2023-10-16T20:42:20.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f166bed6640 1 Processor -- start 2023-10-16T20:42:20.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f166bed6640 1 -- start start 2023-10-16T20:42:20.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f166bed6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f166406d6b0 0x7f16641585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:20.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f166bed6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16640c58d0 0x7f16641f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:20.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f166bed6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1664157b10 0x7f16641507e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:20.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f166bed6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166415cc10 con 0x7f16640c58d0 2023-10-16T20:42:20.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f166bed6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1664070730 con 0x7f166406d6b0 2023-10-16T20:42:20.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f166bed6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1664160480 con 0x7f1664157b10 2023-10-16T20:42:20.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f1669471640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16640c58d0 0x7f16641f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:20.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f1669471640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16640c58d0 0x7f16641f3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47758/0 (socket says 172.21.15.88:47758) 2023-10-16T20:42:20.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.033+0000 7f1669471640 1 -- 172.21.15.88:0/2954009533 learned_addr learned my addr 172.21.15.88:0/2954009533 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:20.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f166a473640 1 --2- 172.21.15.88:0/2954009533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1664157b10 0x7f16641507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:20.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f1669c72640 1 --2- 172.21.15.88:0/2954009533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f166406d6b0 0x7f16641585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:20.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f1669471640 1 -- 172.21.15.88:0/2954009533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1664157b10 msgr2=0x7f16641507e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:20.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f1669471640 1 --2- 172.21.15.88:0/2954009533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1664157b10 0x7f16641507e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:20.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f1669471640 1 -- 172.21.15.88:0/2954009533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f166406d6b0 msgr2=0x7f16641585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:20.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f1669471640 1 --2- 172.21.15.88:0/2954009533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f166406d6b0 0x7f16641585d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:20.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f1669471640 1 -- 172.21.15.88:0/2954009533 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16540094d0 con 0x7f16640c58d0 2023-10-16T20:42:20.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f1669471640 1 --2- 172.21.15.88:0/2954009533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16640c58d0 0x7f16641f3a80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f165401a0e0 tx=0x7f165401acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:20.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f165a7fc640 1 -- 172.21.15.88:0/2954009533 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16540b7650 con 0x7f16640c58d0 2023-10-16T20:42:20.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f165a7fc640 1 -- 172.21.15.88:0/2954009533 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16540ace90 con 0x7f16640c58d0 2023-10-16T20:42:20.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f165a7fc640 1 -- 172.21.15.88:0/2954009533 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16540b79d0 con 0x7f16640c58d0 2023-10-16T20:42:20.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f166bed6640 1 -- 172.21.15.88:0/2954009533 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16641f6800 con 0x7f16640c58d0 2023-10-16T20:42:20.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f166bed6640 1 -- 172.21.15.88:0/2954009533 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166406e030 con 0x7f16640c58d0 2023-10-16T20:42:20.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f166bed6640 1 -- 172.21.15.88:0/2954009533 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1664074700 con 0x7f16640c58d0 2023-10-16T20:42:20.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f165a7fc640 1 -- 172.21.15.88:0/2954009533 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f16540b3070 con 0x7f16640c58d0 2023-10-16T20:42:20.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f165a7fc640 1 --2- 172.21.15.88:0/2954009533 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f16480664a0 0x7f1648068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:20.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f165a7fc640 1 -- 172.21.15.88:0/2954009533 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f165411e250 con 0x7f16640c58d0 2023-10-16T20:42:20.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.037+0000 7f1669c72640 1 --2- 172.21.15.88:0/2954009533 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f16480664a0 0x7f1648068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:20.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.041+0000 7f1669c72640 1 --2- 172.21.15.88:0/2954009533 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f16480664a0 0x7f1648068960 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f166000b9d0 tx=0x7f1660008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:20.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.041+0000 7f165a7fc640 1 -- 172.21.15.88:0/2954009533 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f16540eb220 con 0x7f16640c58d0 2023-10-16T20:42:20.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.225+0000 7f166bed6640 1 -- 172.21.15.88:0/2954009533 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16640c6410 con 0x7f16640c58d0 2023-10-16T20:42:20.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.225+0000 7f165a7fc640 1 -- 172.21.15.88:0/2954009533 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5776 (secure 0 0 0) 0x7f16540ee840 con 0x7f16640c58d0 2023-10-16T20:42:20.232 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:20.232 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:16.016135+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:resolve","state_seq":83,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:20.234 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 25 2023-10-16T20:42:20.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.225+0000 7f166bed6640 1 -- 172.21.15.88:0/2954009533 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f16480664a0 msgr2=0x7f1648068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:20.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.225+0000 7f166bed6640 1 --2- 172.21.15.88:0/2954009533 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f16480664a0 0x7f1648068960 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f166000b9d0 tx=0x7f1660008040 comp rx=0 tx=0).stop 2023-10-16T20:42:20.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.225+0000 7f166bed6640 1 -- 172.21.15.88:0/2954009533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16640c58d0 msgr2=0x7f16641f3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:20.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.225+0000 7f166bed6640 1 --2- 172.21.15.88:0/2954009533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16640c58d0 0x7f16641f3a80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f165401a0e0 tx=0x7f165401acf0 comp rx=0 tx=0).stop 2023-10-16T20:42:20.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.225+0000 7f166bed6640 1 -- 172.21.15.88:0/2954009533 shutdown_connections 2023-10-16T20:42:20.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.225+0000 7f166bed6640 1 --2- 172.21.15.88:0/2954009533 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f16480664a0 0x7f1648068960 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:20.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.225+0000 7f166bed6640 1 --2- 172.21.15.88:0/2954009533 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1664157b10 0x7f16641507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:20.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.225+0000 7f166bed6640 1 --2- 172.21.15.88:0/2954009533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16640c58d0 0x7f16641f3a80 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:20.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.225+0000 7f166bed6640 1 --2- 172.21.15.88:0/2954009533 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f166406d6b0 0x7f16641585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:20.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.225+0000 7f166bed6640 1 -- 172.21.15.88:0/2954009533 >> 172.21.15.88:0/2954009533 conn(0x7f16640cad50 msgr2=0x7f166406ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:20.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.225+0000 7f166bed6640 1 -- 172.21.15.88:0/2954009533 shutdown_connections 2023-10-16T20:42:20.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:20.225+0000 7f166bed6640 1 -- 172.21.15.88:0/2954009533 wait complete. 2023-10-16T20:42:20.252 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:42:22.254 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:22.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.393+0000 7f344abc4640 1 Processor -- start 2023-10-16T20:42:22.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.393+0000 7f344abc4640 1 -- start start 2023-10-16T20:42:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f344abc4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3444151650 0x7f344406e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f344abc4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f344406ec90 0x7f344415adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f344abc4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f344415b300 0x7f34440c5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f344abc4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f344415a970 con 0x7f3444151650 2023-10-16T20:42:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f344abc4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f344415b770 con 0x7f344415b300 2023-10-16T20:42:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f344abc4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3444157350 con 0x7f344406ec90 2023-10-16T20:42:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f3448960640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3444151650 0x7f344406e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f3448960640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3444151650 0x7f344406e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47770/0 (socket says 172.21.15.88:47770) 2023-10-16T20:42:22.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f3448960640 1 -- 172.21.15.88:0/1131932258 learned_addr learned my addr 172.21.15.88:0/1131932258 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:22.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f3449161640 1 --2- 172.21.15.88:0/1131932258 >> v2:172.21.15.191:3300/0 conn(0x7f344415b300 0x7f34440c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:22.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f3448960640 1 -- 172.21.15.88:0/1131932258 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f344406ec90 msgr2=0x7f344415adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:22.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f343bfff640 1 --2- 172.21.15.88:0/1131932258 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f344406ec90 0x7f344415adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:22.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f3448960640 1 --2- 172.21.15.88:0/1131932258 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f344406ec90 0x7f344415adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:22.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f3448960640 1 -- 172.21.15.88:0/1131932258 >> v2:172.21.15.191:3300/0 conn(0x7f344415b300 msgr2=0x7f34440c5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:22.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f3448960640 1 --2- 172.21.15.88:0/1131932258 >> v2:172.21.15.191:3300/0 conn(0x7f344415b300 0x7f34440c5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:22.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f3448960640 1 -- 172.21.15.88:0/1131932258 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3444157610 con 0x7f3444151650 2023-10-16T20:42:22.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f3449161640 1 --2- 172.21.15.88:0/1131932258 >> v2:172.21.15.191:3300/0 conn(0x7f344415b300 0x7f34440c5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:42:22.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.397+0000 7f343bfff640 1 --2- 172.21.15.88:0/1131932258 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f344406ec90 0x7f344415adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:42:22.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.401+0000 7f3448960640 1 --2- 172.21.15.88:0/1131932258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3444151650 0x7f344406e750 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f34440cba40 tx=0x7f344001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ac0212decb6b12 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:22.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.401+0000 7f343b7fe640 1 -- 172.21.15.88:0/1131932258 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34400a89b0 con 0x7f3444151650 2023-10-16T20:42:22.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.401+0000 7f343b7fe640 1 -- 172.21.15.88:0/1131932258 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34400a8b10 con 0x7f3444151650 2023-10-16T20:42:22.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.401+0000 7f343b7fe640 1 -- 172.21.15.88:0/1131932258 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34400a8ea0 con 0x7f3444151650 2023-10-16T20:42:22.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.401+0000 7f344abc4640 1 -- 172.21.15.88:0/1131932258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3444151650 msgr2=0x7f344406e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:22.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.401+0000 7f344abc4640 1 --2- 172.21.15.88:0/1131932258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3444151650 0x7f344406e750 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f34440cba40 tx=0x7f344001ecf0 comp rx=0 tx=0).stop 2023-10-16T20:42:22.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.401+0000 7f344abc4640 1 -- 172.21.15.88:0/1131932258 shutdown_connections 2023-10-16T20:42:22.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.401+0000 7f344abc4640 1 --2- 172.21.15.88:0/1131932258 >> v2:172.21.15.191:3300/0 conn(0x7f344415b300 0x7f34440c5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:22.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.401+0000 7f344abc4640 1 --2- 172.21.15.88:0/1131932258 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f344406ec90 0x7f344415adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:22.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.401+0000 7f344abc4640 1 --2- 172.21.15.88:0/1131932258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3444151650 0x7f344406e750 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:22.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.401+0000 7f344abc4640 1 -- 172.21.15.88:0/1131932258 >> 172.21.15.88:0/1131932258 conn(0x7f34440cbeb0 msgr2=0x7f34440cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:22.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.401+0000 7f344abc4640 1 -- 172.21.15.88:0/1131932258 shutdown_connections 2023-10-16T20:42:22.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.401+0000 7f344abc4640 1 -- 172.21.15.88:0/1131932258 wait complete. 2023-10-16T20:42:22.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.401+0000 7f344abc4640 1 Processor -- start 2023-10-16T20:42:22.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.401+0000 7f344abc4640 1 -- start start 2023-10-16T20:42:22.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f344abc4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f344406ec90 0x7f3444155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:22.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f344abc4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3444151650 0x7f34441428b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:22.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f344abc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f344415b300 0x7f3444142df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f344abc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34441ee060 con 0x7f344415b300 2023-10-16T20:42:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f344abc4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34441f1750 con 0x7f344406ec90 2023-10-16T20:42:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f344abc4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34441f1a30 con 0x7f3444151650 2023-10-16T20:42:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f3449161640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f344415b300 0x7f3444142df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:22.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f3449161640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f344415b300 0x7f3444142df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47776/0 (socket says 172.21.15.88:47776) 2023-10-16T20:42:22.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f3449161640 1 -- 172.21.15.88:0/2076282354 learned_addr learned my addr 172.21.15.88:0/2076282354 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:22.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f343bfff640 1 --2- 172.21.15.88:0/2076282354 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3444151650 0x7f34441428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:22.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f3448960640 1 --2- 172.21.15.88:0/2076282354 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f344406ec90 0x7f3444155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:22.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f3449161640 1 -- 172.21.15.88:0/2076282354 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3444151650 msgr2=0x7f34441428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:22.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f3449161640 1 --2- 172.21.15.88:0/2076282354 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3444151650 0x7f34441428b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:22.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f3449161640 1 -- 172.21.15.88:0/2076282354 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f344406ec90 msgr2=0x7f3444155400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:22.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f3449161640 1 --2- 172.21.15.88:0/2076282354 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f344406ec90 0x7f3444155400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:22.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f3449161640 1 -- 172.21.15.88:0/2076282354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3440009480 con 0x7f344415b300 2023-10-16T20:42:22.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f3449161640 1 --2- 172.21.15.88:0/2076282354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f344415b300 0x7f3444142df0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f343c00cd50 tx=0x7f343c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:22.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f34397fa640 1 -- 172.21.15.88:0/2076282354 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c00f680 con 0x7f344415b300 2023-10-16T20:42:22.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f34397fa640 1 -- 172.21.15.88:0/2076282354 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343c00f800 con 0x7f344415b300 2023-10-16T20:42:22.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f34397fa640 1 -- 172.21.15.88:0/2076282354 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c00fbb0 con 0x7f344415b300 2023-10-16T20:42:22.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f344abc4640 1 -- 172.21.15.88:0/2076282354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34441f1d10 con 0x7f344415b300 2023-10-16T20:42:22.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f344abc4640 1 -- 172.21.15.88:0/2076282354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34441f22d0 con 0x7f344415b300 2023-10-16T20:42:22.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f344abc4640 1 -- 172.21.15.88:0/2076282354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34441ee1d0 con 0x7f344415b300 2023-10-16T20:42:22.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.405+0000 7f34397fa640 1 -- 172.21.15.88:0/2076282354 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f343c00b040 con 0x7f344415b300 2023-10-16T20:42:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.409+0000 7f34397fa640 1 --2- 172.21.15.88:0/2076282354 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34300664a0 0x7f3430068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.409+0000 7f34397fa640 1 -- 172.21.15.88:0/2076282354 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f343c084b30 con 0x7f344415b300 2023-10-16T20:42:22.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.409+0000 7f3448960640 1 --2- 172.21.15.88:0/2076282354 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34300664a0 0x7f3430068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:22.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.409+0000 7f3448960640 1 --2- 172.21.15.88:0/2076282354 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34300664a0 0x7f3430068960 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f34400b2000 tx=0x7f34400aa070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:22.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.413+0000 7f34397fa640 1 -- 172.21.15.88:0/2076282354 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f34441ee1d0 con 0x7f344415b300 2023-10-16T20:42:22.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.593+0000 7f344abc4640 1 -- 172.21.15.88:0/2076282354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34441ee1a0 con 0x7f344415b300 2023-10-16T20:42:22.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.597+0000 7f34397fa640 1 -- 172.21.15.88:0/2076282354 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5778 (secure 0 0 0) 0x7f343c01d020 con 0x7f344415b300 2023-10-16T20:42:22.603 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:22.604 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:21.404748+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:reconnect","state_seq":85,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:22.604 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 26 2023-10-16T20:42:22.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.597+0000 7f344abc4640 1 -- 172.21.15.88:0/2076282354 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34300664a0 msgr2=0x7f3430068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:22.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.597+0000 7f344abc4640 1 --2- 172.21.15.88:0/2076282354 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34300664a0 0x7f3430068960 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f34400b2000 tx=0x7f34400aa070 comp rx=0 tx=0).stop 2023-10-16T20:42:22.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.597+0000 7f344abc4640 1 -- 172.21.15.88:0/2076282354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f344415b300 msgr2=0x7f3444142df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:22.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.597+0000 7f344abc4640 1 --2- 172.21.15.88:0/2076282354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f344415b300 0x7f3444142df0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f343c00cd50 tx=0x7f343c00a650 comp rx=0 tx=0).stop 2023-10-16T20:42:22.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.597+0000 7f344abc4640 1 -- 172.21.15.88:0/2076282354 shutdown_connections 2023-10-16T20:42:22.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.597+0000 7f344abc4640 1 --2- 172.21.15.88:0/2076282354 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34300664a0 0x7f3430068960 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:22.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.597+0000 7f344abc4640 1 --2- 172.21.15.88:0/2076282354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f344415b300 0x7f3444142df0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:22.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.597+0000 7f344abc4640 1 --2- 172.21.15.88:0/2076282354 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3444151650 0x7f34441428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:22.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.597+0000 7f344abc4640 1 --2- 172.21.15.88:0/2076282354 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f344406ec90 0x7f3444155400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:22.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.597+0000 7f344abc4640 1 -- 172.21.15.88:0/2076282354 >> 172.21.15.88:0/2076282354 conn(0x7f34440cbeb0 msgr2=0x7f3444156820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:22.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.597+0000 7f344abc4640 1 -- 172.21.15.88:0/2076282354 shutdown_connections 2023-10-16T20:42:22.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:22.597+0000 7f344abc4640 1 -- 172.21.15.88:0/2076282354 wait complete. 2023-10-16T20:42:22.625 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 26, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 26, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4244": { "addr": "172.21.15.88:6839/1987579860", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6837", "nonce": 1987579860, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 1987579860, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 24, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:reconnect", "state_seq": 85 }, "gid_4263": { "addr": "172.21.15.88:6835/518281867", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 518281867, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 518281867, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4263, "incarnation": 16, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 43 }, "gid_4275": { "addr": "172.21.15.88:6838/321173984", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 321173984, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 321173984, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4275, "incarnation": 14, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 41 }, "gid_4279": { "addr": "172.21.15.191:6833/2446850309", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 2446850309, "type": "v2" }, { "addr": "172.21.15.191:6833", "nonce": 2446850309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 40 }, "gid_4288": { "addr": "172.21.15.191:6835/3059699574", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3059699574, "type": "v2" }, { "addr": "172.21.15.191:6835", "nonce": 3059699574, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:42:21.404748+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 4279, "mds_2": 4275, "mds_3": 4263, "mds_4": 4244 } } } ], "standbys": [] } 2023-10-16T20:42:24.626 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:24.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.765+0000 7fa3fbf0f640 1 Processor -- start 2023-10-16T20:42:24.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.765+0000 7fa3fbf0f640 1 -- start start 2023-10-16T20:42:24.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.769+0000 7fa3fbf0f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa3f4150280 0x7fa3f40c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:24.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.769+0000 7fa3fbf0f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f40c5c60 con 0x7fa3f4150660 2023-10-16T20:42:24.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.769+0000 7fa3fbf0f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f40c5370 con 0x7fa3f40c6bc0 2023-10-16T20:42:24.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.769+0000 7fa3fbf0f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f4156d10 con 0x7fa3f4150280 2023-10-16T20:42:24.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.769+0000 7fa3f9cab640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa3f4150280 0x7fa3f40c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:24.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.769+0000 7fa3f94aa640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa3f4150660 0x7fa3f406d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:53844/0 (socket says 172.21.15.88:53844) 2023-10-16T20:42:24.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.769+0000 7fa3f9cab640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa3f4150280 0x7fa3f40c60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47780/0 (socket says 172.21.15.88:47780) 2023-10-16T20:42:24.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.769+0000 7fa3f94aa640 1 -- 172.21.15.88:0/3109842669 learned_addr learned my addr 172.21.15.88:0/3109842669 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:42:24.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.769+0000 7fa3f9cab640 1 -- 172.21.15.88:0/3109842669 >> v1:172.21.15.191:6789/0 conn(0x7fa3f40c6bc0 legacy=0x7fa3f4156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:24.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.769+0000 7fa3f9cab640 1 -- 172.21.15.88:0/3109842669 >> v1:172.21.15.88:6789/0 conn(0x7fa3f4150660 legacy=0x7fa3f406d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:24.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.769+0000 7fa3f9cab640 1 -- 172.21.15.88:0/3109842669 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f40c5610 con 0x7fa3f4150280 2023-10-16T20:42:24.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.769+0000 7fa3f8ca9640 1 -- 172.21.15.88:0/3109842669 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144725316 0 0) 0x7fa3f40c5c60 con 0x7fa3f4150660 2023-10-16T20:42:24.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.769+0000 7fa3f9cab640 1 --2- 172.21.15.88:0/3109842669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f4150280 0x7fa3f40c60f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00097c0 tx=0x7fa3f001ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3887b6b249b8e462 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:24.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.769+0000 7fa3f8ca9640 1 -- 172.21.15.88:0/3109842669 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f00ab670 con 0x7fa3f4150280 2023-10-16T20:42:24.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.769+0000 7fa3f8ca9640 1 -- 172.21.15.88:0/3109842669 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f00ab7d0 con 0x7fa3f4150280 2023-10-16T20:42:24.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.769+0000 7fa3f8ca9640 1 -- 172.21.15.88:0/3109842669 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f00abb60 con 0x7fa3f4150280 2023-10-16T20:42:24.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fbf0f640 1 -- 172.21.15.88:0/3109842669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f4150280 msgr2=0x7fa3f40c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:24.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fbf0f640 1 --2- 172.21.15.88:0/3109842669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f4150280 0x7fa3f40c60f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00097c0 tx=0x7fa3f001ceb0 comp rx=0 tx=0).stop 2023-10-16T20:42:24.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fbf0f640 1 -- 172.21.15.88:0/3109842669 shutdown_connections 2023-10-16T20:42:24.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fbf0f640 1 --2- 172.21.15.88:0/3109842669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f4150280 0x7fa3f40c60f0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:24.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fbf0f640 1 -- 172.21.15.88:0/3109842669 >> 172.21.15.88:0/3109842669 conn(0x7fa3f40cad50 msgr2=0x7fa3f40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:24.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fbf0f640 1 -- 172.21.15.88:0/3109842669 shutdown_connections 2023-10-16T20:42:24.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fbf0f640 1 -- 172.21.15.88:0/3109842669 wait complete. 2023-10-16T20:42:24.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fbf0f640 1 Processor -- start 2023-10-16T20:42:24.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fbf0f640 1 -- start start 2023-10-16T20:42:24.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fbf0f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3f40c6bc0 0x7fa3f4071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:24.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fbf0f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3f4150280 0x7fa3f4155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:24.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fbf0f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f4150660 0x7fa3f41f33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:24.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fbf0f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f4150b60 con 0x7fa3f4150660 2023-10-16T20:42:24.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fbf0f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f41f2f40 con 0x7fa3f40c6bc0 2023-10-16T20:42:24.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fbf0f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f414d860 con 0x7fa3f4150280 2023-10-16T20:42:24.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fa4ac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f4150660 0x7fa3f41f33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:24.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fa4ac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f4150660 0x7fa3f41f33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47788/0 (socket says 172.21.15.88:47788) 2023-10-16T20:42:24.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fa4ac640 1 -- 172.21.15.88:0/2951493855 learned_addr learned my addr 172.21.15.88:0/2951493855 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:24.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3f9cab640 1 --2- 172.21.15.88:0/2951493855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3f40c6bc0 0x7fa3f4071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:24.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3f94aa640 1 --2- 172.21.15.88:0/2951493855 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3f4150280 0x7fa3f4155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:24.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fa4ac640 1 -- 172.21.15.88:0/2951493855 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3f4150280 msgr2=0x7fa3f4155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:24.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fa4ac640 1 --2- 172.21.15.88:0/2951493855 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3f4150280 0x7fa3f4155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:24.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fa4ac640 1 -- 172.21.15.88:0/2951493855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3f40c6bc0 msgr2=0x7fa3f4071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:24.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fa4ac640 1 --2- 172.21.15.88:0/2951493855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3f40c6bc0 0x7fa3f4071ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:24.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.773+0000 7fa3fa4ac640 1 -- 172.21.15.88:0/2951493855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f0009420 con 0x7fa3f4150660 2023-10-16T20:42:24.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.777+0000 7fa3fa4ac640 1 --2- 172.21.15.88:0/2951493855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f4150660 0x7fa3f41f33d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00c770 tx=0x7fa3ec00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:24.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.777+0000 7fa3ea7fc640 1 -- 172.21.15.88:0/2951493855 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec003b80 con 0x7fa3f4150660 2023-10-16T20:42:24.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.777+0000 7fa3ea7fc640 1 -- 172.21.15.88:0/2951493855 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ec003ce0 con 0x7fa3f4150660 2023-10-16T20:42:24.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.777+0000 7fa3ea7fc640 1 -- 172.21.15.88:0/2951493855 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec00aa90 con 0x7fa3f4150660 2023-10-16T20:42:24.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.777+0000 7fa3fbf0f640 1 -- 172.21.15.88:0/2951493855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f414d580 con 0x7fa3f4150660 2023-10-16T20:42:24.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.777+0000 7fa3fbf0f640 1 -- 172.21.15.88:0/2951493855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f4154bd0 con 0x7fa3f4150660 2023-10-16T20:42:24.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.777+0000 7fa3fbf0f640 1 -- 172.21.15.88:0/2951493855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f41465b0 con 0x7fa3f4150660 2023-10-16T20:42:24.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.777+0000 7fa3ea7fc640 1 -- 172.21.15.88:0/2951493855 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa3ec00abf0 con 0x7fa3f4150660 2023-10-16T20:42:24.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.777+0000 7fa3ea7fc640 1 --2- 172.21.15.88:0/2951493855 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa3d80664a0 0x7fa3d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:24.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.781+0000 7fa3ea7fc640 1 -- 172.21.15.88:0/2951493855 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7fa3ec0840c0 con 0x7fa3f4150660 2023-10-16T20:42:24.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.781+0000 7fa3f9cab640 1 --2- 172.21.15.88:0/2951493855 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa3d80664a0 0x7fa3d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:24.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.781+0000 7fa3f9cab640 1 --2- 172.21.15.88:0/2951493855 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa3d80664a0 0x7fa3d8068960 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00af000 tx=0x7fa3f0004740 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:24.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.785+0000 7fa3ea7fc640 1 -- 172.21.15.88:0/2951493855 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa3ec0510a0 con 0x7fa3f4150660 2023-10-16T20:42:24.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.965+0000 7fa3fbf0f640 1 -- 172.21.15.88:0/2951493855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3f4074700 con 0x7fa3f4150660 2023-10-16T20:42:24.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.965+0000 7fa3ea7fc640 1 -- 172.21.15.88:0/2951493855 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5778 (secure 0 0 0) 0x7fa3ec0546c0 con 0x7fa3f4150660 2023-10-16T20:42:24.973 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:24.973 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:21.404748+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:reconnect","state_seq":85,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:24.973 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 26 2023-10-16T20:42:24.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.965+0000 7fa3fbf0f640 1 -- 172.21.15.88:0/2951493855 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa3d80664a0 msgr2=0x7fa3d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:24.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.965+0000 7fa3fbf0f640 1 --2- 172.21.15.88:0/2951493855 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa3d80664a0 0x7fa3d8068960 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00af000 tx=0x7fa3f0004740 comp rx=0 tx=0).stop 2023-10-16T20:42:24.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.965+0000 7fa3fbf0f640 1 -- 172.21.15.88:0/2951493855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f4150660 msgr2=0x7fa3f41f33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:24.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.965+0000 7fa3fbf0f640 1 --2- 172.21.15.88:0/2951493855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f4150660 0x7fa3f41f33d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00c770 tx=0x7fa3ec00cc30 comp rx=0 tx=0).stop 2023-10-16T20:42:24.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.965+0000 7fa3fbf0f640 1 -- 172.21.15.88:0/2951493855 shutdown_connections 2023-10-16T20:42:24.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.965+0000 7fa3fbf0f640 1 --2- 172.21.15.88:0/2951493855 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa3d80664a0 0x7fa3d8068960 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:24.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.965+0000 7fa3fbf0f640 1 --2- 172.21.15.88:0/2951493855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f4150660 0x7fa3f41f33d0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:24.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.965+0000 7fa3fbf0f640 1 --2- 172.21.15.88:0/2951493855 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3f4150280 0x7fa3f4155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:24.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.965+0000 7fa3fbf0f640 1 --2- 172.21.15.88:0/2951493855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3f40c6bc0 0x7fa3f4071ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:24.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.965+0000 7fa3fbf0f640 1 -- 172.21.15.88:0/2951493855 >> 172.21.15.88:0/2951493855 conn(0x7fa3f40cad50 msgr2=0x7fa3f406f710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:24.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.965+0000 7fa3fbf0f640 1 -- 172.21.15.88:0/2951493855 shutdown_connections 2023-10-16T20:42:24.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:24.965+0000 7fa3fbf0f640 1 -- 172.21.15.88:0/2951493855 wait complete. 2023-10-16T20:42:24.993 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:42:25.115 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:42:25.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:42:26.994 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.137+0000 7fcb2ee54640 1 Processor -- start 2023-10-16T20:42:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.137+0000 7fcb2ee54640 1 -- start start 2023-10-16T20:42:27.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.137+0000 7fcb2ee54640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb281406d0 0x7fcb280c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.137+0000 7fcb2ee54640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcb2815a810 0x7fcb2815ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.137+0000 7fcb2ee54640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb2815b3c0 con 0x7fcb2806a970 2023-10-16T20:42:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.137+0000 7fcb2ee54640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb2815b1c0 con 0x7fcb2815a810 2023-10-16T20:42:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.137+0000 7fcb2ee54640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2815b680 con 0x7fcb281406d0 2023-10-16T20:42:27.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.137+0000 7fcb2d3f1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcb2815a810 0x7fcb2815ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:27.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.137+0000 7fcb2d3f1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcb2815a810 0x7fcb2815ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47802/0 (socket says 172.21.15.88:47802) 2023-10-16T20:42:27.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.137+0000 7fcb2d3f1640 1 -- 172.21.15.88:0/1944230697 learned_addr learned my addr 172.21.15.88:0/1944230697 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:27.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.137+0000 7fcb2cbf0640 1 --2- 172.21.15.88:0/1944230697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb281406d0 0x7fcb280c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:27.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.137+0000 7fcb2d3f1640 1 -- 172.21.15.88:0/1944230697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb281406d0 msgr2=0x7fcb280c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:27.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.137+0000 7fcb2d3f1640 1 --2- 172.21.15.88:0/1944230697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb281406d0 0x7fcb280c1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:27.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.137+0000 7fcb2d3f1640 1 -- 172.21.15.88:0/1944230697 >> v1:172.21.15.191:6789/0 conn(0x7fcb2806a970 legacy=0x7fcb28071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:27.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.137+0000 7fcb2d3f1640 1 -- 172.21.15.88:0/1944230697 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb28157570 con 0x7fcb2815a810 2023-10-16T20:42:27.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.137+0000 7fcb2cbf0640 1 --2- 172.21.15.88:0/1944230697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb281406d0 0x7fcb280c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:42:27.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.141+0000 7fcb2d3f1640 1 --2- 172.21.15.88:0/1944230697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb2815a810 0x7fcb2815ac80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcb20009e50 tx=0x7fcb2001c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9636adbd66ef77b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:27.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.141+0000 7fcb1f7fe640 1 -- 172.21.15.88:0/1944230697 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb200a58a0 con 0x7fcb2815a810 2023-10-16T20:42:27.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.141+0000 7fcb1f7fe640 1 -- 172.21.15.88:0/1944230697 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb200a5a00 con 0x7fcb2815a810 2023-10-16T20:42:27.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.141+0000 7fcb1f7fe640 1 -- 172.21.15.88:0/1944230697 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb200a5d90 con 0x7fcb2815a810 2023-10-16T20:42:27.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.141+0000 7fcb2ee54640 1 -- 172.21.15.88:0/1944230697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb2815a810 msgr2=0x7fcb2815ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:27.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.141+0000 7fcb2ee54640 1 --2- 172.21.15.88:0/1944230697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb2815a810 0x7fcb2815ac80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcb20009e50 tx=0x7fcb2001c9e0 comp rx=0 tx=0).stop 2023-10-16T20:42:27.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.141+0000 7fcb2ee54640 1 -- 172.21.15.88:0/1944230697 shutdown_connections 2023-10-16T20:42:27.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.141+0000 7fcb2ee54640 1 --2- 172.21.15.88:0/1944230697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb2815a810 0x7fcb2815ac80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:27.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.141+0000 7fcb2ee54640 1 --2- 172.21.15.88:0/1944230697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb281406d0 0x7fcb280c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:27.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.141+0000 7fcb2ee54640 1 -- 172.21.15.88:0/1944230697 >> 172.21.15.88:0/1944230697 conn(0x7fcb280cad50 msgr2=0x7fcb280cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:27.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.141+0000 7fcb2ee54640 1 -- 172.21.15.88:0/1944230697 shutdown_connections 2023-10-16T20:42:27.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.141+0000 7fcb2ee54640 1 -- 172.21.15.88:0/1944230697 wait complete. 2023-10-16T20:42:27.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2ee54640 1 Processor -- start 2023-10-16T20:42:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2ee54640 1 -- start start 2023-10-16T20:42:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2ee54640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb2806a970 0x7fcb281534f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2ee54640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb281406d0 0x7fcb281f38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2ee54640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb2815a810 0x7fcb281f3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2ee54640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb28152800 con 0x7fcb2815a810 2023-10-16T20:42:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2ee54640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb28152140 con 0x7fcb2806a970 2023-10-16T20:42:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2ee54640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb281546d0 con 0x7fcb281406d0 2023-10-16T20:42:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2d3f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb2815a810 0x7fcb281f3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2d3f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb2815a810 0x7fcb281f3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47804/0 (socket says 172.21.15.88:47804) 2023-10-16T20:42:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2d3f1640 1 -- 172.21.15.88:0/3542670516 learned_addr learned my addr 172.21.15.88:0/3542670516 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:27.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb1ffff640 1 --2- 172.21.15.88:0/3542670516 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb281406d0 0x7fcb281f38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:27.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2cbf0640 1 --2- 172.21.15.88:0/3542670516 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb2806a970 0x7fcb281534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:27.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2d3f1640 1 -- 172.21.15.88:0/3542670516 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb281406d0 msgr2=0x7fcb281f38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:27.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2d3f1640 1 --2- 172.21.15.88:0/3542670516 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb281406d0 0x7fcb281f38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:27.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2d3f1640 1 -- 172.21.15.88:0/3542670516 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb2806a970 msgr2=0x7fcb281534f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2d3f1640 1 --2- 172.21.15.88:0/3542670516 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb2806a970 0x7fcb281534f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2d3f1640 1 -- 172.21.15.88:0/3542670516 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb20009b30 con 0x7fcb2815a810 2023-10-16T20:42:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2d3f1640 1 --2- 172.21.15.88:0/3542670516 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb2815a810 0x7fcb281f3e00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcb200af040 tx=0x7fcb20019780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb1d7fa640 1 -- 172.21.15.88:0/3542670516 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb200b8640 con 0x7fcb2815a810 2023-10-16T20:42:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb1d7fa640 1 -- 172.21.15.88:0/3542670516 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb200b87a0 con 0x7fcb2815a810 2023-10-16T20:42:27.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2ee54640 1 -- 172.21.15.88:0/3542670516 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb281f3100 con 0x7fcb2815a810 2023-10-16T20:42:27.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb1d7fa640 1 -- 172.21.15.88:0/3542670516 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb200b8b10 con 0x7fcb2815a810 2023-10-16T20:42:27.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.145+0000 7fcb2ee54640 1 -- 172.21.15.88:0/3542670516 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb28154110 con 0x7fcb2815a810 2023-10-16T20:42:27.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.149+0000 7fcb2ee54640 1 -- 172.21.15.88:0/3542670516 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb28074700 con 0x7fcb2815a810 2023-10-16T20:42:27.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.149+0000 7fcb1d7fa640 1 -- 172.21.15.88:0/3542670516 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcb200b8c70 con 0x7fcb2815a810 2023-10-16T20:42:27.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.149+0000 7fcb1d7fa640 1 --2- 172.21.15.88:0/3542670516 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb140664f0 0x7fcb140689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:27.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.149+0000 7fcb1d7fa640 1 -- 172.21.15.88:0/3542670516 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7fcb2011f590 con 0x7fcb2815a810 2023-10-16T20:42:27.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.149+0000 7fcb2cbf0640 1 --2- 172.21.15.88:0/3542670516 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb140664f0 0x7fcb140689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.149+0000 7fcb2cbf0640 1 --2- 172.21.15.88:0/3542670516 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb140664f0 0x7fcb140689b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcb24006e90 tx=0x7fcb24008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.153+0000 7fcb1d7fa640 1 -- 172.21.15.88:0/3542670516 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcb200ec840 con 0x7fcb2815a810 2023-10-16T20:42:27.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.337+0000 7fcb2ee54640 1 -- 172.21.15.88:0/3542670516 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb28154c40 con 0x7fcb2815a810 2023-10-16T20:42:27.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.337+0000 7fcb1d7fa640 1 -- 172.21.15.88:0/3542670516 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5775 (secure 0 0 0) 0x7fcb200efe60 con 0x7fcb2815a810 2023-10-16T20:42:27.346 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:27.346 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:26.420109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:rejoin","state_seq":87,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:27.347 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 27 2023-10-16T20:42:27.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.341+0000 7fcb2ee54640 1 -- 172.21.15.88:0/3542670516 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb140664f0 msgr2=0x7fcb140689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:27.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.341+0000 7fcb2ee54640 1 --2- 172.21.15.88:0/3542670516 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb140664f0 0x7fcb140689b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcb24006e90 tx=0x7fcb24008040 comp rx=0 tx=0).stop 2023-10-16T20:42:27.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.341+0000 7fcb2ee54640 1 -- 172.21.15.88:0/3542670516 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb2815a810 msgr2=0x7fcb281f3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:27.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.341+0000 7fcb2ee54640 1 --2- 172.21.15.88:0/3542670516 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb2815a810 0x7fcb281f3e00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcb200af040 tx=0x7fcb20019780 comp rx=0 tx=0).stop 2023-10-16T20:42:27.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.341+0000 7fcb2ee54640 1 -- 172.21.15.88:0/3542670516 shutdown_connections 2023-10-16T20:42:27.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.341+0000 7fcb2ee54640 1 --2- 172.21.15.88:0/3542670516 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb140664f0 0x7fcb140689b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:27.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.341+0000 7fcb2ee54640 1 --2- 172.21.15.88:0/3542670516 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb2815a810 0x7fcb281f3e00 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:27.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.341+0000 7fcb2ee54640 1 --2- 172.21.15.88:0/3542670516 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb281406d0 0x7fcb281f38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:27.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.341+0000 7fcb2ee54640 1 --2- 172.21.15.88:0/3542670516 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb2806a970 0x7fcb281534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:27.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.341+0000 7fcb2ee54640 1 -- 172.21.15.88:0/3542670516 >> 172.21.15.88:0/3542670516 conn(0x7fcb280cad50 msgr2=0x7fcb2813d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:27.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.341+0000 7fcb2ee54640 1 -- 172.21.15.88:0/3542670516 shutdown_connections 2023-10-16T20:42:27.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:27.341+0000 7fcb2ee54640 1 -- 172.21.15.88:0/3542670516 wait complete. 2023-10-16T20:42:27.367 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:42:29.368 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:29.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.509+0000 7f3f283d0640 1 Processor -- start 2023-10-16T20:42:29.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.509+0000 7f3f283d0640 1 -- start start 2023-10-16T20:42:29.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.513+0000 7f3f283d0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3f2006a970 0x7f3f20071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:29.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.513+0000 7f3f283d0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3f20071e60 0x7f3f2015a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:29.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.513+0000 7f3f283d0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2015b3c0 con 0x7f3f201406d0 2023-10-16T20:42:29.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.513+0000 7f3f283d0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f200c2460 con 0x7f3f2006a970 2023-10-16T20:42:29.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.513+0000 7f3f283d0640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f2015b680 con 0x7f3f20071e60 2023-10-16T20:42:29.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.513+0000 7f3f2596b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3f2006a970 0x7f3f20071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:29.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.513+0000 7f3f2596b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3f2006a970 0x7f3f20071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36854/0 (socket says 172.21.15.88:36854) 2023-10-16T20:42:29.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.513+0000 7f3f2616c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3f201406d0 0x7f3f200c1d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36686/0 (socket says 172.21.15.88:36686) 2023-10-16T20:42:29.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.513+0000 7f3f2596b640 1 -- 172.21.15.88:0/2473819101 learned_addr learned my addr 172.21.15.88:0/2473819101 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:29.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.513+0000 7f3f2696d640 1 --2- 172.21.15.88:0/2473819101 >> v2:172.21.15.191:3300/0 conn(0x7f3f20071e60 0x7f3f2015a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:29.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.513+0000 7f3f2596b640 1 -- 172.21.15.88:0/2473819101 >> v2:172.21.15.191:3300/0 conn(0x7f3f20071e60 msgr2=0x7f3f2015a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:29.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.513+0000 7f3f2596b640 1 --2- 172.21.15.88:0/2473819101 >> v2:172.21.15.191:3300/0 conn(0x7f3f20071e60 0x7f3f2015a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:29.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.513+0000 7f3f2596b640 1 -- 172.21.15.88:0/2473819101 >> v1:172.21.15.88:6789/0 conn(0x7f3f201406d0 legacy=0x7f3f200c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:29.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.513+0000 7f3f2596b640 1 -- 172.21.15.88:0/2473819101 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f20157570 con 0x7f3f2006a970 2023-10-16T20:42:29.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.513+0000 7f3f2516a640 1 -- 172.21.15.88:0/2473819101 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625876111 0 0) 0x7f3f2015b3c0 con 0x7f3f201406d0 2023-10-16T20:42:29.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.517+0000 7f3f2596b640 1 --2- 172.21.15.88:0/2473819101 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2006a970 0x7f3f20071920 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3f200c41e0 tx=0x7f3f1001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ccdc9e4e20ad88 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:29.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.517+0000 7f3f2516a640 1 -- 172.21.15.88:0/2473819101 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f100ac640 con 0x7f3f2006a970 2023-10-16T20:42:29.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.517+0000 7f3f2516a640 1 -- 172.21.15.88:0/2473819101 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f100ac7a0 con 0x7f3f2006a970 2023-10-16T20:42:29.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.517+0000 7f3f2516a640 1 -- 172.21.15.88:0/2473819101 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f100acb30 con 0x7f3f2006a970 2023-10-16T20:42:29.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.517+0000 7f3f283d0640 1 -- 172.21.15.88:0/2473819101 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2006a970 msgr2=0x7f3f20071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:29.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.517+0000 7f3f283d0640 1 --2- 172.21.15.88:0/2473819101 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2006a970 0x7f3f20071920 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3f200c41e0 tx=0x7f3f1001c4d0 comp rx=0 tx=0).stop 2023-10-16T20:42:29.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.517+0000 7f3f283d0640 1 -- 172.21.15.88:0/2473819101 shutdown_connections 2023-10-16T20:42:29.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.517+0000 7f3f283d0640 1 --2- 172.21.15.88:0/2473819101 >> v2:172.21.15.191:3300/0 conn(0x7f3f20071e60 0x7f3f2015a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:29.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.517+0000 7f3f283d0640 1 --2- 172.21.15.88:0/2473819101 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2006a970 0x7f3f20071920 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:29.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.517+0000 7f3f283d0640 1 -- 172.21.15.88:0/2473819101 >> 172.21.15.88:0/2473819101 conn(0x7f3f200cad50 msgr2=0x7f3f200cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:29.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.517+0000 7f3f283d0640 1 -- 172.21.15.88:0/2473819101 shutdown_connections 2023-10-16T20:42:29.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.517+0000 7f3f283d0640 1 -- 172.21.15.88:0/2473819101 wait complete. 2023-10-16T20:42:29.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.517+0000 7f3f283d0640 1 Processor -- start 2023-10-16T20:42:29.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.517+0000 7f3f283d0640 1 -- start start 2023-10-16T20:42:29.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f283d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2006a970 0x7f3f20154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:29.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f283d0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f20071e60 0x7f3f20159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:29.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f283d0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f201406d0 0x7f3f201523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:29.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f283d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f20154930 con 0x7f3f2006a970 2023-10-16T20:42:29.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f283d0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f200c6ee0 con 0x7f3f201406d0 2023-10-16T20:42:29.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f283d0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f200c71c0 con 0x7f3f20071e60 2023-10-16T20:42:29.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f2616c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2006a970 0x7f3f20154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:29.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f2616c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2006a970 0x7f3f20154280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36856/0 (socket says 172.21.15.88:36856) 2023-10-16T20:42:29.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f2616c640 1 -- 172.21.15.88:0/3428493040 learned_addr learned my addr 172.21.15.88:0/3428493040 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:29.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f2596b640 1 --2- 172.21.15.88:0/3428493040 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f20071e60 0x7f3f20159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:29.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f2616c640 1 -- 172.21.15.88:0/3428493040 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f20071e60 msgr2=0x7f3f20159ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:29.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f2696d640 1 --2- 172.21.15.88:0/3428493040 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f201406d0 0x7f3f201523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:29.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f2616c640 1 --2- 172.21.15.88:0/3428493040 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f20071e60 0x7f3f20159ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:29.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f2616c640 1 -- 172.21.15.88:0/3428493040 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f201406d0 msgr2=0x7f3f201523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:29.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f2616c640 1 --2- 172.21.15.88:0/3428493040 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f201406d0 0x7f3f201523c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:29.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f2616c640 1 -- 172.21.15.88:0/3428493040 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f100094d0 con 0x7f3f2006a970 2023-10-16T20:42:29.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f2596b640 1 --2- 172.21.15.88:0/3428493040 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f20071e60 0x7f3f20159ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:42:29.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f2696d640 1 --2- 172.21.15.88:0/3428493040 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f201406d0 0x7f3f201523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:42:29.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f2616c640 1 --2- 172.21.15.88:0/3428493040 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2006a970 0x7f3f20154280 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00c480 tx=0x7f3f1c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:29.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f16ffd640 1 -- 172.21.15.88:0/3428493040 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c0074e0 con 0x7f3f2006a970 2023-10-16T20:42:29.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f16ffd640 1 -- 172.21.15.88:0/3428493040 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1c007660 con 0x7f3f2006a970 2023-10-16T20:42:29.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f283d0640 1 -- 172.21.15.88:0/3428493040 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f200c74a0 con 0x7f3f2006a970 2023-10-16T20:42:29.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f16ffd640 1 -- 172.21.15.88:0/3428493040 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c004a30 con 0x7f3f2006a970 2023-10-16T20:42:29.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f283d0640 1 -- 172.21.15.88:0/3428493040 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2015a420 con 0x7f3f2006a970 2023-10-16T20:42:29.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f283d0640 1 -- 172.21.15.88:0/3428493040 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f20074700 con 0x7f3f2006a970 2023-10-16T20:42:29.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.521+0000 7f3f16ffd640 1 -- 172.21.15.88:0/3428493040 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3f1c004b90 con 0x7f3f2006a970 2023-10-16T20:42:29.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.525+0000 7f3f16ffd640 1 --2- 172.21.15.88:0/3428493040 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3f040664a0 0x7f3f04068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:29.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.525+0000 7f3f16ffd640 1 -- 172.21.15.88:0/3428493040 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f3f1c083820 con 0x7f3f2006a970 2023-10-16T20:42:29.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.525+0000 7f3f2596b640 1 --2- 172.21.15.88:0/3428493040 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3f040664a0 0x7f3f04068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:29.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.525+0000 7f3f2596b640 1 --2- 172.21.15.88:0/3428493040 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3f040664a0 0x7f3f04068960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3f100ae000 tx=0x7f3f10008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:29.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.529+0000 7f3f16ffd640 1 -- 172.21.15.88:0/3428493040 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3f1c054310 con 0x7f3f2006a970 2023-10-16T20:42:29.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.713+0000 7f3f283d0640 1 -- 172.21.15.88:0/3428493040 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f20154c40 con 0x7f3f2006a970 2023-10-16T20:42:29.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.713+0000 7f3f16ffd640 1 -- 172.21.15.88:0/3428493040 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5775 (secure 0 0 0) 0x7f3f1c053ea0 con 0x7f3f2006a970 2023-10-16T20:42:29.720 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:29.720 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:26.420109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:rejoin","state_seq":87,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:29.721 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 27 2023-10-16T20:42:29.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.713+0000 7f3f283d0640 1 -- 172.21.15.88:0/3428493040 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3f040664a0 msgr2=0x7f3f04068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:29.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.713+0000 7f3f283d0640 1 --2- 172.21.15.88:0/3428493040 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3f040664a0 0x7f3f04068960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3f100ae000 tx=0x7f3f10008040 comp rx=0 tx=0).stop 2023-10-16T20:42:29.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.713+0000 7f3f283d0640 1 -- 172.21.15.88:0/3428493040 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2006a970 msgr2=0x7f3f20154280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:29.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.713+0000 7f3f283d0640 1 --2- 172.21.15.88:0/3428493040 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2006a970 0x7f3f20154280 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00c480 tx=0x7f3f1c00c940 comp rx=0 tx=0).stop 2023-10-16T20:42:29.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.713+0000 7f3f283d0640 1 -- 172.21.15.88:0/3428493040 shutdown_connections 2023-10-16T20:42:29.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.713+0000 7f3f283d0640 1 --2- 172.21.15.88:0/3428493040 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3f040664a0 0x7f3f04068960 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:29.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.713+0000 7f3f283d0640 1 --2- 172.21.15.88:0/3428493040 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3f201406d0 0x7f3f201523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:29.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.713+0000 7f3f283d0640 1 --2- 172.21.15.88:0/3428493040 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3f20071e60 0x7f3f20159ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:29.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.713+0000 7f3f283d0640 1 --2- 172.21.15.88:0/3428493040 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f2006a970 0x7f3f20154280 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:29.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.713+0000 7f3f283d0640 1 -- 172.21.15.88:0/3428493040 >> 172.21.15.88:0/3428493040 conn(0x7f3f200cad50 msgr2=0x7f3f200cb130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:29.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.713+0000 7f3f283d0640 1 -- 172.21.15.88:0/3428493040 shutdown_connections 2023-10-16T20:42:29.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:29.713+0000 7f3f283d0640 1 -- 172.21.15.88:0/3428493040 wait complete. 2023-10-16T20:42:29.742 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:42:31.742 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:31.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.881+0000 7fd309f35640 1 Processor -- start 2023-10-16T20:42:31.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.881+0000 7fd309f35640 1 -- start start 2023-10-16T20:42:31.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.885+0000 7fd309f35640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd3041406d0 0x7fd3040c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:31.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.885+0000 7fd309f35640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd30406a970 0x7fd304071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:31.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.885+0000 7fd309f35640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd304071e60 0x7fd30415a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:31.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.885+0000 7fd309f35640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd30415b3c0 con 0x7fd30406a970 2023-10-16T20:42:31.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.885+0000 7fd309f35640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd30415b680 con 0x7fd3041406d0 2023-10-16T20:42:31.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.885+0000 7fd309f35640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd304157570 con 0x7fd304071e60 2023-10-16T20:42:31.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.885+0000 7fd302ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd30406a970 0x7fd304071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:31.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.885+0000 7fd302ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd30406a970 0x7fd304071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36866/0 (socket says 172.21.15.88:36866) 2023-10-16T20:42:31.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.885+0000 7fd302ffd640 1 -- 172.21.15.88:0/1199340990 learned_addr learned my addr 172.21.15.88:0/1199340990 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:31.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.885+0000 7fd3037fe640 1 --2- 172.21.15.88:0/1199340990 >> v2:172.21.15.191:3300/0 conn(0x7fd3041406d0 0x7fd3040c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:31.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.885+0000 7fd303fff640 1 --2- 172.21.15.88:0/1199340990 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd304071e60 0x7fd30415a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:31.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.885+0000 7fd302ffd640 1 -- 172.21.15.88:0/1199340990 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd304071e60 msgr2=0x7fd30415a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:31.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.885+0000 7fd302ffd640 1 --2- 172.21.15.88:0/1199340990 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd304071e60 0x7fd30415a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:31.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.885+0000 7fd302ffd640 1 -- 172.21.15.88:0/1199340990 >> v2:172.21.15.191:3300/0 conn(0x7fd3041406d0 msgr2=0x7fd3040c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:31.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.885+0000 7fd302ffd640 1 --2- 172.21.15.88:0/1199340990 >> v2:172.21.15.191:3300/0 conn(0x7fd3041406d0 0x7fd3040c1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:31.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.885+0000 7fd302ffd640 1 -- 172.21.15.88:0/1199340990 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd304157850 con 0x7fd30406a970 2023-10-16T20:42:31.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.889+0000 7fd302ffd640 1 --2- 172.21.15.88:0/1199340990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd30406a970 0x7fd304071920 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd30406e1f0 tx=0x7fd2f801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0491cde3eedf94 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:31.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.889+0000 7fd3027fc640 1 -- 172.21.15.88:0/1199340990 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f80ab600 con 0x7fd30406a970 2023-10-16T20:42:31.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.889+0000 7fd3027fc640 1 -- 172.21.15.88:0/1199340990 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f80a7050 con 0x7fd30406a970 2023-10-16T20:42:31.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.889+0000 7fd3027fc640 1 -- 172.21.15.88:0/1199340990 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f80ab980 con 0x7fd30406a970 2023-10-16T20:42:31.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.889+0000 7fd309f35640 1 -- 172.21.15.88:0/1199340990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd30406a970 msgr2=0x7fd304071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:31.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.889+0000 7fd309f35640 1 --2- 172.21.15.88:0/1199340990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd30406a970 0x7fd304071920 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd30406e1f0 tx=0x7fd2f801c500 comp rx=0 tx=0).stop 2023-10-16T20:42:31.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.889+0000 7fd309f35640 1 -- 172.21.15.88:0/1199340990 shutdown_connections 2023-10-16T20:42:31.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.889+0000 7fd309f35640 1 --2- 172.21.15.88:0/1199340990 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd304071e60 0x7fd30415a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:31.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.889+0000 7fd309f35640 1 --2- 172.21.15.88:0/1199340990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd30406a970 0x7fd304071920 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:31.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.889+0000 7fd309f35640 1 --2- 172.21.15.88:0/1199340990 >> v2:172.21.15.191:3300/0 conn(0x7fd3041406d0 0x7fd3040c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:31.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.889+0000 7fd309f35640 1 -- 172.21.15.88:0/1199340990 >> 172.21.15.88:0/1199340990 conn(0x7fd3040cad50 msgr2=0x7fd3040cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:31.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.889+0000 7fd309f35640 1 -- 172.21.15.88:0/1199340990 shutdown_connections 2023-10-16T20:42:31.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.889+0000 7fd309f35640 1 -- 172.21.15.88:0/1199340990 wait complete. 2023-10-16T20:42:31.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.889+0000 7fd309f35640 1 Processor -- start 2023-10-16T20:42:31.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.889+0000 7fd309f35640 1 -- start start 2023-10-16T20:42:31.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd309f35640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd30406a970 0x7fd3041f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:31.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd309f35640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd304071e60 0x7fd304070090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:31.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd309f35640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3041406d0 0x7fd3041370b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:31.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd309f35640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd304159460 con 0x7fd30406a970 2023-10-16T20:42:31.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd309f35640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3041f22f0 con 0x7fd3041406d0 2023-10-16T20:42:31.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd309f35640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd30406a070 con 0x7fd304071e60 2023-10-16T20:42:31.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd3037fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd30406a970 0x7fd3041f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:31.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd3037fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd30406a970 0x7fd3041f1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36872/0 (socket says 172.21.15.88:36872) 2023-10-16T20:42:31.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd3037fe640 1 -- 172.21.15.88:0/1633961581 learned_addr learned my addr 172.21.15.88:0/1633961581 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:31.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd302ffd640 1 --2- 172.21.15.88:0/1633961581 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd304071e60 0x7fd304070090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:31.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd303fff640 1 --2- 172.21.15.88:0/1633961581 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3041406d0 0x7fd3041370b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:31.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd3037fe640 1 -- 172.21.15.88:0/1633961581 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd304071e60 msgr2=0x7fd304070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:31.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd3037fe640 1 --2- 172.21.15.88:0/1633961581 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd304071e60 0x7fd304070090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:31.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd3037fe640 1 -- 172.21.15.88:0/1633961581 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3041406d0 msgr2=0x7fd3041370b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:31.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd3037fe640 1 --2- 172.21.15.88:0/1633961581 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3041406d0 0x7fd3041370b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:31.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd3037fe640 1 -- 172.21.15.88:0/1633961581 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f80094d0 con 0x7fd30406a970 2023-10-16T20:42:31.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd303fff640 1 --2- 172.21.15.88:0/1633961581 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3041406d0 0x7fd3041370b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:42:31.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd3037fe640 1 --2- 172.21.15.88:0/1633961581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd30406a970 0x7fd3041f1db0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400e870 tx=0x7fd2f400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:31.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd2ebfff640 1 -- 172.21.15.88:0/1633961581 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4009510 con 0x7fd30406a970 2023-10-16T20:42:31.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd2ebfff640 1 -- 172.21.15.88:0/1633961581 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f400eec0 con 0x7fd30406a970 2023-10-16T20:42:31.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd309f35640 1 -- 172.21.15.88:0/1633961581 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3041581d0 con 0x7fd30406a970 2023-10-16T20:42:31.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd2ebfff640 1 -- 172.21.15.88:0/1633961581 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f400cb40 con 0x7fd30406a970 2023-10-16T20:42:31.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd309f35640 1 -- 172.21.15.88:0/1633961581 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd304070c10 con 0x7fd30406a970 2023-10-16T20:42:31.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.893+0000 7fd309f35640 1 -- 172.21.15.88:0/1633961581 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd304074700 con 0x7fd30406a970 2023-10-16T20:42:31.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.897+0000 7fd2ebfff640 1 -- 172.21.15.88:0/1633961581 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd2f400cca0 con 0x7fd30406a970 2023-10-16T20:42:31.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.897+0000 7fd2ebfff640 1 --2- 172.21.15.88:0/1633961581 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2f00664a0 0x7fd2f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:31.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.897+0000 7fd2ebfff640 1 -- 172.21.15.88:0/1633961581 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7fd2f40842e0 con 0x7fd30406a970 2023-10-16T20:42:31.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.897+0000 7fd302ffd640 1 --2- 172.21.15.88:0/1633961581 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2f00664a0 0x7fd2f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:31.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.897+0000 7fd302ffd640 1 --2- 172.21.15.88:0/1633961581 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2f00664a0 0x7fd2f0068960 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80ad000 tx=0x7fd2f8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:31.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:31.901+0000 7fd2ebfff640 1 -- 172.21.15.88:0/1633961581 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd2f40513e0 con 0x7fd30406a970 2023-10-16T20:42:32.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:32.085+0000 7fd309f35640 1 -- 172.21.15.88:0/1633961581 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd304154c40 con 0x7fd30406a970 2023-10-16T20:42:32.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:32.085+0000 7fd2ebfff640 1 -- 172.21.15.88:0/1633961581 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5775 (secure 0 0 0) 0x7fd2f4054a00 con 0x7fd30406a970 2023-10-16T20:42:32.095 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:32.095 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:26.420109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:rejoin","state_seq":87,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 27 2023-10-16T20:42:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:32.089+0000 7fd309f35640 1 -- 172.21.15.88:0/1633961581 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2f00664a0 msgr2=0x7fd2f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:32.089+0000 7fd309f35640 1 --2- 172.21.15.88:0/1633961581 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2f00664a0 0x7fd2f0068960 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd2f80ad000 tx=0x7fd2f8008040 comp rx=0 tx=0).stop 2023-10-16T20:42:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:32.089+0000 7fd309f35640 1 -- 172.21.15.88:0/1633961581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd30406a970 msgr2=0x7fd3041f1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:32.089+0000 7fd309f35640 1 --2- 172.21.15.88:0/1633961581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd30406a970 0x7fd3041f1db0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400e870 tx=0x7fd2f400ec70 comp rx=0 tx=0).stop 2023-10-16T20:42:32.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:32.089+0000 7fd309f35640 1 -- 172.21.15.88:0/1633961581 shutdown_connections 2023-10-16T20:42:32.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:32.089+0000 7fd309f35640 1 --2- 172.21.15.88:0/1633961581 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2f00664a0 0x7fd2f0068960 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:32.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:32.089+0000 7fd309f35640 1 --2- 172.21.15.88:0/1633961581 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3041406d0 0x7fd3041370b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:32.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:32.089+0000 7fd309f35640 1 --2- 172.21.15.88:0/1633961581 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd304071e60 0x7fd304070090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:32.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:32.089+0000 7fd309f35640 1 --2- 172.21.15.88:0/1633961581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd30406a970 0x7fd3041f1db0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:32.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:32.089+0000 7fd309f35640 1 -- 172.21.15.88:0/1633961581 >> 172.21.15.88:0/1633961581 conn(0x7fd3040cad50 msgr2=0x7fd30413b290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:32.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:32.089+0000 7fd309f35640 1 -- 172.21.15.88:0/1633961581 shutdown_connections 2023-10-16T20:42:32.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:32.089+0000 7fd309f35640 1 -- 172.21.15.88:0/1633961581 wait complete. 2023-10-16T20:42:32.115 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:42:34.116 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:34.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.257+0000 7f8a568d6640 1 Processor -- start 2023-10-16T20:42:34.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.257+0000 7f8a568d6640 1 -- start start 2023-10-16T20:42:34.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.257+0000 7f8a568d6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8a500c58d0 0x7f8a5006a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:34.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.257+0000 7f8a568d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a5006f810 0x7f8a5006fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:34.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.257+0000 7f8a568d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a500c5e80 con 0x7f8a50153670 2023-10-16T20:42:34.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.257+0000 7f8a568d6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a500c6860 con 0x7f8a500c58d0 2023-10-16T20:42:34.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.257+0000 7f8a568d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a500701a0 con 0x7f8a5006f810 2023-10-16T20:42:34.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8a50153670 0x7f8a5006d170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36700/0 (socket says 172.21.15.88:36700) 2023-10-16T20:42:34.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4ffff640 1 -- 172.21.15.88:0/1455210210 learned_addr learned my addr 172.21.15.88:0/1455210210 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:42:34.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a54e73640 1 --2- 172.21.15.88:0/1455210210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a5006f810 0x7f8a5006fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:34.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4f7fe640 1 --2- 172.21.15.88:0/1455210210 >> v2:172.21.15.191:3300/0 conn(0x7f8a500c58d0 0x7f8a5006a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:34.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4effd640 1 -- 172.21.15.88:0/1455210210 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534894402 0 0) 0x7f8a40001120 con 0x7f8a50153670 2023-10-16T20:42:34.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4effd640 1 -- 172.21.15.88:0/1455210210 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a38003530 con 0x7f8a50153670 2023-10-16T20:42:34.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4effd640 1 -- 172.21.15.88:0/1455210210 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 820710695 0 0) 0x7f8a38003530 con 0x7f8a50153670 2023-10-16T20:42:34.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4effd640 1 -- 172.21.15.88:0/1455210210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a5006f810 msgr2=0x7f8a5006fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:34.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4effd640 1 --2- 172.21.15.88:0/1455210210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a5006f810 0x7f8a5006fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4effd640 1 -- 172.21.15.88:0/1455210210 >> v2:172.21.15.191:3300/0 conn(0x7f8a500c58d0 msgr2=0x7f8a5006a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:34.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4effd640 1 --2- 172.21.15.88:0/1455210210 >> v2:172.21.15.191:3300/0 conn(0x7f8a500c58d0 0x7f8a5006a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4effd640 1 -- 172.21.15.88:0/1455210210 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5015cb30 con 0x7f8a50153670 2023-10-16T20:42:34.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4effd640 1 -- 172.21.15.88:0/1455210210 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f8a40001810 con 0x7f8a50153670 2023-10-16T20:42:34.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4effd640 1 -- 172.21.15.88:0/1455210210 >> v1:172.21.15.88:6789/0 conn(0x7f8a50153670 legacy=0x7f8a5006d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:34.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4effd640 1 --2- 172.21.15.88:0/1455210210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a5006f810 0x7f8a38003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:34.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4effd640 1 --2- 172.21.15.88:0/1455210210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a38003f00 0x7f8a380047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:34.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4effd640 1 --2- 172.21.15.88:0/1455210210 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a38004d10 0x7f8a38005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:34.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4effd640 1 -- 172.21.15.88:0/1455210210 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a500c6860 con 0x7f8a38003f00 2023-10-16T20:42:34.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4effd640 1 -- 172.21.15.88:0/1455210210 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a500701a0 con 0x7f8a38004d10 2023-10-16T20:42:34.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4effd640 1 -- 172.21.15.88:0/1455210210 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3800b990 con 0x7f8a5006f810 2023-10-16T20:42:34.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4f7fe640 1 --2- 172.21.15.88:0/1455210210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a38003f00 0x7f8a380047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:34.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4ffff640 1 --2- 172.21.15.88:0/1455210210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a5006f810 0x7f8a38003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:34.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a54e73640 1 --2- 172.21.15.88:0/1455210210 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a38004d10 0x7f8a38005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:34.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4f7fe640 1 -- 172.21.15.88:0/1455210210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a5006f810 msgr2=0x7f8a38003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:34.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4f7fe640 1 --2- 172.21.15.88:0/1455210210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a5006f810 0x7f8a38003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4f7fe640 1 -- 172.21.15.88:0/1455210210 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a38004d10 msgr2=0x7f8a38005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:34.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4f7fe640 1 --2- 172.21.15.88:0/1455210210 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a38004d10 0x7f8a38005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a4f7fe640 1 -- 172.21.15.88:0/1455210210 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a38003530 con 0x7f8a38003f00 2023-10-16T20:42:34.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.261+0000 7f8a54e73640 1 --2- 172.21.15.88:0/1455210210 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a38004d10 0x7f8a38005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:42:34.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.265+0000 7f8a4f7fe640 1 --2- 172.21.15.88:0/1455210210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a38003f00 0x7f8a380047d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009ae0 tx=0x7f8a4401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32bb0f8863060e4b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:34.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.265+0000 7f8a4effd640 1 -- 172.21.15.88:0/1455210210 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a440a6470 con 0x7f8a38003f00 2023-10-16T20:42:34.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.265+0000 7f8a4effd640 1 -- 172.21.15.88:0/1455210210 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a440a65d0 con 0x7f8a38003f00 2023-10-16T20:42:34.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.265+0000 7f8a4effd640 1 -- 172.21.15.88:0/1455210210 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a440a6980 con 0x7f8a38003f00 2023-10-16T20:42:34.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.265+0000 7f8a568d6640 1 -- 172.21.15.88:0/1455210210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a38003f00 msgr2=0x7f8a380047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:34.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.265+0000 7f8a568d6640 1 --2- 172.21.15.88:0/1455210210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a38003f00 0x7f8a380047d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009ae0 tx=0x7f8a4401ca30 comp rx=0 tx=0).stop 2023-10-16T20:42:34.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.265+0000 7f8a54e73640 1 -- 172.21.15.88:0/1455210210 reap_dead start 2023-10-16T20:42:34.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.265+0000 7f8a568d6640 1 -- 172.21.15.88:0/1455210210 shutdown_connections 2023-10-16T20:42:34.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.265+0000 7f8a568d6640 1 -- 172.21.15.88:0/1455210210 >> 172.21.15.88:0/1455210210 conn(0x7f8a500cad50 msgr2=0x7f8a500cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:34.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.265+0000 7f8a568d6640 1 -- 172.21.15.88:0/1455210210 shutdown_connections 2023-10-16T20:42:34.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.265+0000 7f8a568d6640 1 -- 172.21.15.88:0/1455210210 wait complete. 2023-10-16T20:42:34.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.265+0000 7f8a568d6640 1 Processor -- start 2023-10-16T20:42:34.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.265+0000 7f8a568d6640 1 -- start start 2023-10-16T20:42:34.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a568d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a38003f00 0x7f8a50159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:34.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a568d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5014e6a0 0x7f8a5014bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:34.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a568d6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a5014eae0 0x7f8a5014c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:34.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a568d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a50157650 con 0x7f8a5014e6a0 2023-10-16T20:42:34.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a568d6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5014d070 con 0x7f8a5014eae0 2023-10-16T20:42:34.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a568d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a50157090 con 0x7f8a38003f00 2023-10-16T20:42:34.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a4f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5014e6a0 0x7f8a5014bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:34.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a4f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5014e6a0 0x7f8a5014bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36888/0 (socket says 172.21.15.88:36888) 2023-10-16T20:42:34.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a4f7fe640 1 -- 172.21.15.88:0/391329798 learned_addr learned my addr 172.21.15.88:0/391329798 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:34.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a4ffff640 1 --2- 172.21.15.88:0/391329798 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a38003f00 0x7f8a50159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:34.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a54e73640 1 --2- 172.21.15.88:0/391329798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a5014eae0 0x7f8a5014c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:34.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a4f7fe640 1 -- 172.21.15.88:0/391329798 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a38003f00 msgr2=0x7f8a50159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:34.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a4f7fe640 1 --2- 172.21.15.88:0/391329798 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a38003f00 0x7f8a50159f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a4f7fe640 1 -- 172.21.15.88:0/391329798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a5014eae0 msgr2=0x7f8a5014c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:34.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a4f7fe640 1 --2- 172.21.15.88:0/391329798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a5014eae0 0x7f8a5014c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a4f7fe640 1 -- 172.21.15.88:0/391329798 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a50157370 con 0x7f8a5014e6a0 2023-10-16T20:42:34.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a54e73640 1 --2- 172.21.15.88:0/391329798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a5014eae0 0x7f8a5014c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:42:34.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a4f7fe640 1 --2- 172.21.15.88:0/391329798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5014e6a0 0x7f8a5014bf70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009ae0 tx=0x7f8a38005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:34.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a4cff9640 1 -- 172.21.15.88:0/391329798 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a440a6470 con 0x7f8a5014e6a0 2023-10-16T20:42:34.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a4cff9640 1 -- 172.21.15.88:0/391329798 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a440bc650 con 0x7f8a5014e6a0 2023-10-16T20:42:34.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a4cff9640 1 -- 172.21.15.88:0/391329798 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a440bc9e0 con 0x7f8a5014e6a0 2023-10-16T20:42:34.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a568d6640 1 -- 172.21.15.88:0/391329798 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5014d4c0 con 0x7f8a5014e6a0 2023-10-16T20:42:34.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a568d6640 1 -- 172.21.15.88:0/391329798 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5014da80 con 0x7f8a5014e6a0 2023-10-16T20:42:34.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a568d6640 1 -- 172.21.15.88:0/391329798 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a50074700 con 0x7f8a5014e6a0 2023-10-16T20:42:34.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.269+0000 7f8a4cff9640 1 -- 172.21.15.88:0/391329798 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8a440bcb40 con 0x7f8a5014e6a0 2023-10-16T20:42:34.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.273+0000 7f8a4cff9640 1 --2- 172.21.15.88:0/391329798 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8a3c0664d0 0x7f8a3c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:34.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.273+0000 7f8a4cff9640 1 -- 172.21.15.88:0/391329798 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f8a4411ead0 con 0x7f8a5014e6a0 2023-10-16T20:42:34.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.273+0000 7f8a4ffff640 1 --2- 172.21.15.88:0/391329798 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8a3c0664d0 0x7f8a3c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:34.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.273+0000 7f8a4ffff640 1 --2- 172.21.15.88:0/391329798 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8a3c0664d0 0x7f8a3c068990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8a400012c0 tx=0x7f8a40007a80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:34.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.273+0000 7f8a4cff9640 1 -- 172.21.15.88:0/391329798 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8a440ebb30 con 0x7f8a5014e6a0 2023-10-16T20:42:34.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.457+0000 7f8a568d6640 1 -- 172.21.15.88:0/391329798 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a50157650 con 0x7f8a5014e6a0 2023-10-16T20:42:34.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.457+0000 7f8a4cff9640 1 -- 172.21.15.88:0/391329798 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5775 (secure 0 0 0) 0x7f8a440c1020 con 0x7f8a5014e6a0 2023-10-16T20:42:34.465 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:34.466 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:31.455971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:active","state_seq":89,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:34.466 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 28 2023-10-16T20:42:34.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.461+0000 7f8a568d6640 1 -- 172.21.15.88:0/391329798 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8a3c0664d0 msgr2=0x7f8a3c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:34.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.461+0000 7f8a568d6640 1 --2- 172.21.15.88:0/391329798 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8a3c0664d0 0x7f8a3c068990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8a400012c0 tx=0x7f8a40007a80 comp rx=0 tx=0).stop 2023-10-16T20:42:34.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.461+0000 7f8a568d6640 1 -- 172.21.15.88:0/391329798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5014e6a0 msgr2=0x7f8a5014bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:34.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.461+0000 7f8a568d6640 1 --2- 172.21.15.88:0/391329798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5014e6a0 0x7f8a5014bf70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009ae0 tx=0x7f8a38005800 comp rx=0 tx=0).stop 2023-10-16T20:42:34.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.461+0000 7f8a568d6640 1 -- 172.21.15.88:0/391329798 shutdown_connections 2023-10-16T20:42:34.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.461+0000 7f8a568d6640 1 --2- 172.21.15.88:0/391329798 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8a3c0664d0 0x7f8a3c068990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.461+0000 7f8a568d6640 1 --2- 172.21.15.88:0/391329798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8a5014eae0 0x7f8a5014c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.461+0000 7f8a568d6640 1 --2- 172.21.15.88:0/391329798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a5014e6a0 0x7f8a5014bf70 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.461+0000 7f8a568d6640 1 --2- 172.21.15.88:0/391329798 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8a38003f00 0x7f8a50159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.461+0000 7f8a568d6640 1 -- 172.21.15.88:0/391329798 >> 172.21.15.88:0/391329798 conn(0x7f8a500cad50 msgr2=0x7f8a500c8730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:34.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.461+0000 7f8a568d6640 1 -- 172.21.15.88:0/391329798 shutdown_connections 2023-10-16T20:42:34.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.461+0000 7f8a568d6640 1 -- 172.21.15.88:0/391329798 wait complete. 2023-10-16T20:42:34.486 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=4, replacing gid=4250 2023-10-16T20:42:34.487 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.b 2023-10-16T20:42:34.487 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-10-16T20:42:34.487 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T20:42:34.487 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T20:42:34.491 INFO:tasks.ceph.mds.b:Started 2023-10-16T20:42:34.491 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:34.582 INFO:tasks.ceph.mds.b.smithi191.stdout:starting mds.b at 2023-10-16T20:42:34.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.621+0000 7f70b1609640 1 Processor -- start 2023-10-16T20:42:34.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70b1609640 1 -- start start 2023-10-16T20:42:34.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70b1609640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac0c9bb0 0x7f70ac0c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:34.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70b1609640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ac153b50 con 0x7f70ac153770 2023-10-16T20:42:34.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70b1609640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ac0cb2f0 con 0x7f70ac0c41f0 2023-10-16T20:42:34.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70b1609640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac0cc250 con 0x7f70ac0c9bb0 2023-10-16T20:42:34.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70aaffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f70ac153770 0x7f70ac159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36714/0 (socket says 172.21.15.88:36714) 2023-10-16T20:42:34.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70aaffd640 1 -- 172.21.15.88:0/2247058474 learned_addr learned my addr 172.21.15.88:0/2247058474 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:42:34.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70aa7fc640 1 --2- 172.21.15.88:0/2247058474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac0c9bb0 0x7f70ac0c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:34.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70a9ffb640 1 -- 172.21.15.88:0/2247058474 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620211617 0 0) 0x7f70ac153b50 con 0x7f70ac153770 2023-10-16T20:42:34.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70a9ffb640 1 -- 172.21.15.88:0/2247058474 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7094003530 con 0x7f70ac153770 2023-10-16T20:42:34.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70a9ffb640 1 -- 172.21.15.88:0/2247058474 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180554566 0 0) 0x7f70ac0cb2f0 con 0x7f70ac0c41f0 2023-10-16T20:42:34.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70a9ffb640 1 -- 172.21.15.88:0/2247058474 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70ac153b50 con 0x7f70ac0c41f0 2023-10-16T20:42:34.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70a9ffb640 1 -- 172.21.15.88:0/2247058474 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 725457708 0 0) 0x7f7094003530 con 0x7f70ac153770 2023-10-16T20:42:34.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70a9ffb640 1 -- 172.21.15.88:0/2247058474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac0c9bb0 msgr2=0x7f70ac0c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:34.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70a9ffb640 1 --2- 172.21.15.88:0/2247058474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac0c9bb0 0x7f70ac0c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70a9ffb640 1 -- 172.21.15.88:0/2247058474 >> v1:172.21.15.191:6789/0 conn(0x7f70ac0c41f0 legacy=0x7f70ac0cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:34.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70a9ffb640 1 -- 172.21.15.88:0/2247058474 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ac0cb5d0 con 0x7f70ac153770 2023-10-16T20:42:34.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70a9ffb640 1 -- 172.21.15.88:0/2247058474 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f709c001800 con 0x7f70ac153770 2023-10-16T20:42:34.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70a9ffb640 1 -- 172.21.15.88:0/2247058474 >> v1:172.21.15.88:6789/0 conn(0x7f70ac153770 legacy=0x7f70ac159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:34.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70a9ffb640 1 --2- 172.21.15.88:0/2247058474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7094003b40 0x7f7094003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:34.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70a9ffb640 1 --2- 172.21.15.88:0/2247058474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac0c9bb0 0x7f70940047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:34.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70a9ffb640 1 --2- 172.21.15.88:0/2247058474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7094004d10 0x7f7094005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:34.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70a9ffb640 1 -- 172.21.15.88:0/2247058474 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac0cc250 con 0x7f7094003b40 2023-10-16T20:42:34.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70a9ffb640 1 -- 172.21.15.88:0/2247058474 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709400ba40 con 0x7f7094004d10 2023-10-16T20:42:34.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70a9ffb640 1 -- 172.21.15.88:0/2247058474 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f709400bc00 con 0x7f70ac0c9bb0 2023-10-16T20:42:34.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70aaffd640 1 --2- 172.21.15.88:0/2247058474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7094003b40 0x7f7094003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:34.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70aa7fc640 1 --2- 172.21.15.88:0/2247058474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac0c9bb0 0x7f70940047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:34.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70ab7fe640 1 --2- 172.21.15.88:0/2247058474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7094004d10 0x7f7094005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:34.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70aaffd640 1 -- 172.21.15.88:0/2247058474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac0c9bb0 msgr2=0x7f70940047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:34.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70aaffd640 1 --2- 172.21.15.88:0/2247058474 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac0c9bb0 0x7f70940047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70aaffd640 1 -- 172.21.15.88:0/2247058474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7094004d10 msgr2=0x7f7094005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:34.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70aaffd640 1 --2- 172.21.15.88:0/2247058474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7094004d10 0x7f7094005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70aaffd640 1 -- 172.21.15.88:0/2247058474 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7094003530 con 0x7f7094003b40 2023-10-16T20:42:34.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.625+0000 7f70ab7fe640 1 --2- 172.21.15.88:0/2247058474 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7094004d10 0x7f7094005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:42:34.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.629+0000 7f70aaffd640 1 --2- 172.21.15.88:0/2247058474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7094003b40 0x7f7094003f40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f709c00c190 tx=0x7f709c01cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a105f0c69672ea server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:34.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.629+0000 7f70a9ffb640 1 -- 172.21.15.88:0/2247058474 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c0070d0 con 0x7f7094003b40 2023-10-16T20:42:34.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.629+0000 7f70a9ffb640 1 -- 172.21.15.88:0/2247058474 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709c007230 con 0x7f7094003b40 2023-10-16T20:42:34.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.629+0000 7f70a9ffb640 1 -- 172.21.15.88:0/2247058474 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c007600 con 0x7f7094003b40 2023-10-16T20:42:34.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.629+0000 7f70b1609640 1 -- 172.21.15.88:0/2247058474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7094003b40 msgr2=0x7f7094003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:34.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.629+0000 7f70b1609640 1 --2- 172.21.15.88:0/2247058474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7094003b40 0x7f7094003f40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f709c00c190 tx=0x7f709c01cbc0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.629+0000 7f70ab7fe640 1 -- 172.21.15.88:0/2247058474 reap_dead start 2023-10-16T20:42:34.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.629+0000 7f70b1609640 1 -- 172.21.15.88:0/2247058474 shutdown_connections 2023-10-16T20:42:34.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.629+0000 7f70b1609640 1 -- 172.21.15.88:0/2247058474 >> 172.21.15.88:0/2247058474 conn(0x7f70ac0715c0 msgr2=0x7f70ac0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:34.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.629+0000 7f70b1609640 1 -- 172.21.15.88:0/2247058474 shutdown_connections 2023-10-16T20:42:34.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.629+0000 7f70b1609640 1 -- 172.21.15.88:0/2247058474 wait complete. 2023-10-16T20:42:34.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70b1609640 1 Processor -- start 2023-10-16T20:42:34.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70b1609640 1 -- start start 2023-10-16T20:42:34.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70b1609640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7094003b40 0x7f70ac0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:34.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70b1609640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac0b4990 0x7f70ac15f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:34.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70b1609640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70ac1f1c00 0x7f70ac1f2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:34.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70b1609640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac0c76b0 con 0x7f70ac1f1c00 2023-10-16T20:42:34.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70b1609640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac0b3990 con 0x7f7094003b40 2023-10-16T20:42:34.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70b1609640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac1f25c0 con 0x7f70ac0b4990 2023-10-16T20:42:34.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70ab7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70ac1f1c00 0x7f70ac1f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:34.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70ab7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70ac1f1c00 0x7f70ac1f2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36898/0 (socket says 172.21.15.88:36898) 2023-10-16T20:42:34.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70ab7fe640 1 -- 172.21.15.88:0/1560445486 learned_addr learned my addr 172.21.15.88:0/1560445486 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:34.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70aaffd640 1 --2- 172.21.15.88:0/1560445486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7094003b40 0x7f70ac0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:34.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70aa7fc640 1 --2- 172.21.15.88:0/1560445486 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac0b4990 0x7f70ac15f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:34.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70ab7fe640 1 -- 172.21.15.88:0/1560445486 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac0b4990 msgr2=0x7f70ac15f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:34.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70ab7fe640 1 --2- 172.21.15.88:0/1560445486 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac0b4990 0x7f70ac15f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70ab7fe640 1 -- 172.21.15.88:0/1560445486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7094003b40 msgr2=0x7f70ac0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:34.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70ab7fe640 1 --2- 172.21.15.88:0/1560445486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7094003b40 0x7f70ac0b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70ab7fe640 1 -- 172.21.15.88:0/1560445486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ac0b33f0 con 0x7f70ac1f1c00 2023-10-16T20:42:34.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70ab7fe640 1 --2- 172.21.15.88:0/1560445486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70ac1f1c00 0x7f70ac1f2080 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f70a4001340 tx=0x7f70a400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:34.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70937fe640 1 -- 172.21.15.88:0/1560445486 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a4006b60 con 0x7f70ac1f1c00 2023-10-16T20:42:34.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70937fe640 1 -- 172.21.15.88:0/1560445486 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a4006cc0 con 0x7f70ac1f1c00 2023-10-16T20:42:34.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70937fe640 1 -- 172.21.15.88:0/1560445486 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a4003bc0 con 0x7f70ac1f1c00 2023-10-16T20:42:34.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70b1609640 1 -- 172.21.15.88:0/1560445486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70ac0b4720 con 0x7f70ac1f1c00 2023-10-16T20:42:34.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.633+0000 7f70b1609640 1 -- 172.21.15.88:0/1560445486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ac0b3e80 con 0x7f70ac1f1c00 2023-10-16T20:42:34.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.637+0000 7f70b1609640 1 -- 172.21.15.88:0/1560445486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70ac074700 con 0x7f70ac1f1c00 2023-10-16T20:42:34.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.637+0000 7f70937fe640 1 -- 172.21.15.88:0/1560445486 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f70a4003d20 con 0x7f70ac1f1c00 2023-10-16T20:42:34.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.637+0000 7f70937fe640 1 --2- 172.21.15.88:0/1560445486 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f70980664d0 0x7f7098068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:34.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.637+0000 7f70937fe640 1 -- 172.21.15.88:0/1560445486 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f70a4083b70 con 0x7f70ac1f1c00 2023-10-16T20:42:34.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.637+0000 7f70aaffd640 1 --2- 172.21.15.88:0/1560445486 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f70980664d0 0x7f7098068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:34.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.637+0000 7f70aaffd640 1 --2- 172.21.15.88:0/1560445486 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f70980664d0 0x7f7098068990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f709c0aa000 tx=0x7f709c0b0040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:34.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.641+0000 7f70937fe640 1 -- 172.21.15.88:0/1560445486 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f70a4050c70 con 0x7f70ac1f1c00 2023-10-16T20:42:34.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.821+0000 7f70b1609640 1 -- 172.21.15.88:0/1560445486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70ac1f18b0 con 0x7f70ac1f1c00 2023-10-16T20:42:34.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.821+0000 7f70937fe640 1 -- 172.21.15.88:0/1560445486 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5775 (secure 0 0 0) 0x7f70a4054290 con 0x7f70ac1f1c00 2023-10-16T20:42:34.830 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:34.830 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:31.455971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:active","state_seq":89,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:34.831 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 28 2023-10-16T20:42:34.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.825+0000 7f70b1609640 1 -- 172.21.15.88:0/1560445486 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f70980664d0 msgr2=0x7f7098068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:34.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.825+0000 7f70b1609640 1 --2- 172.21.15.88:0/1560445486 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f70980664d0 0x7f7098068990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f709c0aa000 tx=0x7f709c0b0040 comp rx=0 tx=0).stop 2023-10-16T20:42:34.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.825+0000 7f70b1609640 1 -- 172.21.15.88:0/1560445486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70ac1f1c00 msgr2=0x7f70ac1f2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:34.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.825+0000 7f70b1609640 1 --2- 172.21.15.88:0/1560445486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70ac1f1c00 0x7f70ac1f2080 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f70a4001340 tx=0x7f70a400eb20 comp rx=0 tx=0).stop 2023-10-16T20:42:34.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.825+0000 7f70b1609640 1 -- 172.21.15.88:0/1560445486 shutdown_connections 2023-10-16T20:42:34.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.825+0000 7f70b1609640 1 --2- 172.21.15.88:0/1560445486 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f70980664d0 0x7f7098068990 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.825+0000 7f70b1609640 1 --2- 172.21.15.88:0/1560445486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70ac1f1c00 0x7f70ac1f2080 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.825+0000 7f70b1609640 1 --2- 172.21.15.88:0/1560445486 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f70ac0b4990 0x7f70ac15f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.825+0000 7f70b1609640 1 --2- 172.21.15.88:0/1560445486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7094003b40 0x7f70ac0b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:34.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.825+0000 7f70b1609640 1 -- 172.21.15.88:0/1560445486 >> 172.21.15.88:0/1560445486 conn(0x7f70ac0715c0 msgr2=0x7f70ac149a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:34.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.825+0000 7f70b1609640 1 -- 172.21.15.88:0/1560445486 shutdown_connections 2023-10-16T20:42:34.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:34.825+0000 7f70b1609640 1 -- 172.21.15.88:0/1560445486 wait complete. 2023-10-16T20:42:34.850 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-10-16T20:42:34.942 INFO:tasks.ceph.mds.f.smithi191.stderr:2023-10-16T20:42:34.940+0000 7fe37f078640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-10-16T20:42:36.851 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:42:36.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.993+0000 7fd983dd9640 1 Processor -- start 2023-10-16T20:42:37.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.993+0000 7fd983dd9640 1 -- start start 2023-10-16T20:42:37.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.993+0000 7fd983dd9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd97c0c7dc0 0x7fd97c159450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:37.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.993+0000 7fd983dd9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd97c0c81a0 0x7fd97c06b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:37.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.993+0000 7fd983dd9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97c0c8810 con 0x7fd97c0c76c0 2023-10-16T20:42:37.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.993+0000 7fd983dd9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd97c06b400 con 0x7fd97c0c7dc0 2023-10-16T20:42:37.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.993+0000 7fd983dd9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd97c06b0d0 con 0x7fd97c0c81a0 2023-10-16T20:42:37.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.993+0000 7fd981b75640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd97c0c7dc0 0x7fd97c159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:37.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.993+0000 7fd981b75640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd97c0c7dc0 0x7fd97c159450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36906/0 (socket says 172.21.15.88:36906) 2023-10-16T20:42:37.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.993+0000 7fd981b75640 1 -- 172.21.15.88:0/1581725735 learned_addr learned my addr 172.21.15.88:0/1581725735 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:37.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.993+0000 7fd982376640 1 --2- 172.21.15.88:0/1581725735 >> v2:172.21.15.191:3300/0 conn(0x7fd97c0c81a0 0x7fd97c06b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:37.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.993+0000 7fd981b75640 1 -- 172.21.15.88:0/1581725735 >> v2:172.21.15.191:3300/0 conn(0x7fd97c0c81a0 msgr2=0x7fd97c06b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:37.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.993+0000 7fd981b75640 1 --2- 172.21.15.88:0/1581725735 >> v2:172.21.15.191:3300/0 conn(0x7fd97c0c81a0 0x7fd97c06b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:37.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.993+0000 7fd981b75640 1 -- 172.21.15.88:0/1581725735 >> v1:172.21.15.191:6789/0 conn(0x7fd97c0c76c0 legacy=0x7fd97c06d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:42:37.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.993+0000 7fd981b75640 1 -- 172.21.15.88:0/1581725735 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97c06e6a0 con 0x7fd97c0c7dc0 2023-10-16T20:42:37.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.997+0000 7fd981b75640 1 --2- 172.21.15.88:0/1581725735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd97c0c7dc0 0x7fd97c159450 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd97c06eda0 tx=0x7fd97801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9f33ce518b1a09 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:37.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.997+0000 7fd980b73640 1 -- 172.21.15.88:0/1581725735 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9780ab600 con 0x7fd97c0c7dc0 2023-10-16T20:42:37.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.997+0000 7fd980b73640 1 -- 172.21.15.88:0/1581725735 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9780a7050 con 0x7fd97c0c7dc0 2023-10-16T20:42:37.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.997+0000 7fd980b73640 1 -- 172.21.15.88:0/1581725735 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9780ab980 con 0x7fd97c0c7dc0 2023-10-16T20:42:37.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.997+0000 7fd983dd9640 1 -- 172.21.15.88:0/1581725735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd97c0c7dc0 msgr2=0x7fd97c159450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:37.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.997+0000 7fd983dd9640 1 --2- 172.21.15.88:0/1581725735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd97c0c7dc0 0x7fd97c159450 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd97c06eda0 tx=0x7fd97801c430 comp rx=0 tx=0).stop 2023-10-16T20:42:37.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.997+0000 7fd983dd9640 1 -- 172.21.15.88:0/1581725735 shutdown_connections 2023-10-16T20:42:37.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.997+0000 7fd983dd9640 1 --2- 172.21.15.88:0/1581725735 >> v2:172.21.15.191:3300/0 conn(0x7fd97c0c81a0 0x7fd97c06b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:37.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.997+0000 7fd983dd9640 1 --2- 172.21.15.88:0/1581725735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd97c0c7dc0 0x7fd97c159450 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:37.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.997+0000 7fd983dd9640 1 -- 172.21.15.88:0/1581725735 >> 172.21.15.88:0/1581725735 conn(0x7fd97c071a60 msgr2=0x7fd97c071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:37.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.997+0000 7fd983dd9640 1 -- 172.21.15.88:0/1581725735 shutdown_connections 2023-10-16T20:42:37.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:36.997+0000 7fd983dd9640 1 -- 172.21.15.88:0/1581725735 wait complete. 2023-10-16T20:42:37.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd983dd9640 1 Processor -- start 2023-10-16T20:42:37.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd983dd9640 1 -- start start 2023-10-16T20:42:37.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd983dd9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd97c0c76c0 0x7fd97c141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:37.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd983dd9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd97c0c7dc0 0x7fd97c1f1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:37.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd983dd9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd97c0c81a0 0x7fd97c1f5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:37.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd983dd9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c1443e0 con 0x7fd97c0c76c0 2023-10-16T20:42:37.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd983dd9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c1447e0 con 0x7fd97c0c7dc0 2023-10-16T20:42:37.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd983dd9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c144c70 con 0x7fd97c0c81a0 2023-10-16T20:42:37.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd981b75640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd97c0c76c0 0x7fd97c141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:37.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd981b75640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd97c0c76c0 0x7fd97c141aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36910/0 (socket says 172.21.15.88:36910) 2023-10-16T20:42:37.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd981b75640 1 -- 172.21.15.88:0/755988935 learned_addr learned my addr 172.21.15.88:0/755988935 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:42:37.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd982376640 1 --2- 172.21.15.88:0/755988935 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd97c0c81a0 0x7fd97c1f5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:37.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd981374640 1 --2- 172.21.15.88:0/755988935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd97c0c7dc0 0x7fd97c1f1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:37.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd981b75640 1 -- 172.21.15.88:0/755988935 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd97c0c81a0 msgr2=0x7fd97c1f5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:37.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd981b75640 1 --2- 172.21.15.88:0/755988935 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd97c0c81a0 0x7fd97c1f5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:37.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd981b75640 1 -- 172.21.15.88:0/755988935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd97c0c7dc0 msgr2=0x7fd97c1f1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:37.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd981b75640 1 --2- 172.21.15.88:0/755988935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd97c0c7dc0 0x7fd97c1f1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd981b75640 1 -- 172.21.15.88:0/755988935 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9780094d0 con 0x7fd97c0c76c0 2023-10-16T20:42:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd981374640 1 --2- 172.21.15.88:0/755988935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd97c0c7dc0 0x7fd97c1f1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:42:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd982376640 1 --2- 172.21.15.88:0/755988935 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd97c0c81a0 0x7fd97c1f5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:42:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd981b75640 1 --2- 172.21.15.88:0/755988935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd97c0c76c0 0x7fd97c141aa0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd97801aae0 tx=0x7fd97801ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd9727fc640 1 -- 172.21.15.88:0/755988935 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9780b6620 con 0x7fd97c0c76c0 2023-10-16T20:42:37.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd9727fc640 1 -- 172.21.15.88:0/755988935 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9780abe90 con 0x7fd97c0c76c0 2023-10-16T20:42:37.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd9727fc640 1 -- 172.21.15.88:0/755988935 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9780b69e0 con 0x7fd97c0c76c0 2023-10-16T20:42:37.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd983dd9640 1 -- 172.21.15.88:0/755988935 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97c144100 con 0x7fd97c0c76c0 2023-10-16T20:42:37.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd983dd9640 1 -- 172.21.15.88:0/755988935 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97c143c50 con 0x7fd97c0c76c0 2023-10-16T20:42:37.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.001+0000 7fd983dd9640 1 -- 172.21.15.88:0/755988935 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97c1445b0 con 0x7fd97c0c76c0 2023-10-16T20:42:37.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.005+0000 7fd9727fc640 1 -- 172.21.15.88:0/755988935 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd9780b2040 con 0x7fd97c0c76c0 2023-10-16T20:42:37.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.005+0000 7fd9727fc640 1 --2- 172.21.15.88:0/755988935 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd9600664a0 0x7fd960068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:42:37.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.005+0000 7fd9727fc640 1 -- 172.21.15.88:0/755988935 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7fd97811e6c0 con 0x7fd97c0c76c0 2023-10-16T20:42:37.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.005+0000 7fd981374640 1 --2- 172.21.15.88:0/755988935 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd9600664a0 0x7fd960068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:42:37.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.005+0000 7fd981374640 1 --2- 172.21.15.88:0/755988935 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd9600664a0 0x7fd960068960 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd97c14aef0 tx=0x7fd96c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:42:37.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.009+0000 7fd9727fc640 1 -- 172.21.15.88:0/755988935 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd97c1445b0 con 0x7fd97c0c76c0 2023-10-16T20:42:37.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.193+0000 7fd983dd9640 1 -- 172.21.15.88:0/755988935 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd97c1445b0 con 0x7fd97c0c76c0 2023-10-16T20:42:37.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.193+0000 7fd9727fc640 1 -- 172.21.15.88:0/755988935 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6568 (secure 0 0 0) 0x7fd97c1445b0 con 0x7fd97c0c76c0 2023-10-16T20:42:37.200 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:42:37.200 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:31.455971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:active","state_seq":89,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:42:37.200 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 29 2023-10-16T20:42:37.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.193+0000 7fd983dd9640 1 -- 172.21.15.88:0/755988935 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd9600664a0 msgr2=0x7fd960068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:37.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.193+0000 7fd983dd9640 1 --2- 172.21.15.88:0/755988935 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd9600664a0 0x7fd960068960 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd97c14aef0 tx=0x7fd96c008040 comp rx=0 tx=0).stop 2023-10-16T20:42:37.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.193+0000 7fd983dd9640 1 -- 172.21.15.88:0/755988935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd97c0c76c0 msgr2=0x7fd97c141aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:42:37.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.193+0000 7fd983dd9640 1 --2- 172.21.15.88:0/755988935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd97c0c76c0 0x7fd97c141aa0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd97801aae0 tx=0x7fd97801ac70 comp rx=0 tx=0).stop 2023-10-16T20:42:37.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.193+0000 7fd983dd9640 1 -- 172.21.15.88:0/755988935 shutdown_connections 2023-10-16T20:42:37.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.193+0000 7fd983dd9640 1 --2- 172.21.15.88:0/755988935 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd9600664a0 0x7fd960068960 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:37.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.193+0000 7fd983dd9640 1 --2- 172.21.15.88:0/755988935 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd97c0c81a0 0x7fd97c1f5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:37.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.193+0000 7fd983dd9640 1 --2- 172.21.15.88:0/755988935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd97c0c7dc0 0x7fd97c1f1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:37.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.193+0000 7fd983dd9640 1 --2- 172.21.15.88:0/755988935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd97c0c76c0 0x7fd97c141aa0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:42:37.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.193+0000 7fd983dd9640 1 -- 172.21.15.88:0/755988935 >> 172.21.15.88:0/755988935 conn(0x7fd97c071a60 msgr2=0x7fd97c14c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:42:37.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.193+0000 7fd983dd9640 1 -- 172.21.15.88:0/755988935 shutdown_connections 2023-10-16T20:42:37.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:42:37.193+0000 7fd983dd9640 1 -- 172.21.15.88:0/755988935 wait complete. 2023-10-16T20:42:37.220 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-10-16T20:42:37.220 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 30 secs before thrashing 2023-10-16T20:42:40.443 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2023-10-16T20:42:40.444 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2023-10-16T20:42:40.444 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2023-10-16T20:42:40.444 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2023-10-16T20:42:40.444 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2023-10-16T20:42:40.444 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2023-10-16T20:42:40.445 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2023-10-16T20:42:40.445 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2023-10-16T20:42:40.445 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2023-10-16T20:42:40.445 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2023-10-16T20:42:40.445 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2023-10-16T20:42:40.446 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2023-10-16T20:42:40.446 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2023-10-16T20:42:40.446 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2023-10-16T20:42:40.446 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2023-10-16T20:42:40.446 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2023-10-16T20:42:40.447 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2023-10-16T20:42:40.447 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2023-10-16T20:42:40.447 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2023-10-16T20:42:40.447 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2023-10-16T20:42:40.447 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2023-10-16T20:42:40.447 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2023-10-16T20:42:40.448 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2023-10-16T20:42:40.448 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2023-10-16T20:42:40.448 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2023-10-16T20:42:40.448 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2023-10-16T20:42:40.448 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2023-10-16T20:42:40.449 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2023-10-16T20:42:40.449 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2023-10-16T20:42:40.449 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2023-10-16T20:42:40.449 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2023-10-16T20:42:40.449 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2023-10-16T20:42:40.450 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2023-10-16T20:42:40.450 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2023-10-16T20:42:40.450 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2023-10-16T20:42:40.450 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2023-10-16T20:42:40.450 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2023-10-16T20:42:40.451 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2023-10-16T20:42:40.451 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2023-10-16T20:42:40.451 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2023-10-16T20:42:40.451 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2023-10-16T20:42:40.451 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2023-10-16T20:42:40.452 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2023-10-16T20:42:40.452 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2023-10-16T20:42:40.452 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2023-10-16T20:42:40.452 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2023-10-16T20:42:40.452 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2023-10-16T20:42:40.453 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2023-10-16T20:42:40.453 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2023-10-16T20:42:40.453 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2023-10-16T20:42:40.453 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2023-10-16T20:42:40.453 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2023-10-16T20:42:40.454 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2023-10-16T20:42:40.454 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2023-10-16T20:42:40.454 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2023-10-16T20:42:40.454 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2023-10-16T20:42:40.454 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2023-10-16T20:42:40.455 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2023-10-16T20:42:40.455 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2023-10-16T20:42:40.455 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2023-10-16T20:42:40.455 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2023-10-16T20:42:40.455 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2023-10-16T20:42:40.456 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2023-10-16T20:42:40.456 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2023-10-16T20:42:40.456 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2023-10-16T20:42:40.456 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2023-10-16T20:42:40.456 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2023-10-16T20:42:40.457 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2023-10-16T20:42:40.457 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2023-10-16T20:42:40.457 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2023-10-16T20:42:40.457 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2023-10-16T20:42:40.457 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2023-10-16T20:42:40.458 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2023-10-16T20:42:40.458 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2023-10-16T20:42:40.458 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2023-10-16T20:42:40.458 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2023-10-16T20:42:40.458 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2023-10-16T20:42:40.458 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2023-10-16T20:42:40.459 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2023-10-16T20:42:40.459 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2023-10-16T20:42:40.459 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2023-10-16T20:42:43.491 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2023-10-16T20:42:43.491 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2023-10-16T20:42:43.491 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2023-10-16T20:42:43.492 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2023-10-16T20:42:43.492 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2023-10-16T20:42:43.492 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2023-10-16T20:42:43.492 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2023-10-16T20:42:43.492 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2023-10-16T20:42:43.493 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2023-10-16T20:42:43.493 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2023-10-16T20:42:43.493 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2023-10-16T20:42:43.493 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2023-10-16T20:42:43.493 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2023-10-16T20:42:43.494 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2023-10-16T20:42:43.494 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2023-10-16T20:42:43.494 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2023-10-16T20:42:43.495 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2023-10-16T20:42:43.495 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2023-10-16T20:42:43.495 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2023-10-16T20:42:43.496 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2023-10-16T20:42:43.496 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2023-10-16T20:42:43.496 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2023-10-16T20:42:43.497 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2023-10-16T20:42:43.497 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2023-10-16T20:42:43.497 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2023-10-16T20:42:43.498 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2023-10-16T20:42:43.498 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2023-10-16T20:42:43.498 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2023-10-16T20:42:43.499 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2023-10-16T20:42:43.499 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2023-10-16T20:42:43.499 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2023-10-16T20:42:43.500 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2023-10-16T20:42:43.500 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2023-10-16T20:42:43.500 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2023-10-16T20:42:43.500 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2023-10-16T20:42:43.501 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2023-10-16T20:42:43.501 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2023-10-16T20:42:43.502 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2023-10-16T20:42:43.502 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2023-10-16T20:42:43.502 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2023-10-16T20:42:43.503 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2023-10-16T20:42:43.503 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2023-10-16T20:42:43.503 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2023-10-16T20:42:43.504 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2023-10-16T20:42:43.504 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2023-10-16T20:42:43.504 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2023-10-16T20:42:43.505 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2023-10-16T20:42:43.505 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2023-10-16T20:42:43.505 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2023-10-16T20:42:43.506 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2023-10-16T20:42:43.506 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2023-10-16T20:42:43.506 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2023-10-16T20:42:43.507 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2023-10-16T20:42:43.507 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2023-10-16T20:42:43.507 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2023-10-16T20:42:43.507 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2023-10-16T20:42:43.508 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2023-10-16T20:42:43.508 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2023-10-16T20:42:43.508 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2023-10-16T20:42:43.509 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2023-10-16T20:42:43.509 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2023-10-16T20:42:43.509 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2023-10-16T20:42:43.510 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2023-10-16T20:42:43.510 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2023-10-16T20:42:43.510 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2023-10-16T20:42:43.511 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2023-10-16T20:42:43.511 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2023-10-16T20:42:43.511 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2023-10-16T20:42:43.511 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2023-10-16T20:42:43.511 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2023-10-16T20:42:43.512 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2023-10-16T20:42:43.512 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2023-10-16T20:42:43.512 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2023-10-16T20:42:43.512 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2023-10-16T20:42:43.512 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2023-10-16T20:42:43.513 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2023-10-16T20:42:43.513 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2023-10-16T20:42:43.513 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2023-10-16T20:42:43.513 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2023-10-16T20:42:43.513 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2023-10-16T20:42:43.513 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2023-10-16T20:42:43.514 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2023-10-16T20:42:46.998 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2023-10-16T20:42:46.999 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2023-10-16T20:42:46.999 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2023-10-16T20:42:46.999 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2023-10-16T20:42:46.999 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2023-10-16T20:42:47.000 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2023-10-16T20:42:47.000 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2023-10-16T20:42:47.000 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2023-10-16T20:42:47.000 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2023-10-16T20:42:47.000 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2023-10-16T20:42:47.001 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2023-10-16T20:42:47.001 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2023-10-16T20:42:47.001 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2023-10-16T20:42:47.001 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2023-10-16T20:42:47.001 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2023-10-16T20:42:47.002 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2023-10-16T20:42:47.002 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2023-10-16T20:42:47.002 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2023-10-16T20:42:47.002 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2023-10-16T20:42:47.002 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2023-10-16T20:42:47.003 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2023-10-16T20:42:47.003 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2023-10-16T20:42:47.003 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2023-10-16T20:42:47.003 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2023-10-16T20:42:47.003 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2023-10-16T20:42:47.003 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2023-10-16T20:42:47.004 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2023-10-16T20:42:47.004 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2023-10-16T20:42:47.004 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2023-10-16T20:42:47.004 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2023-10-16T20:42:47.004 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2023-10-16T20:42:47.005 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2023-10-16T20:42:47.005 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2023-10-16T20:42:47.005 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2023-10-16T20:42:47.005 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2023-10-16T20:42:47.005 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2023-10-16T20:42:47.006 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2023-10-16T20:42:47.006 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2023-10-16T20:42:47.006 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2023-10-16T20:42:47.006 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2023-10-16T20:42:47.006 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2023-10-16T20:42:47.007 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2023-10-16T20:42:47.007 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2023-10-16T20:42:47.007 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2023-10-16T20:42:47.007 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2023-10-16T20:42:47.007 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2023-10-16T20:42:47.007 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2023-10-16T20:42:47.008 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2023-10-16T20:42:47.008 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2023-10-16T20:42:47.008 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2023-10-16T20:42:47.008 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2023-10-16T20:42:47.008 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2023-10-16T20:42:47.009 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2023-10-16T20:42:47.009 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2023-10-16T20:42:47.009 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2023-10-16T20:42:47.009 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2023-10-16T20:42:47.009 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2023-10-16T20:42:47.010 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2023-10-16T20:42:47.010 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2023-10-16T20:42:47.010 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2023-10-16T20:42:47.010 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2023-10-16T20:42:47.010 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2023-10-16T20:42:47.011 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2023-10-16T20:42:47.011 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2023-10-16T20:42:47.011 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2023-10-16T20:42:47.011 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2023-10-16T20:42:47.011 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2023-10-16T20:42:47.012 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2023-10-16T20:42:47.012 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2023-10-16T20:42:47.012 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2023-10-16T20:42:47.012 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2023-10-16T20:42:47.012 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2023-10-16T20:42:47.013 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2023-10-16T20:42:47.013 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2023-10-16T20:42:47.013 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2023-10-16T20:42:47.013 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2023-10-16T20:42:47.013 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2023-10-16T20:42:47.014 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2023-10-16T20:42:47.014 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2023-10-16T20:42:47.014 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2023-10-16T20:42:47.014 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2023-10-16T20:42:53.725 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2023-10-16T20:42:53.726 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2023-10-16T20:42:53.726 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2023-10-16T20:42:53.726 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2023-10-16T20:42:53.726 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2023-10-16T20:42:53.727 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2023-10-16T20:42:53.727 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2023-10-16T20:42:53.727 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2023-10-16T20:42:53.727 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2023-10-16T20:42:53.727 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2023-10-16T20:42:53.728 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2023-10-16T20:42:53.728 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2023-10-16T20:42:53.728 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2023-10-16T20:42:53.728 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2023-10-16T20:42:53.728 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2023-10-16T20:42:53.729 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2023-10-16T20:42:53.729 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2023-10-16T20:42:53.729 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2023-10-16T20:42:53.729 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2023-10-16T20:42:53.729 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2023-10-16T20:42:53.730 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2023-10-16T20:42:53.730 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2023-10-16T20:42:53.730 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2023-10-16T20:42:53.730 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2023-10-16T20:42:53.730 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2023-10-16T20:42:53.731 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2023-10-16T20:42:53.731 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2023-10-16T20:42:53.731 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2023-10-16T20:42:53.731 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2023-10-16T20:42:53.731 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2023-10-16T20:42:53.732 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2023-10-16T20:42:53.733 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2023-10-16T20:42:53.733 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2023-10-16T20:42:53.733 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2023-10-16T20:42:53.734 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2023-10-16T20:42:53.734 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2023-10-16T20:42:53.734 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2023-10-16T20:42:53.734 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2023-10-16T20:42:53.734 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2023-10-16T20:42:53.735 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2023-10-16T20:42:53.735 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2023-10-16T20:42:53.735 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2023-10-16T20:42:53.735 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2023-10-16T20:42:53.735 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2023-10-16T20:42:53.736 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2023-10-16T20:42:53.736 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2023-10-16T20:42:53.736 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2023-10-16T20:42:53.736 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2023-10-16T20:42:53.736 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2023-10-16T20:42:53.736 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2023-10-16T20:42:53.737 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2023-10-16T20:42:53.737 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2023-10-16T20:42:53.737 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2023-10-16T20:42:53.737 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2023-10-16T20:42:53.737 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2023-10-16T20:42:53.738 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2023-10-16T20:42:53.738 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2023-10-16T20:42:53.738 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2023-10-16T20:42:53.738 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2023-10-16T20:42:53.738 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2023-10-16T20:42:53.739 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2023-10-16T20:42:53.739 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2023-10-16T20:42:53.739 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2023-10-16T20:42:53.739 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2023-10-16T20:42:53.739 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2023-10-16T20:42:53.739 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2023-10-16T20:42:53.740 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2023-10-16T20:42:53.740 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2023-10-16T20:42:53.740 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2023-10-16T20:42:53.740 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2023-10-16T20:42:53.740 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2023-10-16T20:42:53.741 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2023-10-16T20:42:53.741 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2023-10-16T20:42:53.741 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2023-10-16T20:42:53.741 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2023-10-16T20:42:53.741 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2023-10-16T20:42:53.741 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2023-10-16T20:42:53.742 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2023-10-16T20:42:53.742 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2023-10-16T20:42:53.742 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2023-10-16T20:42:53.742 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2023-10-16T20:42:55.178 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:42:55.183 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:42:57.567 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2023-10-16T20:42:57.567 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2023-10-16T20:42:57.568 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2023-10-16T20:42:57.568 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2023-10-16T20:42:57.568 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2023-10-16T20:42:57.568 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2023-10-16T20:42:57.568 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2023-10-16T20:42:57.569 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2023-10-16T20:42:57.569 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2023-10-16T20:42:57.569 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2023-10-16T20:42:57.569 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2023-10-16T20:42:57.569 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2023-10-16T20:42:57.570 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2023-10-16T20:42:57.570 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2023-10-16T20:42:57.570 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2023-10-16T20:42:57.570 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2023-10-16T20:42:57.570 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2023-10-16T20:42:57.571 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2023-10-16T20:42:57.571 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2023-10-16T20:42:57.571 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2023-10-16T20:42:57.571 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2023-10-16T20:42:57.571 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2023-10-16T20:42:57.572 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2023-10-16T20:42:57.572 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2023-10-16T20:42:57.572 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2023-10-16T20:42:57.572 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2023-10-16T20:42:57.572 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2023-10-16T20:42:57.573 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2023-10-16T20:42:57.573 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2023-10-16T20:42:57.573 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2023-10-16T20:42:57.573 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2023-10-16T20:42:57.573 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2023-10-16T20:42:57.573 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2023-10-16T20:42:57.574 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2023-10-16T20:42:57.574 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2023-10-16T20:42:57.574 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2023-10-16T20:42:57.574 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2023-10-16T20:42:57.574 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2023-10-16T20:42:57.575 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2023-10-16T20:42:57.575 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2023-10-16T20:42:57.575 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2023-10-16T20:42:57.575 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2023-10-16T20:42:57.575 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2023-10-16T20:42:57.576 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2023-10-16T20:42:57.576 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2023-10-16T20:42:57.576 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2023-10-16T20:42:57.576 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2023-10-16T20:42:57.576 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2023-10-16T20:42:57.577 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2023-10-16T20:42:57.577 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2023-10-16T20:42:57.577 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2023-10-16T20:42:57.577 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2023-10-16T20:42:57.577 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2023-10-16T20:42:57.578 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2023-10-16T20:42:57.578 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2023-10-16T20:42:57.578 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2023-10-16T20:42:57.578 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2023-10-16T20:42:57.578 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2023-10-16T20:42:57.579 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2023-10-16T20:42:57.579 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2023-10-16T20:42:57.579 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2023-10-16T20:42:57.579 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2023-10-16T20:42:57.579 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2023-10-16T20:42:57.579 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2023-10-16T20:42:57.580 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2023-10-16T20:42:57.580 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2023-10-16T20:42:57.580 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2023-10-16T20:42:57.580 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2023-10-16T20:42:57.580 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2023-10-16T20:42:57.581 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2023-10-16T20:42:57.581 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2023-10-16T20:42:57.581 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2023-10-16T20:42:57.581 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2023-10-16T20:42:57.581 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2023-10-16T20:42:57.582 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2023-10-16T20:42:57.582 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2023-10-16T20:42:57.582 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2023-10-16T20:42:57.582 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2023-10-16T20:42:57.582 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2023-10-16T20:42:57.583 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2023-10-16T20:42:57.583 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2023-10-16T20:43:05.307 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2023-10-16T20:43:05.308 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2023-10-16T20:43:05.308 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2023-10-16T20:43:05.309 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2023-10-16T20:43:05.309 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2023-10-16T20:43:05.310 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2023-10-16T20:43:05.310 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2023-10-16T20:43:05.310 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2023-10-16T20:43:05.311 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2023-10-16T20:43:05.311 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2023-10-16T20:43:05.311 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2023-10-16T20:43:05.312 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2023-10-16T20:43:05.312 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2023-10-16T20:43:05.312 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2023-10-16T20:43:05.313 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2023-10-16T20:43:05.313 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2023-10-16T20:43:05.313 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2023-10-16T20:43:05.314 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2023-10-16T20:43:05.314 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2023-10-16T20:43:05.314 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2023-10-16T20:43:05.315 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2023-10-16T20:43:05.315 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2023-10-16T20:43:05.315 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2023-10-16T20:43:05.315 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2023-10-16T20:43:05.316 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2023-10-16T20:43:05.316 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2023-10-16T20:43:05.316 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2023-10-16T20:43:05.316 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2023-10-16T20:43:05.317 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2023-10-16T20:43:05.317 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2023-10-16T20:43:05.317 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2023-10-16T20:43:05.317 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2023-10-16T20:43:05.318 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2023-10-16T20:43:05.318 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2023-10-16T20:43:05.318 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2023-10-16T20:43:05.318 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2023-10-16T20:43:05.319 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2023-10-16T20:43:05.319 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2023-10-16T20:43:05.319 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2023-10-16T20:43:05.319 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2023-10-16T20:43:05.320 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2023-10-16T20:43:05.320 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2023-10-16T20:43:05.320 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2023-10-16T20:43:05.320 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2023-10-16T20:43:05.321 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2023-10-16T20:43:05.321 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2023-10-16T20:43:05.321 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2023-10-16T20:43:05.322 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2023-10-16T20:43:05.322 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2023-10-16T20:43:05.322 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2023-10-16T20:43:05.322 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2023-10-16T20:43:05.323 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2023-10-16T20:43:05.323 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2023-10-16T20:43:05.323 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2023-10-16T20:43:05.323 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2023-10-16T20:43:05.324 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2023-10-16T20:43:05.324 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2023-10-16T20:43:05.324 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2023-10-16T20:43:05.324 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2023-10-16T20:43:05.325 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2023-10-16T20:43:05.325 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2023-10-16T20:43:05.325 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2023-10-16T20:43:05.326 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2023-10-16T20:43:05.326 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2023-10-16T20:43:05.326 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2023-10-16T20:43:05.326 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2023-10-16T20:43:05.327 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2023-10-16T20:43:05.327 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2023-10-16T20:43:05.327 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2023-10-16T20:43:05.328 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2023-10-16T20:43:05.328 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2023-10-16T20:43:05.328 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2023-10-16T20:43:05.328 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2023-10-16T20:43:05.329 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2023-10-16T20:43:05.329 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2023-10-16T20:43:05.329 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2023-10-16T20:43:05.329 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2023-10-16T20:43:05.330 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2023-10-16T20:43:05.330 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2023-10-16T20:43:05.330 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2023-10-16T20:43:05.331 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2023-10-16T20:43:07.221 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:07.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.377+0000 7f0234ad6640 1 Processor -- start 2023-10-16T20:43:07.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.377+0000 7f0234ad6640 1 -- start start 2023-10-16T20:43:07.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.381+0000 7f0234ad6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0230153770 0x7f0230159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:07.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.381+0000 7f0234ad6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f02300c9bb0 0x7f02300c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:07.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.381+0000 7f0234ad6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02300c41f0 0x7f02300cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:07.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.381+0000 7f0234ad6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02300cc250 con 0x7f02300c9bb0 2023-10-16T20:43:07.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.381+0000 7f0234ad6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02300cc670 con 0x7f0230153770 2023-10-16T20:43:07.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.381+0000 7f0234ad6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f023015a410 con 0x7f02300c41f0 2023-10-16T20:43:07.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.381+0000 7f022e59c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0230153770 0x7f0230159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:07.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.381+0000 7f022e59c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0230153770 0x7f0230159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:43182/0 (socket says 172.21.15.88:43182) 2023-10-16T20:43:07.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.381+0000 7f022e59c640 1 -- 172.21.15.88:0/4142285719 learned_addr learned my addr 172.21.15.88:0/4142285719 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:07.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.381+0000 7f022ed9d640 1 --2- 172.21.15.88:0/4142285719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02300c41f0 0x7f02300cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:07.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.381+0000 7f022dd9b640 1 --2- 172.21.15.88:0/4142285719 >> v2:172.21.15.88:3300/0 conn(0x7f02300c9bb0 0x7f02300c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:07.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.381+0000 7f022e59c640 1 -- 172.21.15.88:0/4142285719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02300c41f0 msgr2=0x7f02300cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:07.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.381+0000 7f022e59c640 1 --2- 172.21.15.88:0/4142285719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02300c41f0 0x7f02300cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:07.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.381+0000 7f022e59c640 1 -- 172.21.15.88:0/4142285719 >> v2:172.21.15.88:3300/0 conn(0x7f02300c9bb0 msgr2=0x7f02300c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:07.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.381+0000 7f022e59c640 1 --2- 172.21.15.88:0/4142285719 >> v2:172.21.15.88:3300/0 conn(0x7f02300c9bb0 0x7f02300c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:07.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.381+0000 7f022e59c640 1 -- 172.21.15.88:0/4142285719 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0230153b50 con 0x7f0230153770 2023-10-16T20:43:07.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.385+0000 7f022e59c640 1 --2- 172.21.15.88:0/4142285719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0230153770 0x7f0230159ed0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f02300cecb0 tx=0x7f022001c500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c682e34fdc2abf97 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:07.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.385+0000 7f022d59a640 1 -- 172.21.15.88:0/4142285719 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02200ac630 con 0x7f0230153770 2023-10-16T20:43:07.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.385+0000 7f022d59a640 1 -- 172.21.15.88:0/4142285719 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02200ac790 con 0x7f0230153770 2023-10-16T20:43:07.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.385+0000 7f022d59a640 1 -- 172.21.15.88:0/4142285719 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02200acb20 con 0x7f0230153770 2023-10-16T20:43:07.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.385+0000 7f0234ad6640 1 -- 172.21.15.88:0/4142285719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0230153770 msgr2=0x7f0230159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:07.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.385+0000 7f0234ad6640 1 --2- 172.21.15.88:0/4142285719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0230153770 0x7f0230159ed0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f02300cecb0 tx=0x7f022001c500 comp rx=0 tx=0).stop 2023-10-16T20:43:07.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.385+0000 7f0234ad6640 1 -- 172.21.15.88:0/4142285719 shutdown_connections 2023-10-16T20:43:07.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.385+0000 7f0234ad6640 1 --2- 172.21.15.88:0/4142285719 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02300c41f0 0x7f02300cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:07.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.385+0000 7f0234ad6640 1 --2- 172.21.15.88:0/4142285719 >> v2:172.21.15.88:3300/0 conn(0x7f02300c9bb0 0x7f02300c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:07.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.385+0000 7f0234ad6640 1 --2- 172.21.15.88:0/4142285719 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0230153770 0x7f0230159ed0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:07.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.385+0000 7f0234ad6640 1 -- 172.21.15.88:0/4142285719 >> 172.21.15.88:0/4142285719 conn(0x7f02300715c0 msgr2=0x7f02300719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:07.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.385+0000 7f0234ad6640 1 -- 172.21.15.88:0/4142285719 shutdown_connections 2023-10-16T20:43:07.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.385+0000 7f0234ad6640 1 -- 172.21.15.88:0/4142285719 wait complete. 2023-10-16T20:43:07.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f0234ad6640 1 Processor -- start 2023-10-16T20:43:07.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f0234ad6640 1 -- start start 2023-10-16T20:43:07.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f0234ad6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02300c41f0 0x7f02300b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:07.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f0234ad6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02300c9bb0 0x7f023015a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:07.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f0234ad6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0230153770 0x7f0230156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:07.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f0234ad6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0230153f90 con 0x7f0230153770 2023-10-16T20:43:07.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f0234ad6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0230156d40 con 0x7f02300c41f0 2023-10-16T20:43:07.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f0234ad6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0230142fe0 con 0x7f02300c9bb0 2023-10-16T20:43:07.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f022ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0230153770 0x7f0230156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:07.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f022ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0230153770 0x7f0230156000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49030/0 (socket says 172.21.15.88:49030) 2023-10-16T20:43:07.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f022ed9d640 1 -- 172.21.15.88:0/2232615528 learned_addr learned my addr 172.21.15.88:0/2232615528 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:07.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f022dd9b640 1 --2- 172.21.15.88:0/2232615528 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02300c9bb0 0x7f023015a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:07.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f022e59c640 1 --2- 172.21.15.88:0/2232615528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02300c41f0 0x7f02300b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:07.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f022ed9d640 1 -- 172.21.15.88:0/2232615528 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02300c9bb0 msgr2=0x7f023015a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:07.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f022ed9d640 1 --2- 172.21.15.88:0/2232615528 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02300c9bb0 0x7f023015a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:07.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f022ed9d640 1 -- 172.21.15.88:0/2232615528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02300c41f0 msgr2=0x7f02300b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:07.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f022ed9d640 1 --2- 172.21.15.88:0/2232615528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02300c41f0 0x7f02300b46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:07.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f022ed9d640 1 -- 172.21.15.88:0/2232615528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0220009480 con 0x7f0230153770 2023-10-16T20:43:07.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.389+0000 7f022ed9d640 1 --2- 172.21.15.88:0/2232615528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0230153770 0x7f0230156000 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0228007e60 tx=0x7f022800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:07.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.393+0000 7f021effd640 1 -- 172.21.15.88:0/2232615528 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0228014620 con 0x7f0230153770 2023-10-16T20:43:07.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.393+0000 7f021effd640 1 -- 172.21.15.88:0/2232615528 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022800ee90 con 0x7f0230153770 2023-10-16T20:43:07.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.393+0000 7f021effd640 1 -- 172.21.15.88:0/2232615528 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02280149e0 con 0x7f0230153770 2023-10-16T20:43:07.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.393+0000 7f0234ad6640 1 -- 172.21.15.88:0/2232615528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0230143280 con 0x7f0230153770 2023-10-16T20:43:07.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.393+0000 7f0234ad6640 1 -- 172.21.15.88:0/2232615528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02300b3ef0 con 0x7f0230153770 2023-10-16T20:43:07.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.393+0000 7f0234ad6640 1 -- 172.21.15.88:0/2232615528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0214005260 con 0x7f0230153770 2023-10-16T20:43:07.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.393+0000 7f021effd640 1 -- 172.21.15.88:0/2232615528 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0228014b40 con 0x7f0230153770 2023-10-16T20:43:07.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.393+0000 7f021effd640 1 --2- 172.21.15.88:0/2232615528 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02100664f0 0x7f02100689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:07.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.393+0000 7f021effd640 1 -- 172.21.15.88:0/2232615528 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f0228084b30 con 0x7f0230153770 2023-10-16T20:43:07.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.397+0000 7f022e59c640 1 --2- 172.21.15.88:0/2232615528 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02100664f0 0x7f02100689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:07.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.397+0000 7f022e59c640 1 --2- 172.21.15.88:0/2232615528 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02100664f0 0x7f02100689b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f02200ae000 tx=0x7f02200a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:07.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.401+0000 7f021effd640 1 -- 172.21.15.88:0/2232615528 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0228051af0 con 0x7f0230153770 2023-10-16T20:43:07.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.653+0000 7f0234ad6640 1 -- 172.21.15.88:0/2232615528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0214005a90 con 0x7f0230153770 2023-10-16T20:43:07.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.653+0000 7f021effd640 1 -- 172.21.15.88:0/2232615528 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6568 (secure 0 0 0) 0x7f0228009980 con 0x7f0230153770 2023-10-16T20:43:07.657 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:07.657 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:31.455971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:active","state_seq":89,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:07.657 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 29 2023-10-16T20:43:07.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.657+0000 7f0234ad6640 1 -- 172.21.15.88:0/2232615528 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02100664f0 msgr2=0x7f02100689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:07.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.657+0000 7f0234ad6640 1 --2- 172.21.15.88:0/2232615528 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02100664f0 0x7f02100689b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f02200ae000 tx=0x7f02200a7000 comp rx=0 tx=0).stop 2023-10-16T20:43:07.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.657+0000 7f0234ad6640 1 -- 172.21.15.88:0/2232615528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0230153770 msgr2=0x7f0230156000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:07.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.657+0000 7f0234ad6640 1 --2- 172.21.15.88:0/2232615528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0230153770 0x7f0230156000 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0228007e60 tx=0x7f022800e4b0 comp rx=0 tx=0).stop 2023-10-16T20:43:07.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.657+0000 7f0234ad6640 1 -- 172.21.15.88:0/2232615528 shutdown_connections 2023-10-16T20:43:07.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.657+0000 7f0234ad6640 1 --2- 172.21.15.88:0/2232615528 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02100664f0 0x7f02100689b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:07.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.657+0000 7f0234ad6640 1 --2- 172.21.15.88:0/2232615528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0230153770 0x7f0230156000 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:07.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.657+0000 7f0234ad6640 1 --2- 172.21.15.88:0/2232615528 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02300c9bb0 0x7f023015a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:07.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.657+0000 7f0234ad6640 1 --2- 172.21.15.88:0/2232615528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02300c41f0 0x7f02300b46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:07.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.657+0000 7f0234ad6640 1 -- 172.21.15.88:0/2232615528 >> 172.21.15.88:0/2232615528 conn(0x7f02300715c0 msgr2=0x7f02301521b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:07.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.657+0000 7f0234ad6640 1 -- 172.21.15.88:0/2232615528 shutdown_connections 2023-10-16T20:43:07.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.657+0000 7f0234ad6640 1 -- 172.21.15.88:0/2232615528 wait complete. 2023-10-16T20:43:07.680 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=4) 2023-10-16T20:43:07.681 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T20:43:07.681 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T20:43:07.698 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T20:43:07.698 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:07.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.877+0000 7f9bca09e640 1 Processor -- start 2023-10-16T20:43:07.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.877+0000 7f9bca09e640 1 -- start start 2023-10-16T20:43:07.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.877+0000 7f9bca09e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9bc40c9bb0 0x7f9bc40c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:07.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.877+0000 7f9bca09e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9bc40c41f0 0x7f9bc40cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.877+0000 7f9bca09e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc4153b50 con 0x7f9bc4153770 2023-10-16T20:43:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.877+0000 7f9bca09e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc40cc250 con 0x7f9bc40c41f0 2023-10-16T20:43:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.877+0000 7f9bca09e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc40cc670 con 0x7f9bc40c9bb0 2023-10-16T20:43:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.877+0000 7f9bc2ffd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9bc40c9bb0 0x7f9bc40c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.877+0000 7f9bc3fff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9bc40c41f0 0x7f9bc40cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:07.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.877+0000 7f9bc37fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f9bc4153770 0x7f9bc4159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:44710/0 (socket says 172.21.15.88:44710) 2023-10-16T20:43:07.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.877+0000 7f9bc37fe640 1 -- 172.21.15.88:0/1252010830 learned_addr learned my addr 172.21.15.88:0/1252010830 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:43:07.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.877+0000 7f9bc3fff640 1 -- 172.21.15.88:0/1252010830 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9bc40c9bb0 msgr2=0x7f9bc40c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:07.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.877+0000 7f9bc27fc640 1 -- 172.21.15.88:0/1252010830 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966508998 0 0) 0x7f9bc4153b50 con 0x7f9bc4153770 2023-10-16T20:43:07.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.877+0000 7f9bc3fff640 1 --2- 172.21.15.88:0/1252010830 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9bc40c9bb0 0x7f9bc40c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:07.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.877+0000 7f9bc3fff640 1 -- 172.21.15.88:0/1252010830 >> v1:172.21.15.191:6789/0 conn(0x7f9bc4153770 legacy=0x7f9bc4159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:07.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.877+0000 7f9bc3fff640 1 -- 172.21.15.88:0/1252010830 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc40cb2f0 con 0x7f9bc40c41f0 2023-10-16T20:43:07.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.877+0000 7f9bc2ffd640 1 --2- 172.21.15.88:0/1252010830 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9bc40c9bb0 0x7f9bc40c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:43:07.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.881+0000 7f9bc3fff640 1 --2- 172.21.15.88:0/1252010830 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9bc40c41f0 0x7f9bc40cece0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009d90 tx=0x7f9bbc01c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdedb557aa859583 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:07.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.881+0000 7f9bc27fc640 1 -- 172.21.15.88:0/1252010830 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc0a5920 con 0x7f9bc40c41f0 2023-10-16T20:43:07.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.881+0000 7f9bc27fc640 1 -- 172.21.15.88:0/1252010830 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bbc0a5a80 con 0x7f9bc40c41f0 2023-10-16T20:43:07.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.881+0000 7f9bc27fc640 1 -- 172.21.15.88:0/1252010830 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc0a5e10 con 0x7f9bc40c41f0 2023-10-16T20:43:07.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.881+0000 7f9bca09e640 1 -- 172.21.15.88:0/1252010830 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9bc40c41f0 msgr2=0x7f9bc40cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:07.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.881+0000 7f9bca09e640 1 --2- 172.21.15.88:0/1252010830 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9bc40c41f0 0x7f9bc40cece0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009d90 tx=0x7f9bbc01c950 comp rx=0 tx=0).stop 2023-10-16T20:43:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.881+0000 7f9bca09e640 1 -- 172.21.15.88:0/1252010830 shutdown_connections 2023-10-16T20:43:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.881+0000 7f9bca09e640 1 --2- 172.21.15.88:0/1252010830 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9bc40c41f0 0x7f9bc40cece0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.881+0000 7f9bca09e640 1 --2- 172.21.15.88:0/1252010830 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9bc40c9bb0 0x7f9bc40c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.881+0000 7f9bca09e640 1 -- 172.21.15.88:0/1252010830 >> 172.21.15.88:0/1252010830 conn(0x7f9bc40715c0 msgr2=0x7f9bc40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.881+0000 7f9bca09e640 1 -- 172.21.15.88:0/1252010830 shutdown_connections 2023-10-16T20:43:07.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.881+0000 7f9bca09e640 1 -- 172.21.15.88:0/1252010830 wait complete. 2023-10-16T20:43:07.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bca09e640 1 Processor -- start 2023-10-16T20:43:07.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bca09e640 1 -- start start 2023-10-16T20:43:07.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bca09e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9bc40c41f0 0x7f9bc41eb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:07.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bca09e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bc40c9bb0 0x7f9bc41ebaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:07.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bca09e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9bc4153770 0x7f9bc415ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:07.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bca09e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc40b2600 con 0x7f9bc40c9bb0 2023-10-16T20:43:07.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bca09e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc41ec0f0 con 0x7f9bc4153770 2023-10-16T20:43:07.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bca09e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc415c600 con 0x7f9bc40c41f0 2023-10-16T20:43:07.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bc2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bc40c9bb0 0x7f9bc41ebaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:07.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bc37fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9bc40c41f0 0x7f9bc41eb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:07.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bc2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bc40c9bb0 0x7f9bc41ebaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49032/0 (socket says 172.21.15.88:49032) 2023-10-16T20:43:07.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bc2ffd640 1 -- 172.21.15.88:0/129254987 learned_addr learned my addr 172.21.15.88:0/129254987 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:07.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bc3fff640 1 --2- 172.21.15.88:0/129254987 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9bc4153770 0x7f9bc415ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:07.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bc2ffd640 1 -- 172.21.15.88:0/129254987 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9bc40c41f0 msgr2=0x7f9bc41eb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:07.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bc2ffd640 1 --2- 172.21.15.88:0/129254987 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9bc40c41f0 0x7f9bc41eb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:07.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bc2ffd640 1 -- 172.21.15.88:0/129254987 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9bc4153770 msgr2=0x7f9bc415ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:07.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bc2ffd640 1 --2- 172.21.15.88:0/129254987 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9bc4153770 0x7f9bc415ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:07.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bc2ffd640 1 -- 172.21.15.88:0/129254987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb8009420 con 0x7f9bc40c9bb0 2023-10-16T20:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bc3fff640 1 --2- 172.21.15.88:0/129254987 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9bc4153770 0x7f9bc415ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bc2ffd640 1 --2- 172.21.15.88:0/129254987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bc40c9bb0 0x7f9bc41ebaa0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800ebe0 tx=0x7f9bb800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9babfff640 1 -- 172.21.15.88:0/129254987 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb800eee0 con 0x7f9bc40c9bb0 2023-10-16T20:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9babfff640 1 -- 172.21.15.88:0/129254987 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb800cc30 con 0x7f9bc40c9bb0 2023-10-16T20:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9babfff640 1 -- 172.21.15.88:0/129254987 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8013470 con 0x7f9bc40c9bb0 2023-10-16T20:43:07.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bca09e640 1 -- 172.21.15.88:0/129254987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bbc009a70 con 0x7f9bc40c9bb0 2023-10-16T20:43:07.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.885+0000 7f9bca09e640 1 -- 172.21.15.88:0/129254987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc40b4860 con 0x7f9bc40c9bb0 2023-10-16T20:43:07.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.889+0000 7f9bca09e640 1 -- 172.21.15.88:0/129254987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc4074700 con 0x7f9bc40c9bb0 2023-10-16T20:43:07.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.889+0000 7f9babfff640 1 -- 172.21.15.88:0/129254987 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9bb80135d0 con 0x7f9bc40c9bb0 2023-10-16T20:43:07.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.889+0000 7f9babfff640 1 --2- 172.21.15.88:0/129254987 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9bb00664d0 0x7f9bb0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:07.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.889+0000 7f9babfff640 1 -- 172.21.15.88:0/129254987 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f9bb8084580 con 0x7f9bc40c9bb0 2023-10-16T20:43:07.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.889+0000 7f9bc37fe640 1 --2- 172.21.15.88:0/129254987 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9bb00664d0 0x7f9bb0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:07.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.889+0000 7f9bc37fe640 1 --2- 172.21.15.88:0/129254987 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9bb00664d0 0x7f9bb0068990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4000fc0 tx=0x7f9bb4007cf0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:07.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:07.893+0000 7f9babfff640 1 -- 172.21.15.88:0/129254987 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9bb8051600 con 0x7f9bc40c9bb0 2023-10-16T20:43:08.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:08.073+0000 7f9bca09e640 1 -- 172.21.15.88:0/129254987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bc415c740 con 0x7f9bc40c9bb0 2023-10-16T20:43:08.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:08.073+0000 7f9babfff640 1 -- 172.21.15.88:0/129254987 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6568 (secure 0 0 0) 0x7f9bb8054c20 con 0x7f9bc40c9bb0 2023-10-16T20:43:08.078 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:08.078 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:31.455971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:active","state_seq":89,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:08.082 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 29 2023-10-16T20:43:08.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:08.077+0000 7f9bca09e640 1 -- 172.21.15.88:0/129254987 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9bb00664d0 msgr2=0x7f9bb0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:08.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:08.077+0000 7f9bca09e640 1 --2- 172.21.15.88:0/129254987 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9bb00664d0 0x7f9bb0068990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4000fc0 tx=0x7f9bb4007cf0 comp rx=0 tx=0).stop 2023-10-16T20:43:08.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:08.077+0000 7f9bca09e640 1 -- 172.21.15.88:0/129254987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bc40c9bb0 msgr2=0x7f9bc41ebaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:08.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:08.077+0000 7f9bca09e640 1 --2- 172.21.15.88:0/129254987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bc40c9bb0 0x7f9bc41ebaa0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800ebe0 tx=0x7f9bb800c520 comp rx=0 tx=0).stop 2023-10-16T20:43:08.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:08.077+0000 7f9bca09e640 1 -- 172.21.15.88:0/129254987 shutdown_connections 2023-10-16T20:43:08.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:08.077+0000 7f9bca09e640 1 --2- 172.21.15.88:0/129254987 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9bb00664d0 0x7f9bb0068990 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:08.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:08.077+0000 7f9bca09e640 1 --2- 172.21.15.88:0/129254987 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9bc4153770 0x7f9bc415ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:08.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:08.077+0000 7f9bca09e640 1 --2- 172.21.15.88:0/129254987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bc40c9bb0 0x7f9bc41ebaa0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:08.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:08.077+0000 7f9bca09e640 1 --2- 172.21.15.88:0/129254987 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9bc40c41f0 0x7f9bc41eb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:08.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:08.077+0000 7f9bca09e640 1 -- 172.21.15.88:0/129254987 >> 172.21.15.88:0/129254987 conn(0x7f9bc40715c0 msgr2=0x7f9bc4149ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:08.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:08.077+0000 7f9bca09e640 1 -- 172.21.15.88:0/129254987 shutdown_connections 2023-10-16T20:43:08.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:08.077+0000 7f9bca09e640 1 -- 172.21.15.88:0/129254987 wait complete. 2023-10-16T20:43:08.099 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-16T20:43:10.100 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:10.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.245+0000 7f38f7d0f640 1 Processor -- start 2023-10-16T20:43:10.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.245+0000 7f38f7d0f640 1 -- start start 2023-10-16T20:43:10.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.245+0000 7f38f7d0f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38f006a970 0x7f38f0071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:10.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.245+0000 7f38f7d0f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f38f0071e60 0x7f38f015a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:10.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.245+0000 7f38f7d0f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f015b3c0 con 0x7f38f01406d0 2023-10-16T20:43:10.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.245+0000 7f38f7d0f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38f00c2460 con 0x7f38f0071e60 2023-10-16T20:43:10.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.245+0000 7f38f7d0f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f015b680 con 0x7f38f006a970 2023-10-16T20:43:10.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.245+0000 7f38f5aab640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f38f01406d0 0x7f38f00c1d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:38912/0 (socket says 172.21.15.88:38912) 2023-10-16T20:43:10.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.245+0000 7f38f5aab640 1 -- 172.21.15.88:0/3832926918 learned_addr learned my addr 172.21.15.88:0/3832926918 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:43:10.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.245+0000 7f38f62ac640 1 --2- 172.21.15.88:0/3832926918 >> v2:172.21.15.191:3300/0 conn(0x7f38f0071e60 0x7f38f015a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:10.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.245+0000 7f38f52aa640 1 --2- 172.21.15.88:0/3832926918 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38f006a970 0x7f38f0071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:10.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.245+0000 7f38f4aa9640 1 -- 172.21.15.88:0/3832926918 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937315654 0 0) 0x7f38f015b3c0 con 0x7f38f01406d0 2023-10-16T20:43:10.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.245+0000 7f38f4aa9640 1 -- 172.21.15.88:0/3832926918 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38dc003530 con 0x7f38f01406d0 2023-10-16T20:43:10.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.249+0000 7f38f4aa9640 1 -- 172.21.15.88:0/3832926918 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3060936962 0 0) 0x7f38dc003530 con 0x7f38f01406d0 2023-10-16T20:43:10.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.249+0000 7f38f62ac640 1 -- 172.21.15.88:0/3832926918 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38f006a970 msgr2=0x7f38f0071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:10.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.249+0000 7f38f62ac640 1 --2- 172.21.15.88:0/3832926918 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38f006a970 0x7f38f0071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:10.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.249+0000 7f38f62ac640 1 -- 172.21.15.88:0/3832926918 >> v1:172.21.15.88:6789/0 conn(0x7f38f01406d0 legacy=0x7f38f00c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:10.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.249+0000 7f38f62ac640 1 -- 172.21.15.88:0/3832926918 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f0157570 con 0x7f38f0071e60 2023-10-16T20:43:10.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.249+0000 7f38f52aa640 1 --2- 172.21.15.88:0/3832926918 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38f006a970 0x7f38f0071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:43:10.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.249+0000 7f38f62ac640 1 --2- 172.21.15.88:0/3832926918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38f0071e60 0x7f38f015a810 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f38e8009d90 tx=0x7f38e801c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e84e83a5541715e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:10.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.249+0000 7f38f4aa9640 1 -- 172.21.15.88:0/3832926918 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec001680 con 0x7f38f0071e60 2023-10-16T20:43:10.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.249+0000 7f38f4aa9640 1 -- 172.21.15.88:0/3832926918 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e80a5550 con 0x7f38f0071e60 2023-10-16T20:43:10.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.249+0000 7f38f4aa9640 1 -- 172.21.15.88:0/3832926918 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e80a58e0 con 0x7f38f0071e60 2023-10-16T20:43:10.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f7d0f640 1 -- 172.21.15.88:0/3832926918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38f0071e60 msgr2=0x7f38f015a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:10.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f7d0f640 1 --2- 172.21.15.88:0/3832926918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38f0071e60 0x7f38f015a810 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f38e8009d90 tx=0x7f38e801c840 comp rx=0 tx=0).stop 2023-10-16T20:43:10.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f7d0f640 1 -- 172.21.15.88:0/3832926918 shutdown_connections 2023-10-16T20:43:10.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f7d0f640 1 --2- 172.21.15.88:0/3832926918 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38f0071e60 0x7f38f015a810 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:10.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f7d0f640 1 --2- 172.21.15.88:0/3832926918 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38f006a970 0x7f38f0071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:10.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f7d0f640 1 -- 172.21.15.88:0/3832926918 >> 172.21.15.88:0/3832926918 conn(0x7f38f00cad50 msgr2=0x7f38f00cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:10.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f7d0f640 1 -- 172.21.15.88:0/3832926918 shutdown_connections 2023-10-16T20:43:10.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f7d0f640 1 -- 172.21.15.88:0/3832926918 wait complete. 2023-10-16T20:43:10.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f7d0f640 1 Processor -- start 2023-10-16T20:43:10.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f7d0f640 1 -- start start 2023-10-16T20:43:10.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f7d0f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38f006a970 0x7f38f01f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:10.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f7d0f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38f0071e60 0x7f38f01598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:10.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f7d0f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38f01406d0 0x7f38f013b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:10.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f7d0f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f0071950 con 0x7f38f01406d0 2023-10-16T20:43:10.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f7d0f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f015aa50 con 0x7f38f0071e60 2023-10-16T20:43:10.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f7d0f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f0158580 con 0x7f38f006a970 2023-10-16T20:43:10.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f62ac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38f01406d0 0x7f38f013b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:10.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f62ac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38f01406d0 0x7f38f013b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39928/0 (socket says 172.21.15.88:39928) 2023-10-16T20:43:10.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f62ac640 1 -- 172.21.15.88:0/1724395912 learned_addr learned my addr 172.21.15.88:0/1724395912 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:10.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f5aab640 1 --2- 172.21.15.88:0/1724395912 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38f006a970 0x7f38f01f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:10.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f52aa640 1 --2- 172.21.15.88:0/1724395912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38f0071e60 0x7f38f01598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:10.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f62ac640 1 -- 172.21.15.88:0/1724395912 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38f006a970 msgr2=0x7f38f01f1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:10.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f62ac640 1 --2- 172.21.15.88:0/1724395912 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38f006a970 0x7f38f01f1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:10.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f62ac640 1 -- 172.21.15.88:0/1724395912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38f0071e60 msgr2=0x7f38f01598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:10.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f62ac640 1 --2- 172.21.15.88:0/1724395912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38f0071e60 0x7f38f01598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:10.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f62ac640 1 -- 172.21.15.88:0/1724395912 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e0009420 con 0x7f38f01406d0 2023-10-16T20:43:10.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.253+0000 7f38f62ac640 1 --2- 172.21.15.88:0/1724395912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38f01406d0 0x7f38f013b330 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f38e80b0040 tx=0x7f38e80195e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:10.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.257+0000 7f38e67fc640 1 -- 172.21.15.88:0/1724395912 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e80b9650 con 0x7f38f01406d0 2023-10-16T20:43:10.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.257+0000 7f38e67fc640 1 -- 172.21.15.88:0/1724395912 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e80b97b0 con 0x7f38f01406d0 2023-10-16T20:43:10.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.257+0000 7f38e67fc640 1 -- 172.21.15.88:0/1724395912 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e80b9b40 con 0x7f38f01406d0 2023-10-16T20:43:10.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.257+0000 7f38f7d0f640 1 -- 172.21.15.88:0/1724395912 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38e8009a70 con 0x7f38f01406d0 2023-10-16T20:43:10.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.257+0000 7f38f7d0f640 1 -- 172.21.15.88:0/1724395912 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f015a490 con 0x7f38f01406d0 2023-10-16T20:43:10.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.257+0000 7f38f7d0f640 1 -- 172.21.15.88:0/1724395912 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f01583e0 con 0x7f38f01406d0 2023-10-16T20:43:10.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.257+0000 7f38e67fc640 1 -- 172.21.15.88:0/1724395912 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f38e80b9ca0 con 0x7f38f01406d0 2023-10-16T20:43:10.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.257+0000 7f38e67fc640 1 --2- 172.21.15.88:0/1724395912 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f38d40664a0 0x7f38d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:10.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.257+0000 7f38e67fc640 1 -- 172.21.15.88:0/1724395912 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f38e811f320 con 0x7f38f01406d0 2023-10-16T20:43:10.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.257+0000 7f38f5aab640 1 --2- 172.21.15.88:0/1724395912 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f38d40664a0 0x7f38d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:10.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.257+0000 7f38f5aab640 1 --2- 172.21.15.88:0/1724395912 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f38d40664a0 0x7f38d4068960 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f38ec001190 tx=0x7f38ec001120 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:10.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.261+0000 7f38e67fc640 1 -- 172.21.15.88:0/1724395912 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f38f01583e0 con 0x7f38f01406d0 2023-10-16T20:43:10.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.449+0000 7f38f7d0f640 1 -- 172.21.15.88:0/1724395912 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38f01583e0 con 0x7f38f01406d0 2023-10-16T20:43:10.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.449+0000 7f38e67fc640 1 -- 172.21.15.88:0/1724395912 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6568 (secure 0 0 0) 0x7f38f01583e0 con 0x7f38f01406d0 2023-10-16T20:43:10.453 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:10.453 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:31.455971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:active","state_seq":89,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:10.453 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 29 2023-10-16T20:43:10.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.453+0000 7f38f7d0f640 1 -- 172.21.15.88:0/1724395912 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f38d40664a0 msgr2=0x7f38d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:10.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.453+0000 7f38f7d0f640 1 --2- 172.21.15.88:0/1724395912 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f38d40664a0 0x7f38d4068960 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f38ec001190 tx=0x7f38ec001120 comp rx=0 tx=0).stop 2023-10-16T20:43:10.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.453+0000 7f38f7d0f640 1 -- 172.21.15.88:0/1724395912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38f01406d0 msgr2=0x7f38f013b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:10.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.453+0000 7f38f7d0f640 1 --2- 172.21.15.88:0/1724395912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38f01406d0 0x7f38f013b330 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f38e80b0040 tx=0x7f38e80195e0 comp rx=0 tx=0).stop 2023-10-16T20:43:10.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.453+0000 7f38f7d0f640 1 -- 172.21.15.88:0/1724395912 shutdown_connections 2023-10-16T20:43:10.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.453+0000 7f38f7d0f640 1 --2- 172.21.15.88:0/1724395912 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f38d40664a0 0x7f38d4068960 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:10.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.453+0000 7f38f7d0f640 1 --2- 172.21.15.88:0/1724395912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38f01406d0 0x7f38f013b330 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:10.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.453+0000 7f38f7d0f640 1 --2- 172.21.15.88:0/1724395912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f38f0071e60 0x7f38f01598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:10.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.453+0000 7f38f7d0f640 1 --2- 172.21.15.88:0/1724395912 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f38f006a970 0x7f38f01f1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:10.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.453+0000 7f38f7d0f640 1 -- 172.21.15.88:0/1724395912 >> 172.21.15.88:0/1724395912 conn(0x7f38f00cad50 msgr2=0x7f38f0073350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:10.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.453+0000 7f38f7d0f640 1 -- 172.21.15.88:0/1724395912 shutdown_connections 2023-10-16T20:43:10.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:10.453+0000 7f38f7d0f640 1 -- 172.21.15.88:0/1724395912 wait complete. 2023-10-16T20:43:10.474 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-16T20:43:12.475 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:12.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.621+0000 7f34a97d9640 1 Processor -- start 2023-10-16T20:43:12.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.621+0000 7f34a97d9640 1 -- start start 2023-10-16T20:43:12.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.621+0000 7f34a97d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f34a41406d0 0x7f34a40c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:12.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.621+0000 7f34a97d9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a415b3c0 con 0x7f34a415a810 2023-10-16T20:43:12.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.621+0000 7f34a97d9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a415b680 con 0x7f34a406a970 2023-10-16T20:43:12.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.625+0000 7f34a97d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34a4157570 con 0x7f34a41406d0 2023-10-16T20:43:12.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.625+0000 7f34a2ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f34a41406d0 0x7f34a40c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:12.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.625+0000 7f34a37fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f34a415a810 0x7f34a4161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:38918/0 (socket says 172.21.15.88:38918) 2023-10-16T20:43:12.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.625+0000 7f34a37fe640 1 -- 172.21.15.88:0/2245110404 learned_addr learned my addr 172.21.15.88:0/2245110404 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:43:12.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.625+0000 7f34a2ffd640 1 -- 172.21.15.88:0/2245110404 >> v1:172.21.15.191:6789/0 conn(0x7f34a406a970 legacy=0x7f34a4071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:12.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.625+0000 7f34a2ffd640 1 -- 172.21.15.88:0/2245110404 >> v1:172.21.15.88:6789/0 conn(0x7f34a415a810 legacy=0x7f34a4161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:12.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.625+0000 7f34a2ffd640 1 -- 172.21.15.88:0/2245110404 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a4157850 con 0x7f34a41406d0 2023-10-16T20:43:12.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.625+0000 7f34a27fc640 1 -- 172.21.15.88:0/2245110404 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404379059 0 0) 0x7f34a415b3c0 con 0x7f34a415a810 2023-10-16T20:43:12.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.625+0000 7f34a2ffd640 1 --2- 172.21.15.88:0/2245110404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34a41406d0 0x7f34a40c1d50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f348c0097c0 tx=0x7f348c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c582d7db8c4b1309 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:12.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.625+0000 7f34a27fc640 1 -- 172.21.15.88:0/2245110404 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348c0a7070 con 0x7f34a41406d0 2023-10-16T20:43:12.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.625+0000 7f34a27fc640 1 -- 172.21.15.88:0/2245110404 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f348c0ab430 con 0x7f34a41406d0 2023-10-16T20:43:12.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.625+0000 7f34a27fc640 1 -- 172.21.15.88:0/2245110404 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348c0ab7c0 con 0x7f34a41406d0 2023-10-16T20:43:12.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a97d9640 1 -- 172.21.15.88:0/2245110404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34a41406d0 msgr2=0x7f34a40c1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:12.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a97d9640 1 --2- 172.21.15.88:0/2245110404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34a41406d0 0x7f34a40c1d50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f348c0097c0 tx=0x7f348c01c450 comp rx=0 tx=0).stop 2023-10-16T20:43:12.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a97d9640 1 -- 172.21.15.88:0/2245110404 shutdown_connections 2023-10-16T20:43:12.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a97d9640 1 --2- 172.21.15.88:0/2245110404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34a41406d0 0x7f34a40c1d50 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:12.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a97d9640 1 -- 172.21.15.88:0/2245110404 >> 172.21.15.88:0/2245110404 conn(0x7f34a40cad50 msgr2=0x7f34a40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:12.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a97d9640 1 -- 172.21.15.88:0/2245110404 shutdown_connections 2023-10-16T20:43:12.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a97d9640 1 -- 172.21.15.88:0/2245110404 wait complete. 2023-10-16T20:43:12.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a97d9640 1 Processor -- start 2023-10-16T20:43:12.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a97d9640 1 -- start start 2023-10-16T20:43:12.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a97d9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34a406a970 0x7f34a41f5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:12.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a97d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34a41406d0 0x7f34a40c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:12.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a97d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34a415a810 0x7f34a41f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:12.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a97d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a4071130 con 0x7f34a415a810 2023-10-16T20:43:12.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a97d9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a41f5e90 con 0x7f34a406a970 2023-10-16T20:43:12.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a97d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a41f19d0 con 0x7f34a41406d0 2023-10-16T20:43:12.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34a415a810 0x7f34a41f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:12.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34a415a810 0x7f34a41f1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39958/0 (socket says 172.21.15.88:39958) 2023-10-16T20:43:12.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a37fe640 1 -- 172.21.15.88:0/3959447430 learned_addr learned my addr 172.21.15.88:0/3959447430 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:12.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f34a2ffd640 1 --2- 172.21.15.88:0/3959447430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34a406a970 0x7f34a41f5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:12.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.629+0000 7f349a7fc640 1 --2- 172.21.15.88:0/3959447430 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34a41406d0 0x7f34a40c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:12.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.633+0000 7f34a37fe640 1 -- 172.21.15.88:0/3959447430 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34a41406d0 msgr2=0x7f34a40c9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:12.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.633+0000 7f34a37fe640 1 --2- 172.21.15.88:0/3959447430 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34a41406d0 0x7f34a40c9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:12.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.633+0000 7f34a37fe640 1 -- 172.21.15.88:0/3959447430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34a406a970 msgr2=0x7f34a41f5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:12.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.633+0000 7f34a37fe640 1 --2- 172.21.15.88:0/3959447430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34a406a970 0x7f34a41f5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:12.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.633+0000 7f34a37fe640 1 -- 172.21.15.88:0/3959447430 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348c009420 con 0x7f34a415a810 2023-10-16T20:43:12.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.633+0000 7f34a2ffd640 1 --2- 172.21.15.88:0/3959447430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34a406a970 0x7f34a41f5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:43:12.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.633+0000 7f34a37fe640 1 --2- 172.21.15.88:0/3959447430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34a415a810 0x7f34a41f1ed0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f349c00c780 tx=0x7f349c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:12.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.633+0000 7f349bfff640 1 -- 172.21.15.88:0/3959447430 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c00ce90 con 0x7f34a415a810 2023-10-16T20:43:12.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.633+0000 7f349bfff640 1 -- 172.21.15.88:0/3959447430 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349c00a640 con 0x7f34a415a810 2023-10-16T20:43:12.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.633+0000 7f34a97d9640 1 -- 172.21.15.88:0/3959447430 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a4071360 con 0x7f34a415a810 2023-10-16T20:43:12.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.633+0000 7f349bfff640 1 -- 172.21.15.88:0/3959447430 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c00a9d0 con 0x7f34a415a810 2023-10-16T20:43:12.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.633+0000 7f34a97d9640 1 -- 172.21.15.88:0/3959447430 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a4070090 con 0x7f34a415a810 2023-10-16T20:43:12.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.633+0000 7f34a97d9640 1 -- 172.21.15.88:0/3959447430 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a4074700 con 0x7f34a415a810 2023-10-16T20:43:12.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.633+0000 7f349bfff640 1 -- 172.21.15.88:0/3959447430 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f349c00ab30 con 0x7f34a415a810 2023-10-16T20:43:12.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.633+0000 7f349bfff640 1 --2- 172.21.15.88:0/3959447430 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34840664a0 0x7f3484068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:12.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.637+0000 7f349bfff640 1 -- 172.21.15.88:0/3959447430 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f349c083a00 con 0x7f34a415a810 2023-10-16T20:43:12.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.637+0000 7f34a2ffd640 1 --2- 172.21.15.88:0/3959447430 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34840664a0 0x7f3484068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:12.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.637+0000 7f34a2ffd640 1 --2- 172.21.15.88:0/3959447430 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34840664a0 0x7f3484068960 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f348c0af000 tx=0x7f348c0a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:12.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.641+0000 7f349bfff640 1 -- 172.21.15.88:0/3959447430 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f349c050a60 con 0x7f34a415a810 2023-10-16T20:43:12.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.825+0000 7f34a97d9640 1 -- 172.21.15.88:0/3959447430 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34a406af30 con 0x7f34a415a810 2023-10-16T20:43:12.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.825+0000 7f349bfff640 1 -- 172.21.15.88:0/3959447430 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6568 (secure 0 0 0) 0x7f349c018090 con 0x7f34a415a810 2023-10-16T20:43:12.828 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:12.828 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:31.455971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:active","state_seq":89,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:12.829 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 29 2023-10-16T20:43:12.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.825+0000 7f34a97d9640 1 -- 172.21.15.88:0/3959447430 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34840664a0 msgr2=0x7f3484068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:12.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.825+0000 7f34a97d9640 1 --2- 172.21.15.88:0/3959447430 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34840664a0 0x7f3484068960 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f348c0af000 tx=0x7f348c0a7000 comp rx=0 tx=0).stop 2023-10-16T20:43:12.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.825+0000 7f34a97d9640 1 -- 172.21.15.88:0/3959447430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34a415a810 msgr2=0x7f34a41f1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:12.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.825+0000 7f34a97d9640 1 --2- 172.21.15.88:0/3959447430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34a415a810 0x7f34a41f1ed0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f349c00c780 tx=0x7f349c00cc40 comp rx=0 tx=0).stop 2023-10-16T20:43:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.825+0000 7f34a97d9640 1 -- 172.21.15.88:0/3959447430 shutdown_connections 2023-10-16T20:43:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.825+0000 7f34a97d9640 1 --2- 172.21.15.88:0/3959447430 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34840664a0 0x7f3484068960 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.825+0000 7f34a97d9640 1 --2- 172.21.15.88:0/3959447430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34a415a810 0x7f34a41f1ed0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.825+0000 7f34a97d9640 1 --2- 172.21.15.88:0/3959447430 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34a41406d0 0x7f34a40c9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.825+0000 7f34a97d9640 1 --2- 172.21.15.88:0/3959447430 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34a406a970 0x7f34a41f5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:12.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.825+0000 7f34a97d9640 1 -- 172.21.15.88:0/3959447430 >> 172.21.15.88:0/3959447430 conn(0x7f34a40cad50 msgr2=0x7f34a40c1360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:12.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.825+0000 7f34a97d9640 1 -- 172.21.15.88:0/3959447430 shutdown_connections 2023-10-16T20:43:12.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:12.825+0000 7f34a97d9640 1 -- 172.21.15.88:0/3959447430 wait complete. 2023-10-16T20:43:12.849 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-16T20:43:14.850 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:14.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe30565640 1 Processor -- start 2023-10-16T20:43:14.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe30565640 1 -- start start 2023-10-16T20:43:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe30565640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28153770 0x7efe28159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe30565640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2815a410 con 0x7efe280c9bb0 2023-10-16T20:43:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe30565640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe28153b50 con 0x7efe280cece0 2023-10-16T20:43:15.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe30565640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe280cc250 con 0x7efe28153770 2023-10-16T20:43:15.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2db00640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7efe280c9bb0 0x7efe280c3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:38934/0 (socket says 172.21.15.88:38934) 2023-10-16T20:43:15.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2db00640 1 -- 172.21.15.88:0/1601977828 learned_addr learned my addr 172.21.15.88:0/1601977828 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:43:15.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2e301640 1 --2- 172.21.15.88:0/1601977828 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28153770 0x7efe28159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:15.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2d2ff640 1 -- 172.21.15.88:0/1601977828 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2390944804 0 0) 0x7efe2815a410 con 0x7efe280c9bb0 2023-10-16T20:43:15.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2d2ff640 1 -- 172.21.15.88:0/1601977828 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe14003530 con 0x7efe280c9bb0 2023-10-16T20:43:15.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2d2ff640 1 -- 172.21.15.88:0/1601977828 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001227825 0 0) 0x7efe28153b50 con 0x7efe280cece0 2023-10-16T20:43:15.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2d2ff640 1 -- 172.21.15.88:0/1601977828 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe2815a410 con 0x7efe280cece0 2023-10-16T20:43:15.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2d2ff640 1 -- 172.21.15.88:0/1601977828 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 797799239 0 0) 0x7efe14003530 con 0x7efe280c9bb0 2023-10-16T20:43:15.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2d2ff640 1 -- 172.21.15.88:0/1601977828 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28153770 msgr2=0x7efe28159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:15.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2d2ff640 1 --2- 172.21.15.88:0/1601977828 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28153770 0x7efe28159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2d2ff640 1 -- 172.21.15.88:0/1601977828 >> v1:172.21.15.191:6789/0 conn(0x7efe280cece0 legacy=0x7efe28163dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2d2ff640 1 -- 172.21.15.88:0/1601977828 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe280cb2f0 con 0x7efe280c9bb0 2023-10-16T20:43:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2d2ff640 1 -- 172.21.15.88:0/1601977828 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7efe18001680 con 0x7efe280c9bb0 2023-10-16T20:43:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2d2ff640 1 -- 172.21.15.88:0/1601977828 >> v1:172.21.15.88:6789/0 conn(0x7efe280c9bb0 legacy=0x7efe280c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2d2ff640 1 --2- 172.21.15.88:0/1601977828 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe14003b40 0x7efe14003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:15.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2d2ff640 1 --2- 172.21.15.88:0/1601977828 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28153770 0x7efe140047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:15.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2d2ff640 1 --2- 172.21.15.88:0/1601977828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe14004d10 0x7efe14005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:15.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2d2ff640 1 -- 172.21.15.88:0/1601977828 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe280cc250 con 0x7efe14004d10 2023-10-16T20:43:15.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2d2ff640 1 -- 172.21.15.88:0/1601977828 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1400ba40 con 0x7efe14003b40 2023-10-16T20:43:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2d2ff640 1 -- 172.21.15.88:0/1601977828 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1400bbb0 con 0x7efe28153770 2023-10-16T20:43:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2eb02640 1 --2- 172.21.15.88:0/1601977828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe14004d10 0x7efe14005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2e301640 1 --2- 172.21.15.88:0/1601977828 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe14003b40 0x7efe14003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2db00640 1 --2- 172.21.15.88:0/1601977828 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28153770 0x7efe140047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2e301640 1 -- 172.21.15.88:0/1601977828 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28153770 msgr2=0x7efe140047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:15.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2e301640 1 --2- 172.21.15.88:0/1601977828 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28153770 0x7efe140047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:15.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2e301640 1 -- 172.21.15.88:0/1601977828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe14004d10 msgr2=0x7efe14005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:15.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2e301640 1 --2- 172.21.15.88:0/1601977828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe14004d10 0x7efe14005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:15.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2e301640 1 -- 172.21.15.88:0/1601977828 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe14003530 con 0x7efe14003b40 2023-10-16T20:43:15.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2db00640 1 --2- 172.21.15.88:0/1601977828 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe28153770 0x7efe140047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:43:15.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:14.997+0000 7efe2eb02640 1 --2- 172.21.15.88:0/1601977828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe14004d10 0x7efe14005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:43:15.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.001+0000 7efe2e301640 1 --2- 172.21.15.88:0/1601977828 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe14003b40 0x7efe14003f40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efe24009ac0 tx=0x7efe2401ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d059d15c30209c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:15.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.001+0000 7efe2d2ff640 1 -- 172.21.15.88:0/1601977828 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe240059a0 con 0x7efe14003b40 2023-10-16T20:43:15.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.001+0000 7efe2d2ff640 1 -- 172.21.15.88:0/1601977828 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe24005b00 con 0x7efe14003b40 2023-10-16T20:43:15.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.001+0000 7efe2d2ff640 1 -- 172.21.15.88:0/1601977828 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe24006030 con 0x7efe14003b40 2023-10-16T20:43:15.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.001+0000 7efe30565640 1 -- 172.21.15.88:0/1601977828 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe14003b40 msgr2=0x7efe14003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:15.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.001+0000 7efe30565640 1 --2- 172.21.15.88:0/1601977828 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe14003b40 0x7efe14003f40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efe24009ac0 tx=0x7efe2401ca30 comp rx=0 tx=0).stop 2023-10-16T20:43:15.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.001+0000 7efe2eb02640 1 -- 172.21.15.88:0/1601977828 reap_dead start 2023-10-16T20:43:15.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.001+0000 7efe30565640 1 -- 172.21.15.88:0/1601977828 shutdown_connections 2023-10-16T20:43:15.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.001+0000 7efe30565640 1 -- 172.21.15.88:0/1601977828 >> 172.21.15.88:0/1601977828 conn(0x7efe280715c0 msgr2=0x7efe280719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:15.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.001+0000 7efe30565640 1 -- 172.21.15.88:0/1601977828 shutdown_connections 2023-10-16T20:43:15.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.001+0000 7efe30565640 1 -- 172.21.15.88:0/1601977828 wait complete. 2023-10-16T20:43:15.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe30565640 1 Processor -- start 2023-10-16T20:43:15.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe30565640 1 -- start start 2023-10-16T20:43:15.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe30565640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe14003b40 0x7efe280c5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:15.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe30565640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe280c8e40 0x7efe28144bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:15.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe30565640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe281e5420 0x7efe28145110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:15.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe30565640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe280b3d80 con 0x7efe280c8e40 2023-10-16T20:43:15.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe30565640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe281f3030 con 0x7efe281e5420 2023-10-16T20:43:15.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe30565640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe28148e90 con 0x7efe14003b40 2023-10-16T20:43:15.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe2db00640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe280c8e40 0x7efe28144bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:15.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe2db00640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe280c8e40 0x7efe28144bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39966/0 (socket says 172.21.15.88:39966) 2023-10-16T20:43:15.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe2db00640 1 -- 172.21.15.88:0/637298013 learned_addr learned my addr 172.21.15.88:0/637298013 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:15.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe2e301640 1 --2- 172.21.15.88:0/637298013 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe14003b40 0x7efe280c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:15.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe2eb02640 1 --2- 172.21.15.88:0/637298013 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe281e5420 0x7efe28145110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:15.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe2db00640 1 -- 172.21.15.88:0/637298013 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe14003b40 msgr2=0x7efe280c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:15.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe2db00640 1 --2- 172.21.15.88:0/637298013 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe14003b40 0x7efe280c5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:15.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe2db00640 1 -- 172.21.15.88:0/637298013 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe281e5420 msgr2=0x7efe28145110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:15.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe2db00640 1 --2- 172.21.15.88:0/637298013 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe281e5420 0x7efe28145110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:15.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe2db00640 1 -- 172.21.15.88:0/637298013 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe281f2ad0 con 0x7efe280c8e40 2023-10-16T20:43:15.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe2db00640 1 --2- 172.21.15.88:0/637298013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe280c8e40 0x7efe28144bd0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7efe18000fc0 tx=0x7efe18011c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:15.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe1effd640 1 -- 172.21.15.88:0/637298013 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1800fb60 con 0x7efe280c8e40 2023-10-16T20:43:15.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe1effd640 1 -- 172.21.15.88:0/637298013 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1800fcc0 con 0x7efe280c8e40 2023-10-16T20:43:15.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe30565640 1 -- 172.21.15.88:0/637298013 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe280b3420 con 0x7efe280c8e40 2023-10-16T20:43:15.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe1effd640 1 -- 172.21.15.88:0/637298013 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe18015610 con 0x7efe280c8e40 2023-10-16T20:43:15.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.005+0000 7efe30565640 1 -- 172.21.15.88:0/637298013 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe280b36c0 con 0x7efe280c8e40 2023-10-16T20:43:15.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.009+0000 7efe30565640 1 -- 172.21.15.88:0/637298013 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe280b15d0 con 0x7efe280c8e40 2023-10-16T20:43:15.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.009+0000 7efe1effd640 1 -- 172.21.15.88:0/637298013 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efe18015770 con 0x7efe280c8e40 2023-10-16T20:43:15.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.009+0000 7efe1effd640 1 --2- 172.21.15.88:0/637298013 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efe0c0664d0 0x7efe0c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:15.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.009+0000 7efe1effd640 1 -- 172.21.15.88:0/637298013 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7efe18087450 con 0x7efe280c8e40 2023-10-16T20:43:15.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.009+0000 7efe2e301640 1 --2- 172.21.15.88:0/637298013 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efe0c0664d0 0x7efe0c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:15.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.009+0000 7efe2e301640 1 --2- 172.21.15.88:0/637298013 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efe0c0664d0 0x7efe0c068990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efe240ae000 tx=0x7efe240046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:15.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.013+0000 7efe1effd640 1 -- 172.21.15.88:0/637298013 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efe18054410 con 0x7efe280c8e40 2023-10-16T20:43:15.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.193+0000 7efe30565640 1 -- 172.21.15.88:0/637298013 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe28074700 con 0x7efe280c8e40 2023-10-16T20:43:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.193+0000 7efe1effd640 1 -- 172.21.15.88:0/637298013 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6568 (secure 0 0 0) 0x7efe18057a30 con 0x7efe280c8e40 2023-10-16T20:43:15.197 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:15.198 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:31.455971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:active","state_seq":89,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 29 2023-10-16T20:43:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.197+0000 7efe30565640 1 -- 172.21.15.88:0/637298013 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efe0c0664d0 msgr2=0x7efe0c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.197+0000 7efe30565640 1 --2- 172.21.15.88:0/637298013 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efe0c0664d0 0x7efe0c068990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efe240ae000 tx=0x7efe240046d0 comp rx=0 tx=0).stop 2023-10-16T20:43:15.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.197+0000 7efe30565640 1 -- 172.21.15.88:0/637298013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe280c8e40 msgr2=0x7efe28144bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:15.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.197+0000 7efe30565640 1 --2- 172.21.15.88:0/637298013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe280c8e40 0x7efe28144bd0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7efe18000fc0 tx=0x7efe18011c00 comp rx=0 tx=0).stop 2023-10-16T20:43:15.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.197+0000 7efe30565640 1 -- 172.21.15.88:0/637298013 shutdown_connections 2023-10-16T20:43:15.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.197+0000 7efe30565640 1 --2- 172.21.15.88:0/637298013 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efe0c0664d0 0x7efe0c068990 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:15.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.197+0000 7efe30565640 1 --2- 172.21.15.88:0/637298013 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efe281e5420 0x7efe28145110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:15.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.197+0000 7efe30565640 1 --2- 172.21.15.88:0/637298013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe280c8e40 0x7efe28144bd0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:15.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.197+0000 7efe30565640 1 --2- 172.21.15.88:0/637298013 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efe14003b40 0x7efe280c5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:15.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.197+0000 7efe30565640 1 -- 172.21.15.88:0/637298013 >> 172.21.15.88:0/637298013 conn(0x7efe280715c0 msgr2=0x7efe280cdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:15.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.197+0000 7efe30565640 1 -- 172.21.15.88:0/637298013 shutdown_connections 2023-10-16T20:43:15.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:15.197+0000 7efe30565640 1 -- 172.21.15.88:0/637298013 wait complete. 2023-10-16T20:43:15.218 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-16T20:43:17.220 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:17.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.365+0000 7fb02e1d9640 1 Processor -- start 2023-10-16T20:43:17.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.365+0000 7fb02e1d9640 1 -- start start 2023-10-16T20:43:17.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.369+0000 7fb02e1d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0280c7dc0 0x7fb028159450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:17.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.369+0000 7fb02e1d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb0280c81a0 0x7fb02806b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:17.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.369+0000 7fb02e1d9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0280c8810 con 0x7fb0280c76c0 2023-10-16T20:43:17.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.369+0000 7fb02e1d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02806b400 con 0x7fb0280c81a0 2023-10-16T20:43:17.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.369+0000 7fb02e1d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02806b0d0 con 0x7fb0280c7dc0 2023-10-16T20:43:17.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.369+0000 7fb027fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb0280c81a0 0x7fb02806b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:17.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.369+0000 7fb027fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb0280c81a0 0x7fb02806b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39968/0 (socket says 172.21.15.88:39968) 2023-10-16T20:43:17.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.369+0000 7fb027fff640 1 -- 172.21.15.88:0/185524212 learned_addr learned my addr 172.21.15.88:0/185524212 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:17.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.369+0000 7fb0277fe640 1 --2- 172.21.15.88:0/185524212 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0280c7dc0 0x7fb028159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:17.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.369+0000 7fb027fff640 1 -- 172.21.15.88:0/185524212 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0280c7dc0 msgr2=0x7fb028159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:17.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.369+0000 7fb027fff640 1 --2- 172.21.15.88:0/185524212 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0280c7dc0 0x7fb028159450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:17.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.369+0000 7fb027fff640 1 -- 172.21.15.88:0/185524212 >> v1:172.21.15.191:6789/0 conn(0x7fb0280c76c0 legacy=0x7fb02806d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:17.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.369+0000 7fb027fff640 1 -- 172.21.15.88:0/185524212 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02806e6a0 con 0x7fb0280c81a0 2023-10-16T20:43:17.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.373+0000 7fb027fff640 1 --2- 172.21.15.88:0/185524212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0280c81a0 0x7fb02806b890 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb020009e50 tx=0x7fb02001c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6079c5f63381c650 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:17.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.373+0000 7fb0267fc640 1 -- 172.21.15.88:0/185524212 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0200a5640 con 0x7fb0280c81a0 2023-10-16T20:43:17.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.373+0000 7fb0267fc640 1 -- 172.21.15.88:0/185524212 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0200a57a0 con 0x7fb0280c81a0 2023-10-16T20:43:17.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.373+0000 7fb0267fc640 1 -- 172.21.15.88:0/185524212 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0200a5b30 con 0x7fb0280c81a0 2023-10-16T20:43:17.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.373+0000 7fb02e1d9640 1 -- 172.21.15.88:0/185524212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0280c81a0 msgr2=0x7fb02806b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:17.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.373+0000 7fb02e1d9640 1 --2- 172.21.15.88:0/185524212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0280c81a0 0x7fb02806b890 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb020009e50 tx=0x7fb02001c980 comp rx=0 tx=0).stop 2023-10-16T20:43:17.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.373+0000 7fb02e1d9640 1 -- 172.21.15.88:0/185524212 shutdown_connections 2023-10-16T20:43:17.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.373+0000 7fb02e1d9640 1 --2- 172.21.15.88:0/185524212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0280c81a0 0x7fb02806b890 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:17.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.373+0000 7fb02e1d9640 1 --2- 172.21.15.88:0/185524212 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0280c7dc0 0x7fb028159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:17.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.373+0000 7fb02e1d9640 1 -- 172.21.15.88:0/185524212 >> 172.21.15.88:0/185524212 conn(0x7fb028071a60 msgr2=0x7fb028071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:17.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.373+0000 7fb02e1d9640 1 -- 172.21.15.88:0/185524212 shutdown_connections 2023-10-16T20:43:17.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.373+0000 7fb02e1d9640 1 -- 172.21.15.88:0/185524212 wait complete. 2023-10-16T20:43:17.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.373+0000 7fb02e1d9640 1 Processor -- start 2023-10-16T20:43:17.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.373+0000 7fb02e1d9640 1 -- start start 2023-10-16T20:43:17.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb02e1d9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0280c76c0 0x7fb028145cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:17.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb02e1d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0280c7dc0 0x7fb0280c4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:17.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb02e1d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0280c81a0 0x7fb02813eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:17.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb02e1d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb028142d20 con 0x7fb0280c81a0 2023-10-16T20:43:17.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb02e1d9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0280c5b60 con 0x7fb0280c76c0 2023-10-16T20:43:17.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb02e1d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb028143b60 con 0x7fb0280c7dc0 2023-10-16T20:43:17.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb027fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0280c81a0 0x7fb02813eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:17.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb027fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0280c81a0 0x7fb02813eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39972/0 (socket says 172.21.15.88:39972) 2023-10-16T20:43:17.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb027fff640 1 -- 172.21.15.88:0/965525222 learned_addr learned my addr 172.21.15.88:0/965525222 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:17.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb026ffd640 1 --2- 172.21.15.88:0/965525222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0280c7dc0 0x7fb0280c4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:17.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb0277fe640 1 --2- 172.21.15.88:0/965525222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0280c76c0 0x7fb028145cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:17.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb027fff640 1 -- 172.21.15.88:0/965525222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0280c7dc0 msgr2=0x7fb0280c4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:17.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb027fff640 1 --2- 172.21.15.88:0/965525222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0280c7dc0 0x7fb0280c4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:17.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb027fff640 1 -- 172.21.15.88:0/965525222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0280c76c0 msgr2=0x7fb028145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:17.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb027fff640 1 --2- 172.21.15.88:0/965525222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0280c76c0 0x7fb028145cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:17.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb027fff640 1 -- 172.21.15.88:0/965525222 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb020009b30 con 0x7fb0280c81a0 2023-10-16T20:43:17.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb0277fe640 1 --2- 172.21.15.88:0/965525222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0280c76c0 0x7fb028145cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:43:17.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb027fff640 1 --2- 172.21.15.88:0/965525222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0280c81a0 0x7fb02813eb00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb0200af040 tx=0x7fb020019720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:17.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb00ffff640 1 -- 172.21.15.88:0/965525222 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0200b8670 con 0x7fb0280c81a0 2023-10-16T20:43:17.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb00ffff640 1 -- 172.21.15.88:0/965525222 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0200b87d0 con 0x7fb0280c81a0 2023-10-16T20:43:17.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb02e1d9640 1 -- 172.21.15.88:0/965525222 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb028143e40 con 0x7fb0280c81a0 2023-10-16T20:43:17.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb00ffff640 1 -- 172.21.15.88:0/965525222 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0200b8b80 con 0x7fb0280c81a0 2023-10-16T20:43:17.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb02e1d9640 1 -- 172.21.15.88:0/965525222 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0280c5fe0 con 0x7fb0280c81a0 2023-10-16T20:43:17.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb02e1d9640 1 -- 172.21.15.88:0/965525222 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb028143fd0 con 0x7fb0280c81a0 2023-10-16T20:43:17.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.377+0000 7fb00ffff640 1 -- 172.21.15.88:0/965525222 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb0200b8ce0 con 0x7fb0280c81a0 2023-10-16T20:43:17.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.381+0000 7fb00ffff640 1 --2- 172.21.15.88:0/965525222 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb0140664d0 0x7fb014068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:17.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.381+0000 7fb00ffff640 1 -- 172.21.15.88:0/965525222 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7fb02011f200 con 0x7fb0280c81a0 2023-10-16T20:43:17.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.381+0000 7fb0277fe640 1 --2- 172.21.15.88:0/965525222 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb0140664d0 0x7fb014068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:17.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.381+0000 7fb0277fe640 1 --2- 172.21.15.88:0/965525222 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb0140664d0 0x7fb014068990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb028147000 tx=0x7fb018008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:17.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.385+0000 7fb00ffff640 1 -- 172.21.15.88:0/965525222 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb0200ec1d0 con 0x7fb0280c81a0 2023-10-16T20:43:17.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.569+0000 7fb02e1d9640 1 -- 172.21.15.88:0/965525222 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb028074700 con 0x7fb0280c81a0 2023-10-16T20:43:17.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.569+0000 7fb00ffff640 1 -- 172.21.15.88:0/965525222 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6568 (secure 0 0 0) 0x7fb020009510 con 0x7fb0280c81a0 2023-10-16T20:43:17.572 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:17.572 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:31.455971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:active","state_seq":89,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:17.572 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 29 2023-10-16T20:43:17.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.569+0000 7fb02e1d9640 1 -- 172.21.15.88:0/965525222 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb0140664d0 msgr2=0x7fb014068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:17.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.569+0000 7fb02e1d9640 1 --2- 172.21.15.88:0/965525222 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb0140664d0 0x7fb014068990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb028147000 tx=0x7fb018008040 comp rx=0 tx=0).stop 2023-10-16T20:43:17.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.569+0000 7fb02e1d9640 1 -- 172.21.15.88:0/965525222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0280c81a0 msgr2=0x7fb02813eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:17.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.569+0000 7fb02e1d9640 1 --2- 172.21.15.88:0/965525222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0280c81a0 0x7fb02813eb00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb0200af040 tx=0x7fb020019720 comp rx=0 tx=0).stop 2023-10-16T20:43:17.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.569+0000 7fb02e1d9640 1 -- 172.21.15.88:0/965525222 shutdown_connections 2023-10-16T20:43:17.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.569+0000 7fb02e1d9640 1 --2- 172.21.15.88:0/965525222 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb0140664d0 0x7fb014068990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:17.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.569+0000 7fb02e1d9640 1 --2- 172.21.15.88:0/965525222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0280c81a0 0x7fb02813eb00 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:17.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.569+0000 7fb02e1d9640 1 --2- 172.21.15.88:0/965525222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb0280c7dc0 0x7fb0280c4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:17.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.569+0000 7fb02e1d9640 1 --2- 172.21.15.88:0/965525222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb0280c76c0 0x7fb028145cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:17.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.569+0000 7fb02e1d9640 1 -- 172.21.15.88:0/965525222 >> 172.21.15.88:0/965525222 conn(0x7fb028071a60 msgr2=0x7fb028150440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:17.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.569+0000 7fb02e1d9640 1 -- 172.21.15.88:0/965525222 shutdown_connections 2023-10-16T20:43:17.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:17.569+0000 7fb02e1d9640 1 -- 172.21.15.88:0/965525222 wait complete. 2023-10-16T20:43:17.598 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-16T20:43:19.599 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:19.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f1976891640 1 Processor -- start 2023-10-16T20:43:19.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f1976891640 1 -- start start 2023-10-16T20:43:19.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f1976891640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1970153770 0x7f1970159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:19.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f1976891640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f19700c9bb0 0x7f19700c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:19.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f1976891640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197015a410 con 0x7f19700c41f0 2023-10-16T20:43:19.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f1976891640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19700cc250 con 0x7f19700c9bb0 2023-10-16T20:43:19.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f1976891640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19700cc670 con 0x7f1970153770 2023-10-16T20:43:19.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f1974e2e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f19700c41f0 0x7f19700cece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47638/0 (socket says 172.21.15.88:47638) 2023-10-16T20:43:19.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f1974e2e640 1 -- 172.21.15.88:0/2126349268 learned_addr learned my addr 172.21.15.88:0/2126349268 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:43:19.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f196ffff640 1 --2- 172.21.15.88:0/2126349268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1970153770 0x7f1970159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:19.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f196f7fe640 1 --2- 172.21.15.88:0/2126349268 >> v2:172.21.15.191:3300/0 conn(0x7f19700c9bb0 0x7f19700c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:19.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f196effd640 1 -- 172.21.15.88:0/2126349268 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591348409 0 0) 0x7f197015a410 con 0x7f19700c41f0 2023-10-16T20:43:19.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f196effd640 1 -- 172.21.15.88:0/2126349268 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1958003530 con 0x7f19700c41f0 2023-10-16T20:43:19.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f196ffff640 1 -- 172.21.15.88:0/2126349268 >> v2:172.21.15.191:3300/0 conn(0x7f19700c9bb0 msgr2=0x7f19700c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:19.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f196ffff640 1 --2- 172.21.15.88:0/2126349268 >> v2:172.21.15.191:3300/0 conn(0x7f19700c9bb0 0x7f19700c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:19.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f196ffff640 1 -- 172.21.15.88:0/2126349268 >> v1:172.21.15.88:6789/0 conn(0x7f19700c41f0 legacy=0x7f19700cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:19.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f196ffff640 1 -- 172.21.15.88:0/2126349268 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1970153b50 con 0x7f1970153770 2023-10-16T20:43:19.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.745+0000 7f196f7fe640 1 --2- 172.21.15.88:0/2126349268 >> v2:172.21.15.191:3300/0 conn(0x7f19700c9bb0 0x7f19700c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:43:19.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.749+0000 7f196ffff640 1 --2- 172.21.15.88:0/2126349268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1970153770 0x7f1970159ed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f19600097a0 tx=0x7f196001c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f48d50ab014530d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:19.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.749+0000 7f196effd640 1 -- 172.21.15.88:0/2126349268 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19600ab600 con 0x7f1970153770 2023-10-16T20:43:19.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.749+0000 7f196effd640 1 -- 172.21.15.88:0/2126349268 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19600a7050 con 0x7f1970153770 2023-10-16T20:43:19.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.749+0000 7f196effd640 1 -- 172.21.15.88:0/2126349268 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19600ab980 con 0x7f1970153770 2023-10-16T20:43:19.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.749+0000 7f1976891640 1 -- 172.21.15.88:0/2126349268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1970153770 msgr2=0x7f1970159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:19.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.749+0000 7f1976891640 1 --2- 172.21.15.88:0/2126349268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1970153770 0x7f1970159ed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f19600097a0 tx=0x7f196001c460 comp rx=0 tx=0).stop 2023-10-16T20:43:19.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.749+0000 7f1976891640 1 -- 172.21.15.88:0/2126349268 shutdown_connections 2023-10-16T20:43:19.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.749+0000 7f1976891640 1 --2- 172.21.15.88:0/2126349268 >> v2:172.21.15.191:3300/0 conn(0x7f19700c9bb0 0x7f19700c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:19.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.749+0000 7f1976891640 1 --2- 172.21.15.88:0/2126349268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1970153770 0x7f1970159ed0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:19.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.749+0000 7f1976891640 1 -- 172.21.15.88:0/2126349268 >> 172.21.15.88:0/2126349268 conn(0x7f19700715c0 msgr2=0x7f19700719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:19.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.749+0000 7f1976891640 1 -- 172.21.15.88:0/2126349268 shutdown_connections 2023-10-16T20:43:19.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f1976891640 1 -- 172.21.15.88:0/2126349268 wait complete. 2023-10-16T20:43:19.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f1976891640 1 Processor -- start 2023-10-16T20:43:19.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f1976891640 1 -- start start 2023-10-16T20:43:19.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f1976891640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19700c41f0 0x7f19700c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:19.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f1976891640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19700c9bb0 0x7f1970142cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:19.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f1976891640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1970153770 0x7f1970143230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:19.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f1976891640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197013fd20 con 0x7f19700c9bb0 2023-10-16T20:43:19.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f1976891640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197013ee90 con 0x7f19700c41f0 2023-10-16T20:43:19.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f1976891640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1970140490 con 0x7f1970153770 2023-10-16T20:43:19.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f196f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19700c9bb0 0x7f1970142cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:19.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f196f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19700c9bb0 0x7f1970142cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37078/0 (socket says 172.21.15.88:37078) 2023-10-16T20:43:19.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f196f7fe640 1 -- 172.21.15.88:0/2779358541 learned_addr learned my addr 172.21.15.88:0/2779358541 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:19.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f1974e2e640 1 --2- 172.21.15.88:0/2779358541 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1970153770 0x7f1970143230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:19.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f196ffff640 1 --2- 172.21.15.88:0/2779358541 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19700c41f0 0x7f19700c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:19.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f196f7fe640 1 -- 172.21.15.88:0/2779358541 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1970153770 msgr2=0x7f1970143230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:19.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f196f7fe640 1 --2- 172.21.15.88:0/2779358541 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1970153770 0x7f1970143230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:19.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f196f7fe640 1 -- 172.21.15.88:0/2779358541 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19700c41f0 msgr2=0x7f19700c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:19.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f196f7fe640 1 --2- 172.21.15.88:0/2779358541 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19700c41f0 0x7f19700c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:19.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f196f7fe640 1 -- 172.21.15.88:0/2779358541 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1964009420 con 0x7f19700c9bb0 2023-10-16T20:43:19.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f196ffff640 1 --2- 172.21.15.88:0/2779358541 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19700c41f0 0x7f19700c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:43:19.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.753+0000 7f196f7fe640 1 --2- 172.21.15.88:0/2779358541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19700c9bb0 0x7f1970142cf0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f196400ec10 tx=0x7f196400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:19.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.757+0000 7f196cff9640 1 -- 172.21.15.88:0/2779358541 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196400cc30 con 0x7f19700c9bb0 2023-10-16T20:43:19.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.757+0000 7f196cff9640 1 -- 172.21.15.88:0/2779358541 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196400cd90 con 0x7f19700c9bb0 2023-10-16T20:43:19.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.757+0000 7f196cff9640 1 -- 172.21.15.88:0/2779358541 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1964013690 con 0x7f19700c9bb0 2023-10-16T20:43:19.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.757+0000 7f1976891640 1 -- 172.21.15.88:0/2779358541 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19600093b0 con 0x7f19700c9bb0 2023-10-16T20:43:19.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.757+0000 7f1976891640 1 -- 172.21.15.88:0/2779358541 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19701422f0 con 0x7f19700c9bb0 2023-10-16T20:43:19.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.757+0000 7f1976891640 1 -- 172.21.15.88:0/2779358541 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1970074700 con 0x7f19700c9bb0 2023-10-16T20:43:19.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.757+0000 7f196cff9640 1 -- 172.21.15.88:0/2779358541 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f19640137f0 con 0x7f19700c9bb0 2023-10-16T20:43:19.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.757+0000 7f196cff9640 1 --2- 172.21.15.88:0/2779358541 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f195c0664a0 0x7f195c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:19.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.757+0000 7f196cff9640 1 -- 172.21.15.88:0/2779358541 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f19640849c0 con 0x7f19700c9bb0 2023-10-16T20:43:19.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.757+0000 7f196ffff640 1 --2- 172.21.15.88:0/2779358541 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f195c0664a0 0x7f195c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:19.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.761+0000 7f196ffff640 1 --2- 172.21.15.88:0/2779358541 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f195c0664a0 0x7f195c068960 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f19600ad000 tx=0x7f19600a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:19.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.761+0000 7f196cff9640 1 -- 172.21.15.88:0/2779358541 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f19640519b0 con 0x7f19700c9bb0 2023-10-16T20:43:19.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.941+0000 7f1976891640 1 -- 172.21.15.88:0/2779358541 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19700ca740 con 0x7f19700c9bb0 2023-10-16T20:43:19.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.941+0000 7f196cff9640 1 -- 172.21.15.88:0/2779358541 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6568 (secure 0 0 0) 0x7f1964017090 con 0x7f19700c9bb0 2023-10-16T20:43:19.946 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:19.946 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:31.455971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:active","state_seq":89,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:19.947 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 29 2023-10-16T20:43:19.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.945+0000 7f1976891640 1 -- 172.21.15.88:0/2779358541 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f195c0664a0 msgr2=0x7f195c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:19.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.945+0000 7f1976891640 1 --2- 172.21.15.88:0/2779358541 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f195c0664a0 0x7f195c068960 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f19600ad000 tx=0x7f19600a6000 comp rx=0 tx=0).stop 2023-10-16T20:43:19.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.945+0000 7f1976891640 1 -- 172.21.15.88:0/2779358541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19700c9bb0 msgr2=0x7f1970142cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:19.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.945+0000 7f1976891640 1 --2- 172.21.15.88:0/2779358541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19700c9bb0 0x7f1970142cf0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f196400ec10 tx=0x7f196400c520 comp rx=0 tx=0).stop 2023-10-16T20:43:19.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.945+0000 7f1976891640 1 -- 172.21.15.88:0/2779358541 shutdown_connections 2023-10-16T20:43:19.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.945+0000 7f1976891640 1 --2- 172.21.15.88:0/2779358541 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f195c0664a0 0x7f195c068960 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:19.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.945+0000 7f1976891640 1 --2- 172.21.15.88:0/2779358541 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1970153770 0x7f1970143230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:19.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.945+0000 7f1976891640 1 --2- 172.21.15.88:0/2779358541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19700c9bb0 0x7f1970142cf0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:19.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.945+0000 7f1976891640 1 --2- 172.21.15.88:0/2779358541 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19700c41f0 0x7f19700c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:19.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.945+0000 7f1976891640 1 -- 172.21.15.88:0/2779358541 >> 172.21.15.88:0/2779358541 conn(0x7f19700715c0 msgr2=0x7f1970147300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:19.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.945+0000 7f1976891640 1 -- 172.21.15.88:0/2779358541 shutdown_connections 2023-10-16T20:43:19.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:19.945+0000 7f1976891640 1 -- 172.21.15.88:0/2779358541 wait complete. 2023-10-16T20:43:19.967 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-16T20:43:21.967 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:22.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.113+0000 7f9d3cc17640 1 Processor -- start 2023-10-16T20:43:22.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.113+0000 7f9d3cc17640 1 -- start start 2023-10-16T20:43:22.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.113+0000 7f9d3cc17640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9d38153770 0x7f9d38159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:22.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.113+0000 7f9d3cc17640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d380cece0 0x7f9d380cf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:22.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.113+0000 7f9d3cc17640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d3815a410 con 0x7f9d380c9bb0 2023-10-16T20:43:22.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.113+0000 7f9d3cc17640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d380cc250 con 0x7f9d38153770 2023-10-16T20:43:22.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.113+0000 7f9d3cc17640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380cc670 con 0x7f9d380cece0 2023-10-16T20:43:22.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.113+0000 7f9d3659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9d38153770 0x7f9d38159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:22.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.113+0000 7f9d3659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9d38153770 0x7f9d38159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37094/0 (socket says 172.21.15.88:37094) 2023-10-16T20:43:22.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.113+0000 7f9d3659c640 1 -- 172.21.15.88:0/2764145483 learned_addr learned my addr 172.21.15.88:0/2764145483 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:22.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.113+0000 7f9d36d9d640 1 --2- 172.21.15.88:0/2764145483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d380cece0 0x7f9d380cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:22.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.113+0000 7f9d3659c640 1 -- 172.21.15.88:0/2764145483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d380cece0 msgr2=0x7f9d380cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:22.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.113+0000 7f9d3659c640 1 --2- 172.21.15.88:0/2764145483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d380cece0 0x7f9d380cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:22.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.113+0000 7f9d3659c640 1 -- 172.21.15.88:0/2764145483 >> v1:172.21.15.191:6789/0 conn(0x7f9d380c9bb0 legacy=0x7f9d380c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:22.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.113+0000 7f9d3659c640 1 -- 172.21.15.88:0/2764145483 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d38153b50 con 0x7f9d38153770 2023-10-16T20:43:22.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.113+0000 7f9d36d9d640 1 --2- 172.21.15.88:0/2764145483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d380cece0 0x7f9d380cf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:43:22.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.117+0000 7f9d3659c640 1 --2- 172.21.15.88:0/2764145483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38153770 0x7f9d38159ed0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9d28006fd0 tx=0x7f9d2801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc38afd2aa645f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:22.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.117+0000 7f9d35d9b640 1 -- 172.21.15.88:0/2764145483 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d280a6040 con 0x7f9d38153770 2023-10-16T20:43:22.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.117+0000 7f9d35d9b640 1 -- 172.21.15.88:0/2764145483 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d280aa600 con 0x7f9d38153770 2023-10-16T20:43:22.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.117+0000 7f9d35d9b640 1 -- 172.21.15.88:0/2764145483 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d280aa990 con 0x7f9d38153770 2023-10-16T20:43:22.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.117+0000 7f9d3cc17640 1 -- 172.21.15.88:0/2764145483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38153770 msgr2=0x7f9d38159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:22.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.117+0000 7f9d3cc17640 1 --2- 172.21.15.88:0/2764145483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38153770 0x7f9d38159ed0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9d28006fd0 tx=0x7f9d2801c4e0 comp rx=0 tx=0).stop 2023-10-16T20:43:22.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.117+0000 7f9d3cc17640 1 -- 172.21.15.88:0/2764145483 shutdown_connections 2023-10-16T20:43:22.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.117+0000 7f9d3cc17640 1 --2- 172.21.15.88:0/2764145483 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d380cece0 0x7f9d380cf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:22.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.117+0000 7f9d3cc17640 1 --2- 172.21.15.88:0/2764145483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38153770 0x7f9d38159ed0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:22.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.117+0000 7f9d3cc17640 1 -- 172.21.15.88:0/2764145483 >> 172.21.15.88:0/2764145483 conn(0x7f9d380715c0 msgr2=0x7f9d380719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:22.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.117+0000 7f9d3cc17640 1 -- 172.21.15.88:0/2764145483 shutdown_connections 2023-10-16T20:43:22.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.117+0000 7f9d3cc17640 1 -- 172.21.15.88:0/2764145483 wait complete. 2023-10-16T20:43:22.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d3cc17640 1 Processor -- start 2023-10-16T20:43:22.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d3cc17640 1 -- start start 2023-10-16T20:43:22.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d3cc17640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d380c9bb0 0x7f9d381f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:22.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d3cc17640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d380cece0 0x7f9d38150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:22.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d3cc17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38153770 0x7f9d38150540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:22.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d3cc17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3814ee70 con 0x7f9d38153770 2023-10-16T20:43:22.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d3cc17640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3814f150 con 0x7f9d380c9bb0 2023-10-16T20:43:22.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d3cc17640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3814f430 con 0x7f9d380cece0 2023-10-16T20:43:22.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d36d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38153770 0x7f9d38150540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:22.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d36d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38153770 0x7f9d38150540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37100/0 (socket says 172.21.15.88:37100) 2023-10-16T20:43:22.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d36d9d640 1 -- 172.21.15.88:0/2263158931 learned_addr learned my addr 172.21.15.88:0/2263158931 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:22.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d3659c640 1 --2- 172.21.15.88:0/2263158931 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d380c9bb0 0x7f9d381f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:22.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d2dd9b640 1 --2- 172.21.15.88:0/2263158931 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d380cece0 0x7f9d38150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:22.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d36d9d640 1 -- 172.21.15.88:0/2263158931 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d380cece0 msgr2=0x7f9d38150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:22.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d36d9d640 1 --2- 172.21.15.88:0/2263158931 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d380cece0 0x7f9d38150000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:22.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d36d9d640 1 -- 172.21.15.88:0/2263158931 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d380c9bb0 msgr2=0x7f9d381f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:22.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d36d9d640 1 --2- 172.21.15.88:0/2263158931 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d380c9bb0 0x7f9d381f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:22.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d36d9d640 1 -- 172.21.15.88:0/2263158931 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d28009480 con 0x7f9d38153770 2023-10-16T20:43:22.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d36d9d640 1 --2- 172.21.15.88:0/2263158931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38153770 0x7f9d38150540 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000cef0 tx=0x7f9d3000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:22.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d2f7fe640 1 -- 172.21.15.88:0/2263158931 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3000d840 con 0x7f9d38153770 2023-10-16T20:43:22.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d2f7fe640 1 -- 172.21.15.88:0/2263158931 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3000d9c0 con 0x7f9d38153770 2023-10-16T20:43:22.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d2f7fe640 1 -- 172.21.15.88:0/2263158931 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3000dd70 con 0x7f9d38153770 2023-10-16T20:43:22.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d3cc17640 1 -- 172.21.15.88:0/2263158931 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3814f710 con 0x7f9d38153770 2023-10-16T20:43:22.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.121+0000 7f9d3cc17640 1 -- 172.21.15.88:0/2263158931 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d38152130 con 0x7f9d38153770 2023-10-16T20:43:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.125+0000 7f9d3cc17640 1 -- 172.21.15.88:0/2263158931 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d381f1dc0 con 0x7f9d38153770 2023-10-16T20:43:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.125+0000 7f9d2f7fe640 1 -- 172.21.15.88:0/2263158931 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9d3001d360 con 0x7f9d38153770 2023-10-16T20:43:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.125+0000 7f9d2f7fe640 1 --2- 172.21.15.88:0/2263158931 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9d180664d0 0x7f9d18068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.125+0000 7f9d2f7fe640 1 -- 172.21.15.88:0/2263158931 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f9d30084dc0 con 0x7f9d38153770 2023-10-16T20:43:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.125+0000 7f9d3659c640 1 --2- 172.21.15.88:0/2263158931 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9d180664d0 0x7f9d18068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:22.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.125+0000 7f9d3659c640 1 --2- 172.21.15.88:0/2263158931 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9d180664d0 0x7f9d18068990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9d280ae000 tx=0x7f9d280a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:22.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.129+0000 7f9d2f7fe640 1 -- 172.21.15.88:0/2263158931 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9d381f1dc0 con 0x7f9d38153770 2023-10-16T20:43:22.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.309+0000 7f9d3cc17640 1 -- 172.21.15.88:0/2263158931 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d381f1dc0 con 0x7f9d38153770 2023-10-16T20:43:22.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.309+0000 7f9d2f7fe640 1 -- 172.21.15.88:0/2263158931 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6568 (secure 0 0 0) 0x7f9d381f1dc0 con 0x7f9d38153770 2023-10-16T20:43:22.312 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:22.313 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:31.455971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:active","state_seq":89,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:22.313 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 29 2023-10-16T20:43:22.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.309+0000 7f9d3cc17640 1 -- 172.21.15.88:0/2263158931 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9d180664d0 msgr2=0x7f9d18068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:22.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.309+0000 7f9d3cc17640 1 --2- 172.21.15.88:0/2263158931 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9d180664d0 0x7f9d18068990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9d280ae000 tx=0x7f9d280a7000 comp rx=0 tx=0).stop 2023-10-16T20:43:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.309+0000 7f9d3cc17640 1 -- 172.21.15.88:0/2263158931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38153770 msgr2=0x7f9d38150540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.309+0000 7f9d3cc17640 1 --2- 172.21.15.88:0/2263158931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38153770 0x7f9d38150540 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000cef0 tx=0x7f9d3000a710 comp rx=0 tx=0).stop 2023-10-16T20:43:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.309+0000 7f9d3cc17640 1 -- 172.21.15.88:0/2263158931 shutdown_connections 2023-10-16T20:43:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.309+0000 7f9d3cc17640 1 --2- 172.21.15.88:0/2263158931 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9d180664d0 0x7f9d18068990 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.309+0000 7f9d3cc17640 1 --2- 172.21.15.88:0/2263158931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38153770 0x7f9d38150540 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:22.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.309+0000 7f9d3cc17640 1 --2- 172.21.15.88:0/2263158931 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9d380cece0 0x7f9d38150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:22.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.309+0000 7f9d3cc17640 1 --2- 172.21.15.88:0/2263158931 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9d380c9bb0 0x7f9d381f20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:22.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.309+0000 7f9d3cc17640 1 -- 172.21.15.88:0/2263158931 >> 172.21.15.88:0/2263158931 conn(0x7f9d380715c0 msgr2=0x7f9d380719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:22.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.309+0000 7f9d3cc17640 1 -- 172.21.15.88:0/2263158931 shutdown_connections 2023-10-16T20:43:22.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:22.309+0000 7f9d3cc17640 1 -- 172.21.15.88:0/2263158931 wait complete. 2023-10-16T20:43:22.333 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-16T20:43:24.334 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:24.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.481+0000 7effacac9640 1 Processor -- start 2023-10-16T20:43:24.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.481+0000 7effacac9640 1 -- start start 2023-10-16T20:43:24.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.481+0000 7effacac9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7effa8153770 0x7effa8159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:24.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.481+0000 7effacac9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effa80c9bb0 0x7effa80c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:24.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.481+0000 7effacac9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7effa80c41f0 0x7effa80cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:24.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.481+0000 7effacac9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa80cc250 con 0x7effa80c41f0 2023-10-16T20:43:24.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.481+0000 7effacac9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa80cc670 con 0x7effa8153770 2023-10-16T20:43:24.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.481+0000 7effacac9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa815a410 con 0x7effa80c9bb0 2023-10-16T20:43:24.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.481+0000 7effa6d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7effa80c41f0 0x7effa80cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:24.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.481+0000 7effa6d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7effa80c41f0 0x7effa80cece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37110/0 (socket says 172.21.15.88:37110) 2023-10-16T20:43:24.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.481+0000 7effa6d9d640 1 -- 172.21.15.88:0/1418609476 learned_addr learned my addr 172.21.15.88:0/1418609476 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:24.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.481+0000 7effa6d9d640 1 -- 172.21.15.88:0/1418609476 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effa80c9bb0 msgr2=0x7effa80c3cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T20:43:24.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.481+0000 7effa6d9d640 1 --2- 172.21.15.88:0/1418609476 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effa80c9bb0 0x7effa80c3cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:24.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.481+0000 7effa6d9d640 1 -- 172.21.15.88:0/1418609476 >> v2:172.21.15.191:3300/0 conn(0x7effa8153770 msgr2=0x7effa8159ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-16T20:43:24.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.481+0000 7effa6d9d640 1 --2- 172.21.15.88:0/1418609476 >> v2:172.21.15.191:3300/0 conn(0x7effa8153770 0x7effa8159ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:24.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.481+0000 7effa6d9d640 1 -- 172.21.15.88:0/1418609476 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa8153b50 con 0x7effa80c41f0 2023-10-16T20:43:24.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.485+0000 7effa6d9d640 1 --2- 172.21.15.88:0/1418609476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effa80c41f0 0x7effa80cece0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7effa80cecb0 tx=0x7effa001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98aee971a43fe57 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:24.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.485+0000 7effa5d9b640 1 -- 172.21.15.88:0/1418609476 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa00a57a0 con 0x7effa80c41f0 2023-10-16T20:43:24.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.485+0000 7effa5d9b640 1 -- 172.21.15.88:0/1418609476 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa00a5900 con 0x7effa80c41f0 2023-10-16T20:43:24.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.485+0000 7effa5d9b640 1 -- 172.21.15.88:0/1418609476 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa00a5c90 con 0x7effa80c41f0 2023-10-16T20:43:24.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.485+0000 7effacac9640 1 -- 172.21.15.88:0/1418609476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effa80c41f0 msgr2=0x7effa80cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:24.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.485+0000 7effacac9640 1 --2- 172.21.15.88:0/1418609476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effa80c41f0 0x7effa80cece0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7effa80cecb0 tx=0x7effa001ca10 comp rx=0 tx=0).stop 2023-10-16T20:43:24.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.485+0000 7effacac9640 1 -- 172.21.15.88:0/1418609476 shutdown_connections 2023-10-16T20:43:24.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.485+0000 7effacac9640 1 --2- 172.21.15.88:0/1418609476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effa80c41f0 0x7effa80cece0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:24.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.485+0000 7effacac9640 1 --2- 172.21.15.88:0/1418609476 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effa80c9bb0 0x7effa80c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:24.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.485+0000 7effacac9640 1 --2- 172.21.15.88:0/1418609476 >> v2:172.21.15.191:3300/0 conn(0x7effa8153770 0x7effa8159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:24.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.485+0000 7effacac9640 1 -- 172.21.15.88:0/1418609476 >> 172.21.15.88:0/1418609476 conn(0x7effa80715c0 msgr2=0x7effa80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:24.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.485+0000 7effacac9640 1 -- 172.21.15.88:0/1418609476 shutdown_connections 2023-10-16T20:43:24.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.485+0000 7effacac9640 1 -- 172.21.15.88:0/1418609476 wait complete. 2023-10-16T20:43:24.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effacac9640 1 Processor -- start 2023-10-16T20:43:24.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effacac9640 1 -- start start 2023-10-16T20:43:24.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effacac9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effa80c41f0 0x7effa80c6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:24.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effacac9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effa80c9bb0 0x7effa81495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:24.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effacac9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effa8153770 0x7effa80c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:24.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effacac9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa80c7b00 con 0x7effa80c41f0 2023-10-16T20:43:24.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effacac9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa80c7240 con 0x7effa80c9bb0 2023-10-16T20:43:24.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effacac9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa80c6b70 con 0x7effa8153770 2023-10-16T20:43:24.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effa659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effa80c41f0 0x7effa80c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:24.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effa659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effa80c41f0 0x7effa80c6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37118/0 (socket says 172.21.15.88:37118) 2023-10-16T20:43:24.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effa659c640 1 -- 172.21.15.88:0/1295861954 learned_addr learned my addr 172.21.15.88:0/1295861954 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:24.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effa6d9d640 1 --2- 172.21.15.88:0/1295861954 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effa8153770 0x7effa80c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:24.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7eff9dd9b640 1 --2- 172.21.15.88:0/1295861954 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effa80c9bb0 0x7effa81495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:24.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effa659c640 1 -- 172.21.15.88:0/1295861954 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effa8153770 msgr2=0x7effa80c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:24.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effa659c640 1 --2- 172.21.15.88:0/1295861954 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effa8153770 0x7effa80c7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:24.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effa659c640 1 -- 172.21.15.88:0/1295861954 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effa80c9bb0 msgr2=0x7effa81495b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:24.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effa659c640 1 --2- 172.21.15.88:0/1295861954 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effa80c9bb0 0x7effa81495b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:24.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effa659c640 1 -- 172.21.15.88:0/1295861954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa0009b30 con 0x7effa80c41f0 2023-10-16T20:43:24.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effa659c640 1 --2- 172.21.15.88:0/1295861954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effa80c41f0 0x7effa80c6630 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7eff9800b4b0 tx=0x7eff9800b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:24.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7eff9f7fe640 1 -- 172.21.15.88:0/1295861954 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff98010610 con 0x7effa80c41f0 2023-10-16T20:43:24.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7eff9f7fe640 1 -- 172.21.15.88:0/1295861954 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff98010770 con 0x7effa80c41f0 2023-10-16T20:43:24.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7eff9f7fe640 1 -- 172.21.15.88:0/1295861954 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff98010b20 con 0x7effa80c41f0 2023-10-16T20:43:24.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effacac9640 1 -- 172.21.15.88:0/1295861954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa81f5b10 con 0x7effa80c41f0 2023-10-16T20:43:24.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.489+0000 7effacac9640 1 -- 172.21.15.88:0/1295861954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa80c74b0 con 0x7effa80c41f0 2023-10-16T20:43:24.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.493+0000 7effacac9640 1 -- 172.21.15.88:0/1295861954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa8074700 con 0x7effa80c41f0 2023-10-16T20:43:24.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.493+0000 7eff9f7fe640 1 -- 172.21.15.88:0/1295861954 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7eff98010c80 con 0x7effa80c41f0 2023-10-16T20:43:24.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.493+0000 7eff9f7fe640 1 --2- 172.21.15.88:0/1295861954 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7eff880664a0 0x7eff88068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:24.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.493+0000 7eff9f7fe640 1 -- 172.21.15.88:0/1295861954 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6749+0+0 (secure 0 0 0) 0x7eff98083ff0 con 0x7effa80c41f0 2023-10-16T20:43:24.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.493+0000 7eff9dd9b640 1 --2- 172.21.15.88:0/1295861954 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7eff880664a0 0x7eff88068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:24.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.493+0000 7eff9dd9b640 1 --2- 172.21.15.88:0/1295861954 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7eff880664a0 0x7eff88068960 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7effa81489a0 tx=0x7eff90008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:24.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.497+0000 7eff9f7fe640 1 -- 172.21.15.88:0/1295861954 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7eff98050f90 con 0x7effa80c41f0 2023-10-16T20:43:24.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.677+0000 7effacac9640 1 -- 172.21.15.88:0/1295861954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effa8156d70 con 0x7effa80c41f0 2023-10-16T20:43:24.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.677+0000 7eff9f7fe640 1 -- 172.21.15.88:0/1295861954 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+6568 (secure 0 0 0) 0x7eff9801e090 con 0x7effa80c41f0 2023-10-16T20:43:24.681 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:24.681 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:42:31.455971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"c","rank":4,"incarnation":24,"state":"up:active","state_seq":89,"addr":"172.21.15.88:6839/1987579860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":1987579860},{"type":"v1","addr":"172.21.15.88:6839","nonce":1987579860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:24.682 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 29 2023-10-16T20:43:24.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.681+0000 7effacac9640 1 -- 172.21.15.88:0/1295861954 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7eff880664a0 msgr2=0x7eff88068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:24.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.681+0000 7effacac9640 1 --2- 172.21.15.88:0/1295861954 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7eff880664a0 0x7eff88068960 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7effa81489a0 tx=0x7eff90008040 comp rx=0 tx=0).stop 2023-10-16T20:43:24.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.681+0000 7effacac9640 1 -- 172.21.15.88:0/1295861954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effa80c41f0 msgr2=0x7effa80c6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:24.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.681+0000 7effacac9640 1 --2- 172.21.15.88:0/1295861954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effa80c41f0 0x7effa80c6630 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7eff9800b4b0 tx=0x7eff9800b970 comp rx=0 tx=0).stop 2023-10-16T20:43:24.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.681+0000 7effacac9640 1 -- 172.21.15.88:0/1295861954 shutdown_connections 2023-10-16T20:43:24.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.681+0000 7effacac9640 1 --2- 172.21.15.88:0/1295861954 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7eff880664a0 0x7eff88068960 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:24.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.681+0000 7effacac9640 1 --2- 172.21.15.88:0/1295861954 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7effa8153770 0x7effa80c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:24.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.681+0000 7effacac9640 1 --2- 172.21.15.88:0/1295861954 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effa80c9bb0 0x7effa81495b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:24.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.681+0000 7effacac9640 1 --2- 172.21.15.88:0/1295861954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effa80c41f0 0x7effa80c6630 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:24.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.681+0000 7effacac9640 1 -- 172.21.15.88:0/1295861954 >> 172.21.15.88:0/1295861954 conn(0x7effa80715c0 msgr2=0x7effa815a820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:24.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.681+0000 7effacac9640 1 -- 172.21.15.88:0/1295861954 shutdown_connections 2023-10-16T20:43:24.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:24.681+0000 7effacac9640 1 -- 172.21.15.88:0/1295861954 wait complete. 2023-10-16T20:43:24.702 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-16T20:43:25.323 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:43:25.328 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:43:26.703 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:26.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.849+0000 7f1dbca46640 1 Processor -- start 2023-10-16T20:43:26.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.849+0000 7f1dbca46640 1 -- start start 2023-10-16T20:43:26.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.849+0000 7f1dbca46640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1db815d840 0x7f1db8159640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:26.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.849+0000 7f1dbca46640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1db80cb890 0x7f1db806ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:26.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.849+0000 7f1dbca46640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db815a590 con 0x7f1db80c5300 2023-10-16T20:43:26.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.849+0000 7f1dbca46640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db815d420 con 0x7f1db80cb890 2023-10-16T20:43:26.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.849+0000 7f1dbca46640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db80c7830 con 0x7f1db815d840 2023-10-16T20:43:26.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db659c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1db80c5300 0x7f1db813e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47642/0 (socket says 172.21.15.88:47642) 2023-10-16T20:43:26.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db659c640 1 -- 172.21.15.88:0/1836863892 learned_addr learned my addr 172.21.15.88:0/1836863892 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:43:26.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db6d9d640 1 --2- 172.21.15.88:0/1836863892 >> v2:172.21.15.191:3300/0 conn(0x7f1db80cb890 0x7f1db806ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:26.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1dadd9b640 1 --2- 172.21.15.88:0/1836863892 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1db815d840 0x7f1db8159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:26.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db5d9b640 1 -- 172.21.15.88:0/1836863892 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946489533 0 0) 0x7f1db815a590 con 0x7f1db80c5300 2023-10-16T20:43:26.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db5d9b640 1 -- 172.21.15.88:0/1836863892 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1da4003530 con 0x7f1db80c5300 2023-10-16T20:43:26.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db5d9b640 1 -- 172.21.15.88:0/1836863892 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2951291776 0 0) 0x7f1da4003530 con 0x7f1db80c5300 2023-10-16T20:43:26.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db5d9b640 1 -- 172.21.15.88:0/1836863892 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1db815d840 msgr2=0x7f1db8159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:26.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db5d9b640 1 --2- 172.21.15.88:0/1836863892 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1db815d840 0x7f1db8159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:26.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db5d9b640 1 -- 172.21.15.88:0/1836863892 >> v2:172.21.15.191:3300/0 conn(0x7f1db80cb890 msgr2=0x7f1db806ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:26.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db5d9b640 1 --2- 172.21.15.88:0/1836863892 >> v2:172.21.15.191:3300/0 conn(0x7f1db80cb890 0x7f1db806ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:26.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db5d9b640 1 -- 172.21.15.88:0/1836863892 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db8159fd0 con 0x7f1db80c5300 2023-10-16T20:43:26.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db5d9b640 1 -- 172.21.15.88:0/1836863892 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f1da8001680 con 0x7f1db80c5300 2023-10-16T20:43:26.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db5d9b640 1 -- 172.21.15.88:0/1836863892 >> v1:172.21.15.88:6789/0 conn(0x7f1db80c5300 legacy=0x7f1db813e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:26.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db5d9b640 1 --2- 172.21.15.88:0/1836863892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1da4003980 0x7f1da4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:26.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db5d9b640 1 --2- 172.21.15.88:0/1836863892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1da40047d0 0x7f1da4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:26.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db5d9b640 1 --2- 172.21.15.88:0/1836863892 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1db815d840 0x7f1da4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:26.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db5d9b640 1 -- 172.21.15.88:0/1836863892 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db815d420 con 0x7f1da40047d0 2023-10-16T20:43:26.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db5d9b640 1 -- 172.21.15.88:0/1836863892 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da400b990 con 0x7f1da4003980 2023-10-16T20:43:26.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db5d9b640 1 -- 172.21.15.88:0/1836863892 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da400bad0 con 0x7f1db815d840 2023-10-16T20:43:26.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1dadd9b640 1 --2- 172.21.15.88:0/1836863892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1da40047d0 0x7f1da4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:26.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db659c640 1 --2- 172.21.15.88:0/1836863892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1da4003980 0x7f1da4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db6d9d640 1 --2- 172.21.15.88:0/1836863892 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1db815d840 0x7f1da4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1dadd9b640 1 -- 172.21.15.88:0/1836863892 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1db815d840 msgr2=0x7f1da4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1dadd9b640 1 --2- 172.21.15.88:0/1836863892 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1db815d840 0x7f1da4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1dadd9b640 1 -- 172.21.15.88:0/1836863892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1da4003980 msgr2=0x7f1da4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1dadd9b640 1 --2- 172.21.15.88:0/1836863892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1da4003980 0x7f1da4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:26.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1dadd9b640 1 -- 172.21.15.88:0/1836863892 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da4003530 con 0x7f1da40047d0 2023-10-16T20:43:26.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.853+0000 7f1db659c640 1 --2- 172.21.15.88:0/1836863892 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1da4003980 0x7f1da4003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:43:26.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.857+0000 7f1dadd9b640 1 --2- 172.21.15.88:0/1836863892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1da40047d0 0x7f1da4004c50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009fc0 tx=0x7f1da001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=602b282f620b5579 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:26.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.857+0000 7f1db5d9b640 1 -- 172.21.15.88:0/1836863892 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da00a5b90 con 0x7f1da40047d0 2023-10-16T20:43:26.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.857+0000 7f1db5d9b640 1 -- 172.21.15.88:0/1836863892 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da000f040 con 0x7f1da40047d0 2023-10-16T20:43:26.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.857+0000 7f1db5d9b640 1 -- 172.21.15.88:0/1836863892 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da00ad440 con 0x7f1da40047d0 2023-10-16T20:43:26.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.857+0000 7f1dbca46640 1 -- 172.21.15.88:0/1836863892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1da40047d0 msgr2=0x7f1da4004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:26.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.857+0000 7f1dbca46640 1 --2- 172.21.15.88:0/1836863892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1da40047d0 0x7f1da4004c50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009fc0 tx=0x7f1da001cc60 comp rx=0 tx=0).stop 2023-10-16T20:43:26.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.857+0000 7f1db6d9d640 1 -- 172.21.15.88:0/1836863892 reap_dead start 2023-10-16T20:43:26.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.857+0000 7f1dbca46640 1 -- 172.21.15.88:0/1836863892 shutdown_connections 2023-10-16T20:43:26.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.857+0000 7f1dbca46640 1 -- 172.21.15.88:0/1836863892 >> 172.21.15.88:0/1836863892 conn(0x7f1db806e0f0 msgr2=0x7f1db80cb180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:26.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.857+0000 7f1dbca46640 1 -- 172.21.15.88:0/1836863892 shutdown_connections 2023-10-16T20:43:26.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.857+0000 7f1dbca46640 1 -- 172.21.15.88:0/1836863892 wait complete. 2023-10-16T20:43:26.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.857+0000 7f1dbca46640 1 Processor -- start 2023-10-16T20:43:26.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.857+0000 7f1dbca46640 1 -- start start 2023-10-16T20:43:26.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1dbca46640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1da40047d0 0x7f1db81567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:26.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1dbca46640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1db813ddf0 0x7f1db813aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:26.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1dbca46640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1db813af50 0x7f1db81f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:26.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1dbca46640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db815c110 con 0x7f1da40047d0 2023-10-16T20:43:26.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1dbca46640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db813b3d0 con 0x7f1db813af50 2023-10-16T20:43:26.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1dbca46640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db80c4250 con 0x7f1db813ddf0 2023-10-16T20:43:26.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1db659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1da40047d0 0x7f1db81567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:26.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1db659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1da40047d0 0x7f1db81567e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37132/0 (socket says 172.21.15.88:37132) 2023-10-16T20:43:26.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1db659c640 1 -- 172.21.15.88:0/3455307392 learned_addr learned my addr 172.21.15.88:0/3455307392 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:26.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1dadd9b640 1 --2- 172.21.15.88:0/3455307392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1db813ddf0 0x7f1db813aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:26.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1db6d9d640 1 --2- 172.21.15.88:0/3455307392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1db813af50 0x7f1db81f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:26.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1db659c640 1 -- 172.21.15.88:0/3455307392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1db813ddf0 msgr2=0x7f1db813aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:26.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1db659c640 1 --2- 172.21.15.88:0/3455307392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1db813ddf0 0x7f1db813aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:26.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1db659c640 1 -- 172.21.15.88:0/3455307392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1db813af50 msgr2=0x7f1db81f1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:26.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1db659c640 1 --2- 172.21.15.88:0/3455307392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1db813af50 0x7f1db81f1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:26.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1db659c640 1 -- 172.21.15.88:0/3455307392 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db80c4af0 con 0x7f1da40047d0 2023-10-16T20:43:26.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1db659c640 1 --2- 172.21.15.88:0/3455307392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1da40047d0 0x7f1db81567e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1da8011810 tx=0x7f1da8011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:26.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1daf7fe640 1 -- 172.21.15.88:0/3455307392 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da800f960 con 0x7f1da40047d0 2023-10-16T20:43:26.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1daf7fe640 1 -- 172.21.15.88:0/3455307392 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da800fac0 con 0x7f1da40047d0 2023-10-16T20:43:26.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1dbca46640 1 -- 172.21.15.88:0/3455307392 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db80c3d10 con 0x7f1da40047d0 2023-10-16T20:43:26.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1daf7fe640 1 -- 172.21.15.88:0/3455307392 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da800fe30 con 0x7f1da40047d0 2023-10-16T20:43:26.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1dbca46640 1 -- 172.21.15.88:0/3455307392 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db80c4780 con 0x7f1da40047d0 2023-10-16T20:43:26.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1dbca46640 1 -- 172.21.15.88:0/3455307392 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db8074700 con 0x7f1da40047d0 2023-10-16T20:43:26.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.861+0000 7f1daf7fe640 1 -- 172.21.15.88:0/3455307392 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1da801f460 con 0x7f1da40047d0 2023-10-16T20:43:26.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.865+0000 7f1daf7fe640 1 --2- 172.21.15.88:0/3455307392 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1d980664d0 0x7f1d98068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:26.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.865+0000 7f1daf7fe640 1 -- 172.21.15.88:0/3455307392 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7f1da80874a0 con 0x7f1da40047d0 2023-10-16T20:43:26.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.865+0000 7f1dadd9b640 1 --2- 172.21.15.88:0/3455307392 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1d980664d0 0x7f1d98068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:26.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.865+0000 7f1dadd9b640 1 --2- 172.21.15.88:0/3455307392 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1d980664d0 0x7f1d98068990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1da00af000 tx=0x7f1da00a7040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:26.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:26.869+0000 7f1daf7fe640 1 -- 172.21.15.88:0/3455307392 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1da80544b0 con 0x7f1da40047d0 2023-10-16T20:43:27.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.045+0000 7f1dbca46640 1 -- 172.21.15.88:0/3455307392 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1db80cb670 con 0x7f1da40047d0 2023-10-16T20:43:27.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.045+0000 7f1daf7fe640 1 -- 172.21.15.88:0/3455307392 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f1da8057ad0 con 0x7f1da40047d0 2023-10-16T20:43:27.050 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:27.050 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:43:25.353464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:27.051 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 30 2023-10-16T20:43:27.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.049+0000 7f1dbca46640 1 -- 172.21.15.88:0/3455307392 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1d980664d0 msgr2=0x7f1d98068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:27.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.049+0000 7f1dbca46640 1 --2- 172.21.15.88:0/3455307392 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1d980664d0 0x7f1d98068990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1da00af000 tx=0x7f1da00a7040 comp rx=0 tx=0).stop 2023-10-16T20:43:27.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.049+0000 7f1dbca46640 1 -- 172.21.15.88:0/3455307392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1da40047d0 msgr2=0x7f1db81567e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:27.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.049+0000 7f1dbca46640 1 --2- 172.21.15.88:0/3455307392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1da40047d0 0x7f1db81567e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1da8011810 tx=0x7f1da8011cd0 comp rx=0 tx=0).stop 2023-10-16T20:43:27.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.049+0000 7f1dbca46640 1 -- 172.21.15.88:0/3455307392 shutdown_connections 2023-10-16T20:43:27.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.049+0000 7f1dbca46640 1 --2- 172.21.15.88:0/3455307392 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1d980664d0 0x7f1d98068990 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:27.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.049+0000 7f1dbca46640 1 --2- 172.21.15.88:0/3455307392 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1db813af50 0x7f1db81f1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:27.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.049+0000 7f1dbca46640 1 --2- 172.21.15.88:0/3455307392 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1db813ddf0 0x7f1db813aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:27.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.049+0000 7f1dbca46640 1 --2- 172.21.15.88:0/3455307392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1da40047d0 0x7f1db81567e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:27.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.049+0000 7f1dbca46640 1 -- 172.21.15.88:0/3455307392 >> 172.21.15.88:0/3455307392 conn(0x7f1db806e0f0 msgr2=0x7f1db80c5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:27.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.049+0000 7f1dbca46640 1 -- 172.21.15.88:0/3455307392 shutdown_connections 2023-10-16T20:43:27.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.049+0000 7f1dbca46640 1 -- 172.21.15.88:0/3455307392 wait complete. 2023-10-16T20:43:27.071 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-16T20:43:27.071 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-16T20:43:27.071 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:27.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.205+0000 7fb38d446640 1 Processor -- start 2023-10-16T20:43:27.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.205+0000 7fb38d446640 1 -- start start 2023-10-16T20:43:27.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb38d446640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb388153770 0x7fb388159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:27.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb38d446640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38815a410 con 0x7fb3880c9bb0 2023-10-16T20:43:27.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb38d446640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb388153b50 con 0x7fb3880cece0 2023-10-16T20:43:27.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb38d446640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3880cc250 con 0x7fb388153770 2023-10-16T20:43:27.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb3867fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb3880c9bb0 0x7fb3880c3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47644/0 (socket says 172.21.15.88:47644) 2023-10-16T20:43:27.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb3867fc640 1 -- 172.21.15.88:0/1858354808 learned_addr learned my addr 172.21.15.88:0/1858354808 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:43:27.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb385ffb640 1 -- 172.21.15.88:0/1858354808 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457281016 0 0) 0x7fb38815a410 con 0x7fb3880c9bb0 2023-10-16T20:43:27.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb385ffb640 1 -- 172.21.15.88:0/1858354808 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb370003530 con 0x7fb3880c9bb0 2023-10-16T20:43:27.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb386ffd640 1 --2- 172.21.15.88:0/1858354808 >> v2:172.21.15.191:3300/0 conn(0x7fb388153770 0x7fb388159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:27.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb385ffb640 1 -- 172.21.15.88:0/1858354808 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2526456688 0 0) 0x7fb370003530 con 0x7fb3880c9bb0 2023-10-16T20:43:27.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb385ffb640 1 -- 172.21.15.88:0/1858354808 >> v2:172.21.15.191:3300/0 conn(0x7fb388153770 msgr2=0x7fb388159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:27.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb385ffb640 1 --2- 172.21.15.88:0/1858354808 >> v2:172.21.15.191:3300/0 conn(0x7fb388153770 0x7fb388159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:27.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb385ffb640 1 -- 172.21.15.88:0/1858354808 >> v1:172.21.15.191:6789/0 conn(0x7fb3880cece0 legacy=0x7fb388163dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:27.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb385ffb640 1 -- 172.21.15.88:0/1858354808 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3880cb2f0 con 0x7fb3880c9bb0 2023-10-16T20:43:27.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb385ffb640 1 -- 172.21.15.88:0/1858354808 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7fb37c001680 con 0x7fb3880c9bb0 2023-10-16T20:43:27.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb385ffb640 1 -- 172.21.15.88:0/1858354808 >> v1:172.21.15.88:6789/0 conn(0x7fb3880c9bb0 legacy=0x7fb3880c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:27.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb385ffb640 1 --2- 172.21.15.88:0/1858354808 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb370003980 0x7fb370003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:27.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb385ffb640 1 --2- 172.21.15.88:0/1858354808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3700047d0 0x7fb370004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:27.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb385ffb640 1 --2- 172.21.15.88:0/1858354808 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb370005190 0x7fb37000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:27.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb385ffb640 1 -- 172.21.15.88:0/1858354808 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3880cc250 con 0x7fb370003980 2023-10-16T20:43:27.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb385ffb640 1 -- 172.21.15.88:0/1858354808 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37000c0e0 con 0x7fb370005190 2023-10-16T20:43:27.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb385ffb640 1 -- 172.21.15.88:0/1858354808 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb37000c2a0 con 0x7fb3700047d0 2023-10-16T20:43:27.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb386ffd640 1 --2- 172.21.15.88:0/1858354808 >> v2:172.21.15.191:3300/0 conn(0x7fb388153770 0x7fb388159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:43:27.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb386ffd640 1 --2- 172.21.15.88:0/1858354808 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb370003980 0x7fb370003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:27.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb3867fc640 1 --2- 172.21.15.88:0/1858354808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3700047d0 0x7fb370004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:27.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb3877fe640 1 --2- 172.21.15.88:0/1858354808 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb370005190 0x7fb37000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:27.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb386ffd640 1 -- 172.21.15.88:0/1858354808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3700047d0 msgr2=0x7fb370004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:27.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb386ffd640 1 --2- 172.21.15.88:0/1858354808 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3700047d0 0x7fb370004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:27.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb386ffd640 1 -- 172.21.15.88:0/1858354808 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb370005190 msgr2=0x7fb37000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:27.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb386ffd640 1 --2- 172.21.15.88:0/1858354808 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb370005190 0x7fb37000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:27.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb386ffd640 1 -- 172.21.15.88:0/1858354808 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb370003530 con 0x7fb370003980 2023-10-16T20:43:27.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.209+0000 7fb3877fe640 1 -- 172.21.15.88:0/1858354808 reap_dead start 2023-10-16T20:43:27.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.213+0000 7fb386ffd640 1 --2- 172.21.15.88:0/1858354808 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb370003980 0x7fb370003da0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb3780098a0 tx=0x7fb37801c570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155540e1cad8bedd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:27.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.213+0000 7fb385ffb640 1 -- 172.21.15.88:0/1858354808 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3780aa610 con 0x7fb370003980 2023-10-16T20:43:27.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.213+0000 7fb385ffb640 1 -- 172.21.15.88:0/1858354808 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3780a6040 con 0x7fb370003980 2023-10-16T20:43:27.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.213+0000 7fb385ffb640 1 -- 172.21.15.88:0/1858354808 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3780aa970 con 0x7fb370003980 2023-10-16T20:43:27.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.213+0000 7fb38d446640 1 -- 172.21.15.88:0/1858354808 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb370003980 msgr2=0x7fb370003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:27.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.213+0000 7fb38d446640 1 --2- 172.21.15.88:0/1858354808 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb370003980 0x7fb370003da0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb3780098a0 tx=0x7fb37801c570 comp rx=0 tx=0).stop 2023-10-16T20:43:27.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.213+0000 7fb38d446640 1 -- 172.21.15.88:0/1858354808 shutdown_connections 2023-10-16T20:43:27.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.213+0000 7fb38d446640 1 --2- 172.21.15.88:0/1858354808 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb370003980 0x7fb370003da0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:27.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.213+0000 7fb38d446640 1 -- 172.21.15.88:0/1858354808 >> 172.21.15.88:0/1858354808 conn(0x7fb3880715c0 msgr2=0x7fb3880719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:27.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.213+0000 7fb38d446640 1 -- 172.21.15.88:0/1858354808 shutdown_connections 2023-10-16T20:43:27.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.213+0000 7fb38d446640 1 -- 172.21.15.88:0/1858354808 wait complete. 2023-10-16T20:43:27.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.213+0000 7fb38d446640 1 Processor -- start 2023-10-16T20:43:27.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.213+0000 7fb38d446640 1 -- start start 2023-10-16T20:43:27.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb38d446640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb370003980 0x7fb38814af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:27.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb38d446640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3881f2a30 0x7fb3881520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:27.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb38d446640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3880c7a60 0x7fb388148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:27.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb38d446640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38815ad70 con 0x7fb3881f2a30 2023-10-16T20:43:27.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb38d446640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3880c75d0 con 0x7fb370003980 2023-10-16T20:43:27.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb38d446640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38815a7b0 con 0x7fb3880c7a60 2023-10-16T20:43:27.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb3867fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3881f2a30 0x7fb3881520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:27.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb3867fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3881f2a30 0x7fb3881520f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37162/0 (socket says 172.21.15.88:37162) 2023-10-16T20:43:27.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb3867fc640 1 -- 172.21.15.88:0/2399704064 learned_addr learned my addr 172.21.15.88:0/2399704064 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:27.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb386ffd640 1 --2- 172.21.15.88:0/2399704064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb370003980 0x7fb38814af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:27.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb3877fe640 1 --2- 172.21.15.88:0/2399704064 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3880c7a60 0x7fb388148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:27.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb3867fc640 1 -- 172.21.15.88:0/2399704064 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3880c7a60 msgr2=0x7fb388148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:27.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb3867fc640 1 --2- 172.21.15.88:0/2399704064 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3880c7a60 0x7fb388148d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:27.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb3867fc640 1 -- 172.21.15.88:0/2399704064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb370003980 msgr2=0x7fb38814af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:27.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb3867fc640 1 --2- 172.21.15.88:0/2399704064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb370003980 0x7fb38814af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:27.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb3867fc640 1 -- 172.21.15.88:0/2399704064 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38815aa90 con 0x7fb3881f2a30 2023-10-16T20:43:27.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb386ffd640 1 --2- 172.21.15.88:0/2399704064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb370003980 0x7fb38814af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:43:27.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb3867fc640 1 --2- 172.21.15.88:0/2399704064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3881f2a30 0x7fb3881520f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0072c0 tx=0x7fb37c007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:27.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb36f7fe640 1 -- 172.21.15.88:0/2399704064 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c00f960 con 0x7fb3881f2a30 2023-10-16T20:43:27.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb36f7fe640 1 -- 172.21.15.88:0/2399704064 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37c00fac0 con 0x7fb3881f2a30 2023-10-16T20:43:27.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb38d446640 1 -- 172.21.15.88:0/2399704064 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3880c7260 con 0x7fb3881f2a30 2023-10-16T20:43:27.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb36f7fe640 1 -- 172.21.15.88:0/2399704064 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c00fe50 con 0x7fb3881f2a30 2023-10-16T20:43:27.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb38d446640 1 -- 172.21.15.88:0/2399704064 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3880c9040 con 0x7fb3881f2a30 2023-10-16T20:43:27.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb38d446640 1 -- 172.21.15.88:0/2399704064 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3880c70c0 con 0x7fb3881f2a30 2023-10-16T20:43:27.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.217+0000 7fb36f7fe640 1 -- 172.21.15.88:0/2399704064 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb37c01d460 con 0x7fb3881f2a30 2023-10-16T20:43:27.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.221+0000 7fb36f7fe640 1 --2- 172.21.15.88:0/2399704064 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb3740664d0 0x7fb374068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:27.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.221+0000 7fb36f7fe640 1 -- 172.21.15.88:0/2399704064 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7fb37c084520 con 0x7fb3881f2a30 2023-10-16T20:43:27.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.221+0000 7fb386ffd640 1 --2- 172.21.15.88:0/2399704064 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb3740664d0 0x7fb374068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:27.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.221+0000 7fb386ffd640 1 --2- 172.21.15.88:0/2399704064 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb3740664d0 0x7fb374068990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb3780ae000 tx=0x7fb3780046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:27.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.225+0000 7fb36f7fe640 1 -- 172.21.15.88:0/2399704064 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb3880c70c0 con 0x7fb3881f2a30 2023-10-16T20:43:27.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.409+0000 7fb38d446640 1 -- 172.21.15.88:0/2399704064 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3880c70c0 con 0x7fb3881f2a30 2023-10-16T20:43:27.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.409+0000 7fb36f7fe640 1 -- 172.21.15.88:0/2399704064 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5777 (secure 0 0 0) 0x7fb3880c70c0 con 0x7fb3881f2a30 2023-10-16T20:43:27.413 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:27.413 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:43:25.353464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:27.414 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 30 2023-10-16T20:43:27.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.413+0000 7fb38d446640 1 -- 172.21.15.88:0/2399704064 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb3740664d0 msgr2=0x7fb374068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:27.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.413+0000 7fb38d446640 1 --2- 172.21.15.88:0/2399704064 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb3740664d0 0x7fb374068990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb3780ae000 tx=0x7fb3780046d0 comp rx=0 tx=0).stop 2023-10-16T20:43:27.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.413+0000 7fb38d446640 1 -- 172.21.15.88:0/2399704064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3881f2a30 msgr2=0x7fb3881520f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:27.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.413+0000 7fb38d446640 1 --2- 172.21.15.88:0/2399704064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3881f2a30 0x7fb3881520f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0072c0 tx=0x7fb37c007780 comp rx=0 tx=0).stop 2023-10-16T20:43:27.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.413+0000 7fb38d446640 1 -- 172.21.15.88:0/2399704064 shutdown_connections 2023-10-16T20:43:27.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.413+0000 7fb38d446640 1 --2- 172.21.15.88:0/2399704064 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb3740664d0 0x7fb374068990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:27.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.413+0000 7fb38d446640 1 --2- 172.21.15.88:0/2399704064 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb3880c7a60 0x7fb388148d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:27.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.413+0000 7fb38d446640 1 --2- 172.21.15.88:0/2399704064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3881f2a30 0x7fb3881520f0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:27.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.413+0000 7fb38d446640 1 --2- 172.21.15.88:0/2399704064 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb370003980 0x7fb38814af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:27.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.413+0000 7fb38d446640 1 -- 172.21.15.88:0/2399704064 >> 172.21.15.88:0/2399704064 conn(0x7fb3880715c0 msgr2=0x7fb3880b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:27.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.413+0000 7fb38d446640 1 -- 172.21.15.88:0/2399704064 shutdown_connections 2023-10-16T20:43:27.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:27.413+0000 7fb38d446640 1 -- 172.21.15.88:0/2399704064 wait complete. 2023-10-16T20:43:27.435 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 30, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 30, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4263": { "addr": "172.21.15.88:6835/518281867", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 518281867, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 518281867, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4263, "incarnation": 16, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 43 }, "gid_4275": { "addr": "172.21.15.88:6838/321173984", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 321173984, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 321173984, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4275, "incarnation": 14, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 41 }, "gid_4279": { "addr": "172.21.15.191:6833/2446850309", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 2446850309, "type": "v2" }, { "addr": "172.21.15.191:6833", "nonce": 2446850309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 40 }, "gid_4288": { "addr": "172.21.15.191:6835/3059699574", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3059699574, "type": "v2" }, { "addr": "172.21.15.191:6835", "nonce": 3059699574, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4889": { "addr": "172.21.15.191:6837/2468647900", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6836", "nonce": 2468647900, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2468647900, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4889, "incarnation": 30, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:43:25.353464+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 4279, "mds_2": 4275, "mds_3": 4263, "mds_4": 4889 } } } ], "standbys": [] } 2023-10-16T20:43:29.436 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:29.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.581+0000 7f0a05854640 1 Processor -- start 2023-10-16T20:43:29.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.581+0000 7f0a05854640 1 -- start start 2023-10-16T20:43:29.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.581+0000 7f0a05854640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0a000ae6c0 0x7f0a000c1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:29.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.581+0000 7f0a05854640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a000c1ec0 con 0x7f0a000c1670 2023-10-16T20:43:29.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.581+0000 7f0a05854640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a00156f40 con 0x7f0a00150450 2023-10-16T20:43:29.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.581+0000 7f0a05854640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a00150830 con 0x7f0a000ae6c0 2023-10-16T20:43:29.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.581+0000 7f09fe7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0a000ae6c0 0x7f0a000c1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:29.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.581+0000 7f09fffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0a000c1670 0x7f0a00071f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36906/0 (socket says 172.21.15.88:36906) 2023-10-16T20:43:29.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.581+0000 7f09fffff640 1 -- 172.21.15.88:0/2999835554 learned_addr learned my addr 172.21.15.88:0/2999835554 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:43:29.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.581+0000 7f09fdffb640 1 -- 172.21.15.88:0/2999835554 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302711456 0 0) 0x7f0a000c1ec0 con 0x7f0a000c1670 2023-10-16T20:43:29.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.581+0000 7f09fdffb640 1 -- 172.21.15.88:0/2999835554 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e8005740 con 0x7f0a000c1670 2023-10-16T20:43:29.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.581+0000 7f09fe7fc640 1 -- 172.21.15.88:0/2999835554 >> v1:172.21.15.191:6789/0 conn(0x7f0a00150450 legacy=0x7f0a00159590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:29.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.581+0000 7f09fe7fc640 1 -- 172.21.15.88:0/2999835554 >> v1:172.21.15.88:6789/0 conn(0x7f0a000c1670 legacy=0x7f0a00071f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:29.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.581+0000 7f09fe7fc640 1 -- 172.21.15.88:0/2999835554 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a000c1be0 con 0x7f0a000ae6c0 2023-10-16T20:43:29.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.585+0000 7f09fe7fc640 1 --2- 172.21.15.88:0/2999835554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a000ae6c0 0x7f0a000c1130 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f09f40097c0 tx=0x7f09f401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a208821b31222e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:29.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.585+0000 7f09fdffb640 1 -- 172.21.15.88:0/2999835554 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f40ad680 con 0x7f0a000ae6c0 2023-10-16T20:43:29.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.585+0000 7f09fdffb640 1 -- 172.21.15.88:0/2999835554 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f40ad7e0 con 0x7f0a000ae6c0 2023-10-16T20:43:29.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.585+0000 7f09fdffb640 1 -- 172.21.15.88:0/2999835554 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f40adb70 con 0x7f0a000ae6c0 2023-10-16T20:43:29.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.585+0000 7f0a05854640 1 -- 172.21.15.88:0/2999835554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a000ae6c0 msgr2=0x7f0a000c1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:29.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.585+0000 7f0a05854640 1 --2- 172.21.15.88:0/2999835554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a000ae6c0 0x7f0a000c1130 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f09f40097c0 tx=0x7f09f401c480 comp rx=0 tx=0).stop 2023-10-16T20:43:29.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.585+0000 7f0a05854640 1 -- 172.21.15.88:0/2999835554 shutdown_connections 2023-10-16T20:43:29.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.585+0000 7f0a05854640 1 --2- 172.21.15.88:0/2999835554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a000ae6c0 0x7f0a000c1130 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:29.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.585+0000 7f0a05854640 1 -- 172.21.15.88:0/2999835554 >> 172.21.15.88:0/2999835554 conn(0x7f0a0006e9d0 msgr2=0x7f0a0006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:29.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.585+0000 7f0a05854640 1 -- 172.21.15.88:0/2999835554 shutdown_connections 2023-10-16T20:43:29.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.585+0000 7f0a05854640 1 -- 172.21.15.88:0/2999835554 wait complete. 2023-10-16T20:43:29.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.585+0000 7f0a05854640 1 Processor -- start 2023-10-16T20:43:29.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f0a05854640 1 -- start start 2023-10-16T20:43:29.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f0a05854640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a000ae6c0 0x7f0a0014f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:29.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f0a05854640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a000c1670 0x7f0a0014b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:29.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f0a05854640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a00150450 0x7f0a00146490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:29.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f0a05854640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a00146000 con 0x7f0a000c1670 2023-10-16T20:43:29.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f0a05854640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0014fcc0 con 0x7f0a000ae6c0 2023-10-16T20:43:29.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f0a05854640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0014dd70 con 0x7f0a00150450 2023-10-16T20:43:29.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f09fe7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a000c1670 0x7f0a0014b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:29.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f09fe7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a000c1670 0x7f0a0014b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49508/0 (socket says 172.21.15.88:49508) 2023-10-16T20:43:29.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f09fe7fc640 1 -- 172.21.15.88:0/3723331829 learned_addr learned my addr 172.21.15.88:0/3723331829 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:29.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f09fffff640 1 --2- 172.21.15.88:0/3723331829 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a00150450 0x7f0a00146490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:29.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f09feffd640 1 --2- 172.21.15.88:0/3723331829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a000ae6c0 0x7f0a0014f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:29.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f09fe7fc640 1 -- 172.21.15.88:0/3723331829 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a00150450 msgr2=0x7f0a00146490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:29.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f09fe7fc640 1 --2- 172.21.15.88:0/3723331829 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a00150450 0x7f0a00146490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:29.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f09fe7fc640 1 -- 172.21.15.88:0/3723331829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a000ae6c0 msgr2=0x7f0a0014f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:29.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f09fe7fc640 1 --2- 172.21.15.88:0/3723331829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a000ae6c0 0x7f0a0014f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:29.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f09fe7fc640 1 -- 172.21.15.88:0/3723331829 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f4009420 con 0x7f0a000c1670 2023-10-16T20:43:29.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f09fe7fc640 1 --2- 172.21.15.88:0/3723331829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a000c1670 0x7f0a0014b650 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f09f4019090 tx=0x7f09f401acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:29.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f09e7fff640 1 -- 172.21.15.88:0/3723331829 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f40add60 con 0x7f0a000c1670 2023-10-16T20:43:29.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f09e7fff640 1 -- 172.21.15.88:0/3723331829 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f40b8450 con 0x7f0a000c1670 2023-10-16T20:43:29.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f09e7fff640 1 -- 172.21.15.88:0/3723331829 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f40b8800 con 0x7f0a000c1670 2023-10-16T20:43:29.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f0a05854640 1 -- 172.21.15.88:0/3723331829 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0014e050 con 0x7f0a000c1670 2023-10-16T20:43:29.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f0a05854640 1 -- 172.21.15.88:0/3723331829 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0014e610 con 0x7f0a000c1670 2023-10-16T20:43:29.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.589+0000 7f0a05854640 1 -- 172.21.15.88:0/3723331829 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a001461b0 con 0x7f0a000c1670 2023-10-16T20:43:29.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.593+0000 7f09e7fff640 1 -- 172.21.15.88:0/3723331829 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f09f40b4070 con 0x7f0a000c1670 2023-10-16T20:43:29.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.593+0000 7f09e7fff640 1 --2- 172.21.15.88:0/3723331829 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f09ec0664a0 0x7f09ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:29.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.593+0000 7f09e7fff640 1 -- 172.21.15.88:0/3723331829 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7f09f411e850 con 0x7f0a000c1670 2023-10-16T20:43:29.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.593+0000 7f09feffd640 1 --2- 172.21.15.88:0/3723331829 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f09ec0664a0 0x7f09ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:29.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.593+0000 7f09feffd640 1 --2- 172.21.15.88:0/3723331829 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f09ec0664a0 0x7f09ec068960 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f09f0000f70 tx=0x7f09f0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:29.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.597+0000 7f09e7fff640 1 -- 172.21.15.88:0/3723331829 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0a001461b0 con 0x7f0a000c1670 2023-10-16T20:43:29.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.777+0000 7f0a05854640 1 -- 172.21.15.88:0/3723331829 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a001461b0 con 0x7f0a000c1670 2023-10-16T20:43:29.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.781+0000 7f09e7fff640 1 -- 172.21.15.88:0/3723331829 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f0a001461b0 con 0x7f0a000c1670 2023-10-16T20:43:29.783 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:29.783 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:43:25.353464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:29.784 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 30 2023-10-16T20:43:29.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.781+0000 7f0a05854640 1 -- 172.21.15.88:0/3723331829 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f09ec0664a0 msgr2=0x7f09ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:29.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.781+0000 7f0a05854640 1 --2- 172.21.15.88:0/3723331829 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f09ec0664a0 0x7f09ec068960 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f09f0000f70 tx=0x7f09f0009040 comp rx=0 tx=0).stop 2023-10-16T20:43:29.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.781+0000 7f0a05854640 1 -- 172.21.15.88:0/3723331829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a000c1670 msgr2=0x7f0a0014b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:29.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.781+0000 7f0a05854640 1 --2- 172.21.15.88:0/3723331829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a000c1670 0x7f0a0014b650 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f09f4019090 tx=0x7f09f401acc0 comp rx=0 tx=0).stop 2023-10-16T20:43:29.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.781+0000 7f0a05854640 1 -- 172.21.15.88:0/3723331829 shutdown_connections 2023-10-16T20:43:29.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.781+0000 7f0a05854640 1 --2- 172.21.15.88:0/3723331829 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f09ec0664a0 0x7f09ec068960 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:29.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.781+0000 7f0a05854640 1 --2- 172.21.15.88:0/3723331829 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a00150450 0x7f0a00146490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:29.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.781+0000 7f0a05854640 1 --2- 172.21.15.88:0/3723331829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a000c1670 0x7f0a0014b650 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:29.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.781+0000 7f0a05854640 1 --2- 172.21.15.88:0/3723331829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a000ae6c0 0x7f0a0014f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:29.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.781+0000 7f0a05854640 1 -- 172.21.15.88:0/3723331829 >> 172.21.15.88:0/3723331829 conn(0x7f0a0006e9d0 msgr2=0x7f0a0006b310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:29.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.781+0000 7f0a05854640 1 -- 172.21.15.88:0/3723331829 shutdown_connections 2023-10-16T20:43:29.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:29.781+0000 7f0a05854640 1 -- 172.21.15.88:0/3723331829 wait complete. 2023-10-16T20:43:29.804 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:43:31.804 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:31.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.949+0000 7f9490617640 1 Processor -- start 2023-10-16T20:43:31.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.949+0000 7f9490617640 1 -- start start 2023-10-16T20:43:31.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f9490617640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f948815a810 0x7f948815ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:31.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f9490617640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f948815b3c0 con 0x7f948806a970 2023-10-16T20:43:31.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f9490617640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f948815b680 con 0x7f94881406d0 2023-10-16T20:43:31.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f9490617640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f948815b1c0 con 0x7f948815a810 2023-10-16T20:43:31.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948dbb2640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f948806a970 0x7f9488071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36908/0 (socket says 172.21.15.88:36908) 2023-10-16T20:43:31.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948dbb2640 1 -- 172.21.15.88:0/3711742194 learned_addr learned my addr 172.21.15.88:0/3711742194 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:43:31.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948ebb4640 1 --2- 172.21.15.88:0/3711742194 >> v2:172.21.15.191:3300/0 conn(0x7f948815a810 0x7f948815ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:31.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948d3b1640 1 -- 172.21.15.88:0/3711742194 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265236617 0 0) 0x7f948815b3c0 con 0x7f948806a970 2023-10-16T20:43:31.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948d3b1640 1 -- 172.21.15.88:0/3711742194 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9474003530 con 0x7f948806a970 2023-10-16T20:43:31.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948d3b1640 1 -- 172.21.15.88:0/3711742194 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 399287443 0 0) 0x7f948815b680 con 0x7f94881406d0 2023-10-16T20:43:31.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948d3b1640 1 -- 172.21.15.88:0/3711742194 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f948815b3c0 con 0x7f94881406d0 2023-10-16T20:43:31.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948d3b1640 1 -- 172.21.15.88:0/3711742194 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2988231546 0 0) 0x7f9474003530 con 0x7f948806a970 2023-10-16T20:43:31.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948d3b1640 1 -- 172.21.15.88:0/3711742194 >> v2:172.21.15.191:3300/0 conn(0x7f948815a810 msgr2=0x7f948815ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:31.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948d3b1640 1 --2- 172.21.15.88:0/3711742194 >> v2:172.21.15.191:3300/0 conn(0x7f948815a810 0x7f948815ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:31.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948d3b1640 1 -- 172.21.15.88:0/3711742194 >> v1:172.21.15.191:6789/0 conn(0x7f94881406d0 legacy=0x7f94880c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:31.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948d3b1640 1 -- 172.21.15.88:0/3711742194 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9488157570 con 0x7f948806a970 2023-10-16T20:43:31.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948d3b1640 1 -- 172.21.15.88:0/3711742194 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f9478001680 con 0x7f948806a970 2023-10-16T20:43:31.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948ebb4640 1 --2- 172.21.15.88:0/3711742194 >> v2:172.21.15.191:3300/0 conn(0x7f948815a810 0x7f948815ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:43:31.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948d3b1640 1 -- 172.21.15.88:0/3711742194 >> v1:172.21.15.88:6789/0 conn(0x7f948806a970 legacy=0x7f9488071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:31.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948d3b1640 1 --2- 172.21.15.88:0/3711742194 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9474003b10 0x7f9474003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:31.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948d3b1640 1 --2- 172.21.15.88:0/3711742194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94740047d0 0x7f9474004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:31.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948d3b1640 1 --2- 172.21.15.88:0/3711742194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9474005190 0x7f947400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:31.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948d3b1640 1 -- 172.21.15.88:0/3711742194 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948815b1c0 con 0x7f9474005190 2023-10-16T20:43:31.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948d3b1640 1 -- 172.21.15.88:0/3711742194 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947400c010 con 0x7f9474003b10 2023-10-16T20:43:31.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948d3b1640 1 -- 172.21.15.88:0/3711742194 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947400c1d0 con 0x7f94740047d0 2023-10-16T20:43:31.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948ebb4640 1 --2- 172.21.15.88:0/3711742194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9474005190 0x7f947400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:31.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948e3b3640 1 --2- 172.21.15.88:0/3711742194 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9474003b10 0x7f9474003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:31.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948dbb2640 1 --2- 172.21.15.88:0/3711742194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94740047d0 0x7f9474004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:31.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948ebb4640 1 -- 172.21.15.88:0/3711742194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94740047d0 msgr2=0x7f9474004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:31.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948ebb4640 1 --2- 172.21.15.88:0/3711742194 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f94740047d0 0x7f9474004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:31.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948ebb4640 1 -- 172.21.15.88:0/3711742194 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9474003b10 msgr2=0x7f9474003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:31.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948ebb4640 1 --2- 172.21.15.88:0/3711742194 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9474003b10 0x7f9474003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:31.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.953+0000 7f948ebb4640 1 -- 172.21.15.88:0/3711742194 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9474003530 con 0x7f9474005190 2023-10-16T20:43:31.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.957+0000 7f948ebb4640 1 -- 172.21.15.88:0/3711742194 reap_dead start 2023-10-16T20:43:31.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.957+0000 7f948ebb4640 1 --2- 172.21.15.88:0/3711742194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9474005190 0x7f947400b990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f948000e350 tx=0x7f948001ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2426a7343abc9969 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:31.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.957+0000 7f948d3b1640 1 -- 172.21.15.88:0/3711742194 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94800a5e70 con 0x7f9474005190 2023-10-16T20:43:31.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.957+0000 7f948d3b1640 1 -- 172.21.15.88:0/3711742194 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948000f040 con 0x7f9474005190 2023-10-16T20:43:31.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.957+0000 7f948d3b1640 1 -- 172.21.15.88:0/3711742194 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94800ad6a0 con 0x7f9474005190 2023-10-16T20:43:31.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.957+0000 7f9490617640 1 -- 172.21.15.88:0/3711742194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9474005190 msgr2=0x7f947400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:31.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.957+0000 7f9490617640 1 --2- 172.21.15.88:0/3711742194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9474005190 0x7f947400b990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f948000e350 tx=0x7f948001ce60 comp rx=0 tx=0).stop 2023-10-16T20:43:31.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.957+0000 7f9490617640 1 -- 172.21.15.88:0/3711742194 shutdown_connections 2023-10-16T20:43:31.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.957+0000 7f9490617640 1 --2- 172.21.15.88:0/3711742194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9474005190 0x7f947400b990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:31.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.957+0000 7f9490617640 1 -- 172.21.15.88:0/3711742194 >> 172.21.15.88:0/3711742194 conn(0x7f94880cad50 msgr2=0x7f94880cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:31.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.957+0000 7f9490617640 1 -- 172.21.15.88:0/3711742194 shutdown_connections 2023-10-16T20:43:31.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.957+0000 7f9490617640 1 -- 172.21.15.88:0/3711742194 wait complete. 2023-10-16T20:43:31.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.957+0000 7f9490617640 1 Processor -- start 2023-10-16T20:43:31.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f9490617640 1 -- start start 2023-10-16T20:43:31.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f9490617640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9474005190 0x7f94881388b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:31.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f9490617640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94881ed160 0x7f94881ed5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:31.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f9490617640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94881502d0 0x7f9488150730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:31.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f9490617640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9488152160 con 0x7f94881ed160 2023-10-16T20:43:31.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f9490617640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94881ecd30 con 0x7f94881502d0 2023-10-16T20:43:31.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f9490617640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94881edb00 con 0x7f9474005190 2023-10-16T20:43:31.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f948dbb2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94881ed160 0x7f94881ed5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:31.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f948dbb2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94881ed160 0x7f94881ed5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49516/0 (socket says 172.21.15.88:49516) 2023-10-16T20:43:31.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f948dbb2640 1 -- 172.21.15.88:0/2293919902 learned_addr learned my addr 172.21.15.88:0/2293919902 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:31.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f948e3b3640 1 --2- 172.21.15.88:0/2293919902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9474005190 0x7f94881388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:31.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f948ebb4640 1 --2- 172.21.15.88:0/2293919902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94881502d0 0x7f9488150730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:31.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f948dbb2640 1 -- 172.21.15.88:0/2293919902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9474005190 msgr2=0x7f94881388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:31.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f948dbb2640 1 --2- 172.21.15.88:0/2293919902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9474005190 0x7f94881388b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:31.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f948dbb2640 1 -- 172.21.15.88:0/2293919902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94881502d0 msgr2=0x7f9488150730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:31.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f948dbb2640 1 --2- 172.21.15.88:0/2293919902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94881502d0 0x7f9488150730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:31.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f948dbb2640 1 -- 172.21.15.88:0/2293919902 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948815a2d0 con 0x7f94881ed160 2023-10-16T20:43:31.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f948ebb4640 1 --2- 172.21.15.88:0/2293919902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94881502d0 0x7f9488150730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:43:31.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f948dbb2640 1 --2- 172.21.15.88:0/2293919902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94881ed160 0x7f94881ed5c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9478012520 tx=0x7f9478012920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:31.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f947effd640 1 -- 172.21.15.88:0/2293919902 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9478014810 con 0x7f94881ed160 2023-10-16T20:43:31.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f947effd640 1 -- 172.21.15.88:0/2293919902 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94780149b0 con 0x7f94881ed160 2023-10-16T20:43:31.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f947effd640 1 -- 172.21.15.88:0/2293919902 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9478014d60 con 0x7f94881ed160 2023-10-16T20:43:31.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f9490617640 1 -- 172.21.15.88:0/2293919902 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f948815a5a0 con 0x7f94881ed160 2023-10-16T20:43:31.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f9490617640 1 -- 172.21.15.88:0/2293919902 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9488158ca0 con 0x7f94881ed160 2023-10-16T20:43:31.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.961+0000 7f9490617640 1 -- 172.21.15.88:0/2293919902 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94881ece80 con 0x7f94881ed160 2023-10-16T20:43:31.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.965+0000 7f947effd640 1 -- 172.21.15.88:0/2293919902 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f94780153e0 con 0x7f94881ed160 2023-10-16T20:43:31.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.965+0000 7f947effd640 1 --2- 172.21.15.88:0/2293919902 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f946c0664d0 0x7f946c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:31.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.965+0000 7f947effd640 1 -- 172.21.15.88:0/2293919902 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7f9478086c40 con 0x7f94881ed160 2023-10-16T20:43:31.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.965+0000 7f948e3b3640 1 --2- 172.21.15.88:0/2293919902 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f946c0664d0 0x7f946c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:31.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.965+0000 7f948e3b3640 1 --2- 172.21.15.88:0/2293919902 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f946c0664d0 0x7f946c068990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9484000c10 tx=0x7f9484009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:31.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:31.969+0000 7f947effd640 1 -- 172.21.15.88:0/2293919902 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f94881ece80 con 0x7f94881ed160 2023-10-16T20:43:32.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:32.148+0000 7f9490617640 1 -- 172.21.15.88:0/2293919902 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94881ece80 con 0x7f94881ed160 2023-10-16T20:43:32.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:32.148+0000 7f947effd640 1 -- 172.21.15.88:0/2293919902 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+5779 (secure 0 0 0) 0x7f94881ece80 con 0x7f94881ed160 2023-10-16T20:43:32.151 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:32.152 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:43:30.424961+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:resolve","state_seq":15,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:32.152 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 31 2023-10-16T20:43:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:32.148+0000 7f9490617640 1 -- 172.21.15.88:0/2293919902 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f946c0664d0 msgr2=0x7f946c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:32.148+0000 7f9490617640 1 --2- 172.21.15.88:0/2293919902 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f946c0664d0 0x7f946c068990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9484000c10 tx=0x7f9484009040 comp rx=0 tx=0).stop 2023-10-16T20:43:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:32.148+0000 7f9490617640 1 -- 172.21.15.88:0/2293919902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94881ed160 msgr2=0x7f94881ed5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:32.148+0000 7f9490617640 1 --2- 172.21.15.88:0/2293919902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94881ed160 0x7f94881ed5c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9478012520 tx=0x7f9478012920 comp rx=0 tx=0).stop 2023-10-16T20:43:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:32.148+0000 7f9490617640 1 -- 172.21.15.88:0/2293919902 shutdown_connections 2023-10-16T20:43:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:32.148+0000 7f9490617640 1 --2- 172.21.15.88:0/2293919902 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f946c0664d0 0x7f946c068990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:32.148+0000 7f9490617640 1 --2- 172.21.15.88:0/2293919902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94881502d0 0x7f9488150730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:32.148+0000 7f9490617640 1 --2- 172.21.15.88:0/2293919902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94881ed160 0x7f94881ed5c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:32.148+0000 7f9490617640 1 --2- 172.21.15.88:0/2293919902 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9474005190 0x7f94881388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:32.148+0000 7f9490617640 1 -- 172.21.15.88:0/2293919902 >> 172.21.15.88:0/2293919902 conn(0x7f94880cad50 msgr2=0x7f948813f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:32.148+0000 7f9490617640 1 -- 172.21.15.88:0/2293919902 shutdown_connections 2023-10-16T20:43:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:32.148+0000 7f9490617640 1 -- 172.21.15.88:0/2293919902 wait complete. 2023-10-16T20:43:32.172 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:43:34.173 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:34.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.320+0000 7f8ca239d640 1 Processor -- start 2023-10-16T20:43:34.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.320+0000 7f8ca239d640 1 -- start start 2023-10-16T20:43:34.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.320+0000 7f8ca239d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8c9c150280 0x7f8c9c0c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:34.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.320+0000 7f8ca239d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c9c0c6bc0 0x7f8c9c0c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:34.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.320+0000 7f8ca239d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c9c0c5c60 con 0x7f8c9c150660 2023-10-16T20:43:34.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.320+0000 7f8ca239d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c9c0c5370 con 0x7f8c9c150280 2023-10-16T20:43:34.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.320+0000 7f8ca239d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9c156d10 con 0x7f8c9c0c6bc0 2023-10-16T20:43:34.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.320+0000 7f8c9bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8c9c150280 0x7f8c9c0c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:34.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.320+0000 7f8c9b7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8c9c150660 0x7f8c9c06d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36924/0 (socket says 172.21.15.88:36924) 2023-10-16T20:43:34.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.320+0000 7f8c9b7fe640 1 -- 172.21.15.88:0/1779382506 learned_addr learned my addr 172.21.15.88:0/1779382506 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:43:34.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.320+0000 7f8ca093a640 1 --2- 172.21.15.88:0/1779382506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c9c0c6bc0 0x7f8c9c0c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:34.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8c9bfff640 1 -- 172.21.15.88:0/1779382506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c9c0c6bc0 msgr2=0x7f8c9c0c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:34.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8c9bfff640 1 --2- 172.21.15.88:0/1779382506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c9c0c6bc0 0x7f8c9c0c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:34.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8c9bfff640 1 -- 172.21.15.88:0/1779382506 >> v1:172.21.15.88:6789/0 conn(0x7f8c9c150660 legacy=0x7f8c9c06d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:34.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8c9bfff640 1 -- 172.21.15.88:0/1779382506 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c9c0c5610 con 0x7f8c9c150280 2023-10-16T20:43:34.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8ca093a640 1 --2- 172.21.15.88:0/1779382506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c9c0c6bc0 0x7f8c9c0c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:43:34.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8c9affd640 1 -- 172.21.15.88:0/1779382506 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298024063 0 0) 0x7f8c9c0c5c60 con 0x7f8c9c150660 2023-10-16T20:43:34.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8c9bfff640 1 --2- 172.21.15.88:0/1779382506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c9c150280 0x7f8c9c0c60f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c0097c0 tx=0x7f8c8c01ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8cd4cf999bf4b15 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:34.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8c9affd640 1 -- 172.21.15.88:0/1779382506 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c0ab670 con 0x7f8c9c150280 2023-10-16T20:43:34.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8c9affd640 1 -- 172.21.15.88:0/1779382506 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8c0ab7d0 con 0x7f8c9c150280 2023-10-16T20:43:34.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8c9affd640 1 -- 172.21.15.88:0/1779382506 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c0abb60 con 0x7f8c9c150280 2023-10-16T20:43:34.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8ca239d640 1 -- 172.21.15.88:0/1779382506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c9c150280 msgr2=0x7f8c9c0c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:34.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8ca239d640 1 --2- 172.21.15.88:0/1779382506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c9c150280 0x7f8c9c0c60f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c0097c0 tx=0x7f8c8c01ceb0 comp rx=0 tx=0).stop 2023-10-16T20:43:34.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8ca239d640 1 -- 172.21.15.88:0/1779382506 shutdown_connections 2023-10-16T20:43:34.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8ca239d640 1 --2- 172.21.15.88:0/1779382506 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c9c0c6bc0 0x7f8c9c0c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:34.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8ca239d640 1 --2- 172.21.15.88:0/1779382506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c9c150280 0x7f8c9c0c60f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:34.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8ca239d640 1 -- 172.21.15.88:0/1779382506 >> 172.21.15.88:0/1779382506 conn(0x7f8c9c0cad50 msgr2=0x7f8c9c0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:34.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8ca239d640 1 -- 172.21.15.88:0/1779382506 shutdown_connections 2023-10-16T20:43:34.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.324+0000 7f8ca239d640 1 -- 172.21.15.88:0/1779382506 wait complete. 2023-10-16T20:43:34.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8ca239d640 1 Processor -- start 2023-10-16T20:43:34.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8ca239d640 1 -- start start 2023-10-16T20:43:34.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8ca239d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c9c0c6bc0 0x7f8c9c14a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:34.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8ca239d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c9c150280 0x7f8c9c06a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:34.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8ca239d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c9c150660 0x7f8c9c1f4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:34.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8ca239d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9c071140 con 0x7f8c9c150280 2023-10-16T20:43:34.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8ca239d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9c071420 con 0x7f8c9c150660 2023-10-16T20:43:34.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8ca239d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9c071700 con 0x7f8c9c0c6bc0 2023-10-16T20:43:34.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8c9b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c9c150280 0x7f8c9c06a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:34.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8c9b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c9c150280 0x7f8c9c06a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49524/0 (socket says 172.21.15.88:49524) 2023-10-16T20:43:34.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8c9b7fe640 1 -- 172.21.15.88:0/2492633700 learned_addr learned my addr 172.21.15.88:0/2492633700 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:34.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8c9bfff640 1 --2- 172.21.15.88:0/2492633700 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c9c0c6bc0 0x7f8c9c14a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:34.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8ca093a640 1 --2- 172.21.15.88:0/2492633700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c9c150660 0x7f8c9c1f4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:34.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8c9b7fe640 1 -- 172.21.15.88:0/2492633700 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c9c0c6bc0 msgr2=0x7f8c9c14a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:34.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8c9b7fe640 1 --2- 172.21.15.88:0/2492633700 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c9c0c6bc0 0x7f8c9c14a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:34.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8c9b7fe640 1 -- 172.21.15.88:0/2492633700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c9c150660 msgr2=0x7f8c9c1f4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:34.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8c9b7fe640 1 --2- 172.21.15.88:0/2492633700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c9c150660 0x7f8c9c1f4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:34.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8c9b7fe640 1 -- 172.21.15.88:0/2492633700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8c009420 con 0x7f8c9c150280 2023-10-16T20:43:34.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8c9bfff640 1 --2- 172.21.15.88:0/2492633700 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c9c0c6bc0 0x7f8c9c14a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:43:34.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8ca093a640 1 --2- 172.21.15.88:0/2492633700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c9c150660 0x7f8c9c1f4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:43:34.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8c9b7fe640 1 --2- 172.21.15.88:0/2492633700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c9c150280 0x7f8c9c06a570 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8c9000c480 tx=0x7f8c9000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:34.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8c98ff9640 1 -- 172.21.15.88:0/2492633700 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c900074e0 con 0x7f8c9c150280 2023-10-16T20:43:34.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8c98ff9640 1 -- 172.21.15.88:0/2492633700 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c90007660 con 0x7f8c9c150280 2023-10-16T20:43:34.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8ca239d640 1 -- 172.21.15.88:0/2492633700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c9c0719e0 con 0x7f8c9c150280 2023-10-16T20:43:34.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8c98ff9640 1 -- 172.21.15.88:0/2492633700 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c90004a30 con 0x7f8c9c150280 2023-10-16T20:43:34.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.328+0000 7f8ca239d640 1 -- 172.21.15.88:0/2492633700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c9c071fa0 con 0x7f8c9c150280 2023-10-16T20:43:34.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.332+0000 7f8ca239d640 1 -- 172.21.15.88:0/2492633700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c9c074700 con 0x7f8c9c150280 2023-10-16T20:43:34.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.332+0000 7f8c98ff9640 1 -- 172.21.15.88:0/2492633700 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8c90004b90 con 0x7f8c9c150280 2023-10-16T20:43:34.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.332+0000 7f8c98ff9640 1 --2- 172.21.15.88:0/2492633700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8c880664a0 0x7f8c88068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:34.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.332+0000 7f8c98ff9640 1 -- 172.21.15.88:0/2492633700 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7f8c90083830 con 0x7f8c9c150280 2023-10-16T20:43:34.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.332+0000 7f8c9bfff640 1 --2- 172.21.15.88:0/2492633700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8c880664a0 0x7f8c88068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:34.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.332+0000 7f8c9bfff640 1 --2- 172.21.15.88:0/2492633700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8c880664a0 0x7f8c88068960 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c0af000 tx=0x7f8c8c004740 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:34.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.336+0000 7f8c98ff9640 1 -- 172.21.15.88:0/2492633700 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8c900542c0 con 0x7f8c9c150280 2023-10-16T20:43:34.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.520+0000 7f8ca239d640 1 -- 172.21.15.88:0/2492633700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c9c0c1650 con 0x7f8c9c150280 2023-10-16T20:43:34.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.520+0000 7f8c98ff9640 1 -- 172.21.15.88:0/2492633700 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+5779 (secure 0 0 0) 0x7f8c90053e50 con 0x7f8c9c150280 2023-10-16T20:43:34.524 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:34.525 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:43:30.424961+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:resolve","state_seq":15,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:34.525 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 31 2023-10-16T20:43:34.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.520+0000 7f8ca239d640 1 -- 172.21.15.88:0/2492633700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8c880664a0 msgr2=0x7f8c88068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:34.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.520+0000 7f8ca239d640 1 --2- 172.21.15.88:0/2492633700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8c880664a0 0x7f8c88068960 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c0af000 tx=0x7f8c8c004740 comp rx=0 tx=0).stop 2023-10-16T20:43:34.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.520+0000 7f8ca239d640 1 -- 172.21.15.88:0/2492633700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c9c150280 msgr2=0x7f8c9c06a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:34.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.520+0000 7f8ca239d640 1 --2- 172.21.15.88:0/2492633700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c9c150280 0x7f8c9c06a570 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8c9000c480 tx=0x7f8c9000c940 comp rx=0 tx=0).stop 2023-10-16T20:43:34.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.524+0000 7f8ca239d640 1 -- 172.21.15.88:0/2492633700 shutdown_connections 2023-10-16T20:43:34.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.524+0000 7f8ca239d640 1 --2- 172.21.15.88:0/2492633700 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8c880664a0 0x7f8c88068960 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.524+0000 7f8ca239d640 1 --2- 172.21.15.88:0/2492633700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c9c150660 0x7f8c9c1f4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.524+0000 7f8ca239d640 1 --2- 172.21.15.88:0/2492633700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c9c150280 0x7f8c9c06a570 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.524+0000 7f8ca239d640 1 --2- 172.21.15.88:0/2492633700 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8c9c0c6bc0 0x7f8c9c14a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.524+0000 7f8ca239d640 1 -- 172.21.15.88:0/2492633700 >> 172.21.15.88:0/2492633700 conn(0x7f8c9c0cad50 msgr2=0x7f8c9c0c8850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.524+0000 7f8ca239d640 1 -- 172.21.15.88:0/2492633700 shutdown_connections 2023-10-16T20:43:34.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:34.524+0000 7f8ca239d640 1 -- 172.21.15.88:0/2492633700 wait complete. 2023-10-16T20:43:34.545 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:43:36.547 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:36.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.692+0000 7fdc6e30f640 1 Processor -- start 2023-10-16T20:43:36.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.692+0000 7fdc6e30f640 1 -- start start 2023-10-16T20:43:36.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.692+0000 7fdc6e30f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc68150280 0x7fdc680c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:36.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.692+0000 7fdc6e30f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdc68150660 0x7fdc680c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:36.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.692+0000 7fdc6e30f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdc680c6bc0 0x7fdc6806d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:36.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.692+0000 7fdc6e30f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc680c5c60 con 0x7fdc680c6bc0 2023-10-16T20:43:36.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.692+0000 7fdc6e30f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc680c5370 con 0x7fdc68150660 2023-10-16T20:43:36.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.692+0000 7fdc6e30f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc68156d10 con 0x7fdc68150280 2023-10-16T20:43:36.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.692+0000 7fdc6c8ac640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdc680c6bc0 0x7fdc6806d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:36.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.692+0000 7fdc6c8ac640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdc680c6bc0 0x7fdc6806d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49530/0 (socket says 172.21.15.88:49530) 2023-10-16T20:43:36.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.692+0000 7fdc6c8ac640 1 -- 172.21.15.88:0/3089896021 learned_addr learned my addr 172.21.15.88:0/3089896021 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:36.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.692+0000 7fdc677fe640 1 --2- 172.21.15.88:0/3089896021 >> v2:172.21.15.191:3300/0 conn(0x7fdc68150660 0x7fdc680c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:36.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.692+0000 7fdc67fff640 1 --2- 172.21.15.88:0/3089896021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc68150280 0x7fdc680c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:36.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.696+0000 7fdc6c8ac640 1 -- 172.21.15.88:0/3089896021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc68150280 msgr2=0x7fdc680c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:36.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.696+0000 7fdc6c8ac640 1 --2- 172.21.15.88:0/3089896021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc68150280 0x7fdc680c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:36.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.696+0000 7fdc6c8ac640 1 -- 172.21.15.88:0/3089896021 >> v2:172.21.15.191:3300/0 conn(0x7fdc68150660 msgr2=0x7fdc680c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:36.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.696+0000 7fdc6c8ac640 1 --2- 172.21.15.88:0/3089896021 >> v2:172.21.15.191:3300/0 conn(0x7fdc68150660 0x7fdc680c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:36.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.696+0000 7fdc6c8ac640 1 -- 172.21.15.88:0/3089896021 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc680c5610 con 0x7fdc680c6bc0 2023-10-16T20:43:36.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.696+0000 7fdc67fff640 1 --2- 172.21.15.88:0/3089896021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc68150280 0x7fdc680c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:43:36.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.696+0000 7fdc6c8ac640 1 --2- 172.21.15.88:0/3089896021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc680c6bc0 0x7fdc6806d200 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fdc680c84f0 tx=0x7fdc6001eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f11fd9fd006a28 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:36.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.696+0000 7fdc66ffd640 1 -- 172.21.15.88:0/3089896021 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6000b2d0 con 0x7fdc680c6bc0 2023-10-16T20:43:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.696+0000 7fdc66ffd640 1 -- 172.21.15.88:0/3089896021 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc6000b430 con 0x7fdc680c6bc0 2023-10-16T20:43:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.696+0000 7fdc66ffd640 1 -- 172.21.15.88:0/3089896021 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc600a7ad0 con 0x7fdc680c6bc0 2023-10-16T20:43:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.696+0000 7fdc6e30f640 1 -- 172.21.15.88:0/3089896021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc680c6bc0 msgr2=0x7fdc6806d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.696+0000 7fdc6e30f640 1 --2- 172.21.15.88:0/3089896021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc680c6bc0 0x7fdc6806d200 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fdc680c84f0 tx=0x7fdc6001eb10 comp rx=0 tx=0).stop 2023-10-16T20:43:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.696+0000 7fdc6e30f640 1 -- 172.21.15.88:0/3089896021 shutdown_connections 2023-10-16T20:43:36.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.696+0000 7fdc6e30f640 1 --2- 172.21.15.88:0/3089896021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc680c6bc0 0x7fdc6806d200 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:36.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.696+0000 7fdc6e30f640 1 --2- 172.21.15.88:0/3089896021 >> v2:172.21.15.191:3300/0 conn(0x7fdc68150660 0x7fdc680c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:36.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.696+0000 7fdc6e30f640 1 --2- 172.21.15.88:0/3089896021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc68150280 0x7fdc680c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:36.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.696+0000 7fdc6e30f640 1 -- 172.21.15.88:0/3089896021 >> 172.21.15.88:0/3089896021 conn(0x7fdc680cad50 msgr2=0x7fdc680cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:36.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc6e30f640 1 -- 172.21.15.88:0/3089896021 shutdown_connections 2023-10-16T20:43:36.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc6e30f640 1 -- 172.21.15.88:0/3089896021 wait complete. 2023-10-16T20:43:36.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc6e30f640 1 Processor -- start 2023-10-16T20:43:36.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc6e30f640 1 -- start start 2023-10-16T20:43:36.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc6e30f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc680c6bc0 0x7fdc6814b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:36.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc6e30f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc68150280 0x7fdc681ee310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:36.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc6e30f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc68150660 0x7fdc6814af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:36.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc6e30f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc681ed850 con 0x7fdc68150280 2023-10-16T20:43:36.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc6e30f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc68155ce0 con 0x7fdc680c6bc0 2023-10-16T20:43:36.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc6e30f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc681f0f60 con 0x7fdc68150660 2023-10-16T20:43:36.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc677fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc68150280 0x7fdc681ee310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:36.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc67fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc680c6bc0 0x7fdc6814b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc677fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc68150280 0x7fdc681ee310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49534/0 (socket says 172.21.15.88:49534) 2023-10-16T20:43:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc6c8ac640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc68150660 0x7fdc6814af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc677fe640 1 -- 172.21.15.88:0/4167842572 learned_addr learned my addr 172.21.15.88:0/4167842572 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc677fe640 1 -- 172.21.15.88:0/4167842572 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc68150660 msgr2=0x7fdc6814af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc677fe640 1 --2- 172.21.15.88:0/4167842572 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc68150660 0x7fdc6814af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:36.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc677fe640 1 -- 172.21.15.88:0/4167842572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc680c6bc0 msgr2=0x7fdc6814b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:36.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc677fe640 1 --2- 172.21.15.88:0/4167842572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc680c6bc0 0x7fdc6814b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:36.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc677fe640 1 -- 172.21.15.88:0/4167842572 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6000bd40 con 0x7fdc68150280 2023-10-16T20:43:36.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.700+0000 7fdc677fe640 1 --2- 172.21.15.88:0/4167842572 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc68150280 0x7fdc681ee310 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c00d5f0 tx=0x7fdc5c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:36.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.704+0000 7fdc64ff9640 1 -- 172.21.15.88:0/4167842572 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c0125f0 con 0x7fdc68150280 2023-10-16T20:43:36.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.704+0000 7fdc64ff9640 1 -- 172.21.15.88:0/4167842572 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5c012750 con 0x7fdc68150280 2023-10-16T20:43:36.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.704+0000 7fdc64ff9640 1 -- 172.21.15.88:0/4167842572 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c012ae0 con 0x7fdc68150280 2023-10-16T20:43:36.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.704+0000 7fdc6e30f640 1 -- 172.21.15.88:0/4167842572 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc681edb10 con 0x7fdc68150280 2023-10-16T20:43:36.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.704+0000 7fdc6e30f640 1 -- 172.21.15.88:0/4167842572 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc68154150 con 0x7fdc68150280 2023-10-16T20:43:36.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.704+0000 7fdc6e30f640 1 -- 172.21.15.88:0/4167842572 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc68074700 con 0x7fdc68150280 2023-10-16T20:43:36.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.704+0000 7fdc64ff9640 1 -- 172.21.15.88:0/4167842572 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdc5c012c40 con 0x7fdc68150280 2023-10-16T20:43:36.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.704+0000 7fdc64ff9640 1 --2- 172.21.15.88:0/4167842572 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdc540664a0 0x7fdc54068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:36.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.704+0000 7fdc64ff9640 1 -- 172.21.15.88:0/4167842572 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7fdc5c084680 con 0x7fdc68150280 2023-10-16T20:43:36.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.704+0000 7fdc67fff640 1 --2- 172.21.15.88:0/4167842572 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdc540664a0 0x7fdc54068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:36.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.708+0000 7fdc67fff640 1 --2- 172.21.15.88:0/4167842572 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdc540664a0 0x7fdc54068960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdc5800ab60 tx=0x7fdc58008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:36.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.708+0000 7fdc64ff9640 1 -- 172.21.15.88:0/4167842572 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdc5c0516e0 con 0x7fdc68150280 2023-10-16T20:43:36.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.892+0000 7fdc6e30f640 1 -- 172.21.15.88:0/4167842572 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc680c1650 con 0x7fdc68150280 2023-10-16T20:43:36.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.892+0000 7fdc64ff9640 1 -- 172.21.15.88:0/4167842572 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5781 (secure 0 0 0) 0x7fdc5c054d00 con 0x7fdc68150280 2023-10-16T20:43:36.895 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:36.896 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:43:36.438999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:36.896 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 32 2023-10-16T20:43:36.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.892+0000 7fdc6e30f640 1 -- 172.21.15.88:0/4167842572 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdc540664a0 msgr2=0x7fdc54068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.892+0000 7fdc6e30f640 1 --2- 172.21.15.88:0/4167842572 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdc540664a0 0x7fdc54068960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdc5800ab60 tx=0x7fdc58008040 comp rx=0 tx=0).stop 2023-10-16T20:43:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.892+0000 7fdc6e30f640 1 -- 172.21.15.88:0/4167842572 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc68150280 msgr2=0x7fdc681ee310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.892+0000 7fdc6e30f640 1 --2- 172.21.15.88:0/4167842572 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc68150280 0x7fdc681ee310 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c00d5f0 tx=0x7fdc5c00dab0 comp rx=0 tx=0).stop 2023-10-16T20:43:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.892+0000 7fdc6e30f640 1 -- 172.21.15.88:0/4167842572 shutdown_connections 2023-10-16T20:43:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.892+0000 7fdc6e30f640 1 --2- 172.21.15.88:0/4167842572 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdc540664a0 0x7fdc54068960 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:36.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.892+0000 7fdc6e30f640 1 --2- 172.21.15.88:0/4167842572 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdc68150660 0x7fdc6814af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:36.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.892+0000 7fdc6e30f640 1 --2- 172.21.15.88:0/4167842572 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc68150280 0x7fdc681ee310 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:36.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.892+0000 7fdc6e30f640 1 --2- 172.21.15.88:0/4167842572 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdc680c6bc0 0x7fdc6814b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:36.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.892+0000 7fdc6e30f640 1 -- 172.21.15.88:0/4167842572 >> 172.21.15.88:0/4167842572 conn(0x7fdc680cad50 msgr2=0x7fdc680cb130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:36.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.892+0000 7fdc6e30f640 1 -- 172.21.15.88:0/4167842572 shutdown_connections 2023-10-16T20:43:36.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:36.892+0000 7fdc6e30f640 1 -- 172.21.15.88:0/4167842572 wait complete. 2023-10-16T20:43:36.916 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:43:38.917 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:39.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.060+0000 7f0def3b8640 1 Processor -- start 2023-10-16T20:43:39.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.060+0000 7f0def3b8640 1 -- start start 2023-10-16T20:43:39.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.064+0000 7f0def3b8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0de8153770 0x7f0de8159f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:39.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.064+0000 7f0def3b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0de80c9040 0x7f0de80c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:39.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.064+0000 7f0def3b8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0de8069e90 con 0x7f0de80c6a20 2023-10-16T20:43:39.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.064+0000 7f0def3b8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0de806aef0 con 0x7f0de80c9040 2023-10-16T20:43:39.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.064+0000 7f0def3b8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de80c6e90 con 0x7f0de8153770 2023-10-16T20:43:39.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.064+0000 7f0dec953640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0de80c9040 0x7f0de80c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:39.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.064+0000 7f0ded955640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0de80c6a20 0x7f0de8071b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58570/0 (socket says 172.21.15.88:58570) 2023-10-16T20:43:39.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.064+0000 7f0ded955640 1 -- 172.21.15.88:0/2148682757 learned_addr learned my addr 172.21.15.88:0/2148682757 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:43:39.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.064+0000 7f0dec953640 1 -- 172.21.15.88:0/2148682757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0de8153770 msgr2=0x7f0de8159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:39.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.064+0000 7f0ded154640 1 --2- 172.21.15.88:0/2148682757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0de8153770 0x7f0de8159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:39.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.064+0000 7f0dec953640 1 --2- 172.21.15.88:0/2148682757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0de8153770 0x7f0de8159f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:39.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.064+0000 7f0dec953640 1 -- 172.21.15.88:0/2148682757 >> v1:172.21.15.88:6789/0 conn(0x7f0de80c6a20 legacy=0x7f0de8071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:39.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.064+0000 7f0ddffff640 1 -- 172.21.15.88:0/2148682757 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087680894 0 0) 0x7f0de8069e90 con 0x7f0de80c6a20 2023-10-16T20:43:39.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.064+0000 7f0dec953640 1 -- 172.21.15.88:0/2148682757 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de806a0c0 con 0x7f0de80c9040 2023-10-16T20:43:39.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.064+0000 7f0ded154640 1 --2- 172.21.15.88:0/2148682757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0de8153770 0x7f0de8159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:43:39.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.068+0000 7f0dec953640 1 --2- 172.21.15.88:0/2148682757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80c9040 0x7f0de80c64e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0dd80097c0 tx=0x7f0dd801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8177521dfb856704 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:39.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.068+0000 7f0ddffff640 1 -- 172.21.15.88:0/2148682757 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd80ad640 con 0x7f0de80c9040 2023-10-16T20:43:39.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.068+0000 7f0ddffff640 1 -- 172.21.15.88:0/2148682757 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd80ad7a0 con 0x7f0de80c9040 2023-10-16T20:43:39.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.068+0000 7f0ddffff640 1 -- 172.21.15.88:0/2148682757 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd80adb30 con 0x7f0de80c9040 2023-10-16T20:43:39.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.068+0000 7f0def3b8640 1 -- 172.21.15.88:0/2148682757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80c9040 msgr2=0x7f0de80c64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:39.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.068+0000 7f0def3b8640 1 --2- 172.21.15.88:0/2148682757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80c9040 0x7f0de80c64e0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0dd80097c0 tx=0x7f0dd801c480 comp rx=0 tx=0).stop 2023-10-16T20:43:39.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.068+0000 7f0def3b8640 1 -- 172.21.15.88:0/2148682757 shutdown_connections 2023-10-16T20:43:39.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.068+0000 7f0def3b8640 1 --2- 172.21.15.88:0/2148682757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80c9040 0x7f0de80c64e0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:39.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.068+0000 7f0def3b8640 1 --2- 172.21.15.88:0/2148682757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0de8153770 0x7f0de8159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:39.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.068+0000 7f0def3b8640 1 -- 172.21.15.88:0/2148682757 >> 172.21.15.88:0/2148682757 conn(0x7f0de80c84e0 msgr2=0x7f0de80c88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:39.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.068+0000 7f0def3b8640 1 -- 172.21.15.88:0/2148682757 shutdown_connections 2023-10-16T20:43:39.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.068+0000 7f0def3b8640 1 -- 172.21.15.88:0/2148682757 wait complete. 2023-10-16T20:43:39.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.068+0000 7f0def3b8640 1 Processor -- start 2023-10-16T20:43:39.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.068+0000 7f0def3b8640 1 -- start start 2023-10-16T20:43:39.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0def3b8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0de80c6a20 0x7f0de80c3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:39.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0def3b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80c9040 0x7f0de814af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:39.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0def3b8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0de8153770 0x7f0de815baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:39.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0def3b8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de80c3e70 con 0x7f0de80c9040 2023-10-16T20:43:39.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0def3b8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de814bab0 con 0x7f0de80c6a20 2023-10-16T20:43:39.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0def3b8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de815b530 con 0x7f0de8153770 2023-10-16T20:43:39.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0dec953640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80c9040 0x7f0de814af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:39.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0dec953640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80c9040 0x7f0de814af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57568/0 (socket says 172.21.15.88:57568) 2023-10-16T20:43:39.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0dec953640 1 -- 172.21.15.88:0/1983741457 learned_addr learned my addr 172.21.15.88:0/1983741457 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:39.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0ded955640 1 --2- 172.21.15.88:0/1983741457 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0de8153770 0x7f0de815baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:39.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0ded154640 1 --2- 172.21.15.88:0/1983741457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0de80c6a20 0x7f0de80c3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:39.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0dec953640 1 -- 172.21.15.88:0/1983741457 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0de8153770 msgr2=0x7f0de815baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:39.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0dec953640 1 --2- 172.21.15.88:0/1983741457 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0de8153770 0x7f0de815baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:39.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0dec953640 1 -- 172.21.15.88:0/1983741457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0de80c6a20 msgr2=0x7f0de80c3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:39.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0dec953640 1 --2- 172.21.15.88:0/1983741457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0de80c6a20 0x7f0de80c3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:39.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0dec953640 1 -- 172.21.15.88:0/1983741457 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd8009420 con 0x7f0de80c9040 2023-10-16T20:43:39.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0dec953640 1 --2- 172.21.15.88:0/1983741457 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80c9040 0x7f0de814af90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0dd801a0b0 tx=0x7f0dd8019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:39.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0dddffb640 1 -- 172.21.15.88:0/1983741457 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd80ad640 con 0x7f0de80c9040 2023-10-16T20:43:39.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0dddffb640 1 -- 172.21.15.88:0/1983741457 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd80a6040 con 0x7f0de80c9040 2023-10-16T20:43:39.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0def3b8640 1 -- 172.21.15.88:0/1983741457 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0de815a640 con 0x7f0de80c9040 2023-10-16T20:43:39.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0dddffb640 1 -- 172.21.15.88:0/1983741457 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd80b6a80 con 0x7f0de80c9040 2023-10-16T20:43:39.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0def3b8640 1 -- 172.21.15.88:0/1983741457 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de815ac00 con 0x7f0de80c9040 2023-10-16T20:43:39.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0def3b8640 1 -- 172.21.15.88:0/1983741457 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de8146b00 con 0x7f0de80c9040 2023-10-16T20:43:39.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.072+0000 7f0dddffb640 1 -- 172.21.15.88:0/1983741457 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0dd80b4040 con 0x7f0de80c9040 2023-10-16T20:43:39.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.076+0000 7f0dddffb640 1 --2- 172.21.15.88:0/1983741457 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0dcc0664a0 0x7f0dcc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:39.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.076+0000 7f0dddffb640 1 -- 172.21.15.88:0/1983741457 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7f0dd811e470 con 0x7f0de80c9040 2023-10-16T20:43:39.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.076+0000 7f0ded154640 1 --2- 172.21.15.88:0/1983741457 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0dcc0664a0 0x7f0dcc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:39.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.076+0000 7f0ded154640 1 --2- 172.21.15.88:0/1983741457 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0dcc0664a0 0x7f0dcc068960 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0de4006fd0 tx=0x7f0de4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:39.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.080+0000 7f0dddffb640 1 -- 172.21.15.88:0/1983741457 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0de8146b00 con 0x7f0de80c9040 2023-10-16T20:43:39.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.260+0000 7f0def3b8640 1 -- 172.21.15.88:0/1983741457 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0de8146b00 con 0x7f0de80c9040 2023-10-16T20:43:39.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.260+0000 7f0dddffb640 1 -- 172.21.15.88:0/1983741457 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5781 (secure 0 0 0) 0x7f0de8146b00 con 0x7f0de80c9040 2023-10-16T20:43:39.264 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:39.264 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:43:36.438999+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:39.265 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 32 2023-10-16T20:43:39.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.260+0000 7f0def3b8640 1 -- 172.21.15.88:0/1983741457 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0dcc0664a0 msgr2=0x7f0dcc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:39.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.260+0000 7f0def3b8640 1 --2- 172.21.15.88:0/1983741457 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0dcc0664a0 0x7f0dcc068960 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0de4006fd0 tx=0x7f0de4008040 comp rx=0 tx=0).stop 2023-10-16T20:43:39.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.260+0000 7f0def3b8640 1 -- 172.21.15.88:0/1983741457 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80c9040 msgr2=0x7f0de814af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:39.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.260+0000 7f0def3b8640 1 --2- 172.21.15.88:0/1983741457 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80c9040 0x7f0de814af90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0dd801a0b0 tx=0x7f0dd8019220 comp rx=0 tx=0).stop 2023-10-16T20:43:39.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.260+0000 7f0def3b8640 1 -- 172.21.15.88:0/1983741457 shutdown_connections 2023-10-16T20:43:39.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.260+0000 7f0def3b8640 1 --2- 172.21.15.88:0/1983741457 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0dcc0664a0 0x7f0dcc068960 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:39.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.260+0000 7f0def3b8640 1 --2- 172.21.15.88:0/1983741457 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0de8153770 0x7f0de815baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:39.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.260+0000 7f0def3b8640 1 --2- 172.21.15.88:0/1983741457 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80c9040 0x7f0de814af90 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:39.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.260+0000 7f0def3b8640 1 --2- 172.21.15.88:0/1983741457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0de80c6a20 0x7f0de80c3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:39.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.260+0000 7f0def3b8640 1 -- 172.21.15.88:0/1983741457 >> 172.21.15.88:0/1983741457 conn(0x7f0de80c84e0 msgr2=0x7f0de80c3020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:39.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.260+0000 7f0def3b8640 1 -- 172.21.15.88:0/1983741457 shutdown_connections 2023-10-16T20:43:39.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:39.260+0000 7f0def3b8640 1 -- 172.21.15.88:0/1983741457 wait complete. 2023-10-16T20:43:39.286 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4263": { "addr": "172.21.15.88:6835/518281867", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 518281867, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 518281867, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4263, "incarnation": 16, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 43 }, "gid_4275": { "addr": "172.21.15.88:6838/321173984", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 321173984, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 321173984, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4275, "incarnation": 14, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 41 }, "gid_4279": { "addr": "172.21.15.191:6833/2446850309", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 2446850309, "type": "v2" }, { "addr": "172.21.15.191:6833", "nonce": 2446850309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 40 }, "gid_4288": { "addr": "172.21.15.191:6835/3059699574", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3059699574, "type": "v2" }, { "addr": "172.21.15.191:6835", "nonce": 3059699574, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4889": { "addr": "172.21.15.191:6837/2468647900", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6836", "nonce": 2468647900, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2468647900, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4889, "incarnation": 30, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:reconnect", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:43:36.438999+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 4279, "mds_2": 4275, "mds_3": 4263, "mds_4": 4889 } } } ], "standbys": [] } 2023-10-16T20:43:41.286 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:41.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.432+0000 7f5700017640 1 Processor -- start 2023-10-16T20:43:41.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.432+0000 7f5700017640 1 -- start start 2023-10-16T20:43:41.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.432+0000 7f5700017640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56f80c9bb0 0x7f56f80c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:41.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.432+0000 7f5700017640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f56f80c41f0 0x7f56f80cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:41.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.432+0000 7f5700017640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f8153b50 con 0x7f56f8153770 2023-10-16T20:43:41.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.432+0000 7f5700017640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f80cc250 con 0x7f56f80c41f0 2023-10-16T20:43:41.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.432+0000 7f5700017640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f80cc670 con 0x7f56f80c9bb0 2023-10-16T20:43:41.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.432+0000 7f56fe5b4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f56f80c41f0 0x7f56f80cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:41.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.432+0000 7f56fe5b4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f56f80c41f0 0x7f56f80cece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57580/0 (socket says 172.21.15.88:57580) 2023-10-16T20:43:41.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.432+0000 7f56fe5b4640 1 -- 172.21.15.88:0/2109460291 learned_addr learned my addr 172.21.15.88:0/2109460291 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:41.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.432+0000 7f56fe5b4640 1 -- 172.21.15.88:0/2109460291 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56f80c9bb0 msgr2=0x7f56f80c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:41.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.432+0000 7f56fd5b2640 1 --2- 172.21.15.88:0/2109460291 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56f80c9bb0 0x7f56f80c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:41.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.432+0000 7f56fe5b4640 1 --2- 172.21.15.88:0/2109460291 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56f80c9bb0 0x7f56f80c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:41.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.432+0000 7f56fe5b4640 1 -- 172.21.15.88:0/2109460291 >> v1:172.21.15.191:6789/0 conn(0x7f56f8153770 legacy=0x7f56f8159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:41.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.432+0000 7f56fe5b4640 1 -- 172.21.15.88:0/2109460291 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f80cb2f0 con 0x7f56f80c41f0 2023-10-16T20:43:41.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.432+0000 7f56fd5b2640 1 --2- 172.21.15.88:0/2109460291 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56f80c9bb0 0x7f56f80c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:43:41.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.436+0000 7f56fe5b4640 1 --2- 172.21.15.88:0/2109460291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56f80c41f0 0x7f56f80cece0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f56f0009e50 tx=0x7f56f001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46bb9aae72944c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:41.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.436+0000 7f56fcdb1640 1 -- 172.21.15.88:0/2109460291 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f00a57f0 con 0x7f56f80c41f0 2023-10-16T20:43:41.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.436+0000 7f56fcdb1640 1 -- 172.21.15.88:0/2109460291 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f00a5950 con 0x7f56f80c41f0 2023-10-16T20:43:41.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.436+0000 7f56fcdb1640 1 -- 172.21.15.88:0/2109460291 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f00a5ce0 con 0x7f56f80c41f0 2023-10-16T20:43:41.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.436+0000 7f5700017640 1 -- 172.21.15.88:0/2109460291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56f80c41f0 msgr2=0x7f56f80cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:41.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.436+0000 7f5700017640 1 --2- 172.21.15.88:0/2109460291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56f80c41f0 0x7f56f80cece0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f56f0009e50 tx=0x7f56f001ca10 comp rx=0 tx=0).stop 2023-10-16T20:43:41.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.436+0000 7f5700017640 1 -- 172.21.15.88:0/2109460291 shutdown_connections 2023-10-16T20:43:41.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.436+0000 7f5700017640 1 --2- 172.21.15.88:0/2109460291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56f80c41f0 0x7f56f80cece0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:41.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.436+0000 7f5700017640 1 --2- 172.21.15.88:0/2109460291 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56f80c9bb0 0x7f56f80c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:41.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.436+0000 7f5700017640 1 -- 172.21.15.88:0/2109460291 >> 172.21.15.88:0/2109460291 conn(0x7f56f80715c0 msgr2=0x7f56f80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:41.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.436+0000 7f5700017640 1 -- 172.21.15.88:0/2109460291 shutdown_connections 2023-10-16T20:43:41.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.436+0000 7f5700017640 1 -- 172.21.15.88:0/2109460291 wait complete. 2023-10-16T20:43:41.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f5700017640 1 Processor -- start 2023-10-16T20:43:41.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f5700017640 1 -- start start 2023-10-16T20:43:41.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f5700017640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56f80c41f0 0x7f56f81f2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:41.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f5700017640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56f80c9bb0 0x7f56f80b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:41.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f5700017640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56f8153770 0x7f56f80b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:41.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f5700017640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f80c86d0 con 0x7f56f80c9bb0 2023-10-16T20:43:41.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f5700017640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f80ccec0 con 0x7f56f8153770 2023-10-16T20:43:41.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f5700017640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f81f5920 con 0x7f56f80c41f0 2023-10-16T20:43:41.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f56fd5b2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56f80c9bb0 0x7f56f80b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:41.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f56fd5b2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56f80c9bb0 0x7f56f80b42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57596/0 (socket says 172.21.15.88:57596) 2023-10-16T20:43:41.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f56fd5b2640 1 -- 172.21.15.88:0/3279261235 learned_addr learned my addr 172.21.15.88:0/3279261235 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:41.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f56fe5b4640 1 --2- 172.21.15.88:0/3279261235 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56f8153770 0x7f56f80b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:41.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f56fddb3640 1 --2- 172.21.15.88:0/3279261235 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56f80c41f0 0x7f56f81f2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:41.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f56fd5b2640 1 -- 172.21.15.88:0/3279261235 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56f80c41f0 msgr2=0x7f56f81f2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:41.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f56fd5b2640 1 --2- 172.21.15.88:0/3279261235 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56f80c41f0 0x7f56f81f2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:41.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f56fd5b2640 1 -- 172.21.15.88:0/3279261235 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56f8153770 msgr2=0x7f56f80b47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:41.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f56fd5b2640 1 --2- 172.21.15.88:0/3279261235 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56f8153770 0x7f56f80b47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:41.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f56fd5b2640 1 -- 172.21.15.88:0/3279261235 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f0009b30 con 0x7f56f80c9bb0 2023-10-16T20:43:41.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f56fd5b2640 1 --2- 172.21.15.88:0/3279261235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56f80c9bb0 0x7f56f80b42a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f56e800c7f0 tx=0x7f56e800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:41.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f56ee7fc640 1 -- 172.21.15.88:0/3279261235 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e800cea0 con 0x7f56f80c9bb0 2023-10-16T20:43:41.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f56ee7fc640 1 -- 172.21.15.88:0/3279261235 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e8007a30 con 0x7f56f80c9bb0 2023-10-16T20:43:41.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f56ee7fc640 1 -- 172.21.15.88:0/3279261235 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e8007de0 con 0x7f56f80c9bb0 2023-10-16T20:43:41.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f5700017640 1 -- 172.21.15.88:0/3279261235 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f81f5e50 con 0x7f56f80c9bb0 2023-10-16T20:43:41.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f5700017640 1 -- 172.21.15.88:0/3279261235 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f81f1eb0 con 0x7f56f80c9bb0 2023-10-16T20:43:41.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.440+0000 7f5700017640 1 -- 172.21.15.88:0/3279261235 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f8074700 con 0x7f56f80c9bb0 2023-10-16T20:43:41.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.444+0000 7f56ee7fc640 1 -- 172.21.15.88:0/3279261235 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f56e80113f0 con 0x7f56f80c9bb0 2023-10-16T20:43:41.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.444+0000 7f56ee7fc640 1 --2- 172.21.15.88:0/3279261235 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56dc0664d0 0x7f56dc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:41.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.444+0000 7f56ee7fc640 1 -- 172.21.15.88:0/3279261235 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7f56e8084280 con 0x7f56f80c9bb0 2023-10-16T20:43:41.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.444+0000 7f56fddb3640 1 --2- 172.21.15.88:0/3279261235 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56dc0664d0 0x7f56dc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:41.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.444+0000 7f56fddb3640 1 --2- 172.21.15.88:0/3279261235 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56dc0664d0 0x7f56dc068990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f56f4000fc0 tx=0x7f56f4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:41.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.448+0000 7f56ee7fc640 1 -- 172.21.15.88:0/3279261235 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f56e8051320 con 0x7f56f80c9bb0 2023-10-16T20:43:41.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.632+0000 7f5700017640 1 -- 172.21.15.88:0/3279261235 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56f80ca740 con 0x7f56f80c9bb0 2023-10-16T20:43:41.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.632+0000 7f56ee7fc640 1 -- 172.21.15.88:0/3279261235 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5778 (secure 0 0 0) 0x7f56e8054940 con 0x7f56f80c9bb0 2023-10-16T20:43:41.637 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:41.637 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:43:41.451904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:41.637 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 33 2023-10-16T20:43:41.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.632+0000 7f5700017640 1 -- 172.21.15.88:0/3279261235 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56dc0664d0 msgr2=0x7f56dc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:41.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.632+0000 7f5700017640 1 --2- 172.21.15.88:0/3279261235 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56dc0664d0 0x7f56dc068990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f56f4000fc0 tx=0x7f56f4009040 comp rx=0 tx=0).stop 2023-10-16T20:43:41.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.632+0000 7f5700017640 1 -- 172.21.15.88:0/3279261235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56f80c9bb0 msgr2=0x7f56f80b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:41.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.632+0000 7f5700017640 1 --2- 172.21.15.88:0/3279261235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56f80c9bb0 0x7f56f80b42a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f56e800c7f0 tx=0x7f56e800ccb0 comp rx=0 tx=0).stop 2023-10-16T20:43:41.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.636+0000 7f5700017640 1 -- 172.21.15.88:0/3279261235 shutdown_connections 2023-10-16T20:43:41.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.636+0000 7f5700017640 1 --2- 172.21.15.88:0/3279261235 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56dc0664d0 0x7f56dc068990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:41.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.636+0000 7f5700017640 1 --2- 172.21.15.88:0/3279261235 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56f8153770 0x7f56f80b47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:41.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.636+0000 7f5700017640 1 --2- 172.21.15.88:0/3279261235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56f80c9bb0 0x7f56f80b42a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:41.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.636+0000 7f5700017640 1 --2- 172.21.15.88:0/3279261235 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56f80c41f0 0x7f56f81f2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:41.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.636+0000 7f5700017640 1 -- 172.21.15.88:0/3279261235 >> 172.21.15.88:0/3279261235 conn(0x7f56f80715c0 msgr2=0x7f56f8152480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:41.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.636+0000 7f5700017640 1 -- 172.21.15.88:0/3279261235 shutdown_connections 2023-10-16T20:43:41.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:41.636+0000 7f5700017640 1 -- 172.21.15.88:0/3279261235 wait complete. 2023-10-16T20:43:41.657 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:43:43.658 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:43.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.804+0000 7f6cf6965640 1 Processor -- start 2023-10-16T20:43:43.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.804+0000 7f6cf6965640 1 -- start start 2023-10-16T20:43:43.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.804+0000 7f6cf6965640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6cf0153770 0x7f6cf0159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:43.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.804+0000 7f6cf6965640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f6cf00c9bb0 0x7f6cf00c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:43.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.804+0000 7f6cf6965640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf015a410 con 0x7f6cf00c41f0 2023-10-16T20:43:43.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.804+0000 7f6cf6965640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf00cc250 con 0x7f6cf0153770 2023-10-16T20:43:43.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.804+0000 7f6cf6965640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf00cc670 con 0x7f6cf00c9bb0 2023-10-16T20:43:43.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.804+0000 7f6ceffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6cf0153770 0x7f6cf0159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:43.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.804+0000 7f6ceffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6cf0153770 0x7f6cf0159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57604/0 (socket says 172.21.15.88:57604) 2023-10-16T20:43:43.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.804+0000 7f6ceffff640 1 -- 172.21.15.88:0/744888501 learned_addr learned my addr 172.21.15.88:0/744888501 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:43.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.804+0000 7f6ceffff640 1 -- 172.21.15.88:0/744888501 >> v2:172.21.15.191:3300/0 conn(0x7f6cf00c9bb0 msgr2=0x7f6cf00c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:43.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.804+0000 7f6ceffff640 1 --2- 172.21.15.88:0/744888501 >> v2:172.21.15.191:3300/0 conn(0x7f6cf00c9bb0 0x7f6cf00c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:43.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.804+0000 7f6ceffff640 1 -- 172.21.15.88:0/744888501 >> v1:172.21.15.191:6789/0 conn(0x7f6cf00c41f0 legacy=0x7f6cf00cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:43.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.804+0000 7f6ceffff640 1 -- 172.21.15.88:0/744888501 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf0153b50 con 0x7f6cf0153770 2023-10-16T20:43:43.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.808+0000 7f6ceffff640 1 --2- 172.21.15.88:0/744888501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf0153770 0x7f6cf0159ed0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6cf00cc7b0 tx=0x7f6ce001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d21f9dd4f0b86 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:43.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.808+0000 7f6ceeffd640 1 -- 172.21.15.88:0/744888501 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce00a6040 con 0x7f6cf0153770 2023-10-16T20:43:43.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.808+0000 7f6ceeffd640 1 -- 172.21.15.88:0/744888501 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce00aa600 con 0x7f6cf0153770 2023-10-16T20:43:43.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.808+0000 7f6ceeffd640 1 -- 172.21.15.88:0/744888501 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce00aa990 con 0x7f6cf0153770 2023-10-16T20:43:43.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.808+0000 7f6cf6965640 1 -- 172.21.15.88:0/744888501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf0153770 msgr2=0x7f6cf0159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:43.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.808+0000 7f6cf6965640 1 --2- 172.21.15.88:0/744888501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf0153770 0x7f6cf0159ed0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6cf00cc7b0 tx=0x7f6ce001c4a0 comp rx=0 tx=0).stop 2023-10-16T20:43:43.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.808+0000 7f6cf6965640 1 -- 172.21.15.88:0/744888501 shutdown_connections 2023-10-16T20:43:43.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.808+0000 7f6cf6965640 1 --2- 172.21.15.88:0/744888501 >> v2:172.21.15.191:3300/0 conn(0x7f6cf00c9bb0 0x7f6cf00c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:43.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.808+0000 7f6cf6965640 1 --2- 172.21.15.88:0/744888501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf0153770 0x7f6cf0159ed0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:43.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.808+0000 7f6cf6965640 1 -- 172.21.15.88:0/744888501 >> 172.21.15.88:0/744888501 conn(0x7f6cf00715c0 msgr2=0x7f6cf00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:43.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.808+0000 7f6cf6965640 1 -- 172.21.15.88:0/744888501 shutdown_connections 2023-10-16T20:43:43.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.808+0000 7f6cf6965640 1 -- 172.21.15.88:0/744888501 wait complete. 2023-10-16T20:43:43.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6cf6965640 1 Processor -- start 2023-10-16T20:43:43.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6cf6965640 1 -- start start 2023-10-16T20:43:43.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6cf6965640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf00c41f0 0x7f6cf0146600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:43.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6cf6965640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6cf00c9bb0 0x7f6cf0147c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:43.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6cf6965640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6cf0153770 0x7f6cf01481d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:43.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6cf6965640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf0147870 con 0x7f6cf00c41f0 2023-10-16T20:43:43.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6cf6965640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf00b42b0 con 0x7f6cf0153770 2023-10-16T20:43:43.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6cf6965640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf01474b0 con 0x7f6cf00c9bb0 2023-10-16T20:43:43.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6ceffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf00c41f0 0x7f6cf0146600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:43.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6ceffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf00c41f0 0x7f6cf0146600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57614/0 (socket says 172.21.15.88:57614) 2023-10-16T20:43:43.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6ceffff640 1 -- 172.21.15.88:0/3004588461 learned_addr learned my addr 172.21.15.88:0/3004588461 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:43.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6cf4f02640 1 --2- 172.21.15.88:0/3004588461 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6cf0153770 0x7f6cf01481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:43.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6cef7fe640 1 --2- 172.21.15.88:0/3004588461 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6cf00c9bb0 0x7f6cf0147c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:43.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6ceffff640 1 -- 172.21.15.88:0/3004588461 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6cf00c9bb0 msgr2=0x7f6cf0147c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:43.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6ceffff640 1 --2- 172.21.15.88:0/3004588461 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6cf00c9bb0 0x7f6cf0147c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:43.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6ceffff640 1 -- 172.21.15.88:0/3004588461 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6cf0153770 msgr2=0x7f6cf01481d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:43.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6ceffff640 1 --2- 172.21.15.88:0/3004588461 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6cf0153770 0x7f6cf01481d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:43.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6ceffff640 1 -- 172.21.15.88:0/3004588461 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce0009480 con 0x7f6cf00c41f0 2023-10-16T20:43:43.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6cef7fe640 1 --2- 172.21.15.88:0/3004588461 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6cf00c9bb0 0x7f6cf0147c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:43:43.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6ceffff640 1 --2- 172.21.15.88:0/3004588461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf00c41f0 0x7f6cf0146600 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00ac040 tx=0x7f6ce0019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:43.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6cecff9640 1 -- 172.21.15.88:0/3004588461 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce00aac40 con 0x7f6cf00c41f0 2023-10-16T20:43:43.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6cecff9640 1 -- 172.21.15.88:0/3004588461 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce00aada0 con 0x7f6cf00c41f0 2023-10-16T20:43:43.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6cecff9640 1 -- 172.21.15.88:0/3004588461 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0003860 con 0x7f6cf00c41f0 2023-10-16T20:43:43.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6cf6965640 1 -- 172.21.15.88:0/3004588461 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf00b44e0 con 0x7f6cf00c41f0 2023-10-16T20:43:43.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6cf6965640 1 -- 172.21.15.88:0/3004588461 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf01f2520 con 0x7f6cf00c41f0 2023-10-16T20:43:43.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.812+0000 7f6cf6965640 1 -- 172.21.15.88:0/3004588461 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf01f2710 con 0x7f6cf00c41f0 2023-10-16T20:43:43.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.816+0000 7f6cecff9640 1 -- 172.21.15.88:0/3004588461 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6ce00039c0 con 0x7f6cf00c41f0 2023-10-16T20:43:43.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.816+0000 7f6cecff9640 1 --2- 172.21.15.88:0/3004588461 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6cdc0664a0 0x7f6cdc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:43.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.816+0000 7f6cecff9640 1 -- 172.21.15.88:0/3004588461 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7f6ce011e400 con 0x7f6cf00c41f0 2023-10-16T20:43:43.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.816+0000 7f6cef7fe640 1 --2- 172.21.15.88:0/3004588461 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6cdc0664a0 0x7f6cdc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:43.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.816+0000 7f6cef7fe640 1 --2- 172.21.15.88:0/3004588461 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6cdc0664a0 0x7f6cdc068960 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4006fd0 tx=0x7f6ce4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:43.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:43.820+0000 7f6cecff9640 1 -- 172.21.15.88:0/3004588461 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6ce00eb370 con 0x7f6cf00c41f0 2023-10-16T20:43:44.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:44.000+0000 7f6cf6965640 1 -- 172.21.15.88:0/3004588461 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cf0074700 con 0x7f6cf00c41f0 2023-10-16T20:43:44.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:44.000+0000 7f6cecff9640 1 -- 172.21.15.88:0/3004588461 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5778 (secure 0 0 0) 0x7f6ce00ee990 con 0x7f6cf00c41f0 2023-10-16T20:43:44.005 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:44.005 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:43:41.451904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:44.005 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 33 2023-10-16T20:43:44.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:44.004+0000 7f6cf6965640 1 -- 172.21.15.88:0/3004588461 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6cdc0664a0 msgr2=0x7f6cdc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:44.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:44.004+0000 7f6cf6965640 1 --2- 172.21.15.88:0/3004588461 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6cdc0664a0 0x7f6cdc068960 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4006fd0 tx=0x7f6ce4008040 comp rx=0 tx=0).stop 2023-10-16T20:43:44.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:44.004+0000 7f6cf6965640 1 -- 172.21.15.88:0/3004588461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf00c41f0 msgr2=0x7f6cf0146600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:44.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:44.004+0000 7f6cf6965640 1 --2- 172.21.15.88:0/3004588461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf00c41f0 0x7f6cf0146600 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00ac040 tx=0x7f6ce0019240 comp rx=0 tx=0).stop 2023-10-16T20:43:44.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:44.004+0000 7f6cf6965640 1 -- 172.21.15.88:0/3004588461 shutdown_connections 2023-10-16T20:43:44.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:44.004+0000 7f6cf6965640 1 --2- 172.21.15.88:0/3004588461 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6cdc0664a0 0x7f6cdc068960 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:44.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:44.004+0000 7f6cf6965640 1 --2- 172.21.15.88:0/3004588461 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6cf0153770 0x7f6cf01481d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:44.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:44.004+0000 7f6cf6965640 1 --2- 172.21.15.88:0/3004588461 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6cf00c9bb0 0x7f6cf0147c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:44.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:44.004+0000 7f6cf6965640 1 --2- 172.21.15.88:0/3004588461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf00c41f0 0x7f6cf0146600 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:44.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:44.004+0000 7f6cf6965640 1 -- 172.21.15.88:0/3004588461 >> 172.21.15.88:0/3004588461 conn(0x7f6cf00715c0 msgr2=0x7f6cf00c4620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:44.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:44.004+0000 7f6cf6965640 1 -- 172.21.15.88:0/3004588461 shutdown_connections 2023-10-16T20:43:44.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:44.004+0000 7f6cf6965640 1 -- 172.21.15.88:0/3004588461 wait complete. 2023-10-16T20:43:44.026 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:43:46.027 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:46.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.172+0000 7f2c732d6640 1 Processor -- start 2023-10-16T20:43:46.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.172+0000 7f2c732d6640 1 -- start start 2023-10-16T20:43:46.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.172+0000 7f2c732d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2c6c150280 0x7f2c6c0c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:46.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.172+0000 7f2c732d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c6c0c6bc0 0x7f2c6c0c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:46.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.172+0000 7f2c732d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6c0c5c60 con 0x7f2c6c150660 2023-10-16T20:43:46.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.172+0000 7f2c732d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6c0c5370 con 0x7f2c6c150280 2023-10-16T20:43:46.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.172+0000 7f2c732d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c156d10 con 0x7f2c6c0c6bc0 2023-10-16T20:43:46.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.172+0000 7f2c71072640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2c6c150280 0x7f2c6c0c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:46.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.172+0000 7f2c71072640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2c6c150280 0x7f2c6c0c60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57620/0 (socket says 172.21.15.88:57620) 2023-10-16T20:43:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.172+0000 7f2c71072640 1 -- 172.21.15.88:0/579423953 learned_addr learned my addr 172.21.15.88:0/579423953 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:46.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.172+0000 7f2c70871640 1 --1- 172.21.15.88:0/579423953 >> v1:172.21.15.88:6789/0 conn(0x7f2c6c150660 0x7f2c6c06d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58572/0 (socket says 172.21.15.88:58572) 2023-10-16T20:43:46.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.172+0000 7f2c71873640 1 --2- 172.21.15.88:0/579423953 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c6c0c6bc0 0x7f2c6c0c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:46.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.172+0000 7f2c71072640 1 -- 172.21.15.88:0/579423953 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c6c0c6bc0 msgr2=0x7f2c6c0c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:46.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.176+0000 7f2c71072640 1 --2- 172.21.15.88:0/579423953 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c6c0c6bc0 0x7f2c6c0c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:46.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.176+0000 7f2c71072640 1 -- 172.21.15.88:0/579423953 >> v1:172.21.15.88:6789/0 conn(0x7f2c6c150660 legacy=0x7f2c6c06d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:46.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.176+0000 7f2c71072640 1 -- 172.21.15.88:0/579423953 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6c0c5610 con 0x7f2c6c150280 2023-10-16T20:43:46.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.176+0000 7f2c71873640 1 --2- 172.21.15.88:0/579423953 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c6c0c6bc0 0x7f2c6c0c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:43:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.176+0000 7f2c71072640 1 --2- 172.21.15.88:0/579423953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c6c150280 0x7f2c6c0c60f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2c68009870 tx=0x7f2c6801cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78781da5c9e647cd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.176+0000 7f2c63fff640 1 -- 172.21.15.88:0/579423953 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c680a95f0 con 0x7f2c6c150280 2023-10-16T20:43:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.176+0000 7f2c63fff640 1 -- 172.21.15.88:0/579423953 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c680a9750 con 0x7f2c6c150280 2023-10-16T20:43:46.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.176+0000 7f2c63fff640 1 -- 172.21.15.88:0/579423953 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c680a9ae0 con 0x7f2c6c150280 2023-10-16T20:43:46.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.176+0000 7f2c732d6640 1 -- 172.21.15.88:0/579423953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c6c150280 msgr2=0x7f2c6c0c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:46.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.176+0000 7f2c732d6640 1 --2- 172.21.15.88:0/579423953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c6c150280 0x7f2c6c0c60f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2c68009870 tx=0x7f2c6801cf70 comp rx=0 tx=0).stop 2023-10-16T20:43:46.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.176+0000 7f2c732d6640 1 -- 172.21.15.88:0/579423953 shutdown_connections 2023-10-16T20:43:46.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.176+0000 7f2c732d6640 1 --2- 172.21.15.88:0/579423953 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c6c0c6bc0 0x7f2c6c0c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:46.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.176+0000 7f2c732d6640 1 --2- 172.21.15.88:0/579423953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c6c150280 0x7f2c6c0c60f0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:46.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.176+0000 7f2c732d6640 1 -- 172.21.15.88:0/579423953 >> 172.21.15.88:0/579423953 conn(0x7f2c6c0cad50 msgr2=0x7f2c6c0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:46.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c732d6640 1 -- 172.21.15.88:0/579423953 shutdown_connections 2023-10-16T20:43:46.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c732d6640 1 -- 172.21.15.88:0/579423953 wait complete. 2023-10-16T20:43:46.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c732d6640 1 Processor -- start 2023-10-16T20:43:46.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c732d6640 1 -- start start 2023-10-16T20:43:46.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c732d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c6c0c6bc0 0x7f2c6c140bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:46.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c732d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c6c150280 0x7f2c6c141b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:46.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c732d6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c6c150660 0x7f2c6c1553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:46.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c732d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c1410f0 con 0x7f2c6c0c6bc0 2023-10-16T20:43:46.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c732d6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c1f18a0 con 0x7f2c6c150660 2023-10-16T20:43:46.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c732d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c1f4d00 con 0x7f2c6c150280 2023-10-16T20:43:46.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c71072640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c6c0c6bc0 0x7f2c6c140bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:46.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c71072640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c6c0c6bc0 0x7f2c6c140bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57626/0 (socket says 172.21.15.88:57626) 2023-10-16T20:43:46.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c71072640 1 -- 172.21.15.88:0/392719245 learned_addr learned my addr 172.21.15.88:0/392719245 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:46.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c71072640 1 -- 172.21.15.88:0/392719245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c6c150280 msgr2=0x7f2c6c141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:46.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c71072640 1 --2- 172.21.15.88:0/392719245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c6c150280 0x7f2c6c141b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:46.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c71072640 1 -- 172.21.15.88:0/392719245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c6c150660 msgr2=0x7f2c6c1553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:46.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c71072640 1 --2- 172.21.15.88:0/392719245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c6c150660 0x7f2c6c1553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:46.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c71072640 1 -- 172.21.15.88:0/392719245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c680094d0 con 0x7f2c6c0c6bc0 2023-10-16T20:43:46.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c71072640 1 --2- 172.21.15.88:0/392719245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c6c0c6bc0 0x7f2c6c140bb0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2c6801a140 tx=0x7f2c6801ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:46.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c61ffb640 1 -- 172.21.15.88:0/392719245 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c680b1800 con 0x7f2c6c0c6bc0 2023-10-16T20:43:46.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.180+0000 7f2c61ffb640 1 -- 172.21.15.88:0/392719245 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c680b1980 con 0x7f2c6c0c6bc0 2023-10-16T20:43:46.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.184+0000 7f2c732d6640 1 -- 172.21.15.88:0/392719245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c6c1f4fc0 con 0x7f2c6c0c6bc0 2023-10-16T20:43:46.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.184+0000 7f2c61ffb640 1 -- 172.21.15.88:0/392719245 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c680b1d30 con 0x7f2c6c0c6bc0 2023-10-16T20:43:46.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.184+0000 7f2c732d6640 1 -- 172.21.15.88:0/392719245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c6c1f5580 con 0x7f2c6c0c6bc0 2023-10-16T20:43:46.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.184+0000 7f2c732d6640 1 -- 172.21.15.88:0/392719245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c6c1f1a70 con 0x7f2c6c0c6bc0 2023-10-16T20:43:46.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.184+0000 7f2c61ffb640 1 -- 172.21.15.88:0/392719245 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c680bc450 con 0x7f2c6c0c6bc0 2023-10-16T20:43:46.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.184+0000 7f2c61ffb640 1 --2- 172.21.15.88:0/392719245 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2c500664a0 0x7f2c50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:46.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.184+0000 7f2c61ffb640 1 -- 172.21.15.88:0/392719245 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7f2c6811f190 con 0x7f2c6c0c6bc0 2023-10-16T20:43:46.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.184+0000 7f2c70871640 1 --2- 172.21.15.88:0/392719245 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2c500664a0 0x7f2c50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:46.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.184+0000 7f2c70871640 1 --2- 172.21.15.88:0/392719245 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2c500664a0 0x7f2c50068960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0c1c30 tx=0x7f2c5c0079d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:46.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.188+0000 7f2c61ffb640 1 -- 172.21.15.88:0/392719245 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c6c1f1a70 con 0x7f2c6c0c6bc0 2023-10-16T20:43:46.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.368+0000 7f2c732d6640 1 -- 172.21.15.88:0/392719245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c6c1f1a70 con 0x7f2c6c0c6bc0 2023-10-16T20:43:46.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.368+0000 7f2c61ffb640 1 -- 172.21.15.88:0/392719245 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5778 (secure 0 0 0) 0x7f2c6c1f1a70 con 0x7f2c6c0c6bc0 2023-10-16T20:43:46.374 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:46.374 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:43:41.451904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:46.375 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 33 2023-10-16T20:43:46.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.372+0000 7f2c732d6640 1 -- 172.21.15.88:0/392719245 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2c500664a0 msgr2=0x7f2c50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:46.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.372+0000 7f2c732d6640 1 --2- 172.21.15.88:0/392719245 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2c500664a0 0x7f2c50068960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0c1c30 tx=0x7f2c5c0079d0 comp rx=0 tx=0).stop 2023-10-16T20:43:46.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.372+0000 7f2c732d6640 1 -- 172.21.15.88:0/392719245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c6c0c6bc0 msgr2=0x7f2c6c140bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:46.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.372+0000 7f2c732d6640 1 --2- 172.21.15.88:0/392719245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c6c0c6bc0 0x7f2c6c140bb0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2c6801a140 tx=0x7f2c6801ad50 comp rx=0 tx=0).stop 2023-10-16T20:43:46.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.372+0000 7f2c732d6640 1 -- 172.21.15.88:0/392719245 shutdown_connections 2023-10-16T20:43:46.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.372+0000 7f2c732d6640 1 --2- 172.21.15.88:0/392719245 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2c500664a0 0x7f2c50068960 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:46.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.372+0000 7f2c732d6640 1 --2- 172.21.15.88:0/392719245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2c6c150660 0x7f2c6c1553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:46.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.372+0000 7f2c732d6640 1 --2- 172.21.15.88:0/392719245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2c6c150280 0x7f2c6c141b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:46.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.372+0000 7f2c732d6640 1 --2- 172.21.15.88:0/392719245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c6c0c6bc0 0x7f2c6c140bb0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:46.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.372+0000 7f2c732d6640 1 -- 172.21.15.88:0/392719245 >> 172.21.15.88:0/392719245 conn(0x7f2c6c0cad50 msgr2=0x7f2c6c13e330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:46.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.372+0000 7f2c732d6640 1 -- 172.21.15.88:0/392719245 shutdown_connections 2023-10-16T20:43:46.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:46.372+0000 7f2c732d6640 1 -- 172.21.15.88:0/392719245 wait complete. 2023-10-16T20:43:46.394 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:43:48.395 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:48.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.540+0000 7f9304491640 1 Processor -- start 2023-10-16T20:43:48.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.540+0000 7f9304491640 1 -- start start 2023-10-16T20:43:48.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.540+0000 7f9304491640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92fc1406d0 0x7f92fc0c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:48.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.540+0000 7f9304491640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92fc15b3c0 con 0x7f92fc15a810 2023-10-16T20:43:48.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.540+0000 7f9304491640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92fc15b680 con 0x7f92fc06a970 2023-10-16T20:43:48.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.540+0000 7f9304491640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc157570 con 0x7f92fc1406d0 2023-10-16T20:43:48.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.540+0000 7f9302a2e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f92fc15a810 0x7f92fc161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58584/0 (socket says 172.21.15.88:58584) 2023-10-16T20:43:48.546 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f9302a2e640 1 -- 172.21.15.88:0/445257604 learned_addr learned my addr 172.21.15.88:0/445257604 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:43:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930222d640 1 --2- 172.21.15.88:0/445257604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92fc1406d0 0x7f92fc0c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:48.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930122b640 1 -- 172.21.15.88:0/445257604 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901078107 0 0) 0x7f92fc15b3c0 con 0x7f92fc15a810 2023-10-16T20:43:48.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930122b640 1 -- 172.21.15.88:0/445257604 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e8003530 con 0x7f92fc15a810 2023-10-16T20:43:48.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930122b640 1 -- 172.21.15.88:0/445257604 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958706548 0 0) 0x7f92fc15b680 con 0x7f92fc06a970 2023-10-16T20:43:48.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930122b640 1 -- 172.21.15.88:0/445257604 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92fc15b3c0 con 0x7f92fc06a970 2023-10-16T20:43:48.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930122b640 1 -- 172.21.15.88:0/445257604 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3051016792 0 0) 0x7f92e8003530 con 0x7f92fc15a810 2023-10-16T20:43:48.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930122b640 1 -- 172.21.15.88:0/445257604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92fc1406d0 msgr2=0x7f92fc0c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:48.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930122b640 1 --2- 172.21.15.88:0/445257604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92fc1406d0 0x7f92fc0c1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:48.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930122b640 1 -- 172.21.15.88:0/445257604 >> v1:172.21.15.191:6789/0 conn(0x7f92fc06a970 legacy=0x7f92fc071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:48.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930122b640 1 -- 172.21.15.88:0/445257604 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92fc157850 con 0x7f92fc15a810 2023-10-16T20:43:48.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930122b640 1 -- 172.21.15.88:0/445257604 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f92f4001a00 con 0x7f92fc15a810 2023-10-16T20:43:48.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930122b640 1 -- 172.21.15.88:0/445257604 >> v1:172.21.15.88:6789/0 conn(0x7f92fc15a810 legacy=0x7f92fc161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:48.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930122b640 1 --2- 172.21.15.88:0/445257604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92e8003b40 0x7f92e8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:48.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930122b640 1 --2- 172.21.15.88:0/445257604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92fc1406d0 0x7f92e80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:48.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930122b640 1 --2- 172.21.15.88:0/445257604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92e8004d10 0x7f92e8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:48.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930122b640 1 -- 172.21.15.88:0/445257604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc157570 con 0x7f92e8003b40 2023-10-16T20:43:48.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930122b640 1 -- 172.21.15.88:0/445257604 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e800ba40 con 0x7f92e8004d10 2023-10-16T20:43:48.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930122b640 1 -- 172.21.15.88:0/445257604 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e800bbb0 con 0x7f92fc1406d0 2023-10-16T20:43:48.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930222d640 1 --2- 172.21.15.88:0/445257604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92e8003b40 0x7f92e8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:48.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f9302a2e640 1 --2- 172.21.15.88:0/445257604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92e8004d10 0x7f92e8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:48.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f9301a2c640 1 --2- 172.21.15.88:0/445257604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92fc1406d0 0x7f92e80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:48.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930222d640 1 -- 172.21.15.88:0/445257604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92fc1406d0 msgr2=0x7f92e80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:48.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930222d640 1 --2- 172.21.15.88:0/445257604 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92fc1406d0 0x7f92e80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:48.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930222d640 1 -- 172.21.15.88:0/445257604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92e8004d10 msgr2=0x7f92e8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:48.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930222d640 1 --2- 172.21.15.88:0/445257604 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92e8004d10 0x7f92e8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:48.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.544+0000 7f930222d640 1 -- 172.21.15.88:0/445257604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e8003530 con 0x7f92e8003b40 2023-10-16T20:43:48.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.548+0000 7f930222d640 1 --2- 172.21.15.88:0/445257604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92e8003b40 0x7f92e8003f40 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f92f8009ae0 tx=0x7f92f801ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d3ae9d8f8d61b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:48.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.548+0000 7f930122b640 1 -- 172.21.15.88:0/445257604 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f80a5550 con 0x7f92e8003b40 2023-10-16T20:43:48.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.548+0000 7f930122b640 1 -- 172.21.15.88:0/445257604 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f80a56b0 con 0x7f92e8003b40 2023-10-16T20:43:48.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.548+0000 7f930122b640 1 -- 172.21.15.88:0/445257604 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f80a5a60 con 0x7f92e8003b40 2023-10-16T20:43:48.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.548+0000 7f9304491640 1 -- 172.21.15.88:0/445257604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92e8003b40 msgr2=0x7f92e8003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:48.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.548+0000 7f9304491640 1 --2- 172.21.15.88:0/445257604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92e8003b40 0x7f92e8003f40 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f92f8009ae0 tx=0x7f92f801ca20 comp rx=0 tx=0).stop 2023-10-16T20:43:48.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.548+0000 7f9302a2e640 1 -- 172.21.15.88:0/445257604 reap_dead start 2023-10-16T20:43:48.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.548+0000 7f9304491640 1 -- 172.21.15.88:0/445257604 shutdown_connections 2023-10-16T20:43:48.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.548+0000 7f9304491640 1 -- 172.21.15.88:0/445257604 >> 172.21.15.88:0/445257604 conn(0x7f92fc0cad50 msgr2=0x7f92fc0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:48.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.548+0000 7f9304491640 1 -- 172.21.15.88:0/445257604 shutdown_connections 2023-10-16T20:43:48.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.548+0000 7f9304491640 1 -- 172.21.15.88:0/445257604 wait complete. 2023-10-16T20:43:48.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.548+0000 7f9304491640 1 Processor -- start 2023-10-16T20:43:48.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.548+0000 7f9304491640 1 -- start start 2023-10-16T20:43:48.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.548+0000 7f9304491640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92e8003b40 0x7f92fc159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:48.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9304491640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92fc151210 0x7f92fc151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:48.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9304491640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92fc06f8c0 0x7f92fc06fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:48.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9304491640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc152e30 con 0x7f92fc06f8c0 2023-10-16T20:43:48.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9304491640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc154600 con 0x7f92fc151210 2023-10-16T20:43:48.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9304491640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc070290 con 0x7f92e8003b40 2023-10-16T20:43:48.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9302a2e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92fc06f8c0 0x7f92fc06fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:48.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9302a2e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92fc06f8c0 0x7f92fc06fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57640/0 (socket says 172.21.15.88:57640) 2023-10-16T20:43:48.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9302a2e640 1 -- 172.21.15.88:0/3333285261 learned_addr learned my addr 172.21.15.88:0/3333285261 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:48.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f930222d640 1 --2- 172.21.15.88:0/3333285261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92e8003b40 0x7f92fc159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:48.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9301a2c640 1 --2- 172.21.15.88:0/3333285261 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92fc151210 0x7f92fc151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:48.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9302a2e640 1 -- 172.21.15.88:0/3333285261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92e8003b40 msgr2=0x7f92fc159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:48.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9302a2e640 1 --2- 172.21.15.88:0/3333285261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92e8003b40 0x7f92fc159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:48.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9302a2e640 1 -- 172.21.15.88:0/3333285261 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92fc151210 msgr2=0x7f92fc151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:48.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9302a2e640 1 --2- 172.21.15.88:0/3333285261 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92fc151210 0x7f92fc151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:48.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9302a2e640 1 -- 172.21.15.88:0/3333285261 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92fc15a530 con 0x7f92fc06f8c0 2023-10-16T20:43:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9302a2e640 1 --2- 172.21.15.88:0/3333285261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92fc06f8c0 0x7f92fc06fd50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f92fc15abf0 tx=0x7f92f4007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f92f2ffd640 1 -- 172.21.15.88:0/3333285261 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f400ab90 con 0x7f92fc06f8c0 2023-10-16T20:43:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f92f2ffd640 1 -- 172.21.15.88:0/3333285261 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f400acf0 con 0x7f92fc06f8c0 2023-10-16T20:43:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f92f2ffd640 1 -- 172.21.15.88:0/3333285261 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f4013630 con 0x7f92fc06f8c0 2023-10-16T20:43:48.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9304491640 1 -- 172.21.15.88:0/3333285261 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92fc153cd0 con 0x7f92fc06f8c0 2023-10-16T20:43:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9304491640 1 -- 172.21.15.88:0/3333285261 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92fc154290 con 0x7f92fc06f8c0 2023-10-16T20:43:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f9304491640 1 -- 172.21.15.88:0/3333285261 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92fc074700 con 0x7f92fc06f8c0 2023-10-16T20:43:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.552+0000 7f92f2ffd640 1 -- 172.21.15.88:0/3333285261 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f92f4013790 con 0x7f92fc06f8c0 2023-10-16T20:43:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.556+0000 7f92f2ffd640 1 --2- 172.21.15.88:0/3333285261 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f92e00664a0 0x7f92e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.556+0000 7f92f2ffd640 1 -- 172.21.15.88:0/3333285261 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7f92f4084ae0 con 0x7f92fc06f8c0 2023-10-16T20:43:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.556+0000 7f930222d640 1 --2- 172.21.15.88:0/3333285261 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f92e00664a0 0x7f92e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.556+0000 7f930222d640 1 --2- 172.21.15.88:0/3333285261 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f92e00664a0 0x7f92e0068960 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f92f80af000 tx=0x7f92f80046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:48.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.560+0000 7f92f2ffd640 1 -- 172.21.15.88:0/3333285261 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f92f40519c0 con 0x7f92fc06f8c0 2023-10-16T20:43:48.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.740+0000 7f9304491640 1 -- 172.21.15.88:0/3333285261 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92fc0c3dc0 con 0x7f92fc06f8c0 2023-10-16T20:43:48.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.740+0000 7f92f2ffd640 1 -- 172.21.15.88:0/3333285261 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5778 (secure 0 0 0) 0x7f92f400ae60 con 0x7f92fc06f8c0 2023-10-16T20:43:48.745 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:48.746 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:43:46.483738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:48.746 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 34 2023-10-16T20:43:48.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.744+0000 7f9304491640 1 -- 172.21.15.88:0/3333285261 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f92e00664a0 msgr2=0x7f92e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:48.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.744+0000 7f9304491640 1 --2- 172.21.15.88:0/3333285261 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f92e00664a0 0x7f92e0068960 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f92f80af000 tx=0x7f92f80046d0 comp rx=0 tx=0).stop 2023-10-16T20:43:48.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.744+0000 7f9304491640 1 -- 172.21.15.88:0/3333285261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92fc06f8c0 msgr2=0x7f92fc06fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:48.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.744+0000 7f9304491640 1 --2- 172.21.15.88:0/3333285261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92fc06f8c0 0x7f92fc06fd50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f92fc15abf0 tx=0x7f92f4007bd0 comp rx=0 tx=0).stop 2023-10-16T20:43:48.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.744+0000 7f9304491640 1 -- 172.21.15.88:0/3333285261 shutdown_connections 2023-10-16T20:43:48.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.744+0000 7f9304491640 1 --2- 172.21.15.88:0/3333285261 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f92e00664a0 0x7f92e0068960 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:48.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.744+0000 7f9304491640 1 --2- 172.21.15.88:0/3333285261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92fc06f8c0 0x7f92fc06fd50 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:48.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.744+0000 7f9304491640 1 --2- 172.21.15.88:0/3333285261 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f92fc151210 0x7f92fc151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:48.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.744+0000 7f9304491640 1 --2- 172.21.15.88:0/3333285261 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f92e8003b40 0x7f92fc159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:48.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.744+0000 7f9304491640 1 -- 172.21.15.88:0/3333285261 >> 172.21.15.88:0/3333285261 conn(0x7f92fc0cad50 msgr2=0x7f92fc13f270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:48.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.744+0000 7f9304491640 1 -- 172.21.15.88:0/3333285261 shutdown_connections 2023-10-16T20:43:48.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:48.744+0000 7f9304491640 1 -- 172.21.15.88:0/3333285261 wait complete. 2023-10-16T20:43:48.766 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=4, replacing gid=4244 2023-10-16T20:43:48.766 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.c 2023-10-16T20:43:53.525 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2023-10-16T20:43:53.525 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2023-10-16T20:43:53.525 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2023-10-16T20:43:53.526 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2023-10-16T20:43:53.526 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2023-10-16T20:43:53.526 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2023-10-16T20:43:53.526 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2023-10-16T20:43:53.526 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2023-10-16T20:43:53.527 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2023-10-16T20:43:53.527 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2023-10-16T20:43:53.527 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2023-10-16T20:43:53.527 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2023-10-16T20:43:53.527 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2023-10-16T20:43:53.529 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2023-10-16T20:43:53.529 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2023-10-16T20:43:53.529 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2023-10-16T20:43:53.530 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2023-10-16T20:43:53.530 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2023-10-16T20:43:53.530 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2023-10-16T20:43:53.530 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2023-10-16T20:43:53.530 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2023-10-16T20:43:53.531 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2023-10-16T20:43:53.531 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2023-10-16T20:43:53.531 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2023-10-16T20:43:53.531 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2023-10-16T20:43:53.531 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2023-10-16T20:43:53.532 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2023-10-16T20:43:53.532 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2023-10-16T20:43:53.532 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2023-10-16T20:43:53.532 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2023-10-16T20:43:53.532 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2023-10-16T20:43:53.533 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2023-10-16T20:43:53.533 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2023-10-16T20:43:53.533 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2023-10-16T20:43:53.533 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2023-10-16T20:43:53.533 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2023-10-16T20:43:53.534 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2023-10-16T20:43:53.534 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2023-10-16T20:43:53.534 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2023-10-16T20:43:53.534 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2023-10-16T20:43:53.534 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2023-10-16T20:43:53.535 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2023-10-16T20:43:53.535 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2023-10-16T20:43:53.535 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2023-10-16T20:43:53.535 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2023-10-16T20:43:53.535 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2023-10-16T20:43:53.535 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2023-10-16T20:43:53.536 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2023-10-16T20:43:53.536 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2023-10-16T20:43:53.536 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2023-10-16T20:43:53.536 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2023-10-16T20:43:53.536 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2023-10-16T20:43:53.537 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2023-10-16T20:43:53.537 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2023-10-16T20:43:53.537 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2023-10-16T20:43:53.537 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2023-10-16T20:43:53.537 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2023-10-16T20:43:53.537 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2023-10-16T20:43:53.538 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2023-10-16T20:43:53.538 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2023-10-16T20:43:53.538 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2023-10-16T20:43:53.538 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2023-10-16T20:43:53.538 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2023-10-16T20:43:53.539 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2023-10-16T20:43:53.539 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2023-10-16T20:43:53.539 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2023-10-16T20:43:53.539 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2023-10-16T20:43:53.539 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2023-10-16T20:43:53.539 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2023-10-16T20:43:53.540 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2023-10-16T20:43:53.540 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2023-10-16T20:43:53.540 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2023-10-16T20:43:53.540 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2023-10-16T20:43:53.540 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2023-10-16T20:43:53.541 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2023-10-16T20:43:53.541 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2023-10-16T20:43:53.541 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2023-10-16T20:43:53.541 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2023-10-16T20:43:53.541 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2023-10-16T20:43:53.542 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2023-10-16T20:43:53.542 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2023-10-16T20:43:53.767 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-16T20:43:53.768 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T20:43:53.768 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T20:43:53.773 INFO:tasks.ceph.mds.c:Started 2023-10-16T20:43:53.773 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:53.862 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2023-10-16T20:43:53.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.912+0000 7ff5514c9640 1 Processor -- start 2023-10-16T20:43:53.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.912+0000 7ff5514c9640 1 -- start start 2023-10-16T20:43:53.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.916+0000 7ff5514c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff54c0c9a00 0x7ff54c0ca660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:53.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.916+0000 7ff5514c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff54c153940 0x7ff54c0caba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:53.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.916+0000 7ff5514c9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff54c159f10 con 0x7ff54c153560 2023-10-16T20:43:53.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.916+0000 7ff5514c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff54c1593f0 con 0x7ff54c0c9a00 2023-10-16T20:43:53.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.916+0000 7ff5514c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c15a3a0 con 0x7ff54c153940 2023-10-16T20:43:53.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.916+0000 7ff5427fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff54c0c9a00 0x7ff54c0ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:53.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.916+0000 7ff5427fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff54c0c9a00 0x7ff54c0ca660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60306/0 (socket says 172.21.15.88:60306) 2023-10-16T20:43:53.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.916+0000 7ff5427fc640 1 -- 172.21.15.88:0/2611832507 learned_addr learned my addr 172.21.15.88:0/2611832507 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:53.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.916+0000 7ff54b7fe640 1 --2- 172.21.15.88:0/2611832507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff54c153940 0x7ff54c0caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:53.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.916+0000 7ff5427fc640 1 -- 172.21.15.88:0/2611832507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff54c153940 msgr2=0x7ff54c0caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:43:53.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.916+0000 7ff5427fc640 1 --2- 172.21.15.88:0/2611832507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff54c153940 0x7ff54c0caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:53.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.916+0000 7ff5427fc640 1 -- 172.21.15.88:0/2611832507 >> v1:172.21.15.191:6789/0 conn(0x7ff54c153560 legacy=0x7ff54c0b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:53.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.916+0000 7ff5427fc640 1 -- 172.21.15.88:0/2611832507 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c15d610 con 0x7ff54c0c9a00 2023-10-16T20:43:53.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff5427fc640 1 --2- 172.21.15.88:0/2611832507 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff54c0c9a00 0x7ff54c0ca660 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff534009e80 tx=0x7ff53401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d125429c85debdc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:53.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff54a7fc640 1 -- 172.21.15.88:0/2611832507 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5340a5880 con 0x7ff54c0c9a00 2023-10-16T20:43:53.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff54a7fc640 1 -- 172.21.15.88:0/2611832507 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5340a59e0 con 0x7ff54c0c9a00 2023-10-16T20:43:53.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff54a7fc640 1 -- 172.21.15.88:0/2611832507 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5340a5d70 con 0x7ff54c0c9a00 2023-10-16T20:43:53.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff5514c9640 1 -- 172.21.15.88:0/2611832507 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff54c0c9a00 msgr2=0x7ff54c0ca660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:53.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff5514c9640 1 --2- 172.21.15.88:0/2611832507 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff54c0c9a00 0x7ff54c0ca660 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff534009e80 tx=0x7ff53401c9c0 comp rx=0 tx=0).stop 2023-10-16T20:43:53.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff5514c9640 1 -- 172.21.15.88:0/2611832507 shutdown_connections 2023-10-16T20:43:53.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff5514c9640 1 --2- 172.21.15.88:0/2611832507 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff54c153940 0x7ff54c0caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:53.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff5514c9640 1 --2- 172.21.15.88:0/2611832507 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff54c0c9a00 0x7ff54c0ca660 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:53.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff5514c9640 1 -- 172.21.15.88:0/2611832507 >> 172.21.15.88:0/2611832507 conn(0x7ff54c066080 msgr2=0x7ff54c0715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:53.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff5514c9640 1 -- 172.21.15.88:0/2611832507 shutdown_connections 2023-10-16T20:43:53.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff5514c9640 1 -- 172.21.15.88:0/2611832507 wait complete. 2023-10-16T20:43:53.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff5514c9640 1 Processor -- start 2023-10-16T20:43:53.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff5514c9640 1 -- start start 2023-10-16T20:43:53.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff5514c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff54c0c9a00 0x7ff54c14d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:53.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff5514c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff54c153560 0x7ff54c0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:53.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff5514c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff54c153940 0x7ff54c0cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:53.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff5514c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c14d600 con 0x7ff54c0c9a00 2023-10-16T20:43:53.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff5514c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c0cc9d0 con 0x7ff54c153940 2023-10-16T20:43:53.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.920+0000 7ff5514c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c0cccb0 con 0x7ff54c153560 2023-10-16T20:43:53.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff54affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff54c0c9a00 0x7ff54c14d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:53.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff54affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff54c0c9a00 0x7ff54c14d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60310/0 (socket says 172.21.15.88:60310) 2023-10-16T20:43:53.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff54affd640 1 -- 172.21.15.88:0/1512645417 learned_addr learned my addr 172.21.15.88:0/1512645417 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:53.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff5427fc640 1 --2- 172.21.15.88:0/1512645417 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff54c153560 0x7ff54c0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:53.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff54b7fe640 1 --2- 172.21.15.88:0/1512645417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff54c153940 0x7ff54c0cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:53.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff54affd640 1 -- 172.21.15.88:0/1512645417 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff54c153560 msgr2=0x7ff54c0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:53.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff54affd640 1 --2- 172.21.15.88:0/1512645417 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff54c153560 0x7ff54c0cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:53.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff54affd640 1 -- 172.21.15.88:0/1512645417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff54c153940 msgr2=0x7ff54c0cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:53.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff54affd640 1 --2- 172.21.15.88:0/1512645417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff54c153940 0x7ff54c0cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:53.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff54affd640 1 -- 172.21.15.88:0/1512645417 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff534009b30 con 0x7ff54c0c9a00 2023-10-16T20:43:53.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff54affd640 1 --2- 172.21.15.88:0/1512645417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff54c0c9a00 0x7ff54c14d0c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff53c00c450 tx=0x7ff53c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:53.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff543fff640 1 -- 172.21.15.88:0/1512645417 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c0052d0 con 0x7ff54c0c9a00 2023-10-16T20:43:53.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff543fff640 1 -- 172.21.15.88:0/1512645417 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53c005430 con 0x7ff54c0c9a00 2023-10-16T20:43:53.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff5514c9640 1 -- 172.21.15.88:0/1512645417 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54c0b2f80 con 0x7ff54c0c9a00 2023-10-16T20:43:53.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff543fff640 1 -- 172.21.15.88:0/1512645417 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c003890 con 0x7ff54c0c9a00 2023-10-16T20:43:53.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff5514c9640 1 -- 172.21.15.88:0/1512645417 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54c1f5970 con 0x7ff54c0c9a00 2023-10-16T20:43:53.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff5514c9640 1 -- 172.21.15.88:0/1512645417 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff54c04e9c0 con 0x7ff54c0c9a00 2023-10-16T20:43:53.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff543fff640 1 -- 172.21.15.88:0/1512645417 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff53c0039f0 con 0x7ff54c0c9a00 2023-10-16T20:43:53.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff543fff640 1 --2- 172.21.15.88:0/1512645417 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff52c0664d0 0x7ff52c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:53.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff543fff640 1 -- 172.21.15.88:0/1512645417 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7ff53c083740 con 0x7ff54c0c9a00 2023-10-16T20:43:53.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.924+0000 7ff5427fc640 1 --2- 172.21.15.88:0/1512645417 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff52c0664d0 0x7ff52c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:53.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.928+0000 7ff5427fc640 1 --2- 172.21.15.88:0/1512645417 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff52c0664d0 0x7ff52c068990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff5340ae000 tx=0x7ff5340a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:53.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:53.928+0000 7ff543fff640 1 -- 172.21.15.88:0/1512645417 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff53c050870 con 0x7ff54c0c9a00 2023-10-16T20:43:54.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:54.112+0000 7ff5514c9640 1 -- 172.21.15.88:0/1512645417 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff54c0cac00 con 0x7ff54c0c9a00 2023-10-16T20:43:54.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:54.112+0000 7ff543fff640 1 -- 172.21.15.88:0/1512645417 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5778 (secure 0 0 0) 0x7ff53c053e90 con 0x7ff54c0c9a00 2023-10-16T20:43:54.115 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:54.116 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:43:46.483738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:54.117 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 34 2023-10-16T20:43:54.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:54.112+0000 7ff5514c9640 1 -- 172.21.15.88:0/1512645417 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff52c0664d0 msgr2=0x7ff52c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:54.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:54.112+0000 7ff5514c9640 1 --2- 172.21.15.88:0/1512645417 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff52c0664d0 0x7ff52c068990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff5340ae000 tx=0x7ff5340a7070 comp rx=0 tx=0).stop 2023-10-16T20:43:54.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:54.112+0000 7ff5514c9640 1 -- 172.21.15.88:0/1512645417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff54c0c9a00 msgr2=0x7ff54c14d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:54.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:54.112+0000 7ff5514c9640 1 --2- 172.21.15.88:0/1512645417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff54c0c9a00 0x7ff54c14d0c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff53c00c450 tx=0x7ff53c00c850 comp rx=0 tx=0).stop 2023-10-16T20:43:54.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:54.112+0000 7ff5514c9640 1 -- 172.21.15.88:0/1512645417 shutdown_connections 2023-10-16T20:43:54.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:54.112+0000 7ff5514c9640 1 --2- 172.21.15.88:0/1512645417 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff52c0664d0 0x7ff52c068990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:54.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:54.112+0000 7ff5514c9640 1 --2- 172.21.15.88:0/1512645417 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff54c153940 0x7ff54c0cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:54.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:54.112+0000 7ff5514c9640 1 --2- 172.21.15.88:0/1512645417 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff54c153560 0x7ff54c0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:54.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:54.112+0000 7ff5514c9640 1 --2- 172.21.15.88:0/1512645417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff54c0c9a00 0x7ff54c14d0c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:54.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:54.112+0000 7ff5514c9640 1 -- 172.21.15.88:0/1512645417 >> 172.21.15.88:0/1512645417 conn(0x7ff54c066080 msgr2=0x7ff54c151420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:54.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:54.112+0000 7ff5514c9640 1 -- 172.21.15.88:0/1512645417 shutdown_connections 2023-10-16T20:43:54.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:54.112+0000 7ff5514c9640 1 -- 172.21.15.88:0/1512645417 wait complete. 2023-10-16T20:43:54.136 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-16T20:43:55.381 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:43:55.385 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:43:56.138 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:43:56.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.280+0000 7fd278394640 1 Processor -- start 2023-10-16T20:43:56.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.280+0000 7fd278394640 1 -- start start 2023-10-16T20:43:56.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.280+0000 7fd278394640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2700c76c0 0x7fd270159450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:56.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.280+0000 7fd278394640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2700c8810 con 0x7fd2700c7dc0 2023-10-16T20:43:56.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.280+0000 7fd278394640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd27006b400 con 0x7fd2700c81a0 2023-10-16T20:43:56.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.280+0000 7fd278394640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd27006b0d0 con 0x7fd2700c76c0 2023-10-16T20:43:56.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.280+0000 7fd276130640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd2700c7dc0 0x7fd27006d650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58358/0 (socket says 172.21.15.88:58358) 2023-10-16T20:43:56.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.280+0000 7fd276130640 1 -- 172.21.15.88:0/1370692522 learned_addr learned my addr 172.21.15.88:0/1370692522 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:43:56.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.280+0000 7fd27592f640 1 --2- 172.21.15.88:0/1370692522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2700c76c0 0x7fd270159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:56.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.280+0000 7fd27512e640 1 -- 172.21.15.88:0/1370692522 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737822204 0 0) 0x7fd2700c8810 con 0x7fd2700c7dc0 2023-10-16T20:43:56.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.280+0000 7fd27512e640 1 -- 172.21.15.88:0/1370692522 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd25c003530 con 0x7fd2700c7dc0 2023-10-16T20:43:56.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.280+0000 7fd27512e640 1 -- 172.21.15.88:0/1370692522 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167619282 0 0) 0x7fd27006b400 con 0x7fd2700c81a0 2023-10-16T20:43:56.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.280+0000 7fd27512e640 1 -- 172.21.15.88:0/1370692522 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2700c8810 con 0x7fd2700c81a0 2023-10-16T20:43:56.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.280+0000 7fd27512e640 1 -- 172.21.15.88:0/1370692522 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1342158565 0 0) 0x7fd25c003530 con 0x7fd2700c7dc0 2023-10-16T20:43:56.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.280+0000 7fd27592f640 1 -- 172.21.15.88:0/1370692522 >> v1:172.21.15.191:6789/0 conn(0x7fd2700c81a0 legacy=0x7fd27006b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:56.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.280+0000 7fd27592f640 1 -- 172.21.15.88:0/1370692522 >> v1:172.21.15.88:6789/0 conn(0x7fd2700c7dc0 legacy=0x7fd27006d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:56.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.280+0000 7fd27592f640 1 -- 172.21.15.88:0/1370692522 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27006e6a0 con 0x7fd2700c76c0 2023-10-16T20:43:56.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.284+0000 7fd27592f640 1 --2- 172.21.15.88:0/1370692522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2700c76c0 0x7fd270159450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd2600097f0 tx=0x7fd26001c090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e5fcd9a6de83107 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:56.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.284+0000 7fd27512e640 1 -- 172.21.15.88:0/1370692522 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c001680 con 0x7fd2700c76c0 2023-10-16T20:43:56.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.284+0000 7fd27512e640 1 -- 172.21.15.88:0/1370692522 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2600a4cb0 con 0x7fd2700c76c0 2023-10-16T20:43:56.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.284+0000 7fd27512e640 1 -- 172.21.15.88:0/1370692522 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2600ac640 con 0x7fd2700c76c0 2023-10-16T20:43:56.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.284+0000 7fd278394640 1 -- 172.21.15.88:0/1370692522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2700c76c0 msgr2=0x7fd270159450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:56.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.284+0000 7fd278394640 1 --2- 172.21.15.88:0/1370692522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2700c76c0 0x7fd270159450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd2600097f0 tx=0x7fd26001c090 comp rx=0 tx=0).stop 2023-10-16T20:43:56.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.284+0000 7fd278394640 1 -- 172.21.15.88:0/1370692522 shutdown_connections 2023-10-16T20:43:56.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.284+0000 7fd278394640 1 --2- 172.21.15.88:0/1370692522 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2700c76c0 0x7fd270159450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:56.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.284+0000 7fd278394640 1 -- 172.21.15.88:0/1370692522 >> 172.21.15.88:0/1370692522 conn(0x7fd270071a60 msgr2=0x7fd270071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:56.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd278394640 1 -- 172.21.15.88:0/1370692522 shutdown_connections 2023-10-16T20:43:56.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd278394640 1 -- 172.21.15.88:0/1370692522 wait complete. 2023-10-16T20:43:56.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd278394640 1 Processor -- start 2023-10-16T20:43:56.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd278394640 1 -- start start 2023-10-16T20:43:56.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd278394640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2700c76c0 0x7fd2701476c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:56.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd278394640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2700c7dc0 0x7fd2701f5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:56.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd278394640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2700c81a0 0x7fd2701f55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:56.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd278394640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2701590a0 con 0x7fd2700c81a0 2023-10-16T20:43:56.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd278394640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd270148a60 con 0x7fd2700c76c0 2023-10-16T20:43:56.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd278394640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd270148d00 con 0x7fd2700c7dc0 2023-10-16T20:43:56.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd276931640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2700c81a0 0x7fd2701f55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:56.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd276931640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2700c81a0 0x7fd2701f55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60318/0 (socket says 172.21.15.88:60318) 2023-10-16T20:43:56.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd276931640 1 -- 172.21.15.88:0/2443102304 learned_addr learned my addr 172.21.15.88:0/2443102304 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:43:56.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd276130640 1 --2- 172.21.15.88:0/2443102304 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2700c76c0 0x7fd2701476c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:56.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd27592f640 1 --2- 172.21.15.88:0/2443102304 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2700c7dc0 0x7fd2701f5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:56.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd276931640 1 -- 172.21.15.88:0/2443102304 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2700c7dc0 msgr2=0x7fd2701f5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:56.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd276931640 1 --2- 172.21.15.88:0/2443102304 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2700c7dc0 0x7fd2701f5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:56.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd276931640 1 -- 172.21.15.88:0/2443102304 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2700c76c0 msgr2=0x7fd2701476c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:56.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd276931640 1 --2- 172.21.15.88:0/2443102304 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2700c76c0 0x7fd2701476c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:56.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd276931640 1 -- 172.21.15.88:0/2443102304 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd260009420 con 0x7fd2700c81a0 2023-10-16T20:43:56.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd276130640 1 --2- 172.21.15.88:0/2443102304 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2700c76c0 0x7fd2701476c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:43:56.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd27592f640 1 --2- 172.21.15.88:0/2443102304 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2700c7dc0 0x7fd2701f5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:43:56.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd276931640 1 --2- 172.21.15.88:0/2443102304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2700c81a0 0x7fd2701f55c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd26800c6c0 tx=0x7fd26800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:56.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd266ffd640 1 -- 172.21.15.88:0/2443102304 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd268003b80 con 0x7fd2700c81a0 2023-10-16T20:43:56.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd266ffd640 1 -- 172.21.15.88:0/2443102304 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd268003ce0 con 0x7fd2700c81a0 2023-10-16T20:43:56.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd278394640 1 -- 172.21.15.88:0/2443102304 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd270158420 con 0x7fd2700c81a0 2023-10-16T20:43:56.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.288+0000 7fd266ffd640 1 -- 172.21.15.88:0/2443102304 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2680052f0 con 0x7fd2700c81a0 2023-10-16T20:43:56.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.292+0000 7fd278394640 1 -- 172.21.15.88:0/2443102304 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2701589e0 con 0x7fd2700c81a0 2023-10-16T20:43:56.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.292+0000 7fd278394640 1 -- 172.21.15.88:0/2443102304 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd270074700 con 0x7fd2700c81a0 2023-10-16T20:43:56.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.292+0000 7fd266ffd640 1 -- 172.21.15.88:0/2443102304 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd268005450 con 0x7fd2700c81a0 2023-10-16T20:43:56.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.292+0000 7fd266ffd640 1 --2- 172.21.15.88:0/2443102304 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2540664a0 0x7fd254068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:43:56.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.292+0000 7fd266ffd640 1 -- 172.21.15.88:0/2443102304 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7fd2680841c0 con 0x7fd2700c81a0 2023-10-16T20:43:56.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.292+0000 7fd276130640 1 --2- 172.21.15.88:0/2443102304 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2540664a0 0x7fd254068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:43:56.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.292+0000 7fd276130640 1 --2- 172.21.15.88:0/2443102304 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2540664a0 0x7fd254068960 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd26c0011f0 tx=0x7fd26c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:43:56.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.296+0000 7fd266ffd640 1 -- 172.21.15.88:0/2443102304 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd268051140 con 0x7fd2700c81a0 2023-10-16T20:43:56.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.492+0000 7fd278394640 1 -- 172.21.15.88:0/2443102304 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2700c3a40 con 0x7fd2700c81a0 2023-10-16T20:43:56.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.492+0000 7fd266ffd640 1 -- 172.21.15.88:0/2443102304 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6568 (secure 0 0 0) 0x7fd268054760 con 0x7fd2700c81a0 2023-10-16T20:43:56.498 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:43:56.498 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:43:46.483738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:43:56.499 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 35 2023-10-16T20:43:56.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.496+0000 7fd278394640 1 -- 172.21.15.88:0/2443102304 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2540664a0 msgr2=0x7fd254068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:56.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.496+0000 7fd278394640 1 --2- 172.21.15.88:0/2443102304 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2540664a0 0x7fd254068960 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd26c0011f0 tx=0x7fd26c009040 comp rx=0 tx=0).stop 2023-10-16T20:43:56.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.496+0000 7fd278394640 1 -- 172.21.15.88:0/2443102304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2700c81a0 msgr2=0x7fd2701f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:43:56.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.496+0000 7fd278394640 1 --2- 172.21.15.88:0/2443102304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2700c81a0 0x7fd2701f55c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd26800c6c0 tx=0x7fd26800cb80 comp rx=0 tx=0).stop 2023-10-16T20:43:56.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.496+0000 7fd278394640 1 -- 172.21.15.88:0/2443102304 shutdown_connections 2023-10-16T20:43:56.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.496+0000 7fd278394640 1 --2- 172.21.15.88:0/2443102304 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd2540664a0 0x7fd254068960 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:56.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.496+0000 7fd278394640 1 --2- 172.21.15.88:0/2443102304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2700c81a0 0x7fd2701f55c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:56.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.496+0000 7fd278394640 1 --2- 172.21.15.88:0/2443102304 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2700c7dc0 0x7fd2701f5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:56.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.496+0000 7fd278394640 1 --2- 172.21.15.88:0/2443102304 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd2700c76c0 0x7fd2701476c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:43:56.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.496+0000 7fd278394640 1 -- 172.21.15.88:0/2443102304 >> 172.21.15.88:0/2443102304 conn(0x7fd270071a60 msgr2=0x7fd27006d650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:43:56.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.496+0000 7fd278394640 1 -- 172.21.15.88:0/2443102304 shutdown_connections 2023-10-16T20:43:56.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:43:56.496+0000 7fd278394640 1 -- 172.21.15.88:0/2443102304 wait complete. 2023-10-16T20:43:56.519 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-16T20:43:56.519 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 15 secs before thrashing 2023-10-16T20:43:58.635 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2023-10-16T20:43:58.636 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2023-10-16T20:43:58.636 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2023-10-16T20:43:58.636 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2023-10-16T20:43:58.636 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2023-10-16T20:43:58.637 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2023-10-16T20:43:58.637 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2023-10-16T20:43:58.637 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2023-10-16T20:43:58.637 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2023-10-16T20:43:58.637 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2023-10-16T20:43:58.638 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2023-10-16T20:43:58.638 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2023-10-16T20:43:58.638 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2023-10-16T20:43:58.638 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2023-10-16T20:43:58.639 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2023-10-16T20:43:58.639 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2023-10-16T20:43:58.639 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2023-10-16T20:43:58.639 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2023-10-16T20:43:58.639 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2023-10-16T20:43:58.640 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2023-10-16T20:43:58.640 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2023-10-16T20:43:58.640 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2023-10-16T20:43:58.640 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2023-10-16T20:43:58.640 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2023-10-16T20:43:58.641 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2023-10-16T20:43:58.641 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2023-10-16T20:43:58.641 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2023-10-16T20:43:58.641 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2023-10-16T20:43:58.641 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2023-10-16T20:43:58.641 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2023-10-16T20:43:58.642 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2023-10-16T20:43:58.642 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2023-10-16T20:43:58.642 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2023-10-16T20:43:58.642 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2023-10-16T20:43:58.642 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2023-10-16T20:43:58.643 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2023-10-16T20:43:58.643 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2023-10-16T20:43:58.643 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2023-10-16T20:43:58.643 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2023-10-16T20:43:58.643 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2023-10-16T20:43:58.644 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2023-10-16T20:43:58.644 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2023-10-16T20:43:58.644 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2023-10-16T20:43:58.644 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2023-10-16T20:43:58.645 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2023-10-16T20:43:58.645 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2023-10-16T20:43:58.645 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2023-10-16T20:43:58.645 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2023-10-16T20:43:58.645 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2023-10-16T20:43:58.646 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2023-10-16T20:43:58.646 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2023-10-16T20:43:58.646 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2023-10-16T20:43:58.646 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2023-10-16T20:43:58.646 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2023-10-16T20:43:58.647 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2023-10-16T20:43:58.647 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2023-10-16T20:43:58.647 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2023-10-16T20:43:58.647 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2023-10-16T20:43:58.647 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2023-10-16T20:43:58.648 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2023-10-16T20:43:58.648 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2023-10-16T20:43:58.648 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2023-10-16T20:43:58.648 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2023-10-16T20:43:58.648 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2023-10-16T20:43:58.648 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2023-10-16T20:43:58.649 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2023-10-16T20:43:58.649 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2023-10-16T20:43:58.649 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2023-10-16T20:43:58.649 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2023-10-16T20:43:58.649 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2023-10-16T20:43:58.650 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2023-10-16T20:43:58.650 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2023-10-16T20:43:58.650 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2023-10-16T20:43:58.650 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2023-10-16T20:43:58.650 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2023-10-16T20:43:58.651 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2023-10-16T20:43:58.651 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2023-10-16T20:43:58.651 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2023-10-16T20:43:58.651 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2023-10-16T20:43:58.651 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2023-10-16T20:43:58.652 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2023-10-16T20:44:00.982 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2023-10-16T20:44:00.983 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2023-10-16T20:44:00.983 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2023-10-16T20:44:00.983 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2023-10-16T20:44:00.983 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2023-10-16T20:44:00.983 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2023-10-16T20:44:00.984 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2023-10-16T20:44:00.984 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2023-10-16T20:44:00.984 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2023-10-16T20:44:00.984 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2023-10-16T20:44:00.984 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2023-10-16T20:44:00.985 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2023-10-16T20:44:00.985 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2023-10-16T20:44:00.985 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2023-10-16T20:44:00.985 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2023-10-16T20:44:00.985 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2023-10-16T20:44:00.986 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2023-10-16T20:44:00.986 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2023-10-16T20:44:00.986 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2023-10-16T20:44:00.986 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2023-10-16T20:44:00.986 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2023-10-16T20:44:00.987 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2023-10-16T20:44:00.987 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2023-10-16T20:44:00.987 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2023-10-16T20:44:00.987 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2023-10-16T20:44:00.987 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2023-10-16T20:44:00.988 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2023-10-16T20:44:00.988 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2023-10-16T20:44:00.988 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2023-10-16T20:44:00.988 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2023-10-16T20:44:00.988 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2023-10-16T20:44:00.988 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2023-10-16T20:44:00.989 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2023-10-16T20:44:00.989 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2023-10-16T20:44:00.989 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2023-10-16T20:44:00.989 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2023-10-16T20:44:00.989 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2023-10-16T20:44:00.989 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2023-10-16T20:44:00.990 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2023-10-16T20:44:00.990 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2023-10-16T20:44:00.990 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2023-10-16T20:44:00.990 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2023-10-16T20:44:00.991 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2023-10-16T20:44:00.991 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2023-10-16T20:44:00.991 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2023-10-16T20:44:00.991 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2023-10-16T20:44:00.991 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2023-10-16T20:44:00.991 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2023-10-16T20:44:00.992 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2023-10-16T20:44:00.992 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2023-10-16T20:44:00.992 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2023-10-16T20:44:00.992 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2023-10-16T20:44:00.992 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2023-10-16T20:44:00.993 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2023-10-16T20:44:00.993 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2023-10-16T20:44:00.993 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2023-10-16T20:44:00.993 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2023-10-16T20:44:00.993 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2023-10-16T20:44:00.993 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2023-10-16T20:44:00.994 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2023-10-16T20:44:00.994 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2023-10-16T20:44:00.994 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2023-10-16T20:44:00.994 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2023-10-16T20:44:00.994 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2023-10-16T20:44:00.995 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2023-10-16T20:44:00.995 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2023-10-16T20:44:00.995 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2023-10-16T20:44:00.995 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2023-10-16T20:44:00.996 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2023-10-16T20:44:00.996 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2023-10-16T20:44:00.996 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2023-10-16T20:44:00.996 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2023-10-16T20:44:00.996 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2023-10-16T20:44:00.996 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2023-10-16T20:44:00.997 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2023-10-16T20:44:00.997 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2023-10-16T20:44:00.997 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2023-10-16T20:44:00.997 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2023-10-16T20:44:00.997 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2023-10-16T20:44:00.998 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2023-10-16T20:44:00.998 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2023-10-16T20:44:02.223 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2023-10-16T20:44:02.224 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2023-10-16T20:44:02.224 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2023-10-16T20:44:02.224 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2023-10-16T20:44:02.225 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2023-10-16T20:44:02.225 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2023-10-16T20:44:02.225 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2023-10-16T20:44:02.225 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2023-10-16T20:44:02.225 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2023-10-16T20:44:02.226 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2023-10-16T20:44:02.226 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2023-10-16T20:44:02.226 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2023-10-16T20:44:02.226 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2023-10-16T20:44:02.226 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2023-10-16T20:44:02.227 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2023-10-16T20:44:02.227 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2023-10-16T20:44:02.227 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2023-10-16T20:44:02.227 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2023-10-16T20:44:02.227 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2023-10-16T20:44:02.228 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2023-10-16T20:44:02.228 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2023-10-16T20:44:02.228 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2023-10-16T20:44:02.228 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2023-10-16T20:44:02.228 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2023-10-16T20:44:02.229 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2023-10-16T20:44:02.229 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2023-10-16T20:44:02.229 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2023-10-16T20:44:02.229 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2023-10-16T20:44:02.229 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2023-10-16T20:44:02.229 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2023-10-16T20:44:02.230 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2023-10-16T20:44:02.230 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2023-10-16T20:44:02.230 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2023-10-16T20:44:02.230 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2023-10-16T20:44:02.230 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2023-10-16T20:44:02.231 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2023-10-16T20:44:02.231 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2023-10-16T20:44:02.231 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2023-10-16T20:44:02.231 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2023-10-16T20:44:02.231 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2023-10-16T20:44:02.232 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2023-10-16T20:44:02.232 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2023-10-16T20:44:02.232 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2023-10-16T20:44:02.232 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2023-10-16T20:44:02.232 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2023-10-16T20:44:02.233 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2023-10-16T20:44:02.233 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2023-10-16T20:44:02.233 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2023-10-16T20:44:02.233 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2023-10-16T20:44:02.233 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2023-10-16T20:44:02.233 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2023-10-16T20:44:02.234 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2023-10-16T20:44:02.234 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2023-10-16T20:44:02.234 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2023-10-16T20:44:02.234 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2023-10-16T20:44:02.234 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2023-10-16T20:44:02.235 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2023-10-16T20:44:02.235 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2023-10-16T20:44:02.235 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2023-10-16T20:44:02.235 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2023-10-16T20:44:02.235 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2023-10-16T20:44:02.236 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2023-10-16T20:44:02.236 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2023-10-16T20:44:02.236 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2023-10-16T20:44:02.236 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2023-10-16T20:44:02.236 INFO:tasks.workunit.client.0.smithi088.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2023-10-16T20:44:02.236 INFO:tasks.workunit.client.0.smithi088.stdout:Take third snapshot .snap/test3 2023-10-16T20:44:02.274 INFO:tasks.workunit.client.0.smithi088.stdout:Delete the snapshots... 2023-10-16T20:44:08.924 INFO:tasks.workunit.client.0.smithi088.stdout:.//.snap/test1 2023-10-16T20:44:08.924 INFO:tasks.workunit.client.0.smithi088.stdout:.//.snap/test2 2023-10-16T20:44:08.925 INFO:tasks.workunit.client.0.smithi088.stdout:.//.snap/test3 2023-10-16T20:44:08.941 INFO:tasks.workunit.client.0.smithi088.stdout:./274/.snap/snap-subdir-test 2023-10-16T20:44:08.958 INFO:tasks.workunit.client.0.smithi088.stdout:./166/.snap/snap-subdir-test 2023-10-16T20:44:08.973 INFO:tasks.workunit.client.0.smithi088.stdout:./256/.snap/snap-subdir-test 2023-10-16T20:44:08.988 INFO:tasks.workunit.client.0.smithi088.stdout:./247/.snap/snap-subdir-test 2023-10-16T20:44:09.003 INFO:tasks.workunit.client.0.smithi088.stdout:./248/.snap/snap-subdir-test 2023-10-16T20:44:09.940 INFO:tasks.workunit.client.0.smithi088.stdout:./147/.snap/snap-subdir-test 2023-10-16T20:44:09.962 INFO:tasks.workunit.client.0.smithi088.stdout:./151/.snap/snap-subdir-test 2023-10-16T20:44:09.986 INFO:tasks.workunit.client.0.smithi088.stdout:./170/.snap/snap-subdir-test 2023-10-16T20:44:10.007 INFO:tasks.workunit.client.0.smithi088.stdout:./167/.snap/snap-subdir-test 2023-10-16T20:44:10.027 INFO:tasks.workunit.client.0.smithi088.stdout:./140/.snap/snap-subdir-test 2023-10-16T20:44:10.048 INFO:tasks.workunit.client.0.smithi088.stdout:./142/.snap/snap-subdir-test 2023-10-16T20:44:10.068 INFO:tasks.workunit.client.0.smithi088.stdout:./150/.snap/snap-subdir-test 2023-10-16T20:44:10.085 INFO:tasks.workunit.client.0.smithi088.stdout:./273/.snap/snap-subdir-test 2023-10-16T20:44:10.146 INFO:tasks.workunit.client.0.smithi088.stdout:./269/.snap/snap-subdir-test 2023-10-16T20:44:10.164 INFO:tasks.workunit.client.0.smithi088.stdout:./105/.snap/snap-subdir-test 2023-10-16T20:44:10.181 INFO:tasks.workunit.client.0.smithi088.stdout:./210/.snap/snap-subdir-test 2023-10-16T20:44:10.200 INFO:tasks.workunit.client.0.smithi088.stdout:./265/.snap/snap-subdir-test 2023-10-16T20:44:10.218 INFO:tasks.workunit.client.0.smithi088.stdout:./218/.snap/snap-subdir-test 2023-10-16T20:44:10.237 INFO:tasks.workunit.client.0.smithi088.stdout:./290/.snap/snap-subdir-test 2023-10-16T20:44:10.258 INFO:tasks.workunit.client.0.smithi088.stdout:./149/.snap/snap-subdir-test 2023-10-16T20:44:10.278 INFO:tasks.workunit.client.0.smithi088.stdout:./109/.snap/snap-subdir-test 2023-10-16T20:44:10.296 INFO:tasks.workunit.client.0.smithi088.stdout:./292/.snap/snap-subdir-test 2023-10-16T20:44:10.315 INFO:tasks.workunit.client.0.smithi088.stdout:./242/.snap/snap-subdir-test 2023-10-16T20:44:10.335 INFO:tasks.workunit.client.0.smithi088.stdout:./158/.snap/snap-subdir-test 2023-10-16T20:44:10.353 INFO:tasks.workunit.client.0.smithi088.stdout:./243/.snap/snap-subdir-test 2023-10-16T20:44:10.372 INFO:tasks.workunit.client.0.smithi088.stdout:./106/.snap/snap-subdir-test 2023-10-16T20:44:10.384 INFO:tasks.workunit.client.0.smithi088.stdout:./250/.snap/snap-subdir-test 2023-10-16T20:44:10.395 INFO:tasks.workunit.client.0.smithi088.stdout:./221/.snap/snap-subdir-test 2023-10-16T20:44:10.407 INFO:tasks.workunit.client.0.smithi088.stdout:./244/.snap/snap-subdir-test 2023-10-16T20:44:10.418 INFO:tasks.workunit.client.0.smithi088.stdout:./258/.snap/snap-subdir-test 2023-10-16T20:44:10.432 INFO:tasks.workunit.client.0.smithi088.stdout:./268/.snap/snap-subdir-test 2023-10-16T20:44:10.447 INFO:tasks.workunit.client.0.smithi088.stdout:./189/.snap/snap-subdir-test 2023-10-16T20:44:10.460 INFO:tasks.workunit.client.0.smithi088.stdout:./285/.snap/snap-subdir-test 2023-10-16T20:44:11.022 INFO:tasks.workunit.client.0.smithi088.stdout:./208/.snap/snap-subdir-test 2023-10-16T20:44:11.040 INFO:tasks.workunit.client.0.smithi088.stdout:./288/.snap/snap-subdir-test 2023-10-16T20:44:11.056 INFO:tasks.workunit.client.0.smithi088.stdout:./209/.snap/snap-subdir-test 2023-10-16T20:44:11.075 INFO:tasks.workunit.client.0.smithi088.stdout:./253/.snap/snap-subdir-test 2023-10-16T20:44:11.091 INFO:tasks.workunit.client.0.smithi088.stdout:./249/.snap/snap-subdir-test 2023-10-16T20:44:11.109 INFO:tasks.workunit.client.0.smithi088.stdout:./133/.snap/snap-subdir-test 2023-10-16T20:44:11.123 INFO:tasks.workunit.client.0.smithi088.stdout:./291/.snap/snap-subdir-test 2023-10-16T20:44:11.139 INFO:tasks.workunit.client.0.smithi088.stdout:./226/.snap/snap-subdir-test 2023-10-16T20:44:11.155 INFO:tasks.workunit.client.0.smithi088.stdout:./173/.snap/snap-subdir-test 2023-10-16T20:44:11.171 INFO:tasks.workunit.client.0.smithi088.stdout:./174/.snap/snap-subdir-test 2023-10-16T20:44:11.185 INFO:tasks.workunit.client.0.smithi088.stdout:./299/.snap/snap-subdir-test 2023-10-16T20:44:11.200 INFO:tasks.workunit.client.0.smithi088.stdout:./262/.snap/snap-subdir-test 2023-10-16T20:44:11.215 INFO:tasks.workunit.client.0.smithi088.stdout:./231/.snap/snap-subdir-test 2023-10-16T20:44:11.230 INFO:tasks.workunit.client.0.smithi088.stdout:./214/.snap/snap-subdir-test 2023-10-16T20:44:11.244 INFO:tasks.workunit.client.0.smithi088.stdout:./281/.snap/snap-subdir-test 2023-10-16T20:44:11.257 INFO:tasks.workunit.client.0.smithi088.stdout:./289/.snap/snap-subdir-test 2023-10-16T20:44:11.270 INFO:tasks.workunit.client.0.smithi088.stdout:./283/.snap/snap-subdir-test 2023-10-16T20:44:11.282 INFO:tasks.workunit.client.0.smithi088.stdout:./232/.snap/snap-subdir-test 2023-10-16T20:44:11.295 INFO:tasks.workunit.client.0.smithi088.stdout:./298/.snap/snap-subdir-test 2023-10-16T20:44:11.307 INFO:tasks.workunit.client.0.smithi088.stdout:./233/.snap/snap-subdir-test 2023-10-16T20:44:11.324 INFO:tasks.workunit.client.0.smithi088.stdout:./155/.snap/snap-subdir-test 2023-10-16T20:44:11.337 INFO:tasks.workunit.client.0.smithi088.stdout:./114/.snap/snap-subdir-test 2023-10-16T20:44:11.350 INFO:tasks.workunit.client.0.smithi088.stdout:./159/.snap/snap-subdir-test 2023-10-16T20:44:11.363 INFO:tasks.workunit.client.0.smithi088.stdout:./187/.snap/snap-subdir-test 2023-10-16T20:44:11.377 INFO:tasks.workunit.client.0.smithi088.stdout:./185/.snap/snap-subdir-test 2023-10-16T20:44:11.389 INFO:tasks.workunit.client.0.smithi088.stdout:./293/.snap/snap-subdir-test 2023-10-16T20:44:11.402 INFO:tasks.workunit.client.0.smithi088.stdout:./228/.snap/snap-subdir-test 2023-10-16T20:44:11.415 INFO:tasks.workunit.client.0.smithi088.stdout:./120/.snap/snap-subdir-test 2023-10-16T20:44:11.428 INFO:tasks.workunit.client.0.smithi088.stdout:./110/.snap/snap-subdir-test 2023-10-16T20:44:11.440 INFO:tasks.workunit.client.0.smithi088.stdout:./267/.snap/snap-subdir-test 2023-10-16T20:44:11.453 INFO:tasks.workunit.client.0.smithi088.stdout:./202/.snap/snap-subdir-test 2023-10-16T20:44:11.466 INFO:tasks.workunit.client.0.smithi088.stdout:./137/.snap/snap-subdir-test 2023-10-16T20:44:11.479 INFO:tasks.workunit.client.0.smithi088.stdout:./246/.snap/snap-subdir-test 2023-10-16T20:44:11.495 INFO:tasks.workunit.client.0.smithi088.stdout:./219/.snap/snap-subdir-test 2023-10-16T20:44:11.509 INFO:tasks.workunit.client.0.smithi088.stdout:./198/.snap/snap-subdir-test 2023-10-16T20:44:11.521 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:11.525 INFO:tasks.workunit.client.0.smithi088.stdout:./136/.snap/snap-subdir-test 2023-10-16T20:44:11.536 INFO:tasks.workunit.client.0.smithi088.stdout:./270/.snap/snap-subdir-test 2023-10-16T20:44:11.551 INFO:tasks.workunit.client.0.smithi088.stdout:./125/.snap/snap-subdir-test 2023-10-16T20:44:11.567 INFO:tasks.workunit.client.0.smithi088.stdout:./194/.snap/snap-subdir-test 2023-10-16T20:44:11.582 INFO:tasks.workunit.client.0.smithi088.stdout:./119/.snap/snap-subdir-test 2023-10-16T20:44:11.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.668+0000 7f8e01668640 1 Processor -- start 2023-10-16T20:44:11.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.668+0000 7f8e01668640 1 -- start start 2023-10-16T20:44:11.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.668+0000 7f8e01668640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dfc1538e0 0x7f8dfc0ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:11.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.668+0000 7f8e01668640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8dfc15cb20 0x7f8dfc0cee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:11.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.668+0000 7f8e01668640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dfc0ca610 con 0x7f8dfc15a2a0 2023-10-16T20:44:11.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.668+0000 7f8e01668640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dfc1595b0 con 0x7f8dfc15cb20 2023-10-16T20:44:11.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.668+0000 7f8e01668640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc15d540 con 0x7f8dfc1538e0 2023-10-16T20:44:11.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.668+0000 7f8dfb7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8dfc15cb20 0x7f8dfc0cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:11.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.668+0000 7f8dfb7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8dfc15cb20 0x7f8dfc0cee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58592/0 (socket says 172.21.15.88:58592) 2023-10-16T20:44:11.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.668+0000 7f8dfb7fe640 1 -- 172.21.15.88:0/654050222 learned_addr learned my addr 172.21.15.88:0/654050222 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:11.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.668+0000 7f8dfaffd640 1 --2- 172.21.15.88:0/654050222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dfc1538e0 0x7f8dfc0ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:11.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.672+0000 7f8dfb7fe640 1 -- 172.21.15.88:0/654050222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dfc1538e0 msgr2=0x7f8dfc0ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:11.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.672+0000 7f8dfb7fe640 1 --2- 172.21.15.88:0/654050222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dfc1538e0 0x7f8dfc0ca0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:11.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.672+0000 7f8dfb7fe640 1 -- 172.21.15.88:0/654050222 >> v1:172.21.15.191:6789/0 conn(0x7f8dfc15a2a0 legacy=0x7f8dfc0c3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:11.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.672+0000 7f8dfb7fe640 1 -- 172.21.15.88:0/654050222 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dfc15d820 con 0x7f8dfc15cb20 2023-10-16T20:44:11.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.672+0000 7f8dfa7fc640 1 -- 172.21.15.88:0/654050222 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67319656 0 0) 0x7f8dfc0ca610 con 0x7f8dfc15a2a0 2023-10-16T20:44:11.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.672+0000 7f8dfb7fe640 1 --2- 172.21.15.88:0/654050222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dfc15cb20 0x7f8dfc0cee40 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8df4009870 tx=0x7f8df401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa09faba6233bc06 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:11.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.672+0000 7f8dfa7fc640 1 -- 172.21.15.88:0/654050222 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df40a7070 con 0x7f8dfc15cb20 2023-10-16T20:44:11.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.672+0000 7f8dfa7fc640 1 -- 172.21.15.88:0/654050222 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df40ab430 con 0x7f8dfc15cb20 2023-10-16T20:44:11.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.672+0000 7f8dfa7fc640 1 -- 172.21.15.88:0/654050222 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df40ab7c0 con 0x7f8dfc15cb20 2023-10-16T20:44:11.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.672+0000 7f8e01668640 1 -- 172.21.15.88:0/654050222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dfc15cb20 msgr2=0x7f8dfc0cee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:11.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.672+0000 7f8e01668640 1 --2- 172.21.15.88:0/654050222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dfc15cb20 0x7f8dfc0cee40 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8df4009870 tx=0x7f8df401c540 comp rx=0 tx=0).stop 2023-10-16T20:44:11.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.672+0000 7f8e01668640 1 -- 172.21.15.88:0/654050222 shutdown_connections 2023-10-16T20:44:11.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.672+0000 7f8e01668640 1 --2- 172.21.15.88:0/654050222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dfc15cb20 0x7f8dfc0cee40 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:11.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.672+0000 7f8e01668640 1 --2- 172.21.15.88:0/654050222 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dfc1538e0 0x7f8dfc0ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:11.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.672+0000 7f8e01668640 1 -- 172.21.15.88:0/654050222 >> 172.21.15.88:0/654050222 conn(0x7f8dfc06bd50 msgr2=0x7f8dfc06c160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:11.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.672+0000 7f8e01668640 1 -- 172.21.15.88:0/654050222 shutdown_connections 2023-10-16T20:44:11.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.672+0000 7f8e01668640 1 -- 172.21.15.88:0/654050222 wait complete. 2023-10-16T20:44:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8e01668640 1 Processor -- start 2023-10-16T20:44:11.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8e01668640 1 -- start start 2023-10-16T20:44:11.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8e01668640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dfc1538e0 0x7f8dfc0b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:11.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8e01668640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dfc15a2a0 0x7f8dfc15c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:11.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8e01668640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dfc15cb20 0x7f8dfc158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:11.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8e01668640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc146110 con 0x7f8dfc15cb20 2023-10-16T20:44:11.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8e01668640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc151370 con 0x7f8dfc1538e0 2023-10-16T20:44:11.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8e01668640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc15b210 con 0x7f8dfc15a2a0 2023-10-16T20:44:11.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8dfb7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dfc15cb20 0x7f8dfc158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:11.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8dfb7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dfc15cb20 0x7f8dfc158470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58594/0 (socket says 172.21.15.88:58594) 2023-10-16T20:44:11.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8dfb7fe640 1 -- 172.21.15.88:0/461349005 learned_addr learned my addr 172.21.15.88:0/461349005 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:11.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8dfaffd640 1 --2- 172.21.15.88:0/461349005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dfc1538e0 0x7f8dfc0b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:11.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8df27fc640 1 --2- 172.21.15.88:0/461349005 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dfc15a2a0 0x7f8dfc15c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:11.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8dfb7fe640 1 -- 172.21.15.88:0/461349005 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dfc15a2a0 msgr2=0x7f8dfc15c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:11.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8dfb7fe640 1 --2- 172.21.15.88:0/461349005 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dfc15a2a0 0x7f8dfc15c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:11.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8dfb7fe640 1 -- 172.21.15.88:0/461349005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dfc1538e0 msgr2=0x7f8dfc0b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:11.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8dfb7fe640 1 --2- 172.21.15.88:0/461349005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dfc1538e0 0x7f8dfc0b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:11.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8dfb7fe640 1 -- 172.21.15.88:0/461349005 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df40094d0 con 0x7f8dfc15cb20 2023-10-16T20:44:11.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8dfb7fe640 1 --2- 172.21.15.88:0/461349005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dfc15cb20 0x7f8dfc158470 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8df40ad040 tx=0x7f8df40192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:11.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.676+0000 7f8df3fff640 1 -- 172.21.15.88:0/461349005 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df40ba070 con 0x7f8dfc15cb20 2023-10-16T20:44:11.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.680+0000 7f8df3fff640 1 -- 172.21.15.88:0/461349005 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df40b4420 con 0x7f8dfc15cb20 2023-10-16T20:44:11.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.680+0000 7f8df3fff640 1 -- 172.21.15.88:0/461349005 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df40b47d0 con 0x7f8dfc15cb20 2023-10-16T20:44:11.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.680+0000 7f8e01668640 1 -- 172.21.15.88:0/461349005 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dfc15ac50 con 0x7f8dfc15cb20 2023-10-16T20:44:11.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.680+0000 7f8e01668640 1 -- 172.21.15.88:0/461349005 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dfc15b520 con 0x7f8dfc15cb20 2023-10-16T20:44:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.680+0000 7f8e01668640 1 -- 172.21.15.88:0/461349005 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dfc074700 con 0x7f8dfc15cb20 2023-10-16T20:44:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.680+0000 7f8df3fff640 1 -- 172.21.15.88:0/461349005 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8df40b4930 con 0x7f8dfc15cb20 2023-10-16T20:44:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.680+0000 7f8df3fff640 1 --2- 172.21.15.88:0/461349005 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ddc0664a0 0x7f8ddc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.680+0000 7f8df3fff640 1 -- 172.21.15.88:0/461349005 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7f8df411db80 con 0x7f8dfc15cb20 2023-10-16T20:44:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.680+0000 7f8dfaffd640 1 --2- 172.21.15.88:0/461349005 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ddc0664a0 0x7f8ddc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.680+0000 7f8dfaffd640 1 --2- 172.21.15.88:0/461349005 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ddc0664a0 0x7f8ddc068960 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8dec0113e0 tx=0x7f8dec00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:11.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.684+0000 7f8df3fff640 1 -- 172.21.15.88:0/461349005 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8df40eab10 con 0x7f8dfc15cb20 2023-10-16T20:44:11.782 INFO:tasks.workunit.client.0.smithi088.stdout:./297/.snap/snap-subdir-test 2023-10-16T20:44:11.796 INFO:tasks.workunit.client.0.smithi088.stdout:./230/.snap/snap-subdir-test 2023-10-16T20:44:11.809 INFO:tasks.workunit.client.0.smithi088.stdout:./260/.snap/snap-subdir-test 2023-10-16T20:44:11.827 INFO:tasks.workunit.client.0.smithi088.stdout:./157/.snap/snap-subdir-test 2023-10-16T20:44:11.845 INFO:tasks.workunit.client.0.smithi088.stdout:./172/.snap/snap-subdir-test 2023-10-16T20:44:11.860 INFO:tasks.workunit.client.0.smithi088.stdout:./277/.snap/snap-subdir-test 2023-10-16T20:44:11.877 INFO:tasks.workunit.client.0.smithi088.stdout:./186/.snap/snap-subdir-test 2023-10-16T20:44:11.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.880+0000 7f8e01668640 1 -- 172.21.15.88:0/461349005 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dfc0ce850 con 0x7f8dfc15cb20 2023-10-16T20:44:11.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.884+0000 7f8df3fff640 1 -- 172.21.15.88:0/461349005 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6575 (secure 0 0 0) 0x7f8df411de60 con 0x7f8dfc15cb20 2023-10-16T20:44:11.886 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:11.886 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:01.507428+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:11.887 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 37 2023-10-16T20:44:11.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.884+0000 7f8e01668640 1 -- 172.21.15.88:0/461349005 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ddc0664a0 msgr2=0x7f8ddc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:11.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.884+0000 7f8e01668640 1 --2- 172.21.15.88:0/461349005 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ddc0664a0 0x7f8ddc068960 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8dec0113e0 tx=0x7f8dec00b040 comp rx=0 tx=0).stop 2023-10-16T20:44:11.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.884+0000 7f8e01668640 1 -- 172.21.15.88:0/461349005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dfc15cb20 msgr2=0x7f8dfc158470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:11.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.884+0000 7f8e01668640 1 --2- 172.21.15.88:0/461349005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dfc15cb20 0x7f8dfc158470 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8df40ad040 tx=0x7f8df40192e0 comp rx=0 tx=0).stop 2023-10-16T20:44:11.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.884+0000 7f8e01668640 1 -- 172.21.15.88:0/461349005 shutdown_connections 2023-10-16T20:44:11.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.884+0000 7f8e01668640 1 --2- 172.21.15.88:0/461349005 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ddc0664a0 0x7f8ddc068960 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:11.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.884+0000 7f8e01668640 1 --2- 172.21.15.88:0/461349005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dfc15cb20 0x7f8dfc158470 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:11.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.884+0000 7f8e01668640 1 --2- 172.21.15.88:0/461349005 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8dfc15a2a0 0x7f8dfc15c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:11.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.884+0000 7f8e01668640 1 --2- 172.21.15.88:0/461349005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8dfc1538e0 0x7f8dfc0b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:11.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.884+0000 7f8e01668640 1 -- 172.21.15.88:0/461349005 >> 172.21.15.88:0/461349005 conn(0x7f8dfc06bd50 msgr2=0x7f8dfc0c9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:11.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.884+0000 7f8e01668640 1 -- 172.21.15.88:0/461349005 shutdown_connections 2023-10-16T20:44:11.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:11.884+0000 7f8e01668640 1 -- 172.21.15.88:0/461349005 wait complete. 2023-10-16T20:44:11.894 INFO:tasks.workunit.client.0.smithi088.stdout:./237/.snap/snap-subdir-test 2023-10-16T20:44:11.909 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=3) 2023-10-16T20:44:11.909 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T20:44:11.909 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T20:44:11.965 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T20:44:11.965 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:12.073 INFO:tasks.workunit.client.0.smithi088.stdout:./190/.snap/snap-subdir-test 2023-10-16T20:44:12.089 INFO:tasks.workunit.client.0.smithi088.stdout:./213/.snap/snap-subdir-test 2023-10-16T20:44:12.108 INFO:tasks.workunit.client.0.smithi088.stdout:./104/.snap/snap-subdir-test 2023-10-16T20:44:12.127 INFO:tasks.workunit.client.0.smithi088.stdout:./164/.snap/snap-subdir-test 2023-10-16T20:44:12.146 INFO:tasks.workunit.client.0.smithi088.stdout:./112/.snap/snap-subdir-test 2023-10-16T20:44:12.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.148+0000 7f6961409640 1 Processor -- start 2023-10-16T20:44:12.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.148+0000 7f6961409640 1 -- start start 2023-10-16T20:44:12.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.152+0000 7f6961409640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f695c0cb580 0x7f695c0cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:12.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.152+0000 7f6961409640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f695c0c9af0 0x7f695c159300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:12.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.152+0000 7f6961409640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f695c0c9fc0 con 0x7f695c0b1a60 2023-10-16T20:44:12.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.152+0000 7f6961409640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695c0b20c0 con 0x7f695c0c9af0 2023-10-16T20:44:12.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.152+0000 7f6961409640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0cbea0 con 0x7f695c0cb580 2023-10-16T20:44:12.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.152+0000 7f695bfff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f695c0cb580 0x7f695c0cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:12.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.152+0000 7f695b7fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f695c0c9af0 0x7f695c159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:12.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.152+0000 7f695bfff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f695c0cb580 0x7f695c0cb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:38188/0 (socket says 172.21.15.88:38188) 2023-10-16T20:44:12.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.152+0000 7f695bfff640 1 -- 172.21.15.88:0/1090301415 learned_addr learned my addr 172.21.15.88:0/1090301415 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:12.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.152+0000 7f6960c08640 1 --1- 172.21.15.88:0/1090301415 >> v1:172.21.15.191:6789/0 conn(0x7f695c0b1a60 0x7f695c0715c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:51386/0 (socket says 172.21.15.88:51386) 2023-10-16T20:44:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.152+0000 7f695bfff640 1 -- 172.21.15.88:0/1090301415 >> v2:172.21.15.191:3300/0 conn(0x7f695c0c9af0 msgr2=0x7f695c159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.152+0000 7f695bfff640 1 --2- 172.21.15.88:0/1090301415 >> v2:172.21.15.191:3300/0 conn(0x7f695c0c9af0 0x7f695c159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.152+0000 7f695bfff640 1 -- 172.21.15.88:0/1090301415 >> v1:172.21.15.191:6789/0 conn(0x7f695c0b1a60 legacy=0x7f695c0715c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:12.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.152+0000 7f695affd640 1 -- 172.21.15.88:0/1090301415 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800521304 0 0) 0x7f695c0c9fc0 con 0x7f695c0b1a60 2023-10-16T20:44:12.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.152+0000 7f695bfff640 1 -- 172.21.15.88:0/1090301415 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c0c47d0 con 0x7f695c0cb580 2023-10-16T20:44:12.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.152+0000 7f695b7fe640 1 --2- 172.21.15.88:0/1090301415 >> v2:172.21.15.191:3300/0 conn(0x7f695c0c9af0 0x7f695c159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:44:12.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.156+0000 7f695bfff640 1 --2- 172.21.15.88:0/1090301415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f695c0cb580 0x7f695c0cb960 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f694c009870 tx=0x7f694c01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c90d2733d8f0360 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:12.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.156+0000 7f695affd640 1 -- 172.21.15.88:0/1090301415 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c0ab680 con 0x7f695c0cb580 2023-10-16T20:44:12.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.156+0000 7f695affd640 1 -- 172.21.15.88:0/1090301415 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694c0ab7e0 con 0x7f695c0cb580 2023-10-16T20:44:12.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.156+0000 7f695affd640 1 -- 172.21.15.88:0/1090301415 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c0abb70 con 0x7f695c0cb580 2023-10-16T20:44:12.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.156+0000 7f6961409640 1 -- 172.21.15.88:0/1090301415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f695c0cb580 msgr2=0x7f695c0cb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:12.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.156+0000 7f6961409640 1 --2- 172.21.15.88:0/1090301415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f695c0cb580 0x7f695c0cb960 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f694c009870 tx=0x7f694c01c480 comp rx=0 tx=0).stop 2023-10-16T20:44:12.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.156+0000 7f6961409640 1 -- 172.21.15.88:0/1090301415 shutdown_connections 2023-10-16T20:44:12.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.156+0000 7f6961409640 1 --2- 172.21.15.88:0/1090301415 >> v2:172.21.15.191:3300/0 conn(0x7f695c0c9af0 0x7f695c159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.156+0000 7f6961409640 1 --2- 172.21.15.88:0/1090301415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f695c0cb580 0x7f695c0cb960 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.156+0000 7f6961409640 1 -- 172.21.15.88:0/1090301415 >> 172.21.15.88:0/1090301415 conn(0x7f695c06bb40 msgr2=0x7f695c06bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.156+0000 7f6961409640 1 -- 172.21.15.88:0/1090301415 shutdown_connections 2023-10-16T20:44:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.156+0000 7f6961409640 1 -- 172.21.15.88:0/1090301415 wait complete. 2023-10-16T20:44:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.156+0000 7f6961409640 1 Processor -- start 2023-10-16T20:44:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.156+0000 7f6961409640 1 -- start start 2023-10-16T20:44:12.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6961409640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f695c0b1a60 0x7f695c1448f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:12.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6961409640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f695c0c9af0 0x7f695c158530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:12.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6961409640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f695c0cb580 0x7f695c0b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:12.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6961409640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0c6930 con 0x7f695c0cb580 2023-10-16T20:44:12.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6961409640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c157ae0 con 0x7f695c0b1a60 2023-10-16T20:44:12.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6961409640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0c6530 con 0x7f695c0c9af0 2023-10-16T20:44:12.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6960c08640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f695c0cb580 0x7f695c0b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:12.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6960c08640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f695c0cb580 0x7f695c0b3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58608/0 (socket says 172.21.15.88:58608) 2023-10-16T20:44:12.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6960c08640 1 -- 172.21.15.88:0/986908306 learned_addr learned my addr 172.21.15.88:0/986908306 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:12.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f695b7fe640 1 --2- 172.21.15.88:0/986908306 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f695c0c9af0 0x7f695c158530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:12.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f695bfff640 1 --2- 172.21.15.88:0/986908306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f695c0b1a60 0x7f695c1448f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:12.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6960c08640 1 -- 172.21.15.88:0/986908306 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f695c0c9af0 msgr2=0x7f695c158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:12.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6960c08640 1 --2- 172.21.15.88:0/986908306 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f695c0c9af0 0x7f695c158530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:12.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6960c08640 1 -- 172.21.15.88:0/986908306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f695c0b1a60 msgr2=0x7f695c1448f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:12.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6960c08640 1 --2- 172.21.15.88:0/986908306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f695c0b1a60 0x7f695c1448f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:12.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6960c08640 1 -- 172.21.15.88:0/986908306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6950009420 con 0x7f695c0cb580 2023-10-16T20:44:12.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6960c08640 1 --2- 172.21.15.88:0/986908306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f695c0cb580 0x7f695c0b3fd0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f695400c780 tx=0x7f695400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:12.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6958ff9640 1 -- 172.21.15.88:0/986908306 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695400ce90 con 0x7f695c0cb580 2023-10-16T20:44:12.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6958ff9640 1 -- 172.21.15.88:0/986908306 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695400a640 con 0x7f695c0cb580 2023-10-16T20:44:12.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6961409640 1 -- 172.21.15.88:0/986908306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694c0094d0 con 0x7f695c0cb580 2023-10-16T20:44:12.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6958ff9640 1 -- 172.21.15.88:0/986908306 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695400aa70 con 0x7f695c0cb580 2023-10-16T20:44:12.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6961409640 1 -- 172.21.15.88:0/986908306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695c144e30 con 0x7f695c0cb580 2023-10-16T20:44:12.172 INFO:tasks.workunit.client.0.smithi088.stdout:./295/.snap/snap-subdir-test 2023-10-16T20:44:12.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.160+0000 7f6961409640 1 -- 172.21.15.88:0/986908306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695c04e9c0 con 0x7f695c0cb580 2023-10-16T20:44:12.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.164+0000 7f6958ff9640 1 -- 172.21.15.88:0/986908306 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f695400abd0 con 0x7f695c0cb580 2023-10-16T20:44:12.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.164+0000 7f6958ff9640 1 --2- 172.21.15.88:0/986908306 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69480664a0 0x7f6948068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:12.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.164+0000 7f6958ff9640 1 -- 172.21.15.88:0/986908306 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7f6954083b20 con 0x7f695c0cb580 2023-10-16T20:44:12.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.164+0000 7f695bfff640 1 --2- 172.21.15.88:0/986908306 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69480664a0 0x7f6948068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:12.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.164+0000 7f695bfff640 1 --2- 172.21.15.88:0/986908306 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69480664a0 0x7f6948068960 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f694c0af000 tx=0x7f694c0a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:12.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.168+0000 7f6958ff9640 1 -- 172.21.15.88:0/986908306 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6954050b40 con 0x7f695c0cb580 2023-10-16T20:44:12.179 INFO:tasks.workunit.client.0.smithi088.stdout:./272/.snap/snap-subdir-test 2023-10-16T20:44:12.196 INFO:tasks.workunit.client.0.smithi088.stdout:./100/.snap/snap-subdir-test 2023-10-16T20:44:12.215 INFO:tasks.workunit.client.0.smithi088.stdout:./134/.snap/snap-subdir-test 2023-10-16T20:44:12.232 INFO:tasks.workunit.client.0.smithi088.stdout:./200/.snap/snap-subdir-test 2023-10-16T20:44:12.251 INFO:tasks.workunit.client.0.smithi088.stdout:./115/.snap/snap-subdir-test 2023-10-16T20:44:12.266 INFO:tasks.workunit.client.0.smithi088.stdout:./284/.snap/snap-subdir-test 2023-10-16T20:44:12.281 INFO:tasks.workunit.client.0.smithi088.stdout:./280/.snap/snap-subdir-test 2023-10-16T20:44:12.298 INFO:tasks.workunit.client.0.smithi088.stdout:./116/.snap/snap-subdir-test 2023-10-16T20:44:12.314 INFO:tasks.workunit.client.0.smithi088.stdout:./245/.snap/snap-subdir-test 2023-10-16T20:44:12.331 INFO:tasks.workunit.client.0.smithi088.stdout:./195/.snap/snap-subdir-test 2023-10-16T20:44:12.347 INFO:tasks.workunit.client.0.smithi088.stdout:./138/.snap/snap-subdir-test 2023-10-16T20:44:12.361 INFO:tasks.workunit.client.0.smithi088.stdout:./135/.snap/snap-subdir-test 2023-10-16T20:44:12.375 INFO:tasks.workunit.client.0.smithi088.stdout:./117/.snap/snap-subdir-test 2023-10-16T20:44:12.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.384+0000 7f6961409640 1 -- 172.21.15.88:0/986908306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f695c156680 con 0x7f695c0cb580 2023-10-16T20:44:12.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.384+0000 7f6958ff9640 1 -- 172.21.15.88:0/986908306 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6575 (secure 0 0 0) 0x7f6954018090 con 0x7f695c0cb580 2023-10-16T20:44:12.386 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:12.387 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:01.507428+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:12.387 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 37 2023-10-16T20:44:12.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.384+0000 7f6961409640 1 -- 172.21.15.88:0/986908306 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69480664a0 msgr2=0x7f6948068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:12.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.384+0000 7f6961409640 1 --2- 172.21.15.88:0/986908306 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69480664a0 0x7f6948068960 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f694c0af000 tx=0x7f694c0a7000 comp rx=0 tx=0).stop 2023-10-16T20:44:12.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.384+0000 7f6961409640 1 -- 172.21.15.88:0/986908306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f695c0cb580 msgr2=0x7f695c0b3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:12.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.384+0000 7f6961409640 1 --2- 172.21.15.88:0/986908306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f695c0cb580 0x7f695c0b3fd0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f695400c780 tx=0x7f695400cc40 comp rx=0 tx=0).stop 2023-10-16T20:44:12.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.384+0000 7f6961409640 1 -- 172.21.15.88:0/986908306 shutdown_connections 2023-10-16T20:44:12.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.384+0000 7f6961409640 1 --2- 172.21.15.88:0/986908306 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69480664a0 0x7f6948068960 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:12.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.384+0000 7f6961409640 1 --2- 172.21.15.88:0/986908306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f695c0cb580 0x7f695c0b3fd0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:12.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.384+0000 7f6961409640 1 --2- 172.21.15.88:0/986908306 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f695c0c9af0 0x7f695c158530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:12.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.384+0000 7f6961409640 1 --2- 172.21.15.88:0/986908306 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f695c0b1a60 0x7f695c1448f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:12.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.384+0000 7f6961409640 1 -- 172.21.15.88:0/986908306 >> 172.21.15.88:0/986908306 conn(0x7f695c06bb40 msgr2=0x7f695c1470b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:12.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.384+0000 7f6961409640 1 -- 172.21.15.88:0/986908306 shutdown_connections 2023-10-16T20:44:12.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:12.384+0000 7f6961409640 1 -- 172.21.15.88:0/986908306 wait complete. 2023-10-16T20:44:12.391 INFO:tasks.workunit.client.0.smithi088.stdout:./241/.snap/snap-subdir-test 2023-10-16T20:44:12.404 INFO:tasks.workunit.client.0.smithi088.stdout:./143/.snap/snap-subdir-test 2023-10-16T20:44:12.410 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-16T20:44:12.420 INFO:tasks.workunit.client.0.smithi088.stdout:./180/.snap/snap-subdir-test 2023-10-16T20:44:12.436 INFO:tasks.workunit.client.0.smithi088.stdout:./126/.snap/snap-subdir-test 2023-10-16T20:44:12.452 INFO:tasks.workunit.client.0.smithi088.stdout:./161/.snap/snap-subdir-test 2023-10-16T20:44:12.470 INFO:tasks.workunit.client.0.smithi088.stdout:./204/.snap/snap-subdir-test 2023-10-16T20:44:12.486 INFO:tasks.workunit.client.0.smithi088.stdout:./176/.snap/snap-subdir-test 2023-10-16T20:44:12.503 INFO:tasks.workunit.client.0.smithi088.stdout:./255/.snap/snap-subdir-test 2023-10-16T20:44:13.260 INFO:tasks.workunit.client.0.smithi088.stdout:./215/.snap/snap-subdir-test 2023-10-16T20:44:13.281 INFO:tasks.workunit.client.0.smithi088.stdout:./111/.snap/snap-subdir-test 2023-10-16T20:44:13.300 INFO:tasks.workunit.client.0.smithi088.stdout:./238/.snap/snap-subdir-test 2023-10-16T20:44:13.316 INFO:tasks.workunit.client.0.smithi088.stdout:./191/.snap/snap-subdir-test 2023-10-16T20:44:13.334 INFO:tasks.workunit.client.0.smithi088.stdout:./184/.snap/snap-subdir-test 2023-10-16T20:44:13.353 INFO:tasks.workunit.client.0.smithi088.stdout:./192/.snap/snap-subdir-test 2023-10-16T20:44:13.370 INFO:tasks.workunit.client.0.smithi088.stdout:./103/.snap/snap-subdir-test 2023-10-16T20:44:13.388 INFO:tasks.workunit.client.0.smithi088.stdout:./128/.snap/snap-subdir-test 2023-10-16T20:44:13.404 INFO:tasks.workunit.client.0.smithi088.stdout:./229/.snap/snap-subdir-test 2023-10-16T20:44:13.421 INFO:tasks.workunit.client.0.smithi088.stdout:./144/.snap/snap-subdir-test 2023-10-16T20:44:14.093 INFO:tasks.workunit.client.0.smithi088.stdout:./146/.snap/snap-subdir-test 2023-10-16T20:44:14.110 INFO:tasks.workunit.client.0.smithi088.stdout:./227/.snap/snap-subdir-test 2023-10-16T20:44:14.126 INFO:tasks.workunit.client.0.smithi088.stdout:./278/.snap/snap-subdir-test 2023-10-16T20:44:14.143 INFO:tasks.workunit.client.0.smithi088.stdout:./141/.snap/snap-subdir-test 2023-10-16T20:44:14.159 INFO:tasks.workunit.client.0.smithi088.stdout:./234/.snap/snap-subdir-test 2023-10-16T20:44:14.174 INFO:tasks.workunit.client.0.smithi088.stdout:./225/.snap/snap-subdir-test 2023-10-16T20:44:14.190 INFO:tasks.workunit.client.0.smithi088.stdout:./182/.snap/snap-subdir-test 2023-10-16T20:44:14.411 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:14.434 INFO:tasks.workunit.client.0.smithi088.stdout:./107/.snap/snap-subdir-test 2023-10-16T20:44:14.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.548+0000 7fa52793e640 1 Processor -- start 2023-10-16T20:44:14.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.548+0000 7fa52793e640 1 -- start start 2023-10-16T20:44:14.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.548+0000 7fa52793e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa52014fc50 0x7fa52006ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:14.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.548+0000 7fa52793e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa520071350 0x7fa520069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:14.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.548+0000 7fa52793e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa520071790 0x7fa52006a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:14.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.548+0000 7fa52793e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5200cc0a0 con 0x7fa52014fc50 2023-10-16T20:44:14.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.548+0000 7fa52793e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa520156980 con 0x7fa520071350 2023-10-16T20:44:14.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.548+0000 7fa52793e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa520155d70 con 0x7fa520071790 2023-10-16T20:44:14.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.552+0000 7fa5256da640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa52014fc50 0x7fa52006ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:14.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.552+0000 7fa5256da640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa52014fc50 0x7fa52006ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58618/0 (socket says 172.21.15.88:58618) 2023-10-16T20:44:14.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.552+0000 7fa5256da640 1 -- 172.21.15.88:0/4015210207 learned_addr learned my addr 172.21.15.88:0/4015210207 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:14.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.552+0000 7fa524ed9640 1 --2- 172.21.15.88:0/4015210207 >> v2:172.21.15.191:3300/0 conn(0x7fa520071350 0x7fa520069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:14.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.552+0000 7fa525edb640 1 --2- 172.21.15.88:0/4015210207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa520071790 0x7fa52006a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:14.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.552+0000 7fa5256da640 1 -- 172.21.15.88:0/4015210207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa520071790 msgr2=0x7fa52006a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:14.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.552+0000 7fa5256da640 1 --2- 172.21.15.88:0/4015210207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa520071790 0x7fa52006a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:14.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.552+0000 7fa5256da640 1 -- 172.21.15.88:0/4015210207 >> v2:172.21.15.191:3300/0 conn(0x7fa520071350 msgr2=0x7fa520069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:14.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.552+0000 7fa5256da640 1 --2- 172.21.15.88:0/4015210207 >> v2:172.21.15.191:3300/0 conn(0x7fa520071350 0x7fa520069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:14.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.552+0000 7fa5256da640 1 -- 172.21.15.88:0/4015210207 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa520156050 con 0x7fa52014fc50 2023-10-16T20:44:14.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.552+0000 7fa5256da640 1 --2- 172.21.15.88:0/4015210207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa52014fc50 0x7fa52006ffe0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa5200cbd00 tx=0x7fa51c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3ac2e3f8ba66f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:14.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.552+0000 7fa517fff640 1 -- 172.21.15.88:0/4015210207 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c0a89b0 con 0x7fa52014fc50 2023-10-16T20:44:14.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.552+0000 7fa517fff640 1 -- 172.21.15.88:0/4015210207 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51c0a8b10 con 0x7fa52014fc50 2023-10-16T20:44:14.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.552+0000 7fa517fff640 1 -- 172.21.15.88:0/4015210207 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c0a8ea0 con 0x7fa52014fc50 2023-10-16T20:44:14.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa52793e640 1 -- 172.21.15.88:0/4015210207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa52014fc50 msgr2=0x7fa52006ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:14.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa52793e640 1 --2- 172.21.15.88:0/4015210207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa52014fc50 0x7fa52006ffe0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa5200cbd00 tx=0x7fa51c01ecf0 comp rx=0 tx=0).stop 2023-10-16T20:44:14.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa52793e640 1 -- 172.21.15.88:0/4015210207 shutdown_connections 2023-10-16T20:44:14.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa52793e640 1 --2- 172.21.15.88:0/4015210207 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa520071790 0x7fa52006a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:14.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa52793e640 1 --2- 172.21.15.88:0/4015210207 >> v2:172.21.15.191:3300/0 conn(0x7fa520071350 0x7fa520069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:14.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa52793e640 1 --2- 172.21.15.88:0/4015210207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa52014fc50 0x7fa52006ffe0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:14.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa52793e640 1 -- 172.21.15.88:0/4015210207 >> 172.21.15.88:0/4015210207 conn(0x7fa52006e0f0 msgr2=0x7fa5200cb180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:14.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa52793e640 1 -- 172.21.15.88:0/4015210207 shutdown_connections 2023-10-16T20:44:14.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa52793e640 1 -- 172.21.15.88:0/4015210207 wait complete. 2023-10-16T20:44:14.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa52793e640 1 Processor -- start 2023-10-16T20:44:14.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa52793e640 1 -- start start 2023-10-16T20:44:14.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa52793e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa520071350 0x7fa5200c9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:14.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa52793e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa520071790 0x7fa5201437b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:14.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa52793e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa52014fc50 0x7fa52006a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:14.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa52793e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5201582f0 con 0x7fa520071350 2023-10-16T20:44:14.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa52793e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5200c1c00 con 0x7fa520071790 2023-10-16T20:44:14.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa52793e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa520142fc0 con 0x7fa52014fc50 2023-10-16T20:44:14.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa5256da640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa520071350 0x7fa5200c9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:14.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa5256da640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa520071350 0x7fa5200c9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58620/0 (socket says 172.21.15.88:58620) 2023-10-16T20:44:14.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa5256da640 1 -- 172.21.15.88:0/110243022 learned_addr learned my addr 172.21.15.88:0/110243022 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:14.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa525edb640 1 --2- 172.21.15.88:0/110243022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa52014fc50 0x7fa52006a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:14.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa524ed9640 1 --2- 172.21.15.88:0/110243022 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa520071790 0x7fa5201437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:14.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa5256da640 1 -- 172.21.15.88:0/110243022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa52014fc50 msgr2=0x7fa52006a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:14.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa5256da640 1 --2- 172.21.15.88:0/110243022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa52014fc50 0x7fa52006a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:14.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa5256da640 1 -- 172.21.15.88:0/110243022 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa520071790 msgr2=0x7fa5201437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:14.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa5256da640 1 --2- 172.21.15.88:0/110243022 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa520071790 0x7fa5201437b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:14.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa5256da640 1 -- 172.21.15.88:0/110243022 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c0094d0 con 0x7fa520071350 2023-10-16T20:44:14.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa524ed9640 1 --2- 172.21.15.88:0/110243022 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa520071790 0x7fa5201437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:44:14.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.556+0000 7fa5256da640 1 --2- 172.21.15.88:0/110243022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa520071350 0x7fa5200c9290 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa51c01d3a0 tx=0x7fa51c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:14.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.560+0000 7fa515ffb640 1 -- 172.21.15.88:0/110243022 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c0ba600 con 0x7fa520071350 2023-10-16T20:44:14.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.560+0000 7fa515ffb640 1 -- 172.21.15.88:0/110243022 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51c0ba760 con 0x7fa520071350 2023-10-16T20:44:14.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.560+0000 7fa515ffb640 1 -- 172.21.15.88:0/110243022 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c0bab10 con 0x7fa520071350 2023-10-16T20:44:14.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.560+0000 7fa52793e640 1 -- 172.21.15.88:0/110243022 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa520143540 con 0x7fa520071350 2023-10-16T20:44:14.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.560+0000 7fa52793e640 1 -- 172.21.15.88:0/110243022 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5200c15b0 con 0x7fa520071350 2023-10-16T20:44:14.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.560+0000 7fa52793e640 1 -- 172.21.15.88:0/110243022 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa520074700 con 0x7fa520071350 2023-10-16T20:44:14.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.560+0000 7fa515ffb640 1 -- 172.21.15.88:0/110243022 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa51c0bac70 con 0x7fa520071350 2023-10-16T20:44:14.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.560+0000 7fa515ffb640 1 --2- 172.21.15.88:0/110243022 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa5040664a0 0x7fa504068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:14.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.560+0000 7fa515ffb640 1 -- 172.21.15.88:0/110243022 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7fa51c122a80 con 0x7fa520071350 2023-10-16T20:44:14.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.560+0000 7fa524ed9640 1 --2- 172.21.15.88:0/110243022 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa5040664a0 0x7fa504068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:14.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.564+0000 7fa524ed9640 1 --2- 172.21.15.88:0/110243022 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa5040664a0 0x7fa504068960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa52006ac00 tx=0x7fa510008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:14.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.564+0000 7fa515ffb640 1 -- 172.21.15.88:0/110243022 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa51c0ef960 con 0x7fa520071350 2023-10-16T20:44:14.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.748+0000 7fa52793e640 1 -- 172.21.15.88:0/110243022 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa52014df00 con 0x7fa520071350 2023-10-16T20:44:14.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.748+0000 7fa515ffb640 1 -- 172.21.15.88:0/110243022 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6575 (secure 0 0 0) 0x7fa51c0b5020 con 0x7fa520071350 2023-10-16T20:44:14.753 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:14.753 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:01.507428+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:14.754 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 37 2023-10-16T20:44:14.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.752+0000 7fa52793e640 1 -- 172.21.15.88:0/110243022 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa5040664a0 msgr2=0x7fa504068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:14.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.752+0000 7fa52793e640 1 --2- 172.21.15.88:0/110243022 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa5040664a0 0x7fa504068960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa52006ac00 tx=0x7fa510008040 comp rx=0 tx=0).stop 2023-10-16T20:44:14.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.752+0000 7fa52793e640 1 -- 172.21.15.88:0/110243022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa520071350 msgr2=0x7fa5200c9290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:14.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.752+0000 7fa52793e640 1 --2- 172.21.15.88:0/110243022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa520071350 0x7fa5200c9290 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa51c01d3a0 tx=0x7fa51c01ba90 comp rx=0 tx=0).stop 2023-10-16T20:44:14.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.752+0000 7fa52793e640 1 -- 172.21.15.88:0/110243022 shutdown_connections 2023-10-16T20:44:14.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.752+0000 7fa52793e640 1 --2- 172.21.15.88:0/110243022 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa5040664a0 0x7fa504068960 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:14.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.752+0000 7fa52793e640 1 --2- 172.21.15.88:0/110243022 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa52014fc50 0x7fa52006a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:14.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.752+0000 7fa52793e640 1 --2- 172.21.15.88:0/110243022 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa520071790 0x7fa5201437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:14.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.752+0000 7fa52793e640 1 --2- 172.21.15.88:0/110243022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa520071350 0x7fa5200c9290 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:14.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.752+0000 7fa52793e640 1 -- 172.21.15.88:0/110243022 >> 172.21.15.88:0/110243022 conn(0x7fa52006e0f0 msgr2=0x7fa520070850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:14.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.752+0000 7fa52793e640 1 -- 172.21.15.88:0/110243022 shutdown_connections 2023-10-16T20:44:14.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:14.752+0000 7fa52793e640 1 -- 172.21.15.88:0/110243022 wait complete. 2023-10-16T20:44:14.776 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-16T20:44:16.777 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:16.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.920+0000 7f74a9d3e640 1 Processor -- start 2023-10-16T20:44:16.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.920+0000 7f74a9d3e640 1 -- start start 2023-10-16T20:44:16.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.924+0000 7f74a9d3e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74a41406d0 0x7f74a40c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:16.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.924+0000 7f74a9d3e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f74a406a970 0x7f74a4071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:16.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.924+0000 7f74a9d3e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a415b3c0 con 0x7f74a4071e60 2023-10-16T20:44:16.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.924+0000 7f74a9d3e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a415b680 con 0x7f74a406a970 2023-10-16T20:44:16.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.924+0000 7f74a9d3e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a4157570 con 0x7f74a41406d0 2023-10-16T20:44:16.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.924+0000 7f74a2ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f74a406a970 0x7f74a4071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:16.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.924+0000 7f74a2ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f74a406a970 0x7f74a4071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58630/0 (socket says 172.21.15.88:58630) 2023-10-16T20:44:16.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.924+0000 7f74a2ffd640 1 -- 172.21.15.88:0/3573217536 learned_addr learned my addr 172.21.15.88:0/3573217536 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:16.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.924+0000 7f74a37fe640 1 --2- 172.21.15.88:0/3573217536 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74a41406d0 0x7f74a40c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:16.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.924+0000 7f74a2ffd640 1 -- 172.21.15.88:0/3573217536 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74a41406d0 msgr2=0x7f74a40c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:16.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.924+0000 7f74a2ffd640 1 --2- 172.21.15.88:0/3573217536 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74a41406d0 0x7f74a40c1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:16.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.924+0000 7f74a2ffd640 1 -- 172.21.15.88:0/3573217536 >> v1:172.21.15.88:6789/0 conn(0x7f74a4071e60 legacy=0x7f74a415a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:16.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.924+0000 7f74a2ffd640 1 -- 172.21.15.88:0/3573217536 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a4157850 con 0x7f74a406a970 2023-10-16T20:44:16.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.928+0000 7f74a2ffd640 1 --2- 172.21.15.88:0/3573217536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a406a970 0x7f74a4071920 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7498009870 tx=0x7f749801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa70019bb55d53c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:16.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.928+0000 7f74a27fc640 1 -- 172.21.15.88:0/3573217536 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74980ab620 con 0x7f74a406a970 2023-10-16T20:44:16.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.928+0000 7f74a27fc640 1 -- 172.21.15.88:0/3573217536 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74980ab780 con 0x7f74a406a970 2023-10-16T20:44:16.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.928+0000 7f74a27fc640 1 -- 172.21.15.88:0/3573217536 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74980abb10 con 0x7f74a406a970 2023-10-16T20:44:16.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.928+0000 7f74a9d3e640 1 -- 172.21.15.88:0/3573217536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a406a970 msgr2=0x7f74a4071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:16.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.928+0000 7f74a9d3e640 1 --2- 172.21.15.88:0/3573217536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a406a970 0x7f74a4071920 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7498009870 tx=0x7f749801c510 comp rx=0 tx=0).stop 2023-10-16T20:44:16.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.928+0000 7f74a9d3e640 1 -- 172.21.15.88:0/3573217536 shutdown_connections 2023-10-16T20:44:16.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.928+0000 7f74a9d3e640 1 --2- 172.21.15.88:0/3573217536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a406a970 0x7f74a4071920 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:16.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.928+0000 7f74a9d3e640 1 --2- 172.21.15.88:0/3573217536 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74a41406d0 0x7f74a40c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:16.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.928+0000 7f74a9d3e640 1 -- 172.21.15.88:0/3573217536 >> 172.21.15.88:0/3573217536 conn(0x7f74a40cad50 msgr2=0x7f74a40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:16.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.928+0000 7f74a9d3e640 1 -- 172.21.15.88:0/3573217536 shutdown_connections 2023-10-16T20:44:16.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.928+0000 7f74a9d3e640 1 -- 172.21.15.88:0/3573217536 wait complete. 2023-10-16T20:44:16.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.928+0000 7f74a9d3e640 1 Processor -- start 2023-10-16T20:44:16.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.928+0000 7f74a9d3e640 1 -- start start 2023-10-16T20:44:16.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a9d3e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74a406a970 0x7f74a40c1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:16.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a9d3e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74a4071e60 0x7f74a41f5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:16.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a9d3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a41406d0 0x7f74a41374d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:16.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a9d3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a40704c0 con 0x7f74a41406d0 2023-10-16T20:44:16.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a9d3e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a40c2850 con 0x7f74a4071e60 2023-10-16T20:44:16.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a9d3e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a41f57b0 con 0x7f74a406a970 2023-10-16T20:44:16.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a41406d0 0x7f74a41374d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:16.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a41406d0 0x7f74a41374d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58642/0 (socket says 172.21.15.88:58642) 2023-10-16T20:44:16.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a3fff640 1 -- 172.21.15.88:0/3210399735 learned_addr learned my addr 172.21.15.88:0/3210399735 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:16.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a37fe640 1 --2- 172.21.15.88:0/3210399735 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74a406a970 0x7f74a40c1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:16.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a2ffd640 1 --2- 172.21.15.88:0/3210399735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74a4071e60 0x7f74a41f5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:16.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a3fff640 1 -- 172.21.15.88:0/3210399735 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74a406a970 msgr2=0x7f74a40c1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:16.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a3fff640 1 --2- 172.21.15.88:0/3210399735 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74a406a970 0x7f74a40c1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:16.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a3fff640 1 -- 172.21.15.88:0/3210399735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74a4071e60 msgr2=0x7f74a41f5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:16.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a3fff640 1 --2- 172.21.15.88:0/3210399735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74a4071e60 0x7f74a41f5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:16.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a3fff640 1 -- 172.21.15.88:0/3210399735 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74980094d0 con 0x7f74a41406d0 2023-10-16T20:44:16.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a37fe640 1 --2- 172.21.15.88:0/3210399735 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74a406a970 0x7f74a40c1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:44:16.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a3fff640 1 --2- 172.21.15.88:0/3210399735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a41406d0 0x7f74a41374d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f749c00c7a0 tx=0x7f749c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:16.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f748bfff640 1 -- 172.21.15.88:0/3210399735 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749c003b80 con 0x7f74a41406d0 2023-10-16T20:44:16.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f748bfff640 1 -- 172.21.15.88:0/3210399735 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749c003ce0 con 0x7f74a41406d0 2023-10-16T20:44:16.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a9d3e640 1 -- 172.21.15.88:0/3210399735 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a41581d0 con 0x7f74a41406d0 2023-10-16T20:44:16.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f748bfff640 1 -- 172.21.15.88:0/3210399735 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749c00aa30 con 0x7f74a41406d0 2023-10-16T20:44:16.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a9d3e640 1 -- 172.21.15.88:0/3210399735 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a40c2570 con 0x7f74a41406d0 2023-10-16T20:44:16.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f74a9d3e640 1 -- 172.21.15.88:0/3210399735 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a4074700 con 0x7f74a41406d0 2023-10-16T20:44:16.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.932+0000 7f748bfff640 1 -- 172.21.15.88:0/3210399735 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f749c00ab90 con 0x7f74a41406d0 2023-10-16T20:44:16.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.936+0000 7f748bfff640 1 --2- 172.21.15.88:0/3210399735 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74900664a0 0x7f7490068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:16.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.936+0000 7f748bfff640 1 -- 172.21.15.88:0/3210399735 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7f749c0841e0 con 0x7f74a41406d0 2023-10-16T20:44:16.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.936+0000 7f74a37fe640 1 --2- 172.21.15.88:0/3210399735 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74900664a0 0x7f7490068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:16.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.936+0000 7f74a37fe640 1 --2- 172.21.15.88:0/3210399735 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74900664a0 0x7f7490068960 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7494007a70 tx=0x7f7494008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:16.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:16.940+0000 7f748bfff640 1 -- 172.21.15.88:0/3210399735 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f749c0510c0 con 0x7f74a41406d0 2023-10-16T20:44:17.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:17.120+0000 7f74a9d3e640 1 -- 172.21.15.88:0/3210399735 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74a40c3dc0 con 0x7f74a41406d0 2023-10-16T20:44:17.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:17.120+0000 7f748bfff640 1 -- 172.21.15.88:0/3210399735 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6575 (secure 0 0 0) 0x7f749c0546e0 con 0x7f74a41406d0 2023-10-16T20:44:17.124 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:17.124 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:01.507428+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:17.125 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 37 2023-10-16T20:44:17.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:17.124+0000 7f74a9d3e640 1 -- 172.21.15.88:0/3210399735 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74900664a0 msgr2=0x7f7490068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:17.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:17.124+0000 7f74a9d3e640 1 --2- 172.21.15.88:0/3210399735 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74900664a0 0x7f7490068960 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7494007a70 tx=0x7f7494008040 comp rx=0 tx=0).stop 2023-10-16T20:44:17.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:17.124+0000 7f74a9d3e640 1 -- 172.21.15.88:0/3210399735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a41406d0 msgr2=0x7f74a41374d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:17.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:17.124+0000 7f74a9d3e640 1 --2- 172.21.15.88:0/3210399735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a41406d0 0x7f74a41374d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f749c00c7a0 tx=0x7f749c00cc60 comp rx=0 tx=0).stop 2023-10-16T20:44:17.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:17.124+0000 7f74a9d3e640 1 -- 172.21.15.88:0/3210399735 shutdown_connections 2023-10-16T20:44:17.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:17.124+0000 7f74a9d3e640 1 --2- 172.21.15.88:0/3210399735 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74900664a0 0x7f7490068960 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:17.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:17.124+0000 7f74a9d3e640 1 --2- 172.21.15.88:0/3210399735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a41406d0 0x7f74a41374d0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:17.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:17.124+0000 7f74a9d3e640 1 --2- 172.21.15.88:0/3210399735 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74a4071e60 0x7f74a41f5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:17.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:17.124+0000 7f74a9d3e640 1 --2- 172.21.15.88:0/3210399735 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74a406a970 0x7f74a40c1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:17.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:17.124+0000 7f74a9d3e640 1 -- 172.21.15.88:0/3210399735 >> 172.21.15.88:0/3210399735 conn(0x7f74a40cad50 msgr2=0x7f74a413e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:17.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:17.124+0000 7f74a9d3e640 1 -- 172.21.15.88:0/3210399735 shutdown_connections 2023-10-16T20:44:17.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:17.124+0000 7f74a9d3e640 1 -- 172.21.15.88:0/3210399735 wait complete. 2023-10-16T20:44:17.145 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-16T20:44:19.146 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:19.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f982193b640 1 Processor -- start 2023-10-16T20:44:19.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f982193b640 1 -- start start 2023-10-16T20:44:19.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f982193b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f981c0cb580 0x7f981c0cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:19.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f982193b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f981c0c9af0 0x7f981c159300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:19.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f982193b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f981c0b1a60 0x7f981c159840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:19.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f982193b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f981c0b20c0 con 0x7f981c0b1a60 2023-10-16T20:44:19.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f982193b640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f981c0cbea0 con 0x7f981c0cb580 2023-10-16T20:44:19.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f982193b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981c0c9fc0 con 0x7f981c0c9af0 2023-10-16T20:44:19.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f982113a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f981c0b1a60 0x7f981c159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:19.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f982113a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f981c0b1a60 0x7f981c159840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43084/0 (socket says 172.21.15.88:43084) 2023-10-16T20:44:19.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f982113a640 1 -- 172.21.15.88:0/3045624064 learned_addr learned my addr 172.21.15.88:0/3045624064 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:19.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f9813fff640 1 --2- 172.21.15.88:0/3045624064 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f981c0c9af0 0x7f981c159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:19.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f9820939640 1 --2- 172.21.15.88:0/3045624064 >> v2:172.21.15.191:3300/0 conn(0x7f981c0cb580 0x7f981c0cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:19.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f982113a640 1 -- 172.21.15.88:0/3045624064 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f981c0c9af0 msgr2=0x7f981c159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:19.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f982113a640 1 --2- 172.21.15.88:0/3045624064 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f981c0c9af0 0x7f981c159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:19.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f982113a640 1 -- 172.21.15.88:0/3045624064 >> v2:172.21.15.191:3300/0 conn(0x7f981c0cb580 msgr2=0x7f981c0cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:19.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f982113a640 1 --2- 172.21.15.88:0/3045624064 >> v2:172.21.15.191:3300/0 conn(0x7f981c0cb580 0x7f981c0cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:19.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.292+0000 7f982113a640 1 -- 172.21.15.88:0/3045624064 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c0c47d0 con 0x7f981c0b1a60 2023-10-16T20:44:19.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.296+0000 7f982113a640 1 --2- 172.21.15.88:0/3045624064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f981c0b1a60 0x7f981c159840 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f981c072780 tx=0x7f981401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ff5c5bc8164ce server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:19.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.296+0000 7f98137fe640 1 -- 172.21.15.88:0/3045624064 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98140a57c0 con 0x7f981c0b1a60 2023-10-16T20:44:19.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.296+0000 7f98137fe640 1 -- 172.21.15.88:0/3045624064 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98140a5920 con 0x7f981c0b1a60 2023-10-16T20:44:19.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.296+0000 7f98137fe640 1 -- 172.21.15.88:0/3045624064 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98140a5cb0 con 0x7f981c0b1a60 2023-10-16T20:44:19.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.296+0000 7f982193b640 1 -- 172.21.15.88:0/3045624064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f981c0b1a60 msgr2=0x7f981c159840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:19.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.296+0000 7f982193b640 1 --2- 172.21.15.88:0/3045624064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f981c0b1a60 0x7f981c159840 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f981c072780 tx=0x7f981401c940 comp rx=0 tx=0).stop 2023-10-16T20:44:19.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.296+0000 7f982193b640 1 -- 172.21.15.88:0/3045624064 shutdown_connections 2023-10-16T20:44:19.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.296+0000 7f982193b640 1 --2- 172.21.15.88:0/3045624064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f981c0b1a60 0x7f981c159840 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:19.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.296+0000 7f982193b640 1 --2- 172.21.15.88:0/3045624064 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f981c0c9af0 0x7f981c159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:19.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.296+0000 7f982193b640 1 --2- 172.21.15.88:0/3045624064 >> v2:172.21.15.191:3300/0 conn(0x7f981c0cb580 0x7f981c0cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:19.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.296+0000 7f982193b640 1 -- 172.21.15.88:0/3045624064 >> 172.21.15.88:0/3045624064 conn(0x7f981c06bb40 msgr2=0x7f981c06bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:19.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.296+0000 7f982193b640 1 -- 172.21.15.88:0/3045624064 shutdown_connections 2023-10-16T20:44:19.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.296+0000 7f982193b640 1 -- 172.21.15.88:0/3045624064 wait complete. 2023-10-16T20:44:19.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f982193b640 1 Processor -- start 2023-10-16T20:44:19.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f982193b640 1 -- start start 2023-10-16T20:44:19.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f982193b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f981c0b1a60 0x7f981c13ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:19.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f982193b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f981c0c9af0 0x7f981c0b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:19.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f982193b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f981c0cb580 0x7f981c144f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:19.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f982193b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c13ef60 con 0x7f981c0b1a60 2023-10-16T20:44:19.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f982193b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c0c7570 con 0x7f981c0c9af0 2023-10-16T20:44:19.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f982193b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981c0c69d0 con 0x7f981c0cb580 2023-10-16T20:44:19.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f9820939640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f981c0b1a60 0x7f981c13ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:19.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f9820939640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f981c0b1a60 0x7f981c13ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43096/0 (socket says 172.21.15.88:43096) 2023-10-16T20:44:19.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f9820939640 1 -- 172.21.15.88:0/3929007459 learned_addr learned my addr 172.21.15.88:0/3929007459 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:19.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f982113a640 1 --2- 172.21.15.88:0/3929007459 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f981c0cb580 0x7f981c144f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:19.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f9813fff640 1 --2- 172.21.15.88:0/3929007459 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f981c0c9af0 0x7f981c0b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:19.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f9820939640 1 -- 172.21.15.88:0/3929007459 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f981c0cb580 msgr2=0x7f981c144f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:19.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f9820939640 1 --2- 172.21.15.88:0/3929007459 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f981c0cb580 0x7f981c144f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:19.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f9820939640 1 -- 172.21.15.88:0/3929007459 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f981c0c9af0 msgr2=0x7f981c0b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:19.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f9820939640 1 --2- 172.21.15.88:0/3929007459 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f981c0c9af0 0x7f981c0b3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:19.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f9820939640 1 -- 172.21.15.88:0/3929007459 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9814009b30 con 0x7f981c0b1a60 2023-10-16T20:44:19.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f9820939640 1 --2- 172.21.15.88:0/3929007459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f981c0b1a60 0x7f981c13ea20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f981800d620 tx=0x7f981800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:19.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f98117fa640 1 -- 172.21.15.88:0/3929007459 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981800b620 con 0x7f981c0b1a60 2023-10-16T20:44:19.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f98117fa640 1 -- 172.21.15.88:0/3929007459 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981800b780 con 0x7f981c0b1a60 2023-10-16T20:44:19.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f98117fa640 1 -- 172.21.15.88:0/3929007459 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981800bb10 con 0x7f981c0b1a60 2023-10-16T20:44:19.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f982193b640 1 -- 172.21.15.88:0/3929007459 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981c0c6c40 con 0x7f981c0b1a60 2023-10-16T20:44:19.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.300+0000 7f982193b640 1 -- 172.21.15.88:0/3929007459 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981c0c7200 con 0x7f981c0b1a60 2023-10-16T20:44:19.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.304+0000 7f982193b640 1 -- 172.21.15.88:0/3929007459 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981c0ca2c0 con 0x7f981c0b1a60 2023-10-16T20:44:19.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.304+0000 7f98117fa640 1 -- 172.21.15.88:0/3929007459 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f981800bc70 con 0x7f981c0b1a60 2023-10-16T20:44:19.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.304+0000 7f98117fa640 1 --2- 172.21.15.88:0/3929007459 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f98080664a0 0x7f9808068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:19.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.304+0000 7f98117fa640 1 -- 172.21.15.88:0/3929007459 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7f98180846f0 con 0x7f981c0b1a60 2023-10-16T20:44:19.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.304+0000 7f9813fff640 1 --2- 172.21.15.88:0/3929007459 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f98080664a0 0x7f9808068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:19.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.304+0000 7f9813fff640 1 --2- 172.21.15.88:0/3929007459 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f98080664a0 0x7f9808068960 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f981c159090 tx=0x7f980c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:19.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.308+0000 7f98117fa640 1 -- 172.21.15.88:0/3929007459 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9818051790 con 0x7f981c0b1a60 2023-10-16T20:44:19.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.492+0000 7f982193b640 1 -- 172.21.15.88:0/3929007459 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f981c04e9c0 con 0x7f981c0b1a60 2023-10-16T20:44:19.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.496+0000 7f98117fa640 1 -- 172.21.15.88:0/3929007459 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6575 (secure 0 0 0) 0x7f9818054db0 con 0x7f981c0b1a60 2023-10-16T20:44:19.498 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:19.499 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:01.507428+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:19.499 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 37 2023-10-16T20:44:19.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.496+0000 7f982193b640 1 -- 172.21.15.88:0/3929007459 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f98080664a0 msgr2=0x7f9808068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:19.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.496+0000 7f982193b640 1 --2- 172.21.15.88:0/3929007459 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f98080664a0 0x7f9808068960 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f981c159090 tx=0x7f980c008040 comp rx=0 tx=0).stop 2023-10-16T20:44:19.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.496+0000 7f982193b640 1 -- 172.21.15.88:0/3929007459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f981c0b1a60 msgr2=0x7f981c13ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:19.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.496+0000 7f982193b640 1 --2- 172.21.15.88:0/3929007459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f981c0b1a60 0x7f981c13ea20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f981800d620 tx=0x7f981800dae0 comp rx=0 tx=0).stop 2023-10-16T20:44:19.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.496+0000 7f982193b640 1 -- 172.21.15.88:0/3929007459 shutdown_connections 2023-10-16T20:44:19.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.496+0000 7f982193b640 1 --2- 172.21.15.88:0/3929007459 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f98080664a0 0x7f9808068960 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:19.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.496+0000 7f982193b640 1 --2- 172.21.15.88:0/3929007459 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f981c0cb580 0x7f981c144f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:19.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.496+0000 7f982193b640 1 --2- 172.21.15.88:0/3929007459 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f981c0c9af0 0x7f981c0b3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:19.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.496+0000 7f982193b640 1 --2- 172.21.15.88:0/3929007459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f981c0b1a60 0x7f981c13ea20 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:19.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.496+0000 7f982193b640 1 -- 172.21.15.88:0/3929007459 >> 172.21.15.88:0/3929007459 conn(0x7f981c06bb40 msgr2=0x7f981c0cd510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:19.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.496+0000 7f982193b640 1 -- 172.21.15.88:0/3929007459 shutdown_connections 2023-10-16T20:44:19.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:19.496+0000 7f982193b640 1 -- 172.21.15.88:0/3929007459 wait complete. 2023-10-16T20:44:19.519 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-16T20:44:21.520 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.668+0000 7fc43f794640 1 Processor -- start 2023-10-16T20:44:21.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.668+0000 7fc43f794640 1 -- start start 2023-10-16T20:44:21.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.672+0000 7fc43f794640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc438150660 0x7fc4380c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:21.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.672+0000 7fc43f794640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc4380c6bc0 0x7fc4380c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:21.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.672+0000 7fc43f794640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4380c5c60 con 0x7fc438150280 2023-10-16T20:44:21.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.672+0000 7fc43f794640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4380c5370 con 0x7fc4380c6bc0 2023-10-16T20:44:21.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.672+0000 7fc43f794640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc438156d10 con 0x7fc438150660 2023-10-16T20:44:21.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.672+0000 7fc43dd31640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc4380c6bc0 0x7fc4380c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:21.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.672+0000 7fc43dd31640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc4380c6bc0 0x7fc4380c3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43100/0 (socket says 172.21.15.88:43100) 2023-10-16T20:44:21.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.672+0000 7fc43dd31640 1 -- 172.21.15.88:0/561017361 learned_addr learned my addr 172.21.15.88:0/561017361 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:21.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.672+0000 7fc43cd2f640 1 --2- 172.21.15.88:0/561017361 >> v2:172.21.15.191:3300/0 conn(0x7fc438150660 0x7fc4380c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:21.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.672+0000 7fc43dd31640 1 -- 172.21.15.88:0/561017361 >> v2:172.21.15.191:3300/0 conn(0x7fc438150660 msgr2=0x7fc4380c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:21.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.672+0000 7fc43dd31640 1 --2- 172.21.15.88:0/561017361 >> v2:172.21.15.191:3300/0 conn(0x7fc438150660 0x7fc4380c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:21.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.672+0000 7fc43dd31640 1 -- 172.21.15.88:0/561017361 >> v1:172.21.15.191:6789/0 conn(0x7fc438150280 legacy=0x7fc43806d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:21.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.672+0000 7fc43dd31640 1 -- 172.21.15.88:0/561017361 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4380c5610 con 0x7fc4380c6bc0 2023-10-16T20:44:21.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.672+0000 7fc43dd31640 1 --2- 172.21.15.88:0/561017361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4380c6bc0 0x7fc4380c3110 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc4380c1290 tx=0x7fc43001d400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ffb4500f2f33066 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:21.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc42ffff640 1 -- 172.21.15.88:0/561017361 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4300a5880 con 0x7fc4380c6bc0 2023-10-16T20:44:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc42ffff640 1 -- 172.21.15.88:0/561017361 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4300a59e0 con 0x7fc4380c6bc0 2023-10-16T20:44:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc42ffff640 1 -- 172.21.15.88:0/561017361 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4300a5d70 con 0x7fc4380c6bc0 2023-10-16T20:44:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43f794640 1 -- 172.21.15.88:0/561017361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4380c6bc0 msgr2=0x7fc4380c3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43f794640 1 --2- 172.21.15.88:0/561017361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4380c6bc0 0x7fc4380c3110 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc4380c1290 tx=0x7fc43001d400 comp rx=0 tx=0).stop 2023-10-16T20:44:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43f794640 1 -- 172.21.15.88:0/561017361 shutdown_connections 2023-10-16T20:44:21.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43f794640 1 --2- 172.21.15.88:0/561017361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4380c6bc0 0x7fc4380c3110 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:21.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43f794640 1 --2- 172.21.15.88:0/561017361 >> v2:172.21.15.191:3300/0 conn(0x7fc438150660 0x7fc4380c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:21.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43f794640 1 -- 172.21.15.88:0/561017361 >> 172.21.15.88:0/561017361 conn(0x7fc4380cad50 msgr2=0x7fc4380cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:21.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43f794640 1 -- 172.21.15.88:0/561017361 shutdown_connections 2023-10-16T20:44:21.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43f794640 1 -- 172.21.15.88:0/561017361 wait complete. 2023-10-16T20:44:21.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43f794640 1 Processor -- start 2023-10-16T20:44:21.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43f794640 1 -- start start 2023-10-16T20:44:21.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43f794640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4380c6bc0 0x7fc438153640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:21.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43f794640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc438150280 0x7fc43814ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:21.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43f794640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc438150660 0x7fc438150a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:21.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43f794640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc438162c70 con 0x7fc4380c6bc0 2023-10-16T20:44:21.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43f794640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4381629b0 con 0x7fc438150660 2023-10-16T20:44:21.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43f794640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43806f8b0 con 0x7fc438150280 2023-10-16T20:44:21.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43d530640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4380c6bc0 0x7fc438153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:21.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43d530640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4380c6bc0 0x7fc438153640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43104/0 (socket says 172.21.15.88:43104) 2023-10-16T20:44:21.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.676+0000 7fc43d530640 1 -- 172.21.15.88:0/605118676 learned_addr learned my addr 172.21.15.88:0/605118676 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:21.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc43cd2f640 1 --2- 172.21.15.88:0/605118676 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc438150280 0x7fc43814ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:21.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc43dd31640 1 --2- 172.21.15.88:0/605118676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc438150660 0x7fc438150a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:21.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc43d530640 1 -- 172.21.15.88:0/605118676 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc438150280 msgr2=0x7fc43814ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:21.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc43d530640 1 --2- 172.21.15.88:0/605118676 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc438150280 0x7fc43814ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:21.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc43d530640 1 -- 172.21.15.88:0/605118676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc438150660 msgr2=0x7fc438150a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:21.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc43d530640 1 --2- 172.21.15.88:0/605118676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc438150660 0x7fc438150a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:21.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc43d530640 1 -- 172.21.15.88:0/605118676 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc430009b30 con 0x7fc4380c6bc0 2023-10-16T20:44:21.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc43dd31640 1 --2- 172.21.15.88:0/605118676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc438150660 0x7fc438150a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:44:21.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc43d530640 1 --2- 172.21.15.88:0/605118676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4380c6bc0 0x7fc438153640 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc43400c500 tx=0x7fc43400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:21.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc42dffb640 1 -- 172.21.15.88:0/605118676 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc434005220 con 0x7fc4380c6bc0 2023-10-16T20:44:21.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc42dffb640 1 -- 172.21.15.88:0/605118676 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc434005380 con 0x7fc4380c6bc0 2023-10-16T20:44:21.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc43f794640 1 -- 172.21.15.88:0/605118676 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43806fb90 con 0x7fc4380c6bc0 2023-10-16T20:44:21.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc42dffb640 1 -- 172.21.15.88:0/605118676 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc434007270 con 0x7fc4380c6bc0 2023-10-16T20:44:21.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc43f794640 1 -- 172.21.15.88:0/605118676 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc438070150 con 0x7fc4380c6bc0 2023-10-16T20:44:21.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc43f794640 1 -- 172.21.15.88:0/605118676 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc438074700 con 0x7fc4380c6bc0 2023-10-16T20:44:21.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc42dffb640 1 -- 172.21.15.88:0/605118676 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc4340073d0 con 0x7fc4380c6bc0 2023-10-16T20:44:21.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc42dffb640 1 --2- 172.21.15.88:0/605118676 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc41c0664d0 0x7fc41c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:21.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.680+0000 7fc42dffb640 1 -- 172.21.15.88:0/605118676 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7fc434083ae0 con 0x7fc4380c6bc0 2023-10-16T20:44:21.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.684+0000 7fc43cd2f640 1 --2- 172.21.15.88:0/605118676 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc41c0664d0 0x7fc41c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:21.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.684+0000 7fc43cd2f640 1 --2- 172.21.15.88:0/605118676 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc41c0664d0 0x7fc41c068990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc43814fc40 tx=0x7fc428008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:21.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.688+0000 7fc42dffb640 1 -- 172.21.15.88:0/605118676 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc434050b00 con 0x7fc4380c6bc0 2023-10-16T20:44:21.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.868+0000 7fc43f794640 1 -- 172.21.15.88:0/605118676 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4380c1650 con 0x7fc4380c6bc0 2023-10-16T20:44:21.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.868+0000 7fc42dffb640 1 -- 172.21.15.88:0/605118676 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6575 (secure 0 0 0) 0x7fc434054120 con 0x7fc4380c6bc0 2023-10-16T20:44:21.872 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:21.873 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:01.507428+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:21.873 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 37 2023-10-16T20:44:21.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.872+0000 7fc43f794640 1 -- 172.21.15.88:0/605118676 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc41c0664d0 msgr2=0x7fc41c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:21.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.872+0000 7fc43f794640 1 --2- 172.21.15.88:0/605118676 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc41c0664d0 0x7fc41c068990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc43814fc40 tx=0x7fc428008040 comp rx=0 tx=0).stop 2023-10-16T20:44:21.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.872+0000 7fc43f794640 1 -- 172.21.15.88:0/605118676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4380c6bc0 msgr2=0x7fc438153640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:21.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.872+0000 7fc43f794640 1 --2- 172.21.15.88:0/605118676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4380c6bc0 0x7fc438153640 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc43400c500 tx=0x7fc43400c9c0 comp rx=0 tx=0).stop 2023-10-16T20:44:21.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.872+0000 7fc43f794640 1 -- 172.21.15.88:0/605118676 shutdown_connections 2023-10-16T20:44:21.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.872+0000 7fc43f794640 1 --2- 172.21.15.88:0/605118676 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc41c0664d0 0x7fc41c068990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:21.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.872+0000 7fc43f794640 1 --2- 172.21.15.88:0/605118676 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc438150660 0x7fc438150a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:21.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.872+0000 7fc43f794640 1 --2- 172.21.15.88:0/605118676 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc438150280 0x7fc43814ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:21.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.872+0000 7fc43f794640 1 --2- 172.21.15.88:0/605118676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4380c6bc0 0x7fc438153640 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:21.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.872+0000 7fc43f794640 1 -- 172.21.15.88:0/605118676 >> 172.21.15.88:0/605118676 conn(0x7fc4380cad50 msgr2=0x7fc438069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:21.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.872+0000 7fc43f794640 1 -- 172.21.15.88:0/605118676 shutdown_connections 2023-10-16T20:44:21.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:21.872+0000 7fc43f794640 1 -- 172.21.15.88:0/605118676 wait complete. 2023-10-16T20:44:21.893 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-16T20:44:23.895 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:24.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.040+0000 7f74c4486640 1 Processor -- start 2023-10-16T20:44:24.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.040+0000 7f74c4486640 1 -- start start 2023-10-16T20:44:24.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.040+0000 7f74c4486640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f74bc153970 0x7f74bc070830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:24.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.040+0000 7f74c4486640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74bc15d910 con 0x7f74bc1596e0 2023-10-16T20:44:24.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.040+0000 7f74c4486640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74bc15cd00 con 0x7f74bc069e90 2023-10-16T20:44:24.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.040+0000 7f74c4486640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74bc070ec0 con 0x7f74bc153970 2023-10-16T20:44:24.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.040+0000 7f74c2a23640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f74bc1596e0 0x7f74bc159b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:53754/0 (socket says 172.21.15.88:53754) 2023-10-16T20:44:24.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.040+0000 7f74c2a23640 1 -- 172.21.15.88:0/3626037115 learned_addr learned my addr 172.21.15.88:0/3626037115 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:44:24.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.040+0000 7f74c2222640 1 --2- 172.21.15.88:0/3626037115 >> v2:172.21.15.191:3300/0 conn(0x7f74bc153970 0x7f74bc070830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:24.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.040+0000 7f74c1220640 1 -- 172.21.15.88:0/3626037115 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477681858 0 0) 0x7f74bc15d910 con 0x7f74bc1596e0 2023-10-16T20:44:24.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.040+0000 7f74c1220640 1 -- 172.21.15.88:0/3626037115 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74a8003530 con 0x7f74bc1596e0 2023-10-16T20:44:24.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.040+0000 7f74c1220640 1 -- 172.21.15.88:0/3626037115 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1796363962 0 0) 0x7f74a8003530 con 0x7f74bc1596e0 2023-10-16T20:44:24.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.040+0000 7f74c1220640 1 -- 172.21.15.88:0/3626037115 >> v2:172.21.15.191:3300/0 conn(0x7f74bc153970 msgr2=0x7f74bc070830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:24.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.040+0000 7f74c1220640 1 --2- 172.21.15.88:0/3626037115 >> v2:172.21.15.191:3300/0 conn(0x7f74bc153970 0x7f74bc070830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:24.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1220640 1 -- 172.21.15.88:0/3626037115 >> v1:172.21.15.191:6789/0 conn(0x7f74bc069e90 legacy=0x7f74bc06e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:24.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1220640 1 -- 172.21.15.88:0/3626037115 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc15cfe0 con 0x7f74bc1596e0 2023-10-16T20:44:24.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1220640 1 -- 172.21.15.88:0/3626037115 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f74b4001a00 con 0x7f74bc1596e0 2023-10-16T20:44:24.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1220640 1 -- 172.21.15.88:0/3626037115 >> v1:172.21.15.88:6789/0 conn(0x7f74bc1596e0 legacy=0x7f74bc159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:24.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1220640 1 --2- 172.21.15.88:0/3626037115 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74a8003980 0x7f74a8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:24.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1220640 1 --2- 172.21.15.88:0/3626037115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a80047d0 0x7f74a8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:24.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1220640 1 --2- 172.21.15.88:0/3626037115 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74a8005190 0x7f74a800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:24.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1220640 1 -- 172.21.15.88:0/3626037115 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc070ec0 con 0x7f74a80047d0 2023-10-16T20:44:24.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1220640 1 -- 172.21.15.88:0/3626037115 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a800c040 con 0x7f74a8003980 2023-10-16T20:44:24.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1220640 1 -- 172.21.15.88:0/3626037115 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a800c200 con 0x7f74a8005190 2023-10-16T20:44:24.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1a21640 1 --2- 172.21.15.88:0/3626037115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a80047d0 0x7f74a8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:24.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c2222640 1 --2- 172.21.15.88:0/3626037115 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74a8003980 0x7f74a8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:24.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c2a23640 1 --2- 172.21.15.88:0/3626037115 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74a8005190 0x7f74a800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:24.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1a21640 1 -- 172.21.15.88:0/3626037115 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74a8005190 msgr2=0x7f74a800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:24.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1a21640 1 --2- 172.21.15.88:0/3626037115 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74a8005190 0x7f74a800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:24.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1a21640 1 -- 172.21.15.88:0/3626037115 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74a8003980 msgr2=0x7f74a8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:24.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1a21640 1 --2- 172.21.15.88:0/3626037115 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74a8003980 0x7f74a8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:24.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1a21640 1 -- 172.21.15.88:0/3626037115 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a8003530 con 0x7f74a80047d0 2023-10-16T20:44:24.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c2a23640 1 -- 172.21.15.88:0/3626037115 reap_dead start 2023-10-16T20:44:24.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1a21640 1 --2- 172.21.15.88:0/3626037115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a80047d0 0x7f74a8004c50 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f74ac001780 tx=0x7f74ac01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d85560703c792f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:24.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.044+0000 7f74c1220640 1 -- 172.21.15.88:0/3626037115 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac0052d0 con 0x7f74a80047d0 2023-10-16T20:44:24.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c1220640 1 -- 172.21.15.88:0/3626037115 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ac0a7050 con 0x7f74a80047d0 2023-10-16T20:44:24.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c1220640 1 -- 172.21.15.88:0/3626037115 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac003600 con 0x7f74a80047d0 2023-10-16T20:44:24.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c4486640 1 -- 172.21.15.88:0/3626037115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a80047d0 msgr2=0x7f74a8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:24.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c4486640 1 --2- 172.21.15.88:0/3626037115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a80047d0 0x7f74a8004c50 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f74ac001780 tx=0x7f74ac01c3e0 comp rx=0 tx=0).stop 2023-10-16T20:44:24.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c4486640 1 -- 172.21.15.88:0/3626037115 shutdown_connections 2023-10-16T20:44:24.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c4486640 1 --2- 172.21.15.88:0/3626037115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a80047d0 0x7f74a8004c50 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:24.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c4486640 1 -- 172.21.15.88:0/3626037115 >> 172.21.15.88:0/3626037115 conn(0x7f74bc0cbeb0 msgr2=0x7f74bc0cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:24.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c4486640 1 -- 172.21.15.88:0/3626037115 shutdown_connections 2023-10-16T20:44:24.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c4486640 1 -- 172.21.15.88:0/3626037115 wait complete. 2023-10-16T20:44:24.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c4486640 1 Processor -- start 2023-10-16T20:44:24.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c4486640 1 -- start start 2023-10-16T20:44:24.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c4486640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a80047d0 0x7f74bc14c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:24.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c4486640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74bc1568f0 0x7f74bc1e75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:24.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c4486640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74bc1e7af0 0x7f74bc1f1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:24.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c4486640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc1557a0 con 0x7f74a80047d0 2023-10-16T20:44:24.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c4486640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc156ec0 con 0x7f74bc1e7af0 2023-10-16T20:44:24.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c4486640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc156630 con 0x7f74bc1568f0 2023-10-16T20:44:24.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c2222640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a80047d0 0x7f74bc14c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:24.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c2222640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a80047d0 0x7f74bc14c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43116/0 (socket says 172.21.15.88:43116) 2023-10-16T20:44:24.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c2222640 1 -- 172.21.15.88:0/4278322666 learned_addr learned my addr 172.21.15.88:0/4278322666 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:24.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c1a21640 1 --2- 172.21.15.88:0/4278322666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74bc1568f0 0x7f74bc1e75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:24.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.048+0000 7f74c2a23640 1 --2- 172.21.15.88:0/4278322666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74bc1e7af0 0x7f74bc1f1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:24.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.052+0000 7f74c2222640 1 -- 172.21.15.88:0/4278322666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74bc1568f0 msgr2=0x7f74bc1e75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:24.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.052+0000 7f74c2222640 1 --2- 172.21.15.88:0/4278322666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74bc1568f0 0x7f74bc1e75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:24.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.052+0000 7f74c2222640 1 -- 172.21.15.88:0/4278322666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74bc1e7af0 msgr2=0x7f74bc1f1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:24.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.052+0000 7f74c2222640 1 --2- 172.21.15.88:0/4278322666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74bc1e7af0 0x7f74bc1f1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:24.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.052+0000 7f74c2222640 1 -- 172.21.15.88:0/4278322666 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc15cac0 con 0x7f74a80047d0 2023-10-16T20:44:24.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.052+0000 7f74c2222640 1 --2- 172.21.15.88:0/4278322666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a80047d0 0x7f74bc14c970 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f74b8011a20 tx=0x7f74b8011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:24.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.052+0000 7f74b2ffd640 1 -- 172.21.15.88:0/4278322666 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b800fbb0 con 0x7f74a80047d0 2023-10-16T20:44:24.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.052+0000 7f74b2ffd640 1 -- 172.21.15.88:0/4278322666 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b800fd10 con 0x7f74a80047d0 2023-10-16T20:44:24.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.052+0000 7f74c4486640 1 -- 172.21.15.88:0/4278322666 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74bc1e7120 con 0x7f74a80047d0 2023-10-16T20:44:24.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.052+0000 7f74b2ffd640 1 -- 172.21.15.88:0/4278322666 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b8005bc0 con 0x7f74a80047d0 2023-10-16T20:44:24.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.052+0000 7f74c4486640 1 -- 172.21.15.88:0/4278322666 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74bc15c540 con 0x7f74a80047d0 2023-10-16T20:44:24.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.052+0000 7f74c4486640 1 -- 172.21.15.88:0/4278322666 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74bc074700 con 0x7f74a80047d0 2023-10-16T20:44:24.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.052+0000 7f74b2ffd640 1 -- 172.21.15.88:0/4278322666 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f74b8005d20 con 0x7f74a80047d0 2023-10-16T20:44:24.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.052+0000 7f74b2ffd640 1 --2- 172.21.15.88:0/4278322666 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74a00664a0 0x7f74a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:24.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.052+0000 7f74b2ffd640 1 -- 172.21.15.88:0/4278322666 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6835+0+0 (secure 0 0 0) 0x7f74b8084810 con 0x7f74a80047d0 2023-10-16T20:44:24.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.056+0000 7f74c1a21640 1 --2- 172.21.15.88:0/4278322666 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74a00664a0 0x7f74a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:24.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.056+0000 7f74c1a21640 1 --2- 172.21.15.88:0/4278322666 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74a00664a0 0x7f74a0068960 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f74ac001750 tx=0x7f74ac0011a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:24.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.056+0000 7f74b2ffd640 1 -- 172.21.15.88:0/4278322666 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f74b80517a0 con 0x7f74a80047d0 2023-10-16T20:44:24.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.240+0000 7f74c4486640 1 -- 172.21.15.88:0/4278322666 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74bc150300 con 0x7f74a80047d0 2023-10-16T20:44:24.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.240+0000 7f74b2ffd640 1 -- 172.21.15.88:0/4278322666 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6575 (secure 0 0 0) 0x7f74b8022020 con 0x7f74a80047d0 2023-10-16T20:44:24.246 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:24.246 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:01.507428+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":4263,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"a","rank":3,"incarnation":16,"state":"up:active","state_seq":43,"addr":"172.21.15.88:6835/518281867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":518281867},{"type":"v1","addr":"172.21.15.88:6835","nonce":518281867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:24.247 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 37 2023-10-16T20:44:24.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.244+0000 7f74c4486640 1 -- 172.21.15.88:0/4278322666 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74a00664a0 msgr2=0x7f74a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:24.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.244+0000 7f74c4486640 1 --2- 172.21.15.88:0/4278322666 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74a00664a0 0x7f74a0068960 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f74ac001750 tx=0x7f74ac0011a0 comp rx=0 tx=0).stop 2023-10-16T20:44:24.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.244+0000 7f74c4486640 1 -- 172.21.15.88:0/4278322666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a80047d0 msgr2=0x7f74bc14c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:24.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.244+0000 7f74c4486640 1 --2- 172.21.15.88:0/4278322666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a80047d0 0x7f74bc14c970 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f74b8011a20 tx=0x7f74b8011ee0 comp rx=0 tx=0).stop 2023-10-16T20:44:24.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.244+0000 7f74c4486640 1 -- 172.21.15.88:0/4278322666 shutdown_connections 2023-10-16T20:44:24.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.244+0000 7f74c4486640 1 --2- 172.21.15.88:0/4278322666 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74a00664a0 0x7f74a0068960 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:24.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.244+0000 7f74c4486640 1 --2- 172.21.15.88:0/4278322666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74bc1e7af0 0x7f74bc1f1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:24.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.244+0000 7f74c4486640 1 --2- 172.21.15.88:0/4278322666 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74bc1568f0 0x7f74bc1e75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:24.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.244+0000 7f74c4486640 1 --2- 172.21.15.88:0/4278322666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a80047d0 0x7f74bc14c970 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:24.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.244+0000 7f74c4486640 1 -- 172.21.15.88:0/4278322666 >> 172.21.15.88:0/4278322666 conn(0x7f74bc0cbeb0 msgr2=0x7f74bc071270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:24.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.244+0000 7f74c4486640 1 -- 172.21.15.88:0/4278322666 shutdown_connections 2023-10-16T20:44:24.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:24.244+0000 7f74c4486640 1 -- 172.21.15.88:0/4278322666 wait complete. 2023-10-16T20:44:24.265 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-16T20:44:25.435 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:44:25.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:44:26.267 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:26.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f53824d6640 1 Processor -- start 2023-10-16T20:44:26.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f53824d6640 1 -- start start 2023-10-16T20:44:26.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f53824d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f537c1538e0 0x7f537c0ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:26.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f53824d6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f537c15cb20 0x7f537c0cee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:26.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f53824d6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537c0ca610 con 0x7f537c15a2a0 2023-10-16T20:44:26.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f53824d6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f537c1595b0 con 0x7f537c15cb20 2023-10-16T20:44:26.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f53824d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537c15d540 con 0x7f537c1538e0 2023-10-16T20:44:26.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f5380a73640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f537c15cb20 0x7f537c0cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:26.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f5380a73640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f537c15cb20 0x7f537c0cee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:59230/0 (socket says 172.21.15.88:59230) 2023-10-16T20:44:26.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f5380a73640 1 -- 172.21.15.88:0/629522135 learned_addr learned my addr 172.21.15.88:0/629522135 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:26.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f537b7fe640 1 --1- 172.21.15.88:0/629522135 >> v1:172.21.15.191:6789/0 conn(0x7f537c15a2a0 0x7f537c0c3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:38160/0 (socket says 172.21.15.88:38160) 2023-10-16T20:44:26.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f537bfff640 1 --2- 172.21.15.88:0/629522135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f537c1538e0 0x7f537c0ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:26.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f5380a73640 1 -- 172.21.15.88:0/629522135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f537c1538e0 msgr2=0x7f537c0ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:26.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f5380a73640 1 --2- 172.21.15.88:0/629522135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f537c1538e0 0x7f537c0ca0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f5380a73640 1 -- 172.21.15.88:0/629522135 >> v1:172.21.15.191:6789/0 conn(0x7f537c15a2a0 legacy=0x7f537c0c3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:26.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f5380a73640 1 -- 172.21.15.88:0/629522135 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537c15d820 con 0x7f537c15cb20 2023-10-16T20:44:26.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f537affd640 1 -- 172.21.15.88:0/629522135 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501698997 0 0) 0x7f537c0ca610 con 0x7f537c15a2a0 2023-10-16T20:44:26.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.412+0000 7f537bfff640 1 --2- 172.21.15.88:0/629522135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f537c1538e0 0x7f537c0ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:44:26.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.416+0000 7f5380a73640 1 --2- 172.21.15.88:0/629522135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f537c15cb20 0x7f537c0cee40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5374009e50 tx=0x7f537401ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dbdc3ba5404bc39 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:26.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.416+0000 7f537affd640 1 -- 172.21.15.88:0/629522135 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53740a57d0 con 0x7f537c15cb20 2023-10-16T20:44:26.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.416+0000 7f537affd640 1 -- 172.21.15.88:0/629522135 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53740a5930 con 0x7f537c15cb20 2023-10-16T20:44:26.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.416+0000 7f537affd640 1 -- 172.21.15.88:0/629522135 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53740a5cc0 con 0x7f537c15cb20 2023-10-16T20:44:26.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.416+0000 7f53824d6640 1 -- 172.21.15.88:0/629522135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f537c15cb20 msgr2=0x7f537c0cee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:26.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.416+0000 7f53824d6640 1 --2- 172.21.15.88:0/629522135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f537c15cb20 0x7f537c0cee40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5374009e50 tx=0x7f537401ca10 comp rx=0 tx=0).stop 2023-10-16T20:44:26.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f53824d6640 1 -- 172.21.15.88:0/629522135 shutdown_connections 2023-10-16T20:44:26.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f53824d6640 1 --2- 172.21.15.88:0/629522135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f537c15cb20 0x7f537c0cee40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f53824d6640 1 --2- 172.21.15.88:0/629522135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f537c1538e0 0x7f537c0ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f53824d6640 1 -- 172.21.15.88:0/629522135 >> 172.21.15.88:0/629522135 conn(0x7f537c06bd50 msgr2=0x7f537c06c160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:26.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f53824d6640 1 -- 172.21.15.88:0/629522135 shutdown_connections 2023-10-16T20:44:26.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f53824d6640 1 -- 172.21.15.88:0/629522135 wait complete. 2023-10-16T20:44:26.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f53824d6640 1 Processor -- start 2023-10-16T20:44:26.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f53824d6640 1 -- start start 2023-10-16T20:44:26.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f53824d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f537c1538e0 0x7f537c1574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:26.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f53824d6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f537c15a2a0 0x7f537c0ce670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:26.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f53824d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f537c15cb20 0x7f537c0cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:26.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f53824d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537c0b4b90 con 0x7f537c1538e0 2023-10-16T20:44:26.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f53824d6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537c158640 con 0x7f537c15a2a0 2023-10-16T20:44:26.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f53824d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537c151e80 con 0x7f537c15cb20 2023-10-16T20:44:26.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f537bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f537c1538e0 0x7f537c1574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:26.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f537bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f537c1538e0 0x7f537c1574b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43126/0 (socket says 172.21.15.88:43126) 2023-10-16T20:44:26.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f537bfff640 1 -- 172.21.15.88:0/240407094 learned_addr learned my addr 172.21.15.88:0/240407094 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:26.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f5380a73640 1 --2- 172.21.15.88:0/240407094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f537c15cb20 0x7f537c0cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:26.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f537b7fe640 1 --2- 172.21.15.88:0/240407094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f537c15a2a0 0x7f537c0ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:26.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f537bfff640 1 -- 172.21.15.88:0/240407094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f537c15cb20 msgr2=0x7f537c0cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:26.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f537bfff640 1 --2- 172.21.15.88:0/240407094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f537c15cb20 0x7f537c0cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f537bfff640 1 -- 172.21.15.88:0/240407094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f537c15a2a0 msgr2=0x7f537c0ce670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:26.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f537bfff640 1 --2- 172.21.15.88:0/240407094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f537c15a2a0 0x7f537c0ce670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f537bfff640 1 -- 172.21.15.88:0/240407094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536c009420 con 0x7f537c1538e0 2023-10-16T20:44:26.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f5380a73640 1 --2- 172.21.15.88:0/240407094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f537c15cb20 0x7f537c0cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:44:26.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.420+0000 7f537bfff640 1 --2- 172.21.15.88:0/240407094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f537c1538e0 0x7f537c1574b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f536c00ec10 tx=0x7f536c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:26.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.424+0000 7f5378ff9640 1 -- 172.21.15.88:0/240407094 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c00cc30 con 0x7f537c1538e0 2023-10-16T20:44:26.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.424+0000 7f5378ff9640 1 -- 172.21.15.88:0/240407094 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536c00cd90 con 0x7f537c1538e0 2023-10-16T20:44:26.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.424+0000 7f53824d6640 1 -- 172.21.15.88:0/240407094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5374009b30 con 0x7f537c1538e0 2023-10-16T20:44:26.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.424+0000 7f5378ff9640 1 -- 172.21.15.88:0/240407094 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c013690 con 0x7f537c1538e0 2023-10-16T20:44:26.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.424+0000 7f53824d6640 1 -- 172.21.15.88:0/240407094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f537c15c740 con 0x7f537c1538e0 2023-10-16T20:44:26.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.424+0000 7f53824d6640 1 -- 172.21.15.88:0/240407094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f537c15c960 con 0x7f537c1538e0 2023-10-16T20:44:26.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.424+0000 7f5378ff9640 1 -- 172.21.15.88:0/240407094 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f536c0137f0 con 0x7f537c1538e0 2023-10-16T20:44:26.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.424+0000 7f5378ff9640 1 --2- 172.21.15.88:0/240407094 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f53680664a0 0x7f5368068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:26.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.424+0000 7f5378ff9640 1 -- 172.21.15.88:0/240407094 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f536c084760 con 0x7f537c1538e0 2023-10-16T20:44:26.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.424+0000 7f537b7fe640 1 --2- 172.21.15.88:0/240407094 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f53680664a0 0x7f5368068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:26.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.424+0000 7f537b7fe640 1 --2- 172.21.15.88:0/240407094 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f53680664a0 0x7f5368068960 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f537c0c4aa0 tx=0x7f5370007b60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:26.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.428+0000 7f5378ff9640 1 -- 172.21.15.88:0/240407094 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f536c0515f0 con 0x7f537c1538e0 2023-10-16T20:44:26.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.612+0000 7f53824d6640 1 -- 172.21.15.88:0/240407094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f537c074700 con 0x7f537c1538e0 2023-10-16T20:44:26.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.612+0000 7f5378ff9640 1 -- 172.21.15.88:0/240407094 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5787 (secure 0 0 0) 0x7f536c054c10 con 0x7f537c1538e0 2023-10-16T20:44:26.616 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:26.616 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:25.369424+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:26.616 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 38 2023-10-16T20:44:26.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.612+0000 7f53824d6640 1 -- 172.21.15.88:0/240407094 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f53680664a0 msgr2=0x7f5368068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:26.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.612+0000 7f53824d6640 1 --2- 172.21.15.88:0/240407094 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f53680664a0 0x7f5368068960 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f537c0c4aa0 tx=0x7f5370007b60 comp rx=0 tx=0).stop 2023-10-16T20:44:26.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.612+0000 7f53824d6640 1 -- 172.21.15.88:0/240407094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f537c1538e0 msgr2=0x7f537c1574b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:26.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.612+0000 7f53824d6640 1 --2- 172.21.15.88:0/240407094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f537c1538e0 0x7f537c1574b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f536c00ec10 tx=0x7f536c00c520 comp rx=0 tx=0).stop 2023-10-16T20:44:26.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.612+0000 7f53824d6640 1 -- 172.21.15.88:0/240407094 shutdown_connections 2023-10-16T20:44:26.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.612+0000 7f53824d6640 1 --2- 172.21.15.88:0/240407094 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f53680664a0 0x7f5368068960 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.612+0000 7f53824d6640 1 --2- 172.21.15.88:0/240407094 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f537c15cb20 0x7f537c0cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.612+0000 7f53824d6640 1 --2- 172.21.15.88:0/240407094 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f537c15a2a0 0x7f537c0ce670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.612+0000 7f53824d6640 1 --2- 172.21.15.88:0/240407094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f537c1538e0 0x7f537c1574b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.612+0000 7f53824d6640 1 -- 172.21.15.88:0/240407094 >> 172.21.15.88:0/240407094 conn(0x7f537c06bd50 msgr2=0x7f537c0cc0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:26.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.616+0000 7f53824d6640 1 -- 172.21.15.88:0/240407094 shutdown_connections 2023-10-16T20:44:26.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.616+0000 7f53824d6640 1 -- 172.21.15.88:0/240407094 wait complete. 2023-10-16T20:44:26.637 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-10-16T20:44:26.637 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-16T20:44:26.638 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.772+0000 7f6e0ac17640 1 Processor -- start 2023-10-16T20:44:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.772+0000 7f6e0ac17640 1 -- start start 2023-10-16T20:44:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.772+0000 7f6e0ac17640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e0406fe20 0x7f6e0406a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.772+0000 7f6e0ac17640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e04156430 con 0x7f6e04069e90 2023-10-16T20:44:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.772+0000 7f6e0ac17640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e04070270 con 0x7f6e0414fd10 2023-10-16T20:44:26.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.772+0000 7f6e0ac17640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e04070500 con 0x7f6e0406fe20 2023-10-16T20:44:26.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfbfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6e04069e90 0x7f6e0406f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:53766/0 (socket says 172.21.15.88:53766) 2023-10-16T20:44:26.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfbfff640 1 -- 172.21.15.88:0/1893086829 learned_addr learned my addr 172.21.15.88:0/1893086829 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:44:26.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6e091b4640 1 --2- 172.21.15.88:0/1893086829 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e0406fe20 0x7f6e0406a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:26.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfb7fe640 1 -- 172.21.15.88:0/1893086829 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876248860 0 0) 0x7f6e04156430 con 0x7f6e04069e90 2023-10-16T20:44:26.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfb7fe640 1 -- 172.21.15.88:0/1893086829 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dec003450 con 0x7f6e04069e90 2023-10-16T20:44:26.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfb7fe640 1 -- 172.21.15.88:0/1893086829 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 169161776 0 0) 0x7f6dec003450 con 0x7f6e04069e90 2023-10-16T20:44:26.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfb7fe640 1 -- 172.21.15.88:0/1893086829 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e0406fe20 msgr2=0x7f6e0406a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:26.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfb7fe640 1 --2- 172.21.15.88:0/1893086829 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e0406fe20 0x7f6e0406a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfb7fe640 1 -- 172.21.15.88:0/1893086829 >> v1:172.21.15.191:6789/0 conn(0x7f6e0414fd10 legacy=0x7f6e04070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:26.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfb7fe640 1 -- 172.21.15.88:0/1893086829 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e04155b00 con 0x7f6e04069e90 2023-10-16T20:44:26.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfb7fe640 1 -- 172.21.15.88:0/1893086829 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f6df4001680 con 0x7f6e04069e90 2023-10-16T20:44:26.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfb7fe640 1 -- 172.21.15.88:0/1893086829 >> v1:172.21.15.88:6789/0 conn(0x7f6e04069e90 legacy=0x7f6e0406f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:26.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfb7fe640 1 --2- 172.21.15.88:0/1893086829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6dec003aa0 0x7f6dec003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:26.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfb7fe640 1 --2- 172.21.15.88:0/1893086829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dec008860 0x7f6dec00ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:26.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfb7fe640 1 --2- 172.21.15.88:0/1893086829 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e0406fe20 0x7f6dec00d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:26.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfb7fe640 1 -- 172.21.15.88:0/1893086829 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e04070500 con 0x7f6dec003aa0 2023-10-16T20:44:26.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfb7fe640 1 -- 172.21.15.88:0/1893086829 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec00db90 con 0x7f6dec008860 2023-10-16T20:44:26.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfb7fe640 1 -- 172.21.15.88:0/1893086829 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec00dd50 con 0x7f6e0406fe20 2023-10-16T20:44:26.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6e089b3640 1 --2- 172.21.15.88:0/1893086829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6dec003aa0 0x7f6dec003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:26.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6dfbfff640 1 --2- 172.21.15.88:0/1893086829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dec008860 0x7f6dec00ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:26.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6e091b4640 1 --2- 172.21.15.88:0/1893086829 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e0406fe20 0x7f6dec00d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:26.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6e089b3640 1 -- 172.21.15.88:0/1893086829 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e0406fe20 msgr2=0x7f6dec00d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:26.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6e089b3640 1 --2- 172.21.15.88:0/1893086829 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e0406fe20 0x7f6dec00d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6e089b3640 1 -- 172.21.15.88:0/1893086829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dec008860 msgr2=0x7f6dec00ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:26.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6e089b3640 1 --2- 172.21.15.88:0/1893086829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dec008860 0x7f6dec00ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.776+0000 7f6e089b3640 1 -- 172.21.15.88:0/1893086829 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dec003450 con 0x7f6dec003aa0 2023-10-16T20:44:26.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.780+0000 7f6e089b3640 1 --2- 172.21.15.88:0/1893086829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6dec003aa0 0x7f6dec003e80 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6e00001910 tx=0x7f6e0001ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc1483bc7d5aebd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:26.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.780+0000 7f6dfb7fe640 1 -- 172.21.15.88:0/1893086829 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e000a88f0 con 0x7f6dec003aa0 2023-10-16T20:44:26.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.780+0000 7f6dfb7fe640 1 -- 172.21.15.88:0/1893086829 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e000a8a70 con 0x7f6dec003aa0 2023-10-16T20:44:26.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.780+0000 7f6dfb7fe640 1 -- 172.21.15.88:0/1893086829 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e000a8e20 con 0x7f6dec003aa0 2023-10-16T20:44:26.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.780+0000 7f6e0ac17640 1 -- 172.21.15.88:0/1893086829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6dec003aa0 msgr2=0x7f6dec003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:26.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.780+0000 7f6e0ac17640 1 --2- 172.21.15.88:0/1893086829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6dec003aa0 0x7f6dec003e80 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6e00001910 tx=0x7f6e0001ecd0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.780+0000 7f6e091b4640 1 -- 172.21.15.88:0/1893086829 reap_dead start 2023-10-16T20:44:26.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.780+0000 7f6e0ac17640 1 -- 172.21.15.88:0/1893086829 shutdown_connections 2023-10-16T20:44:26.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.780+0000 7f6e0ac17640 1 -- 172.21.15.88:0/1893086829 >> 172.21.15.88:0/1893086829 conn(0x7f6e040cad50 msgr2=0x7f6e040cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:26.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.780+0000 7f6e0ac17640 1 -- 172.21.15.88:0/1893086829 shutdown_connections 2023-10-16T20:44:26.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.780+0000 7f6e0ac17640 1 -- 172.21.15.88:0/1893086829 wait complete. 2023-10-16T20:44:26.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.780+0000 7f6e0ac17640 1 Processor -- start 2023-10-16T20:44:26.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.780+0000 7f6e0ac17640 1 -- start start 2023-10-16T20:44:26.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e0ac17640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dec003aa0 0x7f6e040c6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:26.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e0ac17640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e04153bd0 0x7f6e041446f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:26.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e0ac17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e04154dd0 0x7f6e04155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:26.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e0ac17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e040c2860 con 0x7f6e04154dd0 2023-10-16T20:44:26.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e0ac17640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e04155790 con 0x7f6dec003aa0 2023-10-16T20:44:26.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e0ac17640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0414a3a0 con 0x7f6e04153bd0 2023-10-16T20:44:26.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e091b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e04154dd0 0x7f6e04155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:26.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e091b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e04154dd0 0x7f6e04155250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43150/0 (socket says 172.21.15.88:43150) 2023-10-16T20:44:26.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e091b4640 1 -- 172.21.15.88:0/1747333464 learned_addr learned my addr 172.21.15.88:0/1747333464 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:26.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e091b4640 1 -- 172.21.15.88:0/1747333464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e04153bd0 msgr2=0x7f6e041446f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:26.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e091b4640 1 --2- 172.21.15.88:0/1747333464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e04153bd0 0x7f6e041446f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e089b3640 1 --2- 172.21.15.88:0/1747333464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dec003aa0 0x7f6e040c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:26.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e091b4640 1 -- 172.21.15.88:0/1747333464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dec003aa0 msgr2=0x7f6e040c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:26.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e091b4640 1 --2- 172.21.15.88:0/1747333464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dec003aa0 0x7f6e040c6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e091b4640 1 -- 172.21.15.88:0/1747333464 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0414af00 con 0x7f6e04154dd0 2023-10-16T20:44:26.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e089b3640 1 --2- 172.21.15.88:0/1747333464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dec003aa0 0x7f6e040c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:44:26.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e091b4640 1 --2- 172.21.15.88:0/1747333464 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e04154dd0 0x7f6e04155250 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6e040c6c30 tx=0x7f6dfc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:26.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6df97fa640 1 -- 172.21.15.88:0/1747333464 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc00e040 con 0x7f6e04154dd0 2023-10-16T20:44:26.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6df97fa640 1 -- 172.21.15.88:0/1747333464 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dfc00f040 con 0x7f6e04154dd0 2023-10-16T20:44:26.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6df97fa640 1 -- 172.21.15.88:0/1747333464 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc015670 con 0x7f6e04154dd0 2023-10-16T20:44:26.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e0ac17640 1 -- 172.21.15.88:0/1747333464 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0414a660 con 0x7f6e04154dd0 2023-10-16T20:44:26.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e0ac17640 1 -- 172.21.15.88:0/1747333464 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0414ac20 con 0x7f6e04154dd0 2023-10-16T20:44:26.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6e0ac17640 1 -- 172.21.15.88:0/1747333464 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e04074700 con 0x7f6e04154dd0 2023-10-16T20:44:26.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.784+0000 7f6df97fa640 1 -- 172.21.15.88:0/1747333464 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6dfc0157d0 con 0x7f6e04154dd0 2023-10-16T20:44:26.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.788+0000 7f6df97fa640 1 --2- 172.21.15.88:0/1747333464 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6df00664d0 0x7f6df0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:26.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.788+0000 7f6df97fa640 1 -- 172.21.15.88:0/1747333464 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f6dfc084b10 con 0x7f6e04154dd0 2023-10-16T20:44:26.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.788+0000 7f6e089b3640 1 --2- 172.21.15.88:0/1747333464 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6df00664d0 0x7f6df0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:26.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.788+0000 7f6e089b3640 1 --2- 172.21.15.88:0/1747333464 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6df00664d0 0x7f6df0068990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6e000b2000 tx=0x7f6e00005040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:26.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.792+0000 7f6df97fa640 1 -- 172.21.15.88:0/1747333464 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6dfc051a40 con 0x7f6e04154dd0 2023-10-16T20:44:26.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.972+0000 7f6e0ac17640 1 -- 172.21.15.88:0/1747333464 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e040c6c60 con 0x7f6e04154dd0 2023-10-16T20:44:26.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.972+0000 7f6df97fa640 1 -- 172.21.15.88:0/1747333464 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5787 (secure 0 0 0) 0x7f6dfc055060 con 0x7f6e04154dd0 2023-10-16T20:44:26.975 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:26.975 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:25.369424+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:26.975 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 38 2023-10-16T20:44:26.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.972+0000 7f6e0ac17640 1 -- 172.21.15.88:0/1747333464 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6df00664d0 msgr2=0x7f6df0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:26.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.972+0000 7f6e0ac17640 1 --2- 172.21.15.88:0/1747333464 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6df00664d0 0x7f6df0068990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6e000b2000 tx=0x7f6e00005040 comp rx=0 tx=0).stop 2023-10-16T20:44:26.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.972+0000 7f6e0ac17640 1 -- 172.21.15.88:0/1747333464 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e04154dd0 msgr2=0x7f6e04155250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:26.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.972+0000 7f6e0ac17640 1 --2- 172.21.15.88:0/1747333464 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e04154dd0 0x7f6e04155250 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6e040c6c30 tx=0x7f6dfc00f580 comp rx=0 tx=0).stop 2023-10-16T20:44:26.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.972+0000 7f6e0ac17640 1 -- 172.21.15.88:0/1747333464 shutdown_connections 2023-10-16T20:44:26.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.972+0000 7f6e0ac17640 1 --2- 172.21.15.88:0/1747333464 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6df00664d0 0x7f6df0068990 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.972+0000 7f6e0ac17640 1 --2- 172.21.15.88:0/1747333464 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e04154dd0 0x7f6e04155250 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.972+0000 7f6e0ac17640 1 --2- 172.21.15.88:0/1747333464 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e04153bd0 0x7f6e041446f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.972+0000 7f6e0ac17640 1 --2- 172.21.15.88:0/1747333464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6dec003aa0 0x7f6e040c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:26.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.972+0000 7f6e0ac17640 1 -- 172.21.15.88:0/1747333464 >> 172.21.15.88:0/1747333464 conn(0x7f6e040cad50 msgr2=0x7f6e040c4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:26.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.972+0000 7f6e0ac17640 1 -- 172.21.15.88:0/1747333464 shutdown_connections 2023-10-16T20:44:26.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:26.972+0000 7f6e0ac17640 1 -- 172.21.15.88:0/1747333464 wait complete. 2023-10-16T20:44:26.996 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 38, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 38, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4275": { "addr": "172.21.15.88:6838/321173984", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 321173984, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 321173984, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4275, "incarnation": 14, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 41 }, "gid_4279": { "addr": "172.21.15.191:6833/2446850309", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 2446850309, "type": "v2" }, { "addr": "172.21.15.191:6833", "nonce": 2446850309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 40 }, "gid_4288": { "addr": "172.21.15.191:6835/3059699574", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3059699574, "type": "v2" }, { "addr": "172.21.15.191:6835", "nonce": 3059699574, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4889": { "addr": "172.21.15.191:6837/2468647900", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6836", "nonce": 2468647900, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2468647900, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4889, "incarnation": 30, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 21 }, "gid_5284": { "addr": "172.21.15.88:6839/2172973775", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6837", "nonce": 2172973775, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2172973775, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5284, "incarnation": 38, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:44:25.369424+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 4279, "mds_2": 4275, "mds_3": 5284, "mds_4": 4889 } } } ], "standbys": [] } 2023-10-16T20:44:28.997 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:29.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.140+0000 7fafab5b8640 1 Processor -- start 2023-10-16T20:44:29.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.140+0000 7fafab5b8640 1 -- start start 2023-10-16T20:44:29.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.144+0000 7fafab5b8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafa4150280 0x7fafa40c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:29.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.144+0000 7fafab5b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fafa4150660 0x7fafa40c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:29.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.144+0000 7fafab5b8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa40c5c60 con 0x7fafa40c6bc0 2023-10-16T20:44:29.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.144+0000 7fafab5b8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa40c5370 con 0x7fafa4150660 2023-10-16T20:44:29.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.144+0000 7fafab5b8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa4156d10 con 0x7fafa4150280 2023-10-16T20:44:29.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.144+0000 7fafa8b53640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fafa4150660 0x7fafa40c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:29.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.144+0000 7fafa8b53640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fafa4150660 0x7fafa40c3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40796/0 (socket says 172.21.15.88:40796) 2023-10-16T20:44:29.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.144+0000 7fafa8b53640 1 -- 172.21.15.88:0/1039872592 learned_addr learned my addr 172.21.15.88:0/1039872592 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:29.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.144+0000 7fafa9354640 1 --2- 172.21.15.88:0/1039872592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafa4150280 0x7fafa40c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:29.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.144+0000 7fafa8b53640 1 -- 172.21.15.88:0/1039872592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafa4150280 msgr2=0x7fafa40c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:29.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.144+0000 7fafa8b53640 1 --2- 172.21.15.88:0/1039872592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafa4150280 0x7fafa40c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:29.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.144+0000 7fafa8b53640 1 -- 172.21.15.88:0/1039872592 >> v1:172.21.15.88:6789/0 conn(0x7fafa40c6bc0 legacy=0x7fafa406d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:29.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.144+0000 7fafa8b53640 1 -- 172.21.15.88:0/1039872592 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa40c5610 con 0x7fafa4150660 2023-10-16T20:44:29.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.148+0000 7fafa8b53640 1 --2- 172.21.15.88:0/1039872592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa4150660 0x7fafa40c3110 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7faf9400bb10 tx=0x7faf9401f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c4533e5358fdfe7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:29.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.148+0000 7faf9bfff640 1 -- 172.21.15.88:0/1039872592 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf940a7c40 con 0x7fafa4150660 2023-10-16T20:44:29.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.148+0000 7faf9bfff640 1 -- 172.21.15.88:0/1039872592 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf940a7da0 con 0x7fafa4150660 2023-10-16T20:44:29.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.148+0000 7faf9bfff640 1 -- 172.21.15.88:0/1039872592 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf940af660 con 0x7fafa4150660 2023-10-16T20:44:29.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.148+0000 7fafab5b8640 1 -- 172.21.15.88:0/1039872592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa4150660 msgr2=0x7fafa40c3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:29.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.148+0000 7fafab5b8640 1 --2- 172.21.15.88:0/1039872592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa4150660 0x7fafa40c3110 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7faf9400bb10 tx=0x7faf9401f5b0 comp rx=0 tx=0).stop 2023-10-16T20:44:29.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.148+0000 7fafab5b8640 1 -- 172.21.15.88:0/1039872592 shutdown_connections 2023-10-16T20:44:29.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.148+0000 7fafab5b8640 1 --2- 172.21.15.88:0/1039872592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa4150660 0x7fafa40c3110 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:29.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.148+0000 7fafab5b8640 1 --2- 172.21.15.88:0/1039872592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafa4150280 0x7fafa40c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:29.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.148+0000 7fafab5b8640 1 -- 172.21.15.88:0/1039872592 >> 172.21.15.88:0/1039872592 conn(0x7fafa40cad50 msgr2=0x7fafa40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:29.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.148+0000 7fafab5b8640 1 -- 172.21.15.88:0/1039872592 shutdown_connections 2023-10-16T20:44:29.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.148+0000 7fafab5b8640 1 -- 172.21.15.88:0/1039872592 wait complete. 2023-10-16T20:44:29.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.148+0000 7fafab5b8640 1 Processor -- start 2023-10-16T20:44:29.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.148+0000 7fafab5b8640 1 -- start start 2023-10-16T20:44:29.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafab5b8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafa40c6bc0 0x7fafa4149630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:29.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafab5b8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafa4150280 0x7fafa406f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:29.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafab5b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa4150660 0x7fafa41f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:29.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafab5b8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa41f2ab0 con 0x7fafa4150660 2023-10-16T20:44:29.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafab5b8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa40c9d00 con 0x7fafa4150280 2023-10-16T20:44:29.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafab5b8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa41f2ed0 con 0x7fafa40c6bc0 2023-10-16T20:44:29.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafa9b55640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa4150660 0x7fafa41f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:29.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafa9b55640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa4150660 0x7fafa41f54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40812/0 (socket says 172.21.15.88:40812) 2023-10-16T20:44:29.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafa9b55640 1 -- 172.21.15.88:0/877748924 learned_addr learned my addr 172.21.15.88:0/877748924 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:29.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafa8b53640 1 --2- 172.21.15.88:0/877748924 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafa4150280 0x7fafa406f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:29.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafa9354640 1 --2- 172.21.15.88:0/877748924 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafa40c6bc0 0x7fafa4149630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:29.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafa9b55640 1 -- 172.21.15.88:0/877748924 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafa40c6bc0 msgr2=0x7fafa4149630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:29.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafa9b55640 1 --2- 172.21.15.88:0/877748924 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafa40c6bc0 0x7fafa4149630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:29.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafa9b55640 1 -- 172.21.15.88:0/877748924 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafa4150280 msgr2=0x7fafa406f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:29.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafa9b55640 1 --2- 172.21.15.88:0/877748924 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafa4150280 0x7fafa406f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:29.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafa9b55640 1 -- 172.21.15.88:0/877748924 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf9400b770 con 0x7fafa4150660 2023-10-16T20:44:29.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafa9b55640 1 --2- 172.21.15.88:0/877748924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa4150660 0x7fafa41f54e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7faf9c00c7a0 tx=0x7faf9c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:29.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7faf99ffb640 1 -- 172.21.15.88:0/877748924 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9c003b80 con 0x7fafa4150660 2023-10-16T20:44:29.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7faf99ffb640 1 -- 172.21.15.88:0/877748924 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9c003ce0 con 0x7fafa4150660 2023-10-16T20:44:29.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7faf99ffb640 1 -- 172.21.15.88:0/877748924 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9c00aa90 con 0x7fafa4150660 2023-10-16T20:44:29.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafab5b8640 1 -- 172.21.15.88:0/877748924 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa40c8cd0 con 0x7fafa4150660 2023-10-16T20:44:29.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafab5b8640 1 -- 172.21.15.88:0/877748924 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa41f23f0 con 0x7fafa4150660 2023-10-16T20:44:29.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7fafab5b8640 1 -- 172.21.15.88:0/877748924 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa41f2bf0 con 0x7fafa4150660 2023-10-16T20:44:29.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.152+0000 7faf99ffb640 1 -- 172.21.15.88:0/877748924 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faf9c00abf0 con 0x7fafa4150660 2023-10-16T20:44:29.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.156+0000 7faf99ffb640 1 --2- 172.21.15.88:0/877748924 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faf880664a0 0x7faf88068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:29.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.156+0000 7faf99ffb640 1 -- 172.21.15.88:0/877748924 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6921+0+0 (secure 0 0 0) 0x7faf9c084170 con 0x7fafa4150660 2023-10-16T20:44:29.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.156+0000 7fafa9354640 1 --2- 172.21.15.88:0/877748924 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faf880664a0 0x7faf88068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:29.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.156+0000 7fafa9354640 1 --2- 172.21.15.88:0/877748924 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faf880664a0 0x7faf88068960 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fafa0007a60 tx=0x7fafa0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:29.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.160+0000 7faf99ffb640 1 -- 172.21.15.88:0/877748924 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fafa41f2bf0 con 0x7fafa4150660 2023-10-16T20:44:29.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.340+0000 7fafab5b8640 1 -- 172.21.15.88:0/877748924 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafa41f2bf0 con 0x7fafa4150660 2023-10-16T20:44:29.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.340+0000 7faf99ffb640 1 -- 172.21.15.88:0/877748924 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5787 (secure 0 0 0) 0x7fafa41f2bf0 con 0x7fafa4150660 2023-10-16T20:44:29.345 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:29.346 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:25.369424+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:29.346 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 38 2023-10-16T20:44:29.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.344+0000 7fafab5b8640 1 -- 172.21.15.88:0/877748924 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faf880664a0 msgr2=0x7faf88068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:29.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.344+0000 7fafab5b8640 1 --2- 172.21.15.88:0/877748924 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faf880664a0 0x7faf88068960 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fafa0007a60 tx=0x7fafa0008040 comp rx=0 tx=0).stop 2023-10-16T20:44:29.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.344+0000 7fafab5b8640 1 -- 172.21.15.88:0/877748924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa4150660 msgr2=0x7fafa41f54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:29.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.344+0000 7fafab5b8640 1 --2- 172.21.15.88:0/877748924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa4150660 0x7fafa41f54e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7faf9c00c7a0 tx=0x7faf9c00cc60 comp rx=0 tx=0).stop 2023-10-16T20:44:29.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.344+0000 7fafab5b8640 1 -- 172.21.15.88:0/877748924 shutdown_connections 2023-10-16T20:44:29.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.344+0000 7fafab5b8640 1 --2- 172.21.15.88:0/877748924 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faf880664a0 0x7faf88068960 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:29.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.344+0000 7fafab5b8640 1 --2- 172.21.15.88:0/877748924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa4150660 0x7fafa41f54e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:29.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.344+0000 7fafab5b8640 1 --2- 172.21.15.88:0/877748924 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafa4150280 0x7fafa406f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:29.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.344+0000 7fafab5b8640 1 --2- 172.21.15.88:0/877748924 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafa40c6bc0 0x7fafa4149630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:29.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.344+0000 7fafab5b8640 1 -- 172.21.15.88:0/877748924 >> 172.21.15.88:0/877748924 conn(0x7fafa40cad50 msgr2=0x7fafa40c4220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:29.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.344+0000 7fafab5b8640 1 -- 172.21.15.88:0/877748924 shutdown_connections 2023-10-16T20:44:29.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:29.344+0000 7fafab5b8640 1 -- 172.21.15.88:0/877748924 wait complete. 2023-10-16T20:44:29.366 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:44:31.367 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:31.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.512+0000 7f9f75f0f640 1 Processor -- start 2023-10-16T20:44:31.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.512+0000 7f9f75f0f640 1 -- start start 2023-10-16T20:44:31.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.516+0000 7f9f75f0f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9f70150280 0x7f9f700c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:31.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.516+0000 7f9f75f0f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f70150660 0x7f9f700c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:31.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.516+0000 7f9f75f0f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f700c5c60 con 0x7f9f700c6bc0 2023-10-16T20:44:31.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.516+0000 7f9f75f0f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f700c5370 con 0x7f9f70150280 2023-10-16T20:44:31.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.516+0000 7f9f75f0f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f70156d10 con 0x7f9f70150660 2023-10-16T20:44:31.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.516+0000 7f9f6f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9f70150280 0x7f9f700c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:31.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.516+0000 7f9f6f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9f70150280 0x7f9f700c60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40820/0 (socket says 172.21.15.88:40820) 2023-10-16T20:44:31.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.516+0000 7f9f6f7fe640 1 -- 172.21.15.88:0/4206444242 learned_addr learned my addr 172.21.15.88:0/4206444242 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:31.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.516+0000 7f9f6effd640 1 --2- 172.21.15.88:0/4206444242 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f70150660 0x7f9f700c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:31.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.516+0000 7f9f6f7fe640 1 -- 172.21.15.88:0/4206444242 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f70150660 msgr2=0x7f9f700c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:31.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.516+0000 7f9f6f7fe640 1 --2- 172.21.15.88:0/4206444242 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f70150660 0x7f9f700c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:31.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.516+0000 7f9f6f7fe640 1 -- 172.21.15.88:0/4206444242 >> v1:172.21.15.88:6789/0 conn(0x7f9f700c6bc0 legacy=0x7f9f7006d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:31.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.516+0000 7f9f6f7fe640 1 -- 172.21.15.88:0/4206444242 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f700c5610 con 0x7f9f70150280 2023-10-16T20:44:31.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.516+0000 7f9f6f7fe640 1 --2- 172.21.15.88:0/4206444242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f70150280 0x7f9f700c60f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009870 tx=0x7f9f6001f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a3cff7eac61ef68 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:31.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f6e7fc640 1 -- 172.21.15.88:0/4206444242 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f600a8680 con 0x7f9f70150280 2023-10-16T20:44:31.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f6e7fc640 1 -- 172.21.15.88:0/4206444242 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f600a87e0 con 0x7f9f70150280 2023-10-16T20:44:31.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f6e7fc640 1 -- 172.21.15.88:0/4206444242 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f600a8b70 con 0x7f9f70150280 2023-10-16T20:44:31.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f75f0f640 1 -- 172.21.15.88:0/4206444242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f70150280 msgr2=0x7f9f700c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:31.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f75f0f640 1 --2- 172.21.15.88:0/4206444242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f70150280 0x7f9f700c60f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009870 tx=0x7f9f6001f0f0 comp rx=0 tx=0).stop 2023-10-16T20:44:31.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f75f0f640 1 -- 172.21.15.88:0/4206444242 shutdown_connections 2023-10-16T20:44:31.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f75f0f640 1 --2- 172.21.15.88:0/4206444242 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f70150660 0x7f9f700c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:31.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f75f0f640 1 --2- 172.21.15.88:0/4206444242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f70150280 0x7f9f700c60f0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:31.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f75f0f640 1 -- 172.21.15.88:0/4206444242 >> 172.21.15.88:0/4206444242 conn(0x7f9f700cad50 msgr2=0x7f9f700cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:31.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f75f0f640 1 -- 172.21.15.88:0/4206444242 shutdown_connections 2023-10-16T20:44:31.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f75f0f640 1 -- 172.21.15.88:0/4206444242 wait complete. 2023-10-16T20:44:31.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f75f0f640 1 Processor -- start 2023-10-16T20:44:31.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f75f0f640 1 -- start start 2023-10-16T20:44:31.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f75f0f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f700c6bc0 0x7f9f700c31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:31.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f75f0f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f70150280 0x7f9f701f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:31.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f75f0f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f70150660 0x7f9f701f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:31.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f75f0f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f701f28a0 con 0x7f9f70150280 2023-10-16T20:44:31.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f75f0f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f70070850 con 0x7f9f70150660 2023-10-16T20:44:31.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f75f0f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f70071f40 con 0x7f9f700c6bc0 2023-10-16T20:44:31.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f6effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f70150280 0x7f9f701f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:31.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f6effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f70150280 0x7f9f701f32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40826/0 (socket says 172.21.15.88:40826) 2023-10-16T20:44:31.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f6effd640 1 -- 172.21.15.88:0/3232040968 learned_addr learned my addr 172.21.15.88:0/3232040968 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:31.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.520+0000 7f9f6f7fe640 1 --2- 172.21.15.88:0/3232040968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f700c6bc0 0x7f9f700c31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:31.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.524+0000 7f9f6ffff640 1 --2- 172.21.15.88:0/3232040968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f70150660 0x7f9f701f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:31.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.524+0000 7f9f6effd640 1 -- 172.21.15.88:0/3232040968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f700c6bc0 msgr2=0x7f9f700c31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:31.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.524+0000 7f9f6effd640 1 --2- 172.21.15.88:0/3232040968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f700c6bc0 0x7f9f700c31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:31.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.524+0000 7f9f6effd640 1 -- 172.21.15.88:0/3232040968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f70150660 msgr2=0x7f9f701f37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:31.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.524+0000 7f9f6effd640 1 --2- 172.21.15.88:0/3232040968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f70150660 0x7f9f701f37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:31.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.524+0000 7f9f6effd640 1 -- 172.21.15.88:0/3232040968 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f600094d0 con 0x7f9f70150280 2023-10-16T20:44:31.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.524+0000 7f9f6ffff640 1 --2- 172.21.15.88:0/3232040968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f70150660 0x7f9f701f37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:44:31.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.524+0000 7f9f6effd640 1 --2- 172.21.15.88:0/3232040968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f70150280 0x7f9f701f32b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9f6400d620 tx=0x7f9f6400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:31.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.524+0000 7f9f57fff640 1 -- 172.21.15.88:0/3232040968 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f640125f0 con 0x7f9f70150280 2023-10-16T20:44:31.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.524+0000 7f9f57fff640 1 -- 172.21.15.88:0/3232040968 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f64012750 con 0x7f9f70150280 2023-10-16T20:44:31.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.524+0000 7f9f57fff640 1 -- 172.21.15.88:0/3232040968 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f64012ae0 con 0x7f9f70150280 2023-10-16T20:44:31.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.524+0000 7f9f75f0f640 1 -- 172.21.15.88:0/3232040968 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f70072220 con 0x7f9f70150280 2023-10-16T20:44:31.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.524+0000 7f9f75f0f640 1 -- 172.21.15.88:0/3232040968 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f701f2ab0 con 0x7f9f70150280 2023-10-16T20:44:31.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.524+0000 7f9f75f0f640 1 -- 172.21.15.88:0/3232040968 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f70074700 con 0x7f9f70150280 2023-10-16T20:44:31.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.524+0000 7f9f57fff640 1 -- 172.21.15.88:0/3232040968 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9f64012c40 con 0x7f9f70150280 2023-10-16T20:44:31.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.524+0000 7f9f57fff640 1 --2- 172.21.15.88:0/3232040968 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9f5c0664a0 0x7f9f5c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:31.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.528+0000 7f9f57fff640 1 -- 172.21.15.88:0/3232040968 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f9f640847c0 con 0x7f9f70150280 2023-10-16T20:44:31.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.528+0000 7f9f6f7fe640 1 --2- 172.21.15.88:0/3232040968 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9f5c0664a0 0x7f9f5c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:31.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.528+0000 7f9f6f7fe640 1 --2- 172.21.15.88:0/3232040968 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9f5c0664a0 0x7f9f5c068960 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9f600af000 tx=0x7f9f60004740 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:31.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.532+0000 7f9f57fff640 1 -- 172.21.15.88:0/3232040968 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9f64051780 con 0x7f9f70150280 2023-10-16T20:44:31.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.712+0000 7f9f75f0f640 1 -- 172.21.15.88:0/3232040968 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f70143110 con 0x7f9f70150280 2023-10-16T20:44:31.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.712+0000 7f9f57fff640 1 -- 172.21.15.88:0/3232040968 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5789 (secure 0 0 0) 0x7f9f64054da0 con 0x7f9f70150280 2023-10-16T20:44:31.716 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:31.716 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:30.443179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:resolve","state_seq":11,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:31.716 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 39 2023-10-16T20:44:31.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.712+0000 7f9f75f0f640 1 -- 172.21.15.88:0/3232040968 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9f5c0664a0 msgr2=0x7f9f5c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:31.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.712+0000 7f9f75f0f640 1 --2- 172.21.15.88:0/3232040968 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9f5c0664a0 0x7f9f5c068960 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9f600af000 tx=0x7f9f60004740 comp rx=0 tx=0).stop 2023-10-16T20:44:31.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.712+0000 7f9f75f0f640 1 -- 172.21.15.88:0/3232040968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f70150280 msgr2=0x7f9f701f32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:31.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.712+0000 7f9f75f0f640 1 --2- 172.21.15.88:0/3232040968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f70150280 0x7f9f701f32b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9f6400d620 tx=0x7f9f6400dae0 comp rx=0 tx=0).stop 2023-10-16T20:44:31.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.712+0000 7f9f75f0f640 1 -- 172.21.15.88:0/3232040968 shutdown_connections 2023-10-16T20:44:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.712+0000 7f9f75f0f640 1 --2- 172.21.15.88:0/3232040968 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9f5c0664a0 0x7f9f5c068960 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.712+0000 7f9f75f0f640 1 --2- 172.21.15.88:0/3232040968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9f70150660 0x7f9f701f37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.712+0000 7f9f75f0f640 1 --2- 172.21.15.88:0/3232040968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f70150280 0x7f9f701f32b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.712+0000 7f9f75f0f640 1 --2- 172.21.15.88:0/3232040968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9f700c6bc0 0x7f9f700c31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.712+0000 7f9f75f0f640 1 -- 172.21.15.88:0/3232040968 >> 172.21.15.88:0/3232040968 conn(0x7f9f700cad50 msgr2=0x7f9f700c4790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:31.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.716+0000 7f9f75f0f640 1 -- 172.21.15.88:0/3232040968 shutdown_connections 2023-10-16T20:44:31.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:31.716+0000 7f9f75f0f640 1 -- 172.21.15.88:0/3232040968 wait complete. 2023-10-16T20:44:31.737 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:44:33.738 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:33.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.884+0000 7f8ec7df8640 1 Processor -- start 2023-10-16T20:44:33.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.884+0000 7f8ec7df8640 1 -- start start 2023-10-16T20:44:33.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.884+0000 7f8ec7df8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8ec00c6760 0x7f8ec00c6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:33.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.884+0000 7f8ec7df8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec006d5c0 0x7f8ec006da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:33.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.884+0000 7f8ec7df8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec006df70 con 0x7f8ec00c72a0 2023-10-16T20:44:33.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.884+0000 7f8ec7df8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ec01563e0 con 0x7f8ec00c6760 2023-10-16T20:44:33.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.884+0000 7f8ec7df8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec00c7080 con 0x7f8ec006d5c0 2023-10-16T20:44:33.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.888+0000 7f8ec5b94640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8ec00c6760 0x7f8ec00c6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:33.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.888+0000 7f8ec5b94640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8ec00c6760 0x7f8ec00c6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40832/0 (socket says 172.21.15.88:40832) 2023-10-16T20:44:33.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.888+0000 7f8ec5b94640 1 -- 172.21.15.88:0/239818731 learned_addr learned my addr 172.21.15.88:0/239818731 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:33.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.888+0000 7f8ec5393640 1 --1- 172.21.15.88:0/239818731 >> v1:172.21.15.88:6789/0 conn(0x7f8ec00c72a0 0x7f8ec00c5a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55550/0 (socket says 172.21.15.88:55550) 2023-10-16T20:44:33.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.888+0000 7f8ec6395640 1 --2- 172.21.15.88:0/239818731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec006d5c0 0x7f8ec006da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:33.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.888+0000 7f8ec5b94640 1 -- 172.21.15.88:0/239818731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec006d5c0 msgr2=0x7f8ec006da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:33.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.888+0000 7f8ec5b94640 1 --2- 172.21.15.88:0/239818731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec006d5c0 0x7f8ec006da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:33.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.888+0000 7f8ec5b94640 1 -- 172.21.15.88:0/239818731 >> v1:172.21.15.88:6789/0 conn(0x7f8ec00c72a0 legacy=0x7f8ec00c5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:33.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.888+0000 7f8ec5b94640 1 -- 172.21.15.88:0/239818731 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec01554f0 con 0x7f8ec00c6760 2023-10-16T20:44:33.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.888+0000 7f8ec6395640 1 --2- 172.21.15.88:0/239818731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec006d5c0 0x7f8ec006da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:44:33.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.888+0000 7f8ec5b94640 1 --2- 172.21.15.88:0/239818731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ec00c6760 0x7f8ec00c6b40 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009f80 tx=0x7f8ebc01d510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d9cd44ed020e4d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:33.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.888+0000 7f8ec4b92640 1 -- 172.21.15.88:0/239818731 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc007ad0 con 0x7f8ec00c6760 2023-10-16T20:44:33.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.888+0000 7f8ec4b92640 1 -- 172.21.15.88:0/239818731 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc007c30 con 0x7f8ec00c6760 2023-10-16T20:44:33.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.888+0000 7f8ec4b92640 1 -- 172.21.15.88:0/239818731 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc0090d0 con 0x7f8ec00c6760 2023-10-16T20:44:33.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec7df8640 1 -- 172.21.15.88:0/239818731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ec00c6760 msgr2=0x7f8ec00c6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:33.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec7df8640 1 --2- 172.21.15.88:0/239818731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ec00c6760 0x7f8ec00c6b40 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009f80 tx=0x7f8ebc01d510 comp rx=0 tx=0).stop 2023-10-16T20:44:33.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec7df8640 1 -- 172.21.15.88:0/239818731 shutdown_connections 2023-10-16T20:44:33.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec7df8640 1 --2- 172.21.15.88:0/239818731 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec006d5c0 0x7f8ec006da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:33.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec7df8640 1 --2- 172.21.15.88:0/239818731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ec00c6760 0x7f8ec00c6b40 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:33.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec7df8640 1 -- 172.21.15.88:0/239818731 >> 172.21.15.88:0/239818731 conn(0x7f8ec00cacc0 msgr2=0x7f8ec00cb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:33.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec7df8640 1 -- 172.21.15.88:0/239818731 shutdown_connections 2023-10-16T20:44:33.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec7df8640 1 -- 172.21.15.88:0/239818731 wait complete. 2023-10-16T20:44:33.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec7df8640 1 Processor -- start 2023-10-16T20:44:33.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec7df8640 1 -- start start 2023-10-16T20:44:33.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec7df8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec006d5c0 0x7f8ec01ef640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:33.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec7df8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ec00c6760 0x7f8ec00c5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:33.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec7df8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ec00c72a0 0x7f8ec01f0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:33.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec7df8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec00aa830 con 0x7f8ec00c6760 2023-10-16T20:44:33.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec7df8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec00aacc0 con 0x7f8ec00c72a0 2023-10-16T20:44:33.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec7df8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec00a9680 con 0x7f8ec006d5c0 2023-10-16T20:44:33.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec5393640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ec00c6760 0x7f8ec00c5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:33.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec5b94640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec006d5c0 0x7f8ec01ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:33.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec5393640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ec00c6760 0x7f8ec00c5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40840/0 (socket says 172.21.15.88:40840) 2023-10-16T20:44:33.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec5393640 1 -- 172.21.15.88:0/4138597186 learned_addr learned my addr 172.21.15.88:0/4138597186 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:33.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.892+0000 7f8ec6395640 1 --2- 172.21.15.88:0/4138597186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ec00c72a0 0x7f8ec01f0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:33.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.896+0000 7f8ec5393640 1 -- 172.21.15.88:0/4138597186 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec006d5c0 msgr2=0x7f8ec01ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:33.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.896+0000 7f8ec5393640 1 --2- 172.21.15.88:0/4138597186 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec006d5c0 0x7f8ec01ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:33.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.896+0000 7f8ec5393640 1 -- 172.21.15.88:0/4138597186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ec00c72a0 msgr2=0x7f8ec01f0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:33.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.896+0000 7f8ec5393640 1 --2- 172.21.15.88:0/4138597186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ec00c72a0 0x7f8ec01f0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:33.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.896+0000 7f8ec5393640 1 -- 172.21.15.88:0/4138597186 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc009df0 con 0x7f8ec00c6760 2023-10-16T20:44:33.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.896+0000 7f8ec5393640 1 --2- 172.21.15.88:0/4138597186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ec00c6760 0x7f8ec00c5b50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000c490 tx=0x7f8eb000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:33.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.896+0000 7f8eb67fc640 1 -- 172.21.15.88:0/4138597186 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0007480 con 0x7f8ec00c6760 2023-10-16T20:44:33.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.896+0000 7f8eb67fc640 1 -- 172.21.15.88:0/4138597186 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb00075e0 con 0x7f8ec00c6760 2023-10-16T20:44:33.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.896+0000 7f8eb67fc640 1 -- 172.21.15.88:0/4138597186 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0004a10 con 0x7f8ec00c6760 2023-10-16T20:44:33.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.896+0000 7f8ec7df8640 1 -- 172.21.15.88:0/4138597186 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec00a9940 con 0x7f8ec00c6760 2023-10-16T20:44:33.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.896+0000 7f8ec7df8640 1 -- 172.21.15.88:0/4138597186 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec00a9f00 con 0x7f8ec00c6760 2023-10-16T20:44:33.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.896+0000 7f8ec7df8640 1 -- 172.21.15.88:0/4138597186 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec0074700 con 0x7f8ec00c6760 2023-10-16T20:44:33.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.896+0000 7f8eb67fc640 1 -- 172.21.15.88:0/4138597186 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8eb0004b70 con 0x7f8ec00c6760 2023-10-16T20:44:33.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.896+0000 7f8eb67fc640 1 --2- 172.21.15.88:0/4138597186 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ea40664a0 0x7f8ea4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:33.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.896+0000 7f8eb67fc640 1 -- 172.21.15.88:0/4138597186 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f8eb0083490 con 0x7f8ec00c6760 2023-10-16T20:44:33.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.896+0000 7f8ec5b94640 1 --2- 172.21.15.88:0/4138597186 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ea40664a0 0x7f8ea4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:33.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.900+0000 7f8ec5b94640 1 --2- 172.21.15.88:0/4138597186 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ea40664a0 0x7f8ea4068960 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc0af000 tx=0x7f8ebc0b4040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:33.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:33.900+0000 7f8eb67fc640 1 -- 172.21.15.88:0/4138597186 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8eb00504f0 con 0x7f8ec00c6760 2023-10-16T20:44:34.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:34.084+0000 7f8ec7df8640 1 -- 172.21.15.88:0/4138597186 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ec00c6540 con 0x7f8ec00c6760 2023-10-16T20:44:34.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:34.084+0000 7f8eb67fc640 1 -- 172.21.15.88:0/4138597186 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5789 (secure 0 0 0) 0x7f8eb0053b10 con 0x7f8ec00c6760 2023-10-16T20:44:34.089 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:34.090 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:30.443179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:resolve","state_seq":11,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:34.091 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 39 2023-10-16T20:44:34.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:34.088+0000 7f8ec7df8640 1 -- 172.21.15.88:0/4138597186 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ea40664a0 msgr2=0x7f8ea4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:34.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:34.088+0000 7f8ec7df8640 1 --2- 172.21.15.88:0/4138597186 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ea40664a0 0x7f8ea4068960 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc0af000 tx=0x7f8ebc0b4040 comp rx=0 tx=0).stop 2023-10-16T20:44:34.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:34.088+0000 7f8ec7df8640 1 -- 172.21.15.88:0/4138597186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ec00c6760 msgr2=0x7f8ec00c5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:34.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:34.088+0000 7f8ec7df8640 1 --2- 172.21.15.88:0/4138597186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ec00c6760 0x7f8ec00c5b50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000c490 tx=0x7f8eb000c950 comp rx=0 tx=0).stop 2023-10-16T20:44:34.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:34.088+0000 7f8ec7df8640 1 -- 172.21.15.88:0/4138597186 shutdown_connections 2023-10-16T20:44:34.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:34.088+0000 7f8ec7df8640 1 --2- 172.21.15.88:0/4138597186 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8ea40664a0 0x7f8ea4068960 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:34.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:34.088+0000 7f8ec7df8640 1 --2- 172.21.15.88:0/4138597186 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ec00c72a0 0x7f8ec01f0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:34.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:34.088+0000 7f8ec7df8640 1 --2- 172.21.15.88:0/4138597186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ec00c6760 0x7f8ec00c5b50 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:34.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:34.088+0000 7f8ec7df8640 1 --2- 172.21.15.88:0/4138597186 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ec006d5c0 0x7f8ec01ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:34.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:34.088+0000 7f8ec7df8640 1 -- 172.21.15.88:0/4138597186 >> 172.21.15.88:0/4138597186 conn(0x7f8ec00cacc0 msgr2=0x7f8ec00c8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:34.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:34.088+0000 7f8ec7df8640 1 -- 172.21.15.88:0/4138597186 shutdown_connections 2023-10-16T20:44:34.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:34.088+0000 7f8ec7df8640 1 -- 172.21.15.88:0/4138597186 wait complete. 2023-10-16T20:44:34.109 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:44:36.110 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:36.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.248+0000 7f74caf65640 1 Processor -- start 2023-10-16T20:44:36.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.248+0000 7f74caf65640 1 -- start start 2023-10-16T20:44:36.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.248+0000 7f74caf65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74c40c5200 0x7f74c40b48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:36.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.252+0000 7f74caf65640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c40c6790 con 0x7f74c406b280 2023-10-16T20:44:36.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.252+0000 7f74caf65640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c415d530 con 0x7f74c40c61d0 2023-10-16T20:44:36.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.252+0000 7f74caf65640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c40c6620 con 0x7f74c40c5200 2023-10-16T20:44:36.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.252+0000 7f74bbfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f74c406b280 0x7f74c415ca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55566/0 (socket says 172.21.15.88:55566) 2023-10-16T20:44:36.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.252+0000 7f74bbfff640 1 -- 172.21.15.88:0/3302269089 learned_addr learned my addr 172.21.15.88:0/3302269089 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:44:36.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.252+0000 7f74c8d01640 1 --2- 172.21.15.88:0/3302269089 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74c40c5200 0x7f74c40b48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:36.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.252+0000 7f74bb7fe640 1 -- 172.21.15.88:0/3302269089 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811647382 0 0) 0x7f74c40c6790 con 0x7f74c406b280 2023-10-16T20:44:36.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.252+0000 7f74bb7fe640 1 -- 172.21.15.88:0/3302269089 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74ac003530 con 0x7f74c406b280 2023-10-16T20:44:36.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.252+0000 7f74bb7fe640 1 -- 172.21.15.88:0/3302269089 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336737946 0 0) 0x7f74c415d530 con 0x7f74c40c61d0 2023-10-16T20:44:36.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.252+0000 7f74bb7fe640 1 -- 172.21.15.88:0/3302269089 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c40c6790 con 0x7f74c40c61d0 2023-10-16T20:44:36.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.252+0000 7f74c8d01640 1 -- 172.21.15.88:0/3302269089 >> v1:172.21.15.191:6789/0 conn(0x7f74c40c61d0 legacy=0x7f74c40c6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:36.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.252+0000 7f74c8d01640 1 -- 172.21.15.88:0/3302269089 >> v1:172.21.15.88:6789/0 conn(0x7f74c406b280 legacy=0x7f74c415ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:36.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.252+0000 7f74bb7fe640 1 -- 172.21.15.88:0/3302269089 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 591906611 0 0) 0x7f74ac003530 con 0x7f74c406b280 2023-10-16T20:44:36.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.252+0000 7f74c8d01640 1 -- 172.21.15.88:0/3302269089 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c40c55e0 con 0x7f74c40c5200 2023-10-16T20:44:36.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.252+0000 7f74c8d01640 1 --2- 172.21.15.88:0/3302269089 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74c40c5200 0x7f74c40b48d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f74c00097f0 tx=0x7f74c001c350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e806bd1004b596c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:36.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74bb7fe640 1 -- 172.21.15.88:0/3302269089 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b4001680 con 0x7f74c40c5200 2023-10-16T20:44:36.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74bb7fe640 1 -- 172.21.15.88:0/3302269089 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c00a7050 con 0x7f74c40c5200 2023-10-16T20:44:36.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74bb7fe640 1 -- 172.21.15.88:0/3302269089 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c00ab620 con 0x7f74c40c5200 2023-10-16T20:44:36.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74caf65640 1 -- 172.21.15.88:0/3302269089 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74c40c5200 msgr2=0x7f74c40b48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:36.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74caf65640 1 --2- 172.21.15.88:0/3302269089 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74c40c5200 0x7f74c40b48d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f74c00097f0 tx=0x7f74c001c350 comp rx=0 tx=0).stop 2023-10-16T20:44:36.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74caf65640 1 -- 172.21.15.88:0/3302269089 shutdown_connections 2023-10-16T20:44:36.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74caf65640 1 --2- 172.21.15.88:0/3302269089 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74c40c5200 0x7f74c40b48d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:36.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74caf65640 1 -- 172.21.15.88:0/3302269089 >> 172.21.15.88:0/3302269089 conn(0x7f74c40cb730 msgr2=0x7f74c40cbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:36.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74caf65640 1 -- 172.21.15.88:0/3302269089 shutdown_connections 2023-10-16T20:44:36.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74caf65640 1 -- 172.21.15.88:0/3302269089 wait complete. 2023-10-16T20:44:36.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74caf65640 1 Processor -- start 2023-10-16T20:44:36.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74caf65640 1 -- start start 2023-10-16T20:44:36.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74caf65640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74c406b280 0x7f74c41f2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:36.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74caf65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74c40c5200 0x7f74c415bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:36.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74caf65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74c40c61d0 0x7f74c4154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:36.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74caf65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c41f1b70 con 0x7f74c40c61d0 2023-10-16T20:44:36.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74caf65640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c4156ea0 con 0x7f74c406b280 2023-10-16T20:44:36.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74caf65640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c41f1d80 con 0x7f74c40c5200 2023-10-16T20:44:36.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74c9502640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74c40c61d0 0x7f74c4154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:36.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74c9502640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74c40c61d0 0x7f74c4154410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40848/0 (socket says 172.21.15.88:40848) 2023-10-16T20:44:36.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.256+0000 7f74c9502640 1 -- 172.21.15.88:0/1506949172 learned_addr learned my addr 172.21.15.88:0/1506949172 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:36.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74c8d01640 1 --2- 172.21.15.88:0/1506949172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74c406b280 0x7f74c41f2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:36.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74bbfff640 1 --2- 172.21.15.88:0/1506949172 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74c40c5200 0x7f74c415bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:36.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74c9502640 1 -- 172.21.15.88:0/1506949172 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74c40c5200 msgr2=0x7f74c415bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:36.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74c9502640 1 --2- 172.21.15.88:0/1506949172 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74c40c5200 0x7f74c415bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:36.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74c9502640 1 -- 172.21.15.88:0/1506949172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74c406b280 msgr2=0x7f74c41f2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:36.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74c9502640 1 --2- 172.21.15.88:0/1506949172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74c406b280 0x7f74c41f2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:36.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74c9502640 1 -- 172.21.15.88:0/1506949172 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c0009420 con 0x7f74c40c61d0 2023-10-16T20:44:36.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74c8d01640 1 --2- 172.21.15.88:0/1506949172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74c406b280 0x7f74c41f2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:44:36.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74c9502640 1 --2- 172.21.15.88:0/1506949172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74c40c61d0 0x7f74c4154410 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f74bc00c760 tx=0x7f74bc00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:36.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74b97fa640 1 -- 172.21.15.88:0/1506949172 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc003b80 con 0x7f74c40c61d0 2023-10-16T20:44:36.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74b97fa640 1 -- 172.21.15.88:0/1506949172 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74bc003ce0 con 0x7f74c40c61d0 2023-10-16T20:44:36.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74b97fa640 1 -- 172.21.15.88:0/1506949172 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc00aa90 con 0x7f74c40c61d0 2023-10-16T20:44:36.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74caf65640 1 -- 172.21.15.88:0/1506949172 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c41568e0 con 0x7f74c40c61d0 2023-10-16T20:44:36.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74caf65640 1 -- 172.21.15.88:0/1506949172 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c41f2020 con 0x7f74c40c61d0 2023-10-16T20:44:36.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74caf65640 1 -- 172.21.15.88:0/1506949172 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c4074700 con 0x7f74c40c61d0 2023-10-16T20:44:36.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74b97fa640 1 -- 172.21.15.88:0/1506949172 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f74bc00abf0 con 0x7f74c40c61d0 2023-10-16T20:44:36.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74b97fa640 1 --2- 172.21.15.88:0/1506949172 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74b00664d0 0x7f74b0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:36.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74b97fa640 1 -- 172.21.15.88:0/1506949172 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f74bc0841c0 con 0x7f74c40c61d0 2023-10-16T20:44:36.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.260+0000 7f74c8d01640 1 --2- 172.21.15.88:0/1506949172 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74b00664d0 0x7f74b0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:36.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.264+0000 7f74c8d01640 1 --2- 172.21.15.88:0/1506949172 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74b00664d0 0x7f74b0068990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f74c00ad000 tx=0x7f74c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:36.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.264+0000 7f74b97fa640 1 -- 172.21.15.88:0/1506949172 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f74bc0510e0 con 0x7f74c40c61d0 2023-10-16T20:44:36.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.452+0000 7f74caf65640 1 -- 172.21.15.88:0/1506949172 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74c40c6db0 con 0x7f74c40c61d0 2023-10-16T20:44:36.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.452+0000 7f74b97fa640 1 -- 172.21.15.88:0/1506949172 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5791 (secure 0 0 0) 0x7f74bc054700 con 0x7f74c40c61d0 2023-10-16T20:44:36.454 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:36.455 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:36.391560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:36.455 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 40 2023-10-16T20:44:36.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.452+0000 7f74caf65640 1 -- 172.21.15.88:0/1506949172 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74b00664d0 msgr2=0x7f74b0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:36.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.452+0000 7f74caf65640 1 --2- 172.21.15.88:0/1506949172 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74b00664d0 0x7f74b0068990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f74c00ad000 tx=0x7f74c0008040 comp rx=0 tx=0).stop 2023-10-16T20:44:36.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.452+0000 7f74caf65640 1 -- 172.21.15.88:0/1506949172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74c40c61d0 msgr2=0x7f74c4154410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:36.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.452+0000 7f74caf65640 1 --2- 172.21.15.88:0/1506949172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74c40c61d0 0x7f74c4154410 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f74bc00c760 tx=0x7f74bc00cc20 comp rx=0 tx=0).stop 2023-10-16T20:44:36.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.452+0000 7f74caf65640 1 -- 172.21.15.88:0/1506949172 shutdown_connections 2023-10-16T20:44:36.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.452+0000 7f74caf65640 1 --2- 172.21.15.88:0/1506949172 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f74b00664d0 0x7f74b0068990 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:36.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.452+0000 7f74caf65640 1 --2- 172.21.15.88:0/1506949172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74c40c61d0 0x7f74c4154410 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:36.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.452+0000 7f74caf65640 1 --2- 172.21.15.88:0/1506949172 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f74c40c5200 0x7f74c415bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:36.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.452+0000 7f74caf65640 1 --2- 172.21.15.88:0/1506949172 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74c406b280 0x7f74c41f2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:36.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.452+0000 7f74caf65640 1 -- 172.21.15.88:0/1506949172 >> 172.21.15.88:0/1506949172 conn(0x7f74c40cb730 msgr2=0x7f74c40c40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:36.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.452+0000 7f74caf65640 1 -- 172.21.15.88:0/1506949172 shutdown_connections 2023-10-16T20:44:36.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:36.452+0000 7f74caf65640 1 -- 172.21.15.88:0/1506949172 wait complete. 2023-10-16T20:44:36.475 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:44:38.476 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:38.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.624+0000 7f605fcc9640 1 Processor -- start 2023-10-16T20:44:38.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.624+0000 7f605fcc9640 1 -- start start 2023-10-16T20:44:38.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.624+0000 7f605fcc9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f605814fc50 0x7f6058071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:38.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.624+0000 7f605fcc9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6058069e90 0x7f605806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:38.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.624+0000 7f605fcc9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60580cc0a0 con 0x7f605806ffe0 2023-10-16T20:44:38.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.624+0000 7f605fcc9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6058066190 con 0x7f6058069e90 2023-10-16T20:44:38.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.624+0000 7f605fcc9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6058156980 con 0x7f605814fc50 2023-10-16T20:44:38.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.624+0000 7f605e266640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6058069e90 0x7f605806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:38.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.624+0000 7f605e266640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6058069e90 0x7f605806a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40862/0 (socket says 172.21.15.88:40862) 2023-10-16T20:44:38.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.624+0000 7f605e266640 1 -- 172.21.15.88:0/617787431 learned_addr learned my addr 172.21.15.88:0/617787431 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:38.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.624+0000 7f605d264640 1 --2- 172.21.15.88:0/617787431 >> v2:172.21.15.191:3300/0 conn(0x7f605814fc50 0x7f6058071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:38.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.624+0000 7f605e266640 1 -- 172.21.15.88:0/617787431 >> v2:172.21.15.191:3300/0 conn(0x7f605814fc50 msgr2=0x7f6058071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:38.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.624+0000 7f605e266640 1 --2- 172.21.15.88:0/617787431 >> v2:172.21.15.191:3300/0 conn(0x7f605814fc50 0x7f6058071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:38.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.624+0000 7f605e266640 1 -- 172.21.15.88:0/617787431 >> v1:172.21.15.191:6789/0 conn(0x7f605806ffe0 legacy=0x7f6058065a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:38.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.624+0000 7f605e266640 1 -- 172.21.15.88:0/617787431 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60580703c0 con 0x7f6058069e90 2023-10-16T20:44:38.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.628+0000 7f605e266640 1 --2- 172.21.15.88:0/617787431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6058069e90 0x7f605806a2e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f60580cbd00 tx=0x7f605001f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e808c78990cf41b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:38.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.628+0000 7f605ca63640 1 -- 172.21.15.88:0/617787431 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60500af400 con 0x7f6058069e90 2023-10-16T20:44:38.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.628+0000 7f605ca63640 1 -- 172.21.15.88:0/617787431 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60500ab050 con 0x7f6058069e90 2023-10-16T20:44:38.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.628+0000 7f605ca63640 1 -- 172.21.15.88:0/617787431 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60500af780 con 0x7f6058069e90 2023-10-16T20:44:38.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.628+0000 7f605fcc9640 1 -- 172.21.15.88:0/617787431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6058069e90 msgr2=0x7f605806a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:38.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.628+0000 7f605fcc9640 1 --2- 172.21.15.88:0/617787431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6058069e90 0x7f605806a2e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f60580cbd00 tx=0x7f605001f180 comp rx=0 tx=0).stop 2023-10-16T20:44:38.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.628+0000 7f605fcc9640 1 -- 172.21.15.88:0/617787431 shutdown_connections 2023-10-16T20:44:38.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.628+0000 7f605fcc9640 1 --2- 172.21.15.88:0/617787431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6058069e90 0x7f605806a2e0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:38.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.628+0000 7f605fcc9640 1 --2- 172.21.15.88:0/617787431 >> v2:172.21.15.191:3300/0 conn(0x7f605814fc50 0x7f6058071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:38.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.628+0000 7f605fcc9640 1 -- 172.21.15.88:0/617787431 >> 172.21.15.88:0/617787431 conn(0x7f605806e0f0 msgr2=0x7f60580cb180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:38.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.628+0000 7f605fcc9640 1 -- 172.21.15.88:0/617787431 shutdown_connections 2023-10-16T20:44:38.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.628+0000 7f605fcc9640 1 -- 172.21.15.88:0/617787431 wait complete. 2023-10-16T20:44:38.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605fcc9640 1 Processor -- start 2023-10-16T20:44:38.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605fcc9640 1 -- start start 2023-10-16T20:44:38.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605fcc9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6058069e90 0x7f6058158980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:38.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605fcc9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f605806ffe0 0x7f605814b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:38.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605fcc9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f605814fc50 0x7f6058156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:38.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605fcc9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6058155180 con 0x7f605814fc50 2023-10-16T20:44:38.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605fcc9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60581eebf0 con 0x7f605806ffe0 2023-10-16T20:44:38.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605fcc9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60580c6d30 con 0x7f6058069e90 2023-10-16T20:44:38.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605e266640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f605814fc50 0x7f6058156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:38.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605e266640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f605814fc50 0x7f6058156de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40878/0 (socket says 172.21.15.88:40878) 2023-10-16T20:44:38.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605e266640 1 -- 172.21.15.88:0/1838883026 learned_addr learned my addr 172.21.15.88:0/1838883026 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:38.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605da65640 1 --2- 172.21.15.88:0/1838883026 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6058069e90 0x7f6058158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:38.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605d264640 1 --2- 172.21.15.88:0/1838883026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f605806ffe0 0x7f605814b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:38.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605e266640 1 -- 172.21.15.88:0/1838883026 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6058069e90 msgr2=0x7f6058158980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:38.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605e266640 1 --2- 172.21.15.88:0/1838883026 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6058069e90 0x7f6058158980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:38.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605e266640 1 -- 172.21.15.88:0/1838883026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f605806ffe0 msgr2=0x7f605814b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:38.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605e266640 1 --2- 172.21.15.88:0/1838883026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f605806ffe0 0x7f605814b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:38.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605e266640 1 -- 172.21.15.88:0/1838883026 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6050009b30 con 0x7f605814fc50 2023-10-16T20:44:38.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605e266640 1 --2- 172.21.15.88:0/1838883026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f605814fc50 0x7f6058156de0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f605001cdb0 tx=0x7f605001bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:38.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f604e7fc640 1 -- 172.21.15.88:0/1838883026 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60500afe00 con 0x7f605814fc50 2023-10-16T20:44:38.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f604e7fc640 1 -- 172.21.15.88:0/1838883026 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6050009510 con 0x7f605814fc50 2023-10-16T20:44:38.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605fcc9640 1 -- 172.21.15.88:0/1838883026 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60581eeeb0 con 0x7f605814fc50 2023-10-16T20:44:38.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f604e7fc640 1 -- 172.21.15.88:0/1838883026 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60500ba660 con 0x7f605814fc50 2023-10-16T20:44:38.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605fcc9640 1 -- 172.21.15.88:0/1838883026 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60581ef470 con 0x7f605814fc50 2023-10-16T20:44:38.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.632+0000 7f605fcc9640 1 -- 172.21.15.88:0/1838883026 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6058074700 con 0x7f605814fc50 2023-10-16T20:44:38.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.636+0000 7f604e7fc640 1 -- 172.21.15.88:0/1838883026 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f60500b6040 con 0x7f605814fc50 2023-10-16T20:44:38.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.636+0000 7f604e7fc640 1 --2- 172.21.15.88:0/1838883026 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f603c0664a0 0x7f603c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:38.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.636+0000 7f604e7fc640 1 -- 172.21.15.88:0/1838883026 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f60501224a0 con 0x7f605814fc50 2023-10-16T20:44:38.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.636+0000 7f605da65640 1 --2- 172.21.15.88:0/1838883026 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f603c0664a0 0x7f603c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:38.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.636+0000 7f605da65640 1 --2- 172.21.15.88:0/1838883026 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f603c0664a0 0x7f603c068960 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6054007fd0 tx=0x7f6054007e40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:38.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.640+0000 7f604e7fc640 1 -- 172.21.15.88:0/1838883026 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f60500ef4f0 con 0x7f605814fc50 2023-10-16T20:44:38.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.824+0000 7f605fcc9640 1 -- 172.21.15.88:0/1838883026 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6058070630 con 0x7f605814fc50 2023-10-16T20:44:38.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.824+0000 7f604e7fc640 1 -- 172.21.15.88:0/1838883026 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5791 (secure 0 0 0) 0x7f60500f2b10 con 0x7f605814fc50 2023-10-16T20:44:38.828 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:38.829 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:36.391560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:38.829 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 40 2023-10-16T20:44:38.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.828+0000 7f605fcc9640 1 -- 172.21.15.88:0/1838883026 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f603c0664a0 msgr2=0x7f603c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:38.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.828+0000 7f605fcc9640 1 --2- 172.21.15.88:0/1838883026 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f603c0664a0 0x7f603c068960 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6054007fd0 tx=0x7f6054007e40 comp rx=0 tx=0).stop 2023-10-16T20:44:38.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.828+0000 7f605fcc9640 1 -- 172.21.15.88:0/1838883026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f605814fc50 msgr2=0x7f6058156de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:38.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.828+0000 7f605fcc9640 1 --2- 172.21.15.88:0/1838883026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f605814fc50 0x7f6058156de0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f605001cdb0 tx=0x7f605001bf20 comp rx=0 tx=0).stop 2023-10-16T20:44:38.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.828+0000 7f605fcc9640 1 -- 172.21.15.88:0/1838883026 shutdown_connections 2023-10-16T20:44:38.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.828+0000 7f605fcc9640 1 --2- 172.21.15.88:0/1838883026 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f603c0664a0 0x7f603c068960 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:38.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.828+0000 7f605fcc9640 1 --2- 172.21.15.88:0/1838883026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f605814fc50 0x7f6058156de0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:38.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.828+0000 7f605fcc9640 1 --2- 172.21.15.88:0/1838883026 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f605806ffe0 0x7f605814b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:38.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.828+0000 7f605fcc9640 1 --2- 172.21.15.88:0/1838883026 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6058069e90 0x7f6058158980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:38.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.828+0000 7f605fcc9640 1 -- 172.21.15.88:0/1838883026 >> 172.21.15.88:0/1838883026 conn(0x7f605806e0f0 msgr2=0x7f60580cb5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:38.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.828+0000 7f605fcc9640 1 -- 172.21.15.88:0/1838883026 shutdown_connections 2023-10-16T20:44:38.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:38.828+0000 7f605fcc9640 1 -- 172.21.15.88:0/1838883026 wait complete. 2023-10-16T20:44:38.849 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 40, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 40, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4275": { "addr": "172.21.15.88:6838/321173984", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 321173984, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 321173984, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4275, "incarnation": 14, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 41 }, "gid_4279": { "addr": "172.21.15.191:6833/2446850309", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 2446850309, "type": "v2" }, { "addr": "172.21.15.191:6833", "nonce": 2446850309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 40 }, "gid_4288": { "addr": "172.21.15.191:6835/3059699574", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3059699574, "type": "v2" }, { "addr": "172.21.15.191:6835", "nonce": 3059699574, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4889": { "addr": "172.21.15.191:6837/2468647900", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6836", "nonce": 2468647900, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2468647900, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4889, "incarnation": 30, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 21 }, "gid_5284": { "addr": "172.21.15.88:6839/2172973775", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6837", "nonce": 2172973775, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2172973775, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5284, "incarnation": 38, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:reconnect", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:44:36.391560+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 4279, "mds_2": 4275, "mds_3": 5284, "mds_4": 4889 } } } ], "standbys": [] } 2023-10-16T20:44:40.851 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:40.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:40.996+0000 7f4bd0246640 1 Processor -- start 2023-10-16T20:44:40.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:40.996+0000 7f4bd0246640 1 -- start start 2023-10-16T20:44:41.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:40.996+0000 7f4bd0246640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc8150280 0x7f4bc80c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:41.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:40.996+0000 7f4bd0246640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc80c5c60 con 0x7f4bc80c6bc0 2023-10-16T20:44:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:40.996+0000 7f4bd0246640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc80c5370 con 0x7f4bc8150660 2023-10-16T20:44:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:40.996+0000 7f4bd0246640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc8156d10 con 0x7f4bc8150280 2023-10-16T20:44:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:40.996+0000 7f4bce7e3640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4bc80c6bc0 0x7f4bc8156110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33714/0 (socket says 172.21.15.88:33714) 2023-10-16T20:44:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:40.996+0000 7f4bce7e3640 1 -- 172.21.15.88:0/2016928806 learned_addr learned my addr 172.21.15.88:0/2016928806 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:44:41.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:40.996+0000 7f4bccfe0640 1 -- 172.21.15.88:0/2016928806 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998681614 0 0) 0x7f4bc80c5c60 con 0x7f4bc80c6bc0 2023-10-16T20:44:41.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:40.996+0000 7f4bccfe0640 1 -- 172.21.15.88:0/2016928806 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb4003530 con 0x7f4bc80c6bc0 2023-10-16T20:44:41.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:40.996+0000 7f4bcdfe2640 1 --2- 172.21.15.88:0/2016928806 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc8150280 0x7f4bc80c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:41.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:40.996+0000 7f4bccfe0640 1 -- 172.21.15.88:0/2016928806 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1673005277 0 0) 0x7f4bb4003530 con 0x7f4bc80c6bc0 2023-10-16T20:44:41.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:40.996+0000 7f4bccfe0640 1 -- 172.21.15.88:0/2016928806 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc8150280 msgr2=0x7f4bc80c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:41.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:40.996+0000 7f4bccfe0640 1 --2- 172.21.15.88:0/2016928806 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc8150280 0x7f4bc80c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:41.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:40.996+0000 7f4bccfe0640 1 -- 172.21.15.88:0/2016928806 >> v1:172.21.15.191:6789/0 conn(0x7f4bc8150660 legacy=0x7f4bc806d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:41.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:40.996+0000 7f4bccfe0640 1 -- 172.21.15.88:0/2016928806 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc80c5610 con 0x7f4bc80c6bc0 2023-10-16T20:44:41.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:40.996+0000 7f4bccfe0640 1 -- 172.21.15.88:0/2016928806 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f4bc0001a00 con 0x7f4bc80c6bc0 2023-10-16T20:44:41.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bccfe0640 1 -- 172.21.15.88:0/2016928806 >> v1:172.21.15.88:6789/0 conn(0x7f4bc80c6bc0 legacy=0x7f4bc8156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:41.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bccfe0640 1 --2- 172.21.15.88:0/2016928806 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc8150280 0x7f4bb40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:41.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bccfe0640 1 --2- 172.21.15.88:0/2016928806 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bb4003f30 0x7f4bb40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:41.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bccfe0640 1 --2- 172.21.15.88:0/2016928806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb4004d10 0x7f4bb4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:41.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bccfe0640 1 -- 172.21.15.88:0/2016928806 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc8156d10 con 0x7f4bb4004d10 2023-10-16T20:44:41.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bccfe0640 1 -- 172.21.15.88:0/2016928806 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb400dd30 con 0x7f4bb4003f30 2023-10-16T20:44:41.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bccfe0640 1 -- 172.21.15.88:0/2016928806 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb400def0 con 0x7f4bc8150280 2023-10-16T20:44:41.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bce7e3640 1 --2- 172.21.15.88:0/2016928806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb4004d10 0x7f4bb4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:41.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bcdfe2640 1 --2- 172.21.15.88:0/2016928806 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc8150280 0x7f4bb40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:41.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bcd7e1640 1 --2- 172.21.15.88:0/2016928806 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bb4003f30 0x7f4bb40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:41.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bce7e3640 1 -- 172.21.15.88:0/2016928806 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc8150280 msgr2=0x7f4bb40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:41.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bce7e3640 1 --2- 172.21.15.88:0/2016928806 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc8150280 0x7f4bb40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:41.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bce7e3640 1 -- 172.21.15.88:0/2016928806 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bb4003f30 msgr2=0x7f4bb40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:41.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bce7e3640 1 --2- 172.21.15.88:0/2016928806 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bb4003f30 0x7f4bb40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:41.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bce7e3640 1 -- 172.21.15.88:0/2016928806 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb4003530 con 0x7f4bb4004d10 2023-10-16T20:44:41.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bcdfe2640 1 --2- 172.21.15.88:0/2016928806 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc8150280 0x7f4bb40039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:44:41.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bce7e3640 1 --2- 172.21.15.88:0/2016928806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb4004d10 0x7f4bb4005190 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000a830 tx=0x7f4bc001faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a54a56c4d206ea7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:41.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.000+0000 7f4bccfe0640 1 -- 172.21.15.88:0/2016928806 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc00aa6d0 con 0x7f4bb4004d10 2023-10-16T20:44:41.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bccfe0640 1 -- 172.21.15.88:0/2016928806 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc00aa850 con 0x7f4bb4004d10 2023-10-16T20:44:41.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bccfe0640 1 -- 172.21.15.88:0/2016928806 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc00aac00 con 0x7f4bb4004d10 2023-10-16T20:44:41.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bd0246640 1 -- 172.21.15.88:0/2016928806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb4004d10 msgr2=0x7f4bb4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:41.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bd0246640 1 --2- 172.21.15.88:0/2016928806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb4004d10 0x7f4bb4005190 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000a830 tx=0x7f4bc001faa0 comp rx=0 tx=0).stop 2023-10-16T20:44:41.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bce7e3640 1 -- 172.21.15.88:0/2016928806 reap_dead start 2023-10-16T20:44:41.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bd0246640 1 -- 172.21.15.88:0/2016928806 shutdown_connections 2023-10-16T20:44:41.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bd0246640 1 -- 172.21.15.88:0/2016928806 >> 172.21.15.88:0/2016928806 conn(0x7f4bc80cad50 msgr2=0x7f4bc80cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:41.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bd0246640 1 -- 172.21.15.88:0/2016928806 shutdown_connections 2023-10-16T20:44:41.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bd0246640 1 -- 172.21.15.88:0/2016928806 wait complete. 2023-10-16T20:44:41.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bd0246640 1 Processor -- start 2023-10-16T20:44:41.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bd0246640 1 -- start start 2023-10-16T20:44:41.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bd0246640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb4004d10 0x7f4bc8145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:41.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bd0246640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bc8150a80 0x7f4bc8153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:41.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bd0246640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc81e5c20 0x7f4bc81e60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:41.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bd0246640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc81515d0 con 0x7f4bb4004d10 2023-10-16T20:44:41.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bd0246640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc8145f80 con 0x7f4bc8150a80 2023-10-16T20:44:41.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bd0246640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc8153840 con 0x7f4bc81e5c20 2023-10-16T20:44:41.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bcdfe2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb4004d10 0x7f4bc8145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:41.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bcdfe2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb4004d10 0x7f4bc8145a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48940/0 (socket says 172.21.15.88:48940) 2023-10-16T20:44:41.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bcdfe2640 1 -- 172.21.15.88:0/4226234590 learned_addr learned my addr 172.21.15.88:0/4226234590 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:41.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.004+0000 7f4bcd7e1640 1 --2- 172.21.15.88:0/4226234590 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bc8150a80 0x7f4bc8153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:41.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.008+0000 7f4bce7e3640 1 --2- 172.21.15.88:0/4226234590 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc81e5c20 0x7f4bc81e60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:41.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.008+0000 7f4bcdfe2640 1 -- 172.21.15.88:0/4226234590 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc81e5c20 msgr2=0x7f4bc81e60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:41.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.008+0000 7f4bcdfe2640 1 --2- 172.21.15.88:0/4226234590 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc81e5c20 0x7f4bc81e60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:41.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.008+0000 7f4bcdfe2640 1 -- 172.21.15.88:0/4226234590 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bc8150a80 msgr2=0x7f4bc8153f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:41.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.008+0000 7f4bcdfe2640 1 --2- 172.21.15.88:0/4226234590 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bc8150a80 0x7f4bc8153f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:41.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.008+0000 7f4bcdfe2640 1 -- 172.21.15.88:0/4226234590 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc81532c0 con 0x7f4bb4004d10 2023-10-16T20:44:41.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.008+0000 7f4bcdfe2640 1 --2- 172.21.15.88:0/4226234590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb4004d10 0x7f4bc8145a40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4011ca0 tx=0x7f4bc400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:41.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.008+0000 7f4bbe7fc640 1 -- 172.21.15.88:0/4226234590 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc400fd20 con 0x7f4bb4004d10 2023-10-16T20:44:41.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.008+0000 7f4bbe7fc640 1 -- 172.21.15.88:0/4226234590 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc400fe80 con 0x7f4bb4004d10 2023-10-16T20:44:41.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.008+0000 7f4bd0246640 1 -- 172.21.15.88:0/4226234590 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc814f520 con 0x7f4bb4004d10 2023-10-16T20:44:41.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.008+0000 7f4bbe7fc640 1 -- 172.21.15.88:0/4226234590 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4004260 con 0x7f4bb4004d10 2023-10-16T20:44:41.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.008+0000 7f4bd0246640 1 -- 172.21.15.88:0/4226234590 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc814faa0 con 0x7f4bb4004d10 2023-10-16T20:44:41.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.008+0000 7f4bd0246640 1 -- 172.21.15.88:0/4226234590 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc8074700 con 0x7f4bb4004d10 2023-10-16T20:44:41.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.008+0000 7f4bbe7fc640 1 -- 172.21.15.88:0/4226234590 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4bc40043c0 con 0x7f4bb4004d10 2023-10-16T20:44:41.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.008+0000 7f4bbe7fc640 1 --2- 172.21.15.88:0/4226234590 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4bac0664d0 0x7f4bac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:41.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.008+0000 7f4bbe7fc640 1 -- 172.21.15.88:0/4226234590 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f4bc4084b60 con 0x7f4bb4004d10 2023-10-16T20:44:41.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.008+0000 7f4bcd7e1640 1 --2- 172.21.15.88:0/4226234590 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4bac0664d0 0x7f4bac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:41.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.012+0000 7f4bcd7e1640 1 --2- 172.21.15.88:0/4226234590 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4bac0664d0 0x7f4bac068990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8156b10 tx=0x7f4bb8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:41.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.012+0000 7f4bbe7fc640 1 -- 172.21.15.88:0/4226234590 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4bc4051a80 con 0x7f4bb4004d10 2023-10-16T20:44:41.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.192+0000 7f4bd0246640 1 -- 172.21.15.88:0/4226234590 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bc8146bb0 con 0x7f4bb4004d10 2023-10-16T20:44:41.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.192+0000 7f4bbe7fc640 1 -- 172.21.15.88:0/4226234590 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5791 (secure 0 0 0) 0x7f4bc4022020 con 0x7f4bb4004d10 2023-10-16T20:44:41.197 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:41.197 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:36.391560+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:41.198 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 40 2023-10-16T20:44:41.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.196+0000 7f4bd0246640 1 -- 172.21.15.88:0/4226234590 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4bac0664d0 msgr2=0x7f4bac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:41.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.196+0000 7f4bd0246640 1 --2- 172.21.15.88:0/4226234590 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4bac0664d0 0x7f4bac068990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8156b10 tx=0x7f4bb8009040 comp rx=0 tx=0).stop 2023-10-16T20:44:41.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.196+0000 7f4bd0246640 1 -- 172.21.15.88:0/4226234590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb4004d10 msgr2=0x7f4bc8145a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:41.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.196+0000 7f4bd0246640 1 --2- 172.21.15.88:0/4226234590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb4004d10 0x7f4bc8145a40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4011ca0 tx=0x7f4bc400f610 comp rx=0 tx=0).stop 2023-10-16T20:44:41.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.196+0000 7f4bd0246640 1 -- 172.21.15.88:0/4226234590 shutdown_connections 2023-10-16T20:44:41.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.196+0000 7f4bd0246640 1 --2- 172.21.15.88:0/4226234590 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4bac0664d0 0x7f4bac068990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:41.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.196+0000 7f4bd0246640 1 --2- 172.21.15.88:0/4226234590 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4bc81e5c20 0x7f4bc81e60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:41.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.196+0000 7f4bd0246640 1 --2- 172.21.15.88:0/4226234590 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4bc8150a80 0x7f4bc8153f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:41.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.196+0000 7f4bd0246640 1 --2- 172.21.15.88:0/4226234590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb4004d10 0x7f4bc8145a40 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:41.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.196+0000 7f4bd0246640 1 -- 172.21.15.88:0/4226234590 >> 172.21.15.88:0/4226234590 conn(0x7f4bc80cad50 msgr2=0x7f4bc80c9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:41.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.196+0000 7f4bd0246640 1 -- 172.21.15.88:0/4226234590 shutdown_connections 2023-10-16T20:44:41.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:41.196+0000 7f4bd0246640 1 -- 172.21.15.88:0/4226234590 wait complete. 2023-10-16T20:44:41.217 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:44:43.218 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:43.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.364+0000 7fcd31246640 1 Processor -- start 2023-10-16T20:44:43.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.364+0000 7fcd31246640 1 -- start start 2023-10-16T20:44:43.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.368+0000 7fcd31246640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd2c150280 0x7fcd2c0c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:43.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.368+0000 7fcd31246640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcd2c0c6bc0 0x7fcd2c0c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:43.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.368+0000 7fcd31246640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd2c0c5c60 con 0x7fcd2c150660 2023-10-16T20:44:43.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.368+0000 7fcd31246640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd2c0c5370 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.368+0000 7fcd31246640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c156d10 con 0x7fcd2c150280 2023-10-16T20:44:43.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.368+0000 7fcd2b59e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcd2c0c6bc0 0x7fcd2c0c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:43.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.368+0000 7fcd2a59c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcd2c150660 0x7fcd2c06d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33720/0 (socket says 172.21.15.88:33720) 2023-10-16T20:44:43.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.368+0000 7fcd2a59c640 1 -- 172.21.15.88:0/2446256549 learned_addr learned my addr 172.21.15.88:0/2446256549 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:44:43.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.368+0000 7fcd2b59e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcd2c0c6bc0 0x7fcd2c0c3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48952/0 (socket says 172.21.15.88:48952) 2023-10-16T20:44:43.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.368+0000 7fcd2b59e640 1 -- 172.21.15.88:0/2446256549 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd2c150280 msgr2=0x7fcd2c0c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:43.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.368+0000 7fcd2b59e640 1 --2- 172.21.15.88:0/2446256549 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd2c150280 0x7fcd2c0c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:43.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.368+0000 7fcd2b59e640 1 -- 172.21.15.88:0/2446256549 >> v1:172.21.15.88:6789/0 conn(0x7fcd2c150660 legacy=0x7fcd2c06d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:43.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.368+0000 7fcd2b59e640 1 -- 172.21.15.88:0/2446256549 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2c0c5610 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.368+0000 7fcd29d9b640 1 -- 172.21.15.88:0/2446256549 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405259215 0 0) 0x7fcd2c0c5c60 con 0x7fcd2c150660 2023-10-16T20:44:43.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.372+0000 7fcd2b59e640 1 --2- 172.21.15.88:0/2446256549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd2c0c6bc0 0x7fcd2c0c3110 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009d90 tx=0x7fcd2401f500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e23a08ed975bd757 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:43.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.372+0000 7fcd29d9b640 1 -- 172.21.15.88:0/2446256549 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd240a7b90 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.372+0000 7fcd29d9b640 1 -- 172.21.15.88:0/2446256549 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd240a7cf0 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.372+0000 7fcd29d9b640 1 -- 172.21.15.88:0/2446256549 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd240af660 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.372+0000 7fcd31246640 1 -- 172.21.15.88:0/2446256549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd2c0c6bc0 msgr2=0x7fcd2c0c3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:43.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.372+0000 7fcd31246640 1 --2- 172.21.15.88:0/2446256549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd2c0c6bc0 0x7fcd2c0c3110 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009d90 tx=0x7fcd2401f500 comp rx=0 tx=0).stop 2023-10-16T20:44:43.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.372+0000 7fcd31246640 1 -- 172.21.15.88:0/2446256549 shutdown_connections 2023-10-16T20:44:43.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.372+0000 7fcd31246640 1 --2- 172.21.15.88:0/2446256549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd2c0c6bc0 0x7fcd2c0c3110 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:43.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.372+0000 7fcd31246640 1 --2- 172.21.15.88:0/2446256549 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd2c150280 0x7fcd2c0c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:43.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.372+0000 7fcd31246640 1 -- 172.21.15.88:0/2446256549 >> 172.21.15.88:0/2446256549 conn(0x7fcd2c0cad50 msgr2=0x7fcd2c0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:43.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.372+0000 7fcd31246640 1 -- 172.21.15.88:0/2446256549 shutdown_connections 2023-10-16T20:44:43.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.372+0000 7fcd31246640 1 -- 172.21.15.88:0/2446256549 wait complete. 2023-10-16T20:44:43.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.372+0000 7fcd31246640 1 Processor -- start 2023-10-16T20:44:43.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.372+0000 7fcd31246640 1 -- start start 2023-10-16T20:44:43.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd31246640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd2c0c6bc0 0x7fcd2c142b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:43.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd31246640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd2c150280 0x7fcd2c1f1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:43.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd31246640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd2c150660 0x7fcd2c13d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:43.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd31246640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c1f58f0 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd31246640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c1f53d0 con 0x7fcd2c150660 2023-10-16T20:44:43.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd31246640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2c1f5650 con 0x7fcd2c150280 2023-10-16T20:44:43.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd2ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd2c0c6bc0 0x7fcd2c142b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:43.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd2ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd2c0c6bc0 0x7fcd2c142b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48958/0 (socket says 172.21.15.88:48958) 2023-10-16T20:44:43.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd2ad9d640 1 -- 172.21.15.88:0/3365528956 learned_addr learned my addr 172.21.15.88:0/3365528956 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:43.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd2a59c640 1 --2- 172.21.15.88:0/3365528956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd2c150280 0x7fcd2c1f1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:43.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd2b59e640 1 --2- 172.21.15.88:0/3365528956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd2c150660 0x7fcd2c13d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:43.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd2ad9d640 1 -- 172.21.15.88:0/3365528956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd2c150280 msgr2=0x7fcd2c1f1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:43.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd2ad9d640 1 --2- 172.21.15.88:0/3365528956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd2c150280 0x7fcd2c1f1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:43.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd2ad9d640 1 -- 172.21.15.88:0/3365528956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd2c150660 msgr2=0x7fcd2c13d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:43.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd2ad9d640 1 --2- 172.21.15.88:0/3365528956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd2c150660 0x7fcd2c13d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:43.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd2ad9d640 1 -- 172.21.15.88:0/3365528956 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd24009a70 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd2b59e640 1 --2- 172.21.15.88:0/3365528956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd2c150660 0x7fcd2c13d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:44:43.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd2ad9d640 1 --2- 172.21.15.88:0/3365528956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd2c0c6bc0 0x7fcd2c142b60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c00b550 tx=0x7fcd1c00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:43.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd137fe640 1 -- 172.21.15.88:0/3365528956 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c010870 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd137fe640 1 -- 172.21.15.88:0/3365528956 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1c0109f0 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd137fe640 1 -- 172.21.15.88:0/3365528956 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c010da0 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd31246640 1 -- 172.21.15.88:0/3365528956 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd2c0ca080 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd31246640 1 -- 172.21.15.88:0/3365528956 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2c0ca640 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.376+0000 7fcd31246640 1 -- 172.21.15.88:0/3365528956 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd2c074700 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.380+0000 7fcd137fe640 1 -- 172.21.15.88:0/3365528956 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcd1c00f3c0 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.380+0000 7fcd137fe640 1 --2- 172.21.15.88:0/3365528956 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcd180664a0 0x7fcd18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:43.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.380+0000 7fcd137fe640 1 -- 172.21.15.88:0/3365528956 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6921+0+0 (secure 0 0 0) 0x7fcd1c0843a0 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.380+0000 7fcd2a59c640 1 --2- 172.21.15.88:0/3365528956 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcd180664a0 0x7fcd18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:43.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.380+0000 7fcd2a59c640 1 --2- 172.21.15.88:0/3365528956 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcd180664a0 0x7fcd18068960 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c072300 tx=0x7fcd20007c10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:43.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.384+0000 7fcd137fe640 1 -- 172.21.15.88:0/3365528956 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcd1c051360 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.568+0000 7fcd31246640 1 -- 172.21.15.88:0/3365528956 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd2c0c1650 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.568+0000 7fcd137fe640 1 -- 172.21.15.88:0/3365528956 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+5788 (secure 0 0 0) 0x7fcd1c054980 con 0x7fcd2c0c6bc0 2023-10-16T20:44:43.572 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:43.572 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:41.409914+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:43.572 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 41 2023-10-16T20:44:43.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.572+0000 7fcd31246640 1 -- 172.21.15.88:0/3365528956 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcd180664a0 msgr2=0x7fcd18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:43.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.572+0000 7fcd31246640 1 --2- 172.21.15.88:0/3365528956 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcd180664a0 0x7fcd18068960 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c072300 tx=0x7fcd20007c10 comp rx=0 tx=0).stop 2023-10-16T20:44:43.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.572+0000 7fcd31246640 1 -- 172.21.15.88:0/3365528956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd2c0c6bc0 msgr2=0x7fcd2c142b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:43.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.572+0000 7fcd31246640 1 --2- 172.21.15.88:0/3365528956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd2c0c6bc0 0x7fcd2c142b60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c00b550 tx=0x7fcd1c00ba10 comp rx=0 tx=0).stop 2023-10-16T20:44:43.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.572+0000 7fcd31246640 1 -- 172.21.15.88:0/3365528956 shutdown_connections 2023-10-16T20:44:43.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.572+0000 7fcd31246640 1 --2- 172.21.15.88:0/3365528956 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcd180664a0 0x7fcd18068960 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.572+0000 7fcd31246640 1 --2- 172.21.15.88:0/3365528956 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcd2c150660 0x7fcd2c13d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.572+0000 7fcd31246640 1 --2- 172.21.15.88:0/3365528956 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcd2c150280 0x7fcd2c1f1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.572+0000 7fcd31246640 1 --2- 172.21.15.88:0/3365528956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd2c0c6bc0 0x7fcd2c142b60 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.572+0000 7fcd31246640 1 -- 172.21.15.88:0/3365528956 >> 172.21.15.88:0/3365528956 conn(0x7fcd2c0cad50 msgr2=0x7fcd2c154530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:43.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.572+0000 7fcd31246640 1 -- 172.21.15.88:0/3365528956 shutdown_connections 2023-10-16T20:44:43.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:43.572+0000 7fcd31246640 1 -- 172.21.15.88:0/3365528956 wait complete. 2023-10-16T20:44:43.592 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:44:45.594 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:45.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.740+0000 7f0674009640 1 Processor -- start 2023-10-16T20:44:45.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.740+0000 7f0674009640 1 -- start start 2023-10-16T20:44:45.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.744+0000 7f0674009640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f066c06a970 0x7f066c071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:45.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.744+0000 7f0674009640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f066c071e60 0x7f066c15a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:45.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.744+0000 7f0674009640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066c15b3c0 con 0x7f066c1406d0 2023-10-16T20:44:45.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.744+0000 7f0674009640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f066c0c2460 con 0x7f066c071e60 2023-10-16T20:44:45.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.744+0000 7f0674009640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066c15b680 con 0x7f066c06a970 2023-10-16T20:44:45.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.744+0000 7f06725a6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f066c071e60 0x7f066c15a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:45.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.744+0000 7f06725a6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f066c071e60 0x7f066c15a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48966/0 (socket says 172.21.15.88:48966) 2023-10-16T20:44:45.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.744+0000 7f06725a6640 1 -- 172.21.15.88:0/2797663499 learned_addr learned my addr 172.21.15.88:0/2797663499 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:45.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.744+0000 7f06715a4640 1 --2- 172.21.15.88:0/2797663499 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f066c06a970 0x7f066c071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:45.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.744+0000 7f06725a6640 1 -- 172.21.15.88:0/2797663499 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f066c06a970 msgr2=0x7f066c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:45.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.744+0000 7f06725a6640 1 --2- 172.21.15.88:0/2797663499 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f066c06a970 0x7f066c071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:45.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.744+0000 7f06725a6640 1 -- 172.21.15.88:0/2797663499 >> v1:172.21.15.191:6789/0 conn(0x7f066c1406d0 legacy=0x7f066c0c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:45.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.744+0000 7f06725a6640 1 -- 172.21.15.88:0/2797663499 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c157570 con 0x7f066c071e60 2023-10-16T20:44:45.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.744+0000 7f06725a6640 1 --2- 172.21.15.88:0/2797663499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c071e60 0x7f066c15a810 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0664009e50 tx=0x7f066401c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4774f942035e22f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:45.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.744+0000 7f0670da3640 1 -- 172.21.15.88:0/2797663499 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06640a57c0 con 0x7f066c071e60 2023-10-16T20:44:45.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0670da3640 1 -- 172.21.15.88:0/2797663499 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06640a5920 con 0x7f066c071e60 2023-10-16T20:44:45.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0670da3640 1 -- 172.21.15.88:0/2797663499 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06640a5cb0 con 0x7f066c071e60 2023-10-16T20:44:45.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0674009640 1 -- 172.21.15.88:0/2797663499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c071e60 msgr2=0x7f066c15a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:45.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0674009640 1 --2- 172.21.15.88:0/2797663499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c071e60 0x7f066c15a810 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0664009e50 tx=0x7f066401c9e0 comp rx=0 tx=0).stop 2023-10-16T20:44:45.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0674009640 1 -- 172.21.15.88:0/2797663499 shutdown_connections 2023-10-16T20:44:45.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0674009640 1 --2- 172.21.15.88:0/2797663499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c071e60 0x7f066c15a810 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:45.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0674009640 1 --2- 172.21.15.88:0/2797663499 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f066c06a970 0x7f066c071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:45.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0674009640 1 -- 172.21.15.88:0/2797663499 >> 172.21.15.88:0/2797663499 conn(0x7f066c0cad50 msgr2=0x7f066c0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:45.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0674009640 1 -- 172.21.15.88:0/2797663499 shutdown_connections 2023-10-16T20:44:45.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0674009640 1 -- 172.21.15.88:0/2797663499 wait complete. 2023-10-16T20:44:45.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0674009640 1 Processor -- start 2023-10-16T20:44:45.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0674009640 1 -- start start 2023-10-16T20:44:45.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0674009640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c06a970 0x7f066c139680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:45.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0674009640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f066c071e60 0x7f066c1f1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:45.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0674009640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f066c1406d0 0x7f066c0c90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:45.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0674009640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066c1f58b0 con 0x7f066c06a970 2023-10-16T20:44:45.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0674009640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066c0ca300 con 0x7f066c071e60 2023-10-16T20:44:45.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0674009640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066c0ca5e0 con 0x7f066c1406d0 2023-10-16T20:44:45.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0671da5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c06a970 0x7f066c139680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:45.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0671da5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c06a970 0x7f066c139680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48972/0 (socket says 172.21.15.88:48972) 2023-10-16T20:44:45.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f0671da5640 1 -- 172.21.15.88:0/1047789813 learned_addr learned my addr 172.21.15.88:0/1047789813 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:45.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f06715a4640 1 --2- 172.21.15.88:0/1047789813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f066c071e60 0x7f066c1f1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:45.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.748+0000 7f06725a6640 1 --2- 172.21.15.88:0/1047789813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f066c1406d0 0x7f066c0c90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:45.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.752+0000 7f0671da5640 1 -- 172.21.15.88:0/1047789813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f066c1406d0 msgr2=0x7f066c0c90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:45.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.752+0000 7f0671da5640 1 --2- 172.21.15.88:0/1047789813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f066c1406d0 0x7f066c0c90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:45.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.752+0000 7f0671da5640 1 -- 172.21.15.88:0/1047789813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f066c071e60 msgr2=0x7f066c1f1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:45.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.752+0000 7f0671da5640 1 --2- 172.21.15.88:0/1047789813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f066c071e60 0x7f066c1f1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:45.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.752+0000 7f0671da5640 1 -- 172.21.15.88:0/1047789813 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0664009b30 con 0x7f066c06a970 2023-10-16T20:44:45.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.752+0000 7f06725a6640 1 --2- 172.21.15.88:0/1047789813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f066c1406d0 0x7f066c0c90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:44:45.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.752+0000 7f0671da5640 1 --2- 172.21.15.88:0/1047789813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c06a970 0x7f066c139680 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f066800c500 tx=0x7f066800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:45.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.752+0000 7f06627fc640 1 -- 172.21.15.88:0/1047789813 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0668005220 con 0x7f066c06a970 2023-10-16T20:44:45.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.752+0000 7f06627fc640 1 -- 172.21.15.88:0/1047789813 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0668005380 con 0x7f066c06a970 2023-10-16T20:44:45.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.752+0000 7f06627fc640 1 -- 172.21.15.88:0/1047789813 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0668007270 con 0x7f066c06a970 2023-10-16T20:44:45.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.752+0000 7f0674009640 1 -- 172.21.15.88:0/1047789813 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f066c1f4f80 con 0x7f066c06a970 2023-10-16T20:44:45.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.752+0000 7f0674009640 1 -- 172.21.15.88:0/1047789813 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f066c1f5540 con 0x7f066c06a970 2023-10-16T20:44:45.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.752+0000 7f0674009640 1 -- 172.21.15.88:0/1047789813 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f066c0c1720 con 0x7f066c06a970 2023-10-16T20:44:45.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.752+0000 7f06627fc640 1 -- 172.21.15.88:0/1047789813 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f06680073d0 con 0x7f066c06a970 2023-10-16T20:44:45.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.752+0000 7f06627fc640 1 --2- 172.21.15.88:0/1047789813 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f06500664a0 0x7f0650068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:45.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.756+0000 7f06627fc640 1 -- 172.21.15.88:0/1047789813 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f0668083ca0 con 0x7f066c06a970 2023-10-16T20:44:45.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.756+0000 7f06715a4640 1 --2- 172.21.15.88:0/1047789813 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f06500664a0 0x7f0650068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:45.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.756+0000 7f06715a4640 1 --2- 172.21.15.88:0/1047789813 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f06500664a0 0x7f0650068960 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f066c1513c0 tx=0x7f065c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:45.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.760+0000 7f06627fc640 1 -- 172.21.15.88:0/1047789813 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0668050bd0 con 0x7f066c06a970 2023-10-16T20:44:45.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.944+0000 7f0674009640 1 -- 172.21.15.88:0/1047789813 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f066c074700 con 0x7f066c06a970 2023-10-16T20:44:45.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.944+0000 7f06627fc640 1 -- 172.21.15.88:0/1047789813 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+5788 (secure 0 0 0) 0x7f06680541f0 con 0x7f066c06a970 2023-10-16T20:44:45.948 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:45.948 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:41.409914+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:45.950 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 41 2023-10-16T20:44:45.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.948+0000 7f0674009640 1 -- 172.21.15.88:0/1047789813 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f06500664a0 msgr2=0x7f0650068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:45.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.948+0000 7f0674009640 1 --2- 172.21.15.88:0/1047789813 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f06500664a0 0x7f0650068960 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f066c1513c0 tx=0x7f065c008040 comp rx=0 tx=0).stop 2023-10-16T20:44:45.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.948+0000 7f0674009640 1 -- 172.21.15.88:0/1047789813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c06a970 msgr2=0x7f066c139680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:45.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.948+0000 7f0674009640 1 --2- 172.21.15.88:0/1047789813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c06a970 0x7f066c139680 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f066800c500 tx=0x7f066800c9c0 comp rx=0 tx=0).stop 2023-10-16T20:44:45.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.948+0000 7f0674009640 1 -- 172.21.15.88:0/1047789813 shutdown_connections 2023-10-16T20:44:45.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.948+0000 7f0674009640 1 --2- 172.21.15.88:0/1047789813 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f06500664a0 0x7f0650068960 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:45.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.948+0000 7f0674009640 1 --2- 172.21.15.88:0/1047789813 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f066c1406d0 0x7f066c0c90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:45.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.948+0000 7f0674009640 1 --2- 172.21.15.88:0/1047789813 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f066c071e60 0x7f066c1f1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:45.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.948+0000 7f0674009640 1 --2- 172.21.15.88:0/1047789813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c06a970 0x7f066c139680 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:45.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.948+0000 7f0674009640 1 -- 172.21.15.88:0/1047789813 >> 172.21.15.88:0/1047789813 conn(0x7f066c0cad50 msgr2=0x7f066c156a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:45.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.948+0000 7f0674009640 1 -- 172.21.15.88:0/1047789813 shutdown_connections 2023-10-16T20:44:45.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:45.948+0000 7f0674009640 1 -- 172.21.15.88:0/1047789813 wait complete. 2023-10-16T20:44:45.968 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:44:47.970 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:48.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.116+0000 7ff931b72640 1 Processor -- start 2023-10-16T20:44:48.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.116+0000 7ff931b72640 1 -- start start 2023-10-16T20:44:48.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.116+0000 7ff931b72640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff92c153770 0x7ff92c159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:48.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.116+0000 7ff931b72640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff92c0cece0 0x7ff92c0cf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:48.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.116+0000 7ff931b72640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92c15a410 con 0x7ff92c0c9bb0 2023-10-16T20:44:48.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.116+0000 7ff931b72640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff92c0cc250 con 0x7ff92c0cece0 2023-10-16T20:44:48.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.116+0000 7ff931b72640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0cc670 con 0x7ff92c153770 2023-10-16T20:44:48.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.116+0000 7ff92b7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff92c153770 0x7ff92c159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:48.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.116+0000 7ff92b7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff92c153770 0x7ff92c159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:38118/0 (socket says 172.21.15.88:38118) 2023-10-16T20:44:48.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.116+0000 7ff92b7fe640 1 -- 172.21.15.88:0/2062208307 learned_addr learned my addr 172.21.15.88:0/2062208307 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:48.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.116+0000 7ff92bfff640 1 --2- 172.21.15.88:0/2062208307 >> v2:172.21.15.191:3300/0 conn(0x7ff92c0cece0 0x7ff92c0cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:48.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.116+0000 7ff92b7fe640 1 -- 172.21.15.88:0/2062208307 >> v2:172.21.15.191:3300/0 conn(0x7ff92c0cece0 msgr2=0x7ff92c0cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:48.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.116+0000 7ff92b7fe640 1 --2- 172.21.15.88:0/2062208307 >> v2:172.21.15.191:3300/0 conn(0x7ff92c0cece0 0x7ff92c0cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:48.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.116+0000 7ff92b7fe640 1 -- 172.21.15.88:0/2062208307 >> v1:172.21.15.191:6789/0 conn(0x7ff92c0c9bb0 legacy=0x7ff92c0c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:48.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.116+0000 7ff92a7fc640 1 -- 172.21.15.88:0/2062208307 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203153168 0 0) 0x7ff92c15a410 con 0x7ff92c0c9bb0 2023-10-16T20:44:48.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.116+0000 7ff92b7fe640 1 -- 172.21.15.88:0/2062208307 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c153b50 con 0x7ff92c153770 2023-10-16T20:44:48.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.120+0000 7ff92b7fe640 1 --2- 172.21.15.88:0/2062208307 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff92c153770 0x7ff92c159ed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff91c006fd0 tx=0x7ff91c01c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4afeb11202762a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:48.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.120+0000 7ff92a7fc640 1 -- 172.21.15.88:0/2062208307 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c0aa680 con 0x7ff92c153770 2023-10-16T20:44:48.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.120+0000 7ff92a7fc640 1 -- 172.21.15.88:0/2062208307 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91c0aa7e0 con 0x7ff92c153770 2023-10-16T20:44:48.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.120+0000 7ff92a7fc640 1 -- 172.21.15.88:0/2062208307 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c0aab70 con 0x7ff92c153770 2023-10-16T20:44:48.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.120+0000 7ff931b72640 1 -- 172.21.15.88:0/2062208307 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff92c153770 msgr2=0x7ff92c159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:48.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.120+0000 7ff931b72640 1 --2- 172.21.15.88:0/2062208307 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff92c153770 0x7ff92c159ed0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff91c006fd0 tx=0x7ff91c01c520 comp rx=0 tx=0).stop 2023-10-16T20:44:48.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.120+0000 7ff931b72640 1 -- 172.21.15.88:0/2062208307 shutdown_connections 2023-10-16T20:44:48.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.120+0000 7ff931b72640 1 --2- 172.21.15.88:0/2062208307 >> v2:172.21.15.191:3300/0 conn(0x7ff92c0cece0 0x7ff92c0cf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:48.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.120+0000 7ff931b72640 1 --2- 172.21.15.88:0/2062208307 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff92c153770 0x7ff92c159ed0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:48.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.120+0000 7ff931b72640 1 -- 172.21.15.88:0/2062208307 >> 172.21.15.88:0/2062208307 conn(0x7ff92c0715c0 msgr2=0x7ff92c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:48.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.120+0000 7ff931b72640 1 -- 172.21.15.88:0/2062208307 shutdown_connections 2023-10-16T20:44:48.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.120+0000 7ff931b72640 1 -- 172.21.15.88:0/2062208307 wait complete. 2023-10-16T20:44:48.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff931b72640 1 Processor -- start 2023-10-16T20:44:48.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff931b72640 1 -- start start 2023-10-16T20:44:48.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff931b72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff92c0c9bb0 0x7ff92c149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:48.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff931b72640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff92c0cece0 0x7ff92c1f47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:48.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff931b72640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c153770 0x7ff92c1f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:48.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff931b72640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c155970 con 0x7ff92c0c9bb0 2023-10-16T20:44:48.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff931b72640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c155520 con 0x7ff92c153770 2023-10-16T20:44:48.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff931b72640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c154750 con 0x7ff92c0cece0 2023-10-16T20:44:48.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff92b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff92c0c9bb0 0x7ff92c149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:48.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff92b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff92c0c9bb0 0x7ff92c149910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48984/0 (socket says 172.21.15.88:48984) 2023-10-16T20:44:48.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff92b7fe640 1 -- 172.21.15.88:0/3576708077 learned_addr learned my addr 172.21.15.88:0/3576708077 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:48.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff92bfff640 1 --2- 172.21.15.88:0/3576708077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c153770 0x7ff92c1f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:48.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff92affd640 1 --2- 172.21.15.88:0/3576708077 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff92c0cece0 0x7ff92c1f47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:48.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff92b7fe640 1 -- 172.21.15.88:0/3576708077 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff92c0cece0 msgr2=0x7ff92c1f47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:48.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff92b7fe640 1 --2- 172.21.15.88:0/3576708077 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff92c0cece0 0x7ff92c1f47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:48.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff92b7fe640 1 -- 172.21.15.88:0/3576708077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c153770 msgr2=0x7ff92c1f4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:48.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff92b7fe640 1 --2- 172.21.15.88:0/3576708077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c153770 0x7ff92c1f4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:48.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff92b7fe640 1 -- 172.21.15.88:0/3576708077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c009480 con 0x7ff92c0c9bb0 2023-10-16T20:44:48.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff92b7fe640 1 --2- 172.21.15.88:0/3576708077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff92c0c9bb0 0x7ff92c149910 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff91c01c560 tx=0x7ff91c0192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:48.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff913fff640 1 -- 172.21.15.88:0/3576708077 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c0aab50 con 0x7ff92c0c9bb0 2023-10-16T20:44:48.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff913fff640 1 -- 172.21.15.88:0/3576708077 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91c0b8070 con 0x7ff92c0c9bb0 2023-10-16T20:44:48.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff913fff640 1 -- 172.21.15.88:0/3576708077 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c003620 con 0x7ff92c0c9bb0 2023-10-16T20:44:48.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff931b72640 1 -- 172.21.15.88:0/3576708077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92c15a340 con 0x7ff92c0c9bb0 2023-10-16T20:44:48.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.124+0000 7ff931b72640 1 -- 172.21.15.88:0/3576708077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92c154c80 con 0x7ff92c0c9bb0 2023-10-16T20:44:48.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.128+0000 7ff931b72640 1 -- 172.21.15.88:0/3576708077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92c074700 con 0x7ff92c0c9bb0 2023-10-16T20:44:48.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.128+0000 7ff913fff640 1 -- 172.21.15.88:0/3576708077 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff91c003780 con 0x7ff92c0c9bb0 2023-10-16T20:44:48.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.128+0000 7ff913fff640 1 --2- 172.21.15.88:0/3576708077 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff9180664a0 0x7ff918068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:48.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.128+0000 7ff913fff640 1 -- 172.21.15.88:0/3576708077 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6921+0+0 (secure 0 0 0) 0x7ff91c0ba070 con 0x7ff92c0c9bb0 2023-10-16T20:44:48.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.128+0000 7ff92affd640 1 --2- 172.21.15.88:0/3576708077 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff9180664a0 0x7ff918068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:48.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.128+0000 7ff92affd640 1 --2- 172.21.15.88:0/3576708077 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff9180664a0 0x7ff918068960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff92c0b20d0 tx=0x7ff920009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:48.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.132+0000 7ff913fff640 1 -- 172.21.15.88:0/3576708077 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff91c0ec100 con 0x7ff92c0c9bb0 2023-10-16T20:44:48.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.312+0000 7ff931b72640 1 -- 172.21.15.88:0/3576708077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff92c14d800 con 0x7ff92c0c9bb0 2023-10-16T20:44:48.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.312+0000 7ff913fff640 1 -- 172.21.15.88:0/3576708077 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5788 (secure 0 0 0) 0x7ff91c0c3090 con 0x7ff92c0c9bb0 2023-10-16T20:44:48.319 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:48.319 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:46.789923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:48.320 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 42 2023-10-16T20:44:48.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.316+0000 7ff931b72640 1 -- 172.21.15.88:0/3576708077 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff9180664a0 msgr2=0x7ff918068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:48.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.316+0000 7ff931b72640 1 --2- 172.21.15.88:0/3576708077 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff9180664a0 0x7ff918068960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff92c0b20d0 tx=0x7ff920009040 comp rx=0 tx=0).stop 2023-10-16T20:44:48.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.316+0000 7ff931b72640 1 -- 172.21.15.88:0/3576708077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff92c0c9bb0 msgr2=0x7ff92c149910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:48.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.316+0000 7ff931b72640 1 --2- 172.21.15.88:0/3576708077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff92c0c9bb0 0x7ff92c149910 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff91c01c560 tx=0x7ff91c0192c0 comp rx=0 tx=0).stop 2023-10-16T20:44:48.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.316+0000 7ff931b72640 1 -- 172.21.15.88:0/3576708077 shutdown_connections 2023-10-16T20:44:48.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.316+0000 7ff931b72640 1 --2- 172.21.15.88:0/3576708077 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff9180664a0 0x7ff918068960 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:48.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.316+0000 7ff931b72640 1 --2- 172.21.15.88:0/3576708077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff92c153770 0x7ff92c1f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:48.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.316+0000 7ff931b72640 1 --2- 172.21.15.88:0/3576708077 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff92c0cece0 0x7ff92c1f47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:48.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.316+0000 7ff931b72640 1 --2- 172.21.15.88:0/3576708077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff92c0c9bb0 0x7ff92c149910 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:48.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.316+0000 7ff931b72640 1 -- 172.21.15.88:0/3576708077 >> 172.21.15.88:0/3576708077 conn(0x7ff92c0715c0 msgr2=0x7ff92c1522c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:48.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.316+0000 7ff931b72640 1 -- 172.21.15.88:0/3576708077 shutdown_connections 2023-10-16T20:44:48.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:48.316+0000 7ff931b72640 1 -- 172.21.15.88:0/3576708077 wait complete. 2023-10-16T20:44:48.337 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=3, replacing gid=4263 2023-10-16T20:44:48.338 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.a 2023-10-16T20:44:49.950 INFO:tasks.ceph.mds.f.smithi191.stderr:2023-10-16T20:44:49.950+0000 7fe37f078640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-10-16T20:44:50.339 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-10-16T20:44:50.339 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-16T20:44:50.339 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-16T20:44:50.343 INFO:tasks.ceph.mds.a:Started 2023-10-16T20:44:50.344 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:50.437 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2023-10-16T20:44:50.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.488+0000 7f57c37f8640 1 Processor -- start 2023-10-16T20:44:50.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.488+0000 7f57c37f8640 1 -- start start 2023-10-16T20:44:50.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.488+0000 7f57c37f8640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f57bc153770 0x7f57bc159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:50.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.488+0000 7f57c37f8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57bc0c9bb0 0x7f57bc0c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:50.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.488+0000 7f57c37f8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57bc15a410 con 0x7f57bc0c41f0 2023-10-16T20:44:50.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.488+0000 7f57c37f8640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57bc0cc250 con 0x7f57bc153770 2023-10-16T20:44:50.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.488+0000 7f57c37f8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc0cc670 con 0x7f57bc0c9bb0 2023-10-16T20:44:50.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.488+0000 7f57c0d93640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57bc0c9bb0 0x7f57bc0c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:50.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.488+0000 7f57c0d93640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57bc0c9bb0 0x7f57bc0c3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:60254/0 (socket says 172.21.15.88:60254) 2023-10-16T20:44:50.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.488+0000 7f57c0d93640 1 -- 172.21.15.88:0/1112333246 learned_addr learned my addr 172.21.15.88:0/1112333246 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:50.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.488+0000 7f57c1594640 1 --2- 172.21.15.88:0/1112333246 >> v2:172.21.15.191:3300/0 conn(0x7f57bc153770 0x7f57bc159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:50.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.488+0000 7f57c1594640 1 -- 172.21.15.88:0/1112333246 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57bc0c9bb0 msgr2=0x7f57bc0c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:50.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.488+0000 7f57c1594640 1 --2- 172.21.15.88:0/1112333246 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57bc0c9bb0 0x7f57bc0c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:50.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.488+0000 7f57c1594640 1 -- 172.21.15.88:0/1112333246 >> v1:172.21.15.191:6789/0 conn(0x7f57bc0c41f0 legacy=0x7f57bc0cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:50.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.488+0000 7f57c1594640 1 -- 172.21.15.88:0/1112333246 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57bc153b50 con 0x7f57bc153770 2023-10-16T20:44:50.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.488+0000 7f57c0d93640 1 --2- 172.21.15.88:0/1112333246 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57bc0c9bb0 0x7f57bc0c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:44:50.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.492+0000 7f57c1594640 1 --2- 172.21.15.88:0/1112333246 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57bc153770 0x7f57bc159ed0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f57b80097a0 tx=0x7f57b801c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=122fcc27b749cfdf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:50.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.492+0000 7f57b3fff640 1 -- 172.21.15.88:0/1112333246 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b80ab600 con 0x7f57bc153770 2023-10-16T20:44:50.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.492+0000 7f57b3fff640 1 -- 172.21.15.88:0/1112333246 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b80a7050 con 0x7f57bc153770 2023-10-16T20:44:50.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.492+0000 7f57b3fff640 1 -- 172.21.15.88:0/1112333246 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b80ab980 con 0x7f57bc153770 2023-10-16T20:44:50.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.492+0000 7f57c37f8640 1 -- 172.21.15.88:0/1112333246 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57bc153770 msgr2=0x7f57bc159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:50.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.492+0000 7f57c37f8640 1 --2- 172.21.15.88:0/1112333246 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57bc153770 0x7f57bc159ed0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f57b80097a0 tx=0x7f57b801c460 comp rx=0 tx=0).stop 2023-10-16T20:44:50.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.492+0000 7f57c37f8640 1 -- 172.21.15.88:0/1112333246 shutdown_connections 2023-10-16T20:44:50.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.492+0000 7f57c37f8640 1 --2- 172.21.15.88:0/1112333246 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57bc0c9bb0 0x7f57bc0c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:50.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.492+0000 7f57c37f8640 1 --2- 172.21.15.88:0/1112333246 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57bc153770 0x7f57bc159ed0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:50.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.492+0000 7f57c37f8640 1 -- 172.21.15.88:0/1112333246 >> 172.21.15.88:0/1112333246 conn(0x7f57bc0715c0 msgr2=0x7f57bc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:50.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.492+0000 7f57c37f8640 1 -- 172.21.15.88:0/1112333246 shutdown_connections 2023-10-16T20:44:50.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.492+0000 7f57c37f8640 1 -- 172.21.15.88:0/1112333246 wait complete. 2023-10-16T20:44:50.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c37f8640 1 Processor -- start 2023-10-16T20:44:50.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c37f8640 1 -- start start 2023-10-16T20:44:50.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c37f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc0c41f0 0x7f57bc151c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:50.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c37f8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57bc0c9bb0 0x7f57bc0b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:50.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c37f8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57bc153770 0x7f57bc0cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:50.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c37f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc148e70 con 0x7f57bc0c41f0 2023-10-16T20:44:50.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c37f8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc152290 con 0x7f57bc153770 2023-10-16T20:44:50.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c37f8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc154950 con 0x7f57bc0c9bb0 2023-10-16T20:44:50.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c1594640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc0c41f0 0x7f57bc151c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:50.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c1594640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc0c41f0 0x7f57bc151c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38120/0 (socket says 172.21.15.88:38120) 2023-10-16T20:44:50.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c1594640 1 -- 172.21.15.88:0/1047105211 learned_addr learned my addr 172.21.15.88:0/1047105211 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:50.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c1594640 1 -- 172.21.15.88:0/1047105211 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57bc0c9bb0 msgr2=0x7f57bc0b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:50.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c1594640 1 --2- 172.21.15.88:0/1047105211 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57bc0c9bb0 0x7f57bc0b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:50.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c1594640 1 -- 172.21.15.88:0/1047105211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57bc153770 msgr2=0x7f57bc0cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:50.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c1d95640 1 --2- 172.21.15.88:0/1047105211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57bc153770 0x7f57bc0cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:50.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c1594640 1 --2- 172.21.15.88:0/1047105211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57bc153770 0x7f57bc0cceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:50.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c1594640 1 -- 172.21.15.88:0/1047105211 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ac0094d0 con 0x7f57bc0c41f0 2023-10-16T20:44:50.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c1d95640 1 --2- 172.21.15.88:0/1047105211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57bc153770 0x7f57bc0cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:44:50.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c1594640 1 --2- 172.21.15.88:0/1047105211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc0c41f0 0x7f57bc151c70 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f57b80ae040 tx=0x7f57b801aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:50.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57b1ffb640 1 -- 172.21.15.88:0/1047105211 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b80b7450 con 0x7f57bc0c41f0 2023-10-16T20:44:50.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57b1ffb640 1 -- 172.21.15.88:0/1047105211 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b80b75d0 con 0x7f57bc0c41f0 2023-10-16T20:44:50.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57b1ffb640 1 -- 172.21.15.88:0/1047105211 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b80b7980 con 0x7f57bc0c41f0 2023-10-16T20:44:50.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c37f8640 1 -- 172.21.15.88:0/1047105211 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b80093b0 con 0x7f57bc0c41f0 2023-10-16T20:44:50.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.496+0000 7f57c37f8640 1 -- 172.21.15.88:0/1047105211 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57bc0ce510 con 0x7f57bc0c41f0 2023-10-16T20:44:50.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.500+0000 7f57c37f8640 1 -- 172.21.15.88:0/1047105211 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57bc1553d0 con 0x7f57bc0c41f0 2023-10-16T20:44:50.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.500+0000 7f57b1ffb640 1 -- 172.21.15.88:0/1047105211 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f57b80b7ae0 con 0x7f57bc0c41f0 2023-10-16T20:44:50.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.500+0000 7f57b1ffb640 1 --2- 172.21.15.88:0/1047105211 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57a00664a0 0x7f57a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:50.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.500+0000 7f57b1ffb640 1 -- 172.21.15.88:0/1047105211 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f57b811ea90 con 0x7f57bc0c41f0 2023-10-16T20:44:50.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.500+0000 7f57c0d93640 1 --2- 172.21.15.88:0/1047105211 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57a00664a0 0x7f57a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:50.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.500+0000 7f57c0d93640 1 --2- 172.21.15.88:0/1047105211 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57a00664a0 0x7f57a0068960 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f57ac006f50 tx=0x7f57ac006ea0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:50.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.504+0000 7f57b1ffb640 1 -- 172.21.15.88:0/1047105211 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f57b80eba40 con 0x7f57bc0c41f0 2023-10-16T20:44:50.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.688+0000 7f57c37f8640 1 -- 172.21.15.88:0/1047105211 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57bc074700 con 0x7f57bc0c41f0 2023-10-16T20:44:50.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.688+0000 7f57b1ffb640 1 -- 172.21.15.88:0/1047105211 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5788 (secure 0 0 0) 0x7f57b80ef060 con 0x7f57bc0c41f0 2023-10-16T20:44:50.692 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:50.692 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:46.789923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:50.692 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 42 2023-10-16T20:44:50.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.688+0000 7f57c37f8640 1 -- 172.21.15.88:0/1047105211 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57a00664a0 msgr2=0x7f57a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.688+0000 7f57c37f8640 1 --2- 172.21.15.88:0/1047105211 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57a00664a0 0x7f57a0068960 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f57ac006f50 tx=0x7f57ac006ea0 comp rx=0 tx=0).stop 2023-10-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.688+0000 7f57c37f8640 1 -- 172.21.15.88:0/1047105211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc0c41f0 msgr2=0x7f57bc151c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.688+0000 7f57c37f8640 1 --2- 172.21.15.88:0/1047105211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc0c41f0 0x7f57bc151c70 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f57b80ae040 tx=0x7f57b801aca0 comp rx=0 tx=0).stop 2023-10-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.688+0000 7f57c37f8640 1 -- 172.21.15.88:0/1047105211 shutdown_connections 2023-10-16T20:44:50.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.688+0000 7f57c37f8640 1 --2- 172.21.15.88:0/1047105211 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57a00664a0 0x7f57a0068960 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.688+0000 7f57c37f8640 1 --2- 172.21.15.88:0/1047105211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57bc153770 0x7f57bc0cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.688+0000 7f57c37f8640 1 --2- 172.21.15.88:0/1047105211 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57bc0c9bb0 0x7f57bc0b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.688+0000 7f57c37f8640 1 --2- 172.21.15.88:0/1047105211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc0c41f0 0x7f57bc151c70 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:50.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.692+0000 7f57c37f8640 1 -- 172.21.15.88:0/1047105211 >> 172.21.15.88:0/1047105211 conn(0x7f57bc0715c0 msgr2=0x7f57bc0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:50.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.692+0000 7f57c37f8640 1 -- 172.21.15.88:0/1047105211 shutdown_connections 2023-10-16T20:44:50.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:50.692+0000 7f57c37f8640 1 -- 172.21.15.88:0/1047105211 wait complete. 2023-10-16T20:44:50.711 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-16T20:44:52.491 INFO:tasks.workunit.client.0.smithi088.stdout:./178/.snap/snap-subdir-test 2023-10-16T20:44:52.712 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:44:52.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.860+0000 7f13122d6640 1 Processor -- start 2023-10-16T20:44:52.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.860+0000 7f13122d6640 1 -- start start 2023-10-16T20:44:52.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.860+0000 7f13122d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f130c150280 0x7f130c0c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:52.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.860+0000 7f13122d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f130c150660 0x7f130c0c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:52.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.860+0000 7f13122d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f130c0c5c60 con 0x7f130c0c6bc0 2023-10-16T20:44:52.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.860+0000 7f13122d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f130c0c5370 con 0x7f130c150280 2023-10-16T20:44:52.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.860+0000 7f13122d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130c156d10 con 0x7f130c150660 2023-10-16T20:44:52.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.860+0000 7f130bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f130c150280 0x7f130c0c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:52.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.860+0000 7f1310873640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f130c0c6bc0 0x7f130c06d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35442/0 (socket says 172.21.15.88:35442) 2023-10-16T20:44:52.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.860+0000 7f1310873640 1 -- 172.21.15.88:0/1418546135 learned_addr learned my addr 172.21.15.88:0/1418546135 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:44:52.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.860+0000 7f130b7fe640 1 --2- 172.21.15.88:0/1418546135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f130c150660 0x7f130c0c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:52.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.860+0000 7f130bfff640 1 -- 172.21.15.88:0/1418546135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f130c150660 msgr2=0x7f130c0c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:44:52.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.860+0000 7f130bfff640 1 --2- 172.21.15.88:0/1418546135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f130c150660 0x7f130c0c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:52.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.860+0000 7f130bfff640 1 -- 172.21.15.88:0/1418546135 >> v1:172.21.15.88:6789/0 conn(0x7f130c0c6bc0 legacy=0x7f130c06d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:52.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.860+0000 7f130bfff640 1 -- 172.21.15.88:0/1418546135 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c0c5610 con 0x7f130c150280 2023-10-16T20:44:52.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.860+0000 7f130affd640 1 -- 172.21.15.88:0/1418546135 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259306691 0 0) 0x7f130c0c5c60 con 0x7f130c0c6bc0 2023-10-16T20:44:52.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.860+0000 7f130b7fe640 1 --2- 172.21.15.88:0/1418546135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f130c150660 0x7f130c0c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:44:52.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.864+0000 7f130bfff640 1 --2- 172.21.15.88:0/1418546135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f130c150280 0x7f130c0c60f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0097c0 tx=0x7f12fc01ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28324455d7802dbe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:52.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.864+0000 7f130affd640 1 -- 172.21.15.88:0/1418546135 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc0ac640 con 0x7f130c150280 2023-10-16T20:44:52.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.864+0000 7f130affd640 1 -- 172.21.15.88:0/1418546135 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12fc0ac7a0 con 0x7f130c150280 2023-10-16T20:44:52.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.864+0000 7f130affd640 1 -- 172.21.15.88:0/1418546135 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc0acb30 con 0x7f130c150280 2023-10-16T20:44:52.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.864+0000 7f13122d6640 1 -- 172.21.15.88:0/1418546135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f130c150280 msgr2=0x7f130c0c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:52.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.864+0000 7f13122d6640 1 --2- 172.21.15.88:0/1418546135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f130c150280 0x7f130c0c60f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0097c0 tx=0x7f12fc01ce20 comp rx=0 tx=0).stop 2023-10-16T20:44:52.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.864+0000 7f13122d6640 1 -- 172.21.15.88:0/1418546135 shutdown_connections 2023-10-16T20:44:52.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.864+0000 7f13122d6640 1 --2- 172.21.15.88:0/1418546135 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f130c150660 0x7f130c0c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:52.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.864+0000 7f13122d6640 1 --2- 172.21.15.88:0/1418546135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f130c150280 0x7f130c0c60f0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:52.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.864+0000 7f13122d6640 1 -- 172.21.15.88:0/1418546135 >> 172.21.15.88:0/1418546135 conn(0x7f130c0cad50 msgr2=0x7f130c0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:52.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.864+0000 7f13122d6640 1 -- 172.21.15.88:0/1418546135 shutdown_connections 2023-10-16T20:44:52.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.864+0000 7f13122d6640 1 -- 172.21.15.88:0/1418546135 wait complete. 2023-10-16T20:44:52.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f13122d6640 1 Processor -- start 2023-10-16T20:44:52.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f13122d6640 1 -- start start 2023-10-16T20:44:52.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f13122d6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f130c0c6bc0 0x7f130c1416e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:52.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f13122d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f130c150280 0x7f130c0718c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:52.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f13122d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f130c150660 0x7f130c071e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:52.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f13122d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c141c20 con 0x7f130c150660 2023-10-16T20:44:52.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f13122d6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c070b90 con 0x7f130c0c6bc0 2023-10-16T20:44:52.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f13122d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130c071430 con 0x7f130c150280 2023-10-16T20:44:52.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f1310873640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f130c150660 0x7f130c071e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:52.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f1310873640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f130c150660 0x7f130c071e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38126/0 (socket says 172.21.15.88:38126) 2023-10-16T20:44:52.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f1310873640 1 -- 172.21.15.88:0/3879900785 learned_addr learned my addr 172.21.15.88:0/3879900785 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:44:52.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f130bfff640 1 --2- 172.21.15.88:0/3879900785 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f130c0c6bc0 0x7f130c1416e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:52.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f130b7fe640 1 --2- 172.21.15.88:0/3879900785 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f130c150280 0x7f130c0718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:52.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f1310873640 1 -- 172.21.15.88:0/3879900785 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f130c150280 msgr2=0x7f130c0718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:52.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f1310873640 1 --2- 172.21.15.88:0/3879900785 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f130c150280 0x7f130c0718c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:52.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f1310873640 1 -- 172.21.15.88:0/3879900785 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f130c0c6bc0 msgr2=0x7f130c1416e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:52.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f1310873640 1 --2- 172.21.15.88:0/3879900785 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f130c0c6bc0 0x7f130c1416e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:52.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f1310873640 1 -- 172.21.15.88:0/3879900785 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc009420 con 0x7f130c150660 2023-10-16T20:44:52.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f130bfff640 1 --2- 172.21.15.88:0/3879900785 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f130c0c6bc0 0x7f130c1416e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:44:52.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f130b7fe640 1 --2- 172.21.15.88:0/3879900785 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f130c150280 0x7f130c0718c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:44:52.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f1310873640 1 --2- 172.21.15.88:0/3879900785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f130c150660 0x7f130c071e00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f130400c810 tx=0x7f130400ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:52.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f1308ff9640 1 -- 172.21.15.88:0/3879900785 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130400a5e0 con 0x7f130c150660 2023-10-16T20:44:52.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f1308ff9640 1 -- 172.21.15.88:0/3879900785 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130400a760 con 0x7f130c150660 2023-10-16T20:44:52.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f1308ff9640 1 -- 172.21.15.88:0/3879900785 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130400ab10 con 0x7f130c150660 2023-10-16T20:44:52.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f13122d6640 1 -- 172.21.15.88:0/3879900785 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130c070520 con 0x7f130c150660 2023-10-16T20:44:52.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.868+0000 7f13122d6640 1 -- 172.21.15.88:0/3879900785 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130c070de0 con 0x7f130c150660 2023-10-16T20:44:52.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.872+0000 7f13122d6640 1 -- 172.21.15.88:0/3879900785 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130c074700 con 0x7f130c150660 2023-10-16T20:44:52.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.872+0000 7f1308ff9640 1 -- 172.21.15.88:0/3879900785 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f130400ac70 con 0x7f130c150660 2023-10-16T20:44:52.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.872+0000 7f1308ff9640 1 --2- 172.21.15.88:0/3879900785 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f12f80664d0 0x7f12f8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:44:52.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.872+0000 7f1308ff9640 1 -- 172.21.15.88:0/3879900785 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6921+0+0 (secure 0 0 0) 0x7f1304083830 con 0x7f130c150660 2023-10-16T20:44:52.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.872+0000 7f130bfff640 1 --2- 172.21.15.88:0/3879900785 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f12f80664d0 0x7f12f8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:44:52.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.872+0000 7f130bfff640 1 --2- 172.21.15.88:0/3879900785 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f12f80664d0 0x7f12f8068990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0ae000 tx=0x7f12fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:44:52.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:52.876+0000 7f1308ff9640 1 -- 172.21.15.88:0/3879900785 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1304050750 con 0x7f130c150660 2023-10-16T20:44:53.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:53.056+0000 7f13122d6640 1 -- 172.21.15.88:0/3879900785 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f130c071570 con 0x7f130c150660 2023-10-16T20:44:53.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:53.056+0000 7f1308ff9640 1 -- 172.21.15.88:0/3879900785 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6578 (secure 0 0 0) 0x7f1304053d70 con 0x7f130c150660 2023-10-16T20:44:53.060 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:44:53.061 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:44:46.789923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:44:53.061 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 43 2023-10-16T20:44:53.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:53.060+0000 7f13122d6640 1 -- 172.21.15.88:0/3879900785 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f12f80664d0 msgr2=0x7f12f8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:53.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:53.060+0000 7f13122d6640 1 --2- 172.21.15.88:0/3879900785 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f12f80664d0 0x7f12f8068990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0ae000 tx=0x7f12fc008040 comp rx=0 tx=0).stop 2023-10-16T20:44:53.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:53.060+0000 7f13122d6640 1 -- 172.21.15.88:0/3879900785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f130c150660 msgr2=0x7f130c071e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:44:53.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:53.060+0000 7f13122d6640 1 --2- 172.21.15.88:0/3879900785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f130c150660 0x7f130c071e00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f130400c810 tx=0x7f130400ccd0 comp rx=0 tx=0).stop 2023-10-16T20:44:53.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:53.060+0000 7f13122d6640 1 -- 172.21.15.88:0/3879900785 shutdown_connections 2023-10-16T20:44:53.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:53.060+0000 7f13122d6640 1 --2- 172.21.15.88:0/3879900785 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f12f80664d0 0x7f12f8068990 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:53.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:53.060+0000 7f13122d6640 1 --2- 172.21.15.88:0/3879900785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f130c150660 0x7f130c071e00 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:53.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:53.060+0000 7f13122d6640 1 --2- 172.21.15.88:0/3879900785 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f130c150280 0x7f130c0718c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:53.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:53.060+0000 7f13122d6640 1 --2- 172.21.15.88:0/3879900785 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f130c0c6bc0 0x7f130c1416e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:44:53.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:53.060+0000 7f13122d6640 1 -- 172.21.15.88:0/3879900785 >> 172.21.15.88:0/3879900785 conn(0x7f130c0cad50 msgr2=0x7f130c06d200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:44:53.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:53.060+0000 7f13122d6640 1 -- 172.21.15.88:0/3879900785 shutdown_connections 2023-10-16T20:44:53.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:44:53.060+0000 7f13122d6640 1 -- 172.21.15.88:0/3879900785 wait complete. 2023-10-16T20:44:53.081 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-10-16T20:44:53.081 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 66 secs before thrashing 2023-10-16T20:44:53.985 INFO:tasks.workunit.client.0.smithi088.stdout:./266/.snap/snap-subdir-test 2023-10-16T20:44:54.003 INFO:tasks.workunit.client.0.smithi088.stdout:./240/.snap/snap-subdir-test 2023-10-16T20:44:54.022 INFO:tasks.workunit.client.0.smithi088.stdout:./156/.snap/snap-subdir-test 2023-10-16T20:44:54.040 INFO:tasks.workunit.client.0.smithi088.stdout:./131/.snap/snap-subdir-test 2023-10-16T20:44:54.057 INFO:tasks.workunit.client.0.smithi088.stdout:./123/.snap/snap-subdir-test 2023-10-16T20:44:54.074 INFO:tasks.workunit.client.0.smithi088.stdout:./160/.snap/snap-subdir-test 2023-10-16T20:44:54.089 INFO:tasks.workunit.client.0.smithi088.stdout:./165/.snap/snap-subdir-test 2023-10-16T20:44:54.106 INFO:tasks.workunit.client.0.smithi088.stdout:./101/.snap/snap-subdir-test 2023-10-16T20:44:54.122 INFO:tasks.workunit.client.0.smithi088.stdout:./148/.snap/snap-subdir-test 2023-10-16T20:44:54.138 INFO:tasks.workunit.client.0.smithi088.stdout:./168/.snap/snap-subdir-test 2023-10-16T20:44:55.032 INFO:tasks.workunit.client.0.smithi088.stdout:./179/.snap/snap-subdir-test 2023-10-16T20:44:55.048 INFO:tasks.workunit.client.0.smithi088.stdout:./257/.snap/snap-subdir-test 2023-10-16T20:44:55.063 INFO:tasks.workunit.client.0.smithi088.stdout:./252/.snap/snap-subdir-test 2023-10-16T20:44:55.080 INFO:tasks.workunit.client.0.smithi088.stdout:./132/.snap/snap-subdir-test 2023-10-16T20:44:55.096 INFO:tasks.workunit.client.0.smithi088.stdout:./236/.snap/snap-subdir-test 2023-10-16T20:44:55.110 INFO:tasks.workunit.client.0.smithi088.stdout:./211/.snap/snap-subdir-test 2023-10-16T20:44:55.126 INFO:tasks.workunit.client.0.smithi088.stdout:./199/.snap/snap-subdir-test 2023-10-16T20:44:55.141 INFO:tasks.workunit.client.0.smithi088.stdout:./205/.snap/snap-subdir-test 2023-10-16T20:44:55.156 INFO:tasks.workunit.client.0.smithi088.stdout:./163/.snap/snap-subdir-test 2023-10-16T20:44:55.170 INFO:tasks.workunit.client.0.smithi088.stdout:./261/.snap/snap-subdir-test 2023-10-16T20:44:55.513 INFO:tasks.workunit.client.0.smithi088.stdout:./201/.snap/snap-subdir-test 2023-10-16T20:44:55.527 INFO:tasks.workunit.client.0.smithi088.stdout:./282/.snap/snap-subdir-test 2023-10-16T20:44:55.543 INFO:tasks.workunit.client.0.smithi088.stdout:./296/.snap/snap-subdir-test 2023-10-16T20:44:55.559 INFO:tasks.workunit.client.0.smithi088.stdout:./217/.snap/snap-subdir-test 2023-10-16T20:44:55.824 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:44:55.829 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:44:56.148 INFO:tasks.workunit.client.0.smithi088.stdout:./153/.snap/snap-subdir-test 2023-10-16T20:44:56.164 INFO:tasks.workunit.client.0.smithi088.stdout:./206/.snap/snap-subdir-test 2023-10-16T20:44:56.182 INFO:tasks.workunit.client.0.smithi088.stdout:./122/.snap/snap-subdir-test 2023-10-16T20:44:56.201 INFO:tasks.workunit.client.0.smithi088.stdout:./162/.snap/snap-subdir-test 2023-10-16T20:44:56.216 INFO:tasks.workunit.client.0.smithi088.stdout:./197/.snap/snap-subdir-test 2023-10-16T20:44:56.232 INFO:tasks.workunit.client.0.smithi088.stdout:./216/.snap/snap-subdir-test 2023-10-16T20:44:59.782 INFO:tasks.workunit.client.0.smithi088.stdout:./254/.snap/snap-subdir-test 2023-10-16T20:44:59.797 INFO:tasks.workunit.client.0.smithi088.stdout:./113/.snap/snap-subdir-test 2023-10-16T20:44:59.813 INFO:tasks.workunit.client.0.smithi088.stdout:./152/.snap/snap-subdir-test 2023-10-16T20:44:59.831 INFO:tasks.workunit.client.0.smithi088.stdout:./121/.snap/snap-subdir-test 2023-10-16T20:44:59.846 INFO:tasks.workunit.client.0.smithi088.stdout:./118/.snap/snap-subdir-test 2023-10-16T20:44:59.863 INFO:tasks.workunit.client.0.smithi088.stdout:./196/.snap/snap-subdir-test 2023-10-16T20:44:59.879 INFO:tasks.workunit.client.0.smithi088.stdout:./124/.snap/snap-subdir-test 2023-10-16T20:44:59.895 INFO:tasks.workunit.client.0.smithi088.stdout:./239/.snap/snap-subdir-test 2023-10-16T20:44:59.912 INFO:tasks.workunit.client.0.smithi088.stdout:./188/.snap/snap-subdir-test 2023-10-16T20:44:59.927 INFO:tasks.workunit.client.0.smithi088.stdout:./287/.snap/snap-subdir-test 2023-10-16T20:44:59.942 INFO:tasks.workunit.client.0.smithi088.stdout:./275/.snap/snap-subdir-test 2023-10-16T20:44:59.956 INFO:tasks.workunit.client.0.smithi088.stdout:./259/.snap/snap-subdir-test 2023-10-16T20:44:59.971 INFO:tasks.workunit.client.0.smithi088.stdout:./224/.snap/snap-subdir-test 2023-10-16T20:44:59.986 INFO:tasks.workunit.client.0.smithi088.stdout:./264/.snap/snap-subdir-test 2023-10-16T20:45:00.003 INFO:tasks.workunit.client.0.smithi088.stdout:./145/.snap/snap-subdir-test 2023-10-16T20:45:00.016 INFO:tasks.workunit.client.0.smithi088.stdout:./279/.snap/snap-subdir-test 2023-10-16T20:45:00.029 INFO:tasks.workunit.client.0.smithi088.stdout:./263/.snap/snap-subdir-test 2023-10-16T20:45:00.044 INFO:tasks.workunit.client.0.smithi088.stdout:./130/.snap/snap-subdir-test 2023-10-16T20:45:00.057 INFO:tasks.workunit.client.0.smithi088.stdout:./193/.snap/snap-subdir-test 2023-10-16T20:45:00.071 INFO:tasks.workunit.client.0.smithi088.stdout:./223/.snap/snap-subdir-test 2023-10-16T20:45:00.083 INFO:tasks.workunit.client.0.smithi088.stdout:./207/.snap/snap-subdir-test 2023-10-16T20:45:00.096 INFO:tasks.workunit.client.0.smithi088.stdout:./276/.snap/snap-subdir-test 2023-10-16T20:45:00.108 INFO:tasks.workunit.client.0.smithi088.stdout:./271/.snap/snap-subdir-test 2023-10-16T20:45:00.121 INFO:tasks.workunit.client.0.smithi088.stdout:./251/.snap/snap-subdir-test 2023-10-16T20:45:00.135 INFO:tasks.workunit.client.0.smithi088.stdout:./175/.snap/snap-subdir-test 2023-10-16T20:45:00.148 INFO:tasks.workunit.client.0.smithi088.stdout:./222/.snap/snap-subdir-test 2023-10-16T20:45:00.161 INFO:tasks.workunit.client.0.smithi088.stdout:./220/.snap/snap-subdir-test 2023-10-16T20:45:00.174 INFO:tasks.workunit.client.0.smithi088.stdout:./203/.snap/snap-subdir-test 2023-10-16T20:45:00.188 INFO:tasks.workunit.client.0.smithi088.stdout:./181/.snap/snap-subdir-test 2023-10-16T20:45:00.200 INFO:tasks.workunit.client.0.smithi088.stdout:./286/.snap/snap-subdir-test 2023-10-16T20:45:00.214 INFO:tasks.workunit.client.0.smithi088.stdout:./127/.snap/snap-subdir-test 2023-10-16T20:45:00.228 INFO:tasks.workunit.client.0.smithi088.stdout:./108/.snap/snap-subdir-test 2023-10-16T20:45:01.001 INFO:tasks.workunit.client.0.smithi088.stdout:./183/.snap/snap-subdir-test 2023-10-16T20:45:01.016 INFO:tasks.workunit.client.0.smithi088.stdout:./294/.snap/snap-subdir-test 2023-10-16T20:45:01.898 INFO:tasks.workunit.client.0.smithi088.stdout:./235/.snap/snap-subdir-test 2023-10-16T20:45:01.914 INFO:tasks.workunit.client.0.smithi088.stdout:./177/.snap/snap-subdir-test 2023-10-16T20:45:01.930 INFO:tasks.workunit.client.0.smithi088.stdout:./169/.snap/snap-subdir-test 2023-10-16T20:45:01.945 INFO:tasks.workunit.client.0.smithi088.stdout:./212/.snap/snap-subdir-test 2023-10-16T20:45:01.959 INFO:tasks.workunit.client.0.smithi088.stdout:./154/.snap/snap-subdir-test 2023-10-16T20:45:01.974 INFO:tasks.workunit.client.0.smithi088.stdout:./171/.snap/snap-subdir-test 2023-10-16T20:45:01.990 INFO:tasks.workunit.client.0.smithi088.stdout:./139/.snap/snap-subdir-test 2023-10-16T20:45:02.004 INFO:tasks.workunit.client.0.smithi088.stdout:./102/.snap/snap-subdir-test 2023-10-16T20:45:02.019 INFO:tasks.workunit.client.0.smithi088.stdout:./129/.snap/snap-subdir-test 2023-10-16T20:45:20.646 INFO:tasks.workunit.client.0.smithi088.stdout:Delete all the files and directories ... 2023-10-16T20:45:23.015 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/19' 2023-10-16T20:45:23.015 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/15' 2023-10-16T20:45:23.016 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/31' 2023-10-16T20:45:23.016 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/36' 2023-10-16T20:45:23.016 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/20' 2023-10-16T20:45:23.016 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/12' 2023-10-16T20:45:23.017 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/39' 2023-10-16T20:45:23.017 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/10' 2023-10-16T20:45:23.017 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/34' 2023-10-16T20:45:23.017 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/38' 2023-10-16T20:45:23.017 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/11' 2023-10-16T20:45:23.018 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/16' 2023-10-16T20:45:23.018 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/32' 2023-10-16T20:45:23.018 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/30' 2023-10-16T20:45:23.018 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/14' 2023-10-16T20:45:23.018 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/18' 2023-10-16T20:45:23.019 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/13' 2023-10-16T20:45:23.019 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/33' 2023-10-16T20:45:23.019 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/17' 2023-10-16T20:45:23.019 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/35' 2023-10-16T20:45:23.020 INFO:tasks.workunit.client.0.smithi088.stdout:removed './100/37' 2023-10-16T20:45:23.020 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './100' 2023-10-16T20:45:23.020 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/19' 2023-10-16T20:45:23.020 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/15' 2023-10-16T20:45:23.020 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/31' 2023-10-16T20:45:23.021 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/36' 2023-10-16T20:45:23.021 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/20' 2023-10-16T20:45:23.021 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/12' 2023-10-16T20:45:23.021 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/39' 2023-10-16T20:45:23.021 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/10' 2023-10-16T20:45:23.021 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/34' 2023-10-16T20:45:23.022 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/38' 2023-10-16T20:45:23.022 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/11' 2023-10-16T20:45:23.022 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/16' 2023-10-16T20:45:23.022 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/32' 2023-10-16T20:45:23.022 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/30' 2023-10-16T20:45:23.023 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/14' 2023-10-16T20:45:23.023 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/18' 2023-10-16T20:45:23.023 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/13' 2023-10-16T20:45:23.023 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/33' 2023-10-16T20:45:23.023 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/17' 2023-10-16T20:45:23.024 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/35' 2023-10-16T20:45:23.024 INFO:tasks.workunit.client.0.smithi088.stdout:removed './101/37' 2023-10-16T20:45:23.024 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './101' 2023-10-16T20:45:23.024 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/19' 2023-10-16T20:45:23.024 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/15' 2023-10-16T20:45:23.025 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/31' 2023-10-16T20:45:23.025 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/36' 2023-10-16T20:45:23.025 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/20' 2023-10-16T20:45:23.025 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/12' 2023-10-16T20:45:23.025 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/39' 2023-10-16T20:45:23.025 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/10' 2023-10-16T20:45:23.026 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/34' 2023-10-16T20:45:23.026 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/38' 2023-10-16T20:45:23.026 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/11' 2023-10-16T20:45:23.026 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/16' 2023-10-16T20:45:23.026 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/32' 2023-10-16T20:45:23.027 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/30' 2023-10-16T20:45:23.027 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/14' 2023-10-16T20:45:23.027 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/18' 2023-10-16T20:45:23.027 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/13' 2023-10-16T20:45:23.027 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/33' 2023-10-16T20:45:23.028 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/17' 2023-10-16T20:45:23.028 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/35' 2023-10-16T20:45:23.028 INFO:tasks.workunit.client.0.smithi088.stdout:removed './102/37' 2023-10-16T20:45:23.028 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './102' 2023-10-16T20:45:23.028 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/19' 2023-10-16T20:45:23.029 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/15' 2023-10-16T20:45:23.029 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/31' 2023-10-16T20:45:23.029 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/36' 2023-10-16T20:45:23.029 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/20' 2023-10-16T20:45:23.029 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/12' 2023-10-16T20:45:23.030 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/39' 2023-10-16T20:45:23.030 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/10' 2023-10-16T20:45:23.030 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/34' 2023-10-16T20:45:23.030 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/38' 2023-10-16T20:45:23.030 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/11' 2023-10-16T20:45:23.031 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/16' 2023-10-16T20:45:23.031 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/32' 2023-10-16T20:45:23.031 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/30' 2023-10-16T20:45:23.031 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/14' 2023-10-16T20:45:23.031 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/18' 2023-10-16T20:45:23.032 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/13' 2023-10-16T20:45:23.032 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/33' 2023-10-16T20:45:23.032 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/17' 2023-10-16T20:45:23.032 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/35' 2023-10-16T20:45:23.032 INFO:tasks.workunit.client.0.smithi088.stdout:removed './103/37' 2023-10-16T20:45:23.033 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './103' 2023-10-16T20:45:23.033 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/19' 2023-10-16T20:45:23.033 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/15' 2023-10-16T20:45:23.033 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/31' 2023-10-16T20:45:23.033 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/36' 2023-10-16T20:45:23.034 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/20' 2023-10-16T20:45:23.034 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/12' 2023-10-16T20:45:23.034 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/39' 2023-10-16T20:45:23.034 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/10' 2023-10-16T20:45:23.034 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/34' 2023-10-16T20:45:23.035 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/38' 2023-10-16T20:45:23.035 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/11' 2023-10-16T20:45:23.035 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/16' 2023-10-16T20:45:23.035 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/32' 2023-10-16T20:45:23.035 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/30' 2023-10-16T20:45:23.036 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/14' 2023-10-16T20:45:23.036 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/18' 2023-10-16T20:45:23.036 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/13' 2023-10-16T20:45:23.036 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/33' 2023-10-16T20:45:23.036 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/17' 2023-10-16T20:45:23.037 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/35' 2023-10-16T20:45:23.037 INFO:tasks.workunit.client.0.smithi088.stdout:removed './104/37' 2023-10-16T20:45:23.037 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './104' 2023-10-16T20:45:23.037 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/19' 2023-10-16T20:45:23.037 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/15' 2023-10-16T20:45:23.038 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/31' 2023-10-16T20:45:23.038 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/36' 2023-10-16T20:45:23.038 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/20' 2023-10-16T20:45:23.038 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/12' 2023-10-16T20:45:23.038 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/39' 2023-10-16T20:45:23.039 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/10' 2023-10-16T20:45:23.039 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/34' 2023-10-16T20:45:23.039 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/38' 2023-10-16T20:45:23.039 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/11' 2023-10-16T20:45:23.039 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/16' 2023-10-16T20:45:23.040 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/32' 2023-10-16T20:45:23.040 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/30' 2023-10-16T20:45:23.040 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/14' 2023-10-16T20:45:23.040 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/18' 2023-10-16T20:45:23.040 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/13' 2023-10-16T20:45:23.040 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/33' 2023-10-16T20:45:23.041 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/17' 2023-10-16T20:45:23.041 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/35' 2023-10-16T20:45:23.041 INFO:tasks.workunit.client.0.smithi088.stdout:removed './105/37' 2023-10-16T20:45:23.041 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './105' 2023-10-16T20:45:23.041 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/19' 2023-10-16T20:45:23.042 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/15' 2023-10-16T20:45:23.042 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/31' 2023-10-16T20:45:23.042 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/36' 2023-10-16T20:45:23.042 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/20' 2023-10-16T20:45:23.042 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/12' 2023-10-16T20:45:23.043 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/39' 2023-10-16T20:45:23.043 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/10' 2023-10-16T20:45:23.043 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/34' 2023-10-16T20:45:23.043 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/38' 2023-10-16T20:45:23.043 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/11' 2023-10-16T20:45:23.044 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/16' 2023-10-16T20:45:23.044 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/32' 2023-10-16T20:45:23.044 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/30' 2023-10-16T20:45:23.044 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/14' 2023-10-16T20:45:23.044 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/18' 2023-10-16T20:45:23.044 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/13' 2023-10-16T20:45:23.045 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/33' 2023-10-16T20:45:23.045 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/17' 2023-10-16T20:45:23.045 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/35' 2023-10-16T20:45:23.045 INFO:tasks.workunit.client.0.smithi088.stdout:removed './106/37' 2023-10-16T20:45:23.045 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './106' 2023-10-16T20:45:23.045 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/19' 2023-10-16T20:45:23.046 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/15' 2023-10-16T20:45:23.046 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/31' 2023-10-16T20:45:23.046 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/36' 2023-10-16T20:45:23.046 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/20' 2023-10-16T20:45:23.046 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/12' 2023-10-16T20:45:23.047 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/39' 2023-10-16T20:45:23.047 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/10' 2023-10-16T20:45:23.047 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/34' 2023-10-16T20:45:23.047 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/38' 2023-10-16T20:45:23.047 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/11' 2023-10-16T20:45:23.048 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/16' 2023-10-16T20:45:23.048 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/32' 2023-10-16T20:45:23.048 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/30' 2023-10-16T20:45:23.048 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/14' 2023-10-16T20:45:23.048 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/18' 2023-10-16T20:45:23.049 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/13' 2023-10-16T20:45:23.049 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/33' 2023-10-16T20:45:23.049 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/17' 2023-10-16T20:45:23.049 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/35' 2023-10-16T20:45:23.049 INFO:tasks.workunit.client.0.smithi088.stdout:removed './107/37' 2023-10-16T20:45:23.049 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './107' 2023-10-16T20:45:23.050 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/19' 2023-10-16T20:45:23.050 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/15' 2023-10-16T20:45:23.050 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/31' 2023-10-16T20:45:23.050 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/36' 2023-10-16T20:45:23.050 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/20' 2023-10-16T20:45:23.051 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/12' 2023-10-16T20:45:23.051 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/39' 2023-10-16T20:45:23.051 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/10' 2023-10-16T20:45:23.051 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/34' 2023-10-16T20:45:23.051 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/38' 2023-10-16T20:45:23.052 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/11' 2023-10-16T20:45:23.052 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/16' 2023-10-16T20:45:23.052 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/32' 2023-10-16T20:45:23.052 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/30' 2023-10-16T20:45:23.052 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/14' 2023-10-16T20:45:23.052 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/18' 2023-10-16T20:45:23.053 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/13' 2023-10-16T20:45:23.053 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/33' 2023-10-16T20:45:23.053 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/17' 2023-10-16T20:45:23.053 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/35' 2023-10-16T20:45:23.053 INFO:tasks.workunit.client.0.smithi088.stdout:removed './108/37' 2023-10-16T20:45:23.054 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './108' 2023-10-16T20:45:23.054 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/19' 2023-10-16T20:45:23.054 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/15' 2023-10-16T20:45:23.054 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/31' 2023-10-16T20:45:23.054 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/36' 2023-10-16T20:45:23.055 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/20' 2023-10-16T20:45:23.055 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/12' 2023-10-16T20:45:23.055 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/39' 2023-10-16T20:45:23.055 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/10' 2023-10-16T20:45:23.055 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/34' 2023-10-16T20:45:23.056 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/38' 2023-10-16T20:45:23.057 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/11' 2023-10-16T20:45:23.057 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/16' 2023-10-16T20:45:23.057 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/32' 2023-10-16T20:45:23.058 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/30' 2023-10-16T20:45:23.590 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/14' 2023-10-16T20:45:23.591 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/18' 2023-10-16T20:45:23.591 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/13' 2023-10-16T20:45:23.591 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/33' 2023-10-16T20:45:23.591 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/17' 2023-10-16T20:45:23.591 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/35' 2023-10-16T20:45:23.592 INFO:tasks.workunit.client.0.smithi088.stdout:removed './109/37' 2023-10-16T20:45:23.592 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './109' 2023-10-16T20:45:23.592 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/19' 2023-10-16T20:45:23.592 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/15' 2023-10-16T20:45:23.592 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/31' 2023-10-16T20:45:23.593 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/36' 2023-10-16T20:45:23.593 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/20' 2023-10-16T20:45:23.593 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/12' 2023-10-16T20:45:23.593 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/39' 2023-10-16T20:45:23.593 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/10' 2023-10-16T20:45:23.594 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/34' 2023-10-16T20:45:23.594 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/38' 2023-10-16T20:45:23.594 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/11' 2023-10-16T20:45:23.594 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/16' 2023-10-16T20:45:23.594 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/32' 2023-10-16T20:45:23.594 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/30' 2023-10-16T20:45:23.595 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/14' 2023-10-16T20:45:23.595 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/18' 2023-10-16T20:45:23.595 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/13' 2023-10-16T20:45:23.595 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/33' 2023-10-16T20:45:23.595 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/17' 2023-10-16T20:45:23.596 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/35' 2023-10-16T20:45:23.596 INFO:tasks.workunit.client.0.smithi088.stdout:removed './110/37' 2023-10-16T20:45:23.596 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './110' 2023-10-16T20:45:23.596 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/19' 2023-10-16T20:45:23.596 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/15' 2023-10-16T20:45:23.597 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/31' 2023-10-16T20:45:23.597 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/36' 2023-10-16T20:45:23.597 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/20' 2023-10-16T20:45:23.597 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/12' 2023-10-16T20:45:23.597 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/39' 2023-10-16T20:45:23.597 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/10' 2023-10-16T20:45:23.598 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/34' 2023-10-16T20:45:23.598 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/38' 2023-10-16T20:45:23.598 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/11' 2023-10-16T20:45:23.598 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/16' 2023-10-16T20:45:23.598 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/32' 2023-10-16T20:45:23.599 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/30' 2023-10-16T20:45:23.599 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/14' 2023-10-16T20:45:23.599 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/18' 2023-10-16T20:45:23.599 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/13' 2023-10-16T20:45:23.599 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/33' 2023-10-16T20:45:23.600 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/17' 2023-10-16T20:45:23.600 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/35' 2023-10-16T20:45:23.600 INFO:tasks.workunit.client.0.smithi088.stdout:removed './111/37' 2023-10-16T20:45:23.600 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './111' 2023-10-16T20:45:23.600 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/19' 2023-10-16T20:45:23.601 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/15' 2023-10-16T20:45:23.601 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/31' 2023-10-16T20:45:23.601 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/36' 2023-10-16T20:45:23.601 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/20' 2023-10-16T20:45:23.601 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/12' 2023-10-16T20:45:23.601 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/39' 2023-10-16T20:45:23.602 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/10' 2023-10-16T20:45:23.602 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/34' 2023-10-16T20:45:23.602 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/38' 2023-10-16T20:45:23.602 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/11' 2023-10-16T20:45:23.602 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/16' 2023-10-16T20:45:23.603 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/32' 2023-10-16T20:45:23.603 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/30' 2023-10-16T20:45:23.603 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/14' 2023-10-16T20:45:23.603 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/18' 2023-10-16T20:45:23.603 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/13' 2023-10-16T20:45:23.604 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/33' 2023-10-16T20:45:23.604 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/17' 2023-10-16T20:45:23.604 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/35' 2023-10-16T20:45:23.604 INFO:tasks.workunit.client.0.smithi088.stdout:removed './112/37' 2023-10-16T20:45:23.604 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './112' 2023-10-16T20:45:23.604 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/19' 2023-10-16T20:45:23.605 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/15' 2023-10-16T20:45:23.605 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/31' 2023-10-16T20:45:23.605 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/36' 2023-10-16T20:45:23.605 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/20' 2023-10-16T20:45:23.605 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/12' 2023-10-16T20:45:23.605 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/39' 2023-10-16T20:45:23.606 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/10' 2023-10-16T20:45:23.606 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/34' 2023-10-16T20:45:23.606 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/38' 2023-10-16T20:45:23.606 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/11' 2023-10-16T20:45:23.606 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/16' 2023-10-16T20:45:23.607 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/32' 2023-10-16T20:45:23.607 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/30' 2023-10-16T20:45:23.607 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/14' 2023-10-16T20:45:23.607 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/18' 2023-10-16T20:45:23.607 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/13' 2023-10-16T20:45:23.607 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/33' 2023-10-16T20:45:23.608 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/17' 2023-10-16T20:45:23.608 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/35' 2023-10-16T20:45:23.608 INFO:tasks.workunit.client.0.smithi088.stdout:removed './113/37' 2023-10-16T20:45:23.608 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './113' 2023-10-16T20:45:23.608 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/19' 2023-10-16T20:45:23.609 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/15' 2023-10-16T20:45:23.609 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/31' 2023-10-16T20:45:23.609 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/36' 2023-10-16T20:45:23.609 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/20' 2023-10-16T20:45:23.609 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/12' 2023-10-16T20:45:23.609 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/39' 2023-10-16T20:45:23.610 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/10' 2023-10-16T20:45:23.610 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/34' 2023-10-16T20:45:23.610 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/38' 2023-10-16T20:45:23.610 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/11' 2023-10-16T20:45:23.610 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/16' 2023-10-16T20:45:23.611 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/32' 2023-10-16T20:45:23.611 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/30' 2023-10-16T20:45:23.611 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/14' 2023-10-16T20:45:23.611 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/18' 2023-10-16T20:45:23.611 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/13' 2023-10-16T20:45:23.611 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/33' 2023-10-16T20:45:23.612 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/17' 2023-10-16T20:45:23.612 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/35' 2023-10-16T20:45:23.612 INFO:tasks.workunit.client.0.smithi088.stdout:removed './114/37' 2023-10-16T20:45:23.612 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './114' 2023-10-16T20:45:23.612 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/19' 2023-10-16T20:45:23.613 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/15' 2023-10-16T20:45:23.613 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/31' 2023-10-16T20:45:23.613 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/36' 2023-10-16T20:45:23.613 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/20' 2023-10-16T20:45:23.613 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/12' 2023-10-16T20:45:23.613 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/39' 2023-10-16T20:45:23.614 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/10' 2023-10-16T20:45:23.614 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/34' 2023-10-16T20:45:23.614 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/38' 2023-10-16T20:45:23.614 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/11' 2023-10-16T20:45:23.614 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/16' 2023-10-16T20:45:23.615 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/32' 2023-10-16T20:45:23.615 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/30' 2023-10-16T20:45:23.615 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/14' 2023-10-16T20:45:23.615 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/18' 2023-10-16T20:45:23.615 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/13' 2023-10-16T20:45:23.615 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/33' 2023-10-16T20:45:23.616 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/17' 2023-10-16T20:45:23.616 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/35' 2023-10-16T20:45:23.616 INFO:tasks.workunit.client.0.smithi088.stdout:removed './115/37' 2023-10-16T20:45:23.616 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './115' 2023-10-16T20:45:23.616 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/19' 2023-10-16T20:45:23.617 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/15' 2023-10-16T20:45:23.617 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/31' 2023-10-16T20:45:23.617 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/36' 2023-10-16T20:45:23.617 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/20' 2023-10-16T20:45:23.617 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/12' 2023-10-16T20:45:23.617 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/39' 2023-10-16T20:45:23.618 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/10' 2023-10-16T20:45:23.618 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/34' 2023-10-16T20:45:23.618 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/38' 2023-10-16T20:45:23.618 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/11' 2023-10-16T20:45:23.618 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/16' 2023-10-16T20:45:23.619 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/32' 2023-10-16T20:45:23.619 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/30' 2023-10-16T20:45:23.619 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/14' 2023-10-16T20:45:23.619 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/18' 2023-10-16T20:45:23.619 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/13' 2023-10-16T20:45:23.619 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/33' 2023-10-16T20:45:23.620 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/17' 2023-10-16T20:45:23.620 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/35' 2023-10-16T20:45:23.620 INFO:tasks.workunit.client.0.smithi088.stdout:removed './116/37' 2023-10-16T20:45:23.620 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './116' 2023-10-16T20:45:23.620 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/19' 2023-10-16T20:45:23.621 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/15' 2023-10-16T20:45:23.621 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/31' 2023-10-16T20:45:23.621 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/36' 2023-10-16T20:45:23.621 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/20' 2023-10-16T20:45:23.621 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/12' 2023-10-16T20:45:23.621 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/39' 2023-10-16T20:45:23.622 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/10' 2023-10-16T20:45:23.622 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/34' 2023-10-16T20:45:23.622 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/38' 2023-10-16T20:45:23.622 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/11' 2023-10-16T20:45:23.622 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/16' 2023-10-16T20:45:23.623 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/32' 2023-10-16T20:45:23.623 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/30' 2023-10-16T20:45:23.623 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/14' 2023-10-16T20:45:23.623 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/18' 2023-10-16T20:45:23.623 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/13' 2023-10-16T20:45:23.623 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/33' 2023-10-16T20:45:23.624 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/17' 2023-10-16T20:45:23.624 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/35' 2023-10-16T20:45:23.624 INFO:tasks.workunit.client.0.smithi088.stdout:removed './117/37' 2023-10-16T20:45:23.624 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './117' 2023-10-16T20:45:23.624 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/19' 2023-10-16T20:45:23.625 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/15' 2023-10-16T20:45:23.625 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/31' 2023-10-16T20:45:23.625 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/36' 2023-10-16T20:45:23.625 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/20' 2023-10-16T20:45:23.625 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/12' 2023-10-16T20:45:23.625 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/39' 2023-10-16T20:45:23.626 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/10' 2023-10-16T20:45:23.626 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/34' 2023-10-16T20:45:23.626 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/38' 2023-10-16T20:45:23.626 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/11' 2023-10-16T20:45:23.626 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/16' 2023-10-16T20:45:23.627 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/32' 2023-10-16T20:45:23.627 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/30' 2023-10-16T20:45:23.627 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/14' 2023-10-16T20:45:23.627 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/18' 2023-10-16T20:45:23.627 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/13' 2023-10-16T20:45:23.627 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/33' 2023-10-16T20:45:23.628 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/17' 2023-10-16T20:45:23.628 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/35' 2023-10-16T20:45:23.628 INFO:tasks.workunit.client.0.smithi088.stdout:removed './118/37' 2023-10-16T20:45:23.628 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './118' 2023-10-16T20:45:23.628 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/19' 2023-10-16T20:45:23.628 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/15' 2023-10-16T20:45:23.629 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/31' 2023-10-16T20:45:23.629 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/36' 2023-10-16T20:45:23.629 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/20' 2023-10-16T20:45:23.629 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/12' 2023-10-16T20:45:25.170 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/39' 2023-10-16T20:45:25.170 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/10' 2023-10-16T20:45:25.170 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/34' 2023-10-16T20:45:25.170 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/38' 2023-10-16T20:45:25.171 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/11' 2023-10-16T20:45:25.171 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/16' 2023-10-16T20:45:25.171 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/32' 2023-10-16T20:45:25.171 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/30' 2023-10-16T20:45:25.171 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/14' 2023-10-16T20:45:25.171 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/18' 2023-10-16T20:45:25.172 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/13' 2023-10-16T20:45:25.172 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/33' 2023-10-16T20:45:25.172 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/17' 2023-10-16T20:45:25.172 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/35' 2023-10-16T20:45:25.172 INFO:tasks.workunit.client.0.smithi088.stdout:removed './119/37' 2023-10-16T20:45:25.173 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './119' 2023-10-16T20:45:25.173 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/19' 2023-10-16T20:45:25.173 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/15' 2023-10-16T20:45:25.173 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/31' 2023-10-16T20:45:25.173 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/36' 2023-10-16T20:45:25.173 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/20' 2023-10-16T20:45:25.174 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/12' 2023-10-16T20:45:25.174 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/39' 2023-10-16T20:45:25.174 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/10' 2023-10-16T20:45:25.174 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/34' 2023-10-16T20:45:25.174 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/38' 2023-10-16T20:45:25.175 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/11' 2023-10-16T20:45:25.175 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/16' 2023-10-16T20:45:25.175 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/32' 2023-10-16T20:45:25.175 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/30' 2023-10-16T20:45:25.175 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/14' 2023-10-16T20:45:25.175 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/18' 2023-10-16T20:45:25.176 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/13' 2023-10-16T20:45:25.176 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/33' 2023-10-16T20:45:25.176 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/17' 2023-10-16T20:45:25.176 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/35' 2023-10-16T20:45:25.177 INFO:tasks.workunit.client.0.smithi088.stdout:removed './120/37' 2023-10-16T20:45:25.177 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './120' 2023-10-16T20:45:25.177 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/19' 2023-10-16T20:45:25.177 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/15' 2023-10-16T20:45:25.177 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/31' 2023-10-16T20:45:25.177 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/36' 2023-10-16T20:45:25.178 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/20' 2023-10-16T20:45:25.178 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/12' 2023-10-16T20:45:25.178 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/39' 2023-10-16T20:45:25.178 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/10' 2023-10-16T20:45:25.178 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/34' 2023-10-16T20:45:25.179 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/38' 2023-10-16T20:45:25.179 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/11' 2023-10-16T20:45:25.179 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/16' 2023-10-16T20:45:25.179 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/32' 2023-10-16T20:45:25.179 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/30' 2023-10-16T20:45:25.180 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/14' 2023-10-16T20:45:25.180 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/18' 2023-10-16T20:45:25.180 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/13' 2023-10-16T20:45:25.180 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/33' 2023-10-16T20:45:25.180 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/17' 2023-10-16T20:45:25.180 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/35' 2023-10-16T20:45:25.181 INFO:tasks.workunit.client.0.smithi088.stdout:removed './121/37' 2023-10-16T20:45:25.181 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './121' 2023-10-16T20:45:25.181 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/19' 2023-10-16T20:45:25.181 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/15' 2023-10-16T20:45:25.181 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/31' 2023-10-16T20:45:25.182 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/36' 2023-10-16T20:45:25.182 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/20' 2023-10-16T20:45:25.182 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/12' 2023-10-16T20:45:25.182 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/39' 2023-10-16T20:45:25.182 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/10' 2023-10-16T20:45:25.183 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/34' 2023-10-16T20:45:25.183 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/38' 2023-10-16T20:45:25.183 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/11' 2023-10-16T20:45:25.183 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/16' 2023-10-16T20:45:25.183 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/32' 2023-10-16T20:45:25.183 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/30' 2023-10-16T20:45:25.184 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/14' 2023-10-16T20:45:25.184 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/18' 2023-10-16T20:45:25.184 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/13' 2023-10-16T20:45:25.184 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/33' 2023-10-16T20:45:25.184 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/17' 2023-10-16T20:45:25.185 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/35' 2023-10-16T20:45:25.185 INFO:tasks.workunit.client.0.smithi088.stdout:removed './122/37' 2023-10-16T20:45:25.185 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './122' 2023-10-16T20:45:25.185 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/19' 2023-10-16T20:45:25.185 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/15' 2023-10-16T20:45:25.186 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/31' 2023-10-16T20:45:25.186 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/36' 2023-10-16T20:45:25.186 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/20' 2023-10-16T20:45:25.186 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/12' 2023-10-16T20:45:25.186 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/39' 2023-10-16T20:45:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/10' 2023-10-16T20:45:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/34' 2023-10-16T20:45:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/38' 2023-10-16T20:45:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/11' 2023-10-16T20:45:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/16' 2023-10-16T20:45:25.188 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/32' 2023-10-16T20:45:25.188 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/30' 2023-10-16T20:45:25.188 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/14' 2023-10-16T20:45:25.188 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/18' 2023-10-16T20:45:25.188 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/13' 2023-10-16T20:45:25.188 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/33' 2023-10-16T20:45:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/17' 2023-10-16T20:45:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/35' 2023-10-16T20:45:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:removed './123/37' 2023-10-16T20:45:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './123' 2023-10-16T20:45:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/19' 2023-10-16T20:45:25.190 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/15' 2023-10-16T20:45:25.190 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/31' 2023-10-16T20:45:25.190 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/36' 2023-10-16T20:45:25.190 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/20' 2023-10-16T20:45:25.190 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/12' 2023-10-16T20:45:25.191 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/39' 2023-10-16T20:45:25.191 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/10' 2023-10-16T20:45:25.191 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/34' 2023-10-16T20:45:25.191 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/38' 2023-10-16T20:45:25.191 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/11' 2023-10-16T20:45:25.191 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/16' 2023-10-16T20:45:25.192 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/32' 2023-10-16T20:45:25.192 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/30' 2023-10-16T20:45:25.192 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/14' 2023-10-16T20:45:25.192 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/18' 2023-10-16T20:45:25.192 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/13' 2023-10-16T20:45:25.193 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/33' 2023-10-16T20:45:25.193 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/17' 2023-10-16T20:45:25.193 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/35' 2023-10-16T20:45:25.193 INFO:tasks.workunit.client.0.smithi088.stdout:removed './124/37' 2023-10-16T20:45:25.193 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './124' 2023-10-16T20:45:25.193 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/19' 2023-10-16T20:45:25.194 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/15' 2023-10-16T20:45:25.194 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/31' 2023-10-16T20:45:25.194 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/36' 2023-10-16T20:45:25.194 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/20' 2023-10-16T20:45:25.194 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/12' 2023-10-16T20:45:25.195 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/39' 2023-10-16T20:45:25.195 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/10' 2023-10-16T20:45:25.195 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/34' 2023-10-16T20:45:25.195 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/38' 2023-10-16T20:45:25.195 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/11' 2023-10-16T20:45:25.196 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/16' 2023-10-16T20:45:25.196 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/32' 2023-10-16T20:45:25.196 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/30' 2023-10-16T20:45:25.196 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/14' 2023-10-16T20:45:25.196 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/18' 2023-10-16T20:45:25.196 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/13' 2023-10-16T20:45:25.197 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/33' 2023-10-16T20:45:25.197 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/17' 2023-10-16T20:45:25.197 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/35' 2023-10-16T20:45:25.197 INFO:tasks.workunit.client.0.smithi088.stdout:removed './125/37' 2023-10-16T20:45:25.197 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './125' 2023-10-16T20:45:25.198 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/19' 2023-10-16T20:45:25.198 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/15' 2023-10-16T20:45:25.198 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/31' 2023-10-16T20:45:25.198 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/36' 2023-10-16T20:45:25.198 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/20' 2023-10-16T20:45:25.199 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/12' 2023-10-16T20:45:25.199 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/39' 2023-10-16T20:45:25.199 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/10' 2023-10-16T20:45:25.199 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/34' 2023-10-16T20:45:25.199 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/38' 2023-10-16T20:45:25.199 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/11' 2023-10-16T20:45:25.200 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/16' 2023-10-16T20:45:25.200 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/32' 2023-10-16T20:45:25.200 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/30' 2023-10-16T20:45:25.200 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/14' 2023-10-16T20:45:25.200 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/18' 2023-10-16T20:45:25.201 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/13' 2023-10-16T20:45:25.201 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/33' 2023-10-16T20:45:25.201 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/17' 2023-10-16T20:45:25.201 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/35' 2023-10-16T20:45:25.201 INFO:tasks.workunit.client.0.smithi088.stdout:removed './126/37' 2023-10-16T20:45:25.201 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './126' 2023-10-16T20:45:25.202 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/19' 2023-10-16T20:45:25.202 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/15' 2023-10-16T20:45:25.202 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/31' 2023-10-16T20:45:25.202 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/36' 2023-10-16T20:45:25.202 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/20' 2023-10-16T20:45:25.203 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/12' 2023-10-16T20:45:25.203 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/39' 2023-10-16T20:45:25.203 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/10' 2023-10-16T20:45:25.203 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/34' 2023-10-16T20:45:25.203 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/38' 2023-10-16T20:45:25.204 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/11' 2023-10-16T20:45:25.204 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/16' 2023-10-16T20:45:25.204 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/32' 2023-10-16T20:45:25.204 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/30' 2023-10-16T20:45:25.204 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/14' 2023-10-16T20:45:25.204 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/18' 2023-10-16T20:45:25.205 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/13' 2023-10-16T20:45:25.205 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/33' 2023-10-16T20:45:25.205 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/17' 2023-10-16T20:45:25.205 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/35' 2023-10-16T20:45:25.205 INFO:tasks.workunit.client.0.smithi088.stdout:removed './127/37' 2023-10-16T20:45:25.206 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './127' 2023-10-16T20:45:25.206 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/19' 2023-10-16T20:45:25.206 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/15' 2023-10-16T20:45:25.206 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/31' 2023-10-16T20:45:25.206 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/36' 2023-10-16T20:45:25.207 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/20' 2023-10-16T20:45:25.207 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/12' 2023-10-16T20:45:25.207 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/39' 2023-10-16T20:45:25.207 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/10' 2023-10-16T20:45:25.207 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/34' 2023-10-16T20:45:25.207 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/38' 2023-10-16T20:45:25.208 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/11' 2023-10-16T20:45:25.208 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/16' 2023-10-16T20:45:25.208 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/32' 2023-10-16T20:45:25.208 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/30' 2023-10-16T20:45:25.208 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/14' 2023-10-16T20:45:25.209 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/18' 2023-10-16T20:45:25.209 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/13' 2023-10-16T20:45:25.209 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/33' 2023-10-16T20:45:25.209 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/17' 2023-10-16T20:45:25.209 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/35' 2023-10-16T20:45:25.901 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:45:25.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:45:26.925 INFO:tasks.workunit.client.0.smithi088.stdout:removed './128/37' 2023-10-16T20:45:26.925 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './128' 2023-10-16T20:45:26.925 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/19' 2023-10-16T20:45:26.926 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/15' 2023-10-16T20:45:26.926 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/31' 2023-10-16T20:45:26.926 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/36' 2023-10-16T20:45:26.926 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/20' 2023-10-16T20:45:26.926 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/12' 2023-10-16T20:45:26.927 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/39' 2023-10-16T20:45:26.927 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/10' 2023-10-16T20:45:26.927 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/34' 2023-10-16T20:45:26.927 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/38' 2023-10-16T20:45:26.927 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/11' 2023-10-16T20:45:26.928 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/16' 2023-10-16T20:45:26.928 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/32' 2023-10-16T20:45:26.928 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/30' 2023-10-16T20:45:26.928 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/14' 2023-10-16T20:45:26.928 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/18' 2023-10-16T20:45:26.928 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/13' 2023-10-16T20:45:26.929 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/33' 2023-10-16T20:45:26.929 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/17' 2023-10-16T20:45:26.929 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/35' 2023-10-16T20:45:26.929 INFO:tasks.workunit.client.0.smithi088.stdout:removed './129/37' 2023-10-16T20:45:26.929 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './129' 2023-10-16T20:45:26.929 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/19' 2023-10-16T20:45:26.930 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/15' 2023-10-16T20:45:26.930 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/31' 2023-10-16T20:45:26.930 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/36' 2023-10-16T20:45:26.930 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/20' 2023-10-16T20:45:26.930 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/12' 2023-10-16T20:45:26.931 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/39' 2023-10-16T20:45:26.931 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/10' 2023-10-16T20:45:26.931 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/34' 2023-10-16T20:45:26.931 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/38' 2023-10-16T20:45:26.931 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/11' 2023-10-16T20:45:26.931 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/16' 2023-10-16T20:45:26.932 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/32' 2023-10-16T20:45:26.932 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/30' 2023-10-16T20:45:26.932 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/14' 2023-10-16T20:45:26.932 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/18' 2023-10-16T20:45:26.932 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/13' 2023-10-16T20:45:26.932 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/33' 2023-10-16T20:45:26.933 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/17' 2023-10-16T20:45:26.933 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/35' 2023-10-16T20:45:26.933 INFO:tasks.workunit.client.0.smithi088.stdout:removed './130/37' 2023-10-16T20:45:26.933 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './130' 2023-10-16T20:45:26.933 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/19' 2023-10-16T20:45:26.934 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/15' 2023-10-16T20:45:26.934 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/31' 2023-10-16T20:45:26.934 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/36' 2023-10-16T20:45:26.934 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/20' 2023-10-16T20:45:26.934 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/12' 2023-10-16T20:45:26.935 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/39' 2023-10-16T20:45:26.935 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/10' 2023-10-16T20:45:26.935 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/34' 2023-10-16T20:45:26.935 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/38' 2023-10-16T20:45:26.935 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/11' 2023-10-16T20:45:26.936 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/16' 2023-10-16T20:45:26.936 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/32' 2023-10-16T20:45:26.936 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/30' 2023-10-16T20:45:26.936 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/14' 2023-10-16T20:45:26.936 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/18' 2023-10-16T20:45:26.936 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/13' 2023-10-16T20:45:26.937 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/33' 2023-10-16T20:45:26.937 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/17' 2023-10-16T20:45:26.937 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/35' 2023-10-16T20:45:26.937 INFO:tasks.workunit.client.0.smithi088.stdout:removed './131/37' 2023-10-16T20:45:26.937 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './131' 2023-10-16T20:45:26.937 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/19' 2023-10-16T20:45:26.938 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/15' 2023-10-16T20:45:26.938 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/31' 2023-10-16T20:45:26.938 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/36' 2023-10-16T20:45:26.938 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/20' 2023-10-16T20:45:26.938 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/12' 2023-10-16T20:45:26.939 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/39' 2023-10-16T20:45:26.939 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/10' 2023-10-16T20:45:26.939 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/34' 2023-10-16T20:45:26.939 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/38' 2023-10-16T20:45:26.939 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/11' 2023-10-16T20:45:26.939 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/16' 2023-10-16T20:45:26.940 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/32' 2023-10-16T20:45:26.940 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/30' 2023-10-16T20:45:26.940 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/14' 2023-10-16T20:45:26.940 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/18' 2023-10-16T20:45:26.940 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/13' 2023-10-16T20:45:26.941 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/33' 2023-10-16T20:45:26.941 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/17' 2023-10-16T20:45:26.941 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/35' 2023-10-16T20:45:26.941 INFO:tasks.workunit.client.0.smithi088.stdout:removed './132/37' 2023-10-16T20:45:26.941 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './132' 2023-10-16T20:45:26.941 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/19' 2023-10-16T20:45:26.942 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/15' 2023-10-16T20:45:26.942 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/31' 2023-10-16T20:45:26.942 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/36' 2023-10-16T20:45:26.942 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/20' 2023-10-16T20:45:26.942 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/12' 2023-10-16T20:45:26.942 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/39' 2023-10-16T20:45:26.943 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/10' 2023-10-16T20:45:26.943 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/34' 2023-10-16T20:45:26.943 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/38' 2023-10-16T20:45:26.943 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/11' 2023-10-16T20:45:26.943 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/16' 2023-10-16T20:45:26.943 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/32' 2023-10-16T20:45:26.944 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/30' 2023-10-16T20:45:26.944 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/14' 2023-10-16T20:45:26.944 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/18' 2023-10-16T20:45:26.944 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/13' 2023-10-16T20:45:26.944 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/33' 2023-10-16T20:45:26.944 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/17' 2023-10-16T20:45:26.945 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/35' 2023-10-16T20:45:26.945 INFO:tasks.workunit.client.0.smithi088.stdout:removed './133/37' 2023-10-16T20:45:26.945 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './133' 2023-10-16T20:45:26.945 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/19' 2023-10-16T20:45:26.945 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/15' 2023-10-16T20:45:26.946 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/31' 2023-10-16T20:45:26.946 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/36' 2023-10-16T20:45:26.946 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/20' 2023-10-16T20:45:26.946 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/12' 2023-10-16T20:45:26.946 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/39' 2023-10-16T20:45:26.946 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/10' 2023-10-16T20:45:26.947 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/34' 2023-10-16T20:45:26.947 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/38' 2023-10-16T20:45:26.947 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/11' 2023-10-16T20:45:26.947 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/16' 2023-10-16T20:45:26.947 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/32' 2023-10-16T20:45:26.947 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/30' 2023-10-16T20:45:26.948 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/14' 2023-10-16T20:45:26.948 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/18' 2023-10-16T20:45:26.948 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/13' 2023-10-16T20:45:26.948 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/33' 2023-10-16T20:45:26.948 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/17' 2023-10-16T20:45:26.949 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/35' 2023-10-16T20:45:26.949 INFO:tasks.workunit.client.0.smithi088.stdout:removed './134/37' 2023-10-16T20:45:26.949 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './134' 2023-10-16T20:45:26.949 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/19' 2023-10-16T20:45:26.949 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/15' 2023-10-16T20:45:26.949 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/31' 2023-10-16T20:45:26.950 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/36' 2023-10-16T20:45:26.950 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/20' 2023-10-16T20:45:26.950 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/12' 2023-10-16T20:45:26.950 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/39' 2023-10-16T20:45:26.950 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/10' 2023-10-16T20:45:26.950 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/34' 2023-10-16T20:45:26.951 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/38' 2023-10-16T20:45:26.951 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/11' 2023-10-16T20:45:26.951 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/16' 2023-10-16T20:45:26.951 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/32' 2023-10-16T20:45:26.951 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/30' 2023-10-16T20:45:26.952 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/14' 2023-10-16T20:45:26.952 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/18' 2023-10-16T20:45:26.952 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/13' 2023-10-16T20:45:26.952 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/33' 2023-10-16T20:45:26.952 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/17' 2023-10-16T20:45:26.952 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/35' 2023-10-16T20:45:26.953 INFO:tasks.workunit.client.0.smithi088.stdout:removed './135/37' 2023-10-16T20:45:26.953 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './135' 2023-10-16T20:45:26.953 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/19' 2023-10-16T20:45:26.953 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/15' 2023-10-16T20:45:26.953 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/31' 2023-10-16T20:45:26.953 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/36' 2023-10-16T20:45:26.954 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/20' 2023-10-16T20:45:26.954 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/12' 2023-10-16T20:45:26.954 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/39' 2023-10-16T20:45:26.954 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/10' 2023-10-16T20:45:26.954 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/34' 2023-10-16T20:45:26.955 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/38' 2023-10-16T20:45:26.955 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/11' 2023-10-16T20:45:26.955 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/16' 2023-10-16T20:45:26.955 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/32' 2023-10-16T20:45:26.955 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/30' 2023-10-16T20:45:26.955 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/14' 2023-10-16T20:45:26.956 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/18' 2023-10-16T20:45:26.956 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/13' 2023-10-16T20:45:26.956 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/33' 2023-10-16T20:45:26.956 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/17' 2023-10-16T20:45:26.956 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/35' 2023-10-16T20:45:26.956 INFO:tasks.workunit.client.0.smithi088.stdout:removed './136/37' 2023-10-16T20:45:26.956 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './136' 2023-10-16T20:45:26.957 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/19' 2023-10-16T20:45:26.957 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/15' 2023-10-16T20:45:26.957 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/31' 2023-10-16T20:45:26.957 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/36' 2023-10-16T20:45:26.957 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/20' 2023-10-16T20:45:26.958 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/12' 2023-10-16T20:45:26.958 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/39' 2023-10-16T20:45:26.958 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/10' 2023-10-16T20:45:26.958 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/34' 2023-10-16T20:45:26.958 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/38' 2023-10-16T20:45:26.959 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/11' 2023-10-16T20:45:26.959 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/16' 2023-10-16T20:45:26.959 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/32' 2023-10-16T20:45:26.959 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/30' 2023-10-16T20:45:26.959 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/14' 2023-10-16T20:45:26.959 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/18' 2023-10-16T20:45:26.960 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/13' 2023-10-16T20:45:26.960 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/33' 2023-10-16T20:45:26.960 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/17' 2023-10-16T20:45:26.960 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/35' 2023-10-16T20:45:26.960 INFO:tasks.workunit.client.0.smithi088.stdout:removed './137/37' 2023-10-16T20:45:26.960 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './137' 2023-10-16T20:45:26.961 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/19' 2023-10-16T20:45:26.961 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/15' 2023-10-16T20:45:26.961 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/31' 2023-10-16T20:45:26.961 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/36' 2023-10-16T20:45:26.961 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/20' 2023-10-16T20:45:26.962 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/12' 2023-10-16T20:45:26.962 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/39' 2023-10-16T20:45:26.962 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/10' 2023-10-16T20:45:26.962 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/34' 2023-10-16T20:45:26.962 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/38' 2023-10-16T20:45:26.963 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/11' 2023-10-16T20:45:26.963 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/16' 2023-10-16T20:45:28.357 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/32' 2023-10-16T20:45:28.358 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/30' 2023-10-16T20:45:28.358 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/14' 2023-10-16T20:45:28.358 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/18' 2023-10-16T20:45:28.358 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/13' 2023-10-16T20:45:28.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/33' 2023-10-16T20:45:28.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/17' 2023-10-16T20:45:28.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/35' 2023-10-16T20:45:28.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './138/37' 2023-10-16T20:45:28.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './138' 2023-10-16T20:45:28.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/19' 2023-10-16T20:45:28.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/15' 2023-10-16T20:45:28.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/31' 2023-10-16T20:45:28.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/36' 2023-10-16T20:45:28.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/20' 2023-10-16T20:45:28.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/12' 2023-10-16T20:45:28.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/39' 2023-10-16T20:45:28.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/10' 2023-10-16T20:45:28.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/34' 2023-10-16T20:45:28.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/38' 2023-10-16T20:45:28.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/11' 2023-10-16T20:45:28.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/16' 2023-10-16T20:45:28.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/32' 2023-10-16T20:45:28.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/30' 2023-10-16T20:45:28.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/14' 2023-10-16T20:45:28.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/18' 2023-10-16T20:45:28.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/13' 2023-10-16T20:45:28.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/33' 2023-10-16T20:45:28.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/17' 2023-10-16T20:45:28.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/35' 2023-10-16T20:45:28.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed './139/37' 2023-10-16T20:45:28.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './139' 2023-10-16T20:45:28.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/19' 2023-10-16T20:45:28.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/15' 2023-10-16T20:45:28.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/31' 2023-10-16T20:45:28.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/36' 2023-10-16T20:45:28.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/20' 2023-10-16T20:45:28.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/12' 2023-10-16T20:45:28.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/39' 2023-10-16T20:45:28.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/10' 2023-10-16T20:45:28.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/34' 2023-10-16T20:45:28.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/38' 2023-10-16T20:45:28.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/11' 2023-10-16T20:45:28.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/16' 2023-10-16T20:45:28.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/32' 2023-10-16T20:45:28.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/30' 2023-10-16T20:45:28.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/14' 2023-10-16T20:45:28.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/18' 2023-10-16T20:45:28.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/13' 2023-10-16T20:45:28.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/33' 2023-10-16T20:45:28.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/17' 2023-10-16T20:45:28.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/35' 2023-10-16T20:45:28.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed './140/37' 2023-10-16T20:45:28.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './140' 2023-10-16T20:45:28.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/19' 2023-10-16T20:45:28.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/15' 2023-10-16T20:45:28.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/31' 2023-10-16T20:45:28.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/36' 2023-10-16T20:45:28.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/20' 2023-10-16T20:45:28.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/12' 2023-10-16T20:45:28.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/39' 2023-10-16T20:45:28.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/10' 2023-10-16T20:45:28.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/34' 2023-10-16T20:45:28.370 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/38' 2023-10-16T20:45:28.370 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/11' 2023-10-16T20:45:28.370 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/16' 2023-10-16T20:45:28.370 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/32' 2023-10-16T20:45:28.370 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/30' 2023-10-16T20:45:28.371 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/14' 2023-10-16T20:45:28.371 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/18' 2023-10-16T20:45:28.371 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/13' 2023-10-16T20:45:28.371 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/33' 2023-10-16T20:45:28.371 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/17' 2023-10-16T20:45:28.371 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/35' 2023-10-16T20:45:28.372 INFO:tasks.workunit.client.0.smithi088.stdout:removed './141/37' 2023-10-16T20:45:28.372 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './141' 2023-10-16T20:45:28.372 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/19' 2023-10-16T20:45:28.372 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/15' 2023-10-16T20:45:28.372 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/31' 2023-10-16T20:45:28.372 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/36' 2023-10-16T20:45:28.373 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/20' 2023-10-16T20:45:28.373 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/12' 2023-10-16T20:45:28.373 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/39' 2023-10-16T20:45:28.373 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/10' 2023-10-16T20:45:28.373 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/34' 2023-10-16T20:45:28.373 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/38' 2023-10-16T20:45:28.374 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/11' 2023-10-16T20:45:28.374 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/16' 2023-10-16T20:45:28.374 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/32' 2023-10-16T20:45:28.374 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/30' 2023-10-16T20:45:28.374 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/14' 2023-10-16T20:45:28.375 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/18' 2023-10-16T20:45:28.375 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/13' 2023-10-16T20:45:28.375 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/33' 2023-10-16T20:45:28.375 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/17' 2023-10-16T20:45:28.375 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/35' 2023-10-16T20:45:28.375 INFO:tasks.workunit.client.0.smithi088.stdout:removed './142/37' 2023-10-16T20:45:28.376 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './142' 2023-10-16T20:45:28.376 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/19' 2023-10-16T20:45:28.376 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/15' 2023-10-16T20:45:28.376 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/31' 2023-10-16T20:45:28.376 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/36' 2023-10-16T20:45:28.376 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/20' 2023-10-16T20:45:28.377 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/12' 2023-10-16T20:45:28.377 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/39' 2023-10-16T20:45:28.377 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/10' 2023-10-16T20:45:28.377 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/34' 2023-10-16T20:45:28.377 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/38' 2023-10-16T20:45:28.378 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/11' 2023-10-16T20:45:28.378 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/16' 2023-10-16T20:45:28.378 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/32' 2023-10-16T20:45:28.378 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/30' 2023-10-16T20:45:28.378 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/14' 2023-10-16T20:45:28.378 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/18' 2023-10-16T20:45:28.379 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/13' 2023-10-16T20:45:28.379 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/33' 2023-10-16T20:45:28.379 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/17' 2023-10-16T20:45:28.379 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/35' 2023-10-16T20:45:28.379 INFO:tasks.workunit.client.0.smithi088.stdout:removed './143/37' 2023-10-16T20:45:28.379 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './143' 2023-10-16T20:45:28.380 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/19' 2023-10-16T20:45:28.380 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/15' 2023-10-16T20:45:28.380 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/31' 2023-10-16T20:45:28.380 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/36' 2023-10-16T20:45:28.380 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/20' 2023-10-16T20:45:28.381 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/12' 2023-10-16T20:45:28.381 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/39' 2023-10-16T20:45:28.381 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/10' 2023-10-16T20:45:28.381 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/34' 2023-10-16T20:45:28.381 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/38' 2023-10-16T20:45:28.381 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/11' 2023-10-16T20:45:28.382 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/16' 2023-10-16T20:45:28.382 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/32' 2023-10-16T20:45:28.382 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/30' 2023-10-16T20:45:28.382 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/14' 2023-10-16T20:45:28.383 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/18' 2023-10-16T20:45:28.383 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/13' 2023-10-16T20:45:28.383 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/33' 2023-10-16T20:45:28.383 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/17' 2023-10-16T20:45:28.383 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/35' 2023-10-16T20:45:28.383 INFO:tasks.workunit.client.0.smithi088.stdout:removed './144/37' 2023-10-16T20:45:28.384 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './144' 2023-10-16T20:45:28.384 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/19' 2023-10-16T20:45:28.384 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/15' 2023-10-16T20:45:28.384 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/31' 2023-10-16T20:45:28.384 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/36' 2023-10-16T20:45:28.384 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/20' 2023-10-16T20:45:28.385 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/12' 2023-10-16T20:45:28.385 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/39' 2023-10-16T20:45:28.385 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/10' 2023-10-16T20:45:28.385 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/34' 2023-10-16T20:45:28.385 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/38' 2023-10-16T20:45:28.385 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/11' 2023-10-16T20:45:28.386 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/16' 2023-10-16T20:45:28.386 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/32' 2023-10-16T20:45:28.386 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/30' 2023-10-16T20:45:28.386 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/14' 2023-10-16T20:45:28.386 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/18' 2023-10-16T20:45:28.387 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/13' 2023-10-16T20:45:28.387 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/33' 2023-10-16T20:45:28.387 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/17' 2023-10-16T20:45:28.387 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/35' 2023-10-16T20:45:28.387 INFO:tasks.workunit.client.0.smithi088.stdout:removed './145/37' 2023-10-16T20:45:28.388 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './145' 2023-10-16T20:45:28.388 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/19' 2023-10-16T20:45:28.388 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/15' 2023-10-16T20:45:28.388 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/31' 2023-10-16T20:45:28.388 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/36' 2023-10-16T20:45:28.388 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/20' 2023-10-16T20:45:28.389 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/12' 2023-10-16T20:45:28.389 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/39' 2023-10-16T20:45:28.389 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/10' 2023-10-16T20:45:28.389 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/34' 2023-10-16T20:45:28.389 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/38' 2023-10-16T20:45:28.390 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/11' 2023-10-16T20:45:28.390 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/16' 2023-10-16T20:45:28.390 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/32' 2023-10-16T20:45:28.390 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/30' 2023-10-16T20:45:28.390 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/14' 2023-10-16T20:45:28.391 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/18' 2023-10-16T20:45:28.391 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/13' 2023-10-16T20:45:28.391 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/33' 2023-10-16T20:45:28.391 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/17' 2023-10-16T20:45:28.391 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/35' 2023-10-16T20:45:28.392 INFO:tasks.workunit.client.0.smithi088.stdout:removed './146/37' 2023-10-16T20:45:28.392 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './146' 2023-10-16T20:45:28.392 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/19' 2023-10-16T20:45:28.392 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/15' 2023-10-16T20:45:28.392 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/31' 2023-10-16T20:45:28.393 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/36' 2023-10-16T20:45:28.393 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/20' 2023-10-16T20:45:28.393 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/12' 2023-10-16T20:45:28.393 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/39' 2023-10-16T20:45:28.393 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/10' 2023-10-16T20:45:28.394 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/34' 2023-10-16T20:45:28.394 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/38' 2023-10-16T20:45:28.394 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/11' 2023-10-16T20:45:28.394 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/16' 2023-10-16T20:45:28.394 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/32' 2023-10-16T20:45:28.395 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/30' 2023-10-16T20:45:28.395 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/14' 2023-10-16T20:45:28.395 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/18' 2023-10-16T20:45:28.395 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/13' 2023-10-16T20:45:28.398 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/33' 2023-10-16T20:45:28.398 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/17' 2023-10-16T20:45:28.398 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/35' 2023-10-16T20:45:28.398 INFO:tasks.workunit.client.0.smithi088.stdout:removed './147/37' 2023-10-16T20:45:28.399 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './147' 2023-10-16T20:45:28.399 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/19' 2023-10-16T20:45:28.399 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/15' 2023-10-16T20:45:28.399 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/31' 2023-10-16T20:45:28.399 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/36' 2023-10-16T20:45:29.345 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/20' 2023-10-16T20:45:29.345 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/12' 2023-10-16T20:45:29.345 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/39' 2023-10-16T20:45:29.345 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/10' 2023-10-16T20:45:29.346 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/34' 2023-10-16T20:45:29.346 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/38' 2023-10-16T20:45:29.346 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/11' 2023-10-16T20:45:29.346 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/16' 2023-10-16T20:45:29.346 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/32' 2023-10-16T20:45:29.347 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/30' 2023-10-16T20:45:29.347 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/14' 2023-10-16T20:45:29.347 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/18' 2023-10-16T20:45:29.347 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/13' 2023-10-16T20:45:29.347 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/33' 2023-10-16T20:45:29.347 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/17' 2023-10-16T20:45:29.348 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/35' 2023-10-16T20:45:29.348 INFO:tasks.workunit.client.0.smithi088.stdout:removed './148/37' 2023-10-16T20:45:29.348 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './148' 2023-10-16T20:45:29.348 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/19' 2023-10-16T20:45:29.348 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/15' 2023-10-16T20:45:29.349 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/31' 2023-10-16T20:45:29.349 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/36' 2023-10-16T20:45:29.349 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/20' 2023-10-16T20:45:29.349 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/12' 2023-10-16T20:45:29.349 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/39' 2023-10-16T20:45:29.350 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/10' 2023-10-16T20:45:29.350 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/34' 2023-10-16T20:45:29.350 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/38' 2023-10-16T20:45:29.350 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/11' 2023-10-16T20:45:29.350 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/16' 2023-10-16T20:45:29.351 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/32' 2023-10-16T20:45:29.351 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/30' 2023-10-16T20:45:29.351 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/14' 2023-10-16T20:45:29.351 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/18' 2023-10-16T20:45:29.351 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/13' 2023-10-16T20:45:29.351 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/33' 2023-10-16T20:45:29.352 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/17' 2023-10-16T20:45:29.352 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/35' 2023-10-16T20:45:29.352 INFO:tasks.workunit.client.0.smithi088.stdout:removed './149/37' 2023-10-16T20:45:29.352 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './149' 2023-10-16T20:45:29.352 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/19' 2023-10-16T20:45:29.353 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/15' 2023-10-16T20:45:29.353 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/31' 2023-10-16T20:45:29.353 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/36' 2023-10-16T20:45:29.353 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/20' 2023-10-16T20:45:29.353 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/12' 2023-10-16T20:45:29.354 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/39' 2023-10-16T20:45:29.354 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/10' 2023-10-16T20:45:29.354 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/34' 2023-10-16T20:45:29.354 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/38' 2023-10-16T20:45:29.354 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/11' 2023-10-16T20:45:29.354 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/16' 2023-10-16T20:45:29.355 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/32' 2023-10-16T20:45:29.355 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/30' 2023-10-16T20:45:29.355 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/14' 2023-10-16T20:45:29.355 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/18' 2023-10-16T20:45:29.355 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/13' 2023-10-16T20:45:29.356 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/33' 2023-10-16T20:45:29.356 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/17' 2023-10-16T20:45:29.356 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/35' 2023-10-16T20:45:29.356 INFO:tasks.workunit.client.0.smithi088.stdout:removed './150/37' 2023-10-16T20:45:29.356 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './150' 2023-10-16T20:45:29.356 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/19' 2023-10-16T20:45:29.356 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/15' 2023-10-16T20:45:29.357 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/31' 2023-10-16T20:45:29.357 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/36' 2023-10-16T20:45:29.357 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/20' 2023-10-16T20:45:29.357 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/12' 2023-10-16T20:45:29.357 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/39' 2023-10-16T20:45:29.358 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/10' 2023-10-16T20:45:29.358 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/34' 2023-10-16T20:45:29.358 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/38' 2023-10-16T20:45:29.358 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/11' 2023-10-16T20:45:29.358 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/16' 2023-10-16T20:45:29.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/32' 2023-10-16T20:45:29.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/30' 2023-10-16T20:45:29.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/14' 2023-10-16T20:45:29.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/18' 2023-10-16T20:45:29.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/13' 2023-10-16T20:45:29.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/33' 2023-10-16T20:45:29.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/17' 2023-10-16T20:45:29.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/35' 2023-10-16T20:45:29.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed './151/37' 2023-10-16T20:45:29.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './151' 2023-10-16T20:45:29.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/19' 2023-10-16T20:45:29.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/15' 2023-10-16T20:45:29.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/31' 2023-10-16T20:45:29.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/36' 2023-10-16T20:45:29.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/20' 2023-10-16T20:45:29.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/12' 2023-10-16T20:45:29.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/39' 2023-10-16T20:45:29.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/10' 2023-10-16T20:45:29.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/34' 2023-10-16T20:45:29.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/38' 2023-10-16T20:45:29.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/11' 2023-10-16T20:45:29.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/16' 2023-10-16T20:45:29.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/32' 2023-10-16T20:45:29.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/30' 2023-10-16T20:45:29.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/14' 2023-10-16T20:45:29.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/18' 2023-10-16T20:45:29.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/13' 2023-10-16T20:45:29.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/33' 2023-10-16T20:45:29.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/17' 2023-10-16T20:45:29.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/35' 2023-10-16T20:45:29.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed './152/37' 2023-10-16T20:45:29.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './152' 2023-10-16T20:45:29.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/19' 2023-10-16T20:45:29.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/15' 2023-10-16T20:45:29.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/31' 2023-10-16T20:45:29.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/36' 2023-10-16T20:45:29.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/20' 2023-10-16T20:45:29.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/12' 2023-10-16T20:45:29.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/39' 2023-10-16T20:45:29.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/10' 2023-10-16T20:45:29.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/34' 2023-10-16T20:45:29.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/38' 2023-10-16T20:45:29.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/11' 2023-10-16T20:45:29.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/16' 2023-10-16T20:45:29.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/32' 2023-10-16T20:45:29.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/30' 2023-10-16T20:45:29.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/14' 2023-10-16T20:45:29.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/18' 2023-10-16T20:45:29.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/13' 2023-10-16T20:45:29.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/33' 2023-10-16T20:45:29.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/17' 2023-10-16T20:45:29.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/35' 2023-10-16T20:45:29.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed './153/37' 2023-10-16T20:45:29.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './153' 2023-10-16T20:45:29.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/19' 2023-10-16T20:45:29.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/15' 2023-10-16T20:45:29.370 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/31' 2023-10-16T20:45:29.370 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/36' 2023-10-16T20:45:29.370 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/20' 2023-10-16T20:45:29.370 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/12' 2023-10-16T20:45:29.371 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/39' 2023-10-16T20:45:29.371 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/10' 2023-10-16T20:45:29.371 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/34' 2023-10-16T20:45:29.371 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/38' 2023-10-16T20:45:29.371 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/11' 2023-10-16T20:45:29.372 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/16' 2023-10-16T20:45:29.372 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/32' 2023-10-16T20:45:29.372 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/30' 2023-10-16T20:45:29.372 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/14' 2023-10-16T20:45:29.372 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/18' 2023-10-16T20:45:29.373 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/13' 2023-10-16T20:45:29.373 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/33' 2023-10-16T20:45:29.373 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/17' 2023-10-16T20:45:29.373 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/35' 2023-10-16T20:45:29.373 INFO:tasks.workunit.client.0.smithi088.stdout:removed './154/37' 2023-10-16T20:45:29.374 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './154' 2023-10-16T20:45:29.374 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/19' 2023-10-16T20:45:29.374 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/15' 2023-10-16T20:45:29.374 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/31' 2023-10-16T20:45:29.375 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/36' 2023-10-16T20:45:29.375 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/20' 2023-10-16T20:45:29.375 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/12' 2023-10-16T20:45:29.375 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/39' 2023-10-16T20:45:29.376 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/10' 2023-10-16T20:45:29.376 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/34' 2023-10-16T20:45:29.376 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/38' 2023-10-16T20:45:29.376 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/11' 2023-10-16T20:45:29.376 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/16' 2023-10-16T20:45:29.377 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/32' 2023-10-16T20:45:29.377 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/30' 2023-10-16T20:45:29.377 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/14' 2023-10-16T20:45:29.377 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/18' 2023-10-16T20:45:29.377 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/13' 2023-10-16T20:45:29.378 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/33' 2023-10-16T20:45:29.378 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/17' 2023-10-16T20:45:29.378 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/35' 2023-10-16T20:45:29.378 INFO:tasks.workunit.client.0.smithi088.stdout:removed './155/37' 2023-10-16T20:45:29.379 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './155' 2023-10-16T20:45:29.379 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/19' 2023-10-16T20:45:29.379 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/15' 2023-10-16T20:45:29.379 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/31' 2023-10-16T20:45:29.379 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/36' 2023-10-16T20:45:29.380 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/20' 2023-10-16T20:45:29.380 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/12' 2023-10-16T20:45:29.380 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/39' 2023-10-16T20:45:29.380 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/10' 2023-10-16T20:45:29.380 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/34' 2023-10-16T20:45:29.381 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/38' 2023-10-16T20:45:29.381 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/11' 2023-10-16T20:45:29.381 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/16' 2023-10-16T20:45:29.381 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/32' 2023-10-16T20:45:29.381 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/30' 2023-10-16T20:45:29.382 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/14' 2023-10-16T20:45:29.382 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/18' 2023-10-16T20:45:29.382 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/13' 2023-10-16T20:45:29.382 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/33' 2023-10-16T20:45:29.382 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/17' 2023-10-16T20:45:29.383 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/35' 2023-10-16T20:45:29.383 INFO:tasks.workunit.client.0.smithi088.stdout:removed './156/37' 2023-10-16T20:45:29.383 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './156' 2023-10-16T20:45:29.383 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/19' 2023-10-16T20:45:29.383 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/15' 2023-10-16T20:45:29.384 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/31' 2023-10-16T20:45:29.384 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/36' 2023-10-16T20:45:29.384 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/20' 2023-10-16T20:45:29.384 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/12' 2023-10-16T20:45:29.384 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/39' 2023-10-16T20:45:29.384 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/10' 2023-10-16T20:45:29.385 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/34' 2023-10-16T20:45:29.385 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/38' 2023-10-16T20:45:29.385 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/11' 2023-10-16T20:45:29.385 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/16' 2023-10-16T20:45:29.385 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/32' 2023-10-16T20:45:29.386 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/30' 2023-10-16T20:45:29.386 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/14' 2023-10-16T20:45:29.386 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/18' 2023-10-16T20:45:29.386 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/13' 2023-10-16T20:45:29.386 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/33' 2023-10-16T20:45:32.014 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/17' 2023-10-16T20:45:32.014 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/35' 2023-10-16T20:45:32.014 INFO:tasks.workunit.client.0.smithi088.stdout:removed './157/37' 2023-10-16T20:45:32.015 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './157' 2023-10-16T20:45:32.015 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/19' 2023-10-16T20:45:32.015 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/15' 2023-10-16T20:45:32.015 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/31' 2023-10-16T20:45:32.015 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/36' 2023-10-16T20:45:32.016 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/20' 2023-10-16T20:45:32.016 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/12' 2023-10-16T20:45:32.016 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/39' 2023-10-16T20:45:32.016 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/10' 2023-10-16T20:45:32.016 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/34' 2023-10-16T20:45:32.016 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/38' 2023-10-16T20:45:32.017 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/11' 2023-10-16T20:45:32.017 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/16' 2023-10-16T20:45:32.017 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/32' 2023-10-16T20:45:32.017 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/30' 2023-10-16T20:45:32.017 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/14' 2023-10-16T20:45:32.017 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/18' 2023-10-16T20:45:32.018 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/13' 2023-10-16T20:45:32.018 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/33' 2023-10-16T20:45:32.018 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/17' 2023-10-16T20:45:32.018 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/35' 2023-10-16T20:45:32.018 INFO:tasks.workunit.client.0.smithi088.stdout:removed './158/37' 2023-10-16T20:45:32.019 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './158' 2023-10-16T20:45:32.019 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/19' 2023-10-16T20:45:32.019 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/15' 2023-10-16T20:45:32.019 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/31' 2023-10-16T20:45:32.019 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/36' 2023-10-16T20:45:32.019 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/20' 2023-10-16T20:45:32.020 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/12' 2023-10-16T20:45:32.020 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/39' 2023-10-16T20:45:32.020 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/10' 2023-10-16T20:45:32.020 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/34' 2023-10-16T20:45:32.020 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/38' 2023-10-16T20:45:32.021 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/11' 2023-10-16T20:45:32.021 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/16' 2023-10-16T20:45:32.021 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/32' 2023-10-16T20:45:32.021 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/30' 2023-10-16T20:45:32.021 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/14' 2023-10-16T20:45:32.021 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/18' 2023-10-16T20:45:32.022 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/13' 2023-10-16T20:45:32.022 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/33' 2023-10-16T20:45:32.022 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/17' 2023-10-16T20:45:32.022 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/35' 2023-10-16T20:45:32.022 INFO:tasks.workunit.client.0.smithi088.stdout:removed './159/37' 2023-10-16T20:45:32.022 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './159' 2023-10-16T20:45:32.023 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/19' 2023-10-16T20:45:32.023 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/15' 2023-10-16T20:45:32.023 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/31' 2023-10-16T20:45:32.023 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/36' 2023-10-16T20:45:32.023 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/20' 2023-10-16T20:45:32.023 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/12' 2023-10-16T20:45:32.024 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/39' 2023-10-16T20:45:32.024 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/10' 2023-10-16T20:45:32.024 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/34' 2023-10-16T20:45:32.024 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/38' 2023-10-16T20:45:32.024 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/11' 2023-10-16T20:45:32.025 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/16' 2023-10-16T20:45:32.025 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/32' 2023-10-16T20:45:32.025 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/30' 2023-10-16T20:45:32.025 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/14' 2023-10-16T20:45:32.025 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/18' 2023-10-16T20:45:32.025 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/13' 2023-10-16T20:45:32.026 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/33' 2023-10-16T20:45:32.026 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/17' 2023-10-16T20:45:32.026 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/35' 2023-10-16T20:45:32.026 INFO:tasks.workunit.client.0.smithi088.stdout:removed './160/37' 2023-10-16T20:45:32.026 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './160' 2023-10-16T20:45:32.027 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/19' 2023-10-16T20:45:32.027 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/15' 2023-10-16T20:45:32.027 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/31' 2023-10-16T20:45:32.027 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/36' 2023-10-16T20:45:32.027 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/20' 2023-10-16T20:45:32.027 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/12' 2023-10-16T20:45:32.028 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/39' 2023-10-16T20:45:32.028 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/10' 2023-10-16T20:45:32.028 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/34' 2023-10-16T20:45:32.028 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/38' 2023-10-16T20:45:32.028 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/11' 2023-10-16T20:45:32.029 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/16' 2023-10-16T20:45:32.029 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/32' 2023-10-16T20:45:32.029 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/30' 2023-10-16T20:45:32.029 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/14' 2023-10-16T20:45:32.029 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/18' 2023-10-16T20:45:32.029 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/13' 2023-10-16T20:45:32.030 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/33' 2023-10-16T20:45:32.030 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/17' 2023-10-16T20:45:32.030 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/35' 2023-10-16T20:45:32.030 INFO:tasks.workunit.client.0.smithi088.stdout:removed './161/37' 2023-10-16T20:45:32.030 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './161' 2023-10-16T20:45:32.031 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/19' 2023-10-16T20:45:32.031 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/15' 2023-10-16T20:45:32.031 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/31' 2023-10-16T20:45:32.031 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/36' 2023-10-16T20:45:32.031 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/20' 2023-10-16T20:45:32.031 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/12' 2023-10-16T20:45:32.031 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/39' 2023-10-16T20:45:32.032 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/10' 2023-10-16T20:45:32.032 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/34' 2023-10-16T20:45:32.032 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/38' 2023-10-16T20:45:32.032 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/11' 2023-10-16T20:45:32.032 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/16' 2023-10-16T20:45:32.032 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/32' 2023-10-16T20:45:32.033 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/30' 2023-10-16T20:45:32.033 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/14' 2023-10-16T20:45:32.033 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/18' 2023-10-16T20:45:32.033 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/13' 2023-10-16T20:45:32.033 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/33' 2023-10-16T20:45:32.034 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/17' 2023-10-16T20:45:32.034 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/35' 2023-10-16T20:45:32.034 INFO:tasks.workunit.client.0.smithi088.stdout:removed './162/37' 2023-10-16T20:45:32.034 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './162' 2023-10-16T20:45:32.034 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/19' 2023-10-16T20:45:32.034 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/15' 2023-10-16T20:45:32.035 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/31' 2023-10-16T20:45:32.035 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/36' 2023-10-16T20:45:32.035 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/20' 2023-10-16T20:45:32.035 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/12' 2023-10-16T20:45:32.035 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/39' 2023-10-16T20:45:32.036 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/10' 2023-10-16T20:45:32.036 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/34' 2023-10-16T20:45:32.036 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/38' 2023-10-16T20:45:32.036 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/11' 2023-10-16T20:45:32.036 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/16' 2023-10-16T20:45:32.036 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/32' 2023-10-16T20:45:32.037 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/30' 2023-10-16T20:45:32.037 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/14' 2023-10-16T20:45:32.037 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/18' 2023-10-16T20:45:32.037 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/13' 2023-10-16T20:45:32.037 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/33' 2023-10-16T20:45:32.038 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/17' 2023-10-16T20:45:32.038 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/35' 2023-10-16T20:45:32.038 INFO:tasks.workunit.client.0.smithi088.stdout:removed './163/37' 2023-10-16T20:45:32.038 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './163' 2023-10-16T20:45:32.038 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/19' 2023-10-16T20:45:32.038 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/15' 2023-10-16T20:45:32.039 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/31' 2023-10-16T20:45:32.039 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/36' 2023-10-16T20:45:32.039 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/20' 2023-10-16T20:45:32.039 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/12' 2023-10-16T20:45:32.039 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/39' 2023-10-16T20:45:32.040 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/10' 2023-10-16T20:45:32.040 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/34' 2023-10-16T20:45:32.040 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/38' 2023-10-16T20:45:32.040 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/11' 2023-10-16T20:45:32.040 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/16' 2023-10-16T20:45:32.040 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/32' 2023-10-16T20:45:32.041 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/30' 2023-10-16T20:45:32.041 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/14' 2023-10-16T20:45:32.041 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/18' 2023-10-16T20:45:32.041 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/13' 2023-10-16T20:45:32.041 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/33' 2023-10-16T20:45:32.042 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/17' 2023-10-16T20:45:32.042 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/35' 2023-10-16T20:45:32.042 INFO:tasks.workunit.client.0.smithi088.stdout:removed './164/37' 2023-10-16T20:45:32.042 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './164' 2023-10-16T20:45:32.042 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/19' 2023-10-16T20:45:32.042 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/15' 2023-10-16T20:45:32.043 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/31' 2023-10-16T20:45:32.043 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/36' 2023-10-16T20:45:32.043 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/20' 2023-10-16T20:45:32.043 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/12' 2023-10-16T20:45:32.043 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/39' 2023-10-16T20:45:32.044 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/10' 2023-10-16T20:45:32.044 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/34' 2023-10-16T20:45:32.044 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/38' 2023-10-16T20:45:32.044 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/11' 2023-10-16T20:45:32.044 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/16' 2023-10-16T20:45:32.044 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/32' 2023-10-16T20:45:32.045 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/30' 2023-10-16T20:45:32.045 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/14' 2023-10-16T20:45:32.045 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/18' 2023-10-16T20:45:32.045 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/13' 2023-10-16T20:45:32.045 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/33' 2023-10-16T20:45:32.045 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/17' 2023-10-16T20:45:32.046 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/35' 2023-10-16T20:45:32.046 INFO:tasks.workunit.client.0.smithi088.stdout:removed './165/37' 2023-10-16T20:45:32.046 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './165' 2023-10-16T20:45:32.046 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/19' 2023-10-16T20:45:32.046 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/15' 2023-10-16T20:45:32.047 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/31' 2023-10-16T20:45:32.047 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/36' 2023-10-16T20:45:32.047 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/20' 2023-10-16T20:45:32.047 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/12' 2023-10-16T20:45:32.047 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/39' 2023-10-16T20:45:32.047 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/10' 2023-10-16T20:45:32.048 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/34' 2023-10-16T20:45:32.048 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/38' 2023-10-16T20:45:32.048 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/11' 2023-10-16T20:45:32.048 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/16' 2023-10-16T20:45:32.048 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/32' 2023-10-16T20:45:32.048 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/30' 2023-10-16T20:45:32.049 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/14' 2023-10-16T20:45:32.049 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/18' 2023-10-16T20:45:32.049 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/13' 2023-10-16T20:45:32.049 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/33' 2023-10-16T20:45:32.049 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/17' 2023-10-16T20:45:32.050 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/35' 2023-10-16T20:45:32.050 INFO:tasks.workunit.client.0.smithi088.stdout:removed './166/37' 2023-10-16T20:45:32.050 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './166' 2023-10-16T20:45:32.050 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/19' 2023-10-16T20:45:32.050 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/15' 2023-10-16T20:45:32.050 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/31' 2023-10-16T20:45:32.051 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/36' 2023-10-16T20:45:32.051 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/20' 2023-10-16T20:45:32.051 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/12' 2023-10-16T20:45:32.051 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/39' 2023-10-16T20:45:32.051 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/10' 2023-10-16T20:45:32.052 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/34' 2023-10-16T20:45:32.052 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/38' 2023-10-16T20:45:34.152 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/11' 2023-10-16T20:45:34.152 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/16' 2023-10-16T20:45:34.152 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/32' 2023-10-16T20:45:34.152 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/30' 2023-10-16T20:45:34.153 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/14' 2023-10-16T20:45:34.153 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/18' 2023-10-16T20:45:34.153 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/13' 2023-10-16T20:45:34.153 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/33' 2023-10-16T20:45:34.153 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/17' 2023-10-16T20:45:34.153 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/35' 2023-10-16T20:45:34.154 INFO:tasks.workunit.client.0.smithi088.stdout:removed './167/37' 2023-10-16T20:45:34.154 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './167' 2023-10-16T20:45:34.154 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/19' 2023-10-16T20:45:34.154 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/15' 2023-10-16T20:45:34.154 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/31' 2023-10-16T20:45:34.155 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/36' 2023-10-16T20:45:34.155 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/20' 2023-10-16T20:45:34.155 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/12' 2023-10-16T20:45:34.155 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/39' 2023-10-16T20:45:34.155 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/10' 2023-10-16T20:45:34.156 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/34' 2023-10-16T20:45:34.156 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/38' 2023-10-16T20:45:34.156 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/11' 2023-10-16T20:45:34.156 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/16' 2023-10-16T20:45:34.156 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/32' 2023-10-16T20:45:34.157 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/30' 2023-10-16T20:45:34.157 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/14' 2023-10-16T20:45:34.157 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/18' 2023-10-16T20:45:34.157 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/13' 2023-10-16T20:45:34.157 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/33' 2023-10-16T20:45:34.157 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/17' 2023-10-16T20:45:34.158 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/35' 2023-10-16T20:45:34.158 INFO:tasks.workunit.client.0.smithi088.stdout:removed './168/37' 2023-10-16T20:45:34.158 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './168' 2023-10-16T20:45:34.158 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/19' 2023-10-16T20:45:34.158 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/15' 2023-10-16T20:45:34.159 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/31' 2023-10-16T20:45:34.159 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/36' 2023-10-16T20:45:34.159 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/20' 2023-10-16T20:45:34.159 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/12' 2023-10-16T20:45:34.159 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/39' 2023-10-16T20:45:34.160 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/10' 2023-10-16T20:45:34.160 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/34' 2023-10-16T20:45:34.160 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/38' 2023-10-16T20:45:34.160 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/11' 2023-10-16T20:45:34.160 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/16' 2023-10-16T20:45:34.160 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/32' 2023-10-16T20:45:34.161 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/30' 2023-10-16T20:45:34.161 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/14' 2023-10-16T20:45:34.161 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/18' 2023-10-16T20:45:34.161 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/13' 2023-10-16T20:45:34.161 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/33' 2023-10-16T20:45:34.161 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/17' 2023-10-16T20:45:34.162 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/35' 2023-10-16T20:45:34.162 INFO:tasks.workunit.client.0.smithi088.stdout:removed './169/37' 2023-10-16T20:45:34.162 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './169' 2023-10-16T20:45:34.162 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/19' 2023-10-16T20:45:34.162 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/15' 2023-10-16T20:45:34.162 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/31' 2023-10-16T20:45:34.163 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/36' 2023-10-16T20:45:34.163 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/20' 2023-10-16T20:45:34.163 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/12' 2023-10-16T20:45:34.163 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/39' 2023-10-16T20:45:34.163 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/10' 2023-10-16T20:45:34.163 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/34' 2023-10-16T20:45:34.164 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/38' 2023-10-16T20:45:34.164 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/11' 2023-10-16T20:45:34.164 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/16' 2023-10-16T20:45:34.164 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/32' 2023-10-16T20:45:34.164 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/30' 2023-10-16T20:45:34.164 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/14' 2023-10-16T20:45:34.165 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/18' 2023-10-16T20:45:34.165 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/13' 2023-10-16T20:45:34.165 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/33' 2023-10-16T20:45:34.165 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/17' 2023-10-16T20:45:34.165 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/35' 2023-10-16T20:45:34.166 INFO:tasks.workunit.client.0.smithi088.stdout:removed './170/37' 2023-10-16T20:45:34.166 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './170' 2023-10-16T20:45:34.166 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/19' 2023-10-16T20:45:34.166 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/15' 2023-10-16T20:45:34.166 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/31' 2023-10-16T20:45:34.167 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/36' 2023-10-16T20:45:34.167 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/20' 2023-10-16T20:45:34.167 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/12' 2023-10-16T20:45:34.167 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/39' 2023-10-16T20:45:34.167 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/10' 2023-10-16T20:45:34.168 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/34' 2023-10-16T20:45:34.168 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/38' 2023-10-16T20:45:34.168 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/11' 2023-10-16T20:45:34.168 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/16' 2023-10-16T20:45:34.168 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/32' 2023-10-16T20:45:34.168 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/30' 2023-10-16T20:45:34.169 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/14' 2023-10-16T20:45:34.169 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/18' 2023-10-16T20:45:34.169 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/13' 2023-10-16T20:45:34.169 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/33' 2023-10-16T20:45:34.169 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/17' 2023-10-16T20:45:34.170 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/35' 2023-10-16T20:45:34.170 INFO:tasks.workunit.client.0.smithi088.stdout:removed './171/37' 2023-10-16T20:45:34.170 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './171' 2023-10-16T20:45:34.170 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/19' 2023-10-16T20:45:34.170 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/15' 2023-10-16T20:45:34.171 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/31' 2023-10-16T20:45:34.171 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/36' 2023-10-16T20:45:34.171 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/20' 2023-10-16T20:45:34.171 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/12' 2023-10-16T20:45:34.171 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/39' 2023-10-16T20:45:34.172 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/10' 2023-10-16T20:45:34.172 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/34' 2023-10-16T20:45:34.172 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/38' 2023-10-16T20:45:34.172 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/11' 2023-10-16T20:45:34.172 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/16' 2023-10-16T20:45:34.172 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/32' 2023-10-16T20:45:34.173 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/30' 2023-10-16T20:45:34.173 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/14' 2023-10-16T20:45:34.173 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/18' 2023-10-16T20:45:34.173 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/13' 2023-10-16T20:45:34.173 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/33' 2023-10-16T20:45:34.173 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/17' 2023-10-16T20:45:34.174 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/35' 2023-10-16T20:45:34.174 INFO:tasks.workunit.client.0.smithi088.stdout:removed './172/37' 2023-10-16T20:45:34.174 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './172' 2023-10-16T20:45:34.174 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/19' 2023-10-16T20:45:34.174 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/15' 2023-10-16T20:45:34.175 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/31' 2023-10-16T20:45:34.175 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/36' 2023-10-16T20:45:34.175 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/20' 2023-10-16T20:45:34.175 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/12' 2023-10-16T20:45:34.175 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/39' 2023-10-16T20:45:34.176 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/10' 2023-10-16T20:45:34.176 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/34' 2023-10-16T20:45:34.176 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/38' 2023-10-16T20:45:34.176 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/11' 2023-10-16T20:45:34.176 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/16' 2023-10-16T20:45:34.177 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/32' 2023-10-16T20:45:34.177 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/30' 2023-10-16T20:45:34.177 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/14' 2023-10-16T20:45:34.177 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/18' 2023-10-16T20:45:34.177 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/13' 2023-10-16T20:45:34.177 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/33' 2023-10-16T20:45:34.178 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/17' 2023-10-16T20:45:34.178 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/35' 2023-10-16T20:45:34.178 INFO:tasks.workunit.client.0.smithi088.stdout:removed './173/37' 2023-10-16T20:45:34.178 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './173' 2023-10-16T20:45:34.178 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/19' 2023-10-16T20:45:34.179 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/15' 2023-10-16T20:45:34.179 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/31' 2023-10-16T20:45:34.179 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/36' 2023-10-16T20:45:34.179 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/20' 2023-10-16T20:45:34.179 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/12' 2023-10-16T20:45:34.180 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/39' 2023-10-16T20:45:34.180 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/10' 2023-10-16T20:45:34.180 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/34' 2023-10-16T20:45:34.180 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/38' 2023-10-16T20:45:34.180 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/11' 2023-10-16T20:45:34.181 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/16' 2023-10-16T20:45:34.181 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/32' 2023-10-16T20:45:34.181 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/30' 2023-10-16T20:45:34.181 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/14' 2023-10-16T20:45:34.181 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/18' 2023-10-16T20:45:34.181 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/13' 2023-10-16T20:45:34.182 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/33' 2023-10-16T20:45:34.182 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/17' 2023-10-16T20:45:34.182 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/35' 2023-10-16T20:45:34.182 INFO:tasks.workunit.client.0.smithi088.stdout:removed './174/37' 2023-10-16T20:45:34.182 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './174' 2023-10-16T20:45:34.183 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/19' 2023-10-16T20:45:34.183 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/15' 2023-10-16T20:45:34.183 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/31' 2023-10-16T20:45:34.183 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/36' 2023-10-16T20:45:34.183 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/20' 2023-10-16T20:45:34.184 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/12' 2023-10-16T20:45:34.184 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/39' 2023-10-16T20:45:34.184 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/10' 2023-10-16T20:45:34.184 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/34' 2023-10-16T20:45:34.184 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/38' 2023-10-16T20:45:34.185 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/11' 2023-10-16T20:45:34.185 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/16' 2023-10-16T20:45:34.185 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/32' 2023-10-16T20:45:34.185 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/30' 2023-10-16T20:45:34.185 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/14' 2023-10-16T20:45:34.185 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/18' 2023-10-16T20:45:34.186 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/13' 2023-10-16T20:45:34.186 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/33' 2023-10-16T20:45:34.186 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/17' 2023-10-16T20:45:34.186 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/35' 2023-10-16T20:45:34.186 INFO:tasks.workunit.client.0.smithi088.stdout:removed './175/37' 2023-10-16T20:45:34.187 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './175' 2023-10-16T20:45:34.187 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/19' 2023-10-16T20:45:34.187 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/15' 2023-10-16T20:45:34.187 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/31' 2023-10-16T20:45:34.187 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/36' 2023-10-16T20:45:34.187 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/20' 2023-10-16T20:45:34.188 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/12' 2023-10-16T20:45:34.188 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/39' 2023-10-16T20:45:34.188 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/10' 2023-10-16T20:45:34.188 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/34' 2023-10-16T20:45:34.188 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/38' 2023-10-16T20:45:34.188 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/11' 2023-10-16T20:45:34.188 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/16' 2023-10-16T20:45:34.189 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/32' 2023-10-16T20:45:34.189 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/30' 2023-10-16T20:45:34.189 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/14' 2023-10-16T20:45:34.189 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/18' 2023-10-16T20:45:34.189 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/13' 2023-10-16T20:45:34.190 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/33' 2023-10-16T20:45:34.190 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/17' 2023-10-16T20:45:34.190 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/35' 2023-10-16T20:45:34.190 INFO:tasks.workunit.client.0.smithi088.stdout:removed './176/37' 2023-10-16T20:45:34.190 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './176' 2023-10-16T20:45:34.191 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/19' 2023-10-16T20:45:34.191 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/15' 2023-10-16T20:45:35.498 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/31' 2023-10-16T20:45:35.498 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/36' 2023-10-16T20:45:35.498 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/20' 2023-10-16T20:45:35.499 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/12' 2023-10-16T20:45:35.499 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/39' 2023-10-16T20:45:35.499 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/10' 2023-10-16T20:45:35.499 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/34' 2023-10-16T20:45:35.499 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/38' 2023-10-16T20:45:35.499 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/11' 2023-10-16T20:45:35.500 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/16' 2023-10-16T20:45:35.500 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/32' 2023-10-16T20:45:35.500 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/30' 2023-10-16T20:45:35.500 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/14' 2023-10-16T20:45:35.500 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/18' 2023-10-16T20:45:35.500 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/13' 2023-10-16T20:45:35.501 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/33' 2023-10-16T20:45:35.501 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/17' 2023-10-16T20:45:35.501 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/35' 2023-10-16T20:45:35.501 INFO:tasks.workunit.client.0.smithi088.stdout:removed './177/37' 2023-10-16T20:45:35.501 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './177' 2023-10-16T20:45:35.502 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/19' 2023-10-16T20:45:35.502 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/15' 2023-10-16T20:45:35.502 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/31' 2023-10-16T20:45:35.502 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/36' 2023-10-16T20:45:35.502 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/20' 2023-10-16T20:45:35.503 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/12' 2023-10-16T20:45:35.503 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/39' 2023-10-16T20:45:35.503 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/10' 2023-10-16T20:45:35.503 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/34' 2023-10-16T20:45:35.503 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/38' 2023-10-16T20:45:35.504 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/11' 2023-10-16T20:45:35.504 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/16' 2023-10-16T20:45:35.504 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/32' 2023-10-16T20:45:35.504 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/30' 2023-10-16T20:45:35.504 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/14' 2023-10-16T20:45:35.504 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/18' 2023-10-16T20:45:35.505 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/13' 2023-10-16T20:45:35.505 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/33' 2023-10-16T20:45:35.505 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/17' 2023-10-16T20:45:35.505 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/35' 2023-10-16T20:45:35.505 INFO:tasks.workunit.client.0.smithi088.stdout:removed './178/37' 2023-10-16T20:45:35.506 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './178' 2023-10-16T20:45:35.506 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/19' 2023-10-16T20:45:35.506 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/15' 2023-10-16T20:45:35.506 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/31' 2023-10-16T20:45:35.506 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/36' 2023-10-16T20:45:35.507 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/20' 2023-10-16T20:45:35.507 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/12' 2023-10-16T20:45:35.507 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/39' 2023-10-16T20:45:35.507 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/10' 2023-10-16T20:45:35.507 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/34' 2023-10-16T20:45:35.507 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/38' 2023-10-16T20:45:35.508 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/11' 2023-10-16T20:45:35.508 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/16' 2023-10-16T20:45:35.508 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/32' 2023-10-16T20:45:35.508 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/30' 2023-10-16T20:45:35.508 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/14' 2023-10-16T20:45:35.508 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/18' 2023-10-16T20:45:35.509 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/13' 2023-10-16T20:45:35.509 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/33' 2023-10-16T20:45:35.509 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/17' 2023-10-16T20:45:35.509 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/35' 2023-10-16T20:45:35.509 INFO:tasks.workunit.client.0.smithi088.stdout:removed './179/37' 2023-10-16T20:45:35.509 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './179' 2023-10-16T20:45:35.510 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/19' 2023-10-16T20:45:35.510 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/15' 2023-10-16T20:45:35.510 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/31' 2023-10-16T20:45:35.510 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/36' 2023-10-16T20:45:35.510 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/20' 2023-10-16T20:45:35.511 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/12' 2023-10-16T20:45:35.511 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/39' 2023-10-16T20:45:35.511 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/10' 2023-10-16T20:45:35.511 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/34' 2023-10-16T20:45:35.511 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/38' 2023-10-16T20:45:35.511 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/11' 2023-10-16T20:45:35.512 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/16' 2023-10-16T20:45:35.512 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/32' 2023-10-16T20:45:35.512 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/30' 2023-10-16T20:45:35.512 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/14' 2023-10-16T20:45:35.512 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/18' 2023-10-16T20:45:35.512 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/13' 2023-10-16T20:45:35.513 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/33' 2023-10-16T20:45:35.513 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/17' 2023-10-16T20:45:35.513 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/35' 2023-10-16T20:45:35.513 INFO:tasks.workunit.client.0.smithi088.stdout:removed './180/37' 2023-10-16T20:45:35.513 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './180' 2023-10-16T20:45:35.513 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/19' 2023-10-16T20:45:35.514 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/15' 2023-10-16T20:45:35.514 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/31' 2023-10-16T20:45:35.514 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/36' 2023-10-16T20:45:35.514 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/20' 2023-10-16T20:45:35.514 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/12' 2023-10-16T20:45:35.515 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/39' 2023-10-16T20:45:35.515 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/10' 2023-10-16T20:45:35.515 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/34' 2023-10-16T20:45:35.515 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/38' 2023-10-16T20:45:35.515 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/11' 2023-10-16T20:45:35.515 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/16' 2023-10-16T20:45:35.516 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/32' 2023-10-16T20:45:35.516 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/30' 2023-10-16T20:45:35.516 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/14' 2023-10-16T20:45:35.516 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/18' 2023-10-16T20:45:35.516 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/13' 2023-10-16T20:45:35.517 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/33' 2023-10-16T20:45:35.517 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/17' 2023-10-16T20:45:35.517 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/35' 2023-10-16T20:45:35.517 INFO:tasks.workunit.client.0.smithi088.stdout:removed './181/37' 2023-10-16T20:45:35.517 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './181' 2023-10-16T20:45:35.518 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/19' 2023-10-16T20:45:35.518 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/15' 2023-10-16T20:45:35.518 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/31' 2023-10-16T20:45:35.518 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/36' 2023-10-16T20:45:35.518 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/20' 2023-10-16T20:45:35.518 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/12' 2023-10-16T20:45:35.519 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/39' 2023-10-16T20:45:35.519 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/10' 2023-10-16T20:45:35.519 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/34' 2023-10-16T20:45:35.519 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/38' 2023-10-16T20:45:35.519 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/11' 2023-10-16T20:45:35.520 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/16' 2023-10-16T20:45:35.520 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/32' 2023-10-16T20:45:35.520 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/30' 2023-10-16T20:45:35.520 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/14' 2023-10-16T20:45:35.520 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/18' 2023-10-16T20:45:35.520 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/13' 2023-10-16T20:45:35.520 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/33' 2023-10-16T20:45:35.521 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/17' 2023-10-16T20:45:35.521 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/35' 2023-10-16T20:45:35.521 INFO:tasks.workunit.client.0.smithi088.stdout:removed './182/37' 2023-10-16T20:45:35.521 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './182' 2023-10-16T20:45:35.521 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/19' 2023-10-16T20:45:35.522 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/15' 2023-10-16T20:45:35.522 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/31' 2023-10-16T20:45:35.522 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/36' 2023-10-16T20:45:35.522 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/20' 2023-10-16T20:45:35.522 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/12' 2023-10-16T20:45:35.523 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/39' 2023-10-16T20:45:35.523 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/10' 2023-10-16T20:45:35.523 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/34' 2023-10-16T20:45:35.523 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/38' 2023-10-16T20:45:35.523 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/11' 2023-10-16T20:45:35.523 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/16' 2023-10-16T20:45:35.524 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/32' 2023-10-16T20:45:35.524 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/30' 2023-10-16T20:45:35.524 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/14' 2023-10-16T20:45:35.524 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/18' 2023-10-16T20:45:35.524 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/13' 2023-10-16T20:45:35.524 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/33' 2023-10-16T20:45:35.525 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/17' 2023-10-16T20:45:35.525 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/35' 2023-10-16T20:45:35.525 INFO:tasks.workunit.client.0.smithi088.stdout:removed './183/37' 2023-10-16T20:45:35.525 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './183' 2023-10-16T20:45:35.525 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/19' 2023-10-16T20:45:35.526 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/15' 2023-10-16T20:45:35.526 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/31' 2023-10-16T20:45:35.526 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/36' 2023-10-16T20:45:35.526 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/20' 2023-10-16T20:45:35.526 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/12' 2023-10-16T20:45:35.526 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/39' 2023-10-16T20:45:35.527 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/10' 2023-10-16T20:45:35.527 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/34' 2023-10-16T20:45:35.527 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/38' 2023-10-16T20:45:35.527 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/11' 2023-10-16T20:45:35.527 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/16' 2023-10-16T20:45:35.527 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/32' 2023-10-16T20:45:35.528 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/30' 2023-10-16T20:45:35.528 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/14' 2023-10-16T20:45:35.528 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/18' 2023-10-16T20:45:35.528 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/13' 2023-10-16T20:45:35.528 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/33' 2023-10-16T20:45:35.528 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/17' 2023-10-16T20:45:35.529 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/35' 2023-10-16T20:45:35.529 INFO:tasks.workunit.client.0.smithi088.stdout:removed './184/37' 2023-10-16T20:45:35.529 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './184' 2023-10-16T20:45:35.529 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/19' 2023-10-16T20:45:35.529 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/15' 2023-10-16T20:45:35.530 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/31' 2023-10-16T20:45:35.530 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/36' 2023-10-16T20:45:35.530 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/20' 2023-10-16T20:45:35.530 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/12' 2023-10-16T20:45:35.530 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/39' 2023-10-16T20:45:35.531 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/10' 2023-10-16T20:45:35.531 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/34' 2023-10-16T20:45:35.531 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/38' 2023-10-16T20:45:35.531 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/11' 2023-10-16T20:45:35.531 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/16' 2023-10-16T20:45:35.531 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/32' 2023-10-16T20:45:35.532 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/30' 2023-10-16T20:45:35.532 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/14' 2023-10-16T20:45:35.532 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/18' 2023-10-16T20:45:35.532 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/13' 2023-10-16T20:45:35.532 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/33' 2023-10-16T20:45:35.533 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/17' 2023-10-16T20:45:35.533 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/35' 2023-10-16T20:45:35.533 INFO:tasks.workunit.client.0.smithi088.stdout:removed './185/37' 2023-10-16T20:45:35.533 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './185' 2023-10-16T20:45:35.533 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/19' 2023-10-16T20:45:35.533 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/15' 2023-10-16T20:45:35.534 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/31' 2023-10-16T20:45:35.534 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/36' 2023-10-16T20:45:35.534 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/20' 2023-10-16T20:45:35.534 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/12' 2023-10-16T20:45:35.534 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/39' 2023-10-16T20:45:35.535 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/10' 2023-10-16T20:45:35.535 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/34' 2023-10-16T20:45:35.535 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/38' 2023-10-16T20:45:35.535 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/11' 2023-10-16T20:45:35.535 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/16' 2023-10-16T20:45:35.535 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/32' 2023-10-16T20:45:35.536 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/30' 2023-10-16T20:45:35.536 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/14' 2023-10-16T20:45:35.536 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/18' 2023-10-16T20:45:37.763 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/13' 2023-10-16T20:45:37.764 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/33' 2023-10-16T20:45:37.764 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/17' 2023-10-16T20:45:37.764 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/35' 2023-10-16T20:45:37.764 INFO:tasks.workunit.client.0.smithi088.stdout:removed './186/37' 2023-10-16T20:45:37.765 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './186' 2023-10-16T20:45:37.765 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/19' 2023-10-16T20:45:37.765 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/15' 2023-10-16T20:45:37.765 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/31' 2023-10-16T20:45:37.765 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/36' 2023-10-16T20:45:37.766 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/20' 2023-10-16T20:45:37.766 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/12' 2023-10-16T20:45:37.766 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/39' 2023-10-16T20:45:37.766 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/10' 2023-10-16T20:45:37.766 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/34' 2023-10-16T20:45:37.767 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/38' 2023-10-16T20:45:37.767 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/11' 2023-10-16T20:45:37.767 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/16' 2023-10-16T20:45:37.767 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/32' 2023-10-16T20:45:37.767 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/30' 2023-10-16T20:45:37.768 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/14' 2023-10-16T20:45:37.768 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/18' 2023-10-16T20:45:37.768 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/13' 2023-10-16T20:45:37.768 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/33' 2023-10-16T20:45:37.768 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/17' 2023-10-16T20:45:37.769 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/35' 2023-10-16T20:45:37.769 INFO:tasks.workunit.client.0.smithi088.stdout:removed './187/37' 2023-10-16T20:45:37.769 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './187' 2023-10-16T20:45:37.769 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/19' 2023-10-16T20:45:37.769 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/15' 2023-10-16T20:45:37.770 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/31' 2023-10-16T20:45:37.770 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/36' 2023-10-16T20:45:37.770 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/20' 2023-10-16T20:45:37.770 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/12' 2023-10-16T20:45:37.770 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/39' 2023-10-16T20:45:37.771 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/10' 2023-10-16T20:45:37.771 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/34' 2023-10-16T20:45:37.771 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/38' 2023-10-16T20:45:37.771 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/11' 2023-10-16T20:45:37.771 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/16' 2023-10-16T20:45:37.772 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/32' 2023-10-16T20:45:37.772 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/30' 2023-10-16T20:45:37.772 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/14' 2023-10-16T20:45:37.772 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/18' 2023-10-16T20:45:37.772 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/13' 2023-10-16T20:45:37.772 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/33' 2023-10-16T20:45:37.773 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/17' 2023-10-16T20:45:37.773 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/35' 2023-10-16T20:45:37.773 INFO:tasks.workunit.client.0.smithi088.stdout:removed './188/37' 2023-10-16T20:45:37.773 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './188' 2023-10-16T20:45:37.773 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/19' 2023-10-16T20:45:37.774 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/15' 2023-10-16T20:45:37.774 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/31' 2023-10-16T20:45:37.774 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/36' 2023-10-16T20:45:37.774 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/20' 2023-10-16T20:45:37.774 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/12' 2023-10-16T20:45:37.775 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/39' 2023-10-16T20:45:37.775 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/10' 2023-10-16T20:45:37.775 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/34' 2023-10-16T20:45:37.775 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/38' 2023-10-16T20:45:37.775 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/11' 2023-10-16T20:45:37.775 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/16' 2023-10-16T20:45:37.776 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/32' 2023-10-16T20:45:37.776 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/30' 2023-10-16T20:45:37.776 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/14' 2023-10-16T20:45:37.776 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/18' 2023-10-16T20:45:37.776 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/13' 2023-10-16T20:45:37.777 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/33' 2023-10-16T20:45:37.777 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/17' 2023-10-16T20:45:37.777 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/35' 2023-10-16T20:45:37.777 INFO:tasks.workunit.client.0.smithi088.stdout:removed './189/37' 2023-10-16T20:45:37.777 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './189' 2023-10-16T20:45:37.778 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/19' 2023-10-16T20:45:37.778 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/15' 2023-10-16T20:45:37.778 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/31' 2023-10-16T20:45:37.778 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/36' 2023-10-16T20:45:37.778 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/20' 2023-10-16T20:45:37.779 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/12' 2023-10-16T20:45:37.779 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/39' 2023-10-16T20:45:37.779 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/10' 2023-10-16T20:45:37.779 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/34' 2023-10-16T20:45:37.779 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/38' 2023-10-16T20:45:37.780 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/11' 2023-10-16T20:45:37.780 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/16' 2023-10-16T20:45:37.780 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/32' 2023-10-16T20:45:37.780 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/30' 2023-10-16T20:45:37.780 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/14' 2023-10-16T20:45:37.781 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/18' 2023-10-16T20:45:37.781 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/13' 2023-10-16T20:45:37.781 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/33' 2023-10-16T20:45:37.781 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/17' 2023-10-16T20:45:37.781 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/35' 2023-10-16T20:45:37.782 INFO:tasks.workunit.client.0.smithi088.stdout:removed './190/37' 2023-10-16T20:45:37.782 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './190' 2023-10-16T20:45:37.782 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/19' 2023-10-16T20:45:37.782 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/15' 2023-10-16T20:45:37.782 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/31' 2023-10-16T20:45:37.783 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/36' 2023-10-16T20:45:37.783 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/20' 2023-10-16T20:45:37.783 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/12' 2023-10-16T20:45:37.783 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/39' 2023-10-16T20:45:37.783 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/10' 2023-10-16T20:45:37.784 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/34' 2023-10-16T20:45:37.784 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/38' 2023-10-16T20:45:37.784 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/11' 2023-10-16T20:45:37.784 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/16' 2023-10-16T20:45:37.784 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/32' 2023-10-16T20:45:37.784 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/30' 2023-10-16T20:45:37.785 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/14' 2023-10-16T20:45:37.785 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/18' 2023-10-16T20:45:37.785 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/13' 2023-10-16T20:45:37.785 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/33' 2023-10-16T20:45:37.785 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/17' 2023-10-16T20:45:37.786 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/35' 2023-10-16T20:45:37.786 INFO:tasks.workunit.client.0.smithi088.stdout:removed './191/37' 2023-10-16T20:45:37.786 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './191' 2023-10-16T20:45:37.786 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/19' 2023-10-16T20:45:37.786 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/15' 2023-10-16T20:45:37.787 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/31' 2023-10-16T20:45:37.787 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/36' 2023-10-16T20:45:37.787 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/20' 2023-10-16T20:45:37.787 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/12' 2023-10-16T20:45:37.787 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/39' 2023-10-16T20:45:37.787 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/10' 2023-10-16T20:45:37.788 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/34' 2023-10-16T20:45:37.788 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/38' 2023-10-16T20:45:37.788 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/11' 2023-10-16T20:45:37.788 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/16' 2023-10-16T20:45:37.788 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/32' 2023-10-16T20:45:37.789 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/30' 2023-10-16T20:45:37.789 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/14' 2023-10-16T20:45:37.789 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/18' 2023-10-16T20:45:37.789 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/13' 2023-10-16T20:45:37.789 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/33' 2023-10-16T20:45:37.789 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/17' 2023-10-16T20:45:37.790 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/35' 2023-10-16T20:45:37.790 INFO:tasks.workunit.client.0.smithi088.stdout:removed './192/37' 2023-10-16T20:45:37.790 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './192' 2023-10-16T20:45:37.790 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/19' 2023-10-16T20:45:37.790 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/15' 2023-10-16T20:45:37.791 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/31' 2023-10-16T20:45:37.791 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/36' 2023-10-16T20:45:37.791 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/20' 2023-10-16T20:45:37.791 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/12' 2023-10-16T20:45:37.791 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/39' 2023-10-16T20:45:37.791 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/10' 2023-10-16T20:45:37.792 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/34' 2023-10-16T20:45:37.792 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/38' 2023-10-16T20:45:37.792 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/11' 2023-10-16T20:45:37.792 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/16' 2023-10-16T20:45:37.792 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/32' 2023-10-16T20:45:37.792 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/30' 2023-10-16T20:45:37.793 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/14' 2023-10-16T20:45:37.793 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/18' 2023-10-16T20:45:37.793 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/13' 2023-10-16T20:45:37.793 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/33' 2023-10-16T20:45:37.793 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/17' 2023-10-16T20:45:37.794 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/35' 2023-10-16T20:45:37.794 INFO:tasks.workunit.client.0.smithi088.stdout:removed './193/37' 2023-10-16T20:45:37.794 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './193' 2023-10-16T20:45:37.794 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/19' 2023-10-16T20:45:37.794 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/15' 2023-10-16T20:45:37.795 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/31' 2023-10-16T20:45:37.795 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/36' 2023-10-16T20:45:37.795 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/20' 2023-10-16T20:45:37.795 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/12' 2023-10-16T20:45:37.795 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/39' 2023-10-16T20:45:37.795 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/10' 2023-10-16T20:45:37.796 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/34' 2023-10-16T20:45:37.796 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/38' 2023-10-16T20:45:37.796 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/11' 2023-10-16T20:45:37.796 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/16' 2023-10-16T20:45:37.796 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/32' 2023-10-16T20:45:37.797 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/30' 2023-10-16T20:45:37.797 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/14' 2023-10-16T20:45:37.797 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/18' 2023-10-16T20:45:37.797 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/13' 2023-10-16T20:45:37.797 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/33' 2023-10-16T20:45:37.798 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/17' 2023-10-16T20:45:37.798 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/35' 2023-10-16T20:45:37.798 INFO:tasks.workunit.client.0.smithi088.stdout:removed './194/37' 2023-10-16T20:45:37.798 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './194' 2023-10-16T20:45:37.798 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/19' 2023-10-16T20:45:37.799 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/15' 2023-10-16T20:45:37.799 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/31' 2023-10-16T20:45:37.799 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/36' 2023-10-16T20:45:37.799 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/20' 2023-10-16T20:45:37.799 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/12' 2023-10-16T20:45:37.799 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/39' 2023-10-16T20:45:37.800 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/10' 2023-10-16T20:45:37.800 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/34' 2023-10-16T20:45:37.800 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/38' 2023-10-16T20:45:37.800 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/11' 2023-10-16T20:45:37.800 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/16' 2023-10-16T20:45:37.800 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/32' 2023-10-16T20:45:37.801 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/30' 2023-10-16T20:45:37.801 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/14' 2023-10-16T20:45:37.801 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/18' 2023-10-16T20:45:37.801 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/13' 2023-10-16T20:45:37.801 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/33' 2023-10-16T20:45:37.801 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/17' 2023-10-16T20:45:37.802 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/35' 2023-10-16T20:45:37.802 INFO:tasks.workunit.client.0.smithi088.stdout:removed './195/37' 2023-10-16T20:45:37.802 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './195' 2023-10-16T20:45:37.802 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/19' 2023-10-16T20:45:37.802 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/15' 2023-10-16T20:45:37.803 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/31' 2023-10-16T20:45:37.803 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/36' 2023-10-16T20:45:37.803 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/20' 2023-10-16T20:45:37.803 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/12' 2023-10-16T20:45:37.803 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/39' 2023-10-16T20:45:37.803 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/10' 2023-10-16T20:45:39.515 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/34' 2023-10-16T20:45:39.516 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/38' 2023-10-16T20:45:39.516 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/11' 2023-10-16T20:45:39.516 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/16' 2023-10-16T20:45:39.516 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/32' 2023-10-16T20:45:39.516 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/30' 2023-10-16T20:45:39.516 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/14' 2023-10-16T20:45:39.517 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/18' 2023-10-16T20:45:39.517 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/13' 2023-10-16T20:45:39.517 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/33' 2023-10-16T20:45:39.517 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/17' 2023-10-16T20:45:39.517 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/35' 2023-10-16T20:45:39.517 INFO:tasks.workunit.client.0.smithi088.stdout:removed './196/37' 2023-10-16T20:45:39.518 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './196' 2023-10-16T20:45:39.518 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/19' 2023-10-16T20:45:39.518 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/15' 2023-10-16T20:45:39.518 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/31' 2023-10-16T20:45:39.518 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/36' 2023-10-16T20:45:39.519 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/20' 2023-10-16T20:45:39.519 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/12' 2023-10-16T20:45:39.519 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/39' 2023-10-16T20:45:39.519 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/10' 2023-10-16T20:45:39.519 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/34' 2023-10-16T20:45:39.519 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/38' 2023-10-16T20:45:39.520 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/11' 2023-10-16T20:45:39.520 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/16' 2023-10-16T20:45:39.520 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/32' 2023-10-16T20:45:39.520 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/30' 2023-10-16T20:45:39.520 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/14' 2023-10-16T20:45:39.520 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/18' 2023-10-16T20:45:39.521 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/13' 2023-10-16T20:45:39.521 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/33' 2023-10-16T20:45:39.521 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/17' 2023-10-16T20:45:39.521 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/35' 2023-10-16T20:45:39.521 INFO:tasks.workunit.client.0.smithi088.stdout:removed './197/37' 2023-10-16T20:45:39.521 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './197' 2023-10-16T20:45:39.522 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/19' 2023-10-16T20:45:39.522 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/15' 2023-10-16T20:45:39.522 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/31' 2023-10-16T20:45:39.522 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/36' 2023-10-16T20:45:39.522 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/20' 2023-10-16T20:45:39.523 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/12' 2023-10-16T20:45:39.523 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/39' 2023-10-16T20:45:39.523 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/10' 2023-10-16T20:45:39.523 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/34' 2023-10-16T20:45:39.523 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/38' 2023-10-16T20:45:39.523 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/11' 2023-10-16T20:45:39.524 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/16' 2023-10-16T20:45:39.524 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/32' 2023-10-16T20:45:39.524 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/30' 2023-10-16T20:45:39.524 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/14' 2023-10-16T20:45:39.524 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/18' 2023-10-16T20:45:39.524 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/13' 2023-10-16T20:45:39.525 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/33' 2023-10-16T20:45:39.525 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/17' 2023-10-16T20:45:39.525 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/35' 2023-10-16T20:45:39.525 INFO:tasks.workunit.client.0.smithi088.stdout:removed './198/37' 2023-10-16T20:45:39.525 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './198' 2023-10-16T20:45:39.525 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/19' 2023-10-16T20:45:39.526 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/15' 2023-10-16T20:45:39.526 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/31' 2023-10-16T20:45:39.526 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/36' 2023-10-16T20:45:39.526 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/20' 2023-10-16T20:45:39.526 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/12' 2023-10-16T20:45:39.527 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/39' 2023-10-16T20:45:39.527 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/10' 2023-10-16T20:45:39.527 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/34' 2023-10-16T20:45:39.527 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/38' 2023-10-16T20:45:39.527 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/11' 2023-10-16T20:45:39.527 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/16' 2023-10-16T20:45:39.528 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/32' 2023-10-16T20:45:39.528 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/30' 2023-10-16T20:45:39.528 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/14' 2023-10-16T20:45:39.528 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/18' 2023-10-16T20:45:39.528 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/13' 2023-10-16T20:45:39.528 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/33' 2023-10-16T20:45:39.529 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/17' 2023-10-16T20:45:39.529 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/35' 2023-10-16T20:45:39.529 INFO:tasks.workunit.client.0.smithi088.stdout:removed './199/37' 2023-10-16T20:45:39.529 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './199' 2023-10-16T20:45:39.529 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/21' 2023-10-16T20:45:39.530 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/27' 2023-10-16T20:45:39.530 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/31' 2023-10-16T20:45:39.530 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/36' 2023-10-16T20:45:39.530 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/20' 2023-10-16T20:45:39.530 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/39' 2023-10-16T20:45:39.531 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/34' 2023-10-16T20:45:39.531 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/38' 2023-10-16T20:45:39.531 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/28' 2023-10-16T20:45:39.531 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/24' 2023-10-16T20:45:39.531 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/29' 2023-10-16T20:45:39.532 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/32' 2023-10-16T20:45:39.532 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/22' 2023-10-16T20:45:39.532 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/25' 2023-10-16T20:45:39.532 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/30' 2023-10-16T20:45:39.532 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/33' 2023-10-16T20:45:39.532 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/23' 2023-10-16T20:45:39.533 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/35' 2023-10-16T20:45:39.533 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/26' 2023-10-16T20:45:39.533 INFO:tasks.workunit.client.0.smithi088.stdout:removed './200/37' 2023-10-16T20:45:39.533 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './200' 2023-10-16T20:45:39.533 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/21' 2023-10-16T20:45:39.534 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/27' 2023-10-16T20:45:39.534 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/31' 2023-10-16T20:45:39.534 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/36' 2023-10-16T20:45:39.534 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/20' 2023-10-16T20:45:39.534 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/39' 2023-10-16T20:45:39.534 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/34' 2023-10-16T20:45:39.535 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/38' 2023-10-16T20:45:39.535 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/28' 2023-10-16T20:45:39.535 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/24' 2023-10-16T20:45:39.535 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/29' 2023-10-16T20:45:39.535 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/32' 2023-10-16T20:45:39.535 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/22' 2023-10-16T20:45:39.536 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/25' 2023-10-16T20:45:39.536 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/30' 2023-10-16T20:45:39.536 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/33' 2023-10-16T20:45:39.536 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/23' 2023-10-16T20:45:39.536 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/35' 2023-10-16T20:45:39.536 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/26' 2023-10-16T20:45:39.537 INFO:tasks.workunit.client.0.smithi088.stdout:removed './201/37' 2023-10-16T20:45:39.537 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './201' 2023-10-16T20:45:39.537 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/21' 2023-10-16T20:45:39.537 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/27' 2023-10-16T20:45:39.537 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/31' 2023-10-16T20:45:39.538 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/36' 2023-10-16T20:45:39.538 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/20' 2023-10-16T20:45:39.538 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/39' 2023-10-16T20:45:39.538 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/34' 2023-10-16T20:45:39.538 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/38' 2023-10-16T20:45:39.539 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/28' 2023-10-16T20:45:39.539 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/24' 2023-10-16T20:45:39.539 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/29' 2023-10-16T20:45:39.539 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/32' 2023-10-16T20:45:39.539 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/22' 2023-10-16T20:45:39.540 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/25' 2023-10-16T20:45:39.540 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/30' 2023-10-16T20:45:39.540 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/33' 2023-10-16T20:45:39.540 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/23' 2023-10-16T20:45:39.540 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/35' 2023-10-16T20:45:39.540 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/26' 2023-10-16T20:45:39.541 INFO:tasks.workunit.client.0.smithi088.stdout:removed './202/37' 2023-10-16T20:45:39.541 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './202' 2023-10-16T20:45:39.541 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/21' 2023-10-16T20:45:39.541 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/27' 2023-10-16T20:45:39.541 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/31' 2023-10-16T20:45:39.541 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/36' 2023-10-16T20:45:39.542 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/20' 2023-10-16T20:45:39.542 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/39' 2023-10-16T20:45:39.542 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/34' 2023-10-16T20:45:39.542 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/38' 2023-10-16T20:45:39.542 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/28' 2023-10-16T20:45:39.542 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/24' 2023-10-16T20:45:39.543 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/29' 2023-10-16T20:45:39.543 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/32' 2023-10-16T20:45:39.543 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/22' 2023-10-16T20:45:39.543 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/25' 2023-10-16T20:45:39.543 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/30' 2023-10-16T20:45:39.543 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/33' 2023-10-16T20:45:39.544 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/23' 2023-10-16T20:45:39.544 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/35' 2023-10-16T20:45:39.544 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/26' 2023-10-16T20:45:39.544 INFO:tasks.workunit.client.0.smithi088.stdout:removed './203/37' 2023-10-16T20:45:39.544 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './203' 2023-10-16T20:45:39.544 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/21' 2023-10-16T20:45:39.545 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/27' 2023-10-16T20:45:39.545 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/31' 2023-10-16T20:45:39.545 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/36' 2023-10-16T20:45:39.545 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/20' 2023-10-16T20:45:39.545 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/39' 2023-10-16T20:45:39.545 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/34' 2023-10-16T20:45:39.546 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/38' 2023-10-16T20:45:39.546 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/28' 2023-10-16T20:45:39.546 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/24' 2023-10-16T20:45:39.546 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/29' 2023-10-16T20:45:39.546 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/32' 2023-10-16T20:45:39.547 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/22' 2023-10-16T20:45:39.547 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/25' 2023-10-16T20:45:39.547 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/30' 2023-10-16T20:45:39.547 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/33' 2023-10-16T20:45:39.547 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/23' 2023-10-16T20:45:39.547 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/35' 2023-10-16T20:45:39.548 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/26' 2023-10-16T20:45:39.548 INFO:tasks.workunit.client.0.smithi088.stdout:removed './204/37' 2023-10-16T20:45:39.548 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './204' 2023-10-16T20:45:39.548 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/21' 2023-10-16T20:45:39.548 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/27' 2023-10-16T20:45:39.548 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/31' 2023-10-16T20:45:39.549 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/36' 2023-10-16T20:45:39.549 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/20' 2023-10-16T20:45:39.549 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/39' 2023-10-16T20:45:39.549 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/34' 2023-10-16T20:45:39.549 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/38' 2023-10-16T20:45:39.550 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/28' 2023-10-16T20:45:39.550 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/24' 2023-10-16T20:45:39.550 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/29' 2023-10-16T20:45:39.550 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/32' 2023-10-16T20:45:39.550 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/22' 2023-10-16T20:45:39.550 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/25' 2023-10-16T20:45:39.551 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/30' 2023-10-16T20:45:39.551 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/33' 2023-10-16T20:45:39.551 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/23' 2023-10-16T20:45:39.551 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/35' 2023-10-16T20:45:39.551 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/26' 2023-10-16T20:45:39.551 INFO:tasks.workunit.client.0.smithi088.stdout:removed './205/37' 2023-10-16T20:45:39.552 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './205' 2023-10-16T20:45:39.552 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/21' 2023-10-16T20:45:39.552 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/27' 2023-10-16T20:45:39.552 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/31' 2023-10-16T20:45:39.552 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/36' 2023-10-16T20:45:39.552 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/20' 2023-10-16T20:45:39.553 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/39' 2023-10-16T20:45:42.680 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/34' 2023-10-16T20:45:42.680 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/38' 2023-10-16T20:45:42.680 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/28' 2023-10-16T20:45:42.680 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/24' 2023-10-16T20:45:42.680 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/29' 2023-10-16T20:45:42.681 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/32' 2023-10-16T20:45:42.681 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/22' 2023-10-16T20:45:42.681 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/25' 2023-10-16T20:45:42.681 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/30' 2023-10-16T20:45:42.681 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/33' 2023-10-16T20:45:42.682 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/23' 2023-10-16T20:45:42.682 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/35' 2023-10-16T20:45:42.682 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/26' 2023-10-16T20:45:42.682 INFO:tasks.workunit.client.0.smithi088.stdout:removed './206/37' 2023-10-16T20:45:42.682 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './206' 2023-10-16T20:45:42.683 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/21' 2023-10-16T20:45:42.683 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/27' 2023-10-16T20:45:42.683 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/31' 2023-10-16T20:45:42.683 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/36' 2023-10-16T20:45:42.683 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/20' 2023-10-16T20:45:42.683 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/39' 2023-10-16T20:45:42.684 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/34' 2023-10-16T20:45:42.684 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/38' 2023-10-16T20:45:42.684 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/28' 2023-10-16T20:45:42.684 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/24' 2023-10-16T20:45:42.684 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/29' 2023-10-16T20:45:42.684 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/32' 2023-10-16T20:45:42.685 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/22' 2023-10-16T20:45:42.685 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/25' 2023-10-16T20:45:42.685 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/30' 2023-10-16T20:45:42.685 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/33' 2023-10-16T20:45:42.685 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/23' 2023-10-16T20:45:42.686 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/35' 2023-10-16T20:45:42.686 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/26' 2023-10-16T20:45:42.686 INFO:tasks.workunit.client.0.smithi088.stdout:removed './207/37' 2023-10-16T20:45:42.686 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './207' 2023-10-16T20:45:42.686 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/21' 2023-10-16T20:45:42.687 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/27' 2023-10-16T20:45:42.687 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/31' 2023-10-16T20:45:42.687 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/36' 2023-10-16T20:45:42.687 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/20' 2023-10-16T20:45:42.687 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/39' 2023-10-16T20:45:42.687 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/34' 2023-10-16T20:45:42.688 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/38' 2023-10-16T20:45:42.688 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/28' 2023-10-16T20:45:42.688 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/24' 2023-10-16T20:45:42.688 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/29' 2023-10-16T20:45:42.688 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/32' 2023-10-16T20:45:42.689 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/22' 2023-10-16T20:45:42.689 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/25' 2023-10-16T20:45:42.689 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/30' 2023-10-16T20:45:42.689 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/33' 2023-10-16T20:45:42.689 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/23' 2023-10-16T20:45:42.689 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/35' 2023-10-16T20:45:42.690 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/26' 2023-10-16T20:45:42.690 INFO:tasks.workunit.client.0.smithi088.stdout:removed './208/37' 2023-10-16T20:45:42.690 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './208' 2023-10-16T20:45:42.690 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/21' 2023-10-16T20:45:42.690 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/27' 2023-10-16T20:45:42.691 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/31' 2023-10-16T20:45:42.691 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/36' 2023-10-16T20:45:42.691 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/20' 2023-10-16T20:45:42.691 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/39' 2023-10-16T20:45:42.691 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/34' 2023-10-16T20:45:42.691 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/38' 2023-10-16T20:45:42.692 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/28' 2023-10-16T20:45:42.692 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/24' 2023-10-16T20:45:42.692 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/29' 2023-10-16T20:45:42.692 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/32' 2023-10-16T20:45:42.692 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/22' 2023-10-16T20:45:42.692 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/25' 2023-10-16T20:45:42.693 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/30' 2023-10-16T20:45:42.693 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/33' 2023-10-16T20:45:42.693 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/23' 2023-10-16T20:45:42.693 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/35' 2023-10-16T20:45:42.693 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/26' 2023-10-16T20:45:42.693 INFO:tasks.workunit.client.0.smithi088.stdout:removed './209/37' 2023-10-16T20:45:42.694 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './209' 2023-10-16T20:45:42.694 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/21' 2023-10-16T20:45:42.694 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/27' 2023-10-16T20:45:42.694 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/31' 2023-10-16T20:45:42.694 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/36' 2023-10-16T20:45:42.695 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/20' 2023-10-16T20:45:42.695 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/39' 2023-10-16T20:45:42.695 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/34' 2023-10-16T20:45:42.695 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/38' 2023-10-16T20:45:42.695 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/28' 2023-10-16T20:45:42.695 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/24' 2023-10-16T20:45:42.695 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/29' 2023-10-16T20:45:42.696 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/32' 2023-10-16T20:45:42.696 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/22' 2023-10-16T20:45:42.696 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/25' 2023-10-16T20:45:42.696 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/30' 2023-10-16T20:45:42.696 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/33' 2023-10-16T20:45:42.696 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/23' 2023-10-16T20:45:42.697 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/35' 2023-10-16T20:45:42.697 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/26' 2023-10-16T20:45:42.697 INFO:tasks.workunit.client.0.smithi088.stdout:removed './210/37' 2023-10-16T20:45:42.697 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './210' 2023-10-16T20:45:42.697 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/21' 2023-10-16T20:45:42.698 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/27' 2023-10-16T20:45:42.698 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/31' 2023-10-16T20:45:42.698 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/36' 2023-10-16T20:45:42.698 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/20' 2023-10-16T20:45:42.699 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/39' 2023-10-16T20:45:42.699 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/34' 2023-10-16T20:45:42.699 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/38' 2023-10-16T20:45:42.699 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/28' 2023-10-16T20:45:42.699 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/24' 2023-10-16T20:45:42.699 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/29' 2023-10-16T20:45:42.700 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/32' 2023-10-16T20:45:42.700 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/22' 2023-10-16T20:45:42.700 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/25' 2023-10-16T20:45:42.700 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/30' 2023-10-16T20:45:42.700 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/33' 2023-10-16T20:45:42.700 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/23' 2023-10-16T20:45:42.701 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/35' 2023-10-16T20:45:42.701 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/26' 2023-10-16T20:45:42.701 INFO:tasks.workunit.client.0.smithi088.stdout:removed './211/37' 2023-10-16T20:45:42.701 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './211' 2023-10-16T20:45:42.701 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/21' 2023-10-16T20:45:42.702 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/27' 2023-10-16T20:45:42.702 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/31' 2023-10-16T20:45:42.702 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/36' 2023-10-16T20:45:42.702 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/20' 2023-10-16T20:45:42.702 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/39' 2023-10-16T20:45:42.702 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/34' 2023-10-16T20:45:42.703 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/38' 2023-10-16T20:45:42.703 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/28' 2023-10-16T20:45:42.703 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/24' 2023-10-16T20:45:42.703 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/29' 2023-10-16T20:45:42.703 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/32' 2023-10-16T20:45:42.703 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/22' 2023-10-16T20:45:42.704 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/25' 2023-10-16T20:45:42.704 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/30' 2023-10-16T20:45:42.704 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/33' 2023-10-16T20:45:42.704 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/23' 2023-10-16T20:45:42.704 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/35' 2023-10-16T20:45:42.704 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/26' 2023-10-16T20:45:42.705 INFO:tasks.workunit.client.0.smithi088.stdout:removed './212/37' 2023-10-16T20:45:42.705 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './212' 2023-10-16T20:45:42.705 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/21' 2023-10-16T20:45:42.705 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/27' 2023-10-16T20:45:42.705 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/31' 2023-10-16T20:45:42.705 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/36' 2023-10-16T20:45:42.706 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/20' 2023-10-16T20:45:42.706 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/39' 2023-10-16T20:45:42.706 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/34' 2023-10-16T20:45:42.706 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/38' 2023-10-16T20:45:42.706 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/28' 2023-10-16T20:45:42.706 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/24' 2023-10-16T20:45:42.707 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/29' 2023-10-16T20:45:42.707 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/32' 2023-10-16T20:45:42.707 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/22' 2023-10-16T20:45:42.707 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/25' 2023-10-16T20:45:42.707 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/30' 2023-10-16T20:45:42.708 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/33' 2023-10-16T20:45:42.708 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/23' 2023-10-16T20:45:42.708 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/35' 2023-10-16T20:45:42.708 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/26' 2023-10-16T20:45:42.708 INFO:tasks.workunit.client.0.smithi088.stdout:removed './213/37' 2023-10-16T20:45:42.708 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './213' 2023-10-16T20:45:42.709 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/21' 2023-10-16T20:45:42.709 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/27' 2023-10-16T20:45:42.709 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/31' 2023-10-16T20:45:42.709 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/36' 2023-10-16T20:45:42.709 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/20' 2023-10-16T20:45:42.709 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/39' 2023-10-16T20:45:42.710 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/34' 2023-10-16T20:45:42.710 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/38' 2023-10-16T20:45:42.710 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/28' 2023-10-16T20:45:42.710 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/24' 2023-10-16T20:45:42.710 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/29' 2023-10-16T20:45:42.711 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/32' 2023-10-16T20:45:42.711 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/22' 2023-10-16T20:45:42.711 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/25' 2023-10-16T20:45:42.711 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/30' 2023-10-16T20:45:42.711 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/33' 2023-10-16T20:45:42.711 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/23' 2023-10-16T20:45:42.712 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/35' 2023-10-16T20:45:42.712 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/26' 2023-10-16T20:45:42.712 INFO:tasks.workunit.client.0.smithi088.stdout:removed './214/37' 2023-10-16T20:45:42.712 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './214' 2023-10-16T20:45:42.712 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/21' 2023-10-16T20:45:42.712 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/27' 2023-10-16T20:45:42.713 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/31' 2023-10-16T20:45:42.713 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/36' 2023-10-16T20:45:42.713 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/20' 2023-10-16T20:45:42.713 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/39' 2023-10-16T20:45:42.713 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/34' 2023-10-16T20:45:42.713 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/38' 2023-10-16T20:45:42.714 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/28' 2023-10-16T20:45:42.714 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/24' 2023-10-16T20:45:42.714 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/29' 2023-10-16T20:45:42.714 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/32' 2023-10-16T20:45:42.714 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/22' 2023-10-16T20:45:42.714 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/25' 2023-10-16T20:45:42.715 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/30' 2023-10-16T20:45:42.715 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/33' 2023-10-16T20:45:42.715 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/23' 2023-10-16T20:45:42.715 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/35' 2023-10-16T20:45:42.715 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/26' 2023-10-16T20:45:42.716 INFO:tasks.workunit.client.0.smithi088.stdout:removed './215/37' 2023-10-16T20:45:42.716 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './215' 2023-10-16T20:45:42.716 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/21' 2023-10-16T20:45:42.716 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/27' 2023-10-16T20:45:42.716 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/31' 2023-10-16T20:45:42.717 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/36' 2023-10-16T20:45:42.717 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/20' 2023-10-16T20:45:42.717 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/39' 2023-10-16T20:45:42.717 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/34' 2023-10-16T20:45:42.717 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/38' 2023-10-16T20:45:45.969 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/28' 2023-10-16T20:45:45.969 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/24' 2023-10-16T20:45:45.970 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/29' 2023-10-16T20:45:45.970 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/32' 2023-10-16T20:45:45.970 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/22' 2023-10-16T20:45:45.970 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/25' 2023-10-16T20:45:45.971 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/30' 2023-10-16T20:45:45.971 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/33' 2023-10-16T20:45:45.971 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/23' 2023-10-16T20:45:45.971 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/35' 2023-10-16T20:45:45.971 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/26' 2023-10-16T20:45:45.971 INFO:tasks.workunit.client.0.smithi088.stdout:removed './216/37' 2023-10-16T20:45:45.972 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './216' 2023-10-16T20:45:45.972 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/21' 2023-10-16T20:45:45.972 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/27' 2023-10-16T20:45:45.972 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/31' 2023-10-16T20:45:45.972 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/36' 2023-10-16T20:45:45.972 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/20' 2023-10-16T20:45:45.973 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/39' 2023-10-16T20:45:45.973 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/34' 2023-10-16T20:45:45.973 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/38' 2023-10-16T20:45:45.973 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/28' 2023-10-16T20:45:45.973 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/24' 2023-10-16T20:45:45.973 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/29' 2023-10-16T20:45:45.974 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/32' 2023-10-16T20:45:45.974 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/22' 2023-10-16T20:45:45.974 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/25' 2023-10-16T20:45:45.974 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/30' 2023-10-16T20:45:45.974 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/33' 2023-10-16T20:45:45.974 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/23' 2023-10-16T20:45:45.975 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/35' 2023-10-16T20:45:45.975 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/26' 2023-10-16T20:45:45.975 INFO:tasks.workunit.client.0.smithi088.stdout:removed './217/37' 2023-10-16T20:45:45.975 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './217' 2023-10-16T20:45:45.975 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/21' 2023-10-16T20:45:45.975 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/27' 2023-10-16T20:45:45.976 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/31' 2023-10-16T20:45:45.976 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/36' 2023-10-16T20:45:45.976 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/20' 2023-10-16T20:45:45.976 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/39' 2023-10-16T20:45:45.976 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/34' 2023-10-16T20:45:45.977 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/38' 2023-10-16T20:45:45.977 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/28' 2023-10-16T20:45:45.977 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/24' 2023-10-16T20:45:45.977 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/29' 2023-10-16T20:45:45.977 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/32' 2023-10-16T20:45:45.977 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/22' 2023-10-16T20:45:45.978 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/25' 2023-10-16T20:45:45.978 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/30' 2023-10-16T20:45:45.978 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/33' 2023-10-16T20:45:45.978 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/23' 2023-10-16T20:45:45.978 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/35' 2023-10-16T20:45:45.979 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/26' 2023-10-16T20:45:45.979 INFO:tasks.workunit.client.0.smithi088.stdout:removed './218/37' 2023-10-16T20:45:45.979 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './218' 2023-10-16T20:45:45.979 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/21' 2023-10-16T20:45:45.979 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/27' 2023-10-16T20:45:45.980 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/31' 2023-10-16T20:45:45.980 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/36' 2023-10-16T20:45:45.980 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/20' 2023-10-16T20:45:45.980 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/39' 2023-10-16T20:45:45.980 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/34' 2023-10-16T20:45:45.980 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/38' 2023-10-16T20:45:45.981 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/28' 2023-10-16T20:45:45.981 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/24' 2023-10-16T20:45:45.981 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/29' 2023-10-16T20:45:45.981 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/32' 2023-10-16T20:45:45.981 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/22' 2023-10-16T20:45:45.982 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/25' 2023-10-16T20:45:45.982 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/30' 2023-10-16T20:45:45.982 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/33' 2023-10-16T20:45:45.982 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/23' 2023-10-16T20:45:45.982 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/35' 2023-10-16T20:45:45.983 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/26' 2023-10-16T20:45:45.983 INFO:tasks.workunit.client.0.smithi088.stdout:removed './219/37' 2023-10-16T20:45:45.983 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './219' 2023-10-16T20:45:45.983 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/21' 2023-10-16T20:45:45.983 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/27' 2023-10-16T20:45:45.983 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/31' 2023-10-16T20:45:45.984 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/36' 2023-10-16T20:45:45.984 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/20' 2023-10-16T20:45:45.984 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/39' 2023-10-16T20:45:45.984 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/34' 2023-10-16T20:45:45.984 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/38' 2023-10-16T20:45:45.984 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/28' 2023-10-16T20:45:45.985 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/24' 2023-10-16T20:45:45.985 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/29' 2023-10-16T20:45:45.985 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/32' 2023-10-16T20:45:45.985 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/22' 2023-10-16T20:45:45.985 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/25' 2023-10-16T20:45:45.985 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/30' 2023-10-16T20:45:45.986 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/33' 2023-10-16T20:45:45.986 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/23' 2023-10-16T20:45:45.986 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/35' 2023-10-16T20:45:45.986 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/26' 2023-10-16T20:45:45.986 INFO:tasks.workunit.client.0.smithi088.stdout:removed './220/37' 2023-10-16T20:45:45.987 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './220' 2023-10-16T20:45:45.987 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/21' 2023-10-16T20:45:45.987 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/27' 2023-10-16T20:45:45.987 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/31' 2023-10-16T20:45:45.987 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/36' 2023-10-16T20:45:45.988 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/20' 2023-10-16T20:45:45.988 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/39' 2023-10-16T20:45:45.988 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/34' 2023-10-16T20:45:45.988 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/38' 2023-10-16T20:45:45.988 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/28' 2023-10-16T20:45:45.989 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/24' 2023-10-16T20:45:45.989 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/29' 2023-10-16T20:45:45.989 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/32' 2023-10-16T20:45:45.989 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/22' 2023-10-16T20:45:45.989 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/25' 2023-10-16T20:45:45.989 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/30' 2023-10-16T20:45:45.990 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/33' 2023-10-16T20:45:45.990 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/23' 2023-10-16T20:45:45.990 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/35' 2023-10-16T20:45:45.990 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/26' 2023-10-16T20:45:45.990 INFO:tasks.workunit.client.0.smithi088.stdout:removed './221/37' 2023-10-16T20:45:45.991 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './221' 2023-10-16T20:45:45.991 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/21' 2023-10-16T20:45:45.991 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/27' 2023-10-16T20:45:45.991 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/31' 2023-10-16T20:45:45.991 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/36' 2023-10-16T20:45:45.991 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/20' 2023-10-16T20:45:45.992 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/39' 2023-10-16T20:45:45.992 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/34' 2023-10-16T20:45:45.992 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/38' 2023-10-16T20:45:45.992 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/28' 2023-10-16T20:45:45.992 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/24' 2023-10-16T20:45:45.992 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/29' 2023-10-16T20:45:45.993 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/32' 2023-10-16T20:45:45.993 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/22' 2023-10-16T20:45:45.993 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/25' 2023-10-16T20:45:45.993 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/30' 2023-10-16T20:45:45.993 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/33' 2023-10-16T20:45:45.993 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/23' 2023-10-16T20:45:45.994 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/35' 2023-10-16T20:45:45.994 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/26' 2023-10-16T20:45:45.994 INFO:tasks.workunit.client.0.smithi088.stdout:removed './222/37' 2023-10-16T20:45:45.994 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './222' 2023-10-16T20:45:45.994 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/21' 2023-10-16T20:45:45.995 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/27' 2023-10-16T20:45:45.995 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/31' 2023-10-16T20:45:45.995 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/36' 2023-10-16T20:45:45.995 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/20' 2023-10-16T20:45:45.995 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/39' 2023-10-16T20:45:45.996 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/34' 2023-10-16T20:45:45.996 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/38' 2023-10-16T20:45:45.996 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/28' 2023-10-16T20:45:45.996 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/24' 2023-10-16T20:45:45.996 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/29' 2023-10-16T20:45:45.996 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/32' 2023-10-16T20:45:45.997 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/22' 2023-10-16T20:45:45.997 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/25' 2023-10-16T20:45:45.997 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/30' 2023-10-16T20:45:45.997 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/33' 2023-10-16T20:45:45.997 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/23' 2023-10-16T20:45:45.997 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/35' 2023-10-16T20:45:45.998 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/26' 2023-10-16T20:45:45.998 INFO:tasks.workunit.client.0.smithi088.stdout:removed './223/37' 2023-10-16T20:45:45.998 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './223' 2023-10-16T20:45:45.998 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/21' 2023-10-16T20:45:45.998 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/27' 2023-10-16T20:45:45.998 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/31' 2023-10-16T20:45:45.999 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/36' 2023-10-16T20:45:45.999 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/20' 2023-10-16T20:45:45.999 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/39' 2023-10-16T20:45:45.999 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/34' 2023-10-16T20:45:45.999 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/38' 2023-10-16T20:45:45.999 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/28' 2023-10-16T20:45:46.000 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/24' 2023-10-16T20:45:46.000 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/29' 2023-10-16T20:45:46.000 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/32' 2023-10-16T20:45:46.000 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/22' 2023-10-16T20:45:46.000 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/25' 2023-10-16T20:45:46.000 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/30' 2023-10-16T20:45:46.001 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/33' 2023-10-16T20:45:46.001 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/23' 2023-10-16T20:45:46.001 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/35' 2023-10-16T20:45:46.001 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/26' 2023-10-16T20:45:46.001 INFO:tasks.workunit.client.0.smithi088.stdout:removed './224/37' 2023-10-16T20:45:46.002 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './224' 2023-10-16T20:45:46.002 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/21' 2023-10-16T20:45:46.002 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/27' 2023-10-16T20:45:46.002 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/31' 2023-10-16T20:45:46.002 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/36' 2023-10-16T20:45:46.002 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/20' 2023-10-16T20:45:46.003 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/39' 2023-10-16T20:45:46.003 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/34' 2023-10-16T20:45:46.003 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/38' 2023-10-16T20:45:46.003 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/28' 2023-10-16T20:45:46.003 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/24' 2023-10-16T20:45:46.004 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/29' 2023-10-16T20:45:46.004 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/32' 2023-10-16T20:45:46.004 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/22' 2023-10-16T20:45:46.004 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/25' 2023-10-16T20:45:46.004 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/30' 2023-10-16T20:45:46.004 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/33' 2023-10-16T20:45:46.005 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/23' 2023-10-16T20:45:46.005 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/35' 2023-10-16T20:45:46.005 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/26' 2023-10-16T20:45:46.005 INFO:tasks.workunit.client.0.smithi088.stdout:removed './225/37' 2023-10-16T20:45:46.005 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './225' 2023-10-16T20:45:46.005 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/21' 2023-10-16T20:45:46.005 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/27' 2023-10-16T20:45:46.006 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/31' 2023-10-16T20:45:46.006 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/36' 2023-10-16T20:45:46.006 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/20' 2023-10-16T20:45:46.006 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/39' 2023-10-16T20:45:46.006 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/34' 2023-10-16T20:45:46.006 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/38' 2023-10-16T20:45:46.007 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/28' 2023-10-16T20:45:46.007 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/24' 2023-10-16T20:45:46.599 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/29' 2023-10-16T20:45:46.600 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/32' 2023-10-16T20:45:46.600 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/22' 2023-10-16T20:45:46.600 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/25' 2023-10-16T20:45:46.600 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/30' 2023-10-16T20:45:46.601 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/33' 2023-10-16T20:45:46.601 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/23' 2023-10-16T20:45:46.601 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/35' 2023-10-16T20:45:46.601 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/26' 2023-10-16T20:45:46.601 INFO:tasks.workunit.client.0.smithi088.stdout:removed './226/37' 2023-10-16T20:45:46.602 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './226' 2023-10-16T20:45:46.602 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/21' 2023-10-16T20:45:46.602 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/27' 2023-10-16T20:45:46.602 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/31' 2023-10-16T20:45:46.602 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/36' 2023-10-16T20:45:46.603 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/20' 2023-10-16T20:45:46.603 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/39' 2023-10-16T20:45:46.603 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/34' 2023-10-16T20:45:46.603 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/38' 2023-10-16T20:45:46.603 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/28' 2023-10-16T20:45:46.604 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/24' 2023-10-16T20:45:46.604 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/29' 2023-10-16T20:45:46.604 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/32' 2023-10-16T20:45:46.604 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/22' 2023-10-16T20:45:46.604 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/25' 2023-10-16T20:45:46.604 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/30' 2023-10-16T20:45:46.605 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/33' 2023-10-16T20:45:46.605 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/23' 2023-10-16T20:45:46.605 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/35' 2023-10-16T20:45:46.605 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/26' 2023-10-16T20:45:46.605 INFO:tasks.workunit.client.0.smithi088.stdout:removed './227/37' 2023-10-16T20:45:46.606 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './227' 2023-10-16T20:45:46.606 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/21' 2023-10-16T20:45:46.606 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/27' 2023-10-16T20:45:46.606 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/31' 2023-10-16T20:45:46.606 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/36' 2023-10-16T20:45:46.606 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/20' 2023-10-16T20:45:46.607 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/39' 2023-10-16T20:45:46.607 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/34' 2023-10-16T20:45:46.607 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/38' 2023-10-16T20:45:46.607 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/28' 2023-10-16T20:45:46.607 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/24' 2023-10-16T20:45:46.608 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/29' 2023-10-16T20:45:46.608 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/32' 2023-10-16T20:45:46.608 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/22' 2023-10-16T20:45:46.608 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/25' 2023-10-16T20:45:46.608 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/30' 2023-10-16T20:45:46.609 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/33' 2023-10-16T20:45:46.609 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/23' 2023-10-16T20:45:46.609 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/35' 2023-10-16T20:45:46.609 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/26' 2023-10-16T20:45:46.609 INFO:tasks.workunit.client.0.smithi088.stdout:removed './228/37' 2023-10-16T20:45:46.610 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './228' 2023-10-16T20:45:46.610 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/21' 2023-10-16T20:45:46.610 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/27' 2023-10-16T20:45:46.610 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/31' 2023-10-16T20:45:46.610 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/36' 2023-10-16T20:45:46.611 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/20' 2023-10-16T20:45:46.611 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/39' 2023-10-16T20:45:46.611 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/34' 2023-10-16T20:45:46.611 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/38' 2023-10-16T20:45:46.611 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/28' 2023-10-16T20:45:46.611 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/24' 2023-10-16T20:45:46.612 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/29' 2023-10-16T20:45:46.612 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/32' 2023-10-16T20:45:46.612 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/22' 2023-10-16T20:45:46.612 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/25' 2023-10-16T20:45:46.612 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/30' 2023-10-16T20:45:46.613 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/33' 2023-10-16T20:45:46.613 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/23' 2023-10-16T20:45:46.613 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/35' 2023-10-16T20:45:46.613 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/26' 2023-10-16T20:45:46.613 INFO:tasks.workunit.client.0.smithi088.stdout:removed './229/37' 2023-10-16T20:45:46.614 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './229' 2023-10-16T20:45:46.614 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/21' 2023-10-16T20:45:46.614 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/27' 2023-10-16T20:45:46.614 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/31' 2023-10-16T20:45:46.614 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/36' 2023-10-16T20:45:46.615 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/20' 2023-10-16T20:45:46.615 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/39' 2023-10-16T20:45:46.615 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/34' 2023-10-16T20:45:46.615 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/38' 2023-10-16T20:45:46.615 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/28' 2023-10-16T20:45:46.615 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/24' 2023-10-16T20:45:46.616 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/29' 2023-10-16T20:45:46.616 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/32' 2023-10-16T20:45:46.616 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/22' 2023-10-16T20:45:46.616 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/25' 2023-10-16T20:45:46.616 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/30' 2023-10-16T20:45:46.616 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/33' 2023-10-16T20:45:46.617 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/23' 2023-10-16T20:45:46.617 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/35' 2023-10-16T20:45:46.617 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/26' 2023-10-16T20:45:46.617 INFO:tasks.workunit.client.0.smithi088.stdout:removed './230/37' 2023-10-16T20:45:46.617 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './230' 2023-10-16T20:45:46.618 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/21' 2023-10-16T20:45:46.618 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/27' 2023-10-16T20:45:46.618 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/31' 2023-10-16T20:45:46.618 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/36' 2023-10-16T20:45:46.619 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/20' 2023-10-16T20:45:46.619 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/39' 2023-10-16T20:45:46.619 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/34' 2023-10-16T20:45:46.619 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/38' 2023-10-16T20:45:46.619 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/28' 2023-10-16T20:45:46.619 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/24' 2023-10-16T20:45:46.620 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/29' 2023-10-16T20:45:46.620 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/32' 2023-10-16T20:45:46.620 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/22' 2023-10-16T20:45:46.620 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/25' 2023-10-16T20:45:46.620 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/30' 2023-10-16T20:45:46.620 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/33' 2023-10-16T20:45:46.621 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/23' 2023-10-16T20:45:46.621 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/35' 2023-10-16T20:45:46.621 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/26' 2023-10-16T20:45:46.621 INFO:tasks.workunit.client.0.smithi088.stdout:removed './231/37' 2023-10-16T20:45:46.621 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './231' 2023-10-16T20:45:46.622 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/21' 2023-10-16T20:45:46.622 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/27' 2023-10-16T20:45:46.622 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/31' 2023-10-16T20:45:46.622 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/36' 2023-10-16T20:45:46.622 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/20' 2023-10-16T20:45:46.623 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/39' 2023-10-16T20:45:46.623 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/34' 2023-10-16T20:45:46.623 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/38' 2023-10-16T20:45:46.623 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/28' 2023-10-16T20:45:46.623 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/24' 2023-10-16T20:45:46.623 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/29' 2023-10-16T20:45:46.624 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/32' 2023-10-16T20:45:46.624 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/22' 2023-10-16T20:45:46.624 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/25' 2023-10-16T20:45:46.624 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/30' 2023-10-16T20:45:46.624 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/33' 2023-10-16T20:45:46.625 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/23' 2023-10-16T20:45:46.625 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/35' 2023-10-16T20:45:46.625 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/26' 2023-10-16T20:45:46.625 INFO:tasks.workunit.client.0.smithi088.stdout:removed './232/37' 2023-10-16T20:45:46.625 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './232' 2023-10-16T20:45:46.626 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/21' 2023-10-16T20:45:46.626 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/27' 2023-10-16T20:45:46.626 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/31' 2023-10-16T20:45:46.626 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/36' 2023-10-16T20:45:46.626 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/20' 2023-10-16T20:45:46.627 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/39' 2023-10-16T20:45:46.627 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/34' 2023-10-16T20:45:46.627 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/38' 2023-10-16T20:45:46.627 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/28' 2023-10-16T20:45:46.627 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/24' 2023-10-16T20:45:46.627 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/29' 2023-10-16T20:45:46.628 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/32' 2023-10-16T20:45:46.628 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/22' 2023-10-16T20:45:46.628 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/25' 2023-10-16T20:45:46.628 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/30' 2023-10-16T20:45:46.628 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/33' 2023-10-16T20:45:46.629 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/23' 2023-10-16T20:45:46.629 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/35' 2023-10-16T20:45:46.629 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/26' 2023-10-16T20:45:46.629 INFO:tasks.workunit.client.0.smithi088.stdout:removed './233/37' 2023-10-16T20:45:46.629 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './233' 2023-10-16T20:45:46.629 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/21' 2023-10-16T20:45:46.630 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/27' 2023-10-16T20:45:46.630 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/31' 2023-10-16T20:45:46.630 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/36' 2023-10-16T20:45:46.630 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/20' 2023-10-16T20:45:46.630 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/39' 2023-10-16T20:45:46.631 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/34' 2023-10-16T20:45:46.631 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/38' 2023-10-16T20:45:46.631 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/28' 2023-10-16T20:45:46.631 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/24' 2023-10-16T20:45:46.631 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/29' 2023-10-16T20:45:46.632 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/32' 2023-10-16T20:45:46.632 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/22' 2023-10-16T20:45:46.632 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/25' 2023-10-16T20:45:46.632 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/30' 2023-10-16T20:45:46.632 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/33' 2023-10-16T20:45:46.632 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/23' 2023-10-16T20:45:46.633 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/35' 2023-10-16T20:45:46.633 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/26' 2023-10-16T20:45:46.633 INFO:tasks.workunit.client.0.smithi088.stdout:removed './234/37' 2023-10-16T20:45:46.633 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './234' 2023-10-16T20:45:46.633 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/21' 2023-10-16T20:45:46.634 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/27' 2023-10-16T20:45:46.634 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/31' 2023-10-16T20:45:46.634 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/36' 2023-10-16T20:45:46.634 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/20' 2023-10-16T20:45:46.634 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/39' 2023-10-16T20:45:46.635 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/34' 2023-10-16T20:45:46.635 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/38' 2023-10-16T20:45:46.635 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/28' 2023-10-16T20:45:46.635 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/24' 2023-10-16T20:45:46.635 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/29' 2023-10-16T20:45:46.636 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/32' 2023-10-16T20:45:46.636 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/22' 2023-10-16T20:45:46.636 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/25' 2023-10-16T20:45:46.636 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/30' 2023-10-16T20:45:46.636 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/33' 2023-10-16T20:45:46.637 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/23' 2023-10-16T20:45:46.637 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/35' 2023-10-16T20:45:46.637 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/26' 2023-10-16T20:45:46.637 INFO:tasks.workunit.client.0.smithi088.stdout:removed './235/37' 2023-10-16T20:45:46.637 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './235' 2023-10-16T20:45:46.637 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/21' 2023-10-16T20:45:46.638 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/27' 2023-10-16T20:45:46.638 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/31' 2023-10-16T20:45:46.638 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/36' 2023-10-16T20:45:46.638 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/20' 2023-10-16T20:45:46.638 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/39' 2023-10-16T20:45:46.639 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/34' 2023-10-16T20:45:46.639 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/38' 2023-10-16T20:45:46.639 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/28' 2023-10-16T20:45:46.639 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/24' 2023-10-16T20:45:46.639 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/29' 2023-10-16T20:45:46.640 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/32' 2023-10-16T20:45:47.417 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/22' 2023-10-16T20:45:47.417 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/25' 2023-10-16T20:45:47.418 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/30' 2023-10-16T20:45:47.418 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/33' 2023-10-16T20:45:47.418 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/23' 2023-10-16T20:45:47.418 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/35' 2023-10-16T20:45:47.418 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/26' 2023-10-16T20:45:47.419 INFO:tasks.workunit.client.0.smithi088.stdout:removed './236/37' 2023-10-16T20:45:47.419 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './236' 2023-10-16T20:45:47.419 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/21' 2023-10-16T20:45:47.419 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/27' 2023-10-16T20:45:47.419 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/31' 2023-10-16T20:45:47.419 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/36' 2023-10-16T20:45:47.420 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/20' 2023-10-16T20:45:47.420 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/39' 2023-10-16T20:45:47.420 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/34' 2023-10-16T20:45:47.420 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/38' 2023-10-16T20:45:47.420 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/28' 2023-10-16T20:45:47.421 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/24' 2023-10-16T20:45:47.421 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/29' 2023-10-16T20:45:47.421 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/32' 2023-10-16T20:45:47.421 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/22' 2023-10-16T20:45:47.421 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/25' 2023-10-16T20:45:47.422 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/30' 2023-10-16T20:45:47.422 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/33' 2023-10-16T20:45:47.422 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/23' 2023-10-16T20:45:47.422 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/35' 2023-10-16T20:45:47.422 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/26' 2023-10-16T20:45:47.422 INFO:tasks.workunit.client.0.smithi088.stdout:removed './237/37' 2023-10-16T20:45:47.423 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './237' 2023-10-16T20:45:47.423 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/21' 2023-10-16T20:45:47.423 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/27' 2023-10-16T20:45:47.423 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/31' 2023-10-16T20:45:47.423 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/36' 2023-10-16T20:45:47.423 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/20' 2023-10-16T20:45:47.423 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/39' 2023-10-16T20:45:47.424 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/34' 2023-10-16T20:45:47.424 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/38' 2023-10-16T20:45:47.424 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/28' 2023-10-16T20:45:47.424 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/24' 2023-10-16T20:45:47.424 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/29' 2023-10-16T20:45:47.425 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/32' 2023-10-16T20:45:47.425 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/22' 2023-10-16T20:45:47.425 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/25' 2023-10-16T20:45:47.425 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/30' 2023-10-16T20:45:47.425 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/33' 2023-10-16T20:45:47.425 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/23' 2023-10-16T20:45:47.426 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/35' 2023-10-16T20:45:47.426 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/26' 2023-10-16T20:45:47.426 INFO:tasks.workunit.client.0.smithi088.stdout:removed './238/37' 2023-10-16T20:45:47.426 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './238' 2023-10-16T20:45:47.426 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/21' 2023-10-16T20:45:47.427 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/27' 2023-10-16T20:45:47.427 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/31' 2023-10-16T20:45:47.427 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/36' 2023-10-16T20:45:47.427 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/20' 2023-10-16T20:45:47.427 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/39' 2023-10-16T20:45:47.427 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/34' 2023-10-16T20:45:47.428 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/38' 2023-10-16T20:45:47.428 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/28' 2023-10-16T20:45:47.428 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/24' 2023-10-16T20:45:47.428 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/29' 2023-10-16T20:45:47.428 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/32' 2023-10-16T20:45:47.428 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/22' 2023-10-16T20:45:47.428 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/25' 2023-10-16T20:45:47.429 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/30' 2023-10-16T20:45:47.429 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/33' 2023-10-16T20:45:47.429 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/23' 2023-10-16T20:45:47.429 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/35' 2023-10-16T20:45:47.429 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/26' 2023-10-16T20:45:47.430 INFO:tasks.workunit.client.0.smithi088.stdout:removed './239/37' 2023-10-16T20:45:47.430 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './239' 2023-10-16T20:45:47.430 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/21' 2023-10-16T20:45:47.430 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/27' 2023-10-16T20:45:47.430 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/31' 2023-10-16T20:45:47.431 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/36' 2023-10-16T20:45:47.431 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/20' 2023-10-16T20:45:47.431 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/39' 2023-10-16T20:45:47.431 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/34' 2023-10-16T20:45:47.431 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/38' 2023-10-16T20:45:47.431 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/28' 2023-10-16T20:45:47.432 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/24' 2023-10-16T20:45:47.432 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/29' 2023-10-16T20:45:47.432 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/32' 2023-10-16T20:45:47.432 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/22' 2023-10-16T20:45:47.432 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/25' 2023-10-16T20:45:47.432 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/30' 2023-10-16T20:45:47.433 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/33' 2023-10-16T20:45:47.433 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/23' 2023-10-16T20:45:47.433 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/35' 2023-10-16T20:45:47.433 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/26' 2023-10-16T20:45:47.433 INFO:tasks.workunit.client.0.smithi088.stdout:removed './240/37' 2023-10-16T20:45:47.433 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './240' 2023-10-16T20:45:47.434 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/21' 2023-10-16T20:45:47.434 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/27' 2023-10-16T20:45:47.434 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/31' 2023-10-16T20:45:47.434 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/36' 2023-10-16T20:45:47.434 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/20' 2023-10-16T20:45:47.435 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/39' 2023-10-16T20:45:47.435 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/34' 2023-10-16T20:45:47.435 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/38' 2023-10-16T20:45:47.435 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/28' 2023-10-16T20:45:47.435 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/24' 2023-10-16T20:45:47.435 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/29' 2023-10-16T20:45:47.436 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/32' 2023-10-16T20:45:47.436 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/22' 2023-10-16T20:45:47.436 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/25' 2023-10-16T20:45:47.436 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/30' 2023-10-16T20:45:47.436 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/33' 2023-10-16T20:45:47.436 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/23' 2023-10-16T20:45:47.437 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/35' 2023-10-16T20:45:47.437 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/26' 2023-10-16T20:45:47.437 INFO:tasks.workunit.client.0.smithi088.stdout:removed './241/37' 2023-10-16T20:45:47.437 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './241' 2023-10-16T20:45:47.437 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/21' 2023-10-16T20:45:47.437 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/27' 2023-10-16T20:45:47.438 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/31' 2023-10-16T20:45:47.438 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/36' 2023-10-16T20:45:47.438 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/20' 2023-10-16T20:45:47.438 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/39' 2023-10-16T20:45:47.438 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/34' 2023-10-16T20:45:47.439 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/38' 2023-10-16T20:45:47.439 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/28' 2023-10-16T20:45:47.439 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/24' 2023-10-16T20:45:47.439 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/29' 2023-10-16T20:45:47.439 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/32' 2023-10-16T20:45:47.439 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/22' 2023-10-16T20:45:47.440 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/25' 2023-10-16T20:45:47.440 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/30' 2023-10-16T20:45:47.440 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/33' 2023-10-16T20:45:47.440 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/23' 2023-10-16T20:45:47.440 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/35' 2023-10-16T20:45:47.440 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/26' 2023-10-16T20:45:47.441 INFO:tasks.workunit.client.0.smithi088.stdout:removed './242/37' 2023-10-16T20:45:47.441 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './242' 2023-10-16T20:45:47.441 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/21' 2023-10-16T20:45:47.441 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/27' 2023-10-16T20:45:47.441 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/31' 2023-10-16T20:45:47.442 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/36' 2023-10-16T20:45:47.442 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/20' 2023-10-16T20:45:47.442 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/39' 2023-10-16T20:45:47.442 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/34' 2023-10-16T20:45:47.442 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/38' 2023-10-16T20:45:47.442 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/28' 2023-10-16T20:45:47.443 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/24' 2023-10-16T20:45:47.443 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/29' 2023-10-16T20:45:47.443 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/32' 2023-10-16T20:45:47.443 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/22' 2023-10-16T20:45:47.443 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/25' 2023-10-16T20:45:47.443 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/30' 2023-10-16T20:45:47.444 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/33' 2023-10-16T20:45:47.444 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/23' 2023-10-16T20:45:47.444 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/35' 2023-10-16T20:45:47.444 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/26' 2023-10-16T20:45:47.444 INFO:tasks.workunit.client.0.smithi088.stdout:removed './243/37' 2023-10-16T20:45:47.444 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './243' 2023-10-16T20:45:47.445 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/21' 2023-10-16T20:45:47.445 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/27' 2023-10-16T20:45:47.445 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/31' 2023-10-16T20:45:47.445 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/36' 2023-10-16T20:45:47.445 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/20' 2023-10-16T20:45:47.445 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/39' 2023-10-16T20:45:47.446 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/34' 2023-10-16T20:45:47.446 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/38' 2023-10-16T20:45:47.446 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/28' 2023-10-16T20:45:47.446 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/24' 2023-10-16T20:45:47.446 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/29' 2023-10-16T20:45:47.447 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/32' 2023-10-16T20:45:47.447 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/22' 2023-10-16T20:45:47.447 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/25' 2023-10-16T20:45:47.447 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/30' 2023-10-16T20:45:47.447 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/33' 2023-10-16T20:45:47.447 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/23' 2023-10-16T20:45:47.448 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/35' 2023-10-16T20:45:47.448 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/26' 2023-10-16T20:45:47.448 INFO:tasks.workunit.client.0.smithi088.stdout:removed './244/37' 2023-10-16T20:45:47.448 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './244' 2023-10-16T20:45:47.448 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/21' 2023-10-16T20:45:47.448 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/27' 2023-10-16T20:45:47.448 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/31' 2023-10-16T20:45:47.449 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/36' 2023-10-16T20:45:47.449 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/20' 2023-10-16T20:45:47.449 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/39' 2023-10-16T20:45:47.449 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/34' 2023-10-16T20:45:47.449 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/38' 2023-10-16T20:45:47.450 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/28' 2023-10-16T20:45:47.450 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/24' 2023-10-16T20:45:47.450 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/29' 2023-10-16T20:45:47.450 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/32' 2023-10-16T20:45:47.450 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/22' 2023-10-16T20:45:47.450 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/25' 2023-10-16T20:45:47.451 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/30' 2023-10-16T20:45:47.451 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/33' 2023-10-16T20:45:47.451 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/23' 2023-10-16T20:45:47.451 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/35' 2023-10-16T20:45:47.451 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/26' 2023-10-16T20:45:47.452 INFO:tasks.workunit.client.0.smithi088.stdout:removed './245/37' 2023-10-16T20:45:47.452 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './245' 2023-10-16T20:45:47.452 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/21' 2023-10-16T20:45:47.452 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/27' 2023-10-16T20:45:47.452 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/31' 2023-10-16T20:45:47.453 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/36' 2023-10-16T20:45:47.453 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/20' 2023-10-16T20:45:47.453 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/39' 2023-10-16T20:45:47.453 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/34' 2023-10-16T20:45:47.453 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/38' 2023-10-16T20:45:47.453 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/28' 2023-10-16T20:45:47.454 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/24' 2023-10-16T20:45:47.454 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/29' 2023-10-16T20:45:47.454 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/32' 2023-10-16T20:45:47.454 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/22' 2023-10-16T20:45:50.863 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/25' 2023-10-16T20:45:50.864 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/30' 2023-10-16T20:45:50.864 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/33' 2023-10-16T20:45:50.864 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/23' 2023-10-16T20:45:50.864 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/35' 2023-10-16T20:45:50.864 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/26' 2023-10-16T20:45:50.865 INFO:tasks.workunit.client.0.smithi088.stdout:removed './246/37' 2023-10-16T20:45:50.865 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './246' 2023-10-16T20:45:50.865 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/21' 2023-10-16T20:45:50.865 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/27' 2023-10-16T20:45:50.865 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/31' 2023-10-16T20:45:50.866 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/36' 2023-10-16T20:45:50.866 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/20' 2023-10-16T20:45:50.866 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/39' 2023-10-16T20:45:50.866 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/34' 2023-10-16T20:45:50.866 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/38' 2023-10-16T20:45:50.867 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/28' 2023-10-16T20:45:50.867 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/24' 2023-10-16T20:45:50.867 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/29' 2023-10-16T20:45:50.867 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/32' 2023-10-16T20:45:50.867 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/22' 2023-10-16T20:45:50.867 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/25' 2023-10-16T20:45:50.868 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/30' 2023-10-16T20:45:50.868 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/33' 2023-10-16T20:45:50.868 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/23' 2023-10-16T20:45:50.868 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/35' 2023-10-16T20:45:50.868 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/26' 2023-10-16T20:45:50.869 INFO:tasks.workunit.client.0.smithi088.stdout:removed './247/37' 2023-10-16T20:45:50.869 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './247' 2023-10-16T20:45:50.869 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/21' 2023-10-16T20:45:50.869 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/27' 2023-10-16T20:45:50.869 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/31' 2023-10-16T20:45:50.870 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/36' 2023-10-16T20:45:50.870 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/20' 2023-10-16T20:45:50.870 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/39' 2023-10-16T20:45:50.870 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/34' 2023-10-16T20:45:50.870 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/38' 2023-10-16T20:45:50.871 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/28' 2023-10-16T20:45:50.871 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/24' 2023-10-16T20:45:50.871 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/29' 2023-10-16T20:45:50.871 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/32' 2023-10-16T20:45:50.871 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/22' 2023-10-16T20:45:50.871 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/25' 2023-10-16T20:45:50.872 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/30' 2023-10-16T20:45:50.872 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/33' 2023-10-16T20:45:50.872 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/23' 2023-10-16T20:45:50.872 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/35' 2023-10-16T20:45:50.872 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/26' 2023-10-16T20:45:50.873 INFO:tasks.workunit.client.0.smithi088.stdout:removed './248/37' 2023-10-16T20:45:50.873 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './248' 2023-10-16T20:45:50.873 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/21' 2023-10-16T20:45:50.873 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/27' 2023-10-16T20:45:50.873 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/31' 2023-10-16T20:45:50.873 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/36' 2023-10-16T20:45:50.874 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/20' 2023-10-16T20:45:50.874 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/39' 2023-10-16T20:45:50.874 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/34' 2023-10-16T20:45:50.874 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/38' 2023-10-16T20:45:50.874 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/28' 2023-10-16T20:45:50.875 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/24' 2023-10-16T20:45:50.875 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/29' 2023-10-16T20:45:50.875 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/32' 2023-10-16T20:45:50.875 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/22' 2023-10-16T20:45:50.875 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/25' 2023-10-16T20:45:50.876 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/30' 2023-10-16T20:45:50.876 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/33' 2023-10-16T20:45:50.876 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/23' 2023-10-16T20:45:50.876 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/35' 2023-10-16T20:45:50.876 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/26' 2023-10-16T20:45:50.877 INFO:tasks.workunit.client.0.smithi088.stdout:removed './249/37' 2023-10-16T20:45:50.877 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './249' 2023-10-16T20:45:50.877 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/21' 2023-10-16T20:45:50.877 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/27' 2023-10-16T20:45:50.877 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/31' 2023-10-16T20:45:50.877 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/36' 2023-10-16T20:45:50.878 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/20' 2023-10-16T20:45:50.878 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/39' 2023-10-16T20:45:50.878 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/34' 2023-10-16T20:45:50.878 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/38' 2023-10-16T20:45:50.878 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/28' 2023-10-16T20:45:50.879 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/24' 2023-10-16T20:45:50.879 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/29' 2023-10-16T20:45:50.879 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/32' 2023-10-16T20:45:50.879 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/22' 2023-10-16T20:45:50.879 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/25' 2023-10-16T20:45:50.879 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/30' 2023-10-16T20:45:50.880 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/33' 2023-10-16T20:45:50.880 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/23' 2023-10-16T20:45:50.880 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/35' 2023-10-16T20:45:50.880 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/26' 2023-10-16T20:45:50.880 INFO:tasks.workunit.client.0.smithi088.stdout:removed './250/37' 2023-10-16T20:45:50.881 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './250' 2023-10-16T20:45:50.881 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/21' 2023-10-16T20:45:50.881 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/27' 2023-10-16T20:45:50.881 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/31' 2023-10-16T20:45:50.881 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/36' 2023-10-16T20:45:50.882 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/20' 2023-10-16T20:45:50.882 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/39' 2023-10-16T20:45:50.882 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/34' 2023-10-16T20:45:50.882 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/38' 2023-10-16T20:45:50.882 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/28' 2023-10-16T20:45:50.883 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/24' 2023-10-16T20:45:50.883 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/29' 2023-10-16T20:45:50.883 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/32' 2023-10-16T20:45:50.883 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/22' 2023-10-16T20:45:50.883 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/25' 2023-10-16T20:45:50.883 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/30' 2023-10-16T20:45:50.884 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/33' 2023-10-16T20:45:50.884 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/23' 2023-10-16T20:45:50.884 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/35' 2023-10-16T20:45:50.884 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/26' 2023-10-16T20:45:50.884 INFO:tasks.workunit.client.0.smithi088.stdout:removed './251/37' 2023-10-16T20:45:50.885 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './251' 2023-10-16T20:45:50.885 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/21' 2023-10-16T20:45:50.885 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/27' 2023-10-16T20:45:50.885 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/31' 2023-10-16T20:45:50.885 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/36' 2023-10-16T20:45:50.885 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/20' 2023-10-16T20:45:50.886 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/39' 2023-10-16T20:45:50.886 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/34' 2023-10-16T20:45:50.886 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/38' 2023-10-16T20:45:50.886 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/28' 2023-10-16T20:45:50.886 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/24' 2023-10-16T20:45:50.887 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/29' 2023-10-16T20:45:50.887 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/32' 2023-10-16T20:45:50.887 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/22' 2023-10-16T20:45:50.887 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/25' 2023-10-16T20:45:50.887 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/30' 2023-10-16T20:45:50.888 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/33' 2023-10-16T20:45:50.888 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/23' 2023-10-16T20:45:50.888 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/35' 2023-10-16T20:45:50.888 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/26' 2023-10-16T20:45:50.888 INFO:tasks.workunit.client.0.smithi088.stdout:removed './252/37' 2023-10-16T20:45:50.888 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './252' 2023-10-16T20:45:50.889 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/21' 2023-10-16T20:45:50.889 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/27' 2023-10-16T20:45:50.889 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/31' 2023-10-16T20:45:50.889 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/36' 2023-10-16T20:45:50.889 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/20' 2023-10-16T20:45:50.890 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/39' 2023-10-16T20:45:50.890 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/34' 2023-10-16T20:45:50.890 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/38' 2023-10-16T20:45:50.890 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/28' 2023-10-16T20:45:50.890 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/24' 2023-10-16T20:45:50.891 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/29' 2023-10-16T20:45:50.891 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/32' 2023-10-16T20:45:50.891 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/22' 2023-10-16T20:45:50.891 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/25' 2023-10-16T20:45:50.891 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/30' 2023-10-16T20:45:50.891 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/33' 2023-10-16T20:45:50.892 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/23' 2023-10-16T20:45:50.892 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/35' 2023-10-16T20:45:50.892 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/26' 2023-10-16T20:45:50.892 INFO:tasks.workunit.client.0.smithi088.stdout:removed './253/37' 2023-10-16T20:45:50.892 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './253' 2023-10-16T20:45:50.893 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/21' 2023-10-16T20:45:50.893 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/27' 2023-10-16T20:45:50.893 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/31' 2023-10-16T20:45:50.893 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/36' 2023-10-16T20:45:50.893 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/20' 2023-10-16T20:45:50.893 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/39' 2023-10-16T20:45:50.894 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/34' 2023-10-16T20:45:50.894 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/38' 2023-10-16T20:45:50.894 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/28' 2023-10-16T20:45:50.894 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/24' 2023-10-16T20:45:50.894 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/29' 2023-10-16T20:45:50.895 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/32' 2023-10-16T20:45:50.895 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/22' 2023-10-16T20:45:50.895 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/25' 2023-10-16T20:45:50.895 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/30' 2023-10-16T20:45:50.895 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/33' 2023-10-16T20:45:50.895 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/23' 2023-10-16T20:45:50.896 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/35' 2023-10-16T20:45:50.896 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/26' 2023-10-16T20:45:50.896 INFO:tasks.workunit.client.0.smithi088.stdout:removed './254/37' 2023-10-16T20:45:50.896 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './254' 2023-10-16T20:45:50.896 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/21' 2023-10-16T20:45:50.897 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/27' 2023-10-16T20:45:50.897 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/31' 2023-10-16T20:45:50.897 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/36' 2023-10-16T20:45:50.897 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/20' 2023-10-16T20:45:50.897 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/39' 2023-10-16T20:45:50.898 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/34' 2023-10-16T20:45:50.898 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/38' 2023-10-16T20:45:50.898 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/28' 2023-10-16T20:45:50.898 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/24' 2023-10-16T20:45:50.898 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/29' 2023-10-16T20:45:50.899 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/32' 2023-10-16T20:45:50.899 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/22' 2023-10-16T20:45:50.899 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/25' 2023-10-16T20:45:50.899 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/30' 2023-10-16T20:45:50.899 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/33' 2023-10-16T20:45:50.900 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/23' 2023-10-16T20:45:50.900 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/35' 2023-10-16T20:45:50.900 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/26' 2023-10-16T20:45:50.900 INFO:tasks.workunit.client.0.smithi088.stdout:removed './255/37' 2023-10-16T20:45:50.900 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './255' 2023-10-16T20:45:50.900 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/21' 2023-10-16T20:45:50.901 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/27' 2023-10-16T20:45:50.901 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/31' 2023-10-16T20:45:50.901 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/36' 2023-10-16T20:45:50.901 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/20' 2023-10-16T20:45:50.901 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/39' 2023-10-16T20:45:50.902 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/34' 2023-10-16T20:45:50.902 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/38' 2023-10-16T20:45:50.902 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/28' 2023-10-16T20:45:50.902 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/24' 2023-10-16T20:45:50.902 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/29' 2023-10-16T20:45:50.903 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/32' 2023-10-16T20:45:50.903 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/22' 2023-10-16T20:45:50.903 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/25' 2023-10-16T20:45:50.903 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/30' 2023-10-16T20:45:54.421 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/33' 2023-10-16T20:45:54.422 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/23' 2023-10-16T20:45:54.422 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/35' 2023-10-16T20:45:54.422 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/26' 2023-10-16T20:45:54.422 INFO:tasks.workunit.client.0.smithi088.stdout:removed './256/37' 2023-10-16T20:45:54.422 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './256' 2023-10-16T20:45:54.423 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/21' 2023-10-16T20:45:54.423 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/27' 2023-10-16T20:45:54.423 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/31' 2023-10-16T20:45:54.423 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/36' 2023-10-16T20:45:54.423 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/20' 2023-10-16T20:45:54.424 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/39' 2023-10-16T20:45:54.424 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/34' 2023-10-16T20:45:54.424 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/38' 2023-10-16T20:45:54.424 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/28' 2023-10-16T20:45:54.424 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/24' 2023-10-16T20:45:54.424 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/29' 2023-10-16T20:45:54.425 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/32' 2023-10-16T20:45:54.425 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/22' 2023-10-16T20:45:54.425 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/25' 2023-10-16T20:45:54.425 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/30' 2023-10-16T20:45:54.425 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/33' 2023-10-16T20:45:54.426 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/23' 2023-10-16T20:45:54.426 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/35' 2023-10-16T20:45:54.426 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/26' 2023-10-16T20:45:54.426 INFO:tasks.workunit.client.0.smithi088.stdout:removed './257/37' 2023-10-16T20:45:54.426 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './257' 2023-10-16T20:45:54.427 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/21' 2023-10-16T20:45:54.427 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/27' 2023-10-16T20:45:54.427 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/31' 2023-10-16T20:45:54.427 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/36' 2023-10-16T20:45:54.427 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/20' 2023-10-16T20:45:54.428 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/39' 2023-10-16T20:45:54.428 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/34' 2023-10-16T20:45:54.428 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/38' 2023-10-16T20:45:54.428 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/28' 2023-10-16T20:45:54.428 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/24' 2023-10-16T20:45:54.429 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/29' 2023-10-16T20:45:54.429 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/32' 2023-10-16T20:45:54.429 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/22' 2023-10-16T20:45:54.429 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/25' 2023-10-16T20:45:54.429 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/30' 2023-10-16T20:45:54.430 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/33' 2023-10-16T20:45:54.430 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/23' 2023-10-16T20:45:54.430 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/35' 2023-10-16T20:45:54.430 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/26' 2023-10-16T20:45:54.430 INFO:tasks.workunit.client.0.smithi088.stdout:removed './258/37' 2023-10-16T20:45:54.431 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './258' 2023-10-16T20:45:54.431 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/21' 2023-10-16T20:45:54.431 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/27' 2023-10-16T20:45:54.431 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/31' 2023-10-16T20:45:54.431 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/36' 2023-10-16T20:45:54.432 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/20' 2023-10-16T20:45:54.432 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/39' 2023-10-16T20:45:54.432 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/34' 2023-10-16T20:45:54.432 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/38' 2023-10-16T20:45:54.432 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/28' 2023-10-16T20:45:54.433 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/24' 2023-10-16T20:45:54.433 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/29' 2023-10-16T20:45:54.433 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/32' 2023-10-16T20:45:54.433 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/22' 2023-10-16T20:45:54.433 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/25' 2023-10-16T20:45:54.434 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/30' 2023-10-16T20:45:54.434 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/33' 2023-10-16T20:45:54.434 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/23' 2023-10-16T20:45:54.434 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/35' 2023-10-16T20:45:54.434 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/26' 2023-10-16T20:45:54.435 INFO:tasks.workunit.client.0.smithi088.stdout:removed './259/37' 2023-10-16T20:45:54.435 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './259' 2023-10-16T20:45:54.435 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/21' 2023-10-16T20:45:54.435 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/27' 2023-10-16T20:45:54.435 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/31' 2023-10-16T20:45:54.435 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/36' 2023-10-16T20:45:54.436 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/20' 2023-10-16T20:45:54.436 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/39' 2023-10-16T20:45:54.436 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/34' 2023-10-16T20:45:54.436 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/38' 2023-10-16T20:45:54.436 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/28' 2023-10-16T20:45:54.437 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/24' 2023-10-16T20:45:54.437 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/29' 2023-10-16T20:45:54.437 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/32' 2023-10-16T20:45:54.437 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/22' 2023-10-16T20:45:54.437 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/25' 2023-10-16T20:45:54.437 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/30' 2023-10-16T20:45:54.438 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/33' 2023-10-16T20:45:54.438 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/23' 2023-10-16T20:45:54.438 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/35' 2023-10-16T20:45:54.438 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/26' 2023-10-16T20:45:54.438 INFO:tasks.workunit.client.0.smithi088.stdout:removed './260/37' 2023-10-16T20:45:54.439 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './260' 2023-10-16T20:45:54.439 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/21' 2023-10-16T20:45:54.439 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/27' 2023-10-16T20:45:54.439 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/31' 2023-10-16T20:45:54.439 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/36' 2023-10-16T20:45:54.440 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/20' 2023-10-16T20:45:54.440 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/39' 2023-10-16T20:45:54.440 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/34' 2023-10-16T20:45:54.440 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/38' 2023-10-16T20:45:54.440 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/28' 2023-10-16T20:45:54.441 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/24' 2023-10-16T20:45:54.441 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/29' 2023-10-16T20:45:54.441 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/32' 2023-10-16T20:45:54.441 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/22' 2023-10-16T20:45:54.441 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/25' 2023-10-16T20:45:54.442 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/30' 2023-10-16T20:45:54.442 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/33' 2023-10-16T20:45:54.442 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/23' 2023-10-16T20:45:54.442 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/35' 2023-10-16T20:45:54.442 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/26' 2023-10-16T20:45:54.443 INFO:tasks.workunit.client.0.smithi088.stdout:removed './261/37' 2023-10-16T20:45:54.443 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './261' 2023-10-16T20:45:54.443 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/21' 2023-10-16T20:45:54.443 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/27' 2023-10-16T20:45:54.443 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/31' 2023-10-16T20:45:54.443 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/36' 2023-10-16T20:45:54.444 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/20' 2023-10-16T20:45:54.444 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/39' 2023-10-16T20:45:54.444 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/34' 2023-10-16T20:45:54.444 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/38' 2023-10-16T20:45:54.444 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/28' 2023-10-16T20:45:54.445 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/24' 2023-10-16T20:45:54.445 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/29' 2023-10-16T20:45:54.445 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/32' 2023-10-16T20:45:54.445 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/22' 2023-10-16T20:45:54.445 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/25' 2023-10-16T20:45:54.446 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/30' 2023-10-16T20:45:54.446 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/33' 2023-10-16T20:45:54.446 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/23' 2023-10-16T20:45:54.446 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/35' 2023-10-16T20:45:54.446 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/26' 2023-10-16T20:45:54.446 INFO:tasks.workunit.client.0.smithi088.stdout:removed './262/37' 2023-10-16T20:45:54.447 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './262' 2023-10-16T20:45:54.447 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/21' 2023-10-16T20:45:54.447 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/27' 2023-10-16T20:45:54.447 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/31' 2023-10-16T20:45:54.447 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/36' 2023-10-16T20:45:54.448 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/20' 2023-10-16T20:45:54.448 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/39' 2023-10-16T20:45:54.448 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/34' 2023-10-16T20:45:54.448 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/38' 2023-10-16T20:45:54.448 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/28' 2023-10-16T20:45:54.449 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/24' 2023-10-16T20:45:54.449 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/29' 2023-10-16T20:45:54.449 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/32' 2023-10-16T20:45:54.449 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/22' 2023-10-16T20:45:54.449 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/25' 2023-10-16T20:45:54.450 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/30' 2023-10-16T20:45:54.450 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/33' 2023-10-16T20:45:54.450 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/23' 2023-10-16T20:45:54.450 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/35' 2023-10-16T20:45:54.450 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/26' 2023-10-16T20:45:54.451 INFO:tasks.workunit.client.0.smithi088.stdout:removed './263/37' 2023-10-16T20:45:54.451 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './263' 2023-10-16T20:45:54.451 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/21' 2023-10-16T20:45:54.451 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/27' 2023-10-16T20:45:54.451 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/31' 2023-10-16T20:45:54.451 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/36' 2023-10-16T20:45:54.452 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/20' 2023-10-16T20:45:54.452 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/39' 2023-10-16T20:45:54.452 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/34' 2023-10-16T20:45:54.452 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/38' 2023-10-16T20:45:54.452 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/28' 2023-10-16T20:45:54.452 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/24' 2023-10-16T20:45:54.453 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/29' 2023-10-16T20:45:54.453 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/32' 2023-10-16T20:45:54.453 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/22' 2023-10-16T20:45:54.453 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/25' 2023-10-16T20:45:54.453 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/30' 2023-10-16T20:45:54.453 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/33' 2023-10-16T20:45:54.454 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/23' 2023-10-16T20:45:54.454 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/35' 2023-10-16T20:45:54.454 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/26' 2023-10-16T20:45:54.454 INFO:tasks.workunit.client.0.smithi088.stdout:removed './264/37' 2023-10-16T20:45:54.455 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './264' 2023-10-16T20:45:54.455 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/21' 2023-10-16T20:45:54.455 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/27' 2023-10-16T20:45:54.455 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/31' 2023-10-16T20:45:54.455 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/36' 2023-10-16T20:45:54.456 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/20' 2023-10-16T20:45:54.456 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/39' 2023-10-16T20:45:54.456 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/34' 2023-10-16T20:45:54.456 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/38' 2023-10-16T20:45:54.456 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/28' 2023-10-16T20:45:54.457 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/24' 2023-10-16T20:45:54.457 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/29' 2023-10-16T20:45:54.457 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/32' 2023-10-16T20:45:54.457 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/22' 2023-10-16T20:45:54.457 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/25' 2023-10-16T20:45:54.457 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/30' 2023-10-16T20:45:54.458 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/33' 2023-10-16T20:45:54.458 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/23' 2023-10-16T20:45:54.458 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/35' 2023-10-16T20:45:54.458 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/26' 2023-10-16T20:45:54.458 INFO:tasks.workunit.client.0.smithi088.stdout:removed './265/37' 2023-10-16T20:45:54.459 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './265' 2023-10-16T20:45:54.459 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/21' 2023-10-16T20:45:54.459 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/27' 2023-10-16T20:45:54.459 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/31' 2023-10-16T20:45:54.459 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/36' 2023-10-16T20:45:54.460 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/20' 2023-10-16T20:45:54.460 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/39' 2023-10-16T20:45:54.460 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/34' 2023-10-16T20:45:54.460 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/38' 2023-10-16T20:45:54.460 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/28' 2023-10-16T20:45:54.460 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/24' 2023-10-16T20:45:54.461 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/29' 2023-10-16T20:45:54.461 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/32' 2023-10-16T20:45:54.461 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/22' 2023-10-16T20:45:54.461 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/25' 2023-10-16T20:45:54.461 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/30' 2023-10-16T20:45:54.462 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/33' 2023-10-16T20:45:54.462 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/23' 2023-10-16T20:45:56.151 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:45:56.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:45:57.060 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/35' 2023-10-16T20:45:57.060 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/26' 2023-10-16T20:45:57.061 INFO:tasks.workunit.client.0.smithi088.stdout:removed './266/37' 2023-10-16T20:45:57.061 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './266' 2023-10-16T20:45:57.061 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/21' 2023-10-16T20:45:57.061 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/27' 2023-10-16T20:45:57.062 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/31' 2023-10-16T20:45:57.062 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/36' 2023-10-16T20:45:57.062 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/20' 2023-10-16T20:45:57.062 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/39' 2023-10-16T20:45:57.062 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/34' 2023-10-16T20:45:57.063 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/38' 2023-10-16T20:45:57.063 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/28' 2023-10-16T20:45:57.063 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/24' 2023-10-16T20:45:57.063 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/29' 2023-10-16T20:45:57.064 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/32' 2023-10-16T20:45:57.064 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/22' 2023-10-16T20:45:57.064 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/25' 2023-10-16T20:45:57.064 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/30' 2023-10-16T20:45:57.064 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/33' 2023-10-16T20:45:57.065 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/23' 2023-10-16T20:45:57.065 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/35' 2023-10-16T20:45:57.065 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/26' 2023-10-16T20:45:57.065 INFO:tasks.workunit.client.0.smithi088.stdout:removed './267/37' 2023-10-16T20:45:57.066 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './267' 2023-10-16T20:45:57.066 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/21' 2023-10-16T20:45:57.066 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/27' 2023-10-16T20:45:57.066 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/31' 2023-10-16T20:45:57.066 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/36' 2023-10-16T20:45:57.067 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/20' 2023-10-16T20:45:57.067 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/39' 2023-10-16T20:45:57.067 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/34' 2023-10-16T20:45:57.067 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/38' 2023-10-16T20:45:57.067 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/28' 2023-10-16T20:45:57.068 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/24' 2023-10-16T20:45:57.068 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/29' 2023-10-16T20:45:57.068 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/32' 2023-10-16T20:45:57.068 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/22' 2023-10-16T20:45:57.069 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/25' 2023-10-16T20:45:57.069 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/30' 2023-10-16T20:45:57.069 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/33' 2023-10-16T20:45:57.069 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/23' 2023-10-16T20:45:57.069 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/35' 2023-10-16T20:45:57.070 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/26' 2023-10-16T20:45:57.070 INFO:tasks.workunit.client.0.smithi088.stdout:removed './268/37' 2023-10-16T20:45:57.070 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './268' 2023-10-16T20:45:57.070 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/21' 2023-10-16T20:45:57.071 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/27' 2023-10-16T20:45:57.071 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/31' 2023-10-16T20:45:57.071 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/36' 2023-10-16T20:45:57.071 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/20' 2023-10-16T20:45:57.071 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/39' 2023-10-16T20:45:57.072 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/34' 2023-10-16T20:45:57.072 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/38' 2023-10-16T20:45:57.072 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/28' 2023-10-16T20:45:57.072 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/24' 2023-10-16T20:45:57.072 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/29' 2023-10-16T20:45:57.073 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/32' 2023-10-16T20:45:57.073 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/22' 2023-10-16T20:45:57.073 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/25' 2023-10-16T20:45:57.073 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/30' 2023-10-16T20:45:57.073 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/33' 2023-10-16T20:45:57.074 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/23' 2023-10-16T20:45:57.074 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/35' 2023-10-16T20:45:57.074 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/26' 2023-10-16T20:45:57.074 INFO:tasks.workunit.client.0.smithi088.stdout:removed './269/37' 2023-10-16T20:45:57.074 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './269' 2023-10-16T20:45:57.075 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/21' 2023-10-16T20:45:57.075 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/27' 2023-10-16T20:45:57.075 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/31' 2023-10-16T20:45:57.075 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/36' 2023-10-16T20:45:57.075 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/20' 2023-10-16T20:45:57.076 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/39' 2023-10-16T20:45:57.076 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/34' 2023-10-16T20:45:57.076 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/38' 2023-10-16T20:45:57.076 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/28' 2023-10-16T20:45:57.076 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/24' 2023-10-16T20:45:57.077 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/29' 2023-10-16T20:45:57.077 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/32' 2023-10-16T20:45:57.077 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/22' 2023-10-16T20:45:57.077 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/25' 2023-10-16T20:45:57.077 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/30' 2023-10-16T20:45:57.078 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/33' 2023-10-16T20:45:57.078 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/23' 2023-10-16T20:45:57.078 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/35' 2023-10-16T20:45:57.078 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/26' 2023-10-16T20:45:57.079 INFO:tasks.workunit.client.0.smithi088.stdout:removed './270/37' 2023-10-16T20:45:57.079 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './270' 2023-10-16T20:45:57.079 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/21' 2023-10-16T20:45:57.079 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/27' 2023-10-16T20:45:57.079 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/31' 2023-10-16T20:45:57.080 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/36' 2023-10-16T20:45:57.080 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/20' 2023-10-16T20:45:57.080 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/39' 2023-10-16T20:45:57.080 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/34' 2023-10-16T20:45:57.080 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/38' 2023-10-16T20:45:57.081 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/28' 2023-10-16T20:45:57.081 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/24' 2023-10-16T20:45:57.081 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/29' 2023-10-16T20:45:57.081 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/32' 2023-10-16T20:45:57.082 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/22' 2023-10-16T20:45:57.082 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/25' 2023-10-16T20:45:57.082 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/30' 2023-10-16T20:45:57.082 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/33' 2023-10-16T20:45:57.083 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/23' 2023-10-16T20:45:57.083 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/35' 2023-10-16T20:45:57.083 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/26' 2023-10-16T20:45:57.083 INFO:tasks.workunit.client.0.smithi088.stdout:removed './271/37' 2023-10-16T20:45:57.083 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './271' 2023-10-16T20:45:57.084 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/21' 2023-10-16T20:45:57.084 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/27' 2023-10-16T20:45:57.084 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/31' 2023-10-16T20:45:57.084 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/36' 2023-10-16T20:45:57.084 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/20' 2023-10-16T20:45:57.085 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/39' 2023-10-16T20:45:57.085 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/34' 2023-10-16T20:45:57.085 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/38' 2023-10-16T20:45:57.085 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/28' 2023-10-16T20:45:57.086 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/24' 2023-10-16T20:45:57.086 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/29' 2023-10-16T20:45:57.086 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/32' 2023-10-16T20:45:57.086 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/22' 2023-10-16T20:45:57.086 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/25' 2023-10-16T20:45:57.087 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/30' 2023-10-16T20:45:57.087 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/33' 2023-10-16T20:45:57.087 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/23' 2023-10-16T20:45:57.087 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/35' 2023-10-16T20:45:57.088 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/26' 2023-10-16T20:45:57.088 INFO:tasks.workunit.client.0.smithi088.stdout:removed './272/37' 2023-10-16T20:45:57.088 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './272' 2023-10-16T20:45:57.088 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/21' 2023-10-16T20:45:57.088 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/27' 2023-10-16T20:45:57.089 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/31' 2023-10-16T20:45:57.089 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/36' 2023-10-16T20:45:57.089 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/20' 2023-10-16T20:45:57.089 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/39' 2023-10-16T20:45:57.089 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/34' 2023-10-16T20:45:57.090 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/38' 2023-10-16T20:45:57.090 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/28' 2023-10-16T20:45:57.090 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/24' 2023-10-16T20:45:57.090 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/29' 2023-10-16T20:45:57.091 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/32' 2023-10-16T20:45:57.091 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/22' 2023-10-16T20:45:57.091 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/25' 2023-10-16T20:45:57.091 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/30' 2023-10-16T20:45:57.091 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/33' 2023-10-16T20:45:57.092 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/23' 2023-10-16T20:45:57.092 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/35' 2023-10-16T20:45:57.092 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/26' 2023-10-16T20:45:57.092 INFO:tasks.workunit.client.0.smithi088.stdout:removed './273/37' 2023-10-16T20:45:57.092 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './273' 2023-10-16T20:45:57.093 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/21' 2023-10-16T20:45:57.093 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/27' 2023-10-16T20:45:57.093 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/31' 2023-10-16T20:45:57.093 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/36' 2023-10-16T20:45:57.093 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/20' 2023-10-16T20:45:57.094 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/39' 2023-10-16T20:45:57.094 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/34' 2023-10-16T20:45:57.094 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/38' 2023-10-16T20:45:57.094 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/28' 2023-10-16T20:45:57.095 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/24' 2023-10-16T20:45:57.095 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/29' 2023-10-16T20:45:57.095 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/32' 2023-10-16T20:45:57.095 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/22' 2023-10-16T20:45:57.095 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/25' 2023-10-16T20:45:57.096 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/30' 2023-10-16T20:45:57.096 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/33' 2023-10-16T20:45:57.096 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/23' 2023-10-16T20:45:57.096 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/35' 2023-10-16T20:45:57.096 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/26' 2023-10-16T20:45:57.097 INFO:tasks.workunit.client.0.smithi088.stdout:removed './274/37' 2023-10-16T20:45:57.097 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './274' 2023-10-16T20:45:57.097 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/21' 2023-10-16T20:45:57.097 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/27' 2023-10-16T20:45:57.097 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/31' 2023-10-16T20:45:57.098 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/36' 2023-10-16T20:45:57.098 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/20' 2023-10-16T20:45:57.098 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/39' 2023-10-16T20:45:57.098 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/34' 2023-10-16T20:45:57.099 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/38' 2023-10-16T20:45:57.099 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/28' 2023-10-16T20:45:57.099 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/24' 2023-10-16T20:45:57.101 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/29' 2023-10-16T20:45:57.101 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/32' 2023-10-16T20:45:57.101 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/22' 2023-10-16T20:45:57.102 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/25' 2023-10-16T20:45:57.102 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/30' 2023-10-16T20:45:57.102 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/33' 2023-10-16T20:45:57.102 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/23' 2023-10-16T20:45:57.102 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/35' 2023-10-16T20:45:57.103 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/26' 2023-10-16T20:45:57.103 INFO:tasks.workunit.client.0.smithi088.stdout:removed './275/37' 2023-10-16T20:45:57.103 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './275' 2023-10-16T20:45:57.103 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/21' 2023-10-16T20:45:57.104 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/27' 2023-10-16T20:45:57.104 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/31' 2023-10-16T20:45:57.104 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/36' 2023-10-16T20:45:57.104 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/20' 2023-10-16T20:45:57.104 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/39' 2023-10-16T20:45:57.105 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/34' 2023-10-16T20:45:57.105 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/38' 2023-10-16T20:45:57.105 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/28' 2023-10-16T20:45:57.105 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/24' 2023-10-16T20:45:57.105 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/29' 2023-10-16T20:45:57.106 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/32' 2023-10-16T20:45:57.106 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/22' 2023-10-16T20:45:57.106 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/25' 2023-10-16T20:45:57.106 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/30' 2023-10-16T20:45:57.106 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/33' 2023-10-16T20:45:57.107 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/23' 2023-10-16T20:45:57.107 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/35' 2023-10-16T20:45:57.107 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/26' 2023-10-16T20:45:58.050 INFO:tasks.workunit.client.0.smithi088.stdout:removed './276/37' 2023-10-16T20:45:58.051 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './276' 2023-10-16T20:45:58.051 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/21' 2023-10-16T20:45:58.051 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/27' 2023-10-16T20:45:58.051 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/31' 2023-10-16T20:45:58.052 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/36' 2023-10-16T20:45:58.052 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/20' 2023-10-16T20:45:58.052 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/39' 2023-10-16T20:45:58.052 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/34' 2023-10-16T20:45:58.052 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/38' 2023-10-16T20:45:58.052 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/28' 2023-10-16T20:45:58.053 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/24' 2023-10-16T20:45:58.053 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/29' 2023-10-16T20:45:58.053 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/32' 2023-10-16T20:45:58.053 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/22' 2023-10-16T20:45:58.053 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/25' 2023-10-16T20:45:58.053 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/30' 2023-10-16T20:45:58.054 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/33' 2023-10-16T20:45:58.054 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/23' 2023-10-16T20:45:58.054 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/35' 2023-10-16T20:45:58.054 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/26' 2023-10-16T20:45:58.054 INFO:tasks.workunit.client.0.smithi088.stdout:removed './277/37' 2023-10-16T20:45:58.055 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './277' 2023-10-16T20:45:58.055 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/21' 2023-10-16T20:45:58.055 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/27' 2023-10-16T20:45:58.055 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/31' 2023-10-16T20:45:58.055 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/36' 2023-10-16T20:45:58.056 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/20' 2023-10-16T20:45:58.056 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/39' 2023-10-16T20:45:58.056 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/34' 2023-10-16T20:45:58.056 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/38' 2023-10-16T20:45:58.056 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/28' 2023-10-16T20:45:58.057 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/24' 2023-10-16T20:45:58.057 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/29' 2023-10-16T20:45:58.057 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/32' 2023-10-16T20:45:58.057 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/22' 2023-10-16T20:45:58.057 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/25' 2023-10-16T20:45:58.057 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/30' 2023-10-16T20:45:58.058 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/33' 2023-10-16T20:45:58.058 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/23' 2023-10-16T20:45:58.058 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/35' 2023-10-16T20:45:58.058 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/26' 2023-10-16T20:45:58.058 INFO:tasks.workunit.client.0.smithi088.stdout:removed './278/37' 2023-10-16T20:45:58.059 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './278' 2023-10-16T20:45:58.059 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/21' 2023-10-16T20:45:58.059 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/27' 2023-10-16T20:45:58.059 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/31' 2023-10-16T20:45:58.059 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/36' 2023-10-16T20:45:58.060 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/20' 2023-10-16T20:45:58.060 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/39' 2023-10-16T20:45:58.060 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/34' 2023-10-16T20:45:58.060 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/38' 2023-10-16T20:45:58.060 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/28' 2023-10-16T20:45:58.060 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/24' 2023-10-16T20:45:58.061 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/29' 2023-10-16T20:45:58.061 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/32' 2023-10-16T20:45:58.061 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/22' 2023-10-16T20:45:58.061 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/25' 2023-10-16T20:45:58.061 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/30' 2023-10-16T20:45:58.061 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/33' 2023-10-16T20:45:58.062 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/23' 2023-10-16T20:45:58.062 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/35' 2023-10-16T20:45:58.062 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/26' 2023-10-16T20:45:58.062 INFO:tasks.workunit.client.0.smithi088.stdout:removed './279/37' 2023-10-16T20:45:58.062 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './279' 2023-10-16T20:45:58.063 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/21' 2023-10-16T20:45:58.063 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/27' 2023-10-16T20:45:58.063 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/31' 2023-10-16T20:45:58.063 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/36' 2023-10-16T20:45:58.063 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/20' 2023-10-16T20:45:58.063 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/39' 2023-10-16T20:45:58.064 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/34' 2023-10-16T20:45:58.064 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/38' 2023-10-16T20:45:58.064 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/28' 2023-10-16T20:45:58.064 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/24' 2023-10-16T20:45:58.064 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/29' 2023-10-16T20:45:58.064 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/32' 2023-10-16T20:45:58.065 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/22' 2023-10-16T20:45:58.065 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/25' 2023-10-16T20:45:58.065 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/30' 2023-10-16T20:45:58.065 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/33' 2023-10-16T20:45:58.065 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/23' 2023-10-16T20:45:58.066 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/35' 2023-10-16T20:45:58.066 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/26' 2023-10-16T20:45:58.066 INFO:tasks.workunit.client.0.smithi088.stdout:removed './280/37' 2023-10-16T20:45:58.066 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './280' 2023-10-16T20:45:58.066 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/21' 2023-10-16T20:45:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/27' 2023-10-16T20:45:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/31' 2023-10-16T20:45:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/36' 2023-10-16T20:45:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/20' 2023-10-16T20:45:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/39' 2023-10-16T20:45:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/34' 2023-10-16T20:45:58.068 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/38' 2023-10-16T20:45:58.068 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/28' 2023-10-16T20:45:58.068 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/24' 2023-10-16T20:45:58.068 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/29' 2023-10-16T20:45:58.068 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/32' 2023-10-16T20:45:58.068 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/22' 2023-10-16T20:45:58.069 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/25' 2023-10-16T20:45:58.069 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/30' 2023-10-16T20:45:58.069 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/33' 2023-10-16T20:45:58.069 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/23' 2023-10-16T20:45:58.069 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/35' 2023-10-16T20:45:58.070 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/26' 2023-10-16T20:45:58.070 INFO:tasks.workunit.client.0.smithi088.stdout:removed './281/37' 2023-10-16T20:45:58.070 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './281' 2023-10-16T20:45:58.070 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/21' 2023-10-16T20:45:58.070 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/27' 2023-10-16T20:45:58.071 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/31' 2023-10-16T20:45:58.071 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/36' 2023-10-16T20:45:58.071 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/20' 2023-10-16T20:45:58.071 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/39' 2023-10-16T20:45:58.071 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/34' 2023-10-16T20:45:58.072 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/38' 2023-10-16T20:45:58.072 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/28' 2023-10-16T20:45:58.072 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/24' 2023-10-16T20:45:58.072 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/29' 2023-10-16T20:45:58.072 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/32' 2023-10-16T20:45:58.073 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/22' 2023-10-16T20:45:58.073 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/25' 2023-10-16T20:45:58.073 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/30' 2023-10-16T20:45:58.073 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/33' 2023-10-16T20:45:58.073 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/23' 2023-10-16T20:45:58.074 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/35' 2023-10-16T20:45:58.074 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/26' 2023-10-16T20:45:58.074 INFO:tasks.workunit.client.0.smithi088.stdout:removed './282/37' 2023-10-16T20:45:58.074 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './282' 2023-10-16T20:45:58.074 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/21' 2023-10-16T20:45:58.075 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/27' 2023-10-16T20:45:58.075 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/31' 2023-10-16T20:45:58.075 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/36' 2023-10-16T20:45:58.075 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/20' 2023-10-16T20:45:58.075 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/39' 2023-10-16T20:45:58.076 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/34' 2023-10-16T20:45:58.076 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/38' 2023-10-16T20:45:58.076 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/28' 2023-10-16T20:45:58.076 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/24' 2023-10-16T20:45:58.076 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/29' 2023-10-16T20:45:58.077 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/32' 2023-10-16T20:45:58.077 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/22' 2023-10-16T20:45:58.077 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/25' 2023-10-16T20:45:58.077 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/30' 2023-10-16T20:45:58.077 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/33' 2023-10-16T20:45:58.078 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/23' 2023-10-16T20:45:58.078 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/35' 2023-10-16T20:45:58.078 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/26' 2023-10-16T20:45:58.078 INFO:tasks.workunit.client.0.smithi088.stdout:removed './283/37' 2023-10-16T20:45:58.078 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './283' 2023-10-16T20:45:58.079 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/21' 2023-10-16T20:45:58.079 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/27' 2023-10-16T20:45:58.079 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/31' 2023-10-16T20:45:58.079 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/36' 2023-10-16T20:45:58.079 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/20' 2023-10-16T20:45:58.080 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/39' 2023-10-16T20:45:58.080 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/34' 2023-10-16T20:45:58.080 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/38' 2023-10-16T20:45:58.080 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/28' 2023-10-16T20:45:58.080 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/24' 2023-10-16T20:45:58.081 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/29' 2023-10-16T20:45:58.081 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/32' 2023-10-16T20:45:58.081 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/22' 2023-10-16T20:45:58.081 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/25' 2023-10-16T20:45:58.081 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/30' 2023-10-16T20:45:58.082 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/33' 2023-10-16T20:45:58.082 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/23' 2023-10-16T20:45:58.082 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/35' 2023-10-16T20:45:58.082 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/26' 2023-10-16T20:45:58.082 INFO:tasks.workunit.client.0.smithi088.stdout:removed './284/37' 2023-10-16T20:45:58.083 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './284' 2023-10-16T20:45:58.083 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/21' 2023-10-16T20:45:58.083 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/27' 2023-10-16T20:45:58.083 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/31' 2023-10-16T20:45:58.083 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/36' 2023-10-16T20:45:58.083 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/20' 2023-10-16T20:45:58.084 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/39' 2023-10-16T20:45:58.084 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/34' 2023-10-16T20:45:58.084 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/38' 2023-10-16T20:45:58.084 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/28' 2023-10-16T20:45:58.084 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/24' 2023-10-16T20:45:58.084 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/29' 2023-10-16T20:45:58.085 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/32' 2023-10-16T20:45:58.085 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/22' 2023-10-16T20:45:58.085 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/25' 2023-10-16T20:45:58.085 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/30' 2023-10-16T20:45:58.085 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/33' 2023-10-16T20:45:58.085 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/23' 2023-10-16T20:45:58.086 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/35' 2023-10-16T20:45:58.086 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/26' 2023-10-16T20:45:58.086 INFO:tasks.workunit.client.0.smithi088.stdout:removed './285/37' 2023-10-16T20:45:58.086 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './285' 2023-10-16T20:45:58.087 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/21' 2023-10-16T20:45:58.087 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/27' 2023-10-16T20:45:58.087 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/31' 2023-10-16T20:45:58.087 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/36' 2023-10-16T20:45:58.087 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/20' 2023-10-16T20:45:58.087 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/39' 2023-10-16T20:45:58.088 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/34' 2023-10-16T20:45:58.088 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/38' 2023-10-16T20:45:58.088 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/28' 2023-10-16T20:45:58.088 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/24' 2023-10-16T20:45:58.088 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/29' 2023-10-16T20:45:58.089 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/32' 2023-10-16T20:45:58.089 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/22' 2023-10-16T20:45:58.089 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/25' 2023-10-16T20:45:58.089 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/30' 2023-10-16T20:45:58.089 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/33' 2023-10-16T20:45:58.090 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/23' 2023-10-16T20:45:58.090 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/35' 2023-10-16T20:45:58.090 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/26' 2023-10-16T20:45:58.090 INFO:tasks.workunit.client.0.smithi088.stdout:removed './286/37' 2023-10-16T20:45:58.090 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './286' 2023-10-16T20:45:59.082 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:45:59.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.219+0000 7fa6fa009640 1 Processor -- start 2023-10-16T20:45:59.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.219+0000 7fa6fa009640 1 -- start start 2023-10-16T20:45:59.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.223+0000 7fa6fa009640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6f4069e90 0x7fa6f406a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:45:59.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.223+0000 7fa6fa009640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa6f406f710 0x7fa6f406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:45:59.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.223+0000 7fa6fa009640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f4156430 con 0x7fa6f414fd10 2023-10-16T20:45:59.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.223+0000 7fa6fa009640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f4070500 con 0x7fa6f406f710 2023-10-16T20:45:59.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.223+0000 7fa6fa009640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f4155b00 con 0x7fa6f4069e90 2023-10-16T20:45:59.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.223+0000 7fa6f3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa6f406f710 0x7fa6f406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:45:59.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.223+0000 7fa6f3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa6f406f710 0x7fa6f406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46128/0 (socket says 172.21.15.88:46128) 2023-10-16T20:45:59.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.223+0000 7fa6f3fff640 1 -- 172.21.15.88:0/1707255140 learned_addr learned my addr 172.21.15.88:0/1707255140 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:45:59.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.223+0000 7fa6ebfff640 1 --2- 172.21.15.88:0/1707255140 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6f4069e90 0x7fa6f406a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:45:59.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.223+0000 7fa6f3fff640 1 -- 172.21.15.88:0/1707255140 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6f4069e90 msgr2=0x7fa6f406a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:45:59.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.223+0000 7fa6f3fff640 1 --2- 172.21.15.88:0/1707255140 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6f4069e90 0x7fa6f406a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.223+0000 7fa6f3fff640 1 -- 172.21.15.88:0/1707255140 >> v1:172.21.15.191:6789/0 conn(0x7fa6f414fd10 legacy=0x7fa6f4070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:45:59.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.223+0000 7fa6f3fff640 1 -- 172.21.15.88:0/1707255140 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f4155de0 con 0x7fa6f406f710 2023-10-16T20:45:59.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.223+0000 7fa6f3fff640 1 --2- 172.21.15.88:0/1707255140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6f406f710 0x7fa6f406fb80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec009e50 tx=0x7fa6ec01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86661cff1a5d341d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:45:59.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.223+0000 7fa6f2ffd640 1 -- 172.21.15.88:0/1707255140 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec0af400 con 0x7fa6f406f710 2023-10-16T20:45:59.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6f2ffd640 1 -- 172.21.15.88:0/1707255140 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ec0ab050 con 0x7fa6f406f710 2023-10-16T20:45:59.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6f2ffd640 1 -- 172.21.15.88:0/1707255140 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec0af780 con 0x7fa6f406f710 2023-10-16T20:45:59.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6fa009640 1 -- 172.21.15.88:0/1707255140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6f406f710 msgr2=0x7fa6f406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:45:59.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6fa009640 1 --2- 172.21.15.88:0/1707255140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6f406f710 0x7fa6f406fb80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec009e50 tx=0x7fa6ec01f1f0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6fa009640 1 -- 172.21.15.88:0/1707255140 shutdown_connections 2023-10-16T20:45:59.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6fa009640 1 --2- 172.21.15.88:0/1707255140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6f406f710 0x7fa6f406fb80 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6fa009640 1 --2- 172.21.15.88:0/1707255140 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6f4069e90 0x7fa6f406a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6fa009640 1 -- 172.21.15.88:0/1707255140 >> 172.21.15.88:0/1707255140 conn(0x7fa6f40cad50 msgr2=0x7fa6f40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:45:59.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6fa009640 1 -- 172.21.15.88:0/1707255140 shutdown_connections 2023-10-16T20:45:59.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6fa009640 1 -- 172.21.15.88:0/1707255140 wait complete. 2023-10-16T20:45:59.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6fa009640 1 Processor -- start 2023-10-16T20:45:59.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6fa009640 1 -- start start 2023-10-16T20:45:59.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6fa009640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6f4069e90 0x7fa6f41f0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:45:59.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6fa009640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6f406f710 0x7fa6f41ec9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:45:59.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6fa009640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6f414fd10 0x7fa6f40c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:45:59.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6fa009640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f41f06a0 con 0x7fa6f4069e90 2023-10-16T20:45:59.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6fa009640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f41ecf10 con 0x7fa6f414fd10 2023-10-16T20:45:59.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6fa009640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f414ad90 con 0x7fa6f406f710 2023-10-16T20:45:59.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6f37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6f4069e90 0x7fa6f41f0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:45:59.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6f37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6f4069e90 0x7fa6f41f0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46144/0 (socket says 172.21.15.88:46144) 2023-10-16T20:45:59.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6f37fe640 1 -- 172.21.15.88:0/3329355675 learned_addr learned my addr 172.21.15.88:0/3329355675 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:45:59.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6f3fff640 1 --2- 172.21.15.88:0/3329355675 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6f414fd10 0x7fa6f40c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:45:59.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6ebfff640 1 --2- 172.21.15.88:0/3329355675 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6f406f710 0x7fa6f41ec9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:45:59.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6f37fe640 1 -- 172.21.15.88:0/3329355675 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6f406f710 msgr2=0x7fa6f41ec9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:45:59.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6f37fe640 1 --2- 172.21.15.88:0/3329355675 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6f406f710 0x7fa6f41ec9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6f37fe640 1 -- 172.21.15.88:0/3329355675 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6f414fd10 msgr2=0x7fa6f40c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:45:59.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6f37fe640 1 --2- 172.21.15.88:0/3329355675 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6f414fd10 0x7fa6f40c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.227+0000 7fa6f37fe640 1 -- 172.21.15.88:0/3329355675 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ec009b30 con 0x7fa6f4069e90 2023-10-16T20:45:59.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.231+0000 7fa6f37fe640 1 --2- 172.21.15.88:0/3329355675 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6f4069e90 0x7fa6f41f0ac0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa6e400c3c0 tx=0x7fa6e400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:45:59.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.231+0000 7fa6f0ff9640 1 -- 172.21.15.88:0/3329355675 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e40052d0 con 0x7fa6f4069e90 2023-10-16T20:45:59.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.231+0000 7fa6f0ff9640 1 -- 172.21.15.88:0/3329355675 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e4005430 con 0x7fa6f4069e90 2023-10-16T20:45:59.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.231+0000 7fa6fa009640 1 -- 172.21.15.88:0/3329355675 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f414b070 con 0x7fa6f4069e90 2023-10-16T20:45:59.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.231+0000 7fa6f0ff9640 1 -- 172.21.15.88:0/3329355675 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e4003840 con 0x7fa6f4069e90 2023-10-16T20:45:59.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.231+0000 7fa6fa009640 1 -- 172.21.15.88:0/3329355675 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f41effe0 con 0x7fa6f4069e90 2023-10-16T20:45:59.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.231+0000 7fa6fa009640 1 -- 172.21.15.88:0/3329355675 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f41f07e0 con 0x7fa6f4069e90 2023-10-16T20:45:59.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.231+0000 7fa6f0ff9640 1 -- 172.21.15.88:0/3329355675 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa6e40039a0 con 0x7fa6f4069e90 2023-10-16T20:45:59.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.231+0000 7fa6f0ff9640 1 --2- 172.21.15.88:0/3329355675 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6d40664a0 0x7fa6d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:45:59.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.231+0000 7fa6f0ff9640 1 -- 172.21.15.88:0/3329355675 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7333+0+0 (secure 0 0 0) 0x7fa6e40839f0 con 0x7fa6f4069e90 2023-10-16T20:45:59.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.231+0000 7fa6ebfff640 1 --2- 172.21.15.88:0/3329355675 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6d40664a0 0x7fa6d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:45:59.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.231+0000 7fa6ebfff640 1 --2- 172.21.15.88:0/3329355675 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6d40664a0 0x7fa6d4068960 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa6f41ea600 tx=0x7fa6dc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:45:59.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.235+0000 7fa6f0ff9640 1 -- 172.21.15.88:0/3329355675 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa6f41f07e0 con 0x7fa6f4069e90 2023-10-16T20:45:59.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.419+0000 7fa6fa009640 1 -- 172.21.15.88:0/3329355675 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6f41f07e0 con 0x7fa6f4069e90 2023-10-16T20:45:59.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.419+0000 7fa6f0ff9640 1 -- 172.21.15.88:0/3329355675 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+6581 (secure 0 0 0) 0x7fa6f41f07e0 con 0x7fa6f4069e90 2023-10-16T20:45:59.424 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:45:59.424 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:45:46.511774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:45:59.425 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 51 2023-10-16T20:45:59.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.423+0000 7fa6fa009640 1 -- 172.21.15.88:0/3329355675 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6d40664a0 msgr2=0x7fa6d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:45:59.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.423+0000 7fa6fa009640 1 --2- 172.21.15.88:0/3329355675 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6d40664a0 0x7fa6d4068960 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa6f41ea600 tx=0x7fa6dc008040 comp rx=0 tx=0).stop 2023-10-16T20:45:59.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.423+0000 7fa6fa009640 1 -- 172.21.15.88:0/3329355675 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6f4069e90 msgr2=0x7fa6f41f0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:45:59.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.423+0000 7fa6fa009640 1 --2- 172.21.15.88:0/3329355675 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6f4069e90 0x7fa6f41f0ac0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa6e400c3c0 tx=0x7fa6e400c880 comp rx=0 tx=0).stop 2023-10-16T20:45:59.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.423+0000 7fa6fa009640 1 -- 172.21.15.88:0/3329355675 shutdown_connections 2023-10-16T20:45:59.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.423+0000 7fa6fa009640 1 --2- 172.21.15.88:0/3329355675 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa6d40664a0 0x7fa6d4068960 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.423+0000 7fa6fa009640 1 --2- 172.21.15.88:0/3329355675 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa6f414fd10 0x7fa6f40c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.423+0000 7fa6fa009640 1 --2- 172.21.15.88:0/3329355675 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa6f406f710 0x7fa6f41ec9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.423+0000 7fa6fa009640 1 --2- 172.21.15.88:0/3329355675 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6f4069e90 0x7fa6f41f0ac0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.423+0000 7fa6fa009640 1 -- 172.21.15.88:0/3329355675 >> 172.21.15.88:0/3329355675 conn(0x7fa6f40cad50 msgr2=0x7fa6f40c7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:45:59.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.423+0000 7fa6fa009640 1 -- 172.21.15.88:0/3329355675 shutdown_connections 2023-10-16T20:45:59.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.423+0000 7fa6fa009640 1 -- 172.21.15.88:0/3329355675 wait complete. 2023-10-16T20:45:59.445 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=2) 2023-10-16T20:45:59.446 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-16T20:45:59.446 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T20:45:59.460 INFO:tasks.ceph.mds.e:Stopped 2023-10-16T20:45:59.460 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:45:59.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.643+0000 7f5339980640 1 Processor -- start 2023-10-16T20:45:59.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.643+0000 7f5339980640 1 -- start start 2023-10-16T20:45:59.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.647+0000 7f5339980640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5334153770 0x7f5334159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:45:59.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.647+0000 7f5339980640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f53340c9bb0 0x7f53340c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:45:59.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.647+0000 7f5339980640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f53340c41f0 0x7f53340cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:45:59.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.647+0000 7f5339980640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53340cc250 con 0x7f53340c41f0 2023-10-16T20:45:59.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.647+0000 7f5339980640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53340cc670 con 0x7f53340c9bb0 2023-10-16T20:45:59.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.647+0000 7f5339980640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533415a410 con 0x7f5334153770 2023-10-16T20:45:59.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.647+0000 7f53337fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f53340c41f0 0x7f53340cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:45:59.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.647+0000 7f53337fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f53340c41f0 0x7f53340cece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46150/0 (socket says 172.21.15.88:46150) 2023-10-16T20:45:59.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.647+0000 7f532bfff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f53340c9bb0 0x7f53340c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:45:59.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.647+0000 7f53337fe640 1 -- 172.21.15.88:0/370418548 learned_addr learned my addr 172.21.15.88:0/370418548 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:45:59.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.647+0000 7f5332ffd640 1 --2- 172.21.15.88:0/370418548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5334153770 0x7f5334159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:45:59.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.647+0000 7f53337fe640 1 -- 172.21.15.88:0/370418548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5334153770 msgr2=0x7f5334159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:45:59.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.647+0000 7f53337fe640 1 --2- 172.21.15.88:0/370418548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5334153770 0x7f5334159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.647+0000 7f53337fe640 1 -- 172.21.15.88:0/370418548 >> v2:172.21.15.191:3300/0 conn(0x7f53340c9bb0 msgr2=0x7f53340c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:45:59.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.647+0000 7f53337fe640 1 --2- 172.21.15.88:0/370418548 >> v2:172.21.15.191:3300/0 conn(0x7f53340c9bb0 0x7f53340c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.647+0000 7f53337fe640 1 -- 172.21.15.88:0/370418548 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5334153b50 con 0x7f53340c41f0 2023-10-16T20:45:59.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f53337fe640 1 --2- 172.21.15.88:0/370418548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53340c41f0 0x7f53340cece0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f53340cc810 tx=0x7f532c01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=173aac773d375a86 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:45:59.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f53327fc640 1 -- 172.21.15.88:0/370418548 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c0a5760 con 0x7f53340c41f0 2023-10-16T20:45:59.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f53327fc640 1 -- 172.21.15.88:0/370418548 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532c0a58c0 con 0x7f53340c41f0 2023-10-16T20:45:59.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f53327fc640 1 -- 172.21.15.88:0/370418548 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c0a5c50 con 0x7f53340c41f0 2023-10-16T20:45:59.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f5339980640 1 -- 172.21.15.88:0/370418548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53340c41f0 msgr2=0x7f53340cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:45:59.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f5339980640 1 --2- 172.21.15.88:0/370418548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53340c41f0 0x7f53340cece0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f53340cc810 tx=0x7f532c01c9d0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f5339980640 1 -- 172.21.15.88:0/370418548 shutdown_connections 2023-10-16T20:45:59.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f5339980640 1 --2- 172.21.15.88:0/370418548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53340c41f0 0x7f53340cece0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f5339980640 1 --2- 172.21.15.88:0/370418548 >> v2:172.21.15.191:3300/0 conn(0x7f53340c9bb0 0x7f53340c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f5339980640 1 --2- 172.21.15.88:0/370418548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5334153770 0x7f5334159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f5339980640 1 -- 172.21.15.88:0/370418548 >> 172.21.15.88:0/370418548 conn(0x7f53340715c0 msgr2=0x7f53340719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:45:59.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f5339980640 1 -- 172.21.15.88:0/370418548 shutdown_connections 2023-10-16T20:45:59.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f5339980640 1 -- 172.21.15.88:0/370418548 wait complete. 2023-10-16T20:45:59.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f5339980640 1 Processor -- start 2023-10-16T20:45:59.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f5339980640 1 -- start start 2023-10-16T20:45:59.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f5339980640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53340c41f0 0x7f53341511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:45:59.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f5339980640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f53340c9bb0 0x7f53340b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:45:59.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f5339980640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5334153770 0x7f53340c6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:45:59.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f5339980640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53341503a0 con 0x7f53340c41f0 2023-10-16T20:45:59.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f5339980640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5334150680 con 0x7f5334153770 2023-10-16T20:45:59.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.651+0000 7f5339980640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5334150960 con 0x7f53340c9bb0 2023-10-16T20:45:59.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f5332ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53340c41f0 0x7f53341511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:45:59.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f5332ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53340c41f0 0x7f53341511c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46154/0 (socket says 172.21.15.88:46154) 2023-10-16T20:45:59.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f5332ffd640 1 -- 172.21.15.88:0/3952123288 learned_addr learned my addr 172.21.15.88:0/3952123288 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:45:59.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f532bfff640 1 --2- 172.21.15.88:0/3952123288 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f53340c9bb0 0x7f53340b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:45:59.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f53337fe640 1 --2- 172.21.15.88:0/3952123288 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5334153770 0x7f53340c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:45:59.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f5332ffd640 1 -- 172.21.15.88:0/3952123288 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f53340c9bb0 msgr2=0x7f53340b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:45:59.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f5332ffd640 1 --2- 172.21.15.88:0/3952123288 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f53340c9bb0 0x7f53340b3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f5332ffd640 1 -- 172.21.15.88:0/3952123288 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5334153770 msgr2=0x7f53340c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:45:59.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f5332ffd640 1 --2- 172.21.15.88:0/3952123288 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5334153770 0x7f53340c6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f5332ffd640 1 -- 172.21.15.88:0/3952123288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532c009b30 con 0x7f53340c41f0 2023-10-16T20:45:59.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f53337fe640 1 --2- 172.21.15.88:0/3952123288 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5334153770 0x7f53340c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:45:59.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f5332ffd640 1 --2- 172.21.15.88:0/3952123288 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53340c41f0 0x7f53341511c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f532400e9b0 tx=0x7f532400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:45:59.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f5330ff9640 1 -- 172.21.15.88:0/3952123288 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5324009670 con 0x7f53340c41f0 2023-10-16T20:45:59.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f5330ff9640 1 -- 172.21.15.88:0/3952123288 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532400c9a0 con 0x7f53340c41f0 2023-10-16T20:45:59.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f5339980640 1 -- 172.21.15.88:0/3952123288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f533414f480 con 0x7f53340c41f0 2023-10-16T20:45:59.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f5330ff9640 1 -- 172.21.15.88:0/3952123288 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532400cd10 con 0x7f53340c41f0 2023-10-16T20:45:59.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f5339980640 1 -- 172.21.15.88:0/3952123288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f533414fa40 con 0x7f53340c41f0 2023-10-16T20:45:59.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f5339980640 1 -- 172.21.15.88:0/3952123288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f533414fc00 con 0x7f53340c41f0 2023-10-16T20:45:59.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.655+0000 7f5330ff9640 1 -- 172.21.15.88:0/3952123288 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5324013470 con 0x7f53340c41f0 2023-10-16T20:45:59.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.659+0000 7f5330ff9640 1 --2- 172.21.15.88:0/3952123288 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f53140664a0 0x7f5314068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:45:59.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.659+0000 7f5330ff9640 1 -- 172.21.15.88:0/3952123288 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7333+0+0 (secure 0 0 0) 0x7f5324084aa0 con 0x7f53340c41f0 2023-10-16T20:45:59.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.659+0000 7f532bfff640 1 --2- 172.21.15.88:0/3952123288 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f53140664a0 0x7f5314068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:45:59.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.659+0000 7f532bfff640 1 --2- 172.21.15.88:0/3952123288 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f53140664a0 0x7f5314068960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f53340caad0 tx=0x7f531c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:45:59.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.663+0000 7f5330ff9640 1 -- 172.21.15.88:0/3952123288 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f53240518d0 con 0x7f53340c41f0 2023-10-16T20:45:59.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.843+0000 7f5339980640 1 -- 172.21.15.88:0/3952123288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5334074700 con 0x7f53340c41f0 2023-10-16T20:45:59.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.843+0000 7f5330ff9640 1 -- 172.21.15.88:0/3952123288 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+6581 (secure 0 0 0) 0x7f5324017090 con 0x7f53340c41f0 2023-10-16T20:45:59.847 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:45:59.847 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:45:46.511774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:45:59.847 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 51 2023-10-16T20:45:59.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.843+0000 7f5339980640 1 -- 172.21.15.88:0/3952123288 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f53140664a0 msgr2=0x7f5314068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:45:59.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.843+0000 7f5339980640 1 --2- 172.21.15.88:0/3952123288 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f53140664a0 0x7f5314068960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f53340caad0 tx=0x7f531c009190 comp rx=0 tx=0).stop 2023-10-16T20:45:59.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.843+0000 7f5339980640 1 -- 172.21.15.88:0/3952123288 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53340c41f0 msgr2=0x7f53341511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:45:59.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.843+0000 7f5339980640 1 --2- 172.21.15.88:0/3952123288 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53340c41f0 0x7f53341511c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f532400e9b0 tx=0x7f532400ee70 comp rx=0 tx=0).stop 2023-10-16T20:45:59.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.843+0000 7f5339980640 1 -- 172.21.15.88:0/3952123288 shutdown_connections 2023-10-16T20:45:59.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.843+0000 7f5339980640 1 --2- 172.21.15.88:0/3952123288 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f53140664a0 0x7f5314068960 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.843+0000 7f5339980640 1 --2- 172.21.15.88:0/3952123288 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5334153770 0x7f53340c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.843+0000 7f5339980640 1 --2- 172.21.15.88:0/3952123288 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f53340c9bb0 0x7f53340b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.843+0000 7f5339980640 1 --2- 172.21.15.88:0/3952123288 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53340c41f0 0x7f53341511c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:45:59.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.843+0000 7f5339980640 1 -- 172.21.15.88:0/3952123288 >> 172.21.15.88:0/3952123288 conn(0x7f53340715c0 msgr2=0x7f5334159aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:45:59.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.843+0000 7f5339980640 1 -- 172.21.15.88:0/3952123288 shutdown_connections 2023-10-16T20:45:59.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:45:59.847+0000 7f5339980640 1 -- 172.21.15.88:0/3952123288 wait complete. 2023-10-16T20:45:59.867 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-16T20:46:01.868 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:02.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.007+0000 7f067abf8640 1 Processor -- start 2023-10-16T20:46:02.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.007+0000 7f067abf8640 1 -- start start 2023-10-16T20:46:02.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.007+0000 7f067abf8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0674150280 0x7f06740c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:02.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.007+0000 7f067abf8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0674150660 0x7f06740c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:02.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.007+0000 7f067abf8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06740c5c60 con 0x7f06740c6bc0 2023-10-16T20:46:02.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.007+0000 7f067abf8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06740c5370 con 0x7f0674150660 2023-10-16T20:46:02.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.007+0000 7f067abf8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0674156d10 con 0x7f0674150280 2023-10-16T20:46:02.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.007+0000 7f066bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0674150660 0x7f06740c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:02.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.007+0000 7f0679195640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f06740c6bc0 0x7f067406d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43866/0 (socket says 172.21.15.88:43866) 2023-10-16T20:46:02.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.007+0000 7f0679195640 1 -- 172.21.15.88:0/1738759331 learned_addr learned my addr 172.21.15.88:0/1738759331 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:46:02.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.007+0000 7f0678994640 1 --2- 172.21.15.88:0/1738759331 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0674150280 0x7f06740c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:02.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.011+0000 7f066bfff640 1 -- 172.21.15.88:0/1738759331 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0674150280 msgr2=0x7f06740c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:02.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.011+0000 7f066bfff640 1 --2- 172.21.15.88:0/1738759331 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0674150280 0x7f06740c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:02.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.011+0000 7f066bfff640 1 -- 172.21.15.88:0/1738759331 >> v1:172.21.15.88:6789/0 conn(0x7f06740c6bc0 legacy=0x7f067406d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:02.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.011+0000 7f066bfff640 1 -- 172.21.15.88:0/1738759331 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06740c5610 con 0x7f0674150660 2023-10-16T20:46:02.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.011+0000 7f066b7fe640 1 -- 172.21.15.88:0/1738759331 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900427155 0 0) 0x7f06740c5c60 con 0x7f06740c6bc0 2023-10-16T20:46:02.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.011+0000 7f066bfff640 1 --2- 172.21.15.88:0/1738759331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0674150660 0x7f06740c3110 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f06640097c0 tx=0x7f066401ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e80bb09f0edf8e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:02.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.011+0000 7f066b7fe640 1 -- 172.21.15.88:0/1738759331 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06640ac640 con 0x7f0674150660 2023-10-16T20:46:02.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.011+0000 7f066b7fe640 1 -- 172.21.15.88:0/1738759331 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06640ac7a0 con 0x7f0674150660 2023-10-16T20:46:02.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.011+0000 7f066b7fe640 1 -- 172.21.15.88:0/1738759331 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06640acb30 con 0x7f0674150660 2023-10-16T20:46:02.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.011+0000 7f067abf8640 1 -- 172.21.15.88:0/1738759331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0674150660 msgr2=0x7f06740c3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:02.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f067abf8640 1 --2- 172.21.15.88:0/1738759331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0674150660 0x7f06740c3110 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f06640097c0 tx=0x7f066401ce20 comp rx=0 tx=0).stop 2023-10-16T20:46:02.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f067abf8640 1 -- 172.21.15.88:0/1738759331 shutdown_connections 2023-10-16T20:46:02.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f067abf8640 1 --2- 172.21.15.88:0/1738759331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0674150660 0x7f06740c3110 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:02.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f067abf8640 1 --2- 172.21.15.88:0/1738759331 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0674150280 0x7f06740c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:02.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f067abf8640 1 -- 172.21.15.88:0/1738759331 >> 172.21.15.88:0/1738759331 conn(0x7f06740cad50 msgr2=0x7f06740cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:02.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f067abf8640 1 -- 172.21.15.88:0/1738759331 shutdown_connections 2023-10-16T20:46:02.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f067abf8640 1 -- 172.21.15.88:0/1738759331 wait complete. 2023-10-16T20:46:02.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f067abf8640 1 Processor -- start 2023-10-16T20:46:02.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f067abf8640 1 -- start start 2023-10-16T20:46:02.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f067abf8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f06740c6bc0 0x7f0674070480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:02.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f067abf8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0674150280 0x7f06740709c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:02.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f067abf8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0674150660 0x7f06741f1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:02.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f067abf8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0674145930 con 0x7f0674150660 2023-10-16T20:46:02.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f067abf8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06741f5ca0 con 0x7f0674150280 2023-10-16T20:46:02.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f067abf8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0674145670 con 0x7f06740c6bc0 2023-10-16T20:46:02.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f0679195640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0674150660 0x7f06741f1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:02.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f0679195640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0674150660 0x7f06741f1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46178/0 (socket says 172.21.15.88:46178) 2023-10-16T20:46:02.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f0679195640 1 -- 172.21.15.88:0/2166091133 learned_addr learned my addr 172.21.15.88:0/2166091133 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:02.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f0678994640 1 --2- 172.21.15.88:0/2166091133 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f06740c6bc0 0x7f0674070480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:02.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f066bfff640 1 --2- 172.21.15.88:0/2166091133 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0674150280 0x7f06740709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:02.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f0679195640 1 -- 172.21.15.88:0/2166091133 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f06740c6bc0 msgr2=0x7f0674070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:02.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f0679195640 1 --2- 172.21.15.88:0/2166091133 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f06740c6bc0 0x7f0674070480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f0679195640 1 -- 172.21.15.88:0/2166091133 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0674150280 msgr2=0x7f06740709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f0679195640 1 --2- 172.21.15.88:0/2166091133 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0674150280 0x7f06740709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f0679195640 1 -- 172.21.15.88:0/2166091133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0664009420 con 0x7f0674150660 2023-10-16T20:46:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f0679195640 1 --2- 172.21.15.88:0/2166091133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0674150660 0x7f06741f1b80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f066c001570 tx=0x7f066c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.015+0000 7f06697fa640 1 -- 172.21.15.88:0/2166091133 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c00a640 con 0x7f0674150660 2023-10-16T20:46:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.019+0000 7f06697fa640 1 -- 172.21.15.88:0/2166091133 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066c00a7c0 con 0x7f0674150660 2023-10-16T20:46:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.019+0000 7f06697fa640 1 -- 172.21.15.88:0/2166091133 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c00ab70 con 0x7f0674150660 2023-10-16T20:46:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.019+0000 7f067abf8640 1 -- 172.21.15.88:0/2166091133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06741f5090 con 0x7f0674150660 2023-10-16T20:46:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.019+0000 7f067abf8640 1 -- 172.21.15.88:0/2166091133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06741f5650 con 0x7f0674150660 2023-10-16T20:46:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.019+0000 7f067abf8640 1 -- 172.21.15.88:0/2166091133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0674074700 con 0x7f0674150660 2023-10-16T20:46:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.019+0000 7f06697fa640 1 -- 172.21.15.88:0/2166091133 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f066c00acd0 con 0x7f0674150660 2023-10-16T20:46:02.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.019+0000 7f06697fa640 1 --2- 172.21.15.88:0/2166091133 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f06600664a0 0x7f0660068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:02.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.019+0000 7f06697fa640 1 -- 172.21.15.88:0/2166091133 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7333+0+0 (secure 0 0 0) 0x7f066c083be0 con 0x7f0674150660 2023-10-16T20:46:02.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.019+0000 7f0678994640 1 --2- 172.21.15.88:0/2166091133 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f06600664a0 0x7f0660068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:02.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.019+0000 7f0678994640 1 --2- 172.21.15.88:0/2166091133 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f06600664a0 0x7f0660068960 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0670009af0 tx=0x7f0670008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:02.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.023+0000 7f06697fa640 1 -- 172.21.15.88:0/2166091133 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f066c0508d0 con 0x7f0674150660 2023-10-16T20:46:02.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.203+0000 7f067abf8640 1 -- 172.21.15.88:0/2166091133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06740c1650 con 0x7f0674150660 2023-10-16T20:46:02.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.203+0000 7f06697fa640 1 -- 172.21.15.88:0/2166091133 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+6581 (secure 0 0 0) 0x7f066c018070 con 0x7f0674150660 2023-10-16T20:46:02.207 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:02.207 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:45:46.511774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:02.208 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 51 2023-10-16T20:46:02.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.207+0000 7f067abf8640 1 -- 172.21.15.88:0/2166091133 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f06600664a0 msgr2=0x7f0660068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:02.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.207+0000 7f067abf8640 1 --2- 172.21.15.88:0/2166091133 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f06600664a0 0x7f0660068960 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0670009af0 tx=0x7f0670008040 comp rx=0 tx=0).stop 2023-10-16T20:46:02.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.207+0000 7f067abf8640 1 -- 172.21.15.88:0/2166091133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0674150660 msgr2=0x7f06741f1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:02.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.207+0000 7f067abf8640 1 --2- 172.21.15.88:0/2166091133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0674150660 0x7f06741f1b80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f066c001570 tx=0x7f066c00cca0 comp rx=0 tx=0).stop 2023-10-16T20:46:02.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.207+0000 7f067abf8640 1 -- 172.21.15.88:0/2166091133 shutdown_connections 2023-10-16T20:46:02.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.207+0000 7f067abf8640 1 --2- 172.21.15.88:0/2166091133 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f06600664a0 0x7f0660068960 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:02.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.207+0000 7f067abf8640 1 --2- 172.21.15.88:0/2166091133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0674150660 0x7f06741f1b80 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:02.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.207+0000 7f067abf8640 1 --2- 172.21.15.88:0/2166091133 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0674150280 0x7f06740709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:02.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.207+0000 7f067abf8640 1 --2- 172.21.15.88:0/2166091133 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f06740c6bc0 0x7f0674070480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:02.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.207+0000 7f067abf8640 1 -- 172.21.15.88:0/2166091133 >> 172.21.15.88:0/2166091133 conn(0x7f06740cad50 msgr2=0x7f0674155190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:02.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.207+0000 7f067abf8640 1 -- 172.21.15.88:0/2166091133 shutdown_connections 2023-10-16T20:46:02.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:02.207+0000 7f067abf8640 1 -- 172.21.15.88:0/2166091133 wait complete. 2023-10-16T20:46:02.228 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-16T20:46:04.229 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:04.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.375+0000 7fc37359e640 1 Processor -- start 2023-10-16T20:46:04.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.375+0000 7fc37359e640 1 -- start start 2023-10-16T20:46:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.375+0000 7fc37359e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc3740c9af0 0x7fc3740cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.375+0000 7fc37359e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc3740b1a60 0x7fc374163b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.375+0000 7fc37359e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3740c9fc0 con 0x7fc3740cb580 2023-10-16T20:46:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.375+0000 7fc37359e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3740b20c0 con 0x7fc3740c9af0 2023-10-16T20:46:04.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.375+0000 7fc37359e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3740cbea0 con 0x7fc3740b1a60 2023-10-16T20:46:04.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.375+0000 7fc37259c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc3740cb580 0x7fc374159300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43876/0 (socket says 172.21.15.88:43876) 2023-10-16T20:46:04.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.375+0000 7fc37259c640 1 -- 172.21.15.88:0/1800578551 learned_addr learned my addr 172.21.15.88:0/1800578551 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:46:04.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.375+0000 7fc371d9b640 1 --2- 172.21.15.88:0/1800578551 >> v2:172.21.15.88:3300/0 conn(0x7fc3740c9af0 0x7fc3740cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:04.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.375+0000 7fc372d9d640 1 --2- 172.21.15.88:0/1800578551 >> v2:172.21.15.191:3300/0 conn(0x7fc3740b1a60 0x7fc374163b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.375+0000 7fc37159a640 1 -- 172.21.15.88:0/1800578551 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875148196 0 0) 0x7fc3740c9fc0 con 0x7fc3740cb580 2023-10-16T20:46:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.375+0000 7fc37159a640 1 -- 172.21.15.88:0/1800578551 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc35c003530 con 0x7fc3740cb580 2023-10-16T20:46:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.375+0000 7fc371d9b640 1 -- 172.21.15.88:0/1800578551 >> v2:172.21.15.191:3300/0 conn(0x7fc3740b1a60 msgr2=0x7fc374163b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.375+0000 7fc371d9b640 1 --2- 172.21.15.88:0/1800578551 >> v2:172.21.15.191:3300/0 conn(0x7fc3740b1a60 0x7fc374163b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.375+0000 7fc371d9b640 1 -- 172.21.15.88:0/1800578551 >> v1:172.21.15.88:6789/0 conn(0x7fc3740cb580 legacy=0x7fc374159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:04.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.375+0000 7fc371d9b640 1 -- 172.21.15.88:0/1800578551 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3740c47d0 con 0x7fc3740c9af0 2023-10-16T20:46:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.379+0000 7fc371d9b640 1 --2- 172.21.15.88:0/1800578551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3740c9af0 0x7fc3740cb960 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc3740cfb80 tx=0x7fc36801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff701dea35f1daf server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.379+0000 7fc37159a640 1 -- 172.21.15.88:0/1800578551 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3680ac600 con 0x7fc3740c9af0 2023-10-16T20:46:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.379+0000 7fc37159a640 1 -- 172.21.15.88:0/1800578551 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3680a8050 con 0x7fc3740c9af0 2023-10-16T20:46:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.379+0000 7fc37159a640 1 -- 172.21.15.88:0/1800578551 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3680ac980 con 0x7fc3740c9af0 2023-10-16T20:46:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.379+0000 7fc37359e640 1 -- 172.21.15.88:0/1800578551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3740c9af0 msgr2=0x7fc3740cb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.379+0000 7fc37359e640 1 --2- 172.21.15.88:0/1800578551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3740c9af0 0x7fc3740cb960 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc3740cfb80 tx=0x7fc36801c3a0 comp rx=0 tx=0).stop 2023-10-16T20:46:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.379+0000 7fc37359e640 1 -- 172.21.15.88:0/1800578551 shutdown_connections 2023-10-16T20:46:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.379+0000 7fc37359e640 1 --2- 172.21.15.88:0/1800578551 >> v2:172.21.15.191:3300/0 conn(0x7fc3740b1a60 0x7fc374163b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.379+0000 7fc37359e640 1 --2- 172.21.15.88:0/1800578551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3740c9af0 0x7fc3740cb960 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.379+0000 7fc37359e640 1 -- 172.21.15.88:0/1800578551 >> 172.21.15.88:0/1800578551 conn(0x7fc37406bb40 msgr2=0x7fc37406bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.379+0000 7fc37359e640 1 -- 172.21.15.88:0/1800578551 shutdown_connections 2023-10-16T20:46:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.379+0000 7fc37359e640 1 -- 172.21.15.88:0/1800578551 wait complete. 2023-10-16T20:46:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc37359e640 1 Processor -- start 2023-10-16T20:46:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc37359e640 1 -- start start 2023-10-16T20:46:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc37359e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3740b1a60 0x7fc3741f50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc37359e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3740c9af0 0x7fc3741f5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc37359e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3740cb580 0x7fc3740b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc37359e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3741589f0 con 0x7fc3740cb580 2023-10-16T20:46:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc37359e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc374157e70 con 0x7fc3740c9af0 2023-10-16T20:46:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc37359e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37414cb60 con 0x7fc3740b1a60 2023-10-16T20:46:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc372d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3740cb580 0x7fc3740b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc372d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3740cb580 0x7fc3740b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46192/0 (socket says 172.21.15.88:46192) 2023-10-16T20:46:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc372d9d640 1 -- 172.21.15.88:0/3933634358 learned_addr learned my addr 172.21.15.88:0/3933634358 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:04.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc371d9b640 1 --2- 172.21.15.88:0/3933634358 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3740c9af0 0x7fc3741f5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:04.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc37259c640 1 --2- 172.21.15.88:0/3933634358 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3740b1a60 0x7fc3741f50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:04.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc372d9d640 1 -- 172.21.15.88:0/3933634358 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3740b1a60 msgr2=0x7fc3741f50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:04.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc372d9d640 1 --2- 172.21.15.88:0/3933634358 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3740b1a60 0x7fc3741f50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:04.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc372d9d640 1 -- 172.21.15.88:0/3933634358 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3740c9af0 msgr2=0x7fc3741f5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:04.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc372d9d640 1 --2- 172.21.15.88:0/3933634358 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3740c9af0 0x7fc3741f5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:04.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc372d9d640 1 -- 172.21.15.88:0/3933634358 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc368009420 con 0x7fc3740cb580 2023-10-16T20:46:04.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc37259c640 1 --2- 172.21.15.88:0/3933634358 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3740b1a60 0x7fc3741f50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:46:04.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc372d9d640 1 --2- 172.21.15.88:0/3933634358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3740cb580 0x7fc3740b2fc0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc36c00ef90 tx=0x7fc36c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:04.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc362ffd640 1 -- 172.21.15.88:0/3933634358 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c009ca0 con 0x7fc3740cb580 2023-10-16T20:46:04.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc362ffd640 1 -- 172.21.15.88:0/3933634358 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36c00f040 con 0x7fc3740cb580 2023-10-16T20:46:04.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc37359e640 1 -- 172.21.15.88:0/3933634358 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3741f5c30 con 0x7fc3740cb580 2023-10-16T20:46:04.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc362ffd640 1 -- 172.21.15.88:0/3933634358 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c0146c0 con 0x7fc3740cb580 2023-10-16T20:46:04.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc37359e640 1 -- 172.21.15.88:0/3933634358 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3741583a0 con 0x7fc3740cb580 2023-10-16T20:46:04.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.383+0000 7fc37359e640 1 -- 172.21.15.88:0/3933634358 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc374158ba0 con 0x7fc3740cb580 2023-10-16T20:46:04.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.387+0000 7fc362ffd640 1 -- 172.21.15.88:0/3933634358 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc36c014820 con 0x7fc3740cb580 2023-10-16T20:46:04.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.387+0000 7fc362ffd640 1 --2- 172.21.15.88:0/3933634358 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc3540664a0 0x7fc354068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:04.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.387+0000 7fc362ffd640 1 -- 172.21.15.88:0/3933634358 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7333+0+0 (secure 0 0 0) 0x7fc36c085440 con 0x7fc3740cb580 2023-10-16T20:46:04.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.387+0000 7fc37259c640 1 --2- 172.21.15.88:0/3933634358 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc3540664a0 0x7fc354068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:04.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.387+0000 7fc37259c640 1 --2- 172.21.15.88:0/3933634358 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc3540664a0 0x7fc354068960 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc364005890 tx=0x7fc364009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:04.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.391+0000 7fc362ffd640 1 -- 172.21.15.88:0/3933634358 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc374158ba0 con 0x7fc3740cb580 2023-10-16T20:46:04.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.571+0000 7fc37359e640 1 -- 172.21.15.88:0/3933634358 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc374158ba0 con 0x7fc3740cb580 2023-10-16T20:46:04.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.571+0000 7fc362ffd640 1 -- 172.21.15.88:0/3933634358 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+6581 (secure 0 0 0) 0x7fc374158ba0 con 0x7fc3740cb580 2023-10-16T20:46:04.576 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:04.576 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:45:46.511774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:04.576 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 51 2023-10-16T20:46:04.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.575+0000 7fc37359e640 1 -- 172.21.15.88:0/3933634358 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc3540664a0 msgr2=0x7fc354068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:04.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.575+0000 7fc37359e640 1 --2- 172.21.15.88:0/3933634358 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc3540664a0 0x7fc354068960 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc364005890 tx=0x7fc364009040 comp rx=0 tx=0).stop 2023-10-16T20:46:04.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.575+0000 7fc37359e640 1 -- 172.21.15.88:0/3933634358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3740cb580 msgr2=0x7fc3740b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:04.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.575+0000 7fc37359e640 1 --2- 172.21.15.88:0/3933634358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3740cb580 0x7fc3740b2fc0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc36c00ef90 tx=0x7fc36c00c490 comp rx=0 tx=0).stop 2023-10-16T20:46:04.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.575+0000 7fc37359e640 1 -- 172.21.15.88:0/3933634358 shutdown_connections 2023-10-16T20:46:04.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.575+0000 7fc37359e640 1 --2- 172.21.15.88:0/3933634358 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc3540664a0 0x7fc354068960 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:04.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.575+0000 7fc37359e640 1 --2- 172.21.15.88:0/3933634358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3740cb580 0x7fc3740b2fc0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:04.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.575+0000 7fc37359e640 1 --2- 172.21.15.88:0/3933634358 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc3740c9af0 0x7fc3741f5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:04.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.575+0000 7fc37359e640 1 --2- 172.21.15.88:0/3933634358 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc3740b1a60 0x7fc3741f50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:04.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.575+0000 7fc37359e640 1 -- 172.21.15.88:0/3933634358 >> 172.21.15.88:0/3933634358 conn(0x7fc37406bb40 msgr2=0x7fc3740cdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:04.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.575+0000 7fc37359e640 1 -- 172.21.15.88:0/3933634358 shutdown_connections 2023-10-16T20:46:04.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:04.575+0000 7fc37359e640 1 -- 172.21.15.88:0/3933634358 wait complete. 2023-10-16T20:46:04.596 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-16T20:46:06.597 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:06.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.743+0000 7fcdcc1d9640 1 Processor -- start 2023-10-16T20:46:06.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.743+0000 7fcdcc1d9640 1 -- start start 2023-10-16T20:46:06.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdcc1d9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcdc4157b10 0x7fcdc415e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:06.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdcc1d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdc4160e70 0x7fcdc4161290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:06.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdcc1d9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdc41617d0 con 0x7fcdc40c58d0 2023-10-16T20:46:06.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdcc1d9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdc40c5e10 con 0x7fcdc4157b10 2023-10-16T20:46:06.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdcc1d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc415e810 con 0x7fcdc4160e70 2023-10-16T20:46:06.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc9774640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcdc40c58d0 0x7fcdc406d170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43890/0 (socket says 172.21.15.88:43890) 2023-10-16T20:46:06.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc9774640 1 -- 172.21.15.88:0/1807007843 learned_addr learned my addr 172.21.15.88:0/1807007843 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:46:06.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc8f73640 1 -- 172.21.15.88:0/1807007843 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2141643373 0 0) 0x7fcdc41617d0 con 0x7fcdc40c58d0 2023-10-16T20:46:06.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc8f73640 1 -- 172.21.15.88:0/1807007843 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdb0003530 con 0x7fcdc40c58d0 2023-10-16T20:46:06.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdca776640 1 --2- 172.21.15.88:0/1807007843 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdc4160e70 0x7fcdc4161290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:06.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc9f75640 1 --2- 172.21.15.88:0/1807007843 >> v2:172.21.15.191:3300/0 conn(0x7fcdc4157b10 0x7fcdc415e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:06.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc8f73640 1 -- 172.21.15.88:0/1807007843 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4010595641 0 0) 0x7fcdb0003530 con 0x7fcdc40c58d0 2023-10-16T20:46:06.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc8f73640 1 -- 172.21.15.88:0/1807007843 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdc4160e70 msgr2=0x7fcdc4161290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:06.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc8f73640 1 --2- 172.21.15.88:0/1807007843 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdc4160e70 0x7fcdc4161290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:06.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc8f73640 1 -- 172.21.15.88:0/1807007843 >> v2:172.21.15.191:3300/0 conn(0x7fcdc4157b10 msgr2=0x7fcdc415e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:06.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc8f73640 1 --2- 172.21.15.88:0/1807007843 >> v2:172.21.15.191:3300/0 conn(0x7fcdc4157b10 0x7fcdc415e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:06.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc8f73640 1 -- 172.21.15.88:0/1807007843 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc406f840 con 0x7fcdc40c58d0 2023-10-16T20:46:06.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc8f73640 1 -- 172.21.15.88:0/1807007843 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7fcdb4001680 con 0x7fcdc40c58d0 2023-10-16T20:46:06.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc8f73640 1 -- 172.21.15.88:0/1807007843 >> v1:172.21.15.88:6789/0 conn(0x7fcdc40c58d0 legacy=0x7fcdc406d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:06.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc8f73640 1 --2- 172.21.15.88:0/1807007843 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdc4160e70 0x7fcdb0003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:06.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc8f73640 1 --2- 172.21.15.88:0/1807007843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdb0003f00 0x7fcdb00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:06.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc8f73640 1 --2- 172.21.15.88:0/1807007843 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdb0004d10 0x7fcdb0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:06.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc8f73640 1 -- 172.21.15.88:0/1807007843 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc40c5e10 con 0x7fcdb0003f00 2023-10-16T20:46:06.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc8f73640 1 -- 172.21.15.88:0/1807007843 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc415e810 con 0x7fcdb0004d10 2023-10-16T20:46:06.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc8f73640 1 -- 172.21.15.88:0/1807007843 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb000ba90 con 0x7fcdc4160e70 2023-10-16T20:46:06.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc9f75640 1 --2- 172.21.15.88:0/1807007843 >> v2:172.21.15.191:3300/0 conn(0x7fcdc4157b10 0x7fcdc415e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:46:06.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc9774640 1 --2- 172.21.15.88:0/1807007843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdb0003f00 0x7fcdb00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:06.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdca776640 1 --2- 172.21.15.88:0/1807007843 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdb0004d10 0x7fcdb0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:06.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc9f75640 1 --2- 172.21.15.88:0/1807007843 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdc4160e70 0x7fcdb0003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:06.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc9774640 1 -- 172.21.15.88:0/1807007843 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdc4160e70 msgr2=0x7fcdb0003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:06.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc9774640 1 --2- 172.21.15.88:0/1807007843 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdc4160e70 0x7fcdb0003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:06.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc9774640 1 -- 172.21.15.88:0/1807007843 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdb0004d10 msgr2=0x7fcdb0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:06.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc9774640 1 --2- 172.21.15.88:0/1807007843 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdb0004d10 0x7fcdb0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:06.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdc9774640 1 -- 172.21.15.88:0/1807007843 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb0003530 con 0x7fcdb0003f00 2023-10-16T20:46:06.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.747+0000 7fcdca776640 1 --2- 172.21.15.88:0/1807007843 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdb0004d10 0x7fcdb0005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:46:06.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.751+0000 7fcdc9774640 1 --2- 172.21.15.88:0/1807007843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdb0003f00 0x7fcdb00047d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fcdb400c240 tx=0x7fcdb401cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83a615520b1ad61 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:06.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.751+0000 7fcdc8f73640 1 -- 172.21.15.88:0/1807007843 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb40058e0 con 0x7fcdb0003f00 2023-10-16T20:46:06.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.751+0000 7fcdc8f73640 1 -- 172.21.15.88:0/1807007843 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb4005a40 con 0x7fcdb0003f00 2023-10-16T20:46:06.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.751+0000 7fcdc8f73640 1 -- 172.21.15.88:0/1807007843 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4005e50 con 0x7fcdb0003f00 2023-10-16T20:46:06.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.751+0000 7fcdcc1d9640 1 -- 172.21.15.88:0/1807007843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdb0003f00 msgr2=0x7fcdb00047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:06.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.751+0000 7fcdcc1d9640 1 --2- 172.21.15.88:0/1807007843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdb0003f00 0x7fcdb00047d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fcdb400c240 tx=0x7fcdb401cc10 comp rx=0 tx=0).stop 2023-10-16T20:46:06.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.751+0000 7fcdca776640 1 -- 172.21.15.88:0/1807007843 reap_dead start 2023-10-16T20:46:06.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.751+0000 7fcdcc1d9640 1 -- 172.21.15.88:0/1807007843 shutdown_connections 2023-10-16T20:46:06.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.751+0000 7fcdcc1d9640 1 -- 172.21.15.88:0/1807007843 >> 172.21.15.88:0/1807007843 conn(0x7fcdc40cad50 msgr2=0x7fcdc40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:06.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.751+0000 7fcdcc1d9640 1 -- 172.21.15.88:0/1807007843 shutdown_connections 2023-10-16T20:46:06.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.751+0000 7fcdcc1d9640 1 -- 172.21.15.88:0/1807007843 wait complete. 2023-10-16T20:46:06.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.751+0000 7fcdcc1d9640 1 Processor -- start 2023-10-16T20:46:06.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdcc1d9640 1 -- start start 2023-10-16T20:46:06.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdcc1d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdb0003f00 0x7fcdc41f94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:06.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdcc1d9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdc41f9a30 0x7fcdc4153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:06.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdcc1d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdc41552c0 0x7fcdc4153ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:06.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdcc1d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc40c29c0 con 0x7fcdc41552c0 2023-10-16T20:46:06.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdcc1d9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc40c2e50 con 0x7fcdc41f9a30 2023-10-16T20:46:06.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdcc1d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc414fbc0 con 0x7fcdb0003f00 2023-10-16T20:46:06.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdca776640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdc41552c0 0x7fcdc4153ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:06.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdca776640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdc41552c0 0x7fcdc4153ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46216/0 (socket says 172.21.15.88:46216) 2023-10-16T20:46:06.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdca776640 1 -- 172.21.15.88:0/1949391976 learned_addr learned my addr 172.21.15.88:0/1949391976 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:06.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdc9f75640 1 --2- 172.21.15.88:0/1949391976 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdb0003f00 0x7fcdc41f94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:06.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdc9774640 1 --2- 172.21.15.88:0/1949391976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdc41f9a30 0x7fcdc4153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:06.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdca776640 1 -- 172.21.15.88:0/1949391976 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdb0003f00 msgr2=0x7fcdc41f94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:06.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdca776640 1 --2- 172.21.15.88:0/1949391976 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdb0003f00 0x7fcdc41f94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:06.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdca776640 1 -- 172.21.15.88:0/1949391976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdc41f9a30 msgr2=0x7fcdc4153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:06.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdca776640 1 --2- 172.21.15.88:0/1949391976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdc41f9a30 0x7fcdc4153590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:06.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdca776640 1 -- 172.21.15.88:0/1949391976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc41fa1f0 con 0x7fcdc41552c0 2023-10-16T20:46:06.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdca776640 1 --2- 172.21.15.88:0/1949391976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdc41552c0 0x7fcdc4153ad0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fcdc406f6c0 tx=0x7fcdbc010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:06.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdba7fc640 1 -- 172.21.15.88:0/1949391976 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc00e050 con 0x7fcdc41552c0 2023-10-16T20:46:06.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdba7fc640 1 -- 172.21.15.88:0/1949391976 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdbc00f040 con 0x7fcdc41552c0 2023-10-16T20:46:06.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdba7fc640 1 -- 172.21.15.88:0/1949391976 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc0039a0 con 0x7fcdc41552c0 2023-10-16T20:46:06.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdcc1d9640 1 -- 172.21.15.88:0/1949391976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc414f900 con 0x7fcdc41552c0 2023-10-16T20:46:06.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdcc1d9640 1 -- 172.21.15.88:0/1949391976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc41f9f10 con 0x7fcdc41552c0 2023-10-16T20:46:06.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.755+0000 7fcdcc1d9640 1 -- 172.21.15.88:0/1949391976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc40c2b70 con 0x7fcdc41552c0 2023-10-16T20:46:06.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.759+0000 7fcdba7fc640 1 -- 172.21.15.88:0/1949391976 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcdbc015440 con 0x7fcdc41552c0 2023-10-16T20:46:06.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.759+0000 7fcdba7fc640 1 --2- 172.21.15.88:0/1949391976 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcda80664a0 0x7fcda8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:06.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.759+0000 7fcdba7fc640 1 -- 172.21.15.88:0/1949391976 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7333+0+0 (secure 0 0 0) 0x7fcdbc0847d0 con 0x7fcdc41552c0 2023-10-16T20:46:06.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.759+0000 7fcdc9f75640 1 --2- 172.21.15.88:0/1949391976 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcda80664a0 0x7fcda8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:06.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.759+0000 7fcdc9f75640 1 --2- 172.21.15.88:0/1949391976 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcda80664a0 0x7fcda8068960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0004490 tx=0x7fcdc0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:06.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.763+0000 7fcdba7fc640 1 -- 172.21.15.88:0/1949391976 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcdc40c2b70 con 0x7fcdc41552c0 2023-10-16T20:46:06.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.947+0000 7fcdcc1d9640 1 -- 172.21.15.88:0/1949391976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdc40c2b70 con 0x7fcdc41552c0 2023-10-16T20:46:06.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.947+0000 7fcdba7fc640 1 -- 172.21.15.88:0/1949391976 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+6581 (secure 0 0 0) 0x7fcdc40c2b70 con 0x7fcdc41552c0 2023-10-16T20:46:06.951 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:06.951 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:45:46.511774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:06.951 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 51 2023-10-16T20:46:06.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.947+0000 7fcdcc1d9640 1 -- 172.21.15.88:0/1949391976 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcda80664a0 msgr2=0x7fcda8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:06.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.947+0000 7fcdcc1d9640 1 --2- 172.21.15.88:0/1949391976 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcda80664a0 0x7fcda8068960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0004490 tx=0x7fcdc0008040 comp rx=0 tx=0).stop 2023-10-16T20:46:06.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.947+0000 7fcdcc1d9640 1 -- 172.21.15.88:0/1949391976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdc41552c0 msgr2=0x7fcdc4153ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:06.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.947+0000 7fcdcc1d9640 1 --2- 172.21.15.88:0/1949391976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdc41552c0 0x7fcdc4153ad0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fcdc406f6c0 tx=0x7fcdbc010590 comp rx=0 tx=0).stop 2023-10-16T20:46:06.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.947+0000 7fcdcc1d9640 1 -- 172.21.15.88:0/1949391976 shutdown_connections 2023-10-16T20:46:06.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.947+0000 7fcdcc1d9640 1 --2- 172.21.15.88:0/1949391976 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcda80664a0 0x7fcda8068960 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:06.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.947+0000 7fcdcc1d9640 1 --2- 172.21.15.88:0/1949391976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdc41552c0 0x7fcdc4153ad0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:06.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.947+0000 7fcdcc1d9640 1 --2- 172.21.15.88:0/1949391976 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdc41f9a30 0x7fcdc4153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:06.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.947+0000 7fcdcc1d9640 1 --2- 172.21.15.88:0/1949391976 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdb0003f00 0x7fcdc41f94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:06.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.947+0000 7fcdcc1d9640 1 -- 172.21.15.88:0/1949391976 >> 172.21.15.88:0/1949391976 conn(0x7fcdc40cad50 msgr2=0x7fcdc40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:06.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.947+0000 7fcdcc1d9640 1 -- 172.21.15.88:0/1949391976 shutdown_connections 2023-10-16T20:46:06.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:06.947+0000 7fcdcc1d9640 1 -- 172.21.15.88:0/1949391976 wait complete. 2023-10-16T20:46:06.971 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-16T20:46:08.972 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:09.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.115+0000 7f69d4005640 1 Processor -- start 2023-10-16T20:46:09.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.115+0000 7f69d4005640 1 -- start start 2023-10-16T20:46:09.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.119+0000 7f69d4005640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69cc0c5300 0x7f69cc13e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:09.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.119+0000 7f69d4005640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f69cc15d840 0x7f69cc159640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:09.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.119+0000 7f69d4005640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69cc15a590 con 0x7f69cc0cb890 2023-10-16T20:46:09.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.119+0000 7f69d4005640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69cc15d420 con 0x7f69cc15d840 2023-10-16T20:46:09.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.119+0000 7f69d4005640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc0c7830 con 0x7f69cc0c5300 2023-10-16T20:46:09.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.119+0000 7f69d15a0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f69cc15d840 0x7f69cc159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:09.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.119+0000 7f69d15a0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f69cc15d840 0x7f69cc159640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58966/0 (socket says 172.21.15.88:58966) 2023-10-16T20:46:09.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.119+0000 7f69d15a0640 1 -- 172.21.15.88:0/3997143546 learned_addr learned my addr 172.21.15.88:0/3997143546 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:09.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.119+0000 7f69d1da1640 1 --2- 172.21.15.88:0/3997143546 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69cc0c5300 0x7f69cc13e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:09.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.119+0000 7f69d15a0640 1 -- 172.21.15.88:0/3997143546 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69cc0c5300 msgr2=0x7f69cc13e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:09.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.119+0000 7f69d15a0640 1 --2- 172.21.15.88:0/3997143546 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69cc0c5300 0x7f69cc13e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:09.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.119+0000 7f69d15a0640 1 -- 172.21.15.88:0/3997143546 >> v1:172.21.15.191:6789/0 conn(0x7f69cc0cb890 legacy=0x7f69cc06ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:09.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.119+0000 7f69d15a0640 1 -- 172.21.15.88:0/3997143546 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc13ed50 con 0x7f69cc15d840 2023-10-16T20:46:09.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.123+0000 7f69d15a0640 1 --2- 172.21.15.88:0/3997143546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69cc15d840 0x7f69cc159640 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009870 tx=0x7f69bc01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2409e6d7fbfdeab4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:09.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.123+0000 7f69d0d9f640 1 -- 172.21.15.88:0/3997143546 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc0aa670 con 0x7f69cc15d840 2023-10-16T20:46:09.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.123+0000 7f69d0d9f640 1 -- 172.21.15.88:0/3997143546 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69bc0aa7d0 con 0x7f69cc15d840 2023-10-16T20:46:09.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.123+0000 7f69d0d9f640 1 -- 172.21.15.88:0/3997143546 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc0aab60 con 0x7f69cc15d840 2023-10-16T20:46:09.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.123+0000 7f69d4005640 1 -- 172.21.15.88:0/3997143546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69cc15d840 msgr2=0x7f69cc159640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:09.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.123+0000 7f69d4005640 1 --2- 172.21.15.88:0/3997143546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69cc15d840 0x7f69cc159640 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009870 tx=0x7f69bc01c510 comp rx=0 tx=0).stop 2023-10-16T20:46:09.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.123+0000 7f69d4005640 1 -- 172.21.15.88:0/3997143546 shutdown_connections 2023-10-16T20:46:09.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.123+0000 7f69d4005640 1 --2- 172.21.15.88:0/3997143546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69cc15d840 0x7f69cc159640 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:09.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.123+0000 7f69d4005640 1 --2- 172.21.15.88:0/3997143546 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69cc0c5300 0x7f69cc13e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:09.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.123+0000 7f69d4005640 1 -- 172.21.15.88:0/3997143546 >> 172.21.15.88:0/3997143546 conn(0x7f69cc06e0f0 msgr2=0x7f69cc0cb180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:09.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.123+0000 7f69d4005640 1 -- 172.21.15.88:0/3997143546 shutdown_connections 2023-10-16T20:46:09.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.123+0000 7f69d4005640 1 -- 172.21.15.88:0/3997143546 wait complete. 2023-10-16T20:46:09.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.123+0000 7f69d4005640 1 Processor -- start 2023-10-16T20:46:09.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.123+0000 7f69d4005640 1 -- start start 2023-10-16T20:46:09.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d4005640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69cc0c5300 0x7f69cc139d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:09.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d4005640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69cc0cb890 0x7f69cc0ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:09.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d4005640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69cc15d840 0x7f69cc0caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:09.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d4005640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc13c060 con 0x7f69cc15d840 2023-10-16T20:46:09.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d4005640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc13b9e0 con 0x7f69cc0cb890 2023-10-16T20:46:09.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d4005640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc1594f0 con 0x7f69cc0c5300 2023-10-16T20:46:09.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d25a2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69cc15d840 0x7f69cc0caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:09.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d1da1640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69cc0c5300 0x7f69cc139d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:09.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d25a2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69cc15d840 0x7f69cc0caed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58968/0 (socket says 172.21.15.88:58968) 2023-10-16T20:46:09.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d25a2640 1 -- 172.21.15.88:0/4059356223 learned_addr learned my addr 172.21.15.88:0/4059356223 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:09.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d15a0640 1 --2- 172.21.15.88:0/4059356223 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69cc0cb890 0x7f69cc0ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:09.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d25a2640 1 -- 172.21.15.88:0/4059356223 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69cc0c5300 msgr2=0x7f69cc139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:09.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d25a2640 1 --2- 172.21.15.88:0/4059356223 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69cc0c5300 0x7f69cc139d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:09.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d25a2640 1 -- 172.21.15.88:0/4059356223 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69cc0cb890 msgr2=0x7f69cc0ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:09.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d25a2640 1 --2- 172.21.15.88:0/4059356223 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69cc0cb890 0x7f69cc0ca990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:09.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d25a2640 1 -- 172.21.15.88:0/4059356223 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69bc0094d0 con 0x7f69cc15d840 2023-10-16T20:46:09.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d25a2640 1 --2- 172.21.15.88:0/4059356223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69cc15d840 0x7f69cc0caed0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f69c400c770 tx=0x7f69c400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:09.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69c27fc640 1 -- 172.21.15.88:0/4059356223 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c4003b80 con 0x7f69cc15d840 2023-10-16T20:46:09.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69c27fc640 1 -- 172.21.15.88:0/4059356223 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c4003ce0 con 0x7f69cc15d840 2023-10-16T20:46:09.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d4005640 1 -- 172.21.15.88:0/4059356223 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69cc13bc80 con 0x7f69cc15d840 2023-10-16T20:46:09.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69c27fc640 1 -- 172.21.15.88:0/4059356223 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c400aa90 con 0x7f69cc15d840 2023-10-16T20:46:09.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d4005640 1 -- 172.21.15.88:0/4059356223 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69cc1597d0 con 0x7f69cc15d840 2023-10-16T20:46:09.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69d4005640 1 -- 172.21.15.88:0/4059356223 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69cc13c1a0 con 0x7f69cc15d840 2023-10-16T20:46:09.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.127+0000 7f69c27fc640 1 -- 172.21.15.88:0/4059356223 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f69c400abf0 con 0x7f69cc15d840 2023-10-16T20:46:09.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.131+0000 7f69c27fc640 1 --2- 172.21.15.88:0/4059356223 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69b00664d0 0x7f69b0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:09.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.131+0000 7f69c27fc640 1 -- 172.21.15.88:0/4059356223 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7333+0+0 (secure 0 0 0) 0x7f69c40841a0 con 0x7f69cc15d840 2023-10-16T20:46:09.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.131+0000 7f69d1da1640 1 --2- 172.21.15.88:0/4059356223 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69b00664d0 0x7f69b0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:09.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.131+0000 7f69d1da1640 1 --2- 172.21.15.88:0/4059356223 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69b00664d0 0x7f69b0068990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f69c8009b60 tx=0x7f69c8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:09.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.135+0000 7f69c27fc640 1 -- 172.21.15.88:0/4059356223 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f69cc13c1a0 con 0x7f69cc15d840 2023-10-16T20:46:09.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.315+0000 7f69d4005640 1 -- 172.21.15.88:0/4059356223 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69cc13c1a0 con 0x7f69cc15d840 2023-10-16T20:46:09.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.315+0000 7f69c27fc640 1 -- 172.21.15.88:0/4059356223 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+6581 (secure 0 0 0) 0x7f69cc13c1a0 con 0x7f69cc15d840 2023-10-16T20:46:09.317 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:09.318 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:45:46.511774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:09.319 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 51 2023-10-16T20:46:09.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.315+0000 7f69d4005640 1 -- 172.21.15.88:0/4059356223 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69b00664d0 msgr2=0x7f69b0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:09.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.315+0000 7f69d4005640 1 --2- 172.21.15.88:0/4059356223 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69b00664d0 0x7f69b0068990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f69c8009b60 tx=0x7f69c8009190 comp rx=0 tx=0).stop 2023-10-16T20:46:09.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.315+0000 7f69d4005640 1 -- 172.21.15.88:0/4059356223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69cc15d840 msgr2=0x7f69cc0caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:09.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.315+0000 7f69d4005640 1 --2- 172.21.15.88:0/4059356223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69cc15d840 0x7f69cc0caed0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f69c400c770 tx=0x7f69c400cc30 comp rx=0 tx=0).stop 2023-10-16T20:46:09.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.315+0000 7f69d4005640 1 -- 172.21.15.88:0/4059356223 shutdown_connections 2023-10-16T20:46:09.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.315+0000 7f69d4005640 1 --2- 172.21.15.88:0/4059356223 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69b00664d0 0x7f69b0068990 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:09.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.315+0000 7f69d4005640 1 --2- 172.21.15.88:0/4059356223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69cc15d840 0x7f69cc0caed0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:09.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.315+0000 7f69d4005640 1 --2- 172.21.15.88:0/4059356223 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69cc0cb890 0x7f69cc0ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:09.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.315+0000 7f69d4005640 1 --2- 172.21.15.88:0/4059356223 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69cc0c5300 0x7f69cc139d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:09.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.315+0000 7f69d4005640 1 -- 172.21.15.88:0/4059356223 >> 172.21.15.88:0/4059356223 conn(0x7f69cc06e0f0 msgr2=0x7f69cc06ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:09.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.315+0000 7f69d4005640 1 -- 172.21.15.88:0/4059356223 shutdown_connections 2023-10-16T20:46:09.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:09.315+0000 7f69d4005640 1 -- 172.21.15.88:0/4059356223 wait complete. 2023-10-16T20:46:09.338 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-16T20:46:11.339 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:11.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.487+0000 7fd53afc4640 1 Processor -- start 2023-10-16T20:46:11.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.487+0000 7fd53afc4640 1 -- start start 2023-10-16T20:46:11.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.487+0000 7fd53afc4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd5340c9bb0 0x7fd5340c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:11.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.487+0000 7fd53afc4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd5340c41f0 0x7fd5340cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:11.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.487+0000 7fd53afc4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd534153b50 con 0x7fd534153770 2023-10-16T20:46:11.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.487+0000 7fd53afc4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5340cc250 con 0x7fd5340c41f0 2023-10-16T20:46:11.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.487+0000 7fd53afc4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5340cc670 con 0x7fd5340c9bb0 2023-10-16T20:46:11.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.487+0000 7fd539561640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd5340c41f0 0x7fd5340cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:11.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.487+0000 7fd533fff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd5340c9bb0 0x7fd5340c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:11.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.487+0000 7fd538d60640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fd534153770 0x7fd534159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:33264/0 (socket says 172.21.15.88:33264) 2023-10-16T20:46:11.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.487+0000 7fd538d60640 1 -- 172.21.15.88:0/3648320758 learned_addr learned my addr 172.21.15.88:0/3648320758 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:46:11.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.487+0000 7fd539561640 1 -- 172.21.15.88:0/3648320758 >> v2:172.21.15.191:3300/0 conn(0x7fd5340c9bb0 msgr2=0x7fd5340c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:11.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.487+0000 7fd539561640 1 --2- 172.21.15.88:0/3648320758 >> v2:172.21.15.191:3300/0 conn(0x7fd5340c9bb0 0x7fd5340c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:11.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.487+0000 7fd539561640 1 -- 172.21.15.88:0/3648320758 >> v1:172.21.15.191:6789/0 conn(0x7fd534153770 legacy=0x7fd534159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:11.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.487+0000 7fd539561640 1 -- 172.21.15.88:0/3648320758 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5340cb2f0 con 0x7fd5340c41f0 2023-10-16T20:46:11.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.487+0000 7fd533fff640 1 --2- 172.21.15.88:0/3648320758 >> v2:172.21.15.191:3300/0 conn(0x7fd5340c9bb0 0x7fd5340c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:46:11.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.491+0000 7fd539561640 1 --2- 172.21.15.88:0/3648320758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5340c41f0 0x7fd5340cece0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd5340cfe50 tx=0x7fd52c01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2003adcda3010a0a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:11.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.491+0000 7fd5337fe640 1 -- 172.21.15.88:0/3648320758 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c0a58e0 con 0x7fd5340c41f0 2023-10-16T20:46:11.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.491+0000 7fd5337fe640 1 -- 172.21.15.88:0/3648320758 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c0a5a40 con 0x7fd5340c41f0 2023-10-16T20:46:11.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.491+0000 7fd5337fe640 1 -- 172.21.15.88:0/3648320758 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c0a5dd0 con 0x7fd5340c41f0 2023-10-16T20:46:11.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.491+0000 7fd53afc4640 1 -- 172.21.15.88:0/3648320758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5340c41f0 msgr2=0x7fd5340cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:11.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.491+0000 7fd53afc4640 1 --2- 172.21.15.88:0/3648320758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5340c41f0 0x7fd5340cece0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd5340cfe50 tx=0x7fd52c01c910 comp rx=0 tx=0).stop 2023-10-16T20:46:11.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.491+0000 7fd53afc4640 1 -- 172.21.15.88:0/3648320758 shutdown_connections 2023-10-16T20:46:11.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.491+0000 7fd53afc4640 1 --2- 172.21.15.88:0/3648320758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5340c41f0 0x7fd5340cece0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:11.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.491+0000 7fd53afc4640 1 --2- 172.21.15.88:0/3648320758 >> v2:172.21.15.191:3300/0 conn(0x7fd5340c9bb0 0x7fd5340c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:11.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.491+0000 7fd53afc4640 1 -- 172.21.15.88:0/3648320758 >> 172.21.15.88:0/3648320758 conn(0x7fd5340715c0 msgr2=0x7fd5340719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:11.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.491+0000 7fd53afc4640 1 -- 172.21.15.88:0/3648320758 shutdown_connections 2023-10-16T20:46:11.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.491+0000 7fd53afc4640 1 -- 172.21.15.88:0/3648320758 wait complete. 2023-10-16T20:46:11.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd53afc4640 1 Processor -- start 2023-10-16T20:46:11.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd53afc4640 1 -- start start 2023-10-16T20:46:11.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd53afc4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5340c41f0 0x7fd5340c6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:11.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd53afc4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5340c9bb0 0x7fd5340c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:11.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd53afc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd534153770 0x7fd5340c8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:11.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd53afc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5340c8d50 con 0x7fd534153770 2023-10-16T20:46:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd53afc4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5340c7b90 con 0x7fd5340c41f0 2023-10-16T20:46:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd53afc4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5341f2610 con 0x7fd5340c9bb0 2023-10-16T20:46:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd539561640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd534153770 0x7fd5340c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd539561640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd534153770 0x7fd5340c8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58986/0 (socket says 172.21.15.88:58986) 2023-10-16T20:46:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd539561640 1 -- 172.21.15.88:0/3237888455 learned_addr learned my addr 172.21.15.88:0/3237888455 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd538d60640 1 --2- 172.21.15.88:0/3237888455 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5340c41f0 0x7fd5340c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd533fff640 1 --2- 172.21.15.88:0/3237888455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5340c9bb0 0x7fd5340c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd539561640 1 -- 172.21.15.88:0/3237888455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5340c9bb0 msgr2=0x7fd5340c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd539561640 1 --2- 172.21.15.88:0/3237888455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5340c9bb0 0x7fd5340c8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd539561640 1 -- 172.21.15.88:0/3237888455 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5340c41f0 msgr2=0x7fd5340c6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:11.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd539561640 1 --2- 172.21.15.88:0/3237888455 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5340c41f0 0x7fd5340c6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:11.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd539561640 1 -- 172.21.15.88:0/3237888455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd520009420 con 0x7fd534153770 2023-10-16T20:46:11.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd539561640 1 --2- 172.21.15.88:0/3237888455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd534153770 0x7fd5340c8560 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd52c019520 tx=0x7fd52c0196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:11.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd5317fa640 1 -- 172.21.15.88:0/3237888455 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c0ad850 con 0x7fd534153770 2023-10-16T20:46:11.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd5317fa640 1 -- 172.21.15.88:0/3237888455 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c0ad9b0 con 0x7fd534153770 2023-10-16T20:46:11.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd53afc4640 1 -- 172.21.15.88:0/3237888455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd52c009a70 con 0x7fd534153770 2023-10-16T20:46:11.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd5317fa640 1 -- 172.21.15.88:0/3237888455 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c0add60 con 0x7fd534153770 2023-10-16T20:46:11.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd53afc4640 1 -- 172.21.15.88:0/3237888455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53415d6a0 con 0x7fd534153770 2023-10-16T20:46:11.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.495+0000 7fd53afc4640 1 -- 172.21.15.88:0/3237888455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd534074700 con 0x7fd534153770 2023-10-16T20:46:11.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.499+0000 7fd5317fa640 1 -- 172.21.15.88:0/3237888455 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd52c0b4050 con 0x7fd534153770 2023-10-16T20:46:11.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.499+0000 7fd5317fa640 1 --2- 172.21.15.88:0/3237888455 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd5180664a0 0x7fd518068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:11.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.499+0000 7fd5317fa640 1 -- 172.21.15.88:0/3237888455 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7333+0+0 (secure 0 0 0) 0x7fd52c11f140 con 0x7fd534153770 2023-10-16T20:46:11.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.499+0000 7fd538d60640 1 --2- 172.21.15.88:0/3237888455 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd5180664a0 0x7fd518068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:11.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.499+0000 7fd538d60640 1 --2- 172.21.15.88:0/3237888455 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd5180664a0 0x7fd518068960 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd5280058e0 tx=0x7fd528007e70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:11.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.503+0000 7fd5317fa640 1 -- 172.21.15.88:0/3237888455 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd52c0ebee0 con 0x7fd534153770 2023-10-16T20:46:11.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.687+0000 7fd53afc4640 1 -- 172.21.15.88:0/3237888455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5340c7cd0 con 0x7fd534153770 2023-10-16T20:46:11.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.687+0000 7fd5317fa640 1 -- 172.21.15.88:0/3237888455 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+6581 (secure 0 0 0) 0x7fd52c0ef500 con 0x7fd534153770 2023-10-16T20:46:11.692 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:11.692 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:45:46.511774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:11.692 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 51 2023-10-16T20:46:11.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.691+0000 7fd53afc4640 1 -- 172.21.15.88:0/3237888455 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd5180664a0 msgr2=0x7fd518068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:11.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.691+0000 7fd53afc4640 1 --2- 172.21.15.88:0/3237888455 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd5180664a0 0x7fd518068960 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd5280058e0 tx=0x7fd528007e70 comp rx=0 tx=0).stop 2023-10-16T20:46:11.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.691+0000 7fd53afc4640 1 -- 172.21.15.88:0/3237888455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd534153770 msgr2=0x7fd5340c8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:11.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.691+0000 7fd53afc4640 1 --2- 172.21.15.88:0/3237888455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd534153770 0x7fd5340c8560 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd52c019520 tx=0x7fd52c0196b0 comp rx=0 tx=0).stop 2023-10-16T20:46:11.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.691+0000 7fd53afc4640 1 -- 172.21.15.88:0/3237888455 shutdown_connections 2023-10-16T20:46:11.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.691+0000 7fd53afc4640 1 --2- 172.21.15.88:0/3237888455 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd5180664a0 0x7fd518068960 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:11.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.691+0000 7fd53afc4640 1 --2- 172.21.15.88:0/3237888455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd534153770 0x7fd5340c8560 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:11.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.691+0000 7fd53afc4640 1 --2- 172.21.15.88:0/3237888455 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5340c9bb0 0x7fd5340c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:11.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.691+0000 7fd53afc4640 1 --2- 172.21.15.88:0/3237888455 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5340c41f0 0x7fd5340c6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:11.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.691+0000 7fd53afc4640 1 -- 172.21.15.88:0/3237888455 >> 172.21.15.88:0/3237888455 conn(0x7fd5340715c0 msgr2=0x7fd5340cdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:11.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.691+0000 7fd53afc4640 1 -- 172.21.15.88:0/3237888455 shutdown_connections 2023-10-16T20:46:11.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:11.691+0000 7fd53afc4640 1 -- 172.21.15.88:0/3237888455 wait complete. 2023-10-16T20:46:11.713 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-16T20:46:13.714 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:13.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.859+0000 7f10f2679640 1 Processor -- start 2023-10-16T20:46:13.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.859+0000 7f10f2679640 1 -- start start 2023-10-16T20:46:13.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.863+0000 7f10f2679640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f10ec0c5300 0x7f10ec13e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:13.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.863+0000 7f10f2679640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10ec0cb890 0x7f10ec06ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:13.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.863+0000 7f10f2679640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ec15a590 con 0x7f10ec15d840 2023-10-16T20:46:13.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.863+0000 7f10f2679640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10ec15d420 con 0x7f10ec0c5300 2023-10-16T20:46:13.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.863+0000 7f10f2679640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec0c7830 con 0x7f10ec0cb890 2023-10-16T20:46:13.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.863+0000 7f10eb7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f10ec15d840 0x7f10ec159640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41674/0 (socket says 172.21.15.88:41674) 2023-10-16T20:46:13.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.863+0000 7f10eb7fe640 1 -- 172.21.15.88:0/2627718150 learned_addr learned my addr 172.21.15.88:0/2627718150 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:46:13.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.863+0000 7f10ebfff640 1 --2- 172.21.15.88:0/2627718150 >> v2:172.21.15.191:3300/0 conn(0x7f10ec0c5300 0x7f10ec13e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:13.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.863+0000 7f10f0c16640 1 --2- 172.21.15.88:0/2627718150 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10ec0cb890 0x7f10ec06ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:13.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.863+0000 7f10eaffd640 1 -- 172.21.15.88:0/2627718150 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339358448 0 0) 0x7f10ec15a590 con 0x7f10ec15d840 2023-10-16T20:46:13.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.863+0000 7f10eaffd640 1 -- 172.21.15.88:0/2627718150 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10d4003530 con 0x7f10ec15d840 2023-10-16T20:46:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.863+0000 7f10ebfff640 1 -- 172.21.15.88:0/2627718150 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10ec0cb890 msgr2=0x7f10ec06ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.863+0000 7f10ebfff640 1 --2- 172.21.15.88:0/2627718150 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10ec0cb890 0x7f10ec06ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.863+0000 7f10ebfff640 1 -- 172.21.15.88:0/2627718150 >> v1:172.21.15.88:6789/0 conn(0x7f10ec15d840 legacy=0x7f10ec159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.863+0000 7f10ebfff640 1 -- 172.21.15.88:0/2627718150 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ec159d50 con 0x7f10ec0c5300 2023-10-16T20:46:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.863+0000 7f10f0c16640 1 --2- 172.21.15.88:0/2627718150 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10ec0cb890 0x7f10ec06ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:46:13.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.867+0000 7f10ebfff640 1 --2- 172.21.15.88:0/2627718150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10ec0c5300 0x7f10ec13e810 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f10dc0097c0 tx=0x7f10dc01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbceafdb8dbffc60 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:13.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.867+0000 7f10eaffd640 1 -- 172.21.15.88:0/2627718150 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc0ad640 con 0x7f10ec0c5300 2023-10-16T20:46:13.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.867+0000 7f10eaffd640 1 -- 172.21.15.88:0/2627718150 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10dc0ad7a0 con 0x7f10ec0c5300 2023-10-16T20:46:13.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.867+0000 7f10eaffd640 1 -- 172.21.15.88:0/2627718150 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc0adb30 con 0x7f10ec0c5300 2023-10-16T20:46:13.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.867+0000 7f10f2679640 1 -- 172.21.15.88:0/2627718150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10ec0c5300 msgr2=0x7f10ec13e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:13.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.867+0000 7f10f2679640 1 --2- 172.21.15.88:0/2627718150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10ec0c5300 0x7f10ec13e810 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f10dc0097c0 tx=0x7f10dc01c480 comp rx=0 tx=0).stop 2023-10-16T20:46:13.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.867+0000 7f10f2679640 1 -- 172.21.15.88:0/2627718150 shutdown_connections 2023-10-16T20:46:13.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.867+0000 7f10f2679640 1 --2- 172.21.15.88:0/2627718150 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10ec0cb890 0x7f10ec06ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:13.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.867+0000 7f10f2679640 1 --2- 172.21.15.88:0/2627718150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10ec0c5300 0x7f10ec13e810 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:13.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.867+0000 7f10f2679640 1 -- 172.21.15.88:0/2627718150 >> 172.21.15.88:0/2627718150 conn(0x7f10ec06e0f0 msgr2=0x7f10ec0cb180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:13.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.867+0000 7f10f2679640 1 -- 172.21.15.88:0/2627718150 shutdown_connections 2023-10-16T20:46:13.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.867+0000 7f10f2679640 1 -- 172.21.15.88:0/2627718150 wait complete. 2023-10-16T20:46:13.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.867+0000 7f10f2679640 1 Processor -- start 2023-10-16T20:46:13.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.867+0000 7f10f2679640 1 -- start start 2023-10-16T20:46:13.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10f2679640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10ec0c5300 0x7f10ec1f4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:13.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10f2679640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10ec0cb890 0x7f10ec0c2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:13.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10f2679640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10ec15d840 0x7f10ec135da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:13.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10f2679640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec1f57d0 con 0x7f10ec0c5300 2023-10-16T20:46:13.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10f2679640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec1f5510 con 0x7f10ec0cb890 2023-10-16T20:46:13.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10f2679640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec13c310 con 0x7f10ec15d840 2023-10-16T20:46:13.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10ebfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10ec0c5300 0x7f10ec1f4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:13.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10ebfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10ec0c5300 0x7f10ec1f4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59002/0 (socket says 172.21.15.88:59002) 2023-10-16T20:46:13.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10ebfff640 1 -- 172.21.15.88:0/3988469855 learned_addr learned my addr 172.21.15.88:0/3988469855 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:13.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10eb7fe640 1 --2- 172.21.15.88:0/3988469855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10ec0cb890 0x7f10ec0c2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:13.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10f0c16640 1 --2- 172.21.15.88:0/3988469855 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10ec15d840 0x7f10ec135da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:13.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10ebfff640 1 -- 172.21.15.88:0/3988469855 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10ec15d840 msgr2=0x7f10ec135da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:13.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10ebfff640 1 --2- 172.21.15.88:0/3988469855 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10ec15d840 0x7f10ec135da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:13.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10ebfff640 1 -- 172.21.15.88:0/3988469855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10ec0cb890 msgr2=0x7f10ec0c2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:13.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10ebfff640 1 --2- 172.21.15.88:0/3988469855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10ec0cb890 0x7f10ec0c2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:13.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10ebfff640 1 -- 172.21.15.88:0/3988469855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e4009a70 con 0x7f10ec0c5300 2023-10-16T20:46:13.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10f0c16640 1 --2- 172.21.15.88:0/3988469855 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10ec15d840 0x7f10ec135da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:46:13.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10eb7fe640 1 --2- 172.21.15.88:0/3988469855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10ec0cb890 0x7f10ec0c2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:46:13.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10ebfff640 1 --2- 172.21.15.88:0/3988469855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10ec0c5300 0x7f10ec1f4c40 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f10dc0b0040 tx=0x7f10dc019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:13.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10e8ff9640 1 -- 172.21.15.88:0/3988469855 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc0b6820 con 0x7f10ec0c5300 2023-10-16T20:46:13.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10e8ff9640 1 -- 172.21.15.88:0/3988469855 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10dc0a6040 con 0x7f10ec0c5300 2023-10-16T20:46:13.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10f2679640 1 -- 172.21.15.88:0/3988469855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10dc009420 con 0x7f10ec0c5300 2023-10-16T20:46:13.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10e8ff9640 1 -- 172.21.15.88:0/3988469855 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc0b6bc0 con 0x7f10ec0c5300 2023-10-16T20:46:13.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10f2679640 1 -- 172.21.15.88:0/3988469855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10ec159380 con 0x7f10ec0c5300 2023-10-16T20:46:13.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10f2679640 1 -- 172.21.15.88:0/3988469855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10ec074700 con 0x7f10ec0c5300 2023-10-16T20:46:13.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.871+0000 7f10e8ff9640 1 -- 172.21.15.88:0/3988469855 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f10dc0b4040 con 0x7f10ec0c5300 2023-10-16T20:46:13.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.875+0000 7f10e8ff9640 1 --2- 172.21.15.88:0/3988469855 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f10d80664a0 0x7f10d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:13.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.875+0000 7f10e8ff9640 1 -- 172.21.15.88:0/3988469855 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7333+0+0 (secure 0 0 0) 0x7f10dc11eb20 con 0x7f10ec0c5300 2023-10-16T20:46:13.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.875+0000 7f10eb7fe640 1 --2- 172.21.15.88:0/3988469855 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f10d80664a0 0x7f10d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:13.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.875+0000 7f10eb7fe640 1 --2- 172.21.15.88:0/3988469855 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f10d80664a0 0x7f10d8068960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f10ec1573c0 tx=0x7f10e0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:13.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:13.875+0000 7f10e8ff9640 1 -- 172.21.15.88:0/3988469855 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f10dc0eb930 con 0x7f10ec0c5300 2023-10-16T20:46:14.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:14.059+0000 7f10f2679640 1 -- 172.21.15.88:0/3988469855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10ec0c4c60 con 0x7f10ec0c5300 2023-10-16T20:46:14.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:14.059+0000 7f10e8ff9640 1 -- 172.21.15.88:0/3988469855 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+6581 (secure 0 0 0) 0x7f10dc0eef50 con 0x7f10ec0c5300 2023-10-16T20:46:14.062 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:14.063 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5535,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:45:46.511774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":4275,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":41,"addr":"172.21.15.88:6838/321173984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":321173984},{"type":"v1","addr":"172.21.15.88:6838","nonce":321173984}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 51 2023-10-16T20:46:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:14.059+0000 7f10f2679640 1 -- 172.21.15.88:0/3988469855 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f10d80664a0 msgr2=0x7f10d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:14.059+0000 7f10f2679640 1 --2- 172.21.15.88:0/3988469855 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f10d80664a0 0x7f10d8068960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f10ec1573c0 tx=0x7f10e0009040 comp rx=0 tx=0).stop 2023-10-16T20:46:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:14.059+0000 7f10f2679640 1 -- 172.21.15.88:0/3988469855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10ec0c5300 msgr2=0x7f10ec1f4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:14.059+0000 7f10f2679640 1 --2- 172.21.15.88:0/3988469855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10ec0c5300 0x7f10ec1f4c40 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f10dc0b0040 tx=0x7f10dc019220 comp rx=0 tx=0).stop 2023-10-16T20:46:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:14.059+0000 7f10f2679640 1 -- 172.21.15.88:0/3988469855 shutdown_connections 2023-10-16T20:46:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:14.059+0000 7f10f2679640 1 --2- 172.21.15.88:0/3988469855 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f10d80664a0 0x7f10d8068960 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:14.059+0000 7f10f2679640 1 --2- 172.21.15.88:0/3988469855 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f10ec15d840 0x7f10ec135da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:14.059+0000 7f10f2679640 1 --2- 172.21.15.88:0/3988469855 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f10ec0cb890 0x7f10ec0c2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:14.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:14.059+0000 7f10f2679640 1 --2- 172.21.15.88:0/3988469855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10ec0c5300 0x7f10ec1f4c40 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:14.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:14.059+0000 7f10f2679640 1 -- 172.21.15.88:0/3988469855 >> 172.21.15.88:0/3988469855 conn(0x7f10ec06e0f0 msgr2=0x7f10ec15c210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:14.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:14.059+0000 7f10f2679640 1 -- 172.21.15.88:0/3988469855 shutdown_connections 2023-10-16T20:46:14.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:14.059+0000 7f10f2679640 1 -- 172.21.15.88:0/3988469855 wait complete. 2023-10-16T20:46:14.083 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-16T20:46:16.085 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:16.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.231+0000 7f717d009640 1 Processor -- start 2023-10-16T20:46:16.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.231+0000 7f717d009640 1 -- start start 2023-10-16T20:46:16.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.231+0000 7f717d009640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71780c9bb0 0x7f71780c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:16.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.231+0000 7f717d009640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f71780c41f0 0x7f71780cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:16.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.231+0000 7f717d009640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7178153b50 con 0x7f7178153770 2023-10-16T20:46:16.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.231+0000 7f717d009640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71780cc250 con 0x7f71780c41f0 2023-10-16T20:46:16.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.231+0000 7f717d009640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71780cc670 con 0x7f71780c9bb0 2023-10-16T20:46:16.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.231+0000 7f717759e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f71780c41f0 0x7f71780cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:16.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.231+0000 7f7176d9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7178153770 0x7f7178159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41676/0 (socket says 172.21.15.88:41676) 2023-10-16T20:46:16.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.231+0000 7f7176d9d640 1 -- 172.21.15.88:0/1311507903 learned_addr learned my addr 172.21.15.88:0/1311507903 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:46:16.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.231+0000 7f716e59c640 1 --2- 172.21.15.88:0/1311507903 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71780c9bb0 0x7f71780c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:16.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.231+0000 7f717759e640 1 -- 172.21.15.88:0/1311507903 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71780c9bb0 msgr2=0x7f71780c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:16.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.231+0000 7f717759e640 1 --2- 172.21.15.88:0/1311507903 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71780c9bb0 0x7f71780c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.231+0000 7f717759e640 1 -- 172.21.15.88:0/1311507903 >> v1:172.21.15.88:6789/0 conn(0x7f7178153770 legacy=0x7f7178159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:16.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.231+0000 7f717759e640 1 -- 172.21.15.88:0/1311507903 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71780cb2f0 con 0x7f71780c41f0 2023-10-16T20:46:16.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.235+0000 7f717759e640 1 --2- 172.21.15.88:0/1311507903 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71780c41f0 0x7f71780cece0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7170009d90 tx=0x7f717001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4343f58a384be37c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:16.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.235+0000 7f717659c640 1 -- 172.21.15.88:0/1311507903 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71700a5920 con 0x7f71780c41f0 2023-10-16T20:46:16.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.235+0000 7f717659c640 1 -- 172.21.15.88:0/1311507903 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71700a5a80 con 0x7f71780c41f0 2023-10-16T20:46:16.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.235+0000 7f717659c640 1 -- 172.21.15.88:0/1311507903 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71700a5e10 con 0x7f71780c41f0 2023-10-16T20:46:16.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.235+0000 7f717d009640 1 -- 172.21.15.88:0/1311507903 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71780c41f0 msgr2=0x7f71780cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:16.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.235+0000 7f717d009640 1 --2- 172.21.15.88:0/1311507903 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71780c41f0 0x7f71780cece0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7170009d90 tx=0x7f717001c950 comp rx=0 tx=0).stop 2023-10-16T20:46:16.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.235+0000 7f717d009640 1 -- 172.21.15.88:0/1311507903 shutdown_connections 2023-10-16T20:46:16.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.235+0000 7f717d009640 1 --2- 172.21.15.88:0/1311507903 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71780c41f0 0x7f71780cece0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.235+0000 7f717d009640 1 --2- 172.21.15.88:0/1311507903 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f71780c9bb0 0x7f71780c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.235+0000 7f717d009640 1 -- 172.21.15.88:0/1311507903 >> 172.21.15.88:0/1311507903 conn(0x7f71780715c0 msgr2=0x7f71780719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:16.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.235+0000 7f717d009640 1 -- 172.21.15.88:0/1311507903 shutdown_connections 2023-10-16T20:46:16.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.235+0000 7f717d009640 1 -- 172.21.15.88:0/1311507903 wait complete. 2023-10-16T20:46:16.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f717d009640 1 Processor -- start 2023-10-16T20:46:16.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f717d009640 1 -- start start 2023-10-16T20:46:16.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f717d009640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71780c41f0 0x7f717813ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:16.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f717d009640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71780c9bb0 0x7f71780b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:16.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f717d009640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7178153770 0x7f717813fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:16.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f717d009640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71780b4100 con 0x7f71780c9bb0 2023-10-16T20:46:16.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f717d009640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71780b4c10 con 0x7f71780c41f0 2023-10-16T20:46:16.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f717d009640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71780b4e30 con 0x7f7178153770 2023-10-16T20:46:16.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f716e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71780c9bb0 0x7f71780b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:16.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f716e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71780c9bb0 0x7f71780b46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59032/0 (socket says 172.21.15.88:59032) 2023-10-16T20:46:16.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f716e59c640 1 -- 172.21.15.88:0/3977230050 learned_addr learned my addr 172.21.15.88:0/3977230050 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:16.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f7176d9d640 1 --2- 172.21.15.88:0/3977230050 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71780c41f0 0x7f717813ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f717759e640 1 --2- 172.21.15.88:0/3977230050 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7178153770 0x7f717813fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f716e59c640 1 -- 172.21.15.88:0/3977230050 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7178153770 msgr2=0x7f717813fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f716e59c640 1 --2- 172.21.15.88:0/3977230050 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7178153770 0x7f717813fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f716e59c640 1 -- 172.21.15.88:0/3977230050 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71780c41f0 msgr2=0x7f717813ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f716e59c640 1 --2- 172.21.15.88:0/3977230050 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71780c41f0 0x7f717813ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f716e59c640 1 -- 172.21.15.88:0/3977230050 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7170009a70 con 0x7f71780c9bb0 2023-10-16T20:46:16.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f716e59c640 1 --2- 172.21.15.88:0/3977230050 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71780c9bb0 0x7f71780b46d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f716000d830 tx=0x7f716000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:16.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f716ffff640 1 -- 172.21.15.88:0/3977230050 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716000b6c0 con 0x7f71780c9bb0 2023-10-16T20:46:16.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f716ffff640 1 -- 172.21.15.88:0/3977230050 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f716000b820 con 0x7f71780c9bb0 2023-10-16T20:46:16.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f717d009640 1 -- 172.21.15.88:0/3977230050 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71780cdfa0 con 0x7f71780c9bb0 2023-10-16T20:46:16.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f716ffff640 1 -- 172.21.15.88:0/3977230050 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716000bb90 con 0x7f71780c9bb0 2023-10-16T20:46:16.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f717d009640 1 -- 172.21.15.88:0/3977230050 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71780ce560 con 0x7f71780c9bb0 2023-10-16T20:46:16.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.239+0000 7f717d009640 1 -- 172.21.15.88:0/3977230050 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7178074700 con 0x7f71780c9bb0 2023-10-16T20:46:16.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.243+0000 7f716ffff640 1 -- 172.21.15.88:0/3977230050 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f716001e460 con 0x7f71780c9bb0 2023-10-16T20:46:16.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.243+0000 7f716ffff640 1 --2- 172.21.15.88:0/3977230050 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f71580664d0 0x7f7158068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:16.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.243+0000 7f716ffff640 1 -- 172.21.15.88:0/3977230050 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7007+0+0 (secure 0 0 0) 0x7f7160084c20 con 0x7f71780c9bb0 2023-10-16T20:46:16.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.243+0000 7f7176d9d640 1 --2- 172.21.15.88:0/3977230050 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f71580664d0 0x7f7158068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:16.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.243+0000 7f7176d9d640 1 --2- 172.21.15.88:0/3977230050 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f71580664d0 0x7f7158068990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f71680010f0 tx=0x7f7168009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:16.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.247+0000 7f716ffff640 1 -- 172.21.15.88:0/3977230050 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7160051af0 con 0x7f71780c9bb0 2023-10-16T20:46:16.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.427+0000 7f717d009640 1 -- 172.21.15.88:0/3977230050 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71780ca740 con 0x7f71780c9bb0 2023-10-16T20:46:16.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.427+0000 7f716ffff640 1 -- 172.21.15.88:0/3977230050 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+5790 (secure 0 0 0) 0x7f7160055110 con 0x7f71780c9bb0 2023-10-16T20:46:16.431 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:16.431 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:46:15.444685+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:16.432 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 52 2023-10-16T20:46:16.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.431+0000 7f717d009640 1 -- 172.21.15.88:0/3977230050 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f71580664d0 msgr2=0x7f7158068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:16.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.431+0000 7f717d009640 1 --2- 172.21.15.88:0/3977230050 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f71580664d0 0x7f7158068990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f71680010f0 tx=0x7f7168009040 comp rx=0 tx=0).stop 2023-10-16T20:46:16.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.431+0000 7f717d009640 1 -- 172.21.15.88:0/3977230050 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71780c9bb0 msgr2=0x7f71780b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:16.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.431+0000 7f717d009640 1 --2- 172.21.15.88:0/3977230050 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71780c9bb0 0x7f71780b46d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f716000d830 tx=0x7f716000dcf0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.431+0000 7f717d009640 1 -- 172.21.15.88:0/3977230050 shutdown_connections 2023-10-16T20:46:16.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.431+0000 7f717d009640 1 --2- 172.21.15.88:0/3977230050 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f71580664d0 0x7f7158068990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.431+0000 7f717d009640 1 --2- 172.21.15.88:0/3977230050 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7178153770 0x7f717813fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.431+0000 7f717d009640 1 --2- 172.21.15.88:0/3977230050 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71780c9bb0 0x7f71780b46d0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.431+0000 7f717d009640 1 --2- 172.21.15.88:0/3977230050 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71780c41f0 0x7f717813ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.431+0000 7f717d009640 1 -- 172.21.15.88:0/3977230050 >> 172.21.15.88:0/3977230050 conn(0x7f71780715c0 msgr2=0x7f71780719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:16.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.431+0000 7f717d009640 1 -- 172.21.15.88:0/3977230050 shutdown_connections 2023-10-16T20:46:16.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.431+0000 7f717d009640 1 -- 172.21.15.88:0/3977230050 wait complete. 2023-10-16T20:46:16.452 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-16T20:46:16.452 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-16T20:46:16.452 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:16.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.587+0000 7f4d6309c640 1 Processor -- start 2023-10-16T20:46:16.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.587+0000 7f4d6309c640 1 -- start start 2023-10-16T20:46:16.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d6309c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4d5c150280 0x7f4d5c0c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:16.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d6309c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0c6bc0 0x7f4d5c0c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:16.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d6309c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d5c0c5c60 con 0x7f4d5c150660 2023-10-16T20:46:16.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d6309c640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d5c0c5370 con 0x7f4d5c150280 2023-10-16T20:46:16.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d6309c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c156d10 con 0x7f4d5c0c6bc0 2023-10-16T20:46:16.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d53fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4d5c150660 0x7f4d5c06d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41682/0 (socket says 172.21.15.88:41682) 2023-10-16T20:46:16.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d53fff640 1 -- 172.21.15.88:0/1674375860 learned_addr learned my addr 172.21.15.88:0/1674375860 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:46:16.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d60e38640 1 --2- 172.21.15.88:0/1674375860 >> v2:172.21.15.191:3300/0 conn(0x7f4d5c150280 0x7f4d5c0c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:16.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d61639640 1 --2- 172.21.15.88:0/1674375860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0c6bc0 0x7f4d5c0c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:16.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d537fe640 1 -- 172.21.15.88:0/1674375860 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314523398 0 0) 0x7f4d5c0c5c60 con 0x7f4d5c150660 2023-10-16T20:46:16.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d537fe640 1 -- 172.21.15.88:0/1674375860 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d44003530 con 0x7f4d5c150660 2023-10-16T20:46:16.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d537fe640 1 -- 172.21.15.88:0/1674375860 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1149558664 0 0) 0x7f4d44003530 con 0x7f4d5c150660 2023-10-16T20:46:16.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d537fe640 1 -- 172.21.15.88:0/1674375860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0c6bc0 msgr2=0x7f4d5c0c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:16.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d537fe640 1 --2- 172.21.15.88:0/1674375860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0c6bc0 0x7f4d5c0c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d537fe640 1 -- 172.21.15.88:0/1674375860 >> v2:172.21.15.191:3300/0 conn(0x7f4d5c150280 msgr2=0x7f4d5c0c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:16.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d537fe640 1 --2- 172.21.15.88:0/1674375860 >> v2:172.21.15.191:3300/0 conn(0x7f4d5c150280 0x7f4d5c0c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d537fe640 1 -- 172.21.15.88:0/1674375860 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5c0c5610 con 0x7f4d5c150660 2023-10-16T20:46:16.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d537fe640 1 -- 172.21.15.88:0/1674375860 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f4d4c001680 con 0x7f4d5c150660 2023-10-16T20:46:16.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d537fe640 1 -- 172.21.15.88:0/1674375860 >> v1:172.21.15.88:6789/0 conn(0x7f4d5c150660 legacy=0x7f4d5c06d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:16.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d537fe640 1 --2- 172.21.15.88:0/1674375860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d44003980 0x7f4d44003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:16.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d537fe640 1 --2- 172.21.15.88:0/1674375860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0c6bc0 0x7f4d440047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:16.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d537fe640 1 --2- 172.21.15.88:0/1674375860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d44004d10 0x7f4d44005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:16.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d537fe640 1 -- 172.21.15.88:0/1674375860 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c0c5370 con 0x7f4d44004d10 2023-10-16T20:46:16.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d537fe640 1 -- 172.21.15.88:0/1674375860 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c156d10 con 0x7f4d44003980 2023-10-16T20:46:16.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d537fe640 1 -- 172.21.15.88:0/1674375860 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4400b990 con 0x7f4d5c0c6bc0 2023-10-16T20:46:16.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d61639640 1 --2- 172.21.15.88:0/1674375860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d44004d10 0x7f4d44005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:16.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d60e38640 1 --2- 172.21.15.88:0/1674375860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d44003980 0x7f4d44003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:16.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d53fff640 1 --2- 172.21.15.88:0/1674375860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0c6bc0 0x7f4d440047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:16.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d61639640 1 -- 172.21.15.88:0/1674375860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0c6bc0 msgr2=0x7f4d440047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:16.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d61639640 1 --2- 172.21.15.88:0/1674375860 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c0c6bc0 0x7f4d440047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d61639640 1 -- 172.21.15.88:0/1674375860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d44003980 msgr2=0x7f4d44003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:16.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d61639640 1 --2- 172.21.15.88:0/1674375860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d44003980 0x7f4d44003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.591+0000 7f4d61639640 1 -- 172.21.15.88:0/1674375860 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d44003530 con 0x7f4d44004d10 2023-10-16T20:46:16.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.595+0000 7f4d61639640 1 --2- 172.21.15.88:0/1674375860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d44004d10 0x7f4d44005190 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4d54009fc0 tx=0x7f4d5401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92556458ff0aee5f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:16.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.595+0000 7f4d537fe640 1 -- 172.21.15.88:0/1674375860 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d540a5c70 con 0x7f4d44004d10 2023-10-16T20:46:16.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.595+0000 7f4d537fe640 1 -- 172.21.15.88:0/1674375860 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5400f040 con 0x7f4d44004d10 2023-10-16T20:46:16.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.595+0000 7f4d537fe640 1 -- 172.21.15.88:0/1674375860 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d540ad440 con 0x7f4d44004d10 2023-10-16T20:46:16.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.595+0000 7f4d6309c640 1 -- 172.21.15.88:0/1674375860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d44004d10 msgr2=0x7f4d44005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:16.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.595+0000 7f4d6309c640 1 --2- 172.21.15.88:0/1674375860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d44004d10 0x7f4d44005190 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4d54009fc0 tx=0x7f4d5401cc60 comp rx=0 tx=0).stop 2023-10-16T20:46:16.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.595+0000 7f4d61639640 1 -- 172.21.15.88:0/1674375860 reap_dead start 2023-10-16T20:46:16.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.595+0000 7f4d6309c640 1 -- 172.21.15.88:0/1674375860 shutdown_connections 2023-10-16T20:46:16.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.595+0000 7f4d6309c640 1 -- 172.21.15.88:0/1674375860 >> 172.21.15.88:0/1674375860 conn(0x7f4d5c0cad50 msgr2=0x7f4d5c0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:16.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.595+0000 7f4d6309c640 1 -- 172.21.15.88:0/1674375860 shutdown_connections 2023-10-16T20:46:16.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.595+0000 7f4d6309c640 1 -- 172.21.15.88:0/1674375860 wait complete. 2023-10-16T20:46:16.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d6309c640 1 Processor -- start 2023-10-16T20:46:16.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d6309c640 1 -- start start 2023-10-16T20:46:16.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d6309c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d5c0716d0 0x7f4d5c1f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:16.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d6309c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c150a80 0x7f4d5c1512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:16.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d6309c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d5c071ab0 0x7f4d5c151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:16.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d6309c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c14c700 con 0x7f4d5c0716d0 2023-10-16T20:46:16.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d6309c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c1f2310 con 0x7f4d5c071ab0 2023-10-16T20:46:16.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d6309c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c14c420 con 0x7f4d5c150a80 2023-10-16T20:46:16.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d60e38640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d5c0716d0 0x7f4d5c1f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d60e38640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d5c0716d0 0x7f4d5c1f27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59042/0 (socket says 172.21.15.88:59042) 2023-10-16T20:46:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d60e38640 1 -- 172.21.15.88:0/3467826658 learned_addr learned my addr 172.21.15.88:0/3467826658 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d53fff640 1 --2- 172.21.15.88:0/3467826658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c150a80 0x7f4d5c1512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d61639640 1 --2- 172.21.15.88:0/3467826658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d5c071ab0 0x7f4d5c151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d60e38640 1 -- 172.21.15.88:0/3467826658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c150a80 msgr2=0x7f4d5c1512f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:16.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d60e38640 1 --2- 172.21.15.88:0/3467826658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c150a80 0x7f4d5c1512f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d60e38640 1 -- 172.21.15.88:0/3467826658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d5c071ab0 msgr2=0x7f4d5c151830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:16.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d60e38640 1 --2- 172.21.15.88:0/3467826658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d5c071ab0 0x7f4d5c151830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d60e38640 1 -- 172.21.15.88:0/3467826658 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5c154910 con 0x7f4d5c0716d0 2023-10-16T20:46:16.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d60e38640 1 --2- 172.21.15.88:0/3467826658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d5c0716d0 0x7f4d5c1f27a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4d580119d0 tx=0x7f4d58011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:16.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d517fa640 1 -- 172.21.15.88:0/3467826658 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5800faf0 con 0x7f4d5c0716d0 2023-10-16T20:46:16.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d517fa640 1 -- 172.21.15.88:0/3467826658 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5800fc50 con 0x7f4d5c0716d0 2023-10-16T20:46:16.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d517fa640 1 -- 172.21.15.88:0/3467826658 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d58014440 con 0x7f4d5c0716d0 2023-10-16T20:46:16.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d6309c640 1 -- 172.21.15.88:0/3467826658 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d5c1f1fa0 con 0x7f4d5c0716d0 2023-10-16T20:46:16.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d6309c640 1 -- 172.21.15.88:0/3467826658 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d5c1f2ce0 con 0x7f4d5c0716d0 2023-10-16T20:46:16.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.599+0000 7f4d6309c640 1 -- 172.21.15.88:0/3467826658 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d5c074700 con 0x7f4d5c0716d0 2023-10-16T20:46:16.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.603+0000 7f4d517fa640 1 -- 172.21.15.88:0/3467826658 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4d580145a0 con 0x7f4d5c0716d0 2023-10-16T20:46:16.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.603+0000 7f4d517fa640 1 --2- 172.21.15.88:0/3467826658 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4d48066430 0x7f4d480688f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:16.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.603+0000 7f4d517fa640 1 -- 172.21.15.88:0/3467826658 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7007+0+0 (secure 0 0 0) 0x7f4d58084bb0 con 0x7f4d5c0716d0 2023-10-16T20:46:16.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.603+0000 7f4d53fff640 1 --2- 172.21.15.88:0/3467826658 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4d48066430 0x7f4d480688f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:16.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.603+0000 7f4d53fff640 1 --2- 172.21.15.88:0/3467826658 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4d48066430 0x7f4d480688f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c0c4140 tx=0x7f4d4c007040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:16.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.607+0000 7f4d517fa640 1 -- 172.21.15.88:0/3467826658 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4d58051b40 con 0x7f4d5c0716d0 2023-10-16T20:46:16.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.795+0000 7f4d6309c640 1 -- 172.21.15.88:0/3467826658 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d5c0c1650 con 0x7f4d5c0716d0 2023-10-16T20:46:16.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.795+0000 7f4d517fa640 1 -- 172.21.15.88:0/3467826658 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+5790 (secure 0 0 0) 0x7f4d58022090 con 0x7f4d5c0716d0 2023-10-16T20:46:16.800 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:16.801 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:46:15.444685+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:16.801 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 52 2023-10-16T20:46:16.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.799+0000 7f4d6309c640 1 -- 172.21.15.88:0/3467826658 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4d48066430 msgr2=0x7f4d480688f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:16.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.799+0000 7f4d6309c640 1 --2- 172.21.15.88:0/3467826658 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4d48066430 0x7f4d480688f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c0c4140 tx=0x7f4d4c007040 comp rx=0 tx=0).stop 2023-10-16T20:46:16.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.799+0000 7f4d6309c640 1 -- 172.21.15.88:0/3467826658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d5c0716d0 msgr2=0x7f4d5c1f27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:16.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.799+0000 7f4d6309c640 1 --2- 172.21.15.88:0/3467826658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d5c0716d0 0x7f4d5c1f27a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4d580119d0 tx=0x7f4d58011dd0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.799+0000 7f4d6309c640 1 -- 172.21.15.88:0/3467826658 shutdown_connections 2023-10-16T20:46:16.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.799+0000 7f4d6309c640 1 --2- 172.21.15.88:0/3467826658 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4d48066430 0x7f4d480688f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.799+0000 7f4d6309c640 1 --2- 172.21.15.88:0/3467826658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d5c071ab0 0x7f4d5c151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.799+0000 7f4d6309c640 1 --2- 172.21.15.88:0/3467826658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d5c150a80 0x7f4d5c1512f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.799+0000 7f4d6309c640 1 --2- 172.21.15.88:0/3467826658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d5c0716d0 0x7f4d5c1f27a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:16.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.799+0000 7f4d6309c640 1 -- 172.21.15.88:0/3467826658 >> 172.21.15.88:0/3467826658 conn(0x7f4d5c0cad50 msgr2=0x7f4d5c0c3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:16.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.799+0000 7f4d6309c640 1 -- 172.21.15.88:0/3467826658 shutdown_connections 2023-10-16T20:46:16.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:16.799+0000 7f4d6309c640 1 -- 172.21.15.88:0/3467826658 wait complete. 2023-10-16T20:46:16.822 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 52, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 52, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4279": { "addr": "172.21.15.191:6833/2446850309", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 2446850309, "type": "v2" }, { "addr": "172.21.15.191:6833", "nonce": 2446850309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 40 }, "gid_4288": { "addr": "172.21.15.191:6835/3059699574", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3059699574, "type": "v2" }, { "addr": "172.21.15.191:6835", "nonce": 3059699574, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4889": { "addr": "172.21.15.191:6837/2468647900", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6836", "nonce": 2468647900, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2468647900, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4889, "incarnation": 30, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 21 }, "gid_5284": { "addr": "172.21.15.88:6839/2172973775", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6837", "nonce": 2172973775, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2172973775, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5284, "incarnation": 38, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 17 }, "gid_5535": { "addr": "172.21.15.88:6835/3476086754", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3476086754, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3476086754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5535, "incarnation": 52, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 50, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:46:15.444685+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 4279, "mds_2": 5535, "mds_3": 5284, "mds_4": 4889 } } } ], "standbys": [] } 2023-10-16T20:46:18.822 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:18.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.967+0000 7f462d79d640 1 Processor -- start 2023-10-16T20:46:18.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.967+0000 7f462d79d640 1 -- start start 2023-10-16T20:46:18.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.967+0000 7f462d79d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46280c58d0 0x7f462806a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:18.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.967+0000 7f462d79d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46280c5e80 con 0x7f462806f810 2023-10-16T20:46:18.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.967+0000 7f462d79d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f462815cb30 con 0x7f4628153670 2023-10-16T20:46:18.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.967+0000 7f462d79d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46280c6860 con 0x7f46280c58d0 2023-10-16T20:46:18.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.967+0000 7f46277fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f462806f810 0x7f462806fc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33078/0 (socket says 172.21.15.88:33078) 2023-10-16T20:46:18.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.967+0000 7f46277fe640 1 -- 172.21.15.88:0/3091756610 learned_addr learned my addr 172.21.15.88:0/3091756610 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:46:18.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.967+0000 7f461ffff640 1 --2- 172.21.15.88:0/3091756610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46280c58d0 0x7f462806a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:18.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.971+0000 7f46267fc640 1 -- 172.21.15.88:0/3091756610 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607538118 0 0) 0x7f46200014a0 con 0x7f462806f810 2023-10-16T20:46:18.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.971+0000 7f46267fc640 1 -- 172.21.15.88:0/3091756610 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4614003530 con 0x7f462806f810 2023-10-16T20:46:18.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.971+0000 7f46267fc640 1 -- 172.21.15.88:0/3091756610 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986210926 0 0) 0x7f462815cb30 con 0x7f4628153670 2023-10-16T20:46:18.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.971+0000 7f46267fc640 1 -- 172.21.15.88:0/3091756610 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46200014a0 con 0x7f4628153670 2023-10-16T20:46:18.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.971+0000 7f461ffff640 1 -- 172.21.15.88:0/3091756610 >> v1:172.21.15.191:6789/0 conn(0x7f4628153670 legacy=0x7f462806d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:18.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.971+0000 7f461ffff640 1 -- 172.21.15.88:0/3091756610 >> v1:172.21.15.88:6789/0 conn(0x7f462806f810 legacy=0x7f462806fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:18.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.971+0000 7f461ffff640 1 -- 172.21.15.88:0/3091756610 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f462815ce10 con 0x7f46280c58d0 2023-10-16T20:46:18.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.971+0000 7f461ffff640 1 --2- 172.21.15.88:0/3091756610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46280c58d0 0x7f462806a8a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f46100097f0 tx=0x7f461001c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdbc2efea475e91e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:18.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.971+0000 7f46267fc640 1 -- 172.21.15.88:0/3091756610 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46100ab600 con 0x7f46280c58d0 2023-10-16T20:46:18.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.971+0000 7f46267fc640 1 -- 172.21.15.88:0/3091756610 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46100a7050 con 0x7f46280c58d0 2023-10-16T20:46:18.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.971+0000 7f46267fc640 1 -- 172.21.15.88:0/3091756610 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46100ab980 con 0x7f46280c58d0 2023-10-16T20:46:18.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f462d79d640 1 -- 172.21.15.88:0/3091756610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46280c58d0 msgr2=0x7f462806a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:18.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f462d79d640 1 --2- 172.21.15.88:0/3091756610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46280c58d0 0x7f462806a8a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f46100097f0 tx=0x7f461001c4c0 comp rx=0 tx=0).stop 2023-10-16T20:46:18.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f462d79d640 1 -- 172.21.15.88:0/3091756610 shutdown_connections 2023-10-16T20:46:18.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f462d79d640 1 --2- 172.21.15.88:0/3091756610 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46280c58d0 0x7f462806a8a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:18.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f462d79d640 1 -- 172.21.15.88:0/3091756610 >> 172.21.15.88:0/3091756610 conn(0x7f46280cad50 msgr2=0x7f46280cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:18.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f462d79d640 1 -- 172.21.15.88:0/3091756610 shutdown_connections 2023-10-16T20:46:18.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f462d79d640 1 -- 172.21.15.88:0/3091756610 wait complete. 2023-10-16T20:46:18.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f462d79d640 1 Processor -- start 2023-10-16T20:46:18.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f462d79d640 1 -- start start 2023-10-16T20:46:18.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f462d79d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f462806f810 0x7f46280c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:18.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f462d79d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46280c58d0 0x7f4628141670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:18.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f462d79d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4628153670 0x7f462813e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:18.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f462d79d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462814bdc0 con 0x7f4628153670 2023-10-16T20:46:18.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f462d79d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46280c96d0 con 0x7f46280c58d0 2023-10-16T20:46:18.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f462d79d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46280c1e80 con 0x7f462806f810 2023-10-16T20:46:18.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f46277fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4628153670 0x7f462813e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:18.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f46277fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4628153670 0x7f462813e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56624/0 (socket says 172.21.15.88:56624) 2023-10-16T20:46:18.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f46277fe640 1 -- 172.21.15.88:0/2535872660 learned_addr learned my addr 172.21.15.88:0/2535872660 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:18.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f4626ffd640 1 --2- 172.21.15.88:0/2535872660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f462806f810 0x7f46280c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:18.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f461ffff640 1 --2- 172.21.15.88:0/2535872660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46280c58d0 0x7f4628141670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:18.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f46277fe640 1 -- 172.21.15.88:0/2535872660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f462806f810 msgr2=0x7f46280c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:18.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f46277fe640 1 --2- 172.21.15.88:0/2535872660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f462806f810 0x7f46280c8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:18.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f46277fe640 1 -- 172.21.15.88:0/2535872660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46280c58d0 msgr2=0x7f4628141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:18.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f46277fe640 1 --2- 172.21.15.88:0/2535872660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46280c58d0 0x7f4628141670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:18.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.975+0000 7f46277fe640 1 -- 172.21.15.88:0/2535872660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4610009420 con 0x7f4628153670 2023-10-16T20:46:18.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.979+0000 7f46277fe640 1 --2- 172.21.15.88:0/2535872660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4628153670 0x7f462813e2c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4620001f40 tx=0x7f462000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:18.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.979+0000 7f461f7fe640 1 -- 172.21.15.88:0/2535872660 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4620012400 con 0x7f4628153670 2023-10-16T20:46:18.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.979+0000 7f461f7fe640 1 -- 172.21.15.88:0/2535872660 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4620012580 con 0x7f4628153670 2023-10-16T20:46:18.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.979+0000 7f461f7fe640 1 -- 172.21.15.88:0/2535872660 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4620012970 con 0x7f4628153670 2023-10-16T20:46:18.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.979+0000 7f462d79d640 1 -- 172.21.15.88:0/2535872660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46280c3e60 con 0x7f4628153670 2023-10-16T20:46:18.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.979+0000 7f462d79d640 1 -- 172.21.15.88:0/2535872660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46280c9090 con 0x7f4628153670 2023-10-16T20:46:18.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.979+0000 7f462d79d640 1 -- 172.21.15.88:0/2535872660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46280c1910 con 0x7f4628153670 2023-10-16T20:46:18.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.979+0000 7f461f7fe640 1 -- 172.21.15.88:0/2535872660 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4620007070 con 0x7f4628153670 2023-10-16T20:46:18.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.979+0000 7f461f7fe640 1 --2- 172.21.15.88:0/2535872660 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f46080664a0 0x7f4608068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:18.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.979+0000 7f461f7fe640 1 -- 172.21.15.88:0/2535872660 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7007+0+0 (secure 0 0 0) 0x7f4620084440 con 0x7f4628153670 2023-10-16T20:46:18.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.979+0000 7f4626ffd640 1 --2- 172.21.15.88:0/2535872660 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f46080664a0 0x7f4608068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:18.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.983+0000 7f4626ffd640 1 --2- 172.21.15.88:0/2535872660 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f46080664a0 0x7f4608068960 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f46180011f0 tx=0x7f4618009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:18.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:18.983+0000 7f461f7fe640 1 -- 172.21.15.88:0/2535872660 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4620051330 con 0x7f4628153670 2023-10-16T20:46:19.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:19.163+0000 7f462d79d640 1 -- 172.21.15.88:0/2535872660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46280c1ff0 con 0x7f4628153670 2023-10-16T20:46:19.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:19.163+0000 7f461f7fe640 1 -- 172.21.15.88:0/2535872660 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+5790 (secure 0 0 0) 0x7f4620017090 con 0x7f4628153670 2023-10-16T20:46:19.167 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:19.167 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:46:15.444685+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:19.167 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 52 2023-10-16T20:46:19.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:19.167+0000 7f462d79d640 1 -- 172.21.15.88:0/2535872660 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f46080664a0 msgr2=0x7f4608068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:19.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:19.167+0000 7f462d79d640 1 --2- 172.21.15.88:0/2535872660 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f46080664a0 0x7f4608068960 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f46180011f0 tx=0x7f4618009040 comp rx=0 tx=0).stop 2023-10-16T20:46:19.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:19.167+0000 7f462d79d640 1 -- 172.21.15.88:0/2535872660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4628153670 msgr2=0x7f462813e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:19.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:19.167+0000 7f462d79d640 1 --2- 172.21.15.88:0/2535872660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4628153670 0x7f462813e2c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f4620001f40 tx=0x7f462000a430 comp rx=0 tx=0).stop 2023-10-16T20:46:19.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:19.167+0000 7f462d79d640 1 -- 172.21.15.88:0/2535872660 shutdown_connections 2023-10-16T20:46:19.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:19.167+0000 7f462d79d640 1 --2- 172.21.15.88:0/2535872660 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f46080664a0 0x7f4608068960 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:19.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:19.167+0000 7f462d79d640 1 --2- 172.21.15.88:0/2535872660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4628153670 0x7f462813e2c0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:19.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:19.167+0000 7f462d79d640 1 --2- 172.21.15.88:0/2535872660 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46280c58d0 0x7f4628141670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:19.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:19.167+0000 7f462d79d640 1 --2- 172.21.15.88:0/2535872660 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f462806f810 0x7f46280c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:19.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:19.167+0000 7f462d79d640 1 -- 172.21.15.88:0/2535872660 >> 172.21.15.88:0/2535872660 conn(0x7f46280cad50 msgr2=0x7f462814ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:19.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:19.167+0000 7f462d79d640 1 -- 172.21.15.88:0/2535872660 shutdown_connections 2023-10-16T20:46:19.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:19.167+0000 7f462d79d640 1 -- 172.21.15.88:0/2535872660 wait complete. 2023-10-16T20:46:19.189 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:46:21.190 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:21.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.327+0000 7fe8a0079640 1 Processor -- start 2023-10-16T20:46:21.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.327+0000 7fe8a0079640 1 -- start start 2023-10-16T20:46:21.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.327+0000 7fe8a0079640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe8981406d0 0x7fe8980c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:21.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.327+0000 7fe8a0079640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe89806a970 0x7fe898071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:21.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.327+0000 7fe8a0079640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe898071e60 0x7fe89815a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:21.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.327+0000 7fe8a0079640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89815b3c0 con 0x7fe89806a970 2023-10-16T20:46:21.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.331+0000 7fe8a0079640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89815b680 con 0x7fe8981406d0 2023-10-16T20:46:21.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.331+0000 7fe8a0079640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe898157570 con 0x7fe898071e60 2023-10-16T20:46:21.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.331+0000 7fe89d614640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe89806a970 0x7fe898071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:21.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.331+0000 7fe89d614640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe89806a970 0x7fe898071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56636/0 (socket says 172.21.15.88:56636) 2023-10-16T20:46:21.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.331+0000 7fe89d614640 1 -- 172.21.15.88:0/1773999154 learned_addr learned my addr 172.21.15.88:0/1773999154 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:21.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.331+0000 7fe89de15640 1 --2- 172.21.15.88:0/1773999154 >> v2:172.21.15.191:3300/0 conn(0x7fe8981406d0 0x7fe8980c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:21.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.331+0000 7fe89e616640 1 --2- 172.21.15.88:0/1773999154 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe898071e60 0x7fe89815a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:21.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.331+0000 7fe89d614640 1 -- 172.21.15.88:0/1773999154 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe898071e60 msgr2=0x7fe89815a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:21.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.331+0000 7fe89d614640 1 --2- 172.21.15.88:0/1773999154 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe898071e60 0x7fe89815a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:21.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.331+0000 7fe89d614640 1 -- 172.21.15.88:0/1773999154 >> v2:172.21.15.191:3300/0 conn(0x7fe8981406d0 msgr2=0x7fe8980c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:21.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.331+0000 7fe89d614640 1 --2- 172.21.15.88:0/1773999154 >> v2:172.21.15.191:3300/0 conn(0x7fe8981406d0 0x7fe8980c1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:21.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.331+0000 7fe89d614640 1 -- 172.21.15.88:0/1773999154 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe898157850 con 0x7fe89806a970 2023-10-16T20:46:21.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.331+0000 7fe89e616640 1 --2- 172.21.15.88:0/1773999154 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe898071e60 0x7fe89815a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:46:21.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.331+0000 7fe89d614640 1 --2- 172.21.15.88:0/1773999154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe89806a970 0x7fe898071920 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe89806e1f0 tx=0x7fe88801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87396cebf9589678 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:21.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.331+0000 7fe89ce13640 1 -- 172.21.15.88:0/1773999154 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8880ab600 con 0x7fe89806a970 2023-10-16T20:46:21.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe89ce13640 1 -- 172.21.15.88:0/1773999154 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8880a7050 con 0x7fe89806a970 2023-10-16T20:46:21.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe89ce13640 1 -- 172.21.15.88:0/1773999154 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8880ab980 con 0x7fe89806a970 2023-10-16T20:46:21.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe8a0079640 1 -- 172.21.15.88:0/1773999154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe89806a970 msgr2=0x7fe898071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:21.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe8a0079640 1 --2- 172.21.15.88:0/1773999154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe89806a970 0x7fe898071920 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe89806e1f0 tx=0x7fe88801c500 comp rx=0 tx=0).stop 2023-10-16T20:46:21.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe8a0079640 1 -- 172.21.15.88:0/1773999154 shutdown_connections 2023-10-16T20:46:21.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe8a0079640 1 --2- 172.21.15.88:0/1773999154 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe898071e60 0x7fe89815a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:21.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe8a0079640 1 --2- 172.21.15.88:0/1773999154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe89806a970 0x7fe898071920 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:21.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe8a0079640 1 --2- 172.21.15.88:0/1773999154 >> v2:172.21.15.191:3300/0 conn(0x7fe8981406d0 0x7fe8980c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:21.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe8a0079640 1 -- 172.21.15.88:0/1773999154 >> 172.21.15.88:0/1773999154 conn(0x7fe8980cad50 msgr2=0x7fe8980cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:21.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe8a0079640 1 -- 172.21.15.88:0/1773999154 shutdown_connections 2023-10-16T20:46:21.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe8a0079640 1 -- 172.21.15.88:0/1773999154 wait complete. 2023-10-16T20:46:21.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe8a0079640 1 Processor -- start 2023-10-16T20:46:21.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe8a0079640 1 -- start start 2023-10-16T20:46:21.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe8a0079640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe89806a970 0x7fe8980c2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:21.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe8a0079640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe898071e60 0x7fe89813b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:21.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe8a0079640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8981406d0 0x7fe89813b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:21.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe8a0079640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe898158da0 con 0x7fe89806a970 2023-10-16T20:46:21.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe8a0079640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89813a660 con 0x7fe898071e60 2023-10-16T20:46:21.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe8a0079640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe898159550 con 0x7fe8981406d0 2023-10-16T20:46:21.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe89de15640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe89806a970 0x7fe8980c2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:21.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe89de15640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe89806a970 0x7fe8980c2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56646/0 (socket says 172.21.15.88:56646) 2023-10-16T20:46:21.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe89de15640 1 -- 172.21.15.88:0/2287221273 learned_addr learned my addr 172.21.15.88:0/2287221273 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:21.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe89d614640 1 --2- 172.21.15.88:0/2287221273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe898071e60 0x7fe89813b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:21.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.335+0000 7fe89e616640 1 --2- 172.21.15.88:0/2287221273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8981406d0 0x7fe89813b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:21.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.339+0000 7fe89de15640 1 -- 172.21.15.88:0/2287221273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8981406d0 msgr2=0x7fe89813b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:21.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.339+0000 7fe89de15640 1 --2- 172.21.15.88:0/2287221273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8981406d0 0x7fe89813b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:21.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.339+0000 7fe89de15640 1 -- 172.21.15.88:0/2287221273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe898071e60 msgr2=0x7fe89813b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:21.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.339+0000 7fe89de15640 1 --2- 172.21.15.88:0/2287221273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe898071e60 0x7fe89813b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:21.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.339+0000 7fe89de15640 1 -- 172.21.15.88:0/2287221273 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8880094d0 con 0x7fe89806a970 2023-10-16T20:46:21.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.339+0000 7fe89e616640 1 --2- 172.21.15.88:0/2287221273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8981406d0 0x7fe89813b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:46:21.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.339+0000 7fe89de15640 1 --2- 172.21.15.88:0/2287221273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe89806a970 0x7fe8980c2ae0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe89400e870 tx=0x7fe89400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:21.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.339+0000 7fe88e7fc640 1 -- 172.21.15.88:0/2287221273 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe894009510 con 0x7fe89806a970 2023-10-16T20:46:21.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.339+0000 7fe88e7fc640 1 -- 172.21.15.88:0/2287221273 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89400eec0 con 0x7fe89806a970 2023-10-16T20:46:21.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.339+0000 7fe8a0079640 1 -- 172.21.15.88:0/2287221273 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8981581d0 con 0x7fe89806a970 2023-10-16T20:46:21.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.339+0000 7fe88e7fc640 1 -- 172.21.15.88:0/2287221273 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89400cb40 con 0x7fe89806a970 2023-10-16T20:46:21.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.339+0000 7fe8a0079640 1 -- 172.21.15.88:0/2287221273 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe898158f90 con 0x7fe89806a970 2023-10-16T20:46:21.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.339+0000 7fe8a0079640 1 -- 172.21.15.88:0/2287221273 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe898074700 con 0x7fe89806a970 2023-10-16T20:46:21.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.339+0000 7fe88e7fc640 1 -- 172.21.15.88:0/2287221273 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe89400cca0 con 0x7fe89806a970 2023-10-16T20:46:21.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.339+0000 7fe88e7fc640 1 --2- 172.21.15.88:0/2287221273 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe87c0664a0 0x7fe87c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:21.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.339+0000 7fe88e7fc640 1 -- 172.21.15.88:0/2287221273 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7007+0+0 (secure 0 0 0) 0x7fe894084350 con 0x7fe89806a970 2023-10-16T20:46:21.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.339+0000 7fe89d614640 1 --2- 172.21.15.88:0/2287221273 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe87c0664a0 0x7fe87c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:21.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.343+0000 7fe89d614640 1 --2- 172.21.15.88:0/2287221273 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe87c0664a0 0x7fe87c068960 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe8880ad000 tx=0x7fe888008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:21.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.343+0000 7fe88e7fc640 1 -- 172.21.15.88:0/2287221273 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe894051360 con 0x7fe89806a970 2023-10-16T20:46:21.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.527+0000 7fe8a0079640 1 -- 172.21.15.88:0/2287221273 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8980c8300 con 0x7fe89806a970 2023-10-16T20:46:21.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.531+0000 7fe88e7fc640 1 -- 172.21.15.88:0/2287221273 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+5792 (secure 0 0 0) 0x7fe894054980 con 0x7fe89806a970 2023-10-16T20:46:21.533 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:21.533 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:46:20.763022+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:resolve","state_seq":24,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:21.534 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 53 2023-10-16T20:46:21.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.531+0000 7fe8a0079640 1 -- 172.21.15.88:0/2287221273 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe87c0664a0 msgr2=0x7fe87c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:21.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.531+0000 7fe8a0079640 1 --2- 172.21.15.88:0/2287221273 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe87c0664a0 0x7fe87c068960 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe8880ad000 tx=0x7fe888008040 comp rx=0 tx=0).stop 2023-10-16T20:46:21.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.531+0000 7fe8a0079640 1 -- 172.21.15.88:0/2287221273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe89806a970 msgr2=0x7fe8980c2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:21.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.531+0000 7fe8a0079640 1 --2- 172.21.15.88:0/2287221273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe89806a970 0x7fe8980c2ae0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe89400e870 tx=0x7fe89400ec70 comp rx=0 tx=0).stop 2023-10-16T20:46:21.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.531+0000 7fe8a0079640 1 -- 172.21.15.88:0/2287221273 shutdown_connections 2023-10-16T20:46:21.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.531+0000 7fe8a0079640 1 --2- 172.21.15.88:0/2287221273 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe87c0664a0 0x7fe87c068960 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:21.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.531+0000 7fe8a0079640 1 --2- 172.21.15.88:0/2287221273 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8981406d0 0x7fe89813b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:21.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.531+0000 7fe8a0079640 1 --2- 172.21.15.88:0/2287221273 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe898071e60 0x7fe89813b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:21.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.531+0000 7fe8a0079640 1 --2- 172.21.15.88:0/2287221273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe89806a970 0x7fe8980c2ae0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:21.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.531+0000 7fe8a0079640 1 -- 172.21.15.88:0/2287221273 >> 172.21.15.88:0/2287221273 conn(0x7fe8980cad50 msgr2=0x7fe89813d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:21.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.531+0000 7fe8a0079640 1 -- 172.21.15.88:0/2287221273 shutdown_connections 2023-10-16T20:46:21.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:21.531+0000 7fe8a0079640 1 -- 172.21.15.88:0/2287221273 wait complete. 2023-10-16T20:46:21.554 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:46:23.555 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:23.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.699+0000 7f702110f640 1 Processor -- start 2023-10-16T20:46:23.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.699+0000 7f702110f640 1 -- start start 2023-10-16T20:46:23.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.699+0000 7f702110f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f701c1538e0 0x7f701c0ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:23.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.699+0000 7f702110f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f701c15a2a0 0x7f701c0c3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:23.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.699+0000 7f702110f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701c0ca610 con 0x7f701c15cb20 2023-10-16T20:46:23.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.699+0000 7f702110f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f701c1595b0 con 0x7f701c1538e0 2023-10-16T20:46:23.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.699+0000 7f702110f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701c15d540 con 0x7f701c15a2a0 2023-10-16T20:46:23.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.699+0000 7f701ad9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f701c1538e0 0x7f701c0ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:23.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.699+0000 7f701b59e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f701c15cb20 0x7f701c0cee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33082/0 (socket says 172.21.15.88:33082) 2023-10-16T20:46:23.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.699+0000 7f701b59e640 1 -- 172.21.15.88:0/2866578332 learned_addr learned my addr 172.21.15.88:0/2866578332 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:46:23.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.699+0000 7f701259c640 1 --2- 172.21.15.88:0/2866578332 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f701c15a2a0 0x7f701c0c3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:23.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.699+0000 7f701ad9d640 1 -- 172.21.15.88:0/2866578332 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f701c15a2a0 msgr2=0x7f701c0c3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:23.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.699+0000 7f701ad9d640 1 --2- 172.21.15.88:0/2866578332 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f701c15a2a0 0x7f701c0c3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:23.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.699+0000 7f701ad9d640 1 -- 172.21.15.88:0/2866578332 >> v1:172.21.15.88:6789/0 conn(0x7f701c15cb20 legacy=0x7f701c0cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:23.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.699+0000 7f701ad9d640 1 -- 172.21.15.88:0/2866578332 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c15d820 con 0x7f701c1538e0 2023-10-16T20:46:23.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.699+0000 7f701a59c640 1 -- 172.21.15.88:0/2866578332 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180555890 0 0) 0x7f701c0ca610 con 0x7f701c15cb20 2023-10-16T20:46:23.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.703+0000 7f701ad9d640 1 --2- 172.21.15.88:0/2866578332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f701c1538e0 0x7f701c0ca0d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f700c0097c0 tx=0x7f700c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554868f434a4d241 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:23.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.703+0000 7f701a59c640 1 -- 172.21.15.88:0/2866578332 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c0ab600 con 0x7f701c1538e0 2023-10-16T20:46:23.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.703+0000 7f701a59c640 1 -- 172.21.15.88:0/2866578332 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700c0a7050 con 0x7f701c1538e0 2023-10-16T20:46:23.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.703+0000 7f701a59c640 1 -- 172.21.15.88:0/2866578332 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c0ab980 con 0x7f701c1538e0 2023-10-16T20:46:23.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.703+0000 7f702110f640 1 -- 172.21.15.88:0/2866578332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f701c1538e0 msgr2=0x7f701c0ca0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:23.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.703+0000 7f702110f640 1 --2- 172.21.15.88:0/2866578332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f701c1538e0 0x7f701c0ca0d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f700c0097c0 tx=0x7f700c01c480 comp rx=0 tx=0).stop 2023-10-16T20:46:23.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.703+0000 7f702110f640 1 -- 172.21.15.88:0/2866578332 shutdown_connections 2023-10-16T20:46:23.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.703+0000 7f702110f640 1 --2- 172.21.15.88:0/2866578332 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f701c15a2a0 0x7f701c0c3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:23.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.703+0000 7f702110f640 1 --2- 172.21.15.88:0/2866578332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f701c1538e0 0x7f701c0ca0d0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:23.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.703+0000 7f702110f640 1 -- 172.21.15.88:0/2866578332 >> 172.21.15.88:0/2866578332 conn(0x7f701c06bd50 msgr2=0x7f701c06c160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:23.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.703+0000 7f702110f640 1 -- 172.21.15.88:0/2866578332 shutdown_connections 2023-10-16T20:46:23.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.703+0000 7f702110f640 1 -- 172.21.15.88:0/2866578332 wait complete. 2023-10-16T20:46:23.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f702110f640 1 Processor -- start 2023-10-16T20:46:23.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f702110f640 1 -- start start 2023-10-16T20:46:23.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f702110f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f701c1538e0 0x7f701c0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:23.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f702110f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f701c15a2a0 0x7f701c0c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:23.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f702110f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f701c15cb20 0x7f701c1498e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:23.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f702110f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c0cfc60 con 0x7f701c1538e0 2023-10-16T20:46:23.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f702110f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c15c920 con 0x7f701c15cb20 2023-10-16T20:46:23.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f702110f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701c0c6d80 con 0x7f701c15a2a0 2023-10-16T20:46:23.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f701ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f701c1538e0 0x7f701c0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:23.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f701ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f701c1538e0 0x7f701c0cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56664/0 (socket says 172.21.15.88:56664) 2023-10-16T20:46:23.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f701ad9d640 1 -- 172.21.15.88:0/1661541350 learned_addr learned my addr 172.21.15.88:0/1661541350 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:23.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f701259c640 1 --2- 172.21.15.88:0/1661541350 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f701c15a2a0 0x7f701c0c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:23.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f701b59e640 1 --2- 172.21.15.88:0/1661541350 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f701c15cb20 0x7f701c1498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:23.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f701ad9d640 1 -- 172.21.15.88:0/1661541350 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f701c15a2a0 msgr2=0x7f701c0c77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:23.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f701ad9d640 1 --2- 172.21.15.88:0/1661541350 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f701c15a2a0 0x7f701c0c77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:23.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f701ad9d640 1 -- 172.21.15.88:0/1661541350 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f701c15cb20 msgr2=0x7f701c1498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:23.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f701ad9d640 1 --2- 172.21.15.88:0/1661541350 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f701c15cb20 0x7f701c1498e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:23.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f701ad9d640 1 -- 172.21.15.88:0/1661541350 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700c009420 con 0x7f701c1538e0 2023-10-16T20:46:23.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f701259c640 1 --2- 172.21.15.88:0/1661541350 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f701c15a2a0 0x7f701c0c77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:46:23.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f701ad9d640 1 --2- 172.21.15.88:0/1661541350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f701c1538e0 0x7f701c0cf720 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f700c01a0b0 tx=0x7f700c019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:23.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f7013fff640 1 -- 172.21.15.88:0/1661541350 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c0b4610 con 0x7f701c1538e0 2023-10-16T20:46:23.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f7013fff640 1 -- 172.21.15.88:0/1661541350 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700c0abe90 con 0x7f701c1538e0 2023-10-16T20:46:23.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f702110f640 1 -- 172.21.15.88:0/1661541350 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701c0c7d10 con 0x7f701c1538e0 2023-10-16T20:46:23.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f7013fff640 1 -- 172.21.15.88:0/1661541350 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c0b49d0 con 0x7f701c1538e0 2023-10-16T20:46:23.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.707+0000 7f702110f640 1 -- 172.21.15.88:0/1661541350 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f701c1f2720 con 0x7f701c1538e0 2023-10-16T20:46:23.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.711+0000 7f702110f640 1 -- 172.21.15.88:0/1661541350 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f701c074700 con 0x7f701c1538e0 2023-10-16T20:46:23.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.711+0000 7f7013fff640 1 -- 172.21.15.88:0/1661541350 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f700c0b2040 con 0x7f701c1538e0 2023-10-16T20:46:23.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.711+0000 7f7013fff640 1 --2- 172.21.15.88:0/1661541350 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6ffc0664a0 0x7f6ffc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:23.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.711+0000 7f7013fff640 1 -- 172.21.15.88:0/1661541350 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7007+0+0 (secure 0 0 0) 0x7f700c11e480 con 0x7f701c1538e0 2023-10-16T20:46:23.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.711+0000 7f701259c640 1 --2- 172.21.15.88:0/1661541350 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6ffc0664a0 0x7f6ffc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:23.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.711+0000 7f701259c640 1 --2- 172.21.15.88:0/1661541350 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6ffc0664a0 0x7f6ffc068960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f701c0c4c90 tx=0x7f7004008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:23.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.715+0000 7f7013fff640 1 -- 172.21.15.88:0/1661541350 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f700c0eb3e0 con 0x7f701c1538e0 2023-10-16T20:46:23.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.895+0000 7f702110f640 1 -- 172.21.15.88:0/1661541350 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f701c159d50 con 0x7f701c1538e0 2023-10-16T20:46:23.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.895+0000 7f7013fff640 1 -- 172.21.15.88:0/1661541350 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+5792 (secure 0 0 0) 0x7f700c0b0020 con 0x7f701c1538e0 2023-10-16T20:46:23.900 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:23.900 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:46:20.763022+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:resolve","state_seq":24,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:23.900 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 53 2023-10-16T20:46:23.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.899+0000 7f702110f640 1 -- 172.21.15.88:0/1661541350 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6ffc0664a0 msgr2=0x7f6ffc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:23.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.899+0000 7f702110f640 1 --2- 172.21.15.88:0/1661541350 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6ffc0664a0 0x7f6ffc068960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f701c0c4c90 tx=0x7f7004008040 comp rx=0 tx=0).stop 2023-10-16T20:46:23.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.899+0000 7f702110f640 1 -- 172.21.15.88:0/1661541350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f701c1538e0 msgr2=0x7f701c0cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:23.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.899+0000 7f702110f640 1 --2- 172.21.15.88:0/1661541350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f701c1538e0 0x7f701c0cf720 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f700c01a0b0 tx=0x7f700c019220 comp rx=0 tx=0).stop 2023-10-16T20:46:23.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.899+0000 7f702110f640 1 -- 172.21.15.88:0/1661541350 shutdown_connections 2023-10-16T20:46:23.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.899+0000 7f702110f640 1 --2- 172.21.15.88:0/1661541350 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6ffc0664a0 0x7f6ffc068960 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:23.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.899+0000 7f702110f640 1 --2- 172.21.15.88:0/1661541350 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f701c15cb20 0x7f701c1498e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:23.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.899+0000 7f702110f640 1 --2- 172.21.15.88:0/1661541350 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f701c15a2a0 0x7f701c0c77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:23.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.899+0000 7f702110f640 1 --2- 172.21.15.88:0/1661541350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f701c1538e0 0x7f701c0cf720 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:23.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.899+0000 7f702110f640 1 -- 172.21.15.88:0/1661541350 >> 172.21.15.88:0/1661541350 conn(0x7f701c06bd50 msgr2=0x7f701c0c9760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:23.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.899+0000 7f702110f640 1 -- 172.21.15.88:0/1661541350 shutdown_connections 2023-10-16T20:46:23.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:23.899+0000 7f702110f640 1 -- 172.21.15.88:0/1661541350 wait complete. 2023-10-16T20:46:23.920 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:46:25.922 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:26.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.067+0000 7f57c2cea640 1 Processor -- start 2023-10-16T20:46:26.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.067+0000 7f57c2cea640 1 -- start start 2023-10-16T20:46:26.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.071+0000 7f57c2cea640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f57bc1406d0 0x7f57bc0c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:26.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.071+0000 7f57c2cea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f57bc15a810 0x7f57bc15ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:26.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.071+0000 7f57c2cea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57bc15b3c0 con 0x7f57bc06a970 2023-10-16T20:46:26.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.071+0000 7f57c2cea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57bc15b1c0 con 0x7f57bc15a810 2023-10-16T20:46:26.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.071+0000 7f57c2cea640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57bc15b680 con 0x7f57bc1406d0 2023-10-16T20:46:26.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.071+0000 7f57bbfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f57bc06a970 0x7f57bc071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33086/0 (socket says 172.21.15.88:33086) 2023-10-16T20:46:26.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.071+0000 7f57bbfff640 1 -- 172.21.15.88:0/4211214856 learned_addr learned my addr 172.21.15.88:0/4211214856 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:46:26.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.071+0000 7f57c1287640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f57bc15a810 0x7f57bc15ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:26.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.071+0000 7f57c0a86640 1 --2- 172.21.15.88:0/4211214856 >> v2:172.21.15.191:3300/0 conn(0x7f57bc1406d0 0x7f57bc0c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:26.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.071+0000 7f57bb7fe640 1 -- 172.21.15.88:0/4211214856 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129215574 0 0) 0x7f57bc15b3c0 con 0x7f57bc06a970 2023-10-16T20:46:26.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.071+0000 7f57bb7fe640 1 -- 172.21.15.88:0/4211214856 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57ac003530 con 0x7f57bc06a970 2023-10-16T20:46:26.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.071+0000 7f57c1287640 1 -- 172.21.15.88:0/4211214856 >> v2:172.21.15.191:3300/0 conn(0x7f57bc1406d0 msgr2=0x7f57bc0c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:26.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.071+0000 7f57c1287640 1 --2- 172.21.15.88:0/4211214856 >> v2:172.21.15.191:3300/0 conn(0x7f57bc1406d0 0x7f57bc0c1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:26.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.071+0000 7f57c1287640 1 -- 172.21.15.88:0/4211214856 >> v1:172.21.15.88:6789/0 conn(0x7f57bc06a970 legacy=0x7f57bc071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:26.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.071+0000 7f57c1287640 1 -- 172.21.15.88:0/4211214856 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57bc157570 con 0x7f57bc15a810 2023-10-16T20:46:26.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57c1287640 1 --2- 172.21.15.88:0/4211214856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc15a810 0x7f57bc15ac80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f57bc06e1f0 tx=0x7f57b401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=205ed071e015cc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:26.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57bb7fe640 1 -- 172.21.15.88:0/4211214856 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b40a59a0 con 0x7f57bc15a810 2023-10-16T20:46:26.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57bb7fe640 1 -- 172.21.15.88:0/4211214856 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b40a5b00 con 0x7f57bc15a810 2023-10-16T20:46:26.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57bb7fe640 1 -- 172.21.15.88:0/4211214856 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b40a5e90 con 0x7f57bc15a810 2023-10-16T20:46:26.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57c2cea640 1 -- 172.21.15.88:0/4211214856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc15a810 msgr2=0x7f57bc15ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:26.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57c2cea640 1 --2- 172.21.15.88:0/4211214856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc15a810 0x7f57bc15ac80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f57bc06e1f0 tx=0x7f57b401c8e0 comp rx=0 tx=0).stop 2023-10-16T20:46:26.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57c2cea640 1 -- 172.21.15.88:0/4211214856 shutdown_connections 2023-10-16T20:46:26.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57c2cea640 1 --2- 172.21.15.88:0/4211214856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc15a810 0x7f57bc15ac80 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:26.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57c2cea640 1 --2- 172.21.15.88:0/4211214856 >> v2:172.21.15.191:3300/0 conn(0x7f57bc1406d0 0x7f57bc0c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:26.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57c2cea640 1 -- 172.21.15.88:0/4211214856 >> 172.21.15.88:0/4211214856 conn(0x7f57bc0cad50 msgr2=0x7f57bc0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:26.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57c2cea640 1 -- 172.21.15.88:0/4211214856 shutdown_connections 2023-10-16T20:46:26.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57c2cea640 1 -- 172.21.15.88:0/4211214856 wait complete. 2023-10-16T20:46:26.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57c2cea640 1 Processor -- start 2023-10-16T20:46:26.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57c2cea640 1 -- start start 2023-10-16T20:46:26.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57c2cea640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57bc06a970 0x7f57bc13df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:26.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57c2cea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc1406d0 0x7f57bc155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:26.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57c2cea640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57bc15a810 0x7f57bc1efd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:26.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57c2cea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc13a620 con 0x7f57bc1406d0 2023-10-16T20:46:26.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57c2cea640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc152370 con 0x7f57bc15a810 2023-10-16T20:46:26.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57c2cea640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc1ef600 con 0x7f57bc06a970 2023-10-16T20:46:26.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57bbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc1406d0 0x7f57bc155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:26.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57bbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc1406d0 0x7f57bc155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56682/0 (socket says 172.21.15.88:56682) 2023-10-16T20:46:26.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.075+0000 7f57bbfff640 1 -- 172.21.15.88:0/71138815 learned_addr learned my addr 172.21.15.88:0/71138815 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:26.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.079+0000 7f57c0a86640 1 --2- 172.21.15.88:0/71138815 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57bc06a970 0x7f57bc13df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:26.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.079+0000 7f57c1287640 1 --2- 172.21.15.88:0/71138815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57bc15a810 0x7f57bc1efd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:26.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.079+0000 7f57bbfff640 1 -- 172.21.15.88:0/71138815 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57bc06a970 msgr2=0x7f57bc13df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:26.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.079+0000 7f57bbfff640 1 --2- 172.21.15.88:0/71138815 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57bc06a970 0x7f57bc13df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:26.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.079+0000 7f57bbfff640 1 -- 172.21.15.88:0/71138815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57bc15a810 msgr2=0x7f57bc1efd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:26.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.079+0000 7f57bbfff640 1 --2- 172.21.15.88:0/71138815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57bc15a810 0x7f57bc1efd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:26.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.079+0000 7f57bbfff640 1 -- 172.21.15.88:0/71138815 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b4009a70 con 0x7f57bc1406d0 2023-10-16T20:46:26.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.079+0000 7f57bbfff640 1 --2- 172.21.15.88:0/71138815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc1406d0 0x7f57bc155c60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f57a800c420 tx=0x7f57a800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:26.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.079+0000 7f57b97fa640 1 -- 172.21.15.88:0/71138815 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a8005cf0 con 0x7f57bc1406d0 2023-10-16T20:46:26.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.079+0000 7f57b97fa640 1 -- 172.21.15.88:0/71138815 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a8005e50 con 0x7f57bc1406d0 2023-10-16T20:46:26.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.079+0000 7f57b97fa640 1 -- 172.21.15.88:0/71138815 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a80072d0 con 0x7f57bc1406d0 2023-10-16T20:46:26.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.079+0000 7f57c2cea640 1 -- 172.21.15.88:0/71138815 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57bc1544e0 con 0x7f57bc1406d0 2023-10-16T20:46:26.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.079+0000 7f57c2cea640 1 -- 172.21.15.88:0/71138815 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57bc153960 con 0x7f57bc1406d0 2023-10-16T20:46:26.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.079+0000 7f57c2cea640 1 -- 172.21.15.88:0/71138815 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57bc074700 con 0x7f57bc1406d0 2023-10-16T20:46:26.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.079+0000 7f57b97fa640 1 -- 172.21.15.88:0/71138815 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f57a8007430 con 0x7f57bc1406d0 2023-10-16T20:46:26.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.079+0000 7f57b97fa640 1 --2- 172.21.15.88:0/71138815 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57a00664a0 0x7f57a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:26.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.079+0000 7f57b97fa640 1 -- 172.21.15.88:0/71138815 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7007+0+0 (secure 0 0 0) 0x7f57a80839c0 con 0x7f57bc1406d0 2023-10-16T20:46:26.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.083+0000 7f57c0a86640 1 --2- 172.21.15.88:0/71138815 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57a00664a0 0x7f57a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:26.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.083+0000 7f57c0a86640 1 --2- 172.21.15.88:0/71138815 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57a00664a0 0x7f57a0068960 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009ac0 tx=0x7f57b0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:26.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.087+0000 7f57b97fa640 1 -- 172.21.15.88:0/71138815 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f57a80508b0 con 0x7f57bc1406d0 2023-10-16T20:46:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.267+0000 7f57c2cea640 1 -- 172.21.15.88:0/71138815 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57bc153c40 con 0x7f57bc1406d0 2023-10-16T20:46:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.267+0000 7f57b97fa640 1 -- 172.21.15.88:0/71138815 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+5792 (secure 0 0 0) 0x7f57a8017020 con 0x7f57bc1406d0 2023-10-16T20:46:26.270 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:26.270 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:46:20.763022+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:resolve","state_seq":24,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 53 2023-10-16T20:46:26.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.267+0000 7f57c2cea640 1 -- 172.21.15.88:0/71138815 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57a00664a0 msgr2=0x7f57a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:26.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.267+0000 7f57c2cea640 1 --2- 172.21.15.88:0/71138815 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57a00664a0 0x7f57a0068960 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009ac0 tx=0x7f57b0009190 comp rx=0 tx=0).stop 2023-10-16T20:46:26.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.267+0000 7f57c2cea640 1 -- 172.21.15.88:0/71138815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc1406d0 msgr2=0x7f57bc155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:26.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.267+0000 7f57c2cea640 1 --2- 172.21.15.88:0/71138815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc1406d0 0x7f57bc155c60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f57a800c420 tx=0x7f57a800c8e0 comp rx=0 tx=0).stop 2023-10-16T20:46:26.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.267+0000 7f57c2cea640 1 -- 172.21.15.88:0/71138815 shutdown_connections 2023-10-16T20:46:26.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.267+0000 7f57c2cea640 1 --2- 172.21.15.88:0/71138815 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57a00664a0 0x7f57a0068960 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:26.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.267+0000 7f57c2cea640 1 --2- 172.21.15.88:0/71138815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f57bc15a810 0x7f57bc1efd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:26.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.267+0000 7f57c2cea640 1 --2- 172.21.15.88:0/71138815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57bc1406d0 0x7f57bc155c60 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:26.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.267+0000 7f57c2cea640 1 --2- 172.21.15.88:0/71138815 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f57bc06a970 0x7f57bc13df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.267+0000 7f57c2cea640 1 -- 172.21.15.88:0/71138815 >> 172.21.15.88:0/71138815 conn(0x7f57bc0cad50 msgr2=0x7f57bc0c24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.267+0000 7f57c2cea640 1 -- 172.21.15.88:0/71138815 shutdown_connections 2023-10-16T20:46:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:26.271+0000 7f57c2cea640 1 -- 172.21.15.88:0/71138815 wait complete. 2023-10-16T20:46:26.291 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:46:27.251 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:46:27.256 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:46:28.292 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:28.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.439+0000 7f3319de4640 1 Processor -- start 2023-10-16T20:46:28.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.439+0000 7f3319de4640 1 -- start start 2023-10-16T20:46:28.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.439+0000 7f3319de4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f33141537c0 0x7f331415c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:28.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.439+0000 7f3319de4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f331415cdb0 0x7f33140c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:28.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.439+0000 7f3319de4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33140cae10 con 0x7f33140ca2f0 2023-10-16T20:46:28.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.439+0000 7f3319de4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33140cac30 con 0x7f331415cdb0 2023-10-16T20:46:28.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.439+0000 7f3319de4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f331415d2b0 con 0x7f33141537c0 2023-10-16T20:46:28.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.443+0000 7f33195e3640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f33140ca2f0 0x7f33140cead0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33088/0 (socket says 172.21.15.88:33088) 2023-10-16T20:46:28.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.443+0000 7f33195e3640 1 -- 172.21.15.88:0/2453364944 learned_addr learned my addr 172.21.15.88:0/2453364944 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:46:28.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.443+0000 7f3313fff640 1 --2- 172.21.15.88:0/2453364944 >> v2:172.21.15.88:3300/0 conn(0x7f331415cdb0 0x7f33140c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:28.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.443+0000 7f3318de2640 1 --2- 172.21.15.88:0/2453364944 >> v2:172.21.15.191:3300/0 conn(0x7f33141537c0 0x7f331415c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:28.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.443+0000 7f33137fe640 1 -- 172.21.15.88:0/2453364944 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266878046 0 0) 0x7f33140cae10 con 0x7f33140ca2f0 2023-10-16T20:46:28.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.443+0000 7f33137fe640 1 -- 172.21.15.88:0/2453364944 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3304003530 con 0x7f33140ca2f0 2023-10-16T20:46:28.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.443+0000 7f3313fff640 1 -- 172.21.15.88:0/2453364944 >> v2:172.21.15.191:3300/0 conn(0x7f33141537c0 msgr2=0x7f331415c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:28.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.443+0000 7f3313fff640 1 --2- 172.21.15.88:0/2453364944 >> v2:172.21.15.191:3300/0 conn(0x7f33141537c0 0x7f331415c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:28.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.443+0000 7f3313fff640 1 -- 172.21.15.88:0/2453364944 >> v1:172.21.15.88:6789/0 conn(0x7f33140ca2f0 legacy=0x7f33140cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:28.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.443+0000 7f3313fff640 1 -- 172.21.15.88:0/2453364944 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331415d590 con 0x7f331415cdb0 2023-10-16T20:46:28.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.443+0000 7f3313fff640 1 --2- 172.21.15.88:0/2453364944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f331415cdb0 0x7f33140c4c80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f33140cfc00 tx=0x7f330001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a43ffc7b38b4a239 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:28.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.443+0000 7f33137fe640 1 -- 172.21.15.88:0/2453364944 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33000a8950 con 0x7f331415cdb0 2023-10-16T20:46:28.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.443+0000 7f33137fe640 1 -- 172.21.15.88:0/2453364944 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33000a8ab0 con 0x7f331415cdb0 2023-10-16T20:46:28.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.443+0000 7f33137fe640 1 -- 172.21.15.88:0/2453364944 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33000a8e40 con 0x7f331415cdb0 2023-10-16T20:46:28.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3319de4640 1 -- 172.21.15.88:0/2453364944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f331415cdb0 msgr2=0x7f33140c4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:28.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3319de4640 1 --2- 172.21.15.88:0/2453364944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f331415cdb0 0x7f33140c4c80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f33140cfc00 tx=0x7f330001ec50 comp rx=0 tx=0).stop 2023-10-16T20:46:28.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3319de4640 1 -- 172.21.15.88:0/2453364944 shutdown_connections 2023-10-16T20:46:28.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3319de4640 1 --2- 172.21.15.88:0/2453364944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f331415cdb0 0x7f33140c4c80 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:28.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3319de4640 1 --2- 172.21.15.88:0/2453364944 >> v2:172.21.15.191:3300/0 conn(0x7f33141537c0 0x7f331415c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:28.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3319de4640 1 -- 172.21.15.88:0/2453364944 >> 172.21.15.88:0/2453364944 conn(0x7f331406c170 msgr2=0x7f33140715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:28.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3319de4640 1 -- 172.21.15.88:0/2453364944 shutdown_connections 2023-10-16T20:46:28.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3319de4640 1 -- 172.21.15.88:0/2453364944 wait complete. 2023-10-16T20:46:28.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3319de4640 1 Processor -- start 2023-10-16T20:46:28.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3319de4640 1 -- start start 2023-10-16T20:46:28.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3319de4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33140ca2f0 0x7f33141e4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:28.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3319de4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33141537c0 0x7f33141f2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:28.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3319de4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f331415cdb0 0x7f33141f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:28.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3319de4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331414d100 con 0x7f33141537c0 2023-10-16T20:46:28.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3319de4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331414e3d0 con 0x7f331415cdb0 2023-10-16T20:46:28.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3319de4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331414cad0 con 0x7f33140ca2f0 2023-10-16T20:46:28.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3313fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33141537c0 0x7f33141f2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:28.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3313fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33141537c0 0x7f33141f2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56700/0 (socket says 172.21.15.88:56700) 2023-10-16T20:46:28.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3313fff640 1 -- 172.21.15.88:0/3664259268 learned_addr learned my addr 172.21.15.88:0/3664259268 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:28.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f3318de2640 1 --2- 172.21.15.88:0/3664259268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33140ca2f0 0x7f33141e4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:28.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.447+0000 7f33195e3640 1 --2- 172.21.15.88:0/3664259268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f331415cdb0 0x7f33141f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:28.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.451+0000 7f3313fff640 1 -- 172.21.15.88:0/3664259268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33140ca2f0 msgr2=0x7f33141e4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:28.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.451+0000 7f3313fff640 1 --2- 172.21.15.88:0/3664259268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33140ca2f0 0x7f33141e4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:28.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.451+0000 7f3313fff640 1 -- 172.21.15.88:0/3664259268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f331415cdb0 msgr2=0x7f33141f2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:28.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.451+0000 7f3313fff640 1 --2- 172.21.15.88:0/3664259268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f331415cdb0 0x7f33141f2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:28.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.451+0000 7f3313fff640 1 -- 172.21.15.88:0/3664259268 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3300009420 con 0x7f33141537c0 2023-10-16T20:46:28.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.451+0000 7f33195e3640 1 --2- 172.21.15.88:0/3664259268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f331415cdb0 0x7f33141f2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:46:28.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.451+0000 7f3313fff640 1 --2- 172.21.15.88:0/3664259268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33141537c0 0x7f33141f2120 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f330001c880 tx=0x7f330001b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:28.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.451+0000 7f33117fa640 1 -- 172.21.15.88:0/3664259268 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33000b0d20 con 0x7f33141537c0 2023-10-16T20:46:28.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.451+0000 7f33117fa640 1 -- 172.21.15.88:0/3664259268 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33000b9430 con 0x7f33141537c0 2023-10-16T20:46:28.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.451+0000 7f33117fa640 1 -- 172.21.15.88:0/3664259268 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33000b0ea0 con 0x7f33141537c0 2023-10-16T20:46:28.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.451+0000 7f3319de4640 1 -- 172.21.15.88:0/3664259268 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331414e600 con 0x7f33141537c0 2023-10-16T20:46:28.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.451+0000 7f3319de4640 1 -- 172.21.15.88:0/3664259268 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331414cd20 con 0x7f33141537c0 2023-10-16T20:46:28.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.451+0000 7f3319de4640 1 -- 172.21.15.88:0/3664259268 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f331414eab0 con 0x7f33141537c0 2023-10-16T20:46:28.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.451+0000 7f33117fa640 1 -- 172.21.15.88:0/3664259268 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f33000b7040 con 0x7f33141537c0 2023-10-16T20:46:28.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.451+0000 7f33117fa640 1 --2- 172.21.15.88:0/3664259268 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f32f80664a0 0x7f32f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:28.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.451+0000 7f33117fa640 1 -- 172.21.15.88:0/3664259268 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7007+0+0 (secure 0 0 0) 0x7f3300121e20 con 0x7f33141537c0 2023-10-16T20:46:28.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.451+0000 7f3318de2640 1 --2- 172.21.15.88:0/3664259268 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f32f80664a0 0x7f32f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:28.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.455+0000 7f3318de2640 1 --2- 172.21.15.88:0/3664259268 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f32f80664a0 0x7f32f8068960 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3308009b60 tx=0x7f3308009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:28.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.455+0000 7f33117fa640 1 -- 172.21.15.88:0/3664259268 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f33000eec60 con 0x7f33141537c0 2023-10-16T20:46:28.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.639+0000 7f3319de4640 1 -- 172.21.15.88:0/3664259268 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33141e4c00 con 0x7f33141537c0 2023-10-16T20:46:28.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.643+0000 7f33117fa640 1 -- 172.21.15.88:0/3664259268 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+5794 (secure 0 0 0) 0x7f33000f2280 con 0x7f33141537c0 2023-10-16T20:46:28.645 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:28.645 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:46:26.508840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:28.645 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 54 2023-10-16T20:46:28.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.643+0000 7f3319de4640 1 -- 172.21.15.88:0/3664259268 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f32f80664a0 msgr2=0x7f32f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:28.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.643+0000 7f3319de4640 1 --2- 172.21.15.88:0/3664259268 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f32f80664a0 0x7f32f8068960 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3308009b60 tx=0x7f3308009190 comp rx=0 tx=0).stop 2023-10-16T20:46:28.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.643+0000 7f3319de4640 1 -- 172.21.15.88:0/3664259268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33141537c0 msgr2=0x7f33141f2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:28.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.643+0000 7f3319de4640 1 --2- 172.21.15.88:0/3664259268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33141537c0 0x7f33141f2120 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f330001c880 tx=0x7f330001b9f0 comp rx=0 tx=0).stop 2023-10-16T20:46:28.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.643+0000 7f3319de4640 1 -- 172.21.15.88:0/3664259268 shutdown_connections 2023-10-16T20:46:28.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.643+0000 7f3319de4640 1 --2- 172.21.15.88:0/3664259268 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f32f80664a0 0x7f32f8068960 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:28.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.643+0000 7f3319de4640 1 --2- 172.21.15.88:0/3664259268 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f331415cdb0 0x7f33141f2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:28.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.643+0000 7f3319de4640 1 --2- 172.21.15.88:0/3664259268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33141537c0 0x7f33141f2120 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:28.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.643+0000 7f3319de4640 1 --2- 172.21.15.88:0/3664259268 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f33140ca2f0 0x7f33141e4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:28.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.643+0000 7f3319de4640 1 -- 172.21.15.88:0/3664259268 >> 172.21.15.88:0/3664259268 conn(0x7f331406c170 msgr2=0x7f33140cb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:28.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.643+0000 7f3319de4640 1 -- 172.21.15.88:0/3664259268 shutdown_connections 2023-10-16T20:46:28.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:28.643+0000 7f3319de4640 1 -- 172.21.15.88:0/3664259268 wait complete. 2023-10-16T20:46:28.666 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 54, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 54, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4279": { "addr": "172.21.15.191:6833/2446850309", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 2446850309, "type": "v2" }, { "addr": "172.21.15.191:6833", "nonce": 2446850309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 40 }, "gid_4288": { "addr": "172.21.15.191:6835/3059699574", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3059699574, "type": "v2" }, { "addr": "172.21.15.191:6835", "nonce": 3059699574, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4889": { "addr": "172.21.15.191:6837/2468647900", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6836", "nonce": 2468647900, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2468647900, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4889, "incarnation": 30, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 21 }, "gid_5284": { "addr": "172.21.15.88:6839/2172973775", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6837", "nonce": 2172973775, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2172973775, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5284, "incarnation": 38, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 17 }, "gid_5535": { "addr": "172.21.15.88:6835/3476086754", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3476086754, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3476086754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5535, "incarnation": 52, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:reconnect", "state_seq": 26 } }, "last_failure": 0, "last_failure_osd_epoch": 50, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:46:26.508840+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 4279, "mds_2": 5535, "mds_3": 5284, "mds_4": 4889 } } } ], "standbys": [] } 2023-10-16T20:46:30.667 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:30.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.815+0000 7ff6d359e640 1 Processor -- start 2023-10-16T20:46:30.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.815+0000 7ff6d359e640 1 -- start start 2023-10-16T20:46:30.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.815+0000 7ff6d359e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6d40cb580 0x7ff6d40cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:30.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.815+0000 7ff6d359e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff6d40b1a60 0x7ff6d4163b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:30.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.815+0000 7ff6d359e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d40c9fc0 con 0x7ff6d40c9af0 2023-10-16T20:46:30.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.815+0000 7ff6d359e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d40b20c0 con 0x7ff6d40b1a60 2023-10-16T20:46:30.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.815+0000 7ff6d359e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d40cbea0 con 0x7ff6d40cb580 2023-10-16T20:46:30.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.815+0000 7ff6d2d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff6d40b1a60 0x7ff6d4163b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:30.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.815+0000 7ff6d2d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff6d40b1a60 0x7ff6d4163b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38892/0 (socket says 172.21.15.88:38892) 2023-10-16T20:46:30.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.815+0000 7ff6d2d9d640 1 -- 172.21.15.88:0/3768295478 learned_addr learned my addr 172.21.15.88:0/3768295478 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:30.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.815+0000 7ff6d259c640 1 --2- 172.21.15.88:0/3768295478 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6d40cb580 0x7ff6d40cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:30.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.815+0000 7ff6d2d9d640 1 -- 172.21.15.88:0/3768295478 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6d40cb580 msgr2=0x7ff6d40cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:30.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.815+0000 7ff6d2d9d640 1 --2- 172.21.15.88:0/3768295478 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6d40cb580 0x7ff6d40cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:30.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.815+0000 7ff6d159a640 1 -- 172.21.15.88:0/3768295478 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595913373 0 0) 0x7ff6d40c9fc0 con 0x7ff6d40c9af0 2023-10-16T20:46:30.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.815+0000 7ff6d2d9d640 1 -- 172.21.15.88:0/3768295478 >> v1:172.21.15.191:6789/0 conn(0x7ff6d40c9af0 legacy=0x7ff6d4159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:30.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.815+0000 7ff6d2d9d640 1 -- 172.21.15.88:0/3768295478 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d40c47d0 con 0x7ff6d40b1a60 2023-10-16T20:46:30.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.819+0000 7ff6d2d9d640 1 --2- 172.21.15.88:0/3768295478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6d40b1a60 0x7ff6d4163b80 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009e50 tx=0x7ff6cc01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81af65f73de32ef7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:30.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.819+0000 7ff6d159a640 1 -- 172.21.15.88:0/3768295478 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc0a5810 con 0x7ff6d40b1a60 2023-10-16T20:46:30.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.819+0000 7ff6d159a640 1 -- 172.21.15.88:0/3768295478 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6cc0a5970 con 0x7ff6d40b1a60 2023-10-16T20:46:30.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.819+0000 7ff6d159a640 1 -- 172.21.15.88:0/3768295478 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc0a5d00 con 0x7ff6d40b1a60 2023-10-16T20:46:30.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.819+0000 7ff6d359e640 1 -- 172.21.15.88:0/3768295478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6d40b1a60 msgr2=0x7ff6d4163b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:30.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.819+0000 7ff6d359e640 1 --2- 172.21.15.88:0/3768295478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6d40b1a60 0x7ff6d4163b80 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009e50 tx=0x7ff6cc01c950 comp rx=0 tx=0).stop 2023-10-16T20:46:30.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.819+0000 7ff6d359e640 1 -- 172.21.15.88:0/3768295478 shutdown_connections 2023-10-16T20:46:30.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.819+0000 7ff6d359e640 1 --2- 172.21.15.88:0/3768295478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6d40b1a60 0x7ff6d4163b80 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:30.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.819+0000 7ff6d359e640 1 --2- 172.21.15.88:0/3768295478 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6d40cb580 0x7ff6d40cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:30.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.819+0000 7ff6d359e640 1 -- 172.21.15.88:0/3768295478 >> 172.21.15.88:0/3768295478 conn(0x7ff6d406bb40 msgr2=0x7ff6d406bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:30.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.819+0000 7ff6d359e640 1 -- 172.21.15.88:0/3768295478 shutdown_connections 2023-10-16T20:46:30.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.819+0000 7ff6d359e640 1 -- 172.21.15.88:0/3768295478 wait complete. 2023-10-16T20:46:30.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d359e640 1 Processor -- start 2023-10-16T20:46:30.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d359e640 1 -- start start 2023-10-16T20:46:30.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d359e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6d40b1a60 0x7ff6d4143070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:30.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d359e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6d40c9af0 0x7ff6d40b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:30.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d359e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6d40cb580 0x7ff6d40b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:30.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d359e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d41f1ab0 con 0x7ff6d40c9af0 2023-10-16T20:46:30.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d359e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d4158ca0 con 0x7ff6d40b1a60 2023-10-16T20:46:30.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d359e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d41f1cc0 con 0x7ff6d40cb580 2023-10-16T20:46:30.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d1d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6d40c9af0 0x7ff6d40b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:30.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d1d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6d40c9af0 0x7ff6d40b4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38900/0 (socket says 172.21.15.88:38900) 2023-10-16T20:46:30.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d1d9b640 1 -- 172.21.15.88:0/3105684220 learned_addr learned my addr 172.21.15.88:0/3105684220 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:30.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d2d9d640 1 --2- 172.21.15.88:0/3105684220 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6d40cb580 0x7ff6d40b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:30.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d259c640 1 --2- 172.21.15.88:0/3105684220 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6d40b1a60 0x7ff6d4143070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:30.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d1d9b640 1 -- 172.21.15.88:0/3105684220 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6d40cb580 msgr2=0x7ff6d40b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:30.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d1d9b640 1 --2- 172.21.15.88:0/3105684220 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6d40cb580 0x7ff6d40b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:30.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d1d9b640 1 -- 172.21.15.88:0/3105684220 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6d40b1a60 msgr2=0x7ff6d4143070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:30.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d1d9b640 1 --2- 172.21.15.88:0/3105684220 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6d40b1a60 0x7ff6d4143070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:30.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d1d9b640 1 -- 172.21.15.88:0/3105684220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6cc009b30 con 0x7ff6d40c9af0 2023-10-16T20:46:30.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d1d9b640 1 --2- 172.21.15.88:0/3105684220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6d40c9af0 0x7ff6d40b4070 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c520 tx=0x7ff6c800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:30.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6c2ffd640 1 -- 172.21.15.88:0/3105684220 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8003600 con 0x7ff6d40c9af0 2023-10-16T20:46:30.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6c2ffd640 1 -- 172.21.15.88:0/3105684220 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c8003780 con 0x7ff6d40c9af0 2023-10-16T20:46:30.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6c2ffd640 1 -- 172.21.15.88:0/3105684220 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8003b30 con 0x7ff6d40c9af0 2023-10-16T20:46:30.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d359e640 1 -- 172.21.15.88:0/3105684220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d41f1fa0 con 0x7ff6d40c9af0 2023-10-16T20:46:30.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.823+0000 7ff6d359e640 1 -- 172.21.15.88:0/3105684220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d41f2560 con 0x7ff6d40c9af0 2023-10-16T20:46:30.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.827+0000 7ff6d359e640 1 -- 172.21.15.88:0/3105684220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d40ca2c0 con 0x7ff6d40c9af0 2023-10-16T20:46:30.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.827+0000 7ff6c2ffd640 1 -- 172.21.15.88:0/3105684220 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff6c8003c90 con 0x7ff6d40c9af0 2023-10-16T20:46:30.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.827+0000 7ff6c2ffd640 1 --2- 172.21.15.88:0/3105684220 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff6b40664d0 0x7ff6b4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:30.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.827+0000 7ff6c2ffd640 1 -- 172.21.15.88:0/3105684220 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7007+0+0 (secure 0 0 0) 0x7ff6c8083930 con 0x7ff6d40c9af0 2023-10-16T20:46:30.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.827+0000 7ff6d259c640 1 --2- 172.21.15.88:0/3105684220 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff6b40664d0 0x7ff6b4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:30.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.827+0000 7ff6d259c640 1 --2- 172.21.15.88:0/3105684220 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff6b40664d0 0x7ff6b4068990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4009ac0 tx=0x7ff6c4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:30.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:30.831+0000 7ff6c2ffd640 1 -- 172.21.15.88:0/3105684220 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff6c8050940 con 0x7ff6d40c9af0 2023-10-16T20:46:31.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:31.011+0000 7ff6d359e640 1 -- 172.21.15.88:0/3105684220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6d404e9c0 con 0x7ff6d40c9af0 2023-10-16T20:46:31.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:31.011+0000 7ff6c2ffd640 1 -- 172.21.15.88:0/3105684220 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+5794 (secure 0 0 0) 0x7ff6c8017020 con 0x7ff6d40c9af0 2023-10-16T20:46:31.015 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:31.015 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:46:26.508840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:31.015 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 54 2023-10-16T20:46:31.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:31.015+0000 7ff6d359e640 1 -- 172.21.15.88:0/3105684220 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff6b40664d0 msgr2=0x7ff6b4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:31.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:31.015+0000 7ff6d359e640 1 --2- 172.21.15.88:0/3105684220 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff6b40664d0 0x7ff6b4068990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4009ac0 tx=0x7ff6c4009190 comp rx=0 tx=0).stop 2023-10-16T20:46:31.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:31.015+0000 7ff6d359e640 1 -- 172.21.15.88:0/3105684220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6d40c9af0 msgr2=0x7ff6d40b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:31.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:31.015+0000 7ff6d359e640 1 --2- 172.21.15.88:0/3105684220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6d40c9af0 0x7ff6d40b4070 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c520 tx=0x7ff6c800c9e0 comp rx=0 tx=0).stop 2023-10-16T20:46:31.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:31.015+0000 7ff6d359e640 1 -- 172.21.15.88:0/3105684220 shutdown_connections 2023-10-16T20:46:31.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:31.015+0000 7ff6d359e640 1 --2- 172.21.15.88:0/3105684220 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff6b40664d0 0x7ff6b4068990 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:31.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:31.015+0000 7ff6d359e640 1 --2- 172.21.15.88:0/3105684220 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6d40cb580 0x7ff6d40b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:31.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:31.015+0000 7ff6d359e640 1 --2- 172.21.15.88:0/3105684220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6d40c9af0 0x7ff6d40b4070 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:31.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:31.015+0000 7ff6d359e640 1 --2- 172.21.15.88:0/3105684220 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6d40b1a60 0x7ff6d4143070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:31.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:31.015+0000 7ff6d359e640 1 -- 172.21.15.88:0/3105684220 >> 172.21.15.88:0/3105684220 conn(0x7ff6d406bb40 msgr2=0x7ff6d40ce0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:31.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:31.015+0000 7ff6d359e640 1 -- 172.21.15.88:0/3105684220 shutdown_connections 2023-10-16T20:46:31.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:31.015+0000 7ff6d359e640 1 -- 172.21.15.88:0/3105684220 wait complete. 2023-10-16T20:46:31.036 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:46:33.037 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:33.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb5453f8640 1 Processor -- start 2023-10-16T20:46:33.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb5453f8640 1 -- start start 2023-10-16T20:46:33.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb5453f8640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb540150660 0x7fb5400c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:33.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb5453f8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb5400c6bc0 0x7fb5400c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:33.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb5453f8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5400c5c60 con 0x7fb540150280 2023-10-16T20:46:33.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb5453f8640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5400c5370 con 0x7fb540150660 2023-10-16T20:46:33.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb5453f8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb540156d10 con 0x7fb5400c6bc0 2023-10-16T20:46:33.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb53effd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb540150280 0x7fb54006d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47214/0 (socket says 172.21.15.88:47214) 2023-10-16T20:46:33.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb53effd640 1 -- 172.21.15.88:0/2052719611 learned_addr learned my addr 172.21.15.88:0/2052719611 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:46:33.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb53e7fc640 1 -- 172.21.15.88:0/2052719611 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341159008 0 0) 0x7fb5400c5c60 con 0x7fb540150280 2023-10-16T20:46:33.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb53f7fe640 1 --2- 172.21.15.88:0/2052719611 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb5400c6bc0 0x7fb5400c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:33.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb5367fc640 1 --2- 172.21.15.88:0/2052719611 >> v2:172.21.15.191:3300/0 conn(0x7fb540150660 0x7fb5400c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:33.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb53e7fc640 1 -- 172.21.15.88:0/2052719611 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb52c003530 con 0x7fb540150280 2023-10-16T20:46:33.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb53e7fc640 1 -- 172.21.15.88:0/2052719611 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1904478495 0 0) 0x7fb52c003530 con 0x7fb540150280 2023-10-16T20:46:33.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb53e7fc640 1 -- 172.21.15.88:0/2052719611 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb5400c6bc0 msgr2=0x7fb5400c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:33.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb53e7fc640 1 --2- 172.21.15.88:0/2052719611 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb5400c6bc0 0x7fb5400c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:33.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb53e7fc640 1 -- 172.21.15.88:0/2052719611 >> v2:172.21.15.191:3300/0 conn(0x7fb540150660 msgr2=0x7fb5400c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:33.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb53e7fc640 1 --2- 172.21.15.88:0/2052719611 >> v2:172.21.15.191:3300/0 conn(0x7fb540150660 0x7fb5400c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:33.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.183+0000 7fb53e7fc640 1 -- 172.21.15.88:0/2052719611 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5400c5610 con 0x7fb540150280 2023-10-16T20:46:33.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb53e7fc640 1 -- 172.21.15.88:0/2052719611 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7fb530001680 con 0x7fb540150280 2023-10-16T20:46:33.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb53e7fc640 1 -- 172.21.15.88:0/2052719611 >> v1:172.21.15.88:6789/0 conn(0x7fb540150280 legacy=0x7fb54006d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:33.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb53e7fc640 1 --2- 172.21.15.88:0/2052719611 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb52c003980 0x7fb52c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:33.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb53e7fc640 1 --2- 172.21.15.88:0/2052719611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52c0047d0 0x7fb52c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:33.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb53e7fc640 1 --2- 172.21.15.88:0/2052719611 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb5400c6bc0 0x7fb52c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:33.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb53e7fc640 1 -- 172.21.15.88:0/2052719611 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400c5370 con 0x7fb52c0047d0 2023-10-16T20:46:33.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb53e7fc640 1 -- 172.21.15.88:0/2052719611 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb540156d10 con 0x7fb52c003980 2023-10-16T20:46:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb53e7fc640 1 -- 172.21.15.88:0/2052719611 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c00dba0 con 0x7fb5400c6bc0 2023-10-16T20:46:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb5367fc640 1 --2- 172.21.15.88:0/2052719611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52c0047d0 0x7fb52c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb53f7fe640 1 --2- 172.21.15.88:0/2052719611 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb5400c6bc0 0x7fb52c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb53effd640 1 --2- 172.21.15.88:0/2052719611 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb52c003980 0x7fb52c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb5367fc640 1 -- 172.21.15.88:0/2052719611 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb5400c6bc0 msgr2=0x7fb52c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb5367fc640 1 --2- 172.21.15.88:0/2052719611 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb5400c6bc0 0x7fb52c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb5367fc640 1 -- 172.21.15.88:0/2052719611 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb52c003980 msgr2=0x7fb52c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb5367fc640 1 --2- 172.21.15.88:0/2052719611 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb52c003980 0x7fb52c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:33.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb5367fc640 1 -- 172.21.15.88:0/2052719611 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52c003530 con 0x7fb52c0047d0 2023-10-16T20:46:33.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb53effd640 1 --2- 172.21.15.88:0/2052719611 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb52c003980 0x7fb52c003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:46:33.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb5367fc640 1 --2- 172.21.15.88:0/2052719611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52c0047d0 0x7fb52c004c50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb528009fc0 tx=0x7fb52801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d7010c29b17649 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:33.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb53e7fc640 1 -- 172.21.15.88:0/2052719611 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5280a5bd0 con 0x7fb52c0047d0 2023-10-16T20:46:33.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb53e7fc640 1 -- 172.21.15.88:0/2052719611 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52800f040 con 0x7fb52c0047d0 2023-10-16T20:46:33.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.187+0000 7fb53e7fc640 1 -- 172.21.15.88:0/2052719611 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5280ad440 con 0x7fb52c0047d0 2023-10-16T20:46:33.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb5453f8640 1 -- 172.21.15.88:0/2052719611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52c0047d0 msgr2=0x7fb52c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:33.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb5453f8640 1 --2- 172.21.15.88:0/2052719611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52c0047d0 0x7fb52c004c50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb528009fc0 tx=0x7fb52801cca0 comp rx=0 tx=0).stop 2023-10-16T20:46:33.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb53f7fe640 1 -- 172.21.15.88:0/2052719611 reap_dead start 2023-10-16T20:46:33.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb5453f8640 1 -- 172.21.15.88:0/2052719611 shutdown_connections 2023-10-16T20:46:33.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb5453f8640 1 -- 172.21.15.88:0/2052719611 >> 172.21.15.88:0/2052719611 conn(0x7fb5400cad50 msgr2=0x7fb5400cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:33.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb5453f8640 1 -- 172.21.15.88:0/2052719611 shutdown_connections 2023-10-16T20:46:33.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb5453f8640 1 -- 172.21.15.88:0/2052719611 wait complete. 2023-10-16T20:46:33.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb5453f8640 1 Processor -- start 2023-10-16T20:46:33.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb5453f8640 1 -- start start 2023-10-16T20:46:33.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb5453f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52c0047d0 0x7fb540150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:33.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb5453f8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb540151170 0x7fb540152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:33.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb5453f8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb540153280 0x7fb5401455c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb5453f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5401485c0 con 0x7fb52c0047d0 2023-10-16T20:46:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb5453f8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5401487a0 con 0x7fb540153280 2023-10-16T20:46:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb5453f8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb540149c00 con 0x7fb540151170 2023-10-16T20:46:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb53effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52c0047d0 0x7fb540150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:33.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb53effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52c0047d0 0x7fb540150c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38910/0 (socket says 172.21.15.88:38910) 2023-10-16T20:46:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb53effd640 1 -- 172.21.15.88:0/3295330049 learned_addr learned my addr 172.21.15.88:0/3295330049 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb5367fc640 1 --2- 172.21.15.88:0/3295330049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb540151170 0x7fb540152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.191+0000 7fb53f7fe640 1 --2- 172.21.15.88:0/3295330049 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb540153280 0x7fb5401455c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.195+0000 7fb53effd640 1 -- 172.21.15.88:0/3295330049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb540151170 msgr2=0x7fb540152d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:33.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.195+0000 7fb53effd640 1 --2- 172.21.15.88:0/3295330049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb540151170 0x7fb540152d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:33.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.195+0000 7fb53effd640 1 -- 172.21.15.88:0/3295330049 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb540153280 msgr2=0x7fb5401455c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:33.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.195+0000 7fb53effd640 1 --2- 172.21.15.88:0/3295330049 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb540153280 0x7fb5401455c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:33.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.195+0000 7fb53effd640 1 -- 172.21.15.88:0/3295330049 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5401515d0 con 0x7fb52c0047d0 2023-10-16T20:46:33.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.195+0000 7fb53effd640 1 --2- 172.21.15.88:0/3295330049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52c0047d0 0x7fb540150c30 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb530011870 tx=0x7fb530011c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:33.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.195+0000 7fb537fff640 1 -- 172.21.15.88:0/3295330049 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53000fb60 con 0x7fb52c0047d0 2023-10-16T20:46:33.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.195+0000 7fb537fff640 1 -- 172.21.15.88:0/3295330049 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53000fcc0 con 0x7fb52c0047d0 2023-10-16T20:46:33.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.195+0000 7fb5453f8640 1 -- 172.21.15.88:0/3295330049 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5400724e0 con 0x7fb52c0047d0 2023-10-16T20:46:33.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.195+0000 7fb537fff640 1 -- 172.21.15.88:0/3295330049 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530011e60 con 0x7fb52c0047d0 2023-10-16T20:46:33.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.195+0000 7fb5453f8640 1 -- 172.21.15.88:0/3295330049 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5401481e0 con 0x7fb52c0047d0 2023-10-16T20:46:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.195+0000 7fb5453f8640 1 -- 172.21.15.88:0/3295330049 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb540074700 con 0x7fb52c0047d0 2023-10-16T20:46:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.195+0000 7fb537fff640 1 -- 172.21.15.88:0/3295330049 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb53001f460 con 0x7fb52c0047d0 2023-10-16T20:46:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.195+0000 7fb537fff640 1 --2- 172.21.15.88:0/3295330049 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb520066430 0x7fb5200688f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:33.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.195+0000 7fb537fff640 1 -- 172.21.15.88:0/3295330049 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7007+0+0 (secure 0 0 0) 0x7fb530087570 con 0x7fb52c0047d0 2023-10-16T20:46:33.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.195+0000 7fb5367fc640 1 --2- 172.21.15.88:0/3295330049 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb520066430 0x7fb5200688f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:33.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.199+0000 7fb5367fc640 1 --2- 172.21.15.88:0/3295330049 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb520066430 0x7fb5200688f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb5280af000 tx=0x7fb5280a7040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:33.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.199+0000 7fb537fff640 1 -- 172.21.15.88:0/3295330049 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb5300544d0 con 0x7fb52c0047d0 2023-10-16T20:46:33.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.379+0000 7fb5453f8640 1 -- 172.21.15.88:0/3295330049 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5400cb3b0 con 0x7fb52c0047d0 2023-10-16T20:46:33.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.383+0000 7fb537fff640 1 -- 172.21.15.88:0/3295330049 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5791 (secure 0 0 0) 0x7fb530057af0 con 0x7fb52c0047d0 2023-10-16T20:46:33.385 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:33.385 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:46:31.530975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:33.385 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 55 2023-10-16T20:46:33.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.383+0000 7fb5453f8640 1 -- 172.21.15.88:0/3295330049 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb520066430 msgr2=0x7fb5200688f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:33.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.383+0000 7fb5453f8640 1 --2- 172.21.15.88:0/3295330049 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb520066430 0x7fb5200688f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb5280af000 tx=0x7fb5280a7040 comp rx=0 tx=0).stop 2023-10-16T20:46:33.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.383+0000 7fb5453f8640 1 -- 172.21.15.88:0/3295330049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52c0047d0 msgr2=0x7fb540150c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:33.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.383+0000 7fb5453f8640 1 --2- 172.21.15.88:0/3295330049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52c0047d0 0x7fb540150c30 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb530011870 tx=0x7fb530011c70 comp rx=0 tx=0).stop 2023-10-16T20:46:33.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.383+0000 7fb5453f8640 1 -- 172.21.15.88:0/3295330049 shutdown_connections 2023-10-16T20:46:33.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.383+0000 7fb5453f8640 1 --2- 172.21.15.88:0/3295330049 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb520066430 0x7fb5200688f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:33.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.383+0000 7fb5453f8640 1 --2- 172.21.15.88:0/3295330049 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb540153280 0x7fb5401455c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:33.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.383+0000 7fb5453f8640 1 --2- 172.21.15.88:0/3295330049 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb540151170 0x7fb540152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:33.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.383+0000 7fb5453f8640 1 --2- 172.21.15.88:0/3295330049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52c0047d0 0x7fb540150c30 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:33.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.383+0000 7fb5453f8640 1 -- 172.21.15.88:0/3295330049 >> 172.21.15.88:0/3295330049 conn(0x7fb5400cad50 msgr2=0x7fb540155f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:33.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.383+0000 7fb5453f8640 1 -- 172.21.15.88:0/3295330049 shutdown_connections 2023-10-16T20:46:33.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:33.383+0000 7fb5453f8640 1 -- 172.21.15.88:0/3295330049 wait complete. 2023-10-16T20:46:33.405 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:46:35.407 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:35.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.559+0000 7fdbb7dc5640 1 Processor -- start 2023-10-16T20:46:35.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.559+0000 7fdbb7dc5640 1 -- start start 2023-10-16T20:46:35.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.559+0000 7fdbb7dc5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdbb00c81a0 0x7fdbb0159450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:35.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.559+0000 7fdbb7dc5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb00c8810 con 0x7fdbb00c76c0 2023-10-16T20:46:35.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.559+0000 7fdbb7dc5640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb006b400 con 0x7fdbb00c7dc0 2023-10-16T20:46:35.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.559+0000 7fdbb7dc5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbb006b0d0 con 0x7fdbb00c81a0 2023-10-16T20:46:35.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.559+0000 7fdbb5360640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdbb00c76c0 0x7fdbb006b890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47222/0 (socket says 172.21.15.88:47222) 2023-10-16T20:46:35.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.559+0000 7fdbb5360640 1 -- 172.21.15.88:0/848368775 learned_addr learned my addr 172.21.15.88:0/848368775 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:46:35.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.559+0000 7fdbb6362640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdbb00c81a0 0x7fdbb0159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:35.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.559+0000 7fdbb4b5f640 1 -- 172.21.15.88:0/848368775 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930449403 0 0) 0x7fdbb00c8810 con 0x7fdbb00c76c0 2023-10-16T20:46:35.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.559+0000 7fdbb4b5f640 1 -- 172.21.15.88:0/848368775 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb9c003530 con 0x7fdbb00c76c0 2023-10-16T20:46:35.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.559+0000 7fdbb6362640 1 -- 172.21.15.88:0/848368775 >> v1:172.21.15.191:6789/0 conn(0x7fdbb00c7dc0 legacy=0x7fdbb006d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:35.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.559+0000 7fdbb4b5f640 1 -- 172.21.15.88:0/848368775 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276603294 0 0) 0x7fdbb006b400 con 0x7fdbb00c7dc0 2023-10-16T20:46:35.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.559+0000 7fdbb6362640 1 -- 172.21.15.88:0/848368775 >> v1:172.21.15.88:6789/0 conn(0x7fdbb00c76c0 legacy=0x7fdbb006b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:35.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.559+0000 7fdbb6362640 1 -- 172.21.15.88:0/848368775 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb006e6a0 con 0x7fdbb00c81a0 2023-10-16T20:46:35.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.563+0000 7fdbb6362640 1 --2- 172.21.15.88:0/848368775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbb00c81a0 0x7fdbb0159450 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fdba8009d90 tx=0x7fdba801c600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd1827b4a56b27a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:35.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.563+0000 7fdbb4b5f640 1 -- 172.21.15.88:0/848368775 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba80a8620 con 0x7fdbb00c81a0 2023-10-16T20:46:35.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.563+0000 7fdbb4b5f640 1 -- 172.21.15.88:0/848368775 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba80a8780 con 0x7fdbb00c81a0 2023-10-16T20:46:35.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.563+0000 7fdbb4b5f640 1 -- 172.21.15.88:0/848368775 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba80a8b10 con 0x7fdbb00c81a0 2023-10-16T20:46:35.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.563+0000 7fdbb7dc5640 1 -- 172.21.15.88:0/848368775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbb00c81a0 msgr2=0x7fdbb0159450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:35.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.563+0000 7fdbb7dc5640 1 --2- 172.21.15.88:0/848368775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbb00c81a0 0x7fdbb0159450 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fdba8009d90 tx=0x7fdba801c600 comp rx=0 tx=0).stop 2023-10-16T20:46:35.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.563+0000 7fdbb7dc5640 1 -- 172.21.15.88:0/848368775 shutdown_connections 2023-10-16T20:46:35.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.563+0000 7fdbb7dc5640 1 --2- 172.21.15.88:0/848368775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbb00c81a0 0x7fdbb0159450 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:35.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.563+0000 7fdbb7dc5640 1 -- 172.21.15.88:0/848368775 >> 172.21.15.88:0/848368775 conn(0x7fdbb0071a60 msgr2=0x7fdbb0071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:35.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.563+0000 7fdbb7dc5640 1 -- 172.21.15.88:0/848368775 shutdown_connections 2023-10-16T20:46:35.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.563+0000 7fdbb7dc5640 1 -- 172.21.15.88:0/848368775 wait complete. 2023-10-16T20:46:35.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb7dc5640 1 Processor -- start 2023-10-16T20:46:35.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb7dc5640 1 -- start start 2023-10-16T20:46:35.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb7dc5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdbb00c76c0 0x7fdbb0151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:35.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb7dc5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbb00c7dc0 0x7fdbb014d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:35.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb7dc5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdbb00c81a0 0x7fdbb014dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:35.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb7dc5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb014e4f0 con 0x7fdbb00c7dc0 2023-10-16T20:46:35.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb7dc5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb00c4d70 con 0x7fdbb00c81a0 2023-10-16T20:46:35.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb7dc5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb00c5050 con 0x7fdbb00c76c0 2023-10-16T20:46:35.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb5360640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbb00c7dc0 0x7fdbb014d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:35.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb5360640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbb00c7dc0 0x7fdbb014d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38920/0 (socket says 172.21.15.88:38920) 2023-10-16T20:46:35.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb5360640 1 -- 172.21.15.88:0/224959336 learned_addr learned my addr 172.21.15.88:0/224959336 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:35.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb5360640 1 -- 172.21.15.88:0/224959336 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdbb00c76c0 msgr2=0x7fdbb0151de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:35.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb5b61640 1 --2- 172.21.15.88:0/224959336 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdbb00c76c0 0x7fdbb0151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:35.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb6362640 1 --2- 172.21.15.88:0/224959336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdbb00c81a0 0x7fdbb014dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:35.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb5360640 1 --2- 172.21.15.88:0/224959336 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdbb00c76c0 0x7fdbb0151de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:35.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb5360640 1 -- 172.21.15.88:0/224959336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdbb00c81a0 msgr2=0x7fdbb014dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:35.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb5360640 1 --2- 172.21.15.88:0/224959336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdbb00c81a0 0x7fdbb014dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:35.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb5360640 1 -- 172.21.15.88:0/224959336 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba8009a70 con 0x7fdbb00c7dc0 2023-10-16T20:46:35.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb5b61640 1 --2- 172.21.15.88:0/224959336 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdbb00c76c0 0x7fdbb0151de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:46:35.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb6362640 1 --2- 172.21.15.88:0/224959336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdbb00c81a0 0x7fdbb014dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:46:35.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb5360640 1 --2- 172.21.15.88:0/224959336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbb00c7dc0 0x7fdbb014d980 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdba000c450 tx=0x7fdba000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:35.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdba67fc640 1 -- 172.21.15.88:0/224959336 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba0005cf0 con 0x7fdbb00c7dc0 2023-10-16T20:46:35.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdba67fc640 1 -- 172.21.15.88:0/224959336 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba0005e50 con 0x7fdbb00c7dc0 2023-10-16T20:46:35.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdba67fc640 1 -- 172.21.15.88:0/224959336 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba0007290 con 0x7fdbb00c7dc0 2023-10-16T20:46:35.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb7dc5640 1 -- 172.21.15.88:0/224959336 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb01f57e0 con 0x7fdbb00c7dc0 2023-10-16T20:46:35.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb7dc5640 1 -- 172.21.15.88:0/224959336 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb00c5910 con 0x7fdbb00c7dc0 2023-10-16T20:46:35.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.567+0000 7fdbb7dc5640 1 -- 172.21.15.88:0/224959336 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb01f5c90 con 0x7fdbb00c7dc0 2023-10-16T20:46:35.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.571+0000 7fdba67fc640 1 -- 172.21.15.88:0/224959336 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdba00073f0 con 0x7fdbb00c7dc0 2023-10-16T20:46:35.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.571+0000 7fdba67fc640 1 --2- 172.21.15.88:0/224959336 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdb940664a0 0x7fdb94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:35.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.571+0000 7fdba67fc640 1 -- 172.21.15.88:0/224959336 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7007+0+0 (secure 0 0 0) 0x7fdba0083980 con 0x7fdbb00c7dc0 2023-10-16T20:46:35.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.571+0000 7fdbb5b61640 1 --2- 172.21.15.88:0/224959336 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdb940664a0 0x7fdb94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:35.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.571+0000 7fdbb5b61640 1 --2- 172.21.15.88:0/224959336 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdb940664a0 0x7fdb94068960 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdbac000f70 tx=0x7fdbac009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:35.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.575+0000 7fdba67fc640 1 -- 172.21.15.88:0/224959336 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdba00508e0 con 0x7fdbb00c7dc0 2023-10-16T20:46:35.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.759+0000 7fdbb7dc5640 1 -- 172.21.15.88:0/224959336 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbb0074700 con 0x7fdbb00c7dc0 2023-10-16T20:46:35.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.759+0000 7fdba67fc640 1 -- 172.21.15.88:0/224959336 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5791 (secure 0 0 0) 0x7fdba0017020 con 0x7fdbb00c7dc0 2023-10-16T20:46:35.762 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:35.762 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:46:31.530975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:35.763 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 55 2023-10-16T20:46:35.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.759+0000 7fdbb7dc5640 1 -- 172.21.15.88:0/224959336 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdb940664a0 msgr2=0x7fdb94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:35.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.759+0000 7fdbb7dc5640 1 --2- 172.21.15.88:0/224959336 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdb940664a0 0x7fdb94068960 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdbac000f70 tx=0x7fdbac009040 comp rx=0 tx=0).stop 2023-10-16T20:46:35.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.759+0000 7fdbb7dc5640 1 -- 172.21.15.88:0/224959336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbb00c7dc0 msgr2=0x7fdbb014d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:35.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.759+0000 7fdbb7dc5640 1 --2- 172.21.15.88:0/224959336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbb00c7dc0 0x7fdbb014d980 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdba000c450 tx=0x7fdba000c910 comp rx=0 tx=0).stop 2023-10-16T20:46:35.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.759+0000 7fdbb7dc5640 1 -- 172.21.15.88:0/224959336 shutdown_connections 2023-10-16T20:46:35.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.759+0000 7fdbb7dc5640 1 --2- 172.21.15.88:0/224959336 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdb940664a0 0x7fdb94068960 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:35.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.759+0000 7fdbb7dc5640 1 --2- 172.21.15.88:0/224959336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdbb00c81a0 0x7fdbb014dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:35.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.759+0000 7fdbb7dc5640 1 --2- 172.21.15.88:0/224959336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbb00c7dc0 0x7fdbb014d980 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:35.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.759+0000 7fdbb7dc5640 1 --2- 172.21.15.88:0/224959336 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdbb00c76c0 0x7fdbb0151de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:35.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.759+0000 7fdbb7dc5640 1 -- 172.21.15.88:0/224959336 >> 172.21.15.88:0/224959336 conn(0x7fdbb0071a60 msgr2=0x7fdbb006b890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:35.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.759+0000 7fdbb7dc5640 1 -- 172.21.15.88:0/224959336 shutdown_connections 2023-10-16T20:46:35.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:35.759+0000 7fdbb7dc5640 1 -- 172.21.15.88:0/224959336 wait complete. 2023-10-16T20:46:35.783 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:46:37.784 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:37.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.931+0000 7f02f8609640 1 Processor -- start 2023-10-16T20:46:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.931+0000 7f02f8609640 1 -- start start 2023-10-16T20:46:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.931+0000 7f02f8609640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f02f0150280 0x7f02f00c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:37.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.931+0000 7f02f8609640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02f00c6bc0 0x7f02f00c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:37.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.931+0000 7f02f8609640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f00c5c60 con 0x7f02f0150660 2023-10-16T20:46:37.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.931+0000 7f02f8609640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02f00c5370 con 0x7f02f0150280 2023-10-16T20:46:37.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.931+0000 7f02f8609640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f0156d10 con 0x7f02f00c6bc0 2023-10-16T20:46:37.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.931+0000 7f02f5ba4640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f02f0150660 0x7f02f006d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47226/0 (socket says 172.21.15.88:47226) 2023-10-16T20:46:37.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.931+0000 7f02f63a5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f02f0150280 0x7f02f00c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:37.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.931+0000 7f02f5ba4640 1 -- 172.21.15.88:0/806262939 learned_addr learned my addr 172.21.15.88:0/806262939 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:46:37.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.931+0000 7f02f6ba6640 1 --2- 172.21.15.88:0/806262939 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02f00c6bc0 0x7f02f00c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:37.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.931+0000 7f02f63a5640 1 -- 172.21.15.88:0/806262939 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02f00c6bc0 msgr2=0x7f02f00c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:37.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.931+0000 7f02f63a5640 1 --2- 172.21.15.88:0/806262939 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02f00c6bc0 0x7f02f00c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:37.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.931+0000 7f02f63a5640 1 -- 172.21.15.88:0/806262939 >> v1:172.21.15.88:6789/0 conn(0x7f02f0150660 legacy=0x7f02f006d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:37.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.931+0000 7f02f63a5640 1 -- 172.21.15.88:0/806262939 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f00c5610 con 0x7f02f0150280 2023-10-16T20:46:37.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.935+0000 7f02f63a5640 1 --2- 172.21.15.88:0/806262939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02f0150280 0x7f02f00c60f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0097c0 tx=0x7f02ec01f0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a537b59fc40dd544 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:37.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.935+0000 7f02f53a3640 1 -- 172.21.15.88:0/806262939 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec0a8650 con 0x7f02f0150280 2023-10-16T20:46:37.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.935+0000 7f02f53a3640 1 -- 172.21.15.88:0/806262939 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02ec0a87b0 con 0x7f02f0150280 2023-10-16T20:46:37.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.935+0000 7f02f53a3640 1 -- 172.21.15.88:0/806262939 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec0a8b40 con 0x7f02f0150280 2023-10-16T20:46:37.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.935+0000 7f02f8609640 1 -- 172.21.15.88:0/806262939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02f0150280 msgr2=0x7f02f00c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:37.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.935+0000 7f02f8609640 1 --2- 172.21.15.88:0/806262939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02f0150280 0x7f02f00c60f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0097c0 tx=0x7f02ec01f0c0 comp rx=0 tx=0).stop 2023-10-16T20:46:37.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.935+0000 7f02f8609640 1 -- 172.21.15.88:0/806262939 shutdown_connections 2023-10-16T20:46:37.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.935+0000 7f02f8609640 1 --2- 172.21.15.88:0/806262939 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02f00c6bc0 0x7f02f00c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:37.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.935+0000 7f02f8609640 1 --2- 172.21.15.88:0/806262939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02f0150280 0x7f02f00c60f0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:37.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.935+0000 7f02f8609640 1 -- 172.21.15.88:0/806262939 >> 172.21.15.88:0/806262939 conn(0x7f02f00cad50 msgr2=0x7f02f00cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:37.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.935+0000 7f02f8609640 1 -- 172.21.15.88:0/806262939 shutdown_connections 2023-10-16T20:46:37.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.935+0000 7f02f8609640 1 -- 172.21.15.88:0/806262939 wait complete. 2023-10-16T20:46:37.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f8609640 1 Processor -- start 2023-10-16T20:46:37.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f8609640 1 -- start start 2023-10-16T20:46:37.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f8609640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02f00c6bc0 0x7f02f01f2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:37.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f8609640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02f0150280 0x7f02f0071cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:37.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f8609640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02f0150660 0x7f02f01f4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:37.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f8609640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f00c99a0 con 0x7f02f0150280 2023-10-16T20:46:37.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f8609640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f01f3200 con 0x7f02f0150660 2023-10-16T20:46:37.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f8609640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f0154370 con 0x7f02f00c6bc0 2023-10-16T20:46:37.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f5ba4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02f0150280 0x7f02f0071cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:37.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f5ba4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02f0150280 0x7f02f0071cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38944/0 (socket says 172.21.15.88:38944) 2023-10-16T20:46:37.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f5ba4640 1 -- 172.21.15.88:0/2091950221 learned_addr learned my addr 172.21.15.88:0/2091950221 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:37.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f6ba6640 1 --2- 172.21.15.88:0/2091950221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02f0150660 0x7f02f01f4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:37.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f63a5640 1 --2- 172.21.15.88:0/2091950221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02f00c6bc0 0x7f02f01f2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:37.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f5ba4640 1 -- 172.21.15.88:0/2091950221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02f00c6bc0 msgr2=0x7f02f01f2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:37.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f5ba4640 1 --2- 172.21.15.88:0/2091950221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02f00c6bc0 0x7f02f01f2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:37.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f5ba4640 1 -- 172.21.15.88:0/2091950221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02f0150660 msgr2=0x7f02f01f4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:37.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f5ba4640 1 --2- 172.21.15.88:0/2091950221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02f0150660 0x7f02f01f4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:37.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f5ba4640 1 -- 172.21.15.88:0/2091950221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ec009420 con 0x7f02f0150280 2023-10-16T20:46:37.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f5ba4640 1 --2- 172.21.15.88:0/2091950221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02f0150280 0x7f02f0071cb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f02e000c400 tx=0x7f02e000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:37.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02e6ffd640 1 -- 172.21.15.88:0/2091950221 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e0005cf0 con 0x7f02f0150280 2023-10-16T20:46:37.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02e6ffd640 1 -- 172.21.15.88:0/2091950221 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e0005e50 con 0x7f02f0150280 2023-10-16T20:46:37.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f8609640 1 -- 172.21.15.88:0/2091950221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f0144310 con 0x7f02f0150280 2023-10-16T20:46:37.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02e6ffd640 1 -- 172.21.15.88:0/2091950221 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e00072d0 con 0x7f02f0150280 2023-10-16T20:46:37.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.939+0000 7f02f8609640 1 -- 172.21.15.88:0/2091950221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f0143d50 con 0x7f02f0150280 2023-10-16T20:46:37.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.943+0000 7f02f8609640 1 -- 172.21.15.88:0/2091950221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f0074700 con 0x7f02f0150280 2023-10-16T20:46:37.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.943+0000 7f02e6ffd640 1 -- 172.21.15.88:0/2091950221 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f02e0007430 con 0x7f02f0150280 2023-10-16T20:46:37.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.943+0000 7f02e6ffd640 1 --2- 172.21.15.88:0/2091950221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02d40664a0 0x7f02d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:37.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.943+0000 7f02e6ffd640 1 -- 172.21.15.88:0/2091950221 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7007+0+0 (secure 0 0 0) 0x7f02e0083860 con 0x7f02f0150280 2023-10-16T20:46:37.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.943+0000 7f02f63a5640 1 --2- 172.21.15.88:0/2091950221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02d40664a0 0x7f02d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:37.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.943+0000 7f02f63a5640 1 --2- 172.21.15.88:0/2091950221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02d40664a0 0x7f02d4068960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0af000 tx=0x7f02ec0ab040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:37.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:37.947+0000 7f02e6ffd640 1 -- 172.21.15.88:0/2091950221 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f02e0050870 con 0x7f02f0150280 2023-10-16T20:46:38.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:38.131+0000 7f02f8609640 1 -- 172.21.15.88:0/2091950221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02f00c1650 con 0x7f02f0150280 2023-10-16T20:46:38.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:38.131+0000 7f02e6ffd640 1 -- 172.21.15.88:0/2091950221 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5797 (secure 0 0 0) 0x7f02e0053e90 con 0x7f02f0150280 2023-10-16T20:46:38.134 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:38.134 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:46:36.563452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:clientreplay","state_seq":30,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:38.135 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 56 2023-10-16T20:46:38.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:38.131+0000 7f02f8609640 1 -- 172.21.15.88:0/2091950221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02d40664a0 msgr2=0x7f02d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:38.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:38.131+0000 7f02f8609640 1 --2- 172.21.15.88:0/2091950221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02d40664a0 0x7f02d4068960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0af000 tx=0x7f02ec0ab040 comp rx=0 tx=0).stop 2023-10-16T20:46:38.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:38.131+0000 7f02f8609640 1 -- 172.21.15.88:0/2091950221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02f0150280 msgr2=0x7f02f0071cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:38.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:38.131+0000 7f02f8609640 1 --2- 172.21.15.88:0/2091950221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02f0150280 0x7f02f0071cb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f02e000c400 tx=0x7f02e000c8c0 comp rx=0 tx=0).stop 2023-10-16T20:46:38.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:38.131+0000 7f02f8609640 1 -- 172.21.15.88:0/2091950221 shutdown_connections 2023-10-16T20:46:38.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:38.131+0000 7f02f8609640 1 --2- 172.21.15.88:0/2091950221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02d40664a0 0x7f02d4068960 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:38.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:38.131+0000 7f02f8609640 1 --2- 172.21.15.88:0/2091950221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02f0150660 0x7f02f01f4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:38.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:38.131+0000 7f02f8609640 1 --2- 172.21.15.88:0/2091950221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02f0150280 0x7f02f0071cb0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:38.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:38.131+0000 7f02f8609640 1 --2- 172.21.15.88:0/2091950221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02f00c6bc0 0x7f02f01f2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:38.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:38.131+0000 7f02f8609640 1 -- 172.21.15.88:0/2091950221 >> 172.21.15.88:0/2091950221 conn(0x7f02f00cad50 msgr2=0x7f02f00c1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:38.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:38.131+0000 7f02f8609640 1 -- 172.21.15.88:0/2091950221 shutdown_connections 2023-10-16T20:46:38.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:38.131+0000 7f02f8609640 1 -- 172.21.15.88:0/2091950221 wait complete. 2023-10-16T20:46:38.154 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:46:40.156 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:40.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.299+0000 7fcb4966b640 1 Processor -- start 2023-10-16T20:46:40.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.299+0000 7fcb4966b640 1 -- start start 2023-10-16T20:46:40.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb4966b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb440c5300 0x7fcb4413e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:40.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb4966b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4415a590 con 0x7fcb440cb890 2023-10-16T20:46:40.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb4966b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb44159d50 con 0x7fcb4415d840 2023-10-16T20:46:40.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb4966b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4415d420 con 0x7fcb440c5300 2023-10-16T20:46:40.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb437fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcb440cb890 0x7fcb4406ffe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:53452/0 (socket says 172.21.15.88:53452) 2023-10-16T20:46:40.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb437fe640 1 -- 172.21.15.88:0/2355391253 learned_addr learned my addr 172.21.15.88:0/2355391253 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:46:40.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb42ffd640 1 --2- 172.21.15.88:0/2355391253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb440c5300 0x7fcb4413e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:40.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb427fc640 1 -- 172.21.15.88:0/2355391253 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888720793 0 0) 0x7fcb4415a590 con 0x7fcb440cb890 2023-10-16T20:46:40.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb427fc640 1 -- 172.21.15.88:0/2355391253 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb30003530 con 0x7fcb440cb890 2023-10-16T20:46:40.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb427fc640 1 -- 172.21.15.88:0/2355391253 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609658619 0 0) 0x7fcb44159d50 con 0x7fcb4415d840 2023-10-16T20:46:40.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb427fc640 1 -- 172.21.15.88:0/2355391253 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb4415a590 con 0x7fcb4415d840 2023-10-16T20:46:40.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb427fc640 1 -- 172.21.15.88:0/2355391253 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2956712487 0 0) 0x7fcb30003530 con 0x7fcb440cb890 2023-10-16T20:46:40.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb427fc640 1 -- 172.21.15.88:0/2355391253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb440c5300 msgr2=0x7fcb4413e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:40.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb427fc640 1 --2- 172.21.15.88:0/2355391253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb440c5300 0x7fcb4413e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:40.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb427fc640 1 -- 172.21.15.88:0/2355391253 >> v1:172.21.15.191:6789/0 conn(0x7fcb4415d840 legacy=0x7fcb44159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:40.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb427fc640 1 -- 172.21.15.88:0/2355391253 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4413ed50 con 0x7fcb440cb890 2023-10-16T20:46:40.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb427fc640 1 -- 172.21.15.88:0/2355391253 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7fcb3c001a00 con 0x7fcb440cb890 2023-10-16T20:46:40.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb427fc640 1 -- 172.21.15.88:0/2355391253 >> v1:172.21.15.88:6789/0 conn(0x7fcb440cb890 legacy=0x7fcb4406ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:40.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb427fc640 1 --2- 172.21.15.88:0/2355391253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb30003b40 0x7fcb30003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:40.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb427fc640 1 --2- 172.21.15.88:0/2355391253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb300047d0 0x7fcb30004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:40.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb427fc640 1 --2- 172.21.15.88:0/2355391253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb30005190 0x7fcb3000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:40.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb427fc640 1 -- 172.21.15.88:0/2355391253 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4415d420 con 0x7fcb30003b40 2023-10-16T20:46:40.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb427fc640 1 -- 172.21.15.88:0/2355391253 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3000bf80 con 0x7fcb300047d0 2023-10-16T20:46:40.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb427fc640 1 -- 172.21.15.88:0/2355391253 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3000c140 con 0x7fcb30005190 2023-10-16T20:46:40.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb42ffd640 1 --2- 172.21.15.88:0/2355391253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb440c5300 0x7fcb4413e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:46:40.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb3bfff640 1 --2- 172.21.15.88:0/2355391253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb300047d0 0x7fcb30004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:40.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb42ffd640 1 --2- 172.21.15.88:0/2355391253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb30003b40 0x7fcb30003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:40.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb437fe640 1 --2- 172.21.15.88:0/2355391253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb30005190 0x7fcb3000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:40.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb42ffd640 1 -- 172.21.15.88:0/2355391253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb30005190 msgr2=0x7fcb3000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:40.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb42ffd640 1 --2- 172.21.15.88:0/2355391253 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb30005190 0x7fcb3000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:40.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb42ffd640 1 -- 172.21.15.88:0/2355391253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb300047d0 msgr2=0x7fcb30004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:40.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb42ffd640 1 --2- 172.21.15.88:0/2355391253 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb300047d0 0x7fcb30004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:40.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.303+0000 7fcb42ffd640 1 -- 172.21.15.88:0/2355391253 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb30003530 con 0x7fcb30003b40 2023-10-16T20:46:40.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.307+0000 7fcb42ffd640 1 --2- 172.21.15.88:0/2355391253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb30003b40 0x7fcb30003f40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcb4406e9e0 tx=0x7fcb3401ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=851ee039c7ac9136 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:40.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.307+0000 7fcb427fc640 1 -- 172.21.15.88:0/2355391253 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34004030 con 0x7fcb30003b40 2023-10-16T20:46:40.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.307+0000 7fcb427fc640 1 -- 172.21.15.88:0/2355391253 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb34004190 con 0x7fcb30003b40 2023-10-16T20:46:40.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.307+0000 7fcb427fc640 1 -- 172.21.15.88:0/2355391253 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34004520 con 0x7fcb30003b40 2023-10-16T20:46:40.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.307+0000 7fcb4966b640 1 -- 172.21.15.88:0/2355391253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb30003b40 msgr2=0x7fcb30003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:40.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.307+0000 7fcb4966b640 1 --2- 172.21.15.88:0/2355391253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb30003b40 0x7fcb30003f40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcb4406e9e0 tx=0x7fcb3401ca90 comp rx=0 tx=0).stop 2023-10-16T20:46:40.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.307+0000 7fcb437fe640 1 -- 172.21.15.88:0/2355391253 reap_dead start 2023-10-16T20:46:40.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.307+0000 7fcb4966b640 1 -- 172.21.15.88:0/2355391253 shutdown_connections 2023-10-16T20:46:40.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.307+0000 7fcb4966b640 1 -- 172.21.15.88:0/2355391253 >> 172.21.15.88:0/2355391253 conn(0x7fcb4406e0f0 msgr2=0x7fcb440cb180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:40.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.307+0000 7fcb4966b640 1 -- 172.21.15.88:0/2355391253 shutdown_connections 2023-10-16T20:46:40.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.307+0000 7fcb4966b640 1 -- 172.21.15.88:0/2355391253 wait complete. 2023-10-16T20:46:40.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb4966b640 1 Processor -- start 2023-10-16T20:46:40.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb4966b640 1 -- start start 2023-10-16T20:46:40.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb4966b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb30003b40 0x7fcb440c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:40.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb4966b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb440c9040 0x7fcb4415af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:40.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb4966b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb4415b480 0x7fcb44137870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:40.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb4966b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb440ca890 con 0x7fcb440c9040 2023-10-16T20:46:40.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb4966b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4415b900 con 0x7fcb4415b480 2023-10-16T20:46:40.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb4966b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb440ca470 con 0x7fcb30003b40 2023-10-16T20:46:40.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb3bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb440c9040 0x7fcb4415af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:40.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb3bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb440c9040 0x7fcb4415af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36130/0 (socket says 172.21.15.88:36130) 2023-10-16T20:46:40.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb3bfff640 1 -- 172.21.15.88:0/2201510314 learned_addr learned my addr 172.21.15.88:0/2201510314 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:40.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb3bfff640 1 -- 172.21.15.88:0/2201510314 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb30003b40 msgr2=0x7fcb440c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:40.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb3bfff640 1 --2- 172.21.15.88:0/2201510314 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb30003b40 0x7fcb440c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:40.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb3bfff640 1 -- 172.21.15.88:0/2201510314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb4415b480 msgr2=0x7fcb44137870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:40.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb3bfff640 1 --2- 172.21.15.88:0/2201510314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb4415b480 0x7fcb44137870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:40.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb3bfff640 1 -- 172.21.15.88:0/2201510314 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb440caae0 con 0x7fcb440c9040 2023-10-16T20:46:40.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb3bfff640 1 --2- 172.21.15.88:0/2201510314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb440c9040 0x7fcb4415af40 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c00f760 tx=0x7fcb2c00fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:40.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb3b7fe640 1 -- 172.21.15.88:0/2201510314 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c00fe70 con 0x7fcb440c9040 2023-10-16T20:46:40.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb3b7fe640 1 -- 172.21.15.88:0/2201510314 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2c0052e0 con 0x7fcb440c9040 2023-10-16T20:46:40.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb4966b640 1 -- 172.21.15.88:0/2201510314 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb441f1bc0 con 0x7fcb440c9040 2023-10-16T20:46:40.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb3b7fe640 1 -- 172.21.15.88:0/2201510314 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c00d8f0 con 0x7fcb440c9040 2023-10-16T20:46:40.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb4966b640 1 -- 172.21.15.88:0/2201510314 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb441f2180 con 0x7fcb440c9040 2023-10-16T20:46:40.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.311+0000 7fcb4966b640 1 -- 172.21.15.88:0/2201510314 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb44074700 con 0x7fcb440c9040 2023-10-16T20:46:40.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.315+0000 7fcb3b7fe640 1 -- 172.21.15.88:0/2201510314 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcb2c00da50 con 0x7fcb440c9040 2023-10-16T20:46:40.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.315+0000 7fcb3b7fe640 1 --2- 172.21.15.88:0/2201510314 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb240664a0 0x7fcb24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:40.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.315+0000 7fcb3b7fe640 1 -- 172.21.15.88:0/2201510314 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7007+0+0 (secure 0 0 0) 0x7fcb2c084460 con 0x7fcb440c9040 2023-10-16T20:46:40.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.315+0000 7fcb42ffd640 1 --2- 172.21.15.88:0/2201510314 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb240664a0 0x7fcb24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:40.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.315+0000 7fcb42ffd640 1 --2- 172.21.15.88:0/2201510314 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb240664a0 0x7fcb24068960 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcb340aa000 tx=0x7fcb34004740 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:40.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.319+0000 7fcb3b7fe640 1 -- 172.21.15.88:0/2201510314 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcb2c051330 con 0x7fcb440c9040 2023-10-16T20:46:40.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.503+0000 7fcb4966b640 1 -- 172.21.15.88:0/2201510314 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb44157580 con 0x7fcb440c9040 2023-10-16T20:46:40.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.503+0000 7fcb3b7fe640 1 -- 172.21.15.88:0/2201510314 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5797 (secure 0 0 0) 0x7fcb2c054950 con 0x7fcb440c9040 2023-10-16T20:46:40.505 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:40.505 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:46:36.563452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:clientreplay","state_seq":30,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:40.505 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 56 2023-10-16T20:46:40.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.503+0000 7fcb4966b640 1 -- 172.21.15.88:0/2201510314 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb240664a0 msgr2=0x7fcb24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:40.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.503+0000 7fcb4966b640 1 --2- 172.21.15.88:0/2201510314 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb240664a0 0x7fcb24068960 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcb340aa000 tx=0x7fcb34004740 comp rx=0 tx=0).stop 2023-10-16T20:46:40.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.503+0000 7fcb4966b640 1 -- 172.21.15.88:0/2201510314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb440c9040 msgr2=0x7fcb4415af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:40.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.503+0000 7fcb4966b640 1 --2- 172.21.15.88:0/2201510314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb440c9040 0x7fcb4415af40 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c00f760 tx=0x7fcb2c00fc20 comp rx=0 tx=0).stop 2023-10-16T20:46:40.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.503+0000 7fcb4966b640 1 -- 172.21.15.88:0/2201510314 shutdown_connections 2023-10-16T20:46:40.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.503+0000 7fcb4966b640 1 --2- 172.21.15.88:0/2201510314 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcb240664a0 0x7fcb24068960 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:40.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.503+0000 7fcb4966b640 1 --2- 172.21.15.88:0/2201510314 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcb4415b480 0x7fcb44137870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:40.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.503+0000 7fcb4966b640 1 --2- 172.21.15.88:0/2201510314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb440c9040 0x7fcb4415af40 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:40.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.503+0000 7fcb4966b640 1 --2- 172.21.15.88:0/2201510314 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcb30003b40 0x7fcb440c7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:40.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.503+0000 7fcb4966b640 1 -- 172.21.15.88:0/2201510314 >> 172.21.15.88:0/2201510314 conn(0x7fcb4406e0f0 msgr2=0x7fcb4413bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:40.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.503+0000 7fcb4966b640 1 -- 172.21.15.88:0/2201510314 shutdown_connections 2023-10-16T20:46:40.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:40.503+0000 7fcb4966b640 1 -- 172.21.15.88:0/2201510314 wait complete. 2023-10-16T20:46:40.526 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 56, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 56, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4279": { "addr": "172.21.15.191:6833/2446850309", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 2446850309, "type": "v2" }, { "addr": "172.21.15.191:6833", "nonce": 2446850309, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:active", "state_seq": 40 }, "gid_4288": { "addr": "172.21.15.191:6835/3059699574", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3059699574, "type": "v2" }, { "addr": "172.21.15.191:6835", "nonce": 3059699574, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4889": { "addr": "172.21.15.191:6837/2468647900", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6836", "nonce": 2468647900, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2468647900, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4889, "incarnation": 30, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 21 }, "gid_5284": { "addr": "172.21.15.88:6839/2172973775", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6837", "nonce": 2172973775, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2172973775, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5284, "incarnation": 38, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 17 }, "gid_5535": { "addr": "172.21.15.88:6835/3476086754", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3476086754, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3476086754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5535, "incarnation": 52, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:clientreplay", "state_seq": 30 } }, "last_failure": 0, "last_failure_osd_epoch": 50, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:46:36.563452+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 4279, "mds_2": 5535, "mds_3": 5284, "mds_4": 4889 } } } ], "standbys": [] } 2023-10-16T20:46:42.527 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:42.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.687+0000 7f0a4b259640 1 Processor -- start 2023-10-16T20:46:42.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.687+0000 7f0a4b259640 1 -- start start 2023-10-16T20:46:42.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.691+0000 7f0a4b259640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0a44150160 0x7f0a44159c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:42.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.691+0000 7f0a4b259640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a441594a0 con 0x7f0a440c37e0 2023-10-16T20:46:42.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.691+0000 7f0a4b259640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a440c20f0 con 0x7f0a440c6e00 2023-10-16T20:46:42.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.691+0000 7f0a4b259640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a440c23d0 con 0x7f0a44150160 2023-10-16T20:46:42.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.691+0000 7f0a4a257640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0a44150160 0x7f0a44159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:42.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.691+0000 7f0a4aa58640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0a440c37e0 0x7f0a44160830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:53464/0 (socket says 172.21.15.88:53464) 2023-10-16T20:46:42.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.691+0000 7f0a4aa58640 1 -- 172.21.15.88:0/1448339333 learned_addr learned my addr 172.21.15.88:0/1448339333 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:46:42.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.691+0000 7f0a49255640 1 -- 172.21.15.88:0/1448339333 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391385724 0 0) 0x7f0a441594a0 con 0x7f0a440c37e0 2023-10-16T20:46:42.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.691+0000 7f0a49255640 1 -- 172.21.15.88:0/1448339333 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a30003530 con 0x7f0a440c37e0 2023-10-16T20:46:42.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.691+0000 7f0a49255640 1 -- 172.21.15.88:0/1448339333 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601723605 0 0) 0x7f0a440c20f0 con 0x7f0a440c6e00 2023-10-16T20:46:42.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.691+0000 7f0a4a257640 1 -- 172.21.15.88:0/1448339333 >> v1:172.21.15.191:6789/0 conn(0x7f0a440c6e00 legacy=0x7f0a440c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:42.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.691+0000 7f0a4a257640 1 -- 172.21.15.88:0/1448339333 >> v1:172.21.15.88:6789/0 conn(0x7f0a440c37e0 legacy=0x7f0a44160830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:42.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.691+0000 7f0a4a257640 1 -- 172.21.15.88:0/1448339333 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a440c26b0 con 0x7f0a44150160 2023-10-16T20:46:42.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.695+0000 7f0a4a257640 1 --2- 172.21.15.88:0/1448339333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a44150160 0x7f0a44159c80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0a400097c0 tx=0x7f0a4001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f7180f12bcb3c37 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:42.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.695+0000 7f0a49255640 1 -- 172.21.15.88:0/1448339333 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a400ab680 con 0x7f0a44150160 2023-10-16T20:46:42.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.695+0000 7f0a49255640 1 -- 172.21.15.88:0/1448339333 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a400ab7e0 con 0x7f0a44150160 2023-10-16T20:46:42.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.695+0000 7f0a49255640 1 -- 172.21.15.88:0/1448339333 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a400abb70 con 0x7f0a44150160 2023-10-16T20:46:42.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.695+0000 7f0a4b259640 1 -- 172.21.15.88:0/1448339333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a44150160 msgr2=0x7f0a44159c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:42.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.695+0000 7f0a4b259640 1 --2- 172.21.15.88:0/1448339333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a44150160 0x7f0a44159c80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0a400097c0 tx=0x7f0a4001c480 comp rx=0 tx=0).stop 2023-10-16T20:46:42.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.695+0000 7f0a4b259640 1 -- 172.21.15.88:0/1448339333 shutdown_connections 2023-10-16T20:46:42.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.695+0000 7f0a4b259640 1 --2- 172.21.15.88:0/1448339333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a44150160 0x7f0a44159c80 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:42.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.695+0000 7f0a4b259640 1 -- 172.21.15.88:0/1448339333 >> 172.21.15.88:0/1448339333 conn(0x7f0a4406d2d0 msgr2=0x7f0a4406d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:42.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.695+0000 7f0a4b259640 1 -- 172.21.15.88:0/1448339333 shutdown_connections 2023-10-16T20:46:42.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.695+0000 7f0a4b259640 1 -- 172.21.15.88:0/1448339333 wait complete. 2023-10-16T20:46:42.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.695+0000 7f0a4b259640 1 Processor -- start 2023-10-16T20:46:42.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.695+0000 7f0a4b259640 1 -- start start 2023-10-16T20:46:42.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a4b259640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a440c37e0 0x7f0a44155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:42.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a4b259640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a440c6e00 0x7f0a44154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:42.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a4b259640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a44150160 0x7f0a440c98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:42.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a4b259640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4413ef20 con 0x7f0a440c6e00 2023-10-16T20:46:42.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a4b259640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a44154c60 con 0x7f0a440c37e0 2023-10-16T20:46:42.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a4b259640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4413e030 con 0x7f0a44150160 2023-10-16T20:46:42.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a49a56640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a440c6e00 0x7f0a44154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:42.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a49a56640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a440c6e00 0x7f0a44154720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36158/0 (socket says 172.21.15.88:36158) 2023-10-16T20:46:42.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a49a56640 1 -- 172.21.15.88:0/1772903645 learned_addr learned my addr 172.21.15.88:0/1772903645 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:42.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a4aa58640 1 --2- 172.21.15.88:0/1772903645 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a44150160 0x7f0a440c98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:42.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a4a257640 1 --2- 172.21.15.88:0/1772903645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a440c37e0 0x7f0a44155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:42.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a49a56640 1 -- 172.21.15.88:0/1772903645 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a44150160 msgr2=0x7f0a440c98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:42.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a49a56640 1 --2- 172.21.15.88:0/1772903645 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a44150160 0x7f0a440c98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:42.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a49a56640 1 -- 172.21.15.88:0/1772903645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a440c37e0 msgr2=0x7f0a44155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:42.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a49a56640 1 --2- 172.21.15.88:0/1772903645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a440c37e0 0x7f0a44155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:42.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a49a56640 1 -- 172.21.15.88:0/1772903645 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a40009420 con 0x7f0a440c6e00 2023-10-16T20:46:42.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a49a56640 1 --2- 172.21.15.88:0/1772903645 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a440c6e00 0x7f0a44154720 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400c3f0 tx=0x7f0a3400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:42.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a3affd640 1 -- 172.21.15.88:0/1772903645 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a340054e0 con 0x7f0a440c6e00 2023-10-16T20:46:42.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a3affd640 1 -- 172.21.15.88:0/1772903645 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a34003600 con 0x7f0a440c6e00 2023-10-16T20:46:42.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a3affd640 1 -- 172.21.15.88:0/1772903645 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34003990 con 0x7f0a440c6e00 2023-10-16T20:46:42.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a4b259640 1 -- 172.21.15.88:0/1772903645 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a4413e310 con 0x7f0a440c6e00 2023-10-16T20:46:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a4b259640 1 -- 172.21.15.88:0/1772903645 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a4413e8d0 con 0x7f0a440c6e00 2023-10-16T20:46:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a4b259640 1 -- 172.21.15.88:0/1772903645 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4404e9c0 con 0x7f0a440c6e00 2023-10-16T20:46:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.699+0000 7f0a3affd640 1 -- 172.21.15.88:0/1772903645 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0a34003af0 con 0x7f0a440c6e00 2023-10-16T20:46:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.703+0000 7f0a3affd640 1 --2- 172.21.15.88:0/1772903645 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0a280664a0 0x7f0a28068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:42.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.703+0000 7f0a3affd640 1 -- 172.21.15.88:0/1772903645 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7007+0+0 (secure 0 0 0) 0x7f0a34083a10 con 0x7f0a440c6e00 2023-10-16T20:46:42.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.703+0000 7f0a4a257640 1 --2- 172.21.15.88:0/1772903645 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0a280664a0 0x7f0a28068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:42.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.703+0000 7f0a4a257640 1 --2- 172.21.15.88:0/1772903645 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0a280664a0 0x7f0a28068960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0a400af000 tx=0x7f0a400a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:42.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.707+0000 7f0a3affd640 1 -- 172.21.15.88:0/1772903645 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0a34050900 con 0x7f0a440c6e00 2023-10-16T20:46:42.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.887+0000 7f0a4b259640 1 -- 172.21.15.88:0/1772903645 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a440c1420 con 0x7f0a440c6e00 2023-10-16T20:46:42.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.887+0000 7f0a3affd640 1 -- 172.21.15.88:0/1772903645 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+5797 (secure 0 0 0) 0x7f0a34017020 con 0x7f0a440c6e00 2023-10-16T20:46:42.891 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:42.891 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:46:36.563452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:clientreplay","state_seq":30,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:42.891 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 56 2023-10-16T20:46:42.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.891+0000 7f0a4b259640 1 -- 172.21.15.88:0/1772903645 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0a280664a0 msgr2=0x7f0a28068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:42.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.891+0000 7f0a4b259640 1 --2- 172.21.15.88:0/1772903645 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0a280664a0 0x7f0a28068960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0a400af000 tx=0x7f0a400a7000 comp rx=0 tx=0).stop 2023-10-16T20:46:42.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.891+0000 7f0a4b259640 1 -- 172.21.15.88:0/1772903645 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a440c6e00 msgr2=0x7f0a44154720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:42.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.891+0000 7f0a4b259640 1 --2- 172.21.15.88:0/1772903645 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a440c6e00 0x7f0a44154720 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400c3f0 tx=0x7f0a3400c8b0 comp rx=0 tx=0).stop 2023-10-16T20:46:42.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.891+0000 7f0a4b259640 1 -- 172.21.15.88:0/1772903645 shutdown_connections 2023-10-16T20:46:42.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.891+0000 7f0a4b259640 1 --2- 172.21.15.88:0/1772903645 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0a280664a0 0x7f0a28068960 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:42.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.891+0000 7f0a4b259640 1 --2- 172.21.15.88:0/1772903645 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0a44150160 0x7f0a440c98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:42.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.891+0000 7f0a4b259640 1 --2- 172.21.15.88:0/1772903645 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a440c6e00 0x7f0a44154720 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:42.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.891+0000 7f0a4b259640 1 --2- 172.21.15.88:0/1772903645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a440c37e0 0x7f0a44155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:42.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.891+0000 7f0a4b259640 1 -- 172.21.15.88:0/1772903645 >> 172.21.15.88:0/1772903645 conn(0x7f0a4406d2d0 msgr2=0x7f0a440c59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:42.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.891+0000 7f0a4b259640 1 -- 172.21.15.88:0/1772903645 shutdown_connections 2023-10-16T20:46:42.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:42.891+0000 7f0a4b259640 1 -- 172.21.15.88:0/1772903645 wait complete. 2023-10-16T20:46:42.911 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:46:44.911 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:44.959 INFO:tasks.ceph.mds.f.smithi191.stderr:2023-10-16T20:46:44.956+0000 7fe37f078640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-16T20:46:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.059+0000 7f3a1c279640 1 Processor -- start 2023-10-16T20:46:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.059+0000 7f3a1c279640 1 -- start start 2023-10-16T20:46:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.059+0000 7f3a1c279640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a14153770 0x7f3a14159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.059+0000 7f3a1c279640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3a140cece0 0x7f3a140cf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.059+0000 7f3a1c279640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1415a410 con 0x7f3a140c9bb0 2023-10-16T20:46:45.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.059+0000 7f3a1c279640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a140cc250 con 0x7f3a140cece0 2023-10-16T20:46:45.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.059+0000 7f3a1c279640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a140cc670 con 0x7f3a14153770 2023-10-16T20:46:45.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.059+0000 7f3a1a015640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a14153770 0x7f3a14159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:45.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.059+0000 7f3a1a015640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a14153770 0x7f3a14159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:56668/0 (socket says 172.21.15.88:56668) 2023-10-16T20:46:45.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.059+0000 7f3a1a015640 1 -- 172.21.15.88:0/2686789756 learned_addr learned my addr 172.21.15.88:0/2686789756 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:45.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.059+0000 7f3a1a816640 1 --2- 172.21.15.88:0/2686789756 >> v2:172.21.15.191:3300/0 conn(0x7f3a140cece0 0x7f3a140cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:45.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.059+0000 7f3a1a816640 1 -- 172.21.15.88:0/2686789756 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a14153770 msgr2=0x7f3a14159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:45.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.059+0000 7f3a1a816640 1 --2- 172.21.15.88:0/2686789756 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a14153770 0x7f3a14159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:45.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.059+0000 7f3a1a816640 1 -- 172.21.15.88:0/2686789756 >> v1:172.21.15.191:6789/0 conn(0x7f3a140c9bb0 legacy=0x7f3a140c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:45.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.059+0000 7f3a1a816640 1 -- 172.21.15.88:0/2686789756 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a14153b50 con 0x7f3a140cece0 2023-10-16T20:46:45.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.059+0000 7f3a1a015640 1 --2- 172.21.15.88:0/2686789756 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a14153770 0x7f3a14159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:46:45.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.063+0000 7f3a1a816640 1 --2- 172.21.15.88:0/2686789756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a140cece0 0x7f3a140cf170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009d90 tx=0x7f3a0c01c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5b462a474a20c56 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:45.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.063+0000 7f3a19013640 1 -- 172.21.15.88:0/2686789756 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c0a5a10 con 0x7f3a140cece0 2023-10-16T20:46:45.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.063+0000 7f3a19013640 1 -- 172.21.15.88:0/2686789756 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0c0a5b70 con 0x7f3a140cece0 2023-10-16T20:46:45.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.063+0000 7f3a19013640 1 -- 172.21.15.88:0/2686789756 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c0ad440 con 0x7f3a140cece0 2023-10-16T20:46:45.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.063+0000 7f3a1c279640 1 -- 172.21.15.88:0/2686789756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a140cece0 msgr2=0x7f3a140cf170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:45.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.063+0000 7f3a1c279640 1 --2- 172.21.15.88:0/2686789756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a140cece0 0x7f3a140cf170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009d90 tx=0x7f3a0c01c950 comp rx=0 tx=0).stop 2023-10-16T20:46:45.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.063+0000 7f3a1c279640 1 -- 172.21.15.88:0/2686789756 shutdown_connections 2023-10-16T20:46:45.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.063+0000 7f3a1c279640 1 --2- 172.21.15.88:0/2686789756 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a140cece0 0x7f3a140cf170 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:45.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.063+0000 7f3a1c279640 1 --2- 172.21.15.88:0/2686789756 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a14153770 0x7f3a14159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:45.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.063+0000 7f3a1c279640 1 -- 172.21.15.88:0/2686789756 >> 172.21.15.88:0/2686789756 conn(0x7f3a140715c0 msgr2=0x7f3a140719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:45.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.063+0000 7f3a1c279640 1 -- 172.21.15.88:0/2686789756 shutdown_connections 2023-10-16T20:46:45.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.063+0000 7f3a1c279640 1 -- 172.21.15.88:0/2686789756 wait complete. 2023-10-16T20:46:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1c279640 1 Processor -- start 2023-10-16T20:46:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1c279640 1 -- start start 2023-10-16T20:46:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1c279640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a140c9bb0 0x7f3a14148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1c279640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a140cece0 0x7f3a1414c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1c279640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a14153770 0x7f3a1414ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1c279640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1414d600 con 0x7f3a140c9bb0 2023-10-16T20:46:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1c279640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1415a610 con 0x7f3a140cece0 2023-10-16T20:46:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1c279640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1414ba90 con 0x7f3a14153770 2023-10-16T20:46:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1a015640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a140c9bb0 0x7f3a14148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1a015640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a140c9bb0 0x7f3a14148340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36162/0 (socket says 172.21.15.88:36162) 2023-10-16T20:46:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1a015640 1 -- 172.21.15.88:0/3739808566 learned_addr learned my addr 172.21.15.88:0/3739808566 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1a816640 1 --2- 172.21.15.88:0/3739808566 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a14153770 0x7f3a1414ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a19814640 1 --2- 172.21.15.88:0/3739808566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a140cece0 0x7f3a1414c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1a015640 1 -- 172.21.15.88:0/3739808566 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a14153770 msgr2=0x7f3a1414ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1a015640 1 --2- 172.21.15.88:0/3739808566 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a14153770 0x7f3a1414ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1a015640 1 -- 172.21.15.88:0/3739808566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a140cece0 msgr2=0x7f3a1414c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1a015640 1 --2- 172.21.15.88:0/3739808566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a140cece0 0x7f3a1414c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1a015640 1 -- 172.21.15.88:0/3739808566 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a10009480 con 0x7f3a140c9bb0 2023-10-16T20:46:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1a015640 1 --2- 172.21.15.88:0/3739808566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a140c9bb0 0x7f3a14148340 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3a1000ebb0 tx=0x7f3a1000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a0affd640 1 -- 172.21.15.88:0/3739808566 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1000eeb0 con 0x7f3a140c9bb0 2023-10-16T20:46:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a0affd640 1 -- 172.21.15.88:0/3739808566 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1000cc30 con 0x7f3a140c9bb0 2023-10-16T20:46:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a0affd640 1 -- 172.21.15.88:0/3739808566 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a100059a0 con 0x7f3a140c9bb0 2023-10-16T20:46:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1c279640 1 -- 172.21.15.88:0/3739808566 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a0c009a70 con 0x7f3a140c9bb0 2023-10-16T20:46:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.067+0000 7f3a1c279640 1 -- 172.21.15.88:0/3739808566 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a1414bd70 con 0x7f3a140c9bb0 2023-10-16T20:46:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.071+0000 7f3a1c279640 1 -- 172.21.15.88:0/3739808566 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a14074700 con 0x7f3a140c9bb0 2023-10-16T20:46:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.071+0000 7f3a0affd640 1 -- 172.21.15.88:0/3739808566 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3a10005b00 con 0x7f3a140c9bb0 2023-10-16T20:46:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.071+0000 7f3a0affd640 1 --2- 172.21.15.88:0/3739808566 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f39f80664a0 0x7f39f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.071+0000 7f3a0affd640 1 -- 172.21.15.88:0/3739808566 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7007+0+0 (secure 0 0 0) 0x7f3a10084930 con 0x7f3a140c9bb0 2023-10-16T20:46:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.071+0000 7f3a19814640 1 --2- 172.21.15.88:0/3739808566 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f39f80664a0 0x7f39f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:45.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.071+0000 7f3a19814640 1 --2- 172.21.15.88:0/3739808566 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f39f80664a0 0x7f39f8068960 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3a14150c10 tx=0x7f3a04009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:45.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.075+0000 7f3a0affd640 1 -- 172.21.15.88:0/3739808566 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3a100518b0 con 0x7f3a140c9bb0 2023-10-16T20:46:45.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.255+0000 7f3a1c279640 1 -- 172.21.15.88:0/3739808566 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a140ca740 con 0x7f3a140c9bb0 2023-10-16T20:46:45.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.255+0000 7f3a0affd640 1 -- 172.21.15.88:0/3739808566 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5791 (secure 0 0 0) 0x7f3a10054ed0 con 0x7f3a140c9bb0 2023-10-16T20:46:45.257 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:45.258 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:46:42.588210+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:45.258 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 57 2023-10-16T20:46:45.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.255+0000 7f3a1c279640 1 -- 172.21.15.88:0/3739808566 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f39f80664a0 msgr2=0x7f39f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:45.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.255+0000 7f3a1c279640 1 --2- 172.21.15.88:0/3739808566 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f39f80664a0 0x7f39f8068960 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3a14150c10 tx=0x7f3a04009040 comp rx=0 tx=0).stop 2023-10-16T20:46:45.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.255+0000 7f3a1c279640 1 -- 172.21.15.88:0/3739808566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a140c9bb0 msgr2=0x7f3a14148340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:45.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.255+0000 7f3a1c279640 1 --2- 172.21.15.88:0/3739808566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a140c9bb0 0x7f3a14148340 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3a1000ebb0 tx=0x7f3a1000c520 comp rx=0 tx=0).stop 2023-10-16T20:46:45.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.255+0000 7f3a1c279640 1 -- 172.21.15.88:0/3739808566 shutdown_connections 2023-10-16T20:46:45.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.255+0000 7f3a1c279640 1 --2- 172.21.15.88:0/3739808566 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f39f80664a0 0x7f39f8068960 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:45.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.255+0000 7f3a1c279640 1 --2- 172.21.15.88:0/3739808566 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3a14153770 0x7f3a1414ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:45.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.255+0000 7f3a1c279640 1 --2- 172.21.15.88:0/3739808566 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a140cece0 0x7f3a1414c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:45.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.255+0000 7f3a1c279640 1 --2- 172.21.15.88:0/3739808566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a140c9bb0 0x7f3a14148340 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:45.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.255+0000 7f3a1c279640 1 -- 172.21.15.88:0/3739808566 >> 172.21.15.88:0/3739808566 conn(0x7f3a140715c0 msgr2=0x7f3a140c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:45.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.255+0000 7f3a1c279640 1 -- 172.21.15.88:0/3739808566 shutdown_connections 2023-10-16T20:46:45.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:45.255+0000 7f3a1c279640 1 -- 172.21.15.88:0/3739808566 wait complete. 2023-10-16T20:46:45.277 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=2, replacing gid=4275 2023-10-16T20:46:45.278 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.e 2023-10-16T20:46:49.750 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/21' 2023-10-16T20:46:49.750 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/27' 2023-10-16T20:46:49.750 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/31' 2023-10-16T20:46:49.750 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/36' 2023-10-16T20:46:49.751 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/20' 2023-10-16T20:46:49.751 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/39' 2023-10-16T20:46:49.751 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/34' 2023-10-16T20:46:49.751 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/38' 2023-10-16T20:46:49.751 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/28' 2023-10-16T20:46:49.752 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/24' 2023-10-16T20:46:49.752 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/29' 2023-10-16T20:46:49.752 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/32' 2023-10-16T20:46:49.752 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/22' 2023-10-16T20:46:49.752 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/25' 2023-10-16T20:46:49.752 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/30' 2023-10-16T20:46:49.753 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/33' 2023-10-16T20:46:49.753 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/23' 2023-10-16T20:46:49.753 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/35' 2023-10-16T20:46:49.753 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/26' 2023-10-16T20:46:49.753 INFO:tasks.workunit.client.0.smithi088.stdout:removed './287/37' 2023-10-16T20:46:49.754 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './287' 2023-10-16T20:46:49.754 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/21' 2023-10-16T20:46:49.754 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/27' 2023-10-16T20:46:49.754 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/31' 2023-10-16T20:46:49.754 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/36' 2023-10-16T20:46:49.755 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/20' 2023-10-16T20:46:49.755 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/39' 2023-10-16T20:46:49.755 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/34' 2023-10-16T20:46:49.755 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/38' 2023-10-16T20:46:49.755 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/28' 2023-10-16T20:46:49.755 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/24' 2023-10-16T20:46:49.756 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/29' 2023-10-16T20:46:49.756 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/32' 2023-10-16T20:46:49.756 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/22' 2023-10-16T20:46:49.756 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/25' 2023-10-16T20:46:49.756 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/30' 2023-10-16T20:46:49.756 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/33' 2023-10-16T20:46:49.757 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/23' 2023-10-16T20:46:49.757 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/35' 2023-10-16T20:46:49.757 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/26' 2023-10-16T20:46:49.757 INFO:tasks.workunit.client.0.smithi088.stdout:removed './288/37' 2023-10-16T20:46:49.757 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './288' 2023-10-16T20:46:49.758 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/21' 2023-10-16T20:46:49.758 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/27' 2023-10-16T20:46:49.758 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/31' 2023-10-16T20:46:49.758 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/36' 2023-10-16T20:46:49.758 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/20' 2023-10-16T20:46:49.759 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/39' 2023-10-16T20:46:49.759 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/34' 2023-10-16T20:46:49.759 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/38' 2023-10-16T20:46:49.759 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/28' 2023-10-16T20:46:49.759 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/24' 2023-10-16T20:46:49.759 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/29' 2023-10-16T20:46:49.760 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/32' 2023-10-16T20:46:49.760 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/22' 2023-10-16T20:46:49.760 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/25' 2023-10-16T20:46:49.760 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/30' 2023-10-16T20:46:49.760 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/33' 2023-10-16T20:46:49.761 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/23' 2023-10-16T20:46:49.761 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/35' 2023-10-16T20:46:49.761 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/26' 2023-10-16T20:46:49.761 INFO:tasks.workunit.client.0.smithi088.stdout:removed './289/37' 2023-10-16T20:46:49.761 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './289' 2023-10-16T20:46:49.761 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/21' 2023-10-16T20:46:49.762 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/27' 2023-10-16T20:46:49.762 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/31' 2023-10-16T20:46:49.762 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/36' 2023-10-16T20:46:49.762 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/20' 2023-10-16T20:46:49.762 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/39' 2023-10-16T20:46:49.763 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/34' 2023-10-16T20:46:49.763 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/38' 2023-10-16T20:46:49.763 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/28' 2023-10-16T20:46:49.763 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/24' 2023-10-16T20:46:49.763 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/29' 2023-10-16T20:46:49.764 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/32' 2023-10-16T20:46:49.764 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/22' 2023-10-16T20:46:49.764 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/25' 2023-10-16T20:46:49.764 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/30' 2023-10-16T20:46:49.764 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/33' 2023-10-16T20:46:49.764 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/23' 2023-10-16T20:46:49.765 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/35' 2023-10-16T20:46:49.765 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/26' 2023-10-16T20:46:49.765 INFO:tasks.workunit.client.0.smithi088.stdout:removed './290/37' 2023-10-16T20:46:49.765 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './290' 2023-10-16T20:46:49.765 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/21' 2023-10-16T20:46:49.766 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/27' 2023-10-16T20:46:49.766 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/31' 2023-10-16T20:46:49.766 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/36' 2023-10-16T20:46:49.766 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/20' 2023-10-16T20:46:49.766 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/39' 2023-10-16T20:46:49.767 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/34' 2023-10-16T20:46:49.767 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/38' 2023-10-16T20:46:49.767 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/28' 2023-10-16T20:46:49.767 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/24' 2023-10-16T20:46:49.767 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/29' 2023-10-16T20:46:49.767 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/32' 2023-10-16T20:46:49.768 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/22' 2023-10-16T20:46:49.768 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/25' 2023-10-16T20:46:49.768 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/30' 2023-10-16T20:46:49.768 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/33' 2023-10-16T20:46:49.768 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/23' 2023-10-16T20:46:49.769 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/35' 2023-10-16T20:46:49.769 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/26' 2023-10-16T20:46:49.769 INFO:tasks.workunit.client.0.smithi088.stdout:removed './291/37' 2023-10-16T20:46:49.769 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './291' 2023-10-16T20:46:49.769 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/21' 2023-10-16T20:46:49.769 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/27' 2023-10-16T20:46:49.770 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/31' 2023-10-16T20:46:49.770 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/36' 2023-10-16T20:46:49.770 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/20' 2023-10-16T20:46:49.770 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/39' 2023-10-16T20:46:49.770 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/34' 2023-10-16T20:46:49.771 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/38' 2023-10-16T20:46:49.771 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/28' 2023-10-16T20:46:49.771 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/24' 2023-10-16T20:46:49.771 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/29' 2023-10-16T20:46:49.771 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/32' 2023-10-16T20:46:49.772 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/22' 2023-10-16T20:46:49.772 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/25' 2023-10-16T20:46:49.772 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/30' 2023-10-16T20:46:49.772 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/33' 2023-10-16T20:46:49.772 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/23' 2023-10-16T20:46:49.772 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/35' 2023-10-16T20:46:49.773 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/26' 2023-10-16T20:46:49.773 INFO:tasks.workunit.client.0.smithi088.stdout:removed './292/37' 2023-10-16T20:46:49.773 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './292' 2023-10-16T20:46:49.773 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/21' 2023-10-16T20:46:49.773 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/27' 2023-10-16T20:46:49.774 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/31' 2023-10-16T20:46:49.774 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/36' 2023-10-16T20:46:49.774 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/20' 2023-10-16T20:46:49.774 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/39' 2023-10-16T20:46:49.774 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/34' 2023-10-16T20:46:49.774 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/38' 2023-10-16T20:46:49.775 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/28' 2023-10-16T20:46:49.775 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/24' 2023-10-16T20:46:49.775 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/29' 2023-10-16T20:46:49.775 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/32' 2023-10-16T20:46:49.775 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/22' 2023-10-16T20:46:49.775 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/25' 2023-10-16T20:46:49.776 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/30' 2023-10-16T20:46:49.776 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/33' 2023-10-16T20:46:49.776 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/23' 2023-10-16T20:46:49.776 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/35' 2023-10-16T20:46:49.776 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/26' 2023-10-16T20:46:49.776 INFO:tasks.workunit.client.0.smithi088.stdout:removed './293/37' 2023-10-16T20:46:49.777 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './293' 2023-10-16T20:46:49.777 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/21' 2023-10-16T20:46:49.777 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/27' 2023-10-16T20:46:49.777 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/31' 2023-10-16T20:46:49.777 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/36' 2023-10-16T20:46:49.777 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/20' 2023-10-16T20:46:49.778 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/39' 2023-10-16T20:46:49.778 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/34' 2023-10-16T20:46:49.778 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/38' 2023-10-16T20:46:49.778 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/28' 2023-10-16T20:46:49.778 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/24' 2023-10-16T20:46:49.779 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/29' 2023-10-16T20:46:49.779 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/32' 2023-10-16T20:46:49.779 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/22' 2023-10-16T20:46:49.779 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/25' 2023-10-16T20:46:49.779 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/30' 2023-10-16T20:46:49.780 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/33' 2023-10-16T20:46:49.780 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/23' 2023-10-16T20:46:49.780 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/35' 2023-10-16T20:46:49.780 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/26' 2023-10-16T20:46:49.780 INFO:tasks.workunit.client.0.smithi088.stdout:removed './294/37' 2023-10-16T20:46:49.780 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './294' 2023-10-16T20:46:49.781 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/21' 2023-10-16T20:46:49.781 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/27' 2023-10-16T20:46:49.781 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/31' 2023-10-16T20:46:49.781 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/36' 2023-10-16T20:46:49.781 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/20' 2023-10-16T20:46:49.781 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/39' 2023-10-16T20:46:49.782 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/34' 2023-10-16T20:46:49.782 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/38' 2023-10-16T20:46:49.782 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/28' 2023-10-16T20:46:49.782 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/24' 2023-10-16T20:46:49.782 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/29' 2023-10-16T20:46:49.783 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/32' 2023-10-16T20:46:49.783 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/22' 2023-10-16T20:46:49.783 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/25' 2023-10-16T20:46:49.783 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/30' 2023-10-16T20:46:49.783 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/33' 2023-10-16T20:46:49.783 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/23' 2023-10-16T20:46:49.784 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/35' 2023-10-16T20:46:49.784 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/26' 2023-10-16T20:46:49.784 INFO:tasks.workunit.client.0.smithi088.stdout:removed './295/37' 2023-10-16T20:46:49.784 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './295' 2023-10-16T20:46:49.784 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/21' 2023-10-16T20:46:49.784 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/27' 2023-10-16T20:46:49.785 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/31' 2023-10-16T20:46:49.785 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/36' 2023-10-16T20:46:49.785 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/20' 2023-10-16T20:46:49.785 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/39' 2023-10-16T20:46:49.785 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/34' 2023-10-16T20:46:49.786 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/38' 2023-10-16T20:46:49.786 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/28' 2023-10-16T20:46:49.786 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/24' 2023-10-16T20:46:49.786 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/29' 2023-10-16T20:46:49.786 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/32' 2023-10-16T20:46:49.786 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/22' 2023-10-16T20:46:49.787 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/25' 2023-10-16T20:46:49.787 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/30' 2023-10-16T20:46:49.787 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/33' 2023-10-16T20:46:49.787 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/23' 2023-10-16T20:46:49.787 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/35' 2023-10-16T20:46:49.787 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/26' 2023-10-16T20:46:49.788 INFO:tasks.workunit.client.0.smithi088.stdout:removed './296/37' 2023-10-16T20:46:49.788 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './296' 2023-10-16T20:46:49.788 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/21' 2023-10-16T20:46:49.788 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/27' 2023-10-16T20:46:52.279 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-16T20:46:52.279 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-16T20:46:52.280 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-16T20:46:52.284 INFO:tasks.ceph.mds.e:Started 2023-10-16T20:46:52.284 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:46:52.371 INFO:tasks.ceph.mds.e.smithi088.stdout:starting mds.e at 2023-10-16T20:46:52.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.423+0000 7faf1bdf8640 1 Processor -- start 2023-10-16T20:46:52.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.423+0000 7faf1bdf8640 1 -- start start 2023-10-16T20:46:52.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.423+0000 7faf1bdf8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf14150280 0x7faf140c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:52.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.423+0000 7faf1bdf8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faf140c6bc0 0x7faf140c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:52.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.423+0000 7faf1bdf8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf140c5c60 con 0x7faf14150660 2023-10-16T20:46:52.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.423+0000 7faf1bdf8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf140c5370 con 0x7faf140c6bc0 2023-10-16T20:46:52.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.423+0000 7faf1bdf8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf14156d10 con 0x7faf14150280 2023-10-16T20:46:52.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.423+0000 7faf1a395640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faf140c6bc0 0x7faf140c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:52.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.423+0000 7faf1a395640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faf140c6bc0 0x7faf140c3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50538/0 (socket says 172.21.15.88:50538) 2023-10-16T20:46:52.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.423+0000 7faf1a395640 1 -- 172.21.15.88:0/59974832 learned_addr learned my addr 172.21.15.88:0/59974832 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:52.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.423+0000 7faf19b94640 1 --2- 172.21.15.88:0/59974832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf14150280 0x7faf140c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:52.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.423+0000 7faf1a395640 1 -- 172.21.15.88:0/59974832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf14150280 msgr2=0x7faf140c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:46:52.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.423+0000 7faf1a395640 1 --2- 172.21.15.88:0/59974832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf14150280 0x7faf140c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:52.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.423+0000 7faf1a395640 1 -- 172.21.15.88:0/59974832 >> v1:172.21.15.191:6789/0 conn(0x7faf14150660 legacy=0x7faf1406d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:52.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.423+0000 7faf1a395640 1 -- 172.21.15.88:0/59974832 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf140c5610 con 0x7faf140c6bc0 2023-10-16T20:46:52.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.427+0000 7faf1a395640 1 --2- 172.21.15.88:0/59974832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf140c6bc0 0x7faf140c3110 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7faf0c009e50 tx=0x7faf0c01d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950c125dc2e5cd9a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:52.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.427+0000 7faf18b92640 1 -- 172.21.15.88:0/59974832 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c0a5830 con 0x7faf140c6bc0 2023-10-16T20:46:52.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.427+0000 7faf18b92640 1 -- 172.21.15.88:0/59974832 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0c0a5990 con 0x7faf140c6bc0 2023-10-16T20:46:52.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.427+0000 7faf18b92640 1 -- 172.21.15.88:0/59974832 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c0a5d20 con 0x7faf140c6bc0 2023-10-16T20:46:52.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.427+0000 7faf1bdf8640 1 -- 172.21.15.88:0/59974832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf140c6bc0 msgr2=0x7faf140c3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:52.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.427+0000 7faf1bdf8640 1 --2- 172.21.15.88:0/59974832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf140c6bc0 0x7faf140c3110 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7faf0c009e50 tx=0x7faf0c01d3b0 comp rx=0 tx=0).stop 2023-10-16T20:46:52.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.427+0000 7faf1bdf8640 1 -- 172.21.15.88:0/59974832 shutdown_connections 2023-10-16T20:46:52.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.427+0000 7faf1bdf8640 1 --2- 172.21.15.88:0/59974832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf140c6bc0 0x7faf140c3110 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:52.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.427+0000 7faf1bdf8640 1 --2- 172.21.15.88:0/59974832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf14150280 0x7faf140c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:52.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.427+0000 7faf1bdf8640 1 -- 172.21.15.88:0/59974832 >> 172.21.15.88:0/59974832 conn(0x7faf140cad50 msgr2=0x7faf140cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:52.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.427+0000 7faf1bdf8640 1 -- 172.21.15.88:0/59974832 shutdown_connections 2023-10-16T20:46:52.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.427+0000 7faf1bdf8640 1 -- 172.21.15.88:0/59974832 wait complete. 2023-10-16T20:46:52.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf1bdf8640 1 Processor -- start 2023-10-16T20:46:52.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf1bdf8640 1 -- start start 2023-10-16T20:46:52.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf1bdf8640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf140c6bc0 0x7faf14142c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:52.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf1bdf8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf14150280 0x7faf1413e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:52.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf1bdf8640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf14150660 0x7faf141f1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:52.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf1bdf8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf141f1890 con 0x7faf14150280 2023-10-16T20:46:52.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf1bdf8640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf141401e0 con 0x7faf14150660 2023-10-16T20:46:52.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf1bdf8640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf140ca6b0 con 0x7faf140c6bc0 2023-10-16T20:46:52.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf19393640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf14150280 0x7faf1413e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:52.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf19393640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf14150280 0x7faf1413e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50542/0 (socket says 172.21.15.88:50542) 2023-10-16T20:46:52.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf19b94640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf140c6bc0 0x7faf14142c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:52.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf19393640 1 -- 172.21.15.88:0/1076635926 learned_addr learned my addr 172.21.15.88:0/1076635926 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:46:52.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf1a395640 1 --2- 172.21.15.88:0/1076635926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf14150660 0x7faf141f1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:52.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf19393640 1 -- 172.21.15.88:0/1076635926 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf140c6bc0 msgr2=0x7faf14142c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:52.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf19393640 1 --2- 172.21.15.88:0/1076635926 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf140c6bc0 0x7faf14142c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:52.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf19393640 1 -- 172.21.15.88:0/1076635926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf14150660 msgr2=0x7faf141f1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:52.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf19393640 1 --2- 172.21.15.88:0/1076635926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf14150660 0x7faf141f1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:52.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf19393640 1 -- 172.21.15.88:0/1076635926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0c009b30 con 0x7faf14150280 2023-10-16T20:46:52.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf19393640 1 --2- 172.21.15.88:0/1076635926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf14150280 0x7faf1413e320 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7faf0400c3f0 tx=0x7faf0400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:52.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf0a7fc640 1 -- 172.21.15.88:0/1076635926 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf040052d0 con 0x7faf14150280 2023-10-16T20:46:52.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf0a7fc640 1 -- 172.21.15.88:0/1076635926 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf04005430 con 0x7faf14150280 2023-10-16T20:46:52.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf1bdf8640 1 -- 172.21.15.88:0/1076635926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf140c9b70 con 0x7faf14150280 2023-10-16T20:46:52.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf0a7fc640 1 -- 172.21.15.88:0/1076635926 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf04003840 con 0x7faf14150280 2023-10-16T20:46:52.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.431+0000 7faf1bdf8640 1 -- 172.21.15.88:0/1076635926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf140ca0f0 con 0x7faf14150280 2023-10-16T20:46:52.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.435+0000 7faf1bdf8640 1 -- 172.21.15.88:0/1076635926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf14074700 con 0x7faf14150280 2023-10-16T20:46:52.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.435+0000 7faf0a7fc640 1 -- 172.21.15.88:0/1076635926 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faf040039a0 con 0x7faf14150280 2023-10-16T20:46:52.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.435+0000 7faf0a7fc640 1 --2- 172.21.15.88:0/1076635926 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faef80664a0 0x7faef8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:46:52.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.435+0000 7faf0a7fc640 1 -- 172.21.15.88:0/1076635926 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7007+0+0 (secure 0 0 0) 0x7faf04083830 con 0x7faf14150280 2023-10-16T20:46:52.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.435+0000 7faf19b94640 1 --2- 172.21.15.88:0/1076635926 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faef80664a0 0x7faef8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:46:52.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.435+0000 7faf19b94640 1 --2- 172.21.15.88:0/1076635926 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faef80664a0 0x7faef8068960 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7faf10009b60 tx=0x7faf10009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:46:52.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.439+0000 7faf0a7fc640 1 -- 172.21.15.88:0/1076635926 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faf04050840 con 0x7faf14150280 2023-10-16T20:46:52.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.627+0000 7faf1bdf8640 1 -- 172.21.15.88:0/1076635926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf140c3850 con 0x7faf14150280 2023-10-16T20:46:52.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.627+0000 7faf0a7fc640 1 -- 172.21.15.88:0/1076635926 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+6581 (secure 0 0 0) 0x7faf04053e60 con 0x7faf14150280 2023-10-16T20:46:52.632 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:46:52.632 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5709,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:46:51.514159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:46:52.633 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 59 2023-10-16T20:46:52.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.631+0000 7faf1bdf8640 1 -- 172.21.15.88:0/1076635926 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faef80664a0 msgr2=0x7faef8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:52.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.631+0000 7faf1bdf8640 1 --2- 172.21.15.88:0/1076635926 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faef80664a0 0x7faef8068960 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7faf10009b60 tx=0x7faf10009190 comp rx=0 tx=0).stop 2023-10-16T20:46:52.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.631+0000 7faf1bdf8640 1 -- 172.21.15.88:0/1076635926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf14150280 msgr2=0x7faf1413e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:46:52.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.631+0000 7faf1bdf8640 1 --2- 172.21.15.88:0/1076635926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf14150280 0x7faf1413e320 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7faf0400c3f0 tx=0x7faf0400c8b0 comp rx=0 tx=0).stop 2023-10-16T20:46:52.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.631+0000 7faf1bdf8640 1 -- 172.21.15.88:0/1076635926 shutdown_connections 2023-10-16T20:46:52.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.631+0000 7faf1bdf8640 1 --2- 172.21.15.88:0/1076635926 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7faef80664a0 0x7faef8068960 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:52.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.631+0000 7faf1bdf8640 1 --2- 172.21.15.88:0/1076635926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf14150660 0x7faf141f1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:52.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.631+0000 7faf1bdf8640 1 --2- 172.21.15.88:0/1076635926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf14150280 0x7faf1413e320 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:52.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.631+0000 7faf1bdf8640 1 --2- 172.21.15.88:0/1076635926 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7faf140c6bc0 0x7faf14142c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:46:52.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.631+0000 7faf1bdf8640 1 -- 172.21.15.88:0/1076635926 >> 172.21.15.88:0/1076635926 conn(0x7faf140cad50 msgr2=0x7faf140c1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:46:52.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.631+0000 7faf1bdf8640 1 -- 172.21.15.88:0/1076635926 shutdown_connections 2023-10-16T20:46:52.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:46:52.631+0000 7faf1bdf8640 1 -- 172.21.15.88:0/1076635926 wait complete. 2023-10-16T20:46:52.652 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-16T20:46:52.652 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 95 secs before thrashing 2023-10-16T20:46:53.545 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/31' 2023-10-16T20:46:53.546 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/36' 2023-10-16T20:46:53.546 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/20' 2023-10-16T20:46:53.546 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/39' 2023-10-16T20:46:53.546 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/34' 2023-10-16T20:46:53.546 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/38' 2023-10-16T20:46:53.547 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/28' 2023-10-16T20:46:53.547 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/24' 2023-10-16T20:46:53.547 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/29' 2023-10-16T20:46:53.547 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/32' 2023-10-16T20:46:53.547 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/22' 2023-10-16T20:46:53.548 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/25' 2023-10-16T20:46:53.549 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/30' 2023-10-16T20:46:53.549 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/33' 2023-10-16T20:46:53.549 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/23' 2023-10-16T20:46:53.550 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/35' 2023-10-16T20:46:53.550 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/26' 2023-10-16T20:46:53.550 INFO:tasks.workunit.client.0.smithi088.stdout:removed './297/37' 2023-10-16T20:46:53.550 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './297' 2023-10-16T20:46:53.550 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/21' 2023-10-16T20:46:53.551 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/27' 2023-10-16T20:46:53.551 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/31' 2023-10-16T20:46:53.551 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/36' 2023-10-16T20:46:53.551 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/20' 2023-10-16T20:46:53.551 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/39' 2023-10-16T20:46:53.552 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/34' 2023-10-16T20:46:53.552 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/38' 2023-10-16T20:46:53.552 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/28' 2023-10-16T20:46:53.552 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/24' 2023-10-16T20:46:53.552 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/29' 2023-10-16T20:46:53.552 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/32' 2023-10-16T20:46:53.553 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/22' 2023-10-16T20:46:53.553 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/25' 2023-10-16T20:46:53.553 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/30' 2023-10-16T20:46:53.553 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/33' 2023-10-16T20:46:53.553 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/23' 2023-10-16T20:46:53.554 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/35' 2023-10-16T20:46:53.554 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/26' 2023-10-16T20:46:53.554 INFO:tasks.workunit.client.0.smithi088.stdout:removed './298/37' 2023-10-16T20:46:53.554 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './298' 2023-10-16T20:46:53.554 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/21' 2023-10-16T20:46:53.555 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/27' 2023-10-16T20:46:53.555 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/31' 2023-10-16T20:46:53.555 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/36' 2023-10-16T20:46:53.555 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/20' 2023-10-16T20:46:53.555 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/39' 2023-10-16T20:46:53.556 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/34' 2023-10-16T20:46:53.556 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/38' 2023-10-16T20:46:53.556 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/28' 2023-10-16T20:46:53.556 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/24' 2023-10-16T20:46:53.556 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/29' 2023-10-16T20:46:53.556 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/32' 2023-10-16T20:46:53.557 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/22' 2023-10-16T20:46:53.557 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/25' 2023-10-16T20:46:53.557 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/30' 2023-10-16T20:46:53.557 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/33' 2023-10-16T20:46:53.557 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/23' 2023-10-16T20:46:53.557 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/35' 2023-10-16T20:46:53.558 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/26' 2023-10-16T20:46:53.558 INFO:tasks.workunit.client.0.smithi088.stdout:removed './299/37' 2023-10-16T20:46:53.558 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './299' 2023-10-16T20:46:53.558 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/166/19' 2023-10-16T20:46:53.558 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/166/15' 2023-10-16T20:46:53.558 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/166/20' 2023-10-16T20:46:53.559 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/166/12' 2023-10-16T20:46:53.559 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/166/10' 2023-10-16T20:46:53.559 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/166/11' 2023-10-16T20:46:53.559 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/166/16' 2023-10-16T20:46:53.559 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/166/14' 2023-10-16T20:46:53.559 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/166/18' 2023-10-16T20:46:53.560 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/166/13' 2023-10-16T20:46:53.560 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/166/17' 2023-10-16T20:46:53.560 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/166' 2023-10-16T20:46:53.560 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/147/19' 2023-10-16T20:46:53.560 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/147/15' 2023-10-16T20:46:53.561 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/147/20' 2023-10-16T20:46:53.561 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/147/12' 2023-10-16T20:46:53.561 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/147/10' 2023-10-16T20:46:53.561 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/147/11' 2023-10-16T20:46:53.561 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/147/16' 2023-10-16T20:46:53.562 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/147/14' 2023-10-16T20:46:53.562 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/147/18' 2023-10-16T20:46:53.562 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/147/13' 2023-10-16T20:46:53.562 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/147/17' 2023-10-16T20:46:53.562 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/147' 2023-10-16T20:46:53.563 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/151/19' 2023-10-16T20:46:53.563 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/151/15' 2023-10-16T20:46:53.563 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/151/20' 2023-10-16T20:46:53.563 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/151/12' 2023-10-16T20:46:53.563 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/151/10' 2023-10-16T20:46:53.563 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/151/11' 2023-10-16T20:46:53.564 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/151/16' 2023-10-16T20:46:53.564 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/151/14' 2023-10-16T20:46:53.564 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/151/18' 2023-10-16T20:46:53.564 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/151/13' 2023-10-16T20:46:53.564 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/151/17' 2023-10-16T20:46:53.565 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/151' 2023-10-16T20:46:53.565 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/170/19' 2023-10-16T20:46:53.565 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/170/15' 2023-10-16T20:46:53.565 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/170/20' 2023-10-16T20:46:53.565 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/170/12' 2023-10-16T20:46:53.565 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/170/10' 2023-10-16T20:46:53.566 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/170/11' 2023-10-16T20:46:53.566 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/170/16' 2023-10-16T20:46:53.566 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/170/14' 2023-10-16T20:46:53.566 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/170/18' 2023-10-16T20:46:53.566 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/170/13' 2023-10-16T20:46:53.567 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/170/17' 2023-10-16T20:46:53.567 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/170' 2023-10-16T20:46:53.567 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/167/19' 2023-10-16T20:46:53.567 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/167/15' 2023-10-16T20:46:53.567 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/167/20' 2023-10-16T20:46:53.568 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/167/12' 2023-10-16T20:46:53.568 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/167/10' 2023-10-16T20:46:53.568 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/167/11' 2023-10-16T20:46:53.568 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/167/16' 2023-10-16T20:46:53.568 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/167/14' 2023-10-16T20:46:53.568 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/167/18' 2023-10-16T20:46:53.569 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/167/13' 2023-10-16T20:46:53.569 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/167/17' 2023-10-16T20:46:53.569 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/167' 2023-10-16T20:46:53.569 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/140/19' 2023-10-16T20:46:53.569 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/140/15' 2023-10-16T20:46:53.570 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/140/20' 2023-10-16T20:46:53.570 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/140/12' 2023-10-16T20:46:53.570 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/140/10' 2023-10-16T20:46:53.570 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/140/11' 2023-10-16T20:46:53.570 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/140/16' 2023-10-16T20:46:53.571 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/140/14' 2023-10-16T20:46:53.571 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/140/18' 2023-10-16T20:46:53.571 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/140/13' 2023-10-16T20:46:53.571 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/140/17' 2023-10-16T20:46:53.571 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/140' 2023-10-16T20:46:53.571 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/142/19' 2023-10-16T20:46:53.572 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/142/15' 2023-10-16T20:46:53.572 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/142/20' 2023-10-16T20:46:53.572 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/142/12' 2023-10-16T20:46:53.572 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/142/10' 2023-10-16T20:46:57.064 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/142/11' 2023-10-16T20:46:57.065 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/142/16' 2023-10-16T20:46:57.065 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/142/14' 2023-10-16T20:46:57.065 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/142/18' 2023-10-16T20:46:57.065 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/142/13' 2023-10-16T20:46:57.065 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/142/17' 2023-10-16T20:46:57.066 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/142' 2023-10-16T20:46:57.066 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/150/19' 2023-10-16T20:46:57.066 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/150/15' 2023-10-16T20:46:57.066 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/150/20' 2023-10-16T20:46:57.067 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/150/12' 2023-10-16T20:46:57.067 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/150/10' 2023-10-16T20:46:57.067 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/150/11' 2023-10-16T20:46:57.067 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/150/16' 2023-10-16T20:46:57.068 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/150/14' 2023-10-16T20:46:57.068 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/150/18' 2023-10-16T20:46:57.068 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/150/13' 2023-10-16T20:46:57.068 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/150/17' 2023-10-16T20:46:57.068 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/150' 2023-10-16T20:46:57.068 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/105/19' 2023-10-16T20:46:57.069 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/105/15' 2023-10-16T20:46:57.069 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/105/20' 2023-10-16T20:46:57.069 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/105/12' 2023-10-16T20:46:57.069 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/105/10' 2023-10-16T20:46:57.070 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/105/11' 2023-10-16T20:46:57.070 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/105/16' 2023-10-16T20:46:57.070 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/105/14' 2023-10-16T20:46:57.070 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/105/18' 2023-10-16T20:46:57.070 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/105/13' 2023-10-16T20:46:57.070 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/105/17' 2023-10-16T20:46:57.071 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/105' 2023-10-16T20:46:57.071 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/149/19' 2023-10-16T20:46:57.071 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/149/15' 2023-10-16T20:46:57.071 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/149/20' 2023-10-16T20:46:57.071 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/149/12' 2023-10-16T20:46:57.072 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/149/10' 2023-10-16T20:46:57.072 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/149/11' 2023-10-16T20:46:57.072 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/149/16' 2023-10-16T20:46:57.072 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/149/14' 2023-10-16T20:46:57.072 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/149/18' 2023-10-16T20:46:57.073 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/149/13' 2023-10-16T20:46:57.073 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/149/17' 2023-10-16T20:46:57.073 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/149' 2023-10-16T20:46:57.073 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/109/19' 2023-10-16T20:46:57.073 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/109/15' 2023-10-16T20:46:57.074 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/109/20' 2023-10-16T20:46:57.074 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/109/12' 2023-10-16T20:46:57.074 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/109/10' 2023-10-16T20:46:57.074 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/109/11' 2023-10-16T20:46:57.074 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/109/16' 2023-10-16T20:46:57.075 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/109/14' 2023-10-16T20:46:57.075 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/109/18' 2023-10-16T20:46:57.075 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/109/13' 2023-10-16T20:46:57.075 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/109/17' 2023-10-16T20:46:57.076 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/109' 2023-10-16T20:46:57.076 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/158/19' 2023-10-16T20:46:57.076 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/158/15' 2023-10-16T20:46:57.076 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/158/20' 2023-10-16T20:46:57.076 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/158/12' 2023-10-16T20:46:57.077 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/158/10' 2023-10-16T20:46:57.077 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/158/11' 2023-10-16T20:46:57.077 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/158/16' 2023-10-16T20:46:57.077 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/158/14' 2023-10-16T20:46:57.077 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/158/18' 2023-10-16T20:46:57.077 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/158/13' 2023-10-16T20:46:57.078 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/158/17' 2023-10-16T20:46:57.078 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/158' 2023-10-16T20:46:57.078 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/106/19' 2023-10-16T20:46:57.078 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/106/15' 2023-10-16T20:46:57.079 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/106/20' 2023-10-16T20:46:57.079 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/106/12' 2023-10-16T20:46:57.079 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/106/10' 2023-10-16T20:46:57.079 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/106/11' 2023-10-16T20:46:57.079 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/106/16' 2023-10-16T20:46:57.080 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/106/14' 2023-10-16T20:46:57.080 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/106/18' 2023-10-16T20:46:57.080 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/106/13' 2023-10-16T20:46:57.080 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/106/17' 2023-10-16T20:46:57.080 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/106' 2023-10-16T20:46:57.081 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/189/19' 2023-10-16T20:46:57.081 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/189/15' 2023-10-16T20:46:57.081 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/189/20' 2023-10-16T20:46:57.081 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/189/12' 2023-10-16T20:46:57.081 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/189/10' 2023-10-16T20:46:57.082 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/189/11' 2023-10-16T20:46:57.082 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/189/16' 2023-10-16T20:46:57.082 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/189/14' 2023-10-16T20:46:57.082 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/189/18' 2023-10-16T20:46:57.082 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/189/13' 2023-10-16T20:46:57.083 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/189/17' 2023-10-16T20:46:57.083 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/189' 2023-10-16T20:46:57.083 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/133/19' 2023-10-16T20:46:57.083 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/133/15' 2023-10-16T20:46:57.083 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/133/20' 2023-10-16T20:46:57.083 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/133/12' 2023-10-16T20:46:57.084 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/133/10' 2023-10-16T20:46:57.084 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/133/11' 2023-10-16T20:46:57.084 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/133/16' 2023-10-16T20:46:57.084 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/133/14' 2023-10-16T20:46:57.084 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/133/18' 2023-10-16T20:46:57.084 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/133/13' 2023-10-16T20:46:57.085 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/133/17' 2023-10-16T20:46:57.085 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/133' 2023-10-16T20:46:57.085 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/173/19' 2023-10-16T20:46:57.085 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/173/15' 2023-10-16T20:46:57.086 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/173/20' 2023-10-16T20:46:57.086 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/173/12' 2023-10-16T20:46:57.086 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/173/10' 2023-10-16T20:46:57.087 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/173/11' 2023-10-16T20:46:57.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:46:57.499 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:46:57.817 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/173/16' 2023-10-16T20:46:57.817 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/173/14' 2023-10-16T20:46:57.817 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/173/18' 2023-10-16T20:46:57.817 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/173/13' 2023-10-16T20:46:57.818 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/173/17' 2023-10-16T20:46:57.818 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/173' 2023-10-16T20:46:57.818 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/174/19' 2023-10-16T20:46:57.818 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/174/15' 2023-10-16T20:46:57.818 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/174/20' 2023-10-16T20:46:57.819 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/174/12' 2023-10-16T20:46:57.819 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/174/10' 2023-10-16T20:46:57.819 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/174/11' 2023-10-16T20:46:57.819 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/174/16' 2023-10-16T20:46:57.820 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/174/14' 2023-10-16T20:46:57.820 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/174/18' 2023-10-16T20:46:57.820 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/174/13' 2023-10-16T20:46:57.820 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/174/17' 2023-10-16T20:46:57.821 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/174' 2023-10-16T20:46:57.821 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/155/19' 2023-10-16T20:46:57.821 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/155/15' 2023-10-16T20:46:57.821 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/155/20' 2023-10-16T20:46:57.821 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/155/12' 2023-10-16T20:46:57.822 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/155/10' 2023-10-16T20:46:57.822 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/155/11' 2023-10-16T20:46:57.822 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/155/16' 2023-10-16T20:46:57.822 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/155/14' 2023-10-16T20:46:57.823 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/155/18' 2023-10-16T20:46:57.823 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/155/13' 2023-10-16T20:46:57.823 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/155/17' 2023-10-16T20:46:57.823 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/155' 2023-10-16T20:46:57.823 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/114/19' 2023-10-16T20:46:57.823 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/114/15' 2023-10-16T20:46:57.824 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/114/20' 2023-10-16T20:46:57.824 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/114/12' 2023-10-16T20:46:57.824 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/114/10' 2023-10-16T20:46:57.825 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/114/11' 2023-10-16T20:46:57.825 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/114/16' 2023-10-16T20:46:57.825 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/114/14' 2023-10-16T20:46:57.825 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/114/18' 2023-10-16T20:46:57.826 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/114/13' 2023-10-16T20:46:57.826 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/114/17' 2023-10-16T20:46:57.826 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/114' 2023-10-16T20:46:57.827 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/159/19' 2023-10-16T20:46:57.827 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/159/15' 2023-10-16T20:46:57.828 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/159/20' 2023-10-16T20:46:57.828 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/159/12' 2023-10-16T20:46:57.828 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/159/10' 2023-10-16T20:46:57.829 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/159/11' 2023-10-16T20:46:57.829 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/159/16' 2023-10-16T20:46:57.829 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/159/14' 2023-10-16T20:46:57.830 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/159/18' 2023-10-16T20:46:57.830 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/159/13' 2023-10-16T20:46:57.831 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/159/17' 2023-10-16T20:46:57.831 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/159' 2023-10-16T20:46:57.831 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/187/19' 2023-10-16T20:46:57.832 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/187/15' 2023-10-16T20:46:57.832 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/187/20' 2023-10-16T20:46:57.832 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/187/12' 2023-10-16T20:46:57.833 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/187/10' 2023-10-16T20:46:57.833 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/187/11' 2023-10-16T20:46:57.833 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/187/16' 2023-10-16T20:46:57.834 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/187/14' 2023-10-16T20:46:57.834 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/187/18' 2023-10-16T20:46:57.835 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/187/13' 2023-10-16T20:46:57.835 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/187/17' 2023-10-16T20:46:57.835 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/187' 2023-10-16T20:46:57.836 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/185/19' 2023-10-16T20:46:57.836 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/185/15' 2023-10-16T20:46:57.836 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/185/20' 2023-10-16T20:46:57.837 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/185/12' 2023-10-16T20:46:57.837 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/185/10' 2023-10-16T20:46:57.838 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/185/11' 2023-10-16T20:46:57.838 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/185/16' 2023-10-16T20:46:57.838 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/185/14' 2023-10-16T20:46:57.839 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/185/18' 2023-10-16T20:46:57.839 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/185/13' 2023-10-16T20:46:57.839 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/185/17' 2023-10-16T20:46:57.839 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/185' 2023-10-16T20:46:57.840 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/120/19' 2023-10-16T20:46:57.840 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/120/15' 2023-10-16T20:46:57.841 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/120/20' 2023-10-16T20:46:57.841 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/120/12' 2023-10-16T20:46:57.841 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/120/10' 2023-10-16T20:46:57.842 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/120/11' 2023-10-16T20:46:57.842 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/120/16' 2023-10-16T20:46:57.842 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/120/14' 2023-10-16T20:46:57.843 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/120/18' 2023-10-16T20:46:57.843 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/120/13' 2023-10-16T20:46:57.845 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/120/17' 2023-10-16T20:46:57.845 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/120' 2023-10-16T20:46:57.845 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/110/19' 2023-10-16T20:46:57.846 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/110/15' 2023-10-16T20:46:57.846 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/110/20' 2023-10-16T20:46:57.847 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/110/12' 2023-10-16T20:46:57.847 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/110/10' 2023-10-16T20:46:57.847 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/110/11' 2023-10-16T20:46:57.848 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/110/16' 2023-10-16T20:46:57.848 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/110/14' 2023-10-16T20:46:57.848 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/110/18' 2023-10-16T20:46:57.849 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/110/13' 2023-10-16T20:46:57.849 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/110/17' 2023-10-16T20:46:57.849 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/110' 2023-10-16T20:46:57.850 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/137/19' 2023-10-16T20:46:57.850 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/137/15' 2023-10-16T20:46:57.851 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/137/20' 2023-10-16T20:46:57.851 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/137/12' 2023-10-16T20:46:57.851 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/137/10' 2023-10-16T20:46:57.851 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/137/11' 2023-10-16T20:46:57.852 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/137/16' 2023-10-16T20:46:58.198 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/137/14' 2023-10-16T20:46:58.198 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/137/18' 2023-10-16T20:46:58.199 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/137/13' 2023-10-16T20:46:58.199 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/137/17' 2023-10-16T20:46:58.199 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/137' 2023-10-16T20:46:58.199 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/198/19' 2023-10-16T20:46:58.199 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/198/15' 2023-10-16T20:46:58.200 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/198/20' 2023-10-16T20:46:58.200 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/198/12' 2023-10-16T20:46:58.200 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/198/10' 2023-10-16T20:46:58.200 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/198/11' 2023-10-16T20:46:58.200 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/198/16' 2023-10-16T20:46:58.201 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/198/14' 2023-10-16T20:46:58.201 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/198/18' 2023-10-16T20:46:58.201 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/198/13' 2023-10-16T20:46:58.201 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/198/17' 2023-10-16T20:46:58.201 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/198' 2023-10-16T20:46:58.202 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/136/19' 2023-10-16T20:46:58.202 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/136/15' 2023-10-16T20:46:58.202 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/136/20' 2023-10-16T20:46:58.202 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/136/12' 2023-10-16T20:46:58.202 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/136/10' 2023-10-16T20:46:58.203 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/136/11' 2023-10-16T20:46:58.203 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/136/16' 2023-10-16T20:46:58.203 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/136/14' 2023-10-16T20:46:58.203 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/136/18' 2023-10-16T20:46:58.203 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/136/13' 2023-10-16T20:46:58.204 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/136/17' 2023-10-16T20:46:58.204 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/136' 2023-10-16T20:46:58.204 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/125/19' 2023-10-16T20:46:58.204 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/125/15' 2023-10-16T20:46:58.204 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/125/20' 2023-10-16T20:46:58.205 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/125/12' 2023-10-16T20:46:58.205 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/125/10' 2023-10-16T20:46:58.205 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/125/11' 2023-10-16T20:46:58.205 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/125/16' 2023-10-16T20:46:58.205 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/125/14' 2023-10-16T20:46:58.205 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/125/18' 2023-10-16T20:46:58.206 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/125/13' 2023-10-16T20:46:58.206 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/125/17' 2023-10-16T20:46:58.206 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/125' 2023-10-16T20:46:58.206 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/194/19' 2023-10-16T20:46:58.206 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/194/15' 2023-10-16T20:46:58.207 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/194/20' 2023-10-16T20:46:58.207 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/194/12' 2023-10-16T20:46:58.207 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/194/10' 2023-10-16T20:46:58.207 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/194/11' 2023-10-16T20:46:58.207 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/194/16' 2023-10-16T20:46:58.208 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/194/14' 2023-10-16T20:46:58.208 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/194/18' 2023-10-16T20:46:58.208 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/194/13' 2023-10-16T20:46:58.208 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/194/17' 2023-10-16T20:46:58.209 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/194' 2023-10-16T20:46:58.209 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/119/19' 2023-10-16T20:46:58.209 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/119/15' 2023-10-16T20:46:58.209 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/119/20' 2023-10-16T20:46:58.209 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/119/12' 2023-10-16T20:46:58.209 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/119/10' 2023-10-16T20:46:58.210 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/119/11' 2023-10-16T20:46:58.210 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/119/16' 2023-10-16T20:46:58.210 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/119/14' 2023-10-16T20:46:58.210 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/119/18' 2023-10-16T20:46:58.211 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/119/13' 2023-10-16T20:46:58.211 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/119/17' 2023-10-16T20:46:58.211 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/119' 2023-10-16T20:46:58.211 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/157/19' 2023-10-16T20:46:58.211 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/157/15' 2023-10-16T20:46:58.212 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/157/20' 2023-10-16T20:46:58.212 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/157/12' 2023-10-16T20:46:58.212 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/157/10' 2023-10-16T20:46:58.212 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/157/11' 2023-10-16T20:46:58.212 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/157/16' 2023-10-16T20:46:58.213 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/157/14' 2023-10-16T20:46:58.213 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/157/18' 2023-10-16T20:46:58.213 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/157/13' 2023-10-16T20:46:58.213 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/157/17' 2023-10-16T20:46:58.213 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/157' 2023-10-16T20:46:58.214 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/172/19' 2023-10-16T20:46:58.214 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/172/15' 2023-10-16T20:46:58.214 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/172/20' 2023-10-16T20:46:58.214 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/172/12' 2023-10-16T20:46:58.214 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/172/10' 2023-10-16T20:46:58.215 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/172/11' 2023-10-16T20:46:58.215 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/172/16' 2023-10-16T20:46:58.215 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/172/14' 2023-10-16T20:46:58.215 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/172/18' 2023-10-16T20:46:58.215 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/172/13' 2023-10-16T20:46:58.216 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/172/17' 2023-10-16T20:46:58.216 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/172' 2023-10-16T20:46:58.216 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/186/19' 2023-10-16T20:46:58.216 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/186/15' 2023-10-16T20:46:58.216 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/186/20' 2023-10-16T20:46:58.217 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/186/12' 2023-10-16T20:46:58.217 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/186/10' 2023-10-16T20:46:58.217 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/186/11' 2023-10-16T20:46:58.217 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/186/16' 2023-10-16T20:46:58.217 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/186/14' 2023-10-16T20:46:58.217 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/186/18' 2023-10-16T20:46:58.218 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/186/13' 2023-10-16T20:46:58.218 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/186/17' 2023-10-16T20:46:58.218 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/186' 2023-10-16T20:46:58.218 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/190/19' 2023-10-16T20:46:58.218 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/190/15' 2023-10-16T20:46:58.219 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/190/20' 2023-10-16T20:46:58.219 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/190/12' 2023-10-16T20:46:58.219 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/190/10' 2023-10-16T20:46:58.219 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/190/11' 2023-10-16T20:46:58.219 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/190/16' 2023-10-16T20:46:58.220 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/190/14' 2023-10-16T20:47:00.434 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/190/18' 2023-10-16T20:47:00.435 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/190/13' 2023-10-16T20:47:00.435 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/190/17' 2023-10-16T20:47:00.435 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/190' 2023-10-16T20:47:00.435 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/104/19' 2023-10-16T20:47:00.436 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/104/15' 2023-10-16T20:47:00.436 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/104/20' 2023-10-16T20:47:00.436 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/104/12' 2023-10-16T20:47:00.436 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/104/10' 2023-10-16T20:47:00.436 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/104/11' 2023-10-16T20:47:00.437 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/104/16' 2023-10-16T20:47:00.437 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/104/14' 2023-10-16T20:47:00.437 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/104/18' 2023-10-16T20:47:00.437 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/104/13' 2023-10-16T20:47:00.437 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/104/17' 2023-10-16T20:47:00.438 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/104' 2023-10-16T20:47:00.438 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/164/19' 2023-10-16T20:47:00.438 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/164/15' 2023-10-16T20:47:00.438 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/164/20' 2023-10-16T20:47:00.438 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/164/12' 2023-10-16T20:47:00.439 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/164/10' 2023-10-16T20:47:00.439 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/164/11' 2023-10-16T20:47:00.439 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/164/16' 2023-10-16T20:47:00.439 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/164/14' 2023-10-16T20:47:00.439 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/164/18' 2023-10-16T20:47:00.440 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/164/13' 2023-10-16T20:47:00.440 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/164/17' 2023-10-16T20:47:00.440 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/164' 2023-10-16T20:47:00.440 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/112/19' 2023-10-16T20:47:00.440 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/112/15' 2023-10-16T20:47:00.441 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/112/20' 2023-10-16T20:47:00.441 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/112/12' 2023-10-16T20:47:00.441 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/112/10' 2023-10-16T20:47:00.441 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/112/11' 2023-10-16T20:47:00.441 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/112/16' 2023-10-16T20:47:00.441 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/112/14' 2023-10-16T20:47:00.442 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/112/18' 2023-10-16T20:47:00.442 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/112/13' 2023-10-16T20:47:00.442 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/112/17' 2023-10-16T20:47:00.442 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/112' 2023-10-16T20:47:00.443 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/100/19' 2023-10-16T20:47:00.443 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/100/15' 2023-10-16T20:47:00.443 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/100/20' 2023-10-16T20:47:00.443 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/100/12' 2023-10-16T20:47:00.444 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/100/10' 2023-10-16T20:47:00.444 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/100/11' 2023-10-16T20:47:00.444 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/100/16' 2023-10-16T20:47:00.444 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/100/14' 2023-10-16T20:47:00.444 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/100/18' 2023-10-16T20:47:00.444 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/100/13' 2023-10-16T20:47:00.445 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/100/17' 2023-10-16T20:47:00.445 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/100' 2023-10-16T20:47:00.445 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/134/19' 2023-10-16T20:47:00.446 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/134/15' 2023-10-16T20:47:00.446 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/134/20' 2023-10-16T20:47:00.446 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/134/12' 2023-10-16T20:47:00.446 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/134/10' 2023-10-16T20:47:00.447 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/134/11' 2023-10-16T20:47:00.447 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/134/16' 2023-10-16T20:47:00.447 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/134/14' 2023-10-16T20:47:00.447 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/134/18' 2023-10-16T20:47:00.447 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/134/13' 2023-10-16T20:47:00.448 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/134/17' 2023-10-16T20:47:00.448 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/134' 2023-10-16T20:47:00.448 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/115/19' 2023-10-16T20:47:00.448 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/115/15' 2023-10-16T20:47:00.448 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/115/20' 2023-10-16T20:47:00.449 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/115/12' 2023-10-16T20:47:00.449 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/115/10' 2023-10-16T20:47:00.449 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/115/11' 2023-10-16T20:47:00.449 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/115/16' 2023-10-16T20:47:00.449 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/115/14' 2023-10-16T20:47:00.450 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/115/18' 2023-10-16T20:47:00.450 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/115/13' 2023-10-16T20:47:00.450 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/115/17' 2023-10-16T20:47:00.450 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/115' 2023-10-16T20:47:00.451 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/116/19' 2023-10-16T20:47:00.451 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/116/15' 2023-10-16T20:47:00.451 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/116/20' 2023-10-16T20:47:00.451 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/116/12' 2023-10-16T20:47:00.451 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/116/10' 2023-10-16T20:47:00.452 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/116/11' 2023-10-16T20:47:00.452 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/116/16' 2023-10-16T20:47:00.452 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/116/14' 2023-10-16T20:47:00.452 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/116/18' 2023-10-16T20:47:00.453 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/116/13' 2023-10-16T20:47:00.453 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/116/17' 2023-10-16T20:47:00.453 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/116' 2023-10-16T20:47:00.453 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/195/19' 2023-10-16T20:47:00.453 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/195/15' 2023-10-16T20:47:00.454 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/195/20' 2023-10-16T20:47:00.454 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/195/12' 2023-10-16T20:47:00.454 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/195/10' 2023-10-16T20:47:00.454 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/195/11' 2023-10-16T20:47:00.455 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/195/16' 2023-10-16T20:47:00.455 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/195/14' 2023-10-16T20:47:00.455 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/195/18' 2023-10-16T20:47:00.455 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/195/13' 2023-10-16T20:47:00.455 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/195/17' 2023-10-16T20:47:00.456 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/195' 2023-10-16T20:47:00.456 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/138/19' 2023-10-16T20:47:00.456 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/138/15' 2023-10-16T20:47:00.456 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/138/20' 2023-10-16T20:47:00.456 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/138/12' 2023-10-16T20:47:00.456 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/138/10' 2023-10-16T20:47:00.457 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/138/11' 2023-10-16T20:47:00.457 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/138/16' 2023-10-16T20:47:00.457 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/138/14' 2023-10-16T20:47:00.457 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/138/18' 2023-10-16T20:47:02.634 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/138/13' 2023-10-16T20:47:02.634 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/138/17' 2023-10-16T20:47:02.634 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/138' 2023-10-16T20:47:02.634 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/135/19' 2023-10-16T20:47:02.635 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/135/15' 2023-10-16T20:47:02.635 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/135/20' 2023-10-16T20:47:02.635 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/135/12' 2023-10-16T20:47:02.635 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/135/10' 2023-10-16T20:47:02.635 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/135/11' 2023-10-16T20:47:02.635 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/135/16' 2023-10-16T20:47:02.636 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/135/14' 2023-10-16T20:47:02.636 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/135/18' 2023-10-16T20:47:02.636 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/135/13' 2023-10-16T20:47:02.636 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/135/17' 2023-10-16T20:47:02.636 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/135' 2023-10-16T20:47:02.636 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/117/19' 2023-10-16T20:47:02.637 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/117/15' 2023-10-16T20:47:02.637 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/117/20' 2023-10-16T20:47:02.637 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/117/12' 2023-10-16T20:47:02.637 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/117/10' 2023-10-16T20:47:02.637 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/117/11' 2023-10-16T20:47:02.637 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/117/16' 2023-10-16T20:47:02.638 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/117/14' 2023-10-16T20:47:02.638 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/117/18' 2023-10-16T20:47:02.638 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/117/13' 2023-10-16T20:47:02.638 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/117/17' 2023-10-16T20:47:02.638 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/117' 2023-10-16T20:47:02.639 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/143/19' 2023-10-16T20:47:02.639 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/143/15' 2023-10-16T20:47:02.639 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/143/20' 2023-10-16T20:47:02.639 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/143/12' 2023-10-16T20:47:02.639 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/143/10' 2023-10-16T20:47:02.640 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/143/11' 2023-10-16T20:47:02.640 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/143/16' 2023-10-16T20:47:02.640 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/143/14' 2023-10-16T20:47:02.640 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/143/18' 2023-10-16T20:47:02.640 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/143/13' 2023-10-16T20:47:02.640 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/143/17' 2023-10-16T20:47:02.641 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/143' 2023-10-16T20:47:02.641 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/180/19' 2023-10-16T20:47:02.641 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/180/15' 2023-10-16T20:47:02.641 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/180/20' 2023-10-16T20:47:02.641 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/180/12' 2023-10-16T20:47:02.641 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/180/10' 2023-10-16T20:47:02.642 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/180/11' 2023-10-16T20:47:02.642 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/180/16' 2023-10-16T20:47:02.642 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/180/14' 2023-10-16T20:47:02.642 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/180/18' 2023-10-16T20:47:02.642 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/180/13' 2023-10-16T20:47:02.643 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/180/17' 2023-10-16T20:47:02.643 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/180' 2023-10-16T20:47:02.643 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/126/19' 2023-10-16T20:47:02.643 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/126/15' 2023-10-16T20:47:02.643 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/126/20' 2023-10-16T20:47:02.643 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/126/12' 2023-10-16T20:47:02.644 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/126/10' 2023-10-16T20:47:02.644 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/126/11' 2023-10-16T20:47:02.644 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/126/16' 2023-10-16T20:47:02.644 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/126/14' 2023-10-16T20:47:02.644 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/126/18' 2023-10-16T20:47:02.645 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/126/13' 2023-10-16T20:47:02.645 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/126/17' 2023-10-16T20:47:02.645 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/126' 2023-10-16T20:47:02.645 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/161/19' 2023-10-16T20:47:02.645 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/161/15' 2023-10-16T20:47:02.645 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/161/20' 2023-10-16T20:47:02.646 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/161/12' 2023-10-16T20:47:02.646 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/161/10' 2023-10-16T20:47:02.646 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/161/11' 2023-10-16T20:47:02.646 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/161/16' 2023-10-16T20:47:02.646 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/161/14' 2023-10-16T20:47:02.647 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/161/18' 2023-10-16T20:47:02.647 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/161/13' 2023-10-16T20:47:02.647 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/161/17' 2023-10-16T20:47:02.647 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/161' 2023-10-16T20:47:02.647 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/176/19' 2023-10-16T20:47:02.648 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/176/15' 2023-10-16T20:47:02.648 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/176/20' 2023-10-16T20:47:02.648 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/176/12' 2023-10-16T20:47:02.648 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/176/10' 2023-10-16T20:47:02.648 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/176/11' 2023-10-16T20:47:02.648 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/176/16' 2023-10-16T20:47:02.649 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/176/14' 2023-10-16T20:47:02.649 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/176/18' 2023-10-16T20:47:02.649 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/176/13' 2023-10-16T20:47:02.649 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/176/17' 2023-10-16T20:47:02.649 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/176' 2023-10-16T20:47:02.650 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/111/19' 2023-10-16T20:47:02.650 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/111/15' 2023-10-16T20:47:02.650 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/111/20' 2023-10-16T20:47:02.650 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/111/12' 2023-10-16T20:47:02.650 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/111/10' 2023-10-16T20:47:02.651 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/111/11' 2023-10-16T20:47:02.651 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/111/16' 2023-10-16T20:47:02.651 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/111/14' 2023-10-16T20:47:02.651 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/111/18' 2023-10-16T20:47:02.651 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/111/13' 2023-10-16T20:47:02.651 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/111/17' 2023-10-16T20:47:02.652 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/111' 2023-10-16T20:47:02.652 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/191/19' 2023-10-16T20:47:02.652 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/191/15' 2023-10-16T20:47:02.652 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/191/20' 2023-10-16T20:47:02.652 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/191/12' 2023-10-16T20:47:02.652 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/191/10' 2023-10-16T20:47:02.652 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/191/11' 2023-10-16T20:47:02.653 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/191/16' 2023-10-16T20:47:02.653 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/191/14' 2023-10-16T20:47:02.653 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/191/18' 2023-10-16T20:47:02.653 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/191/13' 2023-10-16T20:47:04.357 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/191/17' 2023-10-16T20:47:04.358 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/191' 2023-10-16T20:47:04.358 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/184/19' 2023-10-16T20:47:04.358 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/184/15' 2023-10-16T20:47:04.358 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/184/20' 2023-10-16T20:47:04.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/184/12' 2023-10-16T20:47:04.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/184/10' 2023-10-16T20:47:04.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/184/11' 2023-10-16T20:47:04.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/184/16' 2023-10-16T20:47:04.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/184/14' 2023-10-16T20:47:04.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/184/18' 2023-10-16T20:47:04.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/184/13' 2023-10-16T20:47:04.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/184/17' 2023-10-16T20:47:04.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/184' 2023-10-16T20:47:04.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/192/19' 2023-10-16T20:47:04.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/192/15' 2023-10-16T20:47:04.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/192/20' 2023-10-16T20:47:04.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/192/12' 2023-10-16T20:47:04.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/192/10' 2023-10-16T20:47:04.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/192/11' 2023-10-16T20:47:04.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/192/16' 2023-10-16T20:47:04.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/192/14' 2023-10-16T20:47:04.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/192/18' 2023-10-16T20:47:04.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/192/13' 2023-10-16T20:47:04.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/192/17' 2023-10-16T20:47:04.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/192' 2023-10-16T20:47:04.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/103/19' 2023-10-16T20:47:04.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/103/15' 2023-10-16T20:47:04.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/103/20' 2023-10-16T20:47:04.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/103/12' 2023-10-16T20:47:04.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/103/10' 2023-10-16T20:47:04.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/103/11' 2023-10-16T20:47:04.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/103/16' 2023-10-16T20:47:04.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/103/14' 2023-10-16T20:47:04.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/103/18' 2023-10-16T20:47:04.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/103/13' 2023-10-16T20:47:04.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/103/17' 2023-10-16T20:47:04.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/103' 2023-10-16T20:47:04.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/128/19' 2023-10-16T20:47:04.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/128/15' 2023-10-16T20:47:04.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/128/20' 2023-10-16T20:47:04.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/128/12' 2023-10-16T20:47:04.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/128/10' 2023-10-16T20:47:04.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/128/11' 2023-10-16T20:47:04.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/128/16' 2023-10-16T20:47:04.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/128/14' 2023-10-16T20:47:04.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/128/18' 2023-10-16T20:47:04.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/128/13' 2023-10-16T20:47:04.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/128/17' 2023-10-16T20:47:04.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/128' 2023-10-16T20:47:04.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/144/19' 2023-10-16T20:47:04.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/144/15' 2023-10-16T20:47:04.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/144/20' 2023-10-16T20:47:04.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/144/12' 2023-10-16T20:47:04.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/144/10' 2023-10-16T20:47:04.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/144/11' 2023-10-16T20:47:04.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/144/16' 2023-10-16T20:47:04.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/144/14' 2023-10-16T20:47:04.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/144/18' 2023-10-16T20:47:04.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/144/13' 2023-10-16T20:47:04.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/144/17' 2023-10-16T20:47:04.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/144' 2023-10-16T20:47:04.370 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/146/19' 2023-10-16T20:47:04.370 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/146/15' 2023-10-16T20:47:04.370 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/146/20' 2023-10-16T20:47:04.370 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/146/12' 2023-10-16T20:47:04.371 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/146/10' 2023-10-16T20:47:04.371 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/146/11' 2023-10-16T20:47:04.371 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/146/16' 2023-10-16T20:47:04.371 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/146/14' 2023-10-16T20:47:04.371 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/146/18' 2023-10-16T20:47:04.372 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/146/13' 2023-10-16T20:47:04.372 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/146/17' 2023-10-16T20:47:04.372 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/146' 2023-10-16T20:47:04.372 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/141/19' 2023-10-16T20:47:04.372 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/141/15' 2023-10-16T20:47:04.372 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/141/20' 2023-10-16T20:47:04.373 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/141/12' 2023-10-16T20:47:04.373 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/141/10' 2023-10-16T20:47:04.373 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/141/11' 2023-10-16T20:47:04.373 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/141/16' 2023-10-16T20:47:04.373 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/141/14' 2023-10-16T20:47:04.374 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/141/18' 2023-10-16T20:47:04.374 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/141/13' 2023-10-16T20:47:04.374 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/141/17' 2023-10-16T20:47:04.374 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/141' 2023-10-16T20:47:04.374 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/182/19' 2023-10-16T20:47:04.375 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/182/15' 2023-10-16T20:47:04.375 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/182/20' 2023-10-16T20:47:04.375 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/182/12' 2023-10-16T20:47:04.375 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/182/10' 2023-10-16T20:47:04.375 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/182/11' 2023-10-16T20:47:04.376 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/182/16' 2023-10-16T20:47:04.376 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/182/14' 2023-10-16T20:47:04.376 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/182/18' 2023-10-16T20:47:04.376 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/182/13' 2023-10-16T20:47:04.376 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/182/17' 2023-10-16T20:47:04.376 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/182' 2023-10-16T20:47:04.377 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/107/19' 2023-10-16T20:47:04.377 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/107/15' 2023-10-16T20:47:04.377 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/107/20' 2023-10-16T20:47:04.377 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/107/12' 2023-10-16T20:47:04.377 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/107/10' 2023-10-16T20:47:04.377 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/107/11' 2023-10-16T20:47:04.378 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/107/16' 2023-10-16T20:47:04.378 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/107/14' 2023-10-16T20:47:04.378 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/107/18' 2023-10-16T20:47:04.378 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/107/13' 2023-10-16T20:47:04.378 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/107/17' 2023-10-16T20:47:06.925 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/107' 2023-10-16T20:47:06.925 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/178/19' 2023-10-16T20:47:06.925 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/178/15' 2023-10-16T20:47:06.926 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/178/20' 2023-10-16T20:47:06.926 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/178/12' 2023-10-16T20:47:06.926 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/178/10' 2023-10-16T20:47:06.926 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/178/11' 2023-10-16T20:47:06.927 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/178/16' 2023-10-16T20:47:06.927 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/178/14' 2023-10-16T20:47:06.927 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/178/18' 2023-10-16T20:47:06.927 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/178/13' 2023-10-16T20:47:06.927 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/178/17' 2023-10-16T20:47:06.928 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/178' 2023-10-16T20:47:06.928 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/156/19' 2023-10-16T20:47:06.928 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/156/15' 2023-10-16T20:47:06.928 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/156/20' 2023-10-16T20:47:06.928 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/156/12' 2023-10-16T20:47:06.929 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/156/10' 2023-10-16T20:47:06.929 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/156/11' 2023-10-16T20:47:06.929 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/156/16' 2023-10-16T20:47:06.929 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/156/14' 2023-10-16T20:47:06.929 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/156/18' 2023-10-16T20:47:06.930 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/156/13' 2023-10-16T20:47:06.930 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/156/17' 2023-10-16T20:47:06.930 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/156' 2023-10-16T20:47:06.930 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/131/19' 2023-10-16T20:47:06.931 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/131/15' 2023-10-16T20:47:06.931 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/131/20' 2023-10-16T20:47:06.931 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/131/12' 2023-10-16T20:47:06.931 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/131/10' 2023-10-16T20:47:06.932 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/131/11' 2023-10-16T20:47:06.932 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/131/16' 2023-10-16T20:47:06.932 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/131/14' 2023-10-16T20:47:06.932 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/131/18' 2023-10-16T20:47:06.932 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/131/13' 2023-10-16T20:47:06.932 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/131/17' 2023-10-16T20:47:06.933 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/131' 2023-10-16T20:47:06.933 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/123/19' 2023-10-16T20:47:06.933 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/123/15' 2023-10-16T20:47:06.933 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/123/20' 2023-10-16T20:47:06.934 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/123/12' 2023-10-16T20:47:06.934 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/123/10' 2023-10-16T20:47:06.934 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/123/11' 2023-10-16T20:47:06.934 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/123/16' 2023-10-16T20:47:06.934 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/123/14' 2023-10-16T20:47:06.935 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/123/18' 2023-10-16T20:47:06.935 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/123/13' 2023-10-16T20:47:06.935 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/123/17' 2023-10-16T20:47:06.935 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/123' 2023-10-16T20:47:06.935 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/160/19' 2023-10-16T20:47:06.936 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/160/15' 2023-10-16T20:47:06.936 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/160/20' 2023-10-16T20:47:06.936 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/160/12' 2023-10-16T20:47:06.936 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/160/10' 2023-10-16T20:47:06.936 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/160/11' 2023-10-16T20:47:06.937 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/160/16' 2023-10-16T20:47:06.937 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/160/14' 2023-10-16T20:47:06.937 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/160/18' 2023-10-16T20:47:06.937 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/160/13' 2023-10-16T20:47:06.937 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/160/17' 2023-10-16T20:47:06.937 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/160' 2023-10-16T20:47:06.938 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/165/19' 2023-10-16T20:47:06.938 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/165/15' 2023-10-16T20:47:06.938 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/165/20' 2023-10-16T20:47:06.938 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/165/12' 2023-10-16T20:47:06.939 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/165/10' 2023-10-16T20:47:06.939 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/165/11' 2023-10-16T20:47:06.939 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/165/16' 2023-10-16T20:47:06.939 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/165/14' 2023-10-16T20:47:06.939 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/165/18' 2023-10-16T20:47:06.940 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/165/13' 2023-10-16T20:47:06.940 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/165/17' 2023-10-16T20:47:06.940 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/165' 2023-10-16T20:47:06.940 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/101/19' 2023-10-16T20:47:06.940 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/101/15' 2023-10-16T20:47:06.941 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/101/20' 2023-10-16T20:47:06.941 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/101/12' 2023-10-16T20:47:06.941 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/101/10' 2023-10-16T20:47:06.941 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/101/11' 2023-10-16T20:47:06.941 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/101/16' 2023-10-16T20:47:06.941 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/101/14' 2023-10-16T20:47:06.942 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/101/18' 2023-10-16T20:47:06.942 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/101/13' 2023-10-16T20:47:06.942 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/101/17' 2023-10-16T20:47:06.942 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/101' 2023-10-16T20:47:06.943 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/148/19' 2023-10-16T20:47:06.943 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/148/15' 2023-10-16T20:47:06.943 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/148/20' 2023-10-16T20:47:06.943 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/148/12' 2023-10-16T20:47:06.943 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/148/10' 2023-10-16T20:47:06.943 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/148/11' 2023-10-16T20:47:06.944 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/148/16' 2023-10-16T20:47:06.944 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/148/14' 2023-10-16T20:47:06.944 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/148/18' 2023-10-16T20:47:06.944 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/148/13' 2023-10-16T20:47:06.944 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/148/17' 2023-10-16T20:47:06.945 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/148' 2023-10-16T20:47:06.945 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/168/19' 2023-10-16T20:47:06.945 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/168/15' 2023-10-16T20:47:06.945 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/168/20' 2023-10-16T20:47:06.945 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/168/12' 2023-10-16T20:47:06.946 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/168/10' 2023-10-16T20:47:06.946 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/168/11' 2023-10-16T20:47:06.946 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/168/16' 2023-10-16T20:47:06.946 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/168/14' 2023-10-16T20:47:06.946 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/168/18' 2023-10-16T20:47:06.947 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/168/13' 2023-10-16T20:47:06.947 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/168/17' 2023-10-16T20:47:06.947 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/168' 2023-10-16T20:47:07.343 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/179/19' 2023-10-16T20:47:07.344 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/179/15' 2023-10-16T20:47:07.344 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/179/20' 2023-10-16T20:47:07.344 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/179/12' 2023-10-16T20:47:07.344 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/179/10' 2023-10-16T20:47:07.344 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/179/11' 2023-10-16T20:47:07.345 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/179/16' 2023-10-16T20:47:07.345 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/179/14' 2023-10-16T20:47:07.345 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/179/18' 2023-10-16T20:47:07.345 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/179/13' 2023-10-16T20:47:07.346 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/179/17' 2023-10-16T20:47:07.346 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/179' 2023-10-16T20:47:07.346 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/132/19' 2023-10-16T20:47:07.346 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/132/15' 2023-10-16T20:47:07.346 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/132/20' 2023-10-16T20:47:07.347 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/132/12' 2023-10-16T20:47:07.347 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/132/10' 2023-10-16T20:47:07.347 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/132/11' 2023-10-16T20:47:07.347 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/132/16' 2023-10-16T20:47:07.347 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/132/14' 2023-10-16T20:47:07.348 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/132/18' 2023-10-16T20:47:07.348 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/132/13' 2023-10-16T20:47:07.348 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/132/17' 2023-10-16T20:47:07.348 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/132' 2023-10-16T20:47:07.348 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/199/19' 2023-10-16T20:47:07.349 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/199/15' 2023-10-16T20:47:07.349 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/199/20' 2023-10-16T20:47:07.349 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/199/12' 2023-10-16T20:47:07.349 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/199/10' 2023-10-16T20:47:07.349 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/199/11' 2023-10-16T20:47:07.350 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/199/16' 2023-10-16T20:47:07.350 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/199/14' 2023-10-16T20:47:07.350 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/199/18' 2023-10-16T20:47:07.350 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/199/13' 2023-10-16T20:47:07.350 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/199/17' 2023-10-16T20:47:07.351 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/199' 2023-10-16T20:47:07.351 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/163/19' 2023-10-16T20:47:07.351 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/163/15' 2023-10-16T20:47:07.351 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/163/20' 2023-10-16T20:47:07.351 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/163/12' 2023-10-16T20:47:07.352 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/163/10' 2023-10-16T20:47:07.352 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/163/11' 2023-10-16T20:47:07.352 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/163/16' 2023-10-16T20:47:07.352 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/163/14' 2023-10-16T20:47:07.353 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/163/18' 2023-10-16T20:47:07.353 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/163/13' 2023-10-16T20:47:07.353 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/163/17' 2023-10-16T20:47:07.353 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/163' 2023-10-16T20:47:07.354 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/153/19' 2023-10-16T20:47:07.354 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/153/15' 2023-10-16T20:47:07.354 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/153/20' 2023-10-16T20:47:07.354 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/153/12' 2023-10-16T20:47:07.355 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/153/10' 2023-10-16T20:47:07.355 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/153/11' 2023-10-16T20:47:07.355 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/153/16' 2023-10-16T20:47:07.355 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/153/14' 2023-10-16T20:47:07.355 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/153/18' 2023-10-16T20:47:07.355 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/153/13' 2023-10-16T20:47:07.356 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/153/17' 2023-10-16T20:47:07.356 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/153' 2023-10-16T20:47:07.356 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/122/19' 2023-10-16T20:47:07.356 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/122/15' 2023-10-16T20:47:07.356 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/122/20' 2023-10-16T20:47:07.356 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/122/12' 2023-10-16T20:47:07.357 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/122/10' 2023-10-16T20:47:07.357 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/122/11' 2023-10-16T20:47:07.357 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/122/16' 2023-10-16T20:47:07.357 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/122/14' 2023-10-16T20:47:07.357 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/122/18' 2023-10-16T20:47:07.358 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/122/13' 2023-10-16T20:47:07.358 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/122/17' 2023-10-16T20:47:07.358 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/122' 2023-10-16T20:47:07.358 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/162/19' 2023-10-16T20:47:07.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/162/15' 2023-10-16T20:47:07.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/162/20' 2023-10-16T20:47:07.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/162/12' 2023-10-16T20:47:07.359 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/162/10' 2023-10-16T20:47:07.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/162/11' 2023-10-16T20:47:07.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/162/16' 2023-10-16T20:47:07.360 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/162/14' 2023-10-16T20:47:07.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/162/18' 2023-10-16T20:47:07.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/162/13' 2023-10-16T20:47:07.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/162/17' 2023-10-16T20:47:07.361 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/162' 2023-10-16T20:47:07.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/197/19' 2023-10-16T20:47:07.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/197/15' 2023-10-16T20:47:07.362 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/197/20' 2023-10-16T20:47:07.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/197/12' 2023-10-16T20:47:07.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/197/10' 2023-10-16T20:47:07.363 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/197/11' 2023-10-16T20:47:07.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/197/16' 2023-10-16T20:47:07.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/197/14' 2023-10-16T20:47:07.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/197/18' 2023-10-16T20:47:07.364 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/197/13' 2023-10-16T20:47:07.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/197/17' 2023-10-16T20:47:07.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/197' 2023-10-16T20:47:07.365 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/113/19' 2023-10-16T20:47:07.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/113/15' 2023-10-16T20:47:07.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/113/20' 2023-10-16T20:47:07.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/113/12' 2023-10-16T20:47:07.366 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/113/10' 2023-10-16T20:47:07.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/113/11' 2023-10-16T20:47:07.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/113/16' 2023-10-16T20:47:07.367 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/113/14' 2023-10-16T20:47:07.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/113/18' 2023-10-16T20:47:07.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/113/13' 2023-10-16T20:47:07.368 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/113/17' 2023-10-16T20:47:07.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/113' 2023-10-16T20:47:07.369 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/152/19' 2023-10-16T20:47:09.279 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/152/15' 2023-10-16T20:47:09.280 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/152/20' 2023-10-16T20:47:09.280 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/152/12' 2023-10-16T20:47:09.280 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/152/10' 2023-10-16T20:47:09.280 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/152/11' 2023-10-16T20:47:09.281 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/152/16' 2023-10-16T20:47:09.281 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/152/14' 2023-10-16T20:47:09.281 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/152/18' 2023-10-16T20:47:09.281 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/152/13' 2023-10-16T20:47:09.282 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/152/17' 2023-10-16T20:47:09.282 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/152' 2023-10-16T20:47:09.282 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/121/19' 2023-10-16T20:47:09.282 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/121/15' 2023-10-16T20:47:09.283 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/121/20' 2023-10-16T20:47:09.283 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/121/12' 2023-10-16T20:47:09.283 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/121/10' 2023-10-16T20:47:09.283 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/121/11' 2023-10-16T20:47:09.283 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/121/16' 2023-10-16T20:47:09.284 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/121/14' 2023-10-16T20:47:09.284 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/121/18' 2023-10-16T20:47:09.284 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/121/13' 2023-10-16T20:47:09.284 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/121/17' 2023-10-16T20:47:09.284 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/121' 2023-10-16T20:47:09.285 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/118/19' 2023-10-16T20:47:09.285 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/118/15' 2023-10-16T20:47:09.285 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/118/20' 2023-10-16T20:47:09.285 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/118/12' 2023-10-16T20:47:09.286 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/118/10' 2023-10-16T20:47:09.286 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/118/11' 2023-10-16T20:47:09.286 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/118/16' 2023-10-16T20:47:09.286 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/118/14' 2023-10-16T20:47:09.286 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/118/18' 2023-10-16T20:47:09.287 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/118/13' 2023-10-16T20:47:09.287 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/118/17' 2023-10-16T20:47:09.287 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/118' 2023-10-16T20:47:09.287 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/196/19' 2023-10-16T20:47:09.287 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/196/15' 2023-10-16T20:47:09.288 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/196/20' 2023-10-16T20:47:09.288 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/196/12' 2023-10-16T20:47:09.288 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/196/10' 2023-10-16T20:47:09.288 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/196/11' 2023-10-16T20:47:09.288 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/196/16' 2023-10-16T20:47:09.289 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/196/14' 2023-10-16T20:47:09.289 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/196/18' 2023-10-16T20:47:09.289 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/196/13' 2023-10-16T20:47:09.289 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/196/17' 2023-10-16T20:47:09.289 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/196' 2023-10-16T20:47:09.290 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/124/19' 2023-10-16T20:47:09.290 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/124/15' 2023-10-16T20:47:09.290 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/124/20' 2023-10-16T20:47:09.290 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/124/12' 2023-10-16T20:47:09.290 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/124/10' 2023-10-16T20:47:09.291 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/124/11' 2023-10-16T20:47:09.291 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/124/16' 2023-10-16T20:47:09.291 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/124/14' 2023-10-16T20:47:09.291 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/124/18' 2023-10-16T20:47:09.291 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/124/13' 2023-10-16T20:47:09.292 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/124/17' 2023-10-16T20:47:09.292 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/124' 2023-10-16T20:47:09.292 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/188/19' 2023-10-16T20:47:09.292 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/188/15' 2023-10-16T20:47:09.292 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/188/20' 2023-10-16T20:47:09.293 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/188/12' 2023-10-16T20:47:09.293 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/188/10' 2023-10-16T20:47:09.293 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/188/11' 2023-10-16T20:47:09.293 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/188/16' 2023-10-16T20:47:09.293 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/188/14' 2023-10-16T20:47:09.294 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/188/18' 2023-10-16T20:47:09.294 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/188/13' 2023-10-16T20:47:09.294 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/188/17' 2023-10-16T20:47:09.294 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/188' 2023-10-16T20:47:09.294 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/145/19' 2023-10-16T20:47:09.295 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/145/15' 2023-10-16T20:47:09.295 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/145/20' 2023-10-16T20:47:09.295 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/145/12' 2023-10-16T20:47:09.295 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/145/10' 2023-10-16T20:47:09.295 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/145/11' 2023-10-16T20:47:09.295 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/145/16' 2023-10-16T20:47:09.296 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/145/14' 2023-10-16T20:47:09.296 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/145/18' 2023-10-16T20:47:09.296 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/145/13' 2023-10-16T20:47:09.296 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/145/17' 2023-10-16T20:47:09.296 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/145' 2023-10-16T20:47:09.297 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/130/19' 2023-10-16T20:47:09.297 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/130/15' 2023-10-16T20:47:09.297 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/130/20' 2023-10-16T20:47:09.297 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/130/12' 2023-10-16T20:47:09.297 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/130/10' 2023-10-16T20:47:09.298 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/130/11' 2023-10-16T20:47:09.298 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/130/16' 2023-10-16T20:47:09.298 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/130/14' 2023-10-16T20:47:09.298 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/130/18' 2023-10-16T20:47:09.299 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/130/13' 2023-10-16T20:47:09.299 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/130/17' 2023-10-16T20:47:09.299 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/130' 2023-10-16T20:47:09.300 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/193/19' 2023-10-16T20:47:09.300 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/193/15' 2023-10-16T20:47:09.300 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/193/20' 2023-10-16T20:47:09.300 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/193/12' 2023-10-16T20:47:09.301 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/193/10' 2023-10-16T20:47:09.301 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/193/11' 2023-10-16T20:47:09.301 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/193/16' 2023-10-16T20:47:09.302 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/193/14' 2023-10-16T20:47:09.302 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/193/18' 2023-10-16T20:47:09.302 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/193/13' 2023-10-16T20:47:09.303 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/193/17' 2023-10-16T20:47:09.303 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/193' 2023-10-16T20:47:09.304 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/175/19' 2023-10-16T20:47:09.304 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/175/15' 2023-10-16T20:47:11.298 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/175/20' 2023-10-16T20:47:11.299 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/175/12' 2023-10-16T20:47:11.299 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/175/10' 2023-10-16T20:47:11.299 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/175/11' 2023-10-16T20:47:11.299 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/175/16' 2023-10-16T20:47:11.299 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/175/14' 2023-10-16T20:47:11.300 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/175/18' 2023-10-16T20:47:11.300 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/175/13' 2023-10-16T20:47:11.300 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/175/17' 2023-10-16T20:47:11.300 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/175' 2023-10-16T20:47:11.300 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/181/19' 2023-10-16T20:47:11.301 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/181/15' 2023-10-16T20:47:11.301 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/181/20' 2023-10-16T20:47:11.301 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/181/12' 2023-10-16T20:47:11.301 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/181/10' 2023-10-16T20:47:11.301 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/181/11' 2023-10-16T20:47:11.302 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/181/16' 2023-10-16T20:47:11.302 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/181/14' 2023-10-16T20:47:11.302 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/181/18' 2023-10-16T20:47:11.302 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/181/13' 2023-10-16T20:47:11.302 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/181/17' 2023-10-16T20:47:11.303 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/181' 2023-10-16T20:47:11.303 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/127/19' 2023-10-16T20:47:11.303 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/127/15' 2023-10-16T20:47:11.303 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/127/20' 2023-10-16T20:47:11.303 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/127/12' 2023-10-16T20:47:11.304 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/127/10' 2023-10-16T20:47:11.304 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/127/11' 2023-10-16T20:47:11.304 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/127/16' 2023-10-16T20:47:11.304 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/127/14' 2023-10-16T20:47:11.304 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/127/18' 2023-10-16T20:47:11.305 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/127/13' 2023-10-16T20:47:11.305 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/127/17' 2023-10-16T20:47:11.305 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/127' 2023-10-16T20:47:11.305 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/108/19' 2023-10-16T20:47:11.305 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/108/15' 2023-10-16T20:47:11.305 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/108/20' 2023-10-16T20:47:11.306 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/108/12' 2023-10-16T20:47:11.306 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/108/10' 2023-10-16T20:47:11.306 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/108/11' 2023-10-16T20:47:11.306 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/108/16' 2023-10-16T20:47:11.307 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/108/14' 2023-10-16T20:47:11.307 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/108/18' 2023-10-16T20:47:11.307 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/108/13' 2023-10-16T20:47:11.307 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/108/17' 2023-10-16T20:47:11.307 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/108' 2023-10-16T20:47:11.307 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/183/19' 2023-10-16T20:47:11.308 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/183/15' 2023-10-16T20:47:11.308 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/183/20' 2023-10-16T20:47:11.308 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/183/12' 2023-10-16T20:47:11.308 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/183/10' 2023-10-16T20:47:11.308 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/183/11' 2023-10-16T20:47:11.309 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/183/16' 2023-10-16T20:47:11.309 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/183/14' 2023-10-16T20:47:11.309 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/183/18' 2023-10-16T20:47:11.309 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/183/13' 2023-10-16T20:47:11.309 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/183/17' 2023-10-16T20:47:11.310 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/183' 2023-10-16T20:47:11.310 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/177/19' 2023-10-16T20:47:11.310 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/177/15' 2023-10-16T20:47:11.310 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/177/20' 2023-10-16T20:47:11.310 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/177/12' 2023-10-16T20:47:11.311 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/177/10' 2023-10-16T20:47:11.311 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/177/11' 2023-10-16T20:47:11.311 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/177/16' 2023-10-16T20:47:11.311 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/177/14' 2023-10-16T20:47:11.311 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/177/18' 2023-10-16T20:47:11.312 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/177/13' 2023-10-16T20:47:11.312 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/177/17' 2023-10-16T20:47:11.312 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/177' 2023-10-16T20:47:11.312 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/169/19' 2023-10-16T20:47:11.312 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/169/15' 2023-10-16T20:47:11.313 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/169/20' 2023-10-16T20:47:11.313 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/169/12' 2023-10-16T20:47:11.313 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/169/10' 2023-10-16T20:47:11.313 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/169/11' 2023-10-16T20:47:11.313 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/169/16' 2023-10-16T20:47:11.314 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/169/14' 2023-10-16T20:47:11.314 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/169/18' 2023-10-16T20:47:11.314 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/169/13' 2023-10-16T20:47:11.314 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/169/17' 2023-10-16T20:47:11.314 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/169' 2023-10-16T20:47:11.315 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/154/19' 2023-10-16T20:47:11.315 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/154/15' 2023-10-16T20:47:11.315 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/154/20' 2023-10-16T20:47:11.315 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/154/12' 2023-10-16T20:47:11.315 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/154/10' 2023-10-16T20:47:11.316 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/154/11' 2023-10-16T20:47:11.316 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/154/16' 2023-10-16T20:47:11.316 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/154/14' 2023-10-16T20:47:11.316 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/154/18' 2023-10-16T20:47:11.316 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/154/13' 2023-10-16T20:47:11.317 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/154/17' 2023-10-16T20:47:11.317 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/154' 2023-10-16T20:47:11.317 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/171/19' 2023-10-16T20:47:11.317 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/171/15' 2023-10-16T20:47:11.317 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/171/20' 2023-10-16T20:47:11.318 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/171/12' 2023-10-16T20:47:11.318 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/171/10' 2023-10-16T20:47:11.318 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/171/11' 2023-10-16T20:47:11.318 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/171/16' 2023-10-16T20:47:11.318 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/171/14' 2023-10-16T20:47:11.319 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/171/18' 2023-10-16T20:47:11.319 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/171/13' 2023-10-16T20:47:11.319 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/171/17' 2023-10-16T20:47:11.319 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/171' 2023-10-16T20:47:11.319 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/139/19' 2023-10-16T20:47:11.320 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/139/15' 2023-10-16T20:47:11.320 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/139/20' 2023-10-16T20:47:12.848 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/139/12' 2023-10-16T20:47:12.848 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/139/10' 2023-10-16T20:47:12.848 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/139/11' 2023-10-16T20:47:12.849 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/139/16' 2023-10-16T20:47:12.849 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/139/14' 2023-10-16T20:47:12.849 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/139/18' 2023-10-16T20:47:12.849 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/139/13' 2023-10-16T20:47:12.849 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/139/17' 2023-10-16T20:47:12.850 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/139' 2023-10-16T20:47:12.850 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/102/19' 2023-10-16T20:47:12.850 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/102/15' 2023-10-16T20:47:12.850 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/102/20' 2023-10-16T20:47:12.851 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/102/12' 2023-10-16T20:47:12.851 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/102/10' 2023-10-16T20:47:12.851 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/102/11' 2023-10-16T20:47:12.851 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/102/16' 2023-10-16T20:47:12.851 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/102/14' 2023-10-16T20:47:12.851 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/102/18' 2023-10-16T20:47:12.852 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/102/13' 2023-10-16T20:47:12.852 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/102/17' 2023-10-16T20:47:12.852 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/102' 2023-10-16T20:47:12.852 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/129/19' 2023-10-16T20:47:12.852 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/129/15' 2023-10-16T20:47:12.852 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/129/20' 2023-10-16T20:47:12.853 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/129/12' 2023-10-16T20:47:12.853 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/129/10' 2023-10-16T20:47:12.853 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/129/11' 2023-10-16T20:47:12.853 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/129/16' 2023-10-16T20:47:12.853 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/129/14' 2023-10-16T20:47:12.854 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/129/18' 2023-10-16T20:47:12.854 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/129/13' 2023-10-16T20:47:12.854 INFO:tasks.workunit.client.0.smithi088.stdout:removed './copyofsnap1/test1/129/17' 2023-10-16T20:47:12.854 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1/129' 2023-10-16T20:47:12.854 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1/test1' 2023-10-16T20:47:12.855 INFO:tasks.workunit.client.0.smithi088.stdout:removed directory './copyofsnap1' 2023-10-16T20:47:12.855 INFO:tasks.workunit.client.0.smithi088.stdout:OK 2023-10-16T20:47:12.856 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-16T20:47:12.856 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-16T20:47:12.891 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2023-10-16T20:47:12.891 DEBUG:teuthology.orchestra.run.smithi088:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=19fd96fe3581520b6d9f50ffdabf29051aee5d4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2023-10-16T20:47:12.956 INFO:tasks.workunit.client.0.smithi088.stderr:+ set -e 2023-10-16T20:47:12.956 INFO:tasks.workunit.client.0.smithi088.stderr:+ touch foo 2023-10-16T20:47:12.959 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod +x foo 2023-10-16T20:47:12.961 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir .snap/s 2023-10-16T20:47:12.992 INFO:tasks.workunit.client.0.smithi088.stderr:+ find .snap/s/foo -executable 2023-10-16T20:47:12.993 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep foo 2023-10-16T20:47:13.013 INFO:tasks.workunit.client.0.smithi088.stdout:.snap/s/foo 2023-10-16T20:47:13.014 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir .snap/s 2023-10-16T20:47:13.026 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm foo 2023-10-16T20:47:13.030 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo OK 2023-10-16T20:47:13.031 INFO:tasks.workunit.client.0.smithi088.stdout:OK 2023-10-16T20:47:13.031 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-16T20:47:13.032 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-16T20:47:13.106 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2023-10-16T20:47:13.106 DEBUG:teuthology.orchestra.run.smithi088:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=19fd96fe3581520b6d9f50ffdabf29051aee5d4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2023-10-16T20:47:13.168 INFO:tasks.workunit.client.0.smithi088.stderr:+ set -e 2023-10-16T20:47:13.168 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir foo 2023-10-16T20:47:13.173 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo x 2023-10-16T20:47:13.174 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:14.222 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir foo/.snap/ss 2023-10-16T20:47:14.850 INFO:tasks.workunit.client.0.smithi088.stderr:+ ln foo/x foo/xx 2023-10-16T20:47:14.855 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat foo/.snap/ss/x 2023-10-16T20:47:14.866 INFO:tasks.workunit.client.0.smithi088.stdout:x 2023-10-16T20:47:14.866 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir foo/.snap/ss 2023-10-16T20:47:14.879 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo a 2023-10-16T20:47:14.881 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo b 2023-10-16T20:47:14.882 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir foo/.snap/s 2023-10-16T20:47:14.895 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat foo/.snap/s/a 2023-10-16T20:47:14.906 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=a 2023-10-16T20:47:14.907 INFO:tasks.workunit.client.0.smithi088.stderr:+ [ -z a ] 2023-10-16T20:47:14.907 INFO:tasks.workunit.client.0.smithi088.stderr:+ ln foo/b foo/b2 2023-10-16T20:47:14.911 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat foo/.snap/s/b 2023-10-16T20:47:14.920 INFO:tasks.workunit.client.0.smithi088.stdout:b 2023-10-16T20:47:14.921 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo this used to hang: 2023-10-16T20:47:14.921 INFO:tasks.workunit.client.0.smithi088.stdout:this used to hang: 2023-10-16T20:47:14.922 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo more 2023-10-16T20:47:14.924 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo oh, it didn't hang! good job. 2023-10-16T20:47:14.925 INFO:tasks.workunit.client.0.smithi088.stdout:oh, it didn't hang! good job. 2023-10-16T20:47:14.925 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat foo/b 2023-10-16T20:47:14.927 INFO:tasks.workunit.client.0.smithi088.stdout:b 2023-10-16T20:47:14.927 INFO:tasks.workunit.client.0.smithi088.stdout:more 2023-10-16T20:47:14.928 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir foo/.snap/s 2023-10-16T20:47:14.937 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm -r foo 2023-10-16T20:47:14.964 INFO:tasks.workunit.client.0.smithi088.stdout:OK 2023-10-16T20:47:14.965 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo OK 2023-10-16T20:47:14.965 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-16T20:47:14.966 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-16T20:47:15.046 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2023-10-16T20:47:15.046 DEBUG:teuthology.orchestra.run.smithi088:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=19fd96fe3581520b6d9f50ffdabf29051aee5d4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2023-10-16T20:47:15.112 INFO:tasks.workunit.client.0.smithi088.stderr:+ set -e 2023-10-16T20:47:15.112 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:15.115 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir .snap/bar 2023-10-16T20:47:15.928 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/foo 2023-10-16T20:47:15.937 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/foo 2023-10-16T20:47:15.946 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:15.951 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/bar/a 2023-10-16T20:47:15.959 INFO:tasks.workunit.client.0.smithi088.stdout: File: .snap/bar/a 2023-10-16T20:47:15.959 INFO:tasks.workunit.client.0.smithi088.stdout: Size: 0 Blocks: 1 IO Block: 4096 directory 2023-10-16T20:47:15.959 INFO:tasks.workunit.client.0.smithi088.stdout:Device: 2bh/43d Inode: 845524441765455 Links: 2 2023-10-16T20:47:15.960 INFO:tasks.workunit.client.0.smithi088.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1271/ ubuntu) 2023-10-16T20:47:15.960 INFO:tasks.workunit.client.0.smithi088.stdout:Access: 2023-10-16 20:47:15.115318308 +0000 2023-10-16T20:47:15.960 INFO:tasks.workunit.client.0.smithi088.stdout:Modify: 2023-10-16 20:47:15.115318308 +0000 2023-10-16T20:47:15.960 INFO:tasks.workunit.client.0.smithi088.stdout:Change: 2023-10-16 20:47:15.115318308 +0000 2023-10-16T20:47:15.960 INFO:tasks.workunit.client.0.smithi088.stdout: Birth: - 2023-10-16T20:47:15.961 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir .snap/bar 2023-10-16T20:47:16.181 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo OK 2023-10-16T20:47:16.182 INFO:tasks.workunit.client.0.smithi088.stdout:OK 2023-10-16T20:47:16.182 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-16T20:47:16.183 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-16T20:47:16.255 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2023-10-16T20:47:16.255 DEBUG:teuthology.orchestra.run.smithi088:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=19fd96fe3581520b6d9f50ffdabf29051aee5d4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2023-10-16T20:47:16.329 INFO:tasks.workunit.client.0.smithi088.stderr:+ set -e 2023-10-16T20:47:16.330 INFO:tasks.workunit.client.0.smithi088.stderr:+ seq 1 20 2023-10-16T20:47:16.331 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:16.335 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:16.339 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:16.352 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:16.361 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:16.363 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:16.535 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:16.550 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:16.551 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:16.551 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:16.551 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:16.551 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:16.551 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:16.552 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:16.552 INFO:tasks.workunit.client.0.smithi088.stdout: 51253 ? 00:00:00 ps 2023-10-16T20:47:16.555 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:16.575 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:16.581 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:16.588 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:16.592 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:16.595 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:16.606 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:17.754 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:17.759 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:17.934 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:17.949 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:17.949 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:17.949 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:17.949 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:17.949 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:17.950 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:17.950 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:17.950 INFO:tasks.workunit.client.0.smithi088.stdout: 51263 ? 00:00:00 ps 2023-10-16T20:47:17.953 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:17.975 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:17.981 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:17.984 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:17.986 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:17.996 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:18.005 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:18.007 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:18.175 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:18.189 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:18.189 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:18.190 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:18.190 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:18.190 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:18.190 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:18.190 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:18.191 INFO:tasks.workunit.client.0.smithi088.stdout: 51273 ? 00:00:00 ps 2023-10-16T20:47:18.194 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:18.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:18.216 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:18.224 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:18.227 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:18.229 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:18.239 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:18.470 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:18.473 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:18.542 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:18.557 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:18.558 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:18.558 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:18.558 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:18.558 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:18.559 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:18.559 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:18.559 INFO:tasks.workunit.client.0.smithi088.stdout: 51283 ? 00:00:00 ps 2023-10-16T20:47:18.561 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:19.099 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:19.104 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:19.111 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:19.114 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:19.116 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:19.128 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:19.137 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:19.139 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:19.218 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:19.232 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:19.233 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:19.233 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:19.233 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:19.233 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:19.233 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:19.234 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:19.234 INFO:tasks.workunit.client.0.smithi088.stdout: 51293 ? 00:00:00 ps 2023-10-16T20:47:19.237 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:19.254 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:19.259 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:19.266 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:19.269 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:19.271 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:19.283 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:19.292 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:19.294 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:19.409 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:19.423 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:19.424 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:19.424 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:19.424 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:19.424 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:19.424 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:19.425 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:19.425 INFO:tasks.workunit.client.0.smithi088.stdout: 51303 ? 00:00:00 ps 2023-10-16T20:47:19.428 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:19.445 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:19.449 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:19.456 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:19.460 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:19.462 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:19.473 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:19.482 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:19.484 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:19.543 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:19.557 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:19.557 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:19.558 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:19.558 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:19.558 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:19.558 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:19.559 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:19.559 INFO:tasks.workunit.client.0.smithi088.stdout: 51313 ? 00:00:00 ps 2023-10-16T20:47:19.563 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:19.580 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:19.584 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:19.592 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:19.595 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:19.597 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:19.609 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:19.618 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:19.620 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:19.701 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:19.716 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:19.717 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:19.717 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:19.717 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:19.717 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:19.717 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:19.718 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:19.718 INFO:tasks.workunit.client.0.smithi088.stdout: 51323 ? 00:00:00 ps 2023-10-16T20:47:19.722 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:19.952 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:19.957 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:19.964 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:19.967 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:20.200 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:20.212 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:20.221 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:20.223 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:20.311 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:20.325 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:20.325 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:20.325 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:20.326 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:20.326 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:20.326 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:20.326 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:20.326 INFO:tasks.workunit.client.0.smithi088.stdout: 51333 ? 00:00:00 ps 2023-10-16T20:47:20.330 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:20.741 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:20.746 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:20.753 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:20.756 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:20.758 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:20.769 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:20.779 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:20.781 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:20.878 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:20.892 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:20.892 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:20.892 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:20.893 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:20.893 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:20.893 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:20.893 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:20.893 INFO:tasks.workunit.client.0.smithi088.stdout: 51343 ? 00:00:00 ps 2023-10-16T20:47:20.897 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:20.913 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:20.917 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:20.924 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:20.927 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:20.930 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:20.941 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:20.950 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:20.952 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:21.011 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:21.025 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:21.025 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:21.025 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:21.026 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:21.026 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:21.026 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:21.026 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:21.026 INFO:tasks.workunit.client.0.smithi088.stdout: 51353 ? 00:00:00 ps 2023-10-16T20:47:21.029 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:21.047 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:21.050 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:21.056 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:21.059 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:21.061 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:21.192 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:21.201 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:21.203 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:21.296 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:21.310 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:21.310 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:21.311 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:21.311 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:21.311 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:21.311 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:21.311 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:21.312 INFO:tasks.workunit.client.0.smithi088.stdout: 51363 ? 00:00:00 ps 2023-10-16T20:47:21.315 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:21.702 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:21.705 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:21.711 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:21.714 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:21.717 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:21.728 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:21.737 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:21.740 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:21.829 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:21.844 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:21.844 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:21.844 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:21.844 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:21.844 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:21.845 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:21.845 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:21.845 INFO:tasks.workunit.client.0.smithi088.stdout: 51373 ? 00:00:00 ps 2023-10-16T20:47:22.566 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:22.593 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:22.601 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:22.621 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:22.634 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:22.643 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:22.716 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:22.783 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:22.800 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:23.279 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:23.294 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:23.294 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:23.294 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:23.294 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:23.295 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:23.295 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:23.295 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:23.295 INFO:tasks.workunit.client.0.smithi088.stdout: 51383 ? 00:00:00 ps 2023-10-16T20:47:23.298 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:23.316 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:23.320 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:23.325 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:23.329 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:23.332 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:23.342 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:23.350 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:23.352 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:23.535 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:23.549 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:23.549 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:23.549 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:23.549 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:23.550 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:23.550 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:23.550 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:23.550 INFO:tasks.workunit.client.0.smithi088.stdout: 51393 ? 00:00:00 ps 2023-10-16T20:47:23.554 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:23.570 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:23.574 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:23.579 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:23.582 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:23.585 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:23.595 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:23.609 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:23.613 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:23.697 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:23.711 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:23.711 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:23.711 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:23.711 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:23.712 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:23.712 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:23.712 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:23.712 INFO:tasks.workunit.client.0.smithi088.stdout: 51403 ? 00:00:00 ps 2023-10-16T20:47:23.715 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:23.737 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:23.740 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:23.745 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:23.748 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:23.750 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:23.984 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:24.659 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:24.661 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:24.806 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:24.821 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:24.821 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:24.821 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:24.821 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:24.821 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:24.822 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:24.822 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:24.822 INFO:tasks.workunit.client.0.smithi088.stdout: 51413 ? 00:00:00 ps 2023-10-16T20:47:24.826 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:25.297 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:25.301 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:25.306 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:25.309 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:25.313 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:25.506 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:25.524 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:25.526 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:25.772 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:25.787 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:25.788 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:25.788 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:25.788 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:25.788 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:25.788 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:25.789 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:25.789 INFO:tasks.workunit.client.0.smithi088.stdout: 51423 ? 00:00:00 ps 2023-10-16T20:47:25.791 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:25.808 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:25.812 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:25.936 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:25.939 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:25.941 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:25.952 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:25.961 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:25.963 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:26.089 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:26.104 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:26.104 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:26.104 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:26.105 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:26.105 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:26.105 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:26.105 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:26.105 INFO:tasks.workunit.client.0.smithi088.stdout: 51433 ? 00:00:00 ps 2023-10-16T20:47:26.109 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:26.126 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:26.130 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:26.137 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a 2023-10-16T20:47:26.141 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/one 2023-10-16T20:47:26.143 INFO:tasks.workunit.client.0.smithi088.stderr:+ cat 2023-10-16T20:47:26.153 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir a/.snap/two 2023-10-16T20:47:26.163 INFO:tasks.workunit.client.0.smithi088.stderr:+ chmod 777 a/foo 2023-10-16T20:47:26.165 INFO:tasks.workunit.client.0.smithi088.stderr:+ sync 2023-10-16T20:47:26.232 INFO:tasks.workunit.client.0.smithi088.stderr:+ ps 2023-10-16T20:47:26.246 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2023-10-16T20:47:26.246 INFO:tasks.workunit.client.0.smithi088.stdout: 1297 ? 00:00:00 systemd 2023-10-16T20:47:26.247 INFO:tasks.workunit.client.0.smithi088.stdout: 1298 ? 00:00:00 (sd-pam) 2023-10-16T20:47:26.247 INFO:tasks.workunit.client.0.smithi088.stdout: 28017 ? 00:00:02 sshd 2023-10-16T20:47:26.247 INFO:tasks.workunit.client.0.smithi088.stdout: 46438 ? 00:00:00 bash 2023-10-16T20:47:26.247 INFO:tasks.workunit.client.0.smithi088.stdout: 51240 ? 00:00:00 timeout 2023-10-16T20:47:26.247 INFO:tasks.workunit.client.0.smithi088.stdout: 51243 ? 00:00:00 snaptest-double 2023-10-16T20:47:26.248 INFO:tasks.workunit.client.0.smithi088.stdout: 51443 ? 00:00:00 ps 2023-10-16T20:47:26.250 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-16T20:47:26.268 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm a/foo 2023-10-16T20:47:26.272 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir a 2023-10-16T20:47:26.278 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo OK 2023-10-16T20:47:26.278 INFO:tasks.workunit.client.0.smithi088.stdout:OK 2023-10-16T20:47:26.279 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-16T20:47:26.279 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-16T20:47:27.274 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2023-10-16T20:47:27.274 DEBUG:teuthology.orchestra.run.smithi088:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=19fd96fe3581520b6d9f50ffdabf29051aee5d4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2023-10-16T20:47:27.297 INFO:tasks.workunit.client.0.smithi088.stderr:+ mkdir .snap/foo 2023-10-16T20:47:27.311 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo We want ENOENT, not ESTALE, here. 2023-10-16T20:47:27.311 INFO:tasks.workunit.client.0.smithi088.stdout:We want ENOENT, not ESTALE, here. 2023-10-16T20:47:27.312 INFO:tasks.workunit.client.0.smithi088.stderr:+ seq 1 100 2023-10-16T20:47:27.313 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/1 2023-10-16T20:47:27.313 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.317 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2023-10-16T20:47:27.318 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/2 2023-10-16T20:47:27.318 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.321 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2023-10-16T20:47:27.322 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/3 2023-10-16T20:47:27.322 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.326 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2023-10-16T20:47:27.326 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/4 2023-10-16T20:47:27.327 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.331 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2023-10-16T20:47:27.332 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/5 2023-10-16T20:47:27.332 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.336 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2023-10-16T20:47:27.337 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/6 2023-10-16T20:47:27.337 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.341 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2023-10-16T20:47:27.342 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/7 2023-10-16T20:47:27.342 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.346 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2023-10-16T20:47:27.347 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/8 2023-10-16T20:47:27.347 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.351 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2023-10-16T20:47:27.352 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/9 2023-10-16T20:47:27.352 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.357 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2023-10-16T20:47:27.358 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/10 2023-10-16T20:47:27.358 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.360 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2023-10-16T20:47:27.361 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/11 2023-10-16T20:47:27.361 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.365 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2023-10-16T20:47:27.366 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/12 2023-10-16T20:47:27.366 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.369 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2023-10-16T20:47:27.370 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/13 2023-10-16T20:47:27.370 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.373 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2023-10-16T20:47:27.374 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/14 2023-10-16T20:47:27.374 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.378 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2023-10-16T20:47:27.379 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/15 2023-10-16T20:47:27.379 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.382 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2023-10-16T20:47:27.383 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/16 2023-10-16T20:47:27.383 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.387 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2023-10-16T20:47:27.388 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/17 2023-10-16T20:47:27.388 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.391 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2023-10-16T20:47:27.392 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/18 2023-10-16T20:47:27.392 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.395 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2023-10-16T20:47:27.396 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/19 2023-10-16T20:47:27.396 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.400 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2023-10-16T20:47:27.401 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/20 2023-10-16T20:47:27.401 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.404 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2023-10-16T20:47:27.405 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/21 2023-10-16T20:47:27.406 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.409 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2023-10-16T20:47:27.410 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/22 2023-10-16T20:47:27.410 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.413 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2023-10-16T20:47:27.414 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/23 2023-10-16T20:47:27.414 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.417 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2023-10-16T20:47:27.418 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/24 2023-10-16T20:47:27.418 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.421 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2023-10-16T20:47:27.422 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/25 2023-10-16T20:47:27.422 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.426 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2023-10-16T20:47:27.427 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/26 2023-10-16T20:47:27.427 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.430 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2023-10-16T20:47:27.431 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/27 2023-10-16T20:47:27.432 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.434 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2023-10-16T20:47:27.435 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/28 2023-10-16T20:47:27.435 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.439 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2023-10-16T20:47:27.440 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/29 2023-10-16T20:47:27.440 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.443 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2023-10-16T20:47:27.444 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/30 2023-10-16T20:47:27.444 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.447 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2023-10-16T20:47:27.448 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/31 2023-10-16T20:47:27.448 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.451 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2023-10-16T20:47:27.452 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/32 2023-10-16T20:47:27.452 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.455 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2023-10-16T20:47:27.456 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/33 2023-10-16T20:47:27.456 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.459 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2023-10-16T20:47:27.460 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/34 2023-10-16T20:47:27.461 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.464 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2023-10-16T20:47:27.465 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/35 2023-10-16T20:47:27.465 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.468 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2023-10-16T20:47:27.469 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/36 2023-10-16T20:47:27.469 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.473 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2023-10-16T20:47:27.475 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/37 2023-10-16T20:47:27.475 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.485 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2023-10-16T20:47:27.486 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/38 2023-10-16T20:47:27.487 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.496 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2023-10-16T20:47:27.500 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.500 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/39 2023-10-16T20:47:27.507 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2023-10-16T20:47:27.509 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/40 2023-10-16T20:47:27.509 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.515 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2023-10-16T20:47:27.516 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/41 2023-10-16T20:47:27.517 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.525 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2023-10-16T20:47:27.526 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/42 2023-10-16T20:47:27.526 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.529 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2023-10-16T20:47:27.530 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/43 2023-10-16T20:47:27.530 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.533 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2023-10-16T20:47:27.534 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/44 2023-10-16T20:47:27.534 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.538 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2023-10-16T20:47:27.539 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/45 2023-10-16T20:47:27.539 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.542 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2023-10-16T20:47:27.543 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/46 2023-10-16T20:47:27.543 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.546 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2023-10-16T20:47:27.547 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/47 2023-10-16T20:47:27.547 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.552 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2023-10-16T20:47:27.552 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/48 2023-10-16T20:47:27.553 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.556 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2023-10-16T20:47:27.557 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/49 2023-10-16T20:47:27.557 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.557 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:47:27.562 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:47:27.565 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2023-10-16T20:47:27.565 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/50 2023-10-16T20:47:27.566 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.566 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2023-10-16T20:47:27.567 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/51 2023-10-16T20:47:27.567 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.570 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2023-10-16T20:47:27.570 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/52 2023-10-16T20:47:27.571 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.574 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2023-10-16T20:47:27.575 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/53 2023-10-16T20:47:27.575 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.578 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2023-10-16T20:47:27.579 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/54 2023-10-16T20:47:27.579 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.583 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2023-10-16T20:47:27.584 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/55 2023-10-16T20:47:27.584 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.586 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2023-10-16T20:47:27.587 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/56 2023-10-16T20:47:27.588 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.590 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2023-10-16T20:47:27.591 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/57 2023-10-16T20:47:27.591 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.595 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2023-10-16T20:47:27.596 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/58 2023-10-16T20:47:27.596 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.599 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2023-10-16T20:47:27.600 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/59 2023-10-16T20:47:27.600 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.603 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2023-10-16T20:47:27.604 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/60 2023-10-16T20:47:27.604 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.607 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2023-10-16T20:47:27.608 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/61 2023-10-16T20:47:27.608 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.612 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2023-10-16T20:47:27.612 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/62 2023-10-16T20:47:27.612 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.615 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2023-10-16T20:47:27.616 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/63 2023-10-16T20:47:27.616 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.619 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2023-10-16T20:47:27.619 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/64 2023-10-16T20:47:27.620 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.623 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2023-10-16T20:47:27.624 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/65 2023-10-16T20:47:27.624 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.626 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2023-10-16T20:47:27.628 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/66 2023-10-16T20:47:27.628 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.630 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2023-10-16T20:47:27.631 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/67 2023-10-16T20:47:27.632 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.635 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2023-10-16T20:47:27.636 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/68 2023-10-16T20:47:27.636 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.638 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2023-10-16T20:47:27.639 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/69 2023-10-16T20:47:27.639 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.643 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2023-10-16T20:47:27.643 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/70 2023-10-16T20:47:27.644 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.647 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2023-10-16T20:47:27.647 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/71 2023-10-16T20:47:27.648 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.651 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2023-10-16T20:47:27.652 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/72 2023-10-16T20:47:27.652 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.655 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2023-10-16T20:47:27.656 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/73 2023-10-16T20:47:27.656 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.659 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2023-10-16T20:47:27.660 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/74 2023-10-16T20:47:27.660 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.663 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2023-10-16T20:47:27.664 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/75 2023-10-16T20:47:27.664 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.666 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2023-10-16T20:47:27.667 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/76 2023-10-16T20:47:27.668 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.671 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2023-10-16T20:47:27.672 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/77 2023-10-16T20:47:27.672 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.675 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2023-10-16T20:47:27.676 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/78 2023-10-16T20:47:27.676 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.678 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2023-10-16T20:47:27.680 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/79 2023-10-16T20:47:27.680 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.682 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2023-10-16T20:47:27.683 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/80 2023-10-16T20:47:27.683 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.686 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2023-10-16T20:47:27.687 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/81 2023-10-16T20:47:27.688 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.690 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2023-10-16T20:47:27.691 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/82 2023-10-16T20:47:27.691 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.695 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2023-10-16T20:47:27.696 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/83 2023-10-16T20:47:27.696 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.699 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2023-10-16T20:47:27.700 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/84 2023-10-16T20:47:27.700 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.703 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2023-10-16T20:47:27.704 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/85 2023-10-16T20:47:27.704 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.707 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2023-10-16T20:47:27.708 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/86 2023-10-16T20:47:27.708 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.710 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2023-10-16T20:47:27.711 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/87 2023-10-16T20:47:27.712 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.715 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2023-10-16T20:47:27.716 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/88 2023-10-16T20:47:27.717 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.719 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2023-10-16T20:47:27.720 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/89 2023-10-16T20:47:27.721 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.724 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2023-10-16T20:47:27.725 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/90 2023-10-16T20:47:27.725 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.728 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2023-10-16T20:47:27.729 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/91 2023-10-16T20:47:27.729 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.732 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2023-10-16T20:47:27.734 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/92 2023-10-16T20:47:27.734 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.736 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2023-10-16T20:47:27.737 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/93 2023-10-16T20:47:27.738 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.742 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2023-10-16T20:47:27.743 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/94 2023-10-16T20:47:27.743 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.745 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2023-10-16T20:47:27.746 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/95 2023-10-16T20:47:27.747 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.749 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2023-10-16T20:47:27.750 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/96 2023-10-16T20:47:27.750 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.753 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2023-10-16T20:47:27.754 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/97 2023-10-16T20:47:27.754 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.757 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2023-10-16T20:47:27.758 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/98 2023-10-16T20:47:27.758 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.762 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2023-10-16T20:47:27.763 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/99 2023-10-16T20:47:27.765 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.766 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2023-10-16T20:47:27.767 INFO:tasks.workunit.client.0.smithi088.stderr:+ stat .snap/foo/100 2023-10-16T20:47:27.767 INFO:tasks.workunit.client.0.smithi088.stderr:+ grep No such file 2023-10-16T20:47:27.770 INFO:tasks.workunit.client.0.smithi088.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2023-10-16T20:47:27.770 INFO:tasks.workunit.client.0.smithi088.stderr:+ rmdir .snap/foo 2023-10-16T20:47:27.881 INFO:tasks.workunit.client.0.smithi088.stdout:OK 2023-10-16T20:47:27.883 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo OK 2023-10-16T20:47:27.883 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-16T20:47:27.883 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-16T20:47:27.951 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2023-10-16T20:47:27.951 DEBUG:teuthology.orchestra.run.smithi088:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=19fd96fe3581520b6d9f50ffdabf29051aee5d4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2023-10-16T20:47:28.036 INFO:tasks.workunit.client.0.smithi088.stderr:+ set -e 2023-10-16T20:47:28.036 INFO:tasks.workunit.client.0.smithi088.stderr:+ sudo git config --global http.sslVerify false 2023-10-16T20:47:28.075 INFO:tasks.workunit.client.0.smithi088.stderr:+ sudo git config --global http.postBuffer 1048576000 2023-10-16T20:47:28.099 INFO:tasks.workunit.client.0.smithi088.stderr:+ retried=false 2023-10-16T20:47:28.099 INFO:tasks.workunit.client.0.smithi088.stderr:+ trap -- retry EXIT 2023-10-16T20:47:28.099 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm -rf ceph 2023-10-16T20:47:28.100 INFO:tasks.workunit.client.0.smithi088.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2023-10-16T20:47:28.107 INFO:tasks.workunit.client.0.smithi088.stderr:Cloning into 'ceph'... 2023-10-16T20:47:57.633 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:47:57.638 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:48:27.653 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:48:27.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.805+0000 7fa1e759e640 1 Processor -- start 2023-10-16T20:48:27.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.805+0000 7fa1e759e640 1 -- start start 2023-10-16T20:48:27.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.809+0000 7fa1e759e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1e80c9af0 0x7fa1e80cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:27.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.809+0000 7fa1e759e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa1e80b1a60 0x7fa1e8163b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:27.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.809+0000 7fa1e759e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e80c9fc0 con 0x7fa1e80cb580 2023-10-16T20:48:27.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.809+0000 7fa1e759e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e80b20c0 con 0x7fa1e80b1a60 2023-10-16T20:48:27.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.809+0000 7fa1e759e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e80cbea0 con 0x7fa1e80c9af0 2023-10-16T20:48:27.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.809+0000 7fa1e659c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa1e80cb580 0x7fa1e8159300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35212/0 (socket says 172.21.15.88:35212) 2023-10-16T20:48:27.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.809+0000 7fa1e659c640 1 -- 172.21.15.88:0/1974478046 learned_addr learned my addr 172.21.15.88:0/1974478046 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:48:27.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.809+0000 7fa1e6d9d640 1 --2- 172.21.15.88:0/1974478046 >> v2:172.21.15.88:3300/0 conn(0x7fa1e80b1a60 0x7fa1e8163b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:27.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.809+0000 7fa1e5d9b640 1 --2- 172.21.15.88:0/1974478046 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1e80c9af0 0x7fa1e80cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:27.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.809+0000 7fa1e559a640 1 -- 172.21.15.88:0/1974478046 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257493246 0 0) 0x7fa1e80c9fc0 con 0x7fa1e80cb580 2023-10-16T20:48:27.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.809+0000 7fa1e559a640 1 -- 172.21.15.88:0/1974478046 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d0003530 con 0x7fa1e80cb580 2023-10-16T20:48:27.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.809+0000 7fa1e6d9d640 1 -- 172.21.15.88:0/1974478046 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1e80c9af0 msgr2=0x7fa1e80cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:27.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.809+0000 7fa1e6d9d640 1 --2- 172.21.15.88:0/1974478046 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1e80c9af0 0x7fa1e80cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:27.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.809+0000 7fa1e6d9d640 1 -- 172.21.15.88:0/1974478046 >> v1:172.21.15.88:6789/0 conn(0x7fa1e80cb580 legacy=0x7fa1e8159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:27.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.809+0000 7fa1e6d9d640 1 -- 172.21.15.88:0/1974478046 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e80c47d0 con 0x7fa1e80b1a60 2023-10-16T20:48:27.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.813+0000 7fa1e6d9d640 1 --2- 172.21.15.88:0/1974478046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e80b1a60 0x7fa1e8163b80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009d90 tx=0x7fa1e001c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f00d900cd4a61321 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:27.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.813+0000 7fa1e559a640 1 -- 172.21.15.88:0/1974478046 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e00a5950 con 0x7fa1e80b1a60 2023-10-16T20:48:27.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.813+0000 7fa1e559a640 1 -- 172.21.15.88:0/1974478046 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e00a5ab0 con 0x7fa1e80b1a60 2023-10-16T20:48:27.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.813+0000 7fa1e559a640 1 -- 172.21.15.88:0/1974478046 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e00a5e40 con 0x7fa1e80b1a60 2023-10-16T20:48:27.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.813+0000 7fa1e759e640 1 -- 172.21.15.88:0/1974478046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e80b1a60 msgr2=0x7fa1e8163b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:27.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.813+0000 7fa1e759e640 1 --2- 172.21.15.88:0/1974478046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e80b1a60 0x7fa1e8163b80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009d90 tx=0x7fa1e001c890 comp rx=0 tx=0).stop 2023-10-16T20:48:27.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.813+0000 7fa1e759e640 1 -- 172.21.15.88:0/1974478046 shutdown_connections 2023-10-16T20:48:27.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.813+0000 7fa1e759e640 1 --2- 172.21.15.88:0/1974478046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e80b1a60 0x7fa1e8163b80 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:27.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.813+0000 7fa1e759e640 1 --2- 172.21.15.88:0/1974478046 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1e80c9af0 0x7fa1e80cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:27.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.813+0000 7fa1e759e640 1 -- 172.21.15.88:0/1974478046 >> 172.21.15.88:0/1974478046 conn(0x7fa1e806bb40 msgr2=0x7fa1e806bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:27.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.813+0000 7fa1e759e640 1 -- 172.21.15.88:0/1974478046 shutdown_connections 2023-10-16T20:48:27.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.813+0000 7fa1e759e640 1 -- 172.21.15.88:0/1974478046 wait complete. 2023-10-16T20:48:27.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.813+0000 7fa1e759e640 1 Processor -- start 2023-10-16T20:48:27.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.813+0000 7fa1e759e640 1 -- start start 2023-10-16T20:48:27.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e759e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1e80b1a60 0x7fa1e80b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:27.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e80c9af0 0x7fa1e813d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:27.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e759e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1e80cb580 0x7fa1e81f2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:27.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e759e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e80b3850 con 0x7fa1e80c9af0 2023-10-16T20:48:27.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e759e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e813ddc0 con 0x7fa1e80b1a60 2023-10-16T20:48:27.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e759e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e80b41a0 con 0x7fa1e80cb580 2023-10-16T20:48:27.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e5d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e80c9af0 0x7fa1e813d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:27.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e5d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e80c9af0 0x7fa1e813d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46478/0 (socket says 172.21.15.88:46478) 2023-10-16T20:48:27.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e5d9b640 1 -- 172.21.15.88:0/1151856824 learned_addr learned my addr 172.21.15.88:0/1151856824 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:27.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e659c640 1 --2- 172.21.15.88:0/1151856824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1e80b1a60 0x7fa1e80b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:27.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e6d9d640 1 --2- 172.21.15.88:0/1151856824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1e80cb580 0x7fa1e81f2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:27.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e5d9b640 1 -- 172.21.15.88:0/1151856824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1e80cb580 msgr2=0x7fa1e81f2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:27.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e5d9b640 1 --2- 172.21.15.88:0/1151856824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1e80cb580 0x7fa1e81f2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:27.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e5d9b640 1 -- 172.21.15.88:0/1151856824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1e80b1a60 msgr2=0x7fa1e80b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:27.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e5d9b640 1 --2- 172.21.15.88:0/1151856824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1e80b1a60 0x7fa1e80b2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:27.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e5d9b640 1 -- 172.21.15.88:0/1151856824 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e0009a70 con 0x7fa1e80c9af0 2023-10-16T20:48:27.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e6d9d640 1 --2- 172.21.15.88:0/1151856824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1e80cb580 0x7fa1e81f2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:48:27.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e659c640 1 --2- 172.21.15.88:0/1151856824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1e80b1a60 0x7fa1e80b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:48:27.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e5d9b640 1 --2- 172.21.15.88:0/1151856824 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e80c9af0 0x7fa1e813d880 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00e900 tx=0x7fa1dc00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:27.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1d6ffd640 1 -- 172.21.15.88:0/1151856824 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc009690 con 0x7fa1e80c9af0 2023-10-16T20:48:27.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1d6ffd640 1 -- 172.21.15.88:0/1151856824 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc00ca00 con 0x7fa1e80c9af0 2023-10-16T20:48:27.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e759e640 1 -- 172.21.15.88:0/1151856824 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e8143530 con 0x7fa1e80c9af0 2023-10-16T20:48:27.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1d6ffd640 1 -- 172.21.15.88:0/1151856824 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc00cdb0 con 0x7fa1e80c9af0 2023-10-16T20:48:27.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e759e640 1 -- 172.21.15.88:0/1151856824 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e8142610 con 0x7fa1e80c9af0 2023-10-16T20:48:27.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.817+0000 7fa1e759e640 1 -- 172.21.15.88:0/1151856824 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e81427a0 con 0x7fa1e80c9af0 2023-10-16T20:48:27.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.821+0000 7fa1d6ffd640 1 -- 172.21.15.88:0/1151856824 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa1dc013470 con 0x7fa1e80c9af0 2023-10-16T20:48:27.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.821+0000 7fa1d6ffd640 1 --2- 172.21.15.88:0/1151856824 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa1c80664d0 0x7fa1c8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:27.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.821+0000 7fa1d6ffd640 1 -- 172.21.15.88:0/1151856824 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7fa1dc084550 con 0x7fa1e80c9af0 2023-10-16T20:48:27.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.821+0000 7fa1e659c640 1 --2- 172.21.15.88:0/1151856824 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa1c80664d0 0x7fa1c8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:27.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.821+0000 7fa1e659c640 1 --2- 172.21.15.88:0/1151856824 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa1c80664d0 0x7fa1c8068990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa1d80011f0 tx=0x7fa1d8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:27.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:27.825+0000 7fa1d6ffd640 1 -- 172.21.15.88:0/1151856824 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa1dc051490 con 0x7fa1e80c9af0 2023-10-16T20:48:28.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.013+0000 7fa1e759e640 1 -- 172.21.15.88:0/1151856824 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1e80ca2c0 con 0x7fa1e80c9af0 2023-10-16T20:48:28.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.013+0000 7fa1d6ffd640 1 -- 172.21.15.88:0/1151856824 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+6586 (secure 0 0 0) 0x7fa1dc054ab0 con 0x7fa1e80c9af0 2023-10-16T20:48:28.017 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:48:28.018 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5709,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:13.446293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:48:28.018 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 64 2023-10-16T20:48:28.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.013+0000 7fa1e759e640 1 -- 172.21.15.88:0/1151856824 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa1c80664d0 msgr2=0x7fa1c8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:28.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.013+0000 7fa1e759e640 1 --2- 172.21.15.88:0/1151856824 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa1c80664d0 0x7fa1c8068990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa1d80011f0 tx=0x7fa1d8009040 comp rx=0 tx=0).stop 2023-10-16T20:48:28.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.013+0000 7fa1e759e640 1 -- 172.21.15.88:0/1151856824 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e80c9af0 msgr2=0x7fa1e813d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:28.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.013+0000 7fa1e759e640 1 --2- 172.21.15.88:0/1151856824 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e80c9af0 0x7fa1e813d880 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00e900 tx=0x7fa1dc00edc0 comp rx=0 tx=0).stop 2023-10-16T20:48:28.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.013+0000 7fa1e759e640 1 -- 172.21.15.88:0/1151856824 shutdown_connections 2023-10-16T20:48:28.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.013+0000 7fa1e759e640 1 --2- 172.21.15.88:0/1151856824 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa1c80664d0 0x7fa1c8068990 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:28.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.013+0000 7fa1e759e640 1 --2- 172.21.15.88:0/1151856824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa1e80cb580 0x7fa1e81f2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:28.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.013+0000 7fa1e759e640 1 --2- 172.21.15.88:0/1151856824 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e80c9af0 0x7fa1e813d880 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:28.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.013+0000 7fa1e759e640 1 --2- 172.21.15.88:0/1151856824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa1e80b1a60 0x7fa1e80b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:28.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.013+0000 7fa1e759e640 1 -- 172.21.15.88:0/1151856824 >> 172.21.15.88:0/1151856824 conn(0x7fa1e806bb40 msgr2=0x7fa1e806bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:28.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.013+0000 7fa1e759e640 1 -- 172.21.15.88:0/1151856824 shutdown_connections 2023-10-16T20:48:28.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.013+0000 7fa1e759e640 1 -- 172.21.15.88:0/1151856824 wait complete. 2023-10-16T20:48:28.038 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=1) 2023-10-16T20:48:28.039 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T20:48:28.039 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T20:48:28.056 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T20:48:28.057 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:48:28.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.189+0000 7fcdf1817640 1 Processor -- start 2023-10-16T20:48:28.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.189+0000 7fcdf1817640 1 -- start start 2023-10-16T20:48:28.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdf1817640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcdec153770 0x7fcdec159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdf1817640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdec15a410 con 0x7fcdec0cece0 2023-10-16T20:48:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdf1817640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdec153b50 con 0x7fcdec0c9bb0 2023-10-16T20:48:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdf1817640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdec0cc250 con 0x7fcdec153770 2023-10-16T20:48:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdeb7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcdec0cece0 0x7fcdec163dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35214/0 (socket says 172.21.15.88:35214) 2023-10-16T20:48:28.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdeb7fe640 1 -- 172.21.15.88:0/3740154106 learned_addr learned my addr 172.21.15.88:0/3740154106 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:48:28.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdeaffd640 1 --2- 172.21.15.88:0/3740154106 >> v2:172.21.15.191:3300/0 conn(0x7fcdec153770 0x7fcdec159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:28.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdea7fc640 1 -- 172.21.15.88:0/3740154106 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499745070 0 0) 0x7fcdec15a410 con 0x7fcdec0cece0 2023-10-16T20:48:28.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdea7fc640 1 -- 172.21.15.88:0/3740154106 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd8003530 con 0x7fcdec0cece0 2023-10-16T20:48:28.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdea7fc640 1 -- 172.21.15.88:0/3740154106 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1628116791 0 0) 0x7fcdd8003530 con 0x7fcdec0cece0 2023-10-16T20:48:28.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdea7fc640 1 -- 172.21.15.88:0/3740154106 >> v2:172.21.15.191:3300/0 conn(0x7fcdec153770 msgr2=0x7fcdec159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:28.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdea7fc640 1 --2- 172.21.15.88:0/3740154106 >> v2:172.21.15.191:3300/0 conn(0x7fcdec153770 0x7fcdec159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:28.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdea7fc640 1 -- 172.21.15.88:0/3740154106 >> v1:172.21.15.191:6789/0 conn(0x7fcdec0c9bb0 legacy=0x7fcdec0c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:28.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdea7fc640 1 -- 172.21.15.88:0/3740154106 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec0cb2f0 con 0x7fcdec0cece0 2023-10-16T20:48:28.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdea7fc640 1 -- 172.21.15.88:0/3740154106 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7fcde4001680 con 0x7fcdec0cece0 2023-10-16T20:48:28.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdea7fc640 1 -- 172.21.15.88:0/3740154106 >> v1:172.21.15.88:6789/0 conn(0x7fcdec0cece0 legacy=0x7fcdec163dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:28.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdea7fc640 1 --2- 172.21.15.88:0/3740154106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdd8003980 0x7fcdd8003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:28.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdea7fc640 1 --2- 172.21.15.88:0/3740154106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdd8004380 0x7fcdd8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:28.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdea7fc640 1 --2- 172.21.15.88:0/3740154106 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdd8005050 0x7fcdd800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:28.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdea7fc640 1 -- 172.21.15.88:0/3740154106 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec0cc250 con 0x7fcdd8004380 2023-10-16T20:48:28.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdea7fc640 1 -- 172.21.15.88:0/3740154106 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd800bfb0 con 0x7fcdd8003980 2023-10-16T20:48:28.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdea7fc640 1 -- 172.21.15.88:0/3740154106 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd800c170 con 0x7fcdd8005050 2023-10-16T20:48:28.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcde27fc640 1 --2- 172.21.15.88:0/3740154106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdd8004380 0x7fcdd8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:28.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdeaffd640 1 --2- 172.21.15.88:0/3740154106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdd8003980 0x7fcdd8003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:28.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdeb7fe640 1 --2- 172.21.15.88:0/3740154106 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdd8005050 0x7fcdd800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:28.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcde27fc640 1 -- 172.21.15.88:0/3740154106 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdd8005050 msgr2=0x7fcdd800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:28.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcde27fc640 1 --2- 172.21.15.88:0/3740154106 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdd8005050 0x7fcdd800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:28.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcde27fc640 1 -- 172.21.15.88:0/3740154106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdd8003980 msgr2=0x7fcdd8003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:28.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcde27fc640 1 --2- 172.21.15.88:0/3740154106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdd8003980 0x7fcdd8003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:28.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcde27fc640 1 -- 172.21.15.88:0/3740154106 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd8003530 con 0x7fcdd8004380 2023-10-16T20:48:28.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.193+0000 7fcdeb7fe640 1 -- 172.21.15.88:0/3740154106 reap_dead start 2023-10-16T20:48:28.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.197+0000 7fcde27fc640 1 --2- 172.21.15.88:0/3740154106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdd8004380 0x7fcdd8004850 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4001860 tx=0x7fcdd401c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6889a64ea81bcbe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:28.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.197+0000 7fcdea7fc640 1 -- 172.21.15.88:0/3740154106 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd40ab600 con 0x7fcdd8004380 2023-10-16T20:48:28.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.197+0000 7fcdea7fc640 1 -- 172.21.15.88:0/3740154106 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd40a7050 con 0x7fcdd8004380 2023-10-16T20:48:28.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.197+0000 7fcdea7fc640 1 -- 172.21.15.88:0/3740154106 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd40ab9e0 con 0x7fcdd8004380 2023-10-16T20:48:28.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.197+0000 7fcdf1817640 1 -- 172.21.15.88:0/3740154106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdd8004380 msgr2=0x7fcdd8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:28.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.197+0000 7fcdf1817640 1 --2- 172.21.15.88:0/3740154106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdd8004380 0x7fcdd8004850 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4001860 tx=0x7fcdd401c4c0 comp rx=0 tx=0).stop 2023-10-16T20:48:28.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.197+0000 7fcdf1817640 1 -- 172.21.15.88:0/3740154106 shutdown_connections 2023-10-16T20:48:28.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.197+0000 7fcdf1817640 1 --2- 172.21.15.88:0/3740154106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdd8004380 0x7fcdd8004850 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:28.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.197+0000 7fcdf1817640 1 -- 172.21.15.88:0/3740154106 >> 172.21.15.88:0/3740154106 conn(0x7fcdec0715c0 msgr2=0x7fcdec0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:28.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.197+0000 7fcdf1817640 1 -- 172.21.15.88:0/3740154106 shutdown_connections 2023-10-16T20:48:28.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.197+0000 7fcdf1817640 1 -- 172.21.15.88:0/3740154106 wait complete. 2023-10-16T20:48:28.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.197+0000 7fcdf1817640 1 Processor -- start 2023-10-16T20:48:28.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.197+0000 7fcdf1817640 1 -- start start 2023-10-16T20:48:28.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdf1817640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdd8004380 0x7fcdec15cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:28.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdf1817640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdec152c60 0x7fcdec14b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:28.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdf1817640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdec1590c0 0x7fcdec14b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:28.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdf1817640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec1457b0 con 0x7fcdec1590c0 2023-10-16T20:48:28.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdf1817640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec14c1e0 con 0x7fcdd8004380 2023-10-16T20:48:28.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdf1817640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec14d340 con 0x7fcdec152c60 2023-10-16T20:48:28.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdeb7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdec1590c0 0x7fcdec14b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:28.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdeb7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdec1590c0 0x7fcdec14b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46502/0 (socket says 172.21.15.88:46502) 2023-10-16T20:48:28.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdeb7fe640 1 -- 172.21.15.88:0/549632534 learned_addr learned my addr 172.21.15.88:0/549632534 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:28.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdeaffd640 1 --2- 172.21.15.88:0/549632534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdd8004380 0x7fcdec15cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:28.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcde27fc640 1 --2- 172.21.15.88:0/549632534 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdec152c60 0x7fcdec14b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:28.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdeaffd640 1 -- 172.21.15.88:0/549632534 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdec152c60 msgr2=0x7fcdec14b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:28.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdeaffd640 1 --2- 172.21.15.88:0/549632534 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdec152c60 0x7fcdec14b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:28.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdeaffd640 1 -- 172.21.15.88:0/549632534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdec1590c0 msgr2=0x7fcdec14b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:28.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdeaffd640 1 --2- 172.21.15.88:0/549632534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdec1590c0 0x7fcdec14b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:28.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdeaffd640 1 -- 172.21.15.88:0/549632534 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec14ac90 con 0x7fcdd8004380 2023-10-16T20:48:28.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdeb7fe640 1 --2- 172.21.15.88:0/549632534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdec1590c0 0x7fcdec14b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:48:28.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdeaffd640 1 --2- 172.21.15.88:0/549632534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdd8004380 0x7fcdec15cc50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcdec068520 tx=0x7fcddc00f580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:28.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcde8ff9640 1 -- 172.21.15.88:0/549632534 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc00e040 con 0x7fcdd8004380 2023-10-16T20:48:28.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcde8ff9640 1 -- 172.21.15.88:0/549632534 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcddc00f040 con 0x7fcdd8004380 2023-10-16T20:48:28.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcde8ff9640 1 -- 172.21.15.88:0/549632534 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc015630 con 0x7fcdd8004380 2023-10-16T20:48:28.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdf1817640 1 -- 172.21.15.88:0/549632534 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdec14d590 con 0x7fcdd8004380 2023-10-16T20:48:28.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdf1817640 1 -- 172.21.15.88:0/549632534 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdec14a640 con 0x7fcdd8004380 2023-10-16T20:48:28.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.201+0000 7fcdf1817640 1 -- 172.21.15.88:0/549632534 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdec074700 con 0x7fcdd8004380 2023-10-16T20:48:28.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.205+0000 7fcde8ff9640 1 -- 172.21.15.88:0/549632534 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcddc006d50 con 0x7fcdd8004380 2023-10-16T20:48:28.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.205+0000 7fcde8ff9640 1 --2- 172.21.15.88:0/549632534 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcdcc0664a0 0x7fcdcc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:28.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.205+0000 7fcde8ff9640 1 -- 172.21.15.88:0/549632534 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7fcddc085140 con 0x7fcdd8004380 2023-10-16T20:48:28.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.205+0000 7fcde27fc640 1 --2- 172.21.15.88:0/549632534 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcdcc0664a0 0x7fcdcc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:28.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.205+0000 7fcde27fc640 1 --2- 172.21.15.88:0/549632534 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcdcc0664a0 0x7fcdcc068960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40ad000 tx=0x7fcdd40a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:28.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.209+0000 7fcde8ff9640 1 -- 172.21.15.88:0/549632534 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcddc051fc0 con 0x7fcdd8004380 2023-10-16T20:48:28.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.397+0000 7fcdf1817640 1 -- 172.21.15.88:0/549632534 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdec0cbe70 con 0x7fcdd8004380 2023-10-16T20:48:28.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.397+0000 7fcde8ff9640 1 -- 172.21.15.88:0/549632534 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+6586 (secure 0 0 0) 0x7fcddc022070 con 0x7fcdd8004380 2023-10-16T20:48:28.401 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:48:28.401 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5709,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:13.446293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:48:28.402 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 64 2023-10-16T20:48:28.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.397+0000 7fcdf1817640 1 -- 172.21.15.88:0/549632534 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcdcc0664a0 msgr2=0x7fcdcc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:28.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.397+0000 7fcdf1817640 1 --2- 172.21.15.88:0/549632534 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcdcc0664a0 0x7fcdcc068960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40ad000 tx=0x7fcdd40a6000 comp rx=0 tx=0).stop 2023-10-16T20:48:28.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.397+0000 7fcdf1817640 1 -- 172.21.15.88:0/549632534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdd8004380 msgr2=0x7fcdec15cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:28.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.397+0000 7fcdf1817640 1 --2- 172.21.15.88:0/549632534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdd8004380 0x7fcdec15cc50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcdec068520 tx=0x7fcddc00f580 comp rx=0 tx=0).stop 2023-10-16T20:48:28.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.397+0000 7fcdf1817640 1 -- 172.21.15.88:0/549632534 shutdown_connections 2023-10-16T20:48:28.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.397+0000 7fcdf1817640 1 --2- 172.21.15.88:0/549632534 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcdcc0664a0 0x7fcdcc068960 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:28.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.397+0000 7fcdf1817640 1 --2- 172.21.15.88:0/549632534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdec1590c0 0x7fcdec14b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:28.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.397+0000 7fcdf1817640 1 --2- 172.21.15.88:0/549632534 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcdec152c60 0x7fcdec14b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:28.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.397+0000 7fcdf1817640 1 --2- 172.21.15.88:0/549632534 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdd8004380 0x7fcdec15cc50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:28.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.397+0000 7fcdf1817640 1 -- 172.21.15.88:0/549632534 >> 172.21.15.88:0/549632534 conn(0x7fcdec0715c0 msgr2=0x7fcdec06c300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:28.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.397+0000 7fcdf1817640 1 -- 172.21.15.88:0/549632534 shutdown_connections 2023-10-16T20:48:28.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:28.397+0000 7fcdf1817640 1 -- 172.21.15.88:0/549632534 wait complete. 2023-10-16T20:48:28.422 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-16T20:48:28.870 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:48:28.874 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:48:30.423 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:48:30.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.561+0000 7fc061d3e640 1 Processor -- start 2023-10-16T20:48:30.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.561+0000 7fc061d3e640 1 -- start start 2023-10-16T20:48:30.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc061d3e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc05c153770 0x7fc05c159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:30.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc061d3e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc05c0c9bb0 0x7fc05c0c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:30.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc061d3e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc05c0c41f0 0x7fc05c0cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:30.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc061d3e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc05c0cc250 con 0x7fc05c153770 2023-10-16T20:48:30.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc061d3e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc05c0cc670 con 0x7fc05c0c41f0 2023-10-16T20:48:30.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc061d3e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc05c15a410 con 0x7fc05c0c9bb0 2023-10-16T20:48:30.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc05b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc05c153770 0x7fc05c159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:30.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc05b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc05c153770 0x7fc05c159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38618/0 (socket says 172.21.15.88:38618) 2023-10-16T20:48:30.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc05b7fe640 1 -- 172.21.15.88:0/1463250857 learned_addr learned my addr 172.21.15.88:0/1463250857 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:30.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc05bfff640 1 --2- 172.21.15.88:0/1463250857 >> v2:172.21.15.191:3300/0 conn(0x7fc05c0c41f0 0x7fc05c0cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:30.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc05b7fe640 1 -- 172.21.15.88:0/1463250857 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc05c0c9bb0 msgr2=0x7fc05c0c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:30.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc05affd640 1 --2- 172.21.15.88:0/1463250857 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc05c0c9bb0 0x7fc05c0c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:30.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc05b7fe640 1 --2- 172.21.15.88:0/1463250857 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc05c0c9bb0 0x7fc05c0c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:30.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc05b7fe640 1 -- 172.21.15.88:0/1463250857 >> v2:172.21.15.191:3300/0 conn(0x7fc05c0c41f0 msgr2=0x7fc05c0cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:30.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc05b7fe640 1 --2- 172.21.15.88:0/1463250857 >> v2:172.21.15.191:3300/0 conn(0x7fc05c0c41f0 0x7fc05c0cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:30.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc05b7fe640 1 -- 172.21.15.88:0/1463250857 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05c153b50 con 0x7fc05c153770 2023-10-16T20:48:30.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc05bfff640 1 --2- 172.21.15.88:0/1463250857 >> v2:172.21.15.191:3300/0 conn(0x7fc05c0c41f0 0x7fc05c0cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:48:30.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.565+0000 7fc05affd640 1 --2- 172.21.15.88:0/1463250857 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc05c0c9bb0 0x7fc05c0c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:48:30.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc05b7fe640 1 --2- 172.21.15.88:0/1463250857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc05c153770 0x7fc05c159ed0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fc05c0b2dc0 tx=0x7fc04c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e7988f8204d74f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:30.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc05a7fc640 1 -- 172.21.15.88:0/1463250857 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04c0ac630 con 0x7fc05c153770 2023-10-16T20:48:30.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc05a7fc640 1 -- 172.21.15.88:0/1463250857 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04c0ac790 con 0x7fc05c153770 2023-10-16T20:48:30.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc05a7fc640 1 -- 172.21.15.88:0/1463250857 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04c0acb20 con 0x7fc05c153770 2023-10-16T20:48:30.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc061d3e640 1 -- 172.21.15.88:0/1463250857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc05c153770 msgr2=0x7fc05c159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:30.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc061d3e640 1 --2- 172.21.15.88:0/1463250857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc05c153770 0x7fc05c159ed0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fc05c0b2dc0 tx=0x7fc04c01c500 comp rx=0 tx=0).stop 2023-10-16T20:48:30.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc061d3e640 1 -- 172.21.15.88:0/1463250857 shutdown_connections 2023-10-16T20:48:30.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc061d3e640 1 --2- 172.21.15.88:0/1463250857 >> v2:172.21.15.191:3300/0 conn(0x7fc05c0c41f0 0x7fc05c0cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:30.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc061d3e640 1 --2- 172.21.15.88:0/1463250857 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc05c0c9bb0 0x7fc05c0c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:30.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc061d3e640 1 --2- 172.21.15.88:0/1463250857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc05c153770 0x7fc05c159ed0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:30.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc061d3e640 1 -- 172.21.15.88:0/1463250857 >> 172.21.15.88:0/1463250857 conn(0x7fc05c0715c0 msgr2=0x7fc05c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:30.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc061d3e640 1 -- 172.21.15.88:0/1463250857 shutdown_connections 2023-10-16T20:48:30.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc061d3e640 1 -- 172.21.15.88:0/1463250857 wait complete. 2023-10-16T20:48:30.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc061d3e640 1 Processor -- start 2023-10-16T20:48:30.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc061d3e640 1 -- start start 2023-10-16T20:48:30.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc061d3e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc05c0c41f0 0x7fc05c0b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:30.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc061d3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc05c0c9bb0 0x7fc05c0ce120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:30.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc061d3e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc05c153770 0x7fc05c0b3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:30.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc061d3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05c0b3640 con 0x7fc05c0c9bb0 2023-10-16T20:48:30.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc061d3e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05c0ceab0 con 0x7fc05c0c41f0 2023-10-16T20:48:30.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.569+0000 7fc061d3e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc05c0cdb00 con 0x7fc05c153770 2023-10-16T20:48:30.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc05affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc05c0c9bb0 0x7fc05c0ce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:30.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc05affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc05c0c9bb0 0x7fc05c0ce120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38632/0 (socket says 172.21.15.88:38632) 2023-10-16T20:48:30.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc05affd640 1 -- 172.21.15.88:0/1134463322 learned_addr learned my addr 172.21.15.88:0/1134463322 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:30.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc05bfff640 1 --2- 172.21.15.88:0/1134463322 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc05c153770 0x7fc05c0b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:30.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc05b7fe640 1 --2- 172.21.15.88:0/1134463322 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc05c0c41f0 0x7fc05c0b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:30.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc05affd640 1 -- 172.21.15.88:0/1134463322 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc05c153770 msgr2=0x7fc05c0b3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:30.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc05affd640 1 --2- 172.21.15.88:0/1134463322 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc05c153770 0x7fc05c0b3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:30.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc05affd640 1 -- 172.21.15.88:0/1134463322 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc05c0c41f0 msgr2=0x7fc05c0b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:30.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc05affd640 1 --2- 172.21.15.88:0/1134463322 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc05c0c41f0 0x7fc05c0b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:30.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc05affd640 1 -- 172.21.15.88:0/1134463322 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c009480 con 0x7fc05c0c9bb0 2023-10-16T20:48:30.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc05b7fe640 1 --2- 172.21.15.88:0/1134463322 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc05c0c41f0 0x7fc05c0b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:48:30.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc05affd640 1 --2- 172.21.15.88:0/1134463322 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc05c0c9bb0 0x7fc05c0ce120 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc05000c7f0 tx=0x7fc05000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:30.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc043fff640 1 -- 172.21.15.88:0/1134463322 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05000cea0 con 0x7fc05c0c9bb0 2023-10-16T20:48:30.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc043fff640 1 -- 172.21.15.88:0/1134463322 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc050007a30 con 0x7fc05c0c9bb0 2023-10-16T20:48:30.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc043fff640 1 -- 172.21.15.88:0/1134463322 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc050007de0 con 0x7fc05c0c9bb0 2023-10-16T20:48:30.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc061d3e640 1 -- 172.21.15.88:0/1134463322 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc05c0b4db0 con 0x7fc05c0c9bb0 2023-10-16T20:48:30.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc061d3e640 1 -- 172.21.15.88:0/1134463322 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc05c0cd790 con 0x7fc05c0c9bb0 2023-10-16T20:48:30.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc061d3e640 1 -- 172.21.15.88:0/1134463322 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc05c074700 con 0x7fc05c0c9bb0 2023-10-16T20:48:30.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc043fff640 1 -- 172.21.15.88:0/1134463322 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc0500113f0 con 0x7fc05c0c9bb0 2023-10-16T20:48:30.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.573+0000 7fc043fff640 1 --2- 172.21.15.88:0/1134463322 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0480664a0 0x7fc048068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:30.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.577+0000 7fc043fff640 1 -- 172.21.15.88:0/1134463322 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7fc0500845b0 con 0x7fc05c0c9bb0 2023-10-16T20:48:30.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.577+0000 7fc05b7fe640 1 --2- 172.21.15.88:0/1134463322 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0480664a0 0x7fc048068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:30.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.577+0000 7fc05b7fe640 1 --2- 172.21.15.88:0/1134463322 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0480664a0 0x7fc048068960 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0ae000 tx=0x7fc04c0a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:30.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.581+0000 7fc043fff640 1 -- 172.21.15.88:0/1134463322 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc050051440 con 0x7fc05c0c9bb0 2023-10-16T20:48:30.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.765+0000 7fc061d3e640 1 -- 172.21.15.88:0/1134463322 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc05c1570a0 con 0x7fc05c0c9bb0 2023-10-16T20:48:30.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.765+0000 7fc043fff640 1 -- 172.21.15.88:0/1134463322 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+6586 (secure 0 0 0) 0x7fc050054a60 con 0x7fc05c0c9bb0 2023-10-16T20:48:30.770 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:48:30.770 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5709,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:13.446293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:48:30.771 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 64 2023-10-16T20:48:30.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.769+0000 7fc061d3e640 1 -- 172.21.15.88:0/1134463322 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0480664a0 msgr2=0x7fc048068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:30.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.769+0000 7fc061d3e640 1 --2- 172.21.15.88:0/1134463322 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0480664a0 0x7fc048068960 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0ae000 tx=0x7fc04c0a7000 comp rx=0 tx=0).stop 2023-10-16T20:48:30.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.769+0000 7fc061d3e640 1 -- 172.21.15.88:0/1134463322 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc05c0c9bb0 msgr2=0x7fc05c0ce120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:30.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.769+0000 7fc061d3e640 1 --2- 172.21.15.88:0/1134463322 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc05c0c9bb0 0x7fc05c0ce120 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc05000c7f0 tx=0x7fc05000ccb0 comp rx=0 tx=0).stop 2023-10-16T20:48:30.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.769+0000 7fc061d3e640 1 -- 172.21.15.88:0/1134463322 shutdown_connections 2023-10-16T20:48:30.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.769+0000 7fc061d3e640 1 --2- 172.21.15.88:0/1134463322 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0480664a0 0x7fc048068960 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:30.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.769+0000 7fc061d3e640 1 --2- 172.21.15.88:0/1134463322 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc05c153770 0x7fc05c0b3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:30.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.769+0000 7fc061d3e640 1 --2- 172.21.15.88:0/1134463322 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc05c0c9bb0 0x7fc05c0ce120 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:30.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.769+0000 7fc061d3e640 1 --2- 172.21.15.88:0/1134463322 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc05c0c41f0 0x7fc05c0b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:30.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.769+0000 7fc061d3e640 1 -- 172.21.15.88:0/1134463322 >> 172.21.15.88:0/1134463322 conn(0x7fc05c0715c0 msgr2=0x7fc05c152140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:30.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.769+0000 7fc061d3e640 1 -- 172.21.15.88:0/1134463322 shutdown_connections 2023-10-16T20:48:30.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:30.769+0000 7fc061d3e640 1 -- 172.21.15.88:0/1134463322 wait complete. 2023-10-16T20:48:30.793 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-16T20:48:32.794 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:48:32.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.941+0000 7f156953e640 1 Processor -- start 2023-10-16T20:48:32.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.941+0000 7f156953e640 1 -- start start 2023-10-16T20:48:32.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.945+0000 7f156953e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f15641538d0 0x7f156406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:32.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.945+0000 7f156953e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f15640cb110 0x7f156406a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:32.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.945+0000 7f156953e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1564069e90 con 0x7f15640cac50 2023-10-16T20:48:32.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.945+0000 7f156953e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15640ca470 con 0x7f15640cb110 2023-10-16T20:48:32.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.945+0000 7f156953e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15640ca680 con 0x7f15641538d0 2023-10-16T20:48:32.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.945+0000 7f15637fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f15640cb110 0x7f156406a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:32.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.945+0000 7f15637fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f15640cb110 0x7f156406a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38638/0 (socket says 172.21.15.88:38638) 2023-10-16T20:48:32.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.945+0000 7f15637fe640 1 -- 172.21.15.88:0/1888211157 learned_addr learned my addr 172.21.15.88:0/1888211157 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:32.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.945+0000 7f1562ffd640 1 --2- 172.21.15.88:0/1888211157 >> v2:172.21.15.191:3300/0 conn(0x7f15641538d0 0x7f156406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:32.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.945+0000 7f15637fe640 1 -- 172.21.15.88:0/1888211157 >> v2:172.21.15.191:3300/0 conn(0x7f15641538d0 msgr2=0x7f156406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:32.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.945+0000 7f15637fe640 1 --2- 172.21.15.88:0/1888211157 >> v2:172.21.15.191:3300/0 conn(0x7f15641538d0 0x7f156406c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:32.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.945+0000 7f15637fe640 1 -- 172.21.15.88:0/1888211157 >> v1:172.21.15.88:6789/0 conn(0x7f15640cac50 legacy=0x7f15640cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:32.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.945+0000 7f15637fe640 1 -- 172.21.15.88:0/1888211157 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15640ca960 con 0x7f15640cb110 2023-10-16T20:48:32.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.945+0000 7f15627fc640 1 -- 172.21.15.88:0/1888211157 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019102380 0 0) 0x7f1564069e90 con 0x7f15640cac50 2023-10-16T20:48:32.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f15637fe640 1 --2- 172.21.15.88:0/1888211157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15640cb110 0x7f156406a600 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f15640cc310 tx=0x7f155c01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0973062e97a6e8a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:32.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f15627fc640 1 -- 172.21.15.88:0/1888211157 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c0a56e0 con 0x7f15640cb110 2023-10-16T20:48:32.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f15627fc640 1 -- 172.21.15.88:0/1888211157 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155c0a5840 con 0x7f15640cb110 2023-10-16T20:48:32.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f15627fc640 1 -- 172.21.15.88:0/1888211157 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155c0a5bd0 con 0x7f15640cb110 2023-10-16T20:48:32.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f156953e640 1 -- 172.21.15.88:0/1888211157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15640cb110 msgr2=0x7f156406a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:32.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f156953e640 1 --2- 172.21.15.88:0/1888211157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15640cb110 0x7f156406a600 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f15640cc310 tx=0x7f155c01c910 comp rx=0 tx=0).stop 2023-10-16T20:48:32.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f156953e640 1 -- 172.21.15.88:0/1888211157 shutdown_connections 2023-10-16T20:48:32.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f156953e640 1 --2- 172.21.15.88:0/1888211157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15640cb110 0x7f156406a600 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:32.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f156953e640 1 --2- 172.21.15.88:0/1888211157 >> v2:172.21.15.191:3300/0 conn(0x7f15641538d0 0x7f156406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:32.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f156953e640 1 -- 172.21.15.88:0/1888211157 >> 172.21.15.88:0/1888211157 conn(0x7f1564071cb0 msgr2=0x7f156406bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:32.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f156953e640 1 -- 172.21.15.88:0/1888211157 shutdown_connections 2023-10-16T20:48:32.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f156953e640 1 -- 172.21.15.88:0/1888211157 wait complete. 2023-10-16T20:48:32.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f156953e640 1 Processor -- start 2023-10-16T20:48:32.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f156953e640 1 -- start start 2023-10-16T20:48:32.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f156953e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15640cac50 0x7f15641f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:32.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f156953e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15640cb110 0x7f15641409f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:32.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f156953e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15641538d0 0x7f15640ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f156953e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156414efd0 con 0x7f15640cb110 2023-10-16T20:48:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f156953e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15640b41c0 con 0x7f15641538d0 2023-10-16T20:48:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.949+0000 7f156953e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15641f2720 con 0x7f15640cac50 2023-10-16T20:48:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f155a7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15640cb110 0x7f15641409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f155a7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15640cb110 0x7f15641409f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38640/0 (socket says 172.21.15.88:38640) 2023-10-16T20:48:32.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f155a7fc640 1 -- 172.21.15.88:0/2461082459 learned_addr learned my addr 172.21.15.88:0/2461082459 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:32.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f15637fe640 1 --2- 172.21.15.88:0/2461082459 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15641538d0 0x7f15640ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:32.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f1562ffd640 1 --2- 172.21.15.88:0/2461082459 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15640cac50 0x7f15641f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:32.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f155a7fc640 1 -- 172.21.15.88:0/2461082459 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15640cac50 msgr2=0x7f15641f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:32.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f155a7fc640 1 --2- 172.21.15.88:0/2461082459 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15640cac50 0x7f15641f21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:32.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f155a7fc640 1 -- 172.21.15.88:0/2461082459 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15641538d0 msgr2=0x7f15640ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:32.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f155a7fc640 1 --2- 172.21.15.88:0/2461082459 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15641538d0 0x7f15640ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:32.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f155a7fc640 1 -- 172.21.15.88:0/2461082459 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c009b30 con 0x7f15640cb110 2023-10-16T20:48:32.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f155a7fc640 1 --2- 172.21.15.88:0/2461082459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15640cb110 0x7f15641409f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f154c00c480 tx=0x7f154c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:32.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f155bfff640 1 -- 172.21.15.88:0/2461082459 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c005480 con 0x7f15640cb110 2023-10-16T20:48:32.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f155bfff640 1 -- 172.21.15.88:0/2461082459 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154c003600 con 0x7f15640cb110 2023-10-16T20:48:32.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f156953e640 1 -- 172.21.15.88:0/2461082459 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15640b3a50 con 0x7f15640cb110 2023-10-16T20:48:32.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f155bfff640 1 -- 172.21.15.88:0/2461082459 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c0039b0 con 0x7f15640cb110 2023-10-16T20:48:32.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f156953e640 1 -- 172.21.15.88:0/2461082459 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15640b3d30 con 0x7f15640cb110 2023-10-16T20:48:32.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f156953e640 1 -- 172.21.15.88:0/2461082459 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1564074700 con 0x7f15640cb110 2023-10-16T20:48:32.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.953+0000 7f155bfff640 1 -- 172.21.15.88:0/2461082459 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f154c003b10 con 0x7f15640cb110 2023-10-16T20:48:32.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.957+0000 7f155bfff640 1 --2- 172.21.15.88:0/2461082459 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f15440664a0 0x7f1544068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:32.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.957+0000 7f155bfff640 1 -- 172.21.15.88:0/2461082459 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f154c083850 con 0x7f15640cb110 2023-10-16T20:48:32.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.957+0000 7f1562ffd640 1 --2- 172.21.15.88:0/2461082459 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f15440664a0 0x7f1544068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:32.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.957+0000 7f1562ffd640 1 --2- 172.21.15.88:0/2461082459 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f15440664a0 0x7f1544068960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1554009b60 tx=0x7f1554009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:32.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:32.961+0000 7f155bfff640 1 -- 172.21.15.88:0/2461082459 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f154c050820 con 0x7f15640cb110 2023-10-16T20:48:33.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:33.141+0000 7f156953e640 1 -- 172.21.15.88:0/2461082459 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15640c9c40 con 0x7f15640cb110 2023-10-16T20:48:33.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:33.141+0000 7f155bfff640 1 -- 172.21.15.88:0/2461082459 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+6586 (secure 0 0 0) 0x7f154c017090 con 0x7f15640cb110 2023-10-16T20:48:33.146 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:48:33.146 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5709,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:13.446293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:48:33.146 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 64 2023-10-16T20:48:33.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:33.141+0000 7f156953e640 1 -- 172.21.15.88:0/2461082459 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f15440664a0 msgr2=0x7f1544068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:33.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:33.141+0000 7f156953e640 1 --2- 172.21.15.88:0/2461082459 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f15440664a0 0x7f1544068960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1554009b60 tx=0x7f1554009190 comp rx=0 tx=0).stop 2023-10-16T20:48:33.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:33.141+0000 7f156953e640 1 -- 172.21.15.88:0/2461082459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15640cb110 msgr2=0x7f15641409f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:33.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:33.141+0000 7f156953e640 1 --2- 172.21.15.88:0/2461082459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15640cb110 0x7f15641409f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f154c00c480 tx=0x7f154c00c940 comp rx=0 tx=0).stop 2023-10-16T20:48:33.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:33.145+0000 7f156953e640 1 -- 172.21.15.88:0/2461082459 shutdown_connections 2023-10-16T20:48:33.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:33.145+0000 7f156953e640 1 --2- 172.21.15.88:0/2461082459 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f15440664a0 0x7f1544068960 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:33.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:33.145+0000 7f156953e640 1 --2- 172.21.15.88:0/2461082459 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f15641538d0 0x7f15640ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:33.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:33.145+0000 7f156953e640 1 --2- 172.21.15.88:0/2461082459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15640cb110 0x7f15641409f0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:33.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:33.145+0000 7f156953e640 1 --2- 172.21.15.88:0/2461082459 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f15640cac50 0x7f15641f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:33.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:33.145+0000 7f156953e640 1 -- 172.21.15.88:0/2461082459 >> 172.21.15.88:0/2461082459 conn(0x7f1564071cb0 msgr2=0x7f15640c45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:33.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:33.145+0000 7f156953e640 1 -- 172.21.15.88:0/2461082459 shutdown_connections 2023-10-16T20:48:33.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:33.145+0000 7f156953e640 1 -- 172.21.15.88:0/2461082459 wait complete. 2023-10-16T20:48:33.166 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-16T20:48:35.167 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:48:35.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.301+0000 7f4986baa640 1 Processor -- start 2023-10-16T20:48:35.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.301+0000 7f4986baa640 1 -- start start 2023-10-16T20:48:35.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.301+0000 7f4986baa640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4980153670 0x7f498006a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:35.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.301+0000 7f4986baa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f49800c58d0 0x7f498006d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:35.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.301+0000 7f4986baa640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49800c5e80 con 0x7f498006d6b0 2023-10-16T20:48:35.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.301+0000 7f4986baa640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49800c6860 con 0x7f49800c58d0 2023-10-16T20:48:35.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.301+0000 7f4986baa640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498015cb30 con 0x7f4980153670 2023-10-16T20:48:35.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.301+0000 7f4977fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f49800c58d0 0x7f498006d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:35.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.301+0000 7f4977fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f49800c58d0 0x7f498006d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38654/0 (socket says 172.21.15.88:38654) 2023-10-16T20:48:35.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.301+0000 7f4977fff640 1 -- 172.21.15.88:0/867012437 learned_addr learned my addr 172.21.15.88:0/867012437 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:35.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.301+0000 7f4984946640 1 --2- 172.21.15.88:0/867012437 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4980153670 0x7f498006a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:35.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.301+0000 7f4977fff640 1 -- 172.21.15.88:0/867012437 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4980153670 msgr2=0x7f498006a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:35.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.301+0000 7f4977fff640 1 --2- 172.21.15.88:0/867012437 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4980153670 0x7f498006a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:35.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.301+0000 7f4977fff640 1 -- 172.21.15.88:0/867012437 >> v1:172.21.15.191:6789/0 conn(0x7f498006d6b0 legacy=0x7f498006f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:35.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.301+0000 7f4977fff640 1 -- 172.21.15.88:0/867012437 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498015ce10 con 0x7f49800c58d0 2023-10-16T20:48:35.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.305+0000 7f4977fff640 1 --2- 172.21.15.88:0/867012437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49800c58d0 0x7f498006d170 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4970009870 tx=0x7f497001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a819a5fdb760c2e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:35.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.305+0000 7f49777fe640 1 -- 172.21.15.88:0/867012437 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49700ab600 con 0x7f49800c58d0 2023-10-16T20:48:35.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.305+0000 7f49777fe640 1 -- 172.21.15.88:0/867012437 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49700a7050 con 0x7f49800c58d0 2023-10-16T20:48:35.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.305+0000 7f49777fe640 1 -- 172.21.15.88:0/867012437 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49700ab980 con 0x7f49800c58d0 2023-10-16T20:48:35.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.305+0000 7f4986baa640 1 -- 172.21.15.88:0/867012437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49800c58d0 msgr2=0x7f498006d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:35.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.305+0000 7f4986baa640 1 --2- 172.21.15.88:0/867012437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49800c58d0 0x7f498006d170 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4970009870 tx=0x7f497001c540 comp rx=0 tx=0).stop 2023-10-16T20:48:35.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.305+0000 7f4986baa640 1 -- 172.21.15.88:0/867012437 shutdown_connections 2023-10-16T20:48:35.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.305+0000 7f4986baa640 1 --2- 172.21.15.88:0/867012437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49800c58d0 0x7f498006d170 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:35.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.305+0000 7f4986baa640 1 --2- 172.21.15.88:0/867012437 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4980153670 0x7f498006a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:35.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.305+0000 7f4986baa640 1 -- 172.21.15.88:0/867012437 >> 172.21.15.88:0/867012437 conn(0x7f49800cad50 msgr2=0x7f49800cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:35.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.305+0000 7f4986baa640 1 -- 172.21.15.88:0/867012437 shutdown_connections 2023-10-16T20:48:35.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.305+0000 7f4986baa640 1 -- 172.21.15.88:0/867012437 wait complete. 2023-10-16T20:48:35.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4986baa640 1 Processor -- start 2023-10-16T20:48:35.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4986baa640 1 -- start start 2023-10-16T20:48:35.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4986baa640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f498006d6b0 0x7f4980147720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:35.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4986baa640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49800c58d0 0x7f49800c2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:35.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4986baa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4980153670 0x7f498014c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:35.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4986baa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49800c2660 con 0x7f4980153670 2023-10-16T20:48:35.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4986baa640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49800c3980 con 0x7f49800c58d0 2023-10-16T20:48:35.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4986baa640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49800c4010 con 0x7f498006d6b0 2023-10-16T20:48:35.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4985147640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4980153670 0x7f498014c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:35.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4985147640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4980153670 0x7f498014c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38666/0 (socket says 172.21.15.88:38666) 2023-10-16T20:48:35.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4985147640 1 -- 172.21.15.88:0/1158932465 learned_addr learned my addr 172.21.15.88:0/1158932465 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:35.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4984946640 1 --2- 172.21.15.88:0/1158932465 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f498006d6b0 0x7f4980147720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:35.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4977fff640 1 --2- 172.21.15.88:0/1158932465 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49800c58d0 0x7f49800c2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:35.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4985147640 1 -- 172.21.15.88:0/1158932465 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f498006d6b0 msgr2=0x7f4980147720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:35.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4985147640 1 --2- 172.21.15.88:0/1158932465 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f498006d6b0 0x7f4980147720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:35.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4985147640 1 -- 172.21.15.88:0/1158932465 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49800c58d0 msgr2=0x7f49800c2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:35.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4985147640 1 --2- 172.21.15.88:0/1158932465 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49800c58d0 0x7f49800c2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:35.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4985147640 1 -- 172.21.15.88:0/1158932465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49700094d0 con 0x7f4980153670 2023-10-16T20:48:35.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4985147640 1 --2- 172.21.15.88:0/1158932465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4980153670 0x7f498014c300 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f497800c770 tx=0x7f497800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:35.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f49757fa640 1 -- 172.21.15.88:0/1158932465 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4978003b80 con 0x7f4980153670 2023-10-16T20:48:35.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f49757fa640 1 -- 172.21.15.88:0/1158932465 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4978003ce0 con 0x7f4980153670 2023-10-16T20:48:35.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f49757fa640 1 -- 172.21.15.88:0/1158932465 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497800aa90 con 0x7f4980153670 2023-10-16T20:48:35.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4986baa640 1 -- 172.21.15.88:0/1158932465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4980147c60 con 0x7f4980153670 2023-10-16T20:48:35.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.309+0000 7f4986baa640 1 -- 172.21.15.88:0/1158932465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498014ba80 con 0x7f4980153670 2023-10-16T20:48:35.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.313+0000 7f4986baa640 1 -- 172.21.15.88:0/1158932465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4980074700 con 0x7f4980153670 2023-10-16T20:48:35.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.313+0000 7f49757fa640 1 -- 172.21.15.88:0/1158932465 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f497800abf0 con 0x7f4980153670 2023-10-16T20:48:35.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.313+0000 7f49757fa640 1 --2- 172.21.15.88:0/1158932465 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f496c0664d0 0x7f496c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:35.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.313+0000 7f49757fa640 1 -- 172.21.15.88:0/1158932465 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f4978083f80 con 0x7f4980153670 2023-10-16T20:48:35.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.313+0000 7f4984946640 1 --2- 172.21.15.88:0/1158932465 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f496c0664d0 0x7f496c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:35.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.313+0000 7f4984946640 1 --2- 172.21.15.88:0/1158932465 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f496c0664d0 0x7f496c068990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f497c0079c0 tx=0x7f497c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:35.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.317+0000 7f49757fa640 1 -- 172.21.15.88:0/1158932465 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4978050ec0 con 0x7f4980153670 2023-10-16T20:48:35.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.505+0000 7f4986baa640 1 -- 172.21.15.88:0/1158932465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49800c6410 con 0x7f4980153670 2023-10-16T20:48:35.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.505+0000 7f49757fa640 1 -- 172.21.15.88:0/1158932465 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+6586 (secure 0 0 0) 0x7f49780544e0 con 0x7f4980153670 2023-10-16T20:48:35.509 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:48:35.509 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5709,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:13.446293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:48:35.509 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 64 2023-10-16T20:48:35.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.505+0000 7f4986baa640 1 -- 172.21.15.88:0/1158932465 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f496c0664d0 msgr2=0x7f496c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:35.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.505+0000 7f4986baa640 1 --2- 172.21.15.88:0/1158932465 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f496c0664d0 0x7f496c068990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f497c0079c0 tx=0x7f497c008040 comp rx=0 tx=0).stop 2023-10-16T20:48:35.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.505+0000 7f4986baa640 1 -- 172.21.15.88:0/1158932465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4980153670 msgr2=0x7f498014c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:35.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.505+0000 7f4986baa640 1 --2- 172.21.15.88:0/1158932465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4980153670 0x7f498014c300 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f497800c770 tx=0x7f497800cc30 comp rx=0 tx=0).stop 2023-10-16T20:48:35.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.505+0000 7f4986baa640 1 -- 172.21.15.88:0/1158932465 shutdown_connections 2023-10-16T20:48:35.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.505+0000 7f4986baa640 1 --2- 172.21.15.88:0/1158932465 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f496c0664d0 0x7f496c068990 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:35.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.505+0000 7f4986baa640 1 --2- 172.21.15.88:0/1158932465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4980153670 0x7f498014c300 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:35.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.505+0000 7f4986baa640 1 --2- 172.21.15.88:0/1158932465 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f49800c58d0 0x7f49800c2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:35.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.505+0000 7f4986baa640 1 --2- 172.21.15.88:0/1158932465 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f498006d6b0 0x7f4980147720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:35.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.505+0000 7f4986baa640 1 -- 172.21.15.88:0/1158932465 >> 172.21.15.88:0/1158932465 conn(0x7f49800cad50 msgr2=0x7f498015bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:35.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.505+0000 7f4986baa640 1 -- 172.21.15.88:0/1158932465 shutdown_connections 2023-10-16T20:48:35.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:35.505+0000 7f4986baa640 1 -- 172.21.15.88:0/1158932465 wait complete. 2023-10-16T20:48:35.529 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-16T20:48:37.531 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:48:37.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.689+0000 7f51b6609640 1 Processor -- start 2023-10-16T20:48:37.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.689+0000 7f51b6609640 1 -- start start 2023-10-16T20:48:37.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.689+0000 7f51b6609640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f51b0070760 0x7f51b00c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:37.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.689+0000 7f51b6609640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f51b0070b40 0x7f51b0159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:37.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.689+0000 7f51b6609640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51b00c5bd0 con 0x7f51b0070f90 2023-10-16T20:48:37.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.689+0000 7f51b6609640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51b006ba10 con 0x7f51b0070b40 2023-10-16T20:48:37.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.689+0000 7f51b6609640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51b0159f10 con 0x7f51b0070760 2023-10-16T20:48:37.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.689+0000 7f51b4e06640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f51b0070b40 0x7f51b0159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:37.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.689+0000 7f51b4e06640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f51b0070b40 0x7f51b0159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38678/0 (socket says 172.21.15.88:38678) 2023-10-16T20:48:37.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.689+0000 7f51b4e06640 1 -- 172.21.15.88:0/903485916 learned_addr learned my addr 172.21.15.88:0/903485916 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:37.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.689+0000 7f51b5607640 1 --2- 172.21.15.88:0/903485916 >> v2:172.21.15.191:3300/0 conn(0x7f51b0070760 0x7f51b00c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:37.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.689+0000 7f51b4e06640 1 -- 172.21.15.88:0/903485916 >> v2:172.21.15.191:3300/0 conn(0x7f51b0070760 msgr2=0x7f51b00c62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:37.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.689+0000 7f51b4e06640 1 --2- 172.21.15.88:0/903485916 >> v2:172.21.15.191:3300/0 conn(0x7f51b0070760 0x7f51b00c62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:37.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.689+0000 7f51b4e06640 1 -- 172.21.15.88:0/903485916 >> v1:172.21.15.191:6789/0 conn(0x7f51b0070f90 legacy=0x7f51b0152e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:37.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.689+0000 7f51b4e06640 1 -- 172.21.15.88:0/903485916 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b015a1f0 con 0x7f51b0070b40 2023-10-16T20:48:37.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.693+0000 7f51b4e06640 1 --2- 172.21.15.88:0/903485916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b0070b40 0x7f51b0159330 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f51b0070240 tx=0x7f51a001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a03b2ae69814e47 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:37.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.693+0000 7f51a7fff640 1 -- 172.21.15.88:0/903485916 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a00a89f0 con 0x7f51b0070b40 2023-10-16T20:48:37.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.693+0000 7f51a7fff640 1 -- 172.21.15.88:0/903485916 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51a00a8b50 con 0x7f51b0070b40 2023-10-16T20:48:37.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.693+0000 7f51a7fff640 1 -- 172.21.15.88:0/903485916 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a00b0440 con 0x7f51b0070b40 2023-10-16T20:48:37.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.693+0000 7f51b6609640 1 -- 172.21.15.88:0/903485916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b0070b40 msgr2=0x7f51b0159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:37.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.693+0000 7f51b6609640 1 --2- 172.21.15.88:0/903485916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b0070b40 0x7f51b0159330 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f51b0070240 tx=0x7f51a001ecf0 comp rx=0 tx=0).stop 2023-10-16T20:48:37.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.693+0000 7f51b6609640 1 -- 172.21.15.88:0/903485916 shutdown_connections 2023-10-16T20:48:37.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.693+0000 7f51b6609640 1 --2- 172.21.15.88:0/903485916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b0070b40 0x7f51b0159330 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:37.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.693+0000 7f51b6609640 1 --2- 172.21.15.88:0/903485916 >> v2:172.21.15.191:3300/0 conn(0x7f51b0070760 0x7f51b00c62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:37.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.693+0000 7f51b6609640 1 -- 172.21.15.88:0/903485916 >> 172.21.15.88:0/903485916 conn(0x7f51b00c83d0 msgr2=0x7f51b00c87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:37.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b6609640 1 -- 172.21.15.88:0/903485916 shutdown_connections 2023-10-16T20:48:37.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b6609640 1 -- 172.21.15.88:0/903485916 wait complete. 2023-10-16T20:48:37.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b6609640 1 Processor -- start 2023-10-16T20:48:37.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b6609640 1 -- start start 2023-10-16T20:48:37.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b6609640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51b0070760 0x7f51b015d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:37.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b6609640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51b0070b40 0x7f51b0154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:37.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b6609640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b0070f90 0x7f51b0154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:37.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b6609640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b0159990 con 0x7f51b0070f90 2023-10-16T20:48:37.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b6609640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b01565f0 con 0x7f51b0070b40 2023-10-16T20:48:37.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b6609640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51b0156860 con 0x7f51b0070760 2023-10-16T20:48:37.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b5e08640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b0070f90 0x7f51b0154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:37.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b5e08640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b0070f90 0x7f51b0154bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38690/0 (socket says 172.21.15.88:38690) 2023-10-16T20:48:37.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b5e08640 1 -- 172.21.15.88:0/1531624082 learned_addr learned my addr 172.21.15.88:0/1531624082 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:37.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b5607640 1 --2- 172.21.15.88:0/1531624082 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51b0070760 0x7f51b015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:37.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b4e06640 1 --2- 172.21.15.88:0/1531624082 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51b0070b40 0x7f51b0154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:37.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b5e08640 1 -- 172.21.15.88:0/1531624082 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51b0070760 msgr2=0x7f51b015d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:37.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b5e08640 1 --2- 172.21.15.88:0/1531624082 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51b0070760 0x7f51b015d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:37.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b5e08640 1 -- 172.21.15.88:0/1531624082 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51b0070b40 msgr2=0x7f51b0154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:37.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b5e08640 1 --2- 172.21.15.88:0/1531624082 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51b0070b40 0x7f51b0154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.697+0000 7f51b5e08640 1 -- 172.21.15.88:0/1531624082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a00094d0 con 0x7f51b0070f90 2023-10-16T20:48:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.701+0000 7f51b5e08640 1 --2- 172.21.15.88:0/1531624082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b0070f90 0x7f51b0154bb0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f51a800c730 tx=0x7f51a800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.701+0000 7f51a5ffb640 1 -- 172.21.15.88:0/1531624082 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a8003b80 con 0x7f51b0070f90 2023-10-16T20:48:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.701+0000 7f51a5ffb640 1 -- 172.21.15.88:0/1531624082 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51a8003ce0 con 0x7f51b0070f90 2023-10-16T20:48:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.701+0000 7f51a5ffb640 1 -- 172.21.15.88:0/1531624082 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a800aa30 con 0x7f51b0070f90 2023-10-16T20:48:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.701+0000 7f51b6609640 1 -- 172.21.15.88:0/1531624082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51b0156b40 con 0x7f51b0070f90 2023-10-16T20:48:37.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.701+0000 7f51b6609640 1 -- 172.21.15.88:0/1531624082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51b01596b0 con 0x7f51b0070f90 2023-10-16T20:48:37.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.701+0000 7f51b6609640 1 -- 172.21.15.88:0/1531624082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51b006b3a0 con 0x7f51b0070f90 2023-10-16T20:48:37.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.701+0000 7f51a5ffb640 1 -- 172.21.15.88:0/1531624082 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f51a800ab90 con 0x7f51b0070f90 2023-10-16T20:48:37.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.701+0000 7f51a5ffb640 1 --2- 172.21.15.88:0/1531624082 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f51940664a0 0x7f5194068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:37.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.705+0000 7f51a5ffb640 1 -- 172.21.15.88:0/1531624082 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f51a8083e70 con 0x7f51b0070f90 2023-10-16T20:48:37.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.705+0000 7f51b5607640 1 --2- 172.21.15.88:0/1531624082 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f51940664a0 0x7f5194068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:37.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.705+0000 7f51b5607640 1 --2- 172.21.15.88:0/1531624082 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f51940664a0 0x7f5194068960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f51ac006aa0 tx=0x7f51ac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:37.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.709+0000 7f51a5ffb640 1 -- 172.21.15.88:0/1531624082 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f51b006b3a0 con 0x7f51b0070f90 2023-10-16T20:48:37.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.925+0000 7f51b6609640 1 -- 172.21.15.88:0/1531624082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51b00c5bd0 con 0x7f51b0070f90 2023-10-16T20:48:37.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.925+0000 7f51a5ffb640 1 -- 172.21.15.88:0/1531624082 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+6586 (secure 0 0 0) 0x7f51b00c5bd0 con 0x7f51b0070f90 2023-10-16T20:48:37.931 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:48:37.932 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5709,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:13.446293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:48:37.932 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 64 2023-10-16T20:48:37.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.929+0000 7f51b6609640 1 -- 172.21.15.88:0/1531624082 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f51940664a0 msgr2=0x7f5194068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:37.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.929+0000 7f51b6609640 1 --2- 172.21.15.88:0/1531624082 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f51940664a0 0x7f5194068960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f51ac006aa0 tx=0x7f51ac008040 comp rx=0 tx=0).stop 2023-10-16T20:48:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.929+0000 7f51b6609640 1 -- 172.21.15.88:0/1531624082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b0070f90 msgr2=0x7f51b0154bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.929+0000 7f51b6609640 1 --2- 172.21.15.88:0/1531624082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b0070f90 0x7f51b0154bb0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f51a800c730 tx=0x7f51a800cbf0 comp rx=0 tx=0).stop 2023-10-16T20:48:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.929+0000 7f51b6609640 1 -- 172.21.15.88:0/1531624082 shutdown_connections 2023-10-16T20:48:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.929+0000 7f51b6609640 1 --2- 172.21.15.88:0/1531624082 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f51940664a0 0x7f5194068960 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.929+0000 7f51b6609640 1 --2- 172.21.15.88:0/1531624082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b0070f90 0x7f51b0154bb0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.929+0000 7f51b6609640 1 --2- 172.21.15.88:0/1531624082 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f51b0070b40 0x7f51b0154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.929+0000 7f51b6609640 1 --2- 172.21.15.88:0/1531624082 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f51b0070760 0x7f51b015d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.929+0000 7f51b6609640 1 -- 172.21.15.88:0/1531624082 >> 172.21.15.88:0/1531624082 conn(0x7f51b00c83d0 msgr2=0x7f51b00c61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.929+0000 7f51b6609640 1 -- 172.21.15.88:0/1531624082 shutdown_connections 2023-10-16T20:48:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:37.929+0000 7f51b6609640 1 -- 172.21.15.88:0/1531624082 wait complete. 2023-10-16T20:48:37.953 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-16T20:48:39.954 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:48:40.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.105+0000 7fb29753e640 1 Processor -- start 2023-10-16T20:48:40.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.105+0000 7fb29753e640 1 -- start start 2023-10-16T20:48:40.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.105+0000 7fb29753e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb290153770 0x7fb290159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:40.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.105+0000 7fb29753e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2900c9bb0 0x7fb2900c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:40.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.105+0000 7fb29753e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29015a410 con 0x7fb2900c41f0 2023-10-16T20:48:40.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.105+0000 7fb29753e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2900cc250 con 0x7fb290153770 2023-10-16T20:48:40.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.105+0000 7fb29753e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2900cc670 con 0x7fb2900c9bb0 2023-10-16T20:48:40.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.105+0000 7fb295adb640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb2900c41f0 0x7fb2900cece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:39280/0 (socket says 172.21.15.88:39280) 2023-10-16T20:48:40.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.105+0000 7fb295adb640 1 -- 172.21.15.88:0/2103963527 learned_addr learned my addr 172.21.15.88:0/2103963527 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:48:40.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.105+0000 7fb2952da640 1 --2- 172.21.15.88:0/2103963527 >> v2:172.21.15.191:3300/0 conn(0x7fb290153770 0x7fb290159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:40.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.105+0000 7fb294ad9640 1 --2- 172.21.15.88:0/2103963527 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2900c9bb0 0x7fb2900c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:40.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.105+0000 7fb287fff640 1 -- 172.21.15.88:0/2103963527 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016308735 0 0) 0x7fb29015a410 con 0x7fb2900c41f0 2023-10-16T20:48:40.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.105+0000 7fb287fff640 1 -- 172.21.15.88:0/2103963527 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb27c003530 con 0x7fb2900c41f0 2023-10-16T20:48:40.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.105+0000 7fb2952da640 1 -- 172.21.15.88:0/2103963527 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2900c9bb0 msgr2=0x7fb2900c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:40.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.105+0000 7fb2952da640 1 --2- 172.21.15.88:0/2103963527 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2900c9bb0 0x7fb2900c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:40.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.109+0000 7fb2952da640 1 -- 172.21.15.88:0/2103963527 >> v1:172.21.15.88:6789/0 conn(0x7fb2900c41f0 legacy=0x7fb2900cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:40.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.109+0000 7fb2952da640 1 -- 172.21.15.88:0/2103963527 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb290153b50 con 0x7fb290153770 2023-10-16T20:48:40.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.109+0000 7fb294ad9640 1 --2- 172.21.15.88:0/2103963527 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2900c9bb0 0x7fb2900c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:48:40.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.109+0000 7fb2952da640 1 --2- 172.21.15.88:0/2103963527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb290153770 0x7fb290159ed0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb28c0097a0 tx=0x7fb28c01c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=472b4004b4441e66 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:40.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.109+0000 7fb287fff640 1 -- 172.21.15.88:0/2103963527 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c0ab600 con 0x7fb290153770 2023-10-16T20:48:40.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb287fff640 1 -- 172.21.15.88:0/2103963527 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb28c0a7050 con 0x7fb290153770 2023-10-16T20:48:40.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb287fff640 1 -- 172.21.15.88:0/2103963527 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c0ab980 con 0x7fb290153770 2023-10-16T20:48:40.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb29753e640 1 -- 172.21.15.88:0/2103963527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb290153770 msgr2=0x7fb290159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:40.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb29753e640 1 --2- 172.21.15.88:0/2103963527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb290153770 0x7fb290159ed0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb28c0097a0 tx=0x7fb28c01c460 comp rx=0 tx=0).stop 2023-10-16T20:48:40.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb29753e640 1 -- 172.21.15.88:0/2103963527 shutdown_connections 2023-10-16T20:48:40.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb29753e640 1 --2- 172.21.15.88:0/2103963527 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2900c9bb0 0x7fb2900c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:40.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb29753e640 1 --2- 172.21.15.88:0/2103963527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb290153770 0x7fb290159ed0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:40.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb29753e640 1 -- 172.21.15.88:0/2103963527 >> 172.21.15.88:0/2103963527 conn(0x7fb2900715c0 msgr2=0x7fb2900719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:40.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb29753e640 1 -- 172.21.15.88:0/2103963527 shutdown_connections 2023-10-16T20:48:40.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb29753e640 1 -- 172.21.15.88:0/2103963527 wait complete. 2023-10-16T20:48:40.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb29753e640 1 Processor -- start 2023-10-16T20:48:40.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb29753e640 1 -- start start 2023-10-16T20:48:40.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb29753e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2900c41f0 0x7fb2901f1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:40.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb29753e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2900c9bb0 0x7fb2901496f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:40.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb29753e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb290153770 0x7fb29013e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:40.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb29753e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2901439d0 con 0x7fb2900c41f0 2023-10-16T20:48:40.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb29753e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb290143450 con 0x7fb290153770 2023-10-16T20:48:40.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.113+0000 7fb29753e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2901436f0 con 0x7fb2900c9bb0 2023-10-16T20:48:40.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb2952da640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2900c41f0 0x7fb2901f1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:40.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb2952da640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2900c41f0 0x7fb2901f1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47408/0 (socket says 172.21.15.88:47408) 2023-10-16T20:48:40.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb2952da640 1 -- 172.21.15.88:0/2516424696 learned_addr learned my addr 172.21.15.88:0/2516424696 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:40.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb294ad9640 1 --2- 172.21.15.88:0/2516424696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2900c9bb0 0x7fb2901496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:40.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb295adb640 1 --2- 172.21.15.88:0/2516424696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb290153770 0x7fb29013e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:40.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb2952da640 1 -- 172.21.15.88:0/2516424696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2900c9bb0 msgr2=0x7fb2901496f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:40.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb2952da640 1 --2- 172.21.15.88:0/2516424696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2900c9bb0 0x7fb2901496f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:40.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb2952da640 1 -- 172.21.15.88:0/2516424696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb290153770 msgr2=0x7fb29013e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:40.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb2952da640 1 --2- 172.21.15.88:0/2516424696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb290153770 0x7fb29013e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:40.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb2952da640 1 -- 172.21.15.88:0/2516424696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb280009420 con 0x7fb2900c41f0 2023-10-16T20:48:40.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb2952da640 1 --2- 172.21.15.88:0/2516424696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2900c41f0 0x7fb2901f1570 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb28c0ae040 tx=0x7fb28c01aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:40.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb285ffb640 1 -- 172.21.15.88:0/2516424696 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c0abcf0 con 0x7fb2900c41f0 2023-10-16T20:48:40.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb285ffb640 1 -- 172.21.15.88:0/2516424696 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb28c0abe70 con 0x7fb2900c41f0 2023-10-16T20:48:40.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb29753e640 1 -- 172.21.15.88:0/2516424696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb28c0093b0 con 0x7fb2900c41f0 2023-10-16T20:48:40.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb285ffb640 1 -- 172.21.15.88:0/2516424696 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c005bc0 con 0x7fb2900c41f0 2023-10-16T20:48:40.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb29753e640 1 -- 172.21.15.88:0/2516424696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb290155c80 con 0x7fb2900c41f0 2023-10-16T20:48:40.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb29753e640 1 -- 172.21.15.88:0/2516424696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb290074700 con 0x7fb2900c41f0 2023-10-16T20:48:40.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.117+0000 7fb285ffb640 1 -- 172.21.15.88:0/2516424696 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb28c0b7450 con 0x7fb2900c41f0 2023-10-16T20:48:40.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.121+0000 7fb285ffb640 1 --2- 172.21.15.88:0/2516424696 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb2740664a0 0x7fb274068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:40.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.121+0000 7fb285ffb640 1 -- 172.21.15.88:0/2516424696 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7fb28c11e3b0 con 0x7fb2900c41f0 2023-10-16T20:48:40.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.121+0000 7fb294ad9640 1 --2- 172.21.15.88:0/2516424696 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb2740664a0 0x7fb274068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:40.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.121+0000 7fb294ad9640 1 --2- 172.21.15.88:0/2516424696 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb2740664a0 0x7fb274068960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb290152c20 tx=0x7fb280009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:40.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.125+0000 7fb285ffb640 1 -- 172.21.15.88:0/2516424696 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb28c0eb380 con 0x7fb2900c41f0 2023-10-16T20:48:40.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.305+0000 7fb29753e640 1 -- 172.21.15.88:0/2516424696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2900ca740 con 0x7fb2900c41f0 2023-10-16T20:48:40.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.305+0000 7fb285ffb640 1 -- 172.21.15.88:0/2516424696 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+6586 (secure 0 0 0) 0x7fb28c0ee9a0 con 0x7fb2900c41f0 2023-10-16T20:48:40.309 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:48:40.309 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5709,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:13.446293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:48:40.309 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 64 2023-10-16T20:48:40.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.305+0000 7fb29753e640 1 -- 172.21.15.88:0/2516424696 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb2740664a0 msgr2=0x7fb274068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:40.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.305+0000 7fb29753e640 1 --2- 172.21.15.88:0/2516424696 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb2740664a0 0x7fb274068960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb290152c20 tx=0x7fb280009190 comp rx=0 tx=0).stop 2023-10-16T20:48:40.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.305+0000 7fb29753e640 1 -- 172.21.15.88:0/2516424696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2900c41f0 msgr2=0x7fb2901f1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:40.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.305+0000 7fb29753e640 1 --2- 172.21.15.88:0/2516424696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2900c41f0 0x7fb2901f1570 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb28c0ae040 tx=0x7fb28c01aca0 comp rx=0 tx=0).stop 2023-10-16T20:48:40.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.305+0000 7fb29753e640 1 -- 172.21.15.88:0/2516424696 shutdown_connections 2023-10-16T20:48:40.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.305+0000 7fb29753e640 1 --2- 172.21.15.88:0/2516424696 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb2740664a0 0x7fb274068960 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:40.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.305+0000 7fb29753e640 1 --2- 172.21.15.88:0/2516424696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb290153770 0x7fb29013e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:40.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.305+0000 7fb29753e640 1 --2- 172.21.15.88:0/2516424696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb2900c9bb0 0x7fb2901496f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:40.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.305+0000 7fb29753e640 1 --2- 172.21.15.88:0/2516424696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2900c41f0 0x7fb2901f1570 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:40.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.305+0000 7fb29753e640 1 -- 172.21.15.88:0/2516424696 >> 172.21.15.88:0/2516424696 conn(0x7fb2900715c0 msgr2=0x7fb2900b3770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:40.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.305+0000 7fb29753e640 1 -- 172.21.15.88:0/2516424696 shutdown_connections 2023-10-16T20:48:40.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:40.305+0000 7fb29753e640 1 -- 172.21.15.88:0/2516424696 wait complete. 2023-10-16T20:48:40.329 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-16T20:48:42.330 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:48:42.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d41194640 1 Processor -- start 2023-10-16T20:48:42.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d41194640 1 -- start start 2023-10-16T20:48:42.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d41194640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1d3c0cece0 0x7f1d3c0cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:42.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d41194640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c153b50 con 0x7f1d3c153770 2023-10-16T20:48:42.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d41194640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c0cb2f0 con 0x7f1d3c0c9bb0 2023-10-16T20:48:42.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d41194640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d3c0cc250 con 0x7f1d3c0cece0 2023-10-16T20:48:42.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3ad9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1d3c153770 0x7f1d3c159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:39288/0 (socket says 172.21.15.88:39288) 2023-10-16T20:48:42.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3ad9d640 1 -- 172.21.15.88:0/1534878202 learned_addr learned my addr 172.21.15.88:0/1534878202 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:48:42.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3a59c640 1 -- 172.21.15.88:0/1534878202 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50139140 0 0) 0x7f1d3c153b50 con 0x7f1d3c153770 2023-10-16T20:48:42.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3a59c640 1 -- 172.21.15.88:0/1534878202 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d28003530 con 0x7f1d3c153770 2023-10-16T20:48:42.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3b59e640 1 --2- 172.21.15.88:0/1534878202 >> v2:172.21.15.191:3300/0 conn(0x7f1d3c0cece0 0x7f1d3c0cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:42.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3a59c640 1 -- 172.21.15.88:0/1534878202 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 70976698 0 0) 0x7f1d28003530 con 0x7f1d3c153770 2023-10-16T20:48:42.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3a59c640 1 -- 172.21.15.88:0/1534878202 >> v2:172.21.15.191:3300/0 conn(0x7f1d3c0cece0 msgr2=0x7f1d3c0cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:42.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3a59c640 1 --2- 172.21.15.88:0/1534878202 >> v2:172.21.15.191:3300/0 conn(0x7f1d3c0cece0 0x7f1d3c0cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:42.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3a59c640 1 -- 172.21.15.88:0/1534878202 >> v1:172.21.15.191:6789/0 conn(0x7f1d3c0c9bb0 legacy=0x7f1d3c0c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:42.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3a59c640 1 -- 172.21.15.88:0/1534878202 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c0cb5d0 con 0x7f1d3c153770 2023-10-16T20:48:42.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3a59c640 1 -- 172.21.15.88:0/1534878202 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f1d2c001800 con 0x7f1d3c153770 2023-10-16T20:48:42.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3a59c640 1 -- 172.21.15.88:0/1534878202 >> v1:172.21.15.88:6789/0 conn(0x7f1d3c153770 legacy=0x7f1d3c159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:42.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3a59c640 1 --2- 172.21.15.88:0/1534878202 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d28003980 0x7f1d28003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:42.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3a59c640 1 --2- 172.21.15.88:0/1534878202 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d280047d0 0x7f1d28004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:42.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3a59c640 1 --2- 172.21.15.88:0/1534878202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d28005190 0x7f1d2800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:42.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3a59c640 1 -- 172.21.15.88:0/1534878202 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c0cc250 con 0x7f1d28005190 2023-10-16T20:48:42.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3a59c640 1 -- 172.21.15.88:0/1534878202 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2800c0c0 con 0x7f1d28003980 2023-10-16T20:48:42.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3a59c640 1 -- 172.21.15.88:0/1534878202 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2800c280 con 0x7f1d280047d0 2023-10-16T20:48:42.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3b59e640 1 --2- 172.21.15.88:0/1534878202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d28005190 0x7f1d2800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:42.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d3ad9d640 1 --2- 172.21.15.88:0/1534878202 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d28003980 0x7f1d28003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:42.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.465+0000 7f1d33fff640 1 --2- 172.21.15.88:0/1534878202 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d280047d0 0x7f1d28004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:42.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.469+0000 7f1d3b59e640 1 -- 172.21.15.88:0/1534878202 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d280047d0 msgr2=0x7f1d28004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:42.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.469+0000 7f1d3b59e640 1 --2- 172.21.15.88:0/1534878202 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d280047d0 0x7f1d28004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:42.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.469+0000 7f1d3b59e640 1 -- 172.21.15.88:0/1534878202 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d28003980 msgr2=0x7f1d28003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:42.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.469+0000 7f1d3b59e640 1 --2- 172.21.15.88:0/1534878202 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d28003980 0x7f1d28003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:42.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.469+0000 7f1d3b59e640 1 -- 172.21.15.88:0/1534878202 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d28003530 con 0x7f1d28005190 2023-10-16T20:48:42.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.469+0000 7f1d3b59e640 1 -- 172.21.15.88:0/1534878202 reap_dead start 2023-10-16T20:48:42.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.469+0000 7f1d3b59e640 1 --2- 172.21.15.88:0/1534878202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d28005190 0x7f1d2800b990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1d34007330 tx=0x7f1d3401ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=158ef1ca610cdbcd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:42.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.469+0000 7f1d3a59c640 1 -- 172.21.15.88:0/1534878202 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d340a5d40 con 0x7f1d28005190 2023-10-16T20:48:42.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.469+0000 7f1d3a59c640 1 -- 172.21.15.88:0/1534878202 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d340039a0 con 0x7f1d28005190 2023-10-16T20:48:42.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.469+0000 7f1d3a59c640 1 -- 172.21.15.88:0/1534878202 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d340a5ea0 con 0x7f1d28005190 2023-10-16T20:48:42.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.469+0000 7f1d41194640 1 -- 172.21.15.88:0/1534878202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d28005190 msgr2=0x7f1d2800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:42.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.469+0000 7f1d41194640 1 --2- 172.21.15.88:0/1534878202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d28005190 0x7f1d2800b990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1d34007330 tx=0x7f1d3401ce40 comp rx=0 tx=0).stop 2023-10-16T20:48:42.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.469+0000 7f1d41194640 1 -- 172.21.15.88:0/1534878202 shutdown_connections 2023-10-16T20:48:42.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.469+0000 7f1d41194640 1 --2- 172.21.15.88:0/1534878202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d28005190 0x7f1d2800b990 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:42.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.469+0000 7f1d41194640 1 -- 172.21.15.88:0/1534878202 >> 172.21.15.88:0/1534878202 conn(0x7f1d3c0715c0 msgr2=0x7f1d3c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:42.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.469+0000 7f1d41194640 1 -- 172.21.15.88:0/1534878202 shutdown_connections 2023-10-16T20:48:42.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.469+0000 7f1d41194640 1 -- 172.21.15.88:0/1534878202 wait complete. 2023-10-16T20:48:42.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d41194640 1 Processor -- start 2023-10-16T20:48:42.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d41194640 1 -- start start 2023-10-16T20:48:42.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d41194640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d28005190 0x7f1d3c0b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:42.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d41194640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d3c0c8e80 0x7f1d3c15c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:42.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d41194640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d3c15b070 0x7f1d3c15c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:42.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d41194640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c157640 con 0x7f1d3c0c8e80 2023-10-16T20:48:42.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d41194640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c0cd060 con 0x7f1d28005190 2023-10-16T20:48:42.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d41194640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c0c71d0 con 0x7f1d3c15b070 2023-10-16T20:48:42.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d33fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d3c0c8e80 0x7f1d3c15c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:42.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d33fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d3c0c8e80 0x7f1d3c15c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47426/0 (socket says 172.21.15.88:47426) 2023-10-16T20:48:42.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d33fff640 1 -- 172.21.15.88:0/1710019908 learned_addr learned my addr 172.21.15.88:0/1710019908 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:42.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d33fff640 1 -- 172.21.15.88:0/1710019908 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d3c15b070 msgr2=0x7f1d3c15c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:42.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d3ad9d640 1 --2- 172.21.15.88:0/1710019908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d28005190 0x7f1d3c0b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:42.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d3b59e640 1 --2- 172.21.15.88:0/1710019908 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d3c15b070 0x7f1d3c15c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:42.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d33fff640 1 --2- 172.21.15.88:0/1710019908 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d3c15b070 0x7f1d3c15c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:42.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d33fff640 1 -- 172.21.15.88:0/1710019908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d28005190 msgr2=0x7f1d3c0b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:42.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d33fff640 1 --2- 172.21.15.88:0/1710019908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d28005190 0x7f1d3c0b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:42.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d33fff640 1 -- 172.21.15.88:0/1710019908 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c0c74b0 con 0x7f1d3c0c8e80 2023-10-16T20:48:42.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d3b59e640 1 --2- 172.21.15.88:0/1710019908 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d3c15b070 0x7f1d3c15c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:48:42.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d3ad9d640 1 --2- 172.21.15.88:0/1710019908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d28005190 0x7f1d3c0b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:48:42.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d33fff640 1 --2- 172.21.15.88:0/1710019908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d3c0c8e80 0x7f1d3c15c200 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400e490 tx=0x7f1d2400e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:42.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d337fe640 1 -- 172.21.15.88:0/1710019908 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d240135f0 con 0x7f1d3c0c8e80 2023-10-16T20:48:42.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d337fe640 1 -- 172.21.15.88:0/1710019908 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d24013750 con 0x7f1d3c0c8e80 2023-10-16T20:48:42.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d337fe640 1 -- 172.21.15.88:0/1710019908 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d24013b00 con 0x7f1d3c0c8e80 2023-10-16T20:48:42.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.473+0000 7f1d41194640 1 -- 172.21.15.88:0/1710019908 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d3c0c7790 con 0x7f1d3c0c8e80 2023-10-16T20:48:42.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.477+0000 7f1d41194640 1 -- 172.21.15.88:0/1710019908 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3c0c7d50 con 0x7f1d3c0c8e80 2023-10-16T20:48:42.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.477+0000 7f1d41194640 1 -- 172.21.15.88:0/1710019908 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d3c15bf20 con 0x7f1d3c0c8e80 2023-10-16T20:48:42.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.477+0000 7f1d337fe640 1 -- 172.21.15.88:0/1710019908 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1d24013c60 con 0x7f1d3c0c8e80 2023-10-16T20:48:42.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.477+0000 7f1d337fe640 1 --2- 172.21.15.88:0/1710019908 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1d1c0664d0 0x7f1d1c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:42.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.477+0000 7f1d337fe640 1 -- 172.21.15.88:0/1710019908 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f1d24084090 con 0x7f1d3c0c8e80 2023-10-16T20:48:42.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.477+0000 7f1d3ad9d640 1 --2- 172.21.15.88:0/1710019908 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1d1c0664d0 0x7f1d1c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:42.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.477+0000 7f1d3ad9d640 1 --2- 172.21.15.88:0/1710019908 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1d1c0664d0 0x7f1d1c068990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c000f00 tx=0x7f1d2c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:42.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.481+0000 7f1d337fe640 1 -- 172.21.15.88:0/1710019908 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1d3c15bf20 con 0x7f1d3c0c8e80 2023-10-16T20:48:42.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.677+0000 7f1d41194640 1 -- 172.21.15.88:0/1710019908 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d3c15bf20 con 0x7f1d3c0c8e80 2023-10-16T20:48:42.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.677+0000 7f1d337fe640 1 -- 172.21.15.88:0/1710019908 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+6584 (secure 0 0 0) 0x7f1d3c15bf20 con 0x7f1d3c0c8e80 2023-10-16T20:48:42.681 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:48:42.681 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5709,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:41.517530+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":4279,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":1,"incarnation":10,"state":"up:active","state_seq":40,"addr":"172.21.15.191:6833/2446850309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":2446850309},{"type":"v1","addr":"172.21.15.191:6833","nonce":2446850309}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:48:42.681 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 65 2023-10-16T20:48:42.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.677+0000 7f1d41194640 1 -- 172.21.15.88:0/1710019908 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1d1c0664d0 msgr2=0x7f1d1c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:42.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.677+0000 7f1d41194640 1 --2- 172.21.15.88:0/1710019908 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1d1c0664d0 0x7f1d1c068990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c000f00 tx=0x7f1d2c009040 comp rx=0 tx=0).stop 2023-10-16T20:48:42.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.677+0000 7f1d41194640 1 -- 172.21.15.88:0/1710019908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d3c0c8e80 msgr2=0x7f1d3c15c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:42.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.677+0000 7f1d41194640 1 --2- 172.21.15.88:0/1710019908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d3c0c8e80 0x7f1d3c15c200 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400e490 tx=0x7f1d2400e950 comp rx=0 tx=0).stop 2023-10-16T20:48:42.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.677+0000 7f1d41194640 1 -- 172.21.15.88:0/1710019908 shutdown_connections 2023-10-16T20:48:42.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.677+0000 7f1d41194640 1 --2- 172.21.15.88:0/1710019908 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1d1c0664d0 0x7f1d1c068990 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:42.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.677+0000 7f1d41194640 1 --2- 172.21.15.88:0/1710019908 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1d3c15b070 0x7f1d3c15c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:42.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.677+0000 7f1d41194640 1 --2- 172.21.15.88:0/1710019908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d3c0c8e80 0x7f1d3c15c200 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:42.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.677+0000 7f1d41194640 1 --2- 172.21.15.88:0/1710019908 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d28005190 0x7f1d3c0b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:42.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.677+0000 7f1d41194640 1 -- 172.21.15.88:0/1710019908 >> 172.21.15.88:0/1710019908 conn(0x7f1d3c0715c0 msgr2=0x7f1d3c0cf110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:42.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.677+0000 7f1d41194640 1 -- 172.21.15.88:0/1710019908 shutdown_connections 2023-10-16T20:48:42.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:42.677+0000 7f1d41194640 1 -- 172.21.15.88:0/1710019908 wait complete. 2023-10-16T20:48:42.702 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-16T20:48:44.703 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:48:44.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd9ab3e640 1 Processor -- start 2023-10-16T20:48:44.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd9ab3e640 1 -- start start 2023-10-16T20:48:44.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd9ab3e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd940cece0 0x7efd940cf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:44.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd9ab3e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd94153b50 con 0x7efd94153770 2023-10-16T20:48:44.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd9ab3e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd940cb2f0 con 0x7efd940c9bb0 2023-10-16T20:48:44.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd9ab3e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd940cc250 con 0x7efd940cece0 2023-10-16T20:48:44.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd988da640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7efd94153770 0x7efd94159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:39296/0 (socket says 172.21.15.88:39296) 2023-10-16T20:48:44.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd988da640 1 -- 172.21.15.88:0/4003177163 learned_addr learned my addr 172.21.15.88:0/4003177163 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:48:44.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd937fe640 1 -- 172.21.15.88:0/4003177163 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180230679 0 0) 0x7efd94153b50 con 0x7efd94153770 2023-10-16T20:48:44.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd937fe640 1 -- 172.21.15.88:0/4003177163 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd84003530 con 0x7efd94153770 2023-10-16T20:48:44.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd990db640 1 --2- 172.21.15.88:0/4003177163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd940cece0 0x7efd940cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:44.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd937fe640 1 -- 172.21.15.88:0/4003177163 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 738416393 0 0) 0x7efd84003530 con 0x7efd94153770 2023-10-16T20:48:44.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd937fe640 1 -- 172.21.15.88:0/4003177163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd940cece0 msgr2=0x7efd940cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:44.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd937fe640 1 --2- 172.21.15.88:0/4003177163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd940cece0 0x7efd940cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:44.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd937fe640 1 -- 172.21.15.88:0/4003177163 >> v1:172.21.15.191:6789/0 conn(0x7efd940c9bb0 legacy=0x7efd940c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:44.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd937fe640 1 -- 172.21.15.88:0/4003177163 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd940cb5d0 con 0x7efd94153770 2023-10-16T20:48:44.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd937fe640 1 -- 172.21.15.88:0/4003177163 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7efd88001800 con 0x7efd94153770 2023-10-16T20:48:44.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd937fe640 1 -- 172.21.15.88:0/4003177163 >> v1:172.21.15.88:6789/0 conn(0x7efd94153770 legacy=0x7efd94159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:44.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd937fe640 1 --2- 172.21.15.88:0/4003177163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd840039b0 0x7efd84003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:44.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd937fe640 1 --2- 172.21.15.88:0/4003177163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd840047d0 0x7efd84004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:44.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd937fe640 1 --2- 172.21.15.88:0/4003177163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd940cece0 0x7efd84005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:44.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd937fe640 1 -- 172.21.15.88:0/4003177163 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd940cc250 con 0x7efd840047d0 2023-10-16T20:48:44.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd937fe640 1 -- 172.21.15.88:0/4003177163 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8400ba40 con 0x7efd840039b0 2023-10-16T20:48:44.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd937fe640 1 -- 172.21.15.88:0/4003177163 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd8400bc00 con 0x7efd940cece0 2023-10-16T20:48:44.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.837+0000 7efd93fff640 1 --2- 172.21.15.88:0/4003177163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd840047d0 0x7efd84004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:44.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.841+0000 7efd988da640 1 --2- 172.21.15.88:0/4003177163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd840039b0 0x7efd84003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:44.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.841+0000 7efd990db640 1 --2- 172.21.15.88:0/4003177163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd940cece0 0x7efd84005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:44.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.841+0000 7efd93fff640 1 -- 172.21.15.88:0/4003177163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd940cece0 msgr2=0x7efd84005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:44.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.841+0000 7efd93fff640 1 --2- 172.21.15.88:0/4003177163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd940cece0 0x7efd84005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:44.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.841+0000 7efd93fff640 1 -- 172.21.15.88:0/4003177163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd840039b0 msgr2=0x7efd84003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:44.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.841+0000 7efd93fff640 1 --2- 172.21.15.88:0/4003177163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd840039b0 0x7efd84003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:44.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.841+0000 7efd93fff640 1 -- 172.21.15.88:0/4003177163 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd84003530 con 0x7efd840047d0 2023-10-16T20:48:44.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.841+0000 7efd93fff640 1 --2- 172.21.15.88:0/4003177163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd840047d0 0x7efd84004c50 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7efd80001750 tx=0x7efd8001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70be0b7b56d14017 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:44.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.841+0000 7efd937fe640 1 -- 172.21.15.88:0/4003177163 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd800052d0 con 0x7efd840047d0 2023-10-16T20:48:44.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.841+0000 7efd937fe640 1 -- 172.21.15.88:0/4003177163 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd800a7050 con 0x7efd840047d0 2023-10-16T20:48:44.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.841+0000 7efd937fe640 1 -- 172.21.15.88:0/4003177163 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd80003600 con 0x7efd840047d0 2023-10-16T20:48:44.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.841+0000 7efd9ab3e640 1 -- 172.21.15.88:0/4003177163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd840047d0 msgr2=0x7efd84004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:44.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.841+0000 7efd9ab3e640 1 --2- 172.21.15.88:0/4003177163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd840047d0 0x7efd84004c50 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7efd80001750 tx=0x7efd8001c3a0 comp rx=0 tx=0).stop 2023-10-16T20:48:44.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.841+0000 7efd990db640 1 -- 172.21.15.88:0/4003177163 reap_dead start 2023-10-16T20:48:44.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd9ab3e640 1 -- 172.21.15.88:0/4003177163 shutdown_connections 2023-10-16T20:48:44.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd9ab3e640 1 -- 172.21.15.88:0/4003177163 >> 172.21.15.88:0/4003177163 conn(0x7efd940715c0 msgr2=0x7efd940719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:44.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd9ab3e640 1 -- 172.21.15.88:0/4003177163 shutdown_connections 2023-10-16T20:48:44.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd9ab3e640 1 -- 172.21.15.88:0/4003177163 wait complete. 2023-10-16T20:48:44.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd9ab3e640 1 Processor -- start 2023-10-16T20:48:44.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd9ab3e640 1 -- start start 2023-10-16T20:48:44.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd9ab3e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd840047d0 0x7efd940cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:44.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd9ab3e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd9415a470 0x7efd940c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:44.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd9ab3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd940c6760 0x7efd94146610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:44.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd9ab3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9415ac10 con 0x7efd940c6760 2023-10-16T20:48:44.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd9ab3e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd94148c90 con 0x7efd9415a470 2023-10-16T20:48:44.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd9ab3e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd9415b200 con 0x7efd840047d0 2023-10-16T20:48:44.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd990db640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd940c6760 0x7efd94146610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:44.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd990db640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd940c6760 0x7efd94146610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47464/0 (socket says 172.21.15.88:47464) 2023-10-16T20:48:44.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd990db640 1 -- 172.21.15.88:0/1638909691 learned_addr learned my addr 172.21.15.88:0/1638909691 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:44.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd93fff640 1 --2- 172.21.15.88:0/1638909691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd9415a470 0x7efd940c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:44.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd988da640 1 --2- 172.21.15.88:0/1638909691 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd840047d0 0x7efd940cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:44.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd990db640 1 -- 172.21.15.88:0/1638909691 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd840047d0 msgr2=0x7efd940cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:44.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd990db640 1 --2- 172.21.15.88:0/1638909691 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd840047d0 0x7efd940cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:44.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd990db640 1 -- 172.21.15.88:0/1638909691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd9415a470 msgr2=0x7efd940c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:44.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd990db640 1 --2- 172.21.15.88:0/1638909691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd9415a470 0x7efd940c6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:44.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd990db640 1 -- 172.21.15.88:0/1638909691 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd940c6be0 con 0x7efd940c6760 2023-10-16T20:48:44.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd93fff640 1 --2- 172.21.15.88:0/1638909691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd9415a470 0x7efd940c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:48:44.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd988da640 1 --2- 172.21.15.88:0/1638909691 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd840047d0 0x7efd940cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:48:44.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.845+0000 7efd990db640 1 --2- 172.21.15.88:0/1638909691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd940c6760 0x7efd94146610 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7efd940cc7e0 tx=0x7efd8c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:44.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.849+0000 7efd917fa640 1 -- 172.21.15.88:0/1638909691 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c00e040 con 0x7efd940c6760 2023-10-16T20:48:44.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.849+0000 7efd917fa640 1 -- 172.21.15.88:0/1638909691 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8c00f040 con 0x7efd940c6760 2023-10-16T20:48:44.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.849+0000 7efd917fa640 1 -- 172.21.15.88:0/1638909691 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c015670 con 0x7efd940c6760 2023-10-16T20:48:44.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.849+0000 7efd9ab3e640 1 -- 172.21.15.88:0/1638909691 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd940c8b40 con 0x7efd940c6760 2023-10-16T20:48:44.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.849+0000 7efd9ab3e640 1 -- 172.21.15.88:0/1638909691 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd94148640 con 0x7efd940c6760 2023-10-16T20:48:44.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.849+0000 7efd9ab3e640 1 -- 172.21.15.88:0/1638909691 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd941484a0 con 0x7efd940c6760 2023-10-16T20:48:44.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.849+0000 7efd917fa640 1 -- 172.21.15.88:0/1638909691 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efd8c0157d0 con 0x7efd940c6760 2023-10-16T20:48:44.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.849+0000 7efd917fa640 1 --2- 172.21.15.88:0/1638909691 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efd780664d0 0x7efd78068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:44.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.849+0000 7efd917fa640 1 -- 172.21.15.88:0/1638909691 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7efd8c084c00 con 0x7efd940c6760 2023-10-16T20:48:44.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.849+0000 7efd988da640 1 --2- 172.21.15.88:0/1638909691 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efd780664d0 0x7efd78068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:44.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.853+0000 7efd988da640 1 --2- 172.21.15.88:0/1638909691 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efd780664d0 0x7efd78068990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7efd88000f00 tx=0x7efd880012a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:44.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:44.853+0000 7efd917fa640 1 -- 172.21.15.88:0/1638909691 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efd941484a0 con 0x7efd940c6760 2023-10-16T20:48:45.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.033+0000 7efd9ab3e640 1 -- 172.21.15.88:0/1638909691 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd941484a0 con 0x7efd940c6760 2023-10-16T20:48:45.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.037+0000 7efd917fa640 1 -- 172.21.15.88:0/1638909691 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5785 (secure 0 0 0) 0x7efd941484a0 con 0x7efd940c6760 2023-10-16T20:48:45.040 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:48:45.040 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:43.641917+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:48:45.040 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 67 2023-10-16T20:48:45.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.037+0000 7efd9ab3e640 1 -- 172.21.15.88:0/1638909691 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efd780664d0 msgr2=0x7efd78068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:45.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.037+0000 7efd9ab3e640 1 --2- 172.21.15.88:0/1638909691 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efd780664d0 0x7efd78068990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7efd88000f00 tx=0x7efd880012a0 comp rx=0 tx=0).stop 2023-10-16T20:48:45.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.037+0000 7efd9ab3e640 1 -- 172.21.15.88:0/1638909691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd940c6760 msgr2=0x7efd94146610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:45.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.037+0000 7efd9ab3e640 1 --2- 172.21.15.88:0/1638909691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd940c6760 0x7efd94146610 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7efd940cc7e0 tx=0x7efd8c00f580 comp rx=0 tx=0).stop 2023-10-16T20:48:45.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.037+0000 7efd9ab3e640 1 -- 172.21.15.88:0/1638909691 shutdown_connections 2023-10-16T20:48:45.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.037+0000 7efd9ab3e640 1 --2- 172.21.15.88:0/1638909691 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efd780664d0 0x7efd78068990 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:45.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.037+0000 7efd9ab3e640 1 --2- 172.21.15.88:0/1638909691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd940c6760 0x7efd94146610 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:45.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.037+0000 7efd9ab3e640 1 --2- 172.21.15.88:0/1638909691 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd9415a470 0x7efd940c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:45.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.037+0000 7efd9ab3e640 1 --2- 172.21.15.88:0/1638909691 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd840047d0 0x7efd940cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:45.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.037+0000 7efd9ab3e640 1 -- 172.21.15.88:0/1638909691 >> 172.21.15.88:0/1638909691 conn(0x7efd940715c0 msgr2=0x7efd940c5630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:45.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.037+0000 7efd9ab3e640 1 -- 172.21.15.88:0/1638909691 shutdown_connections 2023-10-16T20:48:45.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.037+0000 7efd9ab3e640 1 -- 172.21.15.88:0/1638909691 wait complete. 2023-10-16T20:48:45.061 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-16T20:48:45.061 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-16T20:48:45.061 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:48:45.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.193+0000 7fc12110e640 1 Processor -- start 2023-10-16T20:48:45.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.193+0000 7fc12110e640 1 -- start start 2023-10-16T20:48:45.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.197+0000 7fc12110e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc11c153790 0x7fc11c0b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:45.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.197+0000 7fc12110e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc11c06d720 0x7fc11c06db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:45.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.197+0000 7fc12110e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11c06d290 con 0x7fc11c0c6a70 2023-10-16T20:48:45.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.197+0000 7fc12110e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc11c159500 con 0x7fc11c153790 2023-10-16T20:48:45.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.197+0000 7fc12110e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c0c6830 con 0x7fc11c06d720 2023-10-16T20:48:45.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.197+0000 7fc11ad9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc11c153790 0x7fc11c0b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:45.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.197+0000 7fc11259c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc11c0c6a70 0x7fc11c0c5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:39308/0 (socket says 172.21.15.88:39308) 2023-10-16T20:48:45.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.197+0000 7fc11259c640 1 -- 172.21.15.88:0/3027637693 learned_addr learned my addr 172.21.15.88:0/3027637693 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:48:45.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.197+0000 7fc11b59e640 1 --2- 172.21.15.88:0/3027637693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc11c06d720 0x7fc11c06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:45.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.197+0000 7fc11ad9d640 1 -- 172.21.15.88:0/3027637693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc11c06d720 msgr2=0x7fc11c06db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:45.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.197+0000 7fc11ad9d640 1 --2- 172.21.15.88:0/3027637693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc11c06d720 0x7fc11c06db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:45.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.197+0000 7fc11ad9d640 1 -- 172.21.15.88:0/3027637693 >> v1:172.21.15.88:6789/0 conn(0x7fc11c0c6a70 legacy=0x7fc11c0c5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:45.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.197+0000 7fc11ad9d640 1 -- 172.21.15.88:0/3027637693 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c06e0b0 con 0x7fc11c153790 2023-10-16T20:48:45.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.197+0000 7fc11a59c640 1 -- 172.21.15.88:0/3027637693 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803740521 0 0) 0x7fc11c06d290 con 0x7fc11c0c6a70 2023-10-16T20:48:45.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.197+0000 7fc11b59e640 1 --2- 172.21.15.88:0/3027637693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc11c06d720 0x7fc11c06db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:48:45.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.201+0000 7fc11ad9d640 1 --2- 172.21.15.88:0/3027637693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc11c153790 0x7fc11c0b0be0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc10c0097c0 tx=0x7fc10c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253f23361345791d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:45.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.201+0000 7fc11a59c640 1 -- 172.21.15.88:0/3027637693 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c0a8970 con 0x7fc11c153790 2023-10-16T20:48:45.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.201+0000 7fc11a59c640 1 -- 172.21.15.88:0/3027637693 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10c0a8ad0 con 0x7fc11c153790 2023-10-16T20:48:45.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.201+0000 7fc11a59c640 1 -- 172.21.15.88:0/3027637693 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c0a8e60 con 0x7fc11c153790 2023-10-16T20:48:45.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.201+0000 7fc12110e640 1 -- 172.21.15.88:0/3027637693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc11c153790 msgr2=0x7fc11c0b0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:45.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.201+0000 7fc12110e640 1 --2- 172.21.15.88:0/3027637693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc11c153790 0x7fc11c0b0be0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc10c0097c0 tx=0x7fc10c01ec70 comp rx=0 tx=0).stop 2023-10-16T20:48:45.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.201+0000 7fc12110e640 1 -- 172.21.15.88:0/3027637693 shutdown_connections 2023-10-16T20:48:45.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.201+0000 7fc12110e640 1 --2- 172.21.15.88:0/3027637693 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc11c06d720 0x7fc11c06db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:45.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.201+0000 7fc12110e640 1 --2- 172.21.15.88:0/3027637693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc11c153790 0x7fc11c0b0be0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:45.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.201+0000 7fc12110e640 1 -- 172.21.15.88:0/3027637693 >> 172.21.15.88:0/3027637693 conn(0x7fc11c0cacc0 msgr2=0x7fc11c0cb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:45.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.201+0000 7fc12110e640 1 -- 172.21.15.88:0/3027637693 shutdown_connections 2023-10-16T20:48:45.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.201+0000 7fc12110e640 1 -- 172.21.15.88:0/3027637693 wait complete. 2023-10-16T20:48:45.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.201+0000 7fc12110e640 1 Processor -- start 2023-10-16T20:48:45.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.201+0000 7fc12110e640 1 -- start start 2023-10-16T20:48:45.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc12110e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc11c06d720 0x7fc11c1554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:45.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc12110e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc11c0c6a70 0x7fc11c0c47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:45.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc12110e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc11c153790 0x7fc11c0c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:45.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc12110e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c1efad0 con 0x7fc11c153790 2023-10-16T20:48:45.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc12110e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c0c4cf0 con 0x7fc11c06d720 2023-10-16T20:48:45.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc12110e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c0c4f10 con 0x7fc11c0c6a70 2023-10-16T20:48:45.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc11b59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc11c153790 0x7fc11c0c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:45.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc11b59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc11c153790 0x7fc11c0c9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47472/0 (socket says 172.21.15.88:47472) 2023-10-16T20:48:45.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc11b59e640 1 -- 172.21.15.88:0/4290417757 learned_addr learned my addr 172.21.15.88:0/4290417757 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:45.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc11ad9d640 1 --2- 172.21.15.88:0/4290417757 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc11c06d720 0x7fc11c1554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:45.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc11259c640 1 --2- 172.21.15.88:0/4290417757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc11c0c6a70 0x7fc11c0c47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:45.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc11b59e640 1 -- 172.21.15.88:0/4290417757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc11c0c6a70 msgr2=0x7fc11c0c47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:45.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc11b59e640 1 --2- 172.21.15.88:0/4290417757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc11c0c6a70 0x7fc11c0c47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:45.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc11b59e640 1 -- 172.21.15.88:0/4290417757 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc11c06d720 msgr2=0x7fc11c1554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:45.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc11b59e640 1 --2- 172.21.15.88:0/4290417757 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc11c06d720 0x7fc11c1554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:45.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc11b59e640 1 -- 172.21.15.88:0/4290417757 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10c009420 con 0x7fc11c153790 2023-10-16T20:48:45.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc11259c640 1 --2- 172.21.15.88:0/4290417757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc11c0c6a70 0x7fc11c0c47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:48:45.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc11b59e640 1 --2- 172.21.15.88:0/4290417757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc11c153790 0x7fc11c0c9f30 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc11400cdb0 tx=0x7fc11400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:45.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc113fff640 1 -- 172.21.15.88:0/4290417757 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11400f680 con 0x7fc11c153790 2023-10-16T20:48:45.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc113fff640 1 -- 172.21.15.88:0/4290417757 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11400f800 con 0x7fc11c153790 2023-10-16T20:48:45.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc12110e640 1 -- 172.21.15.88:0/4290417757 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11c1eee50 con 0x7fc11c153790 2023-10-16T20:48:45.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc113fff640 1 -- 172.21.15.88:0/4290417757 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11400fbb0 con 0x7fc11c153790 2023-10-16T20:48:45.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc12110e640 1 -- 172.21.15.88:0/4290417757 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11c1ef410 con 0x7fc11c153790 2023-10-16T20:48:45.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.205+0000 7fc12110e640 1 -- 172.21.15.88:0/4290417757 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc11c1efc10 con 0x7fc11c153790 2023-10-16T20:48:45.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.209+0000 7fc113fff640 1 -- 172.21.15.88:0/4290417757 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc11400b040 con 0x7fc11c153790 2023-10-16T20:48:45.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.209+0000 7fc113fff640 1 --2- 172.21.15.88:0/4290417757 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0fc0664a0 0x7fc0fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:45.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.209+0000 7fc113fff640 1 -- 172.21.15.88:0/4290417757 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7fc114084ed0 con 0x7fc11c153790 2023-10-16T20:48:45.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.209+0000 7fc11ad9d640 1 --2- 172.21.15.88:0/4290417757 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0fc0664a0 0x7fc0fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:45.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.209+0000 7fc11ad9d640 1 --2- 172.21.15.88:0/4290417757 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0fc0664a0 0x7fc0fc068960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc10c0b2000 tx=0x7fc10c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:45.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.213+0000 7fc113fff640 1 -- 172.21.15.88:0/4290417757 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc11c1efc10 con 0x7fc11c153790 2023-10-16T20:48:45.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.413+0000 7fc12110e640 1 -- 172.21.15.88:0/4290417757 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc11c1efc10 con 0x7fc11c153790 2023-10-16T20:48:45.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.413+0000 7fc113fff640 1 -- 172.21.15.88:0/4290417757 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5785 (secure 0 0 0) 0x7fc11c1efc10 con 0x7fc11c153790 2023-10-16T20:48:45.420 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:48:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:43.641917+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:48:45.420 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 67 2023-10-16T20:48:45.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.417+0000 7fc12110e640 1 -- 172.21.15.88:0/4290417757 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0fc0664a0 msgr2=0x7fc0fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:45.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.417+0000 7fc12110e640 1 --2- 172.21.15.88:0/4290417757 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0fc0664a0 0x7fc0fc068960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc10c0b2000 tx=0x7fc10c008040 comp rx=0 tx=0).stop 2023-10-16T20:48:45.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.417+0000 7fc12110e640 1 -- 172.21.15.88:0/4290417757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc11c153790 msgr2=0x7fc11c0c9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:45.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.417+0000 7fc12110e640 1 --2- 172.21.15.88:0/4290417757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc11c153790 0x7fc11c0c9f30 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc11400cdb0 tx=0x7fc11400a650 comp rx=0 tx=0).stop 2023-10-16T20:48:45.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.417+0000 7fc12110e640 1 -- 172.21.15.88:0/4290417757 shutdown_connections 2023-10-16T20:48:45.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.417+0000 7fc12110e640 1 --2- 172.21.15.88:0/4290417757 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0fc0664a0 0x7fc0fc068960 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:45.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.417+0000 7fc12110e640 1 --2- 172.21.15.88:0/4290417757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc11c153790 0x7fc11c0c9f30 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:45.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.417+0000 7fc12110e640 1 --2- 172.21.15.88:0/4290417757 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc11c0c6a70 0x7fc11c0c47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:45.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.417+0000 7fc12110e640 1 --2- 172.21.15.88:0/4290417757 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc11c06d720 0x7fc11c1554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:45.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.417+0000 7fc12110e640 1 -- 172.21.15.88:0/4290417757 >> 172.21.15.88:0/4290417757 conn(0x7fc11c0cacc0 msgr2=0x7fc11c0cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:45.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.417+0000 7fc12110e640 1 -- 172.21.15.88:0/4290417757 shutdown_connections 2023-10-16T20:48:45.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:45.417+0000 7fc12110e640 1 -- 172.21.15.88:0/4290417757 wait complete. 2023-10-16T20:48:45.442 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 67, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 67, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4288": { "addr": "172.21.15.191:6835/3059699574", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3059699574, "type": "v2" }, { "addr": "172.21.15.191:6835", "nonce": 3059699574, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4889": { "addr": "172.21.15.191:6837/2468647900", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6836", "nonce": 2468647900, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2468647900, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4889, "incarnation": 30, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 21 }, "gid_5284": { "addr": "172.21.15.88:6839/2172973775", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6837", "nonce": 2172973775, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2172973775, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5284, "incarnation": 38, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 17 }, "gid_5535": { "addr": "172.21.15.88:6835/3476086754", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3476086754, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3476086754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5535, "incarnation": 52, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_5709": { "addr": "172.21.15.88:6838/2600891060", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 2600891060, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 2600891060, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5709, "incarnation": 67, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:48:43.641917+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 5709, "mds_2": 5535, "mds_3": 5284, "mds_4": 4889 } } } ], "standbys": [] } 2023-10-16T20:48:47.443 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:48:47.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e6c17640 1 Processor -- start 2023-10-16T20:48:47.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e6c17640 1 -- start start 2023-10-16T20:48:47.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e6c17640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5e01537c0 0x7ff5e015c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:47.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e6c17640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff5e015cdb0 0x7ff5e00c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:47.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e6c17640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff5e00ca2f0 0x7ff5e00c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:47.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e6c17640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e00cae10 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e6c17640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e00cac30 con 0x7ff5e015cdb0 2023-10-16T20:48:47.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e6c17640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e015d2b0 con 0x7ff5e01537c0 2023-10-16T20:48:47.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e6416640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff5e00ca2f0 0x7ff5e00c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:47.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e6416640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff5e00ca2f0 0x7ff5e00c51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47486/0 (socket says 172.21.15.88:47486) 2023-10-16T20:48:47.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e6416640 1 -- 172.21.15.88:0/373654670 learned_addr learned my addr 172.21.15.88:0/373654670 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:47.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e5414640 1 --2- 172.21.15.88:0/373654670 >> v2:172.21.15.191:3300/0 conn(0x7ff5e015cdb0 0x7ff5e00c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:47.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e5c15640 1 --2- 172.21.15.88:0/373654670 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5e01537c0 0x7ff5e015c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:47.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e6416640 1 -- 172.21.15.88:0/373654670 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5e01537c0 msgr2=0x7ff5e015c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:47.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e6416640 1 --2- 172.21.15.88:0/373654670 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5e01537c0 0x7ff5e015c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:47.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e6416640 1 -- 172.21.15.88:0/373654670 >> v2:172.21.15.191:3300/0 conn(0x7ff5e015cdb0 msgr2=0x7ff5e00c4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:47.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e6416640 1 --2- 172.21.15.88:0/373654670 >> v2:172.21.15.191:3300/0 conn(0x7ff5e015cdb0 0x7ff5e00c4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:47.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e6416640 1 -- 172.21.15.88:0/373654670 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e015d590 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.605+0000 7ff5e5414640 1 --2- 172.21.15.88:0/373654670 >> v2:172.21.15.191:3300/0 conn(0x7ff5e015cdb0 0x7ff5e00c4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:48:47.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.609+0000 7ff5e6416640 1 --2- 172.21.15.88:0/373654670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5e00ca2f0 0x7ff5e00c51c0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff5e00cc650 tx=0x7ff5d801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b52bfa7bdffc2d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:47.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.609+0000 7ff5e4c13640 1 -- 172.21.15.88:0/373654670 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d80af400 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.609+0000 7ff5e4c13640 1 -- 172.21.15.88:0/373654670 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d80ab050 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.609+0000 7ff5e4c13640 1 -- 172.21.15.88:0/373654670 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d80af780 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6c17640 1 -- 172.21.15.88:0/373654670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5e00ca2f0 msgr2=0x7ff5e00c51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:47.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6c17640 1 --2- 172.21.15.88:0/373654670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5e00ca2f0 0x7ff5e00c51c0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff5e00cc650 tx=0x7ff5d801f1c0 comp rx=0 tx=0).stop 2023-10-16T20:48:47.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6c17640 1 -- 172.21.15.88:0/373654670 shutdown_connections 2023-10-16T20:48:47.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6c17640 1 --2- 172.21.15.88:0/373654670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5e00ca2f0 0x7ff5e00c51c0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:47.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6c17640 1 --2- 172.21.15.88:0/373654670 >> v2:172.21.15.191:3300/0 conn(0x7ff5e015cdb0 0x7ff5e00c4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:47.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6c17640 1 --2- 172.21.15.88:0/373654670 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5e01537c0 0x7ff5e015c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:47.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6c17640 1 -- 172.21.15.88:0/373654670 >> 172.21.15.88:0/373654670 conn(0x7ff5e006c170 msgr2=0x7ff5e00715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:47.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6c17640 1 -- 172.21.15.88:0/373654670 shutdown_connections 2023-10-16T20:48:47.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6c17640 1 -- 172.21.15.88:0/373654670 wait complete. 2023-10-16T20:48:47.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6c17640 1 Processor -- start 2023-10-16T20:48:47.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6c17640 1 -- start start 2023-10-16T20:48:47.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6c17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5e00ca2f0 0x7ff5e00c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:47.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6c17640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5e01537c0 0x7ff5e00c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:47.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6c17640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5e015cdb0 0x7ff5e01eff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:47.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6c17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e01efaa0 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6c17640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e00ceef0 con 0x7ff5e01537c0 2023-10-16T20:48:47.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6c17640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e00cec50 con 0x7ff5e015cdb0 2023-10-16T20:48:47.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e5c15640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5e00ca2f0 0x7ff5e00c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:47.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e5c15640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5e00ca2f0 0x7ff5e00c5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47496/0 (socket says 172.21.15.88:47496) 2023-10-16T20:48:47.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e5c15640 1 -- 172.21.15.88:0/2202870613 learned_addr learned my addr 172.21.15.88:0/2202870613 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:47.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e6416640 1 --2- 172.21.15.88:0/2202870613 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5e015cdb0 0x7ff5e01eff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:47.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.613+0000 7ff5e5414640 1 --2- 172.21.15.88:0/2202870613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5e01537c0 0x7ff5e00c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:47.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.617+0000 7ff5e5c15640 1 -- 172.21.15.88:0/2202870613 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5e015cdb0 msgr2=0x7ff5e01eff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:47.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.617+0000 7ff5e5c15640 1 --2- 172.21.15.88:0/2202870613 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5e015cdb0 0x7ff5e01eff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:47.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.617+0000 7ff5e5c15640 1 -- 172.21.15.88:0/2202870613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5e01537c0 msgr2=0x7ff5e00c6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:47.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.617+0000 7ff5e5c15640 1 --2- 172.21.15.88:0/2202870613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5e01537c0 0x7ff5e00c6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:47.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.617+0000 7ff5e5c15640 1 -- 172.21.15.88:0/2202870613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d8009b30 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.617+0000 7ff5e5c15640 1 --2- 172.21.15.88:0/2202870613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5e00ca2f0 0x7ff5e00c5fc0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00d830 tx=0x7ff5dc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:47.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.617+0000 7ff5d67fc640 1 -- 172.21.15.88:0/2202870613 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc00b6c0 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.617+0000 7ff5d67fc640 1 -- 172.21.15.88:0/2202870613 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5dc00b820 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.617+0000 7ff5d67fc640 1 -- 172.21.15.88:0/2202870613 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc00bb90 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.617+0000 7ff5e6c17640 1 -- 172.21.15.88:0/2202870613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e00cdcd0 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.617+0000 7ff5e6c17640 1 -- 172.21.15.88:0/2202870613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e00ce250 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.617+0000 7ff5e6c17640 1 -- 172.21.15.88:0/2202870613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e004e9c0 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.617+0000 7ff5d67fc640 1 -- 172.21.15.88:0/2202870613 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff5dc01e460 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.621+0000 7ff5d67fc640 1 --2- 172.21.15.88:0/2202870613 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff5c40664d0 0x7ff5c4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:47.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.621+0000 7ff5d67fc640 1 -- 172.21.15.88:0/2202870613 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7ff5dc0849f0 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.621+0000 7ff5e5414640 1 --2- 172.21.15.88:0/2202870613 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff5c40664d0 0x7ff5c4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:47.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.621+0000 7ff5e5414640 1 --2- 172.21.15.88:0/2202870613 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff5c40664d0 0x7ff5c4068990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff5e01f2bd0 tx=0x7ff5d0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:47.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.625+0000 7ff5d67fc640 1 -- 172.21.15.88:0/2202870613 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5dc0519a0 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.837+0000 7ff5e6c17640 1 -- 172.21.15.88:0/2202870613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5e00b1ad0 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.837+0000 7ff5d67fc640 1 -- 172.21.15.88:0/2202870613 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5785 (secure 0 0 0) 0x7ff5dc054fc0 con 0x7ff5e00ca2f0 2023-10-16T20:48:47.843 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:48:47.844 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:43.641917+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:48:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 67 2023-10-16T20:48:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.841+0000 7ff5e6c17640 1 -- 172.21.15.88:0/2202870613 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff5c40664d0 msgr2=0x7ff5c4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.841+0000 7ff5e6c17640 1 --2- 172.21.15.88:0/2202870613 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff5c40664d0 0x7ff5c4068990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff5e01f2bd0 tx=0x7ff5d0008040 comp rx=0 tx=0).stop 2023-10-16T20:48:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.841+0000 7ff5e6c17640 1 -- 172.21.15.88:0/2202870613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5e00ca2f0 msgr2=0x7ff5e00c5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:47.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.841+0000 7ff5e6c17640 1 --2- 172.21.15.88:0/2202870613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5e00ca2f0 0x7ff5e00c5fc0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc00d830 tx=0x7ff5dc00dcf0 comp rx=0 tx=0).stop 2023-10-16T20:48:47.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.841+0000 7ff5e6c17640 1 -- 172.21.15.88:0/2202870613 shutdown_connections 2023-10-16T20:48:47.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.841+0000 7ff5e6c17640 1 --2- 172.21.15.88:0/2202870613 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff5c40664d0 0x7ff5c4068990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:47.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.841+0000 7ff5e6c17640 1 --2- 172.21.15.88:0/2202870613 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5e015cdb0 0x7ff5e01eff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:47.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.841+0000 7ff5e6c17640 1 --2- 172.21.15.88:0/2202870613 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5e01537c0 0x7ff5e00c6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:47.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.841+0000 7ff5e6c17640 1 --2- 172.21.15.88:0/2202870613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5e00ca2f0 0x7ff5e00c5fc0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:47.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.841+0000 7ff5e6c17640 1 -- 172.21.15.88:0/2202870613 >> 172.21.15.88:0/2202870613 conn(0x7ff5e006c170 msgr2=0x7ff5e015b380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:47.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.841+0000 7ff5e6c17640 1 -- 172.21.15.88:0/2202870613 shutdown_connections 2023-10-16T20:48:47.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:47.841+0000 7ff5e6c17640 1 -- 172.21.15.88:0/2202870613 wait complete. 2023-10-16T20:48:47.863 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:48:49.865 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:48:50.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.021+0000 7f0e92594640 1 Processor -- start 2023-10-16T20:48:50.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.021+0000 7f0e92594640 1 -- start start 2023-10-16T20:48:50.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.021+0000 7f0e92594640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0e8c1406d0 0x7f0e8c0c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:50.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.021+0000 7f0e92594640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f0e8c15a810 0x7f0e8c15ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:50.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.021+0000 7f0e92594640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e8c15b3c0 con 0x7f0e8c06a970 2023-10-16T20:48:50.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.021+0000 7f0e92594640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8c15b1c0 con 0x7f0e8c1406d0 2023-10-16T20:48:50.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.021+0000 7f0e92594640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8c15b680 con 0x7f0e8c15a810 2023-10-16T20:48:50.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.021+0000 7f0e8bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0e8c1406d0 0x7f0e8c0c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:50.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.021+0000 7f0e8bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0e8c1406d0 0x7f0e8c0c1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34728/0 (socket says 172.21.15.88:34728) 2023-10-16T20:48:50.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.021+0000 7f0e8bfff640 1 -- 172.21.15.88:0/1512911092 learned_addr learned my addr 172.21.15.88:0/1512911092 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:50.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.021+0000 7f0e90b31640 1 --2- 172.21.15.88:0/1512911092 >> v2:172.21.15.191:3300/0 conn(0x7f0e8c15a810 0x7f0e8c15ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:50.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.021+0000 7f0e8bfff640 1 -- 172.21.15.88:0/1512911092 >> v2:172.21.15.191:3300/0 conn(0x7f0e8c15a810 msgr2=0x7f0e8c15ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:50.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.021+0000 7f0e8bfff640 1 --2- 172.21.15.88:0/1512911092 >> v2:172.21.15.191:3300/0 conn(0x7f0e8c15a810 0x7f0e8c15ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:50.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.021+0000 7f0e8bfff640 1 -- 172.21.15.88:0/1512911092 >> v1:172.21.15.88:6789/0 conn(0x7f0e8c06a970 legacy=0x7f0e8c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:50.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.021+0000 7f0e8bfff640 1 -- 172.21.15.88:0/1512911092 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8c157570 con 0x7f0e8c1406d0 2023-10-16T20:48:50.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.021+0000 7f0e8affd640 1 -- 172.21.15.88:0/1512911092 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150142746 0 0) 0x7f0e8c15b3c0 con 0x7f0e8c06a970 2023-10-16T20:48:50.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.025+0000 7f0e8bfff640 1 --2- 172.21.15.88:0/1512911092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e8c1406d0 0x7f0e8c0c1d50 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0c41e0 tx=0x7f0e7c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=841fbd52ef10446a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:50.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.025+0000 7f0e8affd640 1 -- 172.21.15.88:0/1512911092 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c0aa430 con 0x7f0e8c1406d0 2023-10-16T20:48:50.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.025+0000 7f0e8affd640 1 -- 172.21.15.88:0/1512911092 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7c0aa590 con 0x7f0e8c1406d0 2023-10-16T20:48:50.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.025+0000 7f0e8affd640 1 -- 172.21.15.88:0/1512911092 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c0aa920 con 0x7f0e8c1406d0 2023-10-16T20:48:50.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.025+0000 7f0e92594640 1 -- 172.21.15.88:0/1512911092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e8c1406d0 msgr2=0x7f0e8c0c1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:50.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.025+0000 7f0e92594640 1 --2- 172.21.15.88:0/1512911092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e8c1406d0 0x7f0e8c0c1d50 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0c41e0 tx=0x7f0e7c01c4d0 comp rx=0 tx=0).stop 2023-10-16T20:48:50.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.025+0000 7f0e92594640 1 -- 172.21.15.88:0/1512911092 shutdown_connections 2023-10-16T20:48:50.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.025+0000 7f0e92594640 1 --2- 172.21.15.88:0/1512911092 >> v2:172.21.15.191:3300/0 conn(0x7f0e8c15a810 0x7f0e8c15ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:50.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.025+0000 7f0e92594640 1 --2- 172.21.15.88:0/1512911092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e8c1406d0 0x7f0e8c0c1d50 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:50.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.025+0000 7f0e92594640 1 -- 172.21.15.88:0/1512911092 >> 172.21.15.88:0/1512911092 conn(0x7f0e8c0cad50 msgr2=0x7f0e8c0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:50.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.029+0000 7f0e92594640 1 -- 172.21.15.88:0/1512911092 shutdown_connections 2023-10-16T20:48:50.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.029+0000 7f0e92594640 1 -- 172.21.15.88:0/1512911092 wait complete. 2023-10-16T20:48:50.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.029+0000 7f0e92594640 1 Processor -- start 2023-10-16T20:48:50.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.029+0000 7f0e92594640 1 -- start start 2023-10-16T20:48:50.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e92594640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e8c06a970 0x7f0e8c13e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:50.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e92594640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e8c1406d0 0x7f0e8c1f1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:50.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e92594640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e8c15a810 0x7f0e8c14fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:50.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e92594640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c1597f0 con 0x7f0e8c15a810 2023-10-16T20:48:50.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e92594640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c1606f0 con 0x7f0e8c06a970 2023-10-16T20:48:50.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e92594640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c1f13d0 con 0x7f0e8c1406d0 2023-10-16T20:48:50.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e90b31640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e8c15a810 0x7f0e8c14fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:50.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e90b31640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e8c15a810 0x7f0e8c14fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34732/0 (socket says 172.21.15.88:34732) 2023-10-16T20:48:50.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e90b31640 1 -- 172.21.15.88:0/3317057041 learned_addr learned my addr 172.21.15.88:0/3317057041 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:50.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e8bfff640 1 --2- 172.21.15.88:0/3317057041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e8c06a970 0x7f0e8c13e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:50.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e8b7fe640 1 --2- 172.21.15.88:0/3317057041 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e8c1406d0 0x7f0e8c1f1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:50.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e8bfff640 1 -- 172.21.15.88:0/3317057041 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e8c1406d0 msgr2=0x7f0e8c1f1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:50.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e8bfff640 1 --2- 172.21.15.88:0/3317057041 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e8c1406d0 0x7f0e8c1f1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:50.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e8bfff640 1 -- 172.21.15.88:0/3317057041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e8c15a810 msgr2=0x7f0e8c14fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:50.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e8bfff640 1 --2- 172.21.15.88:0/3317057041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e8c15a810 0x7f0e8c14fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:50.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e8bfff640 1 -- 172.21.15.88:0/3317057041 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7c0094d0 con 0x7f0e8c06a970 2023-10-16T20:48:50.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e8bfff640 1 --2- 172.21.15.88:0/3317057041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e8c06a970 0x7f0e8c13e180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c0ab000 tx=0x7f0e7c01ad10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:50.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e88ff9640 1 -- 172.21.15.88:0/3317057041 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c0aa430 con 0x7f0e8c06a970 2023-10-16T20:48:50.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e88ff9640 1 -- 172.21.15.88:0/3317057041 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7c0b3820 con 0x7f0e8c06a970 2023-10-16T20:48:50.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e88ff9640 1 -- 172.21.15.88:0/3317057041 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c0b3bd0 con 0x7f0e8c06a970 2023-10-16T20:48:50.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e92594640 1 -- 172.21.15.88:0/3317057041 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e8c159a60 con 0x7f0e8c06a970 2023-10-16T20:48:50.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e92594640 1 -- 172.21.15.88:0/3317057041 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e8c15a020 con 0x7f0e8c06a970 2023-10-16T20:48:50.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.033+0000 7f0e92594640 1 -- 172.21.15.88:0/3317057041 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e8c074700 con 0x7f0e8c06a970 2023-10-16T20:48:50.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.037+0000 7f0e88ff9640 1 -- 172.21.15.88:0/3317057041 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0e7c0b3d30 con 0x7f0e8c06a970 2023-10-16T20:48:50.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.037+0000 7f0e88ff9640 1 --2- 172.21.15.88:0/3317057041 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0e780664a0 0x7f0e78068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:50.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.037+0000 7f0e88ff9640 1 -- 172.21.15.88:0/3317057041 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7f0e7c11e590 con 0x7f0e8c06a970 2023-10-16T20:48:50.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.037+0000 7f0e8b7fe640 1 --2- 172.21.15.88:0/3317057041 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0e780664a0 0x7f0e78068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:50.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.037+0000 7f0e8b7fe640 1 --2- 172.21.15.88:0/3317057041 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0e780664a0 0x7f0e78068960 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c158bf0 tx=0x7f0e80009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:50.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.041+0000 7f0e88ff9640 1 -- 172.21.15.88:0/3317057041 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0e7c0eb540 con 0x7f0e8c06a970 2023-10-16T20:48:50.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.253+0000 7f0e92594640 1 -- 172.21.15.88:0/3317057041 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e8c154c40 con 0x7f0e8c06a970 2023-10-16T20:48:50.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.253+0000 7f0e88ff9640 1 -- 172.21.15.88:0/3317057041 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+5787 (secure 0 0 0) 0x7f0e7c0eeb60 con 0x7f0e8c06a970 2023-10-16T20:48:50.258 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:48:50.258 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:48.991696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:resolve","state_seq":31,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:48:50.259 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 68 2023-10-16T20:48:50.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.257+0000 7f0e92594640 1 -- 172.21.15.88:0/3317057041 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0e780664a0 msgr2=0x7f0e78068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:50.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.257+0000 7f0e92594640 1 --2- 172.21.15.88:0/3317057041 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0e780664a0 0x7f0e78068960 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c158bf0 tx=0x7f0e80009040 comp rx=0 tx=0).stop 2023-10-16T20:48:50.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.257+0000 7f0e92594640 1 -- 172.21.15.88:0/3317057041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e8c06a970 msgr2=0x7f0e8c13e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:50.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.257+0000 7f0e92594640 1 --2- 172.21.15.88:0/3317057041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e8c06a970 0x7f0e8c13e180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c0ab000 tx=0x7f0e7c01ad10 comp rx=0 tx=0).stop 2023-10-16T20:48:50.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.257+0000 7f0e92594640 1 -- 172.21.15.88:0/3317057041 shutdown_connections 2023-10-16T20:48:50.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.257+0000 7f0e92594640 1 --2- 172.21.15.88:0/3317057041 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0e780664a0 0x7f0e78068960 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:50.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.257+0000 7f0e92594640 1 --2- 172.21.15.88:0/3317057041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e8c15a810 0x7f0e8c14fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:50.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.257+0000 7f0e92594640 1 --2- 172.21.15.88:0/3317057041 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0e8c1406d0 0x7f0e8c1f1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:50.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.257+0000 7f0e92594640 1 --2- 172.21.15.88:0/3317057041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0e8c06a970 0x7f0e8c13e180 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:50.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.257+0000 7f0e92594640 1 -- 172.21.15.88:0/3317057041 >> 172.21.15.88:0/3317057041 conn(0x7f0e8c0cad50 msgr2=0x7f0e8c13ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:50.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.257+0000 7f0e92594640 1 -- 172.21.15.88:0/3317057041 shutdown_connections 2023-10-16T20:48:50.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:50.257+0000 7f0e92594640 1 -- 172.21.15.88:0/3317057041 wait complete. 2023-10-16T20:48:50.279 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:48:52.280 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:48:52.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.421+0000 7efdf5bab640 1 Processor -- start 2023-10-16T20:48:52.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.421+0000 7efdf5bab640 1 -- start start 2023-10-16T20:48:52.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.421+0000 7efdf5bab640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efdf01517f0 0x7efdf00c6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:52.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.421+0000 7efdf5bab640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7efdf00c7bd0 0x7efdf00c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:52.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.421+0000 7efdf5bab640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf015b740 con 0x7efdf0151410 2023-10-16T20:48:52.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.421+0000 7efdf5bab640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf00c4860 con 0x7efdf00c7bd0 2023-10-16T20:48:52.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.421+0000 7efdf5bab640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf015b460 con 0x7efdf01517f0 2023-10-16T20:48:52.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.421+0000 7efdef7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7efdf0151410 0x7efdf00c4150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52304/0 (socket says 172.21.15.88:52304) 2023-10-16T20:48:52.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.421+0000 7efdef7fe640 1 -- 172.21.15.88:0/2174667060 learned_addr learned my addr 172.21.15.88:0/2174667060 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:48:52.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.421+0000 7efdeffff640 1 --2- 172.21.15.88:0/2174667060 >> v2:172.21.15.191:3300/0 conn(0x7efdf00c7bd0 0x7efdf00c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:52.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.421+0000 7efdeeffd640 1 --2- 172.21.15.88:0/2174667060 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efdf01517f0 0x7efdf00c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:52.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.421+0000 7efdee7fc640 1 -- 172.21.15.88:0/2174667060 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378302314 0 0) 0x7efdf015b740 con 0x7efdf0151410 2023-10-16T20:48:52.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.421+0000 7efdee7fc640 1 -- 172.21.15.88:0/2174667060 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdd8003530 con 0x7efdf0151410 2023-10-16T20:48:52.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdee7fc640 1 -- 172.21.15.88:0/2174667060 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4154873373 0 0) 0x7efdd8003530 con 0x7efdf0151410 2023-10-16T20:48:52.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdee7fc640 1 -- 172.21.15.88:0/2174667060 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efdf01517f0 msgr2=0x7efdf00c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:52.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdee7fc640 1 --2- 172.21.15.88:0/2174667060 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efdf01517f0 0x7efdf00c6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:52.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdee7fc640 1 -- 172.21.15.88:0/2174667060 >> v2:172.21.15.191:3300/0 conn(0x7efdf00c7bd0 msgr2=0x7efdf00c6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:52.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdee7fc640 1 --2- 172.21.15.88:0/2174667060 >> v2:172.21.15.191:3300/0 conn(0x7efdf00c7bd0 0x7efdf00c6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:52.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdee7fc640 1 -- 172.21.15.88:0/2174667060 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf006d780 con 0x7efdf0151410 2023-10-16T20:48:52.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdee7fc640 1 -- 172.21.15.88:0/2174667060 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7efde0001680 con 0x7efdf0151410 2023-10-16T20:48:52.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdee7fc640 1 -- 172.21.15.88:0/2174667060 >> v1:172.21.15.88:6789/0 conn(0x7efdf0151410 legacy=0x7efdf00c4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:52.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdee7fc640 1 --2- 172.21.15.88:0/2174667060 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efdd8003980 0x7efdd8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:52.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdee7fc640 1 --2- 172.21.15.88:0/2174667060 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efdd80047d0 0x7efdd8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:52.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdee7fc640 1 --2- 172.21.15.88:0/2174667060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdd8005150 0x7efdd800fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:52.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdee7fc640 1 -- 172.21.15.88:0/2174667060 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf00c4860 con 0x7efdd8005150 2023-10-16T20:48:52.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdee7fc640 1 -- 172.21.15.88:0/2174667060 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf015b460 con 0x7efdd8003980 2023-10-16T20:48:52.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdee7fc640 1 -- 172.21.15.88:0/2174667060 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd80045d0 con 0x7efdd80047d0 2023-10-16T20:48:52.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdeffff640 1 --2- 172.21.15.88:0/2174667060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdd8005150 0x7efdd800fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:52.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdef7fe640 1 --2- 172.21.15.88:0/2174667060 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efdd8003980 0x7efdd8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:52.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdeeffd640 1 --2- 172.21.15.88:0/2174667060 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efdd80047d0 0x7efdd8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:52.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdeffff640 1 -- 172.21.15.88:0/2174667060 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efdd80047d0 msgr2=0x7efdd8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:52.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdeffff640 1 --2- 172.21.15.88:0/2174667060 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efdd80047d0 0x7efdd8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:52.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdeffff640 1 -- 172.21.15.88:0/2174667060 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efdd8003980 msgr2=0x7efdd8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:52.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdeffff640 1 --2- 172.21.15.88:0/2174667060 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efdd8003980 0x7efdd8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:52.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdeffff640 1 -- 172.21.15.88:0/2174667060 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd8003530 con 0x7efdd8005150 2023-10-16T20:48:52.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdeeffd640 1 --2- 172.21.15.88:0/2174667060 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efdd80047d0 0x7efdd8004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:48:52.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.425+0000 7efdeffff640 1 --2- 172.21.15.88:0/2174667060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdd8005150 0x7efdd800fda0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7efde8009f90 tx=0x7efde801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec44ce4581788c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:52.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdee7fc640 1 -- 172.21.15.88:0/2174667060 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde80a5c70 con 0x7efdd8005150 2023-10-16T20:48:52.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdee7fc640 1 -- 172.21.15.88:0/2174667060 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde800f040 con 0x7efdd8005150 2023-10-16T20:48:52.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdee7fc640 1 -- 172.21.15.88:0/2174667060 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde80ad440 con 0x7efdd8005150 2023-10-16T20:48:52.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdf5bab640 1 -- 172.21.15.88:0/2174667060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdd8005150 msgr2=0x7efdd800fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:52.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdf5bab640 1 --2- 172.21.15.88:0/2174667060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdd8005150 0x7efdd800fda0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7efde8009f90 tx=0x7efde801cc60 comp rx=0 tx=0).stop 2023-10-16T20:48:52.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdeffff640 1 -- 172.21.15.88:0/2174667060 reap_dead start 2023-10-16T20:48:52.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdf5bab640 1 -- 172.21.15.88:0/2174667060 shutdown_connections 2023-10-16T20:48:52.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdf5bab640 1 -- 172.21.15.88:0/2174667060 >> 172.21.15.88:0/2174667060 conn(0x7efdf0071a60 msgr2=0x7efdf0071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:52.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdf5bab640 1 -- 172.21.15.88:0/2174667060 shutdown_connections 2023-10-16T20:48:52.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdf5bab640 1 -- 172.21.15.88:0/2174667060 wait complete. 2023-10-16T20:48:52.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdf5bab640 1 Processor -- start 2023-10-16T20:48:52.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdf5bab640 1 -- start start 2023-10-16T20:48:52.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdf5bab640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdd8005150 0x7efdf014a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:52.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdf5bab640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efdf00cc760 0x7efdf01ea040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:52.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdf5bab640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efdf01550e0 0x7efdf01ea580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:52.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdf5bab640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf01580d0 con 0x7efdd8005150 2023-10-16T20:48:52.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdf5bab640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf01ee190 con 0x7efdf00cc760 2023-10-16T20:48:52.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdf5bab640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf01ed5c0 con 0x7efdf01550e0 2023-10-16T20:48:52.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdef7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdd8005150 0x7efdf014a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:52.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdef7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdd8005150 0x7efdf014a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34748/0 (socket says 172.21.15.88:34748) 2023-10-16T20:48:52.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.429+0000 7efdef7fe640 1 -- 172.21.15.88:0/4256682897 learned_addr learned my addr 172.21.15.88:0/4256682897 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:52.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.433+0000 7efdef7fe640 1 -- 172.21.15.88:0/4256682897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efdf01550e0 msgr2=0x7efdf01ea580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:52.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.433+0000 7efdef7fe640 1 --2- 172.21.15.88:0/4256682897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efdf01550e0 0x7efdf01ea580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:52.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.433+0000 7efdef7fe640 1 -- 172.21.15.88:0/4256682897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efdf00cc760 msgr2=0x7efdf01ea040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:52.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.433+0000 7efdef7fe640 1 --2- 172.21.15.88:0/4256682897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efdf00cc760 0x7efdf01ea040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:52.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.433+0000 7efdef7fe640 1 -- 172.21.15.88:0/4256682897 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf01ed860 con 0x7efdd8005150 2023-10-16T20:48:52.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.433+0000 7efdef7fe640 1 --2- 172.21.15.88:0/4256682897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdd8005150 0x7efdf014a730 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7efde0007310 tx=0x7efde00077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:52.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.433+0000 7efdd7fff640 1 -- 172.21.15.88:0/4256682897 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde000f940 con 0x7efdd8005150 2023-10-16T20:48:52.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.433+0000 7efdd7fff640 1 -- 172.21.15.88:0/4256682897 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde000fac0 con 0x7efdd8005150 2023-10-16T20:48:52.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.433+0000 7efdd7fff640 1 -- 172.21.15.88:0/4256682897 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde000fe70 con 0x7efdd8005150 2023-10-16T20:48:52.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.433+0000 7efdf5bab640 1 -- 172.21.15.88:0/4256682897 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf01edb40 con 0x7efdd8005150 2023-10-16T20:48:52.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.433+0000 7efdf5bab640 1 -- 172.21.15.88:0/4256682897 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf0156430 con 0x7efdd8005150 2023-10-16T20:48:52.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.433+0000 7efdf5bab640 1 -- 172.21.15.88:0/4256682897 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf0074700 con 0x7efdd8005150 2023-10-16T20:48:52.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.433+0000 7efdd7fff640 1 -- 172.21.15.88:0/4256682897 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efde001d460 con 0x7efdd8005150 2023-10-16T20:48:52.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.433+0000 7efdd7fff640 1 --2- 172.21.15.88:0/4256682897 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efddc0664a0 0x7efddc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:52.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.433+0000 7efdd7fff640 1 -- 172.21.15.88:0/4256682897 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7efde00846d0 con 0x7efdd8005150 2023-10-16T20:48:52.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.433+0000 7efdeeffd640 1 --2- 172.21.15.88:0/4256682897 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efddc0664a0 0x7efddc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:52.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.437+0000 7efdeeffd640 1 --2- 172.21.15.88:0/4256682897 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efddc0664a0 0x7efddc068960 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efdf006b1a0 tx=0x7efde4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:52.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.437+0000 7efdd7fff640 1 -- 172.21.15.88:0/4256682897 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efde0051560 con 0x7efdd8005150 2023-10-16T20:48:52.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.645+0000 7efdf5bab640 1 -- 172.21.15.88:0/4256682897 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdf014cf70 con 0x7efdd8005150 2023-10-16T20:48:52.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.645+0000 7efdd7fff640 1 -- 172.21.15.88:0/4256682897 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+5787 (secure 0 0 0) 0x7efde0054b80 con 0x7efdd8005150 2023-10-16T20:48:52.651 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:48:52.651 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:48.991696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:resolve","state_seq":31,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:48:52.651 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 68 2023-10-16T20:48:52.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.649+0000 7efdf5bab640 1 -- 172.21.15.88:0/4256682897 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efddc0664a0 msgr2=0x7efddc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:52.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.649+0000 7efdf5bab640 1 --2- 172.21.15.88:0/4256682897 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efddc0664a0 0x7efddc068960 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efdf006b1a0 tx=0x7efde4008040 comp rx=0 tx=0).stop 2023-10-16T20:48:52.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.649+0000 7efdf5bab640 1 -- 172.21.15.88:0/4256682897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdd8005150 msgr2=0x7efdf014a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:52.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.649+0000 7efdf5bab640 1 --2- 172.21.15.88:0/4256682897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdd8005150 0x7efdf014a730 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7efde0007310 tx=0x7efde00077d0 comp rx=0 tx=0).stop 2023-10-16T20:48:52.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.649+0000 7efdf5bab640 1 -- 172.21.15.88:0/4256682897 shutdown_connections 2023-10-16T20:48:52.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.649+0000 7efdf5bab640 1 --2- 172.21.15.88:0/4256682897 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efddc0664a0 0x7efddc068960 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:52.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.649+0000 7efdf5bab640 1 --2- 172.21.15.88:0/4256682897 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efdf01550e0 0x7efdf01ea580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:52.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.649+0000 7efdf5bab640 1 --2- 172.21.15.88:0/4256682897 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efdf00cc760 0x7efdf01ea040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:52.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.649+0000 7efdf5bab640 1 --2- 172.21.15.88:0/4256682897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdd8005150 0x7efdf014a730 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:52.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.649+0000 7efdf5bab640 1 -- 172.21.15.88:0/4256682897 >> 172.21.15.88:0/4256682897 conn(0x7efdf0071a60 msgr2=0x7efdf00c6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:52.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.649+0000 7efdf5bab640 1 -- 172.21.15.88:0/4256682897 shutdown_connections 2023-10-16T20:48:52.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:52.649+0000 7efdf5bab640 1 -- 172.21.15.88:0/4256682897 wait complete. 2023-10-16T20:48:52.675 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:48:54.676 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:48:54.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.813+0000 7f3c1aa09640 1 Processor -- start 2023-10-16T20:48:54.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.813+0000 7f3c1aa09640 1 -- start start 2023-10-16T20:48:54.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.813+0000 7f3c1aa09640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c14150280 0x7f3c140c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:54.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.813+0000 7f3c1aa09640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3c140c6bc0 0x7f3c140c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:54.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.813+0000 7f3c1aa09640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c140c5c60 con 0x7f3c14150660 2023-10-16T20:48:54.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.813+0000 7f3c1aa09640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c140c5370 con 0x7f3c140c6bc0 2023-10-16T20:48:54.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.813+0000 7f3c1aa09640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c14156d10 con 0x7f3c14150280 2023-10-16T20:48:54.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.813+0000 7f3c18fa6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3c140c6bc0 0x7f3c140c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:54.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.813+0000 7f3c18fa6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3c140c6bc0 0x7f3c140c3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34756/0 (socket says 172.21.15.88:34756) 2023-10-16T20:48:54.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.813+0000 7f3c18fa6640 1 -- 172.21.15.88:0/356264320 learned_addr learned my addr 172.21.15.88:0/356264320 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:54.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.813+0000 7f3c13fff640 1 --2- 172.21.15.88:0/356264320 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c14150280 0x7f3c140c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:54.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.813+0000 7f3c18fa6640 1 -- 172.21.15.88:0/356264320 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c14150280 msgr2=0x7f3c140c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:54.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.813+0000 7f3c18fa6640 1 --2- 172.21.15.88:0/356264320 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c14150280 0x7f3c140c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:54.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.813+0000 7f3c18fa6640 1 -- 172.21.15.88:0/356264320 >> v1:172.21.15.191:6789/0 conn(0x7f3c14150660 legacy=0x7f3c1406d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:54.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.813+0000 7f3c18fa6640 1 -- 172.21.15.88:0/356264320 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c140c5610 con 0x7f3c140c6bc0 2023-10-16T20:48:54.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.817+0000 7f3c18fa6640 1 --2- 172.21.15.88:0/356264320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c140c6bc0 0x7f3c140c3110 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009e50 tx=0x7f3c0c01d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce612314dd0b4741 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:54.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.817+0000 7f3c12ffd640 1 -- 172.21.15.88:0/356264320 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c0a5830 con 0x7f3c140c6bc0 2023-10-16T20:48:54.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.817+0000 7f3c12ffd640 1 -- 172.21.15.88:0/356264320 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0c0a5990 con 0x7f3c140c6bc0 2023-10-16T20:48:54.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.817+0000 7f3c12ffd640 1 -- 172.21.15.88:0/356264320 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c0a5d20 con 0x7f3c140c6bc0 2023-10-16T20:48:54.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.817+0000 7f3c1aa09640 1 -- 172.21.15.88:0/356264320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c140c6bc0 msgr2=0x7f3c140c3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:54.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.817+0000 7f3c1aa09640 1 --2- 172.21.15.88:0/356264320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c140c6bc0 0x7f3c140c3110 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009e50 tx=0x7f3c0c01d3b0 comp rx=0 tx=0).stop 2023-10-16T20:48:54.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.817+0000 7f3c1aa09640 1 -- 172.21.15.88:0/356264320 shutdown_connections 2023-10-16T20:48:54.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.817+0000 7f3c1aa09640 1 --2- 172.21.15.88:0/356264320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c140c6bc0 0x7f3c140c3110 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:54.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.817+0000 7f3c1aa09640 1 --2- 172.21.15.88:0/356264320 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c14150280 0x7f3c140c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:54.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.817+0000 7f3c1aa09640 1 -- 172.21.15.88:0/356264320 >> 172.21.15.88:0/356264320 conn(0x7f3c140cad50 msgr2=0x7f3c140cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:54.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.817+0000 7f3c1aa09640 1 -- 172.21.15.88:0/356264320 shutdown_connections 2023-10-16T20:48:54.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.817+0000 7f3c1aa09640 1 -- 172.21.15.88:0/356264320 wait complete. 2023-10-16T20:48:54.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c1aa09640 1 Processor -- start 2023-10-16T20:48:54.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c1aa09640 1 -- start start 2023-10-16T20:48:54.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c1aa09640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c140c6bc0 0x7f3c14142c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:54.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c1aa09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c14150280 0x7f3c1413e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:54.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c1aa09640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c14150660 0x7f3c141f1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:54.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c1aa09640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c141f1890 con 0x7f3c14150280 2023-10-16T20:48:54.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c1aa09640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c141401e0 con 0x7f3c14150660 2023-10-16T20:48:54.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c1aa09640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c140ca6b0 con 0x7f3c140c6bc0 2023-10-16T20:48:54.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c137fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c14150280 0x7f3c1413e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:54.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c137fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c14150280 0x7f3c1413e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34762/0 (socket says 172.21.15.88:34762) 2023-10-16T20:48:54.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c137fe640 1 -- 172.21.15.88:0/3316979845 learned_addr learned my addr 172.21.15.88:0/3316979845 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:54.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c18fa6640 1 --2- 172.21.15.88:0/3316979845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c14150660 0x7f3c141f1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:54.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c13fff640 1 --2- 172.21.15.88:0/3316979845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c140c6bc0 0x7f3c14142c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:54.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c137fe640 1 -- 172.21.15.88:0/3316979845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c140c6bc0 msgr2=0x7f3c14142c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:54.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c137fe640 1 --2- 172.21.15.88:0/3316979845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c140c6bc0 0x7f3c14142c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:54.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c137fe640 1 -- 172.21.15.88:0/3316979845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c14150660 msgr2=0x7f3c141f1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:54.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c137fe640 1 --2- 172.21.15.88:0/3316979845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c14150660 0x7f3c141f1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:54.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c137fe640 1 -- 172.21.15.88:0/3316979845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c009b30 con 0x7f3c14150280 2023-10-16T20:48:54.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c137fe640 1 --2- 172.21.15.88:0/3316979845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c14150280 0x7f3c1413e320 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3c0800c3f0 tx=0x7f3c0800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:54.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c10ff9640 1 -- 172.21.15.88:0/3316979845 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c080052d0 con 0x7f3c14150280 2023-10-16T20:48:54.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c10ff9640 1 -- 172.21.15.88:0/3316979845 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c08005430 con 0x7f3c14150280 2023-10-16T20:48:54.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c1aa09640 1 -- 172.21.15.88:0/3316979845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c140c9b70 con 0x7f3c14150280 2023-10-16T20:48:54.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c10ff9640 1 -- 172.21.15.88:0/3316979845 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c08003840 con 0x7f3c14150280 2023-10-16T20:48:54.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.821+0000 7f3c1aa09640 1 -- 172.21.15.88:0/3316979845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c140ca0f0 con 0x7f3c14150280 2023-10-16T20:48:54.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.825+0000 7f3c1aa09640 1 -- 172.21.15.88:0/3316979845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c14074700 con 0x7f3c14150280 2023-10-16T20:48:54.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.825+0000 7f3c10ff9640 1 -- 172.21.15.88:0/3316979845 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3c080039a0 con 0x7f3c14150280 2023-10-16T20:48:54.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.825+0000 7f3c10ff9640 1 --2- 172.21.15.88:0/3316979845 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3c000664a0 0x7f3c00068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:54.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.825+0000 7f3c10ff9640 1 -- 172.21.15.88:0/3316979845 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7f3c08083890 con 0x7f3c14150280 2023-10-16T20:48:54.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.825+0000 7f3c13fff640 1 --2- 172.21.15.88:0/3316979845 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3c000664a0 0x7f3c00068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:54.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.825+0000 7f3c13fff640 1 --2- 172.21.15.88:0/3316979845 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3c000664a0 0x7f3c00068960 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3c04007a60 tx=0x7f3c04008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:54.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:54.829+0000 7f3c10ff9640 1 -- 172.21.15.88:0/3316979845 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3c08050840 con 0x7f3c14150280 2023-10-16T20:48:55.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:55.017+0000 7f3c1aa09640 1 -- 172.21.15.88:0/3316979845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c140c3850 con 0x7f3c14150280 2023-10-16T20:48:55.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:55.017+0000 7f3c10ff9640 1 -- 172.21.15.88:0/3316979845 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+5789 (secure 0 0 0) 0x7f3c08053e60 con 0x7f3c14150280 2023-10-16T20:48:55.020 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:48:55.020 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:54.848833+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:reconnect","state_seq":33,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:48:55.021 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 69 2023-10-16T20:48:55.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:55.017+0000 7f3c1aa09640 1 -- 172.21.15.88:0/3316979845 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3c000664a0 msgr2=0x7f3c00068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:55.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:55.017+0000 7f3c1aa09640 1 --2- 172.21.15.88:0/3316979845 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3c000664a0 0x7f3c00068960 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3c04007a60 tx=0x7f3c04008040 comp rx=0 tx=0).stop 2023-10-16T20:48:55.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:55.017+0000 7f3c1aa09640 1 -- 172.21.15.88:0/3316979845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c14150280 msgr2=0x7f3c1413e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:55.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:55.017+0000 7f3c1aa09640 1 --2- 172.21.15.88:0/3316979845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c14150280 0x7f3c1413e320 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3c0800c3f0 tx=0x7f3c0800c8b0 comp rx=0 tx=0).stop 2023-10-16T20:48:55.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:55.017+0000 7f3c1aa09640 1 -- 172.21.15.88:0/3316979845 shutdown_connections 2023-10-16T20:48:55.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:55.017+0000 7f3c1aa09640 1 --2- 172.21.15.88:0/3316979845 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3c000664a0 0x7f3c00068960 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:55.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:55.017+0000 7f3c1aa09640 1 --2- 172.21.15.88:0/3316979845 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c14150660 0x7f3c141f1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:55.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:55.017+0000 7f3c1aa09640 1 --2- 172.21.15.88:0/3316979845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c14150280 0x7f3c1413e320 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:55.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:55.017+0000 7f3c1aa09640 1 --2- 172.21.15.88:0/3316979845 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c140c6bc0 0x7f3c14142c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:55.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:55.017+0000 7f3c1aa09640 1 -- 172.21.15.88:0/3316979845 >> 172.21.15.88:0/3316979845 conn(0x7f3c140cad50 msgr2=0x7f3c140c1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:55.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:55.017+0000 7f3c1aa09640 1 -- 172.21.15.88:0/3316979845 shutdown_connections 2023-10-16T20:48:55.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:55.017+0000 7f3c1aa09640 1 -- 172.21.15.88:0/3316979845 wait complete. 2023-10-16T20:48:55.040 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:48:57.042 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:48:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.197+0000 7f1bb131d640 1 Processor -- start 2023-10-16T20:48:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.197+0000 7f1bb131d640 1 -- start start 2023-10-16T20:48:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.197+0000 7f1bb131d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1bac151440 0x7f1bac0ae950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.197+0000 7f1bb131d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bac071380 con 0x7f1bac069e90 2023-10-16T20:48:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.197+0000 7f1bb131d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bac151820 con 0x7f1bac071810 2023-10-16T20:48:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.197+0000 7f1bb131d640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bac06b580 con 0x7f1bac151440 2023-10-16T20:48:57.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.197+0000 7f1baa7fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1bac069e90 0x7f1bac157bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52308/0 (socket says 172.21.15.88:52308) 2023-10-16T20:48:57.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.197+0000 7f1baa7fc640 1 -- 172.21.15.88:0/3431904773 learned_addr learned my addr 172.21.15.88:0/3431904773 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:48:57.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.197+0000 7f1baaffd640 1 --2- 172.21.15.88:0/3431904773 >> v2:172.21.15.191:3300/0 conn(0x7f1bac151440 0x7f1bac0ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:57.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.197+0000 7f1ba9ffb640 1 -- 172.21.15.88:0/3431904773 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035259702 0 0) 0x7f1bac071380 con 0x7f1bac069e90 2023-10-16T20:48:57.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.197+0000 7f1ba9ffb640 1 -- 172.21.15.88:0/3431904773 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b94003530 con 0x7f1bac069e90 2023-10-16T20:48:57.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.197+0000 7f1baaffd640 1 -- 172.21.15.88:0/3431904773 >> v1:172.21.15.191:6789/0 conn(0x7f1bac071810 legacy=0x7f1bac0cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:57.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.197+0000 7f1ba9ffb640 1 -- 172.21.15.88:0/3431904773 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196073943 0 0) 0x7f1bac151820 con 0x7f1bac071810 2023-10-16T20:48:57.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.197+0000 7f1baaffd640 1 -- 172.21.15.88:0/3431904773 >> v1:172.21.15.88:6789/0 conn(0x7f1bac069e90 legacy=0x7f1bac157bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.197+0000 7f1baaffd640 1 -- 172.21.15.88:0/3431904773 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bac0c6bb0 con 0x7f1bac151440 2023-10-16T20:48:57.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.201+0000 7f1baaffd640 1 --2- 172.21.15.88:0/3431904773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bac151440 0x7f1bac0ae950 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c0097c0 tx=0x7f1b9c01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1f6e86eea5312e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.201+0000 7f1ba9ffb640 1 -- 172.21.15.88:0/3431904773 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c0ab600 con 0x7f1bac151440 2023-10-16T20:48:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.201+0000 7f1ba9ffb640 1 -- 172.21.15.88:0/3431904773 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9c0a7050 con 0x7f1bac151440 2023-10-16T20:48:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.201+0000 7f1ba9ffb640 1 -- 172.21.15.88:0/3431904773 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c0ab980 con 0x7f1bac151440 2023-10-16T20:48:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.201+0000 7f1bb131d640 1 -- 172.21.15.88:0/3431904773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bac151440 msgr2=0x7f1bac0ae950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.201+0000 7f1bb131d640 1 --2- 172.21.15.88:0/3431904773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bac151440 0x7f1bac0ae950 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c0097c0 tx=0x7f1b9c01c480 comp rx=0 tx=0).stop 2023-10-16T20:48:57.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1bb131d640 1 -- 172.21.15.88:0/3431904773 shutdown_connections 2023-10-16T20:48:57.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1bb131d640 1 --2- 172.21.15.88:0/3431904773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bac151440 0x7f1bac0ae950 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:57.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1bb131d640 1 -- 172.21.15.88:0/3431904773 >> 172.21.15.88:0/3431904773 conn(0x7f1bac06e9d0 msgr2=0x7f1bac06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:57.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1bb131d640 1 -- 172.21.15.88:0/3431904773 shutdown_connections 2023-10-16T20:48:57.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1bb131d640 1 -- 172.21.15.88:0/3431904773 wait complete. 2023-10-16T20:48:57.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1bb131d640 1 Processor -- start 2023-10-16T20:48:57.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1bb131d640 1 -- start start 2023-10-16T20:48:57.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1bb131d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bac069e90 0x7f1bac0c2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:57.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1bb131d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac071810 0x7f1bac153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:57.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1bb131d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bac151440 0x7f1bac144ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:57.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1bb131d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac145f20 con 0x7f1bac071810 2023-10-16T20:48:57.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1bb131d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac0c2cc0 con 0x7f1bac151440 2023-10-16T20:48:57.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1bb131d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac1f26b0 con 0x7f1bac069e90 2023-10-16T20:48:57.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1baa7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac071810 0x7f1bac153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:57.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1baa7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac071810 0x7f1bac153760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34770/0 (socket says 172.21.15.88:34770) 2023-10-16T20:48:57.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1baa7fc640 1 -- 172.21.15.88:0/3842523168 learned_addr learned my addr 172.21.15.88:0/3842523168 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:57.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1babfff640 1 --2- 172.21.15.88:0/3842523168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bac151440 0x7f1bac144ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:57.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1baaffd640 1 --2- 172.21.15.88:0/3842523168 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bac069e90 0x7f1bac0c2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:57.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1baa7fc640 1 -- 172.21.15.88:0/3842523168 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bac069e90 msgr2=0x7f1bac0c2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:57.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1baa7fc640 1 --2- 172.21.15.88:0/3842523168 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bac069e90 0x7f1bac0c2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:57.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1baa7fc640 1 -- 172.21.15.88:0/3842523168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bac151440 msgr2=0x7f1bac144ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:57.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.205+0000 7f1baa7fc640 1 --2- 172.21.15.88:0/3842523168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bac151440 0x7f1bac144ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:57.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.209+0000 7f1baa7fc640 1 -- 172.21.15.88:0/3842523168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c009420 con 0x7f1bac071810 2023-10-16T20:48:57.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.209+0000 7f1baa7fc640 1 --2- 172.21.15.88:0/3842523168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac071810 0x7f1bac153760 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000c420 tx=0x7f1ba000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:57.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.209+0000 7f1b937fe640 1 -- 172.21.15.88:0/3842523168 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0005cf0 con 0x7f1bac071810 2023-10-16T20:48:57.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.209+0000 7f1b937fe640 1 -- 172.21.15.88:0/3842523168 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba0005e70 con 0x7f1bac071810 2023-10-16T20:48:57.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.209+0000 7f1bb131d640 1 -- 172.21.15.88:0/3842523168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bac146100 con 0x7f1bac071810 2023-10-16T20:48:57.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.209+0000 7f1b937fe640 1 -- 172.21.15.88:0/3842523168 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba00072f0 con 0x7f1bac071810 2023-10-16T20:48:57.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.209+0000 7f1bb131d640 1 -- 172.21.15.88:0/3842523168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bac1466c0 con 0x7f1bac071810 2023-10-16T20:48:57.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.209+0000 7f1bb131d640 1 -- 172.21.15.88:0/3842523168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bac074700 con 0x7f1bac071810 2023-10-16T20:48:57.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.209+0000 7f1b937fe640 1 -- 172.21.15.88:0/3842523168 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1ba00074b0 con 0x7f1bac071810 2023-10-16T20:48:57.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.209+0000 7f1b937fe640 1 --2- 172.21.15.88:0/3842523168 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1b980664a0 0x7f1b98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:57.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.209+0000 7f1b937fe640 1 -- 172.21.15.88:0/3842523168 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7f1ba0083ac0 con 0x7f1bac071810 2023-10-16T20:48:57.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.209+0000 7f1baaffd640 1 --2- 172.21.15.88:0/3842523168 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1b980664a0 0x7f1b98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:57.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.213+0000 7f1baaffd640 1 --2- 172.21.15.88:0/3842523168 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1b980664a0 0x7f1b98068960 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c0ad000 tx=0x7f1b9c0a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:57.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.217+0000 7f1b937fe640 1 -- 172.21.15.88:0/3842523168 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1ba0054490 con 0x7f1bac071810 2023-10-16T20:48:57.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.441+0000 7f1bb131d640 1 -- 172.21.15.88:0/3842523168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bac154b50 con 0x7f1bac071810 2023-10-16T20:48:57.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.441+0000 7f1b937fe640 1 -- 172.21.15.88:0/3842523168 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+5789 (secure 0 0 0) 0x7f1ba0083e60 con 0x7f1bac071810 2023-10-16T20:48:57.446 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:48:57.446 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:54.848833+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:reconnect","state_seq":33,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:48:57.447 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 69 2023-10-16T20:48:57.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.445+0000 7f1bb131d640 1 -- 172.21.15.88:0/3842523168 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1b980664a0 msgr2=0x7f1b98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:57.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.445+0000 7f1bb131d640 1 --2- 172.21.15.88:0/3842523168 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1b980664a0 0x7f1b98068960 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c0ad000 tx=0x7f1b9c0a6000 comp rx=0 tx=0).stop 2023-10-16T20:48:57.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.445+0000 7f1bb131d640 1 -- 172.21.15.88:0/3842523168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac071810 msgr2=0x7f1bac153760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:57.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.445+0000 7f1bb131d640 1 --2- 172.21.15.88:0/3842523168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac071810 0x7f1bac153760 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000c420 tx=0x7f1ba000c8e0 comp rx=0 tx=0).stop 2023-10-16T20:48:57.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.445+0000 7f1bb131d640 1 -- 172.21.15.88:0/3842523168 shutdown_connections 2023-10-16T20:48:57.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.445+0000 7f1bb131d640 1 --2- 172.21.15.88:0/3842523168 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1b980664a0 0x7f1b98068960 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:57.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.445+0000 7f1bb131d640 1 --2- 172.21.15.88:0/3842523168 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1bac151440 0x7f1bac144ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:57.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.445+0000 7f1bb131d640 1 --2- 172.21.15.88:0/3842523168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bac071810 0x7f1bac153760 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:57.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.445+0000 7f1bb131d640 1 --2- 172.21.15.88:0/3842523168 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1bac069e90 0x7f1bac0c2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:57.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.445+0000 7f1bb131d640 1 -- 172.21.15.88:0/3842523168 >> 172.21.15.88:0/3842523168 conn(0x7f1bac06e9d0 msgr2=0x7f1bac14f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:57.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.445+0000 7f1bb131d640 1 -- 172.21.15.88:0/3842523168 shutdown_connections 2023-10-16T20:48:57.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:57.445+0000 7f1bb131d640 1 -- 172.21.15.88:0/3842523168 wait complete. 2023-10-16T20:48:57.471 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 69, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 69, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4288": { "addr": "172.21.15.191:6835/3059699574", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3059699574, "type": "v2" }, { "addr": "172.21.15.191:6835", "nonce": 3059699574, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4288, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4889": { "addr": "172.21.15.191:6837/2468647900", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6836", "nonce": 2468647900, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2468647900, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4889, "incarnation": 30, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 21 }, "gid_5284": { "addr": "172.21.15.88:6839/2172973775", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6837", "nonce": 2172973775, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2172973775, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5284, "incarnation": 38, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 17 }, "gid_5535": { "addr": "172.21.15.88:6835/3476086754", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3476086754, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3476086754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5535, "incarnation": 52, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_5709": { "addr": "172.21.15.88:6838/2600891060", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 2600891060, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 2600891060, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5709, "incarnation": 67, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:reconnect", "state_seq": 33 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:48:54.848833+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288, "mds_1": 5709, "mds_2": 5535, "mds_3": 5284, "mds_4": 4889 } } } ], "standbys": [] } 2023-10-16T20:48:58.923 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:48:58.927 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:48:59.473 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:48:59.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.617+0000 7f13feb62640 1 Processor -- start 2023-10-16T20:48:59.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13feb62640 1 -- start start 2023-10-16T20:48:59.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13feb62640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f13f81537c0 0x7f13f815c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:59.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13feb62640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f13f815cdb0 0x7f13f80c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:59.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13feb62640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13f80ca2f0 0x7f13f80c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:59.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13feb62640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f80cae10 con 0x7f13f81537c0 2023-10-16T20:48:59.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13feb62640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f80cac30 con 0x7f13f815cdb0 2023-10-16T20:48:59.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13feb62640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f815d2b0 con 0x7f13f80ca2f0 2023-10-16T20:48:59.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13fdb60640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f13f81537c0 0x7f13f815c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:59.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13fdb60640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f13f81537c0 0x7f13f815c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37794/0 (socket says 172.21.15.88:37794) 2023-10-16T20:48:59.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13fdb60640 1 -- 172.21.15.88:0/509069361 learned_addr learned my addr 172.21.15.88:0/509069361 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:59.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13fdb60640 1 -- 172.21.15.88:0/509069361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13f80ca2f0 msgr2=0x7f13f80c51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:59.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13fe361640 1 --2- 172.21.15.88:0/509069361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13f80ca2f0 0x7f13f80c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:59.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13fd35f640 1 --2- 172.21.15.88:0/509069361 >> v2:172.21.15.191:3300/0 conn(0x7f13f815cdb0 0x7f13f80c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:59.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13fdb60640 1 --2- 172.21.15.88:0/509069361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13f80ca2f0 0x7f13f80c51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:59.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13fdb60640 1 -- 172.21.15.88:0/509069361 >> v2:172.21.15.191:3300/0 conn(0x7f13f815cdb0 msgr2=0x7f13f80c4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:48:59.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13fdb60640 1 --2- 172.21.15.88:0/509069361 >> v2:172.21.15.191:3300/0 conn(0x7f13f815cdb0 0x7f13f80c4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:59.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13fdb60640 1 -- 172.21.15.88:0/509069361 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f815d590 con 0x7f13f81537c0 2023-10-16T20:48:59.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13fd35f640 1 --2- 172.21.15.88:0/509069361 >> v2:172.21.15.191:3300/0 conn(0x7f13f815cdb0 0x7f13f80c4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:48:59.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.621+0000 7f13fe361640 1 --2- 172.21.15.88:0/509069361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13f80ca2f0 0x7f13f80c51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:48:59.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.625+0000 7f13fdb60640 1 --2- 172.21.15.88:0/509069361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13f81537c0 0x7f13f815c870 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f13f80cfb20 tx=0x7f13f401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70e4b5b9ab55a17 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:59.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.625+0000 7f13fcb5e640 1 -- 172.21.15.88:0/509069361 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f40aa5f0 con 0x7f13f81537c0 2023-10-16T20:48:59.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.625+0000 7f13fcb5e640 1 -- 172.21.15.88:0/509069361 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f40aa750 con 0x7f13f81537c0 2023-10-16T20:48:59.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.625+0000 7f13fcb5e640 1 -- 172.21.15.88:0/509069361 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f40aaae0 con 0x7f13f81537c0 2023-10-16T20:48:59.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.625+0000 7f13feb62640 1 -- 172.21.15.88:0/509069361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13f81537c0 msgr2=0x7f13f815c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:59.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.625+0000 7f13feb62640 1 --2- 172.21.15.88:0/509069361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13f81537c0 0x7f13f815c870 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f13f80cfb20 tx=0x7f13f401c4d0 comp rx=0 tx=0).stop 2023-10-16T20:48:59.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.625+0000 7f13feb62640 1 -- 172.21.15.88:0/509069361 shutdown_connections 2023-10-16T20:48:59.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.625+0000 7f13feb62640 1 --2- 172.21.15.88:0/509069361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13f80ca2f0 0x7f13f80c51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:59.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.625+0000 7f13feb62640 1 --2- 172.21.15.88:0/509069361 >> v2:172.21.15.191:3300/0 conn(0x7f13f815cdb0 0x7f13f80c4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:59.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.625+0000 7f13feb62640 1 --2- 172.21.15.88:0/509069361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13f81537c0 0x7f13f815c870 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:59.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.625+0000 7f13feb62640 1 -- 172.21.15.88:0/509069361 >> 172.21.15.88:0/509069361 conn(0x7f13f806c170 msgr2=0x7f13f80715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:59.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.625+0000 7f13feb62640 1 -- 172.21.15.88:0/509069361 shutdown_connections 2023-10-16T20:48:59.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.625+0000 7f13feb62640 1 -- 172.21.15.88:0/509069361 wait complete. 2023-10-16T20:48:59.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.625+0000 7f13feb62640 1 Processor -- start 2023-10-16T20:48:59.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.625+0000 7f13feb62640 1 -- start start 2023-10-16T20:48:59.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13feb62640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13f80ca2f0 0x7f13f81f5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:59.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13feb62640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13f81537c0 0x7f13f80cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:59.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13feb62640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13f815cdb0 0x7f13f815b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:59.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13feb62640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f81f5690 con 0x7f13f81537c0 2023-10-16T20:48:59.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13feb62640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f81f6060 con 0x7f13f815cdb0 2023-10-16T20:48:59.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13feb62640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f814b5a0 con 0x7f13f80ca2f0 2023-10-16T20:48:59.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13fd35f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13f81537c0 0x7f13f80cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:59.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13fd35f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13f81537c0 0x7f13f80cb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37798/0 (socket says 172.21.15.88:37798) 2023-10-16T20:48:59.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13fd35f640 1 -- 172.21.15.88:0/3731417730 learned_addr learned my addr 172.21.15.88:0/3731417730 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:48:59.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13fd35f640 1 -- 172.21.15.88:0/3731417730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13f80ca2f0 msgr2=0x7f13f81f5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:59.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13fd35f640 1 --2- 172.21.15.88:0/3731417730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13f80ca2f0 0x7f13f81f5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:59.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13fd35f640 1 -- 172.21.15.88:0/3731417730 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13f815cdb0 msgr2=0x7f13f815b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:59.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13fe361640 1 --2- 172.21.15.88:0/3731417730 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13f815cdb0 0x7f13f815b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:59.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13fd35f640 1 --2- 172.21.15.88:0/3731417730 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13f815cdb0 0x7f13f815b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:59.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13fd35f640 1 -- 172.21.15.88:0/3731417730 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f40094d0 con 0x7f13f81537c0 2023-10-16T20:48:59.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13fe361640 1 --2- 172.21.15.88:0/3731417730 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13f815cdb0 0x7f13f815b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:48:59.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13fd35f640 1 --2- 172.21.15.88:0/3731417730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13f81537c0 0x7f13f80cb950 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f13e800c7f0 tx=0x7f13e800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:59.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13ee7fc640 1 -- 172.21.15.88:0/3731417730 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e8004220 con 0x7f13f81537c0 2023-10-16T20:48:59.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13ee7fc640 1 -- 172.21.15.88:0/3731417730 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e8004380 con 0x7f13f81537c0 2023-10-16T20:48:59.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13ee7fc640 1 -- 172.21.15.88:0/3731417730 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e8003840 con 0x7f13f81537c0 2023-10-16T20:48:59.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13feb62640 1 -- 172.21.15.88:0/3731417730 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f814abd0 con 0x7f13f81537c0 2023-10-16T20:48:59.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13feb62640 1 -- 172.21.15.88:0/3731417730 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f81f5320 con 0x7f13f81537c0 2023-10-16T20:48:59.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13feb62640 1 -- 172.21.15.88:0/3731417730 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f81f5180 con 0x7f13f81537c0 2023-10-16T20:48:59.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.629+0000 7f13ee7fc640 1 -- 172.21.15.88:0/3731417730 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f13e8003bb0 con 0x7f13f81537c0 2023-10-16T20:48:59.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.633+0000 7f13ee7fc640 1 --2- 172.21.15.88:0/3731417730 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f13dc0664d0 0x7f13dc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:48:59.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.633+0000 7f13ee7fc640 1 -- 172.21.15.88:0/3731417730 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7f13e80843d0 con 0x7f13f81537c0 2023-10-16T20:48:59.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.633+0000 7f13fdb60640 1 --2- 172.21.15.88:0/3731417730 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f13dc0664d0 0x7f13dc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:48:59.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.633+0000 7f13fdb60640 1 --2- 172.21.15.88:0/3731417730 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f13dc0664d0 0x7f13dc068990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f13f40ae000 tx=0x7f13f4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:48:59.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.637+0000 7f13ee7fc640 1 -- 172.21.15.88:0/3731417730 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f13f81f5180 con 0x7f13f81537c0 2023-10-16T20:48:59.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.821+0000 7f13feb62640 1 -- 172.21.15.88:0/3731417730 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13f81f5180 con 0x7f13f81537c0 2023-10-16T20:48:59.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.825+0000 7f13ee7fc640 1 -- 172.21.15.88:0/3731417730 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5787 (secure 0 0 0) 0x7f13f81f5180 con 0x7f13f81537c0 2023-10-16T20:48:59.827 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:48:59.828 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:56.518144+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:reconnect","state_seq":33,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:48:59.828 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 70 2023-10-16T20:48:59.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.825+0000 7f13feb62640 1 -- 172.21.15.88:0/3731417730 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f13dc0664d0 msgr2=0x7f13dc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:59.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.825+0000 7f13feb62640 1 --2- 172.21.15.88:0/3731417730 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f13dc0664d0 0x7f13dc068990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f13f40ae000 tx=0x7f13f4008040 comp rx=0 tx=0).stop 2023-10-16T20:48:59.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.825+0000 7f13feb62640 1 -- 172.21.15.88:0/3731417730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13f81537c0 msgr2=0x7f13f80cb950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:48:59.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.825+0000 7f13feb62640 1 --2- 172.21.15.88:0/3731417730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13f81537c0 0x7f13f80cb950 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f13e800c7f0 tx=0x7f13e800ccb0 comp rx=0 tx=0).stop 2023-10-16T20:48:59.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.825+0000 7f13feb62640 1 -- 172.21.15.88:0/3731417730 shutdown_connections 2023-10-16T20:48:59.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.825+0000 7f13feb62640 1 --2- 172.21.15.88:0/3731417730 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f13dc0664d0 0x7f13dc068990 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:59.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.825+0000 7f13feb62640 1 --2- 172.21.15.88:0/3731417730 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13f815cdb0 0x7f13f815b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:59.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.825+0000 7f13feb62640 1 --2- 172.21.15.88:0/3731417730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13f81537c0 0x7f13f80cb950 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:59.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.825+0000 7f13feb62640 1 --2- 172.21.15.88:0/3731417730 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f13f80ca2f0 0x7f13f81f5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:48:59.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.825+0000 7f13feb62640 1 -- 172.21.15.88:0/3731417730 >> 172.21.15.88:0/3731417730 conn(0x7f13f806c170 msgr2=0x7f13f80cafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:48:59.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.825+0000 7f13feb62640 1 -- 172.21.15.88:0/3731417730 shutdown_connections 2023-10-16T20:48:59.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:48:59.825+0000 7f13feb62640 1 -- 172.21.15.88:0/3731417730 wait complete. 2023-10-16T20:48:59.850 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:49:01.851 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:49:01.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.989+0000 7f48e19d9640 1 Processor -- start 2023-10-16T20:49:01.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.993+0000 7f48e19d9640 1 -- start start 2023-10-16T20:49:01.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.993+0000 7f48e19d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48dc1406d0 0x7f48dc0c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:01.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.993+0000 7f48e19d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f48dc06a970 0x7f48dc071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:01.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.993+0000 7f48e19d9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48dc15b3c0 con 0x7f48dc071e60 2023-10-16T20:49:01.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.993+0000 7f48e19d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48dc15b680 con 0x7f48dc06a970 2023-10-16T20:49:01.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.993+0000 7f48e19d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc157570 con 0x7f48dc1406d0 2023-10-16T20:49:01.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.993+0000 7f48d27fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f48dc06a970 0x7f48dc071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:01.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.993+0000 7f48d27fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f48dc06a970 0x7f48dc071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37820/0 (socket says 172.21.15.88:37820) 2023-10-16T20:49:01.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.993+0000 7f48d27fc640 1 -- 172.21.15.88:0/2562964534 learned_addr learned my addr 172.21.15.88:0/2562964534 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:49:01.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.993+0000 7f48db7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f48dc071e60 0x7f48dc15a810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54034/0 (socket says 172.21.15.88:54034) 2023-10-16T20:49:01.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.993+0000 7f48daffd640 1 --2- 172.21.15.88:0/2562964534 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48dc1406d0 0x7f48dc0c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:01.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.993+0000 7f48d27fc640 1 -- 172.21.15.88:0/2562964534 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48dc1406d0 msgr2=0x7f48dc0c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:49:01.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.993+0000 7f48d27fc640 1 --2- 172.21.15.88:0/2562964534 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48dc1406d0 0x7f48dc0c1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:01.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.993+0000 7f48d27fc640 1 -- 172.21.15.88:0/2562964534 >> v1:172.21.15.88:6789/0 conn(0x7f48dc071e60 legacy=0x7f48dc15a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:01.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.993+0000 7f48d27fc640 1 -- 172.21.15.88:0/2562964534 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48dc157850 con 0x7f48dc06a970 2023-10-16T20:49:01.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.997+0000 7f48d27fc640 1 --2- 172.21.15.88:0/2562964534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48dc06a970 0x7f48dc071920 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f48c4009e50 tx=0x7f48c401c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd97b6f5f080ea4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:49:01.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.997+0000 7f48da7fc640 1 -- 172.21.15.88:0/2562964534 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c40a57c0 con 0x7f48dc06a970 2023-10-16T20:49:01.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.997+0000 7f48da7fc640 1 -- 172.21.15.88:0/2562964534 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c40a5920 con 0x7f48dc06a970 2023-10-16T20:49:01.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.997+0000 7f48da7fc640 1 -- 172.21.15.88:0/2562964534 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c40a5cb0 con 0x7f48dc06a970 2023-10-16T20:49:01.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.997+0000 7f48e19d9640 1 -- 172.21.15.88:0/2562964534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48dc06a970 msgr2=0x7f48dc071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:01.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.997+0000 7f48e19d9640 1 --2- 172.21.15.88:0/2562964534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48dc06a970 0x7f48dc071920 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f48c4009e50 tx=0x7f48c401c9e0 comp rx=0 tx=0).stop 2023-10-16T20:49:01.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.997+0000 7f48e19d9640 1 -- 172.21.15.88:0/2562964534 shutdown_connections 2023-10-16T20:49:01.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.997+0000 7f48e19d9640 1 --2- 172.21.15.88:0/2562964534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48dc06a970 0x7f48dc071920 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:01.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.997+0000 7f48e19d9640 1 --2- 172.21.15.88:0/2562964534 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48dc1406d0 0x7f48dc0c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:01.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.997+0000 7f48e19d9640 1 -- 172.21.15.88:0/2562964534 >> 172.21.15.88:0/2562964534 conn(0x7f48dc0cad50 msgr2=0x7f48dc0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:49:01.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.997+0000 7f48e19d9640 1 -- 172.21.15.88:0/2562964534 shutdown_connections 2023-10-16T20:49:01.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.997+0000 7f48e19d9640 1 -- 172.21.15.88:0/2562964534 wait complete. 2023-10-16T20:49:02.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.997+0000 7f48e19d9640 1 Processor -- start 2023-10-16T20:49:02.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:01.997+0000 7f48e19d9640 1 -- start start 2023-10-16T20:49:02.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48e19d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48dc06a970 0x7f48dc070440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:02.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48e19d9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48dc071e60 0x7f48dc160f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:02.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48e19d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48dc1406d0 0x7f48dc151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:02.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48e19d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc070980 con 0x7f48dc1406d0 2023-10-16T20:49:02.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48e19d9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc070b80 con 0x7f48dc071e60 2023-10-16T20:49:02.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48e19d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48dc070e60 con 0x7f48dc06a970 2023-10-16T20:49:02.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48db7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48dc1406d0 0x7f48dc151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:02.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48db7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48dc1406d0 0x7f48dc151f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37828/0 (socket says 172.21.15.88:37828) 2023-10-16T20:49:02.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48db7fe640 1 -- 172.21.15.88:0/2226672714 learned_addr learned my addr 172.21.15.88:0/2226672714 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:49:02.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48daffd640 1 --2- 172.21.15.88:0/2226672714 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48dc06a970 0x7f48dc070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:02.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48d27fc640 1 --2- 172.21.15.88:0/2226672714 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48dc071e60 0x7f48dc160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:02.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48db7fe640 1 -- 172.21.15.88:0/2226672714 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48dc06a970 msgr2=0x7f48dc070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:02.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48db7fe640 1 --2- 172.21.15.88:0/2226672714 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48dc06a970 0x7f48dc070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:02.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48db7fe640 1 -- 172.21.15.88:0/2226672714 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48dc071e60 msgr2=0x7f48dc160f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:02.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48db7fe640 1 --2- 172.21.15.88:0/2226672714 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48dc071e60 0x7f48dc160f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:02.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48db7fe640 1 -- 172.21.15.88:0/2226672714 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c4009b30 con 0x7f48dc1406d0 2023-10-16T20:49:02.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48d27fc640 1 --2- 172.21.15.88:0/2226672714 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48dc071e60 0x7f48dc160f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:49:02.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48db7fe640 1 --2- 172.21.15.88:0/2226672714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48dc1406d0 0x7f48dc151f70 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f48d400c410 tx=0x7f48d400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:49:02.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48d8ff9640 1 -- 172.21.15.88:0/2226672714 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d40074e0 con 0x7f48dc1406d0 2023-10-16T20:49:02.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48d8ff9640 1 -- 172.21.15.88:0/2226672714 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d4007640 con 0x7f48dc1406d0 2023-10-16T20:49:02.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48d8ff9640 1 -- 172.21.15.88:0/2226672714 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d4004a10 con 0x7f48dc1406d0 2023-10-16T20:49:02.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48e19d9640 1 -- 172.21.15.88:0/2226672714 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48dc1581d0 con 0x7f48dc1406d0 2023-10-16T20:49:02.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48e19d9640 1 -- 172.21.15.88:0/2226672714 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48dc071420 con 0x7f48dc1406d0 2023-10-16T20:49:02.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48e19d9640 1 -- 172.21.15.88:0/2226672714 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48dc151c90 con 0x7f48dc1406d0 2023-10-16T20:49:02.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.001+0000 7f48d8ff9640 1 -- 172.21.15.88:0/2226672714 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f48d4004b70 con 0x7f48dc1406d0 2023-10-16T20:49:02.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.005+0000 7f48d8ff9640 1 --2- 172.21.15.88:0/2226672714 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f48bc0664a0 0x7f48bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:02.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.005+0000 7f48d8ff9640 1 -- 172.21.15.88:0/2226672714 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7f48d4083dc0 con 0x7f48dc1406d0 2023-10-16T20:49:02.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.005+0000 7f48daffd640 1 --2- 172.21.15.88:0/2226672714 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f48bc0664a0 0x7f48bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:02.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.005+0000 7f48daffd640 1 --2- 172.21.15.88:0/2226672714 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f48bc0664a0 0x7f48bc068960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00b9d0 tx=0x7f48cc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:49:02.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.009+0000 7f48d8ff9640 1 -- 172.21.15.88:0/2226672714 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f48dc151c90 con 0x7f48dc1406d0 2023-10-16T20:49:02.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.189+0000 7f48e19d9640 1 -- 172.21.15.88:0/2226672714 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48dc151c90 con 0x7f48dc1406d0 2023-10-16T20:49:02.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.189+0000 7f48d8ff9640 1 -- 172.21.15.88:0/2226672714 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+5784 (secure 0 0 0) 0x7f48dc151c90 con 0x7f48dc1406d0 2023-10-16T20:49:02.191 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:49:02.191 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:59.902582+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:rejoin","state_seq":35,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:49:02.191 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 71 2023-10-16T20:49:02.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.193+0000 7f48e19d9640 1 -- 172.21.15.88:0/2226672714 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f48bc0664a0 msgr2=0x7f48bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:02.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.193+0000 7f48e19d9640 1 --2- 172.21.15.88:0/2226672714 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f48bc0664a0 0x7f48bc068960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00b9d0 tx=0x7f48cc008040 comp rx=0 tx=0).stop 2023-10-16T20:49:02.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.193+0000 7f48e19d9640 1 -- 172.21.15.88:0/2226672714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48dc1406d0 msgr2=0x7f48dc151f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:02.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.193+0000 7f48e19d9640 1 --2- 172.21.15.88:0/2226672714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48dc1406d0 0x7f48dc151f70 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f48d400c410 tx=0x7f48d400c8d0 comp rx=0 tx=0).stop 2023-10-16T20:49:02.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.193+0000 7f48e19d9640 1 -- 172.21.15.88:0/2226672714 shutdown_connections 2023-10-16T20:49:02.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.193+0000 7f48e19d9640 1 --2- 172.21.15.88:0/2226672714 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f48bc0664a0 0x7f48bc068960 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:02.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.193+0000 7f48e19d9640 1 --2- 172.21.15.88:0/2226672714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48dc1406d0 0x7f48dc151f70 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:02.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.193+0000 7f48e19d9640 1 --2- 172.21.15.88:0/2226672714 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48dc071e60 0x7f48dc160f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:02.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.193+0000 7f48e19d9640 1 --2- 172.21.15.88:0/2226672714 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48dc06a970 0x7f48dc070440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:02.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.193+0000 7f48e19d9640 1 -- 172.21.15.88:0/2226672714 >> 172.21.15.88:0/2226672714 conn(0x7f48dc0cad50 msgr2=0x7f48dc0ca060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:49:02.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.193+0000 7f48e19d9640 1 -- 172.21.15.88:0/2226672714 shutdown_connections 2023-10-16T20:49:02.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:02.193+0000 7f48e19d9640 1 -- 172.21.15.88:0/2226672714 wait complete. 2023-10-16T20:49:02.211 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:49:04.212 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:49:04.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.357+0000 7ff499577640 1 Processor -- start 2023-10-16T20:49:04.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.357+0000 7ff499577640 1 -- start start 2023-10-16T20:49:04.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.357+0000 7ff499577640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff4940cb580 0x7ff4940cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:04.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.357+0000 7ff499577640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4940b1a60 0x7ff494163b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:04.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.357+0000 7ff499577640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4940c9fc0 con 0x7ff4940c9af0 2023-10-16T20:49:04.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.357+0000 7ff499577640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4940b20c0 con 0x7ff4940cb580 2023-10-16T20:49:04.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.357+0000 7ff499577640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4940cbea0 con 0x7ff4940b1a60 2023-10-16T20:49:04.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.357+0000 7ff493fff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff4940cb580 0x7ff4940cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:04.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.357+0000 7ff498d76640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4940b1a60 0x7ff494163b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:04.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.357+0000 7ff493fff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff4940cb580 0x7ff4940cb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:41972/0 (socket says 172.21.15.88:41972) 2023-10-16T20:49:04.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.357+0000 7ff493fff640 1 -- 172.21.15.88:0/3977613627 learned_addr learned my addr 172.21.15.88:0/3977613627 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:49:04.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.357+0000 7ff493fff640 1 -- 172.21.15.88:0/3977613627 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4940b1a60 msgr2=0x7ff494163b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:49:04.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.357+0000 7ff493fff640 1 --2- 172.21.15.88:0/3977613627 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4940b1a60 0x7ff494163b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:04.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.357+0000 7ff493fff640 1 -- 172.21.15.88:0/3977613627 >> v1:172.21.15.191:6789/0 conn(0x7ff4940c9af0 legacy=0x7ff494159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:04.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.357+0000 7ff493fff640 1 -- 172.21.15.88:0/3977613627 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4940c47d0 con 0x7ff4940cb580 2023-10-16T20:49:04.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.357+0000 7ff498d76640 1 --2- 172.21.15.88:0/3977613627 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4940b1a60 0x7ff494163b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:49:04.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.357+0000 7ff492ffd640 1 -- 172.21.15.88:0/3977613627 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393653754 0 0) 0x7ff4940c9fc0 con 0x7ff4940c9af0 2023-10-16T20:49:04.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.361+0000 7ff493fff640 1 --2- 172.21.15.88:0/3977613627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4940cb580 0x7ff4940cb960 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff484009870 tx=0x7ff48401c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7a5aa25bbf7187e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:49:04.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.361+0000 7ff492ffd640 1 -- 172.21.15.88:0/3977613627 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4840ab680 con 0x7ff4940cb580 2023-10-16T20:49:04.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.361+0000 7ff492ffd640 1 -- 172.21.15.88:0/3977613627 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4840ab7e0 con 0x7ff4940cb580 2023-10-16T20:49:04.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.361+0000 7ff492ffd640 1 -- 172.21.15.88:0/3977613627 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4840abb70 con 0x7ff4940cb580 2023-10-16T20:49:04.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.361+0000 7ff499577640 1 -- 172.21.15.88:0/3977613627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4940cb580 msgr2=0x7ff4940cb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:04.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.361+0000 7ff499577640 1 --2- 172.21.15.88:0/3977613627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4940cb580 0x7ff4940cb960 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff484009870 tx=0x7ff48401c480 comp rx=0 tx=0).stop 2023-10-16T20:49:04.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.361+0000 7ff499577640 1 -- 172.21.15.88:0/3977613627 shutdown_connections 2023-10-16T20:49:04.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.361+0000 7ff499577640 1 --2- 172.21.15.88:0/3977613627 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4940b1a60 0x7ff494163b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:04.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.361+0000 7ff499577640 1 --2- 172.21.15.88:0/3977613627 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4940cb580 0x7ff4940cb960 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:04.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.361+0000 7ff499577640 1 -- 172.21.15.88:0/3977613627 >> 172.21.15.88:0/3977613627 conn(0x7ff49406bb40 msgr2=0x7ff49406bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:49:04.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.361+0000 7ff499577640 1 -- 172.21.15.88:0/3977613627 shutdown_connections 2023-10-16T20:49:04.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.361+0000 7ff499577640 1 -- 172.21.15.88:0/3977613627 wait complete. 2023-10-16T20:49:04.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff499577640 1 Processor -- start 2023-10-16T20:49:04.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff499577640 1 -- start start 2023-10-16T20:49:04.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff499577640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4940b1a60 0x7ff4941f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:04.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff499577640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4940c9af0 0x7ff4940cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:04.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff499577640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4940cb580 0x7ff4940cf230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:04.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff499577640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4940cca70 con 0x7ff4940b1a60 2023-10-16T20:49:04.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff499577640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4941f23e0 con 0x7ff4940c9af0 2023-10-16T20:49:04.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff499577640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4940cd9e0 con 0x7ff4940cb580 2023-10-16T20:49:04.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff493fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4940b1a60 0x7ff4941f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff493fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4940b1a60 0x7ff4941f1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37834/0 (socket says 172.21.15.88:37834) 2023-10-16T20:49:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff493fff640 1 -- 172.21.15.88:0/3999924269 learned_addr learned my addr 172.21.15.88:0/3999924269 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:49:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff498d76640 1 --2- 172.21.15.88:0/3999924269 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4940cb580 0x7ff4940cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff493fff640 1 -- 172.21.15.88:0/3999924269 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4940cb580 msgr2=0x7ff4940cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff4937fe640 1 --2- 172.21.15.88:0/3999924269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4940c9af0 0x7ff4940cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:04.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff493fff640 1 --2- 172.21.15.88:0/3999924269 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4940cb580 0x7ff4940cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:04.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff493fff640 1 -- 172.21.15.88:0/3999924269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4940c9af0 msgr2=0x7ff4940cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:04.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff493fff640 1 --2- 172.21.15.88:0/3999924269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4940c9af0 0x7ff4940cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:04.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff493fff640 1 -- 172.21.15.88:0/3999924269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48c009a70 con 0x7ff4940b1a60 2023-10-16T20:49:04.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff498d76640 1 --2- 172.21.15.88:0/3999924269 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4940cb580 0x7ff4940cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:49:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff4937fe640 1 --2- 172.21.15.88:0/3999924269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4940c9af0 0x7ff4940cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:49:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff493fff640 1 --2- 172.21.15.88:0/3999924269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4940b1a60 0x7ff4941f1ea0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff4840ad040 tx=0x7ff48401acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:49:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff490ff9640 1 -- 172.21.15.88:0/3999924269 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4840b4800 con 0x7ff4940b1a60 2023-10-16T20:49:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff490ff9640 1 -- 172.21.15.88:0/3999924269 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4840b4980 con 0x7ff4940b1a60 2023-10-16T20:49:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff490ff9640 1 -- 172.21.15.88:0/3999924269 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4840b4d30 con 0x7ff4940b1a60 2023-10-16T20:49:04.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff499577640 1 -- 172.21.15.88:0/3999924269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4840094d0 con 0x7ff4940b1a60 2023-10-16T20:49:04.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff499577640 1 -- 172.21.15.88:0/3999924269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4941f2950 con 0x7ff4940b1a60 2023-10-16T20:49:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.365+0000 7ff499577640 1 -- 172.21.15.88:0/3999924269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4940ca2c0 con 0x7ff4940b1a60 2023-10-16T20:49:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.369+0000 7ff490ff9640 1 -- 172.21.15.88:0/3999924269 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff4840b3380 con 0x7ff4940b1a60 2023-10-16T20:49:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.369+0000 7ff490ff9640 1 --2- 172.21.15.88:0/3999924269 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff4800664a0 0x7ff480068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.369+0000 7ff490ff9640 1 -- 172.21.15.88:0/3999924269 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7ff48411eda0 con 0x7ff4940b1a60 2023-10-16T20:49:04.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.369+0000 7ff4937fe640 1 --2- 172.21.15.88:0/3999924269 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff4800664a0 0x7ff480068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:04.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.369+0000 7ff4937fe640 1 --2- 172.21.15.88:0/3999924269 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff4800664a0 0x7ff480068960 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff494148ff0 tx=0x7ff488009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:49:04.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.373+0000 7ff490ff9640 1 -- 172.21.15.88:0/3999924269 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff4840ebcc0 con 0x7ff4940b1a60 2023-10-16T20:49:04.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.561+0000 7ff499577640 1 -- 172.21.15.88:0/3999924269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff49404e9c0 con 0x7ff4940b1a60 2023-10-16T20:49:04.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.561+0000 7ff490ff9640 1 -- 172.21.15.88:0/3999924269 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+5784 (secure 0 0 0) 0x7ff4840c4090 con 0x7ff4940b1a60 2023-10-16T20:49:04.561 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:49:04.561 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:48:59.902582+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:rejoin","state_seq":35,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:49:04.561 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 71 2023-10-16T20:49:04.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.561+0000 7ff499577640 1 -- 172.21.15.88:0/3999924269 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff4800664a0 msgr2=0x7ff480068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:04.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.561+0000 7ff499577640 1 --2- 172.21.15.88:0/3999924269 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff4800664a0 0x7ff480068960 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff494148ff0 tx=0x7ff488009040 comp rx=0 tx=0).stop 2023-10-16T20:49:04.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.561+0000 7ff499577640 1 -- 172.21.15.88:0/3999924269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4940b1a60 msgr2=0x7ff4941f1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:04.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.561+0000 7ff499577640 1 --2- 172.21.15.88:0/3999924269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4940b1a60 0x7ff4941f1ea0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff4840ad040 tx=0x7ff48401acc0 comp rx=0 tx=0).stop 2023-10-16T20:49:04.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.561+0000 7ff499577640 1 -- 172.21.15.88:0/3999924269 shutdown_connections 2023-10-16T20:49:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.561+0000 7ff499577640 1 --2- 172.21.15.88:0/3999924269 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff4800664a0 0x7ff480068960 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.561+0000 7ff499577640 1 --2- 172.21.15.88:0/3999924269 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff4940cb580 0x7ff4940cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.561+0000 7ff499577640 1 --2- 172.21.15.88:0/3999924269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff4940c9af0 0x7ff4940cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.561+0000 7ff499577640 1 --2- 172.21.15.88:0/3999924269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4940b1a60 0x7ff4941f1ea0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:04.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.561+0000 7ff499577640 1 -- 172.21.15.88:0/3999924269 >> 172.21.15.88:0/3999924269 conn(0x7ff49406bb40 msgr2=0x7ff494150a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:49:04.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.561+0000 7ff499577640 1 -- 172.21.15.88:0/3999924269 shutdown_connections 2023-10-16T20:49:04.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:04.561+0000 7ff499577640 1 -- 172.21.15.88:0/3999924269 wait complete. 2023-10-16T20:49:04.581 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:49:06.582 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:49:06.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.733+0000 7f64b59d0640 1 Processor -- start 2023-10-16T20:49:06.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.733+0000 7f64b59d0640 1 -- start start 2023-10-16T20:49:06.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.737+0000 7f64b59d0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64b0153770 0x7f64b0159f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:06.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.737+0000 7f64b59d0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b0069e90 con 0x7f64b00c9040 2023-10-16T20:49:06.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.737+0000 7f64b59d0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b006aef0 con 0x7f64b00c6bf0 2023-10-16T20:49:06.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.737+0000 7f64b59d0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b006a0c0 con 0x7f64b0153770 2023-10-16T20:49:06.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.737+0000 7f64a67fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f64b00c9040 0x7f64b00c64e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54044/0 (socket says 172.21.15.88:54044) 2023-10-16T20:49:06.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.737+0000 7f64a67fc640 1 -- 172.21.15.88:0/4252336226 learned_addr learned my addr 172.21.15.88:0/4252336226 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:49:06.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.737+0000 7f64aeffd640 1 --2- 172.21.15.88:0/4252336226 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64b0153770 0x7f64b0159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:06.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.737+0000 7f64ae7fc640 1 -- 172.21.15.88:0/4252336226 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971869452 0 0) 0x7f64b0069e90 con 0x7f64b00c9040 2023-10-16T20:49:06.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.737+0000 7f64ae7fc640 1 -- 172.21.15.88:0/4252336226 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f649c003530 con 0x7f64b00c9040 2023-10-16T20:49:06.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.737+0000 7f64aeffd640 1 -- 172.21.15.88:0/4252336226 >> v1:172.21.15.191:6789/0 conn(0x7f64b00c6bf0 legacy=0x7f64b0071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:06.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.737+0000 7f64aeffd640 1 -- 172.21.15.88:0/4252336226 >> v1:172.21.15.88:6789/0 conn(0x7f64b00c9040 legacy=0x7f64b00c64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:06.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.737+0000 7f64aeffd640 1 -- 172.21.15.88:0/4252336226 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b006a360 con 0x7f64b0153770 2023-10-16T20:49:06.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64aeffd640 1 --2- 172.21.15.88:0/4252336226 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64b0153770 0x7f64b0159f00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f64980097f0 tx=0x7f649801c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c78995d89576773f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:49:06.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64ae7fc640 1 -- 172.21.15.88:0/4252336226 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64980ac630 con 0x7f64b0153770 2023-10-16T20:49:06.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64ae7fc640 1 -- 172.21.15.88:0/4252336226 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64980ac790 con 0x7f64b0153770 2023-10-16T20:49:06.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64ae7fc640 1 -- 172.21.15.88:0/4252336226 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64980acb20 con 0x7f64b0153770 2023-10-16T20:49:06.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64b59d0640 1 -- 172.21.15.88:0/4252336226 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64b0153770 msgr2=0x7f64b0159f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:06.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64b59d0640 1 --2- 172.21.15.88:0/4252336226 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64b0153770 0x7f64b0159f00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f64980097f0 tx=0x7f649801c4c0 comp rx=0 tx=0).stop 2023-10-16T20:49:06.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64b59d0640 1 -- 172.21.15.88:0/4252336226 shutdown_connections 2023-10-16T20:49:06.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64b59d0640 1 --2- 172.21.15.88:0/4252336226 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64b0153770 0x7f64b0159f00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:06.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64b59d0640 1 -- 172.21.15.88:0/4252336226 >> 172.21.15.88:0/4252336226 conn(0x7f64b00c84e0 msgr2=0x7f64b00c88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:49:06.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64b59d0640 1 -- 172.21.15.88:0/4252336226 shutdown_connections 2023-10-16T20:49:06.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64b59d0640 1 -- 172.21.15.88:0/4252336226 wait complete. 2023-10-16T20:49:06.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64b59d0640 1 Processor -- start 2023-10-16T20:49:06.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64b59d0640 1 -- start start 2023-10-16T20:49:06.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64b59d0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64b00c6bf0 0x7f64b0151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:06.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64b59d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64b00c9040 0x7f64b01586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:06.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64b59d0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64b0153770 0x7f64b0158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:06.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64b59d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b01f3840 con 0x7f64b00c9040 2023-10-16T20:49:06.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64b59d0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b01f3e00 con 0x7f64b00c6bf0 2023-10-16T20:49:06.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.741+0000 7f64b59d0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b0155270 con 0x7f64b0153770 2023-10-16T20:49:06.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64a67fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64b00c9040 0x7f64b01586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:06.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64a67fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64b00c9040 0x7f64b01586e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37850/0 (socket says 172.21.15.88:37850) 2023-10-16T20:49:06.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64a67fc640 1 -- 172.21.15.88:0/602375235 learned_addr learned my addr 172.21.15.88:0/602375235 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:49:06.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64aeffd640 1 --2- 172.21.15.88:0/602375235 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64b00c6bf0 0x7f64b0151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:06.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64af7fe640 1 --2- 172.21.15.88:0/602375235 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64b0153770 0x7f64b0158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:06.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64a67fc640 1 -- 172.21.15.88:0/602375235 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64b0153770 msgr2=0x7f64b0158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:06.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64a67fc640 1 --2- 172.21.15.88:0/602375235 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64b0153770 0x7f64b0158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:06.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64a67fc640 1 -- 172.21.15.88:0/602375235 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64b00c6bf0 msgr2=0x7f64b0151890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:06.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64a67fc640 1 --2- 172.21.15.88:0/602375235 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64b00c6bf0 0x7f64b0151890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:06.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64a67fc640 1 -- 172.21.15.88:0/602375235 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6498009420 con 0x7f64b00c9040 2023-10-16T20:49:06.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64a67fc640 1 --2- 172.21.15.88:0/602375235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64b00c9040 0x7f64b01586e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f64a000c500 tx=0x7f64a000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:49:06.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64acff9640 1 -- 172.21.15.88:0/602375235 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0005cf0 con 0x7f64b00c9040 2023-10-16T20:49:06.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64acff9640 1 -- 172.21.15.88:0/602375235 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a0005e50 con 0x7f64b00c9040 2023-10-16T20:49:06.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64b59d0640 1 -- 172.21.15.88:0/602375235 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b0153d90 con 0x7f64b00c9040 2023-10-16T20:49:06.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64acff9640 1 -- 172.21.15.88:0/602375235 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a00072f0 con 0x7f64b00c9040 2023-10-16T20:49:06.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64b59d0640 1 -- 172.21.15.88:0/602375235 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b01f3a90 con 0x7f64b00c9040 2023-10-16T20:49:06.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64b59d0640 1 -- 172.21.15.88:0/602375235 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b0074700 con 0x7f64b00c9040 2023-10-16T20:49:06.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.745+0000 7f64acff9640 1 -- 172.21.15.88:0/602375235 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f64a0007450 con 0x7f64b00c9040 2023-10-16T20:49:06.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.749+0000 7f64acff9640 1 --2- 172.21.15.88:0/602375235 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f64900664d0 0x7f6490068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:06.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.749+0000 7f64acff9640 1 -- 172.21.15.88:0/602375235 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7f64a00838d0 con 0x7f64b00c9040 2023-10-16T20:49:06.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.749+0000 7f64aeffd640 1 --2- 172.21.15.88:0/602375235 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f64900664d0 0x7f6490068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:06.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.749+0000 7f64aeffd640 1 --2- 172.21.15.88:0/602375235 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f64900664d0 0x7f6490068990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f64980ae000 tx=0x7f6498008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:49:06.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.753+0000 7f64acff9640 1 -- 172.21.15.88:0/602375235 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f64a0050940 con 0x7f64b00c9040 2023-10-16T20:49:06.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.933+0000 7f64b59d0640 1 -- 172.21.15.88:0/602375235 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64b00715c0 con 0x7f64b00c9040 2023-10-16T20:49:06.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.933+0000 7f64acff9640 1 -- 172.21.15.88:0/602375235 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5784 (secure 0 0 0) 0x7f64a0017090 con 0x7f64b00c9040 2023-10-16T20:49:06.933 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:49:06.934 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:49:05.529555+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:49:06.934 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 72 2023-10-16T20:49:06.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.933+0000 7f64b59d0640 1 -- 172.21.15.88:0/602375235 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f64900664d0 msgr2=0x7f6490068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:06.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.933+0000 7f64b59d0640 1 --2- 172.21.15.88:0/602375235 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f64900664d0 0x7f6490068990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f64980ae000 tx=0x7f6498008040 comp rx=0 tx=0).stop 2023-10-16T20:49:06.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.933+0000 7f64b59d0640 1 -- 172.21.15.88:0/602375235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64b00c9040 msgr2=0x7f64b01586e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:06.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.933+0000 7f64b59d0640 1 --2- 172.21.15.88:0/602375235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64b00c9040 0x7f64b01586e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f64a000c500 tx=0x7f64a000c9c0 comp rx=0 tx=0).stop 2023-10-16T20:49:06.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.933+0000 7f64b59d0640 1 -- 172.21.15.88:0/602375235 shutdown_connections 2023-10-16T20:49:06.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.933+0000 7f64b59d0640 1 --2- 172.21.15.88:0/602375235 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f64900664d0 0x7f6490068990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:06.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.933+0000 7f64b59d0640 1 --2- 172.21.15.88:0/602375235 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64b0153770 0x7f64b0158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:06.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.933+0000 7f64b59d0640 1 --2- 172.21.15.88:0/602375235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64b00c9040 0x7f64b01586e0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:06.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.933+0000 7f64b59d0640 1 --2- 172.21.15.88:0/602375235 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64b00c6bf0 0x7f64b0151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:06.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.933+0000 7f64b59d0640 1 -- 172.21.15.88:0/602375235 >> 172.21.15.88:0/602375235 conn(0x7f64b00c84e0 msgr2=0x7f64b00c3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:49:06.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.937+0000 7f64b59d0640 1 -- 172.21.15.88:0/602375235 shutdown_connections 2023-10-16T20:49:06.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:06.937+0000 7f64b59d0640 1 -- 172.21.15.88:0/602375235 wait complete. 2023-10-16T20:49:06.954 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=1, replacing gid=4279 2023-10-16T20:49:06.954 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.d 2023-10-16T20:49:14.955 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-16T20:49:14.956 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T20:49:14.956 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T20:49:14.960 INFO:tasks.ceph.mds.d:Started 2023-10-16T20:49:14.961 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:49:15.057 INFO:tasks.ceph.mds.d.smithi191.stdout:starting mds.d at 2023-10-16T20:49:15.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.113+0000 7f43830dc640 1 Processor -- start 2023-10-16T20:49:15.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.113+0000 7f43830dc640 1 -- start start 2023-10-16T20:49:15.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.117+0000 7f43830dc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f437c0cb580 0x7f437c0cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:15.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.117+0000 7f43830dc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f437c0b1a60 0x7f437c163b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:15.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.117+0000 7f43830dc640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437c0c9fc0 con 0x7f437c0c9af0 2023-10-16T20:49:15.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.117+0000 7f43830dc640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f437c0b20c0 con 0x7f437c0b1a60 2023-10-16T20:49:15.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.117+0000 7f43830dc640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437c0cbea0 con 0x7f437c0cb580 2023-10-16T20:49:15.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.117+0000 7f43828db640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f437c0b1a60 0x7f437c163b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:15.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.117+0000 7f43828db640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f437c0b1a60 0x7f437c163b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57868/0 (socket says 172.21.15.88:57868) 2023-10-16T20:49:15.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.117+0000 7f43828db640 1 -- 172.21.15.88:0/4220602047 learned_addr learned my addr 172.21.15.88:0/4220602047 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:49:15.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.117+0000 7f43820da640 1 --2- 172.21.15.88:0/4220602047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f437c0cb580 0x7f437c0cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:15.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.117+0000 7f43828db640 1 -- 172.21.15.88:0/4220602047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f437c0cb580 msgr2=0x7f437c0cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:49:15.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.117+0000 7f43828db640 1 --2- 172.21.15.88:0/4220602047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f437c0cb580 0x7f437c0cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:15.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.117+0000 7f43828db640 1 -- 172.21.15.88:0/4220602047 >> v1:172.21.15.191:6789/0 conn(0x7f437c0c9af0 legacy=0x7f437c159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:15.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.117+0000 7f43828db640 1 -- 172.21.15.88:0/4220602047 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c0c47d0 con 0x7f437c0b1a60 2023-10-16T20:49:15.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.121+0000 7f43828db640 1 --2- 172.21.15.88:0/4220602047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f437c0b1a60 0x7f437c163b80 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f4374009e50 tx=0x7f437401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a94f396a7b56e0a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:49:15.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.121+0000 7f43810d8640 1 -- 172.21.15.88:0/4220602047 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43740a5810 con 0x7f437c0b1a60 2023-10-16T20:49:15.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.121+0000 7f43810d8640 1 -- 172.21.15.88:0/4220602047 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43740a5970 con 0x7f437c0b1a60 2023-10-16T20:49:15.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.121+0000 7f43810d8640 1 -- 172.21.15.88:0/4220602047 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43740a5d00 con 0x7f437c0b1a60 2023-10-16T20:49:15.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.121+0000 7f43830dc640 1 -- 172.21.15.88:0/4220602047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f437c0b1a60 msgr2=0x7f437c163b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:15.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.121+0000 7f43830dc640 1 --2- 172.21.15.88:0/4220602047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f437c0b1a60 0x7f437c163b80 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f4374009e50 tx=0x7f437401c950 comp rx=0 tx=0).stop 2023-10-16T20:49:15.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.121+0000 7f43830dc640 1 -- 172.21.15.88:0/4220602047 shutdown_connections 2023-10-16T20:49:15.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.121+0000 7f43830dc640 1 --2- 172.21.15.88:0/4220602047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f437c0b1a60 0x7f437c163b80 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:15.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.121+0000 7f43830dc640 1 --2- 172.21.15.88:0/4220602047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f437c0cb580 0x7f437c0cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:15.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.121+0000 7f43830dc640 1 -- 172.21.15.88:0/4220602047 >> 172.21.15.88:0/4220602047 conn(0x7f437c06bb40 msgr2=0x7f437c06bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:49:15.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.121+0000 7f43830dc640 1 -- 172.21.15.88:0/4220602047 shutdown_connections 2023-10-16T20:49:15.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.121+0000 7f43830dc640 1 -- 172.21.15.88:0/4220602047 wait complete. 2023-10-16T20:49:15.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43830dc640 1 Processor -- start 2023-10-16T20:49:15.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43830dc640 1 -- start start 2023-10-16T20:49:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43830dc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f437c0b1a60 0x7f437c155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43830dc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437c0c9af0 0x7f437c156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43830dc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f437c0cb580 0x7f437c1f4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43830dc640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c155de0 con 0x7f437c0b1a60 2023-10-16T20:49:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43830dc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c155b00 con 0x7f437c0c9af0 2023-10-16T20:49:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43830dc640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437c14d880 con 0x7f437c0cb580 2023-10-16T20:49:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43820da640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f437c0b1a60 0x7f437c155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43820da640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f437c0b1a60 0x7f437c155450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57870/0 (socket says 172.21.15.88:57870) 2023-10-16T20:49:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43820da640 1 -- 172.21.15.88:0/1102062649 learned_addr learned my addr 172.21.15.88:0/1102062649 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:49:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43818d9640 1 --2- 172.21.15.88:0/1102062649 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437c0c9af0 0x7f437c156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43828db640 1 --2- 172.21.15.88:0/1102062649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f437c0cb580 0x7f437c1f4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43820da640 1 -- 172.21.15.88:0/1102062649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f437c0cb580 msgr2=0x7f437c1f4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:15.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43820da640 1 --2- 172.21.15.88:0/1102062649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f437c0cb580 0x7f437c1f4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:15.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43820da640 1 -- 172.21.15.88:0/1102062649 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437c0c9af0 msgr2=0x7f437c156480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:15.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43820da640 1 --2- 172.21.15.88:0/1102062649 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437c0c9af0 0x7f437c156480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:15.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43820da640 1 -- 172.21.15.88:0/1102062649 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4374009b30 con 0x7f437c0b1a60 2023-10-16T20:49:15.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43820da640 1 --2- 172.21.15.88:0/1102062649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f437c0b1a60 0x7f437c155450 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f437800e870 tx=0x7f437800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:49:15.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f4372ffd640 1 -- 172.21.15.88:0/1102062649 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4378009510 con 0x7f437c0b1a60 2023-10-16T20:49:15.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f4372ffd640 1 -- 172.21.15.88:0/1102062649 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437800eec0 con 0x7f437c0b1a60 2023-10-16T20:49:15.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43830dc640 1 -- 172.21.15.88:0/1102062649 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f437c14db60 con 0x7f437c0b1a60 2023-10-16T20:49:15.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f4372ffd640 1 -- 172.21.15.88:0/1102062649 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437800cb40 con 0x7f437c0b1a60 2023-10-16T20:49:15.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.125+0000 7f43830dc640 1 -- 172.21.15.88:0/1102062649 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f437c14e120 con 0x7f437c0b1a60 2023-10-16T20:49:15.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.129+0000 7f43830dc640 1 -- 172.21.15.88:0/1102062649 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f437c0ca2c0 con 0x7f437c0b1a60 2023-10-16T20:49:15.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.129+0000 7f4372ffd640 1 -- 172.21.15.88:0/1102062649 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f437800cca0 con 0x7f437c0b1a60 2023-10-16T20:49:15.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.129+0000 7f4372ffd640 1 --2- 172.21.15.88:0/1102062649 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f43600664d0 0x7f4360068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:15.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.129+0000 7f4372ffd640 1 -- 172.21.15.88:0/1102062649 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7f43780844a0 con 0x7f437c0b1a60 2023-10-16T20:49:15.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.129+0000 7f43818d9640 1 --2- 172.21.15.88:0/1102062649 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f43600664d0 0x7f4360068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:15.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.129+0000 7f43818d9640 1 --2- 172.21.15.88:0/1102062649 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f43600664d0 0x7f4360068990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f437c0748e0 tx=0x7f436c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:49:15.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.133+0000 7f4372ffd640 1 -- 172.21.15.88:0/1102062649 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4378051450 con 0x7f437c0b1a60 2023-10-16T20:49:15.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.317+0000 7f43830dc640 1 -- 172.21.15.88:0/1102062649 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f437c04e9c0 con 0x7f437c0b1a60 2023-10-16T20:49:15.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.321+0000 7f4372ffd640 1 -- 172.21.15.88:0/1102062649 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5784 (secure 0 0 0) 0x7f4378054a70 con 0x7f437c0b1a60 2023-10-16T20:49:15.320 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:49:15.320 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:49:05.529555+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:49:15.320 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 72 2023-10-16T20:49:15.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.321+0000 7f43830dc640 1 -- 172.21.15.88:0/1102062649 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f43600664d0 msgr2=0x7f4360068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:15.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.321+0000 7f43830dc640 1 --2- 172.21.15.88:0/1102062649 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f43600664d0 0x7f4360068990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f437c0748e0 tx=0x7f436c009040 comp rx=0 tx=0).stop 2023-10-16T20:49:15.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.321+0000 7f43830dc640 1 -- 172.21.15.88:0/1102062649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f437c0b1a60 msgr2=0x7f437c155450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:15.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.321+0000 7f43830dc640 1 --2- 172.21.15.88:0/1102062649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f437c0b1a60 0x7f437c155450 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f437800e870 tx=0x7f437800ec70 comp rx=0 tx=0).stop 2023-10-16T20:49:15.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.321+0000 7f43830dc640 1 -- 172.21.15.88:0/1102062649 shutdown_connections 2023-10-16T20:49:15.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.321+0000 7f43830dc640 1 --2- 172.21.15.88:0/1102062649 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f43600664d0 0x7f4360068990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:15.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.321+0000 7f43830dc640 1 --2- 172.21.15.88:0/1102062649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f437c0cb580 0x7f437c1f4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:15.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.321+0000 7f43830dc640 1 --2- 172.21.15.88:0/1102062649 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f437c0c9af0 0x7f437c156480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:15.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.321+0000 7f43830dc640 1 --2- 172.21.15.88:0/1102062649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f437c0b1a60 0x7f437c155450 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:15.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.321+0000 7f43830dc640 1 -- 172.21.15.88:0/1102062649 >> 172.21.15.88:0/1102062649 conn(0x7f437c06bb40 msgr2=0x7f437c158e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:49:15.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.321+0000 7f43830dc640 1 -- 172.21.15.88:0/1102062649 shutdown_connections 2023-10-16T20:49:15.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:15.321+0000 7f43830dc640 1 -- 172.21.15.88:0/1102062649 wait complete. 2023-10-16T20:49:15.341 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-16T20:49:17.343 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:49:17.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.481+0000 7f6a02f80640 1 Processor -- start 2023-10-16T20:49:17.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.481+0000 7f6a02f80640 1 -- start start 2023-10-16T20:49:17.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f6a02f80640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69fc15a810 0x7f69fc15ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:17.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f6a02f80640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69fc15b190 con 0x7f69fc06a970 2023-10-16T20:49:17.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f6a02f80640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69fc15b3c0 con 0x7f69fc1406d0 2023-10-16T20:49:17.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f6a02f80640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc0c2460 con 0x7f69fc15a810 2023-10-16T20:49:17.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fbfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f69fc06a970 0x7f69fc071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55914/0 (socket says 172.21.15.88:55914) 2023-10-16T20:49:17.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fbfff640 1 -- 172.21.15.88:0/93785071 learned_addr learned my addr 172.21.15.88:0/93785071 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:49:17.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fb7fe640 1 -- 172.21.15.88:0/93785071 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479826524 0 0) 0x7f69fc15b190 con 0x7f69fc06a970 2023-10-16T20:49:17.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fb7fe640 1 -- 172.21.15.88:0/93785071 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69ec003530 con 0x7f69fc06a970 2023-10-16T20:49:17.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f6a0151d640 1 --2- 172.21.15.88:0/93785071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69fc15a810 0x7f69fc15ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:17.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fb7fe640 1 -- 172.21.15.88:0/93785071 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1001043776 0 0) 0x7f69ec003530 con 0x7f69fc06a970 2023-10-16T20:49:17.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fb7fe640 1 -- 172.21.15.88:0/93785071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69fc15a810 msgr2=0x7f69fc15ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:49:17.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fb7fe640 1 --2- 172.21.15.88:0/93785071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69fc15a810 0x7f69fc15ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:17.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fb7fe640 1 -- 172.21.15.88:0/93785071 >> v1:172.21.15.191:6789/0 conn(0x7f69fc1406d0 legacy=0x7f69fc0c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:17.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fb7fe640 1 -- 172.21.15.88:0/93785071 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69fc15b680 con 0x7f69fc06a970 2023-10-16T20:49:17.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fb7fe640 1 -- 172.21.15.88:0/93785071 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f69e8001680 con 0x7f69fc06a970 2023-10-16T20:49:17.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fb7fe640 1 -- 172.21.15.88:0/93785071 >> v1:172.21.15.88:6789/0 conn(0x7f69fc06a970 legacy=0x7f69fc071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:17.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fb7fe640 1 --2- 172.21.15.88:0/93785071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ec0039b0 0x7f69ec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:17.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fb7fe640 1 --2- 172.21.15.88:0/93785071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69fc15a810 0x7f69ec0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:17.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fb7fe640 1 --2- 172.21.15.88:0/93785071 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69ec004d10 0x7f69ec005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:17.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fb7fe640 1 -- 172.21.15.88:0/93785071 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc0c2460 con 0x7f69ec0039b0 2023-10-16T20:49:17.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fb7fe640 1 -- 172.21.15.88:0/93785071 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec00b990 con 0x7f69ec004d10 2023-10-16T20:49:17.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fb7fe640 1 -- 172.21.15.88:0/93785071 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec00bb50 con 0x7f69fc15a810 2023-10-16T20:49:17.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f6a00d1c640 1 --2- 172.21.15.88:0/93785071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ec0039b0 0x7f69ec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:17.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f69fbfff640 1 --2- 172.21.15.88:0/93785071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69fc15a810 0x7f69ec0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:17.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f6a0151d640 1 --2- 172.21.15.88:0/93785071 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69ec004d10 0x7f69ec005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:17.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f6a00d1c640 1 -- 172.21.15.88:0/93785071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69fc15a810 msgr2=0x7f69ec0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:49:17.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f6a00d1c640 1 --2- 172.21.15.88:0/93785071 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69fc15a810 0x7f69ec0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:17.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f6a00d1c640 1 -- 172.21.15.88:0/93785071 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69ec004d10 msgr2=0x7f69ec005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:49:17.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f6a00d1c640 1 --2- 172.21.15.88:0/93785071 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69ec004d10 0x7f69ec005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:17.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.485+0000 7f6a00d1c640 1 -- 172.21.15.88:0/93785071 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ec003530 con 0x7f69ec0039b0 2023-10-16T20:49:17.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.489+0000 7f6a00d1c640 1 --2- 172.21.15.88:0/93785071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ec0039b0 0x7f69ec003d90 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f69f0001860 tx=0x7f69f001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d3ee3e002cad495 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:49:17.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.489+0000 7f69fb7fe640 1 -- 172.21.15.88:0/93785071 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f00ab600 con 0x7f69ec0039b0 2023-10-16T20:49:17.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.489+0000 7f69fb7fe640 1 -- 172.21.15.88:0/93785071 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f00a7050 con 0x7f69ec0039b0 2023-10-16T20:49:17.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.489+0000 7f69fb7fe640 1 -- 172.21.15.88:0/93785071 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f00ab9a0 con 0x7f69ec0039b0 2023-10-16T20:49:17.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.489+0000 7f6a02f80640 1 -- 172.21.15.88:0/93785071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ec0039b0 msgr2=0x7f69ec003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:17.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.489+0000 7f6a02f80640 1 --2- 172.21.15.88:0/93785071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ec0039b0 0x7f69ec003d90 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f69f0001860 tx=0x7f69f001c410 comp rx=0 tx=0).stop 2023-10-16T20:49:17.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.489+0000 7f6a0151d640 1 -- 172.21.15.88:0/93785071 reap_dead start 2023-10-16T20:49:17.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.489+0000 7f6a02f80640 1 -- 172.21.15.88:0/93785071 shutdown_connections 2023-10-16T20:49:17.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.489+0000 7f6a02f80640 1 -- 172.21.15.88:0/93785071 >> 172.21.15.88:0/93785071 conn(0x7f69fc0cad50 msgr2=0x7f69fc0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:49:17.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.489+0000 7f6a02f80640 1 -- 172.21.15.88:0/93785071 shutdown_connections 2023-10-16T20:49:17.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.489+0000 7f6a02f80640 1 -- 172.21.15.88:0/93785071 wait complete. 2023-10-16T20:49:17.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.489+0000 7f6a02f80640 1 Processor -- start 2023-10-16T20:49:17.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a02f80640 1 -- start start 2023-10-16T20:49:17.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a02f80640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ec0039b0 0x7f69fc1f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:17.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a02f80640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69fc1583e0 0x7f69fc15ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:17.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a02f80640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69fc06fd80 0x7f69fc135ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:17.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a02f80640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc158a10 con 0x7f69ec0039b0 2023-10-16T20:49:17.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a02f80640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc1f19b0 con 0x7f69fc06fd80 2023-10-16T20:49:17.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a02f80640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc06fbb0 con 0x7f69fc1583e0 2023-10-16T20:49:17.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a00d1c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ec0039b0 0x7f69fc1f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:17.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a00d1c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ec0039b0 0x7f69fc1f23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57892/0 (socket says 172.21.15.88:57892) 2023-10-16T20:49:17.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a00d1c640 1 -- 172.21.15.88:0/2649964434 learned_addr learned my addr 172.21.15.88:0/2649964434 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:49:17.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a0151d640 1 --2- 172.21.15.88:0/2649964434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69fc06fd80 0x7f69fc135ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:17.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f69fbfff640 1 --2- 172.21.15.88:0/2649964434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69fc1583e0 0x7f69fc15ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:17.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a00d1c640 1 -- 172.21.15.88:0/2649964434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69fc1583e0 msgr2=0x7f69fc15ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:17.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a00d1c640 1 --2- 172.21.15.88:0/2649964434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69fc1583e0 0x7f69fc15ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:17.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a00d1c640 1 -- 172.21.15.88:0/2649964434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69fc06fd80 msgr2=0x7f69fc135ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:17.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a00d1c640 1 --2- 172.21.15.88:0/2649964434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69fc06fd80 0x7f69fc135ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:17.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a00d1c640 1 -- 172.21.15.88:0/2649964434 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69fc1f1f30 con 0x7f69ec0039b0 2023-10-16T20:49:17.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a00d1c640 1 --2- 172.21.15.88:0/2649964434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ec0039b0 0x7f69fc1f23c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f69f001c240 tx=0x7f69ec005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:49:17.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f69f97fa640 1 -- 172.21.15.88:0/2649964434 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f0005390 con 0x7f69ec0039b0 2023-10-16T20:49:17.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f69f97fa640 1 -- 172.21.15.88:0/2649964434 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f00aba80 con 0x7f69ec0039b0 2023-10-16T20:49:17.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a02f80640 1 -- 172.21.15.88:0/2649964434 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69fc1f2900 con 0x7f69ec0039b0 2023-10-16T20:49:17.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f69f97fa640 1 -- 172.21.15.88:0/2649964434 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f00abdf0 con 0x7f69ec0039b0 2023-10-16T20:49:17.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a02f80640 1 -- 172.21.15.88:0/2649964434 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69fc13eb20 con 0x7f69ec0039b0 2023-10-16T20:49:17.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.493+0000 7f6a02f80640 1 -- 172.21.15.88:0/2649964434 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69fc13ed10 con 0x7f69ec0039b0 2023-10-16T20:49:17.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.497+0000 7f69f97fa640 1 -- 172.21.15.88:0/2649964434 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f69f00b2050 con 0x7f69ec0039b0 2023-10-16T20:49:17.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.497+0000 7f69f97fa640 1 --2- 172.21.15.88:0/2649964434 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69e00664d0 0x7f69e0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:49:17.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.497+0000 7f69f97fa640 1 -- 172.21.15.88:0/2649964434 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7f69f011e430 con 0x7f69ec0039b0 2023-10-16T20:49:17.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.497+0000 7f69fbfff640 1 --2- 172.21.15.88:0/2649964434 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69e00664d0 0x7f69e0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:49:17.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.497+0000 7f69fbfff640 1 --2- 172.21.15.88:0/2649964434 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69e00664d0 0x7f69e0068990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f69fc13e720 tx=0x7f69e8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:49:17.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.501+0000 7f69f97fa640 1 -- 172.21.15.88:0/2649964434 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f69f00eb340 con 0x7f69ec0039b0 2023-10-16T20:49:17.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.709+0000 7f6a02f80640 1 -- 172.21.15.88:0/2649964434 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69fc074700 con 0x7f69ec0039b0 2023-10-16T20:49:17.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.713+0000 7f69f97fa640 1 -- 172.21.15.88:0/2649964434 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+6575 (secure 0 0 0) 0x7f69f00ee960 con 0x7f69ec0039b0 2023-10-16T20:49:17.712 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:49:17.712 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:49:17.469846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:49:17.712 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 74 2023-10-16T20:49:17.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.713+0000 7f6a02f80640 1 -- 172.21.15.88:0/2649964434 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69e00664d0 msgr2=0x7f69e0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:17.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.713+0000 7f6a02f80640 1 --2- 172.21.15.88:0/2649964434 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69e00664d0 0x7f69e0068990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f69fc13e720 tx=0x7f69e8009040 comp rx=0 tx=0).stop 2023-10-16T20:49:17.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.713+0000 7f6a02f80640 1 -- 172.21.15.88:0/2649964434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ec0039b0 msgr2=0x7f69fc1f23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:49:17.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.713+0000 7f6a02f80640 1 --2- 172.21.15.88:0/2649964434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ec0039b0 0x7f69fc1f23c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f69f001c240 tx=0x7f69ec005800 comp rx=0 tx=0).stop 2023-10-16T20:49:17.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.713+0000 7f6a02f80640 1 -- 172.21.15.88:0/2649964434 shutdown_connections 2023-10-16T20:49:17.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.713+0000 7f6a02f80640 1 --2- 172.21.15.88:0/2649964434 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f69e00664d0 0x7f69e0068990 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:17.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.713+0000 7f6a02f80640 1 --2- 172.21.15.88:0/2649964434 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f69fc06fd80 0x7f69fc135ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:17.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.713+0000 7f6a02f80640 1 --2- 172.21.15.88:0/2649964434 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f69fc1583e0 0x7f69fc15ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:17.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.713+0000 7f6a02f80640 1 --2- 172.21.15.88:0/2649964434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ec0039b0 0x7f69fc1f23c0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:49:17.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.713+0000 7f6a02f80640 1 -- 172.21.15.88:0/2649964434 >> 172.21.15.88:0/2649964434 conn(0x7f69fc0cad50 msgr2=0x7f69fc13a360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:49:17.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.713+0000 7f6a02f80640 1 -- 172.21.15.88:0/2649964434 shutdown_connections 2023-10-16T20:49:17.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:49:17.713+0000 7f6a02f80640 1 -- 172.21.15.88:0/2649964434 wait complete. 2023-10-16T20:49:17.732 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-16T20:49:17.733 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 107 secs before thrashing 2023-10-16T20:49:28.985 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:49:28.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:49:59.065 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:49:59.070 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:50:29.129 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:50:29.202 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:50:59.259 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:50:59.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:51:04.734 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:04.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.887+0000 7f238bc68640 1 Processor -- start 2023-10-16T20:51:04.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.887+0000 7f238bc68640 1 -- start start 2023-10-16T20:51:04.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.887+0000 7f238bc68640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2384153770 0x7f2384159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:04.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.887+0000 7f238bc68640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f23840cece0 0x7f23840cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:04.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.887+0000 7f238bc68640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238415a410 con 0x7f23840c9bb0 2023-10-16T20:51:04.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.887+0000 7f238bc68640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23840cc250 con 0x7f23840cece0 2023-10-16T20:51:04.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.887+0000 7f238bc68640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23840cc670 con 0x7f2384153770 2023-10-16T20:51:04.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.887+0000 7f238a205640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f23840cece0 0x7f23840cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:04.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.887+0000 7f238a205640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f23840cece0 0x7f23840cf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39344/0 (socket says 172.21.15.88:39344) 2023-10-16T20:51:04.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.887+0000 7f238a205640 1 -- 172.21.15.88:0/533512422 learned_addr learned my addr 172.21.15.88:0/533512422 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:04.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.887+0000 7f2389a04640 1 --2- 172.21.15.88:0/533512422 >> v2:172.21.15.191:3300/0 conn(0x7f2384153770 0x7f2384159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.887+0000 7f238a205640 1 -- 172.21.15.88:0/533512422 >> v2:172.21.15.191:3300/0 conn(0x7f2384153770 msgr2=0x7f2384159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.887+0000 7f238a205640 1 --2- 172.21.15.88:0/533512422 >> v2:172.21.15.191:3300/0 conn(0x7f2384153770 0x7f2384159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.887+0000 7f238a205640 1 -- 172.21.15.88:0/533512422 >> v1:172.21.15.88:6789/0 conn(0x7f23840c9bb0 legacy=0x7f23840c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:04.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.887+0000 7f238a205640 1 -- 172.21.15.88:0/533512422 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2384153b50 con 0x7f23840cece0 2023-10-16T20:51:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.891+0000 7f238a205640 1 --2- 172.21.15.88:0/533512422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23840cece0 0x7f23840cf130 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f23840cc7b0 tx=0x7f237c01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a45a7a91cef44a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.891+0000 7f2388a02640 1 -- 172.21.15.88:0/533512422 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c0a5890 con 0x7f23840cece0 2023-10-16T20:51:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.891+0000 7f2388a02640 1 -- 172.21.15.88:0/533512422 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237c0a59f0 con 0x7f23840cece0 2023-10-16T20:51:04.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.891+0000 7f2388a02640 1 -- 172.21.15.88:0/533512422 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c0a5d80 con 0x7f23840cece0 2023-10-16T20:51:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f238bc68640 1 -- 172.21.15.88:0/533512422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23840cece0 msgr2=0x7f23840cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:04.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f238bc68640 1 --2- 172.21.15.88:0/533512422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23840cece0 0x7f23840cf130 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f23840cc7b0 tx=0x7f237c01c9d0 comp rx=0 tx=0).stop 2023-10-16T20:51:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f238bc68640 1 -- 172.21.15.88:0/533512422 shutdown_connections 2023-10-16T20:51:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f238bc68640 1 --2- 172.21.15.88:0/533512422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23840cece0 0x7f23840cf130 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f238bc68640 1 --2- 172.21.15.88:0/533512422 >> v2:172.21.15.191:3300/0 conn(0x7f2384153770 0x7f2384159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f238bc68640 1 -- 172.21.15.88:0/533512422 >> 172.21.15.88:0/533512422 conn(0x7f23840715c0 msgr2=0x7f23840719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:04.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f238bc68640 1 -- 172.21.15.88:0/533512422 shutdown_connections 2023-10-16T20:51:04.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f238bc68640 1 -- 172.21.15.88:0/533512422 wait complete. 2023-10-16T20:51:04.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f238bc68640 1 Processor -- start 2023-10-16T20:51:04.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f238bc68640 1 -- start start 2023-10-16T20:51:04.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f238bc68640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23840c9bb0 0x7f238415b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:04.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f238bc68640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23840cece0 0x7f23841f47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f238bc68640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2384153770 0x7f23840b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f238bc68640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23840b3740 con 0x7f23840c9bb0 2023-10-16T20:51:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f238bc68640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238415b960 con 0x7f2384153770 2023-10-16T20:51:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f238bc68640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23840b2e30 con 0x7f23840cece0 2023-10-16T20:51:04.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f2389a04640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23840c9bb0 0x7f238415b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f2389a04640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23840c9bb0 0x7f238415b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39354/0 (socket says 172.21.15.88:39354) 2023-10-16T20:51:04.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f2389a04640 1 -- 172.21.15.88:0/3789787259 learned_addr learned my addr 172.21.15.88:0/3789787259 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:04.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.895+0000 7f2389203640 1 --2- 172.21.15.88:0/3789787259 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23840cece0 0x7f23841f47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:04.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.899+0000 7f238a205640 1 --2- 172.21.15.88:0/3789787259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2384153770 0x7f23840b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:04.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.899+0000 7f2389a04640 1 -- 172.21.15.88:0/3789787259 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23840cece0 msgr2=0x7f23841f47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:04.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.899+0000 7f2389a04640 1 --2- 172.21.15.88:0/3789787259 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23840cece0 0x7f23841f47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:04.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.899+0000 7f2389a04640 1 -- 172.21.15.88:0/3789787259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2384153770 msgr2=0x7f23840b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:04.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.899+0000 7f2389a04640 1 --2- 172.21.15.88:0/3789787259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2384153770 0x7f23840b3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.899+0000 7f2389a04640 1 -- 172.21.15.88:0/3789787259 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c009b30 con 0x7f23840c9bb0 2023-10-16T20:51:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.899+0000 7f2389203640 1 --2- 172.21.15.88:0/3789787259 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23840cece0 0x7f23841f47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:51:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.899+0000 7f2389a04640 1 --2- 172.21.15.88:0/3789787259 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23840c9bb0 0x7f238415b1d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f238000d7d0 tx=0x7f238000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:04.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.899+0000 7f237a7fc640 1 -- 172.21.15.88:0/3789787259 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238000b6c0 con 0x7f23840c9bb0 2023-10-16T20:51:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.899+0000 7f237a7fc640 1 -- 172.21.15.88:0/3789787259 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238000b820 con 0x7f23840c9bb0 2023-10-16T20:51:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.899+0000 7f238bc68640 1 -- 172.21.15.88:0/3789787259 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23841f5210 con 0x7f23840c9bb0 2023-10-16T20:51:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.899+0000 7f237a7fc640 1 -- 172.21.15.88:0/3789787259 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238000bbd0 con 0x7f23840c9bb0 2023-10-16T20:51:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.899+0000 7f238bc68640 1 -- 172.21.15.88:0/3789787259 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23840b33d0 con 0x7f23840c9bb0 2023-10-16T20:51:04.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.899+0000 7f238bc68640 1 -- 172.21.15.88:0/3789787259 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23840b3590 con 0x7f23840c9bb0 2023-10-16T20:51:04.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.899+0000 7f237a7fc640 1 -- 172.21.15.88:0/3789787259 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f238000bd30 con 0x7f23840c9bb0 2023-10-16T20:51:04.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.903+0000 7f237a7fc640 1 --2- 172.21.15.88:0/3789787259 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f23680664a0 0x7f2368068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:04.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.903+0000 7f237a7fc640 1 -- 172.21.15.88:0/3789787259 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7f2380084950 con 0x7f23840c9bb0 2023-10-16T20:51:04.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.903+0000 7f2389203640 1 --2- 172.21.15.88:0/3789787259 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f23680664a0 0x7f2368068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:04.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.903+0000 7f2389203640 1 --2- 172.21.15.88:0/3789787259 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f23680664a0 0x7f2368068960 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f238414d770 tx=0x7f2374009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:04.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:04.907+0000 7f237a7fc640 1 -- 172.21.15.88:0/3789787259 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2380051730 con 0x7f23840c9bb0 2023-10-16T20:51:05.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.115+0000 7f238bc68640 1 -- 172.21.15.88:0/3789787259 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2384074700 con 0x7f23840c9bb0 2023-10-16T20:51:05.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.119+0000 7f237a7fc640 1 -- 172.21.15.88:0/3789787259 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6567 (secure 0 0 0) 0x7f2380054d50 con 0x7f23840c9bb0 2023-10-16T20:51:05.119 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:05.120 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:49:52.334480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:05.120 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 80 2023-10-16T20:51:05.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.119+0000 7f238bc68640 1 -- 172.21.15.88:0/3789787259 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f23680664a0 msgr2=0x7f2368068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:05.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.119+0000 7f238bc68640 1 --2- 172.21.15.88:0/3789787259 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f23680664a0 0x7f2368068960 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f238414d770 tx=0x7f2374009040 comp rx=0 tx=0).stop 2023-10-16T20:51:05.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.119+0000 7f238bc68640 1 -- 172.21.15.88:0/3789787259 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23840c9bb0 msgr2=0x7f238415b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:05.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.119+0000 7f238bc68640 1 --2- 172.21.15.88:0/3789787259 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23840c9bb0 0x7f238415b1d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f238000d7d0 tx=0x7f238000dc90 comp rx=0 tx=0).stop 2023-10-16T20:51:05.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.119+0000 7f238bc68640 1 -- 172.21.15.88:0/3789787259 shutdown_connections 2023-10-16T20:51:05.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.119+0000 7f238bc68640 1 --2- 172.21.15.88:0/3789787259 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f23680664a0 0x7f2368068960 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:05.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.119+0000 7f238bc68640 1 --2- 172.21.15.88:0/3789787259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2384153770 0x7f23840b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:05.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.119+0000 7f238bc68640 1 --2- 172.21.15.88:0/3789787259 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f23840cece0 0x7f23841f47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:05.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.119+0000 7f238bc68640 1 --2- 172.21.15.88:0/3789787259 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23840c9bb0 0x7f238415b1d0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:05.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.119+0000 7f238bc68640 1 -- 172.21.15.88:0/3789787259 >> 172.21.15.88:0/3789787259 conn(0x7f23840715c0 msgr2=0x7f2384151380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:05.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.119+0000 7f238bc68640 1 -- 172.21.15.88:0/3789787259 shutdown_connections 2023-10-16T20:51:05.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.119+0000 7f238bc68640 1 -- 172.21.15.88:0/3789787259 wait complete. 2023-10-16T20:51:05.142 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2023-10-16T20:51:05.143 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-16T20:51:05.143 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T20:51:05.163 INFO:tasks.ceph.mds.f:Stopped 2023-10-16T20:51:05.163 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:05.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.319+0000 7fc97b59e640 1 Processor -- start 2023-10-16T20:51:05.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.319+0000 7fc97b59e640 1 -- start start 2023-10-16T20:51:05.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.319+0000 7fc97b59e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc97c0c7ab0 0x7fc97c06f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:05.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.319+0000 7fc97b59e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c157970 con 0x7fc97c071c30 2023-10-16T20:51:05.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.319+0000 7fc97b59e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c157c50 con 0x7fc97c0c76d0 2023-10-16T20:51:05.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.319+0000 7fc97b59e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c157780 con 0x7fc97c0c7ab0 2023-10-16T20:51:05.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.319+0000 7fc97ad9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc97c071c30 0x7fc97c150df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42468/0 (socket says 172.21.15.88:42468) 2023-10-16T20:51:05.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.319+0000 7fc97ad9d640 1 -- 172.21.15.88:0/2242041394 learned_addr learned my addr 172.21.15.88:0/2242041394 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:51:05.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.319+0000 7fc979d9b640 1 --2- 172.21.15.88:0/2242041394 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc97c0c7ab0 0x7fc97c06f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:05.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.319+0000 7fc97959a640 1 -- 172.21.15.88:0/2242041394 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125893275 0 0) 0x7fc97c157970 con 0x7fc97c071c30 2023-10-16T20:51:05.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.319+0000 7fc97959a640 1 -- 172.21.15.88:0/2242041394 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc964003530 con 0x7fc97c071c30 2023-10-16T20:51:05.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.319+0000 7fc979d9b640 1 -- 172.21.15.88:0/2242041394 >> v1:172.21.15.191:6789/0 conn(0x7fc97c0c76d0 legacy=0x7fc97c157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:05.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.319+0000 7fc97959a640 1 -- 172.21.15.88:0/2242041394 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243795187 0 0) 0x7fc97c157c50 con 0x7fc97c0c76d0 2023-10-16T20:51:05.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.319+0000 7fc979d9b640 1 -- 172.21.15.88:0/2242041394 >> v1:172.21.15.88:6789/0 conn(0x7fc97c071c30 legacy=0x7fc97c150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:05.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.319+0000 7fc979d9b640 1 -- 172.21.15.88:0/2242041394 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97c157f30 con 0x7fc97c0c7ab0 2023-10-16T20:51:05.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.323+0000 7fc979d9b640 1 --2- 172.21.15.88:0/2242041394 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc97c0c7ab0 0x7fc97c06f510 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc9700097f0 tx=0x7fc97001c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7310e854e9ff050 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:05.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.323+0000 7fc97959a640 1 -- 172.21.15.88:0/2242041394 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9700ab600 con 0x7fc97c0c7ab0 2023-10-16T20:51:05.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.323+0000 7fc97959a640 1 -- 172.21.15.88:0/2242041394 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9700a7050 con 0x7fc97c0c7ab0 2023-10-16T20:51:05.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.323+0000 7fc97959a640 1 -- 172.21.15.88:0/2242041394 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9700ab960 con 0x7fc97c0c7ab0 2023-10-16T20:51:05.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97b59e640 1 -- 172.21.15.88:0/2242041394 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc97c0c7ab0 msgr2=0x7fc97c06f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:05.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97b59e640 1 --2- 172.21.15.88:0/2242041394 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc97c0c7ab0 0x7fc97c06f510 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc9700097f0 tx=0x7fc97001c4c0 comp rx=0 tx=0).stop 2023-10-16T20:51:05.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97b59e640 1 -- 172.21.15.88:0/2242041394 shutdown_connections 2023-10-16T20:51:05.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97b59e640 1 --2- 172.21.15.88:0/2242041394 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc97c0c7ab0 0x7fc97c06f510 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:05.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97b59e640 1 -- 172.21.15.88:0/2242041394 >> 172.21.15.88:0/2242041394 conn(0x7fc97c06d2d0 msgr2=0x7fc97c06d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:05.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97b59e640 1 -- 172.21.15.88:0/2242041394 shutdown_connections 2023-10-16T20:51:05.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97b59e640 1 -- 172.21.15.88:0/2242041394 wait complete. 2023-10-16T20:51:05.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97b59e640 1 Processor -- start 2023-10-16T20:51:05.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97b59e640 1 -- start start 2023-10-16T20:51:05.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97b59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc97c071c30 0x7fc97c0c6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:05.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97b59e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc97c0c76d0 0x7fc97c0c25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:05.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97b59e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc97c0c7ab0 0x7fc97c0ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:05.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97b59e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c0c5950 con 0x7fc97c071c30 2023-10-16T20:51:05.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97b59e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c1f5fe0 con 0x7fc97c0c7ab0 2023-10-16T20:51:05.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97b59e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c14f3b0 con 0x7fc97c0c76d0 2023-10-16T20:51:05.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97a59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc97c071c30 0x7fc97c0c6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:05.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97a59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc97c071c30 0x7fc97c0c6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39368/0 (socket says 172.21.15.88:39368) 2023-10-16T20:51:05.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97a59c640 1 -- 172.21.15.88:0/1966532751 learned_addr learned my addr 172.21.15.88:0/1966532751 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:05.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc979d9b640 1 --2- 172.21.15.88:0/1966532751 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc97c0c76d0 0x7fc97c0c25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:05.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.327+0000 7fc97ad9d640 1 --2- 172.21.15.88:0/1966532751 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc97c0c7ab0 0x7fc97c0ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:05.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.331+0000 7fc97a59c640 1 -- 172.21.15.88:0/1966532751 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc97c0c76d0 msgr2=0x7fc97c0c25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:05.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.331+0000 7fc97a59c640 1 --2- 172.21.15.88:0/1966532751 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc97c0c76d0 0x7fc97c0c25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:05.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.331+0000 7fc97a59c640 1 -- 172.21.15.88:0/1966532751 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc97c0c7ab0 msgr2=0x7fc97c0ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:05.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.331+0000 7fc97a59c640 1 --2- 172.21.15.88:0/1966532751 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc97c0c7ab0 0x7fc97c0ca7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:05.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.331+0000 7fc97a59c640 1 -- 172.21.15.88:0/1966532751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc970009420 con 0x7fc97c071c30 2023-10-16T20:51:05.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.331+0000 7fc97a59c640 1 --2- 172.21.15.88:0/1966532751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc97c071c30 0x7fc97c0c6400 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc96c00c480 tx=0x7fc96c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:05.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.331+0000 7fc96affd640 1 -- 172.21.15.88:0/1966532751 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c005480 con 0x7fc97c071c30 2023-10-16T20:51:05.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.331+0000 7fc96affd640 1 -- 172.21.15.88:0/1966532751 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc96c003600 con 0x7fc97c071c30 2023-10-16T20:51:05.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.331+0000 7fc97b59e640 1 -- 172.21.15.88:0/1966532751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97c14ded0 con 0x7fc97c071c30 2023-10-16T20:51:05.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.331+0000 7fc96affd640 1 -- 172.21.15.88:0/1966532751 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c0039b0 con 0x7fc97c071c30 2023-10-16T20:51:05.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.331+0000 7fc97b59e640 1 -- 172.21.15.88:0/1966532751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97c1f5500 con 0x7fc97c071c30 2023-10-16T20:51:05.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.331+0000 7fc97b59e640 1 -- 172.21.15.88:0/1966532751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc97c04e9c0 con 0x7fc97c071c30 2023-10-16T20:51:05.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.331+0000 7fc96affd640 1 -- 172.21.15.88:0/1966532751 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc96c003b10 con 0x7fc97c071c30 2023-10-16T20:51:05.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.335+0000 7fc96affd640 1 --2- 172.21.15.88:0/1966532751 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc95c0664d0 0x7fc95c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:05.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.335+0000 7fc96affd640 1 -- 172.21.15.88:0/1966532751 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7fc96c083ec0 con 0x7fc97c071c30 2023-10-16T20:51:05.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.335+0000 7fc979d9b640 1 --2- 172.21.15.88:0/1966532751 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc95c0664d0 0x7fc95c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:05.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.335+0000 7fc979d9b640 1 --2- 172.21.15.88:0/1966532751 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc95c0664d0 0x7fc95c068990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc9700ad000 tx=0x7fc9700a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:05.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.339+0000 7fc96affd640 1 -- 172.21.15.88:0/1966532751 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc96c050de0 con 0x7fc97c071c30 2023-10-16T20:51:05.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.567+0000 7fc97b59e640 1 -- 172.21.15.88:0/1966532751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc97c145870 con 0x7fc97c071c30 2023-10-16T20:51:05.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.567+0000 7fc96affd640 1 -- 172.21.15.88:0/1966532751 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6567 (secure 0 0 0) 0x7fc96c054400 con 0x7fc97c071c30 2023-10-16T20:51:05.568 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:05.568 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:49:52.334480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:05.569 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 80 2023-10-16T20:51:05.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.567+0000 7fc97b59e640 1 -- 172.21.15.88:0/1966532751 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc95c0664d0 msgr2=0x7fc95c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:05.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.567+0000 7fc97b59e640 1 --2- 172.21.15.88:0/1966532751 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc95c0664d0 0x7fc95c068990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc9700ad000 tx=0x7fc9700a6000 comp rx=0 tx=0).stop 2023-10-16T20:51:05.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.567+0000 7fc97b59e640 1 -- 172.21.15.88:0/1966532751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc97c071c30 msgr2=0x7fc97c0c6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:05.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.567+0000 7fc97b59e640 1 --2- 172.21.15.88:0/1966532751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc97c071c30 0x7fc97c0c6400 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc96c00c480 tx=0x7fc96c00c940 comp rx=0 tx=0).stop 2023-10-16T20:51:05.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.567+0000 7fc97b59e640 1 -- 172.21.15.88:0/1966532751 shutdown_connections 2023-10-16T20:51:05.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.567+0000 7fc97b59e640 1 --2- 172.21.15.88:0/1966532751 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc95c0664d0 0x7fc95c068990 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:05.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.567+0000 7fc97b59e640 1 --2- 172.21.15.88:0/1966532751 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc97c0c7ab0 0x7fc97c0ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:05.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.571+0000 7fc97b59e640 1 --2- 172.21.15.88:0/1966532751 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc97c0c76d0 0x7fc97c0c25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:05.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.571+0000 7fc97b59e640 1 --2- 172.21.15.88:0/1966532751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc97c071c30 0x7fc97c0c6400 secure :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc96c00c480 tx=0x7fc96c00c940 comp rx=0 tx=0).stop 2023-10-16T20:51:05.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.571+0000 7fc97b59e640 1 -- 172.21.15.88:0/1966532751 >> 172.21.15.88:0/1966532751 conn(0x7fc97c06d2d0 msgr2=0x7fc97c06fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:05.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.571+0000 7fc97b59e640 1 -- 172.21.15.88:0/1966532751 shutdown_connections 2023-10-16T20:51:05.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:05.571+0000 7fc97b59e640 1 -- 172.21.15.88:0/1966532751 wait complete. 2023-10-16T20:51:05.594 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-16T20:51:07.595 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:07.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.751+0000 7fa254903640 1 Processor -- start 2023-10-16T20:51:07.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.751+0000 7fa254903640 1 -- start start 2023-10-16T20:51:07.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.755+0000 7fa254903640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa2501537c0 0x7fa25015c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:07.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.755+0000 7fa254903640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25015cdb0 0x7fa2500c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:07.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.755+0000 7fa254903640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2500cae10 con 0x7fa2500ca2f0 2023-10-16T20:51:07.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.755+0000 7fa254903640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2500cac30 con 0x7fa2501537c0 2023-10-16T20:51:07.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.755+0000 7fa254903640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25015d2b0 con 0x7fa25015cdb0 2023-10-16T20:51:07.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.755+0000 7fa24f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa2501537c0 0x7fa25015c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:07.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.755+0000 7fa24f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa2501537c0 0x7fa25015c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39380/0 (socket says 172.21.15.88:39380) 2023-10-16T20:51:07.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.755+0000 7fa24f7fe640 1 -- 172.21.15.88:0/525881669 learned_addr learned my addr 172.21.15.88:0/525881669 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:07.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.755+0000 7fa24effd640 1 --2- 172.21.15.88:0/525881669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25015cdb0 0x7fa2500c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:07.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.755+0000 7fa24f7fe640 1 -- 172.21.15.88:0/525881669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25015cdb0 msgr2=0x7fa2500c4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:07.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.755+0000 7fa24f7fe640 1 --2- 172.21.15.88:0/525881669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25015cdb0 0x7fa2500c4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:07.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.755+0000 7fa24f7fe640 1 -- 172.21.15.88:0/525881669 >> v1:172.21.15.191:6789/0 conn(0x7fa2500ca2f0 legacy=0x7fa2500cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:07.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.755+0000 7fa24f7fe640 1 -- 172.21.15.88:0/525881669 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25015d590 con 0x7fa2501537c0 2023-10-16T20:51:07.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.759+0000 7fa24f7fe640 1 --2- 172.21.15.88:0/525881669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2501537c0 0x7fa25015c870 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa240009870 tx=0x7fa24001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b43f69428d5af3c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:07.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.759+0000 7fa24e7fc640 1 -- 172.21.15.88:0/525881669 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2400a89b0 con 0x7fa2501537c0 2023-10-16T20:51:07.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.759+0000 7fa24e7fc640 1 -- 172.21.15.88:0/525881669 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2400a8b10 con 0x7fa2501537c0 2023-10-16T20:51:07.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.759+0000 7fa24e7fc640 1 -- 172.21.15.88:0/525881669 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2400a8ea0 con 0x7fa2501537c0 2023-10-16T20:51:07.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.759+0000 7fa254903640 1 -- 172.21.15.88:0/525881669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2501537c0 msgr2=0x7fa25015c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:07.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.759+0000 7fa254903640 1 --2- 172.21.15.88:0/525881669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2501537c0 0x7fa25015c870 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa240009870 tx=0x7fa24001ecf0 comp rx=0 tx=0).stop 2023-10-16T20:51:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.759+0000 7fa254903640 1 -- 172.21.15.88:0/525881669 shutdown_connections 2023-10-16T20:51:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.759+0000 7fa254903640 1 --2- 172.21.15.88:0/525881669 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25015cdb0 0x7fa2500c4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.759+0000 7fa254903640 1 --2- 172.21.15.88:0/525881669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2501537c0 0x7fa25015c870 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.759+0000 7fa254903640 1 -- 172.21.15.88:0/525881669 >> 172.21.15.88:0/525881669 conn(0x7fa25006c170 msgr2=0x7fa2500715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.759+0000 7fa254903640 1 -- 172.21.15.88:0/525881669 shutdown_connections 2023-10-16T20:51:07.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.759+0000 7fa254903640 1 -- 172.21.15.88:0/525881669 wait complete. 2023-10-16T20:51:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.759+0000 7fa254903640 1 Processor -- start 2023-10-16T20:51:07.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.759+0000 7fa254903640 1 -- start start 2023-10-16T20:51:07.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa254903640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2500ca2f0 0x7fa2501ef620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:07.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa254903640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2501537c0 0x7fa250150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:07.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa254903640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25015cdb0 0x7fa25014b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:07.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa254903640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2500cf0e0 con 0x7fa2500ca2f0 2023-10-16T20:51:07.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa254903640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2501eec10 con 0x7fa2501537c0 2023-10-16T20:51:07.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa254903640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2501efb60 con 0x7fa25015cdb0 2023-10-16T20:51:07.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa24f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2500ca2f0 0x7fa2501ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:07.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa24f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2500ca2f0 0x7fa2501ef620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39390/0 (socket says 172.21.15.88:39390) 2023-10-16T20:51:07.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa24f7fe640 1 -- 172.21.15.88:0/3022744057 learned_addr learned my addr 172.21.15.88:0/3022744057 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:07.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa24effd640 1 --2- 172.21.15.88:0/3022744057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2501537c0 0x7fa250150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:07.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa24ffff640 1 --2- 172.21.15.88:0/3022744057 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25015cdb0 0x7fa25014b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:07.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa24f7fe640 1 -- 172.21.15.88:0/3022744057 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25015cdb0 msgr2=0x7fa25014b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:07.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa24f7fe640 1 --2- 172.21.15.88:0/3022744057 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25015cdb0 0x7fa25014b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:07.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa24f7fe640 1 -- 172.21.15.88:0/3022744057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2501537c0 msgr2=0x7fa250150b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:07.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa24f7fe640 1 --2- 172.21.15.88:0/3022744057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2501537c0 0x7fa250150b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:07.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa24f7fe640 1 -- 172.21.15.88:0/3022744057 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2400094d0 con 0x7fa2500ca2f0 2023-10-16T20:51:07.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa24f7fe640 1 --2- 172.21.15.88:0/3022744057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2500ca2f0 0x7fa2501ef620 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa2400b3040 tx=0x7fa24001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:07.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa237fff640 1 -- 172.21.15.88:0/3022744057 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2400ba600 con 0x7fa2500ca2f0 2023-10-16T20:51:07.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa237fff640 1 -- 172.21.15.88:0/3022744057 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2400ba780 con 0x7fa2500ca2f0 2023-10-16T20:51:07.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa237fff640 1 -- 172.21.15.88:0/3022744057 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2400bab30 con 0x7fa2500ca2f0 2023-10-16T20:51:07.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa254903640 1 -- 172.21.15.88:0/3022744057 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2500ceb20 con 0x7fa2500ca2f0 2023-10-16T20:51:07.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa254903640 1 -- 172.21.15.88:0/3022744057 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2501eee20 con 0x7fa2500ca2f0 2023-10-16T20:51:07.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa254903640 1 -- 172.21.15.88:0/3022744057 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25004e9c0 con 0x7fa2500ca2f0 2023-10-16T20:51:07.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.763+0000 7fa237fff640 1 -- 172.21.15.88:0/3022744057 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa2400bac90 con 0x7fa2500ca2f0 2023-10-16T20:51:07.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.767+0000 7fa237fff640 1 --2- 172.21.15.88:0/3022744057 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa23c0664d0 0x7fa23c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:07.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.767+0000 7fa237fff640 1 -- 172.21.15.88:0/3022744057 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7fa240122800 con 0x7fa2500ca2f0 2023-10-16T20:51:07.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.767+0000 7fa24effd640 1 --2- 172.21.15.88:0/3022744057 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa23c0664d0 0x7fa23c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:07.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.767+0000 7fa24effd640 1 --2- 172.21.15.88:0/3022744057 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa23c0664d0 0x7fa23c068990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa2501568a0 tx=0x7fa244008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:07.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:07.771+0000 7fa237fff640 1 -- 172.21.15.88:0/3022744057 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa2400ef850 con 0x7fa2500ca2f0 2023-10-16T20:51:08.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:08.003+0000 7fa254903640 1 -- 172.21.15.88:0/3022744057 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa25004e9c0 con 0x7fa2500ca2f0 2023-10-16T20:51:08.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:08.003+0000 7fa237fff640 1 -- 172.21.15.88:0/3022744057 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6567 (secure 0 0 0) 0x7fa2400b5090 con 0x7fa2500ca2f0 2023-10-16T20:51:08.007 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:08.007 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:49:52.334480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:08.007 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 80 2023-10-16T20:51:08.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:08.007+0000 7fa254903640 1 -- 172.21.15.88:0/3022744057 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa23c0664d0 msgr2=0x7fa23c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:08.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:08.007+0000 7fa254903640 1 --2- 172.21.15.88:0/3022744057 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa23c0664d0 0x7fa23c068990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa2501568a0 tx=0x7fa244008040 comp rx=0 tx=0).stop 2023-10-16T20:51:08.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:08.007+0000 7fa254903640 1 -- 172.21.15.88:0/3022744057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2500ca2f0 msgr2=0x7fa2501ef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:08.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:08.007+0000 7fa254903640 1 --2- 172.21.15.88:0/3022744057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2500ca2f0 0x7fa2501ef620 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa2400b3040 tx=0x7fa24001ba90 comp rx=0 tx=0).stop 2023-10-16T20:51:08.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:08.007+0000 7fa254903640 1 -- 172.21.15.88:0/3022744057 shutdown_connections 2023-10-16T20:51:08.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:08.007+0000 7fa254903640 1 --2- 172.21.15.88:0/3022744057 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa23c0664d0 0x7fa23c068990 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:08.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:08.007+0000 7fa254903640 1 --2- 172.21.15.88:0/3022744057 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa25015cdb0 0x7fa25014b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:08.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:08.007+0000 7fa254903640 1 --2- 172.21.15.88:0/3022744057 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2501537c0 0x7fa250150b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:08.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:08.007+0000 7fa254903640 1 --2- 172.21.15.88:0/3022744057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2500ca2f0 0x7fa2501ef620 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:08.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:08.007+0000 7fa254903640 1 -- 172.21.15.88:0/3022744057 >> 172.21.15.88:0/3022744057 conn(0x7fa25006c170 msgr2=0x7fa2500c4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:08.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:08.007+0000 7fa254903640 1 -- 172.21.15.88:0/3022744057 shutdown_connections 2023-10-16T20:51:08.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:08.007+0000 7fa254903640 1 -- 172.21.15.88:0/3022744057 wait complete. 2023-10-16T20:51:08.029 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-16T20:51:10.030 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:10.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.187+0000 7fa6567d9640 1 Processor -- start 2023-10-16T20:51:10.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.191+0000 7fa6567d9640 1 -- start start 2023-10-16T20:51:10.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.191+0000 7fa6567d9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa65014fd10 0x7fa650070c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:10.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.191+0000 7fa6567d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa65006f710 0x7fa65006fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:10.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.191+0000 7fa6567d9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa650156430 con 0x7fa650071170 2023-10-16T20:51:10.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.191+0000 7fa6567d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa650070500 con 0x7fa65006f710 2023-10-16T20:51:10.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.191+0000 7fa6567d9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa650155b00 con 0x7fa65014fd10 2023-10-16T20:51:10.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.191+0000 7fa654d76640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa65006f710 0x7fa65006fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:10.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.191+0000 7fa654d76640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa65006f710 0x7fa65006fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55646/0 (socket says 172.21.15.88:55646) 2023-10-16T20:51:10.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.191+0000 7fa654d76640 1 -- 172.21.15.88:0/1038939346 learned_addr learned my addr 172.21.15.88:0/1038939346 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:10.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.191+0000 7fa64f7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa650071170 0x7fa650069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:51942/0 (socket says 172.21.15.88:51942) 2023-10-16T20:51:10.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.191+0000 7fa64ffff640 1 --2- 172.21.15.88:0/1038939346 >> v2:172.21.15.191:3300/0 conn(0x7fa65014fd10 0x7fa650070c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.191+0000 7fa654d76640 1 -- 172.21.15.88:0/1038939346 >> v2:172.21.15.191:3300/0 conn(0x7fa65014fd10 msgr2=0x7fa650070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.191+0000 7fa654d76640 1 --2- 172.21.15.88:0/1038939346 >> v2:172.21.15.191:3300/0 conn(0x7fa65014fd10 0x7fa650070c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.191+0000 7fa654d76640 1 -- 172.21.15.88:0/1038939346 >> v1:172.21.15.88:6789/0 conn(0x7fa650071170 legacy=0x7fa650069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.191+0000 7fa654d76640 1 -- 172.21.15.88:0/1038939346 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa650155de0 con 0x7fa65006f710 2023-10-16T20:51:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.191+0000 7fa64ffff640 1 --2- 172.21.15.88:0/1038939346 >> v2:172.21.15.191:3300/0 conn(0x7fa65014fd10 0x7fa650070c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:51:10.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.195+0000 7fa654d76640 1 --2- 172.21.15.88:0/1038939346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa65006f710 0x7fa65006fb40 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fa650072840 tx=0x7fa64801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=731dca62852ea64c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:10.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.195+0000 7fa64effd640 1 -- 172.21.15.88:0/1038939346 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6480af400 con 0x7fa65006f710 2023-10-16T20:51:10.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.195+0000 7fa64effd640 1 -- 172.21.15.88:0/1038939346 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6480ab050 con 0x7fa65006f710 2023-10-16T20:51:10.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.195+0000 7fa64effd640 1 -- 172.21.15.88:0/1038939346 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6480af780 con 0x7fa65006f710 2023-10-16T20:51:10.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.195+0000 7fa6567d9640 1 -- 172.21.15.88:0/1038939346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa65006f710 msgr2=0x7fa65006fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:10.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.195+0000 7fa6567d9640 1 --2- 172.21.15.88:0/1038939346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa65006f710 0x7fa65006fb40 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fa650072840 tx=0x7fa64801f1c0 comp rx=0 tx=0).stop 2023-10-16T20:51:10.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.195+0000 7fa6567d9640 1 -- 172.21.15.88:0/1038939346 shutdown_connections 2023-10-16T20:51:10.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.195+0000 7fa6567d9640 1 --2- 172.21.15.88:0/1038939346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa65006f710 0x7fa65006fb40 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:10.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.195+0000 7fa6567d9640 1 --2- 172.21.15.88:0/1038939346 >> v2:172.21.15.191:3300/0 conn(0x7fa65014fd10 0x7fa650070c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:10.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.195+0000 7fa6567d9640 1 -- 172.21.15.88:0/1038939346 >> 172.21.15.88:0/1038939346 conn(0x7fa6500cad50 msgr2=0x7fa6500cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:10.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.195+0000 7fa6567d9640 1 -- 172.21.15.88:0/1038939346 shutdown_connections 2023-10-16T20:51:10.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.195+0000 7fa6567d9640 1 -- 172.21.15.88:0/1038939346 wait complete. 2023-10-16T20:51:10.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa6567d9640 1 Processor -- start 2023-10-16T20:51:10.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa6567d9640 1 -- start start 2023-10-16T20:51:10.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa6567d9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa65006f710 0x7fa65006a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:10.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa6567d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa650071170 0x7fa6500c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:10.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa6567d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa65014fd10 0x7fa6500c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:10.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa6567d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65006a970 con 0x7fa650071170 2023-10-16T20:51:10.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa6567d9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa650149e80 con 0x7fa65006f710 2023-10-16T20:51:10.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa6567d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6500c5f00 con 0x7fa65014fd10 2023-10-16T20:51:10.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa64f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa650071170 0x7fa6500c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:10.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa64f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa650071170 0x7fa6500c9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55652/0 (socket says 172.21.15.88:55652) 2023-10-16T20:51:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa64f7fe640 1 -- 172.21.15.88:0/4036310310 learned_addr learned my addr 172.21.15.88:0/4036310310 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa64ffff640 1 --2- 172.21.15.88:0/4036310310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa65006f710 0x7fa65006a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa654d76640 1 --2- 172.21.15.88:0/4036310310 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa65014fd10 0x7fa6500c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa64f7fe640 1 -- 172.21.15.88:0/4036310310 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa65014fd10 msgr2=0x7fa6500c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa64f7fe640 1 --2- 172.21.15.88:0/4036310310 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa65014fd10 0x7fa6500c9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa64f7fe640 1 -- 172.21.15.88:0/4036310310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa65006f710 msgr2=0x7fa65006a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa64f7fe640 1 --2- 172.21.15.88:0/4036310310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa65006f710 0x7fa65006a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa64f7fe640 1 -- 172.21.15.88:0/4036310310 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa648009b30 con 0x7fa650071170 2023-10-16T20:51:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa64f7fe640 1 --2- 172.21.15.88:0/4036310310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa650071170 0x7fa6500c9930 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa64400c500 tx=0x7fa64400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa64cff9640 1 -- 172.21.15.88:0/4036310310 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa644007500 con 0x7fa650071170 2023-10-16T20:51:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa64cff9640 1 -- 172.21.15.88:0/4036310310 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa644007660 con 0x7fa650071170 2023-10-16T20:51:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa6567d9640 1 -- 172.21.15.88:0/4036310310 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65014aca0 con 0x7fa650071170 2023-10-16T20:51:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa64cff9640 1 -- 172.21.15.88:0/4036310310 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa644004a10 con 0x7fa650071170 2023-10-16T20:51:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.199+0000 7fa6567d9640 1 -- 172.21.15.88:0/4036310310 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65014a370 con 0x7fa650071170 2023-10-16T20:51:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.203+0000 7fa6567d9640 1 -- 172.21.15.88:0/4036310310 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa650074700 con 0x7fa650071170 2023-10-16T20:51:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.203+0000 7fa64cff9640 1 -- 172.21.15.88:0/4036310310 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa644004b70 con 0x7fa650071170 2023-10-16T20:51:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.203+0000 7fa64cff9640 1 --2- 172.21.15.88:0/4036310310 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa63c0664a0 0x7fa63c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.203+0000 7fa64cff9640 1 -- 172.21.15.88:0/4036310310 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7fa644083830 con 0x7fa650071170 2023-10-16T20:51:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.203+0000 7fa64ffff640 1 --2- 172.21.15.88:0/4036310310 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa63c0664a0 0x7fa63c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:10.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.203+0000 7fa64ffff640 1 --2- 172.21.15.88:0/4036310310 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa63c0664a0 0x7fa63c068960 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa6400045f0 tx=0x7fa640008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:10.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.207+0000 7fa64cff9640 1 -- 172.21.15.88:0/4036310310 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa644050880 con 0x7fa650071170 2023-10-16T20:51:10.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.403+0000 7fa6567d9640 1 -- 172.21.15.88:0/4036310310 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6500c7de0 con 0x7fa650071170 2023-10-16T20:51:10.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.403+0000 7fa64cff9640 1 -- 172.21.15.88:0/4036310310 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6567 (secure 0 0 0) 0x7fa644053ea0 con 0x7fa650071170 2023-10-16T20:51:10.404 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:10.405 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:49:52.334480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:10.405 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 80 2023-10-16T20:51:10.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.403+0000 7fa6567d9640 1 -- 172.21.15.88:0/4036310310 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa63c0664a0 msgr2=0x7fa63c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:10.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.403+0000 7fa6567d9640 1 --2- 172.21.15.88:0/4036310310 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa63c0664a0 0x7fa63c068960 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa6400045f0 tx=0x7fa640008040 comp rx=0 tx=0).stop 2023-10-16T20:51:10.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.403+0000 7fa6567d9640 1 -- 172.21.15.88:0/4036310310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa650071170 msgr2=0x7fa6500c9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:10.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.403+0000 7fa6567d9640 1 --2- 172.21.15.88:0/4036310310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa650071170 0x7fa6500c9930 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa64400c500 tx=0x7fa64400c9c0 comp rx=0 tx=0).stop 2023-10-16T20:51:10.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.407+0000 7fa6567d9640 1 -- 172.21.15.88:0/4036310310 shutdown_connections 2023-10-16T20:51:10.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.407+0000 7fa6567d9640 1 --2- 172.21.15.88:0/4036310310 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa63c0664a0 0x7fa63c068960 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:10.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.407+0000 7fa6567d9640 1 --2- 172.21.15.88:0/4036310310 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa65014fd10 0x7fa6500c9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:10.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.407+0000 7fa6567d9640 1 --2- 172.21.15.88:0/4036310310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa650071170 0x7fa6500c9930 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:10.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.407+0000 7fa6567d9640 1 --2- 172.21.15.88:0/4036310310 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa65006f710 0x7fa65006a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:10.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.407+0000 7fa6567d9640 1 -- 172.21.15.88:0/4036310310 >> 172.21.15.88:0/4036310310 conn(0x7fa6500cad50 msgr2=0x7fa65014edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:10.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.407+0000 7fa6567d9640 1 -- 172.21.15.88:0/4036310310 shutdown_connections 2023-10-16T20:51:10.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:10.407+0000 7fa6567d9640 1 -- 172.21.15.88:0/4036310310 wait complete. 2023-10-16T20:51:10.428 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-16T20:51:12.429 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:12.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.587+0000 7f39ddff2640 1 Processor -- start 2023-10-16T20:51:12.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.587+0000 7f39ddff2640 1 -- start start 2023-10-16T20:51:12.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.591+0000 7f39ddff2640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f39d80c9af0 0x7f39d80cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:12.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.591+0000 7f39ddff2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d80c9fc0 con 0x7f39d80b1a60 2023-10-16T20:51:12.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.591+0000 7f39ddff2640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d80c47d0 con 0x7f39d80cb580 2023-10-16T20:51:12.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.591+0000 7f39ddff2640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39d80b20c0 con 0x7f39d80c9af0 2023-10-16T20:51:12.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.591+0000 7f39dd7f1640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f39d80b1a60 0x7f39d8163b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:51950/0 (socket says 172.21.15.88:51950) 2023-10-16T20:51:12.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.591+0000 7f39dd7f1640 1 -- 172.21.15.88:0/3370306870 learned_addr learned my addr 172.21.15.88:0/3370306870 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:51:12.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.591+0000 7f39d7fff640 1 --2- 172.21.15.88:0/3370306870 >> v2:172.21.15.191:3300/0 conn(0x7f39d80c9af0 0x7f39d80cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:12.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.591+0000 7f39d77fe640 1 -- 172.21.15.88:0/3370306870 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117607752 0 0) 0x7f39d80c9fc0 con 0x7f39d80b1a60 2023-10-16T20:51:12.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.591+0000 7f39d77fe640 1 -- 172.21.15.88:0/3370306870 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c8003530 con 0x7f39d80b1a60 2023-10-16T20:51:12.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.591+0000 7f39d7fff640 1 -- 172.21.15.88:0/3370306870 >> v1:172.21.15.191:6789/0 conn(0x7f39d80cb580 legacy=0x7f39d8159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:12.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.591+0000 7f39d7fff640 1 -- 172.21.15.88:0/3370306870 >> v1:172.21.15.88:6789/0 conn(0x7f39d80b1a60 legacy=0x7f39d8163b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:12.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.591+0000 7f39d7fff640 1 -- 172.21.15.88:0/3370306870 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d80c4ab0 con 0x7f39d80c9af0 2023-10-16T20:51:12.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.595+0000 7f39d7fff640 1 --2- 172.21.15.88:0/3370306870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39d80c9af0 0x7f39d80cb960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f39c40097c0 tx=0x7f39c401c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d92217c536085ece server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:12.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.595+0000 7f39d77fe640 1 -- 172.21.15.88:0/3370306870 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c40ab600 con 0x7f39d80c9af0 2023-10-16T20:51:12.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.595+0000 7f39d77fe640 1 -- 172.21.15.88:0/3370306870 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c40a7050 con 0x7f39d80c9af0 2023-10-16T20:51:12.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.595+0000 7f39d77fe640 1 -- 172.21.15.88:0/3370306870 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c40ab980 con 0x7f39d80c9af0 2023-10-16T20:51:12.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.595+0000 7f39ddff2640 1 -- 172.21.15.88:0/3370306870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39d80c9af0 msgr2=0x7f39d80cb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:12.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.595+0000 7f39ddff2640 1 --2- 172.21.15.88:0/3370306870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39d80c9af0 0x7f39d80cb960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f39c40097c0 tx=0x7f39c401c3f0 comp rx=0 tx=0).stop 2023-10-16T20:51:12.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.595+0000 7f39ddff2640 1 -- 172.21.15.88:0/3370306870 shutdown_connections 2023-10-16T20:51:12.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.595+0000 7f39ddff2640 1 --2- 172.21.15.88:0/3370306870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39d80c9af0 0x7f39d80cb960 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:12.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.595+0000 7f39ddff2640 1 -- 172.21.15.88:0/3370306870 >> 172.21.15.88:0/3370306870 conn(0x7f39d806bb40 msgr2=0x7f39d806bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:12.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.595+0000 7f39ddff2640 1 -- 172.21.15.88:0/3370306870 shutdown_connections 2023-10-16T20:51:12.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.595+0000 7f39ddff2640 1 -- 172.21.15.88:0/3370306870 wait complete. 2023-10-16T20:51:12.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39ddff2640 1 Processor -- start 2023-10-16T20:51:12.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39ddff2640 1 -- start start 2023-10-16T20:51:12.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39ddff2640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39d80b1a60 0x7f39d80c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:12.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39ddff2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39d80c9af0 0x7f39d80c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:12.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39ddff2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39d80cb580 0x7f39d8157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:12.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39ddff2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d81479a0 con 0x7f39d80c9af0 2023-10-16T20:51:12.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39ddff2640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d80c61d0 con 0x7f39d80cb580 2023-10-16T20:51:12.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39ddff2640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d8156ec0 con 0x7f39d80b1a60 2023-10-16T20:51:12.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39d7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39d80c9af0 0x7f39d80c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:12.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39d7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39d80c9af0 0x7f39d80c76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55664/0 (socket says 172.21.15.88:55664) 2023-10-16T20:51:12.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39d7fff640 1 -- 172.21.15.88:0/2963662041 learned_addr learned my addr 172.21.15.88:0/2963662041 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:12.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39dd7f1640 1 --2- 172.21.15.88:0/2963662041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39d80cb580 0x7f39d8157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:12.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39dcff0640 1 --2- 172.21.15.88:0/2963662041 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39d80b1a60 0x7f39d80c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:12.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39d7fff640 1 -- 172.21.15.88:0/2963662041 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39d80b1a60 msgr2=0x7f39d80c87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:12.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39d7fff640 1 --2- 172.21.15.88:0/2963662041 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39d80b1a60 0x7f39d80c87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:12.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39d7fff640 1 -- 172.21.15.88:0/2963662041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39d80cb580 msgr2=0x7f39d8157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:12.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39d7fff640 1 --2- 172.21.15.88:0/2963662041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39d80cb580 0x7f39d8157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:12.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39d7fff640 1 -- 172.21.15.88:0/2963662041 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c4009420 con 0x7f39d80c9af0 2023-10-16T20:51:12.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39d7fff640 1 --2- 172.21.15.88:0/2963662041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39d80c9af0 0x7f39d80c76f0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f39c4019000 tx=0x7f39c4019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:12.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39d57fa640 1 -- 172.21.15.88:0/2963662041 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c40b4410 con 0x7f39d80c9af0 2023-10-16T20:51:12.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39d57fa640 1 -- 172.21.15.88:0/2963662041 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c40b4590 con 0x7f39d80c9af0 2023-10-16T20:51:12.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39ddff2640 1 -- 172.21.15.88:0/2963662041 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d81571a0 con 0x7f39d80c9af0 2023-10-16T20:51:12.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39d57fa640 1 -- 172.21.15.88:0/2963662041 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c40b4940 con 0x7f39d80c9af0 2023-10-16T20:51:12.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39ddff2640 1 -- 172.21.15.88:0/2963662041 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d8157760 con 0x7f39d80c9af0 2023-10-16T20:51:12.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.599+0000 7f39ddff2640 1 -- 172.21.15.88:0/2963662041 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d80ca2c0 con 0x7f39d80c9af0 2023-10-16T20:51:12.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.603+0000 7f39d57fa640 1 -- 172.21.15.88:0/2963662041 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f39c40b2040 con 0x7f39d80c9af0 2023-10-16T20:51:12.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.603+0000 7f39d57fa640 1 --2- 172.21.15.88:0/2963662041 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f39bc0664a0 0x7f39bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:12.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.603+0000 7f39d57fa640 1 -- 172.21.15.88:0/2963662041 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7f39c411e330 con 0x7f39d80c9af0 2023-10-16T20:51:12.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.603+0000 7f39dcff0640 1 --2- 172.21.15.88:0/2963662041 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f39bc0664a0 0x7f39bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:12.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.603+0000 7f39dcff0640 1 --2- 172.21.15.88:0/2963662041 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f39bc0664a0 0x7f39bc068960 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f39cc007fd0 tx=0x7f39cc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:12.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.607+0000 7f39d57fa640 1 -- 172.21.15.88:0/2963662041 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f39c40eb2f0 con 0x7f39d80c9af0 2023-10-16T20:51:12.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.799+0000 7f39ddff2640 1 -- 172.21.15.88:0/2963662041 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39d804e9c0 con 0x7f39d80c9af0 2023-10-16T20:51:12.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.799+0000 7f39d57fa640 1 -- 172.21.15.88:0/2963662041 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6567 (secure 0 0 0) 0x7f39c40ee910 con 0x7f39d80c9af0 2023-10-16T20:51:12.800 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:12.800 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:49:52.334480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:12.801 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 80 2023-10-16T20:51:12.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.799+0000 7f39ddff2640 1 -- 172.21.15.88:0/2963662041 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f39bc0664a0 msgr2=0x7f39bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:12.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.799+0000 7f39ddff2640 1 --2- 172.21.15.88:0/2963662041 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f39bc0664a0 0x7f39bc068960 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f39cc007fd0 tx=0x7f39cc009040 comp rx=0 tx=0).stop 2023-10-16T20:51:12.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.799+0000 7f39ddff2640 1 -- 172.21.15.88:0/2963662041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39d80c9af0 msgr2=0x7f39d80c76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:12.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.799+0000 7f39ddff2640 1 --2- 172.21.15.88:0/2963662041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39d80c9af0 0x7f39d80c76f0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f39c4019000 tx=0x7f39c4019190 comp rx=0 tx=0).stop 2023-10-16T20:51:12.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.799+0000 7f39ddff2640 1 -- 172.21.15.88:0/2963662041 shutdown_connections 2023-10-16T20:51:12.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.799+0000 7f39ddff2640 1 --2- 172.21.15.88:0/2963662041 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f39bc0664a0 0x7f39bc068960 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:12.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.799+0000 7f39ddff2640 1 --2- 172.21.15.88:0/2963662041 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39d80cb580 0x7f39d8157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:12.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.799+0000 7f39ddff2640 1 --2- 172.21.15.88:0/2963662041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39d80c9af0 0x7f39d80c76f0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:12.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.799+0000 7f39ddff2640 1 --2- 172.21.15.88:0/2963662041 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39d80b1a60 0x7f39d80c87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:12.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.799+0000 7f39ddff2640 1 -- 172.21.15.88:0/2963662041 >> 172.21.15.88:0/2963662041 conn(0x7f39d806bb40 msgr2=0x7f39d8152890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:12.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.799+0000 7f39ddff2640 1 -- 172.21.15.88:0/2963662041 shutdown_connections 2023-10-16T20:51:12.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:12.799+0000 7f39ddff2640 1 -- 172.21.15.88:0/2963662041 wait complete. 2023-10-16T20:51:12.821 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-16T20:51:14.822 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:14.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.971+0000 7f3bfd90f640 1 Processor -- start 2023-10-16T20:51:14.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.971+0000 7f3bfd90f640 1 -- start start 2023-10-16T20:51:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.971+0000 7f3bfd90f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3bf80c9bb0 0x7f3bf80c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.971+0000 7f3bfd90f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3bf80c41f0 0x7f3bf80cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.971+0000 7f3bfd90f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf8153b50 con 0x7f3bf8153770 2023-10-16T20:51:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.971+0000 7f3bfd90f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf80cc250 con 0x7f3bf80c9bb0 2023-10-16T20:51:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.971+0000 7f3bfd90f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf80cc670 con 0x7f3bf80c41f0 2023-10-16T20:51:14.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.971+0000 7f3bf67fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3bf80c9bb0 0x7f3bf80c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:14.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.971+0000 7f3bf67fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3bf80c9bb0 0x7f3bf80c3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55666/0 (socket says 172.21.15.88:55666) 2023-10-16T20:51:14.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.971+0000 7f3bf67fc640 1 -- 172.21.15.88:0/3135753074 learned_addr learned my addr 172.21.15.88:0/3135753074 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:14.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.971+0000 7f3bf77fe640 1 --2- 172.21.15.88:0/3135753074 >> v2:172.21.15.191:3300/0 conn(0x7f3bf80c41f0 0x7f3bf80cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:14.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.971+0000 7f3bf67fc640 1 -- 172.21.15.88:0/3135753074 >> v2:172.21.15.191:3300/0 conn(0x7f3bf80c41f0 msgr2=0x7f3bf80cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:14.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.971+0000 7f3bf67fc640 1 --2- 172.21.15.88:0/3135753074 >> v2:172.21.15.191:3300/0 conn(0x7f3bf80c41f0 0x7f3bf80cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:14.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.971+0000 7f3bf67fc640 1 -- 172.21.15.88:0/3135753074 >> v1:172.21.15.191:6789/0 conn(0x7f3bf8153770 legacy=0x7f3bf8159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:14.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.971+0000 7f3bf67fc640 1 -- 172.21.15.88:0/3135753074 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf80cb2f0 con 0x7f3bf80c9bb0 2023-10-16T20:51:14.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.975+0000 7f3bf67fc640 1 --2- 172.21.15.88:0/3135753074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf80c9bb0 0x7f3bf80c3cb0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80b19e0 tx=0x7f3bec01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b2c10fd4770ba6e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:14.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.975+0000 7f3bf5ffb640 1 -- 172.21.15.88:0/3135753074 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec0ac630 con 0x7f3bf80c9bb0 2023-10-16T20:51:14.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.975+0000 7f3bf5ffb640 1 -- 172.21.15.88:0/3135753074 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bec0ac790 con 0x7f3bf80c9bb0 2023-10-16T20:51:14.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.975+0000 7f3bf5ffb640 1 -- 172.21.15.88:0/3135753074 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec0acb20 con 0x7f3bf80c9bb0 2023-10-16T20:51:14.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.975+0000 7f3bfd90f640 1 -- 172.21.15.88:0/3135753074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf80c9bb0 msgr2=0x7f3bf80c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:14.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.975+0000 7f3bfd90f640 1 --2- 172.21.15.88:0/3135753074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf80c9bb0 0x7f3bf80c3cb0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80b19e0 tx=0x7f3bec01c4c0 comp rx=0 tx=0).stop 2023-10-16T20:51:14.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.975+0000 7f3bfd90f640 1 -- 172.21.15.88:0/3135753074 shutdown_connections 2023-10-16T20:51:14.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.975+0000 7f3bfd90f640 1 --2- 172.21.15.88:0/3135753074 >> v2:172.21.15.191:3300/0 conn(0x7f3bf80c41f0 0x7f3bf80cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:14.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.975+0000 7f3bfd90f640 1 --2- 172.21.15.88:0/3135753074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf80c9bb0 0x7f3bf80c3cb0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:14.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.975+0000 7f3bfd90f640 1 -- 172.21.15.88:0/3135753074 >> 172.21.15.88:0/3135753074 conn(0x7f3bf80715c0 msgr2=0x7f3bf80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:14.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.975+0000 7f3bfd90f640 1 -- 172.21.15.88:0/3135753074 shutdown_connections 2023-10-16T20:51:14.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.975+0000 7f3bfd90f640 1 -- 172.21.15.88:0/3135753074 wait complete. 2023-10-16T20:51:14.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.975+0000 7f3bfd90f640 1 Processor -- start 2023-10-16T20:51:14.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bfd90f640 1 -- start start 2023-10-16T20:51:14.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bfd90f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf80c41f0 0x7f3bf814da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:14.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bfd90f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bf80c9bb0 0x7f3bf81f5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:14.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bfd90f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3bf8153770 0x7f3bf8145b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:14.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bfd90f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf80c8a70 con 0x7f3bf80c41f0 2023-10-16T20:51:14.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bfd90f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf81f5e80 con 0x7f3bf80c9bb0 2023-10-16T20:51:14.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bfd90f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf80c69f0 con 0x7f3bf8153770 2023-10-16T20:51:14.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bf6ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf80c41f0 0x7f3bf814da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:14.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bf6ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf80c41f0 0x7f3bf814da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55668/0 (socket says 172.21.15.88:55668) 2023-10-16T20:51:14.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bf6ffd640 1 -- 172.21.15.88:0/3204426415 learned_addr learned my addr 172.21.15.88:0/3204426415 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:14.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bf77fe640 1 --2- 172.21.15.88:0/3204426415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3bf8153770 0x7f3bf8145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:14.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bf67fc640 1 --2- 172.21.15.88:0/3204426415 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bf80c9bb0 0x7f3bf81f5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:14.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bf6ffd640 1 -- 172.21.15.88:0/3204426415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3bf8153770 msgr2=0x7f3bf8145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:14.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bf6ffd640 1 --2- 172.21.15.88:0/3204426415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3bf8153770 0x7f3bf8145b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:14.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bf6ffd640 1 -- 172.21.15.88:0/3204426415 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bf80c9bb0 msgr2=0x7f3bf81f5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:14.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bf6ffd640 1 --2- 172.21.15.88:0/3204426415 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bf80c9bb0 0x7f3bf81f5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:14.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bf6ffd640 1 -- 172.21.15.88:0/3204426415 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bec0094d0 con 0x7f3bf80c41f0 2023-10-16T20:51:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bf67fc640 1 --2- 172.21.15.88:0/3204426415 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bf80c9bb0 0x7f3bf81f5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:51:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bf77fe640 1 --2- 172.21.15.88:0/3204426415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3bf8153770 0x7f3bf8145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:51:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bf6ffd640 1 --2- 172.21.15.88:0/3204426415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf80c41f0 0x7f3bf814da20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3be800b3f0 tx=0x7f3be800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bdffff640 1 -- 172.21.15.88:0/3204426415 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be8011830 con 0x7f3bf80c41f0 2023-10-16T20:51:14.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bdffff640 1 -- 172.21.15.88:0/3204426415 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be80119b0 con 0x7f3bf80c41f0 2023-10-16T20:51:14.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bdffff640 1 -- 172.21.15.88:0/3204426415 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be8011d60 con 0x7f3bf80c41f0 2023-10-16T20:51:14.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bfd90f640 1 -- 172.21.15.88:0/3204426415 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf8146040 con 0x7f3bf80c41f0 2023-10-16T20:51:14.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bfd90f640 1 -- 172.21.15.88:0/3204426415 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf814e0d0 con 0x7f3bf80c41f0 2023-10-16T20:51:14.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.979+0000 7f3bfd90f640 1 -- 172.21.15.88:0/3204426415 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf8074700 con 0x7f3bf80c41f0 2023-10-16T20:51:14.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.983+0000 7f3bdffff640 1 -- 172.21.15.88:0/3204426415 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3be801a410 con 0x7f3bf80c41f0 2023-10-16T20:51:14.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.983+0000 7f3bdffff640 1 --2- 172.21.15.88:0/3204426415 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3be40664d0 0x7f3be4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:14.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.983+0000 7f3bdffff640 1 -- 172.21.15.88:0/3204426415 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7f3be80841c0 con 0x7f3bf80c41f0 2023-10-16T20:51:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.983+0000 7f3bf67fc640 1 --2- 172.21.15.88:0/3204426415 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3be40664d0 0x7f3be4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.983+0000 7f3bf67fc640 1 --2- 172.21.15.88:0/3204426415 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3be40664d0 0x7f3be4068990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0ae000 tx=0x7f3bec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:14.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:14.987+0000 7f3bdffff640 1 -- 172.21.15.88:0/3204426415 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3be80510c0 con 0x7f3bf80c41f0 2023-10-16T20:51:15.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:15.167+0000 7f3bfd90f640 1 -- 172.21.15.88:0/3204426415 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bf80cbb90 con 0x7f3bf80c41f0 2023-10-16T20:51:15.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:15.167+0000 7f3bdffff640 1 -- 172.21.15.88:0/3204426415 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6567 (secure 0 0 0) 0x7f3be80546e0 con 0x7f3bf80c41f0 2023-10-16T20:51:15.170 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:15.170 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:49:52.334480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:15.171 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 80 2023-10-16T20:51:15.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:15.167+0000 7f3bfd90f640 1 -- 172.21.15.88:0/3204426415 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3be40664d0 msgr2=0x7f3be4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:15.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:15.167+0000 7f3bfd90f640 1 --2- 172.21.15.88:0/3204426415 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3be40664d0 0x7f3be4068990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0ae000 tx=0x7f3bec008040 comp rx=0 tx=0).stop 2023-10-16T20:51:15.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:15.167+0000 7f3bfd90f640 1 -- 172.21.15.88:0/3204426415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf80c41f0 msgr2=0x7f3bf814da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:15.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:15.167+0000 7f3bfd90f640 1 --2- 172.21.15.88:0/3204426415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf80c41f0 0x7f3bf814da20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3be800b3f0 tx=0x7f3be800b8b0 comp rx=0 tx=0).stop 2023-10-16T20:51:15.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:15.167+0000 7f3bfd90f640 1 -- 172.21.15.88:0/3204426415 shutdown_connections 2023-10-16T20:51:15.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:15.167+0000 7f3bfd90f640 1 --2- 172.21.15.88:0/3204426415 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3be40664d0 0x7f3be4068990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:15.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:15.167+0000 7f3bfd90f640 1 --2- 172.21.15.88:0/3204426415 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3bf8153770 0x7f3bf8145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:15.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:15.167+0000 7f3bfd90f640 1 --2- 172.21.15.88:0/3204426415 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3bf80c9bb0 0x7f3bf81f5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:15.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:15.167+0000 7f3bfd90f640 1 --2- 172.21.15.88:0/3204426415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf80c41f0 0x7f3bf814da20 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:15.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:15.167+0000 7f3bfd90f640 1 -- 172.21.15.88:0/3204426415 >> 172.21.15.88:0/3204426415 conn(0x7f3bf80715c0 msgr2=0x7f3bf8150d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:15.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:15.167+0000 7f3bfd90f640 1 -- 172.21.15.88:0/3204426415 shutdown_connections 2023-10-16T20:51:15.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:15.167+0000 7f3bfd90f640 1 -- 172.21.15.88:0/3204426415 wait complete. 2023-10-16T20:51:15.189 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-16T20:51:17.190 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:17.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.343+0000 7f559dc91640 1 Processor -- start 2023-10-16T20:51:17.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.343+0000 7f559dc91640 1 -- start start 2023-10-16T20:51:17.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f559dc91640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f55980c76c0 0x7f5598159450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:17.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f559dc91640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55980c8810 con 0x7f55980c7dc0 2023-10-16T20:51:17.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f559dc91640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559806b400 con 0x7f55980c81a0 2023-10-16T20:51:17.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f559dc91640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f559806b0d0 con 0x7f55980c76c0 2023-10-16T20:51:17.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55977fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f55980c7dc0 0x7f559806d650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:51952/0 (socket says 172.21.15.88:51952) 2023-10-16T20:51:17.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55977fe640 1 -- 172.21.15.88:0/1689700409 learned_addr learned my addr 172.21.15.88:0/1689700409 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:51:17.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f5596ffd640 1 --2- 172.21.15.88:0/1689700409 >> v2:172.21.15.191:3300/0 conn(0x7f55980c76c0 0x7f5598159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:17.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55967fc640 1 -- 172.21.15.88:0/1689700409 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875675101 0 0) 0x7f55980c8810 con 0x7f55980c7dc0 2023-10-16T20:51:17.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55967fc640 1 -- 172.21.15.88:0/1689700409 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5580003530 con 0x7f55980c7dc0 2023-10-16T20:51:17.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55967fc640 1 -- 172.21.15.88:0/1689700409 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3137533590 0 0) 0x7f5580003530 con 0x7f55980c7dc0 2023-10-16T20:51:17.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55967fc640 1 -- 172.21.15.88:0/1689700409 >> v2:172.21.15.191:3300/0 conn(0x7f55980c76c0 msgr2=0x7f5598159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:17.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55967fc640 1 --2- 172.21.15.88:0/1689700409 >> v2:172.21.15.191:3300/0 conn(0x7f55980c76c0 0x7f5598159450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:17.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55967fc640 1 -- 172.21.15.88:0/1689700409 >> v1:172.21.15.191:6789/0 conn(0x7f55980c81a0 legacy=0x7f559806b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:17.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55967fc640 1 -- 172.21.15.88:0/1689700409 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559806e6a0 con 0x7f55980c7dc0 2023-10-16T20:51:17.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55967fc640 1 -- 172.21.15.88:0/1689700409 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f5588001680 con 0x7f55980c7dc0 2023-10-16T20:51:17.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55967fc640 1 -- 172.21.15.88:0/1689700409 >> v1:172.21.15.88:6789/0 conn(0x7f55980c7dc0 legacy=0x7f559806d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:17.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55967fc640 1 --2- 172.21.15.88:0/1689700409 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5580003980 0x7f5580003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:17.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55967fc640 1 --2- 172.21.15.88:0/1689700409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55800047d0 0x7f5580004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:17.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55967fc640 1 --2- 172.21.15.88:0/1689700409 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5580005190 0x7f558000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:17.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55967fc640 1 -- 172.21.15.88:0/1689700409 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559806b0d0 con 0x7f55800047d0 2023-10-16T20:51:17.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55967fc640 1 -- 172.21.15.88:0/1689700409 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558000c130 con 0x7f5580005190 2023-10-16T20:51:17.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55967fc640 1 -- 172.21.15.88:0/1689700409 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558000c2f0 con 0x7f5580003980 2023-10-16T20:51:17.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f5596ffd640 1 --2- 172.21.15.88:0/1689700409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55800047d0 0x7f5580004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:17.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f55977fe640 1 --2- 172.21.15.88:0/1689700409 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5580003980 0x7f5580003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:17.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f5597fff640 1 --2- 172.21.15.88:0/1689700409 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5580005190 0x7f558000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:17.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f5596ffd640 1 -- 172.21.15.88:0/1689700409 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5580003980 msgr2=0x7f5580003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:17.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f5596ffd640 1 --2- 172.21.15.88:0/1689700409 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5580003980 0x7f5580003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:17.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f5596ffd640 1 -- 172.21.15.88:0/1689700409 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5580005190 msgr2=0x7f558000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:17.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f5596ffd640 1 --2- 172.21.15.88:0/1689700409 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5580005190 0x7f558000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:17.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f5596ffd640 1 -- 172.21.15.88:0/1689700409 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5580003530 con 0x7f55800047d0 2023-10-16T20:51:17.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.347+0000 7f5597fff640 1 -- 172.21.15.88:0/1689700409 reap_dead start 2023-10-16T20:51:17.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.351+0000 7f5596ffd640 1 --2- 172.21.15.88:0/1689700409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55800047d0 0x7f5580004c50 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f558c009ae0 tx=0x7f558c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a0177c464b7e148 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:17.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.351+0000 7f55967fc640 1 -- 172.21.15.88:0/1689700409 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c0a6430 con 0x7f55800047d0 2023-10-16T20:51:17.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.351+0000 7f55967fc640 1 -- 172.21.15.88:0/1689700409 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558c0a6590 con 0x7f55800047d0 2023-10-16T20:51:17.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.351+0000 7f55967fc640 1 -- 172.21.15.88:0/1689700409 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c0a6940 con 0x7f55800047d0 2023-10-16T20:51:17.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.351+0000 7f559dc91640 1 -- 172.21.15.88:0/1689700409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55800047d0 msgr2=0x7f5580004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:17.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.351+0000 7f559dc91640 1 --2- 172.21.15.88:0/1689700409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55800047d0 0x7f5580004c50 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f558c009ae0 tx=0x7f558c01ca50 comp rx=0 tx=0).stop 2023-10-16T20:51:17.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.351+0000 7f559dc91640 1 -- 172.21.15.88:0/1689700409 shutdown_connections 2023-10-16T20:51:17.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.351+0000 7f559dc91640 1 --2- 172.21.15.88:0/1689700409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55800047d0 0x7f5580004c50 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:17.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.351+0000 7f559dc91640 1 -- 172.21.15.88:0/1689700409 >> 172.21.15.88:0/1689700409 conn(0x7f5598071a60 msgr2=0x7f5598071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:17.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.351+0000 7f559dc91640 1 -- 172.21.15.88:0/1689700409 shutdown_connections 2023-10-16T20:51:17.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.351+0000 7f559dc91640 1 -- 172.21.15.88:0/1689700409 wait complete. 2023-10-16T20:51:17.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f559dc91640 1 Processor -- start 2023-10-16T20:51:17.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f559dc91640 1 -- start start 2023-10-16T20:51:17.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f559dc91640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55800047d0 0x7f559814f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:17.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f559dc91640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f559814c520 0x7f559814ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:17.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f559dc91640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5598154020 0x7f5598146a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:17.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f559dc91640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55980c68c0 con 0x7f5598154020 2023-10-16T20:51:17.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f559dc91640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55981f6020 con 0x7f559814c520 2023-10-16T20:51:17.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f559dc91640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55981f6300 con 0x7f55800047d0 2023-10-16T20:51:17.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f5597fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5598154020 0x7f5598146a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:17.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f5597fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5598154020 0x7f5598146a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55680/0 (socket says 172.21.15.88:55680) 2023-10-16T20:51:17.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f5597fff640 1 -- 172.21.15.88:0/2957192171 learned_addr learned my addr 172.21.15.88:0/2957192171 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:17.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f5596ffd640 1 --2- 172.21.15.88:0/2957192171 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f559814c520 0x7f559814ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:17.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f55977fe640 1 --2- 172.21.15.88:0/2957192171 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55800047d0 0x7f559814f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:17.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f5597fff640 1 -- 172.21.15.88:0/2957192171 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55800047d0 msgr2=0x7f559814f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:17.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f5597fff640 1 --2- 172.21.15.88:0/2957192171 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55800047d0 0x7f559814f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:17.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f5597fff640 1 -- 172.21.15.88:0/2957192171 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f559814c520 msgr2=0x7f559814ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:17.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f5597fff640 1 --2- 172.21.15.88:0/2957192171 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f559814c520 0x7f559814ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:17.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f5597fff640 1 -- 172.21.15.88:0/2957192171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55981f22e0 con 0x7f5598154020 2023-10-16T20:51:17.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f5597fff640 1 --2- 172.21.15.88:0/2957192171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5598154020 0x7f5598146a90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f55980719f0 tx=0x7f559000fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:17.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f557ffff640 1 -- 172.21.15.88:0/2957192171 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5590006d70 con 0x7f5598154020 2023-10-16T20:51:17.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f557ffff640 1 -- 172.21.15.88:0/2957192171 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5590003980 con 0x7f5598154020 2023-10-16T20:51:17.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f557ffff640 1 -- 172.21.15.88:0/2957192171 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5590003d30 con 0x7f5598154020 2023-10-16T20:51:17.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f559dc91640 1 -- 172.21.15.88:0/2957192171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55981f25c0 con 0x7f5598154020 2023-10-16T20:51:17.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f559dc91640 1 -- 172.21.15.88:0/2957192171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55981f1f90 con 0x7f5598154020 2023-10-16T20:51:17.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.355+0000 7f559dc91640 1 -- 172.21.15.88:0/2957192171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5598074700 con 0x7f5598154020 2023-10-16T20:51:17.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.359+0000 7f557ffff640 1 -- 172.21.15.88:0/2957192171 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5590005070 con 0x7f5598154020 2023-10-16T20:51:17.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.359+0000 7f557ffff640 1 --2- 172.21.15.88:0/2957192171 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55840664d0 0x7f5584068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:17.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.359+0000 7f557ffff640 1 -- 172.21.15.88:0/2957192171 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7f5590083840 con 0x7f5598154020 2023-10-16T20:51:17.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.359+0000 7f55977fe640 1 --2- 172.21.15.88:0/2957192171 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55840664d0 0x7f5584068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:17.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.359+0000 7f55977fe640 1 --2- 172.21.15.88:0/2957192171 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55840664d0 0x7f5584068990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5588001060 tx=0x7f5588000f90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:17.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.363+0000 7f557ffff640 1 -- 172.21.15.88:0/2957192171 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f55900506d0 con 0x7f5598154020 2023-10-16T20:51:17.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.543+0000 7f559dc91640 1 -- 172.21.15.88:0/2957192171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55981f2a70 con 0x7f5598154020 2023-10-16T20:51:17.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.543+0000 7f557ffff640 1 -- 172.21.15.88:0/2957192171 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6567 (secure 0 0 0) 0x7f5590053cf0 con 0x7f5598154020 2023-10-16T20:51:17.545 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:17.545 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:49:52.334480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:17.546 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 80 2023-10-16T20:51:17.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.543+0000 7f559dc91640 1 -- 172.21.15.88:0/2957192171 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55840664d0 msgr2=0x7f5584068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:17.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.543+0000 7f559dc91640 1 --2- 172.21.15.88:0/2957192171 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55840664d0 0x7f5584068990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5588001060 tx=0x7f5588000f90 comp rx=0 tx=0).stop 2023-10-16T20:51:17.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.543+0000 7f559dc91640 1 -- 172.21.15.88:0/2957192171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5598154020 msgr2=0x7f5598146a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:17.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.543+0000 7f559dc91640 1 --2- 172.21.15.88:0/2957192171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5598154020 0x7f5598146a90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f55980719f0 tx=0x7f559000fc70 comp rx=0 tx=0).stop 2023-10-16T20:51:17.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.543+0000 7f559dc91640 1 -- 172.21.15.88:0/2957192171 shutdown_connections 2023-10-16T20:51:17.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.543+0000 7f559dc91640 1 --2- 172.21.15.88:0/2957192171 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f55840664d0 0x7f5584068990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:17.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.543+0000 7f559dc91640 1 --2- 172.21.15.88:0/2957192171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5598154020 0x7f5598146a90 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:17.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.543+0000 7f559dc91640 1 --2- 172.21.15.88:0/2957192171 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f559814c520 0x7f559814ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:17.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.543+0000 7f559dc91640 1 --2- 172.21.15.88:0/2957192171 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f55800047d0 0x7f559814f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:17.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.543+0000 7f559dc91640 1 -- 172.21.15.88:0/2957192171 >> 172.21.15.88:0/2957192171 conn(0x7f5598071a60 msgr2=0x7f55980cbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:17.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.547+0000 7f559dc91640 1 -- 172.21.15.88:0/2957192171 shutdown_connections 2023-10-16T20:51:17.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:17.547+0000 7f559dc91640 1 -- 172.21.15.88:0/2957192171 wait complete. 2023-10-16T20:51:17.565 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-16T20:51:19.566 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:19.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f444a068640 1 Processor -- start 2023-10-16T20:51:19.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f444a068640 1 -- start start 2023-10-16T20:51:19.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f444a068640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44440cece0 0x7f44440cf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:19.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f444a068640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4444153b50 con 0x7f44440c9bb0 2023-10-16T20:51:19.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f444a068640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44440cb2f0 con 0x7f4444153770 2023-10-16T20:51:19.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f444a068640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44440cc250 con 0x7f44440cece0 2023-10-16T20:51:19.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f4442ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f44440c9bb0 0x7f44440c3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54568/0 (socket says 172.21.15.88:54568) 2023-10-16T20:51:19.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f4442ffd640 1 -- 172.21.15.88:0/4139235245 learned_addr learned my addr 172.21.15.88:0/4139235245 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:51:19.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f4443fff640 1 --2- 172.21.15.88:0/4139235245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44440cece0 0x7f44440cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:19.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f44427fc640 1 -- 172.21.15.88:0/4139235245 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874428978 0 0) 0x7f4444153b50 con 0x7f44440c9bb0 2023-10-16T20:51:19.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f44427fc640 1 -- 172.21.15.88:0/4139235245 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f442c003530 con 0x7f44440c9bb0 2023-10-16T20:51:19.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f44427fc640 1 -- 172.21.15.88:0/4139235245 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3566019732 0 0) 0x7f442c003530 con 0x7f44440c9bb0 2023-10-16T20:51:19.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f44427fc640 1 -- 172.21.15.88:0/4139235245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44440cece0 msgr2=0x7f44440cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:19.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f44427fc640 1 --2- 172.21.15.88:0/4139235245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44440cece0 0x7f44440cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:19.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f44427fc640 1 -- 172.21.15.88:0/4139235245 >> v1:172.21.15.191:6789/0 conn(0x7f4444153770 legacy=0x7f4444159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:19.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f44427fc640 1 -- 172.21.15.88:0/4139235245 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44440cb5d0 con 0x7f44440c9bb0 2023-10-16T20:51:19.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f44427fc640 1 -- 172.21.15.88:0/4139235245 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f4438001680 con 0x7f44440c9bb0 2023-10-16T20:51:19.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f44427fc640 1 -- 172.21.15.88:0/4139235245 >> v1:172.21.15.88:6789/0 conn(0x7f44440c9bb0 legacy=0x7f44440c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:19.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f44427fc640 1 --2- 172.21.15.88:0/4139235245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f442c0039b0 0x7f442c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:19.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f44427fc640 1 --2- 172.21.15.88:0/4139235245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44440cece0 0x7f442c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:19.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f44427fc640 1 --2- 172.21.15.88:0/4139235245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f442c004d10 0x7f442c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:19.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f44427fc640 1 -- 172.21.15.88:0/4139235245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44440cc250 con 0x7f442c004d10 2023-10-16T20:51:19.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f44427fc640 1 -- 172.21.15.88:0/4139235245 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c00ba40 con 0x7f442c0039b0 2023-10-16T20:51:19.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f44427fc640 1 -- 172.21.15.88:0/4139235245 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442c00bc00 con 0x7f44440cece0 2023-10-16T20:51:19.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f4443fff640 1 --2- 172.21.15.88:0/4139235245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f442c004d10 0x7f442c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:19.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f4442ffd640 1 --2- 172.21.15.88:0/4139235245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44440cece0 0x7f442c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:19.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.715+0000 7f44437fe640 1 --2- 172.21.15.88:0/4139235245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f442c0039b0 0x7f442c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:19.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.719+0000 7f4443fff640 1 -- 172.21.15.88:0/4139235245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44440cece0 msgr2=0x7f442c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:19.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.719+0000 7f4443fff640 1 --2- 172.21.15.88:0/4139235245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44440cece0 0x7f442c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:19.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.719+0000 7f4443fff640 1 -- 172.21.15.88:0/4139235245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f442c0039b0 msgr2=0x7f442c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:19.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.719+0000 7f4443fff640 1 --2- 172.21.15.88:0/4139235245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f442c0039b0 0x7f442c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:19.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.719+0000 7f4443fff640 1 -- 172.21.15.88:0/4139235245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c003530 con 0x7f442c004d10 2023-10-16T20:51:19.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.719+0000 7f4442ffd640 1 --2- 172.21.15.88:0/4139235245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f44440cece0 0x7f442c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:51:19.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.719+0000 7f4443fff640 1 --2- 172.21.15.88:0/4139235245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f442c004d10 0x7f442c005190 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f443c009f90 tx=0x7f443c01cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6df0d3700497e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:19.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.719+0000 7f44427fc640 1 -- 172.21.15.88:0/4139235245 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c0a5c30 con 0x7f442c004d10 2023-10-16T20:51:19.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.719+0000 7f44427fc640 1 -- 172.21.15.88:0/4139235245 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443c00f040 con 0x7f442c004d10 2023-10-16T20:51:19.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.719+0000 7f44427fc640 1 -- 172.21.15.88:0/4139235245 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c0ad440 con 0x7f442c004d10 2023-10-16T20:51:19.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.719+0000 7f444a068640 1 -- 172.21.15.88:0/4139235245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f442c004d10 msgr2=0x7f442c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:19.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.719+0000 7f444a068640 1 --2- 172.21.15.88:0/4139235245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f442c004d10 0x7f442c005190 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f443c009f90 tx=0x7f443c01cc20 comp rx=0 tx=0).stop 2023-10-16T20:51:19.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.719+0000 7f4443fff640 1 -- 172.21.15.88:0/4139235245 reap_dead start 2023-10-16T20:51:19.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.719+0000 7f444a068640 1 -- 172.21.15.88:0/4139235245 shutdown_connections 2023-10-16T20:51:19.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.719+0000 7f444a068640 1 -- 172.21.15.88:0/4139235245 >> 172.21.15.88:0/4139235245 conn(0x7f44440715c0 msgr2=0x7f44440719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:19.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.719+0000 7f444a068640 1 -- 172.21.15.88:0/4139235245 shutdown_connections 2023-10-16T20:51:19.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.719+0000 7f444a068640 1 -- 172.21.15.88:0/4139235245 wait complete. 2023-10-16T20:51:19.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f444a068640 1 Processor -- start 2023-10-16T20:51:19.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f444a068640 1 -- start start 2023-10-16T20:51:19.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f444a068640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f442c004d10 0x7f44441421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:19.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f444a068640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4444150ac0 0x7f44440b3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:19.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f444a068640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44440b4d80 0x7f4444142bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:19.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f444a068640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44441eee50 con 0x7f44440b4d80 2023-10-16T20:51:19.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f444a068640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44441ef380 con 0x7f4444150ac0 2023-10-16T20:51:19.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f444a068640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44441ef0a0 con 0x7f442c004d10 2023-10-16T20:51:19.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f4443fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44440b4d80 0x7f4444142bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:19.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f4443fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44440b4d80 0x7f4444142bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33158/0 (socket says 172.21.15.88:33158) 2023-10-16T20:51:19.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f4443fff640 1 -- 172.21.15.88:0/3591551158 learned_addr learned my addr 172.21.15.88:0/3591551158 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:19.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f44437fe640 1 --2- 172.21.15.88:0/3591551158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f442c004d10 0x7f44441421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:19.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f4442ffd640 1 --2- 172.21.15.88:0/3591551158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4444150ac0 0x7f44440b3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:19.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f4443fff640 1 -- 172.21.15.88:0/3591551158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f442c004d10 msgr2=0x7f44441421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:19.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f4443fff640 1 --2- 172.21.15.88:0/3591551158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f442c004d10 0x7f44441421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:19.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f4443fff640 1 -- 172.21.15.88:0/3591551158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4444150ac0 msgr2=0x7f44440b3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:19.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f4443fff640 1 --2- 172.21.15.88:0/3591551158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4444150ac0 0x7f44440b3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:19.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f4443fff640 1 -- 172.21.15.88:0/3591551158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44440c8aa0 con 0x7f44440b4d80 2023-10-16T20:51:19.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f4443fff640 1 --2- 172.21.15.88:0/3591551158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44440b4d80 0x7f4444142bf0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f443c0b0040 tx=0x7f443c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:19.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f442bfff640 1 -- 172.21.15.88:0/3591551158 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c0039a0 con 0x7f44440b4d80 2023-10-16T20:51:19.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f442bfff640 1 -- 172.21.15.88:0/3591551158 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443c003b00 con 0x7f44440b4d80 2023-10-16T20:51:19.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f444a068640 1 -- 172.21.15.88:0/3591551158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44440c8d80 con 0x7f44440b4d80 2023-10-16T20:51:19.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.723+0000 7f442bfff640 1 -- 172.21.15.88:0/3591551158 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c0ae890 con 0x7f44440b4d80 2023-10-16T20:51:19.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.727+0000 7f444a068640 1 -- 172.21.15.88:0/3591551158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44440c9060 con 0x7f44440b4d80 2023-10-16T20:51:19.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.727+0000 7f444a068640 1 -- 172.21.15.88:0/3591551158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4444074700 con 0x7f44440b4d80 2023-10-16T20:51:19.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.727+0000 7f442bfff640 1 -- 172.21.15.88:0/3591551158 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f443c0b4040 con 0x7f44440b4d80 2023-10-16T20:51:19.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.727+0000 7f442bfff640 1 --2- 172.21.15.88:0/3591551158 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44300664d0 0x7f4430068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:19.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.727+0000 7f442bfff640 1 -- 172.21.15.88:0/3591551158 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7093+0+0 (secure 0 0 0) 0x7f443c11e150 con 0x7f44440b4d80 2023-10-16T20:51:19.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.727+0000 7f44437fe640 1 --2- 172.21.15.88:0/3591551158 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44300664d0 0x7f4430068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:19.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.727+0000 7f44437fe640 1 --2- 172.21.15.88:0/3591551158 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44300664d0 0x7f4430068990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4434000c10 tx=0x7f4434009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:19.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.731+0000 7f442bfff640 1 -- 172.21.15.88:0/3591551158 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f443c0eb100 con 0x7f44440b4d80 2023-10-16T20:51:19.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.915+0000 7f444a068640 1 -- 172.21.15.88:0/3591551158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44440ca740 con 0x7f44440b4d80 2023-10-16T20:51:19.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.915+0000 7f442bfff640 1 -- 172.21.15.88:0/3591551158 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6567 (secure 0 0 0) 0x7f443c0ee720 con 0x7f44440b4d80 2023-10-16T20:51:19.916 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:19.917 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:49:52.334480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4288,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6835/3059699574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3059699574},{"type":"v1","addr":"172.21.15.191:6835","nonce":3059699574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:19.917 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 80 2023-10-16T20:51:19.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.915+0000 7f444a068640 1 -- 172.21.15.88:0/3591551158 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44300664d0 msgr2=0x7f4430068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:19.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.915+0000 7f444a068640 1 --2- 172.21.15.88:0/3591551158 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44300664d0 0x7f4430068990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4434000c10 tx=0x7f4434009040 comp rx=0 tx=0).stop 2023-10-16T20:51:19.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.915+0000 7f444a068640 1 -- 172.21.15.88:0/3591551158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44440b4d80 msgr2=0x7f4444142bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:19.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.915+0000 7f444a068640 1 --2- 172.21.15.88:0/3591551158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44440b4d80 0x7f4444142bf0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f443c0b0040 tx=0x7f443c009750 comp rx=0 tx=0).stop 2023-10-16T20:51:19.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.915+0000 7f444a068640 1 -- 172.21.15.88:0/3591551158 shutdown_connections 2023-10-16T20:51:19.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.915+0000 7f444a068640 1 --2- 172.21.15.88:0/3591551158 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44300664d0 0x7f4430068990 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:19.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.915+0000 7f444a068640 1 --2- 172.21.15.88:0/3591551158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44440b4d80 0x7f4444142bf0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:19.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.915+0000 7f444a068640 1 --2- 172.21.15.88:0/3591551158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4444150ac0 0x7f44440b3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:19.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.915+0000 7f444a068640 1 --2- 172.21.15.88:0/3591551158 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f442c004d10 0x7f44441421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:19.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.915+0000 7f444a068640 1 -- 172.21.15.88:0/3591551158 >> 172.21.15.88:0/3591551158 conn(0x7f44440715c0 msgr2=0x7f4444149ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:19.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.915+0000 7f444a068640 1 -- 172.21.15.88:0/3591551158 shutdown_connections 2023-10-16T20:51:19.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:19.919+0000 7f444a068640 1 -- 172.21.15.88:0/3591551158 wait complete. 2023-10-16T20:51:19.938 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-16T20:51:21.939 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:22.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.087+0000 7f55080ea640 1 Processor -- start 2023-10-16T20:51:22.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.087+0000 7f55080ea640 1 -- start start 2023-10-16T20:51:22.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.087+0000 7f55080ea640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5500153670 0x7f550006a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:22.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.087+0000 7f55080ea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f55000c58d0 0x7f550006d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:22.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.087+0000 7f55080ea640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f550006d6b0 0x7f550006f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:22.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.087+0000 7f55080ea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55000c6860 con 0x7f55000c58d0 2023-10-16T20:51:22.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.087+0000 7f55080ea640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55000c5e80 con 0x7f550006d6b0 2023-10-16T20:51:22.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.087+0000 7f55080ea640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f550015cb30 con 0x7f5500153670 2023-10-16T20:51:22.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.087+0000 7f5505685640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f55000c58d0 0x7f550006d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:22.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.087+0000 7f5505685640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f55000c58d0 0x7f550006d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33160/0 (socket says 172.21.15.88:33160) 2023-10-16T20:51:22.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.087+0000 7f5505685640 1 -- 172.21.15.88:0/3485425560 learned_addr learned my addr 172.21.15.88:0/3485425560 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:22.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.091+0000 7f5505e86640 1 --2- 172.21.15.88:0/3485425560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5500153670 0x7f550006a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:22.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.091+0000 7f5505685640 1 -- 172.21.15.88:0/3485425560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5500153670 msgr2=0x7f550006a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:22.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.091+0000 7f5505685640 1 --2- 172.21.15.88:0/3485425560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5500153670 0x7f550006a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.091+0000 7f5506687640 1 --2- 172.21.15.88:0/3485425560 >> v2:172.21.15.191:3300/0 conn(0x7f550006d6b0 0x7f550006f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:22.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.091+0000 7f5505685640 1 -- 172.21.15.88:0/3485425560 >> v2:172.21.15.191:3300/0 conn(0x7f550006d6b0 msgr2=0x7f550006f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:22.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.091+0000 7f5505685640 1 --2- 172.21.15.88:0/3485425560 >> v2:172.21.15.191:3300/0 conn(0x7f550006d6b0 0x7f550006f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.091+0000 7f5505685640 1 -- 172.21.15.88:0/3485425560 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550015ce10 con 0x7f55000c58d0 2023-10-16T20:51:22.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.091+0000 7f5505e86640 1 --2- 172.21.15.88:0/3485425560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5500153670 0x7f550006a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:51:22.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.091+0000 7f5506687640 1 --2- 172.21.15.88:0/3485425560 >> v2:172.21.15.191:3300/0 conn(0x7f550006d6b0 0x7f550006f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:51:22.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.091+0000 7f5505685640 1 --2- 172.21.15.88:0/3485425560 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55000c58d0 0x7f550006d170 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f55000c7290 tx=0x7f54f001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b9302ec33102f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:22.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.091+0000 7f5504e84640 1 -- 172.21.15.88:0/3485425560 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f00ab600 con 0x7f55000c58d0 2023-10-16T20:51:22.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.091+0000 7f5504e84640 1 -- 172.21.15.88:0/3485425560 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f00a7050 con 0x7f55000c58d0 2023-10-16T20:51:22.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.091+0000 7f5504e84640 1 -- 172.21.15.88:0/3485425560 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f00ab980 con 0x7f55000c58d0 2023-10-16T20:51:22.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f55080ea640 1 -- 172.21.15.88:0/3485425560 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55000c58d0 msgr2=0x7f550006d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:22.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f55080ea640 1 --2- 172.21.15.88:0/3485425560 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55000c58d0 0x7f550006d170 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f55000c7290 tx=0x7f54f001c500 comp rx=0 tx=0).stop 2023-10-16T20:51:22.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f55080ea640 1 -- 172.21.15.88:0/3485425560 shutdown_connections 2023-10-16T20:51:22.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f55080ea640 1 --2- 172.21.15.88:0/3485425560 >> v2:172.21.15.191:3300/0 conn(0x7f550006d6b0 0x7f550006f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f55080ea640 1 --2- 172.21.15.88:0/3485425560 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55000c58d0 0x7f550006d170 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f55080ea640 1 --2- 172.21.15.88:0/3485425560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5500153670 0x7f550006a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f55080ea640 1 -- 172.21.15.88:0/3485425560 >> 172.21.15.88:0/3485425560 conn(0x7f55000cad50 msgr2=0x7f55000cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:22.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f55080ea640 1 -- 172.21.15.88:0/3485425560 shutdown_connections 2023-10-16T20:51:22.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f55080ea640 1 -- 172.21.15.88:0/3485425560 wait complete. 2023-10-16T20:51:22.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f55080ea640 1 Processor -- start 2023-10-16T20:51:22.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f55080ea640 1 -- start start 2023-10-16T20:51:22.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f55080ea640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f550006d6b0 0x7f55000c2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:22.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f55080ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55000c58d0 0x7f5500150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:22.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f55080ea640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5500153670 0x7f55001f5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:22.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f55080ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55000c3610 con 0x7f55000c58d0 2023-10-16T20:51:22.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f55080ea640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550014d760 con 0x7f5500153670 2023-10-16T20:51:22.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f55080ea640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f550014d9b0 con 0x7f550006d6b0 2023-10-16T20:51:22.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f5505685640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55000c58d0 0x7f5500150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:22.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f5505685640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55000c58d0 0x7f5500150010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33172/0 (socket says 172.21.15.88:33172) 2023-10-16T20:51:22.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f5505685640 1 -- 172.21.15.88:0/1029268475 learned_addr learned my addr 172.21.15.88:0/1029268475 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:22.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f5505e86640 1 --2- 172.21.15.88:0/1029268475 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f550006d6b0 0x7f55000c2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:22.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f5506687640 1 --2- 172.21.15.88:0/1029268475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5500153670 0x7f55001f5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:22.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f5505685640 1 -- 172.21.15.88:0/1029268475 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f550006d6b0 msgr2=0x7f55000c2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:22.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f5505685640 1 --2- 172.21.15.88:0/1029268475 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f550006d6b0 0x7f55000c2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f5505685640 1 -- 172.21.15.88:0/1029268475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5500153670 msgr2=0x7f55001f5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:22.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f5505685640 1 --2- 172.21.15.88:0/1029268475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5500153670 0x7f55001f5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.095+0000 7f5505685640 1 -- 172.21.15.88:0/1029268475 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f00094d0 con 0x7f55000c58d0 2023-10-16T20:51:22.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.099+0000 7f5505685640 1 --2- 172.21.15.88:0/1029268475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55000c58d0 0x7f5500150010 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f54f001abb0 tx=0x7f550015a440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:22.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.099+0000 7f54f67fc640 1 -- 172.21.15.88:0/1029268475 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f00b37d0 con 0x7f55000c58d0 2023-10-16T20:51:22.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.099+0000 7f54f67fc640 1 -- 172.21.15.88:0/1029268475 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f00b3930 con 0x7f55000c58d0 2023-10-16T20:51:22.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.099+0000 7f54f67fc640 1 -- 172.21.15.88:0/1029268475 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f00b3ce0 con 0x7f55000c58d0 2023-10-16T20:51:22.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.099+0000 7f55080ea640 1 -- 172.21.15.88:0/1029268475 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f550014e0e0 con 0x7f55000c58d0 2023-10-16T20:51:22.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.099+0000 7f55080ea640 1 -- 172.21.15.88:0/1029268475 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f550014de00 con 0x7f55000c58d0 2023-10-16T20:51:22.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.099+0000 7f55080ea640 1 -- 172.21.15.88:0/1029268475 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f550014dc60 con 0x7f55000c58d0 2023-10-16T20:51:22.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.099+0000 7f54f67fc640 1 -- 172.21.15.88:0/1029268475 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f54f00b2070 con 0x7f55000c58d0 2023-10-16T20:51:22.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.099+0000 7f54f67fc640 1 --2- 172.21.15.88:0/1029268475 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f54e40664d0 0x7f54e4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:22.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.099+0000 7f54f67fc640 1 -- 172.21.15.88:0/1029268475 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7f54f011e580 con 0x7f55000c58d0 2023-10-16T20:51:22.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.099+0000 7f5505e86640 1 --2- 172.21.15.88:0/1029268475 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f54e40664d0 0x7f54e4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:22.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.103+0000 7f5505e86640 1 --2- 172.21.15.88:0/1029268475 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f54e40664d0 0x7f54e4068990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f54fc006fd0 tx=0x7f54fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:22.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.103+0000 7f54f67fc640 1 -- 172.21.15.88:0/1029268475 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f550014dc60 con 0x7f55000c58d0 2023-10-16T20:51:22.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.283+0000 7f55080ea640 1 -- 172.21.15.88:0/1029268475 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f550014dc60 con 0x7f55000c58d0 2023-10-16T20:51:22.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.283+0000 7f54f67fc640 1 -- 172.21.15.88:0/1029268475 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5775 (secure 0 0 0) 0x7f550014dc60 con 0x7f55000c58d0 2023-10-16T20:51:22.287 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:22.287 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:51:20.522658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:22.289 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 81 2023-10-16T20:51:22.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.287+0000 7f55080ea640 1 -- 172.21.15.88:0/1029268475 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f54e40664d0 msgr2=0x7f54e4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:22.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.287+0000 7f55080ea640 1 --2- 172.21.15.88:0/1029268475 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f54e40664d0 0x7f54e4068990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f54fc006fd0 tx=0x7f54fc008040 comp rx=0 tx=0).stop 2023-10-16T20:51:22.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.287+0000 7f55080ea640 1 -- 172.21.15.88:0/1029268475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55000c58d0 msgr2=0x7f5500150010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:22.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.287+0000 7f55080ea640 1 --2- 172.21.15.88:0/1029268475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55000c58d0 0x7f5500150010 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f54f001abb0 tx=0x7f550015a440 comp rx=0 tx=0).stop 2023-10-16T20:51:22.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.287+0000 7f55080ea640 1 -- 172.21.15.88:0/1029268475 shutdown_connections 2023-10-16T20:51:22.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.287+0000 7f55080ea640 1 --2- 172.21.15.88:0/1029268475 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f54e40664d0 0x7f54e4068990 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.287+0000 7f55080ea640 1 --2- 172.21.15.88:0/1029268475 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5500153670 0x7f55001f5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.287+0000 7f55080ea640 1 --2- 172.21.15.88:0/1029268475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55000c58d0 0x7f5500150010 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.287+0000 7f55080ea640 1 --2- 172.21.15.88:0/1029268475 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f550006d6b0 0x7f55000c2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.287+0000 7f55080ea640 1 -- 172.21.15.88:0/1029268475 >> 172.21.15.88:0/1029268475 conn(0x7f55000cad50 msgr2=0x7f5500158cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.287+0000 7f55080ea640 1 -- 172.21.15.88:0/1029268475 shutdown_connections 2023-10-16T20:51:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.287+0000 7f55080ea640 1 -- 172.21.15.88:0/1029268475 wait complete. 2023-10-16T20:51:22.306 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-16T20:51:22.306 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-16T20:51:22.307 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:22.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34ba017640 1 Processor -- start 2023-10-16T20:51:22.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34ba017640 1 -- start start 2023-10-16T20:51:22.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34ba017640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f34b4153770 0x7f34b4159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:22.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34ba017640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b40c9bb0 0x7f34b40c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:22.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34ba017640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f34b40c41f0 0x7f34b40cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:22.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34ba017640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b40cc250 con 0x7f34b4153770 2023-10-16T20:51:22.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34ba017640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b40cc670 con 0x7f34b40c41f0 2023-10-16T20:51:22.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34ba017640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b415a410 con 0x7f34b40c9bb0 2023-10-16T20:51:22.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34b37fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f34b4153770 0x7f34b4159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:22.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34b37fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f34b4153770 0x7f34b4159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33174/0 (socket says 172.21.15.88:33174) 2023-10-16T20:51:22.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34b37fe640 1 -- 172.21.15.88:0/2960436076 learned_addr learned my addr 172.21.15.88:0/2960436076 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:22.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34b3fff640 1 --2- 172.21.15.88:0/2960436076 >> v2:172.21.15.191:3300/0 conn(0x7f34b40c41f0 0x7f34b40cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:22.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34b2ffd640 1 --2- 172.21.15.88:0/2960436076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b40c9bb0 0x7f34b40c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:22.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34b37fe640 1 -- 172.21.15.88:0/2960436076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b40c9bb0 msgr2=0x7f34b40c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:22.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34b37fe640 1 --2- 172.21.15.88:0/2960436076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b40c9bb0 0x7f34b40c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34b37fe640 1 -- 172.21.15.88:0/2960436076 >> v2:172.21.15.191:3300/0 conn(0x7f34b40c41f0 msgr2=0x7f34b40cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:22.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34b37fe640 1 --2- 172.21.15.88:0/2960436076 >> v2:172.21.15.191:3300/0 conn(0x7f34b40c41f0 0x7f34b40cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.443+0000 7f34b37fe640 1 -- 172.21.15.88:0/2960436076 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b4153b50 con 0x7f34b4153770 2023-10-16T20:51:22.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.447+0000 7f34b37fe640 1 --2- 172.21.15.88:0/2960436076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b4153770 0x7f34b4159ed0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f34b40b2dc0 tx=0x7f34a401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b156741e006badc4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:22.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.447+0000 7f34b27fc640 1 -- 172.21.15.88:0/2960436076 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a40ac630 con 0x7f34b4153770 2023-10-16T20:51:22.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.447+0000 7f34b27fc640 1 -- 172.21.15.88:0/2960436076 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a40ac790 con 0x7f34b4153770 2023-10-16T20:51:22.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.447+0000 7f34b27fc640 1 -- 172.21.15.88:0/2960436076 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a40acb20 con 0x7f34b4153770 2023-10-16T20:51:22.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.447+0000 7f34ba017640 1 -- 172.21.15.88:0/2960436076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b4153770 msgr2=0x7f34b4159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:22.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.447+0000 7f34ba017640 1 --2- 172.21.15.88:0/2960436076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b4153770 0x7f34b4159ed0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f34b40b2dc0 tx=0x7f34a401c500 comp rx=0 tx=0).stop 2023-10-16T20:51:22.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.447+0000 7f34ba017640 1 -- 172.21.15.88:0/2960436076 shutdown_connections 2023-10-16T20:51:22.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.447+0000 7f34ba017640 1 --2- 172.21.15.88:0/2960436076 >> v2:172.21.15.191:3300/0 conn(0x7f34b40c41f0 0x7f34b40cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.447+0000 7f34ba017640 1 --2- 172.21.15.88:0/2960436076 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b40c9bb0 0x7f34b40c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.447+0000 7f34ba017640 1 --2- 172.21.15.88:0/2960436076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b4153770 0x7f34b4159ed0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.447+0000 7f34ba017640 1 -- 172.21.15.88:0/2960436076 >> 172.21.15.88:0/2960436076 conn(0x7f34b40715c0 msgr2=0x7f34b40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:22.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.447+0000 7f34ba017640 1 -- 172.21.15.88:0/2960436076 shutdown_connections 2023-10-16T20:51:22.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.447+0000 7f34ba017640 1 -- 172.21.15.88:0/2960436076 wait complete. 2023-10-16T20:51:22.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34ba017640 1 Processor -- start 2023-10-16T20:51:22.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34ba017640 1 -- start start 2023-10-16T20:51:22.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34ba017640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b40c41f0 0x7f34b41f5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:22.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34ba017640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34b40c9bb0 0x7f34b4158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:22.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34ba017640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b4153770 0x7f34b4159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:22.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34ba017640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b4147dd0 con 0x7f34b4153770 2023-10-16T20:51:22.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34ba017640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b41480b0 con 0x7f34b40c9bb0 2023-10-16T20:51:22.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34ba017640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b41f5f70 con 0x7f34b40c41f0 2023-10-16T20:51:22.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34b3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b4153770 0x7f34b4159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:22.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34b3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b4153770 0x7f34b4159520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33176/0 (socket says 172.21.15.88:33176) 2023-10-16T20:51:22.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34b3fff640 1 -- 172.21.15.88:0/1620367937 learned_addr learned my addr 172.21.15.88:0/1620367937 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:22.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34b37fe640 1 --2- 172.21.15.88:0/1620367937 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b40c41f0 0x7f34b41f5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:22.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34b2ffd640 1 --2- 172.21.15.88:0/1620367937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34b40c9bb0 0x7f34b4158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:22.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34b3fff640 1 -- 172.21.15.88:0/1620367937 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b40c41f0 msgr2=0x7f34b41f5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:22.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34b3fff640 1 --2- 172.21.15.88:0/1620367937 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b40c41f0 0x7f34b41f5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34b3fff640 1 -- 172.21.15.88:0/1620367937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34b40c9bb0 msgr2=0x7f34b4158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:22.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34b3fff640 1 --2- 172.21.15.88:0/1620367937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34b40c9bb0 0x7f34b4158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34b3fff640 1 -- 172.21.15.88:0/1620367937 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a4009480 con 0x7f34b4153770 2023-10-16T20:51:22.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34b2ffd640 1 --2- 172.21.15.88:0/1620367937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34b40c9bb0 0x7f34b4158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:51:22.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34b37fe640 1 --2- 172.21.15.88:0/1620367937 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b40c41f0 0x7f34b41f5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:51:22.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34b3fff640 1 --2- 172.21.15.88:0/1620367937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b4153770 0x7f34b4159520 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f34ac007e60 tx=0x7f34ac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:22.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f349bfff640 1 -- 172.21.15.88:0/1620367937 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac014620 con 0x7f34b4153770 2023-10-16T20:51:22.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f349bfff640 1 -- 172.21.15.88:0/1620367937 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ac00ee90 con 0x7f34b4153770 2023-10-16T20:51:22.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f349bfff640 1 -- 172.21.15.88:0/1620367937 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac0149e0 con 0x7f34b4153770 2023-10-16T20:51:22.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34ba017640 1 -- 172.21.15.88:0/1620367937 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b41f5c90 con 0x7f34b4153770 2023-10-16T20:51:22.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.451+0000 7f34ba017640 1 -- 172.21.15.88:0/1620367937 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b4146650 con 0x7f34b4153770 2023-10-16T20:51:22.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.455+0000 7f34ba017640 1 -- 172.21.15.88:0/1620367937 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b4148260 con 0x7f34b4153770 2023-10-16T20:51:22.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.455+0000 7f349bfff640 1 -- 172.21.15.88:0/1620367937 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f34ac014b40 con 0x7f34b4153770 2023-10-16T20:51:22.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.455+0000 7f349bfff640 1 --2- 172.21.15.88:0/1620367937 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34a00664a0 0x7f34a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:22.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.455+0000 7f349bfff640 1 -- 172.21.15.88:0/1620367937 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7f34ac084cc0 con 0x7f34b4153770 2023-10-16T20:51:22.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.455+0000 7f34b37fe640 1 --2- 172.21.15.88:0/1620367937 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34a00664a0 0x7f34a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:22.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.455+0000 7f34b37fe640 1 --2- 172.21.15.88:0/1620367937 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34a00664a0 0x7f34a0068960 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f34a40ae000 tx=0x7f34a40a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:22.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.459+0000 7f349bfff640 1 -- 172.21.15.88:0/1620367937 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f34b4148260 con 0x7f34b4153770 2023-10-16T20:51:22.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.643+0000 7f34ba017640 1 -- 172.21.15.88:0/1620367937 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34b4148220 con 0x7f34b4153770 2023-10-16T20:51:22.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.647+0000 7f349bfff640 1 -- 172.21.15.88:0/1620367937 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5775 (secure 0 0 0) 0x7f34ac051c10 con 0x7f34b4153770 2023-10-16T20:51:22.647 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:22.647 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:51:20.522658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:22.648 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 81 2023-10-16T20:51:22.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.647+0000 7f34ba017640 1 -- 172.21.15.88:0/1620367937 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34a00664a0 msgr2=0x7f34a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:22.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.647+0000 7f34ba017640 1 --2- 172.21.15.88:0/1620367937 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34a00664a0 0x7f34a0068960 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f34a40ae000 tx=0x7f34a40a7000 comp rx=0 tx=0).stop 2023-10-16T20:51:22.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.647+0000 7f34ba017640 1 -- 172.21.15.88:0/1620367937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b4153770 msgr2=0x7f34b4159520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:22.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.647+0000 7f34ba017640 1 --2- 172.21.15.88:0/1620367937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b4153770 0x7f34b4159520 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f34ac007e60 tx=0x7f34ac00e4b0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.647+0000 7f34ba017640 1 -- 172.21.15.88:0/1620367937 shutdown_connections 2023-10-16T20:51:22.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.647+0000 7f34ba017640 1 --2- 172.21.15.88:0/1620367937 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34a00664a0 0x7f34a0068960 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.647+0000 7f34ba017640 1 --2- 172.21.15.88:0/1620367937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b4153770 0x7f34b4159520 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.647+0000 7f34ba017640 1 --2- 172.21.15.88:0/1620367937 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34b40c9bb0 0x7f34b4158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.647+0000 7f34ba017640 1 --2- 172.21.15.88:0/1620367937 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34b40c41f0 0x7f34b41f5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:22.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.647+0000 7f34ba017640 1 -- 172.21.15.88:0/1620367937 >> 172.21.15.88:0/1620367937 conn(0x7f34b40715c0 msgr2=0x7f34b4146a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:22.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.647+0000 7f34ba017640 1 -- 172.21.15.88:0/1620367937 shutdown_connections 2023-10-16T20:51:22.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:22.647+0000 7f34ba017640 1 -- 172.21.15.88:0/1620367937 wait complete. 2023-10-16T20:51:22.668 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 81, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 81, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4889": { "addr": "172.21.15.191:6837/2468647900", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6836", "nonce": 2468647900, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2468647900, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4889, "incarnation": 30, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 21 }, "gid_5282": { "addr": "172.21.15.191:6833/3633326602", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 3633326602, "type": "v2" }, { "addr": "172.21.15.191:6833", "nonce": 3633326602, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5282, "incarnation": 81, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_5284": { "addr": "172.21.15.88:6839/2172973775", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6837", "nonce": 2172973775, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2172973775, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5284, "incarnation": 38, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 17 }, "gid_5535": { "addr": "172.21.15.88:6835/3476086754", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3476086754, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3476086754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5535, "incarnation": 52, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_5709": { "addr": "172.21.15.88:6838/2600891060", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 2600891060, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 2600891060, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5709, "incarnation": 67, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:51:20.522658+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5282, "mds_1": 5709, "mds_2": 5535, "mds_3": 5284, "mds_4": 4889 } } } ], "standbys": [] } 2023-10-16T20:51:24.669 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:24.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.819+0000 7f2cf4b65640 1 Processor -- start 2023-10-16T20:51:24.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.819+0000 7f2cf4b65640 1 -- start start 2023-10-16T20:51:24.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.819+0000 7f2cf4b65640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f2cf0153770 0x7f2cf0159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:24.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.819+0000 7f2cf4b65640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2cf00cece0 0x7f2cf00cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:24.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.819+0000 7f2cf4b65640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf015a410 con 0x7f2cf00c9bb0 2023-10-16T20:51:24.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.819+0000 7f2cf4b65640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf00cc250 con 0x7f2cf00cece0 2023-10-16T20:51:24.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.819+0000 7f2cf4b65640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf00cc670 con 0x7f2cf0153770 2023-10-16T20:51:24.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.819+0000 7f2ceed9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2cf00cece0 0x7f2cf00cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:24.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.819+0000 7f2ceed9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2cf00cece0 0x7f2cf00cf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33186/0 (socket says 172.21.15.88:33186) 2023-10-16T20:51:24.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.819+0000 7f2ceed9d640 1 -- 172.21.15.88:0/1563030875 learned_addr learned my addr 172.21.15.88:0/1563030875 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:24.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.819+0000 7f2cee59c640 1 --2- 172.21.15.88:0/1563030875 >> v2:172.21.15.191:3300/0 conn(0x7f2cf0153770 0x7f2cf0159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:24.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.819+0000 7f2ceed9d640 1 -- 172.21.15.88:0/1563030875 >> v2:172.21.15.191:3300/0 conn(0x7f2cf0153770 msgr2=0x7f2cf0159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:24.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.819+0000 7f2ceed9d640 1 --2- 172.21.15.88:0/1563030875 >> v2:172.21.15.191:3300/0 conn(0x7f2cf0153770 0x7f2cf0159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:24.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.819+0000 7f2ceed9d640 1 -- 172.21.15.88:0/1563030875 >> v1:172.21.15.191:6789/0 conn(0x7f2cf00c9bb0 legacy=0x7f2cf00c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:24.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.819+0000 7f2ceed9d640 1 -- 172.21.15.88:0/1563030875 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf0153b50 con 0x7f2cf00cece0 2023-10-16T20:51:24.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.823+0000 7f2ceed9d640 1 --2- 172.21.15.88:0/1563030875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cf00cece0 0x7f2cf00cf130 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2cf00cc7b0 tx=0x7f2ce801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6212a11fe1d75e72 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:24.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.823+0000 7f2ced59a640 1 -- 172.21.15.88:0/1563030875 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce80a5890 con 0x7f2cf00cece0 2023-10-16T20:51:24.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.823+0000 7f2ced59a640 1 -- 172.21.15.88:0/1563030875 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce80a59f0 con 0x7f2cf00cece0 2023-10-16T20:51:24.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.823+0000 7f2ced59a640 1 -- 172.21.15.88:0/1563030875 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce80a5d80 con 0x7f2cf00cece0 2023-10-16T20:51:24.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.823+0000 7f2cf4b65640 1 -- 172.21.15.88:0/1563030875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cf00cece0 msgr2=0x7f2cf00cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:24.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.823+0000 7f2cf4b65640 1 --2- 172.21.15.88:0/1563030875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cf00cece0 0x7f2cf00cf130 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2cf00cc7b0 tx=0x7f2ce801c9d0 comp rx=0 tx=0).stop 2023-10-16T20:51:24.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.823+0000 7f2cf4b65640 1 -- 172.21.15.88:0/1563030875 shutdown_connections 2023-10-16T20:51:24.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.823+0000 7f2cf4b65640 1 --2- 172.21.15.88:0/1563030875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cf00cece0 0x7f2cf00cf130 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:24.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.823+0000 7f2cf4b65640 1 --2- 172.21.15.88:0/1563030875 >> v2:172.21.15.191:3300/0 conn(0x7f2cf0153770 0x7f2cf0159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:24.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.823+0000 7f2cf4b65640 1 -- 172.21.15.88:0/1563030875 >> 172.21.15.88:0/1563030875 conn(0x7f2cf00715c0 msgr2=0x7f2cf00719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:24.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.823+0000 7f2cf4b65640 1 -- 172.21.15.88:0/1563030875 shutdown_connections 2023-10-16T20:51:24.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.823+0000 7f2cf4b65640 1 -- 172.21.15.88:0/1563030875 wait complete. 2023-10-16T20:51:24.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cf4b65640 1 Processor -- start 2023-10-16T20:51:24.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cf4b65640 1 -- start start 2023-10-16T20:51:24.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cf4b65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cf00c9bb0 0x7f2cf015b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:24.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cf4b65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cf00cece0 0x7f2cf01f47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:24.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cf4b65640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cf0153770 0x7f2cf00b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:24.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cf4b65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf00b3740 con 0x7f2cf00cece0 2023-10-16T20:51:24.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cf4b65640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf015b960 con 0x7f2cf0153770 2023-10-16T20:51:24.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cf4b65640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf00b2e30 con 0x7f2cf00c9bb0 2023-10-16T20:51:24.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cedd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cf00cece0 0x7f2cf01f47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:24.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cedd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cf00cece0 0x7f2cf01f47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33202/0 (socket says 172.21.15.88:33202) 2023-10-16T20:51:24.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cedd9b640 1 -- 172.21.15.88:0/797849825 learned_addr learned my addr 172.21.15.88:0/797849825 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:24.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2ceed9d640 1 --2- 172.21.15.88:0/797849825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cf0153770 0x7f2cf00b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:24.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cee59c640 1 --2- 172.21.15.88:0/797849825 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cf00c9bb0 0x7f2cf015b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:24.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cedd9b640 1 -- 172.21.15.88:0/797849825 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cf00c9bb0 msgr2=0x7f2cf015b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:24.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cedd9b640 1 --2- 172.21.15.88:0/797849825 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cf00c9bb0 0x7f2cf015b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:24.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cedd9b640 1 -- 172.21.15.88:0/797849825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cf0153770 msgr2=0x7f2cf00b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:24.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cedd9b640 1 --2- 172.21.15.88:0/797849825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cf0153770 0x7f2cf00b3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:24.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cedd9b640 1 -- 172.21.15.88:0/797849825 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce8009b30 con 0x7f2cf00cece0 2023-10-16T20:51:24.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2ceed9d640 1 --2- 172.21.15.88:0/797849825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cf0153770 0x7f2cf00b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:51:24.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cee59c640 1 --2- 172.21.15.88:0/797849825 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cf00c9bb0 0x7f2cf015b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:51:24.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cedd9b640 1 --2- 172.21.15.88:0/797849825 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cf00cece0 0x7f2cf01f47d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2ce400c500 tx=0x7f2ce400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:24.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cdeffd640 1 -- 172.21.15.88:0/797849825 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4005220 con 0x7f2cf00cece0 2023-10-16T20:51:24.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cdeffd640 1 -- 172.21.15.88:0/797849825 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce4005380 con 0x7f2cf00cece0 2023-10-16T20:51:24.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cdeffd640 1 -- 172.21.15.88:0/797849825 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4007270 con 0x7f2cf00cece0 2023-10-16T20:51:24.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cf4b65640 1 -- 172.21.15.88:0/797849825 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf01f5210 con 0x7f2cf00cece0 2023-10-16T20:51:24.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cf4b65640 1 -- 172.21.15.88:0/797849825 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf00b33d0 con 0x7f2cf00cece0 2023-10-16T20:51:24.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.827+0000 7f2cf4b65640 1 -- 172.21.15.88:0/797849825 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf0074700 con 0x7f2cf00cece0 2023-10-16T20:51:24.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.831+0000 7f2cdeffd640 1 -- 172.21.15.88:0/797849825 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2ce40073d0 con 0x7f2cf00cece0 2023-10-16T20:51:24.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.831+0000 7f2cdeffd640 1 --2- 172.21.15.88:0/797849825 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2cd00664a0 0x7f2cd0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:24.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.831+0000 7f2cdeffd640 1 -- 172.21.15.88:0/797849825 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7f2ce4083d20 con 0x7f2cf00cece0 2023-10-16T20:51:24.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.831+0000 7f2cee59c640 1 --2- 172.21.15.88:0/797849825 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2cd00664a0 0x7f2cd0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:24.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.831+0000 7f2cee59c640 1 --2- 172.21.15.88:0/797849825 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2cd00664a0 0x7f2cd0068960 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009af0 tx=0x7f2ce0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:24.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:24.835+0000 7f2cdeffd640 1 -- 172.21.15.88:0/797849825 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2ce4050bf0 con 0x7f2cf00cece0 2023-10-16T20:51:25.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:25.015+0000 7f2cf4b65640 1 -- 172.21.15.88:0/797849825 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cf00ca740 con 0x7f2cf00cece0 2023-10-16T20:51:25.019 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:25.019+0000 7f2cdeffd640 1 -- 172.21.15.88:0/797849825 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5775 (secure 0 0 0) 0x7f2ce4054210 con 0x7f2cf00cece0 2023-10-16T20:51:25.019 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:25.020 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:51:20.522658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:25.020 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 81 2023-10-16T20:51:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:25.019+0000 7f2cf4b65640 1 -- 172.21.15.88:0/797849825 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2cd00664a0 msgr2=0x7f2cd0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:25.019+0000 7f2cf4b65640 1 --2- 172.21.15.88:0/797849825 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2cd00664a0 0x7f2cd0068960 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009af0 tx=0x7f2ce0008040 comp rx=0 tx=0).stop 2023-10-16T20:51:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:25.019+0000 7f2cf4b65640 1 -- 172.21.15.88:0/797849825 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cf00cece0 msgr2=0x7f2cf01f47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:25.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:25.019+0000 7f2cf4b65640 1 --2- 172.21.15.88:0/797849825 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cf00cece0 0x7f2cf01f47d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2ce400c500 tx=0x7f2ce400c9c0 comp rx=0 tx=0).stop 2023-10-16T20:51:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:25.019+0000 7f2cf4b65640 1 -- 172.21.15.88:0/797849825 shutdown_connections 2023-10-16T20:51:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:25.019+0000 7f2cf4b65640 1 --2- 172.21.15.88:0/797849825 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2cd00664a0 0x7f2cd0068960 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:25.019+0000 7f2cf4b65640 1 --2- 172.21.15.88:0/797849825 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2cf0153770 0x7f2cf00b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:25.019+0000 7f2cf4b65640 1 --2- 172.21.15.88:0/797849825 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cf00cece0 0x7f2cf01f47d0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:25.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:25.019+0000 7f2cf4b65640 1 --2- 172.21.15.88:0/797849825 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2cf00c9bb0 0x7f2cf015b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:25.019+0000 7f2cf4b65640 1 -- 172.21.15.88:0/797849825 >> 172.21.15.88:0/797849825 conn(0x7f2cf00715c0 msgr2=0x7f2cf0151380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:25.019+0000 7f2cf4b65640 1 -- 172.21.15.88:0/797849825 shutdown_connections 2023-10-16T20:51:25.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:25.019+0000 7f2cf4b65640 1 -- 172.21.15.88:0/797849825 wait complete. 2023-10-16T20:51:25.039 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:51:27.040 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:27.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.187+0000 7f1ac173e640 1 Processor -- start 2023-10-16T20:51:27.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.187+0000 7f1ac173e640 1 -- start start 2023-10-16T20:51:27.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.191+0000 7f1ac173e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1abc153770 0x7f1abc159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:27.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.191+0000 7f1ac173e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1abc0c9bb0 0x7f1abc0c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:27.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.191+0000 7f1ac173e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1abc0c41f0 0x7f1abc0cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:27.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.191+0000 7f1ac173e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1abc0cc250 con 0x7f1abc0c9bb0 2023-10-16T20:51:27.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.191+0000 7f1ac173e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1abc0cc670 con 0x7f1abc0c41f0 2023-10-16T20:51:27.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.191+0000 7f1ac173e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc15a410 con 0x7f1abc153770 2023-10-16T20:51:27.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.191+0000 7f1aba7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1abc0c9bb0 0x7f1abc0c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:27.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.191+0000 7f1aba7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1abc0c9bb0 0x7f1abc0c3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33216/0 (socket says 172.21.15.88:33216) 2023-10-16T20:51:27.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.191+0000 7f1aba7fc640 1 -- 172.21.15.88:0/1636009105 learned_addr learned my addr 172.21.15.88:0/1636009105 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:27.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.191+0000 7f1abaffd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1abc153770 0x7f1abc159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:27.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.191+0000 7f1abb7fe640 1 --2- 172.21.15.88:0/1636009105 >> v2:172.21.15.191:3300/0 conn(0x7f1abc0c41f0 0x7f1abc0cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:27.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.191+0000 7f1aba7fc640 1 -- 172.21.15.88:0/1636009105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1abc153770 msgr2=0x7f1abc159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:27.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.191+0000 7f1aba7fc640 1 --2- 172.21.15.88:0/1636009105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1abc153770 0x7f1abc159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:27.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.191+0000 7f1aba7fc640 1 -- 172.21.15.88:0/1636009105 >> v2:172.21.15.191:3300/0 conn(0x7f1abc0c41f0 msgr2=0x7f1abc0cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:27.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.191+0000 7f1aba7fc640 1 --2- 172.21.15.88:0/1636009105 >> v2:172.21.15.191:3300/0 conn(0x7f1abc0c41f0 0x7f1abc0cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:27.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.191+0000 7f1aba7fc640 1 -- 172.21.15.88:0/1636009105 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc153b50 con 0x7f1abc0c9bb0 2023-10-16T20:51:27.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1aba7fc640 1 --2- 172.21.15.88:0/1636009105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc0c9bb0 0x7f1abc0c3cb0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0b2dc0 tx=0x7f1ab001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ec4c99af8300a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:27.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ab9ffb640 1 -- 172.21.15.88:0/1636009105 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00ac630 con 0x7f1abc0c9bb0 2023-10-16T20:51:27.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ab9ffb640 1 -- 172.21.15.88:0/1636009105 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab00ac790 con 0x7f1abc0c9bb0 2023-10-16T20:51:27.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ab9ffb640 1 -- 172.21.15.88:0/1636009105 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00acb20 con 0x7f1abc0c9bb0 2023-10-16T20:51:27.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ac173e640 1 -- 172.21.15.88:0/1636009105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc0c9bb0 msgr2=0x7f1abc0c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:27.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ac173e640 1 --2- 172.21.15.88:0/1636009105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc0c9bb0 0x7f1abc0c3cb0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0b2dc0 tx=0x7f1ab001c500 comp rx=0 tx=0).stop 2023-10-16T20:51:27.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ac173e640 1 -- 172.21.15.88:0/1636009105 shutdown_connections 2023-10-16T20:51:27.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ac173e640 1 --2- 172.21.15.88:0/1636009105 >> v2:172.21.15.191:3300/0 conn(0x7f1abc0c41f0 0x7f1abc0cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:27.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ac173e640 1 --2- 172.21.15.88:0/1636009105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc0c9bb0 0x7f1abc0c3cb0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:27.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ac173e640 1 --2- 172.21.15.88:0/1636009105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1abc153770 0x7f1abc159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:27.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ac173e640 1 -- 172.21.15.88:0/1636009105 >> 172.21.15.88:0/1636009105 conn(0x7f1abc0715c0 msgr2=0x7f1abc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:27.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ac173e640 1 -- 172.21.15.88:0/1636009105 shutdown_connections 2023-10-16T20:51:27.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ac173e640 1 -- 172.21.15.88:0/1636009105 wait complete. 2023-10-16T20:51:27.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ac173e640 1 Processor -- start 2023-10-16T20:51:27.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ac173e640 1 -- start start 2023-10-16T20:51:27.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ac173e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1abc0c41f0 0x7f1abc13b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:27.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ac173e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1abc0c9bb0 0x7f1abc139690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:27.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ac173e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc153770 0x7f1abc1f50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:27.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ac173e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc0b39f0 con 0x7f1abc153770 2023-10-16T20:51:27.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ac173e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc0b4570 con 0x7f1abc0c9bb0 2023-10-16T20:51:27.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.195+0000 7f1ac173e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc145680 con 0x7f1abc0c41f0 2023-10-16T20:51:27.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1abb7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc153770 0x7f1abc1f50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:27.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1aba7fc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1abc0c9bb0 0x7f1abc139690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:27.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1aba7fc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1abc0c9bb0 0x7f1abc139690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:47572/0 (socket says 172.21.15.88:47572) 2023-10-16T20:51:27.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1aba7fc640 1 -- 172.21.15.88:0/3453788702 learned_addr learned my addr 172.21.15.88:0/3453788702 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:27.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1abaffd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1abc0c41f0 0x7f1abc13b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:27.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1abb7fe640 1 --2- 172.21.15.88:0/3453788702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc153770 0x7f1abc1f50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33222/0 (socket says 172.21.15.88:33222) 2023-10-16T20:51:27.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1abb7fe640 1 -- 172.21.15.88:0/3453788702 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1abc0c41f0 msgr2=0x7f1abc13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:27.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1abb7fe640 1 --2- 172.21.15.88:0/3453788702 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1abc0c41f0 0x7f1abc13b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:27.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1abb7fe640 1 -- 172.21.15.88:0/3453788702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1abc0c9bb0 msgr2=0x7f1abc139690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:27.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1abb7fe640 1 --2- 172.21.15.88:0/3453788702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1abc0c9bb0 0x7f1abc139690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:27.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1abb7fe640 1 -- 172.21.15.88:0/3453788702 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab00094d0 con 0x7f1abc153770 2023-10-16T20:51:27.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1aba7fc640 1 --2- 172.21.15.88:0/3453788702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1abc0c9bb0 0x7f1abc139690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:51:27.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1abb7fe640 1 --2- 172.21.15.88:0/3453788702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc153770 0x7f1abc1f50d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1ab400eab0 tx=0x7f1ab400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:27.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1aa37fe640 1 -- 172.21.15.88:0/3453788702 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4010040 con 0x7f1abc153770 2023-10-16T20:51:27.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1aa37fe640 1 -- 172.21.15.88:0/3453788702 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab400bc00 con 0x7f1abc153770 2023-10-16T20:51:27.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1ac173e640 1 -- 172.21.15.88:0/3453788702 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1abc145b20 con 0x7f1abc153770 2023-10-16T20:51:27.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1aa37fe640 1 -- 172.21.15.88:0/3453788702 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4014400 con 0x7f1abc153770 2023-10-16T20:51:27.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1ac173e640 1 -- 172.21.15.88:0/3453788702 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1abc0b3c40 con 0x7f1abc153770 2023-10-16T20:51:27.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1ac173e640 1 -- 172.21.15.88:0/3453788702 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1abc074700 con 0x7f1abc153770 2023-10-16T20:51:27.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1aa37fe640 1 -- 172.21.15.88:0/3453788702 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1ab4014560 con 0x7f1abc153770 2023-10-16T20:51:27.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1aa37fe640 1 --2- 172.21.15.88:0/3453788702 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1aa80664a0 0x7f1aa8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:27.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.199+0000 7f1aa37fe640 1 -- 172.21.15.88:0/3453788702 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7f1ab4084d90 con 0x7f1abc153770 2023-10-16T20:51:27.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.203+0000 7f1abaffd640 1 --2- 172.21.15.88:0/3453788702 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1aa80664a0 0x7f1aa8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:27.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.203+0000 7f1abaffd640 1 --2- 172.21.15.88:0/3453788702 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1aa80664a0 0x7f1aa8068960 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009b20 tx=0x7f1aac006f90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:27.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.207+0000 7f1aa37fe640 1 -- 172.21.15.88:0/3453788702 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1ab4051c60 con 0x7f1abc153770 2023-10-16T20:51:27.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.391+0000 7f1ac173e640 1 -- 172.21.15.88:0/3453788702 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1abc0ce390 con 0x7f1abc153770 2023-10-16T20:51:27.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.391+0000 7f1aa37fe640 1 -- 172.21.15.88:0/3453788702 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5775 (secure 0 0 0) 0x7f1ab4018090 con 0x7f1abc153770 2023-10-16T20:51:27.392 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:27.392 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:51:20.522658+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:27.393 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 81 2023-10-16T20:51:27.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.391+0000 7f1ac173e640 1 -- 172.21.15.88:0/3453788702 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1aa80664a0 msgr2=0x7f1aa8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:27.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.391+0000 7f1ac173e640 1 --2- 172.21.15.88:0/3453788702 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1aa80664a0 0x7f1aa8068960 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009b20 tx=0x7f1aac006f90 comp rx=0 tx=0).stop 2023-10-16T20:51:27.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.391+0000 7f1ac173e640 1 -- 172.21.15.88:0/3453788702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc153770 msgr2=0x7f1abc1f50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:27.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.391+0000 7f1ac173e640 1 --2- 172.21.15.88:0/3453788702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc153770 0x7f1abc1f50d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1ab400eab0 tx=0x7f1ab400ef70 comp rx=0 tx=0).stop 2023-10-16T20:51:27.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.391+0000 7f1ac173e640 1 -- 172.21.15.88:0/3453788702 shutdown_connections 2023-10-16T20:51:27.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.391+0000 7f1ac173e640 1 --2- 172.21.15.88:0/3453788702 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1aa80664a0 0x7f1aa8068960 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:27.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.391+0000 7f1ac173e640 1 --2- 172.21.15.88:0/3453788702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc153770 0x7f1abc1f50d0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:27.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.391+0000 7f1ac173e640 1 --2- 172.21.15.88:0/3453788702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1abc0c9bb0 0x7f1abc139690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:27.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.391+0000 7f1ac173e640 1 --2- 172.21.15.88:0/3453788702 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1abc0c41f0 0x7f1abc13b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:27.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.391+0000 7f1ac173e640 1 -- 172.21.15.88:0/3453788702 >> 172.21.15.88:0/3453788702 conn(0x7f1abc0715c0 msgr2=0x7f1abc146b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:27.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.391+0000 7f1ac173e640 1 -- 172.21.15.88:0/3453788702 shutdown_connections 2023-10-16T20:51:27.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:27.391+0000 7f1ac173e640 1 -- 172.21.15.88:0/3453788702 wait complete. 2023-10-16T20:51:27.413 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:51:29.321 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:51:29.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:51:29.414 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:29.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b98454640 1 Processor -- start 2023-10-16T20:51:29.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b98454640 1 -- start start 2023-10-16T20:51:29.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b98454640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b900cece0 0x7f8b900cf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:29.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b98454640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b90153b50 con 0x7f8b900c9bb0 2023-10-16T20:51:29.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b98454640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b900cb2f0 con 0x7f8b90153770 2023-10-16T20:51:29.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b98454640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b900cc250 con 0x7f8b900cece0 2023-10-16T20:51:29.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b959ef640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8b900c9bb0 0x7f8b900c3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44440/0 (socket says 172.21.15.88:44440) 2023-10-16T20:51:29.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b959ef640 1 -- 172.21.15.88:0/3407893968 learned_addr learned my addr 172.21.15.88:0/3407893968 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:51:29.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b969f1640 1 --2- 172.21.15.88:0/3407893968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b900cece0 0x7f8b900cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:29.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b951ee640 1 -- 172.21.15.88:0/3407893968 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656453063 0 0) 0x7f8b90153b50 con 0x7f8b900c9bb0 2023-10-16T20:51:29.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b951ee640 1 -- 172.21.15.88:0/3407893968 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b7c003530 con 0x7f8b900c9bb0 2023-10-16T20:51:29.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b951ee640 1 -- 172.21.15.88:0/3407893968 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642833035 0 0) 0x7f8b900cb2f0 con 0x7f8b90153770 2023-10-16T20:51:29.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b951ee640 1 -- 172.21.15.88:0/3407893968 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b90153b50 con 0x7f8b90153770 2023-10-16T20:51:29.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b951ee640 1 -- 172.21.15.88:0/3407893968 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2168680352 0 0) 0x7f8b7c003530 con 0x7f8b900c9bb0 2023-10-16T20:51:29.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b951ee640 1 -- 172.21.15.88:0/3407893968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b900cece0 msgr2=0x7f8b900cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:29.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b951ee640 1 --2- 172.21.15.88:0/3407893968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b900cece0 0x7f8b900cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:29.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b951ee640 1 -- 172.21.15.88:0/3407893968 >> v1:172.21.15.191:6789/0 conn(0x7f8b90153770 legacy=0x7f8b90159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:29.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.563+0000 7f8b951ee640 1 -- 172.21.15.88:0/3407893968 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b900cb5d0 con 0x7f8b900c9bb0 2023-10-16T20:51:29.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b951ee640 1 -- 172.21.15.88:0/3407893968 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f8b80001680 con 0x7f8b900c9bb0 2023-10-16T20:51:29.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b951ee640 1 -- 172.21.15.88:0/3407893968 >> v1:172.21.15.88:6789/0 conn(0x7f8b900c9bb0 legacy=0x7f8b900c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:29.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b951ee640 1 --2- 172.21.15.88:0/3407893968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b900cece0 0x7f8b7c003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:29.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b951ee640 1 --2- 172.21.15.88:0/3407893968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b7c0040a0 0x7f8b7c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:29.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b951ee640 1 --2- 172.21.15.88:0/3407893968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b7c004d10 0x7f8b7c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:29.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b951ee640 1 -- 172.21.15.88:0/3407893968 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b900cc250 con 0x7f8b7c004d10 2023-10-16T20:51:29.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b951ee640 1 -- 172.21.15.88:0/3407893968 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c00ba70 con 0x7f8b7c0040a0 2023-10-16T20:51:29.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b951ee640 1 -- 172.21.15.88:0/3407893968 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c00bc30 con 0x7f8b900cece0 2023-10-16T20:51:29.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b969f1640 1 --2- 172.21.15.88:0/3407893968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b7c004d10 0x7f8b7c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:29.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b961f0640 1 --2- 172.21.15.88:0/3407893968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b900cece0 0x7f8b7c003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:29.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b959ef640 1 --2- 172.21.15.88:0/3407893968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b7c0040a0 0x7f8b7c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:29.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b969f1640 1 -- 172.21.15.88:0/3407893968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b900cece0 msgr2=0x7f8b7c003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:29.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b969f1640 1 --2- 172.21.15.88:0/3407893968 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b900cece0 0x7f8b7c003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:29.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b969f1640 1 -- 172.21.15.88:0/3407893968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b7c0040a0 msgr2=0x7f8b7c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:29.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b969f1640 1 --2- 172.21.15.88:0/3407893968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b7c0040a0 0x7f8b7c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:29.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b969f1640 1 -- 172.21.15.88:0/3407893968 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b7c003530 con 0x7f8b7c004d10 2023-10-16T20:51:29.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b959ef640 1 --2- 172.21.15.88:0/3407893968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b7c0040a0 0x7f8b7c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:51:29.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b969f1640 1 --2- 172.21.15.88:0/3407893968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b7c004d10 0x7f8b7c005190 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8b88009fc0 tx=0x7f8b8801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c99c0cde21b4b176 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:29.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b951ee640 1 -- 172.21.15.88:0/3407893968 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b880a5b60 con 0x7f8b7c004d10 2023-10-16T20:51:29.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b951ee640 1 -- 172.21.15.88:0/3407893968 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8800f040 con 0x7f8b7c004d10 2023-10-16T20:51:29.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.567+0000 7f8b951ee640 1 -- 172.21.15.88:0/3407893968 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b880ad440 con 0x7f8b7c004d10 2023-10-16T20:51:29.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b98454640 1 -- 172.21.15.88:0/3407893968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b7c004d10 msgr2=0x7f8b7c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:29.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b98454640 1 --2- 172.21.15.88:0/3407893968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b7c004d10 0x7f8b7c005190 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8b88009fc0 tx=0x7f8b8801cc60 comp rx=0 tx=0).stop 2023-10-16T20:51:29.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b969f1640 1 -- 172.21.15.88:0/3407893968 reap_dead start 2023-10-16T20:51:29.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b98454640 1 -- 172.21.15.88:0/3407893968 shutdown_connections 2023-10-16T20:51:29.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b98454640 1 -- 172.21.15.88:0/3407893968 >> 172.21.15.88:0/3407893968 conn(0x7f8b900715c0 msgr2=0x7f8b900719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:29.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b98454640 1 -- 172.21.15.88:0/3407893968 shutdown_connections 2023-10-16T20:51:29.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b98454640 1 -- 172.21.15.88:0/3407893968 wait complete. 2023-10-16T20:51:29.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b98454640 1 Processor -- start 2023-10-16T20:51:29.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b98454640 1 -- start start 2023-10-16T20:51:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b98454640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b7c004d10 0x7f8b901516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b98454640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b900c8940 0x7f8b9014ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b98454640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b9014dbf0 0x7f8b901f17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b98454640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b901ee1f0 con 0x7f8b900c8940 2023-10-16T20:51:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b98454640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b901ef1c0 con 0x7f8b7c004d10 2023-10-16T20:51:29.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b98454640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b900c9160 con 0x7f8b9014dbf0 2023-10-16T20:51:29.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b959ef640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b900c8940 0x7f8b9014ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:29.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b959ef640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b900c8940 0x7f8b9014ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47654/0 (socket says 172.21.15.88:47654) 2023-10-16T20:51:29.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b959ef640 1 -- 172.21.15.88:0/2578439859 learned_addr learned my addr 172.21.15.88:0/2578439859 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:29.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b961f0640 1 --2- 172.21.15.88:0/2578439859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b7c004d10 0x7f8b901516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:29.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.571+0000 7f8b969f1640 1 --2- 172.21.15.88:0/2578439859 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b9014dbf0 0x7f8b901f17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:29.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.575+0000 7f8b959ef640 1 -- 172.21.15.88:0/2578439859 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b9014dbf0 msgr2=0x7f8b901f17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:29.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.575+0000 7f8b959ef640 1 --2- 172.21.15.88:0/2578439859 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b9014dbf0 0x7f8b901f17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:29.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.575+0000 7f8b959ef640 1 -- 172.21.15.88:0/2578439859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b7c004d10 msgr2=0x7f8b901516f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:29.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.575+0000 7f8b959ef640 1 --2- 172.21.15.88:0/2578439859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b7c004d10 0x7f8b901516f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:29.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.575+0000 7f8b959ef640 1 -- 172.21.15.88:0/2578439859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b900c8e80 con 0x7f8b900c8940 2023-10-16T20:51:29.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.575+0000 7f8b959ef640 1 --2- 172.21.15.88:0/2578439859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b900c8940 0x7f8b9014ce10 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8b800118a0 tx=0x7f8b80011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:29.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.575+0000 7f8b86ffd640 1 -- 172.21.15.88:0/2578439859 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8000fb00 con 0x7f8b900c8940 2023-10-16T20:51:29.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.575+0000 7f8b86ffd640 1 -- 172.21.15.88:0/2578439859 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8000fc60 con 0x7f8b900c8940 2023-10-16T20:51:29.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.575+0000 7f8b86ffd640 1 -- 172.21.15.88:0/2578439859 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b80015410 con 0x7f8b900c8940 2023-10-16T20:51:29.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.575+0000 7f8b98454640 1 -- 172.21.15.88:0/2578439859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b901ee400 con 0x7f8b900c8940 2023-10-16T20:51:29.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.575+0000 7f8b98454640 1 -- 172.21.15.88:0/2578439859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b901ee9c0 con 0x7f8b900c8940 2023-10-16T20:51:29.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.575+0000 7f8b98454640 1 -- 172.21.15.88:0/2578439859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b90074700 con 0x7f8b900c8940 2023-10-16T20:51:29.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.575+0000 7f8b86ffd640 1 -- 172.21.15.88:0/2578439859 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8b80015570 con 0x7f8b900c8940 2023-10-16T20:51:29.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.575+0000 7f8b86ffd640 1 --2- 172.21.15.88:0/2578439859 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8b740664a0 0x7f8b74068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:29.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.575+0000 7f8b86ffd640 1 -- 172.21.15.88:0/2578439859 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7f8b80087600 con 0x7f8b900c8940 2023-10-16T20:51:29.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.575+0000 7f8b961f0640 1 --2- 172.21.15.88:0/2578439859 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8b740664a0 0x7f8b74068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:29.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.579+0000 7f8b961f0640 1 --2- 172.21.15.88:0/2578439859 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8b740664a0 0x7f8b74068960 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c001970 tx=0x7f8b8c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:29.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.579+0000 7f8b86ffd640 1 -- 172.21.15.88:0/2578439859 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8b800544c0 con 0x7f8b900c8940 2023-10-16T20:51:29.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.767+0000 7f8b98454640 1 -- 172.21.15.88:0/2578439859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b900ca740 con 0x7f8b900c8940 2023-10-16T20:51:29.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.767+0000 7f8b86ffd640 1 -- 172.21.15.88:0/2578439859 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f8b80057ae0 con 0x7f8b900c8940 2023-10-16T20:51:29.769 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:29.769 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:51:27.190500+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:resolve","state_seq":35,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:29.769 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 82 2023-10-16T20:51:29.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.767+0000 7f8b98454640 1 -- 172.21.15.88:0/2578439859 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8b740664a0 msgr2=0x7f8b74068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:29.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.767+0000 7f8b98454640 1 --2- 172.21.15.88:0/2578439859 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8b740664a0 0x7f8b74068960 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c001970 tx=0x7f8b8c009040 comp rx=0 tx=0).stop 2023-10-16T20:51:29.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.767+0000 7f8b98454640 1 -- 172.21.15.88:0/2578439859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b900c8940 msgr2=0x7f8b9014ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:29.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.767+0000 7f8b98454640 1 --2- 172.21.15.88:0/2578439859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b900c8940 0x7f8b9014ce10 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8b800118a0 tx=0x7f8b80011ca0 comp rx=0 tx=0).stop 2023-10-16T20:51:29.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.771+0000 7f8b98454640 1 -- 172.21.15.88:0/2578439859 shutdown_connections 2023-10-16T20:51:29.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.771+0000 7f8b98454640 1 --2- 172.21.15.88:0/2578439859 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8b740664a0 0x7f8b74068960 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:29.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.771+0000 7f8b98454640 1 --2- 172.21.15.88:0/2578439859 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8b9014dbf0 0x7f8b901f17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:29.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.771+0000 7f8b98454640 1 --2- 172.21.15.88:0/2578439859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b900c8940 0x7f8b9014ce10 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:29.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.771+0000 7f8b98454640 1 --2- 172.21.15.88:0/2578439859 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8b7c004d10 0x7f8b901516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:29.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.771+0000 7f8b98454640 1 -- 172.21.15.88:0/2578439859 >> 172.21.15.88:0/2578439859 conn(0x7f8b900715c0 msgr2=0x7f8b900cddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:29.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.771+0000 7f8b98454640 1 -- 172.21.15.88:0/2578439859 shutdown_connections 2023-10-16T20:51:29.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:29.771+0000 7f8b98454640 1 -- 172.21.15.88:0/2578439859 wait complete. 2023-10-16T20:51:29.789 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:51:31.790 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:31.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.935+0000 7f02e453e640 1 Processor -- start 2023-10-16T20:51:31.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.935+0000 7f02e453e640 1 -- start start 2023-10-16T20:51:31.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.939+0000 7f02e453e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f02dc0c5300 0x7f02dc13e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:31.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.939+0000 7f02e453e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f02dc15d840 0x7f02dc159640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:31.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.939+0000 7f02e453e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02dc15a590 con 0x7f02dc0cb890 2023-10-16T20:51:31.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.939+0000 7f02e453e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02dc15d420 con 0x7f02dc15d840 2023-10-16T20:51:31.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.939+0000 7f02e453e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02dc0c7830 con 0x7f02dc0c5300 2023-10-16T20:51:31.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.939+0000 7f02e1ad9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f02dc15d840 0x7f02dc159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:31.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.939+0000 7f02e1ad9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f02dc15d840 0x7f02dc159640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47664/0 (socket says 172.21.15.88:47664) 2023-10-16T20:51:31.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.939+0000 7f02e1ad9640 1 -- 172.21.15.88:0/44579666 learned_addr learned my addr 172.21.15.88:0/44579666 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:31.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.939+0000 7f02e22da640 1 --2- 172.21.15.88:0/44579666 >> v2:172.21.15.191:3300/0 conn(0x7f02dc0c5300 0x7f02dc13e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:31.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.939+0000 7f02e1ad9640 1 -- 172.21.15.88:0/44579666 >> v2:172.21.15.191:3300/0 conn(0x7f02dc0c5300 msgr2=0x7f02dc13e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:31.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.939+0000 7f02e1ad9640 1 --2- 172.21.15.88:0/44579666 >> v2:172.21.15.191:3300/0 conn(0x7f02dc0c5300 0x7f02dc13e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:31.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.939+0000 7f02e1ad9640 1 -- 172.21.15.88:0/44579666 >> v1:172.21.15.191:6789/0 conn(0x7f02dc0cb890 legacy=0x7f02dc06ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:31.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.939+0000 7f02e1ad9640 1 -- 172.21.15.88:0/44579666 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02dc13ed50 con 0x7f02dc15d840 2023-10-16T20:51:31.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.943+0000 7f02e1ad9640 1 --2- 172.21.15.88:0/44579666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02dc15d840 0x7f02dc159640 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f02dc0c8d20 tx=0x7f02cc01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36af04235db61827 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:31.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.943+0000 7f02e12d8640 1 -- 172.21.15.88:0/44579666 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc0ab600 con 0x7f02dc15d840 2023-10-16T20:51:31.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.943+0000 7f02e12d8640 1 -- 172.21.15.88:0/44579666 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02cc0ab760 con 0x7f02dc15d840 2023-10-16T20:51:31.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.943+0000 7f02e12d8640 1 -- 172.21.15.88:0/44579666 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc0abaf0 con 0x7f02dc15d840 2023-10-16T20:51:31.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.943+0000 7f02e453e640 1 -- 172.21.15.88:0/44579666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02dc15d840 msgr2=0x7f02dc159640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:31.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.943+0000 7f02e453e640 1 --2- 172.21.15.88:0/44579666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02dc15d840 0x7f02dc159640 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f02dc0c8d20 tx=0x7f02cc01c400 comp rx=0 tx=0).stop 2023-10-16T20:51:31.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.943+0000 7f02e453e640 1 -- 172.21.15.88:0/44579666 shutdown_connections 2023-10-16T20:51:31.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.943+0000 7f02e453e640 1 --2- 172.21.15.88:0/44579666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02dc15d840 0x7f02dc159640 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:31.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.943+0000 7f02e453e640 1 --2- 172.21.15.88:0/44579666 >> v2:172.21.15.191:3300/0 conn(0x7f02dc0c5300 0x7f02dc13e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:31.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.943+0000 7f02e453e640 1 -- 172.21.15.88:0/44579666 >> 172.21.15.88:0/44579666 conn(0x7f02dc06e0f0 msgr2=0x7f02dc0cb180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:31.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.943+0000 7f02e453e640 1 -- 172.21.15.88:0/44579666 shutdown_connections 2023-10-16T20:51:31.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.943+0000 7f02e453e640 1 -- 172.21.15.88:0/44579666 wait complete. 2023-10-16T20:51:31.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.943+0000 7f02e453e640 1 Processor -- start 2023-10-16T20:51:31.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.943+0000 7f02e453e640 1 -- start start 2023-10-16T20:51:31.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e453e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02dc0c5300 0x7f02dc06ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:31.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e453e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02dc0cb890 0x7f02dc1ebf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:31.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e453e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02dc15d840 0x7f02dc158570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:31.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e453e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc157fa0 con 0x7f02dc0c5300 2023-10-16T20:51:31.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e453e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc0c3360 con 0x7f02dc0cb890 2023-10-16T20:51:31.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e453e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc0c3640 con 0x7f02dc15d840 2023-10-16T20:51:31.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e22da640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02dc0c5300 0x7f02dc06ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:31.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e22da640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02dc0c5300 0x7f02dc06ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47668/0 (socket says 172.21.15.88:47668) 2023-10-16T20:51:31.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e22da640 1 -- 172.21.15.88:0/291352651 learned_addr learned my addr 172.21.15.88:0/291352651 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:31.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e1ad9640 1 --2- 172.21.15.88:0/291352651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02dc0cb890 0x7f02dc1ebf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:31.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e2adb640 1 --2- 172.21.15.88:0/291352651 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02dc15d840 0x7f02dc158570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:31.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e22da640 1 -- 172.21.15.88:0/291352651 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02dc15d840 msgr2=0x7f02dc158570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:31.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e22da640 1 --2- 172.21.15.88:0/291352651 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02dc15d840 0x7f02dc158570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:31.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e22da640 1 -- 172.21.15.88:0/291352651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02dc0cb890 msgr2=0x7f02dc1ebf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:31.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e22da640 1 --2- 172.21.15.88:0/291352651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02dc0cb890 0x7f02dc1ebf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:31.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e22da640 1 -- 172.21.15.88:0/291352651 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc0094d0 con 0x7f02dc0c5300 2023-10-16T20:51:31.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e22da640 1 --2- 172.21.15.88:0/291352651 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02dc0c5300 0x7f02dc06ffe0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f02d800d5f0 tx=0x7f02d800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:31.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02d2ffd640 1 -- 172.21.15.88:0/291352651 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d80125f0 con 0x7f02dc0c5300 2023-10-16T20:51:31.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02d2ffd640 1 -- 172.21.15.88:0/291352651 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d8012750 con 0x7f02dc0c5300 2023-10-16T20:51:31.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e453e640 1 -- 172.21.15.88:0/291352651 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02dc0c3920 con 0x7f02dc0c5300 2023-10-16T20:51:31.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02d2ffd640 1 -- 172.21.15.88:0/291352651 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d8012ae0 con 0x7f02dc0c5300 2023-10-16T20:51:31.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e453e640 1 -- 172.21.15.88:0/291352651 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02dc0c3ee0 con 0x7f02dc0c5300 2023-10-16T20:51:31.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02e453e640 1 -- 172.21.15.88:0/291352651 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02dc074700 con 0x7f02dc0c5300 2023-10-16T20:51:31.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.947+0000 7f02d2ffd640 1 -- 172.21.15.88:0/291352651 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f02d8012c40 con 0x7f02dc0c5300 2023-10-16T20:51:31.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.951+0000 7f02d2ffd640 1 --2- 172.21.15.88:0/291352651 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02c00664d0 0x7f02c0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:31.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.951+0000 7f02d2ffd640 1 -- 172.21.15.88:0/291352651 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7f02d80849a0 con 0x7f02dc0c5300 2023-10-16T20:51:31.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.951+0000 7f02e1ad9640 1 --2- 172.21.15.88:0/291352651 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02c00664d0 0x7f02c0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:31.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.951+0000 7f02e1ad9640 1 --2- 172.21.15.88:0/291352651 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02c00664d0 0x7f02c0068990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0af000 tx=0x7f02cc0b6040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:31.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:31.955+0000 7f02d2ffd640 1 -- 172.21.15.88:0/291352651 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f02d8051730 con 0x7f02dc0c5300 2023-10-16T20:51:32.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:32.135+0000 7f02e453e640 1 -- 172.21.15.88:0/291352651 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02dc06aa70 con 0x7f02dc0c5300 2023-10-16T20:51:32.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:32.135+0000 7f02d2ffd640 1 -- 172.21.15.88:0/291352651 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f02d8054d50 con 0x7f02dc0c5300 2023-10-16T20:51:32.138 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:32.138 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:51:27.190500+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:resolve","state_seq":35,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:32.139 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 82 2023-10-16T20:51:32.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:32.139+0000 7f02e453e640 1 -- 172.21.15.88:0/291352651 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02c00664d0 msgr2=0x7f02c0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:32.139+0000 7f02e453e640 1 --2- 172.21.15.88:0/291352651 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02c00664d0 0x7f02c0068990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0af000 tx=0x7f02cc0b6040 comp rx=0 tx=0).stop 2023-10-16T20:51:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:32.139+0000 7f02e453e640 1 -- 172.21.15.88:0/291352651 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02dc0c5300 msgr2=0x7f02dc06ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:32.139+0000 7f02e453e640 1 --2- 172.21.15.88:0/291352651 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02dc0c5300 0x7f02dc06ffe0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f02d800d5f0 tx=0x7f02d800dab0 comp rx=0 tx=0).stop 2023-10-16T20:51:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:32.139+0000 7f02e453e640 1 -- 172.21.15.88:0/291352651 shutdown_connections 2023-10-16T20:51:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:32.139+0000 7f02e453e640 1 --2- 172.21.15.88:0/291352651 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f02c00664d0 0x7f02c0068990 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:32.139+0000 7f02e453e640 1 --2- 172.21.15.88:0/291352651 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f02dc15d840 0x7f02dc158570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:32.139+0000 7f02e453e640 1 --2- 172.21.15.88:0/291352651 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f02dc0cb890 0x7f02dc1ebf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:32.139+0000 7f02e453e640 1 --2- 172.21.15.88:0/291352651 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02dc0c5300 0x7f02dc06ffe0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:32.139+0000 7f02e453e640 1 -- 172.21.15.88:0/291352651 >> 172.21.15.88:0/291352651 conn(0x7f02dc06e0f0 msgr2=0x7f02dc13d320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:32.139+0000 7f02e453e640 1 -- 172.21.15.88:0/291352651 shutdown_connections 2023-10-16T20:51:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:32.139+0000 7f02e453e640 1 -- 172.21.15.88:0/291352651 wait complete. 2023-10-16T20:51:32.158 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:51:34.159 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:34.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03eda74640 1 Processor -- start 2023-10-16T20:51:34.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03eda74640 1 -- start start 2023-10-16T20:51:34.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03eda74640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f03e80c9af0 0x7f03e80cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:34.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03eda74640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03e80b1a60 0x7f03e8163b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:34.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03eda74640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03e80c9fc0 con 0x7f03e80cb580 2023-10-16T20:51:34.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03eda74640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03e80b20c0 con 0x7f03e80c9af0 2023-10-16T20:51:34.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03eda74640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e80cbea0 con 0x7f03e80b1a60 2023-10-16T20:51:34.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03eca72640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f03e80cb580 0x7f03e8159300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44450/0 (socket says 172.21.15.88:44450) 2023-10-16T20:51:34.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03eca72640 1 -- 172.21.15.88:0/316565933 learned_addr learned my addr 172.21.15.88:0/316565933 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:51:34.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e77fe640 1 -- 172.21.15.88:0/316565933 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170757623 0 0) 0x7f03e80c9fc0 con 0x7f03e80cb580 2023-10-16T20:51:34.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e77fe640 1 -- 172.21.15.88:0/316565933 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03d8003530 con 0x7f03e80cb580 2023-10-16T20:51:34.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03ed273640 1 --2- 172.21.15.88:0/316565933 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03e80b1a60 0x7f03e8163b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:34.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e7fff640 1 --2- 172.21.15.88:0/316565933 >> v2:172.21.15.191:3300/0 conn(0x7f03e80c9af0 0x7f03e80cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:34.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e77fe640 1 -- 172.21.15.88:0/316565933 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3919600584 0 0) 0x7f03d8003530 con 0x7f03e80cb580 2023-10-16T20:51:34.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e77fe640 1 -- 172.21.15.88:0/316565933 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03e80b1a60 msgr2=0x7f03e8163b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:34.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e77fe640 1 --2- 172.21.15.88:0/316565933 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03e80b1a60 0x7f03e8163b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:34.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e77fe640 1 -- 172.21.15.88:0/316565933 >> v2:172.21.15.191:3300/0 conn(0x7f03e80c9af0 msgr2=0x7f03e80cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:34.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e77fe640 1 --2- 172.21.15.88:0/316565933 >> v2:172.21.15.191:3300/0 conn(0x7f03e80c9af0 0x7f03e80cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:34.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e77fe640 1 -- 172.21.15.88:0/316565933 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e80c47d0 con 0x7f03e80cb580 2023-10-16T20:51:34.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e77fe640 1 -- 172.21.15.88:0/316565933 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f03dc001680 con 0x7f03e80cb580 2023-10-16T20:51:34.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e77fe640 1 -- 172.21.15.88:0/316565933 >> v1:172.21.15.88:6789/0 conn(0x7f03e80cb580 legacy=0x7f03e8159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:34.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e77fe640 1 --2- 172.21.15.88:0/316565933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03d8003980 0x7f03d8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:34.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e77fe640 1 --2- 172.21.15.88:0/316565933 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03e80b1a60 0x7f03d80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:34.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e77fe640 1 --2- 172.21.15.88:0/316565933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d8004d10 0x7f03d8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:34.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e77fe640 1 -- 172.21.15.88:0/316565933 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e80b20c0 con 0x7f03d8004d10 2023-10-16T20:51:34.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e77fe640 1 -- 172.21.15.88:0/316565933 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e80cbea0 con 0x7f03d8003980 2023-10-16T20:51:34.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e77fe640 1 -- 172.21.15.88:0/316565933 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d800b9c0 con 0x7f03e80b1a60 2023-10-16T20:51:34.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03ed273640 1 --2- 172.21.15.88:0/316565933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d8004d10 0x7f03d8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:34.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03e7fff640 1 --2- 172.21.15.88:0/316565933 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03e80b1a60 0x7f03d80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:34.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03eca72640 1 --2- 172.21.15.88:0/316565933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03d8003980 0x7f03d8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:34.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03ed273640 1 -- 172.21.15.88:0/316565933 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03e80b1a60 msgr2=0x7f03d80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:34.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03ed273640 1 --2- 172.21.15.88:0/316565933 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03e80b1a60 0x7f03d80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:34.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03ed273640 1 -- 172.21.15.88:0/316565933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03d8003980 msgr2=0x7f03d8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:34.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03ed273640 1 --2- 172.21.15.88:0/316565933 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03d8003980 0x7f03d8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:34.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.307+0000 7f03ed273640 1 -- 172.21.15.88:0/316565933 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d8003530 con 0x7f03d8004d10 2023-10-16T20:51:34.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.311+0000 7f03ed273640 1 --2- 172.21.15.88:0/316565933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d8004d10 0x7f03d8005190 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f03e0009f90 tx=0x7f03e001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff3785312f654b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:34.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.311+0000 7f03e77fe640 1 -- 172.21.15.88:0/316565933 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e00a5c30 con 0x7f03d8004d10 2023-10-16T20:51:34.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.311+0000 7f03e77fe640 1 -- 172.21.15.88:0/316565933 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e000f040 con 0x7f03d8004d10 2023-10-16T20:51:34.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.311+0000 7f03e77fe640 1 -- 172.21.15.88:0/316565933 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e00ad440 con 0x7f03d8004d10 2023-10-16T20:51:34.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.311+0000 7f03eda74640 1 -- 172.21.15.88:0/316565933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d8004d10 msgr2=0x7f03d8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:34.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.311+0000 7f03eda74640 1 --2- 172.21.15.88:0/316565933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d8004d10 0x7f03d8005190 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f03e0009f90 tx=0x7f03e001cc20 comp rx=0 tx=0).stop 2023-10-16T20:51:34.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.311+0000 7f03ed273640 1 -- 172.21.15.88:0/316565933 reap_dead start 2023-10-16T20:51:34.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.311+0000 7f03eda74640 1 -- 172.21.15.88:0/316565933 shutdown_connections 2023-10-16T20:51:34.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.311+0000 7f03eda74640 1 -- 172.21.15.88:0/316565933 >> 172.21.15.88:0/316565933 conn(0x7f03e806bb40 msgr2=0x7f03e806bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:34.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.311+0000 7f03eda74640 1 -- 172.21.15.88:0/316565933 shutdown_connections 2023-10-16T20:51:34.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.311+0000 7f03eda74640 1 -- 172.21.15.88:0/316565933 wait complete. 2023-10-16T20:51:34.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03eda74640 1 Processor -- start 2023-10-16T20:51:34.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03eda74640 1 -- start start 2023-10-16T20:51:34.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03eda74640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03d8004d10 0x7f03e8158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:34.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03eda74640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03e814d1b0 0x7f03e81585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:34.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03eda74640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e80c64f0 0x7f03e80c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:34.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03eda74640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e814a4d0 con 0x7f03e80c64f0 2023-10-16T20:51:34.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03eda74640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e814ce30 con 0x7f03e814d1b0 2023-10-16T20:51:34.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03eda74640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e80c60a0 con 0x7f03d8004d10 2023-10-16T20:51:34.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03ed273640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e80c64f0 0x7f03e80c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:34.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03ed273640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e80c64f0 0x7f03e80c6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47686/0 (socket says 172.21.15.88:47686) 2023-10-16T20:51:34.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03ed273640 1 -- 172.21.15.88:0/1145940958 learned_addr learned my addr 172.21.15.88:0/1145940958 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:34.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03eca72640 1 --2- 172.21.15.88:0/1145940958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03d8004d10 0x7f03e8158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:34.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03e7fff640 1 --2- 172.21.15.88:0/1145940958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03e814d1b0 0x7f03e81585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:34.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03ed273640 1 -- 172.21.15.88:0/1145940958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03d8004d10 msgr2=0x7f03e8158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:34.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03ed273640 1 --2- 172.21.15.88:0/1145940958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03d8004d10 0x7f03e8158080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:34.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03ed273640 1 -- 172.21.15.88:0/1145940958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03e814d1b0 msgr2=0x7f03e81585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:34.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03ed273640 1 --2- 172.21.15.88:0/1145940958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03e814d1b0 0x7f03e81585c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:34.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03ed273640 1 -- 172.21.15.88:0/1145940958 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e80c6eb0 con 0x7f03e80c64f0 2023-10-16T20:51:34.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03ed273640 1 --2- 172.21.15.88:0/1145940958 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e80c64f0 0x7f03e80c6970 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f03e00b0040 tx=0x7f03e0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:34.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03e57fa640 1 -- 172.21.15.88:0/1145940958 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e00039a0 con 0x7f03e80c64f0 2023-10-16T20:51:34.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03e57fa640 1 -- 172.21.15.88:0/1145940958 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e0003b00 con 0x7f03e80c64f0 2023-10-16T20:51:34.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03eda74640 1 -- 172.21.15.88:0/1145940958 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e80c89b0 con 0x7f03e80c64f0 2023-10-16T20:51:34.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03e57fa640 1 -- 172.21.15.88:0/1145940958 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e00ad650 con 0x7f03e80c64f0 2023-10-16T20:51:34.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.315+0000 7f03eda74640 1 -- 172.21.15.88:0/1145940958 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e80c8c70 con 0x7f03e80c64f0 2023-10-16T20:51:34.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.319+0000 7f03eda74640 1 -- 172.21.15.88:0/1145940958 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e804e9c0 con 0x7f03e80c64f0 2023-10-16T20:51:34.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.319+0000 7f03e57fa640 1 -- 172.21.15.88:0/1145940958 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f03e00b4040 con 0x7f03e80c64f0 2023-10-16T20:51:34.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.319+0000 7f03e57fa640 1 --2- 172.21.15.88:0/1145940958 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f03cc0664a0 0x7f03cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:34.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.319+0000 7f03e57fa640 1 -- 172.21.15.88:0/1145940958 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7f03e011e0b0 con 0x7f03e80c64f0 2023-10-16T20:51:34.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.319+0000 7f03eca72640 1 --2- 172.21.15.88:0/1145940958 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f03cc0664a0 0x7f03cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:34.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.319+0000 7f03eca72640 1 --2- 172.21.15.88:0/1145940958 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f03cc0664a0 0x7f03cc068960 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f03dc000c10 tx=0x7f03dc007900 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:34.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.323+0000 7f03e57fa640 1 -- 172.21.15.88:0/1145940958 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f03e00eaf20 con 0x7f03e80c64f0 2023-10-16T20:51:34.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.503+0000 7f03eda74640 1 -- 172.21.15.88:0/1145940958 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03e80ca650 con 0x7f03e80c64f0 2023-10-16T20:51:34.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.507+0000 7f03e57fa640 1 -- 172.21.15.88:0/1145940958 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5779 (secure 0 0 0) 0x7f03e00b2090 con 0x7f03e80c64f0 2023-10-16T20:51:34.507 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:34.507 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:51:32.638867+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:34.508 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 83 2023-10-16T20:51:34.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.507+0000 7f03eda74640 1 -- 172.21.15.88:0/1145940958 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f03cc0664a0 msgr2=0x7f03cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:34.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.507+0000 7f03eda74640 1 --2- 172.21.15.88:0/1145940958 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f03cc0664a0 0x7f03cc068960 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f03dc000c10 tx=0x7f03dc007900 comp rx=0 tx=0).stop 2023-10-16T20:51:34.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.507+0000 7f03eda74640 1 -- 172.21.15.88:0/1145940958 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e80c64f0 msgr2=0x7f03e80c6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:34.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.507+0000 7f03eda74640 1 --2- 172.21.15.88:0/1145940958 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e80c64f0 0x7f03e80c6970 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f03e00b0040 tx=0x7f03e0009750 comp rx=0 tx=0).stop 2023-10-16T20:51:34.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.507+0000 7f03eda74640 1 -- 172.21.15.88:0/1145940958 shutdown_connections 2023-10-16T20:51:34.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.507+0000 7f03eda74640 1 --2- 172.21.15.88:0/1145940958 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f03cc0664a0 0x7f03cc068960 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:34.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.507+0000 7f03eda74640 1 --2- 172.21.15.88:0/1145940958 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e80c64f0 0x7f03e80c6970 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:34.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.507+0000 7f03eda74640 1 --2- 172.21.15.88:0/1145940958 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03e814d1b0 0x7f03e81585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:34.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.507+0000 7f03eda74640 1 --2- 172.21.15.88:0/1145940958 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f03d8004d10 0x7f03e8158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:34.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.507+0000 7f03eda74640 1 -- 172.21.15.88:0/1145940958 >> 172.21.15.88:0/1145940958 conn(0x7f03e806bb40 msgr2=0x7f03e814de60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:34.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.507+0000 7f03eda74640 1 -- 172.21.15.88:0/1145940958 shutdown_connections 2023-10-16T20:51:34.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:34.507+0000 7f03eda74640 1 -- 172.21.15.88:0/1145940958 wait complete. 2023-10-16T20:51:34.531 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 83, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 83, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4889": { "addr": "172.21.15.191:6837/2468647900", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6836", "nonce": 2468647900, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2468647900, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4889, "incarnation": 30, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 21 }, "gid_5282": { "addr": "172.21.15.191:6833/3633326602", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 3633326602, "type": "v2" }, { "addr": "172.21.15.191:6833", "nonce": 3633326602, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5282, "incarnation": 81, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 37 }, "gid_5284": { "addr": "172.21.15.88:6839/2172973775", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6837", "nonce": 2172973775, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2172973775, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5284, "incarnation": 38, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 17 }, "gid_5535": { "addr": "172.21.15.88:6835/3476086754", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3476086754, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3476086754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5535, "incarnation": 52, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_5709": { "addr": "172.21.15.88:6838/2600891060", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 2600891060, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 2600891060, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5709, "incarnation": 67, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:51:32.638867+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5282, "mds_1": 5709, "mds_2": 5535, "mds_3": 5284, "mds_4": 4889 } } } ], "standbys": [] } 2023-10-16T20:51:36.533 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:36.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.683+0000 7f68dfaea640 1 Processor -- start 2023-10-16T20:51:36.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.683+0000 7f68dfaea640 1 -- start start 2023-10-16T20:51:36.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.687+0000 7f68dfaea640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f68d80c6bf0 0x7f68d8159f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:36.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.687+0000 7f68dfaea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f68d80c9040 0x7f68d8071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:36.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.687+0000 7f68dfaea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d8069e90 con 0x7f68d8153770 2023-10-16T20:51:36.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.687+0000 7f68dfaea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d806aef0 con 0x7f68d80c9040 2023-10-16T20:51:36.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.687+0000 7f68dfaea640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d806a0c0 con 0x7f68d80c6bf0 2023-10-16T20:51:36.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.687+0000 7f68de087640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f68d80c9040 0x7f68d8071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:36.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.687+0000 7f68dd886640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f68d8153770 0x7f68d80c64e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44460/0 (socket says 172.21.15.88:44460) 2023-10-16T20:51:36.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.687+0000 7f68de087640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f68d80c9040 0x7f68d8071b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47702/0 (socket says 172.21.15.88:47702) 2023-10-16T20:51:36.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.687+0000 7f68dd886640 1 -- 172.21.15.88:0/2826870489 learned_addr learned my addr 172.21.15.88:0/2826870489 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:51:36.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.687+0000 7f68dd085640 1 --2- 172.21.15.88:0/2826870489 >> v2:172.21.15.191:3300/0 conn(0x7f68d80c6bf0 0x7f68d8159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:36.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.687+0000 7f68de087640 1 -- 172.21.15.88:0/2826870489 >> v2:172.21.15.191:3300/0 conn(0x7f68d80c6bf0 msgr2=0x7f68d8159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:36.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.687+0000 7f68de087640 1 --2- 172.21.15.88:0/2826870489 >> v2:172.21.15.191:3300/0 conn(0x7f68d80c6bf0 0x7f68d8159f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:36.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.687+0000 7f68de087640 1 -- 172.21.15.88:0/2826870489 >> v1:172.21.15.88:6789/0 conn(0x7f68d8153770 legacy=0x7f68d80c64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:36.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.687+0000 7f68de087640 1 -- 172.21.15.88:0/2826870489 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d806a360 con 0x7f68d80c9040 2023-10-16T20:51:36.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.687+0000 7f68dc884640 1 -- 172.21.15.88:0/2826870489 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212428440 0 0) 0x7f68d8069e90 con 0x7f68d8153770 2023-10-16T20:51:36.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.691+0000 7f68de087640 1 --2- 172.21.15.88:0/2826870489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68d80c9040 0x7f68d8071b20 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f68d80c95d0 tx=0x7f68d001c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83eb99ebb207d15 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:36.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.691+0000 7f68dc884640 1 -- 172.21.15.88:0/2826870489 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d00a58c0 con 0x7f68d80c9040 2023-10-16T20:51:36.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.691+0000 7f68dc884640 1 -- 172.21.15.88:0/2826870489 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d00a5a20 con 0x7f68d80c9040 2023-10-16T20:51:36.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.691+0000 7f68dc884640 1 -- 172.21.15.88:0/2826870489 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d00a5db0 con 0x7f68d80c9040 2023-10-16T20:51:36.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.691+0000 7f68dfaea640 1 -- 172.21.15.88:0/2826870489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68d80c9040 msgr2=0x7f68d8071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:36.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.691+0000 7f68dfaea640 1 --2- 172.21.15.88:0/2826870489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68d80c9040 0x7f68d8071b20 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f68d80c95d0 tx=0x7f68d001c8f0 comp rx=0 tx=0).stop 2023-10-16T20:51:36.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.691+0000 7f68dfaea640 1 -- 172.21.15.88:0/2826870489 shutdown_connections 2023-10-16T20:51:36.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.691+0000 7f68dfaea640 1 --2- 172.21.15.88:0/2826870489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68d80c9040 0x7f68d8071b20 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:36.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.691+0000 7f68dfaea640 1 --2- 172.21.15.88:0/2826870489 >> v2:172.21.15.191:3300/0 conn(0x7f68d80c6bf0 0x7f68d8159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:36.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.691+0000 7f68dfaea640 1 -- 172.21.15.88:0/2826870489 >> 172.21.15.88:0/2826870489 conn(0x7f68d80c84e0 msgr2=0x7f68d80c88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:36.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.691+0000 7f68dfaea640 1 -- 172.21.15.88:0/2826870489 shutdown_connections 2023-10-16T20:51:36.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.691+0000 7f68dfaea640 1 -- 172.21.15.88:0/2826870489 wait complete. 2023-10-16T20:51:36.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.691+0000 7f68dfaea640 1 Processor -- start 2023-10-16T20:51:36.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.691+0000 7f68dfaea640 1 -- start start 2023-10-16T20:51:36.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68dfaea640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68d80c6bf0 0x7f68d8151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:36.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68dfaea640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d80c9040 0x7f68d8158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:36.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68dfaea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68d8153770 0x7f68d81eae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:36.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68dfaea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d8152ff0 con 0x7f68d8153770 2023-10-16T20:51:36.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68dfaea640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d815d510 con 0x7f68d80c6bf0 2023-10-16T20:51:36.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68dfaea640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d8152770 con 0x7f68d80c9040 2023-10-16T20:51:36.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68de087640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68d8153770 0x7f68d81eae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:36.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68de087640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68d8153770 0x7f68d81eae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47712/0 (socket says 172.21.15.88:47712) 2023-10-16T20:51:36.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68de087640 1 -- 172.21.15.88:0/3757912733 learned_addr learned my addr 172.21.15.88:0/3757912733 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:36.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68dd886640 1 --2- 172.21.15.88:0/3757912733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68d80c6bf0 0x7f68d8151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:36.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68dd085640 1 --2- 172.21.15.88:0/3757912733 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d80c9040 0x7f68d8158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:36.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68de087640 1 -- 172.21.15.88:0/3757912733 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d80c9040 msgr2=0x7f68d8158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:36.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68de087640 1 --2- 172.21.15.88:0/3757912733 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d80c9040 0x7f68d8158d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:36.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68de087640 1 -- 172.21.15.88:0/3757912733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68d80c6bf0 msgr2=0x7f68d8151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:36.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68de087640 1 --2- 172.21.15.88:0/3757912733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68d80c6bf0 0x7f68d8151cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:36.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68de087640 1 -- 172.21.15.88:0/3757912733 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d0009a70 con 0x7f68d8153770 2023-10-16T20:51:36.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68de087640 1 --2- 172.21.15.88:0/3757912733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68d8153770 0x7f68d81eae20 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f68d0019500 tx=0x7f68d0019690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:36.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68ce7fc640 1 -- 172.21.15.88:0/3757912733 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d00b9860 con 0x7f68d8153770 2023-10-16T20:51:36.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68ce7fc640 1 -- 172.21.15.88:0/3757912733 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d00b99c0 con 0x7f68d8153770 2023-10-16T20:51:36.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68ce7fc640 1 -- 172.21.15.88:0/3757912733 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d00b9d70 con 0x7f68d8153770 2023-10-16T20:51:36.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68dfaea640 1 -- 172.21.15.88:0/3757912733 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d8152a30 con 0x7f68d8153770 2023-10-16T20:51:36.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68dfaea640 1 -- 172.21.15.88:0/3757912733 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d81f4ec0 con 0x7f68d8153770 2023-10-16T20:51:36.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68dfaea640 1 -- 172.21.15.88:0/3757912733 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d81f4d20 con 0x7f68d8153770 2023-10-16T20:51:36.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68ce7fc640 1 -- 172.21.15.88:0/3757912733 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f68d00c4460 con 0x7f68d8153770 2023-10-16T20:51:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.695+0000 7f68ce7fc640 1 --2- 172.21.15.88:0/3757912733 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f68bc0664a0 0x7f68bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.699+0000 7f68ce7fc640 1 -- 172.21.15.88:0/3757912733 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7f68d011f7a0 con 0x7f68d8153770 2023-10-16T20:51:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.699+0000 7f68dd886640 1 --2- 172.21.15.88:0/3757912733 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f68bc0664a0 0x7f68bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.699+0000 7f68dd886640 1 --2- 172.21.15.88:0/3757912733 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f68bc0664a0 0x7f68bc068960 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f68d4000f70 tx=0x7f68d4007b60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.703+0000 7f68ce7fc640 1 -- 172.21.15.88:0/3757912733 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f68d81f4d20 con 0x7f68d8153770 2023-10-16T20:51:36.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.883+0000 7f68dfaea640 1 -- 172.21.15.88:0/3757912733 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68d81f56c0 con 0x7f68d8153770 2023-10-16T20:51:36.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.883+0000 7f68ce7fc640 1 -- 172.21.15.88:0/3757912733 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5779 (secure 0 0 0) 0x7f68d81f56c0 con 0x7f68d8153770 2023-10-16T20:51:36.886 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:36.887 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:51:32.638867+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:36.887 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 83 2023-10-16T20:51:36.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.887+0000 7f68dfaea640 1 -- 172.21.15.88:0/3757912733 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f68bc0664a0 msgr2=0x7f68bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:36.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.887+0000 7f68dfaea640 1 --2- 172.21.15.88:0/3757912733 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f68bc0664a0 0x7f68bc068960 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f68d4000f70 tx=0x7f68d4007b60 comp rx=0 tx=0).stop 2023-10-16T20:51:36.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.887+0000 7f68dfaea640 1 -- 172.21.15.88:0/3757912733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68d8153770 msgr2=0x7f68d81eae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:36.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.887+0000 7f68dfaea640 1 --2- 172.21.15.88:0/3757912733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68d8153770 0x7f68d81eae20 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f68d0019500 tx=0x7f68d0019690 comp rx=0 tx=0).stop 2023-10-16T20:51:36.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.887+0000 7f68dfaea640 1 -- 172.21.15.88:0/3757912733 shutdown_connections 2023-10-16T20:51:36.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.887+0000 7f68dfaea640 1 --2- 172.21.15.88:0/3757912733 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f68bc0664a0 0x7f68bc068960 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:36.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.887+0000 7f68dfaea640 1 --2- 172.21.15.88:0/3757912733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68d8153770 0x7f68d81eae20 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:36.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.887+0000 7f68dfaea640 1 --2- 172.21.15.88:0/3757912733 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68d80c9040 0x7f68d8158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:36.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.887+0000 7f68dfaea640 1 --2- 172.21.15.88:0/3757912733 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68d80c6bf0 0x7f68d8151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:36.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.887+0000 7f68dfaea640 1 -- 172.21.15.88:0/3757912733 >> 172.21.15.88:0/3757912733 conn(0x7f68d80c84e0 msgr2=0x7f68d80cb2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:36.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.887+0000 7f68dfaea640 1 -- 172.21.15.88:0/3757912733 shutdown_connections 2023-10-16T20:51:36.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:36.887+0000 7f68dfaea640 1 -- 172.21.15.88:0/3757912733 wait complete. 2023-10-16T20:51:36.907 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:51:38.908 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:39.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.055+0000 7f089d93e640 1 Processor -- start 2023-10-16T20:51:39.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.055+0000 7f089d93e640 1 -- start start 2023-10-16T20:51:39.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.059+0000 7f089d93e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f089806ffe0 0x7f08980714b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:39.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.059+0000 7f089d93e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f08981595e0 0x7f0898159a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:39.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.059+0000 7f089d93e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08980719f0 con 0x7f08980c5370 2023-10-16T20:51:39.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.059+0000 7f089d93e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0898159fb0 con 0x7f08981595e0 2023-10-16T20:51:39.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.059+0000 7f089d93e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0898070400 con 0x7f089806ffe0 2023-10-16T20:51:39.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.059+0000 7f08977fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f08981595e0 0x7f0898159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:39.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.059+0000 7f088e7fc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f089806ffe0 0x7f08980714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:39.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.059+0000 7f088e7fc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f089806ffe0 0x7f08980714b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:53488/0 (socket says 172.21.15.88:53488) 2023-10-16T20:51:39.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.059+0000 7f088e7fc640 1 -- 172.21.15.88:0/1674543119 learned_addr learned my addr 172.21.15.88:0/1674543119 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:39.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.059+0000 7f08977fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f08981595e0 0x7f0898159a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:58502/0 (socket says 172.21.15.88:58502) 2023-10-16T20:51:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.059+0000 7f0896ffd640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f08980c5370 0x7f0898069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:47370/0 (socket says 172.21.15.88:47370) 2023-10-16T20:51:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.059+0000 7f088e7fc640 1 -- 172.21.15.88:0/1674543119 >> v2:172.21.15.191:3300/0 conn(0x7f08981595e0 msgr2=0x7f0898159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.059+0000 7f088e7fc640 1 --2- 172.21.15.88:0/1674543119 >> v2:172.21.15.191:3300/0 conn(0x7f08981595e0 0x7f0898159a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.059+0000 7f088e7fc640 1 -- 172.21.15.88:0/1674543119 >> v1:172.21.15.191:6789/0 conn(0x7f08980c5370 legacy=0x7f0898069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:39.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.059+0000 7f088e7fc640 1 -- 172.21.15.88:0/1674543119 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0898070650 con 0x7f089806ffe0 2023-10-16T20:51:39.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.059+0000 7f08977fe640 1 --2- 172.21.15.88:0/1674543119 >> v2:172.21.15.191:3300/0 conn(0x7f08981595e0 0x7f0898159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:51:39.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.059+0000 7f08967fc640 1 -- 172.21.15.88:0/1674543119 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692482447 0 0) 0x7f0888001120 con 0x7f08980c5370 2023-10-16T20:51:39.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.063+0000 7f088e7fc640 1 --2- 172.21.15.88:0/1674543119 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f089806ffe0 0x7f08980714b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0880009e80 tx=0x7f088001f1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c5e3832aa001f8f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:39.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.063+0000 7f08967fc640 1 -- 172.21.15.88:0/1674543119 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08800af400 con 0x7f089806ffe0 2023-10-16T20:51:39.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.063+0000 7f08967fc640 1 -- 172.21.15.88:0/1674543119 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08800ab050 con 0x7f089806ffe0 2023-10-16T20:51:39.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.063+0000 7f08967fc640 1 -- 172.21.15.88:0/1674543119 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08800af780 con 0x7f089806ffe0 2023-10-16T20:51:39.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.063+0000 7f089d93e640 1 -- 172.21.15.88:0/1674543119 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f089806ffe0 msgr2=0x7f08980714b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:39.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.063+0000 7f089d93e640 1 --2- 172.21.15.88:0/1674543119 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f089806ffe0 0x7f08980714b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0880009e80 tx=0x7f088001f1e0 comp rx=0 tx=0).stop 2023-10-16T20:51:39.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.063+0000 7f089d93e640 1 -- 172.21.15.88:0/1674543119 shutdown_connections 2023-10-16T20:51:39.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.063+0000 7f089d93e640 1 --2- 172.21.15.88:0/1674543119 >> v2:172.21.15.191:3300/0 conn(0x7f08981595e0 0x7f0898159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:39.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.063+0000 7f089d93e640 1 --2- 172.21.15.88:0/1674543119 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f089806ffe0 0x7f08980714b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:39.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.063+0000 7f089d93e640 1 -- 172.21.15.88:0/1674543119 >> 172.21.15.88:0/1674543119 conn(0x7f089806e0f0 msgr2=0x7f08980cb180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:39.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.063+0000 7f089d93e640 1 -- 172.21.15.88:0/1674543119 shutdown_connections 2023-10-16T20:51:39.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.063+0000 7f089d93e640 1 -- 172.21.15.88:0/1674543119 wait complete. 2023-10-16T20:51:39.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.063+0000 7f089d93e640 1 Processor -- start 2023-10-16T20:51:39.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.063+0000 7f089d93e640 1 -- start start 2023-10-16T20:51:39.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f089d93e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f089806ffe0 0x7f08981f6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:39.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f089d93e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08980c5370 0x7f08981f5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:39.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f089d93e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08981595e0 0x7f08981f6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:39.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f089d93e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0898140870 con 0x7f08981595e0 2023-10-16T20:51:39.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f089d93e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08981416f0 con 0x7f089806ffe0 2023-10-16T20:51:39.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f089d93e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0898140590 con 0x7f08980c5370 2023-10-16T20:51:39.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f08977fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08981595e0 0x7f08981f6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:39.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f08977fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08981595e0 0x7f08981f6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48360/0 (socket says 172.21.15.88:48360) 2023-10-16T20:51:39.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f08977fe640 1 -- 172.21.15.88:0/3494795640 learned_addr learned my addr 172.21.15.88:0/3494795640 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:39.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f0896ffd640 1 --2- 172.21.15.88:0/3494795640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f089806ffe0 0x7f08981f6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:39.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f088e7fc640 1 --2- 172.21.15.88:0/3494795640 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08980c5370 0x7f08981f5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:39.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f08977fe640 1 -- 172.21.15.88:0/3494795640 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08980c5370 msgr2=0x7f08981f5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:39.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f08977fe640 1 --2- 172.21.15.88:0/3494795640 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08980c5370 0x7f08981f5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:39.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f08977fe640 1 -- 172.21.15.88:0/3494795640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f089806ffe0 msgr2=0x7f08981f6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:39.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f08977fe640 1 --2- 172.21.15.88:0/3494795640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f089806ffe0 0x7f08981f6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:39.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f08977fe640 1 -- 172.21.15.88:0/3494795640 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0890009420 con 0x7f08981595e0 2023-10-16T20:51:39.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f0896ffd640 1 --2- 172.21.15.88:0/3494795640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f089806ffe0 0x7f08981f6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:51:39.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f08977fe640 1 --2- 172.21.15.88:0/3494795640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08981595e0 0x7f08981f6040 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f089000e960 tx=0x7f089000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:39.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f0894ff9640 1 -- 172.21.15.88:0/3494795640 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890009680 con 0x7f08981595e0 2023-10-16T20:51:39.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f0894ff9640 1 -- 172.21.15.88:0/3494795640 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089000ca00 con 0x7f08981595e0 2023-10-16T20:51:39.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f089d93e640 1 -- 172.21.15.88:0/3494795640 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0880009b30 con 0x7f08981595e0 2023-10-16T20:51:39.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f0894ff9640 1 -- 172.21.15.88:0/3494795640 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089000cdd0 con 0x7f08981595e0 2023-10-16T20:51:39.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f089d93e640 1 -- 172.21.15.88:0/3494795640 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08981f7540 con 0x7f08981595e0 2023-10-16T20:51:39.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f089d93e640 1 -- 172.21.15.88:0/3494795640 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0898141830 con 0x7f08981595e0 2023-10-16T20:51:39.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.067+0000 7f0894ff9640 1 -- 172.21.15.88:0/3494795640 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0890013470 con 0x7f08981595e0 2023-10-16T20:51:39.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.071+0000 7f0894ff9640 1 --2- 172.21.15.88:0/3494795640 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08780664a0 0x7f0878068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:39.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.071+0000 7f0894ff9640 1 -- 172.21.15.88:0/3494795640 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7f08900843d0 con 0x7f08981595e0 2023-10-16T20:51:39.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.071+0000 7f0896ffd640 1 --2- 172.21.15.88:0/3494795640 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08780664a0 0x7f0878068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:39.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.071+0000 7f0896ffd640 1 --2- 172.21.15.88:0/3494795640 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08780664a0 0x7f0878068960 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0888007fd0 tx=0x7f0888007ba0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:39.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.075+0000 7f0894ff9640 1 -- 172.21.15.88:0/3494795640 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0898141830 con 0x7f08981595e0 2023-10-16T20:51:39.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.251+0000 7f089d93e640 1 -- 172.21.15.88:0/3494795640 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0898141830 con 0x7f08981595e0 2023-10-16T20:51:39.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.255+0000 7f0894ff9640 1 -- 172.21.15.88:0/3494795640 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5776 (secure 0 0 0) 0x7f0898141830 con 0x7f08981595e0 2023-10-16T20:51:39.255 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:39.255 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:51:37.661868+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:39.256 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 84 2023-10-16T20:51:39.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.255+0000 7f089d93e640 1 -- 172.21.15.88:0/3494795640 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08780664a0 msgr2=0x7f0878068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:39.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.255+0000 7f089d93e640 1 --2- 172.21.15.88:0/3494795640 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08780664a0 0x7f0878068960 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0888007fd0 tx=0x7f0888007ba0 comp rx=0 tx=0).stop 2023-10-16T20:51:39.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.255+0000 7f089d93e640 1 -- 172.21.15.88:0/3494795640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08981595e0 msgr2=0x7f08981f6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:39.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.255+0000 7f089d93e640 1 --2- 172.21.15.88:0/3494795640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08981595e0 0x7f08981f6040 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f089000e960 tx=0x7f089000ee20 comp rx=0 tx=0).stop 2023-10-16T20:51:39.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.255+0000 7f089d93e640 1 -- 172.21.15.88:0/3494795640 shutdown_connections 2023-10-16T20:51:39.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.255+0000 7f089d93e640 1 --2- 172.21.15.88:0/3494795640 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08780664a0 0x7f0878068960 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:39.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.255+0000 7f089d93e640 1 --2- 172.21.15.88:0/3494795640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08981595e0 0x7f08981f6040 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:39.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.255+0000 7f089d93e640 1 --2- 172.21.15.88:0/3494795640 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08980c5370 0x7f08981f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:39.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.255+0000 7f089d93e640 1 --2- 172.21.15.88:0/3494795640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f089806ffe0 0x7f08981f6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:39.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.255+0000 7f089d93e640 1 -- 172.21.15.88:0/3494795640 >> 172.21.15.88:0/3494795640 conn(0x7f089806e0f0 msgr2=0x7f089813dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:39.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.255+0000 7f089d93e640 1 -- 172.21.15.88:0/3494795640 shutdown_connections 2023-10-16T20:51:39.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:39.255+0000 7f089d93e640 1 -- 172.21.15.88:0/3494795640 wait complete. 2023-10-16T20:51:39.275 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:51:41.276 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:41.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.422+0000 7fdef7e46640 1 Processor -- start 2023-10-16T20:51:41.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.422+0000 7fdef7e46640 1 -- start start 2023-10-16T20:51:41.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.422+0000 7fdef7e46640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdef01406d0 0x7fdef00c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:41.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.422+0000 7fdef7e46640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdef006a970 0x7fdef0071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:41.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.422+0000 7fdef7e46640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef015b3c0 con 0x7fdef0071e60 2023-10-16T20:51:41.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.422+0000 7fdef7e46640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef015b680 con 0x7fdef006a970 2023-10-16T20:51:41.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.422+0000 7fdef7e46640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef0157570 con 0x7fdef01406d0 2023-10-16T20:51:41.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.422+0000 7fdef53e1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdef006a970 0x7fdef0071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:41.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.422+0000 7fdef53e1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdef006a970 0x7fdef0071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48376/0 (socket says 172.21.15.88:48376) 2023-10-16T20:51:41.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.422+0000 7fdef53e1640 1 -- 172.21.15.88:0/2635070174 learned_addr learned my addr 172.21.15.88:0/2635070174 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:41.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.426+0000 7fdef5be2640 1 --2- 172.21.15.88:0/2635070174 >> v2:172.21.15.191:3300/0 conn(0x7fdef01406d0 0x7fdef00c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:41.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.426+0000 7fdef53e1640 1 -- 172.21.15.88:0/2635070174 >> v2:172.21.15.191:3300/0 conn(0x7fdef01406d0 msgr2=0x7fdef00c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:41.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.426+0000 7fdef53e1640 1 --2- 172.21.15.88:0/2635070174 >> v2:172.21.15.191:3300/0 conn(0x7fdef01406d0 0x7fdef00c1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:41.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.426+0000 7fdef53e1640 1 -- 172.21.15.88:0/2635070174 >> v1:172.21.15.191:6789/0 conn(0x7fdef0071e60 legacy=0x7fdef015a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:41.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.426+0000 7fdef53e1640 1 -- 172.21.15.88:0/2635070174 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef0157850 con 0x7fdef006a970 2023-10-16T20:51:41.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.426+0000 7fdef53e1640 1 --2- 172.21.15.88:0/2635070174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdef006a970 0x7fdef0071920 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fdef006e1f0 tx=0x7fdee001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4447f98c91b4564 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:41.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.426+0000 7fdef4be0640 1 -- 172.21.15.88:0/2635070174 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee00ac640 con 0x7fdef006a970 2023-10-16T20:51:41.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.426+0000 7fdef4be0640 1 -- 172.21.15.88:0/2635070174 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee00ac7a0 con 0x7fdef006a970 2023-10-16T20:51:41.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.426+0000 7fdef4be0640 1 -- 172.21.15.88:0/2635070174 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee00acb30 con 0x7fdef006a970 2023-10-16T20:51:41.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.426+0000 7fdef7e46640 1 -- 172.21.15.88:0/2635070174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdef006a970 msgr2=0x7fdef0071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:41.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.426+0000 7fdef7e46640 1 --2- 172.21.15.88:0/2635070174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdef006a970 0x7fdef0071920 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fdef006e1f0 tx=0x7fdee001c4d0 comp rx=0 tx=0).stop 2023-10-16T20:51:41.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef7e46640 1 -- 172.21.15.88:0/2635070174 shutdown_connections 2023-10-16T20:51:41.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef7e46640 1 --2- 172.21.15.88:0/2635070174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdef006a970 0x7fdef0071920 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:41.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef7e46640 1 --2- 172.21.15.88:0/2635070174 >> v2:172.21.15.191:3300/0 conn(0x7fdef01406d0 0x7fdef00c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:41.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef7e46640 1 -- 172.21.15.88:0/2635070174 >> 172.21.15.88:0/2635070174 conn(0x7fdef00cad50 msgr2=0x7fdef00cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:41.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef7e46640 1 -- 172.21.15.88:0/2635070174 shutdown_connections 2023-10-16T20:51:41.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef7e46640 1 -- 172.21.15.88:0/2635070174 wait complete. 2023-10-16T20:51:41.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef7e46640 1 Processor -- start 2023-10-16T20:51:41.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef7e46640 1 -- start start 2023-10-16T20:51:41.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef7e46640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdef006a970 0x7fdef01f1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:41.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef7e46640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdef0071e60 0x7fdef0153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:41.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef7e46640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdef01406d0 0x7fdef0158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:41.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef7e46640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef0153990 con 0x7fdef01406d0 2023-10-16T20:51:41.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef7e46640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef0151810 con 0x7fdef006a970 2023-10-16T20:51:41.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef7e46640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef0151ad0 con 0x7fdef0071e60 2023-10-16T20:51:41.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef63e3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdef01406d0 0x7fdef0158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:41.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef5be2640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdef006a970 0x7fdef01f1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:41.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef63e3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdef01406d0 0x7fdef0158900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48388/0 (socket says 172.21.15.88:48388) 2023-10-16T20:51:41.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef63e3640 1 -- 172.21.15.88:0/3568069185 learned_addr learned my addr 172.21.15.88:0/3568069185 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:41.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef53e1640 1 --2- 172.21.15.88:0/3568069185 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdef0071e60 0x7fdef0153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:41.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef63e3640 1 -- 172.21.15.88:0/3568069185 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdef0071e60 msgr2=0x7fdef0153450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:41.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef63e3640 1 --2- 172.21.15.88:0/3568069185 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdef0071e60 0x7fdef0153450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:41.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef63e3640 1 -- 172.21.15.88:0/3568069185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdef006a970 msgr2=0x7fdef01f1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:41.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef63e3640 1 --2- 172.21.15.88:0/3568069185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdef006a970 0x7fdef01f1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:41.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.430+0000 7fdef63e3640 1 -- 172.21.15.88:0/3568069185 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee00094d0 con 0x7fdef01406d0 2023-10-16T20:51:41.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.434+0000 7fdef63e3640 1 --2- 172.21.15.88:0/3568069185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdef01406d0 0x7fdef0158900 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fdee800c770 tx=0x7fdee800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:41.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.434+0000 7fdee67fc640 1 -- 172.21.15.88:0/3568069185 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee8003b80 con 0x7fdef01406d0 2023-10-16T20:51:41.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.434+0000 7fdee67fc640 1 -- 172.21.15.88:0/3568069185 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee8003ce0 con 0x7fdef01406d0 2023-10-16T20:51:41.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.434+0000 7fdee67fc640 1 -- 172.21.15.88:0/3568069185 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee800aa90 con 0x7fdef01406d0 2023-10-16T20:51:41.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.434+0000 7fdef7e46640 1 -- 172.21.15.88:0/3568069185 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef01581d0 con 0x7fdef01406d0 2023-10-16T20:51:41.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.434+0000 7fdef7e46640 1 -- 172.21.15.88:0/3568069185 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef0152090 con 0x7fdef01406d0 2023-10-16T20:51:41.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.434+0000 7fdef7e46640 1 -- 172.21.15.88:0/3568069185 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef0074700 con 0x7fdef01406d0 2023-10-16T20:51:41.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.434+0000 7fdee67fc640 1 -- 172.21.15.88:0/3568069185 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdee800abf0 con 0x7fdef01406d0 2023-10-16T20:51:41.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.434+0000 7fdee67fc640 1 --2- 172.21.15.88:0/3568069185 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fded40664a0 0x7fded4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:41.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.434+0000 7fdee67fc640 1 -- 172.21.15.88:0/3568069185 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7fdee80841d0 con 0x7fdef01406d0 2023-10-16T20:51:41.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.434+0000 7fdef5be2640 1 --2- 172.21.15.88:0/3568069185 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fded40664a0 0x7fded4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:41.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.438+0000 7fdef5be2640 1 --2- 172.21.15.88:0/3568069185 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fded40664a0 0x7fded4068960 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdeec006aa0 tx=0x7fdeec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:41.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.438+0000 7fdee67fc640 1 -- 172.21.15.88:0/3568069185 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdee80510a0 con 0x7fdef01406d0 2023-10-16T20:51:41.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.622+0000 7fdef7e46640 1 -- 172.21.15.88:0/3568069185 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdef0154c40 con 0x7fdef01406d0 2023-10-16T20:51:41.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.622+0000 7fdee67fc640 1 -- 172.21.15.88:0/3568069185 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5776 (secure 0 0 0) 0x7fdee80546c0 con 0x7fdef01406d0 2023-10-16T20:51:41.626 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:41.627 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:51:37.661868+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:41.627 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 84 2023-10-16T20:51:41.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.626+0000 7fdef7e46640 1 -- 172.21.15.88:0/3568069185 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fded40664a0 msgr2=0x7fded4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:41.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.626+0000 7fdef7e46640 1 --2- 172.21.15.88:0/3568069185 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fded40664a0 0x7fded4068960 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdeec006aa0 tx=0x7fdeec008040 comp rx=0 tx=0).stop 2023-10-16T20:51:41.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.626+0000 7fdef7e46640 1 -- 172.21.15.88:0/3568069185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdef01406d0 msgr2=0x7fdef0158900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:41.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.626+0000 7fdef7e46640 1 --2- 172.21.15.88:0/3568069185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdef01406d0 0x7fdef0158900 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fdee800c770 tx=0x7fdee800cc30 comp rx=0 tx=0).stop 2023-10-16T20:51:41.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.626+0000 7fdef7e46640 1 -- 172.21.15.88:0/3568069185 shutdown_connections 2023-10-16T20:51:41.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.626+0000 7fdef7e46640 1 --2- 172.21.15.88:0/3568069185 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fded40664a0 0x7fded4068960 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:41.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.626+0000 7fdef7e46640 1 --2- 172.21.15.88:0/3568069185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdef01406d0 0x7fdef0158900 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:41.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.626+0000 7fdef7e46640 1 --2- 172.21.15.88:0/3568069185 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdef0071e60 0x7fdef0153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:41.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.626+0000 7fdef7e46640 1 --2- 172.21.15.88:0/3568069185 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdef006a970 0x7fdef01f1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:41.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.626+0000 7fdef7e46640 1 -- 172.21.15.88:0/3568069185 >> 172.21.15.88:0/3568069185 conn(0x7fdef00cad50 msgr2=0x7fdef00c9450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:41.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.626+0000 7fdef7e46640 1 -- 172.21.15.88:0/3568069185 shutdown_connections 2023-10-16T20:51:41.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:41.626+0000 7fdef7e46640 1 -- 172.21.15.88:0/3568069185 wait complete. 2023-10-16T20:51:41.646 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:51:43.648 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:43.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.794+0000 7fbbeaa91640 1 Processor -- start 2023-10-16T20:51:43.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.794+0000 7fbbeaa91640 1 -- start start 2023-10-16T20:51:43.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.794+0000 7fbbeaa91640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbbe4150660 0x7fbbe40c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:43.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.794+0000 7fbbeaa91640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe40c5c60 con 0x7fbbe4150280 2023-10-16T20:51:43.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.794+0000 7fbbeaa91640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe40c5370 con 0x7fbbe40c6bc0 2023-10-16T20:51:43.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.794+0000 7fbbeaa91640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbe4156d10 con 0x7fbbe4150660 2023-10-16T20:51:43.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.798+0000 7fbbe3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbbe4150660 0x7fbbe40c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:43.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.798+0000 7fbbe882d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbbe4150280 0x7fbbe406d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55226/0 (socket says 172.21.15.88:55226) 2023-10-16T20:51:43.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.798+0000 7fbbe882d640 1 -- 172.21.15.88:0/4011715415 learned_addr learned my addr 172.21.15.88:0/4011715415 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:51:43.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.798+0000 7fbbe3fff640 1 -- 172.21.15.88:0/4011715415 >> v1:172.21.15.191:6789/0 conn(0x7fbbe40c6bc0 legacy=0x7fbbe4156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:43.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.798+0000 7fbbe3fff640 1 -- 172.21.15.88:0/4011715415 >> v1:172.21.15.88:6789/0 conn(0x7fbbe4150280 legacy=0x7fbbe406d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:43.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.798+0000 7fbbe3fff640 1 -- 172.21.15.88:0/4011715415 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe40c5610 con 0x7fbbe4150660 2023-10-16T20:51:43.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.798+0000 7fbbe37fe640 1 -- 172.21.15.88:0/4011715415 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993095651 0 0) 0x7fbbe40c5c60 con 0x7fbbe4150280 2023-10-16T20:51:43.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.798+0000 7fbbe3fff640 1 --2- 172.21.15.88:0/4011715415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbe4150660 0x7fbbe40c60f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fbbd00097c0 tx=0x7fbbd001ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de8731d88199a6d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:43.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.798+0000 7fbbe37fe640 1 -- 172.21.15.88:0/4011715415 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd00ab670 con 0x7fbbe4150660 2023-10-16T20:51:43.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.798+0000 7fbbe37fe640 1 -- 172.21.15.88:0/4011715415 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd00ab7d0 con 0x7fbbe4150660 2023-10-16T20:51:43.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.798+0000 7fbbe37fe640 1 -- 172.21.15.88:0/4011715415 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd00abb60 con 0x7fbbe4150660 2023-10-16T20:51:43.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbeaa91640 1 -- 172.21.15.88:0/4011715415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbe4150660 msgr2=0x7fbbe40c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:43.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbeaa91640 1 --2- 172.21.15.88:0/4011715415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbe4150660 0x7fbbe40c60f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fbbd00097c0 tx=0x7fbbd001ceb0 comp rx=0 tx=0).stop 2023-10-16T20:51:43.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbeaa91640 1 -- 172.21.15.88:0/4011715415 shutdown_connections 2023-10-16T20:51:43.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbeaa91640 1 --2- 172.21.15.88:0/4011715415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbe4150660 0x7fbbe40c60f0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:43.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbeaa91640 1 -- 172.21.15.88:0/4011715415 >> 172.21.15.88:0/4011715415 conn(0x7fbbe40cad50 msgr2=0x7fbbe40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:43.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbeaa91640 1 -- 172.21.15.88:0/4011715415 shutdown_connections 2023-10-16T20:51:43.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbeaa91640 1 -- 172.21.15.88:0/4011715415 wait complete. 2023-10-16T20:51:43.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbeaa91640 1 Processor -- start 2023-10-16T20:51:43.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbeaa91640 1 -- start start 2023-10-16T20:51:43.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbeaa91640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbe40c6bc0 0x7fbbe4155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:43.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbeaa91640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbe4150280 0x7fbbe4153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:43.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbeaa91640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbbe4150660 0x7fbbe41440f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:43.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbeaa91640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe4143550 con 0x7fbbe4150280 2023-10-16T20:51:43.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbeaa91640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe4155780 con 0x7fbbe40c6bc0 2023-10-16T20:51:43.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbeaa91640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe4155a00 con 0x7fbbe4150660 2023-10-16T20:51:43.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbe3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbe4150280 0x7fbbe4153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:43.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbe3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbe4150280 0x7fbbe4153700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48408/0 (socket says 172.21.15.88:48408) 2023-10-16T20:51:43.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbe3fff640 1 -- 172.21.15.88:0/3042166526 learned_addr learned my addr 172.21.15.88:0/3042166526 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:43.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbe882d640 1 --2- 172.21.15.88:0/3042166526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbe40c6bc0 0x7fbbe4155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:43.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbe902e640 1 --2- 172.21.15.88:0/3042166526 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbbe4150660 0x7fbbe41440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:43.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbe3fff640 1 -- 172.21.15.88:0/3042166526 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbbe4150660 msgr2=0x7fbbe41440f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:43.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbe3fff640 1 --2- 172.21.15.88:0/3042166526 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbbe4150660 0x7fbbe41440f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:43.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbe3fff640 1 -- 172.21.15.88:0/3042166526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbe40c6bc0 msgr2=0x7fbbe4155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:43.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.802+0000 7fbbe3fff640 1 --2- 172.21.15.88:0/3042166526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbe40c6bc0 0x7fbbe4155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:43.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.806+0000 7fbbe3fff640 1 -- 172.21.15.88:0/3042166526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd0009420 con 0x7fbbe4150280 2023-10-16T20:51:43.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.806+0000 7fbbe3fff640 1 --2- 172.21.15.88:0/3042166526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbe4150280 0x7fbbe4153700 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fbbd00ad040 tx=0x7fbbd001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:43.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.806+0000 7fbbe17fa640 1 -- 172.21.15.88:0/3042166526 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd00b5810 con 0x7fbbe4150280 2023-10-16T20:51:43.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.806+0000 7fbbe17fa640 1 -- 172.21.15.88:0/3042166526 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd00b5990 con 0x7fbbe4150280 2023-10-16T20:51:43.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.806+0000 7fbbe17fa640 1 -- 172.21.15.88:0/3042166526 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd00b5d40 con 0x7fbbe4150280 2023-10-16T20:51:43.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.806+0000 7fbbeaa91640 1 -- 172.21.15.88:0/3042166526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe4152680 con 0x7fbbe4150280 2023-10-16T20:51:43.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.806+0000 7fbbeaa91640 1 -- 172.21.15.88:0/3042166526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe4152c20 con 0x7fbbe4150280 2023-10-16T20:51:43.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.806+0000 7fbbeaa91640 1 -- 172.21.15.88:0/3042166526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbe4151c00 con 0x7fbbe4150280 2023-10-16T20:51:43.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.806+0000 7fbbe17fa640 1 -- 172.21.15.88:0/3042166526 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbbd00b4390 con 0x7fbbe4150280 2023-10-16T20:51:43.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.806+0000 7fbbe17fa640 1 --2- 172.21.15.88:0/3042166526 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbbc80664a0 0x7fbbc8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:43.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.806+0000 7fbbe17fa640 1 -- 172.21.15.88:0/3042166526 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7fbbd011ec50 con 0x7fbbe4150280 2023-10-16T20:51:43.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.810+0000 7fbbe882d640 1 --2- 172.21.15.88:0/3042166526 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbbc80664a0 0x7fbbc8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:43.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.810+0000 7fbbe882d640 1 --2- 172.21.15.88:0/3042166526 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbbc80664a0 0x7fbbc8068960 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8000f70 tx=0x7fbbd8007240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:43.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.814+0000 7fbbe17fa640 1 -- 172.21.15.88:0/3042166526 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbbd00ebbb0 con 0x7fbbe4150280 2023-10-16T20:51:43.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.994+0000 7fbbeaa91640 1 -- 172.21.15.88:0/3042166526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbbe4074700 con 0x7fbbe4150280 2023-10-16T20:51:43.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.994+0000 7fbbe17fa640 1 -- 172.21.15.88:0/3042166526 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5776 (secure 0 0 0) 0x7fbbd00ef1d0 con 0x7fbbe4150280 2023-10-16T20:51:43.996 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:43.996 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:51:42.711013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:43.996 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 85 2023-10-16T20:51:43.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.994+0000 7fbbeaa91640 1 -- 172.21.15.88:0/3042166526 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbbc80664a0 msgr2=0x7fbbc8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:43.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.994+0000 7fbbeaa91640 1 --2- 172.21.15.88:0/3042166526 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbbc80664a0 0x7fbbc8068960 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8000f70 tx=0x7fbbd8007240 comp rx=0 tx=0).stop 2023-10-16T20:51:43.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.994+0000 7fbbeaa91640 1 -- 172.21.15.88:0/3042166526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbe4150280 msgr2=0x7fbbe4153700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:43.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.994+0000 7fbbeaa91640 1 --2- 172.21.15.88:0/3042166526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbe4150280 0x7fbbe4153700 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fbbd00ad040 tx=0x7fbbd001ac90 comp rx=0 tx=0).stop 2023-10-16T20:51:43.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.994+0000 7fbbeaa91640 1 -- 172.21.15.88:0/3042166526 shutdown_connections 2023-10-16T20:51:43.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.994+0000 7fbbeaa91640 1 --2- 172.21.15.88:0/3042166526 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbbc80664a0 0x7fbbc8068960 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:43.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.994+0000 7fbbeaa91640 1 --2- 172.21.15.88:0/3042166526 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbbe4150660 0x7fbbe41440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:43.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.994+0000 7fbbeaa91640 1 --2- 172.21.15.88:0/3042166526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbe4150280 0x7fbbe4153700 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:43.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.994+0000 7fbbeaa91640 1 --2- 172.21.15.88:0/3042166526 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbbe40c6bc0 0x7fbbe4155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:43.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.994+0000 7fbbeaa91640 1 -- 172.21.15.88:0/3042166526 >> 172.21.15.88:0/3042166526 conn(0x7fbbe40cad50 msgr2=0x7fbbe4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:44.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.994+0000 7fbbeaa91640 1 -- 172.21.15.88:0/3042166526 shutdown_connections 2023-10-16T20:51:44.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:43.994+0000 7fbbeaa91640 1 -- 172.21.15.88:0/3042166526 wait complete. 2023-10-16T20:51:44.017 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=4288 2023-10-16T20:51:44.017 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.f 2023-10-16T20:51:49.167 INFO:tasks.workunit.client.0.smithi088.stderr:error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated. 2023-10-16T20:51:49.172 INFO:tasks.workunit.client.0.smithi088.stderr:fetch-pack: unexpected disconnect while reading sideband packet 2023-10-16T20:51:49.177 INFO:tasks.workunit.client.0.smithi088.stderr:fatal: early EOF 2023-10-16T20:51:49.185 INFO:tasks.workunit.client.0.smithi088.stderr:fatal: fetch-pack: invalid index-pack output 2023-10-16T20:51:50.184 INFO:tasks.workunit.client.0.smithi088.stderr:+ retry 2023-10-16T20:51:50.185 INFO:tasks.workunit.client.0.smithi088.stderr:+ rm -rf ceph 2023-10-16T20:51:50.186 INFO:tasks.workunit.client.0.smithi088.stderr:+ timeout 3600 git clone https://git.ceph.com/ceph.git 2023-10-16T20:51:50.193 INFO:tasks.workunit.client.0.smithi088.stderr:Cloning into 'ceph'... 2023-10-16T20:51:50.572 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-16T20:51:50.570+0000 7f9515270640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T20:51:50.572 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-16T20:51:50.570+0000 7f9515270640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-16T20:51:50.573 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-16T20:51:50.570+0000 7f9515270640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-16T20:51:50.573 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-16T20:51:50.570+0000 7f9515270640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-10-16T20:51:50.573 INFO:tasks.ceph.mds.d.smithi191.stderr:2023-10-16T20:51:50.570+0000 7f9515270640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-10-16T20:51:52.018 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-16T20:51:52.019 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-16T20:51:52.019 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-16T20:51:52.023 INFO:tasks.ceph.mds.f:Started 2023-10-16T20:51:52.023 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:52.109 INFO:tasks.ceph.mds.f.smithi191.stdout:starting mds.f at 2023-10-16T20:51:52.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb7530c9640 1 Processor -- start 2023-10-16T20:51:52.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb7530c9640 1 -- start start 2023-10-16T20:51:52.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb7530c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb74c150280 0x7fb74c0c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:52.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb7530c9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb74c150660 0x7fb74c0c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:52.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb7530c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb74c0c6bc0 0x7fb74c06d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:52.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb7530c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74c0c5c60 con 0x7fb74c0c6bc0 2023-10-16T20:51:52.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb7530c9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74c0c5370 con 0x7fb74c150660 2023-10-16T20:51:52.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb7530c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c156d10 con 0x7fb74c150280 2023-10-16T20:51:52.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb751666640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb74c0c6bc0 0x7fb74c06d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:52.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb751666640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb74c0c6bc0 0x7fb74c06d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36336/0 (socket says 172.21.15.88:36336) 2023-10-16T20:51:52.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb751666640 1 -- 172.21.15.88:0/327753312 learned_addr learned my addr 172.21.15.88:0/327753312 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:52.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb750e65640 1 --2- 172.21.15.88:0/327753312 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb74c150280 0x7fb74c0c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:52.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb743fff640 1 --2- 172.21.15.88:0/327753312 >> v2:172.21.15.191:3300/0 conn(0x7fb74c150660 0x7fb74c0c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:52.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb751666640 1 -- 172.21.15.88:0/327753312 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb74c150280 msgr2=0x7fb74c0c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:52.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb751666640 1 --2- 172.21.15.88:0/327753312 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb74c150280 0x7fb74c0c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:52.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb751666640 1 -- 172.21.15.88:0/327753312 >> v2:172.21.15.191:3300/0 conn(0x7fb74c150660 msgr2=0x7fb74c0c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:52.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb751666640 1 --2- 172.21.15.88:0/327753312 >> v2:172.21.15.191:3300/0 conn(0x7fb74c150660 0x7fb74c0c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:52.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb751666640 1 -- 172.21.15.88:0/327753312 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c0c5610 con 0x7fb74c0c6bc0 2023-10-16T20:51:52.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.162+0000 7fb743fff640 1 --2- 172.21.15.88:0/327753312 >> v2:172.21.15.191:3300/0 conn(0x7fb74c150660 0x7fb74c0c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:51:52.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.166+0000 7fb751666640 1 --2- 172.21.15.88:0/327753312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb74c0c6bc0 0x7fb74c06d200 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb74c0c84f0 tx=0x7fb74401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b68455fa7e0be8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:52.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.166+0000 7fb7437fe640 1 -- 172.21.15.88:0/327753312 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7440a56a0 con 0x7fb74c0c6bc0 2023-10-16T20:51:52.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.166+0000 7fb7437fe640 1 -- 172.21.15.88:0/327753312 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7440a5800 con 0x7fb74c0c6bc0 2023-10-16T20:51:52.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.166+0000 7fb7437fe640 1 -- 172.21.15.88:0/327753312 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7440a5b90 con 0x7fb74c0c6bc0 2023-10-16T20:51:52.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.166+0000 7fb7530c9640 1 -- 172.21.15.88:0/327753312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb74c0c6bc0 msgr2=0x7fb74c06d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:52.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.166+0000 7fb7530c9640 1 --2- 172.21.15.88:0/327753312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb74c0c6bc0 0x7fb74c06d200 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb74c0c84f0 tx=0x7fb74401c910 comp rx=0 tx=0).stop 2023-10-16T20:51:52.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.166+0000 7fb7530c9640 1 -- 172.21.15.88:0/327753312 shutdown_connections 2023-10-16T20:51:52.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.166+0000 7fb7530c9640 1 --2- 172.21.15.88:0/327753312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb74c0c6bc0 0x7fb74c06d200 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:52.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.166+0000 7fb7530c9640 1 --2- 172.21.15.88:0/327753312 >> v2:172.21.15.191:3300/0 conn(0x7fb74c150660 0x7fb74c0c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:52.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.166+0000 7fb7530c9640 1 --2- 172.21.15.88:0/327753312 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb74c150280 0x7fb74c0c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:52.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.166+0000 7fb7530c9640 1 -- 172.21.15.88:0/327753312 >> 172.21.15.88:0/327753312 conn(0x7fb74c0cad50 msgr2=0x7fb74c0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:52.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.166+0000 7fb7530c9640 1 -- 172.21.15.88:0/327753312 shutdown_connections 2023-10-16T20:51:52.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.166+0000 7fb7530c9640 1 -- 172.21.15.88:0/327753312 wait complete. 2023-10-16T20:51:52.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb7530c9640 1 Processor -- start 2023-10-16T20:51:52.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb7530c9640 1 -- start start 2023-10-16T20:51:52.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb7530c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb74c0c6bc0 0x7fb74c158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:52.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb7530c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb74c150280 0x7fb74c15e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:52.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb7530c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb74c150660 0x7fb74c15ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:52.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb7530c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c1f1b10 con 0x7fb74c150660 2023-10-16T20:51:52.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb7530c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c14a870 con 0x7fb74c0c6bc0 2023-10-16T20:51:52.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb7530c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c1f1dd0 con 0x7fb74c150280 2023-10-16T20:51:52.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb751666640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb74c150660 0x7fb74c15ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:52.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb751666640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb74c150660 0x7fb74c15ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36348/0 (socket says 172.21.15.88:36348) 2023-10-16T20:51:52.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb751666640 1 -- 172.21.15.88:0/2343720842 learned_addr learned my addr 172.21.15.88:0/2343720842 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:52.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb750e65640 1 --2- 172.21.15.88:0/2343720842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb74c0c6bc0 0x7fb74c158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:52.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb743fff640 1 --2- 172.21.15.88:0/2343720842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb74c150280 0x7fb74c15e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:52.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb751666640 1 -- 172.21.15.88:0/2343720842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb74c150280 msgr2=0x7fb74c15e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:52.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb751666640 1 --2- 172.21.15.88:0/2343720842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb74c150280 0x7fb74c15e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:52.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb751666640 1 -- 172.21.15.88:0/2343720842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb74c0c6bc0 msgr2=0x7fb74c158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:52.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb751666640 1 --2- 172.21.15.88:0/2343720842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb74c0c6bc0 0x7fb74c158730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:52.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb751666640 1 -- 172.21.15.88:0/2343720842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb744009b30 con 0x7fb74c150660 2023-10-16T20:51:52.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb751666640 1 --2- 172.21.15.88:0/2343720842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb74c150660 0x7fb74c15ea50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb7440af040 tx=0x7fb7440196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:52.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb7417fa640 1 -- 172.21.15.88:0/2343720842 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7440b8670 con 0x7fb74c150660 2023-10-16T20:51:52.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb7417fa640 1 -- 172.21.15.88:0/2343720842 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7440b87d0 con 0x7fb74c150660 2023-10-16T20:51:52.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb7417fa640 1 -- 172.21.15.88:0/2343720842 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7440b8b80 con 0x7fb74c150660 2023-10-16T20:51:52.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb7530c9640 1 -- 172.21.15.88:0/2343720842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74c1f20b0 con 0x7fb74c150660 2023-10-16T20:51:52.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.170+0000 7fb7530c9640 1 -- 172.21.15.88:0/2343720842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74c1f2670 con 0x7fb74c150660 2023-10-16T20:51:52.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.174+0000 7fb7530c9640 1 -- 172.21.15.88:0/2343720842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74c074700 con 0x7fb74c150660 2023-10-16T20:51:52.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.174+0000 7fb7417fa640 1 -- 172.21.15.88:0/2343720842 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb7440b8ce0 con 0x7fb74c150660 2023-10-16T20:51:52.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.174+0000 7fb7417fa640 1 --2- 172.21.15.88:0/2343720842 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb7380664a0 0x7fb738068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:52.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.174+0000 7fb7417fa640 1 -- 172.21.15.88:0/2343720842 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7fb74411f850 con 0x7fb74c150660 2023-10-16T20:51:52.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.174+0000 7fb750e65640 1 --2- 172.21.15.88:0/2343720842 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb7380664a0 0x7fb738068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:52.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.174+0000 7fb750e65640 1 --2- 172.21.15.88:0/2343720842 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb7380664a0 0x7fb738068960 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb748009b60 tx=0x7fb748009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:52.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.178+0000 7fb7417fa640 1 -- 172.21.15.88:0/2343720842 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb7440ec7e0 con 0x7fb74c150660 2023-10-16T20:51:52.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.362+0000 7fb7530c9640 1 -- 172.21.15.88:0/2343720842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb74c0c1650 con 0x7fb74c150660 2023-10-16T20:51:52.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.362+0000 7fb7417fa640 1 -- 172.21.15.88:0/2343720842 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5777 (secure 0 0 0) 0x7fb744009510 con 0x7fb74c150660 2023-10-16T20:51:52.366 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:52.366 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:51:51.499697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:52.367 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 86 2023-10-16T20:51:52.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.366+0000 7fb7530c9640 1 -- 172.21.15.88:0/2343720842 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb7380664a0 msgr2=0x7fb738068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:52.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.366+0000 7fb7530c9640 1 --2- 172.21.15.88:0/2343720842 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb7380664a0 0x7fb738068960 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb748009b60 tx=0x7fb748009190 comp rx=0 tx=0).stop 2023-10-16T20:51:52.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.366+0000 7fb7530c9640 1 -- 172.21.15.88:0/2343720842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb74c150660 msgr2=0x7fb74c15ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:52.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.366+0000 7fb7530c9640 1 --2- 172.21.15.88:0/2343720842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb74c150660 0x7fb74c15ea50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb7440af040 tx=0x7fb7440196b0 comp rx=0 tx=0).stop 2023-10-16T20:51:52.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.366+0000 7fb7530c9640 1 -- 172.21.15.88:0/2343720842 shutdown_connections 2023-10-16T20:51:52.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.366+0000 7fb7530c9640 1 --2- 172.21.15.88:0/2343720842 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb7380664a0 0x7fb738068960 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:52.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.366+0000 7fb7530c9640 1 --2- 172.21.15.88:0/2343720842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb74c150660 0x7fb74c15ea50 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:52.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.366+0000 7fb7530c9640 1 --2- 172.21.15.88:0/2343720842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb74c150280 0x7fb74c15e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:52.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.366+0000 7fb7530c9640 1 --2- 172.21.15.88:0/2343720842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb74c0c6bc0 0x7fb74c158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:52.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.366+0000 7fb7530c9640 1 -- 172.21.15.88:0/2343720842 >> 172.21.15.88:0/2343720842 conn(0x7fb74c0cad50 msgr2=0x7fb74c0cb130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:52.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.366+0000 7fb7530c9640 1 -- 172.21.15.88:0/2343720842 shutdown_connections 2023-10-16T20:51:52.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:52.366+0000 7fb7530c9640 1 -- 172.21.15.88:0/2343720842 wait complete. 2023-10-16T20:51:52.387 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-16T20:51:54.388 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:51:54.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.534+0000 7f2162e09640 1 Processor -- start 2023-10-16T20:51:54.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.534+0000 7f2162e09640 1 -- start start 2023-10-16T20:51:54.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.534+0000 7f2162e09640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f215c1406d0 0x7f215c0c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:54.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.534+0000 7f2162e09640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f215c06a970 0x7f215c071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:54.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.534+0000 7f2162e09640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f215c15b3c0 con 0x7f215c071e60 2023-10-16T20:51:54.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.534+0000 7f2162e09640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f215c15b680 con 0x7f215c1406d0 2023-10-16T20:51:54.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.534+0000 7f2162e09640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f215c157570 con 0x7f215c06a970 2023-10-16T20:51:54.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.538+0000 7f2160ba5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f215c1406d0 0x7f215c0c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:54.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.538+0000 7f21613a6640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f215c071e60 0x7f215c15a810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55502/0 (socket says 172.21.15.88:55502) 2023-10-16T20:51:54.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.538+0000 7f21613a6640 1 -- 172.21.15.88:0/4010915534 learned_addr learned my addr 172.21.15.88:0/4010915534 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:51:54.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.538+0000 7f2160ba5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f215c1406d0 0x7f215c0c1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36358/0 (socket says 172.21.15.88:36358) 2023-10-16T20:51:54.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.538+0000 7f2153fff640 1 --2- 172.21.15.88:0/4010915534 >> v2:172.21.15.191:3300/0 conn(0x7f215c06a970 0x7f215c071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:54.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.538+0000 7f2160ba5640 1 -- 172.21.15.88:0/4010915534 >> v2:172.21.15.191:3300/0 conn(0x7f215c06a970 msgr2=0x7f215c071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:51:54.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.538+0000 7f2160ba5640 1 --2- 172.21.15.88:0/4010915534 >> v2:172.21.15.191:3300/0 conn(0x7f215c06a970 0x7f215c071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:54.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.538+0000 7f2160ba5640 1 -- 172.21.15.88:0/4010915534 >> v1:172.21.15.88:6789/0 conn(0x7f215c071e60 legacy=0x7f215c15a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:54.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.538+0000 7f2160ba5640 1 -- 172.21.15.88:0/4010915534 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215c157850 con 0x7f215c1406d0 2023-10-16T20:51:54.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.538+0000 7f2160ba5640 1 --2- 172.21.15.88:0/4010915534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f215c1406d0 0x7f215c0c1d50 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f215c151ab0 tx=0x7f215801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cbd2ff85e3dcf87 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:54.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.538+0000 7f21537fe640 1 -- 172.21.15.88:0/4010915534 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21580ab600 con 0x7f215c1406d0 2023-10-16T20:51:54.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.538+0000 7f21537fe640 1 -- 172.21.15.88:0/4010915534 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21580a7050 con 0x7f215c1406d0 2023-10-16T20:51:54.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.538+0000 7f21537fe640 1 -- 172.21.15.88:0/4010915534 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21580ab980 con 0x7f215c1406d0 2023-10-16T20:51:54.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2162e09640 1 -- 172.21.15.88:0/4010915534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f215c1406d0 msgr2=0x7f215c0c1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:54.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2162e09640 1 --2- 172.21.15.88:0/4010915534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f215c1406d0 0x7f215c0c1d50 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f215c151ab0 tx=0x7f215801c430 comp rx=0 tx=0).stop 2023-10-16T20:51:54.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2162e09640 1 -- 172.21.15.88:0/4010915534 shutdown_connections 2023-10-16T20:51:54.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2162e09640 1 --2- 172.21.15.88:0/4010915534 >> v2:172.21.15.191:3300/0 conn(0x7f215c06a970 0x7f215c071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:54.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2162e09640 1 --2- 172.21.15.88:0/4010915534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f215c1406d0 0x7f215c0c1d50 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:54.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2162e09640 1 -- 172.21.15.88:0/4010915534 >> 172.21.15.88:0/4010915534 conn(0x7f215c0cad50 msgr2=0x7f215c0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:54.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2162e09640 1 -- 172.21.15.88:0/4010915534 shutdown_connections 2023-10-16T20:51:54.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2162e09640 1 -- 172.21.15.88:0/4010915534 wait complete. 2023-10-16T20:51:54.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2162e09640 1 Processor -- start 2023-10-16T20:51:54.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2162e09640 1 -- start start 2023-10-16T20:51:54.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2162e09640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f215c06a970 0x7f215c13eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:54.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2162e09640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f215c071e60 0x7f215c139620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:54.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2162e09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f215c1406d0 0x7f215c139b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:54.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2162e09640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c06a140 con 0x7f215c1406d0 2023-10-16T20:51:54.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2162e09640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c158730 con 0x7f215c071e60 2023-10-16T20:51:54.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2162e09640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f215c1583e0 con 0x7f215c06a970 2023-10-16T20:51:54.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f21613a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f215c1406d0 0x7f215c139b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:54.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f21613a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f215c1406d0 0x7f215c139b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36362/0 (socket says 172.21.15.88:36362) 2023-10-16T20:51:54.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f21613a6640 1 -- 172.21.15.88:0/211135835 learned_addr learned my addr 172.21.15.88:0/211135835 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:51:54.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2153fff640 1 --2- 172.21.15.88:0/211135835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f215c071e60 0x7f215c139620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:54.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2160ba5640 1 --2- 172.21.15.88:0/211135835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f215c06a970 0x7f215c13eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:54.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f21613a6640 1 -- 172.21.15.88:0/211135835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f215c06a970 msgr2=0x7f215c13eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:54.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f21613a6640 1 --2- 172.21.15.88:0/211135835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f215c06a970 0x7f215c13eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:54.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f21613a6640 1 -- 172.21.15.88:0/211135835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f215c071e60 msgr2=0x7f215c139620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:54.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f21613a6640 1 --2- 172.21.15.88:0/211135835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f215c071e60 0x7f215c139620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:54.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f21613a6640 1 -- 172.21.15.88:0/211135835 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2158009420 con 0x7f215c1406d0 2023-10-16T20:51:54.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2153fff640 1 --2- 172.21.15.88:0/211135835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f215c071e60 0x7f215c139620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:51:54.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.542+0000 7f2160ba5640 1 --2- 172.21.15.88:0/211135835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f215c06a970 0x7f215c13eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:51:54.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.546+0000 7f21613a6640 1 --2- 172.21.15.88:0/211135835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f215c1406d0 0x7f215c139b60 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f215400cbf0 tx=0x7f215400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:54.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.546+0000 7f21517fa640 1 -- 172.21.15.88:0/211135835 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215400ad40 con 0x7f215c1406d0 2023-10-16T20:51:54.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.546+0000 7f21517fa640 1 -- 172.21.15.88:0/211135835 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2154012400 con 0x7f215c1406d0 2023-10-16T20:51:54.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.546+0000 7f21517fa640 1 -- 172.21.15.88:0/211135835 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215400aea0 con 0x7f215c1406d0 2023-10-16T20:51:54.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.546+0000 7f2162e09640 1 -- 172.21.15.88:0/211135835 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215c1581d0 con 0x7f215c1406d0 2023-10-16T20:51:54.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.546+0000 7f2162e09640 1 -- 172.21.15.88:0/211135835 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f215c159e90 con 0x7f215c1406d0 2023-10-16T20:51:54.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.546+0000 7f2162e09640 1 -- 172.21.15.88:0/211135835 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f215c158870 con 0x7f215c1406d0 2023-10-16T20:51:54.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.546+0000 7f21517fa640 1 -- 172.21.15.88:0/211135835 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2154007070 con 0x7f215c1406d0 2023-10-16T20:51:54.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.546+0000 7f21517fa640 1 --2- 172.21.15.88:0/211135835 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21480664a0 0x7f2148068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:51:54.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.546+0000 7f21517fa640 1 -- 172.21.15.88:0/211135835 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7f2154084250 con 0x7f215c1406d0 2023-10-16T20:51:54.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.546+0000 7f2160ba5640 1 --2- 172.21.15.88:0/211135835 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21480664a0 0x7f2148068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:51:54.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.550+0000 7f2160ba5640 1 --2- 172.21.15.88:0/211135835 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21480664a0 0x7f2148068960 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f215c0c41e0 tx=0x7f2158008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:51:54.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.550+0000 7f21517fa640 1 -- 172.21.15.88:0/211135835 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f215c158870 con 0x7f215c1406d0 2023-10-16T20:51:54.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.734+0000 7f2162e09640 1 -- 172.21.15.88:0/211135835 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f215c158870 con 0x7f215c1406d0 2023-10-16T20:51:54.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.738+0000 7f21517fa640 1 -- 172.21.15.88:0/211135835 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+6570 (secure 0 0 0) 0x7f215c158870 con 0x7f215c1406d0 2023-10-16T20:51:54.739 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:51:54.740 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5378,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:51:51.499697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:51:54.741 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 87 2023-10-16T20:51:54.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.738+0000 7f2162e09640 1 -- 172.21.15.88:0/211135835 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21480664a0 msgr2=0x7f2148068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:54.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.738+0000 7f2162e09640 1 --2- 172.21.15.88:0/211135835 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21480664a0 0x7f2148068960 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f215c0c41e0 tx=0x7f2158008040 comp rx=0 tx=0).stop 2023-10-16T20:51:54.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.738+0000 7f2162e09640 1 -- 172.21.15.88:0/211135835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f215c1406d0 msgr2=0x7f215c139b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:51:54.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.738+0000 7f2162e09640 1 --2- 172.21.15.88:0/211135835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f215c1406d0 0x7f215c139b60 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f215400cbf0 tx=0x7f215400a430 comp rx=0 tx=0).stop 2023-10-16T20:51:54.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.738+0000 7f2162e09640 1 -- 172.21.15.88:0/211135835 shutdown_connections 2023-10-16T20:51:54.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.738+0000 7f2162e09640 1 --2- 172.21.15.88:0/211135835 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f21480664a0 0x7f2148068960 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:54.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.738+0000 7f2162e09640 1 --2- 172.21.15.88:0/211135835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f215c1406d0 0x7f215c139b60 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:54.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.738+0000 7f2162e09640 1 --2- 172.21.15.88:0/211135835 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f215c071e60 0x7f215c139620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:54.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.738+0000 7f2162e09640 1 --2- 172.21.15.88:0/211135835 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f215c06a970 0x7f215c13eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:51:54.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.738+0000 7f2162e09640 1 -- 172.21.15.88:0/211135835 >> 172.21.15.88:0/211135835 conn(0x7f215c0cad50 msgr2=0x7f215c13d490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:51:54.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.738+0000 7f2162e09640 1 -- 172.21.15.88:0/211135835 shutdown_connections 2023-10-16T20:51:54.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:51:54.738+0000 7f2162e09640 1 -- 172.21.15.88:0/211135835 wait complete. 2023-10-16T20:51:54.760 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-16T20:51:54.760 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 77 secs before thrashing 2023-10-16T20:51:59.386 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:51:59.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:52:29.461 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:52:29.466 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:52:59.657 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:52:59.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:53:11.761 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:11.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd7a891e640 1 Processor -- start 2023-10-16T20:53:11.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd7a891e640 1 -- start start 2023-10-16T20:53:11.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd7a891e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7a40ca2f0 0x7fd7a40c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd7a891e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd7a40c51c0 0x7fd7a40cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd7a891e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a40cae10 con 0x7fd7a41537c0 2023-10-16T20:53:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd7a891e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7a40cac30 con 0x7fd7a40c51c0 2023-10-16T20:53:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd7a891e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a415d2b0 con 0x7fd7a40ca2f0 2023-10-16T20:53:11.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd7a37fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd7a41537c0 0x7fd7a415c870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50778/0 (socket says 172.21.15.88:50778) 2023-10-16T20:53:11.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd7a37fe640 1 -- 172.21.15.88:0/1919500513 learned_addr learned my addr 172.21.15.88:0/1919500513 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:53:11.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd7a2ffd640 1 -- 172.21.15.88:0/1919500513 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125517095 0 0) 0x7fd7a40cae10 con 0x7fd7a41537c0 2023-10-16T20:53:11.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd7a2ffd640 1 -- 172.21.15.88:0/1919500513 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd790003530 con 0x7fd7a41537c0 2023-10-16T20:53:11.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd7a3fff640 1 --2- 172.21.15.88:0/1919500513 >> v2:172.21.15.191:3300/0 conn(0x7fd7a40c51c0 0x7fd7a40cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:11.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd79bfff640 1 --2- 172.21.15.88:0/1919500513 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7a40ca2f0 0x7fd7a40c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:11.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd7a2ffd640 1 -- 172.21.15.88:0/1919500513 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2580705571 0 0) 0x7fd790003530 con 0x7fd7a41537c0 2023-10-16T20:53:11.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd7a2ffd640 1 -- 172.21.15.88:0/1919500513 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7a40ca2f0 msgr2=0x7fd7a40c4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:11.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd7a2ffd640 1 --2- 172.21.15.88:0/1919500513 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7a40ca2f0 0x7fd7a40c4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:11.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd7a2ffd640 1 -- 172.21.15.88:0/1919500513 >> v2:172.21.15.191:3300/0 conn(0x7fd7a40c51c0 msgr2=0x7fd7a40cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:11.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.909+0000 7fd7a2ffd640 1 --2- 172.21.15.88:0/1919500513 >> v2:172.21.15.191:3300/0 conn(0x7fd7a40c51c0 0x7fd7a40cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:11.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd7a2ffd640 1 -- 172.21.15.88:0/1919500513 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a415d590 con 0x7fd7a41537c0 2023-10-16T20:53:11.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd7a2ffd640 1 -- 172.21.15.88:0/1919500513 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7fd794001680 con 0x7fd7a41537c0 2023-10-16T20:53:11.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd7a2ffd640 1 -- 172.21.15.88:0/1919500513 >> v1:172.21.15.88:6789/0 conn(0x7fd7a41537c0 legacy=0x7fd7a415c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:11.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd7a2ffd640 1 --2- 172.21.15.88:0/1919500513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd790003980 0x7fd790003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:11.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd7a2ffd640 1 --2- 172.21.15.88:0/1919500513 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7a40ca2f0 0x7fd7900047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:11.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd7a2ffd640 1 --2- 172.21.15.88:0/1919500513 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd790004d10 0x7fd790005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:11.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd7a2ffd640 1 -- 172.21.15.88:0/1919500513 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a40cac30 con 0x7fd790003980 2023-10-16T20:53:11.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd7a2ffd640 1 -- 172.21.15.88:0/1919500513 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a415d2b0 con 0x7fd790004d10 2023-10-16T20:53:11.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd7a2ffd640 1 -- 172.21.15.88:0/1919500513 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79000bac0 con 0x7fd7a40ca2f0 2023-10-16T20:53:11.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd7a37fe640 1 --2- 172.21.15.88:0/1919500513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd790003980 0x7fd790003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:11.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd79bfff640 1 --2- 172.21.15.88:0/1919500513 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7a40ca2f0 0x7fd7900047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:11.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd7a3fff640 1 --2- 172.21.15.88:0/1919500513 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd790004d10 0x7fd790005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:11.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd7a37fe640 1 -- 172.21.15.88:0/1919500513 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7a40ca2f0 msgr2=0x7fd7900047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:11.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd7a37fe640 1 --2- 172.21.15.88:0/1919500513 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7a40ca2f0 0x7fd7900047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:11.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd7a37fe640 1 -- 172.21.15.88:0/1919500513 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd790004d10 msgr2=0x7fd790005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:11.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd7a37fe640 1 --2- 172.21.15.88:0/1919500513 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd790004d10 0x7fd790005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:11.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd7a37fe640 1 -- 172.21.15.88:0/1919500513 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd790003530 con 0x7fd790003980 2023-10-16T20:53:11.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.913+0000 7fd7a37fe640 1 --2- 172.21.15.88:0/1919500513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd790003980 0x7fd790003da0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd79400c240 tx=0x7fd79401f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be5e242333b76d39 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:11.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a2ffd640 1 -- 172.21.15.88:0/1919500513 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd794005ae0 con 0x7fd790003980 2023-10-16T20:53:11.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a2ffd640 1 -- 172.21.15.88:0/1919500513 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7940ab070 con 0x7fd790003980 2023-10-16T20:53:11.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a2ffd640 1 -- 172.21.15.88:0/1919500513 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd794005ea0 con 0x7fd790003980 2023-10-16T20:53:11.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a891e640 1 -- 172.21.15.88:0/1919500513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd790003980 msgr2=0x7fd790003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:11.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a891e640 1 --2- 172.21.15.88:0/1919500513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd790003980 0x7fd790003da0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd79400c240 tx=0x7fd79401f3d0 comp rx=0 tx=0).stop 2023-10-16T20:53:11.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a3fff640 1 -- 172.21.15.88:0/1919500513 reap_dead start 2023-10-16T20:53:11.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a891e640 1 -- 172.21.15.88:0/1919500513 shutdown_connections 2023-10-16T20:53:11.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a891e640 1 -- 172.21.15.88:0/1919500513 >> 172.21.15.88:0/1919500513 conn(0x7fd7a406c170 msgr2=0x7fd7a40715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:11.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a891e640 1 -- 172.21.15.88:0/1919500513 shutdown_connections 2023-10-16T20:53:11.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a891e640 1 -- 172.21.15.88:0/1919500513 wait complete. 2023-10-16T20:53:11.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a891e640 1 Processor -- start 2023-10-16T20:53:11.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a891e640 1 -- start start 2023-10-16T20:53:11.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a891e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd790003980 0x7fd7a414e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:11.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a891e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7a414bde0 0x7fd7a40c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:11.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a891e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7a40ceca0 0x7fd7a414a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:11.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a891e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a414ed40 con 0x7fd790003980 2023-10-16T20:53:11.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a891e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a414e370 con 0x7fd7a414bde0 2023-10-16T20:53:11.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a891e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a40c62a0 con 0x7fd7a40ceca0 2023-10-16T20:53:11.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd790003980 0x7fd7a414e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:11.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd790003980 0x7fd7a414e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34616/0 (socket says 172.21.15.88:34616) 2023-10-16T20:53:11.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.917+0000 7fd7a37fe640 1 -- 172.21.15.88:0/2817671646 learned_addr learned my addr 172.21.15.88:0/2817671646 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:11.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.921+0000 7fd79bfff640 1 --2- 172.21.15.88:0/2817671646 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7a414bde0 0x7fd7a40c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:11.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.921+0000 7fd7a3fff640 1 --2- 172.21.15.88:0/2817671646 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7a40ceca0 0x7fd7a414a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:11.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.921+0000 7fd7a37fe640 1 -- 172.21.15.88:0/2817671646 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7a40ceca0 msgr2=0x7fd7a414a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:11.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.921+0000 7fd7a37fe640 1 --2- 172.21.15.88:0/2817671646 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7a40ceca0 0x7fd7a414a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:11.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.921+0000 7fd7a37fe640 1 -- 172.21.15.88:0/2817671646 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7a414bde0 msgr2=0x7fd7a40c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:11.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.921+0000 7fd7a37fe640 1 --2- 172.21.15.88:0/2817671646 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7a414bde0 0x7fd7a40c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:11.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.921+0000 7fd7a37fe640 1 -- 172.21.15.88:0/2817671646 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a414ad00 con 0x7fd790003980 2023-10-16T20:53:11.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.921+0000 7fd7a37fe640 1 --2- 172.21.15.88:0/2817671646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd790003980 0x7fd7a414e800 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd79400c240 tx=0x7fd790005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:11.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.921+0000 7fd7a0ff9640 1 -- 172.21.15.88:0/2817671646 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7940b7070 con 0x7fd790003980 2023-10-16T20:53:11.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.921+0000 7fd7a0ff9640 1 -- 172.21.15.88:0/2817671646 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7940048b0 con 0x7fd790003980 2023-10-16T20:53:11.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.921+0000 7fd7a891e640 1 -- 172.21.15.88:0/2817671646 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a40c6530 con 0x7fd790003980 2023-10-16T20:53:11.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.921+0000 7fd7a0ff9640 1 -- 172.21.15.88:0/2817671646 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd794004c20 con 0x7fd790003980 2023-10-16T20:53:11.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.921+0000 7fd7a891e640 1 -- 172.21.15.88:0/2817671646 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a414dd20 con 0x7fd790003980 2023-10-16T20:53:11.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.921+0000 7fd7a891e640 1 -- 172.21.15.88:0/2817671646 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a404e9c0 con 0x7fd790003980 2023-10-16T20:53:11.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.921+0000 7fd7a0ff9640 1 -- 172.21.15.88:0/2817671646 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd7940ad020 con 0x7fd790003980 2023-10-16T20:53:11.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.921+0000 7fd7a0ff9640 1 --2- 172.21.15.88:0/2817671646 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd7840664d0 0x7fd784068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:11.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.921+0000 7fd7a0ff9640 1 -- 172.21.15.88:0/2817671646 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7fd794120c40 con 0x7fd790003980 2023-10-16T20:53:11.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.925+0000 7fd79bfff640 1 --2- 172.21.15.88:0/2817671646 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd7840664d0 0x7fd784068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:11.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.925+0000 7fd79bfff640 1 --2- 172.21.15.88:0/2817671646 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd7840664d0 0x7fd784068990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd7a415bb20 tx=0x7fd78c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:11.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:11.925+0000 7fd7a0ff9640 1 -- 172.21.15.88:0/2817671646 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd7940edb10 con 0x7fd790003980 2023-10-16T20:53:12.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.105+0000 7fd7a891e640 1 -- 172.21.15.88:0/2817671646 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7a40cf280 con 0x7fd790003980 2023-10-16T20:53:12.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.105+0000 7fd7a0ff9640 1 -- 172.21.15.88:0/2817671646 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+6571 (secure 0 0 0) 0x7fd7940c1020 con 0x7fd790003980 2023-10-16T20:53:12.109 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:12.109 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5378,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:52:08.548787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:12.110 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 88 2023-10-16T20:53:12.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.109+0000 7fd7a891e640 1 -- 172.21.15.88:0/2817671646 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd7840664d0 msgr2=0x7fd784068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:12.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.109+0000 7fd7a891e640 1 --2- 172.21.15.88:0/2817671646 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd7840664d0 0x7fd784068990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd7a415bb20 tx=0x7fd78c008040 comp rx=0 tx=0).stop 2023-10-16T20:53:12.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.109+0000 7fd7a891e640 1 -- 172.21.15.88:0/2817671646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd790003980 msgr2=0x7fd7a414e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:12.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.109+0000 7fd7a891e640 1 --2- 172.21.15.88:0/2817671646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd790003980 0x7fd7a414e800 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd79400c240 tx=0x7fd790005800 comp rx=0 tx=0).stop 2023-10-16T20:53:12.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.109+0000 7fd7a891e640 1 -- 172.21.15.88:0/2817671646 shutdown_connections 2023-10-16T20:53:12.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.109+0000 7fd7a891e640 1 --2- 172.21.15.88:0/2817671646 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd7840664d0 0x7fd784068990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:12.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.109+0000 7fd7a891e640 1 --2- 172.21.15.88:0/2817671646 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd7a40ceca0 0x7fd7a414a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:12.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.109+0000 7fd7a891e640 1 --2- 172.21.15.88:0/2817671646 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd7a414bde0 0x7fd7a40c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:12.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.109+0000 7fd7a891e640 1 --2- 172.21.15.88:0/2817671646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd790003980 0x7fd7a414e800 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:12.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.109+0000 7fd7a891e640 1 -- 172.21.15.88:0/2817671646 >> 172.21.15.88:0/2817671646 conn(0x7fd7a406c170 msgr2=0x7fd7a40cadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:12.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.109+0000 7fd7a891e640 1 -- 172.21.15.88:0/2817671646 shutdown_connections 2023-10-16T20:53:12.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.109+0000 7fd7a891e640 1 -- 172.21.15.88:0/2817671646 wait complete. 2023-10-16T20:53:12.130 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=4) 2023-10-16T20:53:12.130 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T20:53:12.131 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T20:53:12.148 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T20:53:12.148 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:12.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.285+0000 7f0958409640 1 Processor -- start 2023-10-16T20:53:12.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.285+0000 7f0958409640 1 -- start start 2023-10-16T20:53:12.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.285+0000 7f0958409640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0950153970 0x7f0950070830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:12.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.285+0000 7f0958409640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f09501596e0 0x7f0950159b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:12.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.285+0000 7f0958409640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095015d910 con 0x7f0950069e90 2023-10-16T20:53:12.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.285+0000 7f0958409640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0950070ec0 con 0x7f09501596e0 2023-10-16T20:53:12.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.285+0000 7f0958409640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095015cb10 con 0x7f0950153970 2023-10-16T20:53:12.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.285+0000 7f09559a4640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0950069e90 0x7f095006e6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50794/0 (socket says 172.21.15.88:50794) 2023-10-16T20:53:12.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.285+0000 7f09559a4640 1 -- 172.21.15.88:0/656958530 learned_addr learned my addr 172.21.15.88:0/656958530 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:53:12.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.285+0000 7f09569a6640 1 --2- 172.21.15.88:0/656958530 >> v2:172.21.15.191:3300/0 conn(0x7f09501596e0 0x7f0950159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:12.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.285+0000 7f09561a5640 1 --2- 172.21.15.88:0/656958530 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0950153970 0x7f0950070830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:12.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.285+0000 7f09551a3640 1 -- 172.21.15.88:0/656958530 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231885450 0 0) 0x7f095015d910 con 0x7f0950069e90 2023-10-16T20:53:12.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.285+0000 7f09551a3640 1 -- 172.21.15.88:0/656958530 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f093c003530 con 0x7f0950069e90 2023-10-16T20:53:12.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09551a3640 1 -- 172.21.15.88:0/656958530 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4090193451 0 0) 0x7f093c003530 con 0x7f0950069e90 2023-10-16T20:53:12.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09551a3640 1 -- 172.21.15.88:0/656958530 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0950153970 msgr2=0x7f0950070830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:12.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09551a3640 1 --2- 172.21.15.88:0/656958530 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0950153970 0x7f0950070830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:12.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09551a3640 1 -- 172.21.15.88:0/656958530 >> v2:172.21.15.191:3300/0 conn(0x7f09501596e0 msgr2=0x7f0950159b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:12.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09551a3640 1 --2- 172.21.15.88:0/656958530 >> v2:172.21.15.191:3300/0 conn(0x7f09501596e0 0x7f0950159b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:12.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09551a3640 1 -- 172.21.15.88:0/656958530 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095015a0b0 con 0x7f0950069e90 2023-10-16T20:53:12.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09551a3640 1 -- 172.21.15.88:0/656958530 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f0940001680 con 0x7f0950069e90 2023-10-16T20:53:12.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09551a3640 1 -- 172.21.15.88:0/656958530 >> v1:172.21.15.88:6789/0 conn(0x7f0950069e90 legacy=0x7f095006e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:12.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09551a3640 1 --2- 172.21.15.88:0/656958530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f093c003980 0x7f093c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:12.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09551a3640 1 --2- 172.21.15.88:0/656958530 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f093c0047d0 0x7f093c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:12.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09551a3640 1 --2- 172.21.15.88:0/656958530 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0950153970 0x7f093c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:12.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09551a3640 1 -- 172.21.15.88:0/656958530 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0950070ec0 con 0x7f093c003980 2023-10-16T20:53:12.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09551a3640 1 -- 172.21.15.88:0/656958530 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093c00b990 con 0x7f093c0047d0 2023-10-16T20:53:12.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09551a3640 1 -- 172.21.15.88:0/656958530 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093c00bad0 con 0x7f0950153970 2023-10-16T20:53:12.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09561a5640 1 --2- 172.21.15.88:0/656958530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f093c003980 0x7f093c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:12.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09559a4640 1 --2- 172.21.15.88:0/656958530 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f093c0047d0 0x7f093c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:12.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09569a6640 1 --2- 172.21.15.88:0/656958530 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0950153970 0x7f093c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:12.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09561a5640 1 -- 172.21.15.88:0/656958530 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0950153970 msgr2=0x7f093c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:12.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09561a5640 1 --2- 172.21.15.88:0/656958530 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0950153970 0x7f093c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:12.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09561a5640 1 -- 172.21.15.88:0/656958530 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f093c0047d0 msgr2=0x7f093c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:12.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09561a5640 1 --2- 172.21.15.88:0/656958530 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f093c0047d0 0x7f093c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:12.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09561a5640 1 -- 172.21.15.88:0/656958530 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093c003530 con 0x7f093c003980 2023-10-16T20:53:12.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.289+0000 7f09561a5640 1 --2- 172.21.15.88:0/656958530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f093c003980 0x7f093c003da0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f094c009ac0 tx=0x7f094c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2a2925d74114c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:12.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f09551a3640 1 -- 172.21.15.88:0/656958530 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c0059a0 con 0x7f093c003980 2023-10-16T20:53:12.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f09551a3640 1 -- 172.21.15.88:0/656958530 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094c005b00 con 0x7f093c003980 2023-10-16T20:53:12.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f09551a3640 1 -- 172.21.15.88:0/656958530 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c006030 con 0x7f093c003980 2023-10-16T20:53:12.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f0958409640 1 -- 172.21.15.88:0/656958530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f093c003980 msgr2=0x7f093c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:12.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f0958409640 1 --2- 172.21.15.88:0/656958530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f093c003980 0x7f093c003da0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f094c009ac0 tx=0x7f094c01ca30 comp rx=0 tx=0).stop 2023-10-16T20:53:12.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f09569a6640 1 -- 172.21.15.88:0/656958530 reap_dead start 2023-10-16T20:53:12.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f0958409640 1 -- 172.21.15.88:0/656958530 shutdown_connections 2023-10-16T20:53:12.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f0958409640 1 -- 172.21.15.88:0/656958530 >> 172.21.15.88:0/656958530 conn(0x7f09500cbeb0 msgr2=0x7f09500cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:12.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f0958409640 1 -- 172.21.15.88:0/656958530 shutdown_connections 2023-10-16T20:53:12.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f0958409640 1 -- 172.21.15.88:0/656958530 wait complete. 2023-10-16T20:53:12.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f0958409640 1 Processor -- start 2023-10-16T20:53:12.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f0958409640 1 -- start start 2023-10-16T20:53:12.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f0958409640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f093c003980 0x7f0950159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:12.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f0958409640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09500c4f10 0x7f09500c42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:12.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f0958409640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09500cb230 0x7f09501495c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:12.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f0958409640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09501589e0 con 0x7f09500c4f10 2023-10-16T20:53:12.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f0958409640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09500c32b0 con 0x7f09500cb230 2023-10-16T20:53:12.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f0958409640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0950158740 con 0x7f093c003980 2023-10-16T20:53:12.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f09559a4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09500c4f10 0x7f09500c42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:12.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f09559a4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09500c4f10 0x7f09500c42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34642/0 (socket says 172.21.15.88:34642) 2023-10-16T20:53:12.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.293+0000 7f09559a4640 1 -- 172.21.15.88:0/1988261204 learned_addr learned my addr 172.21.15.88:0/1988261204 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:12.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f09569a6640 1 --2- 172.21.15.88:0/1988261204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09500cb230 0x7f09501495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:12.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f09561a5640 1 --2- 172.21.15.88:0/1988261204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f093c003980 0x7f0950159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:12.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f09559a4640 1 -- 172.21.15.88:0/1988261204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f093c003980 msgr2=0x7f0950159040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:12.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f09559a4640 1 --2- 172.21.15.88:0/1988261204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f093c003980 0x7f0950159040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:12.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f09559a4640 1 -- 172.21.15.88:0/1988261204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09500cb230 msgr2=0x7f09501495c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:12.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f09559a4640 1 --2- 172.21.15.88:0/1988261204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09500cb230 0x7f09501495c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:12.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f09559a4640 1 -- 172.21.15.88:0/1988261204 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09500c3500 con 0x7f09500c4f10 2023-10-16T20:53:12.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f09559a4640 1 --2- 172.21.15.88:0/1988261204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09500c4f10 0x7f09500c42c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f09400122e0 tx=0x7f09400127a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:12.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f0946ffd640 1 -- 172.21.15.88:0/1988261204 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0940014750 con 0x7f09500c4f10 2023-10-16T20:53:12.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f0946ffd640 1 -- 172.21.15.88:0/1988261204 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09400148d0 con 0x7f09500c4f10 2023-10-16T20:53:12.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f0946ffd640 1 -- 172.21.15.88:0/1988261204 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0940014c80 con 0x7f09500c4f10 2023-10-16T20:53:12.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f0958409640 1 -- 172.21.15.88:0/1988261204 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09500c37e0 con 0x7f09500c4f10 2023-10-16T20:53:12.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f0958409640 1 -- 172.21.15.88:0/1988261204 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09500c3da0 con 0x7f09500c4f10 2023-10-16T20:53:12.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f0958409640 1 -- 172.21.15.88:0/1988261204 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0950074700 con 0x7f09500c4f10 2023-10-16T20:53:12.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f0946ffd640 1 -- 172.21.15.88:0/1988261204 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f09400153e0 con 0x7f09500c4f10 2023-10-16T20:53:12.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f0946ffd640 1 --2- 172.21.15.88:0/1988261204 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f09340664a0 0x7f0934068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:12.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f0946ffd640 1 -- 172.21.15.88:0/1988261204 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7f0940086b00 con 0x7f09500c4f10 2023-10-16T20:53:12.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.297+0000 7f09561a5640 1 --2- 172.21.15.88:0/1988261204 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f09340664a0 0x7f0934068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:12.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.301+0000 7f09561a5640 1 --2- 172.21.15.88:0/1988261204 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f09340664a0 0x7f0934068960 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f094c0ae000 tx=0x7f094c0046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:12.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.305+0000 7f0946ffd640 1 -- 172.21.15.88:0/1988261204 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0940088e60 con 0x7f09500c4f10 2023-10-16T20:53:12.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.485+0000 7f0958409640 1 -- 172.21.15.88:0/1988261204 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0950071050 con 0x7f09500c4f10 2023-10-16T20:53:12.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.485+0000 7f0946ffd640 1 -- 172.21.15.88:0/1988261204 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+6571 (secure 0 0 0) 0x7f094001c090 con 0x7f09500c4f10 2023-10-16T20:53:12.488 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:12.488 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5378,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:52:08.548787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:12.489 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 88 2023-10-16T20:53:12.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.485+0000 7f0958409640 1 -- 172.21.15.88:0/1988261204 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f09340664a0 msgr2=0x7f0934068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:12.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.485+0000 7f0958409640 1 --2- 172.21.15.88:0/1988261204 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f09340664a0 0x7f0934068960 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f094c0ae000 tx=0x7f094c0046d0 comp rx=0 tx=0).stop 2023-10-16T20:53:12.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.485+0000 7f0958409640 1 -- 172.21.15.88:0/1988261204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09500c4f10 msgr2=0x7f09500c42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:12.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.485+0000 7f0958409640 1 --2- 172.21.15.88:0/1988261204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09500c4f10 0x7f09500c42c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f09400122e0 tx=0x7f09400127a0 comp rx=0 tx=0).stop 2023-10-16T20:53:12.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.489+0000 7f0958409640 1 -- 172.21.15.88:0/1988261204 shutdown_connections 2023-10-16T20:53:12.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.489+0000 7f0958409640 1 --2- 172.21.15.88:0/1988261204 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f09340664a0 0x7f0934068960 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:12.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.489+0000 7f0958409640 1 --2- 172.21.15.88:0/1988261204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09500cb230 0x7f09501495c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:12.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.489+0000 7f0958409640 1 --2- 172.21.15.88:0/1988261204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09500c4f10 0x7f09500c42c0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:12.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.489+0000 7f0958409640 1 --2- 172.21.15.88:0/1988261204 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f093c003980 0x7f0950159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:12.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.489+0000 7f0958409640 1 -- 172.21.15.88:0/1988261204 >> 172.21.15.88:0/1988261204 conn(0x7f09500cbeb0 msgr2=0x7f09500cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:12.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.489+0000 7f0958409640 1 -- 172.21.15.88:0/1988261204 shutdown_connections 2023-10-16T20:53:12.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:12.489+0000 7f0958409640 1 -- 172.21.15.88:0/1988261204 wait complete. 2023-10-16T20:53:12.508 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-16T20:53:14.510 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:14.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.657+0000 7f434c69e640 1 Processor -- start 2023-10-16T20:53:14.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.657+0000 7f434c69e640 1 -- start start 2023-10-16T20:53:14.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.657+0000 7f434c69e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f43441406d0 0x7f43440c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:14.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.657+0000 7f434c69e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f434415a810 0x7f434415ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:14.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.657+0000 7f434c69e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f434415b190 con 0x7f434406a970 2023-10-16T20:53:14.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.657+0000 7f434c69e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f434415b3c0 con 0x7f43441406d0 2023-10-16T20:53:14.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.657+0000 7f434c69e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434415b680 con 0x7f434415a810 2023-10-16T20:53:14.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.657+0000 7f434ac3b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f434415a810 0x7f434415ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:14.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.657+0000 7f434a43a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f43441406d0 0x7f43440c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:14.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.657+0000 7f4349c39640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f434406a970 0x7f4344071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:58040/0 (socket says 172.21.15.88:58040) 2023-10-16T20:53:14.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.657+0000 7f4349c39640 1 -- 172.21.15.88:0/2664429697 learned_addr learned my addr 172.21.15.88:0/2664429697 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:53:14.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.657+0000 7f434ac3b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f434415a810 0x7f434415ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:60166/0 (socket says 172.21.15.88:60166) 2023-10-16T20:53:14.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.657+0000 7f434a43a640 1 -- 172.21.15.88:0/2664429697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f434415a810 msgr2=0x7f434415ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:14.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.657+0000 7f434a43a640 1 --2- 172.21.15.88:0/2664429697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f434415a810 0x7f434415ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:14.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.657+0000 7f434a43a640 1 -- 172.21.15.88:0/2664429697 >> v1:172.21.15.191:6789/0 conn(0x7f434406a970 legacy=0x7f4344071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:14.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.657+0000 7f434a43a640 1 -- 172.21.15.88:0/2664429697 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4344157570 con 0x7f43441406d0 2023-10-16T20:53:14.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.657+0000 7f434ac3b640 1 --2- 172.21.15.88:0/2664429697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f434415a810 0x7f434415ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:53:14.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.661+0000 7f434a43a640 1 --2- 172.21.15.88:0/2664429697 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43441406d0 0x7f43440c1d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f43400097c0 tx=0x7f434001c310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53d49fb709322c7c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:14.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.661+0000 7f4349438640 1 -- 172.21.15.88:0/2664429697 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43400ab600 con 0x7f43441406d0 2023-10-16T20:53:14.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.661+0000 7f4349438640 1 -- 172.21.15.88:0/2664429697 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43400a7050 con 0x7f43441406d0 2023-10-16T20:53:14.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.661+0000 7f4349438640 1 -- 172.21.15.88:0/2664429697 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43400ab980 con 0x7f43441406d0 2023-10-16T20:53:14.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.661+0000 7f434c69e640 1 -- 172.21.15.88:0/2664429697 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43441406d0 msgr2=0x7f43440c1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:14.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.661+0000 7f434c69e640 1 --2- 172.21.15.88:0/2664429697 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43441406d0 0x7f43440c1d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f43400097c0 tx=0x7f434001c310 comp rx=0 tx=0).stop 2023-10-16T20:53:14.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434c69e640 1 -- 172.21.15.88:0/2664429697 shutdown_connections 2023-10-16T20:53:14.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434c69e640 1 --2- 172.21.15.88:0/2664429697 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f434415a810 0x7f434415ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:14.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434c69e640 1 --2- 172.21.15.88:0/2664429697 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43441406d0 0x7f43440c1d50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:14.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434c69e640 1 -- 172.21.15.88:0/2664429697 >> 172.21.15.88:0/2664429697 conn(0x7f43440cad50 msgr2=0x7f43440cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:14.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434c69e640 1 -- 172.21.15.88:0/2664429697 shutdown_connections 2023-10-16T20:53:14.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434c69e640 1 -- 172.21.15.88:0/2664429697 wait complete. 2023-10-16T20:53:14.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434c69e640 1 Processor -- start 2023-10-16T20:53:14.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434c69e640 1 -- start start 2023-10-16T20:53:14.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434c69e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f434406a970 0x7f434413d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:14.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434c69e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43441406d0 0x7f4344153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:14.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434c69e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f434415a810 0x7f43441544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:14.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434c69e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434415a5e0 con 0x7f434406a970 2023-10-16T20:53:14.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434c69e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434413dd90 con 0x7f434415a810 2023-10-16T20:53:14.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434c69e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43441597c0 con 0x7f43441406d0 2023-10-16T20:53:14.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434a43a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f434406a970 0x7f434413d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:14.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434a43a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f434406a970 0x7f434413d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34648/0 (socket says 172.21.15.88:34648) 2023-10-16T20:53:14.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434a43a640 1 -- 172.21.15.88:0/2468505883 learned_addr learned my addr 172.21.15.88:0/2468505883 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:14.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f4349c39640 1 --2- 172.21.15.88:0/2468505883 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43441406d0 0x7f4344153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:14.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434ac3b640 1 --2- 172.21.15.88:0/2468505883 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f434415a810 0x7f43441544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:14.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434a43a640 1 -- 172.21.15.88:0/2468505883 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43441406d0 msgr2=0x7f4344153f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:14.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434a43a640 1 --2- 172.21.15.88:0/2468505883 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43441406d0 0x7f4344153f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:14.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434a43a640 1 -- 172.21.15.88:0/2468505883 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f434415a810 msgr2=0x7f43441544c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:14.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434a43a640 1 --2- 172.21.15.88:0/2468505883 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f434415a810 0x7f43441544c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:14.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434a43a640 1 -- 172.21.15.88:0/2468505883 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c009a70 con 0x7f434406a970 2023-10-16T20:53:14.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.665+0000 7f434a43a640 1 --2- 172.21.15.88:0/2468505883 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f434406a970 0x7f434413d850 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f4340019f40 tx=0x7f43400190b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:14.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.669+0000 7f433affd640 1 -- 172.21.15.88:0/2468505883 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43400b4810 con 0x7f434406a970 2023-10-16T20:53:14.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.669+0000 7f433affd640 1 -- 172.21.15.88:0/2468505883 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43400b4970 con 0x7f434406a970 2023-10-16T20:53:14.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.669+0000 7f433affd640 1 -- 172.21.15.88:0/2468505883 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43400b4d20 con 0x7f434406a970 2023-10-16T20:53:14.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.669+0000 7f434c69e640 1 -- 172.21.15.88:0/2468505883 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4340009420 con 0x7f434406a970 2023-10-16T20:53:14.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.669+0000 7f434c69e640 1 -- 172.21.15.88:0/2468505883 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4344159d40 con 0x7f434406a970 2023-10-16T20:53:14.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.669+0000 7f434c69e640 1 -- 172.21.15.88:0/2468505883 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4344074700 con 0x7f434406a970 2023-10-16T20:53:14.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.669+0000 7f433affd640 1 -- 172.21.15.88:0/2468505883 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f43400b2040 con 0x7f434406a970 2023-10-16T20:53:14.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.669+0000 7f433affd640 1 --2- 172.21.15.88:0/2468505883 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f43280664a0 0x7f4328068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:14.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.669+0000 7f433affd640 1 -- 172.21.15.88:0/2468505883 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7f434011d8c0 con 0x7f434406a970 2023-10-16T20:53:14.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.669+0000 7f4349c39640 1 --2- 172.21.15.88:0/2468505883 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f43280664a0 0x7f4328068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:14.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.669+0000 7f4349c39640 1 --2- 172.21.15.88:0/2468505883 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f43280664a0 0x7f4328068960 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4334009fd0 tx=0x7f433400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:14.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.673+0000 7f433affd640 1 -- 172.21.15.88:0/2468505883 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f43400ea790 con 0x7f434406a970 2023-10-16T20:53:14.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.853+0000 7f434c69e640 1 -- 172.21.15.88:0/2468505883 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f434406af30 con 0x7f434406a970 2023-10-16T20:53:14.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.857+0000 7f433affd640 1 -- 172.21.15.88:0/2468505883 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+6571 (secure 0 0 0) 0x7f43400eddb0 con 0x7f434406a970 2023-10-16T20:53:14.858 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:14.858 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5378,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:52:08.548787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:14.859 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 88 2023-10-16T20:53:14.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.857+0000 7f434c69e640 1 -- 172.21.15.88:0/2468505883 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f43280664a0 msgr2=0x7f4328068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:14.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.857+0000 7f434c69e640 1 --2- 172.21.15.88:0/2468505883 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f43280664a0 0x7f4328068960 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4334009fd0 tx=0x7f433400b040 comp rx=0 tx=0).stop 2023-10-16T20:53:14.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.857+0000 7f434c69e640 1 -- 172.21.15.88:0/2468505883 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f434406a970 msgr2=0x7f434413d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:14.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.857+0000 7f434c69e640 1 --2- 172.21.15.88:0/2468505883 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f434406a970 0x7f434413d850 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f4340019f40 tx=0x7f43400190b0 comp rx=0 tx=0).stop 2023-10-16T20:53:14.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.857+0000 7f434c69e640 1 -- 172.21.15.88:0/2468505883 shutdown_connections 2023-10-16T20:53:14.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.857+0000 7f434c69e640 1 --2- 172.21.15.88:0/2468505883 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f43280664a0 0x7f4328068960 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:14.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.857+0000 7f434c69e640 1 --2- 172.21.15.88:0/2468505883 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f434415a810 0x7f43441544c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:14.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.857+0000 7f434c69e640 1 --2- 172.21.15.88:0/2468505883 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f43441406d0 0x7f4344153f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:14.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.857+0000 7f434c69e640 1 --2- 172.21.15.88:0/2468505883 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f434406a970 0x7f434413d850 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:14.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.857+0000 7f434c69e640 1 -- 172.21.15.88:0/2468505883 >> 172.21.15.88:0/2468505883 conn(0x7f43440cad50 msgr2=0x7f43440cb130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:14.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.857+0000 7f434c69e640 1 -- 172.21.15.88:0/2468505883 shutdown_connections 2023-10-16T20:53:14.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:14.857+0000 7f434c69e640 1 -- 172.21.15.88:0/2468505883 wait complete. 2023-10-16T20:53:14.878 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-16T20:53:16.879 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:17.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.037+0000 7ff79bad6640 1 Processor -- start 2023-10-16T20:53:17.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.037+0000 7ff79bad6640 1 -- start start 2023-10-16T20:53:17.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.037+0000 7ff79bad6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff79414fd10 0x7ff794070c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:17.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.037+0000 7ff79bad6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff79406f710 0x7ff79406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:17.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.037+0000 7ff79bad6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff794156430 con 0x7ff794071170 2023-10-16T20:53:17.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.037+0000 7ff79bad6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff794070500 con 0x7ff79414fd10 2023-10-16T20:53:17.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.037+0000 7ff79bad6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff794155b00 con 0x7ff79406f710 2023-10-16T20:53:17.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.037+0000 7ff799872640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff79414fd10 0x7ff794070c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:17.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.037+0000 7ff79a073640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff79406f710 0x7ff79406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:17.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.037+0000 7ff799872640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff79414fd10 0x7ff794070c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:51452/0 (socket says 172.21.15.88:51452) 2023-10-16T20:53:17.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.037+0000 7ff799872640 1 -- 172.21.15.88:0/3545777964 learned_addr learned my addr 172.21.15.88:0/3545777964 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:17.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.037+0000 7ff79a073640 1 -- 172.21.15.88:0/3545777964 >> v2:172.21.15.191:3300/0 conn(0x7ff79414fd10 msgr2=0x7ff794070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:17.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.037+0000 7ff79a073640 1 --2- 172.21.15.88:0/3545777964 >> v2:172.21.15.191:3300/0 conn(0x7ff79414fd10 0x7ff794070c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:17.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.037+0000 7ff79a073640 1 -- 172.21.15.88:0/3545777964 >> v1:172.21.15.191:6789/0 conn(0x7ff794071170 legacy=0x7ff794069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:17.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.037+0000 7ff79a073640 1 -- 172.21.15.88:0/3545777964 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff794155de0 con 0x7ff79406f710 2023-10-16T20:53:17.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.041+0000 7ff79a073640 1 --2- 172.21.15.88:0/3545777964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff79406f710 0x7ff79406fb80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff78c00aea0 tx=0x7ff78c020240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e4e5a859af97ff2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:17.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.041+0000 7ff798870640 1 -- 172.21.15.88:0/3545777964 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c00a2d0 con 0x7ff79406f710 2023-10-16T20:53:17.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.041+0000 7ff798870640 1 -- 172.21.15.88:0/3545777964 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78c0ac050 con 0x7ff79406f710 2023-10-16T20:53:17.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.041+0000 7ff798870640 1 -- 172.21.15.88:0/3545777964 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c008660 con 0x7ff79406f710 2023-10-16T20:53:17.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.041+0000 7ff79bad6640 1 -- 172.21.15.88:0/3545777964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff79406f710 msgr2=0x7ff79406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:17.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.041+0000 7ff79bad6640 1 --2- 172.21.15.88:0/3545777964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff79406f710 0x7ff79406fb80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff78c00aea0 tx=0x7ff78c020240 comp rx=0 tx=0).stop 2023-10-16T20:53:17.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.041+0000 7ff79bad6640 1 -- 172.21.15.88:0/3545777964 shutdown_connections 2023-10-16T20:53:17.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.041+0000 7ff79bad6640 1 --2- 172.21.15.88:0/3545777964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff79406f710 0x7ff79406fb80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:17.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.041+0000 7ff79bad6640 1 --2- 172.21.15.88:0/3545777964 >> v2:172.21.15.191:3300/0 conn(0x7ff79414fd10 0x7ff794070c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:17.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.041+0000 7ff79bad6640 1 -- 172.21.15.88:0/3545777964 >> 172.21.15.88:0/3545777964 conn(0x7ff7940cad50 msgr2=0x7ff7940cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:17.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.041+0000 7ff79bad6640 1 -- 172.21.15.88:0/3545777964 shutdown_connections 2023-10-16T20:53:17.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.041+0000 7ff79bad6640 1 -- 172.21.15.88:0/3545777964 wait complete. 2023-10-16T20:53:17.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff79bad6640 1 Processor -- start 2023-10-16T20:53:17.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff79bad6640 1 -- start start 2023-10-16T20:53:17.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff79bad6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff79406f710 0x7ff7941dfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:17.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff79bad6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff794071170 0x7ff7940c5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:17.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff79bad6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff79414fd10 0x7ff794140be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:17.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff79bad6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff794147c80 con 0x7ff79406f710 2023-10-16T20:53:17.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff79bad6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7941e0200 con 0x7ff79414fd10 2023-10-16T20:53:17.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff79bad6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79406aa80 con 0x7ff794071170 2023-10-16T20:53:17.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff799872640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff79406f710 0x7ff7941dfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:17.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff799872640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff79406f710 0x7ff7941dfcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34664/0 (socket says 172.21.15.88:34664) 2023-10-16T20:53:17.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff799872640 1 -- 172.21.15.88:0/3979298073 learned_addr learned my addr 172.21.15.88:0/3979298073 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:17.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff79a073640 1 --2- 172.21.15.88:0/3979298073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff79414fd10 0x7ff794140be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:17.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff799071640 1 --2- 172.21.15.88:0/3979298073 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff794071170 0x7ff7940c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:17.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff799872640 1 -- 172.21.15.88:0/3979298073 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff794071170 msgr2=0x7ff7940c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:17.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff799872640 1 --2- 172.21.15.88:0/3979298073 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff794071170 0x7ff7940c5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:17.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff799872640 1 -- 172.21.15.88:0/3979298073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff79414fd10 msgr2=0x7ff794140be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:17.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff799872640 1 --2- 172.21.15.88:0/3979298073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff79414fd10 0x7ff794140be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:17.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff799872640 1 -- 172.21.15.88:0/3979298073 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78c00ab80 con 0x7ff79406f710 2023-10-16T20:53:17.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff79a073640 1 --2- 172.21.15.88:0/3979298073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff79414fd10 0x7ff794140be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:53:17.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.045+0000 7ff799872640 1 --2- 172.21.15.88:0/3979298073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff79406f710 0x7ff7941dfcc0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ff79000e7f0 tx=0x7ff79000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:17.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.049+0000 7ff78a7fc640 1 -- 172.21.15.88:0/3979298073 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff790009580 con 0x7ff79406f710 2023-10-16T20:53:17.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.049+0000 7ff78a7fc640 1 -- 172.21.15.88:0/3979298073 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79000c940 con 0x7ff79406f710 2023-10-16T20:53:17.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.049+0000 7ff79bad6640 1 -- 172.21.15.88:0/3979298073 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7941e1330 con 0x7ff79406f710 2023-10-16T20:53:17.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.049+0000 7ff78a7fc640 1 -- 172.21.15.88:0/3979298073 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79000ccf0 con 0x7ff79406f710 2023-10-16T20:53:17.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.049+0000 7ff79bad6640 1 -- 172.21.15.88:0/3979298073 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7941ec690 con 0x7ff79406f710 2023-10-16T20:53:17.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.049+0000 7ff79bad6640 1 -- 172.21.15.88:0/3979298073 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff794074700 con 0x7ff79406f710 2023-10-16T20:53:17.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.049+0000 7ff78a7fc640 1 -- 172.21.15.88:0/3979298073 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff790013470 con 0x7ff79406f710 2023-10-16T20:53:17.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.049+0000 7ff78a7fc640 1 --2- 172.21.15.88:0/3979298073 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff7780664a0 0x7ff778068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:17.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.049+0000 7ff78a7fc640 1 -- 172.21.15.88:0/3979298073 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7ff790084440 con 0x7ff79406f710 2023-10-16T20:53:17.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.049+0000 7ff799071640 1 --2- 172.21.15.88:0/3979298073 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff7780664a0 0x7ff778068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:17.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.053+0000 7ff799071640 1 --2- 172.21.15.88:0/3979298073 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff7780664a0 0x7ff778068960 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff79414d330 tx=0x7ff784009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:17.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.057+0000 7ff78a7fc640 1 -- 172.21.15.88:0/3979298073 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff7900513b0 con 0x7ff79406f710 2023-10-16T20:53:17.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.261+0000 7ff79bad6640 1 -- 172.21.15.88:0/3979298073 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7940c6c60 con 0x7ff79406f710 2023-10-16T20:53:17.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.261+0000 7ff78a7fc640 1 -- 172.21.15.88:0/3979298073 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+6571 (secure 0 0 0) 0x7ff7900549d0 con 0x7ff79406f710 2023-10-16T20:53:17.265 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:17.265 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5378,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:52:08.548787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:17.266 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 88 2023-10-16T20:53:17.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.265+0000 7ff79bad6640 1 -- 172.21.15.88:0/3979298073 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff7780664a0 msgr2=0x7ff778068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:17.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.265+0000 7ff79bad6640 1 --2- 172.21.15.88:0/3979298073 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff7780664a0 0x7ff778068960 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff79414d330 tx=0x7ff784009040 comp rx=0 tx=0).stop 2023-10-16T20:53:17.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.265+0000 7ff79bad6640 1 -- 172.21.15.88:0/3979298073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff79406f710 msgr2=0x7ff7941dfcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:17.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.265+0000 7ff79bad6640 1 --2- 172.21.15.88:0/3979298073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff79406f710 0x7ff7941dfcc0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ff79000e7f0 tx=0x7ff79000ecb0 comp rx=0 tx=0).stop 2023-10-16T20:53:17.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.265+0000 7ff79bad6640 1 -- 172.21.15.88:0/3979298073 shutdown_connections 2023-10-16T20:53:17.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.265+0000 7ff79bad6640 1 --2- 172.21.15.88:0/3979298073 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff7780664a0 0x7ff778068960 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:17.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.265+0000 7ff79bad6640 1 --2- 172.21.15.88:0/3979298073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff79414fd10 0x7ff794140be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:17.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.265+0000 7ff79bad6640 1 --2- 172.21.15.88:0/3979298073 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff794071170 0x7ff7940c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:17.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.265+0000 7ff79bad6640 1 --2- 172.21.15.88:0/3979298073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff79406f710 0x7ff7941dfcc0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:17.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.265+0000 7ff79bad6640 1 -- 172.21.15.88:0/3979298073 >> 172.21.15.88:0/3979298073 conn(0x7ff7940cad50 msgr2=0x7ff79406d140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:17.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.265+0000 7ff79bad6640 1 -- 172.21.15.88:0/3979298073 shutdown_connections 2023-10-16T20:53:17.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:17.265+0000 7ff79bad6640 1 -- 172.21.15.88:0/3979298073 wait complete. 2023-10-16T20:53:17.287 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-16T20:53:19.288 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:19.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.449+0000 7f58115d9640 1 Processor -- start 2023-10-16T20:53:19.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.449+0000 7f58115d9640 1 -- start start 2023-10-16T20:53:19.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.453+0000 7f58115d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f580c153560 0x7f580c0b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:19.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.453+0000 7f58115d9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f580c153940 0x7f580c15c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:19.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.453+0000 7f58115d9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580c159f10 con 0x7f580c0c9a00 2023-10-16T20:53:19.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.453+0000 7f58115d9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f580c1593f0 con 0x7f580c153940 2023-10-16T20:53:19.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.453+0000 7f58115d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580c15a3a0 con 0x7f580c153560 2023-10-16T20:53:19.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.453+0000 7f580affd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f580c153560 0x7f580c0b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:19.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.453+0000 7f580a7fc640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f580c0c9a00 0x7f580c0ca660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:44394/0 (socket says 172.21.15.88:44394) 2023-10-16T20:53:19.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.453+0000 7f580a7fc640 1 -- 172.21.15.88:0/2602174964 learned_addr learned my addr 172.21.15.88:0/2602174964 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:53:19.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.453+0000 7f580b7fe640 1 --2- 172.21.15.88:0/2602174964 >> v2:172.21.15.191:3300/0 conn(0x7f580c153940 0x7f580c15c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:19.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.453+0000 7f580affd640 1 -- 172.21.15.88:0/2602174964 >> v2:172.21.15.191:3300/0 conn(0x7f580c153940 msgr2=0x7f580c15c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:19.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.453+0000 7f580affd640 1 --2- 172.21.15.88:0/2602174964 >> v2:172.21.15.191:3300/0 conn(0x7f580c153940 0x7f580c15c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:19.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.453+0000 7f5809ffb640 1 -- 172.21.15.88:0/2602174964 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187448030 0 0) 0x7f580c159f10 con 0x7f580c0c9a00 2023-10-16T20:53:19.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.453+0000 7f580affd640 1 -- 172.21.15.88:0/2602174964 >> v1:172.21.15.191:6789/0 conn(0x7f580c0c9a00 legacy=0x7f580c0ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:19.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.453+0000 7f580affd640 1 -- 172.21.15.88:0/2602174964 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580c15d610 con 0x7f580c153560 2023-10-16T20:53:19.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.457+0000 7f580affd640 1 --2- 172.21.15.88:0/2602174964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f580c153560 0x7f580c0b26f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f57fc0097a0 tx=0x7f57fc01c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95c14e19b4da46ac server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:19.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.457+0000 7f5809ffb640 1 -- 172.21.15.88:0/2602174964 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc0a7070 con 0x7f580c153560 2023-10-16T20:53:19.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.457+0000 7f5809ffb640 1 -- 172.21.15.88:0/2602174964 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57fc0ab430 con 0x7f580c153560 2023-10-16T20:53:19.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.457+0000 7f5809ffb640 1 -- 172.21.15.88:0/2602174964 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc0ab7c0 con 0x7f580c153560 2023-10-16T20:53:19.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.457+0000 7f58115d9640 1 -- 172.21.15.88:0/2602174964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f580c153560 msgr2=0x7f580c0b26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:19.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.457+0000 7f58115d9640 1 --2- 172.21.15.88:0/2602174964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f580c153560 0x7f580c0b26f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f57fc0097a0 tx=0x7f57fc01c430 comp rx=0 tx=0).stop 2023-10-16T20:53:19.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.457+0000 7f58115d9640 1 -- 172.21.15.88:0/2602174964 shutdown_connections 2023-10-16T20:53:19.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.457+0000 7f58115d9640 1 --2- 172.21.15.88:0/2602174964 >> v2:172.21.15.191:3300/0 conn(0x7f580c153940 0x7f580c15c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:19.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.457+0000 7f58115d9640 1 --2- 172.21.15.88:0/2602174964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f580c153560 0x7f580c0b26f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:19.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.457+0000 7f58115d9640 1 -- 172.21.15.88:0/2602174964 >> 172.21.15.88:0/2602174964 conn(0x7f580c066080 msgr2=0x7f580c0715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:19.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.457+0000 7f58115d9640 1 -- 172.21.15.88:0/2602174964 shutdown_connections 2023-10-16T20:53:19.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.457+0000 7f58115d9640 1 -- 172.21.15.88:0/2602174964 wait complete. 2023-10-16T20:53:19.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.457+0000 7f58115d9640 1 Processor -- start 2023-10-16T20:53:19.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f58115d9640 1 -- start start 2023-10-16T20:53:19.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f58115d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f580c0c9a00 0x7f580c143fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:19.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f58115d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f580c153560 0x7f580c0c2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:19.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f58115d9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f580c153940 0x7f580c148290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:19.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f58115d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c0c2dd0 con 0x7f580c0c9a00 2023-10-16T20:53:19.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f58115d9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580c0c8c80 con 0x7f580c153940 2023-10-16T20:53:19.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f58115d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580c0c6eb0 con 0x7f580c153560 2023-10-16T20:53:19.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f580affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f580c0c9a00 0x7f580c143fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:19.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f580affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f580c0c9a00 0x7f580c143fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40576/0 (socket says 172.21.15.88:40576) 2023-10-16T20:53:19.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f580affd640 1 -- 172.21.15.88:0/77795639 learned_addr learned my addr 172.21.15.88:0/77795639 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:19.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f580a7fc640 1 --2- 172.21.15.88:0/77795639 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f580c153560 0x7f580c0c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:19.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f580b7fe640 1 --2- 172.21.15.88:0/77795639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f580c153940 0x7f580c148290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:19.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f580affd640 1 -- 172.21.15.88:0/77795639 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f580c153560 msgr2=0x7f580c0c2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:19.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f580affd640 1 --2- 172.21.15.88:0/77795639 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f580c153560 0x7f580c0c2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:19.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f580affd640 1 -- 172.21.15.88:0/77795639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f580c153940 msgr2=0x7f580c148290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:19.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f580affd640 1 --2- 172.21.15.88:0/77795639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f580c153940 0x7f580c148290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:19.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f580affd640 1 -- 172.21.15.88:0/77795639 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57fc0093b0 con 0x7f580c0c9a00 2023-10-16T20:53:19.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f580affd640 1 --2- 172.21.15.88:0/77795639 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f580c0c9a00 0x7f580c143fb0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f57fc01a060 tx=0x7f57fc0191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:19.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f57f37fe640 1 -- 172.21.15.88:0/77795639 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc0b4600 con 0x7f580c0c9a00 2023-10-16T20:53:19.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f57f37fe640 1 -- 172.21.15.88:0/77795639 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57fc0b4760 con 0x7f580c0c9a00 2023-10-16T20:53:19.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f57f37fe640 1 -- 172.21.15.88:0/77795639 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc0b4af0 con 0x7f580c0c9a00 2023-10-16T20:53:19.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f58115d9640 1 -- 172.21.15.88:0/77795639 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f580c0c7be0 con 0x7f580c0c9a00 2023-10-16T20:53:19.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f58115d9640 1 -- 172.21.15.88:0/77795639 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f580c0c81a0 con 0x7f580c0c9a00 2023-10-16T20:53:19.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.461+0000 7f58115d9640 1 -- 172.21.15.88:0/77795639 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f580c04e9c0 con 0x7f580c0c9a00 2023-10-16T20:53:19.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.465+0000 7f57f37fe640 1 -- 172.21.15.88:0/77795639 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f57fc0b4c50 con 0x7f580c0c9a00 2023-10-16T20:53:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.465+0000 7f57f37fe640 1 --2- 172.21.15.88:0/77795639 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57f80664a0 0x7f57f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.465+0000 7f57f37fe640 1 -- 172.21.15.88:0/77795639 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7f57fc11e7e0 con 0x7f580c0c9a00 2023-10-16T20:53:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.465+0000 7f580a7fc640 1 --2- 172.21.15.88:0/77795639 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57f80664a0 0x7f57f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.465+0000 7f580a7fc640 1 --2- 172.21.15.88:0/77795639 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57f80664a0 0x7f57f8068960 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f580c0c7460 tx=0x7f5800007d60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:19.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.469+0000 7f57f37fe640 1 -- 172.21.15.88:0/77795639 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f57fc0eb6b0 con 0x7f580c0c9a00 2023-10-16T20:53:19.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.689+0000 7f58115d9640 1 -- 172.21.15.88:0/77795639 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f580c0cbe40 con 0x7f580c0c9a00 2023-10-16T20:53:19.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.689+0000 7f57f37fe640 1 -- 172.21.15.88:0/77795639 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+6571 (secure 0 0 0) 0x7f57fc0eecd0 con 0x7f580c0c9a00 2023-10-16T20:53:19.692 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:19.692 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5378,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:52:08.548787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:19.692 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 88 2023-10-16T20:53:19.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.689+0000 7f58115d9640 1 -- 172.21.15.88:0/77795639 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57f80664a0 msgr2=0x7f57f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:19.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.689+0000 7f58115d9640 1 --2- 172.21.15.88:0/77795639 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57f80664a0 0x7f57f8068960 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f580c0c7460 tx=0x7f5800007d60 comp rx=0 tx=0).stop 2023-10-16T20:53:19.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.689+0000 7f58115d9640 1 -- 172.21.15.88:0/77795639 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f580c0c9a00 msgr2=0x7f580c143fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:19.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.689+0000 7f58115d9640 1 --2- 172.21.15.88:0/77795639 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f580c0c9a00 0x7f580c143fb0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f57fc01a060 tx=0x7f57fc0191d0 comp rx=0 tx=0).stop 2023-10-16T20:53:19.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.689+0000 7f58115d9640 1 -- 172.21.15.88:0/77795639 shutdown_connections 2023-10-16T20:53:19.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.689+0000 7f58115d9640 1 --2- 172.21.15.88:0/77795639 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f57f80664a0 0x7f57f8068960 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:19.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.689+0000 7f58115d9640 1 --2- 172.21.15.88:0/77795639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f580c153940 0x7f580c148290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:19.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.689+0000 7f58115d9640 1 --2- 172.21.15.88:0/77795639 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f580c153560 0x7f580c0c2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:19.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.689+0000 7f58115d9640 1 --2- 172.21.15.88:0/77795639 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f580c0c9a00 0x7f580c143fb0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:19.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.689+0000 7f58115d9640 1 -- 172.21.15.88:0/77795639 >> 172.21.15.88:0/77795639 conn(0x7f580c066080 msgr2=0x7f580c15a760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:19.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.693+0000 7f58115d9640 1 -- 172.21.15.88:0/77795639 shutdown_connections 2023-10-16T20:53:19.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:19.693+0000 7f58115d9640 1 -- 172.21.15.88:0/77795639 wait complete. 2023-10-16T20:53:19.715 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-16T20:53:21.716 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:21.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.869+0000 7f6756288640 1 Processor -- start 2023-10-16T20:53:21.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.869+0000 7f6756288640 1 -- start start 2023-10-16T20:53:21.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.869+0000 7f6756288640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f67500cb580 0x7f67500cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:21.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.869+0000 7f6756288640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f67500b1a60 0x7f6750163b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:21.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.869+0000 7f6756288640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67500c9fc0 con 0x7f67500c9af0 2023-10-16T20:53:21.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.869+0000 7f6756288640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67500b20c0 con 0x7f67500cb580 2023-10-16T20:53:21.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.869+0000 7f6756288640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67500cbea0 con 0x7f67500b1a60 2023-10-16T20:53:21.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.869+0000 7f6755286640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f67500cb580 0x7f67500cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:21.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.869+0000 7f6755286640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f67500cb580 0x7f67500cb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40586/0 (socket says 172.21.15.88:40586) 2023-10-16T20:53:21.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.869+0000 7f6755286640 1 -- 172.21.15.88:0/486608724 learned_addr learned my addr 172.21.15.88:0/486608724 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:21.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.869+0000 7f6755a87640 1 --2- 172.21.15.88:0/486608724 >> v2:172.21.15.191:3300/0 conn(0x7f67500b1a60 0x7f6750163b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:21.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.869+0000 7f6755286640 1 -- 172.21.15.88:0/486608724 >> v2:172.21.15.191:3300/0 conn(0x7f67500b1a60 msgr2=0x7f6750163b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:21.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.869+0000 7f6755286640 1 --2- 172.21.15.88:0/486608724 >> v2:172.21.15.191:3300/0 conn(0x7f67500b1a60 0x7f6750163b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:21.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.869+0000 7f6755286640 1 -- 172.21.15.88:0/486608724 >> v1:172.21.15.191:6789/0 conn(0x7f67500c9af0 legacy=0x7f6750159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:21.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.869+0000 7f6755286640 1 -- 172.21.15.88:0/486608724 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67500c47d0 con 0x7f67500cb580 2023-10-16T20:53:21.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.873+0000 7f6755286640 1 --2- 172.21.15.88:0/486608724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500cb580 0x7f67500cb960 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f67500cfb80 tx=0x7f674c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be63a30c055fe90 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:21.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.873+0000 7f6747fff640 1 -- 172.21.15.88:0/486608724 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c0ab640 con 0x7f67500cb580 2023-10-16T20:53:21.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.873+0000 7f6747fff640 1 -- 172.21.15.88:0/486608724 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674c0a7070 con 0x7f67500cb580 2023-10-16T20:53:21.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.873+0000 7f6747fff640 1 -- 172.21.15.88:0/486608724 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c0ab9c0 con 0x7f67500cb580 2023-10-16T20:53:21.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.873+0000 7f6756288640 1 -- 172.21.15.88:0/486608724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500cb580 msgr2=0x7f67500cb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:21.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.873+0000 7f6756288640 1 --2- 172.21.15.88:0/486608724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500cb580 0x7f67500cb960 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f67500cfb80 tx=0x7f674c01c440 comp rx=0 tx=0).stop 2023-10-16T20:53:21.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.873+0000 7f6756288640 1 -- 172.21.15.88:0/486608724 shutdown_connections 2023-10-16T20:53:21.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.873+0000 7f6756288640 1 --2- 172.21.15.88:0/486608724 >> v2:172.21.15.191:3300/0 conn(0x7f67500b1a60 0x7f6750163b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:21.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.873+0000 7f6756288640 1 --2- 172.21.15.88:0/486608724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500cb580 0x7f67500cb960 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:21.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.873+0000 7f6756288640 1 -- 172.21.15.88:0/486608724 >> 172.21.15.88:0/486608724 conn(0x7f675006bb40 msgr2=0x7f675006bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:21.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.873+0000 7f6756288640 1 -- 172.21.15.88:0/486608724 shutdown_connections 2023-10-16T20:53:21.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.873+0000 7f6756288640 1 -- 172.21.15.88:0/486608724 wait complete. 2023-10-16T20:53:21.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6756288640 1 Processor -- start 2023-10-16T20:53:21.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6756288640 1 -- start start 2023-10-16T20:53:21.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6756288640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500b1a60 0x7f67500c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:21.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6756288640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67500c9af0 0x7f67500c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:21.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6756288640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f67500cb580 0x7f6750157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:21.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6756288640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67501f27d0 con 0x7f67500b1a60 2023-10-16T20:53:21.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6756288640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67501f1c30 con 0x7f67500c9af0 2023-10-16T20:53:21.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6756288640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6750158c10 con 0x7f67500cb580 2023-10-16T20:53:21.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6755286640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500b1a60 0x7f67500c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:21.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6755286640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500b1a60 0x7f67500c5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40590/0 (socket says 172.21.15.88:40590) 2023-10-16T20:53:21.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6755286640 1 -- 172.21.15.88:0/364478127 learned_addr learned my addr 172.21.15.88:0/364478127 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:21.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6755a87640 1 --2- 172.21.15.88:0/364478127 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f67500cb580 0x7f6750157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:21.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6754a85640 1 --2- 172.21.15.88:0/364478127 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67500c9af0 0x7f67500c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:21.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6755286640 1 -- 172.21.15.88:0/364478127 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f67500cb580 msgr2=0x7f6750157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:21.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6755286640 1 --2- 172.21.15.88:0/364478127 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f67500cb580 0x7f6750157070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:21.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6755286640 1 -- 172.21.15.88:0/364478127 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67500c9af0 msgr2=0x7f67500c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:21.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6755286640 1 --2- 172.21.15.88:0/364478127 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67500c9af0 0x7f67500c63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:21.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6755286640 1 -- 172.21.15.88:0/364478127 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c0094d0 con 0x7f67500b1a60 2023-10-16T20:53:21.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.877+0000 7f6755286640 1 --2- 172.21.15.88:0/364478127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500b1a60 0x7f67500c5e70 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f674c019050 tx=0x7f674c01ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:21.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.881+0000 7f6745ffb640 1 -- 172.21.15.88:0/364478127 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c0abe80 con 0x7f67500b1a60 2023-10-16T20:53:21.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.881+0000 7f6745ffb640 1 -- 172.21.15.88:0/364478127 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674c0be3f0 con 0x7f67500b1a60 2023-10-16T20:53:21.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.881+0000 7f6745ffb640 1 -- 172.21.15.88:0/364478127 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c0be780 con 0x7f67500b1a60 2023-10-16T20:53:21.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.881+0000 7f6756288640 1 -- 172.21.15.88:0/364478127 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6750158eb0 con 0x7f67500b1a60 2023-10-16T20:53:21.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.881+0000 7f6756288640 1 -- 172.21.15.88:0/364478127 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67501f2180 con 0x7f67500b1a60 2023-10-16T20:53:21.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.881+0000 7f6756288640 1 -- 172.21.15.88:0/364478127 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67500ca2c0 con 0x7f67500b1a60 2023-10-16T20:53:21.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.881+0000 7f6745ffb640 1 -- 172.21.15.88:0/364478127 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f674c0b43f0 con 0x7f67500b1a60 2023-10-16T20:53:21.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.881+0000 7f6745ffb640 1 --2- 172.21.15.88:0/364478127 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f67340664a0 0x7f6734068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:21.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.881+0000 7f6745ffb640 1 -- 172.21.15.88:0/364478127 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7f674c11e7d0 con 0x7f67500b1a60 2023-10-16T20:53:21.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.885+0000 7f6754a85640 1 --2- 172.21.15.88:0/364478127 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f67340664a0 0x7f6734068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:21.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.885+0000 7f6754a85640 1 --2- 172.21.15.88:0/364478127 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f67340664a0 0x7f6734068960 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f675015a2c0 tx=0x7f6740009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:21.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:21.889+0000 7f6745ffb640 1 -- 172.21.15.88:0/364478127 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f674c0eb610 con 0x7f67500b1a60 2023-10-16T20:53:22.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:22.097+0000 7f6756288640 1 -- 172.21.15.88:0/364478127 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f675004e9c0 con 0x7f67500b1a60 2023-10-16T20:53:22.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:22.097+0000 7f6745ffb640 1 -- 172.21.15.88:0/364478127 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+6571 (secure 0 0 0) 0x7f674c0eec30 con 0x7f67500b1a60 2023-10-16T20:53:22.100 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:22.101 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5378,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:52:08.548787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:22.101 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 88 2023-10-16T20:53:22.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:22.101+0000 7f6756288640 1 -- 172.21.15.88:0/364478127 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f67340664a0 msgr2=0x7f6734068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:22.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:22.101+0000 7f6756288640 1 --2- 172.21.15.88:0/364478127 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f67340664a0 0x7f6734068960 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f675015a2c0 tx=0x7f6740009040 comp rx=0 tx=0).stop 2023-10-16T20:53:22.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:22.101+0000 7f6756288640 1 -- 172.21.15.88:0/364478127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500b1a60 msgr2=0x7f67500c5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:22.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:22.101+0000 7f6756288640 1 --2- 172.21.15.88:0/364478127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500b1a60 0x7f67500c5e70 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f674c019050 tx=0x7f674c01ac80 comp rx=0 tx=0).stop 2023-10-16T20:53:22.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:22.101+0000 7f6756288640 1 -- 172.21.15.88:0/364478127 shutdown_connections 2023-10-16T20:53:22.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:22.101+0000 7f6756288640 1 --2- 172.21.15.88:0/364478127 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f67340664a0 0x7f6734068960 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:22.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:22.101+0000 7f6756288640 1 --2- 172.21.15.88:0/364478127 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f67500cb580 0x7f6750157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:22.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:22.101+0000 7f6756288640 1 --2- 172.21.15.88:0/364478127 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67500c9af0 0x7f67500c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:22.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:22.101+0000 7f6756288640 1 --2- 172.21.15.88:0/364478127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500b1a60 0x7f67500c5e70 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:22.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:22.101+0000 7f6756288640 1 -- 172.21.15.88:0/364478127 >> 172.21.15.88:0/364478127 conn(0x7f675006bb40 msgr2=0x7f675006bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:22.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:22.101+0000 7f6756288640 1 -- 172.21.15.88:0/364478127 shutdown_connections 2023-10-16T20:53:22.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:22.101+0000 7f6756288640 1 -- 172.21.15.88:0/364478127 wait complete. 2023-10-16T20:53:22.122 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-16T20:53:24.123 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:24.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.269+0000 7f461742f640 1 Processor -- start 2023-10-16T20:53:24.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.269+0000 7f461742f640 1 -- start start 2023-10-16T20:53:24.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.269+0000 7f461742f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4610070b40 0x7f4610159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:24.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.269+0000 7f461742f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4610070f90 0x7f4610159870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:24.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.269+0000 7f461742f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46100c5bd0 con 0x7f4610070760 2023-10-16T20:53:24.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.269+0000 7f461742f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461006ba10 con 0x7f4610070f90 2023-10-16T20:53:24.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.269+0000 7f461742f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4610159f10 con 0x7f4610070b40 2023-10-16T20:53:24.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.269+0000 7f4616c2e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4610070f90 0x7f4610159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:24.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.269+0000 7f4616c2e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4610070f90 0x7f4610159870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40594/0 (socket says 172.21.15.88:40594) 2023-10-16T20:53:24.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.269+0000 7f4616c2e640 1 -- 172.21.15.88:0/2659033694 learned_addr learned my addr 172.21.15.88:0/2659033694 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:24.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.269+0000 7f4615c2c640 1 --2- 172.21.15.88:0/2659033694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4610070b40 0x7f4610159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:24.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.269+0000 7f4616c2e640 1 -- 172.21.15.88:0/2659033694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4610070b40 msgr2=0x7f4610159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:24.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.269+0000 7f4616c2e640 1 --2- 172.21.15.88:0/2659033694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4610070b40 0x7f4610159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:24.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.269+0000 7f4616c2e640 1 -- 172.21.15.88:0/2659033694 >> v1:172.21.15.191:6789/0 conn(0x7f4610070760 legacy=0x7f46100c62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:24.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.269+0000 7f4616c2e640 1 -- 172.21.15.88:0/2659033694 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461015a1f0 con 0x7f4610070f90 2023-10-16T20:53:24.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.273+0000 7f4616c2e640 1 --2- 172.21.15.88:0/2659033694 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4610070f90 0x7f4610159870 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4608009e50 tx=0x7f460801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2217b632fd176a03 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:24.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.273+0000 7f461542b640 1 -- 172.21.15.88:0/2659033694 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46080af400 con 0x7f4610070f90 2023-10-16T20:53:24.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.273+0000 7f461542b640 1 -- 172.21.15.88:0/2659033694 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46080ab050 con 0x7f4610070f90 2023-10-16T20:53:24.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.273+0000 7f461542b640 1 -- 172.21.15.88:0/2659033694 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46080af780 con 0x7f4610070f90 2023-10-16T20:53:24.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f461742f640 1 -- 172.21.15.88:0/2659033694 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4610070f90 msgr2=0x7f4610159870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:24.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f461742f640 1 --2- 172.21.15.88:0/2659033694 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4610070f90 0x7f4610159870 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4608009e50 tx=0x7f460801f1f0 comp rx=0 tx=0).stop 2023-10-16T20:53:24.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f461742f640 1 -- 172.21.15.88:0/2659033694 shutdown_connections 2023-10-16T20:53:24.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f461742f640 1 --2- 172.21.15.88:0/2659033694 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4610070f90 0x7f4610159870 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:24.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f461742f640 1 --2- 172.21.15.88:0/2659033694 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4610070b40 0x7f4610159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:24.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f461742f640 1 -- 172.21.15.88:0/2659033694 >> 172.21.15.88:0/2659033694 conn(0x7f46100c83d0 msgr2=0x7f46100c87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:24.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f461742f640 1 -- 172.21.15.88:0/2659033694 shutdown_connections 2023-10-16T20:53:24.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f461742f640 1 -- 172.21.15.88:0/2659033694 wait complete. 2023-10-16T20:53:24.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f461742f640 1 Processor -- start 2023-10-16T20:53:24.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f461742f640 1 -- start start 2023-10-16T20:53:24.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f461742f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4610070760 0x7f4610156470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:24.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f461742f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4610070b40 0x7f4610153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:24.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f461742f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4610070f90 0x7f4610153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:24.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f461742f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4610155650 con 0x7f4610070f90 2023-10-16T20:53:24.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f461742f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461015d330 con 0x7f4610070760 2023-10-16T20:53:24.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f461742f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46101f21f0 con 0x7f4610070b40 2023-10-16T20:53:24.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f4616c2e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4610070f90 0x7f4610153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:24.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f4616c2e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4610070f90 0x7f4610153fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40598/0 (socket says 172.21.15.88:40598) 2023-10-16T20:53:24.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f4616c2e640 1 -- 172.21.15.88:0/1606832220 learned_addr learned my addr 172.21.15.88:0/1606832220 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:24.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f4615c2c640 1 --2- 172.21.15.88:0/1606832220 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4610070b40 0x7f4610153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:24.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.277+0000 7f461642d640 1 --2- 172.21.15.88:0/1606832220 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4610070760 0x7f4610156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:24.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.281+0000 7f4616c2e640 1 -- 172.21.15.88:0/1606832220 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4610070b40 msgr2=0x7f4610153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:24.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.281+0000 7f4616c2e640 1 --2- 172.21.15.88:0/1606832220 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4610070b40 0x7f4610153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:24.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.281+0000 7f4616c2e640 1 -- 172.21.15.88:0/1606832220 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4610070760 msgr2=0x7f4610156470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:24.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.281+0000 7f4616c2e640 1 --2- 172.21.15.88:0/1606832220 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4610070760 0x7f4610156470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:24.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.281+0000 7f4616c2e640 1 -- 172.21.15.88:0/1606832220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4608009b30 con 0x7f4610070f90 2023-10-16T20:53:24.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.281+0000 7f4616c2e640 1 --2- 172.21.15.88:0/1606832220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4610070f90 0x7f4610153fc0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f460801d8a0 tx=0x7f460801bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:24.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.281+0000 7f4606ffd640 1 -- 172.21.15.88:0/1606832220 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4608007000 con 0x7f4610070f90 2023-10-16T20:53:24.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.281+0000 7f4606ffd640 1 -- 172.21.15.88:0/1606832220 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46080afe60 con 0x7f4610070f90 2023-10-16T20:53:24.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.281+0000 7f461742f640 1 -- 172.21.15.88:0/1606832220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46101f24d0 con 0x7f4610070f90 2023-10-16T20:53:24.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.281+0000 7f4606ffd640 1 -- 172.21.15.88:0/1606832220 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46080b9650 con 0x7f4610070f90 2023-10-16T20:53:24.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.281+0000 7f461742f640 1 -- 172.21.15.88:0/1606832220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f461015cce0 con 0x7f4610070f90 2023-10-16T20:53:24.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.281+0000 7f461742f640 1 -- 172.21.15.88:0/1606832220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f461015d4e0 con 0x7f4610070f90 2023-10-16T20:53:24.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.281+0000 7f4606ffd640 1 -- 172.21.15.88:0/1606832220 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f46080b6070 con 0x7f4610070f90 2023-10-16T20:53:24.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.285+0000 7f4606ffd640 1 --2- 172.21.15.88:0/1606832220 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f45f40664d0 0x7f45f4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:24.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.285+0000 7f4606ffd640 1 -- 172.21.15.88:0/1606832220 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7179+0+0 (secure 0 0 0) 0x7f4608122430 con 0x7f4610070f90 2023-10-16T20:53:24.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.285+0000 7f461642d640 1 --2- 172.21.15.88:0/1606832220 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f45f40664d0 0x7f45f4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:24.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.285+0000 7f461642d640 1 --2- 172.21.15.88:0/1606832220 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f45f40664d0 0x7f45f4068990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f460c001810 tx=0x7f460c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:24.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.289+0000 7f4606ffd640 1 -- 172.21.15.88:0/1606832220 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f461015d4e0 con 0x7f4610070f90 2023-10-16T20:53:24.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.525+0000 7f461742f640 1 -- 172.21.15.88:0/1606832220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46100c7bb0 con 0x7f4610070f90 2023-10-16T20:53:24.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.525+0000 7f4606ffd640 1 -- 172.21.15.88:0/1606832220 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+6571 (secure 0 0 0) 0x7f46080ef270 con 0x7f4610070f90 2023-10-16T20:53:24.529 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:24.529 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5378,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:52:08.548787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":4889},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4889":{"gid":4889,"name":"b","rank":4,"incarnation":30,"state":"up:active","state_seq":21,"addr":"172.21.15.191:6837/2468647900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2468647900},{"type":"v1","addr":"172.21.15.191:6837","nonce":2468647900}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:24.530 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 88 2023-10-16T20:53:24.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.529+0000 7f461742f640 1 -- 172.21.15.88:0/1606832220 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f45f40664d0 msgr2=0x7f45f4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:24.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.529+0000 7f461742f640 1 --2- 172.21.15.88:0/1606832220 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f45f40664d0 0x7f45f4068990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f460c001810 tx=0x7f460c009040 comp rx=0 tx=0).stop 2023-10-16T20:53:24.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.529+0000 7f461742f640 1 -- 172.21.15.88:0/1606832220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4610070f90 msgr2=0x7f4610153fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:24.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.529+0000 7f461742f640 1 --2- 172.21.15.88:0/1606832220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4610070f90 0x7f4610153fc0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f460801d8a0 tx=0x7f460801bf90 comp rx=0 tx=0).stop 2023-10-16T20:53:24.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.529+0000 7f461742f640 1 -- 172.21.15.88:0/1606832220 shutdown_connections 2023-10-16T20:53:24.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.529+0000 7f461742f640 1 --2- 172.21.15.88:0/1606832220 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f45f40664d0 0x7f45f4068990 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:24.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.529+0000 7f461742f640 1 --2- 172.21.15.88:0/1606832220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4610070f90 0x7f4610153fc0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:24.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.529+0000 7f461742f640 1 --2- 172.21.15.88:0/1606832220 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4610070b40 0x7f4610153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:24.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.529+0000 7f461742f640 1 --2- 172.21.15.88:0/1606832220 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4610070760 0x7f4610156470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:24.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.529+0000 7f461742f640 1 -- 172.21.15.88:0/1606832220 >> 172.21.15.88:0/1606832220 conn(0x7f46100c83d0 msgr2=0x7f46100c3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:24.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.529+0000 7f461742f640 1 -- 172.21.15.88:0/1606832220 shutdown_connections 2023-10-16T20:53:24.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:24.529+0000 7f461742f640 1 -- 172.21.15.88:0/1606832220 wait complete. 2023-10-16T20:53:24.553 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-16T20:53:26.554 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:26.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.709+0000 7fe7f373e640 1 Processor -- start 2023-10-16T20:53:26.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.709+0000 7fe7f373e640 1 -- start start 2023-10-16T20:53:26.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.713+0000 7fe7f373e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe7ec071b00 0x7fe7ec0c6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:26.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.713+0000 7fe7f373e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7ec071ee0 0x7fe7ec0c6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:26.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.713+0000 7fe7f373e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7ec158160 con 0x7fe7ec0c7ce0 2023-10-16T20:53:26.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.713+0000 7fe7f373e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7ec157360 con 0x7fe7ec071b00 2023-10-16T20:53:26.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.713+0000 7fe7f373e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec0c71c0 con 0x7fe7ec071ee0 2023-10-16T20:53:26.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.713+0000 7fe7f0cd9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7ec071ee0 0x7fe7ec0c6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:26.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.713+0000 7fe7f14da640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe7ec071b00 0x7fe7ec0c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:26.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.713+0000 7fe7f0cd9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7ec071ee0 0x7fe7ec0c6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:57266/0 (socket says 172.21.15.88:57266) 2023-10-16T20:53:26.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.713+0000 7fe7f0cd9640 1 -- 172.21.15.88:0/2241153104 learned_addr learned my addr 172.21.15.88:0/2241153104 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:26.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.713+0000 7fe7f1cdb640 1 --1- 172.21.15.88:0/2241153104 >> v1:172.21.15.191:6789/0 conn(0x7fe7ec0c7ce0 0x7fe7ec151180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:44424/0 (socket says 172.21.15.88:44424) 2023-10-16T20:53:26.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.713+0000 7fe7e3fff640 1 -- 172.21.15.88:0/2241153104 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862066289 0 0) 0x7fe7ec158160 con 0x7fe7ec0c7ce0 2023-10-16T20:53:26.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.713+0000 7fe7f0cd9640 1 -- 172.21.15.88:0/2241153104 >> v2:172.21.15.191:3300/0 conn(0x7fe7ec071b00 msgr2=0x7fe7ec0c6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:26.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.713+0000 7fe7f0cd9640 1 --2- 172.21.15.88:0/2241153104 >> v2:172.21.15.191:3300/0 conn(0x7fe7ec071b00 0x7fe7ec0c6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:26.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.713+0000 7fe7f0cd9640 1 -- 172.21.15.88:0/2241153104 >> v1:172.21.15.191:6789/0 conn(0x7fe7ec0c7ce0 legacy=0x7fe7ec151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:26.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.713+0000 7fe7f0cd9640 1 -- 172.21.15.88:0/2241153104 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ec157550 con 0x7fe7ec071ee0 2023-10-16T20:53:26.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.713+0000 7fe7f14da640 1 --2- 172.21.15.88:0/2241153104 >> v2:172.21.15.191:3300/0 conn(0x7fe7ec071b00 0x7fe7ec0c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:53:26.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.717+0000 7fe7f0cd9640 1 --2- 172.21.15.88:0/2241153104 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7ec071ee0 0x7fe7ec0c6b50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc009870 tx=0x7fe7dc01c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94d3db7f6e92158f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:26.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.717+0000 7fe7e3fff640 1 -- 172.21.15.88:0/2241153104 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc0aa670 con 0x7fe7ec071ee0 2023-10-16T20:53:26.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.717+0000 7fe7e3fff640 1 -- 172.21.15.88:0/2241153104 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7dc0aa7d0 con 0x7fe7ec071ee0 2023-10-16T20:53:26.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.717+0000 7fe7e3fff640 1 -- 172.21.15.88:0/2241153104 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc0aab60 con 0x7fe7ec071ee0 2023-10-16T20:53:26.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.717+0000 7fe7f373e640 1 -- 172.21.15.88:0/2241153104 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7ec071ee0 msgr2=0x7fe7ec0c6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:26.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.717+0000 7fe7f373e640 1 --2- 172.21.15.88:0/2241153104 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7ec071ee0 0x7fe7ec0c6b50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc009870 tx=0x7fe7dc01c510 comp rx=0 tx=0).stop 2023-10-16T20:53:26.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.717+0000 7fe7f373e640 1 -- 172.21.15.88:0/2241153104 shutdown_connections 2023-10-16T20:53:26.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.717+0000 7fe7f373e640 1 --2- 172.21.15.88:0/2241153104 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7ec071ee0 0x7fe7ec0c6b50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:26.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.717+0000 7fe7f373e640 1 --2- 172.21.15.88:0/2241153104 >> v2:172.21.15.191:3300/0 conn(0x7fe7ec071b00 0x7fe7ec0c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:26.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.717+0000 7fe7f373e640 1 -- 172.21.15.88:0/2241153104 >> 172.21.15.88:0/2241153104 conn(0x7fe7ec06d450 msgr2=0x7fe7ec06d860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:26.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.717+0000 7fe7f373e640 1 -- 172.21.15.88:0/2241153104 shutdown_connections 2023-10-16T20:53:26.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.717+0000 7fe7f373e640 1 -- 172.21.15.88:0/2241153104 wait complete. 2023-10-16T20:53:26.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f373e640 1 Processor -- start 2023-10-16T20:53:26.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f373e640 1 -- start start 2023-10-16T20:53:26.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f373e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7ec071b00 0x7fe7ec145970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:26.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f373e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7ec071ee0 0x7fe7ec1f54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:26.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f373e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7ec0c7ce0 0x7fe7ec1f5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:26.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f373e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec1f5040 con 0x7fe7ec071b00 2023-10-16T20:53:26.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f373e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec0703a0 con 0x7fe7ec0c7ce0 2023-10-16T20:53:26.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f373e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec070640 con 0x7fe7ec071ee0 2023-10-16T20:53:26.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f14da640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7ec071b00 0x7fe7ec145970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:26.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f14da640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7ec071b00 0x7fe7ec145970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40610/0 (socket says 172.21.15.88:40610) 2023-10-16T20:53:26.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f14da640 1 -- 172.21.15.88:0/1557157935 learned_addr learned my addr 172.21.15.88:0/1557157935 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:26.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f0cd9640 1 --2- 172.21.15.88:0/1557157935 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7ec071ee0 0x7fe7ec1f54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:26.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f1cdb640 1 --2- 172.21.15.88:0/1557157935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7ec0c7ce0 0x7fe7ec1f5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:26.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f14da640 1 -- 172.21.15.88:0/1557157935 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7ec071ee0 msgr2=0x7fe7ec1f54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:26.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f14da640 1 --2- 172.21.15.88:0/1557157935 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7ec071ee0 0x7fe7ec1f54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:26.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f14da640 1 -- 172.21.15.88:0/1557157935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7ec0c7ce0 msgr2=0x7fe7ec1f5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:26.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f14da640 1 --2- 172.21.15.88:0/1557157935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7ec0c7ce0 0x7fe7ec1f5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:26.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f14da640 1 -- 172.21.15.88:0/1557157935 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e8009420 con 0x7fe7ec071b00 2023-10-16T20:53:26.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7f14da640 1 --2- 172.21.15.88:0/1557157935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7ec071b00 0x7fe7ec145970 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800ebe0 tx=0x7fe7e800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:26.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.721+0000 7fe7e1ffb640 1 -- 172.21.15.88:0/1557157935 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e800eee0 con 0x7fe7ec071b00 2023-10-16T20:53:26.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.725+0000 7fe7e1ffb640 1 -- 172.21.15.88:0/1557157935 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e800cc30 con 0x7fe7ec071b00 2023-10-16T20:53:26.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.725+0000 7fe7f373e640 1 -- 172.21.15.88:0/1557157935 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7dc0094d0 con 0x7fe7ec071b00 2023-10-16T20:53:26.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.725+0000 7fe7e1ffb640 1 -- 172.21.15.88:0/1557157935 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8013470 con 0x7fe7ec071b00 2023-10-16T20:53:26.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.725+0000 7fe7f373e640 1 -- 172.21.15.88:0/1557157935 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7ec070900 con 0x7fe7ec071b00 2023-10-16T20:53:26.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.725+0000 7fe7f373e640 1 -- 172.21.15.88:0/1557157935 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ec074700 con 0x7fe7ec071b00 2023-10-16T20:53:26.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.725+0000 7fe7e1ffb640 1 -- 172.21.15.88:0/1557157935 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe7e80135d0 con 0x7fe7ec071b00 2023-10-16T20:53:26.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.725+0000 7fe7e1ffb640 1 --2- 172.21.15.88:0/1557157935 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe7d00664d0 0x7fe7d0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:26.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.725+0000 7fe7e1ffb640 1 -- 172.21.15.88:0/1557157935 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7fe7e80847f0 con 0x7fe7ec071b00 2023-10-16T20:53:26.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.725+0000 7fe7f0cd9640 1 --2- 172.21.15.88:0/1557157935 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe7d00664d0 0x7fe7d0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:26.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.729+0000 7fe7f0cd9640 1 --2- 172.21.15.88:0/1557157935 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe7d00664d0 0x7fe7d0068990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0ae000 tx=0x7fe7dc0b5040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:26.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.729+0000 7fe7e1ffb640 1 -- 172.21.15.88:0/1557157935 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe7e80515d0 con 0x7fe7ec071b00 2023-10-16T20:53:26.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.957+0000 7fe7f373e640 1 -- 172.21.15.88:0/1557157935 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7ec06d960 con 0x7fe7ec071b00 2023-10-16T20:53:26.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.957+0000 7fe7e1ffb640 1 -- 172.21.15.88:0/1557157935 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+5777 (secure 0 0 0) 0x7fe7e8054bf0 con 0x7fe7ec071b00 2023-10-16T20:53:26.961 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:26.961 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:53:25.553651+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:26.962 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 89 2023-10-16T20:53:26.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.961+0000 7fe7f373e640 1 -- 172.21.15.88:0/1557157935 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe7d00664d0 msgr2=0x7fe7d0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:26.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.961+0000 7fe7f373e640 1 --2- 172.21.15.88:0/1557157935 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe7d00664d0 0x7fe7d0068990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0ae000 tx=0x7fe7dc0b5040 comp rx=0 tx=0).stop 2023-10-16T20:53:26.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.961+0000 7fe7f373e640 1 -- 172.21.15.88:0/1557157935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7ec071b00 msgr2=0x7fe7ec145970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:26.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.961+0000 7fe7f373e640 1 --2- 172.21.15.88:0/1557157935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7ec071b00 0x7fe7ec145970 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800ebe0 tx=0x7fe7e800c520 comp rx=0 tx=0).stop 2023-10-16T20:53:26.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.961+0000 7fe7f373e640 1 -- 172.21.15.88:0/1557157935 shutdown_connections 2023-10-16T20:53:26.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.961+0000 7fe7f373e640 1 --2- 172.21.15.88:0/1557157935 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe7d00664d0 0x7fe7d0068990 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:26.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.961+0000 7fe7f373e640 1 --2- 172.21.15.88:0/1557157935 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7ec0c7ce0 0x7fe7ec1f5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:26.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.961+0000 7fe7f373e640 1 --2- 172.21.15.88:0/1557157935 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe7ec071ee0 0x7fe7ec1f54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:26.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.961+0000 7fe7f373e640 1 --2- 172.21.15.88:0/1557157935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7ec071b00 0x7fe7ec145970 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:26.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.961+0000 7fe7f373e640 1 -- 172.21.15.88:0/1557157935 >> 172.21.15.88:0/1557157935 conn(0x7fe7ec06d450 msgr2=0x7fe7ec0c2610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:26.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.961+0000 7fe7f373e640 1 -- 172.21.15.88:0/1557157935 shutdown_connections 2023-10-16T20:53:26.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:26.961+0000 7fe7f373e640 1 -- 172.21.15.88:0/1557157935 wait complete. 2023-10-16T20:53:26.982 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-10-16T20:53:26.983 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-16T20:53:26.983 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.157+0000 7f2f9433e640 1 Processor -- start 2023-10-16T20:53:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.157+0000 7f2f9433e640 1 -- start start 2023-10-16T20:53:27.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.157+0000 7f2f9433e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2f8c06a970 0x7f2f8c071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:27.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.157+0000 7f2f9433e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f8c071e60 0x7f2f8c15a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:27.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.157+0000 7f2f9433e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c15b3c0 con 0x7f2f8c1406d0 2023-10-16T20:53:27.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.157+0000 7f2f9433e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f8c0c2460 con 0x7f2f8c06a970 2023-10-16T20:53:27.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.157+0000 7f2f9433e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c15b680 con 0x7f2f8c071e60 2023-10-16T20:53:27.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.157+0000 7f2f918d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2f8c06a970 0x7f2f8c071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:27.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.157+0000 7f2f918d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2f8c06a970 0x7f2f8c071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40620/0 (socket says 172.21.15.88:40620) 2023-10-16T20:53:27.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.157+0000 7f2f918d9640 1 -- 172.21.15.88:0/198816638 learned_addr learned my addr 172.21.15.88:0/198816638 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.157+0000 7f2f918d9640 1 -- 172.21.15.88:0/198816638 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f8c071e60 msgr2=0x7f2f8c15a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.157+0000 7f2f918d9640 1 --2- 172.21.15.88:0/198816638 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f8c071e60 0x7f2f8c15a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.157+0000 7f2f918d9640 1 -- 172.21.15.88:0/198816638 >> v1:172.21.15.191:6789/0 conn(0x7f2f8c1406d0 legacy=0x7f2f8c0c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.157+0000 7f2f918d9640 1 -- 172.21.15.88:0/198816638 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8c157570 con 0x7f2f8c06a970 2023-10-16T20:53:27.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.161+0000 7f2f918d9640 1 --2- 172.21.15.88:0/198816638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f8c06a970 0x7f2f8c071920 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c009870 tx=0x7f2f7c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d74c44d4434c6c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:27.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.161+0000 7f2f910d8640 1 -- 172.21.15.88:0/198816638 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c0ab600 con 0x7f2f8c06a970 2023-10-16T20:53:27.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.161+0000 7f2f910d8640 1 -- 172.21.15.88:0/198816638 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f7c0a7050 con 0x7f2f8c06a970 2023-10-16T20:53:27.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.161+0000 7f2f910d8640 1 -- 172.21.15.88:0/198816638 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c0ab980 con 0x7f2f8c06a970 2023-10-16T20:53:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.161+0000 7f2f9433e640 1 -- 172.21.15.88:0/198816638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f8c06a970 msgr2=0x7f2f8c071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.161+0000 7f2f9433e640 1 --2- 172.21.15.88:0/198816638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f8c06a970 0x7f2f8c071920 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c009870 tx=0x7f2f7c01c500 comp rx=0 tx=0).stop 2023-10-16T20:53:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.161+0000 7f2f9433e640 1 -- 172.21.15.88:0/198816638 shutdown_connections 2023-10-16T20:53:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.161+0000 7f2f9433e640 1 --2- 172.21.15.88:0/198816638 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f8c071e60 0x7f2f8c15a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.161+0000 7f2f9433e640 1 --2- 172.21.15.88:0/198816638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f8c06a970 0x7f2f8c071920 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:27.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.161+0000 7f2f9433e640 1 -- 172.21.15.88:0/198816638 >> 172.21.15.88:0/198816638 conn(0x7f2f8c0cad50 msgr2=0x7f2f8c0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:27.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.161+0000 7f2f9433e640 1 -- 172.21.15.88:0/198816638 shutdown_connections 2023-10-16T20:53:27.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.161+0000 7f2f9433e640 1 -- 172.21.15.88:0/198816638 wait complete. 2023-10-16T20:53:27.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f9433e640 1 Processor -- start 2023-10-16T20:53:27.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f9433e640 1 -- start start 2023-10-16T20:53:27.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f9433e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f8c06a970 0x7f2f8c153ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:27.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f9433e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f8c071e60 0x7f2f8c1524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f9433e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f8c1406d0 0x7f2f8c1f0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f9433e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c152a30 con 0x7f2f8c1406d0 2023-10-16T20:53:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f9433e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c154580 con 0x7f2f8c06a970 2023-10-16T20:53:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f9433e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c154860 con 0x7f2f8c071e60 2023-10-16T20:53:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f928db640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f8c1406d0 0x7f2f8c1f0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:27.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f928db640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f8c1406d0 0x7f2f8c1f0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40628/0 (socket says 172.21.15.88:40628) 2023-10-16T20:53:27.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f928db640 1 -- 172.21.15.88:0/1398361759 learned_addr learned my addr 172.21.15.88:0/1398361759 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:27.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f918d9640 1 --2- 172.21.15.88:0/1398361759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f8c071e60 0x7f2f8c1524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:27.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f920da640 1 --2- 172.21.15.88:0/1398361759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f8c06a970 0x7f2f8c153ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:27.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f928db640 1 -- 172.21.15.88:0/1398361759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f8c071e60 msgr2=0x7f2f8c1524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:27.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f928db640 1 --2- 172.21.15.88:0/1398361759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f8c071e60 0x7f2f8c1524f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:27.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f928db640 1 -- 172.21.15.88:0/1398361759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f8c06a970 msgr2=0x7f2f8c153ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:27.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f928db640 1 --2- 172.21.15.88:0/1398361759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f8c06a970 0x7f2f8c153ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:27.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f928db640 1 -- 172.21.15.88:0/1398361759 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7c0094d0 con 0x7f2f8c1406d0 2023-10-16T20:53:27.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f920da640 1 --2- 172.21.15.88:0/1398361759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f8c06a970 0x7f2f8c153ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:53:27.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f928db640 1 --2- 172.21.15.88:0/1398361759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f8c1406d0 0x7f2f8c1f0140 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f2f8400b970 tx=0x7f2f8400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:27.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f82ffd640 1 -- 172.21.15.88:0/1398361759 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8400cb40 con 0x7f2f8c1406d0 2023-10-16T20:53:27.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f82ffd640 1 -- 172.21.15.88:0/1398361759 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8400ccc0 con 0x7f2f8c1406d0 2023-10-16T20:53:27.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f82ffd640 1 -- 172.21.15.88:0/1398361759 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f84014680 con 0x7f2f8c1406d0 2023-10-16T20:53:27.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f9433e640 1 -- 172.21.15.88:0/1398361759 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8c1f3850 con 0x7f2f8c1406d0 2023-10-16T20:53:27.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.165+0000 7f2f9433e640 1 -- 172.21.15.88:0/1398361759 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8c1f3e10 con 0x7f2f8c1406d0 2023-10-16T20:53:27.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.169+0000 7f2f9433e640 1 -- 172.21.15.88:0/1398361759 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8c1f36b0 con 0x7f2f8c1406d0 2023-10-16T20:53:27.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.169+0000 7f2f82ffd640 1 -- 172.21.15.88:0/1398361759 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2f8401d3b0 con 0x7f2f8c1406d0 2023-10-16T20:53:27.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.169+0000 7f2f82ffd640 1 --2- 172.21.15.88:0/1398361759 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2f700664a0 0x7f2f70068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:27.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.169+0000 7f2f82ffd640 1 -- 172.21.15.88:0/1398361759 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f2f84084400 con 0x7f2f8c1406d0 2023-10-16T20:53:27.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.169+0000 7f2f920da640 1 --2- 172.21.15.88:0/1398361759 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2f700664a0 0x7f2f70068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:27.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.173+0000 7f2f920da640 1 --2- 172.21.15.88:0/1398361759 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2f700664a0 0x7f2f70068960 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2f88001760 tx=0x7f2f88009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:27.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.173+0000 7f2f82ffd640 1 -- 172.21.15.88:0/1398361759 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2f8c1f36b0 con 0x7f2f8c1406d0 2023-10-16T20:53:27.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.405+0000 7f2f9433e640 1 -- 172.21.15.88:0/1398361759 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f8c1f36b0 con 0x7f2f8c1406d0 2023-10-16T20:53:27.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.405+0000 7f2f82ffd640 1 -- 172.21.15.88:0/1398361759 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f2f8c1f36b0 con 0x7f2f8c1406d0 2023-10-16T20:53:27.407 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:27.407 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:53:25.553651+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:27.408 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 89 2023-10-16T20:53:27.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.405+0000 7f2f9433e640 1 -- 172.21.15.88:0/1398361759 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2f700664a0 msgr2=0x7f2f70068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:27.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.405+0000 7f2f9433e640 1 --2- 172.21.15.88:0/1398361759 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2f700664a0 0x7f2f70068960 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2f88001760 tx=0x7f2f88009040 comp rx=0 tx=0).stop 2023-10-16T20:53:27.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.405+0000 7f2f9433e640 1 -- 172.21.15.88:0/1398361759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f8c1406d0 msgr2=0x7f2f8c1f0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:27.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.405+0000 7f2f9433e640 1 --2- 172.21.15.88:0/1398361759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f8c1406d0 0x7f2f8c1f0140 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f2f8400b970 tx=0x7f2f8400be30 comp rx=0 tx=0).stop 2023-10-16T20:53:27.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.405+0000 7f2f9433e640 1 -- 172.21.15.88:0/1398361759 shutdown_connections 2023-10-16T20:53:27.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.405+0000 7f2f9433e640 1 --2- 172.21.15.88:0/1398361759 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f2f700664a0 0x7f2f70068960 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:27.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.405+0000 7f2f9433e640 1 --2- 172.21.15.88:0/1398361759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f8c1406d0 0x7f2f8c1f0140 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:27.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.405+0000 7f2f9433e640 1 --2- 172.21.15.88:0/1398361759 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f2f8c071e60 0x7f2f8c1524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:27.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.405+0000 7f2f9433e640 1 --2- 172.21.15.88:0/1398361759 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2f8c06a970 0x7f2f8c153ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:27.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.405+0000 7f2f9433e640 1 -- 172.21.15.88:0/1398361759 >> 172.21.15.88:0/1398361759 conn(0x7f2f8c0cad50 msgr2=0x7f2f8c155090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:27.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.409+0000 7f2f9433e640 1 -- 172.21.15.88:0/1398361759 shutdown_connections 2023-10-16T20:53:27.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:27.409+0000 7f2f9433e640 1 -- 172.21.15.88:0/1398361759 wait complete. 2023-10-16T20:53:27.431 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 89, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 89, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5282": { "addr": "172.21.15.191:6833/3633326602", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 3633326602, "type": "v2" }, { "addr": "172.21.15.191:6833", "nonce": 3633326602, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5282, "incarnation": 81, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 41 }, "gid_5284": { "addr": "172.21.15.88:6839/2172973775", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6837", "nonce": 2172973775, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2172973775, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5284, "incarnation": 38, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 17 }, "gid_5378": { "addr": "172.21.15.191:6835/3119045295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3119045295, "type": "v2" }, { "addr": "172.21.15.191:6835", "nonce": 3119045295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5378, "incarnation": 89, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:replay", "state_seq": 1 }, "gid_5535": { "addr": "172.21.15.88:6835/3476086754", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3476086754, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3476086754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5535, "incarnation": 52, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_5709": { "addr": "172.21.15.88:6838/2600891060", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 2600891060, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 2600891060, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5709, "incarnation": 67, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:53:25.553651+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5282, "mds_1": 5709, "mds_2": 5535, "mds_3": 5284, "mds_4": 5378 } } } ], "standbys": [] } 2023-10-16T20:53:29.432 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:29.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.597+0000 7fc908a17640 1 Processor -- start 2023-10-16T20:53:29.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.597+0000 7fc908a17640 1 -- start start 2023-10-16T20:53:29.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.597+0000 7fc908a17640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc9040cb110 0x7fc90406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:29.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.597+0000 7fc908a17640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc904069e90 con 0x7fc9040cac50 2023-10-16T20:53:29.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.597+0000 7fc908a17640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9040ca680 con 0x7fc9041538d0 2023-10-16T20:53:29.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.597+0000 7fc908a17640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9040ca470 con 0x7fc9040cb110 2023-10-16T20:53:29.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.597+0000 7fc90259c640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fc9041538d0 0x7fc9040cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:39972/0 (socket says 172.21.15.88:39972) 2023-10-16T20:53:29.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.597+0000 7fc90259c640 1 -- 172.21.15.88:0/2400966875 learned_addr learned my addr 172.21.15.88:0/2400966875 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:53:29.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.597+0000 7fc902d9d640 1 --2- 172.21.15.88:0/2400966875 >> v2:172.21.15.88:3300/0 conn(0x7fc9040cb110 0x7fc90406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:29.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.601+0000 7fc90159a640 1 -- 172.21.15.88:0/2400966875 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551818565 0 0) 0x7fc9040ca680 con 0x7fc9041538d0 2023-10-16T20:53:29.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.601+0000 7fc90159a640 1 -- 172.21.15.88:0/2400966875 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8ec003530 con 0x7fc9041538d0 2023-10-16T20:53:29.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.601+0000 7fc902d9d640 1 -- 172.21.15.88:0/2400966875 >> v1:172.21.15.191:6789/0 conn(0x7fc9041538d0 legacy=0x7fc9040cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:29.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.601+0000 7fc902d9d640 1 -- 172.21.15.88:0/2400966875 >> v1:172.21.15.88:6789/0 conn(0x7fc9040cac50 legacy=0x7fc90406a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:29.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.601+0000 7fc902d9d640 1 -- 172.21.15.88:0/2400966875 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9040ca960 con 0x7fc9040cb110 2023-10-16T20:53:29.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.601+0000 7fc902d9d640 1 --2- 172.21.15.88:0/2400966875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9040cb110 0x7fc90406c730 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc009d90 tx=0x7fc8fc01c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd7bf02b31f9455 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:29.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.605+0000 7fc90159a640 1 -- 172.21.15.88:0/2400966875 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc0a5950 con 0x7fc9040cb110 2023-10-16T20:53:29.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.605+0000 7fc90159a640 1 -- 172.21.15.88:0/2400966875 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8fc0a5ab0 con 0x7fc9040cb110 2023-10-16T20:53:29.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.605+0000 7fc90159a640 1 -- 172.21.15.88:0/2400966875 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc0a5e20 con 0x7fc9040cb110 2023-10-16T20:53:29.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.605+0000 7fc908a17640 1 -- 172.21.15.88:0/2400966875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9040cb110 msgr2=0x7fc90406c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:29.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.605+0000 7fc908a17640 1 --2- 172.21.15.88:0/2400966875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9040cb110 0x7fc90406c730 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc009d90 tx=0x7fc8fc01c890 comp rx=0 tx=0).stop 2023-10-16T20:53:29.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.605+0000 7fc908a17640 1 -- 172.21.15.88:0/2400966875 shutdown_connections 2023-10-16T20:53:29.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.605+0000 7fc908a17640 1 --2- 172.21.15.88:0/2400966875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9040cb110 0x7fc90406c730 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:29.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.605+0000 7fc908a17640 1 -- 172.21.15.88:0/2400966875 >> 172.21.15.88:0/2400966875 conn(0x7fc904071cb0 msgr2=0x7fc90406bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:29.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.605+0000 7fc908a17640 1 -- 172.21.15.88:0/2400966875 shutdown_connections 2023-10-16T20:53:29.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.605+0000 7fc908a17640 1 -- 172.21.15.88:0/2400966875 wait complete. 2023-10-16T20:53:29.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.605+0000 7fc908a17640 1 Processor -- start 2023-10-16T20:53:29.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.605+0000 7fc908a17640 1 -- start start 2023-10-16T20:53:29.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc908a17640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9040cac50 0x7fc9041439a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:29.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc908a17640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9040cb110 0x7fc9040b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:29.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc908a17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9041538d0 0x7fc9040c6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:29.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc908a17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9041f5d80 con 0x7fc9041538d0 2023-10-16T20:53:29.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc908a17640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9040c60e0 con 0x7fc9040cb110 2023-10-16T20:53:29.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc908a17640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9041f57e0 con 0x7fc9040cac50 2023-10-16T20:53:29.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc902d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9041538d0 0x7fc9040c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:29.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc902d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9041538d0 0x7fc9040c6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32982/0 (socket says 172.21.15.88:32982) 2023-10-16T20:53:29.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc902d9d640 1 -- 172.21.15.88:0/1401805231 learned_addr learned my addr 172.21.15.88:0/1401805231 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:29.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc90259c640 1 --2- 172.21.15.88:0/1401805231 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9040cac50 0x7fc9041439a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:29.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc901d9b640 1 --2- 172.21.15.88:0/1401805231 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9040cb110 0x7fc9040b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:29.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc902d9d640 1 -- 172.21.15.88:0/1401805231 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9040cac50 msgr2=0x7fc9041439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:29.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc902d9d640 1 --2- 172.21.15.88:0/1401805231 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9040cac50 0x7fc9041439a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:29.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc902d9d640 1 -- 172.21.15.88:0/1401805231 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9040cb110 msgr2=0x7fc9040b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:29.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc902d9d640 1 --2- 172.21.15.88:0/1401805231 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9040cb110 0x7fc9040b35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:29.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc902d9d640 1 -- 172.21.15.88:0/1401805231 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8fc009a70 con 0x7fc9041538d0 2023-10-16T20:53:29.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc902d9d640 1 --2- 172.21.15.88:0/1401805231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9041538d0 0x7fc9040c6570 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc01cd70 tx=0x7fc8fc019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:29.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc8f2ffd640 1 -- 172.21.15.88:0/1401805231 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc003db0 con 0x7fc9041538d0 2023-10-16T20:53:29.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc8f2ffd640 1 -- 172.21.15.88:0/1401805231 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8fc004040 con 0x7fc9041538d0 2023-10-16T20:53:29.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc908a17640 1 -- 172.21.15.88:0/1401805231 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9041f5aa0 con 0x7fc9041538d0 2023-10-16T20:53:29.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc8f2ffd640 1 -- 172.21.15.88:0/1401805231 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc0043f0 con 0x7fc9041538d0 2023-10-16T20:53:29.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc908a17640 1 -- 172.21.15.88:0/1401805231 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9040c8b00 con 0x7fc9041538d0 2023-10-16T20:53:29.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.609+0000 7fc908a17640 1 -- 172.21.15.88:0/1401805231 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc904074700 con 0x7fc9041538d0 2023-10-16T20:53:29.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.613+0000 7fc8f2ffd640 1 -- 172.21.15.88:0/1401805231 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc8fc0b4070 con 0x7fc9041538d0 2023-10-16T20:53:29.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.613+0000 7fc8f2ffd640 1 --2- 172.21.15.88:0/1401805231 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc8e40664d0 0x7fc8e4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:29.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.613+0000 7fc8f2ffd640 1 -- 172.21.15.88:0/1401805231 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7fc8fc11f340 con 0x7fc9041538d0 2023-10-16T20:53:29.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.613+0000 7fc90259c640 1 --2- 172.21.15.88:0/1401805231 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc8e40664d0 0x7fc8e4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:29.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.613+0000 7fc90259c640 1 --2- 172.21.15.88:0/1401805231 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc8e40664d0 0x7fc8e4068990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc8f40011f0 tx=0x7fc8f4007d40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:29.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.617+0000 7fc8f2ffd640 1 -- 172.21.15.88:0/1401805231 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc8fc0ec360 con 0x7fc9041538d0 2023-10-16T20:53:29.711 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:53:29.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:53:29.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.825+0000 7fc908a17640 1 -- 172.21.15.88:0/1401805231 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9040c6220 con 0x7fc9041538d0 2023-10-16T20:53:29.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.825+0000 7fc8f2ffd640 1 -- 172.21.15.88:0/1401805231 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+5777 (secure 0 0 0) 0x7fc8fc0ef980 con 0x7fc9041538d0 2023-10-16T20:53:29.828 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:29.828 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:53:25.553651+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:29.829 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 89 2023-10-16T20:53:29.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.829+0000 7fc908a17640 1 -- 172.21.15.88:0/1401805231 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc8e40664d0 msgr2=0x7fc8e4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:29.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.829+0000 7fc908a17640 1 --2- 172.21.15.88:0/1401805231 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc8e40664d0 0x7fc8e4068990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc8f40011f0 tx=0x7fc8f4007d40 comp rx=0 tx=0).stop 2023-10-16T20:53:29.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.829+0000 7fc908a17640 1 -- 172.21.15.88:0/1401805231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9041538d0 msgr2=0x7fc9040c6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:29.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.829+0000 7fc908a17640 1 --2- 172.21.15.88:0/1401805231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9041538d0 0x7fc9040c6570 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc01cd70 tx=0x7fc8fc019630 comp rx=0 tx=0).stop 2023-10-16T20:53:29.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.829+0000 7fc908a17640 1 -- 172.21.15.88:0/1401805231 shutdown_connections 2023-10-16T20:53:29.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.829+0000 7fc908a17640 1 --2- 172.21.15.88:0/1401805231 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc8e40664d0 0x7fc8e4068990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:29.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.829+0000 7fc908a17640 1 --2- 172.21.15.88:0/1401805231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9041538d0 0x7fc9040c6570 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:29.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.829+0000 7fc908a17640 1 --2- 172.21.15.88:0/1401805231 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9040cb110 0x7fc9040b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:29.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.829+0000 7fc908a17640 1 --2- 172.21.15.88:0/1401805231 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9040cac50 0x7fc9041439a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:29.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.829+0000 7fc908a17640 1 -- 172.21.15.88:0/1401805231 >> 172.21.15.88:0/1401805231 conn(0x7fc904071cb0 msgr2=0x7fc90406a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:29.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.829+0000 7fc908a17640 1 -- 172.21.15.88:0/1401805231 shutdown_connections 2023-10-16T20:53:29.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:29.829+0000 7fc908a17640 1 -- 172.21.15.88:0/1401805231 wait complete. 2023-10-16T20:53:29.852 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:53:31.853 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:32.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.017+0000 7fc159935640 1 Processor -- start 2023-10-16T20:53:32.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.017+0000 7fc159935640 1 -- start start 2023-10-16T20:53:32.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.017+0000 7fc159935640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc154153560 0x7fc1540b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:32.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.017+0000 7fc159935640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc1540c9a00 0x7fc1540ca660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:32.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.017+0000 7fc159935640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc154159f10 con 0x7fc154153940 2023-10-16T20:53:32.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.017+0000 7fc159935640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1541593f0 con 0x7fc1540c9a00 2023-10-16T20:53:32.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.017+0000 7fc159935640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc15415a3a0 con 0x7fc154153560 2023-10-16T20:53:32.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.021+0000 7fc1537fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc154153940 0x7fc15415c960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44352/0 (socket says 172.21.15.88:44352) 2023-10-16T20:53:32.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.021+0000 7fc1537fe640 1 -- 172.21.15.88:0/1877568853 learned_addr learned my addr 172.21.15.88:0/1877568853 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:53:32.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.021+0000 7fc1527fc640 1 --2- 172.21.15.88:0/1877568853 >> v2:172.21.15.191:3300/0 conn(0x7fc1540c9a00 0x7fc1540ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:32.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.021+0000 7fc152ffd640 1 --2- 172.21.15.88:0/1877568853 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc154153560 0x7fc1540b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:32.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.021+0000 7fc151ffb640 1 -- 172.21.15.88:0/1877568853 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929997522 0 0) 0x7fc154159f10 con 0x7fc154153940 2023-10-16T20:53:32.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.021+0000 7fc151ffb640 1 -- 172.21.15.88:0/1877568853 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc13c003530 con 0x7fc154153940 2023-10-16T20:53:32.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.021+0000 7fc1527fc640 1 -- 172.21.15.88:0/1877568853 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc154153560 msgr2=0x7fc1540b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:32.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.021+0000 7fc1527fc640 1 --2- 172.21.15.88:0/1877568853 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc154153560 0x7fc1540b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:32.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.021+0000 7fc1527fc640 1 -- 172.21.15.88:0/1877568853 >> v1:172.21.15.88:6789/0 conn(0x7fc154153940 legacy=0x7fc15415c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:32.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.021+0000 7fc1527fc640 1 -- 172.21.15.88:0/1877568853 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15415d610 con 0x7fc1540c9a00 2023-10-16T20:53:32.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.021+0000 7fc152ffd640 1 --2- 172.21.15.88:0/1877568853 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc154153560 0x7fc1540b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:53:32.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.025+0000 7fc1527fc640 1 --2- 172.21.15.88:0/1877568853 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1540c9a00 0x7fc1540ca660 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc1480097f0 tx=0x7fc14801c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93edfed68c35d0c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:32.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.025+0000 7fc151ffb640 1 -- 172.21.15.88:0/1877568853 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1480ab650 con 0x7fc1540c9a00 2023-10-16T20:53:32.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.025+0000 7fc151ffb640 1 -- 172.21.15.88:0/1877568853 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1480ab7b0 con 0x7fc1540c9a00 2023-10-16T20:53:32.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.025+0000 7fc151ffb640 1 -- 172.21.15.88:0/1877568853 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1480abb40 con 0x7fc1540c9a00 2023-10-16T20:53:32.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.025+0000 7fc159935640 1 -- 172.21.15.88:0/1877568853 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1540c9a00 msgr2=0x7fc1540ca660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:32.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.025+0000 7fc159935640 1 --2- 172.21.15.88:0/1877568853 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1540c9a00 0x7fc1540ca660 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc1480097f0 tx=0x7fc14801c450 comp rx=0 tx=0).stop 2023-10-16T20:53:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.025+0000 7fc159935640 1 -- 172.21.15.88:0/1877568853 shutdown_connections 2023-10-16T20:53:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.025+0000 7fc159935640 1 --2- 172.21.15.88:0/1877568853 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1540c9a00 0x7fc1540ca660 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.025+0000 7fc159935640 1 --2- 172.21.15.88:0/1877568853 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc154153560 0x7fc1540b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.025+0000 7fc159935640 1 -- 172.21.15.88:0/1877568853 >> 172.21.15.88:0/1877568853 conn(0x7fc154066080 msgr2=0x7fc1540715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.025+0000 7fc159935640 1 -- 172.21.15.88:0/1877568853 shutdown_connections 2023-10-16T20:53:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.025+0000 7fc159935640 1 -- 172.21.15.88:0/1877568853 wait complete. 2023-10-16T20:53:32.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.025+0000 7fc159935640 1 Processor -- start 2023-10-16T20:53:32.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.025+0000 7fc159935640 1 -- start start 2023-10-16T20:53:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc159935640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1540c9a00 0x7fc1540c2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc159935640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc154153560 0x7fc1540cf210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc159935640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc154153940 0x7fc154147fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc159935640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1540cd140 con 0x7fc1540c9a00 2023-10-16T20:53:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc159935640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1541ec1c0 con 0x7fc154153940 2023-10-16T20:53:32.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc159935640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1540c3580 con 0x7fc154153560 2023-10-16T20:53:32.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc152ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1540c9a00 0x7fc1540c2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:32.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc152ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1540c9a00 0x7fc1540c2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33004/0 (socket says 172.21.15.88:33004) 2023-10-16T20:53:32.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc152ffd640 1 -- 172.21.15.88:0/2436480389 learned_addr learned my addr 172.21.15.88:0/2436480389 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:32.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc1537fe640 1 --2- 172.21.15.88:0/2436480389 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc154153940 0x7fc154147fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:32.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc1527fc640 1 --2- 172.21.15.88:0/2436480389 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc154153560 0x7fc1540cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:32.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc152ffd640 1 -- 172.21.15.88:0/2436480389 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc154153560 msgr2=0x7fc1540cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:32.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc152ffd640 1 --2- 172.21.15.88:0/2436480389 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc154153560 0x7fc1540cf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:32.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc152ffd640 1 -- 172.21.15.88:0/2436480389 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc154153940 msgr2=0x7fc154147fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:32.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc152ffd640 1 --2- 172.21.15.88:0/2436480389 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc154153940 0x7fc154147fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:32.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc152ffd640 1 -- 172.21.15.88:0/2436480389 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1440093b0 con 0x7fc1540c9a00 2023-10-16T20:53:32.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc152ffd640 1 --2- 172.21.15.88:0/2436480389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1540c9a00 0x7fc1540c2ef0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fc14400ebe0 tx=0x7fc14400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:32.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc13bfff640 1 -- 172.21.15.88:0/2436480389 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc144009760 con 0x7fc1540c9a00 2023-10-16T20:53:32.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc13bfff640 1 -- 172.21.15.88:0/2436480389 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14400cc30 con 0x7fc1540c9a00 2023-10-16T20:53:32.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc159935640 1 -- 172.21.15.88:0/2436480389 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc148009420 con 0x7fc1540c9a00 2023-10-16T20:53:32.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc13bfff640 1 -- 172.21.15.88:0/2436480389 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14400eee0 con 0x7fc1540c9a00 2023-10-16T20:53:32.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc159935640 1 -- 172.21.15.88:0/2436480389 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1541eb540 con 0x7fc1540c9a00 2023-10-16T20:53:32.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.029+0000 7fc159935640 1 -- 172.21.15.88:0/2436480389 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15404e9c0 con 0x7fc1540c9a00 2023-10-16T20:53:32.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.033+0000 7fc13bfff640 1 -- 172.21.15.88:0/2436480389 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc1440059a0 con 0x7fc1540c9a00 2023-10-16T20:53:32.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.033+0000 7fc13bfff640 1 --2- 172.21.15.88:0/2436480389 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc1400664a0 0x7fc140068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:32.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.033+0000 7fc13bfff640 1 -- 172.21.15.88:0/2436480389 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7fc1440849c0 con 0x7fc1540c9a00 2023-10-16T20:53:32.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.033+0000 7fc1527fc640 1 --2- 172.21.15.88:0/2436480389 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc1400664a0 0x7fc140068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:32.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.033+0000 7fc1527fc640 1 --2- 172.21.15.88:0/2436480389 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc1400664a0 0x7fc140068960 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc1480af000 tx=0x7fc1480a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:32.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.037+0000 7fc13bfff640 1 -- 172.21.15.88:0/2436480389 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc144051800 con 0x7fc1540c9a00 2023-10-16T20:53:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.305+0000 7fc159935640 1 -- 172.21.15.88:0/2436480389 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc15414e6f0 con 0x7fc1540c9a00 2023-10-16T20:53:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.305+0000 7fc13bfff640 1 -- 172.21.15.88:0/2436480389 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5779 (secure 0 0 0) 0x7fc144020090 con 0x7fc1540c9a00 2023-10-16T20:53:32.307 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:32.307 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:53:30.839029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:resolve","state_seq":26,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 90 2023-10-16T20:53:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.305+0000 7fc159935640 1 -- 172.21.15.88:0/2436480389 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc1400664a0 msgr2=0x7fc140068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.305+0000 7fc159935640 1 --2- 172.21.15.88:0/2436480389 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc1400664a0 0x7fc140068960 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc1480af000 tx=0x7fc1480a7000 comp rx=0 tx=0).stop 2023-10-16T20:53:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.305+0000 7fc159935640 1 -- 172.21.15.88:0/2436480389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1540c9a00 msgr2=0x7fc1540c2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.305+0000 7fc159935640 1 --2- 172.21.15.88:0/2436480389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1540c9a00 0x7fc1540c2ef0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fc14400ebe0 tx=0x7fc14400c520 comp rx=0 tx=0).stop 2023-10-16T20:53:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.305+0000 7fc159935640 1 -- 172.21.15.88:0/2436480389 shutdown_connections 2023-10-16T20:53:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.305+0000 7fc159935640 1 --2- 172.21.15.88:0/2436480389 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc1400664a0 0x7fc140068960 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.305+0000 7fc159935640 1 --2- 172.21.15.88:0/2436480389 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc154153940 0x7fc154147fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.305+0000 7fc159935640 1 --2- 172.21.15.88:0/2436480389 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc154153560 0x7fc1540cf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.305+0000 7fc159935640 1 --2- 172.21.15.88:0/2436480389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1540c9a00 0x7fc1540c2ef0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.305+0000 7fc159935640 1 -- 172.21.15.88:0/2436480389 >> 172.21.15.88:0/2436480389 conn(0x7fc154066080 msgr2=0x7fc154151ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.305+0000 7fc159935640 1 -- 172.21.15.88:0/2436480389 shutdown_connections 2023-10-16T20:53:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:32.305+0000 7fc159935640 1 -- 172.21.15.88:0/2436480389 wait complete. 2023-10-16T20:53:32.331 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:53:34.332 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:34.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.489+0000 7f1f7e0d3640 1 Processor -- start 2023-10-16T20:53:34.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.489+0000 7f1f7e0d3640 1 -- start start 2023-10-16T20:53:34.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.493+0000 7f1f7e0d3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1f781537c0 0x7f1f7815c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:34.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.493+0000 7f1f7e0d3640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f780ca2f0 0x7f1f780cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:34.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.493+0000 7f1f7e0d3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f780cae10 con 0x7f1f7815cdb0 2023-10-16T20:53:34.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.493+0000 7f1f7e0d3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f780cac30 con 0x7f1f781537c0 2023-10-16T20:53:34.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.493+0000 7f1f7e0d3640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7815d2b0 con 0x7f1f780ca2f0 2023-10-16T20:53:34.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.493+0000 7f1f7d0d1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1f781537c0 0x7f1f7815c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:34.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.493+0000 7f1f7c8d0640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1f7815cdb0 0x7f1f780c4c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44362/0 (socket says 172.21.15.88:44362) 2023-10-16T20:53:34.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.493+0000 7f1f7d0d1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1f781537c0 0x7f1f7815c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33016/0 (socket says 172.21.15.88:33016) 2023-10-16T20:53:34.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.493+0000 7f1f7c8d0640 1 -- 172.21.15.88:0/2464672469 learned_addr learned my addr 172.21.15.88:0/2464672469 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:53:34.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.493+0000 7f1f7d8d2640 1 --2- 172.21.15.88:0/2464672469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f780ca2f0 0x7f1f780cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:34.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.493+0000 7f1f7d0d1640 1 -- 172.21.15.88:0/2464672469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f780ca2f0 msgr2=0x7f1f780cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:34.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.493+0000 7f1f7d0d1640 1 --2- 172.21.15.88:0/2464672469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f780ca2f0 0x7f1f780cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:34.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.493+0000 7f1f7d0d1640 1 -- 172.21.15.88:0/2464672469 >> v1:172.21.15.88:6789/0 conn(0x7f1f7815cdb0 legacy=0x7f1f780c4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:34.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.493+0000 7f1f7d0d1640 1 -- 172.21.15.88:0/2464672469 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7815d590 con 0x7f1f781537c0 2023-10-16T20:53:34.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.493+0000 7f1f6ffff640 1 -- 172.21.15.88:0/2464672469 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421158130 0 0) 0x7f1f780cae10 con 0x7f1f7815cdb0 2023-10-16T20:53:34.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.497+0000 7f1f7d0d1640 1 --2- 172.21.15.88:0/2464672469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f781537c0 0x7f1f7815c870 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f1f740097c0 tx=0x7f1f7401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c391514a649db67b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:34.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.497+0000 7f1f6ffff640 1 -- 172.21.15.88:0/2464672469 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f740ad660 con 0x7f1f781537c0 2023-10-16T20:53:34.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.497+0000 7f1f6ffff640 1 -- 172.21.15.88:0/2464672469 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f740ad7c0 con 0x7f1f781537c0 2023-10-16T20:53:34.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.497+0000 7f1f6ffff640 1 -- 172.21.15.88:0/2464672469 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f740adb50 con 0x7f1f781537c0 2023-10-16T20:53:34.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.497+0000 7f1f7e0d3640 1 -- 172.21.15.88:0/2464672469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f781537c0 msgr2=0x7f1f7815c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:34.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.497+0000 7f1f7e0d3640 1 --2- 172.21.15.88:0/2464672469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f781537c0 0x7f1f7815c870 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f1f740097c0 tx=0x7f1f7401c450 comp rx=0 tx=0).stop 2023-10-16T20:53:34.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.497+0000 7f1f7e0d3640 1 -- 172.21.15.88:0/2464672469 shutdown_connections 2023-10-16T20:53:34.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.497+0000 7f1f7e0d3640 1 --2- 172.21.15.88:0/2464672469 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f780ca2f0 0x7f1f780cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:34.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.497+0000 7f1f7e0d3640 1 --2- 172.21.15.88:0/2464672469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f781537c0 0x7f1f7815c870 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:34.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.497+0000 7f1f7e0d3640 1 -- 172.21.15.88:0/2464672469 >> 172.21.15.88:0/2464672469 conn(0x7f1f7806c170 msgr2=0x7f1f780715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:34.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.497+0000 7f1f7e0d3640 1 -- 172.21.15.88:0/2464672469 shutdown_connections 2023-10-16T20:53:34.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.497+0000 7f1f7e0d3640 1 -- 172.21.15.88:0/2464672469 wait complete. 2023-10-16T20:53:34.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.497+0000 7f1f7e0d3640 1 Processor -- start 2023-10-16T20:53:34.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.497+0000 7f1f7e0d3640 1 -- start start 2023-10-16T20:53:34.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7e0d3640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f780ca2f0 0x7f1f7814df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:34.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7e0d3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f781537c0 0x7f1f78154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:34.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7e0d3640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f7815cdb0 0x7f1f781552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:34.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7e0d3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f78156ff0 con 0x7f1f781537c0 2023-10-16T20:53:34.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7e0d3640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f781f4b20 con 0x7f1f780ca2f0 2023-10-16T20:53:34.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7e0d3640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f78153d10 con 0x7f1f7815cdb0 2023-10-16T20:53:34.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7c8d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f781537c0 0x7f1f78154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:34.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7c8d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f781537c0 0x7f1f78154db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33020/0 (socket says 172.21.15.88:33020) 2023-10-16T20:53:34.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7c8d0640 1 -- 172.21.15.88:0/3099788984 learned_addr learned my addr 172.21.15.88:0/3099788984 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:34.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7d0d1640 1 --2- 172.21.15.88:0/3099788984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f780ca2f0 0x7f1f7814df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:34.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7d8d2640 1 --2- 172.21.15.88:0/3099788984 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f7815cdb0 0x7f1f781552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:34.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7c8d0640 1 -- 172.21.15.88:0/3099788984 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f7815cdb0 msgr2=0x7f1f781552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:34.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7c8d0640 1 --2- 172.21.15.88:0/3099788984 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f7815cdb0 0x7f1f781552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:34.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7c8d0640 1 -- 172.21.15.88:0/3099788984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f780ca2f0 msgr2=0x7f1f7814df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:34.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7c8d0640 1 --2- 172.21.15.88:0/3099788984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f780ca2f0 0x7f1f7814df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:34.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7c8d0640 1 -- 172.21.15.88:0/3099788984 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f74009420 con 0x7f1f781537c0 2023-10-16T20:53:34.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7c8d0640 1 --2- 172.21.15.88:0/3099788984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f781537c0 0x7f1f78154db0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800c480 tx=0x7f1f6800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:34.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f6dffb640 1 -- 172.21.15.88:0/3099788984 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f680074e0 con 0x7f1f781537c0 2023-10-16T20:53:34.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f6dffb640 1 -- 172.21.15.88:0/3099788984 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f68007640 con 0x7f1f781537c0 2023-10-16T20:53:34.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7e0d3640 1 -- 172.21.15.88:0/3099788984 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f78153ff0 con 0x7f1f781537c0 2023-10-16T20:53:34.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f6dffb640 1 -- 172.21.15.88:0/3099788984 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f680049f0 con 0x7f1f781537c0 2023-10-16T20:53:34.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7e0d3640 1 -- 172.21.15.88:0/3099788984 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f781545b0 con 0x7f1f781537c0 2023-10-16T20:53:34.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.501+0000 7f1f7e0d3640 1 -- 172.21.15.88:0/3099788984 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7804e9c0 con 0x7f1f781537c0 2023-10-16T20:53:34.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.505+0000 7f1f6dffb640 1 -- 172.21.15.88:0/3099788984 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1f68004b50 con 0x7f1f781537c0 2023-10-16T20:53:34.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.505+0000 7f1f6dffb640 1 --2- 172.21.15.88:0/3099788984 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1f5c0664d0 0x7f1f5c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:34.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.505+0000 7f1f6dffb640 1 -- 172.21.15.88:0/3099788984 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f1f68083a80 con 0x7f1f781537c0 2023-10-16T20:53:34.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.505+0000 7f1f7d0d1640 1 --2- 172.21.15.88:0/3099788984 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1f5c0664d0 0x7f1f5c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:34.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.505+0000 7f1f7d0d1640 1 --2- 172.21.15.88:0/3099788984 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1f5c0664d0 0x7f1f5c068990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1f740af000 tx=0x7f1f740a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:34.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.509+0000 7f1f6dffb640 1 -- 172.21.15.88:0/3099788984 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1f680508d0 con 0x7f1f781537c0 2023-10-16T20:53:34.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.733+0000 7f1f7e0d3640 1 -- 172.21.15.88:0/3099788984 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f78071750 con 0x7f1f781537c0 2023-10-16T20:53:34.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.733+0000 7f1f6dffb640 1 -- 172.21.15.88:0/3099788984 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5779 (secure 0 0 0) 0x7f1f68018090 con 0x7f1f781537c0 2023-10-16T20:53:34.735 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:34.736 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:53:30.839029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:resolve","state_seq":26,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:34.736 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 90 2023-10-16T20:53:34.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.733+0000 7f1f7e0d3640 1 -- 172.21.15.88:0/3099788984 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1f5c0664d0 msgr2=0x7f1f5c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:34.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.733+0000 7f1f7e0d3640 1 --2- 172.21.15.88:0/3099788984 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1f5c0664d0 0x7f1f5c068990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1f740af000 tx=0x7f1f740a7000 comp rx=0 tx=0).stop 2023-10-16T20:53:34.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.733+0000 7f1f7e0d3640 1 -- 172.21.15.88:0/3099788984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f781537c0 msgr2=0x7f1f78154db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:34.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.733+0000 7f1f7e0d3640 1 --2- 172.21.15.88:0/3099788984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f781537c0 0x7f1f78154db0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1f6800c480 tx=0x7f1f6800c940 comp rx=0 tx=0).stop 2023-10-16T20:53:34.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.733+0000 7f1f7e0d3640 1 -- 172.21.15.88:0/3099788984 shutdown_connections 2023-10-16T20:53:34.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.733+0000 7f1f7e0d3640 1 --2- 172.21.15.88:0/3099788984 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1f5c0664d0 0x7f1f5c068990 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:34.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.733+0000 7f1f7e0d3640 1 --2- 172.21.15.88:0/3099788984 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1f7815cdb0 0x7f1f781552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:34.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.733+0000 7f1f7e0d3640 1 --2- 172.21.15.88:0/3099788984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f781537c0 0x7f1f78154db0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:34.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.733+0000 7f1f7e0d3640 1 --2- 172.21.15.88:0/3099788984 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1f780ca2f0 0x7f1f7814df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:34.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.733+0000 7f1f7e0d3640 1 -- 172.21.15.88:0/3099788984 >> 172.21.15.88:0/3099788984 conn(0x7f1f7806c170 msgr2=0x7f1f780cd850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:34.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.733+0000 7f1f7e0d3640 1 -- 172.21.15.88:0/3099788984 shutdown_connections 2023-10-16T20:53:34.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:34.733+0000 7f1f7e0d3640 1 -- 172.21.15.88:0/3099788984 wait complete. 2023-10-16T20:53:34.759 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:53:36.760 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:36.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.913+0000 7f410d6bb640 1 Processor -- start 2023-10-16T20:53:36.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.913+0000 7f410d6bb640 1 -- start start 2023-10-16T20:53:36.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.913+0000 7f410d6bb640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f41080c9380 0x7f410815b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:36.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.913+0000 7f410d6bb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f41080c67b0 0x7f4108150fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:36.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.913+0000 7f410d6bb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410815abc0 con 0x7f41080c63d0 2023-10-16T20:53:36.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.913+0000 7f410d6bb640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41081578b0 con 0x7f41080c67b0 2023-10-16T20:53:36.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.913+0000 7f410d6bb640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4108157b90 con 0x7f41080c9380 2023-10-16T20:53:36.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.913+0000 7f410ceba640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f41080c67b0 0x7f4108150fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:36.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.913+0000 7f410ceba640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f41080c67b0 0x7f4108150fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33026/0 (socket says 172.21.15.88:33026) 2023-10-16T20:53:36.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.913+0000 7f410ceba640 1 -- 172.21.15.88:0/2725868935 learned_addr learned my addr 172.21.15.88:0/2725868935 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:36.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.913+0000 7f41077fe640 1 --2- 172.21.15.88:0/2725868935 >> v2:172.21.15.191:3300/0 conn(0x7f41080c9380 0x7f410815b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:36.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.913+0000 7f410ceba640 1 -- 172.21.15.88:0/2725868935 >> v2:172.21.15.191:3300/0 conn(0x7f41080c9380 msgr2=0x7f410815b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:36.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.913+0000 7f410ceba640 1 --2- 172.21.15.88:0/2725868935 >> v2:172.21.15.191:3300/0 conn(0x7f41080c9380 0x7f410815b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:36.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.913+0000 7f410ceba640 1 -- 172.21.15.88:0/2725868935 >> v1:172.21.15.88:6789/0 conn(0x7f41080c63d0 legacy=0x7f41081570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:36.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.913+0000 7f410ceba640 1 -- 172.21.15.88:0/2725868935 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4108157e70 con 0x7f41080c67b0 2023-10-16T20:53:36.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.917+0000 7f410ceba640 1 --2- 172.21.15.88:0/2725868935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41080c67b0 0x7f4108150fc0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f4108071700 tx=0x7f410001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95502d9db0876e60 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:36.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.917+0000 7f4106ffd640 1 -- 172.21.15.88:0/2725868935 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41000af400 con 0x7f41080c67b0 2023-10-16T20:53:36.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.917+0000 7f4106ffd640 1 -- 172.21.15.88:0/2725868935 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41000ab050 con 0x7f41080c67b0 2023-10-16T20:53:36.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.917+0000 7f4106ffd640 1 -- 172.21.15.88:0/2725868935 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41000af780 con 0x7f41080c67b0 2023-10-16T20:53:36.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.917+0000 7f410d6bb640 1 -- 172.21.15.88:0/2725868935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41080c67b0 msgr2=0x7f4108150fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:36.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.917+0000 7f410d6bb640 1 --2- 172.21.15.88:0/2725868935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41080c67b0 0x7f4108150fc0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f4108071700 tx=0x7f410001f1c0 comp rx=0 tx=0).stop 2023-10-16T20:53:36.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f410d6bb640 1 -- 172.21.15.88:0/2725868935 shutdown_connections 2023-10-16T20:53:36.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f410d6bb640 1 --2- 172.21.15.88:0/2725868935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41080c67b0 0x7f4108150fc0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:36.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f410d6bb640 1 --2- 172.21.15.88:0/2725868935 >> v2:172.21.15.191:3300/0 conn(0x7f41080c9380 0x7f410815b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:36.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f410d6bb640 1 -- 172.21.15.88:0/2725868935 >> 172.21.15.88:0/2725868935 conn(0x7f4108065a80 msgr2=0x7f4108065e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:36.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f410d6bb640 1 -- 172.21.15.88:0/2725868935 shutdown_connections 2023-10-16T20:53:36.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f410d6bb640 1 -- 172.21.15.88:0/2725868935 wait complete. 2023-10-16T20:53:36.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f410d6bb640 1 Processor -- start 2023-10-16T20:53:36.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f410d6bb640 1 -- start start 2023-10-16T20:53:36.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f410d6bb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41080c63d0 0x7f4108150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:36.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f410d6bb640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41080c67b0 0x7f410814f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:36.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f410d6bb640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41080c9380 0x7f410814f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:36.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f410d6bb640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41081ef7c0 con 0x7f41080c63d0 2023-10-16T20:53:36.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f410d6bb640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41081ee960 con 0x7f41080c9380 2023-10-16T20:53:36.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f410d6bb640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41081eec40 con 0x7f41080c67b0 2023-10-16T20:53:36.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f4107fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41080c63d0 0x7f4108150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:36.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f4107fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41080c63d0 0x7f4108150260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33032/0 (socket says 172.21.15.88:33032) 2023-10-16T20:53:36.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f4107fff640 1 -- 172.21.15.88:0/3393987407 learned_addr learned my addr 172.21.15.88:0/3393987407 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:36.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f41077fe640 1 --2- 172.21.15.88:0/3393987407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41080c67b0 0x7f410814f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:36.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.921+0000 7f410ceba640 1 --2- 172.21.15.88:0/3393987407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41080c9380 0x7f410814f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:36.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.925+0000 7f4107fff640 1 -- 172.21.15.88:0/3393987407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41080c67b0 msgr2=0x7f410814f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:36.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.925+0000 7f4107fff640 1 --2- 172.21.15.88:0/3393987407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41080c67b0 0x7f410814f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:36.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.925+0000 7f4107fff640 1 -- 172.21.15.88:0/3393987407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41080c9380 msgr2=0x7f410814f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:36.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.925+0000 7f4107fff640 1 --2- 172.21.15.88:0/3393987407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41080c9380 0x7f410814f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:36.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.925+0000 7f4107fff640 1 -- 172.21.15.88:0/3393987407 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4100009b30 con 0x7f41080c63d0 2023-10-16T20:53:36.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.925+0000 7f410ceba640 1 --2- 172.21.15.88:0/3393987407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41080c9380 0x7f410814f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:53:36.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.925+0000 7f4107fff640 1 --2- 172.21.15.88:0/3393987407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41080c63d0 0x7f4108150260 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f40f800c500 tx=0x7f40f800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:36.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.925+0000 7f4104ff9640 1 -- 172.21.15.88:0/3393987407 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f8005220 con 0x7f41080c63d0 2023-10-16T20:53:36.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.925+0000 7f4104ff9640 1 -- 172.21.15.88:0/3393987407 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f8005380 con 0x7f41080c63d0 2023-10-16T20:53:36.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.925+0000 7f410d6bb640 1 -- 172.21.15.88:0/3393987407 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41081eef20 con 0x7f41080c63d0 2023-10-16T20:53:36.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.925+0000 7f4104ff9640 1 -- 172.21.15.88:0/3393987407 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f8007270 con 0x7f41080c63d0 2023-10-16T20:53:36.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.925+0000 7f410d6bb640 1 -- 172.21.15.88:0/3393987407 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41081ef4e0 con 0x7f41080c63d0 2023-10-16T20:53:36.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.925+0000 7f410d6bb640 1 -- 172.21.15.88:0/3393987407 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41081594e0 con 0x7f41080c63d0 2023-10-16T20:53:36.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.925+0000 7f4104ff9640 1 -- 172.21.15.88:0/3393987407 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f40f80073d0 con 0x7f41080c63d0 2023-10-16T20:53:36.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.929+0000 7f4104ff9640 1 --2- 172.21.15.88:0/3393987407 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f40f40664a0 0x7f40f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:36.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.929+0000 7f4104ff9640 1 -- 172.21.15.88:0/3393987407 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f40f8083e70 con 0x7f41080c63d0 2023-10-16T20:53:36.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.929+0000 7f41077fe640 1 --2- 172.21.15.88:0/3393987407 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f40f40664a0 0x7f40f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:36.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.929+0000 7f41077fe640 1 --2- 172.21.15.88:0/3393987407 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f40f40664a0 0x7f40f4068960 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f41081e9110 tx=0x7f40fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:36.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:36.933+0000 7f4104ff9640 1 -- 172.21.15.88:0/3393987407 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f40f8050c40 con 0x7f41080c63d0 2023-10-16T20:53:37.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:37.137+0000 7f410d6bb640 1 -- 172.21.15.88:0/3393987407 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f410804e9c0 con 0x7f41080c63d0 2023-10-16T20:53:37.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:37.137+0000 7f4104ff9640 1 -- 172.21.15.88:0/3393987407 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5781 (secure 0 0 0) 0x7f40f8054260 con 0x7f41080c63d0 2023-10-16T20:53:37.140 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:37.140 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:53:36.598006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:reconnect","state_seq":28,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:37.141 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 91 2023-10-16T20:53:37.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:37.141+0000 7f410d6bb640 1 -- 172.21.15.88:0/3393987407 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f40f40664a0 msgr2=0x7f40f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:37.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:37.141+0000 7f410d6bb640 1 --2- 172.21.15.88:0/3393987407 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f40f40664a0 0x7f40f4068960 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f41081e9110 tx=0x7f40fc008040 comp rx=0 tx=0).stop 2023-10-16T20:53:37.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:37.141+0000 7f410d6bb640 1 -- 172.21.15.88:0/3393987407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41080c63d0 msgr2=0x7f4108150260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:37.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:37.141+0000 7f410d6bb640 1 --2- 172.21.15.88:0/3393987407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41080c63d0 0x7f4108150260 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f40f800c500 tx=0x7f40f800c9c0 comp rx=0 tx=0).stop 2023-10-16T20:53:37.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:37.141+0000 7f410d6bb640 1 -- 172.21.15.88:0/3393987407 shutdown_connections 2023-10-16T20:53:37.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:37.141+0000 7f410d6bb640 1 --2- 172.21.15.88:0/3393987407 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f40f40664a0 0x7f40f4068960 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:37.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:37.141+0000 7f410d6bb640 1 --2- 172.21.15.88:0/3393987407 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41080c9380 0x7f410814f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:37.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:37.141+0000 7f410d6bb640 1 --2- 172.21.15.88:0/3393987407 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41080c67b0 0x7f410814f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:37.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:37.141+0000 7f410d6bb640 1 --2- 172.21.15.88:0/3393987407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41080c63d0 0x7f4108150260 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:37.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:37.141+0000 7f410d6bb640 1 -- 172.21.15.88:0/3393987407 >> 172.21.15.88:0/3393987407 conn(0x7f4108065a80 msgr2=0x7f41080c2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:37.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:37.141+0000 7f410d6bb640 1 -- 172.21.15.88:0/3393987407 shutdown_connections 2023-10-16T20:53:37.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:37.141+0000 7f410d6bb640 1 -- 172.21.15.88:0/3393987407 wait complete. 2023-10-16T20:53:37.162 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:53:39.163 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:39.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.313+0000 7fafd753b640 1 Processor -- start 2023-10-16T20:53:39.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.313+0000 7fafd753b640 1 -- start start 2023-10-16T20:53:39.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.313+0000 7fafd753b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fafd00ca2f0 0x7fafd00c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:39.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.313+0000 7fafd753b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd00cae10 con 0x7fafd01537c0 2023-10-16T20:53:39.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.313+0000 7fafd753b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd00cac30 con 0x7fafd00c51c0 2023-10-16T20:53:39.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.313+0000 7fafd753b640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafd00c55d0 con 0x7fafd00ca2f0 2023-10-16T20:53:39.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.313+0000 7fafd6539640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fafd01537c0 0x7fafd015c870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41112/0 (socket says 172.21.15.88:41112) 2023-10-16T20:53:39.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.313+0000 7fafd6539640 1 -- 172.21.15.88:0/1226174638 learned_addr learned my addr 172.21.15.88:0/1226174638 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:53:39.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.313+0000 7fafd5d38640 1 --2- 172.21.15.88:0/1226174638 >> v2:172.21.15.191:3300/0 conn(0x7fafd00ca2f0 0x7fafd00c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:39.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.313+0000 7fafd5537640 1 -- 172.21.15.88:0/1226174638 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818138926 0 0) 0x7fafd00cae10 con 0x7fafd01537c0 2023-10-16T20:53:39.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.313+0000 7fafd5537640 1 -- 172.21.15.88:0/1226174638 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafbc003530 con 0x7fafd01537c0 2023-10-16T20:53:39.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.313+0000 7fafd5537640 1 -- 172.21.15.88:0/1226174638 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936821860 0 0) 0x7fafd00cac30 con 0x7fafd00c51c0 2023-10-16T20:53:39.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.313+0000 7fafd5537640 1 -- 172.21.15.88:0/1226174638 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafd00cae10 con 0x7fafd00c51c0 2023-10-16T20:53:39.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.313+0000 7fafd5d38640 1 -- 172.21.15.88:0/1226174638 >> v1:172.21.15.191:6789/0 conn(0x7fafd00c51c0 legacy=0x7fafd00cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:39.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.313+0000 7fafd5d38640 1 -- 172.21.15.88:0/1226174638 >> v1:172.21.15.88:6789/0 conn(0x7fafd01537c0 legacy=0x7fafd015c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:39.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.313+0000 7fafd5d38640 1 -- 172.21.15.88:0/1226174638 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd015d2b0 con 0x7fafd00ca2f0 2023-10-16T20:53:39.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.317+0000 7fafd5d38640 1 --2- 172.21.15.88:0/1226174638 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafd00ca2f0 0x7fafd00c4c80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fafc00097c0 tx=0x7fafc001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43147f7de12b5e2e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:39.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.317+0000 7fafd5537640 1 -- 172.21.15.88:0/1226174638 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc00ab650 con 0x7fafd00ca2f0 2023-10-16T20:53:39.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.317+0000 7fafd5537640 1 -- 172.21.15.88:0/1226174638 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc00ab7b0 con 0x7fafd00ca2f0 2023-10-16T20:53:39.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.317+0000 7fafd5537640 1 -- 172.21.15.88:0/1226174638 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc00abb40 con 0x7fafd00ca2f0 2023-10-16T20:53:39.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.317+0000 7fafd753b640 1 -- 172.21.15.88:0/1226174638 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafd00ca2f0 msgr2=0x7fafd00c4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:39.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.317+0000 7fafd753b640 1 --2- 172.21.15.88:0/1226174638 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafd00ca2f0 0x7fafd00c4c80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fafc00097c0 tx=0x7fafc001c450 comp rx=0 tx=0).stop 2023-10-16T20:53:39.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.317+0000 7fafd753b640 1 -- 172.21.15.88:0/1226174638 shutdown_connections 2023-10-16T20:53:39.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.317+0000 7fafd753b640 1 --2- 172.21.15.88:0/1226174638 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafd00ca2f0 0x7fafd00c4c80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:39.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.317+0000 7fafd753b640 1 -- 172.21.15.88:0/1226174638 >> 172.21.15.88:0/1226174638 conn(0x7fafd006c170 msgr2=0x7fafd00715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:39.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.317+0000 7fafd753b640 1 -- 172.21.15.88:0/1226174638 shutdown_connections 2023-10-16T20:53:39.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.317+0000 7fafd753b640 1 -- 172.21.15.88:0/1226174638 wait complete. 2023-10-16T20:53:39.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd753b640 1 Processor -- start 2023-10-16T20:53:39.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd753b640 1 -- start start 2023-10-16T20:53:39.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd753b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafd00c51c0 0x7fafd00ce640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:39.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd753b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafd00ca2f0 0x7fafd01f4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:39.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd753b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafd01537c0 0x7fafd0141860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:39.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd753b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd014c510 con 0x7fafd00ca2f0 2023-10-16T20:53:39.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd753b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd00ceb80 con 0x7fafd01537c0 2023-10-16T20:53:39.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd753b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd01f52a0 con 0x7fafd00c51c0 2023-10-16T20:53:39.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd5d38640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafd00ca2f0 0x7fafd01f4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:39.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd5d38640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafd00ca2f0 0x7fafd01f4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39862/0 (socket says 172.21.15.88:39862) 2023-10-16T20:53:39.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd5d38640 1 -- 172.21.15.88:0/2457696851 learned_addr learned my addr 172.21.15.88:0/2457696851 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:39.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd6539640 1 --2- 172.21.15.88:0/2457696851 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafd00c51c0 0x7fafd00ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:39.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd6d3a640 1 --2- 172.21.15.88:0/2457696851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafd01537c0 0x7fafd0141860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:39.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd5d38640 1 -- 172.21.15.88:0/2457696851 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafd00c51c0 msgr2=0x7fafd00ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:39.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd5d38640 1 --2- 172.21.15.88:0/2457696851 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafd00c51c0 0x7fafd00ce640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:39.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd5d38640 1 -- 172.21.15.88:0/2457696851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafd01537c0 msgr2=0x7fafd0141860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:39.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd5d38640 1 --2- 172.21.15.88:0/2457696851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafd01537c0 0x7fafd0141860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:39.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd5d38640 1 -- 172.21.15.88:0/2457696851 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc0009420 con 0x7fafd00ca2f0 2023-10-16T20:53:39.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd6d3a640 1 --2- 172.21.15.88:0/2457696851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafd01537c0 0x7fafd0141860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:53:39.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.321+0000 7fafd5d38640 1 --2- 172.21.15.88:0/2457696851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafd00ca2f0 0x7fafd01f4880 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fafc00ad040 tx=0x7fafc001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:39.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.325+0000 7fafc6ffd640 1 -- 172.21.15.88:0/2457696851 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc00bf610 con 0x7fafd00ca2f0 2023-10-16T20:53:39.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.325+0000 7fafc6ffd640 1 -- 172.21.15.88:0/2457696851 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc00bf7b0 con 0x7fafd00ca2f0 2023-10-16T20:53:39.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.325+0000 7fafd753b640 1 -- 172.21.15.88:0/2457696851 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd014c7e0 con 0x7fafd00ca2f0 2023-10-16T20:53:39.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.325+0000 7fafc6ffd640 1 -- 172.21.15.88:0/2457696851 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc00bfb60 con 0x7fafd00ca2f0 2023-10-16T20:53:39.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.325+0000 7fafd753b640 1 -- 172.21.15.88:0/2457696851 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd0142850 con 0x7fafd00ca2f0 2023-10-16T20:53:39.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.325+0000 7fafd753b640 1 -- 172.21.15.88:0/2457696851 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd004e9c0 con 0x7fafd00ca2f0 2023-10-16T20:53:39.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.325+0000 7fafc6ffd640 1 -- 172.21.15.88:0/2457696851 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fafc00b5400 con 0x7fafd00ca2f0 2023-10-16T20:53:39.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.325+0000 7fafc6ffd640 1 --2- 172.21.15.88:0/2457696851 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fafb40664d0 0x7fafb4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:39.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.325+0000 7fafc6ffd640 1 -- 172.21.15.88:0/2457696851 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7fafc011f100 con 0x7fafd00ca2f0 2023-10-16T20:53:39.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.325+0000 7fafd6539640 1 --2- 172.21.15.88:0/2457696851 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fafb40664d0 0x7fafb4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:39.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.329+0000 7fafd6539640 1 --2- 172.21.15.88:0/2457696851 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fafb40664d0 0x7fafb4068990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fafcc007fd0 tx=0x7fafcc007d80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:39.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.333+0000 7fafc6ffd640 1 -- 172.21.15.88:0/2457696851 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fafc00ec010 con 0x7fafd00ca2f0 2023-10-16T20:53:39.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.557+0000 7fafd753b640 1 -- 172.21.15.88:0/2457696851 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafd00ca0d0 con 0x7fafd00ca2f0 2023-10-16T20:53:39.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.557+0000 7fafc6ffd640 1 -- 172.21.15.88:0/2457696851 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5781 (secure 0 0 0) 0x7fafc00ef630 con 0x7fafd00ca2f0 2023-10-16T20:53:39.561 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:39.561 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:53:36.598006+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:reconnect","state_seq":28,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:39.562 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 91 2023-10-16T20:53:39.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.561+0000 7fafd753b640 1 -- 172.21.15.88:0/2457696851 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fafb40664d0 msgr2=0x7fafb4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:39.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.561+0000 7fafd753b640 1 --2- 172.21.15.88:0/2457696851 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fafb40664d0 0x7fafb4068990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fafcc007fd0 tx=0x7fafcc007d80 comp rx=0 tx=0).stop 2023-10-16T20:53:39.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.561+0000 7fafd753b640 1 -- 172.21.15.88:0/2457696851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafd00ca2f0 msgr2=0x7fafd01f4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:39.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.561+0000 7fafd753b640 1 --2- 172.21.15.88:0/2457696851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafd00ca2f0 0x7fafd01f4880 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fafc00ad040 tx=0x7fafc001ac90 comp rx=0 tx=0).stop 2023-10-16T20:53:39.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.561+0000 7fafd753b640 1 -- 172.21.15.88:0/2457696851 shutdown_connections 2023-10-16T20:53:39.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.561+0000 7fafd753b640 1 --2- 172.21.15.88:0/2457696851 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fafb40664d0 0x7fafb4068990 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:39.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.561+0000 7fafd753b640 1 --2- 172.21.15.88:0/2457696851 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fafd01537c0 0x7fafd0141860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:39.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.561+0000 7fafd753b640 1 --2- 172.21.15.88:0/2457696851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafd00ca2f0 0x7fafd01f4880 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:39.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.561+0000 7fafd753b640 1 --2- 172.21.15.88:0/2457696851 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fafd00c51c0 0x7fafd00ce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:39.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.561+0000 7fafd753b640 1 -- 172.21.15.88:0/2457696851 >> 172.21.15.88:0/2457696851 conn(0x7fafd006c170 msgr2=0x7fafd00cb570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:39.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.561+0000 7fafd753b640 1 -- 172.21.15.88:0/2457696851 shutdown_connections 2023-10-16T20:53:39.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:39.561+0000 7fafd753b640 1 -- 172.21.15.88:0/2457696851 wait complete. 2023-10-16T20:53:39.586 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 91, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5282": { "addr": "172.21.15.191:6833/3633326602", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 3633326602, "type": "v2" }, { "addr": "172.21.15.191:6833", "nonce": 3633326602, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5282, "incarnation": 81, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 41 }, "gid_5284": { "addr": "172.21.15.88:6839/2172973775", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6837", "nonce": 2172973775, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2172973775, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5284, "incarnation": 38, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 17 }, "gid_5378": { "addr": "172.21.15.191:6835/3119045295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3119045295, "type": "v2" }, { "addr": "172.21.15.191:6835", "nonce": 3119045295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5378, "incarnation": 89, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:reconnect", "state_seq": 28 }, "gid_5535": { "addr": "172.21.15.88:6835/3476086754", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3476086754, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3476086754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5535, "incarnation": 52, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_5709": { "addr": "172.21.15.88:6838/2600891060", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 2600891060, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 2600891060, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5709, "incarnation": 67, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:53:36.598006+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5282, "mds_1": 5709, "mds_2": 5535, "mds_3": 5284, "mds_4": 5378 } } } ], "standbys": [] } 2023-10-16T20:53:41.588 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:41.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.749+0000 7fdf2da17640 1 Processor -- start 2023-10-16T20:53:41.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.749+0000 7fdf2da17640 1 -- start start 2023-10-16T20:53:41.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.753+0000 7fdf2da17640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf281534f0 0x7fdf280c3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:41.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.753+0000 7fdf2da17640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdf280c9a50 0x7fdf28071b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:41.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.753+0000 7fdf2da17640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf28159ec0 con 0x7fdf28072070 2023-10-16T20:53:41.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.753+0000 7fdf2da17640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf2815d270 con 0x7fdf280c9a50 2023-10-16T20:53:41.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.753+0000 7fdf2da17640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2815a160 con 0x7fdf281534f0 2023-10-16T20:53:41.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.753+0000 7fdf1ffff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdf280c9a50 0x7fdf28071b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:41.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.753+0000 7fdf1ffff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fdf280c9a50 0x7fdf28071b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:33674/0 (socket says 172.21.15.88:33674) 2023-10-16T20:53:41.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.753+0000 7fdf1ffff640 1 -- 172.21.15.88:0/2341317077 learned_addr learned my addr 172.21.15.88:0/2341317077 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:41.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.753+0000 7fdf26ffd640 1 --2- 172.21.15.88:0/2341317077 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf281534f0 0x7fdf280c3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:41.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.753+0000 7fdf1ffff640 1 -- 172.21.15.88:0/2341317077 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf281534f0 msgr2=0x7fdf280c3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:41.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.753+0000 7fdf1ffff640 1 --2- 172.21.15.88:0/2341317077 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf281534f0 0x7fdf280c3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:41.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.753+0000 7fdf1ffff640 1 -- 172.21.15.88:0/2341317077 >> v1:172.21.15.191:6789/0 conn(0x7fdf28072070 legacy=0x7fdf281592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:41.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.753+0000 7fdf1ffff640 1 -- 172.21.15.88:0/2341317077 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2815c8d0 con 0x7fdf280c9a50 2023-10-16T20:53:41.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.757+0000 7fdf1ffff640 1 --2- 172.21.15.88:0/2341317077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf280c9a50 0x7fdf28071b30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdf10009870 tx=0x7fdf1001ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7ed934dbd114eff server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:41.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.757+0000 7fdf267fc640 1 -- 172.21.15.88:0/2341317077 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf100a8a30 con 0x7fdf280c9a50 2023-10-16T20:53:41.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.757+0000 7fdf267fc640 1 -- 172.21.15.88:0/2341317077 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf100a8b90 con 0x7fdf280c9a50 2023-10-16T20:53:41.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.757+0000 7fdf267fc640 1 -- 172.21.15.88:0/2341317077 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf100b0440 con 0x7fdf280c9a50 2023-10-16T20:53:41.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.757+0000 7fdf2da17640 1 -- 172.21.15.88:0/2341317077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf280c9a50 msgr2=0x7fdf28071b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:41.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.757+0000 7fdf2da17640 1 --2- 172.21.15.88:0/2341317077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf280c9a50 0x7fdf28071b30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdf10009870 tx=0x7fdf1001ed30 comp rx=0 tx=0).stop 2023-10-16T20:53:41.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.757+0000 7fdf2da17640 1 -- 172.21.15.88:0/2341317077 shutdown_connections 2023-10-16T20:53:41.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.757+0000 7fdf2da17640 1 --2- 172.21.15.88:0/2341317077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf280c9a50 0x7fdf28071b30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:41.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.757+0000 7fdf2da17640 1 --2- 172.21.15.88:0/2341317077 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf281534f0 0x7fdf280c3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:41.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.757+0000 7fdf2da17640 1 -- 172.21.15.88:0/2341317077 >> 172.21.15.88:0/2341317077 conn(0x7fdf280cf260 msgr2=0x7fdf280cf670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:41.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.757+0000 7fdf2da17640 1 -- 172.21.15.88:0/2341317077 shutdown_connections 2023-10-16T20:53:41.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.757+0000 7fdf2da17640 1 -- 172.21.15.88:0/2341317077 wait complete. 2023-10-16T20:53:41.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.757+0000 7fdf2da17640 1 Processor -- start 2023-10-16T20:53:41.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.757+0000 7fdf2da17640 1 -- start start 2023-10-16T20:53:41.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf2da17640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf28072070 0x7fdf28149aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:41.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf2da17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf280c9a50 0x7fdf281e5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:41.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf2da17640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf281534f0 0x7fdf281e49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:41.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf2da17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf281e2a90 con 0x7fdf280c9a50 2023-10-16T20:53:41.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf2da17640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf281e2fc0 con 0x7fdf281534f0 2023-10-16T20:53:41.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf2da17640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf280c5f80 con 0x7fdf28072070 2023-10-16T20:53:41.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf1ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf280c9a50 0x7fdf281e5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:41.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf1ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf280c9a50 0x7fdf281e5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39874/0 (socket says 172.21.15.88:39874) 2023-10-16T20:53:41.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf1ffff640 1 -- 172.21.15.88:0/1608130062 learned_addr learned my addr 172.21.15.88:0/1608130062 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:41.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf277fe640 1 --2- 172.21.15.88:0/1608130062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf281534f0 0x7fdf281e49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:41.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf26ffd640 1 --2- 172.21.15.88:0/1608130062 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf28072070 0x7fdf28149aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:41.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf1ffff640 1 -- 172.21.15.88:0/1608130062 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf28072070 msgr2=0x7fdf28149aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:41.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf1ffff640 1 --2- 172.21.15.88:0/1608130062 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf28072070 0x7fdf28149aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:41.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf1ffff640 1 -- 172.21.15.88:0/1608130062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf281534f0 msgr2=0x7fdf281e49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:41.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf1ffff640 1 --2- 172.21.15.88:0/1608130062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf281534f0 0x7fdf281e49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:41.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf1ffff640 1 -- 172.21.15.88:0/1608130062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf100094d0 con 0x7fdf280c9a50 2023-10-16T20:53:41.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf1ffff640 1 --2- 172.21.15.88:0/1608130062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf280c9a50 0x7fdf281e5ab0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fdf100b3040 tx=0x7fdf1001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:41.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf24ff9640 1 -- 172.21.15.88:0/1608130062 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf100b0dd0 con 0x7fdf280c9a50 2023-10-16T20:53:41.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf24ff9640 1 -- 172.21.15.88:0/1608130062 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf100bb460 con 0x7fdf280c9a50 2023-10-16T20:53:41.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf2da17640 1 -- 172.21.15.88:0/1608130062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf281ef920 con 0x7fdf280c9a50 2023-10-16T20:53:41.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf24ff9640 1 -- 172.21.15.88:0/1608130062 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf100bb810 con 0x7fdf280c9a50 2023-10-16T20:53:41.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf2da17640 1 -- 172.21.15.88:0/1608130062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf281e41f0 con 0x7fdf280c9a50 2023-10-16T20:53:41.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.761+0000 7fdf2da17640 1 -- 172.21.15.88:0/1608130062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf2804e9c0 con 0x7fdf280c9a50 2023-10-16T20:53:41.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.765+0000 7fdf24ff9640 1 -- 172.21.15.88:0/1608130062 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdf100b83f0 con 0x7fdf280c9a50 2023-10-16T20:53:41.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.765+0000 7fdf24ff9640 1 --2- 172.21.15.88:0/1608130062 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdf080664a0 0x7fdf08068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:41.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.765+0000 7fdf24ff9640 1 -- 172.21.15.88:0/1608130062 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7fdf10121d40 con 0x7fdf280c9a50 2023-10-16T20:53:41.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.765+0000 7fdf26ffd640 1 --2- 172.21.15.88:0/1608130062 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdf080664a0 0x7fdf08068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:41.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.765+0000 7fdf26ffd640 1 --2- 172.21.15.88:0/1608130062 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdf080664a0 0x7fdf08068960 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdf18009ac0 tx=0x7fdf18009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:41.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.769+0000 7fdf24ff9640 1 -- 172.21.15.88:0/1608130062 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdf100eece0 con 0x7fdf280c9a50 2023-10-16T20:53:41.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.993+0000 7fdf2da17640 1 -- 172.21.15.88:0/1608130062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf28150b00 con 0x7fdf280c9a50 2023-10-16T20:53:41.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.993+0000 7fdf24ff9640 1 -- 172.21.15.88:0/1608130062 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5778 (secure 0 0 0) 0x7fdf100b5020 con 0x7fdf280c9a50 2023-10-16T20:53:41.996 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:41.997 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:53:41.616280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:rejoin","state_seq":30,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:41.997 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 92 2023-10-16T20:53:41.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.997+0000 7fdf2da17640 1 -- 172.21.15.88:0/1608130062 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdf080664a0 msgr2=0x7fdf08068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:41.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.997+0000 7fdf2da17640 1 --2- 172.21.15.88:0/1608130062 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdf080664a0 0x7fdf08068960 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdf18009ac0 tx=0x7fdf18009190 comp rx=0 tx=0).stop 2023-10-16T20:53:41.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.997+0000 7fdf2da17640 1 -- 172.21.15.88:0/1608130062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf280c9a50 msgr2=0x7fdf281e5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:41.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.997+0000 7fdf2da17640 1 --2- 172.21.15.88:0/1608130062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf280c9a50 0x7fdf281e5ab0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fdf100b3040 tx=0x7fdf1001bad0 comp rx=0 tx=0).stop 2023-10-16T20:53:41.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.997+0000 7fdf2da17640 1 -- 172.21.15.88:0/1608130062 shutdown_connections 2023-10-16T20:53:42.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.997+0000 7fdf2da17640 1 --2- 172.21.15.88:0/1608130062 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdf080664a0 0x7fdf08068960 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:42.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.997+0000 7fdf2da17640 1 --2- 172.21.15.88:0/1608130062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdf281534f0 0x7fdf281e49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:42.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.997+0000 7fdf2da17640 1 --2- 172.21.15.88:0/1608130062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf280c9a50 0x7fdf281e5ab0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:42.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.997+0000 7fdf2da17640 1 --2- 172.21.15.88:0/1608130062 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdf28072070 0x7fdf28149aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:42.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.997+0000 7fdf2da17640 1 -- 172.21.15.88:0/1608130062 >> 172.21.15.88:0/1608130062 conn(0x7fdf280cf260 msgr2=0x7fdf28069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:42.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.997+0000 7fdf2da17640 1 -- 172.21.15.88:0/1608130062 shutdown_connections 2023-10-16T20:53:42.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:41.997+0000 7fdf2da17640 1 -- 172.21.15.88:0/1608130062 wait complete. 2023-10-16T20:53:42.019 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:53:44.020 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:44.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.181+0000 7fa239f9d640 1 Processor -- start 2023-10-16T20:53:44.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.181+0000 7fa239f9d640 1 -- start start 2023-10-16T20:53:44.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.181+0000 7fa239f9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa234153770 0x7fa234159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:44.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.181+0000 7fa239f9d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa2340cece0 0x7fa2340cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:44.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.181+0000 7fa239f9d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa23415a410 con 0x7fa2340c9bb0 2023-10-16T20:53:44.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.181+0000 7fa239f9d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2340cc250 con 0x7fa234153770 2023-10-16T20:53:44.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.181+0000 7fa239f9d640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2340cc670 con 0x7fa2340cece0 2023-10-16T20:53:44.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.181+0000 7fa2337fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa234153770 0x7fa234159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:44.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.181+0000 7fa2337fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa234153770 0x7fa234159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39878/0 (socket says 172.21.15.88:39878) 2023-10-16T20:53:44.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.181+0000 7fa2337fe640 1 -- 172.21.15.88:0/4254420327 learned_addr learned my addr 172.21.15.88:0/4254420327 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:44.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.181+0000 7fa232ffd640 1 --1- 172.21.15.88:0/4254420327 >> v1:172.21.15.88:6789/0 conn(0x7fa2340c9bb0 0x7fa2340c3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41114/0 (socket says 172.21.15.88:41114) 2023-10-16T20:53:44.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.181+0000 7fa233fff640 1 --2- 172.21.15.88:0/4254420327 >> v2:172.21.15.191:3300/0 conn(0x7fa2340cece0 0x7fa2340cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:44.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.181+0000 7fa2337fe640 1 -- 172.21.15.88:0/4254420327 >> v2:172.21.15.191:3300/0 conn(0x7fa2340cece0 msgr2=0x7fa2340cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:44.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.181+0000 7fa2337fe640 1 --2- 172.21.15.88:0/4254420327 >> v2:172.21.15.191:3300/0 conn(0x7fa2340cece0 0x7fa2340cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:44.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.181+0000 7fa2337fe640 1 -- 172.21.15.88:0/4254420327 >> v1:172.21.15.88:6789/0 conn(0x7fa2340c9bb0 legacy=0x7fa2340c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:44.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.181+0000 7fa2337fe640 1 -- 172.21.15.88:0/4254420327 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa234153b50 con 0x7fa234153770 2023-10-16T20:53:44.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.181+0000 7fa2327fc640 1 -- 172.21.15.88:0/4254420327 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224789890 0 0) 0x7fa23415a410 con 0x7fa2340c9bb0 2023-10-16T20:53:44.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.185+0000 7fa2337fe640 1 --2- 172.21.15.88:0/4254420327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa234153770 0x7fa234159ed0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa2340cc7b0 tx=0x7fa22401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b6e215f48f52a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:44.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.185+0000 7fa2327fc640 1 -- 172.21.15.88:0/4254420327 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2240aa640 con 0x7fa234153770 2023-10-16T20:53:44.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.185+0000 7fa2327fc640 1 -- 172.21.15.88:0/4254420327 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2240aa7a0 con 0x7fa234153770 2023-10-16T20:53:44.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.185+0000 7fa2327fc640 1 -- 172.21.15.88:0/4254420327 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2240aab30 con 0x7fa234153770 2023-10-16T20:53:44.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.185+0000 7fa239f9d640 1 -- 172.21.15.88:0/4254420327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa234153770 msgr2=0x7fa234159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:44.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.185+0000 7fa239f9d640 1 --2- 172.21.15.88:0/4254420327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa234153770 0x7fa234159ed0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa2340cc7b0 tx=0x7fa22401c4e0 comp rx=0 tx=0).stop 2023-10-16T20:53:44.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.185+0000 7fa239f9d640 1 -- 172.21.15.88:0/4254420327 shutdown_connections 2023-10-16T20:53:44.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.185+0000 7fa239f9d640 1 --2- 172.21.15.88:0/4254420327 >> v2:172.21.15.191:3300/0 conn(0x7fa2340cece0 0x7fa2340cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:44.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.185+0000 7fa239f9d640 1 --2- 172.21.15.88:0/4254420327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa234153770 0x7fa234159ed0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:44.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.185+0000 7fa239f9d640 1 -- 172.21.15.88:0/4254420327 >> 172.21.15.88:0/4254420327 conn(0x7fa2340715c0 msgr2=0x7fa2340719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:44.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.185+0000 7fa239f9d640 1 -- 172.21.15.88:0/4254420327 shutdown_connections 2023-10-16T20:53:44.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.185+0000 7fa239f9d640 1 -- 172.21.15.88:0/4254420327 wait complete. 2023-10-16T20:53:44.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa239f9d640 1 Processor -- start 2023-10-16T20:53:44.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa239f9d640 1 -- start start 2023-10-16T20:53:44.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa239f9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2340c9bb0 0x7fa2341495f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:44.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa239f9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2340cece0 0x7fa2341f1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:44.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa239f9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa234153770 0x7fa2341f1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:44.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa239f9d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2341f4b90 con 0x7fa2340c9bb0 2023-10-16T20:53:44.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa239f9d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2341f4e50 con 0x7fa2340cece0 2023-10-16T20:53:44.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa239f9d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2341f5130 con 0x7fa234153770 2023-10-16T20:53:44.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa2337fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2340c9bb0 0x7fa2341495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:44.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa2337fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2340c9bb0 0x7fa2341495f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39892/0 (socket says 172.21.15.88:39892) 2023-10-16T20:53:44.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa2337fe640 1 -- 172.21.15.88:0/1865454105 learned_addr learned my addr 172.21.15.88:0/1865454105 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:44.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa232ffd640 1 --2- 172.21.15.88:0/1865454105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2340cece0 0x7fa2341f1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:44.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa233fff640 1 --2- 172.21.15.88:0/1865454105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa234153770 0x7fa2341f1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:44.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa2337fe640 1 -- 172.21.15.88:0/1865454105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa234153770 msgr2=0x7fa2341f1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:44.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa2337fe640 1 --2- 172.21.15.88:0/1865454105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa234153770 0x7fa2341f1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:44.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa2337fe640 1 -- 172.21.15.88:0/1865454105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2340cece0 msgr2=0x7fa2341f1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:44.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa2337fe640 1 --2- 172.21.15.88:0/1865454105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2340cece0 0x7fa2341f1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:44.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa2337fe640 1 -- 172.21.15.88:0/1865454105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa224009480 con 0x7fa2340c9bb0 2023-10-16T20:53:44.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa2337fe640 1 --2- 172.21.15.88:0/1865454105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2340c9bb0 0x7fa2341495f0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa22401a110 tx=0x7fa224019280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:44.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa21bfff640 1 -- 172.21.15.88:0/1865454105 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa224005ba0 con 0x7fa2340c9bb0 2023-10-16T20:53:44.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa21bfff640 1 -- 172.21.15.88:0/1865454105 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa224005d00 con 0x7fa2340c9bb0 2023-10-16T20:53:44.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa239f9d640 1 -- 172.21.15.88:0/1865454105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2341f5410 con 0x7fa2340c9bb0 2023-10-16T20:53:44.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa21bfff640 1 -- 172.21.15.88:0/1865454105 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2240aab30 con 0x7fa2340c9bb0 2023-10-16T20:53:44.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.189+0000 7fa239f9d640 1 -- 172.21.15.88:0/1865454105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa23415e140 con 0x7fa2340c9bb0 2023-10-16T20:53:44.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.193+0000 7fa239f9d640 1 -- 172.21.15.88:0/1865454105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa23415e3f0 con 0x7fa2340c9bb0 2023-10-16T20:53:44.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.193+0000 7fa21bfff640 1 -- 172.21.15.88:0/1865454105 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa2240a8050 con 0x7fa2340c9bb0 2023-10-16T20:53:44.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.193+0000 7fa21bfff640 1 --2- 172.21.15.88:0/1865454105 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa2200664a0 0x7fa220068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:44.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.193+0000 7fa21bfff640 1 -- 172.21.15.88:0/1865454105 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7fa22411dcf0 con 0x7fa2340c9bb0 2023-10-16T20:53:44.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.193+0000 7fa232ffd640 1 --2- 172.21.15.88:0/1865454105 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa2200664a0 0x7fa220068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:44.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.193+0000 7fa232ffd640 1 --2- 172.21.15.88:0/1865454105 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa2200664a0 0x7fa220068960 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa23415cb80 tx=0x7fa228007d60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:44.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.197+0000 7fa21bfff640 1 -- 172.21.15.88:0/1865454105 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa2240eaad0 con 0x7fa2340c9bb0 2023-10-16T20:53:44.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.413+0000 7fa239f9d640 1 -- 172.21.15.88:0/1865454105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2341f58c0 con 0x7fa2340c9bb0 2023-10-16T20:53:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.413+0000 7fa21bfff640 1 -- 172.21.15.88:0/1865454105 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5778 (secure 0 0 0) 0x7fa2341f58c0 con 0x7fa2340c9bb0 2023-10-16T20:53:44.415 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:44.415 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:53:41.616280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:rejoin","state_seq":30,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:44.416 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 92 2023-10-16T20:53:44.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.413+0000 7fa239f9d640 1 -- 172.21.15.88:0/1865454105 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa2200664a0 msgr2=0x7fa220068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:44.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.413+0000 7fa239f9d640 1 --2- 172.21.15.88:0/1865454105 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa2200664a0 0x7fa220068960 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa23415cb80 tx=0x7fa228007d60 comp rx=0 tx=0).stop 2023-10-16T20:53:44.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.413+0000 7fa239f9d640 1 -- 172.21.15.88:0/1865454105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2340c9bb0 msgr2=0x7fa2341495f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:44.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.413+0000 7fa239f9d640 1 --2- 172.21.15.88:0/1865454105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2340c9bb0 0x7fa2341495f0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa22401a110 tx=0x7fa224019280 comp rx=0 tx=0).stop 2023-10-16T20:53:44.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.413+0000 7fa239f9d640 1 -- 172.21.15.88:0/1865454105 shutdown_connections 2023-10-16T20:53:44.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.413+0000 7fa239f9d640 1 --2- 172.21.15.88:0/1865454105 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa2200664a0 0x7fa220068960 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:44.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.413+0000 7fa239f9d640 1 --2- 172.21.15.88:0/1865454105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa234153770 0x7fa2341f1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:44.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.413+0000 7fa239f9d640 1 --2- 172.21.15.88:0/1865454105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2340cece0 0x7fa2341f1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:44.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.413+0000 7fa239f9d640 1 --2- 172.21.15.88:0/1865454105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2340c9bb0 0x7fa2341495f0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:44.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.413+0000 7fa239f9d640 1 -- 172.21.15.88:0/1865454105 >> 172.21.15.88:0/1865454105 conn(0x7fa2340715c0 msgr2=0x7fa234151920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:44.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.417+0000 7fa239f9d640 1 -- 172.21.15.88:0/1865454105 shutdown_connections 2023-10-16T20:53:44.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:44.417+0000 7fa239f9d640 1 -- 172.21.15.88:0/1865454105 wait complete. 2023-10-16T20:53:44.438 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:53:46.439 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.597+0000 7fd235f0f640 1 Processor -- start 2023-10-16T20:53:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.597+0000 7fd235f0f640 1 -- start start 2023-10-16T20:53:46.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.601+0000 7fd235f0f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd230153770 0x7fd230159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.601+0000 7fd235f0f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd2300c9bb0 0x7fd2300c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.601+0000 7fd235f0f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd23015a410 con 0x7fd2300c41f0 2023-10-16T20:53:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.601+0000 7fd235f0f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2300cc250 con 0x7fd2300c9bb0 2023-10-16T20:53:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.601+0000 7fd235f0f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2300cc670 con 0x7fd230153770 2023-10-16T20:53:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.601+0000 7fd22effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd2300c9bb0 0x7fd2300c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:46.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.601+0000 7fd22effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd2300c9bb0 0x7fd2300c3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39894/0 (socket says 172.21.15.88:39894) 2023-10-16T20:53:46.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.601+0000 7fd22effd640 1 -- 172.21.15.88:0/1610503280 learned_addr learned my addr 172.21.15.88:0/1610503280 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:46.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.601+0000 7fd22f7fe640 1 --2- 172.21.15.88:0/1610503280 >> v2:172.21.15.191:3300/0 conn(0x7fd230153770 0x7fd230159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:46.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.601+0000 7fd22effd640 1 -- 172.21.15.88:0/1610503280 >> v2:172.21.15.191:3300/0 conn(0x7fd230153770 msgr2=0x7fd230159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:46.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.601+0000 7fd22effd640 1 --2- 172.21.15.88:0/1610503280 >> v2:172.21.15.191:3300/0 conn(0x7fd230153770 0x7fd230159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:46.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.601+0000 7fd22effd640 1 -- 172.21.15.88:0/1610503280 >> v1:172.21.15.191:6789/0 conn(0x7fd2300c41f0 legacy=0x7fd2300cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:46.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.601+0000 7fd22effd640 1 -- 172.21.15.88:0/1610503280 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd230153b50 con 0x7fd2300c9bb0 2023-10-16T20:53:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.605+0000 7fd22effd640 1 --2- 172.21.15.88:0/1610503280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2300c9bb0 0x7fd2300c3cb0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd2300cc7b0 tx=0x7fd22401c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33bc310c41a88e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.605+0000 7fd22e7fc640 1 -- 172.21.15.88:0/1610503280 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2240aa620 con 0x7fd2300c9bb0 2023-10-16T20:53:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.605+0000 7fd22e7fc640 1 -- 172.21.15.88:0/1610503280 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2240aa780 con 0x7fd2300c9bb0 2023-10-16T20:53:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.605+0000 7fd22e7fc640 1 -- 172.21.15.88:0/1610503280 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2240aab10 con 0x7fd2300c9bb0 2023-10-16T20:53:46.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.605+0000 7fd235f0f640 1 -- 172.21.15.88:0/1610503280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2300c9bb0 msgr2=0x7fd2300c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:46.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.605+0000 7fd235f0f640 1 --2- 172.21.15.88:0/1610503280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2300c9bb0 0x7fd2300c3cb0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd2300cc7b0 tx=0x7fd22401c4c0 comp rx=0 tx=0).stop 2023-10-16T20:53:46.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.605+0000 7fd235f0f640 1 -- 172.21.15.88:0/1610503280 shutdown_connections 2023-10-16T20:53:46.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.605+0000 7fd235f0f640 1 --2- 172.21.15.88:0/1610503280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2300c9bb0 0x7fd2300c3cb0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:46.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.605+0000 7fd235f0f640 1 --2- 172.21.15.88:0/1610503280 >> v2:172.21.15.191:3300/0 conn(0x7fd230153770 0x7fd230159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:46.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.605+0000 7fd235f0f640 1 -- 172.21.15.88:0/1610503280 >> 172.21.15.88:0/1610503280 conn(0x7fd2300715c0 msgr2=0x7fd2300719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:46.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.605+0000 7fd235f0f640 1 -- 172.21.15.88:0/1610503280 shutdown_connections 2023-10-16T20:53:46.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.605+0000 7fd235f0f640 1 -- 172.21.15.88:0/1610503280 wait complete. 2023-10-16T20:53:46.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.605+0000 7fd235f0f640 1 Processor -- start 2023-10-16T20:53:46.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.605+0000 7fd235f0f640 1 -- start start 2023-10-16T20:53:46.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd235f0f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2300c41f0 0x7fd230143ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:46.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd235f0f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2300c9bb0 0x7fd230146580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:46.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd235f0f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd230153770 0x7fd230146ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:46.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd235f0f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2300b4180 con 0x7fd2300c9bb0 2023-10-16T20:53:46.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd235f0f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd230144000 con 0x7fd230153770 2023-10-16T20:53:46.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd235f0f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2300b4740 con 0x7fd2300c41f0 2023-10-16T20:53:46.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd22effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2300c9bb0 0x7fd230146580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:46.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd22effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2300c9bb0 0x7fd230146580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39896/0 (socket says 172.21.15.88:39896) 2023-10-16T20:53:46.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd22effd640 1 -- 172.21.15.88:0/3154437164 learned_addr learned my addr 172.21.15.88:0/3154437164 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:46.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd22f7fe640 1 --2- 172.21.15.88:0/3154437164 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2300c41f0 0x7fd230143ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:46.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd22ffff640 1 --2- 172.21.15.88:0/3154437164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd230153770 0x7fd230146ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:46.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd22effd640 1 -- 172.21.15.88:0/3154437164 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2300c41f0 msgr2=0x7fd230143ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:46.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd22effd640 1 --2- 172.21.15.88:0/3154437164 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2300c41f0 0x7fd230143ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:46.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd22effd640 1 -- 172.21.15.88:0/3154437164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd230153770 msgr2=0x7fd230146ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:46.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd22effd640 1 --2- 172.21.15.88:0/3154437164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd230153770 0x7fd230146ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:46.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd22effd640 1 -- 172.21.15.88:0/3154437164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2240094d0 con 0x7fd2300c9bb0 2023-10-16T20:53:46.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd22ffff640 1 --2- 172.21.15.88:0/3154437164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd230153770 0x7fd230146ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:53:46.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd22effd640 1 --2- 172.21.15.88:0/3154437164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2300c9bb0 0x7fd230146580 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd22401a0f0 tx=0x7fd22401ad00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:46.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd217fff640 1 -- 172.21.15.88:0/3154437164 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2240bd3f0 con 0x7fd2300c9bb0 2023-10-16T20:53:46.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd217fff640 1 -- 172.21.15.88:0/3154437164 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2240bd550 con 0x7fd2300c9bb0 2023-10-16T20:53:46.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd235f0f640 1 -- 172.21.15.88:0/3154437164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2300b49f0 con 0x7fd2300c9bb0 2023-10-16T20:53:46.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd217fff640 1 -- 172.21.15.88:0/3154437164 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2240bd900 con 0x7fd2300c9bb0 2023-10-16T20:53:46.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd235f0f640 1 -- 172.21.15.88:0/3154437164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2300b43d0 con 0x7fd2300c9bb0 2023-10-16T20:53:46.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.609+0000 7fd235f0f640 1 -- 172.21.15.88:0/3154437164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd230074700 con 0x7fd2300c9bb0 2023-10-16T20:53:46.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.613+0000 7fd217fff640 1 -- 172.21.15.88:0/3154437164 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd2240b33f0 con 0x7fd2300c9bb0 2023-10-16T20:53:46.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.613+0000 7fd217fff640 1 --2- 172.21.15.88:0/3154437164 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd21c0664a0 0x7fd21c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:46.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.613+0000 7fd217fff640 1 -- 172.21.15.88:0/3154437164 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7fd22411ea50 con 0x7fd2300c9bb0 2023-10-16T20:53:46.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.613+0000 7fd22f7fe640 1 --2- 172.21.15.88:0/3154437164 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd21c0664a0 0x7fd21c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:46.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.613+0000 7fd22f7fe640 1 --2- 172.21.15.88:0/3154437164 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd21c0664a0 0x7fd21c068960 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd2200079a0 tx=0x7fd220008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:46.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.617+0000 7fd217fff640 1 -- 172.21.15.88:0/3154437164 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd2240eb950 con 0x7fd2300c9bb0 2023-10-16T20:53:46.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.829+0000 7fd235f0f640 1 -- 172.21.15.88:0/3154437164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd230150ae0 con 0x7fd2300c9bb0 2023-10-16T20:53:46.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.829+0000 7fd217fff640 1 -- 172.21.15.88:0/3154437164 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5778 (secure 0 0 0) 0x7fd2240eef70 con 0x7fd2300c9bb0 2023-10-16T20:53:46.831 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:46.831 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:53:41.616280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:rejoin","state_seq":30,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:46.832 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 92 2023-10-16T20:53:46.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.829+0000 7fd235f0f640 1 -- 172.21.15.88:0/3154437164 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd21c0664a0 msgr2=0x7fd21c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:46.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.829+0000 7fd235f0f640 1 --2- 172.21.15.88:0/3154437164 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd21c0664a0 0x7fd21c068960 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd2200079a0 tx=0x7fd220008040 comp rx=0 tx=0).stop 2023-10-16T20:53:46.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.829+0000 7fd235f0f640 1 -- 172.21.15.88:0/3154437164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2300c9bb0 msgr2=0x7fd230146580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:46.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.829+0000 7fd235f0f640 1 --2- 172.21.15.88:0/3154437164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2300c9bb0 0x7fd230146580 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd22401a0f0 tx=0x7fd22401ad00 comp rx=0 tx=0).stop 2023-10-16T20:53:46.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.833+0000 7fd235f0f640 1 -- 172.21.15.88:0/3154437164 shutdown_connections 2023-10-16T20:53:46.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.833+0000 7fd235f0f640 1 --2- 172.21.15.88:0/3154437164 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd21c0664a0 0x7fd21c068960 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.833+0000 7fd235f0f640 1 --2- 172.21.15.88:0/3154437164 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd230153770 0x7fd230146ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.833+0000 7fd235f0f640 1 --2- 172.21.15.88:0/3154437164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2300c9bb0 0x7fd230146580 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.833+0000 7fd235f0f640 1 --2- 172.21.15.88:0/3154437164 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd2300c41f0 0x7fd230143ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.833+0000 7fd235f0f640 1 -- 172.21.15.88:0/3154437164 >> 172.21.15.88:0/3154437164 conn(0x7fd2300715c0 msgr2=0x7fd2300cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.833+0000 7fd235f0f640 1 -- 172.21.15.88:0/3154437164 shutdown_connections 2023-10-16T20:53:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:46.833+0000 7fd235f0f640 1 -- 172.21.15.88:0/3154437164 wait complete. 2023-10-16T20:53:46.856 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:53:48.857 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:49.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.053+0000 7f36ad5d9640 1 Processor -- start 2023-10-16T20:53:49.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.053+0000 7f36ad5d9640 1 -- start start 2023-10-16T20:53:49.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.053+0000 7f36ad5d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36a80cadb0 0x7f36a80cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:49.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.053+0000 7f36ad5d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f36a806e720 0x7f36a806eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:49.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.053+0000 7f36ad5d9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a81536f0 con 0x7f36a8153310 2023-10-16T20:53:49.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.053+0000 7f36ad5d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a80cc160 con 0x7f36a806e720 2023-10-16T20:53:49.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.053+0000 7f36ad5d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a80c5630 con 0x7f36a80cadb0 2023-10-16T20:53:49.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.053+0000 7f36a7fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f36a806e720 0x7f36a806eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:49.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.053+0000 7f36a7fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f36a806e720 0x7f36a806eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39940/0 (socket says 172.21.15.88:39940) 2023-10-16T20:53:49.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.053+0000 7f36a7fff640 1 -- 172.21.15.88:0/1030457831 learned_addr learned my addr 172.21.15.88:0/1030457831 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:49.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.053+0000 7f36a67fc640 1 --2- 172.21.15.88:0/1030457831 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36a80cadb0 0x7f36a80cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:49.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.053+0000 7f36a7fff640 1 -- 172.21.15.88:0/1030457831 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36a80cadb0 msgr2=0x7f36a80cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:49.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.053+0000 7f36a7fff640 1 --2- 172.21.15.88:0/1030457831 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36a80cadb0 0x7f36a80cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:49.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.053+0000 7f36a7fff640 1 -- 172.21.15.88:0/1030457831 >> v1:172.21.15.191:6789/0 conn(0x7f36a8153310 legacy=0x7f36a8159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:49.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.053+0000 7f36a7fff640 1 -- 172.21.15.88:0/1030457831 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a80c5910 con 0x7f36a806e720 2023-10-16T20:53:49.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.057+0000 7f36a7fff640 1 --2- 172.21.15.88:0/1030457831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a806e720 0x7f36a806eb90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009e50 tx=0x7f36a001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a656a97c6dd5723 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:49.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.057+0000 7f36a5ffb640 1 -- 172.21.15.88:0/1030457831 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a00af400 con 0x7f36a806e720 2023-10-16T20:53:49.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.057+0000 7f36a5ffb640 1 -- 172.21.15.88:0/1030457831 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a00ab050 con 0x7f36a806e720 2023-10-16T20:53:49.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.057+0000 7f36a5ffb640 1 -- 172.21.15.88:0/1030457831 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a00af780 con 0x7f36a806e720 2023-10-16T20:53:49.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.057+0000 7f36ad5d9640 1 -- 172.21.15.88:0/1030457831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a806e720 msgr2=0x7f36a806eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:49.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.057+0000 7f36ad5d9640 1 --2- 172.21.15.88:0/1030457831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a806e720 0x7f36a806eb90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009e50 tx=0x7f36a001f1f0 comp rx=0 tx=0).stop 2023-10-16T20:53:49.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.057+0000 7f36ad5d9640 1 -- 172.21.15.88:0/1030457831 shutdown_connections 2023-10-16T20:53:49.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.057+0000 7f36ad5d9640 1 --2- 172.21.15.88:0/1030457831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a806e720 0x7f36a806eb90 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:49.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.057+0000 7f36ad5d9640 1 --2- 172.21.15.88:0/1030457831 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36a80cadb0 0x7f36a80cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:49.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.057+0000 7f36ad5d9640 1 -- 172.21.15.88:0/1030457831 >> 172.21.15.88:0/1030457831 conn(0x7f36a806cf80 msgr2=0x7f36a80d0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:49.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.057+0000 7f36ad5d9640 1 -- 172.21.15.88:0/1030457831 shutdown_connections 2023-10-16T20:53:49.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.057+0000 7f36ad5d9640 1 -- 172.21.15.88:0/1030457831 wait complete. 2023-10-16T20:53:49.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36ad5d9640 1 Processor -- start 2023-10-16T20:53:49.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36ad5d9640 1 -- start start 2023-10-16T20:53:49.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36ad5d9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36a806e720 0x7f36a814be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:49.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36ad5d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a80cadb0 0x7f36a814b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:49.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36ad5d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36a8153310 0x7f36a8149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:49.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36ad5d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a806b270 con 0x7f36a80cadb0 2023-10-16T20:53:49.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36ad5d9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a806c0b0 con 0x7f36a806e720 2023-10-16T20:53:49.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36ad5d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a806c370 con 0x7f36a8153310 2023-10-16T20:53:49.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36a67fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a80cadb0 0x7f36a814b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:49.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36a67fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a80cadb0 0x7f36a814b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39950/0 (socket says 172.21.15.88:39950) 2023-10-16T20:53:49.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36a67fc640 1 -- 172.21.15.88:0/4058614075 learned_addr learned my addr 172.21.15.88:0/4058614075 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:49.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36a7fff640 1 --2- 172.21.15.88:0/4058614075 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36a8153310 0x7f36a8149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:49.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36a6ffd640 1 --2- 172.21.15.88:0/4058614075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36a806e720 0x7f36a814be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:49.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36a7fff640 1 -- 172.21.15.88:0/4058614075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36a806e720 msgr2=0x7f36a814be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:49.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36a7fff640 1 --2- 172.21.15.88:0/4058614075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36a806e720 0x7f36a814be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:49.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36a7fff640 1 -- 172.21.15.88:0/4058614075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a80cadb0 msgr2=0x7f36a814b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:49.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36a7fff640 1 --2- 172.21.15.88:0/4058614075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a80cadb0 0x7f36a814b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:49.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36a7fff640 1 -- 172.21.15.88:0/4058614075 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a0009b30 con 0x7f36a8153310 2023-10-16T20:53:49.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36a67fc640 1 --2- 172.21.15.88:0/4058614075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a80cadb0 0x7f36a814b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:53:49.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36a7fff640 1 --2- 172.21.15.88:0/4058614075 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36a8153310 0x7f36a8149560 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f36a00ae000 tx=0x7f36a001bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:49.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f368f7fe640 1 -- 172.21.15.88:0/4058614075 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a00c0070 con 0x7f36a8153310 2023-10-16T20:53:49.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f368f7fe640 1 -- 172.21.15.88:0/4058614075 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a0009510 con 0x7f36a8153310 2023-10-16T20:53:49.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36ad5d9640 1 -- 172.21.15.88:0/4058614075 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a806a940 con 0x7f36a8153310 2023-10-16T20:53:49.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f368f7fe640 1 -- 172.21.15.88:0/4058614075 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a00afe80 con 0x7f36a8153310 2023-10-16T20:53:49.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.061+0000 7f36ad5d9640 1 -- 172.21.15.88:0/4058614075 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a806af00 con 0x7f36a8153310 2023-10-16T20:53:49.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.065+0000 7f36ad5d9640 1 -- 172.21.15.88:0/4058614075 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a806b3e0 con 0x7f36a8153310 2023-10-16T20:53:49.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.065+0000 7f368f7fe640 1 -- 172.21.15.88:0/4058614075 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f36a00b6040 con 0x7f36a8153310 2023-10-16T20:53:49.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.065+0000 7f368f7fe640 1 --2- 172.21.15.88:0/4058614075 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f36940664a0 0x7f3694068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:49.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.065+0000 7f368f7fe640 1 -- 172.21.15.88:0/4058614075 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f36a0121b50 con 0x7f36a8153310 2023-10-16T20:53:49.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.065+0000 7f36a6ffd640 1 --2- 172.21.15.88:0/4058614075 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f36940664a0 0x7f3694068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:49.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.065+0000 7f36a6ffd640 1 --2- 172.21.15.88:0/4058614075 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f36940664a0 0x7f3694068960 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3698007fd0 tx=0x7f3698009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:49.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.069+0000 7f368f7fe640 1 -- 172.21.15.88:0/4058614075 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f36a00eeaf0 con 0x7f36a8153310 2023-10-16T20:53:49.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.293+0000 7f36ad5d9640 1 -- 172.21.15.88:0/4058614075 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36a8074700 con 0x7f36a8153310 2023-10-16T20:53:49.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.297+0000 7f368f7fe640 1 -- 172.21.15.88:0/4058614075 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5778 (secure 0 0 0) 0x7f36a00f2110 con 0x7f36a8153310 2023-10-16T20:53:49.298 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:49.298 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:53:47.037731+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:49.298 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 93 2023-10-16T20:53:49.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.297+0000 7f36ad5d9640 1 -- 172.21.15.88:0/4058614075 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f36940664a0 msgr2=0x7f3694068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:49.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.297+0000 7f36ad5d9640 1 --2- 172.21.15.88:0/4058614075 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f36940664a0 0x7f3694068960 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3698007fd0 tx=0x7f3698009040 comp rx=0 tx=0).stop 2023-10-16T20:53:49.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.297+0000 7f36ad5d9640 1 -- 172.21.15.88:0/4058614075 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36a8153310 msgr2=0x7f36a8149560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:49.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.297+0000 7f36ad5d9640 1 --2- 172.21.15.88:0/4058614075 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36a8153310 0x7f36a8149560 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f36a00ae000 tx=0x7f36a001bf90 comp rx=0 tx=0).stop 2023-10-16T20:53:49.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.297+0000 7f36ad5d9640 1 -- 172.21.15.88:0/4058614075 shutdown_connections 2023-10-16T20:53:49.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.297+0000 7f36ad5d9640 1 --2- 172.21.15.88:0/4058614075 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f36940664a0 0x7f3694068960 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:49.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.297+0000 7f36ad5d9640 1 --2- 172.21.15.88:0/4058614075 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f36a8153310 0x7f36a8149560 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:49.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.297+0000 7f36ad5d9640 1 --2- 172.21.15.88:0/4058614075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36a80cadb0 0x7f36a814b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:49.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.297+0000 7f36ad5d9640 1 --2- 172.21.15.88:0/4058614075 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f36a806e720 0x7f36a814be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:49.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.297+0000 7f36ad5d9640 1 -- 172.21.15.88:0/4058614075 >> 172.21.15.88:0/4058614075 conn(0x7f36a806cf80 msgr2=0x7f36a80cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:49.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.297+0000 7f36ad5d9640 1 -- 172.21.15.88:0/4058614075 shutdown_connections 2023-10-16T20:53:49.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:49.297+0000 7f36ad5d9640 1 -- 172.21.15.88:0/4058614075 wait complete. 2023-10-16T20:53:49.324 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=4, replacing gid=4889 2023-10-16T20:53:49.325 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.b 2023-10-16T20:53:52.325 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-10-16T20:53:52.326 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-16T20:53:52.326 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-16T20:53:52.330 INFO:tasks.ceph.mds.b:Started 2023-10-16T20:53:52.330 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:53:52.421 INFO:tasks.ceph.mds.b.smithi191.stdout:starting mds.b at 2023-10-16T20:53:52.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40cad72640 1 Processor -- start 2023-10-16T20:53:52.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40cad72640 1 -- start start 2023-10-16T20:53:52.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40cad72640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f40c4150280 0x7f40c40c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:52.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40cad72640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f40c4150660 0x7f40c40c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:52.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40cad72640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40c40c6bc0 0x7f40c406d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:52.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40cad72640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c40c5c60 con 0x7f40c4150280 2023-10-16T20:53:52.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40cad72640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c40c5370 con 0x7f40c4150660 2023-10-16T20:53:52.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40cad72640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c4156d10 con 0x7f40c40c6bc0 2023-10-16T20:53:52.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40c8b0e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f40c4150280 0x7f40c40c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:52.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40c8b0e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f40c4150280 0x7f40c40c60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39952/0 (socket says 172.21.15.88:39952) 2023-10-16T20:53:52.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40c8b0e640 1 -- 172.21.15.88:0/1962968461 learned_addr learned my addr 172.21.15.88:0/1962968461 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:52.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40c930f640 1 --2- 172.21.15.88:0/1962968461 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40c40c6bc0 0x7f40c406d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:52.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40c3fff640 1 --2- 172.21.15.88:0/1962968461 >> v2:172.21.15.191:3300/0 conn(0x7f40c4150660 0x7f40c40c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:52.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40c8b0e640 1 -- 172.21.15.88:0/1962968461 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40c40c6bc0 msgr2=0x7f40c406d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:52.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40c8b0e640 1 --2- 172.21.15.88:0/1962968461 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40c40c6bc0 0x7f40c406d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:52.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40c8b0e640 1 -- 172.21.15.88:0/1962968461 >> v2:172.21.15.191:3300/0 conn(0x7f40c4150660 msgr2=0x7f40c40c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:53:52.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40c8b0e640 1 --2- 172.21.15.88:0/1962968461 >> v2:172.21.15.191:3300/0 conn(0x7f40c4150660 0x7f40c40c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:52.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.485+0000 7f40c8b0e640 1 -- 172.21.15.88:0/1962968461 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c40c5610 con 0x7f40c4150280 2023-10-16T20:53:52.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.489+0000 7f40c8b0e640 1 --2- 172.21.15.88:0/1962968461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40c4150280 0x7f40c40c60f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f40c40c1290 tx=0x7f40b801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca86eca19652947 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:52.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.489+0000 7f40c37fe640 1 -- 172.21.15.88:0/1962968461 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b80ac600 con 0x7f40c4150280 2023-10-16T20:53:52.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.489+0000 7f40c37fe640 1 -- 172.21.15.88:0/1962968461 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b80a8050 con 0x7f40c4150280 2023-10-16T20:53:52.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.489+0000 7f40c37fe640 1 -- 172.21.15.88:0/1962968461 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b80ac980 con 0x7f40c4150280 2023-10-16T20:53:52.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.489+0000 7f40cad72640 1 -- 172.21.15.88:0/1962968461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40c4150280 msgr2=0x7f40c40c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:52.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.489+0000 7f40cad72640 1 --2- 172.21.15.88:0/1962968461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40c4150280 0x7f40c40c60f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f40c40c1290 tx=0x7f40b801c4d0 comp rx=0 tx=0).stop 2023-10-16T20:53:52.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.489+0000 7f40cad72640 1 -- 172.21.15.88:0/1962968461 shutdown_connections 2023-10-16T20:53:52.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.489+0000 7f40cad72640 1 --2- 172.21.15.88:0/1962968461 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40c40c6bc0 0x7f40c406d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.489+0000 7f40cad72640 1 --2- 172.21.15.88:0/1962968461 >> v2:172.21.15.191:3300/0 conn(0x7f40c4150660 0x7f40c40c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.489+0000 7f40cad72640 1 --2- 172.21.15.88:0/1962968461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40c4150280 0x7f40c40c60f0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.489+0000 7f40cad72640 1 -- 172.21.15.88:0/1962968461 >> 172.21.15.88:0/1962968461 conn(0x7f40c40cad50 msgr2=0x7f40c40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.489+0000 7f40cad72640 1 -- 172.21.15.88:0/1962968461 shutdown_connections 2023-10-16T20:53:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.493+0000 7f40cad72640 1 -- 172.21.15.88:0/1962968461 wait complete. 2023-10-16T20:53:52.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.493+0000 7f40cad72640 1 Processor -- start 2023-10-16T20:53:52.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.493+0000 7f40cad72640 1 -- start start 2023-10-16T20:53:52.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.493+0000 7f40cad72640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40c40c6bc0 0x7f40c4146470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:52.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.493+0000 7f40cad72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40c4150280 0x7f40c41f22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:52.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.493+0000 7f40cad72640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40c4150660 0x7f40c41f2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:52.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.493+0000 7f40cad72640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c4158e50 con 0x7f40c4150280 2023-10-16T20:53:52.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.493+0000 7f40cad72640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c41f1e50 con 0x7f40c4150660 2023-10-16T20:53:52.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.493+0000 7f40cad72640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c415e300 con 0x7f40c40c6bc0 2023-10-16T20:53:52.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.493+0000 7f40c3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40c4150280 0x7f40c41f22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:52.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.493+0000 7f40c3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40c4150280 0x7f40c41f22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39960/0 (socket says 172.21.15.88:39960) 2023-10-16T20:53:52.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.493+0000 7f40c3fff640 1 -- 172.21.15.88:0/61531774 learned_addr learned my addr 172.21.15.88:0/61531774 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:53:52.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.493+0000 7f40c8b0e640 1 --2- 172.21.15.88:0/61531774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40c40c6bc0 0x7f40c4146470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:52.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.493+0000 7f40c930f640 1 --2- 172.21.15.88:0/61531774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40c4150660 0x7f40c41f2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:52.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.493+0000 7f40c3fff640 1 -- 172.21.15.88:0/61531774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40c40c6bc0 msgr2=0x7f40c4146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:52.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.493+0000 7f40c3fff640 1 --2- 172.21.15.88:0/61531774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40c40c6bc0 0x7f40c4146470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.497+0000 7f40c3fff640 1 -- 172.21.15.88:0/61531774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40c4150660 msgr2=0x7f40c41f2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.497+0000 7f40c3fff640 1 --2- 172.21.15.88:0/61531774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40c4150660 0x7f40c41f2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.497+0000 7f40c3fff640 1 -- 172.21.15.88:0/61531774 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b80094d0 con 0x7f40c4150280 2023-10-16T20:53:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.497+0000 7f40c8b0e640 1 --2- 172.21.15.88:0/61531774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40c40c6bc0 0x7f40c4146470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:53:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.497+0000 7f40c3fff640 1 --2- 172.21.15.88:0/61531774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40c4150280 0x7f40c41f22e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f40b000e8a0 tx=0x7f40b000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:52.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.497+0000 7f40c17fa640 1 -- 172.21.15.88:0/61531774 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0009510 con 0x7f40c4150280 2023-10-16T20:53:52.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.497+0000 7f40c17fa640 1 -- 172.21.15.88:0/61531774 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b000c940 con 0x7f40c4150280 2023-10-16T20:53:52.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.497+0000 7f40c17fa640 1 -- 172.21.15.88:0/61531774 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b000ccb0 con 0x7f40c4150280 2023-10-16T20:53:52.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.497+0000 7f40cad72640 1 -- 172.21.15.88:0/61531774 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c415e5e0 con 0x7f40c4150280 2023-10-16T20:53:52.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.497+0000 7f40cad72640 1 -- 172.21.15.88:0/61531774 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c4159380 con 0x7f40c4150280 2023-10-16T20:53:52.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.497+0000 7f40cad72640 1 -- 172.21.15.88:0/61531774 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c41f2000 con 0x7f40c4150280 2023-10-16T20:53:52.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.497+0000 7f40c17fa640 1 -- 172.21.15.88:0/61531774 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f40b0013470 con 0x7f40c4150280 2023-10-16T20:53:52.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.497+0000 7f40c17fa640 1 --2- 172.21.15.88:0/61531774 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f40a80664d0 0x7f40a8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:53:52.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.497+0000 7f40c17fa640 1 -- 172.21.15.88:0/61531774 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f40b0084460 con 0x7f40c4150280 2023-10-16T20:53:52.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.501+0000 7f40c8b0e640 1 --2- 172.21.15.88:0/61531774 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f40a80664d0 0x7f40a8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:53:52.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.501+0000 7f40c8b0e640 1 --2- 172.21.15.88:0/61531774 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f40a80664d0 0x7f40a8068990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f40b80ae000 tx=0x7f40b80a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:53:52.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.505+0000 7f40c17fa640 1 -- 172.21.15.88:0/61531774 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f40c41f2000 con 0x7f40c4150280 2023-10-16T20:53:52.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.733+0000 7f40cad72640 1 -- 172.21.15.88:0/61531774 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40c41f2000 con 0x7f40c4150280 2023-10-16T20:53:52.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.733+0000 7f40c17fa640 1 -- 172.21.15.88:0/61531774 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6571 (secure 0 0 0) 0x7f40c41f2000 con 0x7f40c4150280 2023-10-16T20:53:52.735 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:53:52.735 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5477,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:53:47.037731+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:53:52.736 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 94 2023-10-16T20:53:52.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.733+0000 7f40cad72640 1 -- 172.21.15.88:0/61531774 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f40a80664d0 msgr2=0x7f40a8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:52.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.733+0000 7f40cad72640 1 --2- 172.21.15.88:0/61531774 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f40a80664d0 0x7f40a8068990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f40b80ae000 tx=0x7f40b80a7000 comp rx=0 tx=0).stop 2023-10-16T20:53:52.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.733+0000 7f40cad72640 1 -- 172.21.15.88:0/61531774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40c4150280 msgr2=0x7f40c41f22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:53:52.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.733+0000 7f40cad72640 1 --2- 172.21.15.88:0/61531774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40c4150280 0x7f40c41f22e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f40b000e8a0 tx=0x7f40b000eca0 comp rx=0 tx=0).stop 2023-10-16T20:53:52.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.733+0000 7f40cad72640 1 -- 172.21.15.88:0/61531774 shutdown_connections 2023-10-16T20:53:52.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.733+0000 7f40cad72640 1 --2- 172.21.15.88:0/61531774 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f40a80664d0 0x7f40a8068990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:52.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.733+0000 7f40cad72640 1 --2- 172.21.15.88:0/61531774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f40c4150660 0x7f40c41f2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:52.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.733+0000 7f40cad72640 1 --2- 172.21.15.88:0/61531774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40c4150280 0x7f40c41f22e0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:52.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.733+0000 7f40cad72640 1 --2- 172.21.15.88:0/61531774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f40c40c6bc0 0x7f40c4146470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:53:52.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.733+0000 7f40cad72640 1 -- 172.21.15.88:0/61531774 >> 172.21.15.88:0/61531774 conn(0x7f40c40cad50 msgr2=0x7f40c4155840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:53:52.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.737+0000 7f40cad72640 1 -- 172.21.15.88:0/61531774 shutdown_connections 2023-10-16T20:53:52.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:53:52.737+0000 7f40cad72640 1 -- 172.21.15.88:0/61531774 wait complete. 2023-10-16T20:53:52.758 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-10-16T20:53:52.759 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 104 secs before thrashing 2023-10-16T20:53:59.772 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:53:59.776 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:54:29.834 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:54:29.839 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:54:59.915 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:54:59.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:55:29.969 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:55:29.974 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:55:36.760 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:55:36.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.900+0000 7f5651467640 1 Processor -- start 2023-10-16T20:55:36.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.900+0000 7f5651467640 1 -- start start 2023-10-16T20:55:36.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.904+0000 7f5651467640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f564c0c9e90 0x7f564c0c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:36.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.904+0000 7f5651467640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f564c0c6620 0x7f564c161a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:36.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.904+0000 7f5651467640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f564c157a90 con 0x7f564c1405e0 2023-10-16T20:55:36.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.904+0000 7f5651467640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564c068370 con 0x7f564c0c6620 2023-10-16T20:55:36.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.904+0000 7f5651467640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564c0c5540 con 0x7f564c0c9e90 2023-10-16T20:55:36.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.904+0000 7f5650c66640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f564c0c6620 0x7f564c161a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:36.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.904+0000 7f564bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f564c1405e0 0x7f564c071730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:38816/0 (socket says 172.21.15.88:38816) 2023-10-16T20:55:36.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.904+0000 7f564bfff640 1 -- 172.21.15.88:0/2912885709 learned_addr learned my addr 172.21.15.88:0/2912885709 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:55:36.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.904+0000 7f564b7fe640 1 --2- 172.21.15.88:0/2912885709 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f564c0c9e90 0x7f564c0c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:36.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.904+0000 7f5650c66640 1 -- 172.21.15.88:0/2912885709 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f564c0c9e90 msgr2=0x7f564c0c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:55:36.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.904+0000 7f5650c66640 1 --2- 172.21.15.88:0/2912885709 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f564c0c9e90 0x7f564c0c60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:36.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.904+0000 7f5650c66640 1 -- 172.21.15.88:0/2912885709 >> v1:172.21.15.88:6789/0 conn(0x7f564c1405e0 legacy=0x7f564c071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:36.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.904+0000 7f5650c66640 1 -- 172.21.15.88:0/2912885709 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c0c4780 con 0x7f564c0c6620 2023-10-16T20:55:36.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.904+0000 7f564affd640 1 -- 172.21.15.88:0/2912885709 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859757118 0 0) 0x7f564c157a90 con 0x7f564c1405e0 2023-10-16T20:55:36.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.904+0000 7f5650c66640 1 --2- 172.21.15.88:0/2912885709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f564c0c6620 0x7f564c161a70 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5644009d90 tx=0x7f564401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae17125f2114ac39 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:36.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f564affd640 1 -- 172.21.15.88:0/2912885709 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56440a5900 con 0x7f564c0c6620 2023-10-16T20:55:36.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f564affd640 1 -- 172.21.15.88:0/2912885709 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56440a5a60 con 0x7f564c0c6620 2023-10-16T20:55:36.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f564affd640 1 -- 172.21.15.88:0/2912885709 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56440a5df0 con 0x7f564c0c6620 2023-10-16T20:55:36.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f5651467640 1 -- 172.21.15.88:0/2912885709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f564c0c6620 msgr2=0x7f564c161a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:36.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f5651467640 1 --2- 172.21.15.88:0/2912885709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f564c0c6620 0x7f564c161a70 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5644009d90 tx=0x7f564401c950 comp rx=0 tx=0).stop 2023-10-16T20:55:36.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f5651467640 1 -- 172.21.15.88:0/2912885709 shutdown_connections 2023-10-16T20:55:36.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f5651467640 1 --2- 172.21.15.88:0/2912885709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f564c0c6620 0x7f564c161a70 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:36.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f5651467640 1 --2- 172.21.15.88:0/2912885709 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f564c0c9e90 0x7f564c0c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:36.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f5651467640 1 -- 172.21.15.88:0/2912885709 >> 172.21.15.88:0/2912885709 conn(0x7f564c0c9370 msgr2=0x7f564c0c9780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:36.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f5651467640 1 -- 172.21.15.88:0/2912885709 shutdown_connections 2023-10-16T20:55:36.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f5651467640 1 -- 172.21.15.88:0/2912885709 wait complete. 2023-10-16T20:55:36.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f5651467640 1 Processor -- start 2023-10-16T20:55:36.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f5651467640 1 -- start start 2023-10-16T20:55:36.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f5651467640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f564c0c6620 0x7f564c0c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:36.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f5651467640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f564c0c9e90 0x7f564c1f0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:36.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f5651467640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f564c1405e0 0x7f564c154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:36.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f5651467640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c151c40 con 0x7f564c0c9e90 2023-10-16T20:55:36.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f5651467640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c1333b0 con 0x7f564c1405e0 2023-10-16T20:55:36.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f5651467640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564c0c7090 con 0x7f564c0c6620 2023-10-16T20:55:36.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f564b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f564c0c9e90 0x7f564c1f0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:36.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f564b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f564c0c9e90 0x7f564c1f0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56234/0 (socket says 172.21.15.88:56234) 2023-10-16T20:55:36.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.908+0000 7f564b7fe640 1 -- 172.21.15.88:0/2903284527 learned_addr learned my addr 172.21.15.88:0/2903284527 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:36.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f564bfff640 1 --2- 172.21.15.88:0/2903284527 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f564c0c6620 0x7f564c0c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:36.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f5650c66640 1 --2- 172.21.15.88:0/2903284527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f564c1405e0 0x7f564c154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:36.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f564b7fe640 1 -- 172.21.15.88:0/2903284527 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f564c0c6620 msgr2=0x7f564c0c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:36.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f564b7fe640 1 --2- 172.21.15.88:0/2903284527 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f564c0c6620 0x7f564c0c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:36.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f564b7fe640 1 -- 172.21.15.88:0/2903284527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f564c1405e0 msgr2=0x7f564c154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:36.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f564b7fe640 1 --2- 172.21.15.88:0/2903284527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f564c1405e0 0x7f564c154230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:36.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f564b7fe640 1 -- 172.21.15.88:0/2903284527 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5644009a70 con 0x7f564c0c9e90 2023-10-16T20:55:36.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f564bfff640 1 --2- 172.21.15.88:0/2903284527 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f564c0c6620 0x7f564c0c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:55:36.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f5650c66640 1 --2- 172.21.15.88:0/2903284527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f564c1405e0 0x7f564c154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:55:36.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f564b7fe640 1 --2- 172.21.15.88:0/2903284527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f564c0c9e90 0x7f564c1f0070 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f564000e8d0 tx=0x7f564000ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:36.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f5648ff9640 1 -- 172.21.15.88:0/2903284527 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564000c940 con 0x7f564c0c9e90 2023-10-16T20:55:36.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f5648ff9640 1 -- 172.21.15.88:0/2903284527 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564000caa0 con 0x7f564c0c9e90 2023-10-16T20:55:36.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f5651467640 1 -- 172.21.15.88:0/2903284527 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564c0c7370 con 0x7f564c0c9e90 2023-10-16T20:55:36.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f5648ff9640 1 -- 172.21.15.88:0/2903284527 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564000ce10 con 0x7f564c0c9e90 2023-10-16T20:55:36.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f5651467640 1 -- 172.21.15.88:0/2903284527 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f564c1efb50 con 0x7f564c0c9e90 2023-10-16T20:55:36.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f5651467640 1 -- 172.21.15.88:0/2903284527 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f564c06a8a0 con 0x7f564c0c9e90 2023-10-16T20:55:36.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f5648ff9640 1 -- 172.21.15.88:0/2903284527 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5640013470 con 0x7f564c0c9e90 2023-10-16T20:55:36.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f5648ff9640 1 --2- 172.21.15.88:0/2903284527 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56380664d0 0x7f5638068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:36.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f5648ff9640 1 -- 172.21.15.88:0/2903284527 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f5640084680 con 0x7f564c0c9e90 2023-10-16T20:55:36.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.912+0000 7f564bfff640 1 --2- 172.21.15.88:0/2903284527 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56380664d0 0x7f5638068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:36.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.916+0000 7f564bfff640 1 --2- 172.21.15.88:0/2903284527 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56380664d0 0x7f5638068990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f563c007fd0 tx=0x7f563c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:36.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:36.916+0000 7f5648ff9640 1 -- 172.21.15.88:0/2903284527 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5640051440 con 0x7f564c0c9e90 2023-10-16T20:55:37.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.100+0000 7f5651467640 1 -- 172.21.15.88:0/2903284527 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f564c04e9c0 con 0x7f564c0c9e90 2023-10-16T20:55:37.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.100+0000 7f5648ff9640 1 -- 172.21.15.88:0/2903284527 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6569 (secure 0 0 0) 0x7f5640054a60 con 0x7f564c0c9e90 2023-10-16T20:55:37.098 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:55:37.098 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5477,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:54:28.552665+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:55:37.098 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 96 2023-10-16T20:55:37.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.100+0000 7f5651467640 1 -- 172.21.15.88:0/2903284527 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56380664d0 msgr2=0x7f5638068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:37.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.100+0000 7f5651467640 1 --2- 172.21.15.88:0/2903284527 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56380664d0 0x7f5638068990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f563c007fd0 tx=0x7f563c009040 comp rx=0 tx=0).stop 2023-10-16T20:55:37.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.100+0000 7f5651467640 1 -- 172.21.15.88:0/2903284527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f564c0c9e90 msgr2=0x7f564c1f0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:37.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.100+0000 7f5651467640 1 --2- 172.21.15.88:0/2903284527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f564c0c9e90 0x7f564c1f0070 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f564000e8d0 tx=0x7f564000ed90 comp rx=0 tx=0).stop 2023-10-16T20:55:37.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.100+0000 7f5651467640 1 -- 172.21.15.88:0/2903284527 shutdown_connections 2023-10-16T20:55:37.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.100+0000 7f5651467640 1 --2- 172.21.15.88:0/2903284527 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f56380664d0 0x7f5638068990 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:37.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.100+0000 7f5651467640 1 --2- 172.21.15.88:0/2903284527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f564c1405e0 0x7f564c154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:37.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.100+0000 7f5651467640 1 --2- 172.21.15.88:0/2903284527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f564c0c9e90 0x7f564c1f0070 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:37.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.100+0000 7f5651467640 1 --2- 172.21.15.88:0/2903284527 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f564c0c6620 0x7f564c0c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:37.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.100+0000 7f5651467640 1 -- 172.21.15.88:0/2903284527 >> 172.21.15.88:0/2903284527 conn(0x7f564c0c9370 msgr2=0x7f564c0c9750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:37.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.100+0000 7f5651467640 1 -- 172.21.15.88:0/2903284527 shutdown_connections 2023-10-16T20:55:37.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.100+0000 7f5651467640 1 -- 172.21.15.88:0/2903284527 wait complete. 2023-10-16T20:55:37.118 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2023-10-16T20:55:37.119 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T20:55:37.119 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T20:55:37.134 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T20:55:37.135 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:55:37.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.296+0000 7fe895a91640 1 Processor -- start 2023-10-16T20:55:37.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.296+0000 7fe895a91640 1 -- start start 2023-10-16T20:55:37.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.300+0000 7fe895a91640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe890153770 0x7fe890159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:37.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.300+0000 7fe895a91640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe8900cece0 0x7fe8900cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:37.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.300+0000 7fe895a91640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89015a410 con 0x7fe8900c9bb0 2023-10-16T20:55:37.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.300+0000 7fe895a91640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8900cc250 con 0x7fe8900cece0 2023-10-16T20:55:37.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.300+0000 7fe895a91640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8900cc670 con 0x7fe890153770 2023-10-16T20:55:37.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.300+0000 7fe88f7fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe890153770 0x7fe890159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:37.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.300+0000 7fe88ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe8900cece0 0x7fe8900cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:37.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.300+0000 7fe88f7fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe890153770 0x7fe890159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:52230/0 (socket says 172.21.15.88:52230) 2023-10-16T20:55:37.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.300+0000 7fe88f7fe640 1 -- 172.21.15.88:0/2681259373 learned_addr learned my addr 172.21.15.88:0/2681259373 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:37.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.300+0000 7fe88ffff640 1 -- 172.21.15.88:0/2681259373 >> v2:172.21.15.191:3300/0 conn(0x7fe890153770 msgr2=0x7fe890159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:55:37.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.300+0000 7fe88ffff640 1 --2- 172.21.15.88:0/2681259373 >> v2:172.21.15.191:3300/0 conn(0x7fe890153770 0x7fe890159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:37.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.300+0000 7fe88ffff640 1 -- 172.21.15.88:0/2681259373 >> v1:172.21.15.88:6789/0 conn(0x7fe8900c9bb0 legacy=0x7fe8900c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:37.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.300+0000 7fe88ffff640 1 -- 172.21.15.88:0/2681259373 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe890153b50 con 0x7fe8900cece0 2023-10-16T20:55:37.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.300+0000 7fe88f7fe640 1 --2- 172.21.15.88:0/2681259373 >> v2:172.21.15.191:3300/0 conn(0x7fe890153770 0x7fe890159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:55:37.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.304+0000 7fe88ffff640 1 --2- 172.21.15.88:0/2681259373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8900cece0 0x7fe8900cf130 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fe8900cc7b0 tx=0x7fe88801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27e9c0a02727d1a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:37.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.304+0000 7fe88e7fc640 1 -- 172.21.15.88:0/2681259373 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8880a59d0 con 0x7fe8900cece0 2023-10-16T20:55:37.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.304+0000 7fe88e7fc640 1 -- 172.21.15.88:0/2681259373 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8880a5b30 con 0x7fe8900cece0 2023-10-16T20:55:37.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.304+0000 7fe88e7fc640 1 -- 172.21.15.88:0/2681259373 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8880ad440 con 0x7fe8900cece0 2023-10-16T20:55:37.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.304+0000 7fe895a91640 1 -- 172.21.15.88:0/2681259373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8900cece0 msgr2=0x7fe8900cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:37.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.304+0000 7fe895a91640 1 --2- 172.21.15.88:0/2681259373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8900cece0 0x7fe8900cf130 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fe8900cc7b0 tx=0x7fe88801c910 comp rx=0 tx=0).stop 2023-10-16T20:55:37.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.304+0000 7fe895a91640 1 -- 172.21.15.88:0/2681259373 shutdown_connections 2023-10-16T20:55:37.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.304+0000 7fe895a91640 1 --2- 172.21.15.88:0/2681259373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8900cece0 0x7fe8900cf130 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:37.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.304+0000 7fe895a91640 1 --2- 172.21.15.88:0/2681259373 >> v2:172.21.15.191:3300/0 conn(0x7fe890153770 0x7fe890159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:37.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.304+0000 7fe895a91640 1 -- 172.21.15.88:0/2681259373 >> 172.21.15.88:0/2681259373 conn(0x7fe8900715c0 msgr2=0x7fe8900719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:37.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.304+0000 7fe895a91640 1 -- 172.21.15.88:0/2681259373 shutdown_connections 2023-10-16T20:55:37.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.304+0000 7fe895a91640 1 -- 172.21.15.88:0/2681259373 wait complete. 2023-10-16T20:55:37.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.308+0000 7fe895a91640 1 Processor -- start 2023-10-16T20:55:37.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.308+0000 7fe895a91640 1 -- start start 2023-10-16T20:55:37.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.308+0000 7fe895a91640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8900c9bb0 0x7fe89013c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:37.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.308+0000 7fe895a91640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8900cece0 0x7fe89013ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:37.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.308+0000 7fe895a91640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe890153770 0x7fe89013a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:37.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.308+0000 7fe895a91640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89013c0c0 con 0x7fe8900cece0 2023-10-16T20:55:37.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.308+0000 7fe895a91640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89013bd70 con 0x7fe890153770 2023-10-16T20:55:37.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.308+0000 7fe895a91640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900b4c90 con 0x7fe8900c9bb0 2023-10-16T20:55:37.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.308+0000 7fe88effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8900cece0 0x7fe89013ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:37.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.308+0000 7fe88effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8900cece0 0x7fe89013ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56258/0 (socket says 172.21.15.88:56258) 2023-10-16T20:55:37.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.308+0000 7fe88effd640 1 -- 172.21.15.88:0/1912303792 learned_addr learned my addr 172.21.15.88:0/1912303792 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:37.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.308+0000 7fe88f7fe640 1 --2- 172.21.15.88:0/1912303792 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8900c9bb0 0x7fe89013c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:37.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.308+0000 7fe88ffff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe890153770 0x7fe89013a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:37.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.312+0000 7fe88effd640 1 -- 172.21.15.88:0/1912303792 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8900c9bb0 msgr2=0x7fe89013c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:37.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.312+0000 7fe88effd640 1 --2- 172.21.15.88:0/1912303792 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8900c9bb0 0x7fe89013c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:37.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.312+0000 7fe88effd640 1 -- 172.21.15.88:0/1912303792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe890153770 msgr2=0x7fe89013a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:37.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.312+0000 7fe88effd640 1 --2- 172.21.15.88:0/1912303792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe890153770 0x7fe89013a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:37.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.312+0000 7fe88effd640 1 -- 172.21.15.88:0/1912303792 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe880009480 con 0x7fe8900cece0 2023-10-16T20:55:37.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.312+0000 7fe88f7fe640 1 --2- 172.21.15.88:0/1912303792 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8900c9bb0 0x7fe89013c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:55:37.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.312+0000 7fe88effd640 1 --2- 172.21.15.88:0/1912303792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8900cece0 0x7fe89013ca20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe88400c500 tx=0x7fe88400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:37.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.312+0000 7fe877fff640 1 -- 172.21.15.88:0/1912303792 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884005220 con 0x7fe8900cece0 2023-10-16T20:55:37.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.312+0000 7fe877fff640 1 -- 172.21.15.88:0/1912303792 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe884005380 con 0x7fe8900cece0 2023-10-16T20:55:37.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.312+0000 7fe895a91640 1 -- 172.21.15.88:0/1912303792 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe888009a70 con 0x7fe8900cece0 2023-10-16T20:55:37.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.312+0000 7fe877fff640 1 -- 172.21.15.88:0/1912303792 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884007270 con 0x7fe8900cece0 2023-10-16T20:55:37.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.312+0000 7fe895a91640 1 -- 172.21.15.88:0/1912303792 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8901439f0 con 0x7fe8900cece0 2023-10-16T20:55:37.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.312+0000 7fe895a91640 1 -- 172.21.15.88:0/1912303792 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe890074700 con 0x7fe8900cece0 2023-10-16T20:55:37.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.316+0000 7fe877fff640 1 -- 172.21.15.88:0/1912303792 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe8840073d0 con 0x7fe8900cece0 2023-10-16T20:55:37.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.316+0000 7fe877fff640 1 --2- 172.21.15.88:0/1912303792 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe87c0664a0 0x7fe87c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:37.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.316+0000 7fe877fff640 1 -- 172.21.15.88:0/1912303792 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7fe884083c80 con 0x7fe8900cece0 2023-10-16T20:55:37.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.316+0000 7fe88f7fe640 1 --2- 172.21.15.88:0/1912303792 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe87c0664a0 0x7fe87c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:37.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.316+0000 7fe88f7fe640 1 --2- 172.21.15.88:0/1912303792 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe87c0664a0 0x7fe87c068960 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe880006fd0 tx=0x7fe880009210 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:37.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.320+0000 7fe877fff640 1 -- 172.21.15.88:0/1912303792 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe884089020 con 0x7fe8900cece0 2023-10-16T20:55:37.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.580+0000 7fe895a91640 1 -- 172.21.15.88:0/1912303792 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8900ca740 con 0x7fe8900cece0 2023-10-16T20:55:37.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.580+0000 7fe877fff640 1 -- 172.21.15.88:0/1912303792 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6569 (secure 0 0 0) 0x7fe884050b80 con 0x7fe8900cece0 2023-10-16T20:55:37.578 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:55:37.579 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5477,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:54:28.552665+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:55:37.579 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 96 2023-10-16T20:55:37.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.584+0000 7fe895a91640 1 -- 172.21.15.88:0/1912303792 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe87c0664a0 msgr2=0x7fe87c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:37.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.584+0000 7fe895a91640 1 --2- 172.21.15.88:0/1912303792 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe87c0664a0 0x7fe87c068960 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe880006fd0 tx=0x7fe880009210 comp rx=0 tx=0).stop 2023-10-16T20:55:37.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.584+0000 7fe895a91640 1 -- 172.21.15.88:0/1912303792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8900cece0 msgr2=0x7fe89013ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:37.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.584+0000 7fe895a91640 1 --2- 172.21.15.88:0/1912303792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8900cece0 0x7fe89013ca20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe88400c500 tx=0x7fe88400c9c0 comp rx=0 tx=0).stop 2023-10-16T20:55:37.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.584+0000 7fe895a91640 1 -- 172.21.15.88:0/1912303792 shutdown_connections 2023-10-16T20:55:37.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.584+0000 7fe895a91640 1 --2- 172.21.15.88:0/1912303792 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe87c0664a0 0x7fe87c068960 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:37.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.584+0000 7fe895a91640 1 --2- 172.21.15.88:0/1912303792 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe890153770 0x7fe89013a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:37.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.584+0000 7fe895a91640 1 --2- 172.21.15.88:0/1912303792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8900cece0 0x7fe89013ca20 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:37.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.584+0000 7fe895a91640 1 --2- 172.21.15.88:0/1912303792 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8900c9bb0 0x7fe89013c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:37.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.584+0000 7fe895a91640 1 -- 172.21.15.88:0/1912303792 >> 172.21.15.88:0/1912303792 conn(0x7fe8900715c0 msgr2=0x7fe8900cf0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:37.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.584+0000 7fe895a91640 1 -- 172.21.15.88:0/1912303792 shutdown_connections 2023-10-16T20:55:37.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:37.584+0000 7fe895a91640 1 -- 172.21.15.88:0/1912303792 wait complete. 2023-10-16T20:55:37.610 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-16T20:55:39.611 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:55:39.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.792+0000 7f39c1846640 1 Processor -- start 2023-10-16T20:55:39.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.792+0000 7f39c1846640 1 -- start start 2023-10-16T20:55:39.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.796+0000 7f39c1846640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f39bc153770 0x7f39bc159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:39.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.796+0000 7f39c1846640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f39bc0c9bb0 0x7f39bc0c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:39.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.796+0000 7f39c1846640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39bc15a410 con 0x7f39bc0c41f0 2023-10-16T20:55:39.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.796+0000 7f39c1846640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39bc0cc250 con 0x7f39bc153770 2023-10-16T20:55:39.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.796+0000 7f39c1846640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39bc0cc670 con 0x7f39bc0c9bb0 2023-10-16T20:55:39.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.796+0000 7f39baffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f39bc153770 0x7f39bc159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:39.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.796+0000 7f39baffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f39bc153770 0x7f39bc159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48192/0 (socket says 172.21.15.88:48192) 2023-10-16T20:55:39.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.796+0000 7f39baffd640 1 -- 172.21.15.88:0/1813718695 learned_addr learned my addr 172.21.15.88:0/1813718695 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:39.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.796+0000 7f39ba7fc640 1 --2- 172.21.15.88:0/1813718695 >> v2:172.21.15.191:3300/0 conn(0x7f39bc0c9bb0 0x7f39bc0c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:39.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.796+0000 7f39baffd640 1 -- 172.21.15.88:0/1813718695 >> v2:172.21.15.191:3300/0 conn(0x7f39bc0c9bb0 msgr2=0x7f39bc0c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:55:39.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.796+0000 7f39baffd640 1 --2- 172.21.15.88:0/1813718695 >> v2:172.21.15.191:3300/0 conn(0x7f39bc0c9bb0 0x7f39bc0c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:39.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.796+0000 7f39baffd640 1 -- 172.21.15.88:0/1813718695 >> v1:172.21.15.191:6789/0 conn(0x7f39bc0c41f0 legacy=0x7f39bc0cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:39.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.796+0000 7f39baffd640 1 -- 172.21.15.88:0/1813718695 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39bc153b50 con 0x7f39bc153770 2023-10-16T20:55:39.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.800+0000 7f39baffd640 1 --2- 172.21.15.88:0/1813718695 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39bc153770 0x7f39bc159ed0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0cc7b0 tx=0x7f39ac01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec09aea52f41eaa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:39.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.800+0000 7f39b9ffb640 1 -- 172.21.15.88:0/1813718695 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac0ab5f0 con 0x7f39bc153770 2023-10-16T20:55:39.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.800+0000 7f39b9ffb640 1 -- 172.21.15.88:0/1813718695 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ac0ab750 con 0x7f39bc153770 2023-10-16T20:55:39.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.800+0000 7f39b9ffb640 1 -- 172.21.15.88:0/1813718695 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac0abae0 con 0x7f39bc153770 2023-10-16T20:55:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39c1846640 1 -- 172.21.15.88:0/1813718695 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39bc153770 msgr2=0x7f39bc159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39c1846640 1 --2- 172.21.15.88:0/1813718695 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39bc153770 0x7f39bc159ed0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0cc7b0 tx=0x7f39ac01c4e0 comp rx=0 tx=0).stop 2023-10-16T20:55:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39c1846640 1 -- 172.21.15.88:0/1813718695 shutdown_connections 2023-10-16T20:55:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39c1846640 1 --2- 172.21.15.88:0/1813718695 >> v2:172.21.15.191:3300/0 conn(0x7f39bc0c9bb0 0x7f39bc0c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:39.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39c1846640 1 --2- 172.21.15.88:0/1813718695 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39bc153770 0x7f39bc159ed0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:39.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39c1846640 1 -- 172.21.15.88:0/1813718695 >> 172.21.15.88:0/1813718695 conn(0x7f39bc0715c0 msgr2=0x7f39bc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:39.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39c1846640 1 -- 172.21.15.88:0/1813718695 shutdown_connections 2023-10-16T20:55:39.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39c1846640 1 -- 172.21.15.88:0/1813718695 wait complete. 2023-10-16T20:55:39.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39c1846640 1 Processor -- start 2023-10-16T20:55:39.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39c1846640 1 -- start start 2023-10-16T20:55:39.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39c1846640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39bc0c41f0 0x7f39bc1499b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:39.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39c1846640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39bc0c9bb0 0x7f39bc0c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:39.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39c1846640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39bc153770 0x7f39bc15b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:39.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39c1846640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc15ca40 con 0x7f39bc0c9bb0 2023-10-16T20:55:39.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39c1846640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc15d540 con 0x7f39bc153770 2023-10-16T20:55:39.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39c1846640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc1f2190 con 0x7f39bc0c41f0 2023-10-16T20:55:39.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39ba7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39bc0c9bb0 0x7f39bc0c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:39.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39baffd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39bc0c41f0 0x7f39bc1499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:39.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39baffd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39bc0c41f0 0x7f39bc1499b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:39050/0 (socket says 172.21.15.88:39050) 2023-10-16T20:55:39.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39baffd640 1 -- 172.21.15.88:0/2423657766 learned_addr learned my addr 172.21.15.88:0/2423657766 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:39.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39ba7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39bc0c9bb0 0x7f39bc0c78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48206/0 (socket says 172.21.15.88:48206) 2023-10-16T20:55:39.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.804+0000 7f39bb7fe640 1 --2- 172.21.15.88:0/2423657766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39bc153770 0x7f39bc15b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:39.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.808+0000 7f39baffd640 1 -- 172.21.15.88:0/2423657766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39bc153770 msgr2=0x7f39bc15b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:39.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.808+0000 7f39baffd640 1 --2- 172.21.15.88:0/2423657766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39bc153770 0x7f39bc15b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:39.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.808+0000 7f39baffd640 1 -- 172.21.15.88:0/2423657766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39bc0c9bb0 msgr2=0x7f39bc0c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:39.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.808+0000 7f39baffd640 1 --2- 172.21.15.88:0/2423657766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39bc0c9bb0 0x7f39bc0c78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:39.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.808+0000 7f39baffd640 1 -- 172.21.15.88:0/2423657766 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac009480 con 0x7f39bc0c41f0 2023-10-16T20:55:39.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.808+0000 7f39bb7fe640 1 --2- 172.21.15.88:0/2423657766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39bc153770 0x7f39bc15b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:55:39.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.808+0000 7f39ba7fc640 1 --2- 172.21.15.88:0/2423657766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39bc0c9bb0 0x7f39bc0c78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:55:39.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.808+0000 7f39baffd640 1 --2- 172.21.15.88:0/2423657766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39bc0c41f0 0x7f39bc1499b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f39ac01a110 tx=0x7f39ac019280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:39.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.808+0000 7f39abfff640 1 -- 172.21.15.88:0/2423657766 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac005ba0 con 0x7f39bc0c41f0 2023-10-16T20:55:39.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.808+0000 7f39abfff640 1 -- 172.21.15.88:0/2423657766 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ac005d00 con 0x7f39bc0c41f0 2023-10-16T20:55:39.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.808+0000 7f39c1846640 1 -- 172.21.15.88:0/2423657766 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39bc1f5b70 con 0x7f39bc0c41f0 2023-10-16T20:55:39.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.808+0000 7f39abfff640 1 -- 172.21.15.88:0/2423657766 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac0abae0 con 0x7f39bc0c41f0 2023-10-16T20:55:39.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.808+0000 7f39c1846640 1 -- 172.21.15.88:0/2423657766 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39bc1f2620 con 0x7f39bc0c41f0 2023-10-16T20:55:39.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.808+0000 7f39c1846640 1 -- 172.21.15.88:0/2423657766 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39bc074700 con 0x7f39bc0c41f0 2023-10-16T20:55:39.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.808+0000 7f39abfff640 1 -- 172.21.15.88:0/2423657766 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f39ac0abc40 con 0x7f39bc0c41f0 2023-10-16T20:55:39.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.812+0000 7f39abfff640 1 --2- 172.21.15.88:0/2423657766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f399c0664d0 0x7f399c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:39.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.812+0000 7f39abfff640 1 -- 172.21.15.88:0/2423657766 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f39ac0a8050 con 0x7f39bc0c41f0 2023-10-16T20:55:39.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.812+0000 7f39ba7fc640 1 --2- 172.21.15.88:0/2423657766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f399c0664d0 0x7f399c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:39.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.812+0000 7f39ba7fc640 1 --2- 172.21.15.88:0/2423657766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f399c0664d0 0x7f399c068990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f39bc15db00 tx=0x7f39a4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:39.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:39.816+0000 7f39abfff640 1 -- 172.21.15.88:0/2423657766 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f39ac0eb8b0 con 0x7f39bc0c41f0 2023-10-16T20:55:40.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:40.096+0000 7f39c1846640 1 -- 172.21.15.88:0/2423657766 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39bc14d800 con 0x7f39bc0c41f0 2023-10-16T20:55:40.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:40.096+0000 7f39abfff640 1 -- 172.21.15.88:0/2423657766 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6569 (secure 0 0 0) 0x7f39ac0eeed0 con 0x7f39bc0c41f0 2023-10-16T20:55:40.095 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:55:40.095 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5477,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:54:28.552665+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:55:40.095 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 96 2023-10-16T20:55:40.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:40.100+0000 7f39c1846640 1 -- 172.21.15.88:0/2423657766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f399c0664d0 msgr2=0x7f399c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:40.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:40.100+0000 7f39c1846640 1 --2- 172.21.15.88:0/2423657766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f399c0664d0 0x7f399c068990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f39bc15db00 tx=0x7f39a4009190 comp rx=0 tx=0).stop 2023-10-16T20:55:40.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:40.100+0000 7f39c1846640 1 -- 172.21.15.88:0/2423657766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39bc0c41f0 msgr2=0x7f39bc1499b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:40.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:40.100+0000 7f39c1846640 1 --2- 172.21.15.88:0/2423657766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39bc0c41f0 0x7f39bc1499b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f39ac01a110 tx=0x7f39ac019280 comp rx=0 tx=0).stop 2023-10-16T20:55:40.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:40.100+0000 7f39c1846640 1 -- 172.21.15.88:0/2423657766 shutdown_connections 2023-10-16T20:55:40.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:40.100+0000 7f39c1846640 1 --2- 172.21.15.88:0/2423657766 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f399c0664d0 0x7f399c068990 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:40.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:40.100+0000 7f39c1846640 1 --2- 172.21.15.88:0/2423657766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f39bc153770 0x7f39bc15b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:40.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:40.100+0000 7f39c1846640 1 --2- 172.21.15.88:0/2423657766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39bc0c9bb0 0x7f39bc0c78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:40.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:40.100+0000 7f39c1846640 1 --2- 172.21.15.88:0/2423657766 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f39bc0c41f0 0x7f39bc1499b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:40.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:40.100+0000 7f39c1846640 1 -- 172.21.15.88:0/2423657766 >> 172.21.15.88:0/2423657766 conn(0x7f39bc0715c0 msgr2=0x7f39bc0719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:40.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:40.100+0000 7f39c1846640 1 -- 172.21.15.88:0/2423657766 shutdown_connections 2023-10-16T20:55:40.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:40.100+0000 7f39c1846640 1 -- 172.21.15.88:0/2423657766 wait complete. 2023-10-16T20:55:40.124 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-16T20:55:42.126 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:55:42.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f349e49c640 1 Processor -- start 2023-10-16T20:55:42.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f349e49c640 1 -- start start 2023-10-16T20:55:42.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f349e49c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34980cece0 0x7f34980cf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:42.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f349e49c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3498153b50 con 0x7f3498153770 2023-10-16T20:55:42.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f349e49c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34980cb2f0 con 0x7f34980c9bb0 2023-10-16T20:55:42.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f349e49c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34980cc250 con 0x7f34980cece0 2023-10-16T20:55:42.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f3497fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3498153770 0x7f3498159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:38894/0 (socket says 172.21.15.88:38894) 2023-10-16T20:55:42.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f3497fff640 1 -- 172.21.15.88:0/4015334007 learned_addr learned my addr 172.21.15.88:0/4015334007 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:55:42.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f349ca39640 1 --2- 172.21.15.88:0/4015334007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34980cece0 0x7f34980cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:42.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f3496ffd640 1 -- 172.21.15.88:0/4015334007 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127320024 0 0) 0x7f3498153b50 con 0x7f3498153770 2023-10-16T20:55:42.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f3496ffd640 1 -- 172.21.15.88:0/4015334007 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3480003530 con 0x7f3498153770 2023-10-16T20:55:42.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f3496ffd640 1 -- 172.21.15.88:0/4015334007 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1755412529 0 0) 0x7f3480003530 con 0x7f3498153770 2023-10-16T20:55:42.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f3496ffd640 1 -- 172.21.15.88:0/4015334007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34980cece0 msgr2=0x7f34980cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:55:42.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f3496ffd640 1 --2- 172.21.15.88:0/4015334007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34980cece0 0x7f34980cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:42.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f3496ffd640 1 -- 172.21.15.88:0/4015334007 >> v1:172.21.15.191:6789/0 conn(0x7f34980c9bb0 legacy=0x7f34980c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:42.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f3496ffd640 1 -- 172.21.15.88:0/4015334007 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34980cb5d0 con 0x7f3498153770 2023-10-16T20:55:42.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f3496ffd640 1 -- 172.21.15.88:0/4015334007 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f3488001800 con 0x7f3498153770 2023-10-16T20:55:42.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f349ca39640 1 --2- 172.21.15.88:0/4015334007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34980cece0 0x7f34980cf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:55:42.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f3496ffd640 1 -- 172.21.15.88:0/4015334007 >> v1:172.21.15.88:6789/0 conn(0x7f3498153770 legacy=0x7f3498159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:42.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f3496ffd640 1 --2- 172.21.15.88:0/4015334007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34980cece0 0x7f34800039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:42.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f3496ffd640 1 --2- 172.21.15.88:0/4015334007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3480003ef0 0x7f34800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:42.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f3496ffd640 1 --2- 172.21.15.88:0/4015334007 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3480004d10 0x7f3480005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:42.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f3496ffd640 1 -- 172.21.15.88:0/4015334007 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34980cc250 con 0x7f3480003ef0 2023-10-16T20:55:42.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f3496ffd640 1 -- 172.21.15.88:0/4015334007 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348000b9f0 con 0x7f3480004d10 2023-10-16T20:55:42.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f3496ffd640 1 -- 172.21.15.88:0/4015334007 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348000bbb0 con 0x7f34980cece0 2023-10-16T20:55:42.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f34977fe640 1 --2- 172.21.15.88:0/4015334007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3480003ef0 0x7f34800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:42.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f34977fe640 1 -- 172.21.15.88:0/4015334007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34980cece0 msgr2=0x7f34800039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:55:42.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f34977fe640 1 --2- 172.21.15.88:0/4015334007 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f34980cece0 0x7f34800039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:42.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f34977fe640 1 -- 172.21.15.88:0/4015334007 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3480004d10 msgr2=0x7f3480005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:55:42.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f349ca39640 1 --2- 172.21.15.88:0/4015334007 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3480004d10 0x7f3480005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:42.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f34977fe640 1 --2- 172.21.15.88:0/4015334007 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3480004d10 0x7f3480005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:42.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.268+0000 7f34977fe640 1 -- 172.21.15.88:0/4015334007 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3490009a70 con 0x7f3480003ef0 2023-10-16T20:55:42.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.272+0000 7f34977fe640 1 --2- 172.21.15.88:0/4015334007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3480003ef0 0x7f34800047d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f348c001910 tx=0x7f348c01ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47fe03b1cb523640 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:42.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.272+0000 7f3496ffd640 1 -- 172.21.15.88:0/4015334007 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348c0a8810 con 0x7f3480003ef0 2023-10-16T20:55:42.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.272+0000 7f3496ffd640 1 -- 172.21.15.88:0/4015334007 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f348c0a8990 con 0x7f3480003ef0 2023-10-16T20:55:42.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.272+0000 7f3496ffd640 1 -- 172.21.15.88:0/4015334007 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348c0a8d40 con 0x7f3480003ef0 2023-10-16T20:55:42.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.272+0000 7f349e49c640 1 -- 172.21.15.88:0/4015334007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3480003ef0 msgr2=0x7f34800047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:42.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.272+0000 7f349e49c640 1 --2- 172.21.15.88:0/4015334007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3480003ef0 0x7f34800047d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f348c001910 tx=0x7f348c01ebf0 comp rx=0 tx=0).stop 2023-10-16T20:55:42.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.272+0000 7f349ca39640 1 -- 172.21.15.88:0/4015334007 reap_dead start 2023-10-16T20:55:42.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.272+0000 7f349e49c640 1 -- 172.21.15.88:0/4015334007 shutdown_connections 2023-10-16T20:55:42.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.272+0000 7f349e49c640 1 -- 172.21.15.88:0/4015334007 >> 172.21.15.88:0/4015334007 conn(0x7f34980715c0 msgr2=0x7f34980719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:42.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.272+0000 7f349e49c640 1 -- 172.21.15.88:0/4015334007 shutdown_connections 2023-10-16T20:55:42.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.272+0000 7f349e49c640 1 -- 172.21.15.88:0/4015334007 wait complete. 2023-10-16T20:55:42.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f349e49c640 1 Processor -- start 2023-10-16T20:55:42.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f349e49c640 1 -- start start 2023-10-16T20:55:42.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f349e49c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3480003ef0 0x7f349815c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f349e49c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f349814d350 0x7f34981455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f349e49c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34980c8180 0x7f3498145b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f349e49c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34980c88d0 con 0x7f3480003ef0 2023-10-16T20:55:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f349e49c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349814e3c0 con 0x7f34980c8180 2023-10-16T20:55:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f349e49c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34980c8ae0 con 0x7f349814d350 2023-10-16T20:55:42.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f3497fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3480003ef0 0x7f349815c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:42.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f3497fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3480003ef0 0x7f349815c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48220/0 (socket says 172.21.15.88:48220) 2023-10-16T20:55:42.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f3497fff640 1 -- 172.21.15.88:0/1697954221 learned_addr learned my addr 172.21.15.88:0/1697954221 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:42.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f34977fe640 1 --2- 172.21.15.88:0/1697954221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f349814d350 0x7f34981455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:42.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f349ca39640 1 --2- 172.21.15.88:0/1697954221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34980c8180 0x7f3498145b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:42.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f3497fff640 1 -- 172.21.15.88:0/1697954221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f349814d350 msgr2=0x7f34981455e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:42.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f3497fff640 1 --2- 172.21.15.88:0/1697954221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f349814d350 0x7f34981455e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:42.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f3497fff640 1 -- 172.21.15.88:0/1697954221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34980c8180 msgr2=0x7f3498145b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:42.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f3497fff640 1 --2- 172.21.15.88:0/1697954221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34980c8180 0x7f3498145b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:42.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f3497fff640 1 -- 172.21.15.88:0/1697954221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349815cc10 con 0x7f3480003ef0 2023-10-16T20:55:42.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f349ca39640 1 --2- 172.21.15.88:0/1697954221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34980c8180 0x7f3498145b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:55:42.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f3497fff640 1 --2- 172.21.15.88:0/1697954221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3480003ef0 0x7f349815c2d0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f3488000f00 tx=0x7f348800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:42.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f3494ff9640 1 -- 172.21.15.88:0/1697954221 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3488007270 con 0x7f3480003ef0 2023-10-16T20:55:42.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f3494ff9640 1 -- 172.21.15.88:0/1697954221 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34880073d0 con 0x7f3480003ef0 2023-10-16T20:55:42.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f3494ff9640 1 -- 172.21.15.88:0/1697954221 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34880057b0 con 0x7f3480003ef0 2023-10-16T20:55:42.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f349e49c640 1 -- 172.21.15.88:0/1697954221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f349815d1b0 con 0x7f3480003ef0 2023-10-16T20:55:42.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.276+0000 7f349e49c640 1 -- 172.21.15.88:0/1697954221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349815ced0 con 0x7f3480003ef0 2023-10-16T20:55:42.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.280+0000 7f349e49c640 1 -- 172.21.15.88:0/1697954221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3498074700 con 0x7f3480003ef0 2023-10-16T20:55:42.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.280+0000 7f3494ff9640 1 -- 172.21.15.88:0/1697954221 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3488005910 con 0x7f3480003ef0 2023-10-16T20:55:42.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.280+0000 7f3494ff9640 1 --2- 172.21.15.88:0/1697954221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34840664d0 0x7f3484068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:42.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.280+0000 7f3494ff9640 1 -- 172.21.15.88:0/1697954221 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f3488083c40 con 0x7f3480003ef0 2023-10-16T20:55:42.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.280+0000 7f34977fe640 1 --2- 172.21.15.88:0/1697954221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34840664d0 0x7f3484068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:42.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.280+0000 7f34977fe640 1 --2- 172.21.15.88:0/1697954221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34840664d0 0x7f3484068990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f348c0b2000 tx=0x7f348c001330 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:42.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.284+0000 7f3494ff9640 1 -- 172.21.15.88:0/1697954221 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3488050a00 con 0x7f3480003ef0 2023-10-16T20:55:42.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.524+0000 7f349e49c640 1 -- 172.21.15.88:0/1697954221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34980cbe40 con 0x7f3480003ef0 2023-10-16T20:55:42.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.524+0000 7f3494ff9640 1 -- 172.21.15.88:0/1697954221 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6569 (secure 0 0 0) 0x7f3488017020 con 0x7f3480003ef0 2023-10-16T20:55:42.521 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:55:42.521 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5477,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:54:28.552665+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:55:42.522 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 96 2023-10-16T20:55:42.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.524+0000 7f349e49c640 1 -- 172.21.15.88:0/1697954221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34840664d0 msgr2=0x7f3484068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:42.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.524+0000 7f349e49c640 1 --2- 172.21.15.88:0/1697954221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34840664d0 0x7f3484068990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f348c0b2000 tx=0x7f348c001330 comp rx=0 tx=0).stop 2023-10-16T20:55:42.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.524+0000 7f349e49c640 1 -- 172.21.15.88:0/1697954221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3480003ef0 msgr2=0x7f349815c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:42.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.524+0000 7f349e49c640 1 --2- 172.21.15.88:0/1697954221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3480003ef0 0x7f349815c2d0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f3488000f00 tx=0x7f348800cb90 comp rx=0 tx=0).stop 2023-10-16T20:55:42.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.524+0000 7f349e49c640 1 -- 172.21.15.88:0/1697954221 shutdown_connections 2023-10-16T20:55:42.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.524+0000 7f349e49c640 1 --2- 172.21.15.88:0/1697954221 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34840664d0 0x7f3484068990 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:42.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.524+0000 7f349e49c640 1 --2- 172.21.15.88:0/1697954221 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f34980c8180 0x7f3498145b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:42.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.524+0000 7f349e49c640 1 --2- 172.21.15.88:0/1697954221 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f349814d350 0x7f34981455e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:42.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.524+0000 7f349e49c640 1 --2- 172.21.15.88:0/1697954221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3480003ef0 0x7f349815c2d0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:42.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.524+0000 7f349e49c640 1 -- 172.21.15.88:0/1697954221 >> 172.21.15.88:0/1697954221 conn(0x7f34980715c0 msgr2=0x7f34980cdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:42.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.524+0000 7f349e49c640 1 -- 172.21.15.88:0/1697954221 shutdown_connections 2023-10-16T20:55:42.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:42.524+0000 7f349e49c640 1 -- 172.21.15.88:0/1697954221 wait complete. 2023-10-16T20:55:42.542 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-16T20:55:44.543 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:55:44.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.712+0000 7f5ab6548640 1 Processor -- start 2023-10-16T20:55:44.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.712+0000 7f5ab6548640 1 -- start start 2023-10-16T20:55:44.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.716+0000 7f5ab6548640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5ab00c9af0 0x7f5ab00cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:44.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.716+0000 7f5ab6548640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f5ab00b1a60 0x7f5ab0163b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:44.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.716+0000 7f5ab6548640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab00c9fc0 con 0x7f5ab00cb580 2023-10-16T20:55:44.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.716+0000 7f5ab6548640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ab00b20c0 con 0x7f5ab00c9af0 2023-10-16T20:55:44.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.716+0000 7f5ab6548640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ab00cbea0 con 0x7f5ab00b1a60 2023-10-16T20:55:44.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.716+0000 7f5ab4d45640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5ab00c9af0 0x7f5ab00cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:44.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.716+0000 7f5ab5546640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f5ab00cb580 0x7f5ab0159300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:43778/0 (socket says 172.21.15.88:43778) 2023-10-16T20:55:44.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.716+0000 7f5ab5546640 1 -- 172.21.15.88:0/286448268 learned_addr learned my addr 172.21.15.88:0/286448268 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:55:44.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.716+0000 7f5ab5d47640 1 --2- 172.21.15.88:0/286448268 >> v2:172.21.15.191:3300/0 conn(0x7f5ab00b1a60 0x7f5ab0163b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:44.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.716+0000 7f5ab4d45640 1 -- 172.21.15.88:0/286448268 >> v2:172.21.15.191:3300/0 conn(0x7f5ab00b1a60 msgr2=0x7f5ab0163b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:55:44.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.716+0000 7f5ab4d45640 1 --2- 172.21.15.88:0/286448268 >> v2:172.21.15.191:3300/0 conn(0x7f5ab00b1a60 0x7f5ab0163b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:44.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.716+0000 7f5ab4d45640 1 -- 172.21.15.88:0/286448268 >> v1:172.21.15.191:6789/0 conn(0x7f5ab00cb580 legacy=0x7f5ab0159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:44.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.716+0000 7f5ab4d45640 1 -- 172.21.15.88:0/286448268 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab00c47d0 con 0x7f5ab00c9af0 2023-10-16T20:55:44.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.720+0000 7f5ab4d45640 1 --2- 172.21.15.88:0/286448268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ab00c9af0 0x7f5ab00cb960 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00cfb80 tx=0x7f5aa001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc043e6125290ccc server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:44.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.720+0000 7f5aa7fff640 1 -- 172.21.15.88:0/286448268 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa00ac600 con 0x7f5ab00c9af0 2023-10-16T20:55:44.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.720+0000 7f5aa7fff640 1 -- 172.21.15.88:0/286448268 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa00a8050 con 0x7f5ab00c9af0 2023-10-16T20:55:44.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.720+0000 7f5aa7fff640 1 -- 172.21.15.88:0/286448268 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa00ac980 con 0x7f5ab00c9af0 2023-10-16T20:55:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.720+0000 7f5ab6548640 1 -- 172.21.15.88:0/286448268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ab00c9af0 msgr2=0x7f5ab00cb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.720+0000 7f5ab6548640 1 --2- 172.21.15.88:0/286448268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ab00c9af0 0x7f5ab00cb960 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00cfb80 tx=0x7f5aa001c3a0 comp rx=0 tx=0).stop 2023-10-16T20:55:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.720+0000 7f5ab6548640 1 -- 172.21.15.88:0/286448268 shutdown_connections 2023-10-16T20:55:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.720+0000 7f5ab6548640 1 --2- 172.21.15.88:0/286448268 >> v2:172.21.15.191:3300/0 conn(0x7f5ab00b1a60 0x7f5ab0163b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.720+0000 7f5ab6548640 1 --2- 172.21.15.88:0/286448268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ab00c9af0 0x7f5ab00cb960 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.720+0000 7f5ab6548640 1 -- 172.21.15.88:0/286448268 >> 172.21.15.88:0/286448268 conn(0x7f5ab006bb40 msgr2=0x7f5ab006bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.724+0000 7f5ab6548640 1 -- 172.21.15.88:0/286448268 shutdown_connections 2023-10-16T20:55:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.724+0000 7f5ab6548640 1 -- 172.21.15.88:0/286448268 wait complete. 2023-10-16T20:55:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.724+0000 7f5ab6548640 1 Processor -- start 2023-10-16T20:55:44.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.724+0000 7f5ab6548640 1 -- start start 2023-10-16T20:55:44.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab6548640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ab00b1a60 0x7f5ab014a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab5546640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ab00b1a60 0x7f5ab014a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab5546640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ab00b1a60 0x7f5ab014a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:39096/0 (socket says 172.21.15.88:39096) 2023-10-16T20:55:44.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab6548640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ab00c9af0 0x7f5ab0158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:44.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab6548640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ab00cb580 0x7f5ab014c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:44.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab5546640 1 -- 172.21.15.88:0/2841418470 learned_addr learned my addr 172.21.15.88:0/2841418470 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:44.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab6548640 1 -- 172.21.15.88:0/2841418470 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab0157d30 con 0x7f5ab00cb580 2023-10-16T20:55:44.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab6548640 1 -- 172.21.15.88:0/2841418470 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab014be00 con 0x7f5ab00c9af0 2023-10-16T20:55:44.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab6548640 1 -- 172.21.15.88:0/2841418470 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab0162de0 con 0x7f5ab00b1a60 2023-10-16T20:55:44.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab4d45640 1 --2- 172.21.15.88:0/2841418470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ab00c9af0 0x7f5ab0158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:44.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab5d47640 1 --2- 172.21.15.88:0/2841418470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ab00cb580 0x7f5ab014c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab5546640 1 -- 172.21.15.88:0/2841418470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ab00c9af0 msgr2=0x7f5ab0158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab5546640 1 --2- 172.21.15.88:0/2841418470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ab00c9af0 0x7f5ab0158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab5546640 1 -- 172.21.15.88:0/2841418470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ab00cb580 msgr2=0x7f5ab014c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab5546640 1 --2- 172.21.15.88:0/2841418470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ab00cb580 0x7f5ab014c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab5546640 1 -- 172.21.15.88:0/2841418470 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa0009420 con 0x7f5ab00b1a60 2023-10-16T20:55:44.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab5546640 1 --2- 172.21.15.88:0/2841418470 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ab00b1a60 0x7f5ab014a4c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00c430 tx=0x7f5aac00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:44.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5aa5ffb640 1 -- 172.21.15.88:0/2841418470 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aac005cf0 con 0x7f5ab00b1a60 2023-10-16T20:55:44.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5aa5ffb640 1 -- 172.21.15.88:0/2841418470 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aac005e50 con 0x7f5ab00b1a60 2023-10-16T20:55:44.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5aa5ffb640 1 -- 172.21.15.88:0/2841418470 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aac0072d0 con 0x7f5ab00b1a60 2023-10-16T20:55:44.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab6548640 1 -- 172.21.15.88:0/2841418470 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab0157420 con 0x7f5ab00b1a60 2023-10-16T20:55:44.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.728+0000 7f5ab6548640 1 -- 172.21.15.88:0/2841418470 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab01632f0 con 0x7f5ab00b1a60 2023-10-16T20:55:44.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.732+0000 7f5ab6548640 1 -- 172.21.15.88:0/2841418470 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab00ca2c0 con 0x7f5ab00b1a60 2023-10-16T20:55:44.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.732+0000 7f5aa5ffb640 1 -- 172.21.15.88:0/2841418470 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5aac007430 con 0x7f5ab00b1a60 2023-10-16T20:55:44.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.732+0000 7f5aa5ffb640 1 --2- 172.21.15.88:0/2841418470 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f5a940664d0 0x7f5a94068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:44.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.732+0000 7f5aa5ffb640 1 -- 172.21.15.88:0/2841418470 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f5aac0839c0 con 0x7f5ab00b1a60 2023-10-16T20:55:44.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.736+0000 7f5ab4d45640 1 --2- 172.21.15.88:0/2841418470 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f5a940664d0 0x7f5a94068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:44.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.736+0000 7f5ab4d45640 1 --2- 172.21.15.88:0/2841418470 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f5a940664d0 0x7f5a94068990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00ae000 tx=0x7f5aa0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:44.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.740+0000 7f5aa5ffb640 1 -- 172.21.15.88:0/2841418470 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5aac050830 con 0x7f5ab00b1a60 2023-10-16T20:55:44.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.928+0000 7f5ab6548640 1 -- 172.21.15.88:0/2841418470 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ab004e9c0 con 0x7f5ab00b1a60 2023-10-16T20:55:44.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.928+0000 7f5aa5ffb640 1 -- 172.21.15.88:0/2841418470 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6569 (secure 0 0 0) 0x7f5aac053e50 con 0x7f5ab00b1a60 2023-10-16T20:55:44.928 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:55:44.928 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5477,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:54:28.552665+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:55:44.928 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 96 2023-10-16T20:55:44.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.932+0000 7f5ab6548640 1 -- 172.21.15.88:0/2841418470 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f5a940664d0 msgr2=0x7f5a94068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:44.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.932+0000 7f5ab6548640 1 --2- 172.21.15.88:0/2841418470 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f5a940664d0 0x7f5a94068990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00ae000 tx=0x7f5aa0008040 comp rx=0 tx=0).stop 2023-10-16T20:55:44.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.932+0000 7f5ab6548640 1 -- 172.21.15.88:0/2841418470 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ab00b1a60 msgr2=0x7f5ab014a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:44.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.932+0000 7f5ab6548640 1 --2- 172.21.15.88:0/2841418470 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ab00b1a60 0x7f5ab014a4c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00c430 tx=0x7f5aac00c8f0 comp rx=0 tx=0).stop 2023-10-16T20:55:44.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.932+0000 7f5ab6548640 1 -- 172.21.15.88:0/2841418470 shutdown_connections 2023-10-16T20:55:44.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.932+0000 7f5ab6548640 1 --2- 172.21.15.88:0/2841418470 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f5a940664d0 0x7f5a94068990 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:44.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.932+0000 7f5ab6548640 1 --2- 172.21.15.88:0/2841418470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ab00cb580 0x7f5ab014c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:44.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.932+0000 7f5ab6548640 1 --2- 172.21.15.88:0/2841418470 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5ab00c9af0 0x7f5ab0158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:44.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.932+0000 7f5ab6548640 1 --2- 172.21.15.88:0/2841418470 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5ab00b1a60 0x7f5ab014a4c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:44.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.932+0000 7f5ab6548640 1 -- 172.21.15.88:0/2841418470 >> 172.21.15.88:0/2841418470 conn(0x7f5ab006bb40 msgr2=0x7f5ab00cd410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:44.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.932+0000 7f5ab6548640 1 -- 172.21.15.88:0/2841418470 shutdown_connections 2023-10-16T20:55:44.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:44.932+0000 7f5ab6548640 1 -- 172.21.15.88:0/2841418470 wait complete. 2023-10-16T20:55:44.949 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-16T20:55:46.950 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:55:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.092+0000 7ff75cd72640 1 Processor -- start 2023-10-16T20:55:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.092+0000 7ff75cd72640 1 -- start start 2023-10-16T20:55:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.096+0000 7ff75cd72640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff7581406d0 0x7ff7580c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.096+0000 7ff75cd72640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff75815a810 0x7ff75815ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.096+0000 7ff75cd72640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff75815b190 con 0x7ff75806a970 2023-10-16T20:55:47.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.096+0000 7ff75cd72640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff75815b3c0 con 0x7ff7581406d0 2023-10-16T20:55:47.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.096+0000 7ff75cd72640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff75815b680 con 0x7ff75815a810 2023-10-16T20:55:47.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.096+0000 7ff75659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff7581406d0 0x7ff7580c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:47.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.096+0000 7ff75659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff7581406d0 0x7ff7580c1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48256/0 (socket says 172.21.15.88:48256) 2023-10-16T20:55:47.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.096+0000 7ff75659c640 1 -- 172.21.15.88:0/867127500 learned_addr learned my addr 172.21.15.88:0/867127500 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:47.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.096+0000 7ff756d9d640 1 --2- 172.21.15.88:0/867127500 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff75815a810 0x7ff75815ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:47.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.096+0000 7ff75659c640 1 -- 172.21.15.88:0/867127500 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff75815a810 msgr2=0x7ff75815ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:55:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.096+0000 7ff75659c640 1 --2- 172.21.15.88:0/867127500 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff75815a810 0x7ff75815ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.096+0000 7ff75659c640 1 -- 172.21.15.88:0/867127500 >> v1:172.21.15.191:6789/0 conn(0x7ff75806a970 legacy=0x7ff758071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.096+0000 7ff75659c640 1 -- 172.21.15.88:0/867127500 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff758157570 con 0x7ff7581406d0 2023-10-16T20:55:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.096+0000 7ff75659c640 1 --2- 172.21.15.88:0/867127500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7581406d0 0x7ff7580c1d50 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7ff748009e50 tx=0x7ff74801c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c888b421ebba7a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.096+0000 7ff755d9b640 1 -- 172.21.15.88:0/867127500 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7480a58a0 con 0x7ff7581406d0 2023-10-16T20:55:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.096+0000 7ff755d9b640 1 -- 172.21.15.88:0/867127500 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7480a5a00 con 0x7ff7581406d0 2023-10-16T20:55:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.096+0000 7ff755d9b640 1 -- 172.21.15.88:0/867127500 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7480a5d90 con 0x7ff7581406d0 2023-10-16T20:55:47.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff75cd72640 1 -- 172.21.15.88:0/867127500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7581406d0 msgr2=0x7ff7580c1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:47.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff75cd72640 1 --2- 172.21.15.88:0/867127500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7581406d0 0x7ff7580c1d50 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7ff748009e50 tx=0x7ff74801c9e0 comp rx=0 tx=0).stop 2023-10-16T20:55:47.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff75cd72640 1 -- 172.21.15.88:0/867127500 shutdown_connections 2023-10-16T20:55:47.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff75cd72640 1 --2- 172.21.15.88:0/867127500 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff75815a810 0x7ff75815ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:47.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff75cd72640 1 --2- 172.21.15.88:0/867127500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7581406d0 0x7ff7580c1d50 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:47.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff75cd72640 1 -- 172.21.15.88:0/867127500 >> 172.21.15.88:0/867127500 conn(0x7ff7580cad50 msgr2=0x7ff7580cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:47.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff75cd72640 1 -- 172.21.15.88:0/867127500 shutdown_connections 2023-10-16T20:55:47.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff75cd72640 1 -- 172.21.15.88:0/867127500 wait complete. 2023-10-16T20:55:47.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff75cd72640 1 Processor -- start 2023-10-16T20:55:47.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff75cd72640 1 -- start start 2023-10-16T20:55:47.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff75cd72640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff75806a970 0x7ff7581520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:47.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff75cd72640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7581406d0 0x7ff758071280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:47.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff75cd72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff75815a810 0x7ff758158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:47.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff75cd72640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff758161770 con 0x7ff75815a810 2023-10-16T20:55:47.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff75cd72640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff758070ac0 con 0x7ff75806a970 2023-10-16T20:55:47.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff75cd72640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff758160ed0 con 0x7ff7581406d0 2023-10-16T20:55:47.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff756d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff75815a810 0x7ff758158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:47.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff756d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff75815a810 0x7ff758158640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48258/0 (socket says 172.21.15.88:48258) 2023-10-16T20:55:47.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff756d9d640 1 -- 172.21.15.88:0/523228095 learned_addr learned my addr 172.21.15.88:0/523228095 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:47.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff75659c640 1 --2- 172.21.15.88:0/523228095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff75806a970 0x7ff7581520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:47.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.100+0000 7ff74dd9b640 1 --2- 172.21.15.88:0/523228095 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7581406d0 0x7ff758071280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:47.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.104+0000 7ff756d9d640 1 -- 172.21.15.88:0/523228095 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7581406d0 msgr2=0x7ff758071280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:47.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.104+0000 7ff756d9d640 1 --2- 172.21.15.88:0/523228095 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7581406d0 0x7ff758071280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:47.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.104+0000 7ff756d9d640 1 -- 172.21.15.88:0/523228095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff75806a970 msgr2=0x7ff7581520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:47.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.104+0000 7ff756d9d640 1 --2- 172.21.15.88:0/523228095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff75806a970 0x7ff7581520b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:47.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.104+0000 7ff756d9d640 1 -- 172.21.15.88:0/523228095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff748009b30 con 0x7ff75815a810 2023-10-16T20:55:47.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.104+0000 7ff74dd9b640 1 --2- 172.21.15.88:0/523228095 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7581406d0 0x7ff758071280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:55:47.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.104+0000 7ff756d9d640 1 --2- 172.21.15.88:0/523228095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff75815a810 0x7ff758158640 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff75000d800 tx=0x7ff75000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:47.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.104+0000 7ff74f7fe640 1 -- 172.21.15.88:0/523228095 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75000b6c0 con 0x7ff75815a810 2023-10-16T20:55:47.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.104+0000 7ff74f7fe640 1 -- 172.21.15.88:0/523228095 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75000b820 con 0x7ff75815a810 2023-10-16T20:55:47.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.104+0000 7ff74f7fe640 1 -- 172.21.15.88:0/523228095 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75000bbd0 con 0x7ff75815a810 2023-10-16T20:55:47.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.104+0000 7ff75cd72640 1 -- 172.21.15.88:0/523228095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7581611b0 con 0x7ff75815a810 2023-10-16T20:55:47.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.104+0000 7ff75cd72640 1 -- 172.21.15.88:0/523228095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff758070d60 con 0x7ff75815a810 2023-10-16T20:55:47.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.104+0000 7ff75cd72640 1 -- 172.21.15.88:0/523228095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff758074700 con 0x7ff75815a810 2023-10-16T20:55:47.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.104+0000 7ff74f7fe640 1 -- 172.21.15.88:0/523228095 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff75000bd30 con 0x7ff75815a810 2023-10-16T20:55:47.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.104+0000 7ff74f7fe640 1 --2- 172.21.15.88:0/523228095 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff7380664a0 0x7ff738068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:47.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.104+0000 7ff74f7fe640 1 -- 172.21.15.88:0/523228095 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7ff7500849c0 con 0x7ff75815a810 2023-10-16T20:55:47.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.104+0000 7ff75659c640 1 --2- 172.21.15.88:0/523228095 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff7380664a0 0x7ff738068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:47.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.108+0000 7ff75659c640 1 --2- 172.21.15.88:0/523228095 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff7380664a0 0x7ff738068960 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff7480ae000 tx=0x7ff7480a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:47.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.108+0000 7ff74f7fe640 1 -- 172.21.15.88:0/523228095 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff750051780 con 0x7ff75815a810 2023-10-16T20:55:47.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.292+0000 7ff75cd72640 1 -- 172.21.15.88:0/523228095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff758154c40 con 0x7ff75815a810 2023-10-16T20:55:47.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.292+0000 7ff74f7fe640 1 -- 172.21.15.88:0/523228095 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6569 (secure 0 0 0) 0x7ff750054da0 con 0x7ff75815a810 2023-10-16T20:55:47.290 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:55:47.291 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5477,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:54:28.552665+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:55:47.292 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 96 2023-10-16T20:55:47.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.292+0000 7ff75cd72640 1 -- 172.21.15.88:0/523228095 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff7380664a0 msgr2=0x7ff738068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:47.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.292+0000 7ff75cd72640 1 --2- 172.21.15.88:0/523228095 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff7380664a0 0x7ff738068960 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff7480ae000 tx=0x7ff7480a7070 comp rx=0 tx=0).stop 2023-10-16T20:55:47.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.292+0000 7ff75cd72640 1 -- 172.21.15.88:0/523228095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff75815a810 msgr2=0x7ff758158640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:47.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.292+0000 7ff75cd72640 1 --2- 172.21.15.88:0/523228095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff75815a810 0x7ff758158640 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff75000d800 tx=0x7ff75000dcc0 comp rx=0 tx=0).stop 2023-10-16T20:55:47.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.292+0000 7ff75cd72640 1 -- 172.21.15.88:0/523228095 shutdown_connections 2023-10-16T20:55:47.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.292+0000 7ff75cd72640 1 --2- 172.21.15.88:0/523228095 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff7380664a0 0x7ff738068960 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:47.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.292+0000 7ff75cd72640 1 --2- 172.21.15.88:0/523228095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff75815a810 0x7ff758158640 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:47.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.292+0000 7ff75cd72640 1 --2- 172.21.15.88:0/523228095 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff7581406d0 0x7ff758071280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:47.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.292+0000 7ff75cd72640 1 --2- 172.21.15.88:0/523228095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff75806a970 0x7ff7581520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:47.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.292+0000 7ff75cd72640 1 -- 172.21.15.88:0/523228095 >> 172.21.15.88:0/523228095 conn(0x7ff7580cad50 msgr2=0x7ff7580c24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:47.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.292+0000 7ff75cd72640 1 -- 172.21.15.88:0/523228095 shutdown_connections 2023-10-16T20:55:47.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:47.292+0000 7ff75cd72640 1 -- 172.21.15.88:0/523228095 wait complete. 2023-10-16T20:55:47.310 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-16T20:55:49.311 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:55:49.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.456+0000 7f347b59e640 1 Processor -- start 2023-10-16T20:55:49.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.456+0000 7f347b59e640 1 -- start start 2023-10-16T20:55:49.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.456+0000 7f347b59e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f347c1537c0 0x7f347c15c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:49.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.456+0000 7f347b59e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f347c15cdb0 0x7f347c0c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:49.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.456+0000 7f347b59e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347c0cae10 con 0x7f347c0ca2f0 2023-10-16T20:55:49.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.456+0000 7f347b59e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f347c0cac30 con 0x7f347c15cdb0 2023-10-16T20:55:49.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.456+0000 7f347b59e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f347c15d2b0 con 0x7f347c1537c0 2023-10-16T20:55:49.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.456+0000 7f3479d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f347c15cdb0 0x7f347c0c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:49.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.456+0000 7f3479d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f347c15cdb0 0x7f347c0c4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36254/0 (socket says 172.21.15.88:36254) 2023-10-16T20:55:49.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.456+0000 7f3479d9b640 1 -- 172.21.15.88:0/3769703967 learned_addr learned my addr 172.21.15.88:0/3769703967 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:49.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.456+0000 7f347a59c640 1 --2- 172.21.15.88:0/3769703967 >> v2:172.21.15.191:3300/0 conn(0x7f347c1537c0 0x7f347c15c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:49.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.456+0000 7f3479d9b640 1 -- 172.21.15.88:0/3769703967 >> v2:172.21.15.191:3300/0 conn(0x7f347c1537c0 msgr2=0x7f347c15c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:55:49.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.456+0000 7f3479d9b640 1 --2- 172.21.15.88:0/3769703967 >> v2:172.21.15.191:3300/0 conn(0x7f347c1537c0 0x7f347c15c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:49.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.456+0000 7f3479d9b640 1 -- 172.21.15.88:0/3769703967 >> v1:172.21.15.191:6789/0 conn(0x7f347c0ca2f0 legacy=0x7f347c0cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:49.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.456+0000 7f3479d9b640 1 -- 172.21.15.88:0/3769703967 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c15d590 con 0x7f347c15cdb0 2023-10-16T20:55:49.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.460+0000 7f3479d9b640 1 --2- 172.21.15.88:0/3769703967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f347c15cdb0 0x7f347c0c4c80 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f347c0cfc00 tx=0x7f347001ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a462db341ac1aed7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:49.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.460+0000 7f347959a640 1 -- 172.21.15.88:0/3769703967 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34700a89b0 con 0x7f347c15cdb0 2023-10-16T20:55:49.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.460+0000 7f347959a640 1 -- 172.21.15.88:0/3769703967 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34700a8b10 con 0x7f347c15cdb0 2023-10-16T20:55:49.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.460+0000 7f347959a640 1 -- 172.21.15.88:0/3769703967 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34700a8ea0 con 0x7f347c15cdb0 2023-10-16T20:55:49.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.460+0000 7f347b59e640 1 -- 172.21.15.88:0/3769703967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f347c15cdb0 msgr2=0x7f347c0c4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:49.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.460+0000 7f347b59e640 1 --2- 172.21.15.88:0/3769703967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f347c15cdb0 0x7f347c0c4c80 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f347c0cfc00 tx=0x7f347001ecb0 comp rx=0 tx=0).stop 2023-10-16T20:55:49.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.460+0000 7f347b59e640 1 -- 172.21.15.88:0/3769703967 shutdown_connections 2023-10-16T20:55:49.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.460+0000 7f347b59e640 1 --2- 172.21.15.88:0/3769703967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f347c15cdb0 0x7f347c0c4c80 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:49.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.460+0000 7f347b59e640 1 --2- 172.21.15.88:0/3769703967 >> v2:172.21.15.191:3300/0 conn(0x7f347c1537c0 0x7f347c15c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:49.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.460+0000 7f347b59e640 1 -- 172.21.15.88:0/3769703967 >> 172.21.15.88:0/3769703967 conn(0x7f347c06c170 msgr2=0x7f347c0715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:49.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.460+0000 7f347b59e640 1 -- 172.21.15.88:0/3769703967 shutdown_connections 2023-10-16T20:55:49.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.460+0000 7f347b59e640 1 -- 172.21.15.88:0/3769703967 wait complete. 2023-10-16T20:55:49.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f347b59e640 1 Processor -- start 2023-10-16T20:55:49.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f347b59e640 1 -- start start 2023-10-16T20:55:49.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f347b59e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f347c0ca2f0 0x7f347c15c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:49.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f347b59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f347c1537c0 0x7f347c1f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:49.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f347b59e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f347c15cdb0 0x7f347c145b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:49.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f347b59e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c1e4440 con 0x7f347c1537c0 2023-10-16T20:55:49.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f347b59e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c15cb90 con 0x7f347c15cdb0 2023-10-16T20:55:49.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f347b59e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347c1e24c0 con 0x7f347c0ca2f0 2023-10-16T20:55:49.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f3479d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f347c1537c0 0x7f347c1f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:49.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f3479d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f347c1537c0 0x7f347c1f2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36268/0 (socket says 172.21.15.88:36268) 2023-10-16T20:55:49.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f3479d9b640 1 -- 172.21.15.88:0/19062543 learned_addr learned my addr 172.21.15.88:0/19062543 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:49.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f347a59c640 1 --2- 172.21.15.88:0/19062543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f347c0ca2f0 0x7f347c15c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:49.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f347ad9d640 1 --2- 172.21.15.88:0/19062543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f347c15cdb0 0x7f347c145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:49.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f3479d9b640 1 -- 172.21.15.88:0/19062543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f347c0ca2f0 msgr2=0x7f347c15c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:49.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f3479d9b640 1 --2- 172.21.15.88:0/19062543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f347c0ca2f0 0x7f347c15c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:49.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f3479d9b640 1 -- 172.21.15.88:0/19062543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f347c15cdb0 msgr2=0x7f347c145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:49.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f3479d9b640 1 --2- 172.21.15.88:0/19062543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f347c15cdb0 0x7f347c145b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:49.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f3479d9b640 1 -- 172.21.15.88:0/19062543 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34700094d0 con 0x7f347c1537c0 2023-10-16T20:55:49.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f3479d9b640 1 --2- 172.21.15.88:0/19062543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f347c1537c0 0x7f347c1f2520 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f347001b8c0 tx=0x7f347001ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:49.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f346affd640 1 -- 172.21.15.88:0/19062543 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34700b1d90 con 0x7f347c1537c0 2023-10-16T20:55:49.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f346affd640 1 -- 172.21.15.88:0/19062543 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34700bb470 con 0x7f347c1537c0 2023-10-16T20:55:49.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f346affd640 1 -- 172.21.15.88:0/19062543 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34700bb800 con 0x7f347c1537c0 2023-10-16T20:55:49.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f347b59e640 1 -- 172.21.15.88:0/19062543 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f347c1f2a60 con 0x7f347c1537c0 2023-10-16T20:55:49.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.464+0000 7f347b59e640 1 -- 172.21.15.88:0/19062543 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347c1e4f30 con 0x7f347c1537c0 2023-10-16T20:55:49.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.468+0000 7f347b59e640 1 -- 172.21.15.88:0/19062543 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f347c04e9c0 con 0x7f347c1537c0 2023-10-16T20:55:49.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.468+0000 7f346affd640 1 -- 172.21.15.88:0/19062543 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f34700b7070 con 0x7f347c1537c0 2023-10-16T20:55:49.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.468+0000 7f346affd640 1 --2- 172.21.15.88:0/19062543 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f345c0664a0 0x7f345c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:49.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.468+0000 7f346affd640 1 -- 172.21.15.88:0/19062543 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7265+0+0 (secure 0 0 0) 0x7f34701224a0 con 0x7f347c1537c0 2023-10-16T20:55:49.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.468+0000 7f347a59c640 1 --2- 172.21.15.88:0/19062543 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f345c0664a0 0x7f345c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:49.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.468+0000 7f347a59c640 1 --2- 172.21.15.88:0/19062543 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f345c0664a0 0x7f345c068960 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f346c007a60 tx=0x7f346c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:49.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.472+0000 7f346affd640 1 -- 172.21.15.88:0/19062543 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f34700ef310 con 0x7f347c1537c0 2023-10-16T20:55:49.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.652+0000 7f347b59e640 1 -- 172.21.15.88:0/19062543 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f347c04e9c0 con 0x7f347c1537c0 2023-10-16T20:55:49.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.652+0000 7f346affd640 1 -- 172.21.15.88:0/19062543 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+6569 (secure 0 0 0) 0x7f34700f2930 con 0x7f347c1537c0 2023-10-16T20:55:49.653 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:55:49.653 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5477,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:54:28.552665+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5282,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5282":{"gid":5282,"name":"d","rank":0,"incarnation":81,"state":"up:active","state_seq":41,"addr":"172.21.15.191:6833/3633326602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3633326602},{"type":"v1","addr":"172.21.15.191:6833","nonce":3633326602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:55:49.655 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 96 2023-10-16T20:55:49.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.656+0000 7f347b59e640 1 -- 172.21.15.88:0/19062543 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f345c0664a0 msgr2=0x7f345c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.656+0000 7f347b59e640 1 --2- 172.21.15.88:0/19062543 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f345c0664a0 0x7f345c068960 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f346c007a60 tx=0x7f346c008040 comp rx=0 tx=0).stop 2023-10-16T20:55:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.656+0000 7f347b59e640 1 -- 172.21.15.88:0/19062543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f347c1537c0 msgr2=0x7f347c1f2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:49.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.656+0000 7f347b59e640 1 --2- 172.21.15.88:0/19062543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f347c1537c0 0x7f347c1f2520 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f347001b8c0 tx=0x7f347001ba50 comp rx=0 tx=0).stop 2023-10-16T20:55:49.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.656+0000 7f347b59e640 1 -- 172.21.15.88:0/19062543 shutdown_connections 2023-10-16T20:55:49.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.656+0000 7f347b59e640 1 --2- 172.21.15.88:0/19062543 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f345c0664a0 0x7f345c068960 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:49.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.656+0000 7f347b59e640 1 --2- 172.21.15.88:0/19062543 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f347c15cdb0 0x7f347c145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:49.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.656+0000 7f347b59e640 1 --2- 172.21.15.88:0/19062543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f347c1537c0 0x7f347c1f2520 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.656+0000 7f347b59e640 1 --2- 172.21.15.88:0/19062543 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f347c0ca2f0 0x7f347c15c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.656+0000 7f347b59e640 1 -- 172.21.15.88:0/19062543 >> 172.21.15.88:0/19062543 conn(0x7f347c06c170 msgr2=0x7f347c0cb480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:49.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.656+0000 7f347b59e640 1 -- 172.21.15.88:0/19062543 shutdown_connections 2023-10-16T20:55:49.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:49.656+0000 7f347b59e640 1 -- 172.21.15.88:0/19062543 wait complete. 2023-10-16T20:55:49.672 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-16T20:55:51.674 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:55:51.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.820+0000 7f17ee19d640 1 Processor -- start 2023-10-16T20:55:51.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.820+0000 7f17ee19d640 1 -- start start 2023-10-16T20:55:51.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.820+0000 7f17ee19d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f17e8153770 0x7f17e8159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:51.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.820+0000 7f17ee19d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e815a410 con 0x7f17e80c9bb0 2023-10-16T20:55:51.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.820+0000 7f17ee19d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e8153b50 con 0x7f17e80cece0 2023-10-16T20:55:51.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.820+0000 7f17ee19d640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e80cc250 con 0x7f17e8153770 2023-10-16T20:55:51.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.820+0000 7f17e6ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f17e80c9bb0 0x7f17e80c3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58706/0 (socket says 172.21.15.88:58706) 2023-10-16T20:55:51.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.820+0000 7f17e6ffd640 1 -- 172.21.15.88:0/3376666623 learned_addr learned my addr 172.21.15.88:0/3376666623 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:55:51.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.820+0000 7f17e77fe640 1 --2- 172.21.15.88:0/3376666623 >> v2:172.21.15.191:3300/0 conn(0x7f17e8153770 0x7f17e8159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:51.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.820+0000 7f17e67fc640 1 -- 172.21.15.88:0/3376666623 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897469280 0 0) 0x7f17e815a410 con 0x7f17e80c9bb0 2023-10-16T20:55:51.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.820+0000 7f17e67fc640 1 -- 172.21.15.88:0/3376666623 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17d0003530 con 0x7f17e80c9bb0 2023-10-16T20:55:51.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.820+0000 7f17e67fc640 1 -- 172.21.15.88:0/3376666623 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600988400 0 0) 0x7f17e8153b50 con 0x7f17e80cece0 2023-10-16T20:55:51.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.820+0000 7f17e67fc640 1 -- 172.21.15.88:0/3376666623 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17e815a410 con 0x7f17e80cece0 2023-10-16T20:55:51.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.820+0000 7f17e77fe640 1 -- 172.21.15.88:0/3376666623 >> v1:172.21.15.191:6789/0 conn(0x7f17e80cece0 legacy=0x7f17e8163dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:51.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.820+0000 7f17e77fe640 1 -- 172.21.15.88:0/3376666623 >> v1:172.21.15.88:6789/0 conn(0x7f17e80c9bb0 legacy=0x7f17e80c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:51.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.820+0000 7f17e77fe640 1 -- 172.21.15.88:0/3376666623 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e80cb2f0 con 0x7f17e8153770 2023-10-16T20:55:51.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.824+0000 7f17e77fe640 1 --2- 172.21.15.88:0/3376666623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17e8153770 0x7f17e8159ed0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f17d80097a0 tx=0x7f17d801c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a685ef58ac4b88 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:51.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.824+0000 7f17e67fc640 1 -- 172.21.15.88:0/3376666623 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d80ab600 con 0x7f17e8153770 2023-10-16T20:55:51.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.824+0000 7f17e67fc640 1 -- 172.21.15.88:0/3376666623 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d80a7050 con 0x7f17e8153770 2023-10-16T20:55:51.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.824+0000 7f17e67fc640 1 -- 172.21.15.88:0/3376666623 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d80ab980 con 0x7f17e8153770 2023-10-16T20:55:51.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.824+0000 7f17ee19d640 1 -- 172.21.15.88:0/3376666623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17e8153770 msgr2=0x7f17e8159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:51.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.824+0000 7f17ee19d640 1 --2- 172.21.15.88:0/3376666623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17e8153770 0x7f17e8159ed0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f17d80097a0 tx=0x7f17d801c460 comp rx=0 tx=0).stop 2023-10-16T20:55:51.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.824+0000 7f17ee19d640 1 -- 172.21.15.88:0/3376666623 shutdown_connections 2023-10-16T20:55:51.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.824+0000 7f17ee19d640 1 --2- 172.21.15.88:0/3376666623 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17e8153770 0x7f17e8159ed0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:51.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.824+0000 7f17ee19d640 1 -- 172.21.15.88:0/3376666623 >> 172.21.15.88:0/3376666623 conn(0x7f17e80715c0 msgr2=0x7f17e80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:51.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.824+0000 7f17ee19d640 1 -- 172.21.15.88:0/3376666623 shutdown_connections 2023-10-16T20:55:51.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.824+0000 7f17ee19d640 1 -- 172.21.15.88:0/3376666623 wait complete. 2023-10-16T20:55:51.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17ee19d640 1 Processor -- start 2023-10-16T20:55:51.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17ee19d640 1 -- start start 2023-10-16T20:55:51.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17ee19d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17e80c9bb0 0x7f17e813cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:51.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17ee19d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17e80cece0 0x7f17e8142a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:51.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17ee19d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17e8153770 0x7f17e8142f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:51.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17ee19d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e8153d40 con 0x7f17e80c9bb0 2023-10-16T20:55:51.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17ee19d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e8149d40 con 0x7f17e80cece0 2023-10-16T20:55:51.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17ee19d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e8149fe0 con 0x7f17e8153770 2023-10-16T20:55:51.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17e77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17e80c9bb0 0x7f17e813cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:51.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17e77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17e80c9bb0 0x7f17e813cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36272/0 (socket says 172.21.15.88:36272) 2023-10-16T20:55:51.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17e77fe640 1 -- 172.21.15.88:0/1228977441 learned_addr learned my addr 172.21.15.88:0/1228977441 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:51.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17e6ffd640 1 --2- 172.21.15.88:0/1228977441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17e80cece0 0x7f17e8142a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17e7fff640 1 --2- 172.21.15.88:0/1228977441 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17e8153770 0x7f17e8142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17e77fe640 1 -- 172.21.15.88:0/1228977441 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17e8153770 msgr2=0x7f17e8142f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17e77fe640 1 --2- 172.21.15.88:0/1228977441 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17e8153770 0x7f17e8142f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17e77fe640 1 -- 172.21.15.88:0/1228977441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17e80cece0 msgr2=0x7f17e8142a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:51.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17e77fe640 1 --2- 172.21.15.88:0/1228977441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17e80cece0 0x7f17e8142a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17e77fe640 1 -- 172.21.15.88:0/1228977441 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d80093b0 con 0x7f17e80c9bb0 2023-10-16T20:55:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17e6ffd640 1 --2- 172.21.15.88:0/1228977441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17e80cece0 0x7f17e8142a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:55:51.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17e77fe640 1 --2- 172.21.15.88:0/1228977441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17e80c9bb0 0x7f17e813cae0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f17d80ae040 tx=0x7f17d8019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:51.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17cffff640 1 -- 172.21.15.88:0/1228977441 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d80059a0 con 0x7f17e80c9bb0 2023-10-16T20:55:51.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17cffff640 1 -- 172.21.15.88:0/1228977441 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d8005b20 con 0x7f17e80c9bb0 2023-10-16T20:55:51.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17ee19d640 1 -- 172.21.15.88:0/1228977441 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e81f59b0 con 0x7f17e80c9bb0 2023-10-16T20:55:51.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17cffff640 1 -- 172.21.15.88:0/1228977441 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d8006020 con 0x7f17e80c9bb0 2023-10-16T20:55:51.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17ee19d640 1 -- 172.21.15.88:0/1228977441 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e8153f90 con 0x7f17e80c9bb0 2023-10-16T20:55:51.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.828+0000 7f17ee19d640 1 -- 172.21.15.88:0/1228977441 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e81f5e60 con 0x7f17e80c9bb0 2023-10-16T20:55:51.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.832+0000 7f17cffff640 1 -- 172.21.15.88:0/1228977441 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f17d80b2040 con 0x7f17e80c9bb0 2023-10-16T20:55:51.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.832+0000 7f17cffff640 1 --2- 172.21.15.88:0/1228977441 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f17d40664a0 0x7f17d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:51.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.832+0000 7f17cffff640 1 -- 172.21.15.88:0/1228977441 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7f17d811d980 con 0x7f17e80c9bb0 2023-10-16T20:55:51.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.832+0000 7f17e6ffd640 1 --2- 172.21.15.88:0/1228977441 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f17d40664a0 0x7f17d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:51.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.832+0000 7f17e6ffd640 1 --2- 172.21.15.88:0/1228977441 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f17d40664a0 0x7f17d4068960 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f17e815a3a0 tx=0x7f17dc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:51.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:51.836+0000 7f17cffff640 1 -- 172.21.15.88:0/1228977441 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f17d80ea5e0 con 0x7f17e80c9bb0 2023-10-16T20:55:52.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.024+0000 7f17ee19d640 1 -- 172.21.15.88:0/1228977441 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17e8147360 con 0x7f17e80c9bb0 2023-10-16T20:55:52.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.028+0000 7f17cffff640 1 -- 172.21.15.88:0/1228977441 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5775 (secure 0 0 0) 0x7f17d80edc00 con 0x7f17e80c9bb0 2023-10-16T20:55:52.025 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:55:52.026 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:55:50.604971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:55:52.026 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 97 2023-10-16T20:55:52.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.028+0000 7f17ee19d640 1 -- 172.21.15.88:0/1228977441 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f17d40664a0 msgr2=0x7f17d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:52.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.028+0000 7f17ee19d640 1 --2- 172.21.15.88:0/1228977441 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f17d40664a0 0x7f17d4068960 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f17e815a3a0 tx=0x7f17dc009040 comp rx=0 tx=0).stop 2023-10-16T20:55:52.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.028+0000 7f17ee19d640 1 -- 172.21.15.88:0/1228977441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17e80c9bb0 msgr2=0x7f17e813cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:52.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.028+0000 7f17ee19d640 1 --2- 172.21.15.88:0/1228977441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17e80c9bb0 0x7f17e813cae0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f17d80ae040 tx=0x7f17d8019200 comp rx=0 tx=0).stop 2023-10-16T20:55:52.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.028+0000 7f17ee19d640 1 -- 172.21.15.88:0/1228977441 shutdown_connections 2023-10-16T20:55:52.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.028+0000 7f17ee19d640 1 --2- 172.21.15.88:0/1228977441 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f17d40664a0 0x7f17d4068960 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:52.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.028+0000 7f17ee19d640 1 --2- 172.21.15.88:0/1228977441 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f17e8153770 0x7f17e8142f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:52.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.028+0000 7f17ee19d640 1 --2- 172.21.15.88:0/1228977441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f17e80cece0 0x7f17e8142a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:52.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.028+0000 7f17ee19d640 1 --2- 172.21.15.88:0/1228977441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17e80c9bb0 0x7f17e813cae0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:52.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.028+0000 7f17ee19d640 1 -- 172.21.15.88:0/1228977441 >> 172.21.15.88:0/1228977441 conn(0x7f17e80715c0 msgr2=0x7f17e80b4140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:52.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.032+0000 7f17ee19d640 1 -- 172.21.15.88:0/1228977441 shutdown_connections 2023-10-16T20:55:52.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.032+0000 7f17ee19d640 1 -- 172.21.15.88:0/1228977441 wait complete. 2023-10-16T20:55:52.056 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-16T20:55:52.057 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-16T20:55:52.057 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:55:52.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.260+0000 7facfcc17640 1 Processor -- start 2023-10-16T20:55:52.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.260+0000 7facfcc17640 1 -- start start 2023-10-16T20:55:52.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.264+0000 7facfcc17640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7facf806a970 0x7facf8071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:52.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.264+0000 7facfcc17640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7facf8071e60 0x7facf815a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:52.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.264+0000 7facfcc17640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf815b3c0 con 0x7facf81406d0 2023-10-16T20:55:52.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.264+0000 7facfcc17640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf80c2460 con 0x7facf806a970 2023-10-16T20:55:52.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.264+0000 7facfcc17640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf815b680 con 0x7facf8071e60 2023-10-16T20:55:52.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.264+0000 7facf5d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7facf806a970 0x7facf8071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:52.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.264+0000 7facf5d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7facf806a970 0x7facf8071920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36282/0 (socket says 172.21.15.88:36282) 2023-10-16T20:55:52.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.264+0000 7facf5d9b640 1 -- 172.21.15.88:0/3612858275 learned_addr learned my addr 172.21.15.88:0/3612858275 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:52.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.264+0000 7facf659c640 1 --1- 172.21.15.88:0/3612858275 >> v1:172.21.15.191:6789/0 conn(0x7facf81406d0 0x7facf80c1d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:51946/0 (socket says 172.21.15.88:51946) 2023-10-16T20:55:52.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.264+0000 7facf6d9d640 1 --2- 172.21.15.88:0/3612858275 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7facf8071e60 0x7facf815a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:52.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.264+0000 7facf5d9b640 1 -- 172.21.15.88:0/3612858275 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7facf8071e60 msgr2=0x7facf815a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:55:52.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.264+0000 7facf5d9b640 1 --2- 172.21.15.88:0/3612858275 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7facf8071e60 0x7facf815a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:52.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.264+0000 7facf559a640 1 -- 172.21.15.88:0/3612858275 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404994413 0 0) 0x7facf815b3c0 con 0x7facf81406d0 2023-10-16T20:55:52.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.264+0000 7facf5d9b640 1 -- 172.21.15.88:0/3612858275 >> v1:172.21.15.191:6789/0 conn(0x7facf81406d0 legacy=0x7facf80c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:52.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.264+0000 7facf5d9b640 1 -- 172.21.15.88:0/3612858275 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf8157570 con 0x7facf806a970 2023-10-16T20:55:52.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.268+0000 7facf5d9b640 1 --2- 172.21.15.88:0/3612858275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facf806a970 0x7facf8071920 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7facec009870 tx=0x7facec01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8c03ec727eea96 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:52.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.268+0000 7facf559a640 1 -- 172.21.15.88:0/3612858275 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec0ab600 con 0x7facf806a970 2023-10-16T20:55:52.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.268+0000 7facf559a640 1 -- 172.21.15.88:0/3612858275 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facec0a7050 con 0x7facf806a970 2023-10-16T20:55:52.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.268+0000 7facf559a640 1 -- 172.21.15.88:0/3612858275 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec0ab980 con 0x7facf806a970 2023-10-16T20:55:52.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.268+0000 7facfcc17640 1 -- 172.21.15.88:0/3612858275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facf806a970 msgr2=0x7facf8071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:52.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.268+0000 7facfcc17640 1 --2- 172.21.15.88:0/3612858275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facf806a970 0x7facf8071920 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7facec009870 tx=0x7facec01c540 comp rx=0 tx=0).stop 2023-10-16T20:55:52.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.268+0000 7facfcc17640 1 -- 172.21.15.88:0/3612858275 shutdown_connections 2023-10-16T20:55:52.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.268+0000 7facfcc17640 1 --2- 172.21.15.88:0/3612858275 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7facf8071e60 0x7facf815a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:52.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.268+0000 7facfcc17640 1 --2- 172.21.15.88:0/3612858275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facf806a970 0x7facf8071920 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:52.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.268+0000 7facfcc17640 1 -- 172.21.15.88:0/3612858275 >> 172.21.15.88:0/3612858275 conn(0x7facf80cad50 msgr2=0x7facf80cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:52.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.268+0000 7facfcc17640 1 -- 172.21.15.88:0/3612858275 shutdown_connections 2023-10-16T20:55:52.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.268+0000 7facfcc17640 1 -- 172.21.15.88:0/3612858275 wait complete. 2023-10-16T20:55:52.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.272+0000 7facfcc17640 1 Processor -- start 2023-10-16T20:55:52.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.272+0000 7facfcc17640 1 -- start start 2023-10-16T20:55:52.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.272+0000 7facfcc17640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7facf806a970 0x7facf80c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:52.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.272+0000 7facfcc17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facf8071e60 0x7facf8153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:52.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.272+0000 7facfcc17640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facf81406d0 0x7facf8154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:52.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.272+0000 7facfcc17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf813d120 con 0x7facf8071e60 2023-10-16T20:55:52.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.272+0000 7facfcc17640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf8070d40 con 0x7facf81406d0 2023-10-16T20:55:52.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.272+0000 7facfcc17640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf81f60d0 con 0x7facf806a970 2023-10-16T20:55:52.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.272+0000 7facf659c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7facf806a970 0x7facf80c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:52.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.272+0000 7facf5d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facf8071e60 0x7facf8153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:52.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.272+0000 7facf659c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7facf806a970 0x7facf80c99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:37150/0 (socket says 172.21.15.88:37150) 2023-10-16T20:55:52.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.272+0000 7facf659c640 1 -- 172.21.15.88:0/4088692198 learned_addr learned my addr 172.21.15.88:0/4088692198 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:52.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.276+0000 7facf6d9d640 1 --2- 172.21.15.88:0/4088692198 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facf81406d0 0x7facf8154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:52.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.276+0000 7facf5d9b640 1 -- 172.21.15.88:0/4088692198 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7facf806a970 msgr2=0x7facf80c99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:52.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.276+0000 7facf5d9b640 1 --2- 172.21.15.88:0/4088692198 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7facf806a970 0x7facf80c99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:52.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.276+0000 7facf5d9b640 1 -- 172.21.15.88:0/4088692198 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facf81406d0 msgr2=0x7facf8154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:52.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.276+0000 7facf5d9b640 1 --2- 172.21.15.88:0/4088692198 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facf81406d0 0x7facf8154520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:52.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.276+0000 7facf5d9b640 1 -- 172.21.15.88:0/4088692198 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facec0094d0 con 0x7facf8071e60 2023-10-16T20:55:52.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.276+0000 7facf659c640 1 --2- 172.21.15.88:0/4088692198 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7facf806a970 0x7facf80c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:55:52.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.276+0000 7facf5d9b640 1 --2- 172.21.15.88:0/4088692198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facf8071e60 0x7facf8153fe0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7facf806e2c0 tx=0x7facec0192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:52.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.276+0000 7face6ffd640 1 -- 172.21.15.88:0/4088692198 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec004220 con 0x7facf8071e60 2023-10-16T20:55:52.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.276+0000 7face6ffd640 1 -- 172.21.15.88:0/4088692198 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facec004380 con 0x7facf8071e60 2023-10-16T20:55:52.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.276+0000 7facfcc17640 1 -- 172.21.15.88:0/4088692198 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf813cb60 con 0x7facf8071e60 2023-10-16T20:55:52.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.276+0000 7face6ffd640 1 -- 172.21.15.88:0/4088692198 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec003820 con 0x7facf8071e60 2023-10-16T20:55:52.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.276+0000 7facfcc17640 1 -- 172.21.15.88:0/4088692198 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf813f9b0 con 0x7facf8071e60 2023-10-16T20:55:52.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.276+0000 7facfcc17640 1 -- 172.21.15.88:0/4088692198 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf813f810 con 0x7facf8071e60 2023-10-16T20:55:52.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.280+0000 7face6ffd640 1 -- 172.21.15.88:0/4088692198 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7facec003980 con 0x7facf8071e60 2023-10-16T20:55:52.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.280+0000 7face6ffd640 1 --2- 172.21.15.88:0/4088692198 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7facd80664f0 0x7facd80689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:52.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.280+0000 7face6ffd640 1 -- 172.21.15.88:0/4088692198 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7facec11db20 con 0x7facf8071e60 2023-10-16T20:55:52.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.280+0000 7facf659c640 1 --2- 172.21.15.88:0/4088692198 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7facd80664f0 0x7facd80689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:52.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.284+0000 7facf659c640 1 --2- 172.21.15.88:0/4088692198 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7facd80664f0 0x7facd80689b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7face8007b20 tx=0x7face8007a70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:52.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.288+0000 7face6ffd640 1 -- 172.21.15.88:0/4088692198 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7facf813f810 con 0x7facf8071e60 2023-10-16T20:55:52.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.568+0000 7facfcc17640 1 -- 172.21.15.88:0/4088692198 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facf813f810 con 0x7facf8071e60 2023-10-16T20:55:52.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.568+0000 7face6ffd640 1 -- 172.21.15.88:0/4088692198 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5775 (secure 0 0 0) 0x7facf813f810 con 0x7facf8071e60 2023-10-16T20:55:52.567 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:55:52.567 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:55:50.604971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:55:52.567 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 97 2023-10-16T20:55:52.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.572+0000 7facfcc17640 1 -- 172.21.15.88:0/4088692198 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7facd80664f0 msgr2=0x7facd80689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:52.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.572+0000 7facfcc17640 1 --2- 172.21.15.88:0/4088692198 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7facd80664f0 0x7facd80689b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7face8007b20 tx=0x7face8007a70 comp rx=0 tx=0).stop 2023-10-16T20:55:52.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.572+0000 7facfcc17640 1 -- 172.21.15.88:0/4088692198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facf8071e60 msgr2=0x7facf8153fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:52.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.572+0000 7facfcc17640 1 --2- 172.21.15.88:0/4088692198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facf8071e60 0x7facf8153fe0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7facf806e2c0 tx=0x7facec0192e0 comp rx=0 tx=0).stop 2023-10-16T20:55:52.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.572+0000 7facfcc17640 1 -- 172.21.15.88:0/4088692198 shutdown_connections 2023-10-16T20:55:52.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.572+0000 7facfcc17640 1 --2- 172.21.15.88:0/4088692198 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7facd80664f0 0x7facd80689b0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:52.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.572+0000 7facfcc17640 1 --2- 172.21.15.88:0/4088692198 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7facf81406d0 0x7facf8154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:52.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.572+0000 7facfcc17640 1 --2- 172.21.15.88:0/4088692198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7facf8071e60 0x7facf8153fe0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:52.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.572+0000 7facfcc17640 1 --2- 172.21.15.88:0/4088692198 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7facf806a970 0x7facf80c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:52.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.572+0000 7facfcc17640 1 -- 172.21.15.88:0/4088692198 >> 172.21.15.88:0/4088692198 conn(0x7facf80cad50 msgr2=0x7facf80c25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:52.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.572+0000 7facfcc17640 1 -- 172.21.15.88:0/4088692198 shutdown_connections 2023-10-16T20:55:52.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:52.572+0000 7facfcc17640 1 -- 172.21.15.88:0/4088692198 wait complete. 2023-10-16T20:55:52.590 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 97, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 97, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5284": { "addr": "172.21.15.88:6839/2172973775", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6837", "nonce": 2172973775, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2172973775, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5284, "incarnation": 38, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 17 }, "gid_5378": { "addr": "172.21.15.191:6835/3119045295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3119045295, "type": "v2" }, { "addr": "172.21.15.191:6835", "nonce": 3119045295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5378, "incarnation": 89, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:active", "state_seq": 32 }, "gid_5477": { "addr": "172.21.15.191:6837/2244284333", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6836", "nonce": 2244284333, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2244284333, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5477, "incarnation": 97, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_5535": { "addr": "172.21.15.88:6835/3476086754", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3476086754, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3476086754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5535, "incarnation": 52, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_5709": { "addr": "172.21.15.88:6838/2600891060", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 2600891060, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 2600891060, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5709, "incarnation": 67, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:55:50.604971+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5477, "mds_1": 5709, "mds_2": 5535, "mds_3": 5284, "mds_4": 5378 } } } ], "standbys": [] } 2023-10-16T20:55:54.591 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:55:54.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.739+0000 7f1ac473e640 1 Processor -- start 2023-10-16T20:55:54.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.739+0000 7f1ac473e640 1 -- start start 2023-10-16T20:55:54.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.743+0000 7f1ac473e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1abc150660 0x7f1abc0c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:54.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.743+0000 7f1ac473e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1abc0c6bc0 0x7f1abc0c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:54.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.743+0000 7f1ac473e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1abc0c5c60 con 0x7f1abc150280 2023-10-16T20:55:54.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.743+0000 7f1ac473e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1abc0c5370 con 0x7f1abc0c6bc0 2023-10-16T20:55:54.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.743+0000 7f1ac473e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1abc156d10 con 0x7f1abc150660 2023-10-16T20:55:54.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.743+0000 7f1ac2cdb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1abc0c6bc0 0x7f1abc0c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:54.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.743+0000 7f1ac2cdb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1abc0c6bc0 0x7f1abc0c3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36308/0 (socket says 172.21.15.88:36308) 2023-10-16T20:55:54.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.743+0000 7f1ac2cdb640 1 -- 172.21.15.88:0/4203044125 learned_addr learned my addr 172.21.15.88:0/4203044125 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:54.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.743+0000 7f1ac24da640 1 --1- 172.21.15.88:0/4203044125 >> v1:172.21.15.88:6789/0 conn(0x7f1abc150280 0x7f1abc06d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58722/0 (socket says 172.21.15.88:58722) 2023-10-16T20:55:54.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.743+0000 7f1ac1cd9640 1 --2- 172.21.15.88:0/4203044125 >> v2:172.21.15.191:3300/0 conn(0x7f1abc150660 0x7f1abc0c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:54.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.743+0000 7f1ac2cdb640 1 -- 172.21.15.88:0/4203044125 >> v2:172.21.15.191:3300/0 conn(0x7f1abc150660 msgr2=0x7f1abc0c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:55:54.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.743+0000 7f1ac2cdb640 1 --2- 172.21.15.88:0/4203044125 >> v2:172.21.15.191:3300/0 conn(0x7f1abc150660 0x7f1abc0c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:54.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.743+0000 7f1ac2cdb640 1 -- 172.21.15.88:0/4203044125 >> v1:172.21.15.88:6789/0 conn(0x7f1abc150280 legacy=0x7f1abc06d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:54.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.743+0000 7f1ac2cdb640 1 -- 172.21.15.88:0/4203044125 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc0c5610 con 0x7f1abc0c6bc0 2023-10-16T20:55:54.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.743+0000 7f1ac14d8640 1 -- 172.21.15.88:0/4203044125 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418218059 0 0) 0x7f1abc0c5c60 con 0x7f1abc150280 2023-10-16T20:55:54.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.743+0000 7f1ac1cd9640 1 --2- 172.21.15.88:0/4203044125 >> v2:172.21.15.191:3300/0 conn(0x7f1abc150660 0x7f1abc0c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:55:54.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac2cdb640 1 --2- 172.21.15.88:0/4203044125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc0c6bc0 0x7f1abc0c3110 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0c1290 tx=0x7f1ab401d400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81682ffdb8ae64d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:54.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac14d8640 1 -- 172.21.15.88:0/4203044125 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab40a5880 con 0x7f1abc0c6bc0 2023-10-16T20:55:54.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac14d8640 1 -- 172.21.15.88:0/4203044125 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab40a59e0 con 0x7f1abc0c6bc0 2023-10-16T20:55:54.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac14d8640 1 -- 172.21.15.88:0/4203044125 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab40a5d70 con 0x7f1abc0c6bc0 2023-10-16T20:55:54.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac473e640 1 -- 172.21.15.88:0/4203044125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc0c6bc0 msgr2=0x7f1abc0c3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:54.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac473e640 1 --2- 172.21.15.88:0/4203044125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc0c6bc0 0x7f1abc0c3110 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0c1290 tx=0x7f1ab401d400 comp rx=0 tx=0).stop 2023-10-16T20:55:54.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac473e640 1 -- 172.21.15.88:0/4203044125 shutdown_connections 2023-10-16T20:55:54.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac473e640 1 --2- 172.21.15.88:0/4203044125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc0c6bc0 0x7f1abc0c3110 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:54.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac473e640 1 --2- 172.21.15.88:0/4203044125 >> v2:172.21.15.191:3300/0 conn(0x7f1abc150660 0x7f1abc0c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:54.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac473e640 1 -- 172.21.15.88:0/4203044125 >> 172.21.15.88:0/4203044125 conn(0x7f1abc0cad50 msgr2=0x7f1abc0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:54.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac473e640 1 -- 172.21.15.88:0/4203044125 shutdown_connections 2023-10-16T20:55:54.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac473e640 1 -- 172.21.15.88:0/4203044125 wait complete. 2023-10-16T20:55:54.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac473e640 1 Processor -- start 2023-10-16T20:55:54.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac473e640 1 -- start start 2023-10-16T20:55:54.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac473e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1abc0c6bc0 0x7f1abc140760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:54.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac473e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1abc150280 0x7f1abc0c3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:54.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac473e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc150660 0x7f1abc0c3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:54.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac473e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc14d650 con 0x7f1abc150660 2023-10-16T20:55:54.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac473e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc150b80 con 0x7f1abc0c6bc0 2023-10-16T20:55:54.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.747+0000 7f1ac473e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc071040 con 0x7f1abc150280 2023-10-16T20:55:54.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ac2cdb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc150660 0x7f1abc0c3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:54.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ac2cdb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc150660 0x7f1abc0c3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36318/0 (socket says 172.21.15.88:36318) 2023-10-16T20:55:54.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ac2cdb640 1 -- 172.21.15.88:0/2535923412 learned_addr learned my addr 172.21.15.88:0/2535923412 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:54.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ac24da640 1 --2- 172.21.15.88:0/2535923412 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1abc0c6bc0 0x7f1abc140760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:54.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ac1cd9640 1 --2- 172.21.15.88:0/2535923412 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1abc150280 0x7f1abc0c3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:54.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ac2cdb640 1 -- 172.21.15.88:0/2535923412 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1abc150280 msgr2=0x7f1abc0c3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:54.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ac2cdb640 1 --2- 172.21.15.88:0/2535923412 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1abc150280 0x7f1abc0c3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:54.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ac2cdb640 1 -- 172.21.15.88:0/2535923412 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1abc0c6bc0 msgr2=0x7f1abc140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:54.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ac2cdb640 1 --2- 172.21.15.88:0/2535923412 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1abc0c6bc0 0x7f1abc140760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:54.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ac2cdb640 1 -- 172.21.15.88:0/2535923412 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab4009b30 con 0x7f1abc150660 2023-10-16T20:55:54.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ac1cd9640 1 --2- 172.21.15.88:0/2535923412 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1abc150280 0x7f1abc0c3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:55:54.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ac2cdb640 1 --2- 172.21.15.88:0/2535923412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc150660 0x7f1abc0c3a40 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40af040 tx=0x7f1ab4019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:54.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ab2ffd640 1 -- 172.21.15.88:0/2535923412 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab40b8880 con 0x7f1abc150660 2023-10-16T20:55:54.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ab2ffd640 1 -- 172.21.15.88:0/2535923412 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab40b8a00 con 0x7f1abc150660 2023-10-16T20:55:54.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ac473e640 1 -- 172.21.15.88:0/2535923412 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1abc071320 con 0x7f1abc150660 2023-10-16T20:55:54.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ab2ffd640 1 -- 172.21.15.88:0/2535923412 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab40b8db0 con 0x7f1abc150660 2023-10-16T20:55:54.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ac473e640 1 -- 172.21.15.88:0/2535923412 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1abc1510b0 con 0x7f1abc150660 2023-10-16T20:55:54.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ac473e640 1 -- 172.21.15.88:0/2535923412 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1abc1512a0 con 0x7f1abc150660 2023-10-16T20:55:54.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ab2ffd640 1 -- 172.21.15.88:0/2535923412 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1ab40c3460 con 0x7f1abc150660 2023-10-16T20:55:54.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.751+0000 7f1ab2ffd640 1 --2- 172.21.15.88:0/2535923412 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1aa00664a0 0x7f1aa0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:54.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.755+0000 7f1ab2ffd640 1 -- 172.21.15.88:0/2535923412 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7f1ab411fbf0 con 0x7f1abc150660 2023-10-16T20:55:54.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.755+0000 7f1ac24da640 1 --2- 172.21.15.88:0/2535923412 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1aa00664a0 0x7f1aa0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:54.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.755+0000 7f1ac24da640 1 --2- 172.21.15.88:0/2535923412 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1aa00664a0 0x7f1aa0068960 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8007fd0 tx=0x7f1ab8007d60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:54.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:54.759+0000 7f1ab2ffd640 1 -- 172.21.15.88:0/2535923412 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1ab40eca80 con 0x7f1abc150660 2023-10-16T20:55:55.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:55.035+0000 7f1ac473e640 1 -- 172.21.15.88:0/2535923412 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1abc0714e0 con 0x7f1abc150660 2023-10-16T20:55:55.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:55.035+0000 7f1ab2ffd640 1 -- 172.21.15.88:0/2535923412 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5775 (secure 0 0 0) 0x7f1ab40f00a0 con 0x7f1abc150660 2023-10-16T20:55:55.034 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:55:55.034 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:55:50.604971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:55:55.035 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 97 2023-10-16T20:55:55.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:55.039+0000 7f1ac473e640 1 -- 172.21.15.88:0/2535923412 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1aa00664a0 msgr2=0x7f1aa0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:55.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:55.039+0000 7f1ac473e640 1 --2- 172.21.15.88:0/2535923412 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1aa00664a0 0x7f1aa0068960 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8007fd0 tx=0x7f1ab8007d60 comp rx=0 tx=0).stop 2023-10-16T20:55:55.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:55.039+0000 7f1ac473e640 1 -- 172.21.15.88:0/2535923412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc150660 msgr2=0x7f1abc0c3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:55.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:55.039+0000 7f1ac473e640 1 --2- 172.21.15.88:0/2535923412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc150660 0x7f1abc0c3a40 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40af040 tx=0x7f1ab4019740 comp rx=0 tx=0).stop 2023-10-16T20:55:55.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:55.039+0000 7f1ac473e640 1 -- 172.21.15.88:0/2535923412 shutdown_connections 2023-10-16T20:55:55.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:55.039+0000 7f1ac473e640 1 --2- 172.21.15.88:0/2535923412 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1aa00664a0 0x7f1aa0068960 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:55.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:55.039+0000 7f1ac473e640 1 --2- 172.21.15.88:0/2535923412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1abc150660 0x7f1abc0c3a40 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:55.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:55.039+0000 7f1ac473e640 1 --2- 172.21.15.88:0/2535923412 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1abc150280 0x7f1abc0c3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:55.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:55.039+0000 7f1ac473e640 1 --2- 172.21.15.88:0/2535923412 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1abc0c6bc0 0x7f1abc140760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:55.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:55.039+0000 7f1ac473e640 1 -- 172.21.15.88:0/2535923412 >> 172.21.15.88:0/2535923412 conn(0x7f1abc0cad50 msgr2=0x7f1abc1416f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:55.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:55.039+0000 7f1ac473e640 1 -- 172.21.15.88:0/2535923412 shutdown_connections 2023-10-16T20:55:55.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:55.039+0000 7f1ac473e640 1 -- 172.21.15.88:0/2535923412 wait complete. 2023-10-16T20:55:55.060 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:55:57.061 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:55:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.199+0000 7fdfba6c9640 1 Processor -- start 2023-10-16T20:55:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.199+0000 7fdfba6c9640 1 -- start start 2023-10-16T20:55:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.203+0000 7fdfba6c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb4150660 0x7fdfb40c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.203+0000 7fdfba6c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdfb40c6bc0 0x7fdfb40c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.203+0000 7fdfba6c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb40c5c60 con 0x7fdfb4150280 2023-10-16T20:55:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.203+0000 7fdfba6c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb40c5370 con 0x7fdfb40c6bc0 2023-10-16T20:55:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.203+0000 7fdfba6c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb4156d10 con 0x7fdfb4150660 2023-10-16T20:55:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.203+0000 7fdfb3fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdfb4150280 0x7fdfb406d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58728/0 (socket says 172.21.15.88:58728) 2023-10-16T20:55:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.203+0000 7fdfb3fff640 1 -- 172.21.15.88:0/1835220574 learned_addr learned my addr 172.21.15.88:0/1835220574 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:55:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.203+0000 7fdfb8c66640 1 --2- 172.21.15.88:0/1835220574 >> v2:172.21.15.88:3300/0 conn(0x7fdfb40c6bc0 0x7fdfb40c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.203+0000 7fdfb37fe640 1 --2- 172.21.15.88:0/1835220574 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb4150660 0x7fdfb40c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:57.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.203+0000 7fdfb2ffd640 1 -- 172.21.15.88:0/1835220574 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635484022 0 0) 0x7fdfb40c5c60 con 0x7fdfb4150280 2023-10-16T20:55:57.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.203+0000 7fdfb2ffd640 1 -- 172.21.15.88:0/1835220574 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf9c005740 con 0x7fdfb4150280 2023-10-16T20:55:57.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.203+0000 7fdfb8c66640 1 -- 172.21.15.88:0/1835220574 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb4150660 msgr2=0x7fdfb40c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:55:57.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.203+0000 7fdfb8c66640 1 --2- 172.21.15.88:0/1835220574 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb4150660 0x7fdfb40c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:57.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.203+0000 7fdfb8c66640 1 -- 172.21.15.88:0/1835220574 >> v1:172.21.15.88:6789/0 conn(0x7fdfb4150280 legacy=0x7fdfb406d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:57.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.203+0000 7fdfb8c66640 1 -- 172.21.15.88:0/1835220574 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb40c5610 con 0x7fdfb40c6bc0 2023-10-16T20:55:57.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.203+0000 7fdfb37fe640 1 --2- 172.21.15.88:0/1835220574 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb4150660 0x7fdfb40c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:55:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.203+0000 7fdfb8c66640 1 --2- 172.21.15.88:0/1835220574 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfb40c6bc0 0x7fdfb40c3110 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009d90 tx=0x7fdfac01d2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221fbd4b1b0abefb server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfb2ffd640 1 -- 172.21.15.88:0/1835220574 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac0a5970 con 0x7fdfb40c6bc0 2023-10-16T20:55:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfb2ffd640 1 -- 172.21.15.88:0/1835220574 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfac0a5ad0 con 0x7fdfb40c6bc0 2023-10-16T20:55:57.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfb2ffd640 1 -- 172.21.15.88:0/1835220574 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac0a5e60 con 0x7fdfb40c6bc0 2023-10-16T20:55:57.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfba6c9640 1 -- 172.21.15.88:0/1835220574 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfb40c6bc0 msgr2=0x7fdfb40c3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:57.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfba6c9640 1 --2- 172.21.15.88:0/1835220574 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfb40c6bc0 0x7fdfb40c3110 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009d90 tx=0x7fdfac01d2f0 comp rx=0 tx=0).stop 2023-10-16T20:55:57.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfba6c9640 1 -- 172.21.15.88:0/1835220574 shutdown_connections 2023-10-16T20:55:57.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfba6c9640 1 --2- 172.21.15.88:0/1835220574 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfb40c6bc0 0x7fdfb40c3110 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfba6c9640 1 --2- 172.21.15.88:0/1835220574 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb4150660 0x7fdfb40c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfba6c9640 1 -- 172.21.15.88:0/1835220574 >> 172.21.15.88:0/1835220574 conn(0x7fdfb40cad50 msgr2=0x7fdfb40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfba6c9640 1 -- 172.21.15.88:0/1835220574 shutdown_connections 2023-10-16T20:55:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfba6c9640 1 -- 172.21.15.88:0/1835220574 wait complete. 2023-10-16T20:55:57.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfba6c9640 1 Processor -- start 2023-10-16T20:55:57.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfba6c9640 1 -- start start 2023-10-16T20:55:57.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfba6c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb40c6bc0 0x7fdfb413d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:57.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfba6c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfb4150280 0x7fdfb4152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:57.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfba6c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfb4150660 0x7fdfb40c3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:57.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfba6c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb41f22b0 con 0x7fdfb4150660 2023-10-16T20:55:57.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfba6c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb41f19a0 con 0x7fdfb4150280 2023-10-16T20:55:57.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfba6c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb40c9020 con 0x7fdfb40c6bc0 2023-10-16T20:55:57.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfb8c66640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfb4150660 0x7fdfb40c3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:57.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfb8c66640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfb4150660 0x7fdfb40c3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36348/0 (socket says 172.21.15.88:36348) 2023-10-16T20:55:57.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.207+0000 7fdfb8c66640 1 -- 172.21.15.88:0/1775940136 learned_addr learned my addr 172.21.15.88:0/1775940136 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:57.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.211+0000 7fdfb3fff640 1 --2- 172.21.15.88:0/1775940136 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb40c6bc0 0x7fdfb413d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:57.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.211+0000 7fdfb37fe640 1 --2- 172.21.15.88:0/1775940136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfb4150280 0x7fdfb4152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:57.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.211+0000 7fdfb8c66640 1 -- 172.21.15.88:0/1775940136 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb40c6bc0 msgr2=0x7fdfb413d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:57.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.211+0000 7fdfb8c66640 1 --2- 172.21.15.88:0/1775940136 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb40c6bc0 0x7fdfb413d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:57.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.211+0000 7fdfb8c66640 1 -- 172.21.15.88:0/1775940136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfb4150280 msgr2=0x7fdfb4152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:57.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.211+0000 7fdfb8c66640 1 --2- 172.21.15.88:0/1775940136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfb4150280 0x7fdfb4152220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:57.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.211+0000 7fdfb8c66640 1 -- 172.21.15.88:0/1775940136 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfac009a70 con 0x7fdfb4150660 2023-10-16T20:55:57.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.211+0000 7fdfb37fe640 1 --2- 172.21.15.88:0/1775940136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfb4150280 0x7fdfb4152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:55:57.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.211+0000 7fdfb8c66640 1 --2- 172.21.15.88:0/1775940136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfb4150660 0x7fdfb40c3f30 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0b0040 tx=0x7fdfac019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:57.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.211+0000 7fdfb0ff9640 1 -- 172.21.15.88:0/1775940136 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac0add70 con 0x7fdfb4150660 2023-10-16T20:55:57.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.211+0000 7fdfb0ff9640 1 -- 172.21.15.88:0/1775940136 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfac009450 con 0x7fdfb4150660 2023-10-16T20:55:57.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.211+0000 7fdfb0ff9640 1 -- 172.21.15.88:0/1775940136 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac0b8660 con 0x7fdfb4150660 2023-10-16T20:55:57.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.211+0000 7fdfba6c9640 1 -- 172.21.15.88:0/1775940136 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb41f1c60 con 0x7fdfb4150660 2023-10-16T20:55:57.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.211+0000 7fdfba6c9640 1 -- 172.21.15.88:0/1775940136 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb4152760 con 0x7fdfb4150660 2023-10-16T20:55:57.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.211+0000 7fdfba6c9640 1 -- 172.21.15.88:0/1775940136 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb41f2420 con 0x7fdfb4150660 2023-10-16T20:55:57.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.211+0000 7fdfb0ff9640 1 -- 172.21.15.88:0/1775940136 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdfac0b4040 con 0x7fdfb4150660 2023-10-16T20:55:57.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.211+0000 7fdfb0ff9640 1 --2- 172.21.15.88:0/1775940136 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdfa00664a0 0x7fdfa0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:57.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.215+0000 7fdfb0ff9640 1 -- 172.21.15.88:0/1775940136 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7fdfac11f910 con 0x7fdfb4150660 2023-10-16T20:55:57.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.215+0000 7fdfb3fff640 1 --2- 172.21.15.88:0/1775940136 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdfa00664a0 0x7fdfa0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:57.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.215+0000 7fdfb3fff640 1 --2- 172.21.15.88:0/1775940136 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdfa00664a0 0x7fdfa0068960 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fdfa40011f0 tx=0x7fdfa4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:57.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.219+0000 7fdfb0ff9640 1 -- 172.21.15.88:0/1775940136 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdfac0ec860 con 0x7fdfb4150660 2023-10-16T20:55:57.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.395+0000 7fdfba6c9640 1 -- 172.21.15.88:0/1775940136 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfb4074700 con 0x7fdfb4150660 2023-10-16T20:55:57.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.399+0000 7fdfb0ff9640 1 -- 172.21.15.88:0/1775940136 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5775 (secure 0 0 0) 0x7fdfac0efe80 con 0x7fdfb4150660 2023-10-16T20:55:57.397 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:55:57.397 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:55:50.604971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:55:57.398 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 97 2023-10-16T20:55:57.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.399+0000 7fdfba6c9640 1 -- 172.21.15.88:0/1775940136 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdfa00664a0 msgr2=0x7fdfa0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:57.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.399+0000 7fdfba6c9640 1 --2- 172.21.15.88:0/1775940136 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdfa00664a0 0x7fdfa0068960 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fdfa40011f0 tx=0x7fdfa4009040 comp rx=0 tx=0).stop 2023-10-16T20:55:57.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.399+0000 7fdfba6c9640 1 -- 172.21.15.88:0/1775940136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfb4150660 msgr2=0x7fdfb40c3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:57.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.399+0000 7fdfba6c9640 1 --2- 172.21.15.88:0/1775940136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfb4150660 0x7fdfb40c3f30 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0b0040 tx=0x7fdfac019630 comp rx=0 tx=0).stop 2023-10-16T20:55:57.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.399+0000 7fdfba6c9640 1 -- 172.21.15.88:0/1775940136 shutdown_connections 2023-10-16T20:55:57.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.399+0000 7fdfba6c9640 1 --2- 172.21.15.88:0/1775940136 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fdfa00664a0 0x7fdfa0068960 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:57.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.399+0000 7fdfba6c9640 1 --2- 172.21.15.88:0/1775940136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfb4150660 0x7fdfb40c3f30 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:57.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.399+0000 7fdfba6c9640 1 --2- 172.21.15.88:0/1775940136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdfb4150280 0x7fdfb4152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:57.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.399+0000 7fdfba6c9640 1 --2- 172.21.15.88:0/1775940136 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fdfb40c6bc0 0x7fdfb413d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:57.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.399+0000 7fdfba6c9640 1 -- 172.21.15.88:0/1775940136 >> 172.21.15.88:0/1775940136 conn(0x7fdfb40cad50 msgr2=0x7fdfb40c2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:57.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.399+0000 7fdfba6c9640 1 -- 172.21.15.88:0/1775940136 shutdown_connections 2023-10-16T20:55:57.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:57.399+0000 7fdfba6c9640 1 -- 172.21.15.88:0/1775940136 wait complete. 2023-10-16T20:55:57.417 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:55:59.418 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:55:59.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f4174b2a640 1 Processor -- start 2023-10-16T20:55:59.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f4174b2a640 1 -- start start 2023-10-16T20:55:59.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f4174b2a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4170153770 0x7f4170159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:59.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f4174b2a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41700c9bb0 0x7f41700c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:59.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f4174b2a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f417015a410 con 0x7f41700c41f0 2023-10-16T20:55:59.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f4174b2a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41700cc250 con 0x7f4170153770 2023-10-16T20:55:59.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f4174b2a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41700cc670 con 0x7f41700c9bb0 2023-10-16T20:55:59.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f416ed9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f41700c41f0 0x7f41700cece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55842/0 (socket says 172.21.15.88:55842) 2023-10-16T20:55:59.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f416ed9d640 1 -- 172.21.15.88:0/3808669772 learned_addr learned my addr 172.21.15.88:0/3808669772 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:55:59.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f416e59c640 1 --2- 172.21.15.88:0/3808669772 >> v2:172.21.15.191:3300/0 conn(0x7f4170153770 0x7f4170159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:59.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f4167fff640 1 --2- 172.21.15.88:0/3808669772 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41700c9bb0 0x7f41700c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:59.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f416dd9b640 1 -- 172.21.15.88:0/3808669772 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431724570 0 0) 0x7f417015a410 con 0x7f41700c41f0 2023-10-16T20:55:59.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f416dd9b640 1 -- 172.21.15.88:0/3808669772 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c003530 con 0x7f41700c41f0 2023-10-16T20:55:59.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f416e59c640 1 -- 172.21.15.88:0/3808669772 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41700c9bb0 msgr2=0x7f41700c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:55:59.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f416e59c640 1 --2- 172.21.15.88:0/3808669772 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41700c9bb0 0x7f41700c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:59.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f416e59c640 1 -- 172.21.15.88:0/3808669772 >> v1:172.21.15.88:6789/0 conn(0x7f41700c41f0 legacy=0x7f41700cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:59.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f416e59c640 1 -- 172.21.15.88:0/3808669772 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4170153b50 con 0x7f4170153770 2023-10-16T20:55:59.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.559+0000 7f4167fff640 1 --2- 172.21.15.88:0/3808669772 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41700c9bb0 0x7f41700c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:55:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.563+0000 7f416e59c640 1 --2- 172.21.15.88:0/3808669772 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4170153770 0x7f4170159ed0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f41600097a0 tx=0x7f416001c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f12e0fda49764f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.563+0000 7f416dd9b640 1 -- 172.21.15.88:0/3808669772 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41600ab600 con 0x7f4170153770 2023-10-16T20:55:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.563+0000 7f416dd9b640 1 -- 172.21.15.88:0/3808669772 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41600a7050 con 0x7f4170153770 2023-10-16T20:55:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.563+0000 7f416dd9b640 1 -- 172.21.15.88:0/3808669772 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41600ab980 con 0x7f4170153770 2023-10-16T20:55:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.563+0000 7f4174b2a640 1 -- 172.21.15.88:0/3808669772 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4170153770 msgr2=0x7f4170159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.563+0000 7f4174b2a640 1 --2- 172.21.15.88:0/3808669772 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4170153770 0x7f4170159ed0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f41600097a0 tx=0x7f416001c460 comp rx=0 tx=0).stop 2023-10-16T20:55:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.563+0000 7f4174b2a640 1 -- 172.21.15.88:0/3808669772 shutdown_connections 2023-10-16T20:55:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.563+0000 7f4174b2a640 1 --2- 172.21.15.88:0/3808669772 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41700c9bb0 0x7f41700c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.563+0000 7f4174b2a640 1 --2- 172.21.15.88:0/3808669772 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4170153770 0x7f4170159ed0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.563+0000 7f4174b2a640 1 -- 172.21.15.88:0/3808669772 >> 172.21.15.88:0/3808669772 conn(0x7f41700715c0 msgr2=0x7f41700719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.563+0000 7f4174b2a640 1 -- 172.21.15.88:0/3808669772 shutdown_connections 2023-10-16T20:55:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.563+0000 7f4174b2a640 1 -- 172.21.15.88:0/3808669772 wait complete. 2023-10-16T20:55:59.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4174b2a640 1 Processor -- start 2023-10-16T20:55:59.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4174b2a640 1 -- start start 2023-10-16T20:55:59.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4174b2a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41700c41f0 0x7f41701f1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:59.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4174b2a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41700c9bb0 0x7f41701496f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:59.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4174b2a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4170153770 0x7f417013e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:59.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4174b2a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41701439d0 con 0x7f41700c9bb0 2023-10-16T20:55:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4174b2a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4170143450 con 0x7f4170153770 2023-10-16T20:55:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4174b2a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41701436f0 con 0x7f41700c41f0 2023-10-16T20:55:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4167fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41700c9bb0 0x7f41701496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4167fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41700c9bb0 0x7f41701496f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46008/0 (socket says 172.21.15.88:46008) 2023-10-16T20:55:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4167fff640 1 -- 172.21.15.88:0/3069155032 learned_addr learned my addr 172.21.15.88:0/3069155032 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:55:59.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f416ed9d640 1 --2- 172.21.15.88:0/3069155032 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4170153770 0x7f417013e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:59.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f416e59c640 1 --2- 172.21.15.88:0/3069155032 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41700c41f0 0x7f41701f1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:59.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4167fff640 1 -- 172.21.15.88:0/3069155032 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41700c41f0 msgr2=0x7f41701f1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:59.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4167fff640 1 --2- 172.21.15.88:0/3069155032 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41700c41f0 0x7f41701f1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:59.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4167fff640 1 -- 172.21.15.88:0/3069155032 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4170153770 msgr2=0x7f417013e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:59.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4167fff640 1 --2- 172.21.15.88:0/3069155032 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4170153770 0x7f417013e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:59.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4167fff640 1 -- 172.21.15.88:0/3069155032 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4158009420 con 0x7f41700c9bb0 2023-10-16T20:55:59.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4167fff640 1 --2- 172.21.15.88:0/3069155032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41700c9bb0 0x7f41701496f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f415800ebe0 tx=0x7f415800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:59.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4166ffd640 1 -- 172.21.15.88:0/3069155032 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415800eee0 con 0x7f41700c9bb0 2023-10-16T20:55:59.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4166ffd640 1 -- 172.21.15.88:0/3069155032 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415800cc30 con 0x7f41700c9bb0 2023-10-16T20:55:59.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4166ffd640 1 -- 172.21.15.88:0/3069155032 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4158013470 con 0x7f41700c9bb0 2023-10-16T20:55:59.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4174b2a640 1 -- 172.21.15.88:0/3069155032 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41600093b0 con 0x7f41700c9bb0 2023-10-16T20:55:59.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.567+0000 7f4174b2a640 1 -- 172.21.15.88:0/3069155032 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4170155c80 con 0x7f41700c9bb0 2023-10-16T20:55:59.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.571+0000 7f4174b2a640 1 -- 172.21.15.88:0/3069155032 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4170074700 con 0x7f41700c9bb0 2023-10-16T20:55:59.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.571+0000 7f4166ffd640 1 -- 172.21.15.88:0/3069155032 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f41580135d0 con 0x7f41700c9bb0 2023-10-16T20:55:59.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.571+0000 7f4166ffd640 1 --2- 172.21.15.88:0/3069155032 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f41500664a0 0x7f4150068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:55:59.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.571+0000 7f4166ffd640 1 -- 172.21.15.88:0/3069155032 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7f4158084a40 con 0x7f41700c9bb0 2023-10-16T20:55:59.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.571+0000 7f416e59c640 1 --2- 172.21.15.88:0/3069155032 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f41500664a0 0x7f4150068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:55:59.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.571+0000 7f416e59c640 1 --2- 172.21.15.88:0/3069155032 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f41500664a0 0x7f4150068960 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f41600ad000 tx=0x7f41600a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:55:59.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.575+0000 7f4166ffd640 1 -- 172.21.15.88:0/3069155032 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4158051970 con 0x7f41700c9bb0 2023-10-16T20:55:59.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.823+0000 7f4174b2a640 1 -- 172.21.15.88:0/3069155032 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41700ca740 con 0x7f41700c9bb0 2023-10-16T20:55:59.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.823+0000 7f4166ffd640 1 -- 172.21.15.88:0/3069155032 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f4158054f90 con 0x7f41700c9bb0 2023-10-16T20:55:59.824 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:55:59.824 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:55:56.935944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:resolve","state_seq":33,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:55:59.825 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 98 2023-10-16T20:55:59.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.827+0000 7f4174b2a640 1 -- 172.21.15.88:0/3069155032 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f41500664a0 msgr2=0x7f4150068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:59.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.827+0000 7f4174b2a640 1 --2- 172.21.15.88:0/3069155032 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f41500664a0 0x7f4150068960 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f41600ad000 tx=0x7f41600a6000 comp rx=0 tx=0).stop 2023-10-16T20:55:59.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.827+0000 7f4174b2a640 1 -- 172.21.15.88:0/3069155032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41700c9bb0 msgr2=0x7f41701496f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:55:59.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.827+0000 7f4174b2a640 1 --2- 172.21.15.88:0/3069155032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41700c9bb0 0x7f41701496f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f415800ebe0 tx=0x7f415800c520 comp rx=0 tx=0).stop 2023-10-16T20:55:59.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.827+0000 7f4174b2a640 1 -- 172.21.15.88:0/3069155032 shutdown_connections 2023-10-16T20:55:59.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.827+0000 7f4174b2a640 1 --2- 172.21.15.88:0/3069155032 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f41500664a0 0x7f4150068960 secure :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f41600ad000 tx=0x7f41600a6000 comp rx=0 tx=0).stop 2023-10-16T20:55:59.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.827+0000 7f4174b2a640 1 --2- 172.21.15.88:0/3069155032 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4170153770 0x7f417013e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:59.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.827+0000 7f4174b2a640 1 --2- 172.21.15.88:0/3069155032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41700c9bb0 0x7f41701496f0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:59.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.827+0000 7f4174b2a640 1 --2- 172.21.15.88:0/3069155032 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f41700c41f0 0x7f41701f1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:55:59.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.827+0000 7f4174b2a640 1 -- 172.21.15.88:0/3069155032 >> 172.21.15.88:0/3069155032 conn(0x7f41700715c0 msgr2=0x7f41700b3770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:55:59.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.827+0000 7f4174b2a640 1 -- 172.21.15.88:0/3069155032 shutdown_connections 2023-10-16T20:55:59.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:55:59.827+0000 7f4174b2a640 1 -- 172.21.15.88:0/3069155032 wait complete. 2023-10-16T20:55:59.853 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:56:00.033 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:56:00.038 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:56:01.854 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:02.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.047+0000 7efd1e777640 1 Processor -- start 2023-10-16T20:56:02.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.047+0000 7efd1e777640 1 -- start start 2023-10-16T20:56:02.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.051+0000 7efd1e777640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd181537c0 0x7efd1815c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:02.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.051+0000 7efd1e777640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efd1815cdb0 0x7efd180c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:02.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.051+0000 7efd1e777640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7efd180ca2f0 0x7efd180c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:02.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.051+0000 7efd1e777640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd180cae10 con 0x7efd1815cdb0 2023-10-16T20:56:02.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.051+0000 7efd1e777640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd180cac30 con 0x7efd180ca2f0 2023-10-16T20:56:02.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.051+0000 7efd1e777640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1815d2b0 con 0x7efd181537c0 2023-10-16T20:56:02.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.051+0000 7efd1d775640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd181537c0 0x7efd1815c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:02.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.051+0000 7efd1d775640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd181537c0 0x7efd1815c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:42106/0 (socket says 172.21.15.88:42106) 2023-10-16T20:56:02.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.051+0000 7efd1d775640 1 -- 172.21.15.88:0/1495701517 learned_addr learned my addr 172.21.15.88:0/1495701517 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:02.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.051+0000 7efd1df76640 1 --2- 172.21.15.88:0/1495701517 >> v2:172.21.15.191:3300/0 conn(0x7efd180ca2f0 0x7efd180c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:02.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.051+0000 7efd1d775640 1 -- 172.21.15.88:0/1495701517 >> v2:172.21.15.191:3300/0 conn(0x7efd180ca2f0 msgr2=0x7efd180c51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:56:02.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.051+0000 7efd1d775640 1 --2- 172.21.15.88:0/1495701517 >> v2:172.21.15.191:3300/0 conn(0x7efd180ca2f0 0x7efd180c51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:02.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.051+0000 7efd1d775640 1 -- 172.21.15.88:0/1495701517 >> v2:172.21.15.88:3300/0 conn(0x7efd1815cdb0 msgr2=0x7efd180c4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:56:02.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.051+0000 7efd1d775640 1 --2- 172.21.15.88:0/1495701517 >> v2:172.21.15.88:3300/0 conn(0x7efd1815cdb0 0x7efd180c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:02.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.051+0000 7efd1d775640 1 -- 172.21.15.88:0/1495701517 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1815d590 con 0x7efd181537c0 2023-10-16T20:56:02.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.055+0000 7efd1d775640 1 --2- 172.21.15.88:0/1495701517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd181537c0 0x7efd1815c870 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efd180cc9f0 tx=0x7efd1401ecf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce1f0b5f77ce8c06 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:02.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.059+0000 7efd0ffff640 1 -- 172.21.15.88:0/1495701517 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd140a89b0 con 0x7efd181537c0 2023-10-16T20:56:02.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.059+0000 7efd0ffff640 1 -- 172.21.15.88:0/1495701517 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd140a8b10 con 0x7efd181537c0 2023-10-16T20:56:02.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.059+0000 7efd0ffff640 1 -- 172.21.15.88:0/1495701517 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd140a8ea0 con 0x7efd181537c0 2023-10-16T20:56:02.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.059+0000 7efd1e777640 1 -- 172.21.15.88:0/1495701517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd181537c0 msgr2=0x7efd1815c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:02.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.059+0000 7efd1e777640 1 --2- 172.21.15.88:0/1495701517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd181537c0 0x7efd1815c870 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efd180cc9f0 tx=0x7efd1401ecf0 comp rx=0 tx=0).stop 2023-10-16T20:56:02.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.059+0000 7efd1e777640 1 -- 172.21.15.88:0/1495701517 shutdown_connections 2023-10-16T20:56:02.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.059+0000 7efd1e777640 1 --2- 172.21.15.88:0/1495701517 >> v2:172.21.15.191:3300/0 conn(0x7efd180ca2f0 0x7efd180c51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:02.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.059+0000 7efd1e777640 1 --2- 172.21.15.88:0/1495701517 >> v2:172.21.15.88:3300/0 conn(0x7efd1815cdb0 0x7efd180c4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:02.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.059+0000 7efd1e777640 1 --2- 172.21.15.88:0/1495701517 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd181537c0 0x7efd1815c870 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:02.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.059+0000 7efd1e777640 1 -- 172.21.15.88:0/1495701517 >> 172.21.15.88:0/1495701517 conn(0x7efd1806c170 msgr2=0x7efd180715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:02.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.059+0000 7efd1e777640 1 -- 172.21.15.88:0/1495701517 shutdown_connections 2023-10-16T20:56:02.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.059+0000 7efd1e777640 1 -- 172.21.15.88:0/1495701517 wait complete. 2023-10-16T20:56:02.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1e777640 1 Processor -- start 2023-10-16T20:56:02.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1e777640 1 -- start start 2023-10-16T20:56:02.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1e777640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd180ca2f0 0x7efd18150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:02.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1e777640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd181537c0 0x7efd181f2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:02.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1e777640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd1815cdb0 0x7efd181f2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:02.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1e777640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd18066920 con 0x7efd1815cdb0 2023-10-16T20:56:02.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1e777640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd180c7b70 con 0x7efd180ca2f0 2023-10-16T20:56:02.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1e777640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd18066660 con 0x7efd181537c0 2023-10-16T20:56:02.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1d775640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd180ca2f0 0x7efd18150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:02.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1d775640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd180ca2f0 0x7efd18150b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:42406/0 (socket says 172.21.15.88:42406) 2023-10-16T20:56:02.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1d775640 1 -- 172.21.15.88:0/1884649447 learned_addr learned my addr 172.21.15.88:0/1884649447 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:02.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1df76640 1 --2- 172.21.15.88:0/1884649447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd1815cdb0 0x7efd181f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:02.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1cf74640 1 --2- 172.21.15.88:0/1884649447 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd181537c0 0x7efd181f2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:02.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1d775640 1 -- 172.21.15.88:0/1884649447 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd181537c0 msgr2=0x7efd181f2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:02.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1d775640 1 --2- 172.21.15.88:0/1884649447 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd181537c0 0x7efd181f2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:02.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1d775640 1 -- 172.21.15.88:0/1884649447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd1815cdb0 msgr2=0x7efd181f2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:02.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1d775640 1 --2- 172.21.15.88:0/1884649447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd1815cdb0 0x7efd181f2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:02.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1d775640 1 -- 172.21.15.88:0/1884649447 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd140094d0 con 0x7efd180ca2f0 2023-10-16T20:56:02.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.063+0000 7efd1d775640 1 --2- 172.21.15.88:0/1884649447 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd180ca2f0 0x7efd18150b90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efd140b3040 tx=0x7efd1401ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:02.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.067+0000 7efd0dffb640 1 -- 172.21.15.88:0/1884649447 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd140ba600 con 0x7efd180ca2f0 2023-10-16T20:56:02.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.067+0000 7efd0dffb640 1 -- 172.21.15.88:0/1884649447 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd140ba7a0 con 0x7efd180ca2f0 2023-10-16T20:56:02.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.067+0000 7efd0dffb640 1 -- 172.21.15.88:0/1884649447 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd140bab50 con 0x7efd180ca2f0 2023-10-16T20:56:02.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.067+0000 7efd1e777640 1 -- 172.21.15.88:0/1884649447 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd180c5f00 con 0x7efd180ca2f0 2023-10-16T20:56:02.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.067+0000 7efd1e777640 1 -- 172.21.15.88:0/1884649447 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd180c64c0 con 0x7efd180ca2f0 2023-10-16T20:56:02.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.067+0000 7efd1e777640 1 -- 172.21.15.88:0/1884649447 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd181f1f30 con 0x7efd180ca2f0 2023-10-16T20:56:02.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.067+0000 7efd0dffb640 1 -- 172.21.15.88:0/1884649447 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efd140bacb0 con 0x7efd180ca2f0 2023-10-16T20:56:02.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.071+0000 7efd0dffb640 1 --2- 172.21.15.88:0/1884649447 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efcfc0664c0 0x7efcfc068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:02.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.071+0000 7efd0dffb640 1 -- 172.21.15.88:0/1884649447 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7efd14122ae0 con 0x7efd180ca2f0 2023-10-16T20:56:02.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.071+0000 7efd1cf74640 1 --2- 172.21.15.88:0/1884649447 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efcfc0664c0 0x7efcfc068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:02.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.071+0000 7efd1cf74640 1 --2- 172.21.15.88:0/1884649447 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efcfc0664c0 0x7efcfc068980 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7efd18151830 tx=0x7efd08006f90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:02.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.075+0000 7efd0dffb640 1 -- 172.21.15.88:0/1884649447 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efd181f1f30 con 0x7efd180ca2f0 2023-10-16T20:56:02.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.303+0000 7efd1e777640 1 -- 172.21.15.88:0/1884649447 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd181f1f30 con 0x7efd180ca2f0 2023-10-16T20:56:02.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.303+0000 7efd0dffb640 1 -- 172.21.15.88:0/1884649447 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+5777 (secure 0 0 0) 0x7efd181f1f30 con 0x7efd180ca2f0 2023-10-16T20:56:02.302 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:02.302 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:55:56.935944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:resolve","state_seq":33,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:02.303 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 98 2023-10-16T20:56:02.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.307+0000 7efd1e777640 1 -- 172.21.15.88:0/1884649447 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efcfc0664c0 msgr2=0x7efcfc068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:02.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.307+0000 7efd1e777640 1 --2- 172.21.15.88:0/1884649447 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efcfc0664c0 0x7efcfc068980 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7efd18151830 tx=0x7efd08006f90 comp rx=0 tx=0).stop 2023-10-16T20:56:02.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.307+0000 7efd1e777640 1 -- 172.21.15.88:0/1884649447 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd180ca2f0 msgr2=0x7efd18150b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:02.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.307+0000 7efd1e777640 1 --2- 172.21.15.88:0/1884649447 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd180ca2f0 0x7efd18150b90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efd140b3040 tx=0x7efd1401ba90 comp rx=0 tx=0).stop 2023-10-16T20:56:02.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.307+0000 7efd1e777640 1 -- 172.21.15.88:0/1884649447 shutdown_connections 2023-10-16T20:56:02.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.307+0000 7efd1e777640 1 --2- 172.21.15.88:0/1884649447 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7efcfc0664c0 0x7efcfc068980 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:02.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.307+0000 7efd1e777640 1 --2- 172.21.15.88:0/1884649447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd1815cdb0 0x7efd181f2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:02.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.307+0000 7efd1e777640 1 --2- 172.21.15.88:0/1884649447 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7efd181537c0 0x7efd181f2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:02.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.307+0000 7efd1e777640 1 --2- 172.21.15.88:0/1884649447 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efd180ca2f0 0x7efd18150b90 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:02.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.307+0000 7efd1e777640 1 -- 172.21.15.88:0/1884649447 >> 172.21.15.88:0/1884649447 conn(0x7efd1806c170 msgr2=0x7efd1815c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:02.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.307+0000 7efd1e777640 1 -- 172.21.15.88:0/1884649447 shutdown_connections 2023-10-16T20:56:02.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:02.307+0000 7efd1e777640 1 -- 172.21.15.88:0/1884649447 wait complete. 2023-10-16T20:56:02.337 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:56:04.339 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:04.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.619+0000 7fd56f362640 1 Processor -- start 2023-10-16T20:56:04.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.619+0000 7fd56f362640 1 -- start start 2023-10-16T20:56:04.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.619+0000 7fd56f362640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd5680cb580 0x7fd5680cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:04.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.619+0000 7fd56f362640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5680c9af0 0x7fd568159300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:04.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.619+0000 7fd56f362640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5680c9fc0 con 0x7fd5680b1a60 2023-10-16T20:56:04.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.619+0000 7fd56f362640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5680b20c0 con 0x7fd5680cb580 2023-10-16T20:56:04.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.619+0000 7fd56f362640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5680cbea0 con 0x7fd5680c9af0 2023-10-16T20:56:04.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.619+0000 7fd56db5f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5680c9af0 0x7fd568159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:04.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.619+0000 7fd56db5f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5680c9af0 0x7fd568159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:42118/0 (socket says 172.21.15.88:42118) 2023-10-16T20:56:04.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.619+0000 7fd56eb61640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fd5680b1a60 0x7fd5680715c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:53772/0 (socket says 172.21.15.88:53772) 2023-10-16T20:56:04.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.619+0000 7fd56eb61640 1 -- 172.21.15.88:0/4096705679 learned_addr learned my addr 172.21.15.88:0/4096705679 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:56:04.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.619+0000 7fd56e360640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fd5680cb580 0x7fd5680cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:04.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.623+0000 7fd56db5f640 1 -- 172.21.15.88:0/4096705679 >> v2:172.21.15.191:3300/0 conn(0x7fd5680cb580 msgr2=0x7fd5680cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:56:04.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.623+0000 7fd56db5f640 1 --2- 172.21.15.88:0/4096705679 >> v2:172.21.15.191:3300/0 conn(0x7fd5680cb580 0x7fd5680cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:04.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.623+0000 7fd56db5f640 1 -- 172.21.15.88:0/4096705679 >> v1:172.21.15.191:6789/0 conn(0x7fd5680b1a60 legacy=0x7fd5680715c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:04.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.623+0000 7fd56db5f640 1 -- 172.21.15.88:0/4096705679 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5680c47d0 con 0x7fd5680c9af0 2023-10-16T20:56:04.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.623+0000 7fd56db5f640 1 --2- 172.21.15.88:0/4096705679 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5680c9af0 0x7fd568159300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd5580097c0 tx=0x7fd55801c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7916e63db8a92ed server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:04.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.623+0000 7fd56d35e640 1 -- 172.21.15.88:0/4096705679 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5580ac620 con 0x7fd5680c9af0 2023-10-16T20:56:04.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.623+0000 7fd56d35e640 1 -- 172.21.15.88:0/4096705679 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5580ac780 con 0x7fd5680c9af0 2023-10-16T20:56:04.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.623+0000 7fd56d35e640 1 -- 172.21.15.88:0/4096705679 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5580acb10 con 0x7fd5680c9af0 2023-10-16T20:56:04.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.627+0000 7fd56f362640 1 -- 172.21.15.88:0/4096705679 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5680c9af0 msgr2=0x7fd568159300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:04.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.627+0000 7fd56f362640 1 --2- 172.21.15.88:0/4096705679 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5680c9af0 0x7fd568159300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd5580097c0 tx=0x7fd55801c3c0 comp rx=0 tx=0).stop 2023-10-16T20:56:04.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.627+0000 7fd56f362640 1 -- 172.21.15.88:0/4096705679 shutdown_connections 2023-10-16T20:56:04.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.627+0000 7fd56f362640 1 --2- 172.21.15.88:0/4096705679 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5680c9af0 0x7fd568159300 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:04.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.627+0000 7fd56f362640 1 --2- 172.21.15.88:0/4096705679 >> v2:172.21.15.191:3300/0 conn(0x7fd5680cb580 0x7fd5680cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:04.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.627+0000 7fd56f362640 1 -- 172.21.15.88:0/4096705679 >> 172.21.15.88:0/4096705679 conn(0x7fd56806bb40 msgr2=0x7fd56806bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:04.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.627+0000 7fd56f362640 1 -- 172.21.15.88:0/4096705679 shutdown_connections 2023-10-16T20:56:04.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.627+0000 7fd56f362640 1 -- 172.21.15.88:0/4096705679 wait complete. 2023-10-16T20:56:04.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.627+0000 7fd56f362640 1 Processor -- start 2023-10-16T20:56:04.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.627+0000 7fd56f362640 1 -- start start 2023-10-16T20:56:04.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.627+0000 7fd56f362640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5680b1a60 0x7fd568157000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:04.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.627+0000 7fd56f362640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5680c9af0 0x7fd5680b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:04.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.627+0000 7fd56f362640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5680cb580 0x7fd5681526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:04.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.627+0000 7fd56f362640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd568147890 con 0x7fd5680b1a60 2023-10-16T20:56:04.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.627+0000 7fd56f362640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd568157540 con 0x7fd5680c9af0 2023-10-16T20:56:04.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.627+0000 7fd56f362640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5681472d0 con 0x7fd5680cb580 2023-10-16T20:56:04.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd56e360640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5680b1a60 0x7fd568157000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:04.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd56e360640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5680b1a60 0x7fd568157000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46034/0 (socket says 172.21.15.88:46034) 2023-10-16T20:56:04.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd56e360640 1 -- 172.21.15.88:0/3563849911 learned_addr learned my addr 172.21.15.88:0/3563849911 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:04.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd56db5f640 1 --2- 172.21.15.88:0/3563849911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5680c9af0 0x7fd5680b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:04.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd56eb61640 1 --2- 172.21.15.88:0/3563849911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5680cb580 0x7fd5681526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:04.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd56e360640 1 -- 172.21.15.88:0/3563849911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5680cb580 msgr2=0x7fd5681526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:04.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd56e360640 1 --2- 172.21.15.88:0/3563849911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5680cb580 0x7fd5681526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:04.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd56e360640 1 -- 172.21.15.88:0/3563849911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5680c9af0 msgr2=0x7fd5680b3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:04.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd56e360640 1 --2- 172.21.15.88:0/3563849911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5680c9af0 0x7fd5680b3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:04.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd56e360640 1 -- 172.21.15.88:0/3563849911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd558009420 con 0x7fd5680b1a60 2023-10-16T20:56:04.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd56e360640 1 --2- 172.21.15.88:0/3563849911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5680b1a60 0x7fd568157000 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd56400e8a0 tx=0x7fd56400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:04.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd55effd640 1 -- 172.21.15.88:0/3563849911 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564009510 con 0x7fd5680b1a60 2023-10-16T20:56:04.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd55effd640 1 -- 172.21.15.88:0/3563849911 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56400c940 con 0x7fd5680b1a60 2023-10-16T20:56:04.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd56f362640 1 -- 172.21.15.88:0/3563849911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5681475b0 con 0x7fd5680b1a60 2023-10-16T20:56:04.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd55effd640 1 -- 172.21.15.88:0/3563849911 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56400ccb0 con 0x7fd5680b1a60 2023-10-16T20:56:04.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd56f362640 1 -- 172.21.15.88:0/3563849911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5681f2180 con 0x7fd5680b1a60 2023-10-16T20:56:04.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd56f362640 1 -- 172.21.15.88:0/3563849911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5681f1d00 con 0x7fd5680b1a60 2023-10-16T20:56:04.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.631+0000 7fd55effd640 1 -- 172.21.15.88:0/3563849911 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd564013470 con 0x7fd5680b1a60 2023-10-16T20:56:04.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.635+0000 7fd55effd640 1 --2- 172.21.15.88:0/3563849911 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd54c0664a0 0x7fd54c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:04.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.635+0000 7fd55effd640 1 -- 172.21.15.88:0/3563849911 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7fd564084990 con 0x7fd5680b1a60 2023-10-16T20:56:04.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.635+0000 7fd56db5f640 1 --2- 172.21.15.88:0/3563849911 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd54c0664a0 0x7fd54c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:04.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.635+0000 7fd56db5f640 1 --2- 172.21.15.88:0/3563849911 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd54c0664a0 0x7fd54c068960 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd5580ae000 tx=0x7fd558008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:04.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.639+0000 7fd55effd640 1 -- 172.21.15.88:0/3563849911 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd5681f1d00 con 0x7fd5680b1a60 2023-10-16T20:56:04.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.967+0000 7fd56f362640 1 -- 172.21.15.88:0/3563849911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5681f1d00 con 0x7fd5680b1a60 2023-10-16T20:56:04.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.967+0000 7fd55effd640 1 -- 172.21.15.88:0/3563849911 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5779 (secure 0 0 0) 0x7fd5681f1d00 con 0x7fd5680b1a60 2023-10-16T20:56:04.967 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:04.967 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:02.625860+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:reconnect","state_seq":35,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:04.968 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 99 2023-10-16T20:56:04.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.971+0000 7fd56f362640 1 -- 172.21.15.88:0/3563849911 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd54c0664a0 msgr2=0x7fd54c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:04.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.971+0000 7fd56f362640 1 --2- 172.21.15.88:0/3563849911 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd54c0664a0 0x7fd54c068960 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd5580ae000 tx=0x7fd558008040 comp rx=0 tx=0).stop 2023-10-16T20:56:04.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.971+0000 7fd56f362640 1 -- 172.21.15.88:0/3563849911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5680b1a60 msgr2=0x7fd568157000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:04.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.971+0000 7fd56f362640 1 --2- 172.21.15.88:0/3563849911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5680b1a60 0x7fd568157000 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd56400e8a0 tx=0x7fd56400eca0 comp rx=0 tx=0).stop 2023-10-16T20:56:04.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.971+0000 7fd56f362640 1 -- 172.21.15.88:0/3563849911 shutdown_connections 2023-10-16T20:56:04.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.971+0000 7fd56f362640 1 --2- 172.21.15.88:0/3563849911 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd54c0664a0 0x7fd54c068960 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:04.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.971+0000 7fd56f362640 1 --2- 172.21.15.88:0/3563849911 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd5680cb580 0x7fd5681526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:04.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.971+0000 7fd56f362640 1 --2- 172.21.15.88:0/3563849911 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd5680c9af0 0x7fd5680b3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:04.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.971+0000 7fd56f362640 1 --2- 172.21.15.88:0/3563849911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5680b1a60 0x7fd568157000 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:04.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.971+0000 7fd56f362640 1 -- 172.21.15.88:0/3563849911 >> 172.21.15.88:0/3563849911 conn(0x7fd56806bb40 msgr2=0x7fd5680715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:04.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.971+0000 7fd56f362640 1 -- 172.21.15.88:0/3563849911 shutdown_connections 2023-10-16T20:56:04.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:04.971+0000 7fd56f362640 1 -- 172.21.15.88:0/3563849911 wait complete. 2023-10-16T20:56:04.998 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 99, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 99, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5284": { "addr": "172.21.15.88:6839/2172973775", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6837", "nonce": 2172973775, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2172973775, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5284, "incarnation": 38, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 17 }, "gid_5378": { "addr": "172.21.15.191:6835/3119045295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3119045295, "type": "v2" }, { "addr": "172.21.15.191:6835", "nonce": 3119045295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5378, "incarnation": 89, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:active", "state_seq": 32 }, "gid_5477": { "addr": "172.21.15.191:6837/2244284333", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6836", "nonce": 2244284333, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2244284333, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5477, "incarnation": 97, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 35 }, "gid_5535": { "addr": "172.21.15.88:6835/3476086754", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3476086754, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3476086754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5535, "incarnation": 52, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_5709": { "addr": "172.21.15.88:6838/2600891060", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 2600891060, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 2600891060, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5709, "incarnation": 67, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:56:02.625860+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5477, "mds_1": 5709, "mds_2": 5535, "mds_3": 5284, "mds_4": 5378 } } } ], "standbys": [] } 2023-10-16T20:56:06.999 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:07.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.179+0000 7f669766b640 1 Processor -- start 2023-10-16T20:56:07.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.179+0000 7f669766b640 1 -- start start 2023-10-16T20:56:07.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.179+0000 7f669766b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66901538e0 0x7f66900ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:07.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.179+0000 7f669766b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f669015a2a0 0x7f66900c3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:07.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.179+0000 7f669766b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66900ca610 con 0x7f669015cb20 2023-10-16T20:56:07.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.179+0000 7f669766b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66901595b0 con 0x7f66901538e0 2023-10-16T20:56:07.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.179+0000 7f669766b640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f669015d540 con 0x7f669015a2a0 2023-10-16T20:56:07.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.179+0000 7f6695407640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66901538e0 0x7f66900ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:07.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.183+0000 7f6695407640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66901538e0 0x7f66900ca0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46036/0 (socket says 172.21.15.88:46036) 2023-10-16T20:56:07.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.183+0000 7f6694c06640 1 --2- 172.21.15.88:0/588402772 >> v2:172.21.15.191:3300/0 conn(0x7f669015a2a0 0x7f66900c3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:07.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.183+0000 7f6695407640 1 -- 172.21.15.88:0/588402772 learned_addr learned my addr 172.21.15.88:0/588402772 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:07.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.183+0000 7f6694c06640 1 -- 172.21.15.88:0/588402772 >> v2:172.21.15.88:3300/0 conn(0x7f66901538e0 msgr2=0x7f66900ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:56:07.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.183+0000 7f6694c06640 1 --2- 172.21.15.88:0/588402772 >> v2:172.21.15.88:3300/0 conn(0x7f66901538e0 0x7f66900ca0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:07.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.183+0000 7f6694c06640 1 -- 172.21.15.88:0/588402772 >> v1:172.21.15.191:6789/0 conn(0x7f669015cb20 legacy=0x7f66900cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:07.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.183+0000 7f6694c06640 1 -- 172.21.15.88:0/588402772 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669015d820 con 0x7f669015a2a0 2023-10-16T20:56:07.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.183+0000 7f6687fff640 1 -- 172.21.15.88:0/588402772 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384863614 0 0) 0x7f66900ca610 con 0x7f669015cb20 2023-10-16T20:56:07.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.183+0000 7f6695407640 1 --2- 172.21.15.88:0/588402772 >> v2:172.21.15.88:3300/0 conn(0x7f66901538e0 0x7f66900ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:56:07.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.187+0000 7f6694c06640 1 --2- 172.21.15.88:0/588402772 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f669015a2a0 0x7f66900c3d60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6690068400 tx=0x7f668001c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9119341356f16884 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:07.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.187+0000 7f6687fff640 1 -- 172.21.15.88:0/588402772 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66800ab600 con 0x7f669015a2a0 2023-10-16T20:56:07.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.187+0000 7f6687fff640 1 -- 172.21.15.88:0/588402772 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66800a7050 con 0x7f669015a2a0 2023-10-16T20:56:07.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.187+0000 7f6687fff640 1 -- 172.21.15.88:0/588402772 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66800ab980 con 0x7f669015a2a0 2023-10-16T20:56:07.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.187+0000 7f669766b640 1 -- 172.21.15.88:0/588402772 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f669015a2a0 msgr2=0x7f66900c3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:07.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.187+0000 7f669766b640 1 --2- 172.21.15.88:0/588402772 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f669015a2a0 0x7f66900c3d60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6690068400 tx=0x7f668001c460 comp rx=0 tx=0).stop 2023-10-16T20:56:07.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.187+0000 7f669766b640 1 -- 172.21.15.88:0/588402772 shutdown_connections 2023-10-16T20:56:07.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.187+0000 7f669766b640 1 --2- 172.21.15.88:0/588402772 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f669015a2a0 0x7f66900c3d60 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:07.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.187+0000 7f669766b640 1 --2- 172.21.15.88:0/588402772 >> v2:172.21.15.88:3300/0 conn(0x7f66901538e0 0x7f66900ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:07.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.187+0000 7f669766b640 1 -- 172.21.15.88:0/588402772 >> 172.21.15.88:0/588402772 conn(0x7f669006bd50 msgr2=0x7f669006c160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:07.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.187+0000 7f669766b640 1 -- 172.21.15.88:0/588402772 shutdown_connections 2023-10-16T20:56:07.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.187+0000 7f669766b640 1 -- 172.21.15.88:0/588402772 wait complete. 2023-10-16T20:56:07.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.191+0000 7f669766b640 1 Processor -- start 2023-10-16T20:56:07.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.191+0000 7f669766b640 1 -- start start 2023-10-16T20:56:07.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.191+0000 7f669766b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66901538e0 0x7f66900ced20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:07.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.191+0000 7f669766b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f669015a2a0 0x7f66900cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:07.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.191+0000 7f669766b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f669015cb20 0x7f66901513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:07.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.191+0000 7f669766b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66901463b0 con 0x7f66901538e0 2023-10-16T20:56:07.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.191+0000 7f669766b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66900cf6b0 con 0x7f669015cb20 2023-10-16T20:56:07.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.191+0000 7f669766b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66900cf3d0 con 0x7f669015a2a0 2023-10-16T20:56:07.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.191+0000 7f6695407640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66901538e0 0x7f66900ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:07.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.191+0000 7f6695407640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66901538e0 0x7f66900ced20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46044/0 (socket says 172.21.15.88:46044) 2023-10-16T20:56:07.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.191+0000 7f6695407640 1 -- 172.21.15.88:0/1417911059 learned_addr learned my addr 172.21.15.88:0/1417911059 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:07.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.191+0000 7f6695c08640 1 --2- 172.21.15.88:0/1417911059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f669015cb20 0x7f66901513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:07.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.191+0000 7f6694c06640 1 --2- 172.21.15.88:0/1417911059 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f669015a2a0 0x7f66900cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.195+0000 7f6695407640 1 -- 172.21.15.88:0/1417911059 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f669015a2a0 msgr2=0x7f66900cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.195+0000 7f6695407640 1 --2- 172.21.15.88:0/1417911059 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f669015a2a0 0x7f66900cdff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.195+0000 7f6695407640 1 -- 172.21.15.88:0/1417911059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f669015cb20 msgr2=0x7f66901513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.195+0000 7f6695407640 1 --2- 172.21.15.88:0/1417911059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f669015cb20 0x7f66901513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.195+0000 7f6695407640 1 -- 172.21.15.88:0/1417911059 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668c0094d0 con 0x7f66901538e0 2023-10-16T20:56:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.195+0000 7f6695407640 1 --2- 172.21.15.88:0/1417911059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66901538e0 0x7f66900ced20 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f668c00e9d0 tx=0x7f668c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.195+0000 7f6685ffb640 1 -- 172.21.15.88:0/1417911059 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668c00ca60 con 0x7f66901538e0 2023-10-16T20:56:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.195+0000 7f6685ffb640 1 -- 172.21.15.88:0/1417911059 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668c00cbc0 con 0x7f66901538e0 2023-10-16T20:56:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.195+0000 7f6685ffb640 1 -- 172.21.15.88:0/1417911059 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668c013470 con 0x7f66901538e0 2023-10-16T20:56:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.195+0000 7f669766b640 1 -- 172.21.15.88:0/1417911059 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6680009420 con 0x7f66901538e0 2023-10-16T20:56:07.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.195+0000 7f669766b640 1 -- 172.21.15.88:0/1417911059 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66900cd040 con 0x7f66901538e0 2023-10-16T20:56:07.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.195+0000 7f669766b640 1 -- 172.21.15.88:0/1417911059 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6690074700 con 0x7f66901538e0 2023-10-16T20:56:07.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.195+0000 7f6685ffb640 1 -- 172.21.15.88:0/1417911059 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f668c0135d0 con 0x7f66901538e0 2023-10-16T20:56:07.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.195+0000 7f6685ffb640 1 --2- 172.21.15.88:0/1417911059 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f66740664d0 0x7f6674068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:07.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.199+0000 7f6685ffb640 1 -- 172.21.15.88:0/1417911059 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7f668c084770 con 0x7f66901538e0 2023-10-16T20:56:07.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.199+0000 7f6694c06640 1 --2- 172.21.15.88:0/1417911059 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f66740664d0 0x7f6674068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:07.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.199+0000 7f6694c06640 1 --2- 172.21.15.88:0/1417911059 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f66740664d0 0x7f6674068990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f66800ad000 tx=0x7f6680008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:07.200 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.203+0000 7f6685ffb640 1 -- 172.21.15.88:0/1417911059 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f668c051620 con 0x7f66901538e0 2023-10-16T20:56:07.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.399+0000 7f669766b640 1 -- 172.21.15.88:0/1417911059 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66900ca7a0 con 0x7f66901538e0 2023-10-16T20:56:07.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.399+0000 7f6685ffb640 1 -- 172.21.15.88:0/1417911059 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5779 (secure 0 0 0) 0x7f668c054c40 con 0x7f66901538e0 2023-10-16T20:56:07.399 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:07.399 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:02.625860+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:reconnect","state_seq":35,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:07.400 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 99 2023-10-16T20:56:07.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.403+0000 7f669766b640 1 -- 172.21.15.88:0/1417911059 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f66740664d0 msgr2=0x7f6674068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:07.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.403+0000 7f669766b640 1 --2- 172.21.15.88:0/1417911059 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f66740664d0 0x7f6674068990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f66800ad000 tx=0x7f6680008040 comp rx=0 tx=0).stop 2023-10-16T20:56:07.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.403+0000 7f669766b640 1 -- 172.21.15.88:0/1417911059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66901538e0 msgr2=0x7f66900ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:07.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.403+0000 7f669766b640 1 --2- 172.21.15.88:0/1417911059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66901538e0 0x7f66900ced20 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f668c00e9d0 tx=0x7f668c00ee90 comp rx=0 tx=0).stop 2023-10-16T20:56:07.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.403+0000 7f669766b640 1 -- 172.21.15.88:0/1417911059 shutdown_connections 2023-10-16T20:56:07.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.403+0000 7f669766b640 1 --2- 172.21.15.88:0/1417911059 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f66740664d0 0x7f6674068990 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:07.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.403+0000 7f669766b640 1 --2- 172.21.15.88:0/1417911059 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f669015cb20 0x7f66901513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:07.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.403+0000 7f669766b640 1 --2- 172.21.15.88:0/1417911059 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f669015a2a0 0x7f66900cdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:07.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.403+0000 7f669766b640 1 --2- 172.21.15.88:0/1417911059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66901538e0 0x7f66900ced20 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:07.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.403+0000 7f669766b640 1 -- 172.21.15.88:0/1417911059 >> 172.21.15.88:0/1417911059 conn(0x7f669006bd50 msgr2=0x7f6690149ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:07.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.403+0000 7f669766b640 1 -- 172.21.15.88:0/1417911059 shutdown_connections 2023-10-16T20:56:07.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:07.403+0000 7f669766b640 1 -- 172.21.15.88:0/1417911059 wait complete. 2023-10-16T20:56:07.420 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:56:09.421 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:09.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.611+0000 7f26ebc3f640 1 Processor -- start 2023-10-16T20:56:09.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.611+0000 7f26ebc3f640 1 -- start start 2023-10-16T20:56:09.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.611+0000 7f26ebc3f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26e4156820 0x7f26e40c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:09.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.611+0000 7f26ebc3f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e40c5c60 con 0x7f26e4150660 2023-10-16T20:56:09.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.611+0000 7f26ebc3f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e40c5370 con 0x7f26e4150280 2023-10-16T20:56:09.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.611+0000 7f26ebc3f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e4156d10 con 0x7f26e4156820 2023-10-16T20:56:09.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.611+0000 7f26e91da640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f26e4150660 0x7f26e4156110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33386/0 (socket says 172.21.15.88:33386) 2023-10-16T20:56:09.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.611+0000 7f26e91da640 1 -- 172.21.15.88:0/74662824 learned_addr learned my addr 172.21.15.88:0/74662824 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:56:09.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.611+0000 7f26ea1dc640 1 --2- 172.21.15.88:0/74662824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26e4156820 0x7f26e40c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:09.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.611+0000 7f26e89d9640 1 -- 172.21.15.88:0/74662824 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771681075 0 0) 0x7f26e40c5c60 con 0x7f26e4150660 2023-10-16T20:56:09.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.611+0000 7f26e89d9640 1 -- 172.21.15.88:0/74662824 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26d0003530 con 0x7f26e4150660 2023-10-16T20:56:09.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.611+0000 7f26e89d9640 1 -- 172.21.15.88:0/74662824 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416513785 0 0) 0x7f26e40c5370 con 0x7f26e4150280 2023-10-16T20:56:09.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.611+0000 7f26e89d9640 1 -- 172.21.15.88:0/74662824 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e40c5c60 con 0x7f26e4150280 2023-10-16T20:56:09.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.611+0000 7f26ea1dc640 1 -- 172.21.15.88:0/74662824 >> v1:172.21.15.191:6789/0 conn(0x7f26e4150280 legacy=0x7f26e406d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:09.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.611+0000 7f26ea1dc640 1 -- 172.21.15.88:0/74662824 >> v1:172.21.15.88:6789/0 conn(0x7f26e4150660 legacy=0x7f26e4156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:09.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.615+0000 7f26ea1dc640 1 -- 172.21.15.88:0/74662824 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e40c5610 con 0x7f26e4156820 2023-10-16T20:56:09.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.619+0000 7f26ea1dc640 1 --2- 172.21.15.88:0/74662824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26e4156820 0x7f26e40c60f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f26dc009dc0 tx=0x7f26dc01d3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5936334185e6b78a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:09.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.619+0000 7f26e89d9640 1 -- 172.21.15.88:0/74662824 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc0a5920 con 0x7f26e4156820 2023-10-16T20:56:09.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.619+0000 7f26e89d9640 1 -- 172.21.15.88:0/74662824 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc0a5a80 con 0x7f26e4156820 2023-10-16T20:56:09.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.619+0000 7f26e89d9640 1 -- 172.21.15.88:0/74662824 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc0a5e10 con 0x7f26e4156820 2023-10-16T20:56:09.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.619+0000 7f26ebc3f640 1 -- 172.21.15.88:0/74662824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26e4156820 msgr2=0x7f26e40c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:09.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.619+0000 7f26ebc3f640 1 --2- 172.21.15.88:0/74662824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26e4156820 0x7f26e40c60f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f26dc009dc0 tx=0x7f26dc01d3a0 comp rx=0 tx=0).stop 2023-10-16T20:56:09.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.619+0000 7f26ebc3f640 1 -- 172.21.15.88:0/74662824 shutdown_connections 2023-10-16T20:56:09.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.619+0000 7f26ebc3f640 1 --2- 172.21.15.88:0/74662824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26e4156820 0x7f26e40c60f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:09.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.619+0000 7f26ebc3f640 1 -- 172.21.15.88:0/74662824 >> 172.21.15.88:0/74662824 conn(0x7f26e40cad50 msgr2=0x7f26e40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:09.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.619+0000 7f26ebc3f640 1 -- 172.21.15.88:0/74662824 shutdown_connections 2023-10-16T20:56:09.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.619+0000 7f26ebc3f640 1 -- 172.21.15.88:0/74662824 wait complete. 2023-10-16T20:56:09.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26ebc3f640 1 Processor -- start 2023-10-16T20:56:09.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26ebc3f640 1 -- start start 2023-10-16T20:56:09.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26ebc3f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26e4150280 0x7f26e40c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:09.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26ebc3f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26e4150660 0x7f26e413b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:09.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26ebc3f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26e4156820 0x7f26e413ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:09.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26ebc3f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e4070bf0 con 0x7f26e4150280 2023-10-16T20:56:09.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26ebc3f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e41f50b0 con 0x7f26e4156820 2023-10-16T20:56:09.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26ebc3f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e41f5390 con 0x7f26e4150660 2023-10-16T20:56:09.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26e99db640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26e4150280 0x7f26e40c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:09.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26e99db640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26e4150280 0x7f26e40c8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48874/0 (socket says 172.21.15.88:48874) 2023-10-16T20:56:09.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26e99db640 1 -- 172.21.15.88:0/1397752282 learned_addr learned my addr 172.21.15.88:0/1397752282 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:09.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26ea1dc640 1 --2- 172.21.15.88:0/1397752282 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26e4156820 0x7f26e413ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:09.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26e91da640 1 --2- 172.21.15.88:0/1397752282 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26e4150660 0x7f26e413b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:09.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26e99db640 1 -- 172.21.15.88:0/1397752282 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26e4150660 msgr2=0x7f26e413b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:09.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26e99db640 1 --2- 172.21.15.88:0/1397752282 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26e4150660 0x7f26e413b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:09.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26e99db640 1 -- 172.21.15.88:0/1397752282 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26e4156820 msgr2=0x7f26e413ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:09.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26e99db640 1 --2- 172.21.15.88:0/1397752282 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26e4156820 0x7f26e413ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:09.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.623+0000 7f26e99db640 1 -- 172.21.15.88:0/1397752282 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc009a70 con 0x7f26e4150280 2023-10-16T20:56:09.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.627+0000 7f26e99db640 1 --2- 172.21.15.88:0/1397752282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26e4150280 0x7f26e40c8bf0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f26d800c450 tx=0x7f26d800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:09.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.627+0000 7f26d67fc640 1 -- 172.21.15.88:0/1397752282 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d80052d0 con 0x7f26e4150280 2023-10-16T20:56:09.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.627+0000 7f26d67fc640 1 -- 172.21.15.88:0/1397752282 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d8005430 con 0x7f26e4150280 2023-10-16T20:56:09.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.627+0000 7f26d67fc640 1 -- 172.21.15.88:0/1397752282 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8012870 con 0x7f26e4150280 2023-10-16T20:56:09.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.627+0000 7f26ebc3f640 1 -- 172.21.15.88:0/1397752282 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e41f5670 con 0x7f26e4150280 2023-10-16T20:56:09.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.627+0000 7f26ebc3f640 1 -- 172.21.15.88:0/1397752282 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e413c5a0 con 0x7f26e4150280 2023-10-16T20:56:09.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.627+0000 7f26ebc3f640 1 -- 172.21.15.88:0/1397752282 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e4074700 con 0x7f26e4150280 2023-10-16T20:56:09.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.631+0000 7f26d67fc640 1 -- 172.21.15.88:0/1397752282 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f26d80129d0 con 0x7f26e4150280 2023-10-16T20:56:09.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.631+0000 7f26d67fc640 1 --2- 172.21.15.88:0/1397752282 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f26c80664f0 0x7f26c80689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:09.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.631+0000 7f26d67fc640 1 -- 172.21.15.88:0/1397752282 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7f26d8083fc0 con 0x7f26e4150280 2023-10-16T20:56:09.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.631+0000 7f26e91da640 1 --2- 172.21.15.88:0/1397752282 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f26c80664f0 0x7f26c80689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:09.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.631+0000 7f26e91da640 1 --2- 172.21.15.88:0/1397752282 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f26c80664f0 0x7f26c80689b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f26e413b660 tx=0x7f26e0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:09.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.639+0000 7f26d67fc640 1 -- 172.21.15.88:0/1397752282 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f26d8050db0 con 0x7f26e4150280 2023-10-16T20:56:09.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.907+0000 7f26ebc3f640 1 -- 172.21.15.88:0/1397752282 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26e406d430 con 0x7f26e4150280 2023-10-16T20:56:09.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.907+0000 7f26d67fc640 1 -- 172.21.15.88:0/1397752282 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f26d80543d0 con 0x7f26e4150280 2023-10-16T20:56:09.908 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:09.908 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:07.648858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:09.908 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 100 2023-10-16T20:56:09.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.911+0000 7f26ebc3f640 1 -- 172.21.15.88:0/1397752282 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f26c80664f0 msgr2=0x7f26c80689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:09.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.911+0000 7f26ebc3f640 1 --2- 172.21.15.88:0/1397752282 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f26c80664f0 0x7f26c80689b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f26e413b660 tx=0x7f26e0009040 comp rx=0 tx=0).stop 2023-10-16T20:56:09.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.911+0000 7f26ebc3f640 1 -- 172.21.15.88:0/1397752282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26e4150280 msgr2=0x7f26e40c8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:09.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.911+0000 7f26ebc3f640 1 --2- 172.21.15.88:0/1397752282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26e4150280 0x7f26e40c8bf0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f26d800c450 tx=0x7f26d800c910 comp rx=0 tx=0).stop 2023-10-16T20:56:09.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.911+0000 7f26ebc3f640 1 -- 172.21.15.88:0/1397752282 shutdown_connections 2023-10-16T20:56:09.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.911+0000 7f26ebc3f640 1 --2- 172.21.15.88:0/1397752282 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f26c80664f0 0x7f26c80689b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:09.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.911+0000 7f26ebc3f640 1 --2- 172.21.15.88:0/1397752282 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f26e4156820 0x7f26e413ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:09.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.911+0000 7f26ebc3f640 1 --2- 172.21.15.88:0/1397752282 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f26e4150660 0x7f26e413b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:09.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.911+0000 7f26ebc3f640 1 --2- 172.21.15.88:0/1397752282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26e4150280 0x7f26e40c8bf0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:09.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.911+0000 7f26ebc3f640 1 -- 172.21.15.88:0/1397752282 >> 172.21.15.88:0/1397752282 conn(0x7f26e40cad50 msgr2=0x7f26e40c3790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:09.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.911+0000 7f26ebc3f640 1 -- 172.21.15.88:0/1397752282 shutdown_connections 2023-10-16T20:56:09.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:09.911+0000 7f26ebc3f640 1 -- 172.21.15.88:0/1397752282 wait complete. 2023-10-16T20:56:09.939 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:56:11.940 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:12.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.147+0000 7fa3c46c9640 1 Processor -- start 2023-10-16T20:56:12.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.147+0000 7fa3c46c9640 1 -- start start 2023-10-16T20:56:12.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.147+0000 7fa3c46c9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa3bc0c9bb0 0x7fa3bc0c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:12.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.147+0000 7fa3c46c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3bc0c41f0 0x7fa3bc0cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:12.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.147+0000 7fa3c46c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3bc153b50 con 0x7fa3bc153770 2023-10-16T20:56:12.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.147+0000 7fa3c46c9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3bc0cc250 con 0x7fa3bc0c9bb0 2023-10-16T20:56:12.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.147+0000 7fa3c46c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3bc0cc670 con 0x7fa3bc0c41f0 2023-10-16T20:56:12.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.147+0000 7fa3c2465640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa3bc153770 0x7fa3bc159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33398/0 (socket says 172.21.15.88:33398) 2023-10-16T20:56:12.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.147+0000 7fa3c2465640 1 -- 172.21.15.88:0/2724816942 learned_addr learned my addr 172.21.15.88:0/2724816942 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:56:12.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.147+0000 7fa3c2c66640 1 --2- 172.21.15.88:0/2724816942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3bc0c41f0 0x7fa3bc0cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:12.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.147+0000 7fa3c1c64640 1 --2- 172.21.15.88:0/2724816942 >> v2:172.21.15.191:3300/0 conn(0x7fa3bc0c9bb0 0x7fa3bc0c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:12.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.151+0000 7fa3c2c66640 1 -- 172.21.15.88:0/2724816942 >> v2:172.21.15.191:3300/0 conn(0x7fa3bc0c9bb0 msgr2=0x7fa3bc0c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:56:12.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.151+0000 7fa3c2c66640 1 --2- 172.21.15.88:0/2724816942 >> v2:172.21.15.191:3300/0 conn(0x7fa3bc0c9bb0 0x7fa3bc0c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:12.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.151+0000 7fa3c2c66640 1 -- 172.21.15.88:0/2724816942 >> v1:172.21.15.88:6789/0 conn(0x7fa3bc153770 legacy=0x7fa3bc159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:12.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.151+0000 7fa3c2c66640 1 -- 172.21.15.88:0/2724816942 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3bc0cb2f0 con 0x7fa3bc0c41f0 2023-10-16T20:56:12.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.155+0000 7fa3c2c66640 1 --2- 172.21.15.88:0/2724816942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3bc0c41f0 0x7fa3bc0cece0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4009d90 tx=0x7fa3b401c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40c99bd4e50611fa server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:12.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.155+0000 7fa3c1463640 1 -- 172.21.15.88:0/2724816942 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b40a6700 con 0x7fa3bc0c41f0 2023-10-16T20:56:12.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.155+0000 7fa3c1463640 1 -- 172.21.15.88:0/2724816942 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b40a6860 con 0x7fa3bc0c41f0 2023-10-16T20:56:12.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.155+0000 7fa3c1463640 1 -- 172.21.15.88:0/2724816942 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b40a6bf0 con 0x7fa3bc0c41f0 2023-10-16T20:56:12.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.155+0000 7fa3c46c9640 1 -- 172.21.15.88:0/2724816942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3bc0c41f0 msgr2=0x7fa3bc0cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:12.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.155+0000 7fa3c46c9640 1 --2- 172.21.15.88:0/2724816942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3bc0c41f0 0x7fa3bc0cece0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4009d90 tx=0x7fa3b401c7e0 comp rx=0 tx=0).stop 2023-10-16T20:56:12.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.155+0000 7fa3c46c9640 1 -- 172.21.15.88:0/2724816942 shutdown_connections 2023-10-16T20:56:12.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.155+0000 7fa3c46c9640 1 --2- 172.21.15.88:0/2724816942 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3bc0c41f0 0x7fa3bc0cece0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:12.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.155+0000 7fa3c46c9640 1 --2- 172.21.15.88:0/2724816942 >> v2:172.21.15.191:3300/0 conn(0x7fa3bc0c9bb0 0x7fa3bc0c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:12.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.155+0000 7fa3c46c9640 1 -- 172.21.15.88:0/2724816942 >> 172.21.15.88:0/2724816942 conn(0x7fa3bc0715c0 msgr2=0x7fa3bc0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:12.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.155+0000 7fa3c46c9640 1 -- 172.21.15.88:0/2724816942 shutdown_connections 2023-10-16T20:56:12.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.155+0000 7fa3c46c9640 1 -- 172.21.15.88:0/2724816942 wait complete. 2023-10-16T20:56:12.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c46c9640 1 Processor -- start 2023-10-16T20:56:12.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c46c9640 1 -- start start 2023-10-16T20:56:12.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c46c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3bc0c41f0 0x7fa3bc0b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:12.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c46c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3bc0c9bb0 0x7fa3bc13e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:12.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c46c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3bc153770 0x7fa3bc13eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c46c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3bc1f29a0 con 0x7fa3bc0c41f0 2023-10-16T20:56:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c46c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3bc1f1ab0 con 0x7fa3bc0c9bb0 2023-10-16T20:56:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c46c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3bc1f1d90 con 0x7fa3bc153770 2023-10-16T20:56:12.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c1c64640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3bc0c9bb0 0x7fa3bc13e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:12.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c2c66640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3bc153770 0x7fa3bc13eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:12.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c1c64640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3bc0c9bb0 0x7fa3bc13e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:46886/0 (socket says 172.21.15.88:46886) 2023-10-16T20:56:12.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c1c64640 1 -- 172.21.15.88:0/1774555826 learned_addr learned my addr 172.21.15.88:0/1774555826 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:12.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c2465640 1 --2- 172.21.15.88:0/1774555826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3bc0c41f0 0x7fa3bc0b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:12.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c1c64640 1 -- 172.21.15.88:0/1774555826 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3bc153770 msgr2=0x7fa3bc13eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:12.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c1c64640 1 --2- 172.21.15.88:0/1774555826 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3bc153770 0x7fa3bc13eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:12.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c1c64640 1 -- 172.21.15.88:0/1774555826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3bc0c41f0 msgr2=0x7fa3bc0b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:12.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c1c64640 1 --2- 172.21.15.88:0/1774555826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3bc0c41f0 0x7fa3bc0b4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:12.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c1c64640 1 -- 172.21.15.88:0/1774555826 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b4009a70 con 0x7fa3bc0c9bb0 2023-10-16T20:56:12.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c2c66640 1 --2- 172.21.15.88:0/1774555826 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3bc153770 0x7fa3bc13eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:56:12.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c2465640 1 --2- 172.21.15.88:0/1774555826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3bc0c41f0 0x7fa3bc0b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:56:12.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c1c64640 1 --2- 172.21.15.88:0/1774555826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3bc0c9bb0 0x7fa3bc13e570 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac00e930 tx=0x7fa3ac00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:12.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3b2ffd640 1 -- 172.21.15.88:0/1774555826 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac009690 con 0x7fa3bc0c9bb0 2023-10-16T20:56:12.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3b2ffd640 1 -- 172.21.15.88:0/1774555826 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ac00ca00 con 0x7fa3bc0c9bb0 2023-10-16T20:56:12.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3b2ffd640 1 -- 172.21.15.88:0/1774555826 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac00cd70 con 0x7fa3bc0c9bb0 2023-10-16T20:56:12.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c46c9640 1 -- 172.21.15.88:0/1774555826 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3bc1f2070 con 0x7fa3bc0c9bb0 2023-10-16T20:56:12.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.159+0000 7fa3c46c9640 1 -- 172.21.15.88:0/1774555826 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3bc1f2630 con 0x7fa3bc0c9bb0 2023-10-16T20:56:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.163+0000 7fa3c46c9640 1 -- 172.21.15.88:0/1774555826 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3bc074700 con 0x7fa3bc0c9bb0 2023-10-16T20:56:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.163+0000 7fa3b2ffd640 1 -- 172.21.15.88:0/1774555826 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa3ac013470 con 0x7fa3bc0c9bb0 2023-10-16T20:56:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.163+0000 7fa3b2ffd640 1 --2- 172.21.15.88:0/1774555826 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa3a00664f0 0x7fa3a00689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.163+0000 7fa3b2ffd640 1 -- 172.21.15.88:0/1774555826 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7fa3ac084750 con 0x7fa3bc0c9bb0 2023-10-16T20:56:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.163+0000 7fa3c2465640 1 --2- 172.21.15.88:0/1774555826 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa3a00664f0 0x7fa3a00689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.163+0000 7fa3c2465640 1 --2- 172.21.15.88:0/1774555826 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa3a00664f0 0x7fa3a00689b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8001870 tx=0x7fa3b8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:12.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.167+0000 7fa3b2ffd640 1 -- 172.21.15.88:0/1774555826 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa3ac051690 con 0x7fa3bc0c9bb0 2023-10-16T20:56:12.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.483+0000 7fa3c46c9640 1 -- 172.21.15.88:0/1774555826 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3bc1427c0 con 0x7fa3bc0c9bb0 2023-10-16T20:56:12.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.483+0000 7fa3b2ffd640 1 -- 172.21.15.88:0/1774555826 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5778 (secure 0 0 0) 0x7fa3ac017090 con 0x7fa3bc0c9bb0 2023-10-16T20:56:12.483 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:12.483 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:07.648858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:12.484 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 100 2023-10-16T20:56:12.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.487+0000 7fa3c46c9640 1 -- 172.21.15.88:0/1774555826 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa3a00664f0 msgr2=0x7fa3a00689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:12.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.487+0000 7fa3c46c9640 1 --2- 172.21.15.88:0/1774555826 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa3a00664f0 0x7fa3a00689b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8001870 tx=0x7fa3b8009040 comp rx=0 tx=0).stop 2023-10-16T20:56:12.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.487+0000 7fa3c46c9640 1 -- 172.21.15.88:0/1774555826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3bc0c9bb0 msgr2=0x7fa3bc13e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:12.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.487+0000 7fa3c46c9640 1 --2- 172.21.15.88:0/1774555826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3bc0c9bb0 0x7fa3bc13e570 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac00e930 tx=0x7fa3ac00edf0 comp rx=0 tx=0).stop 2023-10-16T20:56:12.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.487+0000 7fa3c46c9640 1 -- 172.21.15.88:0/1774555826 shutdown_connections 2023-10-16T20:56:12.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.487+0000 7fa3c46c9640 1 --2- 172.21.15.88:0/1774555826 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa3a00664f0 0x7fa3a00689b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:12.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.487+0000 7fa3c46c9640 1 --2- 172.21.15.88:0/1774555826 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa3bc153770 0x7fa3bc13eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:12.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.487+0000 7fa3c46c9640 1 --2- 172.21.15.88:0/1774555826 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa3bc0c9bb0 0x7fa3bc13e570 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:12.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.487+0000 7fa3c46c9640 1 --2- 172.21.15.88:0/1774555826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3bc0c41f0 0x7fa3bc0b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:12.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.487+0000 7fa3c46c9640 1 -- 172.21.15.88:0/1774555826 >> 172.21.15.88:0/1774555826 conn(0x7fa3bc0715c0 msgr2=0x7fa3bc1526a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:12.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.487+0000 7fa3c46c9640 1 -- 172.21.15.88:0/1774555826 shutdown_connections 2023-10-16T20:56:12.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:12.487+0000 7fa3c46c9640 1 -- 172.21.15.88:0/1774555826 wait complete. 2023-10-16T20:56:12.516 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:56:14.516 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:14.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.663+0000 7fea7889c640 1 Processor -- start 2023-10-16T20:56:14.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.663+0000 7fea7889c640 1 -- start start 2023-10-16T20:56:14.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.663+0000 7fea7889c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea74153770 0x7fea74159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:14.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.663+0000 7fea7889c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fea740c9bb0 0x7fea740c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:14.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.663+0000 7fea7889c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7415a410 con 0x7fea740c41f0 2023-10-16T20:56:14.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.663+0000 7fea7889c640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea740cc250 con 0x7fea740c9bb0 2023-10-16T20:56:14.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.663+0000 7fea7889c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea740cc670 con 0x7fea74153770 2023-10-16T20:56:14.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.663+0000 7fea7259c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea74153770 0x7fea74159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:14.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.663+0000 7fea71d9b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fea740c9bb0 0x7fea740c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:14.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.663+0000 7fea72d9d640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fea740c41f0 0x7fea740cece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:45648/0 (socket says 172.21.15.88:45648) 2023-10-16T20:56:14.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.663+0000 7fea72d9d640 1 -- 172.21.15.88:0/988530806 learned_addr learned my addr 172.21.15.88:0/988530806 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:56:14.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.667+0000 7fea7159a640 1 -- 172.21.15.88:0/988530806 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004565082 0 0) 0x7fea7415a410 con 0x7fea740c41f0 2023-10-16T20:56:14.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.667+0000 7fea7159a640 1 -- 172.21.15.88:0/988530806 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea5c003530 con 0x7fea740c41f0 2023-10-16T20:56:14.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.667+0000 7fea7259c640 1 -- 172.21.15.88:0/988530806 >> v2:172.21.15.191:3300/0 conn(0x7fea740c9bb0 msgr2=0x7fea740c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:56:14.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.667+0000 7fea7259c640 1 --2- 172.21.15.88:0/988530806 >> v2:172.21.15.191:3300/0 conn(0x7fea740c9bb0 0x7fea740c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:14.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.667+0000 7fea7259c640 1 -- 172.21.15.88:0/988530806 >> v1:172.21.15.191:6789/0 conn(0x7fea740c41f0 legacy=0x7fea740cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:14.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.667+0000 7fea7259c640 1 -- 172.21.15.88:0/988530806 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea74153b50 con 0x7fea74153770 2023-10-16T20:56:14.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.667+0000 7fea71d9b640 1 --2- 172.21.15.88:0/988530806 >> v2:172.21.15.191:3300/0 conn(0x7fea740c9bb0 0x7fea740c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:56:14.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.667+0000 7fea7259c640 1 --2- 172.21.15.88:0/988530806 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea74153770 0x7fea74159ed0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fea640097a0 tx=0x7fea6401c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55810ee7c1f0655e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:14.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.667+0000 7fea7159a640 1 -- 172.21.15.88:0/988530806 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea640ab600 con 0x7fea74153770 2023-10-16T20:56:14.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.667+0000 7fea7159a640 1 -- 172.21.15.88:0/988530806 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea640a7050 con 0x7fea74153770 2023-10-16T20:56:14.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.667+0000 7fea7159a640 1 -- 172.21.15.88:0/988530806 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea640ab980 con 0x7fea74153770 2023-10-16T20:56:14.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7889c640 1 -- 172.21.15.88:0/988530806 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea74153770 msgr2=0x7fea74159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:14.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7889c640 1 --2- 172.21.15.88:0/988530806 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea74153770 0x7fea74159ed0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fea640097a0 tx=0x7fea6401c460 comp rx=0 tx=0).stop 2023-10-16T20:56:14.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7889c640 1 -- 172.21.15.88:0/988530806 shutdown_connections 2023-10-16T20:56:14.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7889c640 1 --2- 172.21.15.88:0/988530806 >> v2:172.21.15.191:3300/0 conn(0x7fea740c9bb0 0x7fea740c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:14.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7889c640 1 --2- 172.21.15.88:0/988530806 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea74153770 0x7fea74159ed0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:14.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7889c640 1 -- 172.21.15.88:0/988530806 >> 172.21.15.88:0/988530806 conn(0x7fea740715c0 msgr2=0x7fea740719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:14.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7889c640 1 -- 172.21.15.88:0/988530806 shutdown_connections 2023-10-16T20:56:14.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7889c640 1 -- 172.21.15.88:0/988530806 wait complete. 2023-10-16T20:56:14.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7889c640 1 Processor -- start 2023-10-16T20:56:14.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7889c640 1 -- start start 2023-10-16T20:56:14.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7889c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea740c41f0 0x7fea740c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:14.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7889c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea740c9bb0 0x7fea74142cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:14.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7889c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea74153770 0x7fea74143230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:14.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7889c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7413fd20 con 0x7fea740c9bb0 2023-10-16T20:56:14.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7889c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7413ee90 con 0x7fea740c41f0 2023-10-16T20:56:14.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7889c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea74140490 con 0x7fea74153770 2023-10-16T20:56:14.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea71d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea740c9bb0 0x7fea74142cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:14.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7259c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea740c41f0 0x7fea740c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:14.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea71d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea740c9bb0 0x7fea74142cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48894/0 (socket says 172.21.15.88:48894) 2023-10-16T20:56:14.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea71d9b640 1 -- 172.21.15.88:0/2529923245 learned_addr learned my addr 172.21.15.88:0/2529923245 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:14.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea72d9d640 1 --2- 172.21.15.88:0/2529923245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea74153770 0x7fea74143230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:14.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea71d9b640 1 -- 172.21.15.88:0/2529923245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea74153770 msgr2=0x7fea74143230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:14.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea71d9b640 1 --2- 172.21.15.88:0/2529923245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea74153770 0x7fea74143230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:14.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea71d9b640 1 -- 172.21.15.88:0/2529923245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea740c41f0 msgr2=0x7fea740c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:14.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea71d9b640 1 --2- 172.21.15.88:0/2529923245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea740c41f0 0x7fea740c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:14.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea71d9b640 1 -- 172.21.15.88:0/2529923245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea68009420 con 0x7fea740c9bb0 2023-10-16T20:56:14.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.671+0000 7fea7259c640 1 --2- 172.21.15.88:0/2529923245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea740c41f0 0x7fea740c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:56:14.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.675+0000 7fea71d9b640 1 --2- 172.21.15.88:0/2529923245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea740c9bb0 0x7fea74142cf0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fea6800ec10 tx=0x7fea6800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:14.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.675+0000 7fea62ffd640 1 -- 172.21.15.88:0/2529923245 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6800cc30 con 0x7fea740c9bb0 2023-10-16T20:56:14.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.675+0000 7fea62ffd640 1 -- 172.21.15.88:0/2529923245 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6800cd90 con 0x7fea740c9bb0 2023-10-16T20:56:14.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.675+0000 7fea62ffd640 1 -- 172.21.15.88:0/2529923245 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea68013690 con 0x7fea740c9bb0 2023-10-16T20:56:14.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.675+0000 7fea7889c640 1 -- 172.21.15.88:0/2529923245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea640093b0 con 0x7fea740c9bb0 2023-10-16T20:56:14.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.675+0000 7fea7889c640 1 -- 172.21.15.88:0/2529923245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea741422f0 con 0x7fea740c9bb0 2023-10-16T20:56:14.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.675+0000 7fea7889c640 1 -- 172.21.15.88:0/2529923245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea74074700 con 0x7fea740c9bb0 2023-10-16T20:56:14.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.675+0000 7fea62ffd640 1 -- 172.21.15.88:0/2529923245 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fea680137f0 con 0x7fea740c9bb0 2023-10-16T20:56:14.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.675+0000 7fea62ffd640 1 --2- 172.21.15.88:0/2529923245 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fea540664a0 0x7fea54068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:14.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.675+0000 7fea62ffd640 1 -- 172.21.15.88:0/2529923245 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7fea68084c20 con 0x7fea740c9bb0 2023-10-16T20:56:14.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.675+0000 7fea7259c640 1 --2- 172.21.15.88:0/2529923245 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fea540664a0 0x7fea54068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:14.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.679+0000 7fea7259c640 1 --2- 172.21.15.88:0/2529923245 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fea540664a0 0x7fea54068960 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fea640ad000 tx=0x7fea640a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:14.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.679+0000 7fea62ffd640 1 -- 172.21.15.88:0/2529923245 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fea680519b0 con 0x7fea740c9bb0 2023-10-16T20:56:14.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.871+0000 7fea7889c640 1 -- 172.21.15.88:0/2529923245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea740ca740 con 0x7fea740c9bb0 2023-10-16T20:56:14.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.871+0000 7fea62ffd640 1 -- 172.21.15.88:0/2529923245 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5778 (secure 0 0 0) 0x7fea68017090 con 0x7fea740c9bb0 2023-10-16T20:56:14.870 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:14.870 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:12.691340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:14.870 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 101 2023-10-16T20:56:14.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.871+0000 7fea7889c640 1 -- 172.21.15.88:0/2529923245 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fea540664a0 msgr2=0x7fea54068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:14.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.871+0000 7fea7889c640 1 --2- 172.21.15.88:0/2529923245 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fea540664a0 0x7fea54068960 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fea640ad000 tx=0x7fea640a6000 comp rx=0 tx=0).stop 2023-10-16T20:56:14.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.871+0000 7fea7889c640 1 -- 172.21.15.88:0/2529923245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea740c9bb0 msgr2=0x7fea74142cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:14.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.871+0000 7fea7889c640 1 --2- 172.21.15.88:0/2529923245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea740c9bb0 0x7fea74142cf0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fea6800ec10 tx=0x7fea6800c520 comp rx=0 tx=0).stop 2023-10-16T20:56:14.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.871+0000 7fea7889c640 1 -- 172.21.15.88:0/2529923245 shutdown_connections 2023-10-16T20:56:14.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.871+0000 7fea7889c640 1 --2- 172.21.15.88:0/2529923245 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fea540664a0 0x7fea54068960 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:14.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.871+0000 7fea7889c640 1 --2- 172.21.15.88:0/2529923245 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fea74153770 0x7fea74143230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:14.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.871+0000 7fea7889c640 1 --2- 172.21.15.88:0/2529923245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea740c9bb0 0x7fea74142cf0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:14.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.871+0000 7fea7889c640 1 --2- 172.21.15.88:0/2529923245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea740c41f0 0x7fea740c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:14.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.871+0000 7fea7889c640 1 -- 172.21.15.88:0/2529923245 >> 172.21.15.88:0/2529923245 conn(0x7fea740715c0 msgr2=0x7fea74147300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:14.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.871+0000 7fea7889c640 1 -- 172.21.15.88:0/2529923245 shutdown_connections 2023-10-16T20:56:14.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:14.871+0000 7fea7889c640 1 -- 172.21.15.88:0/2529923245 wait complete. 2023-10-16T20:56:14.891 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5282 2023-10-16T20:56:14.891 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.d 2023-10-16T20:56:22.689 INFO:tasks.ceph.mds.b.smithi191.stderr:2023-10-16T20:56:22.685+0000 7f118c629640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-16T20:56:22.689 INFO:tasks.ceph.mds.b.smithi191.stderr:2023-10-16T20:56:22.685+0000 7f118c629640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-16T20:56:22.690 INFO:tasks.ceph.mds.b.smithi191.stderr:2023-10-16T20:56:22.685+0000 7f118c629640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-16T20:56:22.690 INFO:tasks.ceph.mds.b.smithi191.stderr:2023-10-16T20:56:22.685+0000 7f118c629640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-10-16T20:56:22.690 INFO:tasks.ceph.mds.b.smithi191.stderr:2023-10-16T20:56:22.685+0000 7f118c629640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-10-16T20:56:22.892 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-16T20:56:22.893 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-16T20:56:22.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-16T20:56:22.897 INFO:tasks.ceph.mds.d:Started 2023-10-16T20:56:22.898 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:22.985 INFO:tasks.ceph.mds.d.smithi191.stdout:starting mds.d at 2023-10-16T20:56:23.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.051+0000 7f465359e640 1 Processor -- start 2023-10-16T20:56:23.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.051+0000 7f465359e640 1 -- start start 2023-10-16T20:56:23.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.051+0000 7f465359e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f46541537c0 0x7f465415c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:23.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.051+0000 7f465359e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46540cae10 con 0x7f465415cdb0 2023-10-16T20:56:23.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.051+0000 7f465359e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46540cac30 con 0x7f46540ca2f0 2023-10-16T20:56:23.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.051+0000 7f465359e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f465415d2b0 con 0x7f46541537c0 2023-10-16T20:56:23.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.051+0000 7f4651d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f465415cdb0 0x7f46540c4c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56228/0 (socket says 172.21.15.88:56228) 2023-10-16T20:56:23.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.051+0000 7f4651d9b640 1 -- 172.21.15.88:0/4230392004 learned_addr learned my addr 172.21.15.88:0/4230392004 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:56:23.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.051+0000 7f465259c640 1 --2- 172.21.15.88:0/4230392004 >> v2:172.21.15.191:3300/0 conn(0x7f46541537c0 0x7f465415c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:23.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.055+0000 7f465159a640 1 -- 172.21.15.88:0/4230392004 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159579901 0 0) 0x7f46540cae10 con 0x7f465415cdb0 2023-10-16T20:56:23.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.055+0000 7f465159a640 1 -- 172.21.15.88:0/4230392004 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f463c003530 con 0x7f465415cdb0 2023-10-16T20:56:23.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.055+0000 7f465259c640 1 -- 172.21.15.88:0/4230392004 >> v1:172.21.15.191:6789/0 conn(0x7f46540ca2f0 legacy=0x7f46540cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:23.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.055+0000 7f465159a640 1 -- 172.21.15.88:0/4230392004 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536865336 0 0) 0x7f46540cac30 con 0x7f46540ca2f0 2023-10-16T20:56:23.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.055+0000 7f465259c640 1 -- 172.21.15.88:0/4230392004 >> v1:172.21.15.88:6789/0 conn(0x7f465415cdb0 legacy=0x7f46540c4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:23.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.055+0000 7f465259c640 1 -- 172.21.15.88:0/4230392004 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465415d590 con 0x7f46541537c0 2023-10-16T20:56:23.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.055+0000 7f465259c640 1 --2- 172.21.15.88:0/4230392004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46541537c0 0x7f465415c870 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f46440097c0 tx=0x7f464401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=402bbed9ef4fbfca server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:23.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465159a640 1 -- 172.21.15.88:0/4230392004 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46440a8970 con 0x7f46541537c0 2023-10-16T20:56:23.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465159a640 1 -- 172.21.15.88:0/4230392004 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46440a8ad0 con 0x7f46541537c0 2023-10-16T20:56:23.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465159a640 1 -- 172.21.15.88:0/4230392004 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46440a8e60 con 0x7f46541537c0 2023-10-16T20:56:23.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465359e640 1 -- 172.21.15.88:0/4230392004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46541537c0 msgr2=0x7f465415c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:23.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465359e640 1 --2- 172.21.15.88:0/4230392004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46541537c0 0x7f465415c870 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f46440097c0 tx=0x7f464401ec70 comp rx=0 tx=0).stop 2023-10-16T20:56:23.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465359e640 1 -- 172.21.15.88:0/4230392004 shutdown_connections 2023-10-16T20:56:23.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465359e640 1 --2- 172.21.15.88:0/4230392004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46541537c0 0x7f465415c870 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:23.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465359e640 1 -- 172.21.15.88:0/4230392004 >> 172.21.15.88:0/4230392004 conn(0x7f465406c170 msgr2=0x7f46540715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:23.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465359e640 1 -- 172.21.15.88:0/4230392004 shutdown_connections 2023-10-16T20:56:23.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465359e640 1 -- 172.21.15.88:0/4230392004 wait complete. 2023-10-16T20:56:23.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465359e640 1 Processor -- start 2023-10-16T20:56:23.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465359e640 1 -- start start 2023-10-16T20:56:23.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465359e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46540ca2f0 0x7f4654065ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:23.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465359e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46541537c0 0x7f4654154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:23.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465359e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f465415cdb0 0x7f46541547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:23.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465359e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4654155320 con 0x7f465415cdb0 2023-10-16T20:56:23.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465359e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46541561b0 con 0x7f46540ca2f0 2023-10-16T20:56:23.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f465359e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46540ccdf0 con 0x7f46541537c0 2023-10-16T20:56:23.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.059+0000 7f4652d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f465415cdb0 0x7f46541547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:23.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f465259c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46540ca2f0 0x7f4654065ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f4651d9b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46541537c0 0x7f4654154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f4652d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f465415cdb0 0x7f46541547d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39932/0 (socket says 172.21.15.88:39932) 2023-10-16T20:56:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f4652d9d640 1 -- 172.21.15.88:0/2170121081 learned_addr learned my addr 172.21.15.88:0/2170121081 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f465259c640 1 -- 172.21.15.88:0/2170121081 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46541537c0 msgr2=0x7f4654154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f465259c640 1 --2- 172.21.15.88:0/2170121081 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46541537c0 0x7f4654154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:23.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f465259c640 1 -- 172.21.15.88:0/2170121081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f465415cdb0 msgr2=0x7f46541547d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:23.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f465259c640 1 --2- 172.21.15.88:0/2170121081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f465415cdb0 0x7f46541547d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:23.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f465259c640 1 -- 172.21.15.88:0/2170121081 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4644009420 con 0x7f46540ca2f0 2023-10-16T20:56:23.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f4651d9b640 1 --2- 172.21.15.88:0/2170121081 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46541537c0 0x7f4654154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:56:23.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f465259c640 1 --2- 172.21.15.88:0/2170121081 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46540ca2f0 0x7f4654065ce0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f46440af000 tx=0x7f464401ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:23.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f4642ffd640 1 -- 172.21.15.88:0/2170121081 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46440b0db0 con 0x7f46540ca2f0 2023-10-16T20:56:23.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f4642ffd640 1 -- 172.21.15.88:0/2170121081 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4644004020 con 0x7f46540ca2f0 2023-10-16T20:56:23.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f4642ffd640 1 -- 172.21.15.88:0/2170121081 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46440043d0 con 0x7f46540ca2f0 2023-10-16T20:56:23.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f465359e640 1 -- 172.21.15.88:0/2170121081 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4654153d90 con 0x7f46540ca2f0 2023-10-16T20:56:23.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f465359e640 1 -- 172.21.15.88:0/2170121081 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46540cd390 con 0x7f46540ca2f0 2023-10-16T20:56:23.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f465359e640 1 -- 172.21.15.88:0/2170121081 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4654153f20 con 0x7f46540ca2f0 2023-10-16T20:56:23.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f4642ffd640 1 -- 172.21.15.88:0/2170121081 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f46440b7040 con 0x7f46540ca2f0 2023-10-16T20:56:23.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f4642ffd640 1 --2- 172.21.15.88:0/2170121081 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f46340664d0 0x7f4634068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:23.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.063+0000 7f4642ffd640 1 -- 172.21.15.88:0/2170121081 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7f46441214b0 con 0x7f46540ca2f0 2023-10-16T20:56:23.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.067+0000 7f4651d9b640 1 --2- 172.21.15.88:0/2170121081 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f46340664d0 0x7f4634068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:23.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.067+0000 7f4651d9b640 1 --2- 172.21.15.88:0/2170121081 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f46340664d0 0x7f4634068990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f465415a9f0 tx=0x7f4648009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:23.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.071+0000 7f4642ffd640 1 -- 172.21.15.88:0/2170121081 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4654153f20 con 0x7f46540ca2f0 2023-10-16T20:56:23.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.251+0000 7f465359e640 1 -- 172.21.15.88:0/2170121081 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4654153f20 con 0x7f46540ca2f0 2023-10-16T20:56:23.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.255+0000 7f4642ffd640 1 -- 172.21.15.88:0/2170121081 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f4654153f20 con 0x7f46540ca2f0 2023-10-16T20:56:23.253 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:23.253 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:12.691340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:23.253 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 101 2023-10-16T20:56:23.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.255+0000 7f465359e640 1 -- 172.21.15.88:0/2170121081 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f46340664d0 msgr2=0x7f4634068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:23.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.255+0000 7f465359e640 1 --2- 172.21.15.88:0/2170121081 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f46340664d0 0x7f4634068990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f465415a9f0 tx=0x7f4648009040 comp rx=0 tx=0).stop 2023-10-16T20:56:23.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.255+0000 7f465359e640 1 -- 172.21.15.88:0/2170121081 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46540ca2f0 msgr2=0x7f4654065ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:23.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.255+0000 7f465359e640 1 --2- 172.21.15.88:0/2170121081 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46540ca2f0 0x7f4654065ce0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f46440af000 tx=0x7f464401ba10 comp rx=0 tx=0).stop 2023-10-16T20:56:23.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.255+0000 7f465359e640 1 -- 172.21.15.88:0/2170121081 shutdown_connections 2023-10-16T20:56:23.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.255+0000 7f465359e640 1 --2- 172.21.15.88:0/2170121081 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f46340664d0 0x7f4634068990 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:23.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.255+0000 7f465359e640 1 --2- 172.21.15.88:0/2170121081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f465415cdb0 0x7f46541547d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:23.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.255+0000 7f465359e640 1 --2- 172.21.15.88:0/2170121081 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f46541537c0 0x7f4654154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:23.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.255+0000 7f465359e640 1 --2- 172.21.15.88:0/2170121081 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f46540ca2f0 0x7f4654065ce0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:23.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.255+0000 7f465359e640 1 -- 172.21.15.88:0/2170121081 >> 172.21.15.88:0/2170121081 conn(0x7f465406c170 msgr2=0x7f465415af00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:23.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.255+0000 7f465359e640 1 -- 172.21.15.88:0/2170121081 shutdown_connections 2023-10-16T20:56:23.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:23.255+0000 7f465359e640 1 -- 172.21.15.88:0/2170121081 wait complete. 2023-10-16T20:56:23.273 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-16T20:56:25.274 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:25.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.435+0000 7f16e9446640 1 Processor -- start 2023-10-16T20:56:25.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.435+0000 7f16e9446640 1 -- start start 2023-10-16T20:56:25.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.439+0000 7f16e9446640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f16e40c9bb0 0x7f16e40c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:25.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.439+0000 7f16e9446640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16e4153b50 con 0x7f16e4153770 2023-10-16T20:56:25.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.439+0000 7f16e9446640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16e40cb2f0 con 0x7f16e40c41f0 2023-10-16T20:56:25.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.439+0000 7f16e9446640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16e40cc250 con 0x7f16e40c9bb0 2023-10-16T20:56:25.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.439+0000 7f16e27fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f16e40c9bb0 0x7f16e40c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:25.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.439+0000 7f16e2ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f16e4153770 0x7f16e4159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56242/0 (socket says 172.21.15.88:56242) 2023-10-16T20:56:25.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.439+0000 7f16e27fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f16e40c9bb0 0x7f16e40c3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39942/0 (socket says 172.21.15.88:39942) 2023-10-16T20:56:25.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.439+0000 7f16e2ffd640 1 -- 172.21.15.88:0/4006019125 learned_addr learned my addr 172.21.15.88:0/4006019125 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:56:25.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.439+0000 7f16e27fc640 1 -- 172.21.15.88:0/4006019125 >> v1:172.21.15.191:6789/0 conn(0x7f16e40c41f0 legacy=0x7f16e40cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:25.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.439+0000 7f16e27fc640 1 -- 172.21.15.88:0/4006019125 >> v1:172.21.15.88:6789/0 conn(0x7f16e4153770 legacy=0x7f16e4159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:25.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.439+0000 7f16e27fc640 1 -- 172.21.15.88:0/4006019125 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e40cb5d0 con 0x7f16e40c9bb0 2023-10-16T20:56:25.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.443+0000 7f16e27fc640 1 --2- 172.21.15.88:0/4006019125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e40c9bb0 0x7f16e40c3cb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f16d80097c0 tx=0x7f16d801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db39858e4dfbf80d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:25.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.443+0000 7f16e1ffb640 1 -- 172.21.15.88:0/4006019125 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d80a8050 con 0x7f16e40c9bb0 2023-10-16T20:56:25.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.443+0000 7f16e1ffb640 1 -- 172.21.15.88:0/4006019125 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d80ac5f0 con 0x7f16e40c9bb0 2023-10-16T20:56:25.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.443+0000 7f16e1ffb640 1 -- 172.21.15.88:0/4006019125 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d80ac980 con 0x7f16e40c9bb0 2023-10-16T20:56:25.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.443+0000 7f16e9446640 1 -- 172.21.15.88:0/4006019125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e40c9bb0 msgr2=0x7f16e40c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:25.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.443+0000 7f16e9446640 1 --2- 172.21.15.88:0/4006019125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e40c9bb0 0x7f16e40c3cb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f16d80097c0 tx=0x7f16d801c480 comp rx=0 tx=0).stop 2023-10-16T20:56:25.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.443+0000 7f16e9446640 1 -- 172.21.15.88:0/4006019125 shutdown_connections 2023-10-16T20:56:25.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.443+0000 7f16e9446640 1 --2- 172.21.15.88:0/4006019125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e40c9bb0 0x7f16e40c3cb0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:25.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.443+0000 7f16e9446640 1 -- 172.21.15.88:0/4006019125 >> 172.21.15.88:0/4006019125 conn(0x7f16e40715c0 msgr2=0x7f16e40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:25.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.443+0000 7f16e9446640 1 -- 172.21.15.88:0/4006019125 shutdown_connections 2023-10-16T20:56:25.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.443+0000 7f16e9446640 1 -- 172.21.15.88:0/4006019125 wait complete. 2023-10-16T20:56:25.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.443+0000 7f16e9446640 1 Processor -- start 2023-10-16T20:56:25.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e9446640 1 -- start start 2023-10-16T20:56:25.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e9446640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16e40c41f0 0x7f16e40c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:25.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e9446640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e40c9bb0 0x7f16e41f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:25.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e9446640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16e4153770 0x7f16e415b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:25.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e9446640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e40c8e20 con 0x7f16e40c9bb0 2023-10-16T20:56:25.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e9446640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e40cd5f0 con 0x7f16e4153770 2023-10-16T20:56:25.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e9446640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16e41f2400 con 0x7f16e40c41f0 2023-10-16T20:56:25.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e27fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e40c9bb0 0x7f16e41f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:25.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e27fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e40c9bb0 0x7f16e41f56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39946/0 (socket says 172.21.15.88:39946) 2023-10-16T20:56:25.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e27fc640 1 -- 172.21.15.88:0/2381638545 learned_addr learned my addr 172.21.15.88:0/2381638545 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:25.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e2ffd640 1 --2- 172.21.15.88:0/2381638545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16e40c41f0 0x7f16e40c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:25.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e37fe640 1 --2- 172.21.15.88:0/2381638545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16e4153770 0x7f16e415b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:25.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e27fc640 1 -- 172.21.15.88:0/2381638545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16e40c41f0 msgr2=0x7f16e40c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:25.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e27fc640 1 --2- 172.21.15.88:0/2381638545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16e40c41f0 0x7f16e40c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:25.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e27fc640 1 -- 172.21.15.88:0/2381638545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16e4153770 msgr2=0x7f16e415b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:25.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e27fc640 1 --2- 172.21.15.88:0/2381638545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16e4153770 0x7f16e415b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:25.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e27fc640 1 -- 172.21.15.88:0/2381638545 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d8009420 con 0x7f16e40c9bb0 2023-10-16T20:56:25.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e27fc640 1 --2- 172.21.15.88:0/2381638545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e40c9bb0 0x7f16e41f56d0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f16d80af040 tx=0x7f16d8019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:25.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16cb7fe640 1 -- 172.21.15.88:0/2381638545 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d80b5620 con 0x7f16e40c9bb0 2023-10-16T20:56:25.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16cb7fe640 1 -- 172.21.15.88:0/2381638545 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d80b5780 con 0x7f16e40c9bb0 2023-10-16T20:56:25.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16cb7fe640 1 -- 172.21.15.88:0/2381638545 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d80b5b30 con 0x7f16e40c9bb0 2023-10-16T20:56:25.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e9446640 1 -- 172.21.15.88:0/2381638545 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e41f5c10 con 0x7f16e40c9bb0 2023-10-16T20:56:25.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e9446640 1 -- 172.21.15.88:0/2381638545 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e41f1ad0 con 0x7f16e40c9bb0 2023-10-16T20:56:25.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.447+0000 7f16e9446640 1 -- 172.21.15.88:0/2381638545 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16e40c91f0 con 0x7f16e40c9bb0 2023-10-16T20:56:25.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.451+0000 7f16cb7fe640 1 -- 172.21.15.88:0/2381638545 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f16d80b3040 con 0x7f16e40c9bb0 2023-10-16T20:56:25.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.451+0000 7f16cb7fe640 1 --2- 172.21.15.88:0/2381638545 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f16d00664d0 0x7f16d0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:25.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.451+0000 7f16cb7fe640 1 -- 172.21.15.88:0/2381638545 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7f16d811ddf0 con 0x7f16e40c9bb0 2023-10-16T20:56:25.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.451+0000 7f16e2ffd640 1 --2- 172.21.15.88:0/2381638545 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f16d00664d0 0x7f16d0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:25.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.451+0000 7f16e2ffd640 1 --2- 172.21.15.88:0/2381638545 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f16d00664d0 0x7f16d0068990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f16d40010f0 tx=0x7f16d4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:25.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.455+0000 7f16cb7fe640 1 -- 172.21.15.88:0/2381638545 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f16d80eacb0 con 0x7f16e40c9bb0 2023-10-16T20:56:25.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.683+0000 7f16e9446640 1 -- 172.21.15.88:0/2381638545 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16e41f5c10 con 0x7f16e40c9bb0 2023-10-16T20:56:25.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.683+0000 7f16cb7fe640 1 -- 172.21.15.88:0/2381638545 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+6572 (secure 0 0 0) 0x7f16e41f5c10 con 0x7f16e40c9bb0 2023-10-16T20:56:25.683 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:25.683 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:12.691340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:25.683 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 102 2023-10-16T20:56:25.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.687+0000 7f16e9446640 1 -- 172.21.15.88:0/2381638545 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f16d00664d0 msgr2=0x7f16d0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:25.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.687+0000 7f16e9446640 1 --2- 172.21.15.88:0/2381638545 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f16d00664d0 0x7f16d0068990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f16d40010f0 tx=0x7f16d4009040 comp rx=0 tx=0).stop 2023-10-16T20:56:25.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.687+0000 7f16e9446640 1 -- 172.21.15.88:0/2381638545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e40c9bb0 msgr2=0x7f16e41f56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:25.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.687+0000 7f16e9446640 1 --2- 172.21.15.88:0/2381638545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e40c9bb0 0x7f16e41f56d0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f16d80af040 tx=0x7f16d8019220 comp rx=0 tx=0).stop 2023-10-16T20:56:25.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.687+0000 7f16e9446640 1 -- 172.21.15.88:0/2381638545 shutdown_connections 2023-10-16T20:56:25.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.687+0000 7f16e9446640 1 --2- 172.21.15.88:0/2381638545 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f16d00664d0 0x7f16d0068990 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.687+0000 7f16e9446640 1 --2- 172.21.15.88:0/2381638545 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16e4153770 0x7f16e415b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.687+0000 7f16e9446640 1 --2- 172.21.15.88:0/2381638545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16e40c9bb0 0x7f16e41f56d0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.687+0000 7f16e9446640 1 --2- 172.21.15.88:0/2381638545 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f16e40c41f0 0x7f16e40c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.687+0000 7f16e9446640 1 -- 172.21.15.88:0/2381638545 >> 172.21.15.88:0/2381638545 conn(0x7f16e40715c0 msgr2=0x7f16e40cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.687+0000 7f16e9446640 1 -- 172.21.15.88:0/2381638545 shutdown_connections 2023-10-16T20:56:25.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.687+0000 7f16e9446640 1 -- 172.21.15.88:0/2381638545 wait complete. 2023-10-16T20:56:25.706 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-16T20:56:25.707 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:25.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.855+0000 7f51734ad640 1 Processor -- start 2023-10-16T20:56:25.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.855+0000 7f51734ad640 1 -- start start 2023-10-16T20:56:25.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.859+0000 7f51734ad640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f516c1537c0 0x7f516c15c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:25.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.859+0000 7f51734ad640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f516c15cdb0 0x7f516c0c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:25.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.859+0000 7f51734ad640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516c0cae10 con 0x7f516c0ca2f0 2023-10-16T20:56:25.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.859+0000 7f51734ad640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f516c0cac30 con 0x7f516c1537c0 2023-10-16T20:56:25.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.859+0000 7f51734ad640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516c15d2b0 con 0x7f516c15cdb0 2023-10-16T20:56:25.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.859+0000 7f51724ab640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f516c1537c0 0x7f516c15c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:25.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.859+0000 7f5171caa640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f516c15cdb0 0x7f516c0c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:25.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.859+0000 7f51724ab640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f516c1537c0 0x7f516c15c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:42724/0 (socket says 172.21.15.88:42724) 2023-10-16T20:56:25.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.859+0000 7f51724ab640 1 -- 172.21.15.88:0/2438142709 learned_addr learned my addr 172.21.15.88:0/2438142709 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:25.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.859+0000 7f5171caa640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f516c15cdb0 0x7f516c0c4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:55140/0 (socket says 172.21.15.88:55140) 2023-10-16T20:56:25.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.859+0000 7f5172cac640 1 --1- 172.21.15.88:0/2438142709 >> v1:172.21.15.191:6789/0 conn(0x7f516c0ca2f0 0x7f516c0cead0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:52674/0 (socket says 172.21.15.88:52674) 2023-10-16T20:56:25.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.859+0000 7f5171caa640 1 -- 172.21.15.88:0/2438142709 >> v2:172.21.15.191:3300/0 conn(0x7f516c1537c0 msgr2=0x7f516c15c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:56:25.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.859+0000 7f5171caa640 1 --2- 172.21.15.88:0/2438142709 >> v2:172.21.15.191:3300/0 conn(0x7f516c1537c0 0x7f516c15c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:25.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.859+0000 7f5171caa640 1 -- 172.21.15.88:0/2438142709 >> v1:172.21.15.191:6789/0 conn(0x7f516c0ca2f0 legacy=0x7f516c0cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:25.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.859+0000 7f5171caa640 1 -- 172.21.15.88:0/2438142709 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c15d590 con 0x7f516c15cdb0 2023-10-16T20:56:25.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.859+0000 7f51724ab640 1 --2- 172.21.15.88:0/2438142709 >> v2:172.21.15.191:3300/0 conn(0x7f516c1537c0 0x7f516c15c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:56:25.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.863+0000 7f5171caa640 1 --2- 172.21.15.88:0/2438142709 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f516c15cdb0 0x7f516c0c4c80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f515c0097c0 tx=0x7f515c01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7233a98372af07e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:25.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.863+0000 7f51714a9640 1 -- 172.21.15.88:0/2438142709 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c0a8970 con 0x7f516c15cdb0 2023-10-16T20:56:25.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.863+0000 7f51714a9640 1 -- 172.21.15.88:0/2438142709 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c0a8ad0 con 0x7f516c15cdb0 2023-10-16T20:56:25.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.863+0000 7f51714a9640 1 -- 172.21.15.88:0/2438142709 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c0a8e60 con 0x7f516c15cdb0 2023-10-16T20:56:25.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.863+0000 7f51734ad640 1 -- 172.21.15.88:0/2438142709 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f516c15cdb0 msgr2=0x7f516c0c4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:25.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.863+0000 7f51734ad640 1 --2- 172.21.15.88:0/2438142709 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f516c15cdb0 0x7f516c0c4c80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f515c0097c0 tx=0x7f515c01ec70 comp rx=0 tx=0).stop 2023-10-16T20:56:25.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.863+0000 7f51734ad640 1 -- 172.21.15.88:0/2438142709 shutdown_connections 2023-10-16T20:56:25.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.863+0000 7f51734ad640 1 --2- 172.21.15.88:0/2438142709 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f516c15cdb0 0x7f516c0c4c80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:25.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.863+0000 7f51734ad640 1 --2- 172.21.15.88:0/2438142709 >> v2:172.21.15.191:3300/0 conn(0x7f516c1537c0 0x7f516c15c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:25.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.863+0000 7f51734ad640 1 -- 172.21.15.88:0/2438142709 >> 172.21.15.88:0/2438142709 conn(0x7f516c06c170 msgr2=0x7f516c0715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:25.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.863+0000 7f51734ad640 1 -- 172.21.15.88:0/2438142709 shutdown_connections 2023-10-16T20:56:25.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.863+0000 7f51734ad640 1 -- 172.21.15.88:0/2438142709 wait complete. 2023-10-16T20:56:25.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.863+0000 7f51734ad640 1 Processor -- start 2023-10-16T20:56:25.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.863+0000 7f51734ad640 1 -- start start 2023-10-16T20:56:25.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f51734ad640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f516c0ca2f0 0x7f516c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:25.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f51734ad640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f516c1537c0 0x7f516c14c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:25.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f51734ad640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f516c15cdb0 0x7f516c14d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:25.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f51734ad640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516c0cee40 con 0x7f516c1537c0 2023-10-16T20:56:25.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f51734ad640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516c1e3a40 con 0x7f516c15cdb0 2023-10-16T20:56:25.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f51734ad640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516c14c2f0 con 0x7f516c0ca2f0 2023-10-16T20:56:25.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f5171caa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f516c1537c0 0x7f516c14c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:25.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f5171caa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f516c1537c0 0x7f516c14c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39958/0 (socket says 172.21.15.88:39958) 2023-10-16T20:56:25.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f5171caa640 1 -- 172.21.15.88:0/2356531657 learned_addr learned my addr 172.21.15.88:0/2356531657 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:25.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f51724ab640 1 --2- 172.21.15.88:0/2356531657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f516c0ca2f0 0x7f516c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:25.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f5172cac640 1 --2- 172.21.15.88:0/2356531657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f516c15cdb0 0x7f516c14d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:25.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f5171caa640 1 -- 172.21.15.88:0/2356531657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f516c0ca2f0 msgr2=0x7f516c0cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:25.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f5171caa640 1 --2- 172.21.15.88:0/2356531657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f516c0ca2f0 0x7f516c0cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:25.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f5171caa640 1 -- 172.21.15.88:0/2356531657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f516c15cdb0 msgr2=0x7f516c14d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:25.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f5171caa640 1 --2- 172.21.15.88:0/2356531657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f516c15cdb0 0x7f516c14d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:25.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f5171caa640 1 -- 172.21.15.88:0/2356531657 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51680094d0 con 0x7f516c1537c0 2023-10-16T20:56:25.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f5172cac640 1 --2- 172.21.15.88:0/2356531657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f516c15cdb0 0x7f516c14d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:56:25.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f5171caa640 1 --2- 172.21.15.88:0/2356531657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f516c1537c0 0x7f516c14c780 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f515c0b3040 tx=0x7f515c01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:25.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f5162ffd640 1 -- 172.21.15.88:0/2356531657 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c0b9630 con 0x7f516c1537c0 2023-10-16T20:56:25.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f5162ffd640 1 -- 172.21.15.88:0/2356531657 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c0b97b0 con 0x7f516c1537c0 2023-10-16T20:56:25.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f5162ffd640 1 -- 172.21.15.88:0/2356531657 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c0b9b60 con 0x7f516c1537c0 2023-10-16T20:56:25.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f51734ad640 1 -- 172.21.15.88:0/2356531657 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515c009420 con 0x7f516c1537c0 2023-10-16T20:56:25.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f51734ad640 1 -- 172.21.15.88:0/2356531657 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f516c0ceb60 con 0x7f516c1537c0 2023-10-16T20:56:25.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.867+0000 7f51734ad640 1 -- 172.21.15.88:0/2356531657 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f516c04e9c0 con 0x7f516c1537c0 2023-10-16T20:56:25.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.871+0000 7f5162ffd640 1 -- 172.21.15.88:0/2356531657 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f515c0b7040 con 0x7f516c1537c0 2023-10-16T20:56:25.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.871+0000 7f5162ffd640 1 --2- 172.21.15.88:0/2356531657 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f51500664d0 0x7f5150068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:25.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.871+0000 7f5162ffd640 1 -- 172.21.15.88:0/2356531657 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7f515c122320 con 0x7f516c1537c0 2023-10-16T20:56:25.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.871+0000 7f51724ab640 1 --2- 172.21.15.88:0/2356531657 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f51500664d0 0x7f5150068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:25.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.871+0000 7f51724ab640 1 --2- 172.21.15.88:0/2356531657 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f51500664d0 0x7f5150068990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5168009c70 tx=0x7f5168008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:25.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:25.875+0000 7f5162ffd640 1 -- 172.21.15.88:0/2356531657 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f515c0ef1c0 con 0x7f516c1537c0 2023-10-16T20:56:26.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.075+0000 7f51734ad640 1 -- 172.21.15.88:0/2356531657 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f516c0c4730 con 0x7f516c1537c0 2023-10-16T20:56:26.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.075+0000 7f5162ffd640 1 -- 172.21.15.88:0/2356531657 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+6572 (secure 0 0 0) 0x7f515c0f27e0 con 0x7f516c1537c0 2023-10-16T20:56:26.076 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:26.077 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:12.691340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:26.078 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 102 2023-10-16T20:56:26.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.079+0000 7f51734ad640 1 -- 172.21.15.88:0/2356531657 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f51500664d0 msgr2=0x7f5150068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:26.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.079+0000 7f51734ad640 1 --2- 172.21.15.88:0/2356531657 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f51500664d0 0x7f5150068990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5168009c70 tx=0x7f5168008040 comp rx=0 tx=0).stop 2023-10-16T20:56:26.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.079+0000 7f51734ad640 1 -- 172.21.15.88:0/2356531657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f516c1537c0 msgr2=0x7f516c14c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:26.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.079+0000 7f51734ad640 1 --2- 172.21.15.88:0/2356531657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f516c1537c0 0x7f516c14c780 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f515c0b3040 tx=0x7f515c01ba10 comp rx=0 tx=0).stop 2023-10-16T20:56:26.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.079+0000 7f51734ad640 1 -- 172.21.15.88:0/2356531657 shutdown_connections 2023-10-16T20:56:26.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.079+0000 7f51734ad640 1 --2- 172.21.15.88:0/2356531657 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f51500664d0 0x7f5150068990 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:26.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.079+0000 7f51734ad640 1 --2- 172.21.15.88:0/2356531657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f516c15cdb0 0x7f516c14d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:26.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.079+0000 7f51734ad640 1 --2- 172.21.15.88:0/2356531657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f516c1537c0 0x7f516c14c780 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:26.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.079+0000 7f51734ad640 1 --2- 172.21.15.88:0/2356531657 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f516c0ca2f0 0x7f516c0cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:26.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.079+0000 7f51734ad640 1 -- 172.21.15.88:0/2356531657 >> 172.21.15.88:0/2356531657 conn(0x7f516c06c170 msgr2=0x7f516c0c4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:26.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.079+0000 7f51734ad640 1 -- 172.21.15.88:0/2356531657 shutdown_connections 2023-10-16T20:56:26.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.079+0000 7f51734ad640 1 -- 172.21.15.88:0/2356531657 wait complete. 2023-10-16T20:56:26.097 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=3) 2023-10-16T20:56:26.098 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T20:56:26.098 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T20:56:26.112 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T20:56:26.113 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:26.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.295+0000 7f8017bc5640 1 Processor -- start 2023-10-16T20:56:26.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.295+0000 7f8017bc5640 1 -- start start 2023-10-16T20:56:26.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.295+0000 7f8017bc5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8010150660 0x7f80100c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:26.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.295+0000 7f8017bc5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80100c5c60 con 0x7f80100c6bc0 2023-10-16T20:56:26.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.295+0000 7f8017bc5640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80100c5370 con 0x7f8010150280 2023-10-16T20:56:26.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.295+0000 7f8017bc5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8010156d10 con 0x7f8010150660 2023-10-16T20:56:26.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.295+0000 7f8016162640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f80100c6bc0 0x7f8010156110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56258/0 (socket says 172.21.15.88:56258) 2023-10-16T20:56:26.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.295+0000 7f8015160640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8010150660 0x7f80100c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:26.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.295+0000 7f8016162640 1 -- 172.21.15.88:0/4049945800 learned_addr learned my addr 172.21.15.88:0/4049945800 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:56:26.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.295+0000 7f801495f640 1 -- 172.21.15.88:0/4049945800 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275812085 0 0) 0x7f80100c5c60 con 0x7f80100c6bc0 2023-10-16T20:56:26.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.295+0000 7f801495f640 1 -- 172.21.15.88:0/4049945800 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ffc003530 con 0x7f80100c6bc0 2023-10-16T20:56:26.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.295+0000 7f8015160640 1 -- 172.21.15.88:0/4049945800 >> v1:172.21.15.191:6789/0 conn(0x7f8010150280 legacy=0x7f801006d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:26.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.295+0000 7f8015160640 1 -- 172.21.15.88:0/4049945800 >> v1:172.21.15.88:6789/0 conn(0x7f80100c6bc0 legacy=0x7f8010156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:26.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.295+0000 7f8015160640 1 -- 172.21.15.88:0/4049945800 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80100c5610 con 0x7f8010150660 2023-10-16T20:56:26.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.299+0000 7f8015160640 1 --2- 172.21.15.88:0/4049945800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8010150660 0x7f80100c60f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f80000097c0 tx=0x7f800001ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9db0f1a72104cb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:26.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.299+0000 7f801495f640 1 -- 172.21.15.88:0/4049945800 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80000ab670 con 0x7f8010150660 2023-10-16T20:56:26.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.299+0000 7f801495f640 1 -- 172.21.15.88:0/4049945800 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80000ab7d0 con 0x7f8010150660 2023-10-16T20:56:26.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.299+0000 7f801495f640 1 -- 172.21.15.88:0/4049945800 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80000abb60 con 0x7f8010150660 2023-10-16T20:56:26.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.299+0000 7f8017bc5640 1 -- 172.21.15.88:0/4049945800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8010150660 msgr2=0x7f80100c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:26.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.299+0000 7f8017bc5640 1 --2- 172.21.15.88:0/4049945800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8010150660 0x7f80100c60f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f80000097c0 tx=0x7f800001ceb0 comp rx=0 tx=0).stop 2023-10-16T20:56:26.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.299+0000 7f8017bc5640 1 -- 172.21.15.88:0/4049945800 shutdown_connections 2023-10-16T20:56:26.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.299+0000 7f8017bc5640 1 --2- 172.21.15.88:0/4049945800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8010150660 0x7f80100c60f0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:26.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.299+0000 7f8017bc5640 1 -- 172.21.15.88:0/4049945800 >> 172.21.15.88:0/4049945800 conn(0x7f80100cad50 msgr2=0x7f80100cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:26.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.299+0000 7f8017bc5640 1 -- 172.21.15.88:0/4049945800 shutdown_connections 2023-10-16T20:56:26.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.299+0000 7f8017bc5640 1 -- 172.21.15.88:0/4049945800 wait complete. 2023-10-16T20:56:26.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8017bc5640 1 Processor -- start 2023-10-16T20:56:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8017bc5640 1 -- start start 2023-10-16T20:56:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8017bc5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80100c6bc0 0x7f801013cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8017bc5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8010150280 0x7f801006a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8017bc5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8010150660 0x7f8010138150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8017bc5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80100ca7d0 con 0x7f80100c6bc0 2023-10-16T20:56:26.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8017bc5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801013d100 con 0x7f8010150660 2023-10-16T20:56:26.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8017bc5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8010144650 con 0x7f8010150280 2023-10-16T20:56:26.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8015961640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80100c6bc0 0x7f801013cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:26.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8015961640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80100c6bc0 0x7f801013cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39970/0 (socket says 172.21.15.88:39970) 2023-10-16T20:56:26.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8015961640 1 -- 172.21.15.88:0/1422260211 learned_addr learned my addr 172.21.15.88:0/1422260211 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:26.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8016162640 1 --2- 172.21.15.88:0/1422260211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8010150660 0x7f8010138150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:26.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8015160640 1 --2- 172.21.15.88:0/1422260211 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8010150280 0x7f801006a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:26.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8015961640 1 -- 172.21.15.88:0/1422260211 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8010150280 msgr2=0x7f801006a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:26.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8015961640 1 --2- 172.21.15.88:0/1422260211 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8010150280 0x7f801006a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:26.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8015961640 1 -- 172.21.15.88:0/1422260211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8010150660 msgr2=0x7f8010138150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:26.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8015961640 1 --2- 172.21.15.88:0/1422260211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8010150660 0x7f8010138150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:26.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8015961640 1 -- 172.21.15.88:0/1422260211 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8000009420 con 0x7f80100c6bc0 2023-10-16T20:56:26.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8016162640 1 --2- 172.21.15.88:0/1422260211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8010150660 0x7f8010138150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:56:26.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8015961640 1 --2- 172.21.15.88:0/1422260211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80100c6bc0 0x7f801013cbc0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f800c00c500 tx=0x7f800c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:26.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f80067fc640 1 -- 172.21.15.88:0/1422260211 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800c005220 con 0x7f80100c6bc0 2023-10-16T20:56:26.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f80067fc640 1 -- 172.21.15.88:0/1422260211 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800c005380 con 0x7f80100c6bc0 2023-10-16T20:56:26.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f80067fc640 1 -- 172.21.15.88:0/1422260211 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800c007270 con 0x7f80100c6bc0 2023-10-16T20:56:26.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8017bc5640 1 -- 172.21.15.88:0/1422260211 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8010070620 con 0x7f80100c6bc0 2023-10-16T20:56:26.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8017bc5640 1 -- 172.21.15.88:0/1422260211 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80100ca210 con 0x7f80100c6bc0 2023-10-16T20:56:26.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.303+0000 7f8017bc5640 1 -- 172.21.15.88:0/1422260211 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8010067200 con 0x7f80100c6bc0 2023-10-16T20:56:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.307+0000 7f80067fc640 1 -- 172.21.15.88:0/1422260211 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f800c0073d0 con 0x7f80100c6bc0 2023-10-16T20:56:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.307+0000 7f80067fc640 1 --2- 172.21.15.88:0/1422260211 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7ff40664d0 0x7f7ff4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.307+0000 7f80067fc640 1 -- 172.21.15.88:0/1422260211 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7f800c083eb0 con 0x7f80100c6bc0 2023-10-16T20:56:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.307+0000 7f8015160640 1 --2- 172.21.15.88:0/1422260211 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7ff40664d0 0x7f7ff4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.307+0000 7f8015160640 1 --2- 172.21.15.88:0/1422260211 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7ff40664d0 0x7f7ff4068990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f80000af000 tx=0x7f8000004740 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:26.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.311+0000 7f80067fc640 1 -- 172.21.15.88:0/1422260211 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f800c050c30 con 0x7f80100c6bc0 2023-10-16T20:56:26.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.491+0000 7f8017bc5640 1 -- 172.21.15.88:0/1422260211 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8010070450 con 0x7f80100c6bc0 2023-10-16T20:56:26.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.495+0000 7f80067fc640 1 -- 172.21.15.88:0/1422260211 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+6572 (secure 0 0 0) 0x7f800c054250 con 0x7f80100c6bc0 2023-10-16T20:56:26.493 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:26.493 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:12.691340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:26.494 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 102 2023-10-16T20:56:26.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.495+0000 7f8017bc5640 1 -- 172.21.15.88:0/1422260211 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7ff40664d0 msgr2=0x7f7ff4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:26.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.495+0000 7f8017bc5640 1 --2- 172.21.15.88:0/1422260211 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7ff40664d0 0x7f7ff4068990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f80000af000 tx=0x7f8000004740 comp rx=0 tx=0).stop 2023-10-16T20:56:26.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.495+0000 7f8017bc5640 1 -- 172.21.15.88:0/1422260211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80100c6bc0 msgr2=0x7f801013cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:26.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.495+0000 7f8017bc5640 1 --2- 172.21.15.88:0/1422260211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80100c6bc0 0x7f801013cbc0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f800c00c500 tx=0x7f800c00c9c0 comp rx=0 tx=0).stop 2023-10-16T20:56:26.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.495+0000 7f8017bc5640 1 -- 172.21.15.88:0/1422260211 shutdown_connections 2023-10-16T20:56:26.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.495+0000 7f8017bc5640 1 --2- 172.21.15.88:0/1422260211 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7ff40664d0 0x7f7ff4068990 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:26.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.495+0000 7f8017bc5640 1 --2- 172.21.15.88:0/1422260211 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8010150660 0x7f8010138150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:26.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.495+0000 7f8017bc5640 1 --2- 172.21.15.88:0/1422260211 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8010150280 0x7f801006a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:26.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.495+0000 7f8017bc5640 1 --2- 172.21.15.88:0/1422260211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80100c6bc0 0x7f801013cbc0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:26.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.495+0000 7f8017bc5640 1 -- 172.21.15.88:0/1422260211 >> 172.21.15.88:0/1422260211 conn(0x7f80100cad50 msgr2=0x7f801006d200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:26.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.495+0000 7f8017bc5640 1 -- 172.21.15.88:0/1422260211 shutdown_connections 2023-10-16T20:56:26.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:26.495+0000 7f8017bc5640 1 -- 172.21.15.88:0/1422260211 wait complete. 2023-10-16T20:56:26.514 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-16T20:56:28.515 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:28.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.723+0000 7fba4ca45640 1 Processor -- start 2023-10-16T20:56:28.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.723+0000 7fba4ca45640 1 -- start start 2023-10-16T20:56:28.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.723+0000 7fba4ca45640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fba480cb580 0x7fba480cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:28.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.723+0000 7fba4ca45640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba480b1a60 0x7fba48163b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:28.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.723+0000 7fba4ca45640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba480c9fc0 con 0x7fba480c9af0 2023-10-16T20:56:28.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.723+0000 7fba4ca45640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba480b20c0 con 0x7fba480cb580 2023-10-16T20:56:28.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.723+0000 7fba4ca45640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba480cbea0 con 0x7fba480b1a60 2023-10-16T20:56:28.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.723+0000 7fba4659c640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fba480c9af0 0x7fba48159300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:52698/0 (socket says 172.21.15.88:52698) 2023-10-16T20:56:28.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.723+0000 7fba4659c640 1 -- 172.21.15.88:0/3955869856 learned_addr learned my addr 172.21.15.88:0/3955869856 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:56:28.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.723+0000 7fba46d9d640 1 --2- 172.21.15.88:0/3955869856 >> v2:172.21.15.191:3300/0 conn(0x7fba480cb580 0x7fba480cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:28.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.723+0000 7fba4759e640 1 --2- 172.21.15.88:0/3955869856 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba480b1a60 0x7fba48163b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:28.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.727+0000 7fba45d9b640 1 -- 172.21.15.88:0/3955869856 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255856747 0 0) 0x7fba480c9fc0 con 0x7fba480c9af0 2023-10-16T20:56:28.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.727+0000 7fba45d9b640 1 -- 172.21.15.88:0/3955869856 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba30003530 con 0x7fba480c9af0 2023-10-16T20:56:28.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.727+0000 7fba46d9d640 1 -- 172.21.15.88:0/3955869856 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba480b1a60 msgr2=0x7fba48163b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:56:28.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.727+0000 7fba46d9d640 1 --2- 172.21.15.88:0/3955869856 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba480b1a60 0x7fba48163b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:28.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.727+0000 7fba46d9d640 1 -- 172.21.15.88:0/3955869856 >> v1:172.21.15.191:6789/0 conn(0x7fba480c9af0 legacy=0x7fba48159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:28.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.727+0000 7fba46d9d640 1 -- 172.21.15.88:0/3955869856 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba480c47d0 con 0x7fba480cb580 2023-10-16T20:56:28.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.727+0000 7fba4759e640 1 --2- 172.21.15.88:0/3955869856 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba480b1a60 0x7fba48163b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:56:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.731+0000 7fba46d9d640 1 --2- 172.21.15.88:0/3955869856 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba480cb580 0x7fba480cb960 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fba380097c0 tx=0x7fba3801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c766a1312836575 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:28.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.731+0000 7fba45d9b640 1 -- 172.21.15.88:0/3955869856 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba380ac620 con 0x7fba480cb580 2023-10-16T20:56:28.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.731+0000 7fba45d9b640 1 -- 172.21.15.88:0/3955869856 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba380ac780 con 0x7fba480cb580 2023-10-16T20:56:28.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.731+0000 7fba45d9b640 1 -- 172.21.15.88:0/3955869856 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba380acb10 con 0x7fba480cb580 2023-10-16T20:56:28.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.731+0000 7fba4ca45640 1 -- 172.21.15.88:0/3955869856 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba480cb580 msgr2=0x7fba480cb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:28.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.731+0000 7fba4ca45640 1 --2- 172.21.15.88:0/3955869856 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba480cb580 0x7fba480cb960 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fba380097c0 tx=0x7fba3801c3c0 comp rx=0 tx=0).stop 2023-10-16T20:56:28.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.731+0000 7fba4ca45640 1 -- 172.21.15.88:0/3955869856 shutdown_connections 2023-10-16T20:56:28.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.731+0000 7fba4ca45640 1 --2- 172.21.15.88:0/3955869856 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba480b1a60 0x7fba48163b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:28.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.731+0000 7fba4ca45640 1 --2- 172.21.15.88:0/3955869856 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba480cb580 0x7fba480cb960 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:28.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.731+0000 7fba4ca45640 1 -- 172.21.15.88:0/3955869856 >> 172.21.15.88:0/3955869856 conn(0x7fba4806bb40 msgr2=0x7fba4806bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:28.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.731+0000 7fba4ca45640 1 -- 172.21.15.88:0/3955869856 shutdown_connections 2023-10-16T20:56:28.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.731+0000 7fba4ca45640 1 -- 172.21.15.88:0/3955869856 wait complete. 2023-10-16T20:56:28.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.735+0000 7fba4ca45640 1 Processor -- start 2023-10-16T20:56:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.735+0000 7fba4ca45640 1 -- start start 2023-10-16T20:56:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.735+0000 7fba4ca45640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba480b1a60 0x7fba481f2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.735+0000 7fba4ca45640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba480c9af0 0x7fba480c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.735+0000 7fba4ca45640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba480cb580 0x7fba480c6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.735+0000 7fba4ca45640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba480c6b50 con 0x7fba480b1a60 2023-10-16T20:56:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.735+0000 7fba4ca45640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba48159870 con 0x7fba480c9af0 2023-10-16T20:56:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.735+0000 7fba4ca45640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba480c8a20 con 0x7fba480cb580 2023-10-16T20:56:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.735+0000 7fba46d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba480b1a60 0x7fba481f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.739+0000 7fba46d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba480b1a60 0x7fba481f2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39984/0 (socket says 172.21.15.88:39984) 2023-10-16T20:56:28.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.739+0000 7fba46d9d640 1 -- 172.21.15.88:0/2323745696 learned_addr learned my addr 172.21.15.88:0/2323745696 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:28.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.739+0000 7fba4659c640 1 --2- 172.21.15.88:0/2323745696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba480c9af0 0x7fba480c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:28.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.739+0000 7fba4759e640 1 --2- 172.21.15.88:0/2323745696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba480cb580 0x7fba480c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:28.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.739+0000 7fba46d9d640 1 -- 172.21.15.88:0/2323745696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba480cb580 msgr2=0x7fba480c6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:28.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.739+0000 7fba46d9d640 1 --2- 172.21.15.88:0/2323745696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba480cb580 0x7fba480c6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:28.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.739+0000 7fba46d9d640 1 -- 172.21.15.88:0/2323745696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba480c9af0 msgr2=0x7fba480c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:28.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.739+0000 7fba46d9d640 1 --2- 172.21.15.88:0/2323745696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba480c9af0 0x7fba480c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:28.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.739+0000 7fba46d9d640 1 -- 172.21.15.88:0/2323745696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba38009420 con 0x7fba480b1a60 2023-10-16T20:56:28.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.739+0000 7fba4759e640 1 --2- 172.21.15.88:0/2323745696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba480cb580 0x7fba480c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:56:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.739+0000 7fba46d9d640 1 --2- 172.21.15.88:0/2323745696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba480b1a60 0x7fba481f2430 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fba38019ff0 tx=0x7fba38019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.739+0000 7fba2f7fe640 1 -- 172.21.15.88:0/2323745696 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba380b5410 con 0x7fba480b1a60 2023-10-16T20:56:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.739+0000 7fba2f7fe640 1 -- 172.21.15.88:0/2323745696 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba380b5590 con 0x7fba480b1a60 2023-10-16T20:56:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.739+0000 7fba2f7fe640 1 -- 172.21.15.88:0/2323745696 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba380b5940 con 0x7fba480b1a60 2023-10-16T20:56:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.739+0000 7fba4ca45640 1 -- 172.21.15.88:0/2323745696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba481527f0 con 0x7fba480b1a60 2023-10-16T20:56:28.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.739+0000 7fba4ca45640 1 -- 172.21.15.88:0/2323745696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba48152a90 con 0x7fba480b1a60 2023-10-16T20:56:28.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.739+0000 7fba4ca45640 1 -- 172.21.15.88:0/2323745696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba480ca2c0 con 0x7fba480b1a60 2023-10-16T20:56:28.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.743+0000 7fba2f7fe640 1 -- 172.21.15.88:0/2323745696 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fba380b3040 con 0x7fba480b1a60 2023-10-16T20:56:28.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.743+0000 7fba2f7fe640 1 --2- 172.21.15.88:0/2323745696 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fba340664f0 0x7fba340689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:28.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.743+0000 7fba2f7fe640 1 -- 172.21.15.88:0/2323745696 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7fba3811e440 con 0x7fba480b1a60 2023-10-16T20:56:28.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.743+0000 7fba4659c640 1 --2- 172.21.15.88:0/2323745696 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fba340664f0 0x7fba340689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:28.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.743+0000 7fba4659c640 1 --2- 172.21.15.88:0/2323745696 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fba340664f0 0x7fba340689b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fba48159500 tx=0x7fba3c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:28.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:28.747+0000 7fba2f7fe640 1 -- 172.21.15.88:0/2323745696 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fba380eb1a0 con 0x7fba480b1a60 2023-10-16T20:56:29.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:29.023+0000 7fba4ca45640 1 -- 172.21.15.88:0/2323745696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba480c5320 con 0x7fba480b1a60 2023-10-16T20:56:29.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:29.023+0000 7fba2f7fe640 1 -- 172.21.15.88:0/2323745696 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+6572 (secure 0 0 0) 0x7fba380b1070 con 0x7fba480b1a60 2023-10-16T20:56:29.024 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:29.024 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:12.691340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:29.025 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 102 2023-10-16T20:56:29.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:29.027+0000 7fba4ca45640 1 -- 172.21.15.88:0/2323745696 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fba340664f0 msgr2=0x7fba340689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:29.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:29.027+0000 7fba4ca45640 1 --2- 172.21.15.88:0/2323745696 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fba340664f0 0x7fba340689b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fba48159500 tx=0x7fba3c009040 comp rx=0 tx=0).stop 2023-10-16T20:56:29.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:29.027+0000 7fba4ca45640 1 -- 172.21.15.88:0/2323745696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba480b1a60 msgr2=0x7fba481f2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:29.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:29.027+0000 7fba4ca45640 1 --2- 172.21.15.88:0/2323745696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba480b1a60 0x7fba481f2430 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fba38019ff0 tx=0x7fba38019160 comp rx=0 tx=0).stop 2023-10-16T20:56:29.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:29.027+0000 7fba4ca45640 1 -- 172.21.15.88:0/2323745696 shutdown_connections 2023-10-16T20:56:29.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:29.027+0000 7fba4ca45640 1 --2- 172.21.15.88:0/2323745696 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fba340664f0 0x7fba340689b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:29.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:29.027+0000 7fba4ca45640 1 --2- 172.21.15.88:0/2323745696 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fba480cb580 0x7fba480c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:29.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:29.027+0000 7fba4ca45640 1 --2- 172.21.15.88:0/2323745696 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fba480c9af0 0x7fba480c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:29.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:29.027+0000 7fba4ca45640 1 --2- 172.21.15.88:0/2323745696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba480b1a60 0x7fba481f2430 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:29.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:29.027+0000 7fba4ca45640 1 -- 172.21.15.88:0/2323745696 >> 172.21.15.88:0/2323745696 conn(0x7fba4806bb40 msgr2=0x7fba480ceb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:29.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:29.027+0000 7fba4ca45640 1 -- 172.21.15.88:0/2323745696 shutdown_connections 2023-10-16T20:56:29.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:29.027+0000 7fba4ca45640 1 -- 172.21.15.88:0/2323745696 wait complete. 2023-10-16T20:56:29.055 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-16T20:56:30.114 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:56:30.117 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:56:31.055 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:31.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.247+0000 7f7e63e9e640 1 Processor -- start 2023-10-16T20:56:31.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.247+0000 7f7e63e9e640 1 -- start start 2023-10-16T20:56:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.251+0000 7f7e63e9e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7e5c153770 0x7f7e5c159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.251+0000 7f7e63e9e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7e5c0cece0 0x7f7e5c0cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.251+0000 7f7e63e9e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e5c15a410 con 0x7f7e5c0c9bb0 2023-10-16T20:56:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.251+0000 7f7e63e9e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5c0cc250 con 0x7f7e5c153770 2023-10-16T20:56:31.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.251+0000 7f7e63e9e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5c0cc670 con 0x7f7e5c0cece0 2023-10-16T20:56:31.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.251+0000 7f7e61c3a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7e5c153770 0x7f7e5c159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:31.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.251+0000 7f7e61c3a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7e5c153770 0x7f7e5c159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52404/0 (socket says 172.21.15.88:52404) 2023-10-16T20:56:31.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.251+0000 7f7e61c3a640 1 -- 172.21.15.88:0/2195376527 learned_addr learned my addr 172.21.15.88:0/2195376527 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:31.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.251+0000 7f7e6243b640 1 --2- 172.21.15.88:0/2195376527 >> v2:172.21.15.191:3300/0 conn(0x7f7e5c0cece0 0x7f7e5c0cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:31.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.251+0000 7f7e61c3a640 1 -- 172.21.15.88:0/2195376527 >> v2:172.21.15.191:3300/0 conn(0x7f7e5c0cece0 msgr2=0x7f7e5c0cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:56:31.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.251+0000 7f7e61c3a640 1 --2- 172.21.15.88:0/2195376527 >> v2:172.21.15.191:3300/0 conn(0x7f7e5c0cece0 0x7f7e5c0cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:31.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.251+0000 7f7e61c3a640 1 -- 172.21.15.88:0/2195376527 >> v1:172.21.15.88:6789/0 conn(0x7f7e5c0c9bb0 legacy=0x7f7e5c0c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:31.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.251+0000 7f7e61c3a640 1 -- 172.21.15.88:0/2195376527 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5c153b50 con 0x7f7e5c153770 2023-10-16T20:56:31.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.255+0000 7f7e61c3a640 1 --2- 172.21.15.88:0/2195376527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5c153770 0x7f7e5c159ed0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c0cc7b0 tx=0x7f7e5801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=460e519813e5ac65 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:31.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.255+0000 7f7e60c38640 1 -- 172.21.15.88:0/2195376527 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e580aa640 con 0x7f7e5c153770 2023-10-16T20:56:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.255+0000 7f7e60c38640 1 -- 172.21.15.88:0/2195376527 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e580aa7a0 con 0x7f7e5c153770 2023-10-16T20:56:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.255+0000 7f7e60c38640 1 -- 172.21.15.88:0/2195376527 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e580aab30 con 0x7f7e5c153770 2023-10-16T20:56:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.255+0000 7f7e63e9e640 1 -- 172.21.15.88:0/2195376527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5c153770 msgr2=0x7f7e5c159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.255+0000 7f7e63e9e640 1 --2- 172.21.15.88:0/2195376527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5c153770 0x7f7e5c159ed0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c0cc7b0 tx=0x7f7e5801c4e0 comp rx=0 tx=0).stop 2023-10-16T20:56:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.255+0000 7f7e63e9e640 1 -- 172.21.15.88:0/2195376527 shutdown_connections 2023-10-16T20:56:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.255+0000 7f7e63e9e640 1 --2- 172.21.15.88:0/2195376527 >> v2:172.21.15.191:3300/0 conn(0x7f7e5c0cece0 0x7f7e5c0cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.255+0000 7f7e63e9e640 1 --2- 172.21.15.88:0/2195376527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5c153770 0x7f7e5c159ed0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.255+0000 7f7e63e9e640 1 -- 172.21.15.88:0/2195376527 >> 172.21.15.88:0/2195376527 conn(0x7f7e5c0715c0 msgr2=0x7f7e5c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.255+0000 7f7e63e9e640 1 -- 172.21.15.88:0/2195376527 shutdown_connections 2023-10-16T20:56:31.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.255+0000 7f7e63e9e640 1 -- 172.21.15.88:0/2195376527 wait complete. 2023-10-16T20:56:31.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e63e9e640 1 Processor -- start 2023-10-16T20:56:31.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e63e9e640 1 -- start start 2023-10-16T20:56:31.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e63e9e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e5c0c9bb0 0x7f7e5c1f1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:31.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e63e9e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e5c0cece0 0x7f7e5c15d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:31.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e63e9e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5c153770 0x7f7e5c0c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:31.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e63e9e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c1f2530 con 0x7f7e5c153770 2023-10-16T20:56:31.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e63e9e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c0c7020 con 0x7f7e5c0cece0 2023-10-16T20:56:31.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e63e9e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c0c7300 con 0x7f7e5c0c9bb0 2023-10-16T20:56:31.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e6243b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5c153770 0x7f7e5c0c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:31.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e61c3a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e5c0c9bb0 0x7f7e5c1f1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:31.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e61c3a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e5c0c9bb0 0x7f7e5c1f1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:49358/0 (socket says 172.21.15.88:49358) 2023-10-16T20:56:31.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e6243b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5c153770 0x7f7e5c0c8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52418/0 (socket says 172.21.15.88:52418) 2023-10-16T20:56:31.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e61c3a640 1 -- 172.21.15.88:0/230155775 learned_addr learned my addr 172.21.15.88:0/230155775 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:31.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e61439640 1 --2- 172.21.15.88:0/230155775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e5c0cece0 0x7f7e5c15d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:31.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e61c3a640 1 -- 172.21.15.88:0/230155775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e5c0cece0 msgr2=0x7f7e5c15d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:31.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e61c3a640 1 --2- 172.21.15.88:0/230155775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e5c0cece0 0x7f7e5c15d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:31.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e61c3a640 1 -- 172.21.15.88:0/230155775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5c153770 msgr2=0x7f7e5c0c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:31.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e61c3a640 1 --2- 172.21.15.88:0/230155775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5c153770 0x7f7e5c0c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:31.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e61c3a640 1 -- 172.21.15.88:0/230155775 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e58009480 con 0x7f7e5c0c9bb0 2023-10-16T20:56:31.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e61c3a640 1 --2- 172.21.15.88:0/230155775 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e5c0c9bb0 0x7f7e5c1f1ff0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7e5801a110 tx=0x7f7e58019280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:31.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e527fc640 1 -- 172.21.15.88:0/230155775 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e58005ba0 con 0x7f7e5c0c9bb0 2023-10-16T20:56:31.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e527fc640 1 -- 172.21.15.88:0/230155775 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e58005d00 con 0x7f7e5c0c9bb0 2023-10-16T20:56:31.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e63e9e640 1 -- 172.21.15.88:0/230155775 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e5c0c75e0 con 0x7f7e5c0c9bb0 2023-10-16T20:56:31.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e527fc640 1 -- 172.21.15.88:0/230155775 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e580aab30 con 0x7f7e5c0c9bb0 2023-10-16T20:56:31.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.259+0000 7f7e63e9e640 1 -- 172.21.15.88:0/230155775 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e5c0c8230 con 0x7f7e5c0c9bb0 2023-10-16T20:56:31.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.263+0000 7f7e63e9e640 1 -- 172.21.15.88:0/230155775 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e5c074700 con 0x7f7e5c0c9bb0 2023-10-16T20:56:31.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.263+0000 7f7e527fc640 1 -- 172.21.15.88:0/230155775 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7e580a8050 con 0x7f7e5c0c9bb0 2023-10-16T20:56:31.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.263+0000 7f7e527fc640 1 --2- 172.21.15.88:0/230155775 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7e400664a0 0x7f7e40068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:31.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.263+0000 7f7e527fc640 1 -- 172.21.15.88:0/230155775 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7f7e5811e760 con 0x7f7e5c0c9bb0 2023-10-16T20:56:31.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.263+0000 7f7e61439640 1 --2- 172.21.15.88:0/230155775 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7e400664a0 0x7f7e40068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:31.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.267+0000 7f7e61439640 1 --2- 172.21.15.88:0/230155775 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7e400664a0 0x7f7e40068960 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c147c30 tx=0x7f7e4c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:31.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.271+0000 7f7e527fc640 1 -- 172.21.15.88:0/230155775 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7e580eb610 con 0x7f7e5c0c9bb0 2023-10-16T20:56:31.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.539+0000 7f7e63e9e640 1 -- 172.21.15.88:0/230155775 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e5c0ca740 con 0x7f7e5c0c9bb0 2023-10-16T20:56:31.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.539+0000 7f7e527fc640 1 -- 172.21.15.88:0/230155775 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+6572 (secure 0 0 0) 0x7f7e580aa600 con 0x7f7e5c0c9bb0 2023-10-16T20:56:31.539 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:31.540 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:12.691340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:31.540 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 102 2023-10-16T20:56:31.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.543+0000 7f7e63e9e640 1 -- 172.21.15.88:0/230155775 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7e400664a0 msgr2=0x7f7e40068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:31.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.543+0000 7f7e63e9e640 1 --2- 172.21.15.88:0/230155775 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7e400664a0 0x7f7e40068960 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c147c30 tx=0x7f7e4c009040 comp rx=0 tx=0).stop 2023-10-16T20:56:31.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.543+0000 7f7e63e9e640 1 -- 172.21.15.88:0/230155775 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e5c0c9bb0 msgr2=0x7f7e5c1f1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:31.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.543+0000 7f7e63e9e640 1 --2- 172.21.15.88:0/230155775 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e5c0c9bb0 0x7f7e5c1f1ff0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7e5801a110 tx=0x7f7e58019280 comp rx=0 tx=0).stop 2023-10-16T20:56:31.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.543+0000 7f7e63e9e640 1 -- 172.21.15.88:0/230155775 shutdown_connections 2023-10-16T20:56:31.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.543+0000 7f7e63e9e640 1 --2- 172.21.15.88:0/230155775 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7e400664a0 0x7f7e40068960 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:31.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.543+0000 7f7e63e9e640 1 --2- 172.21.15.88:0/230155775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5c153770 0x7f7e5c0c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:31.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.543+0000 7f7e63e9e640 1 --2- 172.21.15.88:0/230155775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e5c0cece0 0x7f7e5c15d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:31.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.543+0000 7f7e63e9e640 1 --2- 172.21.15.88:0/230155775 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e5c0c9bb0 0x7f7e5c1f1ff0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:31.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.543+0000 7f7e63e9e640 1 -- 172.21.15.88:0/230155775 >> 172.21.15.88:0/230155775 conn(0x7f7e5c0715c0 msgr2=0x7f7e5c1518a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:31.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.543+0000 7f7e63e9e640 1 -- 172.21.15.88:0/230155775 shutdown_connections 2023-10-16T20:56:31.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:31.543+0000 7f7e63e9e640 1 -- 172.21.15.88:0/230155775 wait complete. 2023-10-16T20:56:31.568 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-16T20:56:33.569 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:33.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.762+0000 7f3dcbc9e640 1 Processor -- start 2023-10-16T20:56:33.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.762+0000 7f3dcbc9e640 1 -- start start 2023-10-16T20:56:33.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.766+0000 7f3dcbc9e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3dc415a810 0x7f3dc415ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:33.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.766+0000 7f3dcbc9e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc415b3c0 con 0x7f3dc406a970 2023-10-16T20:56:33.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.766+0000 7f3dcbc9e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc415b680 con 0x7f3dc41406d0 2023-10-16T20:56:33.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.766+0000 7f3dcbc9e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc415b1c0 con 0x7f3dc415a810 2023-10-16T20:56:33.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.766+0000 7f3dca23b640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3dc415a810 0x7f3dc415ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:33.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.766+0000 7f3dc9239640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3dc406a970 0x7f3dc4071920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55246/0 (socket says 172.21.15.88:55246) 2023-10-16T20:56:33.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.766+0000 7f3dc9239640 1 -- 172.21.15.88:0/873804657 learned_addr learned my addr 172.21.15.88:0/873804657 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:56:33.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.766+0000 7f3dca23b640 1 -- 172.21.15.88:0/873804657 >> v1:172.21.15.191:6789/0 conn(0x7f3dc41406d0 legacy=0x7f3dc40c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:33.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.766+0000 7f3dca23b640 1 -- 172.21.15.88:0/873804657 >> v1:172.21.15.88:6789/0 conn(0x7f3dc406a970 legacy=0x7f3dc4071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:33.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.766+0000 7f3dc8a38640 1 -- 172.21.15.88:0/873804657 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850756364 0 0) 0x7f3dc415b680 con 0x7f3dc41406d0 2023-10-16T20:56:33.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.766+0000 7f3dca23b640 1 -- 172.21.15.88:0/873804657 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc4157570 con 0x7f3dc415a810 2023-10-16T20:56:33.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.770+0000 7f3dca23b640 1 --2- 172.21.15.88:0/873804657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dc415a810 0x7f3dc415ac80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc009d90 tx=0x7f3dbc01c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ec6720151b616ee server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:33.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.770+0000 7f3dc8a38640 1 -- 172.21.15.88:0/873804657 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc0a5900 con 0x7f3dc415a810 2023-10-16T20:56:33.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.770+0000 7f3dc8a38640 1 -- 172.21.15.88:0/873804657 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dbc0a5a60 con 0x7f3dc415a810 2023-10-16T20:56:33.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.770+0000 7f3dc8a38640 1 -- 172.21.15.88:0/873804657 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc0a5df0 con 0x7f3dc415a810 2023-10-16T20:56:33.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.770+0000 7f3dcbc9e640 1 -- 172.21.15.88:0/873804657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dc415a810 msgr2=0x7f3dc415ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:33.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.770+0000 7f3dcbc9e640 1 --2- 172.21.15.88:0/873804657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dc415a810 0x7f3dc415ac80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc009d90 tx=0x7f3dbc01c950 comp rx=0 tx=0).stop 2023-10-16T20:56:33.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.770+0000 7f3dcbc9e640 1 -- 172.21.15.88:0/873804657 shutdown_connections 2023-10-16T20:56:33.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.770+0000 7f3dcbc9e640 1 --2- 172.21.15.88:0/873804657 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dc415a810 0x7f3dc415ac80 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:33.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.770+0000 7f3dcbc9e640 1 -- 172.21.15.88:0/873804657 >> 172.21.15.88:0/873804657 conn(0x7f3dc40cad50 msgr2=0x7f3dc40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:33.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.770+0000 7f3dcbc9e640 1 -- 172.21.15.88:0/873804657 shutdown_connections 2023-10-16T20:56:33.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.770+0000 7f3dcbc9e640 1 -- 172.21.15.88:0/873804657 wait complete. 2023-10-16T20:56:33.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dcbc9e640 1 Processor -- start 2023-10-16T20:56:33.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dcbc9e640 1 -- start start 2023-10-16T20:56:33.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dcbc9e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3dc406a970 0x7f3dc41f5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:33.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dcbc9e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dc41406d0 0x7f3dc41552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:33.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dcbc9e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3dc415a810 0x7f3dc4153390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:33.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dcbc9e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc413eab0 con 0x7f3dc406a970 2023-10-16T20:56:33.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dcbc9e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc4152f00 con 0x7f3dc41406d0 2023-10-16T20:56:33.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dcbc9e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc4152c00 con 0x7f3dc415a810 2023-10-16T20:56:33.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dc9239640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dc41406d0 0x7f3dc41552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:33.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dca23b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3dc415a810 0x7f3dc4153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:33.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dca23b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3dc415a810 0x7f3dc4153390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:49364/0 (socket says 172.21.15.88:49364) 2023-10-16T20:56:33.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dc9a3a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3dc406a970 0x7f3dc41f5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:33.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dca23b640 1 -- 172.21.15.88:0/1738454560 learned_addr learned my addr 172.21.15.88:0/1738454560 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:33.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dca23b640 1 -- 172.21.15.88:0/1738454560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dc41406d0 msgr2=0x7f3dc41552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:33.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dca23b640 1 --2- 172.21.15.88:0/1738454560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dc41406d0 0x7f3dc41552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:33.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dca23b640 1 -- 172.21.15.88:0/1738454560 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3dc406a970 msgr2=0x7f3dc41f5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:33.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dca23b640 1 --2- 172.21.15.88:0/1738454560 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3dc406a970 0x7f3dc41f5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:33.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.774+0000 7f3dca23b640 1 -- 172.21.15.88:0/1738454560 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dbc009a70 con 0x7f3dc415a810 2023-10-16T20:56:33.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.778+0000 7f3dca23b640 1 --2- 172.21.15.88:0/1738454560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3dc415a810 0x7f3dc4153390 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc009d90 tx=0x7f3dbc0196f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:33.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.778+0000 7f3dba7fc640 1 -- 172.21.15.88:0/1738454560 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc0ad8d0 con 0x7f3dc415a810 2023-10-16T20:56:33.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.778+0000 7f3dba7fc640 1 -- 172.21.15.88:0/1738454560 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dbc0ada30 con 0x7f3dc415a810 2023-10-16T20:56:33.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.778+0000 7f3dba7fc640 1 -- 172.21.15.88:0/1738454560 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc0adde0 con 0x7f3dc415a810 2023-10-16T20:56:33.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.778+0000 7f3dcbc9e640 1 -- 172.21.15.88:0/1738454560 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc41f63b0 con 0x7f3dc415a810 2023-10-16T20:56:33.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.778+0000 7f3dcbc9e640 1 -- 172.21.15.88:0/1738454560 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc40c5320 con 0x7f3dc415a810 2023-10-16T20:56:33.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.778+0000 7f3dcbc9e640 1 -- 172.21.15.88:0/1738454560 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc4074700 con 0x7f3dc415a810 2023-10-16T20:56:33.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.778+0000 7f3dba7fc640 1 -- 172.21.15.88:0/1738454560 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3dbc0c4460 con 0x7f3dc415a810 2023-10-16T20:56:33.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.782+0000 7f3dba7fc640 1 --2- 172.21.15.88:0/1738454560 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3da80664a0 0x7f3da8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:33.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.782+0000 7f3dba7fc640 1 -- 172.21.15.88:0/1738454560 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7f3dbc11f930 con 0x7f3dc415a810 2023-10-16T20:56:33.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.782+0000 7f3dc9a3a640 1 --2- 172.21.15.88:0/1738454560 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3da80664a0 0x7f3da8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:33.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.782+0000 7f3dc9a3a640 1 --2- 172.21.15.88:0/1738454560 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3da80664a0 0x7f3da8068960 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0000f70 tx=0x7f3dc0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:33.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:33.790+0000 7f3dba7fc640 1 -- 172.21.15.88:0/1738454560 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3dbc0ec820 con 0x7f3dc415a810 2023-10-16T20:56:34.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:34.050+0000 7f3dcbc9e640 1 -- 172.21.15.88:0/1738454560 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3dc4156560 con 0x7f3dc415a810 2023-10-16T20:56:34.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:34.050+0000 7f3dba7fc640 1 -- 172.21.15.88:0/1738454560 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+6572 (secure 0 0 0) 0x7f3dbc0efe40 con 0x7f3dc415a810 2023-10-16T20:56:34.051 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:34.052 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:12.691340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:34.052 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 102 2023-10-16T20:56:34.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:34.054+0000 7f3dcbc9e640 1 -- 172.21.15.88:0/1738454560 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3da80664a0 msgr2=0x7f3da8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:34.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:34.054+0000 7f3dcbc9e640 1 --2- 172.21.15.88:0/1738454560 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3da80664a0 0x7f3da8068960 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0000f70 tx=0x7f3dc0009040 comp rx=0 tx=0).stop 2023-10-16T20:56:34.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:34.054+0000 7f3dcbc9e640 1 -- 172.21.15.88:0/1738454560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3dc415a810 msgr2=0x7f3dc4153390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:34.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:34.054+0000 7f3dcbc9e640 1 --2- 172.21.15.88:0/1738454560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3dc415a810 0x7f3dc4153390 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc009d90 tx=0x7f3dbc0196f0 comp rx=0 tx=0).stop 2023-10-16T20:56:34.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:34.054+0000 7f3dcbc9e640 1 -- 172.21.15.88:0/1738454560 shutdown_connections 2023-10-16T20:56:34.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:34.054+0000 7f3dcbc9e640 1 --2- 172.21.15.88:0/1738454560 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3da80664a0 0x7f3da8068960 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:34.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:34.054+0000 7f3dcbc9e640 1 --2- 172.21.15.88:0/1738454560 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3dc415a810 0x7f3dc4153390 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:34.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:34.054+0000 7f3dcbc9e640 1 --2- 172.21.15.88:0/1738454560 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3dc41406d0 0x7f3dc41552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:34.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:34.054+0000 7f3dcbc9e640 1 --2- 172.21.15.88:0/1738454560 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3dc406a970 0x7f3dc41f5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:34.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:34.054+0000 7f3dcbc9e640 1 -- 172.21.15.88:0/1738454560 >> 172.21.15.88:0/1738454560 conn(0x7f3dc40cad50 msgr2=0x7f3dc4155d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:34.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:34.054+0000 7f3dcbc9e640 1 -- 172.21.15.88:0/1738454560 shutdown_connections 2023-10-16T20:56:34.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:34.054+0000 7f3dcbc9e640 1 -- 172.21.15.88:0/1738454560 wait complete. 2023-10-16T20:56:34.078 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-16T20:56:36.079 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:36.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.218+0000 7f77233d0640 1 Processor -- start 2023-10-16T20:56:36.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.218+0000 7f77233d0640 1 -- start start 2023-10-16T20:56:36.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.218+0000 7f77233d0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c7dc0 0x7f771c159450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:36.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.218+0000 7f77233d0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f771c0c81a0 0x7f771c06b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:36.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.218+0000 7f77233d0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771c0c8810 con 0x7f771c0c76c0 2023-10-16T20:56:36.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.218+0000 7f77233d0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f771c06b400 con 0x7f771c0c81a0 2023-10-16T20:56:36.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.218+0000 7f77233d0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771c06b0d0 con 0x7f771c0c7dc0 2023-10-16T20:56:36.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.218+0000 7f772196d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f771c0c81a0 0x7f771c06b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:36.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.218+0000 7f772196d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f771c0c81a0 0x7f771c06b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52442/0 (socket says 172.21.15.88:52442) 2023-10-16T20:56:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.218+0000 7f772196d640 1 -- 172.21.15.88:0/144929832 learned_addr learned my addr 172.21.15.88:0/144929832 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.218+0000 7f772116c640 1 --2- 172.21.15.88:0/144929832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c7dc0 0x7f771c159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.218+0000 7f772196d640 1 -- 172.21.15.88:0/144929832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c7dc0 msgr2=0x7f771c159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:56:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.218+0000 7f772196d640 1 --2- 172.21.15.88:0/144929832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c7dc0 0x7f771c159450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.218+0000 7f772196d640 1 -- 172.21.15.88:0/144929832 >> v1:172.21.15.191:6789/0 conn(0x7f771c0c76c0 legacy=0x7f771c06d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.218+0000 7f772196d640 1 -- 172.21.15.88:0/144929832 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771c06e6a0 con 0x7f771c0c81a0 2023-10-16T20:56:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.222+0000 7f772196d640 1 --2- 172.21.15.88:0/144929832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c0c81a0 0x7f771c06b890 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f7714009e50 tx=0x7f771401c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7428549236f2b33 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:36.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.222+0000 7f7713fff640 1 -- 172.21.15.88:0/144929832 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77140a5640 con 0x7f771c0c81a0 2023-10-16T20:56:36.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.222+0000 7f7713fff640 1 -- 172.21.15.88:0/144929832 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77140a57a0 con 0x7f771c0c81a0 2023-10-16T20:56:36.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.222+0000 7f7713fff640 1 -- 172.21.15.88:0/144929832 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77140a5b30 con 0x7f771c0c81a0 2023-10-16T20:56:36.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.222+0000 7f77233d0640 1 -- 172.21.15.88:0/144929832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c0c81a0 msgr2=0x7f771c06b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:36.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.222+0000 7f77233d0640 1 --2- 172.21.15.88:0/144929832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c0c81a0 0x7f771c06b890 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f7714009e50 tx=0x7f771401c980 comp rx=0 tx=0).stop 2023-10-16T20:56:36.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.222+0000 7f77233d0640 1 -- 172.21.15.88:0/144929832 shutdown_connections 2023-10-16T20:56:36.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.222+0000 7f77233d0640 1 --2- 172.21.15.88:0/144929832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c0c81a0 0x7f771c06b890 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.222+0000 7f77233d0640 1 --2- 172.21.15.88:0/144929832 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c7dc0 0x7f771c159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.222+0000 7f77233d0640 1 -- 172.21.15.88:0/144929832 >> 172.21.15.88:0/144929832 conn(0x7f771c071a60 msgr2=0x7f771c071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.222+0000 7f77233d0640 1 -- 172.21.15.88:0/144929832 shutdown_connections 2023-10-16T20:56:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.222+0000 7f77233d0640 1 -- 172.21.15.88:0/144929832 wait complete. 2023-10-16T20:56:36.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f77233d0640 1 Processor -- start 2023-10-16T20:56:36.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f77233d0640 1 -- start start 2023-10-16T20:56:36.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f77233d0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c76c0 0x7f771c154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:36.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f77233d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c0c7dc0 0x7f771c14ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:36.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f77233d0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f771c0c81a0 0x7f771c1572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:36.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f77233d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c1f5810 con 0x7f771c0c7dc0 2023-10-16T20:56:36.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f77233d0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c155240 con 0x7f771c0c81a0 2023-10-16T20:56:36.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f77233d0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771c157bb0 con 0x7f771c0c76c0 2023-10-16T20:56:36.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f772096b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c0c7dc0 0x7f771c14ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:36.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f772096b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c0c7dc0 0x7f771c14ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52458/0 (socket says 172.21.15.88:52458) 2023-10-16T20:56:36.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f772096b640 1 -- 172.21.15.88:0/4016653643 learned_addr learned my addr 172.21.15.88:0/4016653643 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:36.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f772116c640 1 --2- 172.21.15.88:0/4016653643 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c76c0 0x7f771c154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:36.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f772196d640 1 --2- 172.21.15.88:0/4016653643 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f771c0c81a0 0x7f771c1572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:36.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f772096b640 1 -- 172.21.15.88:0/4016653643 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c76c0 msgr2=0x7f771c154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:36.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f772096b640 1 --2- 172.21.15.88:0/4016653643 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c76c0 0x7f771c154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:36.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f772096b640 1 -- 172.21.15.88:0/4016653643 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f771c0c81a0 msgr2=0x7f771c1572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:36.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f772096b640 1 --2- 172.21.15.88:0/4016653643 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f771c0c81a0 0x7f771c1572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:36.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f772096b640 1 -- 172.21.15.88:0/4016653643 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7714009b30 con 0x7f771c0c7dc0 2023-10-16T20:56:36.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f772096b640 1 --2- 172.21.15.88:0/4016653643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c0c7dc0 0x7f771c14ed50 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f770c00c3c0 tx=0x7f770c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:36.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f7711ffb640 1 -- 172.21.15.88:0/4016653643 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c0052d0 con 0x7f771c0c7dc0 2023-10-16T20:56:36.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f7711ffb640 1 -- 172.21.15.88:0/4016653643 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770c005430 con 0x7f771c0c7dc0 2023-10-16T20:56:36.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f7711ffb640 1 -- 172.21.15.88:0/4016653643 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c003840 con 0x7f771c0c7dc0 2023-10-16T20:56:36.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f77233d0640 1 -- 172.21.15.88:0/4016653643 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771c1f5ab0 con 0x7f771c0c7dc0 2023-10-16T20:56:36.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.226+0000 7f77233d0640 1 -- 172.21.15.88:0/4016653643 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c1f6070 con 0x7f771c0c7dc0 2023-10-16T20:56:36.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.230+0000 7f77233d0640 1 -- 172.21.15.88:0/4016653643 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f771c074700 con 0x7f771c0c7dc0 2023-10-16T20:56:36.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.230+0000 7f7711ffb640 1 -- 172.21.15.88:0/4016653643 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f770c0039a0 con 0x7f771c0c7dc0 2023-10-16T20:56:36.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.230+0000 7f7711ffb640 1 --2- 172.21.15.88:0/4016653643 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f77000664a0 0x7f7700068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:36.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.230+0000 7f7711ffb640 1 -- 172.21.15.88:0/4016653643 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7f770c083af0 con 0x7f771c0c7dc0 2023-10-16T20:56:36.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.230+0000 7f772116c640 1 --2- 172.21.15.88:0/4016653643 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f77000664a0 0x7f7700068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:36.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.230+0000 7f772116c640 1 --2- 172.21.15.88:0/4016653643 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f77000664a0 0x7f7700068960 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7718009b60 tx=0x7f7718009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:36.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.234+0000 7f7711ffb640 1 -- 172.21.15.88:0/4016653643 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f770c050880 con 0x7f771c0c7dc0 2023-10-16T20:56:36.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.414+0000 7f77233d0640 1 -- 172.21.15.88:0/4016653643 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f771c0c3a40 con 0x7f771c0c7dc0 2023-10-16T20:56:36.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.414+0000 7f7711ffb640 1 -- 172.21.15.88:0/4016653643 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+6572 (secure 0 0 0) 0x7f770c017090 con 0x7f771c0c7dc0 2023-10-16T20:56:36.415 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:36.416 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:12.691340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:36.416 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 102 2023-10-16T20:56:36.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.418+0000 7f77233d0640 1 -- 172.21.15.88:0/4016653643 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f77000664a0 msgr2=0x7f7700068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:36.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.418+0000 7f77233d0640 1 --2- 172.21.15.88:0/4016653643 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f77000664a0 0x7f7700068960 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7718009b60 tx=0x7f7718009190 comp rx=0 tx=0).stop 2023-10-16T20:56:36.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.418+0000 7f77233d0640 1 -- 172.21.15.88:0/4016653643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c0c7dc0 msgr2=0x7f771c14ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:36.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.418+0000 7f77233d0640 1 --2- 172.21.15.88:0/4016653643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c0c7dc0 0x7f771c14ed50 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f770c00c3c0 tx=0x7f770c00c880 comp rx=0 tx=0).stop 2023-10-16T20:56:36.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.418+0000 7f77233d0640 1 -- 172.21.15.88:0/4016653643 shutdown_connections 2023-10-16T20:56:36.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.418+0000 7f77233d0640 1 --2- 172.21.15.88:0/4016653643 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f77000664a0 0x7f7700068960 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:36.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.418+0000 7f77233d0640 1 --2- 172.21.15.88:0/4016653643 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f771c0c81a0 0x7f771c1572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:36.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.418+0000 7f77233d0640 1 --2- 172.21.15.88:0/4016653643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c0c7dc0 0x7f771c14ed50 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:36.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.418+0000 7f77233d0640 1 --2- 172.21.15.88:0/4016653643 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f771c0c76c0 0x7f771c154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:36.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.418+0000 7f77233d0640 1 -- 172.21.15.88:0/4016653643 >> 172.21.15.88:0/4016653643 conn(0x7f771c071a60 msgr2=0x7f771c150420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:36.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.418+0000 7f77233d0640 1 -- 172.21.15.88:0/4016653643 shutdown_connections 2023-10-16T20:56:36.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:36.418+0000 7f77233d0640 1 -- 172.21.15.88:0/4016653643 wait complete. 2023-10-16T20:56:36.437 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-16T20:56:38.439 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:38.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.582+0000 7f4d66d1d640 1 Processor -- start 2023-10-16T20:56:38.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.582+0000 7f4d66d1d640 1 -- start start 2023-10-16T20:56:38.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.586+0000 7f4d66d1d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4d60150280 0x7f4d600c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:38.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.586+0000 7f4d66d1d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4d60150660 0x7f4d600c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:38.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.586+0000 7f4d66d1d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d600c6bc0 0x7f4d6006d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:38.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.586+0000 7f4d66d1d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d600c5c60 con 0x7f4d60150660 2023-10-16T20:56:38.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.586+0000 7f4d66d1d640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d600c5370 con 0x7f4d60150280 2023-10-16T20:56:38.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.586+0000 7f4d66d1d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60156d10 con 0x7f4d600c6bc0 2023-10-16T20:56:38.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.586+0000 7f4d57fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4d60150660 0x7f4d600c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:38.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.586+0000 7f4d57fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4d60150660 0x7f4d600c3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52470/0 (socket says 172.21.15.88:52470) 2023-10-16T20:56:38.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.586+0000 7f4d57fff640 1 -- 172.21.15.88:0/1596369510 learned_addr learned my addr 172.21.15.88:0/1596369510 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:38.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.586+0000 7f4d64ab9640 1 --2- 172.21.15.88:0/1596369510 >> v2:172.21.15.191:3300/0 conn(0x7f4d60150280 0x7f4d600c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:38.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.586+0000 7f4d652ba640 1 --2- 172.21.15.88:0/1596369510 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d600c6bc0 0x7f4d6006d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:38.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.586+0000 7f4d57fff640 1 -- 172.21.15.88:0/1596369510 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d600c6bc0 msgr2=0x7f4d6006d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:56:38.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.586+0000 7f4d57fff640 1 --2- 172.21.15.88:0/1596369510 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d600c6bc0 0x7f4d6006d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:38.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.586+0000 7f4d57fff640 1 -- 172.21.15.88:0/1596369510 >> v2:172.21.15.191:3300/0 conn(0x7f4d60150280 msgr2=0x7f4d600c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:56:38.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.586+0000 7f4d57fff640 1 --2- 172.21.15.88:0/1596369510 >> v2:172.21.15.191:3300/0 conn(0x7f4d60150280 0x7f4d600c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:38.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.586+0000 7f4d57fff640 1 -- 172.21.15.88:0/1596369510 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d600c5610 con 0x7f4d60150660 2023-10-16T20:56:38.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.590+0000 7f4d57fff640 1 --2- 172.21.15.88:0/1596369510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d60150660 0x7f4d600c3110 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f4d6006f060 tx=0x7f4d5001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78f764fb103165e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:38.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.590+0000 7f4d577fe640 1 -- 172.21.15.88:0/1596369510 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d500ac600 con 0x7f4d60150660 2023-10-16T20:56:38.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.590+0000 7f4d577fe640 1 -- 172.21.15.88:0/1596369510 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d500a8050 con 0x7f4d60150660 2023-10-16T20:56:38.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.590+0000 7f4d577fe640 1 -- 172.21.15.88:0/1596369510 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d500ac980 con 0x7f4d60150660 2023-10-16T20:56:38.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.590+0000 7f4d66d1d640 1 -- 172.21.15.88:0/1596369510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d60150660 msgr2=0x7f4d600c3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:38.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.590+0000 7f4d66d1d640 1 --2- 172.21.15.88:0/1596369510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d60150660 0x7f4d600c3110 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f4d6006f060 tx=0x7f4d5001c4d0 comp rx=0 tx=0).stop 2023-10-16T20:56:38.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.590+0000 7f4d66d1d640 1 -- 172.21.15.88:0/1596369510 shutdown_connections 2023-10-16T20:56:38.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.590+0000 7f4d66d1d640 1 --2- 172.21.15.88:0/1596369510 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d600c6bc0 0x7f4d6006d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:38.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.590+0000 7f4d66d1d640 1 --2- 172.21.15.88:0/1596369510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d60150660 0x7f4d600c3110 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:38.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.590+0000 7f4d66d1d640 1 --2- 172.21.15.88:0/1596369510 >> v2:172.21.15.191:3300/0 conn(0x7f4d60150280 0x7f4d600c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:38.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.590+0000 7f4d66d1d640 1 -- 172.21.15.88:0/1596369510 >> 172.21.15.88:0/1596369510 conn(0x7f4d600cad50 msgr2=0x7f4d600cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:38.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.590+0000 7f4d66d1d640 1 -- 172.21.15.88:0/1596369510 shutdown_connections 2023-10-16T20:56:38.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.590+0000 7f4d66d1d640 1 -- 172.21.15.88:0/1596369510 wait complete. 2023-10-16T20:56:38.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.590+0000 7f4d66d1d640 1 Processor -- start 2023-10-16T20:56:38.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.590+0000 7f4d66d1d640 1 -- start start 2023-10-16T20:56:38.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d66d1d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d600c6bc0 0x7f4d60070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:38.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d66d1d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d60150280 0x7f4d600c3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:38.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d66d1d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d60150660 0x7f4d601efd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:38.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d66d1d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d601f3ea0 con 0x7f4d600c6bc0 2023-10-16T20:56:38.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d66d1d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d601f2fd0 con 0x7f4d60150280 2023-10-16T20:56:38.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d66d1d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d601f3290 con 0x7f4d60150660 2023-10-16T20:56:38.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d64ab9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d600c6bc0 0x7f4d60070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:38.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d64ab9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d600c6bc0 0x7f4d60070430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52476/0 (socket says 172.21.15.88:52476) 2023-10-16T20:56:38.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d64ab9640 1 -- 172.21.15.88:0/3557607446 learned_addr learned my addr 172.21.15.88:0/3557607446 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:38.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d652ba640 1 --2- 172.21.15.88:0/3557607446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d60150660 0x7f4d601efd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:38.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d57fff640 1 --2- 172.21.15.88:0/3557607446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d60150280 0x7f4d600c3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:38.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d64ab9640 1 -- 172.21.15.88:0/3557607446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d60150660 msgr2=0x7f4d601efd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:38.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d64ab9640 1 --2- 172.21.15.88:0/3557607446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d60150660 0x7f4d601efd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:38.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d64ab9640 1 -- 172.21.15.88:0/3557607446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d60150280 msgr2=0x7f4d600c3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:38.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d64ab9640 1 --2- 172.21.15.88:0/3557607446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d60150280 0x7f4d600c3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:38.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d64ab9640 1 -- 172.21.15.88:0/3557607446 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d500094d0 con 0x7f4d600c6bc0 2023-10-16T20:56:38.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d64ab9640 1 --2- 172.21.15.88:0/3557607446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d600c6bc0 0x7f4d60070430 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c00e930 tx=0x7f4d5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:38.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d557fa640 1 -- 172.21.15.88:0/3557607446 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c009690 con 0x7f4d600c6bc0 2023-10-16T20:56:38.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d557fa640 1 -- 172.21.15.88:0/3557607446 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5c00ca00 con 0x7f4d600c6bc0 2023-10-16T20:56:38.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d557fa640 1 -- 172.21.15.88:0/3557607446 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c00cd70 con 0x7f4d600c6bc0 2023-10-16T20:56:38.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d66d1d640 1 -- 172.21.15.88:0/3557607446 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d601f3570 con 0x7f4d600c6bc0 2023-10-16T20:56:38.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d66d1d640 1 -- 172.21.15.88:0/3557607446 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d601f3b30 con 0x7f4d600c6bc0 2023-10-16T20:56:38.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d66d1d640 1 -- 172.21.15.88:0/3557607446 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d60074700 con 0x7f4d600c6bc0 2023-10-16T20:56:38.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.594+0000 7f4d557fa640 1 -- 172.21.15.88:0/3557607446 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4d5c013470 con 0x7f4d600c6bc0 2023-10-16T20:56:38.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.598+0000 7f4d557fa640 1 --2- 172.21.15.88:0/3557607446 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4d4c0664d0 0x7f4d4c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:38.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.598+0000 7f4d557fa640 1 -- 172.21.15.88:0/3557607446 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7351+0+0 (secure 0 0 0) 0x7f4d5c084680 con 0x7f4d600c6bc0 2023-10-16T20:56:38.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.598+0000 7f4d57fff640 1 --2- 172.21.15.88:0/3557607446 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4d4c0664d0 0x7f4d4c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:38.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.598+0000 7f4d57fff640 1 --2- 172.21.15.88:0/3557607446 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4d4c0664d0 0x7f4d4c068990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4d500ae000 tx=0x7f4d50008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:38.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.602+0000 7f4d557fa640 1 -- 172.21.15.88:0/3557607446 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4d5c051540 con 0x7f4d600c6bc0 2023-10-16T20:56:38.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.870+0000 7f4d66d1d640 1 -- 172.21.15.88:0/3557607446 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d600c1650 con 0x7f4d600c6bc0 2023-10-16T20:56:38.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.870+0000 7f4d557fa640 1 -- 172.21.15.88:0/3557607446 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+6572 (secure 0 0 0) 0x7f4d5c054b60 con 0x7f4d600c6bc0 2023-10-16T20:56:38.871 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:38.871 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:12.691340+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5284,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5284":{"gid":5284,"name":"c","rank":3,"incarnation":38,"state":"up:active","state_seq":17,"addr":"172.21.15.88:6839/2172973775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2172973775},{"type":"v1","addr":"172.21.15.88:6839","nonce":2172973775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:38.871 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 102 2023-10-16T20:56:38.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.874+0000 7f4d66d1d640 1 -- 172.21.15.88:0/3557607446 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4d4c0664d0 msgr2=0x7f4d4c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:38.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.874+0000 7f4d66d1d640 1 --2- 172.21.15.88:0/3557607446 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4d4c0664d0 0x7f4d4c068990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4d500ae000 tx=0x7f4d50008040 comp rx=0 tx=0).stop 2023-10-16T20:56:38.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.874+0000 7f4d66d1d640 1 -- 172.21.15.88:0/3557607446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d600c6bc0 msgr2=0x7f4d60070430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:38.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.874+0000 7f4d66d1d640 1 --2- 172.21.15.88:0/3557607446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d600c6bc0 0x7f4d60070430 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c00e930 tx=0x7f4d5c00edf0 comp rx=0 tx=0).stop 2023-10-16T20:56:38.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.874+0000 7f4d66d1d640 1 -- 172.21.15.88:0/3557607446 shutdown_connections 2023-10-16T20:56:38.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.874+0000 7f4d66d1d640 1 --2- 172.21.15.88:0/3557607446 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4d4c0664d0 0x7f4d4c068990 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:38.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.874+0000 7f4d66d1d640 1 --2- 172.21.15.88:0/3557607446 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4d60150660 0x7f4d601efd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:38.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.874+0000 7f4d66d1d640 1 --2- 172.21.15.88:0/3557607446 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4d60150280 0x7f4d600c3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:38.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.874+0000 7f4d66d1d640 1 --2- 172.21.15.88:0/3557607446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d600c6bc0 0x7f4d60070430 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:38.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.874+0000 7f4d66d1d640 1 -- 172.21.15.88:0/3557607446 >> 172.21.15.88:0/3557607446 conn(0x7f4d600cad50 msgr2=0x7f4d601557b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:38.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.874+0000 7f4d66d1d640 1 -- 172.21.15.88:0/3557607446 shutdown_connections 2023-10-16T20:56:38.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:38.874+0000 7f4d66d1d640 1 -- 172.21.15.88:0/3557607446 wait complete. 2023-10-16T20:56:38.897 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-16T20:56:40.897 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:41.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.046+0000 7f8714a17640 1 Processor -- start 2023-10-16T20:56:41.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.046+0000 7f8714a17640 1 -- start start 2023-10-16T20:56:41.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.046+0000 7f8714a17640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8710153770 0x7f8710159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:41.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.046+0000 7f8714a17640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871015a410 con 0x7f87100cece0 2023-10-16T20:56:41.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.046+0000 7f8714a17640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8710153b50 con 0x7f87100c9bb0 2023-10-16T20:56:41.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.046+0000 7f8714a17640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87100cc250 con 0x7f8710153770 2023-10-16T20:56:41.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.046+0000 7f870ed9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f87100cece0 0x7f8710163dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33340/0 (socket says 172.21.15.88:33340) 2023-10-16T20:56:41.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.046+0000 7f870ed9d640 1 -- 172.21.15.88:0/3323000192 learned_addr learned my addr 172.21.15.88:0/3323000192 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:56:41.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.046+0000 7f870e59c640 1 --2- 172.21.15.88:0/3323000192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8710153770 0x7f8710159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:41.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.046+0000 7f870d59a640 1 -- 172.21.15.88:0/3323000192 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58973546 0 0) 0x7f871015a410 con 0x7f87100cece0 2023-10-16T20:56:41.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.046+0000 7f870d59a640 1 -- 172.21.15.88:0/3323000192 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f8003530 con 0x7f87100cece0 2023-10-16T20:56:41.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.046+0000 7f870d59a640 1 -- 172.21.15.88:0/3323000192 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617878457 0 0) 0x7f8710153b50 con 0x7f87100c9bb0 2023-10-16T20:56:41.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.046+0000 7f870d59a640 1 -- 172.21.15.88:0/3323000192 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f871015a410 con 0x7f87100c9bb0 2023-10-16T20:56:41.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.046+0000 7f870e59c640 1 -- 172.21.15.88:0/3323000192 >> v1:172.21.15.191:6789/0 conn(0x7f87100c9bb0 legacy=0x7f87100c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:41.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.046+0000 7f870e59c640 1 -- 172.21.15.88:0/3323000192 >> v1:172.21.15.88:6789/0 conn(0x7f87100cece0 legacy=0x7f8710163dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:41.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.046+0000 7f870e59c640 1 -- 172.21.15.88:0/3323000192 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87100cb2f0 con 0x7f8710153770 2023-10-16T20:56:41.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.050+0000 7f870e59c640 1 --2- 172.21.15.88:0/3323000192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8710153770 0x7f8710159ed0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f87000097d0 tx=0x7f870001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f419cf61c4bf1a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:41.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.050+0000 7f870d59a640 1 -- 172.21.15.88:0/3323000192 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87000ac410 con 0x7f8710153770 2023-10-16T20:56:41.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.050+0000 7f870d59a640 1 -- 172.21.15.88:0/3323000192 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87000ac570 con 0x7f8710153770 2023-10-16T20:56:41.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.050+0000 7f870d59a640 1 -- 172.21.15.88:0/3323000192 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87000ac900 con 0x7f8710153770 2023-10-16T20:56:41.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.050+0000 7f8714a17640 1 -- 172.21.15.88:0/3323000192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8710153770 msgr2=0x7f8710159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:41.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.050+0000 7f8714a17640 1 --2- 172.21.15.88:0/3323000192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8710153770 0x7f8710159ed0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f87000097d0 tx=0x7f870001c4a0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.050+0000 7f8714a17640 1 -- 172.21.15.88:0/3323000192 shutdown_connections 2023-10-16T20:56:41.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.050+0000 7f8714a17640 1 --2- 172.21.15.88:0/3323000192 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8710153770 0x7f8710159ed0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.050+0000 7f8714a17640 1 -- 172.21.15.88:0/3323000192 >> 172.21.15.88:0/3323000192 conn(0x7f87100715c0 msgr2=0x7f87100719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:41.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.050+0000 7f8714a17640 1 -- 172.21.15.88:0/3323000192 shutdown_connections 2023-10-16T20:56:41.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.050+0000 7f8714a17640 1 -- 172.21.15.88:0/3323000192 wait complete. 2023-10-16T20:56:41.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f8714a17640 1 Processor -- start 2023-10-16T20:56:41.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f8714a17640 1 -- start start 2023-10-16T20:56:41.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f8714a17640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87100c9bb0 0x7f871013d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:41.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f8714a17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87100cece0 0x7f87101f1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:41.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f8714a17640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8710153770 0x7f87101437a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:41.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f8714a17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87101543e0 con 0x7f87100cece0 2023-10-16T20:56:41.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f8714a17640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87101f5180 con 0x7f8710153770 2023-10-16T20:56:41.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f8714a17640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871013c750 con 0x7f87100c9bb0 2023-10-16T20:56:41.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f870dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87100cece0 0x7f87101f1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:41.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f870dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87100cece0 0x7f87101f1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33024/0 (socket says 172.21.15.88:33024) 2023-10-16T20:56:41.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f870dd9b640 1 -- 172.21.15.88:0/374229218 learned_addr learned my addr 172.21.15.88:0/374229218 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:41.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f870e59c640 1 --2- 172.21.15.88:0/374229218 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87100c9bb0 0x7f871013d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:41.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f870ed9d640 1 --2- 172.21.15.88:0/374229218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8710153770 0x7f87101437a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:41.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f870dd9b640 1 -- 172.21.15.88:0/374229218 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87100c9bb0 msgr2=0x7f871013d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:41.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f870dd9b640 1 --2- 172.21.15.88:0/374229218 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87100c9bb0 0x7f871013d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f870dd9b640 1 -- 172.21.15.88:0/374229218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8710153770 msgr2=0x7f87101437a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:41.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f870dd9b640 1 --2- 172.21.15.88:0/374229218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8710153770 0x7f87101437a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f870dd9b640 1 -- 172.21.15.88:0/374229218 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87000093b0 con 0x7f87100cece0 2023-10-16T20:56:41.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f870ed9d640 1 --2- 172.21.15.88:0/374229218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8710153770 0x7f87101437a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:56:41.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f870dd9b640 1 --2- 172.21.15.88:0/374229218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87100cece0 0x7f87101f1b70 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f870400c490 tx=0x7f870400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:41.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f86feffd640 1 -- 172.21.15.88:0/374229218 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87040052d0 con 0x7f87100cece0 2023-10-16T20:56:41.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f86feffd640 1 -- 172.21.15.88:0/374229218 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8704005430 con 0x7f87100cece0 2023-10-16T20:56:41.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f8714a17640 1 -- 172.21.15.88:0/374229218 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87101f5420 con 0x7f87100cece0 2023-10-16T20:56:41.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.054+0000 7f86feffd640 1 -- 172.21.15.88:0/374229218 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704012870 con 0x7f87100cece0 2023-10-16T20:56:41.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.058+0000 7f8714a17640 1 -- 172.21.15.88:0/374229218 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87101f59e0 con 0x7f87100cece0 2023-10-16T20:56:41.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.058+0000 7f8714a17640 1 -- 172.21.15.88:0/374229218 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87101f5bd0 con 0x7f87100cece0 2023-10-16T20:56:41.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.058+0000 7f86feffd640 1 -- 172.21.15.88:0/374229218 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f87040129d0 con 0x7f87100cece0 2023-10-16T20:56:41.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.058+0000 7f86feffd640 1 --2- 172.21.15.88:0/374229218 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f86f00664a0 0x7f86f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:41.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.058+0000 7f86feffd640 1 -- 172.21.15.88:0/374229218 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7f8704084280 con 0x7f87100cece0 2023-10-16T20:56:41.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.058+0000 7f870e59c640 1 --2- 172.21.15.88:0/374229218 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f86f00664a0 0x7f86f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:41.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.058+0000 7f870e59c640 1 --2- 172.21.15.88:0/374229218 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f86f00664a0 0x7f86f0068960 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f87000ae000 tx=0x7f87000a6040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:41.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.062+0000 7f86feffd640 1 -- 172.21.15.88:0/374229218 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8704050f10 con 0x7f87100cece0 2023-10-16T20:56:41.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.246+0000 7f8714a17640 1 -- 172.21.15.88:0/374229218 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8710074700 con 0x7f87100cece0 2023-10-16T20:56:41.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.246+0000 7f86feffd640 1 -- 172.21.15.88:0/374229218 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5781 (secure 0 0 0) 0x7f8704054530 con 0x7f87100cece0 2023-10-16T20:56:41.248 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:41.248 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:40.633696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:41.249 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 103 2023-10-16T20:56:41.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.250+0000 7f8714a17640 1 -- 172.21.15.88:0/374229218 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f86f00664a0 msgr2=0x7f86f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:41.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.250+0000 7f8714a17640 1 --2- 172.21.15.88:0/374229218 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f86f00664a0 0x7f86f0068960 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f87000ae000 tx=0x7f87000a6040 comp rx=0 tx=0).stop 2023-10-16T20:56:41.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.250+0000 7f8714a17640 1 -- 172.21.15.88:0/374229218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87100cece0 msgr2=0x7f87101f1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:41.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.250+0000 7f8714a17640 1 --2- 172.21.15.88:0/374229218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87100cece0 0x7f87101f1b70 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f870400c490 tx=0x7f870400c950 comp rx=0 tx=0).stop 2023-10-16T20:56:41.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.250+0000 7f8714a17640 1 -- 172.21.15.88:0/374229218 shutdown_connections 2023-10-16T20:56:41.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.250+0000 7f8714a17640 1 --2- 172.21.15.88:0/374229218 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f86f00664a0 0x7f86f0068960 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.250+0000 7f8714a17640 1 --2- 172.21.15.88:0/374229218 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8710153770 0x7f87101437a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.250+0000 7f8714a17640 1 --2- 172.21.15.88:0/374229218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87100cece0 0x7f87101f1b70 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.250+0000 7f8714a17640 1 --2- 172.21.15.88:0/374229218 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f87100c9bb0 0x7f871013d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.250+0000 7f8714a17640 1 -- 172.21.15.88:0/374229218 >> 172.21.15.88:0/374229218 conn(0x7f87100715c0 msgr2=0x7f87100719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:41.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.250+0000 7f8714a17640 1 -- 172.21.15.88:0/374229218 shutdown_connections 2023-10-16T20:56:41.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.250+0000 7f8714a17640 1 -- 172.21.15.88:0/374229218 wait complete. 2023-10-16T20:56:41.268 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-16T20:56:41.269 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-16T20:56:41.269 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:41.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.406+0000 7f3c4c73e640 1 Processor -- start 2023-10-16T20:56:41.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.406+0000 7f3c4c73e640 1 -- start start 2023-10-16T20:56:41.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.410+0000 7f3c4c73e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f3c440c7dc0 0x7f3c44159450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:41.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.410+0000 7f3c4c73e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c440c81a0 0x7f3c4406b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:41.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.410+0000 7f3c4c73e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c440c8810 con 0x7f3c440c76c0 2023-10-16T20:56:41.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.410+0000 7f3c4c73e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c4406b400 con 0x7f3c440c7dc0 2023-10-16T20:56:41.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.410+0000 7f3c4c73e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4406b0d0 con 0x7f3c440c81a0 2023-10-16T20:56:41.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.410+0000 7f3c49cd9640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3c440c76c0 0x7f3c4406d650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33350/0 (socket says 172.21.15.88:33350) 2023-10-16T20:56:41.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.410+0000 7f3c49cd9640 1 -- 172.21.15.88:0/1335437068 learned_addr learned my addr 172.21.15.88:0/1335437068 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:56:41.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.410+0000 7f3c4acdb640 1 --2- 172.21.15.88:0/1335437068 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c440c81a0 0x7f3c4406b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:41.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.410+0000 7f3c4a4da640 1 --2- 172.21.15.88:0/1335437068 >> v2:172.21.15.191:3300/0 conn(0x7f3c440c7dc0 0x7f3c44159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:41.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.410+0000 7f3c494d8640 1 -- 172.21.15.88:0/1335437068 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366568252 0 0) 0x7f3c440c8810 con 0x7f3c440c76c0 2023-10-16T20:56:41.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.410+0000 7f3c494d8640 1 -- 172.21.15.88:0/1335437068 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c30003530 con 0x7f3c440c76c0 2023-10-16T20:56:41.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.410+0000 7f3c4acdb640 1 -- 172.21.15.88:0/1335437068 >> v2:172.21.15.191:3300/0 conn(0x7f3c440c7dc0 msgr2=0x7f3c44159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:56:41.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.410+0000 7f3c4acdb640 1 --2- 172.21.15.88:0/1335437068 >> v2:172.21.15.191:3300/0 conn(0x7f3c440c7dc0 0x7f3c44159450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.410+0000 7f3c4acdb640 1 -- 172.21.15.88:0/1335437068 >> v1:172.21.15.88:6789/0 conn(0x7f3c440c76c0 legacy=0x7f3c4406d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:41.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.410+0000 7f3c4acdb640 1 -- 172.21.15.88:0/1335437068 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4406e6a0 con 0x7f3c440c81a0 2023-10-16T20:56:41.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.410+0000 7f3c4a4da640 1 --2- 172.21.15.88:0/1335437068 >> v2:172.21.15.191:3300/0 conn(0x7f3c440c7dc0 0x7f3c44159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:56:41.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.410+0000 7f3c4acdb640 1 --2- 172.21.15.88:0/1335437068 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c440c81a0 0x7f3c4406b890 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009d90 tx=0x7f3c3c01c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f137f3e6dbff7f57 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:41.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.414+0000 7f3c494d8640 1 -- 172.21.15.88:0/1335437068 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c0a5780 con 0x7f3c440c81a0 2023-10-16T20:56:41.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.414+0000 7f3c494d8640 1 -- 172.21.15.88:0/1335437068 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3c0a58e0 con 0x7f3c440c81a0 2023-10-16T20:56:41.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.414+0000 7f3c494d8640 1 -- 172.21.15.88:0/1335437068 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c0a5c70 con 0x7f3c440c81a0 2023-10-16T20:56:41.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.414+0000 7f3c4c73e640 1 -- 172.21.15.88:0/1335437068 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c440c81a0 msgr2=0x7f3c4406b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:41.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.414+0000 7f3c4c73e640 1 --2- 172.21.15.88:0/1335437068 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c440c81a0 0x7f3c4406b890 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009d90 tx=0x7f3c3c01c8c0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.414+0000 7f3c4c73e640 1 -- 172.21.15.88:0/1335437068 shutdown_connections 2023-10-16T20:56:41.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.414+0000 7f3c4c73e640 1 --2- 172.21.15.88:0/1335437068 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c440c81a0 0x7f3c4406b890 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.414+0000 7f3c4c73e640 1 --2- 172.21.15.88:0/1335437068 >> v2:172.21.15.191:3300/0 conn(0x7f3c440c7dc0 0x7f3c44159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.414+0000 7f3c4c73e640 1 -- 172.21.15.88:0/1335437068 >> 172.21.15.88:0/1335437068 conn(0x7f3c44071a60 msgr2=0x7f3c44071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:41.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.414+0000 7f3c4c73e640 1 -- 172.21.15.88:0/1335437068 shutdown_connections 2023-10-16T20:56:41.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.414+0000 7f3c4c73e640 1 -- 172.21.15.88:0/1335437068 wait complete. 2023-10-16T20:56:41.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.414+0000 7f3c4c73e640 1 Processor -- start 2023-10-16T20:56:41.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.414+0000 7f3c4c73e640 1 -- start start 2023-10-16T20:56:41.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.414+0000 7f3c4c73e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c440c76c0 0x7f3c441f54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:41.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.414+0000 7f3c4c73e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c440c7dc0 0x7f3c44157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:41.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.414+0000 7f3c4c73e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c440c81a0 0x7f3c44157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:41.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c4c73e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c440cd4d0 con 0x7f3c440c81a0 2023-10-16T20:56:41.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c4c73e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c441572c0 con 0x7f3c440c76c0 2023-10-16T20:56:41.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c4c73e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c441575a0 con 0x7f3c440c7dc0 2023-10-16T20:56:41.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c4acdb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c440c81a0 0x7f3c44157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:41.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c4acdb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c440c81a0 0x7f3c44157f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33040/0 (socket says 172.21.15.88:33040) 2023-10-16T20:56:41.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c4acdb640 1 -- 172.21.15.88:0/1033535092 learned_addr learned my addr 172.21.15.88:0/1033535092 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:41.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c49cd9640 1 --2- 172.21.15.88:0/1033535092 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c440c7dc0 0x7f3c44157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:41.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c4a4da640 1 --2- 172.21.15.88:0/1033535092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c440c76c0 0x7f3c441f54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:41.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c4acdb640 1 -- 172.21.15.88:0/1033535092 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c440c7dc0 msgr2=0x7f3c44157a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:41.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c4acdb640 1 --2- 172.21.15.88:0/1033535092 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c440c7dc0 0x7f3c44157a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c4acdb640 1 -- 172.21.15.88:0/1033535092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c440c76c0 msgr2=0x7f3c441f54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:41.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c4acdb640 1 --2- 172.21.15.88:0/1033535092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c440c76c0 0x7f3c441f54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c4acdb640 1 -- 172.21.15.88:0/1033535092 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c40009420 con 0x7f3c440c81a0 2023-10-16T20:56:41.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c4acdb640 1 --2- 172.21.15.88:0/1033535092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c440c81a0 0x7f3c44157f70 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c01cda0 tx=0x7f3c3c019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:41.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c3affd640 1 -- 172.21.15.88:0/1033535092 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c0ad640 con 0x7f3c440c81a0 2023-10-16T20:56:41.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c3affd640 1 -- 172.21.15.88:0/1033535092 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3c0ad7a0 con 0x7f3c440c81a0 2023-10-16T20:56:41.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c4c73e640 1 -- 172.21.15.88:0/1033535092 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c3c009a70 con 0x7f3c440c81a0 2023-10-16T20:56:41.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c3affd640 1 -- 172.21.15.88:0/1033535092 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c0adb50 con 0x7f3c440c81a0 2023-10-16T20:56:41.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c4c73e640 1 -- 172.21.15.88:0/1033535092 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c440cd720 con 0x7f3c440c81a0 2023-10-16T20:56:41.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c4c73e640 1 -- 172.21.15.88:0/1033535092 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c44157750 con 0x7f3c440c81a0 2023-10-16T20:56:41.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.418+0000 7f3c3affd640 1 -- 172.21.15.88:0/1033535092 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3c3c0b4070 con 0x7f3c440c81a0 2023-10-16T20:56:41.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.422+0000 7f3c3affd640 1 --2- 172.21.15.88:0/1033535092 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3c280664a0 0x7f3c28068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:41.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.422+0000 7f3c3affd640 1 -- 172.21.15.88:0/1033535092 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7f3c3c11f3a0 con 0x7f3c440c81a0 2023-10-16T20:56:41.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.422+0000 7f3c4a4da640 1 --2- 172.21.15.88:0/1033535092 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3c280664a0 0x7f3c28068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:41.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.422+0000 7f3c4a4da640 1 --2- 172.21.15.88:0/1033535092 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3c280664a0 0x7f3c28068960 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3c40009c80 tx=0x7f3c40008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:41.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.426+0000 7f3c3affd640 1 -- 172.21.15.88:0/1033535092 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3c44157750 con 0x7f3c440c81a0 2023-10-16T20:56:41.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.610+0000 7f3c4c73e640 1 -- 172.21.15.88:0/1033535092 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c44157710 con 0x7f3c440c81a0 2023-10-16T20:56:41.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.610+0000 7f3c3affd640 1 -- 172.21.15.88:0/1033535092 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5781 (secure 0 0 0) 0x7f3c3c0ec160 con 0x7f3c440c81a0 2023-10-16T20:56:41.610 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:41.610 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:40.633696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:41.610 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 103 2023-10-16T20:56:41.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.610+0000 7f3c4c73e640 1 -- 172.21.15.88:0/1033535092 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3c280664a0 msgr2=0x7f3c28068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:41.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.610+0000 7f3c4c73e640 1 --2- 172.21.15.88:0/1033535092 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3c280664a0 0x7f3c28068960 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3c40009c80 tx=0x7f3c40008040 comp rx=0 tx=0).stop 2023-10-16T20:56:41.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.610+0000 7f3c4c73e640 1 -- 172.21.15.88:0/1033535092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c440c81a0 msgr2=0x7f3c44157f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:41.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.610+0000 7f3c4c73e640 1 --2- 172.21.15.88:0/1033535092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c440c81a0 0x7f3c44157f70 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c01cda0 tx=0x7f3c3c019660 comp rx=0 tx=0).stop 2023-10-16T20:56:41.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.610+0000 7f3c4c73e640 1 -- 172.21.15.88:0/1033535092 shutdown_connections 2023-10-16T20:56:41.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.610+0000 7f3c4c73e640 1 --2- 172.21.15.88:0/1033535092 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f3c280664a0 0x7f3c28068960 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.610+0000 7f3c4c73e640 1 --2- 172.21.15.88:0/1033535092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c440c81a0 0x7f3c44157f70 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.610+0000 7f3c4c73e640 1 --2- 172.21.15.88:0/1033535092 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3c440c7dc0 0x7f3c44157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.610+0000 7f3c4c73e640 1 --2- 172.21.15.88:0/1033535092 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c440c76c0 0x7f3c441f54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:41.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.610+0000 7f3c4c73e640 1 -- 172.21.15.88:0/1033535092 >> 172.21.15.88:0/1033535092 conn(0x7f3c44071a60 msgr2=0x7f3c44150320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:41.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.610+0000 7f3c4c73e640 1 -- 172.21.15.88:0/1033535092 shutdown_connections 2023-10-16T20:56:41.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:41.610+0000 7f3c4c73e640 1 -- 172.21.15.88:0/1033535092 wait complete. 2023-10-16T20:56:41.630 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 103, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 103, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5378": { "addr": "172.21.15.191:6835/3119045295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3119045295, "type": "v2" }, { "addr": "172.21.15.191:6835", "nonce": 3119045295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5378, "incarnation": 89, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:active", "state_seq": 32 }, "gid_5477": { "addr": "172.21.15.191:6837/2244284333", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6836", "nonce": 2244284333, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2244284333, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5477, "incarnation": 97, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 39 }, "gid_5535": { "addr": "172.21.15.88:6835/3476086754", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3476086754, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3476086754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5535, "incarnation": 52, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_5570": { "addr": "172.21.15.191:6833/3394064120", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 3394064120, "type": "v2" }, { "addr": "172.21.15.191:6833", "nonce": 3394064120, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5570, "incarnation": 103, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:replay", "state_seq": 1 }, "gid_5709": { "addr": "172.21.15.88:6838/2600891060", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 2600891060, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 2600891060, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5709, "incarnation": 67, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:56:40.633696+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5477, "mds_1": 5709, "mds_2": 5535, "mds_3": 5570, "mds_4": 5378 } } } ], "standbys": [] } 2023-10-16T20:56:43.631 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:43.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.814+0000 7f52556d6640 1 Processor -- start 2023-10-16T20:56:43.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.814+0000 7f52556d6640 1 -- start start 2023-10-16T20:56:43.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.814+0000 7f52556d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5250153310 0x7f5250159c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:43.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.814+0000 7f52556d6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f525006e720 0x7f525006eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:43.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.814+0000 7f52556d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52501536f0 con 0x7f52500cadb0 2023-10-16T20:56:43.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.814+0000 7f52556d6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52500ccd00 con 0x7f525006e720 2023-10-16T20:56:43.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.814+0000 7f52556d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52500cc160 con 0x7f5250153310 2023-10-16T20:56:43.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.818+0000 7f524e7fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f52500cadb0 0x7f52500cc5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33354/0 (socket says 172.21.15.88:33354) 2023-10-16T20:56:43.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.818+0000 7f524e7fc640 1 -- 172.21.15.88:0/1549976810 learned_addr learned my addr 172.21.15.88:0/1549976810 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:56:43.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.818+0000 7f524ffff640 1 --2- 172.21.15.88:0/1549976810 >> v2:172.21.15.191:3300/0 conn(0x7f525006e720 0x7f525006eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:43.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.818+0000 7f524effd640 1 --2- 172.21.15.88:0/1549976810 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5250153310 0x7f5250159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:43.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.818+0000 7f524dffb640 1 -- 172.21.15.88:0/1549976810 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271051421 0 0) 0x7f52501536f0 con 0x7f52500cadb0 2023-10-16T20:56:43.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.818+0000 7f524dffb640 1 -- 172.21.15.88:0/1549976810 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5238003530 con 0x7f52500cadb0 2023-10-16T20:56:43.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.818+0000 7f524effd640 1 -- 172.21.15.88:0/1549976810 >> v2:172.21.15.191:3300/0 conn(0x7f525006e720 msgr2=0x7f525006eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:56:43.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.818+0000 7f524effd640 1 --2- 172.21.15.88:0/1549976810 >> v2:172.21.15.191:3300/0 conn(0x7f525006e720 0x7f525006eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:43.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.818+0000 7f524effd640 1 -- 172.21.15.88:0/1549976810 >> v1:172.21.15.88:6789/0 conn(0x7f52500cadb0 legacy=0x7f52500cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:43.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.818+0000 7f524effd640 1 -- 172.21.15.88:0/1549976810 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525015a1a0 con 0x7f5250153310 2023-10-16T20:56:43.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.818+0000 7f524ffff640 1 --2- 172.21.15.88:0/1549976810 >> v2:172.21.15.191:3300/0 conn(0x7f525006e720 0x7f525006eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:56:43.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.822+0000 7f524effd640 1 --2- 172.21.15.88:0/1549976810 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5250153310 0x7f5250159c60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f52400097c0 tx=0x7f524001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e66ee88b5d1d1e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:43.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.822+0000 7f524dffb640 1 -- 172.21.15.88:0/1549976810 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52400a8970 con 0x7f5250153310 2023-10-16T20:56:43.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.822+0000 7f524dffb640 1 -- 172.21.15.88:0/1549976810 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52400a8ad0 con 0x7f5250153310 2023-10-16T20:56:43.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.822+0000 7f524dffb640 1 -- 172.21.15.88:0/1549976810 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52400a8e60 con 0x7f5250153310 2023-10-16T20:56:43.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.822+0000 7f52556d6640 1 -- 172.21.15.88:0/1549976810 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5250153310 msgr2=0x7f5250159c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:43.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.822+0000 7f52556d6640 1 --2- 172.21.15.88:0/1549976810 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5250153310 0x7f5250159c60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f52400097c0 tx=0x7f524001ec70 comp rx=0 tx=0).stop 2023-10-16T20:56:43.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.822+0000 7f52556d6640 1 -- 172.21.15.88:0/1549976810 shutdown_connections 2023-10-16T20:56:43.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.822+0000 7f52556d6640 1 --2- 172.21.15.88:0/1549976810 >> v2:172.21.15.191:3300/0 conn(0x7f525006e720 0x7f525006eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:43.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.822+0000 7f52556d6640 1 --2- 172.21.15.88:0/1549976810 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f5250153310 0x7f5250159c60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:43.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.822+0000 7f52556d6640 1 -- 172.21.15.88:0/1549976810 >> 172.21.15.88:0/1549976810 conn(0x7f525006cf80 msgr2=0x7f52500d0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:43.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.822+0000 7f52556d6640 1 -- 172.21.15.88:0/1549976810 shutdown_connections 2023-10-16T20:56:43.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.822+0000 7f52556d6640 1 -- 172.21.15.88:0/1549976810 wait complete. 2023-10-16T20:56:43.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.822+0000 7f52556d6640 1 Processor -- start 2023-10-16T20:56:43.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.822+0000 7f52556d6640 1 -- start start 2023-10-16T20:56:43.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f52556d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f525006e720 0x7f525015a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:43.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f52556d6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52500cadb0 0x7f525014f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:43.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f52556d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5250153310 0x7f5250149cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:43.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f52556d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52500c8a20 con 0x7f5250153310 2023-10-16T20:56:43.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f52556d6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52501ef700 con 0x7f52500cadb0 2023-10-16T20:56:43.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f52556d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52501eeb80 con 0x7f525006e720 2023-10-16T20:56:43.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f524ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5250153310 0x7f5250149cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:43.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f524effd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f525006e720 0x7f525015a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:43.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f524e7fc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52500cadb0 0x7f525014f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:43.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f524effd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f525006e720 0x7f525015a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:46012/0 (socket says 172.21.15.88:46012) 2023-10-16T20:56:43.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f524effd640 1 -- 172.21.15.88:0/1391706486 learned_addr learned my addr 172.21.15.88:0/1391706486 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:43.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f524ffff640 1 -- 172.21.15.88:0/1391706486 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f525006e720 msgr2=0x7f525015a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:43.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f524ffff640 1 --2- 172.21.15.88:0/1391706486 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f525006e720 0x7f525015a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:43.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f524ffff640 1 -- 172.21.15.88:0/1391706486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52500cadb0 msgr2=0x7f525014f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:43.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f524ffff640 1 --2- 172.21.15.88:0/1391706486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52500cadb0 0x7f525014f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:43.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f524ffff640 1 -- 172.21.15.88:0/1391706486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5248009a70 con 0x7f5250153310 2023-10-16T20:56:43.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f524e7fc640 1 --2- 172.21.15.88:0/1391706486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52500cadb0 0x7f525014f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:56:43.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f524ffff640 1 --2- 172.21.15.88:0/1391706486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5250153310 0x7f5250149cd0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f524800efc0 tx=0x7f524800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:43.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f52377fe640 1 -- 172.21.15.88:0/1391706486 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524800cd30 con 0x7f5250153310 2023-10-16T20:56:43.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f52377fe640 1 -- 172.21.15.88:0/1391706486 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f524800f040 con 0x7f5250153310 2023-10-16T20:56:43.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f52556d6640 1 -- 172.21.15.88:0/1391706486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5240009420 con 0x7f5250153310 2023-10-16T20:56:43.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f52377fe640 1 -- 172.21.15.88:0/1391706486 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5248003be0 con 0x7f5250153310 2023-10-16T20:56:43.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f52556d6640 1 -- 172.21.15.88:0/1391706486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52501ef140 con 0x7f5250153310 2023-10-16T20:56:43.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f52556d6640 1 -- 172.21.15.88:0/1391706486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52500c8b60 con 0x7f5250153310 2023-10-16T20:56:43.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.826+0000 7f52377fe640 1 -- 172.21.15.88:0/1391706486 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5248014480 con 0x7f5250153310 2023-10-16T20:56:43.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.830+0000 7f52377fe640 1 --2- 172.21.15.88:0/1391706486 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f523c0664a0 0x7f523c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:43.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.830+0000 7f52377fe640 1 -- 172.21.15.88:0/1391706486 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7f5248084ff0 con 0x7f5250153310 2023-10-16T20:56:43.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.830+0000 7f524effd640 1 --2- 172.21.15.88:0/1391706486 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f523c0664a0 0x7f523c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:43.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.830+0000 7f524effd640 1 --2- 172.21.15.88:0/1391706486 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f523c0664a0 0x7f523c068960 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f52400097c0 tx=0x7f5240008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:43.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:43.834+0000 7f52377fe640 1 -- 172.21.15.88:0/1391706486 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f52480b6090 con 0x7f5250153310 2023-10-16T20:56:44.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:44.030+0000 7f52556d6640 1 -- 172.21.15.88:0/1391706486 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f525006f0a0 con 0x7f5250153310 2023-10-16T20:56:44.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:44.030+0000 7f52377fe640 1 -- 172.21.15.88:0/1391706486 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5781 (secure 0 0 0) 0x7f5248051de0 con 0x7f5250153310 2023-10-16T20:56:44.031 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:44.031 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:40.633696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:44.031 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 103 2023-10-16T20:56:44.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:44.030+0000 7f52556d6640 1 -- 172.21.15.88:0/1391706486 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f523c0664a0 msgr2=0x7f523c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:44.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:44.030+0000 7f52556d6640 1 --2- 172.21.15.88:0/1391706486 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f523c0664a0 0x7f523c068960 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f52400097c0 tx=0x7f5240008040 comp rx=0 tx=0).stop 2023-10-16T20:56:44.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:44.030+0000 7f52556d6640 1 -- 172.21.15.88:0/1391706486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5250153310 msgr2=0x7f5250149cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:44.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:44.030+0000 7f52556d6640 1 --2- 172.21.15.88:0/1391706486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5250153310 0x7f5250149cd0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f524800efc0 tx=0x7f524800c490 comp rx=0 tx=0).stop 2023-10-16T20:56:44.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:44.030+0000 7f52556d6640 1 -- 172.21.15.88:0/1391706486 shutdown_connections 2023-10-16T20:56:44.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:44.030+0000 7f52556d6640 1 --2- 172.21.15.88:0/1391706486 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f523c0664a0 0x7f523c068960 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:44.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:44.030+0000 7f52556d6640 1 --2- 172.21.15.88:0/1391706486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5250153310 0x7f5250149cd0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:44.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:44.030+0000 7f52556d6640 1 --2- 172.21.15.88:0/1391706486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52500cadb0 0x7f525014f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:44.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:44.030+0000 7f52556d6640 1 --2- 172.21.15.88:0/1391706486 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f525006e720 0x7f525015a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:44.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:44.030+0000 7f52556d6640 1 -- 172.21.15.88:0/1391706486 >> 172.21.15.88:0/1391706486 conn(0x7f525006cf80 msgr2=0x7f5250071c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:44.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:44.030+0000 7f52556d6640 1 -- 172.21.15.88:0/1391706486 shutdown_connections 2023-10-16T20:56:44.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:44.030+0000 7f52556d6640 1 -- 172.21.15.88:0/1391706486 wait complete. 2023-10-16T20:56:44.051 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:56:46.052 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:46.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.210+0000 7f6ec06d6640 1 Processor -- start 2023-10-16T20:56:46.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.210+0000 7f6ec06d6640 1 -- start start 2023-10-16T20:56:46.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.210+0000 7f6ec06d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6eb8153770 0x7f6eb8159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:46.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.214+0000 7f6ec06d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb815a410 con 0x7f6eb80cece0 2023-10-16T20:56:46.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.214+0000 7f6ec06d6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb8153b50 con 0x7f6eb80c9bb0 2023-10-16T20:56:46.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.214+0000 7f6ec06d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eb80cc250 con 0x7f6eb8153770 2023-10-16T20:56:46.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.214+0000 7f6ebe472640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6eb8153770 0x7f6eb8159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:46.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.214+0000 7f6ebe472640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6eb8153770 0x7f6eb8159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33048/0 (socket says 172.21.15.88:33048) 2023-10-16T20:56:46.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.214+0000 7f6ebe472640 1 -- 172.21.15.88:0/4007537521 learned_addr learned my addr 172.21.15.88:0/4007537521 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:46.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.214+0000 7f6ebe472640 1 -- 172.21.15.88:0/4007537521 >> v1:172.21.15.191:6789/0 conn(0x7f6eb80c9bb0 legacy=0x7f6eb80c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:46.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.214+0000 7f6ebe472640 1 -- 172.21.15.88:0/4007537521 >> v1:172.21.15.88:6789/0 conn(0x7f6eb80cece0 legacy=0x7f6eb8163dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:46.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.214+0000 7f6ebe472640 1 -- 172.21.15.88:0/4007537521 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb80cb2f0 con 0x7f6eb8153770 2023-10-16T20:56:46.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.218+0000 7f6ebe472640 1 --2- 172.21.15.88:0/4007537521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eb8153770 0x7f6eb8159ed0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4006fd0 tx=0x7f6eb401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5976d7092c65679d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:46.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.218+0000 7f6ebd470640 1 -- 172.21.15.88:0/4007537521 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb40a7070 con 0x7f6eb8153770 2023-10-16T20:56:46.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.218+0000 7f6ebd470640 1 -- 172.21.15.88:0/4007537521 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb40ab430 con 0x7f6eb8153770 2023-10-16T20:56:46.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.218+0000 7f6ebd470640 1 -- 172.21.15.88:0/4007537521 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb40ab7c0 con 0x7f6eb8153770 2023-10-16T20:56:46.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.218+0000 7f6ec06d6640 1 -- 172.21.15.88:0/4007537521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eb8153770 msgr2=0x7f6eb8159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:46.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.218+0000 7f6ec06d6640 1 --2- 172.21.15.88:0/4007537521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eb8153770 0x7f6eb8159ed0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4006fd0 tx=0x7f6eb401c520 comp rx=0 tx=0).stop 2023-10-16T20:56:46.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.218+0000 7f6ec06d6640 1 -- 172.21.15.88:0/4007537521 shutdown_connections 2023-10-16T20:56:46.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.218+0000 7f6ec06d6640 1 --2- 172.21.15.88:0/4007537521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eb8153770 0x7f6eb8159ed0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:46.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.218+0000 7f6ec06d6640 1 -- 172.21.15.88:0/4007537521 >> 172.21.15.88:0/4007537521 conn(0x7f6eb80715c0 msgr2=0x7f6eb80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:46.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.218+0000 7f6ec06d6640 1 -- 172.21.15.88:0/4007537521 shutdown_connections 2023-10-16T20:56:46.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.218+0000 7f6ec06d6640 1 -- 172.21.15.88:0/4007537521 wait complete. 2023-10-16T20:56:46.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.218+0000 7f6ec06d6640 1 Processor -- start 2023-10-16T20:56:46.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.218+0000 7f6ec06d6640 1 -- start start 2023-10-16T20:56:46.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ec06d6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6eb80c9bb0 0x7f6eb80b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:46.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ec06d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6eb80cece0 0x7f6eb81f53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:46.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ec06d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eb8153770 0x7f6eb81f5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:46.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ec06d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb81f4fc0 con 0x7f6eb8153770 2023-10-16T20:56:46.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ec06d6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb815a610 con 0x7f6eb80c9bb0 2023-10-16T20:56:46.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ec06d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb815c620 con 0x7f6eb80cece0 2023-10-16T20:56:46.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ebec73640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eb8153770 0x7f6eb81f5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:46.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ebec73640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eb8153770 0x7f6eb81f5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33052/0 (socket says 172.21.15.88:33052) 2023-10-16T20:56:46.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ebec73640 1 -- 172.21.15.88:0/1345710308 learned_addr learned my addr 172.21.15.88:0/1345710308 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:46.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ebe472640 1 --2- 172.21.15.88:0/1345710308 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6eb80c9bb0 0x7f6eb80b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:46.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ebdc71640 1 --2- 172.21.15.88:0/1345710308 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6eb80cece0 0x7f6eb81f53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:46.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ebec73640 1 -- 172.21.15.88:0/1345710308 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6eb80cece0 msgr2=0x7f6eb81f53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:46.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ebec73640 1 --2- 172.21.15.88:0/1345710308 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6eb80cece0 0x7f6eb81f53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:46.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ebec73640 1 -- 172.21.15.88:0/1345710308 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6eb80c9bb0 msgr2=0x7f6eb80b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:46.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ebec73640 1 --2- 172.21.15.88:0/1345710308 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6eb80c9bb0 0x7f6eb80b3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:46.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ebec73640 1 -- 172.21.15.88:0/1345710308 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb4009480 con 0x7f6eb8153770 2023-10-16T20:56:46.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ebec73640 1 --2- 172.21.15.88:0/1345710308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eb8153770 0x7f6eb81f5920 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f6eb000c770 tx=0x7f6eb000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:46.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6eaeffd640 1 -- 172.21.15.88:0/1345710308 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0003b80 con 0x7f6eb8153770 2023-10-16T20:56:46.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6eaeffd640 1 -- 172.21.15.88:0/1345710308 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb0003ce0 con 0x7f6eb8153770 2023-10-16T20:56:46.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ec06d6640 1 -- 172.21.15.88:0/1345710308 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb815a890 con 0x7f6eb8153770 2023-10-16T20:56:46.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6eaeffd640 1 -- 172.21.15.88:0/1345710308 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb000aa90 con 0x7f6eb8153770 2023-10-16T20:56:46.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ec06d6640 1 -- 172.21.15.88:0/1345710308 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb815ae50 con 0x7f6eb8153770 2023-10-16T20:56:46.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.222+0000 7f6ec06d6640 1 -- 172.21.15.88:0/1345710308 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb81f5100 con 0x7f6eb8153770 2023-10-16T20:56:46.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.226+0000 7f6eaeffd640 1 -- 172.21.15.88:0/1345710308 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6eb000abf0 con 0x7f6eb8153770 2023-10-16T20:56:46.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.226+0000 7f6eaeffd640 1 --2- 172.21.15.88:0/1345710308 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6e9c0664a0 0x7f6e9c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:46.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.226+0000 7f6eaeffd640 1 -- 172.21.15.88:0/1345710308 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7f6eb0084440 con 0x7f6eb8153770 2023-10-16T20:56:46.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.226+0000 7f6ebe472640 1 --2- 172.21.15.88:0/1345710308 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6e9c0664a0 0x7f6e9c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:46.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.226+0000 7f6ebe472640 1 --2- 172.21.15.88:0/1345710308 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6e9c0664a0 0x7f6e9c068960 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6eb40af000 tx=0x7f6eb4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:46.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.230+0000 7f6eaeffd640 1 -- 172.21.15.88:0/1345710308 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6eb81f5100 con 0x7f6eb8153770 2023-10-16T20:56:46.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.486+0000 7f6ec06d6640 1 -- 172.21.15.88:0/1345710308 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6eb81f5100 con 0x7f6eb8153770 2023-10-16T20:56:46.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.486+0000 7f6eaeffd640 1 -- 172.21.15.88:0/1345710308 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5781 (secure 0 0 0) 0x7f6eb81f5100 con 0x7f6eb8153770 2023-10-16T20:56:46.486 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:46.487 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:40.633696+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:replay","state_seq":1,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:46.487 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 103 2023-10-16T20:56:46.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.486+0000 7f6ec06d6640 1 -- 172.21.15.88:0/1345710308 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6e9c0664a0 msgr2=0x7f6e9c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:46.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.486+0000 7f6ec06d6640 1 --2- 172.21.15.88:0/1345710308 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6e9c0664a0 0x7f6e9c068960 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6eb40af000 tx=0x7f6eb4009040 comp rx=0 tx=0).stop 2023-10-16T20:56:46.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.486+0000 7f6ec06d6640 1 -- 172.21.15.88:0/1345710308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eb8153770 msgr2=0x7f6eb81f5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:46.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.486+0000 7f6ec06d6640 1 --2- 172.21.15.88:0/1345710308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eb8153770 0x7f6eb81f5920 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f6eb000c770 tx=0x7f6eb000cc30 comp rx=0 tx=0).stop 2023-10-16T20:56:46.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.486+0000 7f6ec06d6640 1 -- 172.21.15.88:0/1345710308 shutdown_connections 2023-10-16T20:56:46.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.486+0000 7f6ec06d6640 1 --2- 172.21.15.88:0/1345710308 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6e9c0664a0 0x7f6e9c068960 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:46.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.486+0000 7f6ec06d6640 1 --2- 172.21.15.88:0/1345710308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eb8153770 0x7f6eb81f5920 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:46.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.486+0000 7f6ec06d6640 1 --2- 172.21.15.88:0/1345710308 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6eb80cece0 0x7f6eb81f53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:46.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.486+0000 7f6ec06d6640 1 --2- 172.21.15.88:0/1345710308 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6eb80c9bb0 0x7f6eb80b3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:46.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.486+0000 7f6ec06d6640 1 -- 172.21.15.88:0/1345710308 >> 172.21.15.88:0/1345710308 conn(0x7f6eb80715c0 msgr2=0x7f6eb81526d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:46.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.486+0000 7f6ec06d6640 1 -- 172.21.15.88:0/1345710308 shutdown_connections 2023-10-16T20:56:46.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:46.486+0000 7f6ec06d6640 1 -- 172.21.15.88:0/1345710308 wait complete. 2023-10-16T20:56:46.509 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:56:48.510 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:48.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.670+0000 7f64a58d6640 1 Processor -- start 2023-10-16T20:56:48.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.670+0000 7f64a58d6640 1 -- start start 2023-10-16T20:56:48.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.674+0000 7f64a58d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a00cd170 0x7f64a0072000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:48.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.674+0000 7f64a58d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a015a190 con 0x7f64a00ccd90 2023-10-16T20:56:48.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.674+0000 7f64a58d6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a0072620 con 0x7f64a00c7560 2023-10-16T20:56:48.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.674+0000 7f64a58d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a0159860 con 0x7f64a00cd170 2023-10-16T20:56:48.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.674+0000 7f649effd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f64a00ccd90 0x7f64a006b280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33372/0 (socket says 172.21.15.88:33372) 2023-10-16T20:56:48.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.674+0000 7f649effd640 1 -- 172.21.15.88:0/819386037 learned_addr learned my addr 172.21.15.88:0/819386037 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:56:48.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.674+0000 7f649e7fc640 1 --2- 172.21.15.88:0/819386037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a00cd170 0x7f64a0072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:48.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.674+0000 7f649dffb640 1 -- 172.21.15.88:0/819386037 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689483847 0 0) 0x7f64a015a190 con 0x7f64a00ccd90 2023-10-16T20:56:48.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.674+0000 7f649dffb640 1 -- 172.21.15.88:0/819386037 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6488003530 con 0x7f64a00ccd90 2023-10-16T20:56:48.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.674+0000 7f649e7fc640 1 -- 172.21.15.88:0/819386037 >> v1:172.21.15.191:6789/0 conn(0x7f64a00c7560 legacy=0x7f64a01423d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:48.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.674+0000 7f649dffb640 1 -- 172.21.15.88:0/819386037 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882385773 0 0) 0x7f64a0072620 con 0x7f64a00c7560 2023-10-16T20:56:48.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.674+0000 7f649e7fc640 1 -- 172.21.15.88:0/819386037 >> v1:172.21.15.88:6789/0 conn(0x7f64a00ccd90 legacy=0x7f64a006b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:48.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.674+0000 7f649e7fc640 1 -- 172.21.15.88:0/819386037 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a0159b40 con 0x7f64a00cd170 2023-10-16T20:56:48.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.678+0000 7f649e7fc640 1 --2- 172.21.15.88:0/819386037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a00cd170 0x7f64a0072000 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f64940097f0 tx=0x7f649401ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4007db8c87dcee6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:48.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.678+0000 7f649dffb640 1 -- 172.21.15.88:0/819386037 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64940a89b0 con 0x7f64a00cd170 2023-10-16T20:56:48.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.678+0000 7f649dffb640 1 -- 172.21.15.88:0/819386037 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64940a8b10 con 0x7f64a00cd170 2023-10-16T20:56:48.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.678+0000 7f649dffb640 1 -- 172.21.15.88:0/819386037 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64940a8ea0 con 0x7f64a00cd170 2023-10-16T20:56:48.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.678+0000 7f64a58d6640 1 -- 172.21.15.88:0/819386037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a00cd170 msgr2=0x7f64a0072000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:48.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.678+0000 7f64a58d6640 1 --2- 172.21.15.88:0/819386037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a00cd170 0x7f64a0072000 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f64940097f0 tx=0x7f649401ecb0 comp rx=0 tx=0).stop 2023-10-16T20:56:48.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.678+0000 7f64a58d6640 1 -- 172.21.15.88:0/819386037 shutdown_connections 2023-10-16T20:56:48.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.678+0000 7f64a58d6640 1 --2- 172.21.15.88:0/819386037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a00cd170 0x7f64a0072000 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:48.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.678+0000 7f64a58d6640 1 -- 172.21.15.88:0/819386037 >> 172.21.15.88:0/819386037 conn(0x7f64a006e040 msgr2=0x7f64a0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:48.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.678+0000 7f64a58d6640 1 -- 172.21.15.88:0/819386037 shutdown_connections 2023-10-16T20:56:48.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.678+0000 7f64a58d6640 1 -- 172.21.15.88:0/819386037 wait complete. 2023-10-16T20:56:48.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f64a58d6640 1 Processor -- start 2023-10-16T20:56:48.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f64a58d6640 1 -- start start 2023-10-16T20:56:48.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f64a58d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a00c7560 0x7f64a0158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:48.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f64a58d6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64a00ccd90 0x7f64a013cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:48.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f64a58d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a00cd170 0x7f64a013f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:48.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f64a58d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a01588c0 con 0x7f64a00cd170 2023-10-16T20:56:48.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f64a58d6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a00c5cf0 con 0x7f64a00ccd90 2023-10-16T20:56:48.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f64a58d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a01564b0 con 0x7f64a00c7560 2023-10-16T20:56:48.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f649effd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a00c7560 0x7f64a0158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:48.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f649effd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a00c7560 0x7f64a0158d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:46028/0 (socket says 172.21.15.88:46028) 2023-10-16T20:56:48.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f649effd640 1 -- 172.21.15.88:0/2052703362 learned_addr learned my addr 172.21.15.88:0/2052703362 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:48.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f649e7fc640 1 --2- 172.21.15.88:0/2052703362 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64a00ccd90 0x7f64a013cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:48.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f649f7fe640 1 --2- 172.21.15.88:0/2052703362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a00cd170 0x7f64a013f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:48.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f649effd640 1 -- 172.21.15.88:0/2052703362 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64a00ccd90 msgr2=0x7f64a013cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:48.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f649effd640 1 --2- 172.21.15.88:0/2052703362 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64a00ccd90 0x7f64a013cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:48.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f649effd640 1 -- 172.21.15.88:0/2052703362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a00cd170 msgr2=0x7f64a013f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:48.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f649effd640 1 --2- 172.21.15.88:0/2052703362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a00cd170 0x7f64a013f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:48.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f649effd640 1 -- 172.21.15.88:0/2052703362 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6494009420 con 0x7f64a00c7560 2023-10-16T20:56:48.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f649effd640 1 --2- 172.21.15.88:0/2052703362 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a00c7560 0x7f64a0158d10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f649000c480 tx=0x7f649000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:48.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f6487fff640 1 -- 172.21.15.88:0/2052703362 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6490005cf0 con 0x7f64a00c7560 2023-10-16T20:56:48.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f6487fff640 1 -- 172.21.15.88:0/2052703362 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6490005e50 con 0x7f64a00c7560 2023-10-16T20:56:48.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f6487fff640 1 -- 172.21.15.88:0/2052703362 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6490007290 con 0x7f64a00c7560 2023-10-16T20:56:48.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f64a58d6640 1 -- 172.21.15.88:0/2052703362 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a00c6b00 con 0x7f64a00c7560 2023-10-16T20:56:48.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.682+0000 7f64a58d6640 1 -- 172.21.15.88:0/2052703362 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a0156720 con 0x7f64a00c7560 2023-10-16T20:56:48.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.686+0000 7f64a58d6640 1 -- 172.21.15.88:0/2052703362 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a0074700 con 0x7f64a00c7560 2023-10-16T20:56:48.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.686+0000 7f6487fff640 1 -- 172.21.15.88:0/2052703362 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f64900073f0 con 0x7f64a00c7560 2023-10-16T20:56:48.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.686+0000 7f6487fff640 1 --2- 172.21.15.88:0/2052703362 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f648c0664f0 0x7f648c0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:48.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.686+0000 7f6487fff640 1 -- 172.21.15.88:0/2052703362 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7f6490083b30 con 0x7f64a00c7560 2023-10-16T20:56:48.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.686+0000 7f649e7fc640 1 --2- 172.21.15.88:0/2052703362 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f648c0664f0 0x7f648c0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:48.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.686+0000 7f649e7fc640 1 --2- 172.21.15.88:0/2052703362 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f648c0664f0 0x7f648c0689b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f64940b2000 tx=0x7f6494008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:48.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.690+0000 7f6487fff640 1 -- 172.21.15.88:0/2052703362 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6490050980 con 0x7f64a00c7560 2023-10-16T20:56:48.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.922+0000 7f64a58d6640 1 -- 172.21.15.88:0/2052703362 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64a0071b10 con 0x7f64a00c7560 2023-10-16T20:56:48.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.922+0000 7f6487fff640 1 -- 172.21.15.88:0/2052703362 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5782 (secure 0 0 0) 0x7f6490017030 con 0x7f64a00c7560 2023-10-16T20:56:48.923 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:48.923 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:45.881755+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:resolve","state_seq":7,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:48.923 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 104 2023-10-16T20:56:48.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.926+0000 7f64a58d6640 1 -- 172.21.15.88:0/2052703362 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f648c0664f0 msgr2=0x7f648c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:48.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.926+0000 7f64a58d6640 1 --2- 172.21.15.88:0/2052703362 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f648c0664f0 0x7f648c0689b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f64940b2000 tx=0x7f6494008040 comp rx=0 tx=0).stop 2023-10-16T20:56:48.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.926+0000 7f64a58d6640 1 -- 172.21.15.88:0/2052703362 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a00c7560 msgr2=0x7f64a0158d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:48.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.926+0000 7f64a58d6640 1 --2- 172.21.15.88:0/2052703362 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a00c7560 0x7f64a0158d10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f649000c480 tx=0x7f649000c940 comp rx=0 tx=0).stop 2023-10-16T20:56:48.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.926+0000 7f64a58d6640 1 -- 172.21.15.88:0/2052703362 shutdown_connections 2023-10-16T20:56:48.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.926+0000 7f64a58d6640 1 --2- 172.21.15.88:0/2052703362 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f648c0664f0 0x7f648c0689b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:48.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.926+0000 7f64a58d6640 1 --2- 172.21.15.88:0/2052703362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a00cd170 0x7f64a013f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:48.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.926+0000 7f64a58d6640 1 --2- 172.21.15.88:0/2052703362 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64a00ccd90 0x7f64a013cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:48.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.926+0000 7f64a58d6640 1 --2- 172.21.15.88:0/2052703362 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a00c7560 0x7f64a0158d10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:48.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.926+0000 7f64a58d6640 1 -- 172.21.15.88:0/2052703362 >> 172.21.15.88:0/2052703362 conn(0x7f64a006e040 msgr2=0x7f64a0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:48.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.926+0000 7f64a58d6640 1 -- 172.21.15.88:0/2052703362 shutdown_connections 2023-10-16T20:56:48.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:48.926+0000 7f64a58d6640 1 -- 172.21.15.88:0/2052703362 wait complete. 2023-10-16T20:56:48.952 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:56:50.953 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:51.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.126+0000 7fe9c96dc640 1 Processor -- start 2023-10-16T20:56:51.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.126+0000 7fe9c96dc640 1 -- start start 2023-10-16T20:56:51.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.130+0000 7fe9c96dc640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe9c40ca2f0 0x7fe9c40c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:51.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.130+0000 7fe9c96dc640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c40cae10 con 0x7fe9c41537c0 2023-10-16T20:56:51.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.130+0000 7fe9c96dc640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c40cac30 con 0x7fe9c40c51c0 2023-10-16T20:56:51.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.130+0000 7fe9c96dc640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c40c55d0 con 0x7fe9c40ca2f0 2023-10-16T20:56:51.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.130+0000 7fe9c37fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe9c40ca2f0 0x7fe9c40c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:51.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.130+0000 7fe9c3fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe9c41537c0 0x7fe9c415c870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59900/0 (socket says 172.21.15.88:59900) 2023-10-16T20:56:51.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.130+0000 7fe9c3fff640 1 -- 172.21.15.88:0/3493608163 learned_addr learned my addr 172.21.15.88:0/3493608163 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:56:51.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.130+0000 7fe9c37fe640 1 -- 172.21.15.88:0/3493608163 >> v1:172.21.15.191:6789/0 conn(0x7fe9c40c51c0 legacy=0x7fe9c40cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:51.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.130+0000 7fe9c37fe640 1 -- 172.21.15.88:0/3493608163 >> v1:172.21.15.88:6789/0 conn(0x7fe9c41537c0 legacy=0x7fe9c415c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:51.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.130+0000 7fe9c2ffd640 1 -- 172.21.15.88:0/3493608163 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217622580 0 0) 0x7fe9c40cae10 con 0x7fe9c41537c0 2023-10-16T20:56:51.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.130+0000 7fe9c37fe640 1 -- 172.21.15.88:0/3493608163 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c415d2b0 con 0x7fe9c40ca2f0 2023-10-16T20:56:51.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.130+0000 7fe9c37fe640 1 --2- 172.21.15.88:0/3493608163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c40ca2f0 0x7fe9c40c4c80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe9b80097c0 tx=0x7fe9b801c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6d2dbd71f754975 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:51.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c2ffd640 1 -- 172.21.15.88:0/3493608163 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b80ab650 con 0x7fe9c40ca2f0 2023-10-16T20:56:51.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c2ffd640 1 -- 172.21.15.88:0/3493608163 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b80ab7b0 con 0x7fe9c40ca2f0 2023-10-16T20:56:51.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c2ffd640 1 -- 172.21.15.88:0/3493608163 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b80abb40 con 0x7fe9c40ca2f0 2023-10-16T20:56:51.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c96dc640 1 -- 172.21.15.88:0/3493608163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c40ca2f0 msgr2=0x7fe9c40c4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:51.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c96dc640 1 --2- 172.21.15.88:0/3493608163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c40ca2f0 0x7fe9c40c4c80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe9b80097c0 tx=0x7fe9b801c450 comp rx=0 tx=0).stop 2023-10-16T20:56:51.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c96dc640 1 -- 172.21.15.88:0/3493608163 shutdown_connections 2023-10-16T20:56:51.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c96dc640 1 --2- 172.21.15.88:0/3493608163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c40ca2f0 0x7fe9c40c4c80 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:51.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c96dc640 1 -- 172.21.15.88:0/3493608163 >> 172.21.15.88:0/3493608163 conn(0x7fe9c406c170 msgr2=0x7fe9c40715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:51.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c96dc640 1 -- 172.21.15.88:0/3493608163 shutdown_connections 2023-10-16T20:56:51.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c96dc640 1 -- 172.21.15.88:0/3493608163 wait complete. 2023-10-16T20:56:51.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c96dc640 1 Processor -- start 2023-10-16T20:56:51.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c96dc640 1 -- start start 2023-10-16T20:56:51.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c96dc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9c40c51c0 0x7fe9c40cb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:51.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c96dc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9c40ca2f0 0x7fe9c4148f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:51.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c96dc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c41537c0 0x7fe9c4147a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:51.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c96dc640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c4146510 con 0x7fe9c40c51c0 2023-10-16T20:56:51.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c96dc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c4149f00 con 0x7fe9c41537c0 2023-10-16T20:56:51.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.134+0000 7fe9c96dc640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c4147570 con 0x7fe9c40ca2f0 2023-10-16T20:56:51.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9c40c51c0 0x7fe9c40cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:51.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9c40c51c0 0x7fe9c40cb050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59472/0 (socket says 172.21.15.88:59472) 2023-10-16T20:56:51.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c3fff640 1 -- 172.21.15.88:0/2347588824 learned_addr learned my addr 172.21.15.88:0/2347588824 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:51.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c37fe640 1 --2- 172.21.15.88:0/2347588824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9c40ca2f0 0x7fe9c4148f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:51.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c8edb640 1 --2- 172.21.15.88:0/2347588824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c41537c0 0x7fe9c4147a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:51.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c3fff640 1 -- 172.21.15.88:0/2347588824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9c40ca2f0 msgr2=0x7fe9c4148f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:51.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c3fff640 1 --2- 172.21.15.88:0/2347588824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9c40ca2f0 0x7fe9c4148f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:51.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c3fff640 1 -- 172.21.15.88:0/2347588824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c41537c0 msgr2=0x7fe9c4147a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:51.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c3fff640 1 --2- 172.21.15.88:0/2347588824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c41537c0 0x7fe9c4147a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:51.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c3fff640 1 -- 172.21.15.88:0/2347588824 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b8009420 con 0x7fe9c40c51c0 2023-10-16T20:56:51.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c3fff640 1 --2- 172.21.15.88:0/2347588824 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9c40c51c0 0x7fe9c40cb050 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe9b400c440 tx=0x7fe9b400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:51.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c0ff9640 1 -- 172.21.15.88:0/2347588824 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b40074e0 con 0x7fe9c40c51c0 2023-10-16T20:56:51.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c0ff9640 1 -- 172.21.15.88:0/2347588824 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b4007640 con 0x7fe9c40c51c0 2023-10-16T20:56:51.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c96dc640 1 -- 172.21.15.88:0/2347588824 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c4149940 con 0x7fe9c40c51c0 2023-10-16T20:56:51.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c0ff9640 1 -- 172.21.15.88:0/2347588824 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4004a10 con 0x7fe9c40c51c0 2023-10-16T20:56:51.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c96dc640 1 -- 172.21.15.88:0/2347588824 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c4146f20 con 0x7fe9c40c51c0 2023-10-16T20:56:51.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c96dc640 1 -- 172.21.15.88:0/2347588824 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c415b1a0 con 0x7fe9c40c51c0 2023-10-16T20:56:51.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.138+0000 7fe9c0ff9640 1 -- 172.21.15.88:0/2347588824 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe9b4004b70 con 0x7fe9c40c51c0 2023-10-16T20:56:51.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.142+0000 7fe9c0ff9640 1 --2- 172.21.15.88:0/2347588824 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe9b00664a0 0x7fe9b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:51.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.142+0000 7fe9c0ff9640 1 -- 172.21.15.88:0/2347588824 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7fe9b4084140 con 0x7fe9c40c51c0 2023-10-16T20:56:51.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.142+0000 7fe9c37fe640 1 --2- 172.21.15.88:0/2347588824 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe9b00664a0 0x7fe9b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:51.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.142+0000 7fe9c37fe640 1 --2- 172.21.15.88:0/2347588824 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe9b00664a0 0x7fe9b0068960 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe9b80af000 tx=0x7fe9b80a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:51.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.146+0000 7fe9c0ff9640 1 -- 172.21.15.88:0/2347588824 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe9b4051020 con 0x7fe9c40c51c0 2023-10-16T20:56:51.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.330+0000 7fe9c96dc640 1 -- 172.21.15.88:0/2347588824 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9c40c4450 con 0x7fe9c40c51c0 2023-10-16T20:56:51.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.330+0000 7fe9c0ff9640 1 -- 172.21.15.88:0/2347588824 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5782 (secure 0 0 0) 0x7fe9b4054640 con 0x7fe9c40c51c0 2023-10-16T20:56:51.331 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:51.331 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:45.881755+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:resolve","state_seq":7,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:51.332 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 104 2023-10-16T20:56:51.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.330+0000 7fe9c96dc640 1 -- 172.21.15.88:0/2347588824 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe9b00664a0 msgr2=0x7fe9b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:51.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.330+0000 7fe9c96dc640 1 --2- 172.21.15.88:0/2347588824 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe9b00664a0 0x7fe9b0068960 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe9b80af000 tx=0x7fe9b80a7000 comp rx=0 tx=0).stop 2023-10-16T20:56:51.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.330+0000 7fe9c96dc640 1 -- 172.21.15.88:0/2347588824 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9c40c51c0 msgr2=0x7fe9c40cb050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:51.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.330+0000 7fe9c96dc640 1 --2- 172.21.15.88:0/2347588824 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9c40c51c0 0x7fe9c40cb050 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe9b400c440 tx=0x7fe9b400c900 comp rx=0 tx=0).stop 2023-10-16T20:56:51.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.334+0000 7fe9c96dc640 1 -- 172.21.15.88:0/2347588824 shutdown_connections 2023-10-16T20:56:51.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.334+0000 7fe9c96dc640 1 --2- 172.21.15.88:0/2347588824 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe9b00664a0 0x7fe9b0068960 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:51.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.334+0000 7fe9c96dc640 1 --2- 172.21.15.88:0/2347588824 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9c41537c0 0x7fe9c4147a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:51.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.334+0000 7fe9c96dc640 1 --2- 172.21.15.88:0/2347588824 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe9c40ca2f0 0x7fe9c4148f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:51.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.334+0000 7fe9c96dc640 1 --2- 172.21.15.88:0/2347588824 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9c40c51c0 0x7fe9c40cb050 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:51.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.334+0000 7fe9c96dc640 1 -- 172.21.15.88:0/2347588824 >> 172.21.15.88:0/2347588824 conn(0x7fe9c406c170 msgr2=0x7fe9c415bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:51.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.334+0000 7fe9c96dc640 1 -- 172.21.15.88:0/2347588824 shutdown_connections 2023-10-16T20:56:51.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:51.334+0000 7fe9c96dc640 1 -- 172.21.15.88:0/2347588824 wait complete. 2023-10-16T20:56:51.353 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:56:53.354 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:53.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.518+0000 7f9e096c9640 1 Processor -- start 2023-10-16T20:56:53.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.518+0000 7f9e096c9640 1 -- start start 2023-10-16T20:56:53.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.522+0000 7f9e096c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e040c9a50 0x7f9e04071b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:53.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.522+0000 7f9e096c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e04159ec0 con 0x7f9e041534f0 2023-10-16T20:56:53.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.522+0000 7f9e096c9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0415a160 con 0x7f9e04072070 2023-10-16T20:56:53.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.522+0000 7f9e096c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0415d270 con 0x7f9e040c9a50 2023-10-16T20:56:53.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.522+0000 7f9e02ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9e041534f0 0x7f9e040c3ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59910/0 (socket says 172.21.15.88:59910) 2023-10-16T20:56:53.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.522+0000 7f9e02ffd640 1 -- 172.21.15.88:0/928167025 learned_addr learned my addr 172.21.15.88:0/928167025 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:56:53.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.522+0000 7f9e027fc640 1 --2- 172.21.15.88:0/928167025 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e040c9a50 0x7f9e04071b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:53.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.522+0000 7f9e01ffb640 1 -- 172.21.15.88:0/928167025 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171182618 0 0) 0x7f9e04159ec0 con 0x7f9e041534f0 2023-10-16T20:56:53.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.522+0000 7f9e01ffb640 1 -- 172.21.15.88:0/928167025 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dec003530 con 0x7f9e041534f0 2023-10-16T20:56:53.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.522+0000 7f9e027fc640 1 -- 172.21.15.88:0/928167025 >> v1:172.21.15.191:6789/0 conn(0x7f9e04072070 legacy=0x7f9e041592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:53.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.522+0000 7f9e01ffb640 1 -- 172.21.15.88:0/928167025 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434622828 0 0) 0x7f9e0415a160 con 0x7f9e04072070 2023-10-16T20:56:53.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.522+0000 7f9e027fc640 1 -- 172.21.15.88:0/928167025 >> v1:172.21.15.88:6789/0 conn(0x7f9e041534f0 legacy=0x7f9e040c3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:53.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.522+0000 7f9e027fc640 1 -- 172.21.15.88:0/928167025 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0415c8d0 con 0x7f9e040c9a50 2023-10-16T20:56:53.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.522+0000 7f9e027fc640 1 --2- 172.21.15.88:0/928167025 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e040c9a50 0x7f9e04071b30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9df80097f0 tx=0x7f9df801ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c6234279513fabe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:53.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e01ffb640 1 -- 172.21.15.88:0/928167025 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df80a89b0 con 0x7f9e040c9a50 2023-10-16T20:56:53.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e01ffb640 1 -- 172.21.15.88:0/928167025 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df80a8b10 con 0x7f9e040c9a50 2023-10-16T20:56:53.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e01ffb640 1 -- 172.21.15.88:0/928167025 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df80a8ea0 con 0x7f9e040c9a50 2023-10-16T20:56:53.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e096c9640 1 -- 172.21.15.88:0/928167025 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e040c9a50 msgr2=0x7f9e04071b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:53.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e096c9640 1 --2- 172.21.15.88:0/928167025 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e040c9a50 0x7f9e04071b30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9df80097f0 tx=0x7f9df801ecb0 comp rx=0 tx=0).stop 2023-10-16T20:56:53.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e096c9640 1 -- 172.21.15.88:0/928167025 shutdown_connections 2023-10-16T20:56:53.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e096c9640 1 --2- 172.21.15.88:0/928167025 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e040c9a50 0x7f9e04071b30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:53.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e096c9640 1 -- 172.21.15.88:0/928167025 >> 172.21.15.88:0/928167025 conn(0x7f9e040cf260 msgr2=0x7f9e040cf670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:53.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e096c9640 1 -- 172.21.15.88:0/928167025 shutdown_connections 2023-10-16T20:56:53.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e096c9640 1 -- 172.21.15.88:0/928167025 wait complete. 2023-10-16T20:56:53.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e096c9640 1 Processor -- start 2023-10-16T20:56:53.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e096c9640 1 -- start start 2023-10-16T20:56:53.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e096c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e04072070 0x7f9e0415b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:53.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e096c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e040c9a50 0x7f9e040c6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:53.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e096c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e041534f0 0x7f9e040c7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:53.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e096c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e040c8f80 con 0x7f9e04072070 2023-10-16T20:56:53.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e096c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e04151d00 con 0x7f9e041534f0 2023-10-16T20:56:53.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.526+0000 7f9e096c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e04151fa0 con 0x7f9e040c9a50 2023-10-16T20:56:53.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9e02ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e04072070 0x7f9e0415b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:53.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9e02ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e04072070 0x7f9e0415b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59488/0 (socket says 172.21.15.88:59488) 2023-10-16T20:56:53.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9e02ffd640 1 -- 172.21.15.88:0/2572478938 learned_addr learned my addr 172.21.15.88:0/2572478938 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:53.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9e027fc640 1 --2- 172.21.15.88:0/2572478938 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e040c9a50 0x7f9e040c6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:53.532 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9e037fe640 1 --2- 172.21.15.88:0/2572478938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e041534f0 0x7f9e040c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:53.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9e02ffd640 1 -- 172.21.15.88:0/2572478938 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e040c9a50 msgr2=0x7f9e040c6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:53.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9e02ffd640 1 --2- 172.21.15.88:0/2572478938 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e040c9a50 0x7f9e040c6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:53.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9e02ffd640 1 -- 172.21.15.88:0/2572478938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e041534f0 msgr2=0x7f9e040c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:53.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9e02ffd640 1 --2- 172.21.15.88:0/2572478938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e041534f0 0x7f9e040c7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:53.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9e02ffd640 1 -- 172.21.15.88:0/2572478938 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df8009420 con 0x7f9e04072070 2023-10-16T20:56:53.533 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9e027fc640 1 --2- 172.21.15.88:0/2572478938 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e040c9a50 0x7f9e040c6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:56:53.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9e02ffd640 1 --2- 172.21.15.88:0/2572478938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e04072070 0x7f9e0415b920 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9df400c3c0 tx=0x7f9df400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:53.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9deb7fe640 1 -- 172.21.15.88:0/2572478938 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df4005cf0 con 0x7f9e04072070 2023-10-16T20:56:53.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9deb7fe640 1 -- 172.21.15.88:0/2572478938 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df4005e50 con 0x7f9e04072070 2023-10-16T20:56:53.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9e096c9640 1 -- 172.21.15.88:0/2572478938 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e040c8ce0 con 0x7f9e04072070 2023-10-16T20:56:53.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9deb7fe640 1 -- 172.21.15.88:0/2572478938 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df40072f0 con 0x7f9e04072070 2023-10-16T20:56:53.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9e096c9640 1 -- 172.21.15.88:0/2572478938 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e041f2aa0 con 0x7f9e04072070 2023-10-16T20:56:53.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9e096c9640 1 -- 172.21.15.88:0/2572478938 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e0404e9c0 con 0x7f9e04072070 2023-10-16T20:56:53.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.530+0000 7f9deb7fe640 1 -- 172.21.15.88:0/2572478938 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9df4007450 con 0x7f9e04072070 2023-10-16T20:56:53.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.534+0000 7f9deb7fe640 1 --2- 172.21.15.88:0/2572478938 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9df00664d0 0x7f9df0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:53.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.534+0000 7f9deb7fe640 1 -- 172.21.15.88:0/2572478938 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7f9df4083ba0 con 0x7f9e04072070 2023-10-16T20:56:53.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.534+0000 7f9e027fc640 1 --2- 172.21.15.88:0/2572478938 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9df00664d0 0x7f9df0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:53.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.534+0000 7f9e027fc640 1 --2- 172.21.15.88:0/2572478938 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9df00664d0 0x7f9df0068990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9df80b2000 tx=0x7f9df80aa070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:53.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.538+0000 7f9deb7fe640 1 -- 172.21.15.88:0/2572478938 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9df40508e0 con 0x7f9e04072070 2023-10-16T20:56:53.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.782+0000 7f9e096c9640 1 -- 172.21.15.88:0/2572478938 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e0415ce60 con 0x7f9e04072070 2023-10-16T20:56:53.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.782+0000 7f9deb7fe640 1 -- 172.21.15.88:0/2572478938 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5784 (secure 0 0 0) 0x7f9df4017070 con 0x7f9e04072070 2023-10-16T20:56:53.782 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:53.782 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:51.658472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:53.783 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 105 2023-10-16T20:56:53.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.782+0000 7f9e096c9640 1 -- 172.21.15.88:0/2572478938 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9df00664d0 msgr2=0x7f9df0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:53.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.782+0000 7f9e096c9640 1 --2- 172.21.15.88:0/2572478938 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9df00664d0 0x7f9df0068990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9df80b2000 tx=0x7f9df80aa070 comp rx=0 tx=0).stop 2023-10-16T20:56:53.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.782+0000 7f9e096c9640 1 -- 172.21.15.88:0/2572478938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e04072070 msgr2=0x7f9e0415b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:53.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.782+0000 7f9e096c9640 1 --2- 172.21.15.88:0/2572478938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e04072070 0x7f9e0415b920 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9df400c3c0 tx=0x7f9df400c880 comp rx=0 tx=0).stop 2023-10-16T20:56:53.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.782+0000 7f9e096c9640 1 -- 172.21.15.88:0/2572478938 shutdown_connections 2023-10-16T20:56:53.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.782+0000 7f9e096c9640 1 --2- 172.21.15.88:0/2572478938 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9df00664d0 0x7f9df0068990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:53.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.782+0000 7f9e096c9640 1 --2- 172.21.15.88:0/2572478938 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e041534f0 0x7f9e040c7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:53.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.782+0000 7f9e096c9640 1 --2- 172.21.15.88:0/2572478938 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9e040c9a50 0x7f9e040c6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:53.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.782+0000 7f9e096c9640 1 --2- 172.21.15.88:0/2572478938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e04072070 0x7f9e0415b920 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:53.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.782+0000 7f9e096c9640 1 -- 172.21.15.88:0/2572478938 >> 172.21.15.88:0/2572478938 conn(0x7f9e040cf260 msgr2=0x7f9e040c3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:53.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.782+0000 7f9e096c9640 1 -- 172.21.15.88:0/2572478938 shutdown_connections 2023-10-16T20:56:53.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:53.782+0000 7f9e096c9640 1 -- 172.21.15.88:0/2572478938 wait complete. 2023-10-16T20:56:53.808 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 105, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 105, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5378": { "addr": "172.21.15.191:6835/3119045295", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6834", "nonce": 3119045295, "type": "v2" }, { "addr": "172.21.15.191:6835", "nonce": 3119045295, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5378, "incarnation": 89, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:active", "state_seq": 32 }, "gid_5477": { "addr": "172.21.15.191:6837/2244284333", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6836", "nonce": 2244284333, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2244284333, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5477, "incarnation": 97, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 39 }, "gid_5535": { "addr": "172.21.15.88:6835/3476086754", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3476086754, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3476086754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5535, "incarnation": 52, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_5570": { "addr": "172.21.15.191:6833/3394064120", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 3394064120, "type": "v2" }, { "addr": "172.21.15.191:6833", "nonce": 3394064120, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5570, "incarnation": 103, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:reconnect", "state_seq": 9 }, "gid_5709": { "addr": "172.21.15.88:6838/2600891060", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 2600891060, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 2600891060, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5709, "incarnation": 67, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:56:51.658472+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5477, "mds_1": 5709, "mds_2": 5535, "mds_3": 5570, "mds_4": 5378 } } } ], "standbys": [] } 2023-10-16T20:56:55.810 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:55.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.970+0000 7f7b5bd01640 1 Processor -- start 2023-10-16T20:56:55.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.970+0000 7f7b5bd01640 1 -- start start 2023-10-16T20:56:55.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.974+0000 7f7b5bd01640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7b54150280 0x7f7b540c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:55.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.974+0000 7f7b5bd01640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b54150660 0x7f7b540c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:55.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.974+0000 7f7b5bd01640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7b540c6bc0 0x7f7b5406d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.974+0000 7f7b5bd01640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b540c5c60 con 0x7f7b540c6bc0 2023-10-16T20:56:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.974+0000 7f7b5bd01640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b540c5370 con 0x7f7b54150280 2023-10-16T20:56:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.974+0000 7f7b5bd01640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b54156d10 con 0x7f7b54150660 2023-10-16T20:56:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.974+0000 7f7b5a29e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7b540c6bc0 0x7f7b5406d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.974+0000 7f7b5a29e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7b540c6bc0 0x7f7b5406d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59498/0 (socket says 172.21.15.88:59498) 2023-10-16T20:56:55.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.974+0000 7f7b5a29e640 1 -- 172.21.15.88:0/3911369183 learned_addr learned my addr 172.21.15.88:0/3911369183 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:55.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.974+0000 7f7b5929c640 1 --2- 172.21.15.88:0/3911369183 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b54150660 0x7f7b540c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:55.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.974+0000 7f7b59a9d640 1 --2- 172.21.15.88:0/3911369183 >> v2:172.21.15.191:3300/0 conn(0x7f7b54150280 0x7f7b540c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:55.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.974+0000 7f7b5a29e640 1 -- 172.21.15.88:0/3911369183 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b54150660 msgr2=0x7f7b540c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:56:55.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.974+0000 7f7b5a29e640 1 --2- 172.21.15.88:0/3911369183 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b54150660 0x7f7b540c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:55.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.974+0000 7f7b5a29e640 1 -- 172.21.15.88:0/3911369183 >> v2:172.21.15.191:3300/0 conn(0x7f7b54150280 msgr2=0x7f7b540c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:56:55.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.974+0000 7f7b5a29e640 1 --2- 172.21.15.88:0/3911369183 >> v2:172.21.15.191:3300/0 conn(0x7f7b54150280 0x7f7b540c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:55.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.974+0000 7f7b5a29e640 1 -- 172.21.15.88:0/3911369183 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b540c5610 con 0x7f7b540c6bc0 2023-10-16T20:56:55.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.978+0000 7f7b5a29e640 1 --2- 172.21.15.88:0/3911369183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b540c6bc0 0x7f7b5406d200 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7b5406f060 tx=0x7f7b4c01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6db1d28fb47c22 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:55.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.978+0000 7f7b58a9b640 1 -- 172.21.15.88:0/3911369183 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c0a56a0 con 0x7f7b540c6bc0 2023-10-16T20:56:55.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.978+0000 7f7b58a9b640 1 -- 172.21.15.88:0/3911369183 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4c0a5800 con 0x7f7b540c6bc0 2023-10-16T20:56:55.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.978+0000 7f7b58a9b640 1 -- 172.21.15.88:0/3911369183 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c0a5b90 con 0x7f7b540c6bc0 2023-10-16T20:56:55.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.978+0000 7f7b5bd01640 1 -- 172.21.15.88:0/3911369183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b540c6bc0 msgr2=0x7f7b5406d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:55.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.978+0000 7f7b5bd01640 1 --2- 172.21.15.88:0/3911369183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b540c6bc0 0x7f7b5406d200 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7b5406f060 tx=0x7f7b4c01c910 comp rx=0 tx=0).stop 2023-10-16T20:56:55.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.978+0000 7f7b5bd01640 1 -- 172.21.15.88:0/3911369183 shutdown_connections 2023-10-16T20:56:55.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.978+0000 7f7b5bd01640 1 --2- 172.21.15.88:0/3911369183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b540c6bc0 0x7f7b5406d200 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:55.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.978+0000 7f7b5bd01640 1 --2- 172.21.15.88:0/3911369183 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b54150660 0x7f7b540c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:55.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.978+0000 7f7b5bd01640 1 --2- 172.21.15.88:0/3911369183 >> v2:172.21.15.191:3300/0 conn(0x7f7b54150280 0x7f7b540c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:55.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.978+0000 7f7b5bd01640 1 -- 172.21.15.88:0/3911369183 >> 172.21.15.88:0/3911369183 conn(0x7f7b540cad50 msgr2=0x7f7b540cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:55.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.978+0000 7f7b5bd01640 1 -- 172.21.15.88:0/3911369183 shutdown_connections 2023-10-16T20:56:55.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.978+0000 7f7b5bd01640 1 -- 172.21.15.88:0/3911369183 wait complete. 2023-10-16T20:56:55.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.978+0000 7f7b5bd01640 1 Processor -- start 2023-10-16T20:56:55.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.978+0000 7f7b5bd01640 1 -- start start 2023-10-16T20:56:55.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5bd01640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b540c6bc0 0x7f7b54152000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:55.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5bd01640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b54150280 0x7f7b5414b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:55.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5bd01640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b54150660 0x7f7b54158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:55.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5bd01640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b541515f0 con 0x7f7b54150280 2023-10-16T20:56:55.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5bd01640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b54151b70 con 0x7f7b540c6bc0 2023-10-16T20:56:55.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5bd01640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b54150ae0 con 0x7f7b54150660 2023-10-16T20:56:55.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5929c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b54150280 0x7f7b5414b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:55.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5929c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b54150280 0x7f7b5414b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59510/0 (socket says 172.21.15.88:59510) 2023-10-16T20:56:55.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5929c640 1 -- 172.21.15.88:0/3601705927 learned_addr learned my addr 172.21.15.88:0/3601705927 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:55.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b59a9d640 1 --2- 172.21.15.88:0/3601705927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b540c6bc0 0x7f7b54152000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:55.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5a29e640 1 --2- 172.21.15.88:0/3601705927 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b54150660 0x7f7b54158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:55.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5929c640 1 -- 172.21.15.88:0/3601705927 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b54150660 msgr2=0x7f7b54158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:55.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5929c640 1 --2- 172.21.15.88:0/3601705927 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b54150660 0x7f7b54158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:55.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5929c640 1 -- 172.21.15.88:0/3601705927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b540c6bc0 msgr2=0x7f7b54152000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:55.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5929c640 1 --2- 172.21.15.88:0/3601705927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b540c6bc0 0x7f7b54152000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:55.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5929c640 1 -- 172.21.15.88:0/3601705927 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4c009b30 con 0x7f7b54150280 2023-10-16T20:56:55.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5929c640 1 --2- 172.21.15.88:0/3601705927 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b54150280 0x7f7b5414b620 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7b4400d620 tx=0x7f7b4400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:55.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b4a7fc640 1 -- 172.21.15.88:0/3601705927 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4400b620 con 0x7f7b54150280 2023-10-16T20:56:55.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b4a7fc640 1 -- 172.21.15.88:0/3601705927 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4400b780 con 0x7f7b54150280 2023-10-16T20:56:55.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5bd01640 1 -- 172.21.15.88:0/3601705927 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b54150d50 con 0x7f7b54150280 2023-10-16T20:56:55.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b4a7fc640 1 -- 172.21.15.88:0/3601705927 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4400bb10 con 0x7f7b54150280 2023-10-16T20:56:55.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5bd01640 1 -- 172.21.15.88:0/3601705927 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b54151310 con 0x7f7b54150280 2023-10-16T20:56:55.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b5bd01640 1 -- 172.21.15.88:0/3601705927 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b54152540 con 0x7f7b54150280 2023-10-16T20:56:55.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.982+0000 7f7b4a7fc640 1 -- 172.21.15.88:0/3601705927 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7b4400bc70 con 0x7f7b54150280 2023-10-16T20:56:55.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.986+0000 7f7b4a7fc640 1 --2- 172.21.15.88:0/3601705927 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7b380664d0 0x7f7b38068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:55.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.986+0000 7f7b4a7fc640 1 -- 172.21.15.88:0/3601705927 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7f7b440848d0 con 0x7f7b54150280 2023-10-16T20:56:55.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.986+0000 7f7b59a9d640 1 --2- 172.21.15.88:0/3601705927 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7b380664d0 0x7f7b38068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:55.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.986+0000 7f7b59a9d640 1 --2- 172.21.15.88:0/3601705927 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7b380664d0 0x7f7b38068990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7b50007a60 tx=0x7f7b50008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:55.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:55.990+0000 7f7b4a7fc640 1 -- 172.21.15.88:0/3601705927 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7b440516a0 con 0x7f7b54150280 2023-10-16T20:56:56.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:56.170+0000 7f7b5bd01640 1 -- 172.21.15.88:0/3601705927 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b54151940 con 0x7f7b54150280 2023-10-16T20:56:56.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:56.170+0000 7f7b4a7fc640 1 -- 172.21.15.88:0/3601705927 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5784 (secure 0 0 0) 0x7f7b44007380 con 0x7f7b54150280 2023-10-16T20:56:56.172 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:56.172 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:51.658472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:56.174 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 105 2023-10-16T20:56:56.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:56.170+0000 7f7b5bd01640 1 -- 172.21.15.88:0/3601705927 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7b380664d0 msgr2=0x7f7b38068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:56.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:56.170+0000 7f7b5bd01640 1 --2- 172.21.15.88:0/3601705927 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7b380664d0 0x7f7b38068990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7b50007a60 tx=0x7f7b50008040 comp rx=0 tx=0).stop 2023-10-16T20:56:56.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:56.170+0000 7f7b5bd01640 1 -- 172.21.15.88:0/3601705927 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b54150280 msgr2=0x7f7b5414b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:56.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:56.170+0000 7f7b5bd01640 1 --2- 172.21.15.88:0/3601705927 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b54150280 0x7f7b5414b620 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7b4400d620 tx=0x7f7b4400dae0 comp rx=0 tx=0).stop 2023-10-16T20:56:56.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:56.174+0000 7f7b5bd01640 1 -- 172.21.15.88:0/3601705927 shutdown_connections 2023-10-16T20:56:56.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:56.174+0000 7f7b5bd01640 1 --2- 172.21.15.88:0/3601705927 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7b380664d0 0x7f7b38068990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:56.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:56.174+0000 7f7b5bd01640 1 --2- 172.21.15.88:0/3601705927 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7b54150660 0x7f7b54158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:56.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:56.174+0000 7f7b5bd01640 1 --2- 172.21.15.88:0/3601705927 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b54150280 0x7f7b5414b620 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:56.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:56.174+0000 7f7b5bd01640 1 --2- 172.21.15.88:0/3601705927 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b540c6bc0 0x7f7b54152000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:56.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:56.174+0000 7f7b5bd01640 1 -- 172.21.15.88:0/3601705927 >> 172.21.15.88:0/3601705927 conn(0x7f7b540cad50 msgr2=0x7f7b54154430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:56.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:56.174+0000 7f7b5bd01640 1 -- 172.21.15.88:0/3601705927 shutdown_connections 2023-10-16T20:56:56.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:56.174+0000 7f7b5bd01640 1 -- 172.21.15.88:0/3601705927 wait complete. 2023-10-16T20:56:56.191 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:56:58.192 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:56:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e2eb35640 1 Processor -- start 2023-10-16T20:56:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e2eb35640 1 -- start start 2023-10-16T20:56:58.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e2eb35640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6e280c76c0 0x7f6e28159450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:58.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e2eb35640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e280c81a0 0x7f6e2806b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:58.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e2eb35640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e280c8810 con 0x7f6e280c7dc0 2023-10-16T20:56:58.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e2eb35640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e2806b400 con 0x7f6e280c76c0 2023-10-16T20:56:58.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e2eb35640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2806b0d0 con 0x7f6e280c81a0 2023-10-16T20:56:58.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e27fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6e280c76c0 0x7f6e28159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:58.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e27fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6e280c76c0 0x7f6e28159450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59522/0 (socket says 172.21.15.88:59522) 2023-10-16T20:56:58.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e27fff640 1 -- 172.21.15.88:0/3581574989 learned_addr learned my addr 172.21.15.88:0/3581574989 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:58.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e2c8d1640 1 --1- 172.21.15.88:0/3581574989 >> v1:172.21.15.88:6789/0 conn(0x7f6e280c7dc0 0x7f6e2806d650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59914/0 (socket says 172.21.15.88:59914) 2023-10-16T20:56:58.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e2d0d2640 1 --2- 172.21.15.88:0/3581574989 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e280c81a0 0x7f6e2806b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:58.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e27fff640 1 -- 172.21.15.88:0/3581574989 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e280c81a0 msgr2=0x7f6e2806b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:56:58.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e27fff640 1 --2- 172.21.15.88:0/3581574989 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e280c81a0 0x7f6e2806b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:58.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e27fff640 1 -- 172.21.15.88:0/3581574989 >> v1:172.21.15.88:6789/0 conn(0x7f6e280c7dc0 legacy=0x7f6e2806d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:58.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e27fff640 1 -- 172.21.15.88:0/3581574989 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2806e6a0 con 0x7f6e280c76c0 2023-10-16T20:56:58.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e2d0d2640 1 --2- 172.21.15.88:0/3581574989 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e280c81a0 0x7f6e2806b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:56:58.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.334+0000 7f6e277fe640 1 -- 172.21.15.88:0/3581574989 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294029970 0 0) 0x7f6e280c8810 con 0x7f6e280c7dc0 2023-10-16T20:56:58.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.338+0000 7f6e27fff640 1 --2- 172.21.15.88:0/3581574989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e280c76c0 0x7f6e28159450 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009870 tx=0x7f6e1401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23dbebd9e7432173 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:58.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.338+0000 7f6e277fe640 1 -- 172.21.15.88:0/3581574989 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e140ab600 con 0x7f6e280c76c0 2023-10-16T20:56:58.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.338+0000 7f6e277fe640 1 -- 172.21.15.88:0/3581574989 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e140a7050 con 0x7f6e280c76c0 2023-10-16T20:56:58.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.338+0000 7f6e277fe640 1 -- 172.21.15.88:0/3581574989 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e140ab980 con 0x7f6e280c76c0 2023-10-16T20:56:58.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.338+0000 7f6e2eb35640 1 -- 172.21.15.88:0/3581574989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e280c76c0 msgr2=0x7f6e28159450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:58.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.338+0000 7f6e2eb35640 1 --2- 172.21.15.88:0/3581574989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e280c76c0 0x7f6e28159450 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009870 tx=0x7f6e1401c4b0 comp rx=0 tx=0).stop 2023-10-16T20:56:58.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.338+0000 7f6e2eb35640 1 -- 172.21.15.88:0/3581574989 shutdown_connections 2023-10-16T20:56:58.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.338+0000 7f6e2eb35640 1 --2- 172.21.15.88:0/3581574989 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e280c81a0 0x7f6e2806b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:58.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.338+0000 7f6e2eb35640 1 --2- 172.21.15.88:0/3581574989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e280c76c0 0x7f6e28159450 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:58.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.338+0000 7f6e2eb35640 1 -- 172.21.15.88:0/3581574989 >> 172.21.15.88:0/3581574989 conn(0x7f6e28071a60 msgr2=0x7f6e28071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:58.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.338+0000 7f6e2eb35640 1 -- 172.21.15.88:0/3581574989 shutdown_connections 2023-10-16T20:56:58.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.338+0000 7f6e2eb35640 1 -- 172.21.15.88:0/3581574989 wait complete. 2023-10-16T20:56:58.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e2eb35640 1 Processor -- start 2023-10-16T20:56:58.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e2eb35640 1 -- start start 2023-10-16T20:56:58.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e2eb35640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e280c76c0 0x7f6e28159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:58.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e2eb35640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e280c7dc0 0x7f6e2814ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:58.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e2eb35640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e280c81a0 0x7f6e28151820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:58.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e2eb35640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e28159960 con 0x7f6e280c7dc0 2023-10-16T20:56:58.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e2eb35640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e28151390 con 0x7f6e280c81a0 2023-10-16T20:56:58.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e2eb35640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e280caff0 con 0x7f6e280c76c0 2023-10-16T20:56:58.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e27fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e280c7dc0 0x7f6e2814ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:58.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e27fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e280c7dc0 0x7f6e2814ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59534/0 (socket says 172.21.15.88:59534) 2023-10-16T20:56:58.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e27fff640 1 -- 172.21.15.88:0/4188759361 learned_addr learned my addr 172.21.15.88:0/4188759361 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:56:58.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e2c8d1640 1 --2- 172.21.15.88:0/4188759361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e280c76c0 0x7f6e28159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:58.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e2d0d2640 1 --2- 172.21.15.88:0/4188759361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e280c81a0 0x7f6e28151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:58.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e27fff640 1 -- 172.21.15.88:0/4188759361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e280c76c0 msgr2=0x7f6e28159420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:58.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e27fff640 1 --2- 172.21.15.88:0/4188759361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e280c76c0 0x7f6e28159420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:58.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e27fff640 1 -- 172.21.15.88:0/4188759361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e280c81a0 msgr2=0x7f6e28151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:58.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e27fff640 1 --2- 172.21.15.88:0/4188759361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e280c81a0 0x7f6e28151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:58.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e27fff640 1 -- 172.21.15.88:0/4188759361 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e140094d0 con 0x7f6e280c7dc0 2023-10-16T20:56:58.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e27fff640 1 --2- 172.21.15.88:0/4188759361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e280c7dc0 0x7f6e2814ebc0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6e140ae040 tx=0x7f6e1401acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:58.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e257fa640 1 -- 172.21.15.88:0/4188759361 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e140b6620 con 0x7f6e280c7dc0 2023-10-16T20:56:58.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e257fa640 1 -- 172.21.15.88:0/4188759361 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e140b67a0 con 0x7f6e280c7dc0 2023-10-16T20:56:58.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e257fa640 1 -- 172.21.15.88:0/4188759361 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e140b6b50 con 0x7f6e280c7dc0 2023-10-16T20:56:58.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e2eb35640 1 -- 172.21.15.88:0/4188759361 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e281585f0 con 0x7f6e280c7dc0 2023-10-16T20:56:58.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.342+0000 7f6e2eb35640 1 -- 172.21.15.88:0/4188759361 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e28158bb0 con 0x7f6e280c7dc0 2023-10-16T20:56:58.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.346+0000 7f6e2eb35640 1 -- 172.21.15.88:0/4188759361 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e28151540 con 0x7f6e280c7dc0 2023-10-16T20:56:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.346+0000 7f6e257fa640 1 -- 172.21.15.88:0/4188759361 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6e140b2040 con 0x7f6e280c7dc0 2023-10-16T20:56:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.346+0000 7f6e257fa640 1 --2- 172.21.15.88:0/4188759361 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6e0c0664a0 0x7f6e0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:56:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.346+0000 7f6e257fa640 1 -- 172.21.15.88:0/4188759361 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7f6e1411e990 con 0x7f6e280c7dc0 2023-10-16T20:56:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.346+0000 7f6e2c8d1640 1 --2- 172.21.15.88:0/4188759361 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6e0c0664a0 0x7f6e0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:56:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.346+0000 7f6e2c8d1640 1 --2- 172.21.15.88:0/4188759361 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6e0c0664a0 0x7f6e0c068960 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c007fd0 tx=0x7f6e1c007d60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:56:58.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.350+0000 7f6e257fa640 1 -- 172.21.15.88:0/4188759361 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6e140eb760 con 0x7f6e280c7dc0 2023-10-16T20:56:58.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.534+0000 7f6e2eb35640 1 -- 172.21.15.88:0/4188759361 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e28074700 con 0x7f6e280c7dc0 2023-10-16T20:56:58.535 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.534+0000 7f6e257fa640 1 -- 172.21.15.88:0/4188759361 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5782 (secure 0 0 0) 0x7f6e140eed80 con 0x7f6e280c7dc0 2023-10-16T20:56:58.535 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:56:58.536 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:57.663588+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:56:58.536 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 106 2023-10-16T20:56:58.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.538+0000 7f6e2eb35640 1 -- 172.21.15.88:0/4188759361 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6e0c0664a0 msgr2=0x7f6e0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:58.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.538+0000 7f6e2eb35640 1 --2- 172.21.15.88:0/4188759361 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6e0c0664a0 0x7f6e0c068960 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c007fd0 tx=0x7f6e1c007d60 comp rx=0 tx=0).stop 2023-10-16T20:56:58.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.538+0000 7f6e2eb35640 1 -- 172.21.15.88:0/4188759361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e280c7dc0 msgr2=0x7f6e2814ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:56:58.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.538+0000 7f6e2eb35640 1 --2- 172.21.15.88:0/4188759361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e280c7dc0 0x7f6e2814ebc0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6e140ae040 tx=0x7f6e1401acf0 comp rx=0 tx=0).stop 2023-10-16T20:56:58.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.538+0000 7f6e2eb35640 1 -- 172.21.15.88:0/4188759361 shutdown_connections 2023-10-16T20:56:58.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.538+0000 7f6e2eb35640 1 --2- 172.21.15.88:0/4188759361 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6e0c0664a0 0x7f6e0c068960 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:58.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.538+0000 7f6e2eb35640 1 --2- 172.21.15.88:0/4188759361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e280c81a0 0x7f6e28151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:58.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.538+0000 7f6e2eb35640 1 --2- 172.21.15.88:0/4188759361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e280c7dc0 0x7f6e2814ebc0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:58.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.538+0000 7f6e2eb35640 1 --2- 172.21.15.88:0/4188759361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6e280c76c0 0x7f6e28159420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:56:58.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.538+0000 7f6e2eb35640 1 -- 172.21.15.88:0/4188759361 >> 172.21.15.88:0/4188759361 conn(0x7f6e28071a60 msgr2=0x7f6e2814fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:56:58.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.538+0000 7f6e2eb35640 1 -- 172.21.15.88:0/4188759361 shutdown_connections 2023-10-16T20:56:58.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:56:58.538+0000 7f6e2eb35640 1 -- 172.21.15.88:0/4188759361 wait complete. 2023-10-16T20:56:58.558 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:57:00.175 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:57:00.180 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:57:00.559 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:57:00.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.722+0000 7fb791bd0640 1 Processor -- start 2023-10-16T20:57:00.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.722+0000 7fb791bd0640 1 -- start start 2023-10-16T20:57:00.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.726+0000 7fb791bd0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb78c1636f0 0x7fb78c167480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:00.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.726+0000 7fb791bd0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb78c0c5ce0 0x7fb78c166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:00.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.726+0000 7fb791bd0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78c163ad0 con 0x7fb78c169f90 2023-10-16T20:57:00.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.726+0000 7fb791bd0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb78c0c44a0 con 0x7fb78c1636f0 2023-10-16T20:57:00.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.726+0000 7fb791bd0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c16a420 con 0x7fb78c0c5ce0 2023-10-16T20:57:00.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.726+0000 7fb790bce640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb78c1636f0 0x7fb78c167480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:00.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.726+0000 7fb790bce640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb78c1636f0 0x7fb78c167480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49294/0 (socket says 172.21.15.88:49294) 2023-10-16T20:57:00.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.726+0000 7fb790bce640 1 -- 172.21.15.88:0/576401361 learned_addr learned my addr 172.21.15.88:0/576401361 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:57:00.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.726+0000 7fb7913cf640 1 --2- 172.21.15.88:0/576401361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb78c0c5ce0 0x7fb78c166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:00.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.726+0000 7fb78bfff640 1 --1- 172.21.15.88:0/576401361 >> v1:172.21.15.88:6789/0 conn(0x7fb78c169f90 0x7fb78c0c7470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49182/0 (socket says 172.21.15.88:49182) 2023-10-16T20:57:00.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.834+0000 7fb78b7fe640 1 -- 172.21.15.88:0/576401361 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340950477 0 0) 0x7fb778001120 con 0x7fb78c169f90 2023-10-16T20:57:00.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.834+0000 7fb78b7fe640 1 -- 172.21.15.88:0/576401361 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb77c003530 con 0x7fb78c169f90 2023-10-16T20:57:00.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.834+0000 7fb790bce640 1 -- 172.21.15.88:0/576401361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb78c0c5ce0 msgr2=0x7fb78c166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:57:00.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.834+0000 7fb790bce640 1 --2- 172.21.15.88:0/576401361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb78c0c5ce0 0x7fb78c166ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:00.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.834+0000 7fb790bce640 1 -- 172.21.15.88:0/576401361 >> v1:172.21.15.88:6789/0 conn(0x7fb78c169f90 legacy=0x7fb78c0c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:00.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.834+0000 7fb790bce640 1 -- 172.21.15.88:0/576401361 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c166010 con 0x7fb78c1636f0 2023-10-16T20:57:00.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.838+0000 7fb790bce640 1 --2- 172.21.15.88:0/576401361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb78c1636f0 0x7fb78c167480 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb780009e50 tx=0x7fb780021410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a626d51de8eea6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:00.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.838+0000 7fb78b7fe640 1 -- 172.21.15.88:0/576401361 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7800b1400 con 0x7fb78c1636f0 2023-10-16T20:57:00.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.842+0000 7fb78b7fe640 1 -- 172.21.15.88:0/576401361 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7800ad050 con 0x7fb78c1636f0 2023-10-16T20:57:00.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.842+0000 7fb78b7fe640 1 -- 172.21.15.88:0/576401361 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7800b1780 con 0x7fb78c1636f0 2023-10-16T20:57:00.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.842+0000 7fb791bd0640 1 -- 172.21.15.88:0/576401361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb78c1636f0 msgr2=0x7fb78c167480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:00.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.842+0000 7fb791bd0640 1 --2- 172.21.15.88:0/576401361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb78c1636f0 0x7fb78c167480 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb780009e50 tx=0x7fb780021410 comp rx=0 tx=0).stop 2023-10-16T20:57:00.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.842+0000 7fb791bd0640 1 -- 172.21.15.88:0/576401361 shutdown_connections 2023-10-16T20:57:00.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.842+0000 7fb791bd0640 1 --2- 172.21.15.88:0/576401361 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb78c0c5ce0 0x7fb78c166ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:00.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.842+0000 7fb791bd0640 1 --2- 172.21.15.88:0/576401361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb78c1636f0 0x7fb78c167480 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:00.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.842+0000 7fb791bd0640 1 -- 172.21.15.88:0/576401361 >> 172.21.15.88:0/576401361 conn(0x7fb78c06d2d0 msgr2=0x7fb78c06d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:00.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.842+0000 7fb791bd0640 1 -- 172.21.15.88:0/576401361 shutdown_connections 2023-10-16T20:57:00.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.842+0000 7fb791bd0640 1 -- 172.21.15.88:0/576401361 wait complete. 2023-10-16T20:57:00.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.842+0000 7fb791bd0640 1 Processor -- start 2023-10-16T20:57:00.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.842+0000 7fb791bd0640 1 -- start start 2023-10-16T20:57:00.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb791bd0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb78c0c5ce0 0x7fb78c161af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:00.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb791bd0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb78c1636f0 0x7fb78c0718c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:00.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb791bd0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb78c071e00 0x7fb78c0c1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:00.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb791bd0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c1f30d0 con 0x7fb78c071e00 2023-10-16T20:57:00.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb791bd0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c162e80 con 0x7fb78c1636f0 2023-10-16T20:57:00.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb791bd0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c1f2df0 con 0x7fb78c0c5ce0 2023-10-16T20:57:00.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb7913cf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb78c071e00 0x7fb78c0c1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:00.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb790bce640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb78c0c5ce0 0x7fb78c161af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:00.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb7913cf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb78c071e00 0x7fb78c0c1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49302/0 (socket says 172.21.15.88:49302) 2023-10-16T20:57:00.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb7913cf640 1 -- 172.21.15.88:0/3775235088 learned_addr learned my addr 172.21.15.88:0/3775235088 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:57:00.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb78bfff640 1 --2- 172.21.15.88:0/3775235088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb78c1636f0 0x7fb78c0718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:00.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb790bce640 1 -- 172.21.15.88:0/3775235088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb78c1636f0 msgr2=0x7fb78c0718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:00.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb790bce640 1 --2- 172.21.15.88:0/3775235088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb78c1636f0 0x7fb78c0718c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:00.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb790bce640 1 -- 172.21.15.88:0/3775235088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb78c071e00 msgr2=0x7fb78c0c1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:00.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb790bce640 1 --2- 172.21.15.88:0/3775235088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb78c071e00 0x7fb78c0c1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:00.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb790bce640 1 -- 172.21.15.88:0/3775235088 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb780009b30 con 0x7fb78c0c5ce0 2023-10-16T20:57:00.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb7913cf640 1 --2- 172.21.15.88:0/3775235088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb78c071e00 0x7fb78c0c1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:57:00.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb790bce640 1 --2- 172.21.15.88:0/3775235088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb78c0c5ce0 0x7fb78c161af0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb7800b0000 tx=0x7fb78001e1b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:00.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb7897fa640 1 -- 172.21.15.88:0/3775235088 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7800b1d60 con 0x7fb78c0c5ce0 2023-10-16T20:57:00.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb7897fa640 1 -- 172.21.15.88:0/3775235088 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb780009510 con 0x7fb78c0c5ce0 2023-10-16T20:57:00.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb791bd0640 1 -- 172.21.15.88:0/3775235088 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78c1628c0 con 0x7fb78c0c5ce0 2023-10-16T20:57:00.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb7897fa640 1 -- 172.21.15.88:0/3775235088 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7800bc660 con 0x7fb78c0c5ce0 2023-10-16T20:57:00.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.846+0000 7fb791bd0640 1 -- 172.21.15.88:0/3775235088 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78c0c15e0 con 0x7fb78c0c5ce0 2023-10-16T20:57:00.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.850+0000 7fb791bd0640 1 -- 172.21.15.88:0/3775235088 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78c0c17d0 con 0x7fb78c0c5ce0 2023-10-16T20:57:00.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.850+0000 7fb7897fa640 1 -- 172.21.15.88:0/3775235088 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb7800b8040 con 0x7fb78c0c5ce0 2023-10-16T20:57:00.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.850+0000 7fb7897fa640 1 --2- 172.21.15.88:0/3775235088 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb7700664a0 0x7fb770068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:00.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.850+0000 7fb7897fa640 1 -- 172.21.15.88:0/3775235088 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7fb780123b00 con 0x7fb78c0c5ce0 2023-10-16T20:57:00.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.850+0000 7fb78bfff640 1 --2- 172.21.15.88:0/3775235088 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb7700664a0 0x7fb770068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:00.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.850+0000 7fb78bfff640 1 --2- 172.21.15.88:0/3775235088 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb7700664a0 0x7fb770068960 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb778005030 tx=0x7fb778004b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:00.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:00.854+0000 7fb7897fa640 1 -- 172.21.15.88:0/3775235088 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb780125e60 con 0x7fb78c0c5ce0 2023-10-16T20:57:01.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:01.050+0000 7fb791bd0640 1 -- 172.21.15.88:0/3775235088 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb78c04e9c0 con 0x7fb78c0c5ce0 2023-10-16T20:57:01.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:01.050+0000 7fb7897fa640 1 -- 172.21.15.88:0/3775235088 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5782 (secure 0 0 0) 0x7fb7800f0990 con 0x7fb78c0c5ce0 2023-10-16T20:57:01.051 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:57:01.052 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:56:57.663588+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:57:01.052 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 106 2023-10-16T20:57:01.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:01.054+0000 7fb791bd0640 1 -- 172.21.15.88:0/3775235088 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb7700664a0 msgr2=0x7fb770068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:01.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:01.054+0000 7fb791bd0640 1 --2- 172.21.15.88:0/3775235088 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb7700664a0 0x7fb770068960 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb778005030 tx=0x7fb778004b50 comp rx=0 tx=0).stop 2023-10-16T20:57:01.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:01.054+0000 7fb791bd0640 1 -- 172.21.15.88:0/3775235088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb78c0c5ce0 msgr2=0x7fb78c161af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:01.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:01.054+0000 7fb791bd0640 1 --2- 172.21.15.88:0/3775235088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb78c0c5ce0 0x7fb78c161af0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb7800b0000 tx=0x7fb78001e1b0 comp rx=0 tx=0).stop 2023-10-16T20:57:01.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:01.054+0000 7fb791bd0640 1 -- 172.21.15.88:0/3775235088 shutdown_connections 2023-10-16T20:57:01.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:01.054+0000 7fb791bd0640 1 --2- 172.21.15.88:0/3775235088 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb7700664a0 0x7fb770068960 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:01.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:01.054+0000 7fb791bd0640 1 --2- 172.21.15.88:0/3775235088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb78c071e00 0x7fb78c0c1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:01.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:01.054+0000 7fb791bd0640 1 --2- 172.21.15.88:0/3775235088 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb78c1636f0 0x7fb78c0718c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:01.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:01.054+0000 7fb791bd0640 1 --2- 172.21.15.88:0/3775235088 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb78c0c5ce0 0x7fb78c161af0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:01.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:01.054+0000 7fb791bd0640 1 -- 172.21.15.88:0/3775235088 >> 172.21.15.88:0/3775235088 conn(0x7fb78c06d2d0 msgr2=0x7fb78c0c7440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:01.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:01.054+0000 7fb791bd0640 1 -- 172.21.15.88:0/3775235088 shutdown_connections 2023-10-16T20:57:01.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:01.054+0000 7fb791bd0640 1 -- 172.21.15.88:0/3775235088 wait complete. 2023-10-16T20:57:01.076 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:57:03.078 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:57:03.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f636b765640 1 Processor -- start 2023-10-16T20:57:03.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f636b765640 1 -- start start 2023-10-16T20:57:03.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f636b765640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6364153770 0x7f6364159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:03.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f636b765640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63640c9bb0 0x7f63640c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:03.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f636b765640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f63640c41f0 0x7f63640cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:03.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f636b765640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63640cc250 con 0x7f6364153770 2023-10-16T20:57:03.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f636b765640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63640cc670 con 0x7f63640c41f0 2023-10-16T20:57:03.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f636b765640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636415a410 con 0x7f63640c9bb0 2023-10-16T20:57:03.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f6369501640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6364153770 0x7f6364159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:03.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f6369501640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6364153770 0x7f6364159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49316/0 (socket says 172.21.15.88:49316) 2023-10-16T20:57:03.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f6369501640 1 -- 172.21.15.88:0/2782129932 learned_addr learned my addr 172.21.15.88:0/2782129932 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:57:03.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f6368d00640 1 --2- 172.21.15.88:0/2782129932 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63640c9bb0 0x7f63640c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:03.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f6369d02640 1 --2- 172.21.15.88:0/2782129932 >> v2:172.21.15.191:3300/0 conn(0x7f63640c41f0 0x7f63640cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:03.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f6369501640 1 -- 172.21.15.88:0/2782129932 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63640c9bb0 msgr2=0x7f63640c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:57:03.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f6369501640 1 --2- 172.21.15.88:0/2782129932 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63640c9bb0 0x7f63640c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f6369501640 1 -- 172.21.15.88:0/2782129932 >> v2:172.21.15.191:3300/0 conn(0x7f63640c41f0 msgr2=0x7f63640cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:57:03.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f6369501640 1 --2- 172.21.15.88:0/2782129932 >> v2:172.21.15.191:3300/0 conn(0x7f63640c41f0 0x7f63640cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.225+0000 7f6369501640 1 -- 172.21.15.88:0/2782129932 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6364153b50 con 0x7f6364153770 2023-10-16T20:57:03.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.229+0000 7f6369501640 1 --2- 172.21.15.88:0/2782129932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6364153770 0x7f6364159ed0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f63640b2dc0 tx=0x7f636001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8419d062e51af704 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:03.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.229+0000 7f635bfff640 1 -- 172.21.15.88:0/2782129932 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63600ac630 con 0x7f6364153770 2023-10-16T20:57:03.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.229+0000 7f635bfff640 1 -- 172.21.15.88:0/2782129932 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63600ac790 con 0x7f6364153770 2023-10-16T20:57:03.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.229+0000 7f635bfff640 1 -- 172.21.15.88:0/2782129932 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63600acb20 con 0x7f6364153770 2023-10-16T20:57:03.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.229+0000 7f636b765640 1 -- 172.21.15.88:0/2782129932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6364153770 msgr2=0x7f6364159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:03.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.229+0000 7f636b765640 1 --2- 172.21.15.88:0/2782129932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6364153770 0x7f6364159ed0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f63640b2dc0 tx=0x7f636001c500 comp rx=0 tx=0).stop 2023-10-16T20:57:03.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.229+0000 7f636b765640 1 -- 172.21.15.88:0/2782129932 shutdown_connections 2023-10-16T20:57:03.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.229+0000 7f636b765640 1 --2- 172.21.15.88:0/2782129932 >> v2:172.21.15.191:3300/0 conn(0x7f63640c41f0 0x7f63640cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.229+0000 7f636b765640 1 --2- 172.21.15.88:0/2782129932 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63640c9bb0 0x7f63640c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.229+0000 7f636b765640 1 --2- 172.21.15.88:0/2782129932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6364153770 0x7f6364159ed0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.229+0000 7f636b765640 1 -- 172.21.15.88:0/2782129932 >> 172.21.15.88:0/2782129932 conn(0x7f63640715c0 msgr2=0x7f63640719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:03.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.229+0000 7f636b765640 1 -- 172.21.15.88:0/2782129932 shutdown_connections 2023-10-16T20:57:03.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.229+0000 7f636b765640 1 -- 172.21.15.88:0/2782129932 wait complete. 2023-10-16T20:57:03.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f636b765640 1 Processor -- start 2023-10-16T20:57:03.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f636b765640 1 -- start start 2023-10-16T20:57:03.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f636b765640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63640c41f0 0x7f63641f5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:03.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f636b765640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63640c9bb0 0x7f6364158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:03.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f636b765640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6364153770 0x7f6364159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:03.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f636b765640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6364147dd0 con 0x7f6364153770 2023-10-16T20:57:03.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f636b765640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63641480b0 con 0x7f63640c41f0 2023-10-16T20:57:03.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f636b765640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63641f5f70 con 0x7f63640c9bb0 2023-10-16T20:57:03.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f6369d02640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6364153770 0x7f6364159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:03.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f6369d02640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6364153770 0x7f6364159520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49330/0 (socket says 172.21.15.88:49330) 2023-10-16T20:57:03.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f6369d02640 1 -- 172.21.15.88:0/2889755662 learned_addr learned my addr 172.21.15.88:0/2889755662 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:57:03.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f6369501640 1 --2- 172.21.15.88:0/2889755662 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63640c41f0 0x7f63641f5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:03.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f6368d00640 1 --2- 172.21.15.88:0/2889755662 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63640c9bb0 0x7f6364158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:03.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f6369d02640 1 -- 172.21.15.88:0/2889755662 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63640c9bb0 msgr2=0x7f6364158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:03.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f6369d02640 1 --2- 172.21.15.88:0/2889755662 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63640c9bb0 0x7f6364158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f6369d02640 1 -- 172.21.15.88:0/2889755662 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63640c41f0 msgr2=0x7f63641f5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:03.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f6369d02640 1 --2- 172.21.15.88:0/2889755662 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63640c41f0 0x7f63641f5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f6369d02640 1 -- 172.21.15.88:0/2889755662 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6360009480 con 0x7f6364153770 2023-10-16T20:57:03.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f6369501640 1 --2- 172.21.15.88:0/2889755662 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63640c41f0 0x7f63641f5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:57:03.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f6369d02640 1 --2- 172.21.15.88:0/2889755662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6364153770 0x7f6364159520 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f635c007e60 tx=0x7f635c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:03.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f6359ffb640 1 -- 172.21.15.88:0/2889755662 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c014620 con 0x7f6364153770 2023-10-16T20:57:03.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f6359ffb640 1 -- 172.21.15.88:0/2889755662 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635c00ee90 con 0x7f6364153770 2023-10-16T20:57:03.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f6359ffb640 1 -- 172.21.15.88:0/2889755662 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c0149e0 con 0x7f6364153770 2023-10-16T20:57:03.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f636b765640 1 -- 172.21.15.88:0/2889755662 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63641f5c90 con 0x7f6364153770 2023-10-16T20:57:03.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f636b765640 1 -- 172.21.15.88:0/2889755662 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6364146650 con 0x7f6364153770 2023-10-16T20:57:03.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.233+0000 7f636b765640 1 -- 172.21.15.88:0/2889755662 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6364148260 con 0x7f6364153770 2023-10-16T20:57:03.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.237+0000 7f6359ffb640 1 -- 172.21.15.88:0/2889755662 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f635c014b40 con 0x7f6364153770 2023-10-16T20:57:03.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.237+0000 7f6359ffb640 1 --2- 172.21.15.88:0/2889755662 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f63480664a0 0x7f6348068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:03.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.237+0000 7f6359ffb640 1 -- 172.21.15.88:0/2889755662 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7f635c084da0 con 0x7f6364153770 2023-10-16T20:57:03.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.237+0000 7f6369501640 1 --2- 172.21.15.88:0/2889755662 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f63480664a0 0x7f6348068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:03.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.237+0000 7f6369501640 1 --2- 172.21.15.88:0/2889755662 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f63480664a0 0x7f6348068960 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f63600ae000 tx=0x7f63600a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:03.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.241+0000 7f6359ffb640 1 -- 172.21.15.88:0/2889755662 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6364148260 con 0x7f6364153770 2023-10-16T20:57:03.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.429+0000 7f636b765640 1 -- 172.21.15.88:0/2889755662 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6364148220 con 0x7f6364153770 2023-10-16T20:57:03.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.433+0000 7f6359ffb640 1 -- 172.21.15.88:0/2889755662 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+5782 (secure 0 0 0) 0x7f635c051b50 con 0x7f6364153770 2023-10-16T20:57:03.433 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:57:03.434 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:57:02.696291+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:57:03.434 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 107 2023-10-16T20:57:03.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.433+0000 7f636b765640 1 -- 172.21.15.88:0/2889755662 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f63480664a0 msgr2=0x7f6348068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:03.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.433+0000 7f636b765640 1 --2- 172.21.15.88:0/2889755662 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f63480664a0 0x7f6348068960 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f63600ae000 tx=0x7f63600a7000 comp rx=0 tx=0).stop 2023-10-16T20:57:03.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.433+0000 7f636b765640 1 -- 172.21.15.88:0/2889755662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6364153770 msgr2=0x7f6364159520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:03.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.433+0000 7f636b765640 1 --2- 172.21.15.88:0/2889755662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6364153770 0x7f6364159520 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f635c007e60 tx=0x7f635c00e4b0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.433+0000 7f636b765640 1 -- 172.21.15.88:0/2889755662 shutdown_connections 2023-10-16T20:57:03.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.433+0000 7f636b765640 1 --2- 172.21.15.88:0/2889755662 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f63480664a0 0x7f6348068960 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.433+0000 7f636b765640 1 --2- 172.21.15.88:0/2889755662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6364153770 0x7f6364159520 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.433+0000 7f636b765640 1 --2- 172.21.15.88:0/2889755662 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f63640c9bb0 0x7f6364158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.433+0000 7f636b765640 1 --2- 172.21.15.88:0/2889755662 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63640c41f0 0x7f63641f5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.433+0000 7f636b765640 1 -- 172.21.15.88:0/2889755662 >> 172.21.15.88:0/2889755662 conn(0x7f63640715c0 msgr2=0x7f6364146a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:03.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.433+0000 7f636b765640 1 -- 172.21.15.88:0/2889755662 shutdown_connections 2023-10-16T20:57:03.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.433+0000 7f636b765640 1 -- 172.21.15.88:0/2889755662 wait complete. 2023-10-16T20:57:03.455 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=3, replacing gid=5284 2023-10-16T20:57:03.455 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.c 2023-10-16T20:57:03.455 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-16T20:57:03.456 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-16T20:57:03.456 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-16T20:57:03.460 INFO:tasks.ceph.mds.c:Started 2023-10-16T20:57:03.460 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:57:03.566 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2023-10-16T20:57:03.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.633+0000 7ff3fbb2a640 1 Processor -- start 2023-10-16T20:57:03.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.633+0000 7ff3fbb2a640 1 -- start start 2023-10-16T20:57:03.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.633+0000 7ff3fbb2a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f40c9bb0 0x7ff3f40c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:03.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.633+0000 7ff3fbb2a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f4153b50 con 0x7ff3f4153770 2023-10-16T20:57:03.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.633+0000 7ff3fbb2a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f40cb2f0 con 0x7ff3f40c41f0 2023-10-16T20:57:03.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.633+0000 7ff3fbb2a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f40cc250 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.633+0000 7ff3f98c6640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff3f4153770 0x7ff3f4159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49190/0 (socket says 172.21.15.88:49190) 2023-10-16T20:57:03.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.633+0000 7ff3f98c6640 1 -- 172.21.15.88:0/1088033070 learned_addr learned my addr 172.21.15.88:0/1088033070 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:57:03.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.633+0000 7ff3f90c5640 1 --2- 172.21.15.88:0/1088033070 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f40c9bb0 0x7ff3f40c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:03.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.633+0000 7ff3f88c4640 1 -- 172.21.15.88:0/1088033070 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947385086 0 0) 0x7ff3f4153b50 con 0x7ff3f4153770 2023-10-16T20:57:03.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.633+0000 7ff3f88c4640 1 -- 172.21.15.88:0/1088033070 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e0003530 con 0x7ff3f4153770 2023-10-16T20:57:03.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.633+0000 7ff3f88c4640 1 -- 172.21.15.88:0/1088033070 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571118440 0 0) 0x7ff3f40cb2f0 con 0x7ff3f40c41f0 2023-10-16T20:57:03.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.633+0000 7ff3f88c4640 1 -- 172.21.15.88:0/1088033070 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f4153b50 con 0x7ff3f40c41f0 2023-10-16T20:57:03.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.633+0000 7ff3f90c5640 1 -- 172.21.15.88:0/1088033070 >> v1:172.21.15.191:6789/0 conn(0x7ff3f40c41f0 legacy=0x7ff3f40cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:03.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.633+0000 7ff3f90c5640 1 -- 172.21.15.88:0/1088033070 >> v1:172.21.15.88:6789/0 conn(0x7ff3f4153770 legacy=0x7ff3f4159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:03.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.633+0000 7ff3f90c5640 1 -- 172.21.15.88:0/1088033070 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f40cb5d0 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.637+0000 7ff3f90c5640 1 --2- 172.21.15.88:0/1088033070 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f40c9bb0 0x7ff3f40c3cb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff3e40097f0 tx=0x7ff3e401c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20015211ed6423e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:03.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.637+0000 7ff3f88c4640 1 -- 172.21.15.88:0/1088033070 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e40ac630 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.637+0000 7ff3f88c4640 1 -- 172.21.15.88:0/1088033070 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e40ac790 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.637+0000 7ff3f88c4640 1 -- 172.21.15.88:0/1088033070 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e40acb20 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.637+0000 7ff3fbb2a640 1 -- 172.21.15.88:0/1088033070 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f40c9bb0 msgr2=0x7ff3f40c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:03.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.637+0000 7ff3fbb2a640 1 --2- 172.21.15.88:0/1088033070 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f40c9bb0 0x7ff3f40c3cb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff3e40097f0 tx=0x7ff3e401c4c0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.637+0000 7ff3fbb2a640 1 -- 172.21.15.88:0/1088033070 shutdown_connections 2023-10-16T20:57:03.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.637+0000 7ff3fbb2a640 1 --2- 172.21.15.88:0/1088033070 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f40c9bb0 0x7ff3f40c3cb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.637+0000 7ff3fbb2a640 1 -- 172.21.15.88:0/1088033070 >> 172.21.15.88:0/1088033070 conn(0x7ff3f40715c0 msgr2=0x7ff3f40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:03.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3fbb2a640 1 -- 172.21.15.88:0/1088033070 shutdown_connections 2023-10-16T20:57:03.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3fbb2a640 1 -- 172.21.15.88:0/1088033070 wait complete. 2023-10-16T20:57:03.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3fbb2a640 1 Processor -- start 2023-10-16T20:57:03.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3fbb2a640 1 -- start start 2023-10-16T20:57:03.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3fbb2a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f40c41f0 0x7ff3f41f2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:03.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3fbb2a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3f40c9bb0 0x7ff3f4149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:03.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3fbb2a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3f4153770 0x7ff3f41455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:03.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3fbb2a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f41520b0 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3fbb2a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f41f15e0 con 0x7ff3f4153770 2023-10-16T20:57:03.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3fbb2a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f41f5600 con 0x7ff3f40c41f0 2023-10-16T20:57:03.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3f90c5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3f40c9bb0 0x7ff3f4149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:03.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3f90c5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3f40c9bb0 0x7ff3f4149970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49346/0 (socket says 172.21.15.88:49346) 2023-10-16T20:57:03.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3f90c5640 1 -- 172.21.15.88:0/4042786592 learned_addr learned my addr 172.21.15.88:0/4042786592 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:57:03.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3f98c6640 1 --2- 172.21.15.88:0/4042786592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f40c41f0 0x7ff3f41f2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:03.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3fa0c7640 1 --2- 172.21.15.88:0/4042786592 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3f4153770 0x7ff3f41455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:03.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3f90c5640 1 -- 172.21.15.88:0/4042786592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f40c41f0 msgr2=0x7ff3f41f2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:03.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3f90c5640 1 --2- 172.21.15.88:0/4042786592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f40c41f0 0x7ff3f41f2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3f90c5640 1 -- 172.21.15.88:0/4042786592 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3f4153770 msgr2=0x7ff3f41455e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:03.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3f90c5640 1 --2- 172.21.15.88:0/4042786592 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3f4153770 0x7ff3f41455e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3f90c5640 1 -- 172.21.15.88:0/4042786592 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e4009420 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3f98c6640 1 --2- 172.21.15.88:0/4042786592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f40c41f0 0x7ff3f41f2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:57:03.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3f90c5640 1 --2- 172.21.15.88:0/4042786592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3f40c9bb0 0x7ff3f4149970 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff3e40097f0 tx=0x7ff3e4019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:03.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3ea7fc640 1 -- 172.21.15.88:0/4042786592 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e40b5610 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3ea7fc640 1 -- 172.21.15.88:0/4042786592 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e40b5770 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3ea7fc640 1 -- 172.21.15.88:0/4042786592 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e40b5b20 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3fbb2a640 1 -- 172.21.15.88:0/4042786592 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f41f58e0 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.641+0000 7ff3fbb2a640 1 -- 172.21.15.88:0/4042786592 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f41f1b80 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.645+0000 7ff3fbb2a640 1 -- 172.21.15.88:0/4042786592 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f40c91f0 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.645+0000 7ff3ea7fc640 1 -- 172.21.15.88:0/4042786592 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff3e40b3040 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.645+0000 7ff3ea7fc640 1 --2- 172.21.15.88:0/4042786592 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff3d80664a0 0x7ff3d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:03.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.645+0000 7ff3ea7fc640 1 -- 172.21.15.88:0/4042786592 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7ff3e411e3b0 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.645+0000 7ff3f98c6640 1 --2- 172.21.15.88:0/4042786592 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff3d80664a0 0x7ff3d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:03.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.645+0000 7ff3f98c6640 1 --2- 172.21.15.88:0/4042786592 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff3d80664a0 0x7ff3d8068960 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0001010 tx=0x7ff3f0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:03.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.649+0000 7ff3ea7fc640 1 -- 172.21.15.88:0/4042786592 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff3e40eb0d0 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.837+0000 7ff3fbb2a640 1 -- 172.21.15.88:0/4042786592 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3f4074700 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.837+0000 7ff3ea7fc640 1 -- 172.21.15.88:0/4042786592 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6573 (secure 0 0 0) 0x7ff3e40ee6f0 con 0x7ff3f40c9bb0 2023-10-16T20:57:03.838 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:57:03.838 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6366,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:57:02.696291+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:57:03.838 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 108 2023-10-16T20:57:03.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.837+0000 7ff3fbb2a640 1 -- 172.21.15.88:0/4042786592 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff3d80664a0 msgr2=0x7ff3d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:03.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.837+0000 7ff3fbb2a640 1 --2- 172.21.15.88:0/4042786592 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff3d80664a0 0x7ff3d8068960 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0001010 tx=0x7ff3f0009040 comp rx=0 tx=0).stop 2023-10-16T20:57:03.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.837+0000 7ff3fbb2a640 1 -- 172.21.15.88:0/4042786592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3f40c9bb0 msgr2=0x7ff3f4149970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:03.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.837+0000 7ff3fbb2a640 1 --2- 172.21.15.88:0/4042786592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3f40c9bb0 0x7ff3f4149970 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff3e40097f0 tx=0x7ff3e4019260 comp rx=0 tx=0).stop 2023-10-16T20:57:03.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.837+0000 7ff3fbb2a640 1 -- 172.21.15.88:0/4042786592 shutdown_connections 2023-10-16T20:57:03.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.837+0000 7ff3fbb2a640 1 --2- 172.21.15.88:0/4042786592 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff3d80664a0 0x7ff3d8068960 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.837+0000 7ff3fbb2a640 1 --2- 172.21.15.88:0/4042786592 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff3f4153770 0x7ff3f41455e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.837+0000 7ff3fbb2a640 1 --2- 172.21.15.88:0/4042786592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3f40c9bb0 0x7ff3f4149970 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.837+0000 7ff3fbb2a640 1 --2- 172.21.15.88:0/4042786592 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff3f40c41f0 0x7ff3f41f2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:03.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.837+0000 7ff3fbb2a640 1 -- 172.21.15.88:0/4042786592 >> 172.21.15.88:0/4042786592 conn(0x7ff3f40715c0 msgr2=0x7ff3f40cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:03.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.837+0000 7ff3fbb2a640 1 -- 172.21.15.88:0/4042786592 shutdown_connections 2023-10-16T20:57:03.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:03.837+0000 7ff3fbb2a640 1 -- 172.21.15.88:0/4042786592 wait complete. 2023-10-16T20:57:03.858 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-16T20:57:03.858 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 39 secs before thrashing 2023-10-16T20:57:31.371 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:57:31.376 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:57:42.860 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:57:43.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.040+0000 7f83583d9640 1 Processor -- start 2023-10-16T20:57:43.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.040+0000 7f83583d9640 1 -- start start 2023-10-16T20:57:43.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.040+0000 7f83583d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f83500c9bb0 0x7f83500c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:43.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.040+0000 7f83583d9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8350153b50 con 0x7f83500c41f0 2023-10-16T20:57:43.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.040+0000 7f83583d9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83500cb2f0 con 0x7f8350153770 2023-10-16T20:57:43.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.040+0000 7f83583d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83500cc250 con 0x7f83500c9bb0 2023-10-16T20:57:43.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.040+0000 7f8356976640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f83500c41f0 0x7f83500cece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52436/0 (socket says 172.21.15.88:52436) 2023-10-16T20:57:43.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.040+0000 7f8356976640 1 -- 172.21.15.88:0/3355605082 learned_addr learned my addr 172.21.15.88:0/3355605082 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:57:43.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.040+0000 7f8355974640 1 --2- 172.21.15.88:0/3355605082 >> v2:172.21.15.88:3300/0 conn(0x7f83500c9bb0 0x7f83500c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:43.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.044+0000 7f8355974640 1 -- 172.21.15.88:0/3355605082 >> v1:172.21.15.191:6789/0 conn(0x7f8350153770 legacy=0x7f8350159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:43.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.044+0000 7f8355173640 1 -- 172.21.15.88:0/3355605082 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926408679 0 0) 0x7f8350153b50 con 0x7f83500c41f0 2023-10-16T20:57:43.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.044+0000 7f8355974640 1 -- 172.21.15.88:0/3355605082 >> v1:172.21.15.88:6789/0 conn(0x7f83500c41f0 legacy=0x7f83500cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:43.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.044+0000 7f8355974640 1 -- 172.21.15.88:0/3355605082 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83500cb5d0 con 0x7f83500c9bb0 2023-10-16T20:57:43.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.044+0000 7f8355974640 1 --2- 172.21.15.88:0/3355605082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83500c9bb0 0x7f83500c3cb0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f8340009740 tx=0x7f834001c2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01b199ec5f58d63 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:43.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.044+0000 7f8355173640 1 -- 172.21.15.88:0/3355605082 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83400ab600 con 0x7f83500c9bb0 2023-10-16T20:57:43.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.044+0000 7f8355173640 1 -- 172.21.15.88:0/3355605082 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83400a7050 con 0x7f83500c9bb0 2023-10-16T20:57:43.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.044+0000 7f8355173640 1 -- 172.21.15.88:0/3355605082 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83400ab980 con 0x7f83500c9bb0 2023-10-16T20:57:43.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f83583d9640 1 -- 172.21.15.88:0/3355605082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83500c9bb0 msgr2=0x7f83500c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:43.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f83583d9640 1 --2- 172.21.15.88:0/3355605082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83500c9bb0 0x7f83500c3cb0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f8340009740 tx=0x7f834001c2b0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f83583d9640 1 -- 172.21.15.88:0/3355605082 shutdown_connections 2023-10-16T20:57:43.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f83583d9640 1 --2- 172.21.15.88:0/3355605082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83500c9bb0 0x7f83500c3cb0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f83583d9640 1 -- 172.21.15.88:0/3355605082 >> 172.21.15.88:0/3355605082 conn(0x7f83500715c0 msgr2=0x7f83500719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:43.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f83583d9640 1 -- 172.21.15.88:0/3355605082 shutdown_connections 2023-10-16T20:57:43.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f83583d9640 1 -- 172.21.15.88:0/3355605082 wait complete. 2023-10-16T20:57:43.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f83583d9640 1 Processor -- start 2023-10-16T20:57:43.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f83583d9640 1 -- start start 2023-10-16T20:57:43.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f83583d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83500c41f0 0x7f8350140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:43.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f83583d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83500c9bb0 0x7f8350148b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:43.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f83583d9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8350153770 0x7f83500b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:43.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f83583d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8350151c40 con 0x7f83500c9bb0 2023-10-16T20:57:43.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f83583d9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83501483d0 con 0x7f8350153770 2023-10-16T20:57:43.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f83583d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8350147890 con 0x7f83500c41f0 2023-10-16T20:57:43.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f8355974640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83500c9bb0 0x7f8350148b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:43.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f8355974640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83500c9bb0 0x7f8350148b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59300/0 (socket says 172.21.15.88:59300) 2023-10-16T20:57:43.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.048+0000 7f8355974640 1 -- 172.21.15.88:0/3982859842 learned_addr learned my addr 172.21.15.88:0/3982859842 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:57:43.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.052+0000 7f8356976640 1 --2- 172.21.15.88:0/3982859842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8350153770 0x7f83500b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:43.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.052+0000 7f8356175640 1 --2- 172.21.15.88:0/3982859842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83500c41f0 0x7f8350140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:43.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.052+0000 7f8355974640 1 -- 172.21.15.88:0/3982859842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83500c41f0 msgr2=0x7f8350140cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:43.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.052+0000 7f8355974640 1 --2- 172.21.15.88:0/3982859842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83500c41f0 0x7f8350140cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.052+0000 7f8355974640 1 -- 172.21.15.88:0/3982859842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8350153770 msgr2=0x7f83500b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:43.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.052+0000 7f8355974640 1 --2- 172.21.15.88:0/3982859842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8350153770 0x7f83500b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.052+0000 7f8355974640 1 -- 172.21.15.88:0/3982859842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8340009420 con 0x7f83500c9bb0 2023-10-16T20:57:43.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.052+0000 7f8355974640 1 --2- 172.21.15.88:0/3982859842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83500c9bb0 0x7f8350148b40 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f8340018ec0 tx=0x7f834001aaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:43.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.052+0000 7f8346ffd640 1 -- 172.21.15.88:0/3982859842 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83400b7650 con 0x7f83500c9bb0 2023-10-16T20:57:43.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.052+0000 7f8346ffd640 1 -- 172.21.15.88:0/3982859842 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83400abe90 con 0x7f83500c9bb0 2023-10-16T20:57:43.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.052+0000 7f8346ffd640 1 -- 172.21.15.88:0/3982859842 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83400b7a10 con 0x7f83500c9bb0 2023-10-16T20:57:43.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.052+0000 7f83583d9640 1 -- 172.21.15.88:0/3982859842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8350142310 con 0x7f83500c9bb0 2023-10-16T20:57:43.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.052+0000 7f83583d9640 1 -- 172.21.15.88:0/3982859842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8350147e30 con 0x7f83500c9bb0 2023-10-16T20:57:43.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.052+0000 7f83583d9640 1 -- 172.21.15.88:0/3982859842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8350074700 con 0x7f83500c9bb0 2023-10-16T20:57:43.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.056+0000 7f8346ffd640 1 -- 172.21.15.88:0/3982859842 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f83400b7b70 con 0x7f83500c9bb0 2023-10-16T20:57:43.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.056+0000 7f8346ffd640 1 --2- 172.21.15.88:0/3982859842 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f83340664a0 0x7f8334068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:43.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.056+0000 7f8346ffd640 1 -- 172.21.15.88:0/3982859842 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7f834011ef10 con 0x7f83500c9bb0 2023-10-16T20:57:43.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.056+0000 7f8356175640 1 --2- 172.21.15.88:0/3982859842 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f83340664a0 0x7f8334068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:43.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.056+0000 7f8356175640 1 --2- 172.21.15.88:0/3982859842 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f83340664a0 0x7f8334068960 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f834c000fc0 tx=0x7f834c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:43.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.060+0000 7f8346ffd640 1 -- 172.21.15.88:0/3982859842 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f83400ebba0 con 0x7f83500c9bb0 2023-10-16T20:57:43.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.348+0000 7f83583d9640 1 -- 172.21.15.88:0/3982859842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83501481b0 con 0x7f83500c9bb0 2023-10-16T20:57:43.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.348+0000 7f8346ffd640 1 -- 172.21.15.88:0/3982859842 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+6580 (secure 0 0 0) 0x7f83400b0070 con 0x7f83500c9bb0 2023-10-16T20:57:43.347 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:57:43.347 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6366,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:57:23.715986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:57:43.347 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 111 2023-10-16T20:57:43.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.352+0000 7f83583d9640 1 -- 172.21.15.88:0/3982859842 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f83340664a0 msgr2=0x7f8334068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:43.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.352+0000 7f83583d9640 1 --2- 172.21.15.88:0/3982859842 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f83340664a0 0x7f8334068960 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f834c000fc0 tx=0x7f834c009040 comp rx=0 tx=0).stop 2023-10-16T20:57:43.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.352+0000 7f83583d9640 1 -- 172.21.15.88:0/3982859842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83500c9bb0 msgr2=0x7f8350148b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:43.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.352+0000 7f83583d9640 1 --2- 172.21.15.88:0/3982859842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83500c9bb0 0x7f8350148b40 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f8340018ec0 tx=0x7f834001aaf0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.352+0000 7f83583d9640 1 -- 172.21.15.88:0/3982859842 shutdown_connections 2023-10-16T20:57:43.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.352+0000 7f83583d9640 1 --2- 172.21.15.88:0/3982859842 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f83340664a0 0x7f8334068960 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.352+0000 7f83583d9640 1 --2- 172.21.15.88:0/3982859842 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8350153770 0x7f83500b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.352+0000 7f83583d9640 1 --2- 172.21.15.88:0/3982859842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83500c9bb0 0x7f8350148b40 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.352+0000 7f83583d9640 1 --2- 172.21.15.88:0/3982859842 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f83500c41f0 0x7f8350140cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.352+0000 7f83583d9640 1 -- 172.21.15.88:0/3982859842 >> 172.21.15.88:0/3982859842 conn(0x7f83500715c0 msgr2=0x7f8350152060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:43.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.352+0000 7f83583d9640 1 -- 172.21.15.88:0/3982859842 shutdown_connections 2023-10-16T20:57:43.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.352+0000 7f83583d9640 1 -- 172.21.15.88:0/3982859842 wait complete. 2023-10-16T20:57:43.378 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=4) 2023-10-16T20:57:43.379 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-16T20:57:43.379 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T20:57:43.394 INFO:tasks.ceph.mds.f:Stopped 2023-10-16T20:57:43.394 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:57:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.584+0000 7f9a7aeea640 1 Processor -- start 2023-10-16T20:57:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.584+0000 7f9a7aeea640 1 -- start start 2023-10-16T20:57:43.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.584+0000 7f9a7aeea640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9a740cece0 0x7f9a740cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:43.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.584+0000 7f9a7aeea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a74153b50 con 0x7f9a74153770 2023-10-16T20:57:43.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.584+0000 7f9a7aeea640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a740cb2f0 con 0x7f9a740c9bb0 2023-10-16T20:57:43.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.584+0000 7f9a7aeea640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a740cc250 con 0x7f9a740cece0 2023-10-16T20:57:43.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.584+0000 7f9a78c86640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9a74153770 0x7f9a74159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52448/0 (socket says 172.21.15.88:52448) 2023-10-16T20:57:43.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.584+0000 7f9a78c86640 1 -- 172.21.15.88:0/4284917799 learned_addr learned my addr 172.21.15.88:0/4284917799 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:57:43.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.584+0000 7f9a79487640 1 --2- 172.21.15.88:0/4284917799 >> v2:172.21.15.191:3300/0 conn(0x7f9a740cece0 0x7f9a740cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:43.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.584+0000 7f9a737fe640 1 -- 172.21.15.88:0/4284917799 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343675220 0 0) 0x7f9a74153b50 con 0x7f9a74153770 2023-10-16T20:57:43.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.584+0000 7f9a737fe640 1 -- 172.21.15.88:0/4284917799 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a64003530 con 0x7f9a74153770 2023-10-16T20:57:43.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.588+0000 7f9a79487640 1 -- 172.21.15.88:0/4284917799 >> v1:172.21.15.191:6789/0 conn(0x7f9a740c9bb0 legacy=0x7f9a740c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:43.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.588+0000 7f9a79487640 1 -- 172.21.15.88:0/4284917799 >> v1:172.21.15.88:6789/0 conn(0x7f9a74153770 legacy=0x7f9a74159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:43.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.588+0000 7f9a79487640 1 -- 172.21.15.88:0/4284917799 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a740cb5d0 con 0x7f9a740cece0 2023-10-16T20:57:43.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.592+0000 7f9a79487640 1 --2- 172.21.15.88:0/4284917799 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a740cece0 0x7f9a740cf130 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009d90 tx=0x7f9a6c01c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bec9c5f066c2b88 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:43.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.592+0000 7f9a737fe640 1 -- 172.21.15.88:0/4284917799 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c0a5920 con 0x7f9a740cece0 2023-10-16T20:57:43.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.592+0000 7f9a737fe640 1 -- 172.21.15.88:0/4284917799 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c0a5a80 con 0x7f9a740cece0 2023-10-16T20:57:43.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.592+0000 7f9a737fe640 1 -- 172.21.15.88:0/4284917799 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c0a5e10 con 0x7f9a740cece0 2023-10-16T20:57:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.592+0000 7f9a7aeea640 1 -- 172.21.15.88:0/4284917799 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a740cece0 msgr2=0x7f9a740cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.592+0000 7f9a7aeea640 1 --2- 172.21.15.88:0/4284917799 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a740cece0 0x7f9a740cf130 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009d90 tx=0x7f9a6c01c950 comp rx=0 tx=0).stop 2023-10-16T20:57:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.592+0000 7f9a7aeea640 1 -- 172.21.15.88:0/4284917799 shutdown_connections 2023-10-16T20:57:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.592+0000 7f9a7aeea640 1 --2- 172.21.15.88:0/4284917799 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a740cece0 0x7f9a740cf130 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.592+0000 7f9a7aeea640 1 -- 172.21.15.88:0/4284917799 >> 172.21.15.88:0/4284917799 conn(0x7f9a740715c0 msgr2=0x7f9a740719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:43.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.592+0000 7f9a7aeea640 1 -- 172.21.15.88:0/4284917799 shutdown_connections 2023-10-16T20:57:43.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.592+0000 7f9a7aeea640 1 -- 172.21.15.88:0/4284917799 wait complete. 2023-10-16T20:57:43.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.592+0000 7f9a7aeea640 1 Processor -- start 2023-10-16T20:57:43.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.592+0000 7f9a7aeea640 1 -- start start 2023-10-16T20:57:43.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a7aeea640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a740c9bb0 0x7f9a741498f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:43.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a7aeea640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9a740cece0 0x7f9a741418d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:43.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a7aeea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a74153770 0x7f9a74141e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a7aeea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7414feb0 con 0x7f9a74153770 2023-10-16T20:57:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a7aeea640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7414d9e0 con 0x7f9a740c9bb0 2023-10-16T20:57:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a7aeea640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7414a790 con 0x7f9a740cece0 2023-10-16T20:57:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a79487640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a74153770 0x7f9a74141e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a79487640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a74153770 0x7f9a74141e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59314/0 (socket says 172.21.15.88:59314) 2023-10-16T20:57:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a79487640 1 -- 172.21.15.88:0/3920900681 learned_addr learned my addr 172.21.15.88:0/3920900681 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:57:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a78c86640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a740c9bb0 0x7f9a741498f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a73fff640 1 --2- 172.21.15.88:0/3920900681 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9a740cece0 0x7f9a741418d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a79487640 1 -- 172.21.15.88:0/3920900681 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9a740cece0 msgr2=0x7f9a741418d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a79487640 1 --2- 172.21.15.88:0/3920900681 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9a740cece0 0x7f9a741418d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a79487640 1 -- 172.21.15.88:0/3920900681 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a740c9bb0 msgr2=0x7f9a741498f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a79487640 1 --2- 172.21.15.88:0/3920900681 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a740c9bb0 0x7f9a741498f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a79487640 1 -- 172.21.15.88:0/3920900681 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6c009a70 con 0x7f9a74153770 2023-10-16T20:57:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a79487640 1 --2- 172.21.15.88:0/3920900681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a74153770 0x7f9a74141e10 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0b0040 tx=0x7f9a6c0196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a717fa640 1 -- 172.21.15.88:0/3920900681 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c0ad850 con 0x7f9a74153770 2023-10-16T20:57:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a717fa640 1 -- 172.21.15.88:0/3920900681 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c0ad9b0 con 0x7f9a74153770 2023-10-16T20:57:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a7aeea640 1 -- 172.21.15.88:0/3920900681 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7414aa70 con 0x7f9a74153770 2023-10-16T20:57:43.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a717fa640 1 -- 172.21.15.88:0/3920900681 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c0add60 con 0x7f9a74153770 2023-10-16T20:57:43.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.596+0000 7f9a7aeea640 1 -- 172.21.15.88:0/3920900681 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a7414b030 con 0x7f9a74153770 2023-10-16T20:57:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.600+0000 7f9a7aeea640 1 -- 172.21.15.88:0/3920900681 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a74074700 con 0x7f9a74153770 2023-10-16T20:57:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.600+0000 7f9a717fa640 1 -- 172.21.15.88:0/3920900681 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9a6c0b4050 con 0x7f9a74153770 2023-10-16T20:57:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.600+0000 7f9a717fa640 1 --2- 172.21.15.88:0/3920900681 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9a580664a0 0x7f9a58068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.600+0000 7f9a717fa640 1 -- 172.21.15.88:0/3920900681 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7f9a6c11f1c0 con 0x7f9a74153770 2023-10-16T20:57:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.600+0000 7f9a78c86640 1 --2- 172.21.15.88:0/3920900681 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9a580664a0 0x7f9a58068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.600+0000 7f9a78c86640 1 --2- 172.21.15.88:0/3920900681 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9a580664a0 0x7f9a58068960 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9a680010f0 tx=0x7f9a68007bf0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:43.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.604+0000 7f9a717fa640 1 -- 172.21.15.88:0/3920900681 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9a6c0ebf00 con 0x7f9a74153770 2023-10-16T20:57:43.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.856+0000 7f9a7aeea640 1 -- 172.21.15.88:0/3920900681 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a7414db20 con 0x7f9a74153770 2023-10-16T20:57:43.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.856+0000 7f9a717fa640 1 -- 172.21.15.88:0/3920900681 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+6580 (secure 0 0 0) 0x7f9a6c0ef520 con 0x7f9a74153770 2023-10-16T20:57:43.855 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:57:43.855 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6366,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:57:23.715986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:57:43.856 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 111 2023-10-16T20:57:43.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.856+0000 7f9a7aeea640 1 -- 172.21.15.88:0/3920900681 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9a580664a0 msgr2=0x7f9a58068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:43.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.856+0000 7f9a7aeea640 1 --2- 172.21.15.88:0/3920900681 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9a580664a0 0x7f9a58068960 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9a680010f0 tx=0x7f9a68007bf0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.856+0000 7f9a7aeea640 1 -- 172.21.15.88:0/3920900681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a74153770 msgr2=0x7f9a74141e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:43.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.856+0000 7f9a7aeea640 1 --2- 172.21.15.88:0/3920900681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a74153770 0x7f9a74141e10 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0b0040 tx=0x7f9a6c0196f0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.860+0000 7f9a7aeea640 1 -- 172.21.15.88:0/3920900681 shutdown_connections 2023-10-16T20:57:43.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.860+0000 7f9a7aeea640 1 --2- 172.21.15.88:0/3920900681 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9a580664a0 0x7f9a58068960 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.860+0000 7f9a7aeea640 1 --2- 172.21.15.88:0/3920900681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a74153770 0x7f9a74141e10 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.860+0000 7f9a7aeea640 1 --2- 172.21.15.88:0/3920900681 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9a740cece0 0x7f9a741418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.860+0000 7f9a7aeea640 1 --2- 172.21.15.88:0/3920900681 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a740c9bb0 0x7f9a741498f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:43.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.860+0000 7f9a7aeea640 1 -- 172.21.15.88:0/3920900681 >> 172.21.15.88:0/3920900681 conn(0x7f9a740715c0 msgr2=0x7f9a74150f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:43.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.860+0000 7f9a7aeea640 1 -- 172.21.15.88:0/3920900681 shutdown_connections 2023-10-16T20:57:43.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:43.860+0000 7f9a7aeea640 1 -- 172.21.15.88:0/3920900681 wait complete. 2023-10-16T20:57:43.880 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-16T20:57:45.881 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:57:46.020 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.020+0000 7f96b1e9e640 1 Processor -- start 2023-10-16T20:57:46.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.020+0000 7f96b1e9e640 1 -- start start 2023-10-16T20:57:46.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.024+0000 7f96b1e9e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f96ac150280 0x7f96ac0c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:46.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.024+0000 7f96b1e9e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f96ac0c6bc0 0x7f96ac0c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:46.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.024+0000 7f96b1e9e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac0c5c60 con 0x7f96ac150660 2023-10-16T20:57:46.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.024+0000 7f96b1e9e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96ac0c5370 con 0x7f96ac150280 2023-10-16T20:57:46.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.024+0000 7f96b1e9e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac156d10 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.024+0000 7f96ab7fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f96ac150280 0x7f96ac0c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:46.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.024+0000 7f96abfff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f96ac0c6bc0 0x7f96ac0c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:46.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.024+0000 7f96ab7fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f96ac150280 0x7f96ac0c60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:55594/0 (socket says 172.21.15.88:55594) 2023-10-16T20:57:46.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.024+0000 7f96abfff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f96ac0c6bc0 0x7f96ac0c3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:49598/0 (socket says 172.21.15.88:49598) 2023-10-16T20:57:46.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.024+0000 7f96ab7fe640 1 -- 172.21.15.88:0/2842161164 learned_addr learned my addr 172.21.15.88:0/2842161164 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:57:46.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.024+0000 7f96aaffd640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f96ac150660 0x7f96ac06d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:45706/0 (socket says 172.21.15.88:45706) 2023-10-16T20:57:46.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.024+0000 7f96abfff640 1 -- 172.21.15.88:0/2842161164 >> v2:172.21.15.191:3300/0 conn(0x7f96ac150280 msgr2=0x7f96ac0c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:57:46.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.024+0000 7f96abfff640 1 --2- 172.21.15.88:0/2842161164 >> v2:172.21.15.191:3300/0 conn(0x7f96ac150280 0x7f96ac0c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:46.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.024+0000 7f96abfff640 1 -- 172.21.15.88:0/2842161164 >> v1:172.21.15.191:6789/0 conn(0x7f96ac150660 legacy=0x7f96ac06d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:46.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.024+0000 7f96abfff640 1 -- 172.21.15.88:0/2842161164 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac0c5610 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.024+0000 7f96aa7fc640 1 -- 172.21.15.88:0/2842161164 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853665731 0 0) 0x7f96ac0c5c60 con 0x7f96ac150660 2023-10-16T20:57:46.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.024+0000 7f96ab7fe640 1 --2- 172.21.15.88:0/2842161164 >> v2:172.21.15.191:3300/0 conn(0x7f96ac150280 0x7f96ac0c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:57:46.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.028+0000 7f96abfff640 1 --2- 172.21.15.88:0/2842161164 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f96ac0c6bc0 0x7f96ac0c3110 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f96a4009d90 tx=0x7f96a401d380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=683d5f75099829fd server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:46.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.028+0000 7f96aa7fc640 1 -- 172.21.15.88:0/2842161164 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a40a5a00 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.028+0000 7f96aa7fc640 1 -- 172.21.15.88:0/2842161164 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a40a5b60 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.028+0000 7f96aa7fc640 1 -- 172.21.15.88:0/2842161164 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a40ad440 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.028+0000 7f96b1e9e640 1 -- 172.21.15.88:0/2842161164 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f96ac0c6bc0 msgr2=0x7f96ac0c3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:46.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.028+0000 7f96b1e9e640 1 --2- 172.21.15.88:0/2842161164 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f96ac0c6bc0 0x7f96ac0c3110 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f96a4009d90 tx=0x7f96a401d380 comp rx=0 tx=0).stop 2023-10-16T20:57:46.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.028+0000 7f96b1e9e640 1 -- 172.21.15.88:0/2842161164 shutdown_connections 2023-10-16T20:57:46.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.028+0000 7f96b1e9e640 1 --2- 172.21.15.88:0/2842161164 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f96ac0c6bc0 0x7f96ac0c3110 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:46.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.028+0000 7f96b1e9e640 1 --2- 172.21.15.88:0/2842161164 >> v2:172.21.15.191:3300/0 conn(0x7f96ac150280 0x7f96ac0c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:46.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.028+0000 7f96b1e9e640 1 -- 172.21.15.88:0/2842161164 >> 172.21.15.88:0/2842161164 conn(0x7f96ac0cad50 msgr2=0x7f96ac0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:46.028 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.028+0000 7f96b1e9e640 1 -- 172.21.15.88:0/2842161164 shutdown_connections 2023-10-16T20:57:46.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.028+0000 7f96b1e9e640 1 -- 172.21.15.88:0/2842161164 wait complete. 2023-10-16T20:57:46.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.028+0000 7f96b1e9e640 1 Processor -- start 2023-10-16T20:57:46.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.028+0000 7f96b1e9e640 1 -- start start 2023-10-16T20:57:46.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96b1e9e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96ac0c6bc0 0x7f96ac146fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:46.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96b1e9e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96ac150280 0x7f96ac152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:46.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96b1e9e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f96ac150660 0x7f96ac150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:46.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96b1e9e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac152f40 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96b1e9e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac071510 con 0x7f96ac150280 2023-10-16T20:57:46.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96b1e9e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac0717f0 con 0x7f96ac150660 2023-10-16T20:57:46.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96ab7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96ac0c6bc0 0x7f96ac146fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:46.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96abfff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f96ac150660 0x7f96ac150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:46.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96ab7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96ac0c6bc0 0x7f96ac146fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59322/0 (socket says 172.21.15.88:59322) 2023-10-16T20:57:46.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96ab7fe640 1 -- 172.21.15.88:0/3742574649 learned_addr learned my addr 172.21.15.88:0/3742574649 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:57:46.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96aaffd640 1 --2- 172.21.15.88:0/3742574649 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96ac150280 0x7f96ac152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:46.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96ab7fe640 1 -- 172.21.15.88:0/3742574649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f96ac150660 msgr2=0x7f96ac150ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:46.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96ab7fe640 1 --2- 172.21.15.88:0/3742574649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f96ac150660 0x7f96ac150ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:46.033 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96ab7fe640 1 -- 172.21.15.88:0/3742574649 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96ac150280 msgr2=0x7f96ac152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:46.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96ab7fe640 1 --2- 172.21.15.88:0/3742574649 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96ac150280 0x7f96ac152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:46.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96ab7fe640 1 -- 172.21.15.88:0/3742574649 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c00b770 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96ab7fe640 1 --2- 172.21.15.88:0/3742574649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96ac0c6bc0 0x7f96ac146fc0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f969c00ed70 tx=0x7f969c00d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:46.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f9693fff640 1 -- 172.21.15.88:0/3742574649 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c010850 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f9693fff640 1 -- 172.21.15.88:0/3742574649 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969c0109b0 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f9693fff640 1 -- 172.21.15.88:0/3742574649 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c010d20 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96b1e9e640 1 -- 172.21.15.88:0/3742574649 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a4009a70 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96b1e9e640 1 -- 172.21.15.88:0/3742574649 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ac071db0 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f96b1e9e640 1 -- 172.21.15.88:0/3742574649 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ac074700 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.032+0000 7f9693fff640 1 -- 172.21.15.88:0/3742574649 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f969c00f3d0 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.036+0000 7f9693fff640 1 --2- 172.21.15.88:0/3742574649 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f96980664a0 0x7f9698068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:46.036 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.036+0000 7f9693fff640 1 -- 172.21.15.88:0/3742574649 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7f969c084e90 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.036+0000 7f96aaffd640 1 --2- 172.21.15.88:0/3742574649 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f96980664a0 0x7f9698068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:46.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.036+0000 7f96aaffd640 1 --2- 172.21.15.88:0/3742574649 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f96980664a0 0x7f9698068960 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0cb680 tx=0x7f96a0007df0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:46.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.040+0000 7f9693fff640 1 -- 172.21.15.88:0/3742574649 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f969c051cf0 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.220+0000 7f96b1e9e640 1 -- 172.21.15.88:0/3742574649 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96ac0c1650 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.220+0000 7f9693fff640 1 -- 172.21.15.88:0/3742574649 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+6580 (secure 0 0 0) 0x7f969c01d090 con 0x7f96ac0c6bc0 2023-10-16T20:57:46.218 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:57:46.219 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6366,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:57:23.715986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:57:46.219 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 111 2023-10-16T20:57:46.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.220+0000 7f96b1e9e640 1 -- 172.21.15.88:0/3742574649 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f96980664a0 msgr2=0x7f9698068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:46.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.220+0000 7f96b1e9e640 1 --2- 172.21.15.88:0/3742574649 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f96980664a0 0x7f9698068960 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0cb680 tx=0x7f96a0007df0 comp rx=0 tx=0).stop 2023-10-16T20:57:46.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.220+0000 7f96b1e9e640 1 -- 172.21.15.88:0/3742574649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96ac0c6bc0 msgr2=0x7f96ac146fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:46.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.220+0000 7f96b1e9e640 1 --2- 172.21.15.88:0/3742574649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96ac0c6bc0 0x7f96ac146fc0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f969c00ed70 tx=0x7f969c00d720 comp rx=0 tx=0).stop 2023-10-16T20:57:46.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.220+0000 7f96b1e9e640 1 -- 172.21.15.88:0/3742574649 shutdown_connections 2023-10-16T20:57:46.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.220+0000 7f96b1e9e640 1 --2- 172.21.15.88:0/3742574649 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f96980664a0 0x7f9698068960 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:46.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.220+0000 7f96b1e9e640 1 --2- 172.21.15.88:0/3742574649 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f96ac150660 0x7f96ac150ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:46.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.220+0000 7f96b1e9e640 1 --2- 172.21.15.88:0/3742574649 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f96ac150280 0x7f96ac152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:46.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.220+0000 7f96b1e9e640 1 --2- 172.21.15.88:0/3742574649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96ac0c6bc0 0x7f96ac146fc0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:46.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.220+0000 7f96b1e9e640 1 -- 172.21.15.88:0/3742574649 >> 172.21.15.88:0/3742574649 conn(0x7f96ac0cad50 msgr2=0x7f96ac0c3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:46.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.220+0000 7f96b1e9e640 1 -- 172.21.15.88:0/3742574649 shutdown_connections 2023-10-16T20:57:46.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:46.220+0000 7f96b1e9e640 1 -- 172.21.15.88:0/3742574649 wait complete. 2023-10-16T20:57:46.239 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-16T20:57:48.240 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:57:48.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.420+0000 7f6c88d80640 1 Processor -- start 2023-10-16T20:57:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.420+0000 7f6c88d80640 1 -- start start 2023-10-16T20:57:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.420+0000 7f6c88d80640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c84153770 0x7f6c84159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.420+0000 7f6c88d80640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8415a410 con 0x7f6c840cece0 2023-10-16T20:57:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.420+0000 7f6c88d80640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c84153b50 con 0x7f6c840c9bb0 2023-10-16T20:57:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.420+0000 7f6c88d80640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c840cc250 con 0x7f6c84153770 2023-10-16T20:57:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.420+0000 7f6c8259c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c84153770 0x7f6c84159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.420+0000 7f6c81d9b640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f6c840c9bb0 0x7f6c840c3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:45716/0 (socket says 172.21.15.88:45716) 2023-10-16T20:57:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.420+0000 7f6c81d9b640 1 -- 172.21.15.88:0/255004921 learned_addr learned my addr 172.21.15.88:0/255004921 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:57:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.420+0000 7f6c82d9d640 1 --1- 172.21.15.88:0/255004921 >> v1:172.21.15.88:6789/0 conn(0x7f6c840cece0 0x7f6c84163dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52452/0 (socket says 172.21.15.88:52452) 2023-10-16T20:57:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.420+0000 7f6c8159a640 1 -- 172.21.15.88:0/255004921 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677647495 0 0) 0x7f6c84153b50 con 0x7f6c840c9bb0 2023-10-16T20:57:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.420+0000 7f6c8259c640 1 -- 172.21.15.88:0/255004921 >> v1:172.21.15.191:6789/0 conn(0x7f6c840c9bb0 legacy=0x7f6c840c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.420+0000 7f6c8259c640 1 -- 172.21.15.88:0/255004921 >> v1:172.21.15.88:6789/0 conn(0x7f6c840cece0 legacy=0x7f6c84163dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.420+0000 7f6c8259c640 1 -- 172.21.15.88:0/255004921 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c840cb2f0 con 0x7f6c84153770 2023-10-16T20:57:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.424+0000 7f6c8259c640 1 --2- 172.21.15.88:0/255004921 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c84153770 0x7f6c84159ed0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6c740097d0 tx=0x7f6c7401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3b801259bcce20 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.424+0000 7f6c8159a640 1 -- 172.21.15.88:0/255004921 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c740ac410 con 0x7f6c84153770 2023-10-16T20:57:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.424+0000 7f6c8159a640 1 -- 172.21.15.88:0/255004921 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c740ac570 con 0x7f6c84153770 2023-10-16T20:57:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.424+0000 7f6c8159a640 1 -- 172.21.15.88:0/255004921 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c740ac900 con 0x7f6c84153770 2023-10-16T20:57:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.428+0000 7f6c88d80640 1 -- 172.21.15.88:0/255004921 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c84153770 msgr2=0x7f6c84159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.428+0000 7f6c88d80640 1 --2- 172.21.15.88:0/255004921 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c84153770 0x7f6c84159ed0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6c740097d0 tx=0x7f6c7401c4a0 comp rx=0 tx=0).stop 2023-10-16T20:57:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.428+0000 7f6c88d80640 1 -- 172.21.15.88:0/255004921 shutdown_connections 2023-10-16T20:57:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.428+0000 7f6c88d80640 1 --2- 172.21.15.88:0/255004921 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c84153770 0x7f6c84159ed0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:48.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.428+0000 7f6c88d80640 1 -- 172.21.15.88:0/255004921 >> 172.21.15.88:0/255004921 conn(0x7f6c840715c0 msgr2=0x7f6c840719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.428+0000 7f6c88d80640 1 -- 172.21.15.88:0/255004921 shutdown_connections 2023-10-16T20:57:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.428+0000 7f6c88d80640 1 -- 172.21.15.88:0/255004921 wait complete. 2023-10-16T20:57:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.428+0000 7f6c88d80640 1 Processor -- start 2023-10-16T20:57:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.428+0000 7f6c88d80640 1 -- start start 2023-10-16T20:57:48.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.428+0000 7f6c88d80640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c840c9bb0 0x7f6c84144cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:48.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c88d80640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c840cece0 0x7f6c840c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:48.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c88d80640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c84153770 0x7f6c840c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:48.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c88d80640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c841f5bd0 con 0x7f6c840cece0 2023-10-16T20:57:48.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c88d80640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c840c6180 con 0x7f6c840c9bb0 2023-10-16T20:57:48.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c88d80640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c840c6460 con 0x7f6c84153770 2023-10-16T20:57:48.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c82d9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c84153770 0x7f6c840c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:48.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c8259c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c840c9bb0 0x7f6c84144cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:48.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c82d9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c84153770 0x7f6c840c8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:49614/0 (socket says 172.21.15.88:49614) 2023-10-16T20:57:48.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c82d9d640 1 -- 172.21.15.88:0/2061352511 learned_addr learned my addr 172.21.15.88:0/2061352511 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:57:48.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c81d9b640 1 --2- 172.21.15.88:0/2061352511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c840cece0 0x7f6c840c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:48.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c82d9d640 1 -- 172.21.15.88:0/2061352511 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c840c9bb0 msgr2=0x7f6c84144cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:48.432 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c82d9d640 1 --2- 172.21.15.88:0/2061352511 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c840c9bb0 0x7f6c84144cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:48.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c82d9d640 1 -- 172.21.15.88:0/2061352511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c840cece0 msgr2=0x7f6c840c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:48.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c82d9d640 1 --2- 172.21.15.88:0/2061352511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c840cece0 0x7f6c840c7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:48.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c82d9d640 1 -- 172.21.15.88:0/2061352511 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c740093b0 con 0x7f6c84153770 2023-10-16T20:57:48.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c82d9d640 1 --2- 172.21.15.88:0/2061352511 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c84153770 0x7f6c840c8290 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00c7a0 tx=0x7f6c7c00cc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:48.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c7affd640 1 -- 172.21.15.88:0/2061352511 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c00a640 con 0x7f6c84153770 2023-10-16T20:57:48.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c7affd640 1 -- 172.21.15.88:0/2061352511 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7c00a7a0 con 0x7f6c84153770 2023-10-16T20:57:48.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c7affd640 1 -- 172.21.15.88:0/2061352511 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c00ab10 con 0x7f6c84153770 2023-10-16T20:57:48.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c88d80640 1 -- 172.21.15.88:0/2061352511 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8413e830 con 0x7f6c84153770 2023-10-16T20:57:48.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c88d80640 1 -- 172.21.15.88:0/2061352511 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c840c6740 con 0x7f6c84153770 2023-10-16T20:57:48.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.432+0000 7f6c88d80640 1 -- 172.21.15.88:0/2061352511 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8413ed50 con 0x7f6c84153770 2023-10-16T20:57:48.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.436+0000 7f6c7affd640 1 -- 172.21.15.88:0/2061352511 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6c7c00ac70 con 0x7f6c84153770 2023-10-16T20:57:48.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.436+0000 7f6c7affd640 1 --2- 172.21.15.88:0/2061352511 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6c6c0664f0 0x7f6c6c0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:48.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.436+0000 7f6c7affd640 1 -- 172.21.15.88:0/2061352511 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7f6c7c083c50 con 0x7f6c84153770 2023-10-16T20:57:48.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.436+0000 7f6c8259c640 1 --2- 172.21.15.88:0/2061352511 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6c6c0664f0 0x7f6c6c0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:48.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.436+0000 7f6c8259c640 1 --2- 172.21.15.88:0/2061352511 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6c6c0664f0 0x7f6c6c0689b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6c740ae000 tx=0x7f6c740a6040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:48.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.440+0000 7f6c7affd640 1 -- 172.21.15.88:0/2061352511 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6c8413ed50 con 0x7f6c84153770 2023-10-16T20:57:48.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.768+0000 7f6c88d80640 1 -- 172.21.15.88:0/2061352511 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c8413ed50 con 0x7f6c84153770 2023-10-16T20:57:48.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.768+0000 7f6c7affd640 1 -- 172.21.15.88:0/2061352511 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+6580 (secure 0 0 0) 0x7f6c8413ed50 con 0x7f6c84153770 2023-10-16T20:57:48.771 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:57:48.771 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6366,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:57:23.715986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:57:48.772 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 111 2023-10-16T20:57:48.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.772+0000 7f6c88d80640 1 -- 172.21.15.88:0/2061352511 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6c6c0664f0 msgr2=0x7f6c6c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:48.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.772+0000 7f6c88d80640 1 --2- 172.21.15.88:0/2061352511 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6c6c0664f0 0x7f6c6c0689b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6c740ae000 tx=0x7f6c740a6040 comp rx=0 tx=0).stop 2023-10-16T20:57:48.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.772+0000 7f6c88d80640 1 -- 172.21.15.88:0/2061352511 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c84153770 msgr2=0x7f6c840c8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:48.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.772+0000 7f6c88d80640 1 --2- 172.21.15.88:0/2061352511 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c84153770 0x7f6c840c8290 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00c7a0 tx=0x7f6c7c00cc60 comp rx=0 tx=0).stop 2023-10-16T20:57:48.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.772+0000 7f6c88d80640 1 -- 172.21.15.88:0/2061352511 shutdown_connections 2023-10-16T20:57:48.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.772+0000 7f6c88d80640 1 --2- 172.21.15.88:0/2061352511 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f6c6c0664f0 0x7f6c6c0689b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:48.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.772+0000 7f6c88d80640 1 --2- 172.21.15.88:0/2061352511 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f6c84153770 0x7f6c840c8290 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:48.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.772+0000 7f6c88d80640 1 --2- 172.21.15.88:0/2061352511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c840cece0 0x7f6c840c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:48.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.772+0000 7f6c88d80640 1 --2- 172.21.15.88:0/2061352511 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6c840c9bb0 0x7f6c84144cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:48.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.772+0000 7f6c88d80640 1 -- 172.21.15.88:0/2061352511 >> 172.21.15.88:0/2061352511 conn(0x7f6c840715c0 msgr2=0x7f6c840ccc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:48.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.776+0000 7f6c88d80640 1 -- 172.21.15.88:0/2061352511 shutdown_connections 2023-10-16T20:57:48.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:48.776+0000 7f6c88d80640 1 -- 172.21.15.88:0/2061352511 wait complete. 2023-10-16T20:57:48.802 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-16T20:57:50.802 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:57:50.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.952+0000 7fbb109d9640 1 Processor -- start 2023-10-16T20:57:50.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.952+0000 7fbb109d9640 1 -- start start 2023-10-16T20:57:50.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.952+0000 7fbb109d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb0c1517f0 0x7fbb0c0c6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:50.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.952+0000 7fbb109d9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbb0c0c7bd0 0x7fbb0c0c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:50.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.952+0000 7fbb109d9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0c15b740 con 0x7fbb0c151410 2023-10-16T20:57:50.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.952+0000 7fbb109d9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb0c0c4860 con 0x7fbb0c0c7bd0 2023-10-16T20:57:50.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.952+0000 7fbb109d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c15b460 con 0x7fbb0c1517f0 2023-10-16T20:57:50.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.952+0000 7fbb01d9b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb0c1517f0 0x7fbb0c0c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:50.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.952+0000 7fbb0ad9d640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fbb0c0c7bd0 0x7fbb0c0c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:50.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.952+0000 7fbb01d9b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb0c1517f0 0x7fbb0c0c6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:53038/0 (socket says 172.21.15.88:53038) 2023-10-16T20:57:50.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.952+0000 7fbb01d9b640 1 -- 172.21.15.88:0/1203520382 learned_addr learned my addr 172.21.15.88:0/1203520382 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:57:50.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.952+0000 7fbb0a59c640 1 --1- 172.21.15.88:0/1203520382 >> v1:172.21.15.191:6789/0 conn(0x7fbb0c151410 0x7fbb0c0c4150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:46988/0 (socket says 172.21.15.88:46988) 2023-10-16T20:57:50.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.952+0000 7fbb01d9b640 1 -- 172.21.15.88:0/1203520382 >> v2:172.21.15.191:3300/0 conn(0x7fbb0c0c7bd0 msgr2=0x7fbb0c0c6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:57:50.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.952+0000 7fbb01d9b640 1 --2- 172.21.15.88:0/1203520382 >> v2:172.21.15.191:3300/0 conn(0x7fbb0c0c7bd0 0x7fbb0c0c6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:50.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.952+0000 7fbb01d9b640 1 -- 172.21.15.88:0/1203520382 >> v1:172.21.15.191:6789/0 conn(0x7fbb0c151410 legacy=0x7fbb0c0c4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:50.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.952+0000 7fbb01d9b640 1 -- 172.21.15.88:0/1203520382 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0c06d780 con 0x7fbb0c1517f0 2023-10-16T20:57:50.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.952+0000 7fbb09d9b640 1 -- 172.21.15.88:0/1203520382 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1570340198 0 0) 0x7fbb0c15b740 con 0x7fbb0c151410 2023-10-16T20:57:50.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.956+0000 7fbb01d9b640 1 --2- 172.21.15.88:0/1203520382 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb0c1517f0 0x7fbb0c0c6840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbafc009870 tx=0x7fbafc01c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1117ea54add34d1c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:50.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.956+0000 7fbb09d9b640 1 -- 172.21.15.88:0/1203520382 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc0aa670 con 0x7fbb0c1517f0 2023-10-16T20:57:50.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.956+0000 7fbb09d9b640 1 -- 172.21.15.88:0/1203520382 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbafc0aa7d0 con 0x7fbb0c1517f0 2023-10-16T20:57:50.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.956+0000 7fbb09d9b640 1 -- 172.21.15.88:0/1203520382 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc0aab60 con 0x7fbb0c1517f0 2023-10-16T20:57:50.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.956+0000 7fbb109d9640 1 -- 172.21.15.88:0/1203520382 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb0c1517f0 msgr2=0x7fbb0c0c6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:50.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.956+0000 7fbb109d9640 1 --2- 172.21.15.88:0/1203520382 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb0c1517f0 0x7fbb0c0c6840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbafc009870 tx=0x7fbafc01c510 comp rx=0 tx=0).stop 2023-10-16T20:57:50.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.956+0000 7fbb109d9640 1 -- 172.21.15.88:0/1203520382 shutdown_connections 2023-10-16T20:57:50.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.956+0000 7fbb109d9640 1 --2- 172.21.15.88:0/1203520382 >> v2:172.21.15.191:3300/0 conn(0x7fbb0c0c7bd0 0x7fbb0c0c6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:50.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.956+0000 7fbb109d9640 1 --2- 172.21.15.88:0/1203520382 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb0c1517f0 0x7fbb0c0c6840 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:50.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.956+0000 7fbb109d9640 1 -- 172.21.15.88:0/1203520382 >> 172.21.15.88:0/1203520382 conn(0x7fbb0c071a60 msgr2=0x7fbb0c071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:50.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.956+0000 7fbb109d9640 1 -- 172.21.15.88:0/1203520382 shutdown_connections 2023-10-16T20:57:50.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.956+0000 7fbb109d9640 1 -- 172.21.15.88:0/1203520382 wait complete. 2023-10-16T20:57:50.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb109d9640 1 Processor -- start 2023-10-16T20:57:50.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb109d9640 1 -- start start 2023-10-16T20:57:50.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb109d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb0c0c7bd0 0x7fbb0c13a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:50.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb109d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb0c151410 0x7fbb0c145660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:50.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb109d9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb0c13f3c0 0x7fbb0c1414d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:50.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb109d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c13da50 con 0x7fbb0c151410 2023-10-16T20:57:50.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb109d9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c1509c0 con 0x7fbb0c13f3c0 2023-10-16T20:57:50.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb109d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c13d800 con 0x7fbb0c0c7bd0 2023-10-16T20:57:50.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb01d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb0c151410 0x7fbb0c145660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:50.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb01d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb0c151410 0x7fbb0c145660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60462/0 (socket says 172.21.15.88:60462) 2023-10-16T20:57:50.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb01d9b640 1 -- 172.21.15.88:0/2596203737 learned_addr learned my addr 172.21.15.88:0/2596203737 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:57:50.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb0a59c640 1 --2- 172.21.15.88:0/2596203737 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb0c0c7bd0 0x7fbb0c13a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:50.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb0ad9d640 1 --2- 172.21.15.88:0/2596203737 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb0c13f3c0 0x7fbb0c1414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:50.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb01d9b640 1 -- 172.21.15.88:0/2596203737 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb0c0c7bd0 msgr2=0x7fbb0c13a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:50.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb01d9b640 1 --2- 172.21.15.88:0/2596203737 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb0c0c7bd0 0x7fbb0c13a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:50.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb01d9b640 1 -- 172.21.15.88:0/2596203737 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb0c13f3c0 msgr2=0x7fbb0c1414d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:50.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb01d9b640 1 --2- 172.21.15.88:0/2596203737 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb0c13f3c0 0x7fbb0c1414d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:50.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb01d9b640 1 -- 172.21.15.88:0/2596203737 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc0094d0 con 0x7fbb0c151410 2023-10-16T20:57:50.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb0ad9d640 1 --2- 172.21.15.88:0/2596203737 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb0c13f3c0 0x7fbb0c1414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:57:50.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb01d9b640 1 --2- 172.21.15.88:0/2596203737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb0c151410 0x7fbb0c145660 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fbafc01abc0 tx=0x7fbafc01ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:50.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb037fe640 1 -- 172.21.15.88:0/2596203737 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc0b3800 con 0x7fbb0c151410 2023-10-16T20:57:50.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb037fe640 1 -- 172.21.15.88:0/2596203737 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbafc0b3980 con 0x7fbb0c151410 2023-10-16T20:57:50.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb109d9640 1 -- 172.21.15.88:0/2596203737 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0c13f870 con 0x7fbb0c151410 2023-10-16T20:57:50.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb037fe640 1 -- 172.21.15.88:0/2596203737 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc0b3d30 con 0x7fbb0c151410 2023-10-16T20:57:50.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.960+0000 7fbb109d9640 1 -- 172.21.15.88:0/2596203737 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb0c1506e0 con 0x7fbb0c151410 2023-10-16T20:57:50.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.964+0000 7fbb109d9640 1 -- 172.21.15.88:0/2596203737 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb0c074700 con 0x7fbb0c151410 2023-10-16T20:57:50.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.964+0000 7fbb037fe640 1 -- 172.21.15.88:0/2596203737 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbafc0c6390 con 0x7fbb0c151410 2023-10-16T20:57:50.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.964+0000 7fbb037fe640 1 --2- 172.21.15.88:0/2596203737 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbaec0664a0 0x7fbaec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:50.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.964+0000 7fbb037fe640 1 -- 172.21.15.88:0/2596203737 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7fbafc11f080 con 0x7fbb0c151410 2023-10-16T20:57:50.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.964+0000 7fbb0a59c640 1 --2- 172.21.15.88:0/2596203737 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbaec0664a0 0x7fbaec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:50.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.964+0000 7fbb0a59c640 1 --2- 172.21.15.88:0/2596203737 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbaec0664a0 0x7fbaec068960 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4007fd0 tx=0x7fbaf4007d60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:50.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:50.968+0000 7fbb037fe640 1 -- 172.21.15.88:0/2596203737 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbafc0ebee0 con 0x7fbb0c151410 2023-10-16T20:57:51.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:51.148+0000 7fbb109d9640 1 -- 172.21.15.88:0/2596203737 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb0c071ee0 con 0x7fbb0c151410 2023-10-16T20:57:51.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:51.148+0000 7fbb037fe640 1 -- 172.21.15.88:0/2596203737 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+6580 (secure 0 0 0) 0x7fbafc0aa630 con 0x7fbb0c151410 2023-10-16T20:57:51.149 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:57:51.149 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6366,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:57:23.715986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:57:51.150 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 111 2023-10-16T20:57:51.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:51.152+0000 7fbb109d9640 1 -- 172.21.15.88:0/2596203737 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbaec0664a0 msgr2=0x7fbaec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:51.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:51.152+0000 7fbb109d9640 1 --2- 172.21.15.88:0/2596203737 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbaec0664a0 0x7fbaec068960 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4007fd0 tx=0x7fbaf4007d60 comp rx=0 tx=0).stop 2023-10-16T20:57:51.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:51.152+0000 7fbb109d9640 1 -- 172.21.15.88:0/2596203737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb0c151410 msgr2=0x7fbb0c145660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:51.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:51.152+0000 7fbb109d9640 1 --2- 172.21.15.88:0/2596203737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb0c151410 0x7fbb0c145660 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fbafc01abc0 tx=0x7fbafc01ad50 comp rx=0 tx=0).stop 2023-10-16T20:57:51.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:51.152+0000 7fbb109d9640 1 -- 172.21.15.88:0/2596203737 shutdown_connections 2023-10-16T20:57:51.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:51.152+0000 7fbb109d9640 1 --2- 172.21.15.88:0/2596203737 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbaec0664a0 0x7fbaec068960 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:51.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:51.152+0000 7fbb109d9640 1 --2- 172.21.15.88:0/2596203737 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbb0c13f3c0 0x7fbb0c1414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:51.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:51.152+0000 7fbb109d9640 1 --2- 172.21.15.88:0/2596203737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb0c151410 0x7fbb0c145660 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:51.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:51.152+0000 7fbb109d9640 1 --2- 172.21.15.88:0/2596203737 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbb0c0c7bd0 0x7fbb0c13a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:51.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:51.152+0000 7fbb109d9640 1 -- 172.21.15.88:0/2596203737 >> 172.21.15.88:0/2596203737 conn(0x7fbb0c071a60 msgr2=0x7fbb0c147880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:51.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:51.152+0000 7fbb109d9640 1 -- 172.21.15.88:0/2596203737 shutdown_connections 2023-10-16T20:57:51.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:51.152+0000 7fbb109d9640 1 -- 172.21.15.88:0/2596203737 wait complete. 2023-10-16T20:57:51.169 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-16T20:57:53.170 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:57:53.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.312+0000 7f225e33e640 1 Processor -- start 2023-10-16T20:57:53.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.312+0000 7f225e33e640 1 -- start start 2023-10-16T20:57:53.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.312+0000 7f225e33e640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f225815a2a0 0x7f22580ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:53.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.312+0000 7f225e33e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225815cb20 0x7f22580cee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:53.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.312+0000 7f225e33e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22580ca610 con 0x7f22581538e0 2023-10-16T20:57:53.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.312+0000 7f225e33e640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22581595b0 con 0x7f225815a2a0 2023-10-16T20:57:53.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.312+0000 7f225e33e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225815d540 con 0x7f225815cb20 2023-10-16T20:57:53.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.312+0000 7f2257fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f22581538e0 0x7f22580c3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:46990/0 (socket says 172.21.15.88:46990) 2023-10-16T20:57:53.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.312+0000 7f22577fe640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f225815a2a0 0x7f22580ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:53.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.312+0000 7f2257fff640 1 -- 172.21.15.88:0/1276617867 learned_addr learned my addr 172.21.15.88:0/1276617867 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:57:53.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.312+0000 7f225c8db640 1 --2- 172.21.15.88:0/1276617867 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225815cb20 0x7f22580cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:53.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.312+0000 7f2256ffd640 1 -- 172.21.15.88:0/1276617867 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689510962 0 0) 0x7f22580ca610 con 0x7f22581538e0 2023-10-16T20:57:53.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.312+0000 7f22577fe640 1 -- 172.21.15.88:0/1276617867 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225815cb20 msgr2=0x7f22580cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:57:53.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.312+0000 7f22577fe640 1 --2- 172.21.15.88:0/1276617867 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225815cb20 0x7f22580cee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:53.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.312+0000 7f22577fe640 1 -- 172.21.15.88:0/1276617867 >> v1:172.21.15.191:6789/0 conn(0x7f22581538e0 legacy=0x7f22580c3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:53.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.312+0000 7f22577fe640 1 -- 172.21.15.88:0/1276617867 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225815d820 con 0x7f225815a2a0 2023-10-16T20:57:53.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.312+0000 7f225c8db640 1 --2- 172.21.15.88:0/1276617867 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225815cb20 0x7f22580cee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:57:53.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.316+0000 7f22577fe640 1 --2- 172.21.15.88:0/1276617867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f225815a2a0 0x7f22580ca0d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f224c0097c0 tx=0x7f224c01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61bfeec9f8adcf41 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:53.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.316+0000 7f2256ffd640 1 -- 172.21.15.88:0/1276617867 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c0ab600 con 0x7f225815a2a0 2023-10-16T20:57:53.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.316+0000 7f2256ffd640 1 -- 172.21.15.88:0/1276617867 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224c0a7050 con 0x7f225815a2a0 2023-10-16T20:57:53.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.316+0000 7f2256ffd640 1 -- 172.21.15.88:0/1276617867 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c0ab980 con 0x7f225815a2a0 2023-10-16T20:57:53.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.316+0000 7f225e33e640 1 -- 172.21.15.88:0/1276617867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f225815a2a0 msgr2=0x7f22580ca0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:53.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.316+0000 7f225e33e640 1 --2- 172.21.15.88:0/1276617867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f225815a2a0 0x7f22580ca0d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f224c0097c0 tx=0x7f224c01c480 comp rx=0 tx=0).stop 2023-10-16T20:57:53.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.316+0000 7f225e33e640 1 -- 172.21.15.88:0/1276617867 shutdown_connections 2023-10-16T20:57:53.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.316+0000 7f225e33e640 1 --2- 172.21.15.88:0/1276617867 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225815cb20 0x7f22580cee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:53.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.316+0000 7f225e33e640 1 --2- 172.21.15.88:0/1276617867 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f225815a2a0 0x7f22580ca0d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:53.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.316+0000 7f225e33e640 1 -- 172.21.15.88:0/1276617867 >> 172.21.15.88:0/1276617867 conn(0x7f225806bd50 msgr2=0x7f225806c160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:53.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.316+0000 7f225e33e640 1 -- 172.21.15.88:0/1276617867 shutdown_connections 2023-10-16T20:57:53.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.316+0000 7f225e33e640 1 -- 172.21.15.88:0/1276617867 wait complete. 2023-10-16T20:57:53.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f225e33e640 1 Processor -- start 2023-10-16T20:57:53.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f225e33e640 1 -- start start 2023-10-16T20:57:53.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f225e33e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22581538e0 0x7f22580cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:53.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f225e33e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225815a2a0 0x7f2258147bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:53.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f225e33e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225815cb20 0x7f22580b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:53.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f225e33e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2258151d20 con 0x7f225815a2a0 2023-10-16T20:57:53.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f225e33e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22581f65d0 con 0x7f22581538e0 2023-10-16T20:57:53.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f225e33e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2258152b40 con 0x7f225815cb20 2023-10-16T20:57:53.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f22577fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225815a2a0 0x7f2258147bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:53.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f22577fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225815a2a0 0x7f2258147bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60464/0 (socket says 172.21.15.88:60464) 2023-10-16T20:57:53.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f22577fe640 1 -- 172.21.15.88:0/2059148542 learned_addr learned my addr 172.21.15.88:0/2059148542 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:57:53.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f2257fff640 1 --2- 172.21.15.88:0/2059148542 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22581538e0 0x7f22580cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:53.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f225c8db640 1 --2- 172.21.15.88:0/2059148542 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225815cb20 0x7f22580b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:53.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f22577fe640 1 -- 172.21.15.88:0/2059148542 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225815cb20 msgr2=0x7f22580b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:53.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f22577fe640 1 --2- 172.21.15.88:0/2059148542 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225815cb20 0x7f22580b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:53.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f22577fe640 1 -- 172.21.15.88:0/2059148542 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22581538e0 msgr2=0x7f22580cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:53.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f22577fe640 1 --2- 172.21.15.88:0/2059148542 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22581538e0 0x7f22580cb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:53.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f22577fe640 1 -- 172.21.15.88:0/2059148542 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2250009a70 con 0x7f225815a2a0 2023-10-16T20:57:53.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f2257fff640 1 --2- 172.21.15.88:0/2059148542 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22581538e0 0x7f22580cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:57:53.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f22577fe640 1 --2- 172.21.15.88:0/2059148542 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225815a2a0 0x7f2258147bb0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f224c0ae040 tx=0x7f224c01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:53.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f2254ff9640 1 -- 172.21.15.88:0/2059148542 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c006e20 con 0x7f225815a2a0 2023-10-16T20:57:53.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f2254ff9640 1 -- 172.21.15.88:0/2059148542 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224c0b7450 con 0x7f225815a2a0 2023-10-16T20:57:53.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f2254ff9640 1 -- 172.21.15.88:0/2059148542 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c0b7800 con 0x7f225815a2a0 2023-10-16T20:57:53.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f225e33e640 1 -- 172.21.15.88:0/2059148542 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224c009420 con 0x7f225815a2a0 2023-10-16T20:57:53.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f225e33e640 1 -- 172.21.15.88:0/2059148542 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22580cd610 con 0x7f225815a2a0 2023-10-16T20:57:53.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.320+0000 7f225e33e640 1 -- 172.21.15.88:0/2059148542 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2258074700 con 0x7f225815a2a0 2023-10-16T20:57:53.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.324+0000 7f2254ff9640 1 -- 172.21.15.88:0/2059148542 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f224c0b7960 con 0x7f225815a2a0 2023-10-16T20:57:53.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.324+0000 7f2254ff9640 1 --2- 172.21.15.88:0/2059148542 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f22440664d0 0x7f2244068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:53.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.324+0000 7f2254ff9640 1 -- 172.21.15.88:0/2059148542 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7f224c11ec10 con 0x7f225815a2a0 2023-10-16T20:57:53.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.324+0000 7f2257fff640 1 --2- 172.21.15.88:0/2059148542 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f22440664d0 0x7f2244068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:53.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.324+0000 7f2257fff640 1 --2- 172.21.15.88:0/2059148542 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f22440664d0 0x7f2244068990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2248007fd0 tx=0x7f2248007c20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:53.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.328+0000 7f2254ff9640 1 -- 172.21.15.88:0/2059148542 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f224c0eba60 con 0x7f225815a2a0 2023-10-16T20:57:53.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.520+0000 7f225e33e640 1 -- 172.21.15.88:0/2059148542 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2258152c80 con 0x7f225815a2a0 2023-10-16T20:57:53.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.520+0000 7f2254ff9640 1 -- 172.21.15.88:0/2059148542 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+6580 (secure 0 0 0) 0x7f224c0ef080 con 0x7f225815a2a0 2023-10-16T20:57:53.521 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:57:53.521 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6366,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:57:23.715986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:57:53.522 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 111 2023-10-16T20:57:53.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.524+0000 7f225e33e640 1 -- 172.21.15.88:0/2059148542 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f22440664d0 msgr2=0x7f2244068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:53.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.524+0000 7f225e33e640 1 --2- 172.21.15.88:0/2059148542 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f22440664d0 0x7f2244068990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2248007fd0 tx=0x7f2248007c20 comp rx=0 tx=0).stop 2023-10-16T20:57:53.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.524+0000 7f225e33e640 1 -- 172.21.15.88:0/2059148542 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225815a2a0 msgr2=0x7f2258147bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:53.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.524+0000 7f225e33e640 1 --2- 172.21.15.88:0/2059148542 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225815a2a0 0x7f2258147bb0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f224c0ae040 tx=0x7f224c01acc0 comp rx=0 tx=0).stop 2023-10-16T20:57:53.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.524+0000 7f225e33e640 1 -- 172.21.15.88:0/2059148542 shutdown_connections 2023-10-16T20:57:53.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.524+0000 7f225e33e640 1 --2- 172.21.15.88:0/2059148542 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f22440664d0 0x7f2244068990 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:53.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.524+0000 7f225e33e640 1 --2- 172.21.15.88:0/2059148542 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f225815cb20 0x7f22580b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:53.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.524+0000 7f225e33e640 1 --2- 172.21.15.88:0/2059148542 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f225815a2a0 0x7f2258147bb0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:53.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.524+0000 7f225e33e640 1 --2- 172.21.15.88:0/2059148542 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22581538e0 0x7f22580cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:53.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.524+0000 7f225e33e640 1 -- 172.21.15.88:0/2059148542 >> 172.21.15.88:0/2059148542 conn(0x7f225806bd50 msgr2=0x7f22581586f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:53.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.524+0000 7f225e33e640 1 -- 172.21.15.88:0/2059148542 shutdown_connections 2023-10-16T20:57:53.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:53.524+0000 7f225e33e640 1 -- 172.21.15.88:0/2059148542 wait complete. 2023-10-16T20:57:53.544 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-16T20:57:55.545 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:57:55.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.700+0000 7f853cca9640 1 Processor -- start 2023-10-16T20:57:55.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.700+0000 7f853cca9640 1 -- start start 2023-10-16T20:57:55.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.704+0000 7f853cca9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8538153770 0x7f8538159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:55.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.704+0000 7f853cca9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f85380c9bb0 0x7f85380c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:55.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.704+0000 7f853cca9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853815a410 con 0x7f85380c41f0 2023-10-16T20:57:55.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.704+0000 7f853cca9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85380cc250 con 0x7f8538153770 2023-10-16T20:57:55.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.704+0000 7f853cca9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85380cc670 con 0x7f85380c9bb0 2023-10-16T20:57:55.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.704+0000 7f853659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8538153770 0x7f8538159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.704+0000 7f853659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8538153770 0x7f8538159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60470/0 (socket says 172.21.15.88:60470) 2023-10-16T20:57:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.704+0000 7f8536d9d640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f85380c41f0 0x7f85380cece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:46998/0 (socket says 172.21.15.88:46998) 2023-10-16T20:57:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.704+0000 7f853659c640 1 -- 172.21.15.88:0/1892471745 learned_addr learned my addr 172.21.15.88:0/1892471745 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:57:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.704+0000 7f8535d9b640 1 --2- 172.21.15.88:0/1892471745 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f85380c9bb0 0x7f85380c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:55.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.704+0000 7f853659c640 1 -- 172.21.15.88:0/1892471745 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f85380c9bb0 msgr2=0x7f85380c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:57:55.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.704+0000 7f853659c640 1 --2- 172.21.15.88:0/1892471745 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f85380c9bb0 0x7f85380c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:55.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.704+0000 7f853559a640 1 -- 172.21.15.88:0/1892471745 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642918882 0 0) 0x7f853815a410 con 0x7f85380c41f0 2023-10-16T20:57:55.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.704+0000 7f853659c640 1 -- 172.21.15.88:0/1892471745 >> v1:172.21.15.191:6789/0 conn(0x7f85380c41f0 legacy=0x7f85380cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:55.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.704+0000 7f853659c640 1 -- 172.21.15.88:0/1892471745 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8538153b50 con 0x7f8538153770 2023-10-16T20:57:55.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.708+0000 7f853659c640 1 --2- 172.21.15.88:0/1892471745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8538153770 0x7f8538159ed0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f8528006fd0 tx=0x7f852801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a146238785385 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:55.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.708+0000 7f853559a640 1 -- 172.21.15.88:0/1892471745 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85280aa680 con 0x7f8538153770 2023-10-16T20:57:55.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.708+0000 7f853559a640 1 -- 172.21.15.88:0/1892471745 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85280aa7e0 con 0x7f8538153770 2023-10-16T20:57:55.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.708+0000 7f853559a640 1 -- 172.21.15.88:0/1892471745 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85280aab70 con 0x7f8538153770 2023-10-16T20:57:55.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.708+0000 7f853cca9640 1 -- 172.21.15.88:0/1892471745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8538153770 msgr2=0x7f8538159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:55.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.708+0000 7f853cca9640 1 --2- 172.21.15.88:0/1892471745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8538153770 0x7f8538159ed0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f8528006fd0 tx=0x7f852801c520 comp rx=0 tx=0).stop 2023-10-16T20:57:55.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.708+0000 7f853cca9640 1 -- 172.21.15.88:0/1892471745 shutdown_connections 2023-10-16T20:57:55.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.708+0000 7f853cca9640 1 --2- 172.21.15.88:0/1892471745 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f85380c9bb0 0x7f85380c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:55.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.708+0000 7f853cca9640 1 --2- 172.21.15.88:0/1892471745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8538153770 0x7f8538159ed0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:55.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.708+0000 7f853cca9640 1 -- 172.21.15.88:0/1892471745 >> 172.21.15.88:0/1892471745 conn(0x7f85380715c0 msgr2=0x7f85380719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:55.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.708+0000 7f853cca9640 1 -- 172.21.15.88:0/1892471745 shutdown_connections 2023-10-16T20:57:55.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.708+0000 7f853cca9640 1 -- 172.21.15.88:0/1892471745 wait complete. 2023-10-16T20:57:55.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.708+0000 7f853cca9640 1 Processor -- start 2023-10-16T20:57:55.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.708+0000 7f853cca9640 1 -- start start 2023-10-16T20:57:55.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f853cca9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85380c41f0 0x7f85380ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:55.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f853cca9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85380c9bb0 0x7f853815c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:55.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f853cca9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8538153770 0x7f853815c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f853cca9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85381f2420 con 0x7f85380c9bb0 2023-10-16T20:57:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f853cca9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85380c8ec0 con 0x7f85380c41f0 2023-10-16T20:57:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f853cca9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853815ab60 con 0x7f8538153770 2023-10-16T20:57:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f8535d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85380c9bb0 0x7f853815c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:55.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f8535d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85380c9bb0 0x7f853815c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60472/0 (socket says 172.21.15.88:60472) 2023-10-16T20:57:55.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f8535d9b640 1 -- 172.21.15.88:0/2297095704 learned_addr learned my addr 172.21.15.88:0/2297095704 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:57:55.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f853659c640 1 --2- 172.21.15.88:0/2297095704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85380c41f0 0x7f85380ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:55.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f8536d9d640 1 --2- 172.21.15.88:0/2297095704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8538153770 0x7f853815c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:55.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f8535d9b640 1 -- 172.21.15.88:0/2297095704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8538153770 msgr2=0x7f853815c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:55.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f8535d9b640 1 --2- 172.21.15.88:0/2297095704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8538153770 0x7f853815c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:55.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f8535d9b640 1 -- 172.21.15.88:0/2297095704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85380c41f0 msgr2=0x7f85380ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:55.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f8535d9b640 1 --2- 172.21.15.88:0/2297095704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85380c41f0 0x7f85380ce7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:55.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f8535d9b640 1 -- 172.21.15.88:0/2297095704 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8528009480 con 0x7f85380c9bb0 2023-10-16T20:57:55.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f8535d9b640 1 --2- 172.21.15.88:0/2297095704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85380c9bb0 0x7f853815c3b0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f852c00e930 tx=0x7f852c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:55.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f8526ffd640 1 -- 172.21.15.88:0/2297095704 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c009690 con 0x7f85380c9bb0 2023-10-16T20:57:55.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f8526ffd640 1 -- 172.21.15.88:0/2297095704 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c00ca00 con 0x7f85380c9bb0 2023-10-16T20:57:55.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f8526ffd640 1 -- 172.21.15.88:0/2297095704 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c00cd70 con 0x7f85380c9bb0 2023-10-16T20:57:55.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f853cca9640 1 -- 172.21.15.88:0/2297095704 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853815ae40 con 0x7f85380c9bb0 2023-10-16T20:57:55.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f853cca9640 1 -- 172.21.15.88:0/2297095704 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853815b400 con 0x7f85380c9bb0 2023-10-16T20:57:55.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f853cca9640 1 -- 172.21.15.88:0/2297095704 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f853815b5c0 con 0x7f85380c9bb0 2023-10-16T20:57:55.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.712+0000 7f8526ffd640 1 -- 172.21.15.88:0/2297095704 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f852c013470 con 0x7f85380c9bb0 2023-10-16T20:57:55.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.716+0000 7f8526ffd640 1 --2- 172.21.15.88:0/2297095704 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f85180664a0 0x7f8518068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:55.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.716+0000 7f8526ffd640 1 -- 172.21.15.88:0/2297095704 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7f852c084a40 con 0x7f85380c9bb0 2023-10-16T20:57:55.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.716+0000 7f853659c640 1 --2- 172.21.15.88:0/2297095704 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f85180664a0 0x7f8518068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:55.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.716+0000 7f853659c640 1 --2- 172.21.15.88:0/2297095704 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f85180664a0 0x7f8518068960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f85280ae000 tx=0x7f85280b5040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:55.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.720+0000 7f8526ffd640 1 -- 172.21.15.88:0/2297095704 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f852c086e60 con 0x7f85380c9bb0 2023-10-16T20:57:55.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.912+0000 7f853cca9640 1 -- 172.21.15.88:0/2297095704 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8538074700 con 0x7f85380c9bb0 2023-10-16T20:57:55.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.912+0000 7f8526ffd640 1 -- 172.21.15.88:0/2297095704 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+6580 (secure 0 0 0) 0x7f852c051a00 con 0x7f85380c9bb0 2023-10-16T20:57:55.913 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:57:55.913 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6366,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:57:23.715986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:57:55.913 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 111 2023-10-16T20:57:55.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.916+0000 7f853cca9640 1 -- 172.21.15.88:0/2297095704 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f85180664a0 msgr2=0x7f8518068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:55.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.916+0000 7f853cca9640 1 --2- 172.21.15.88:0/2297095704 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f85180664a0 0x7f8518068960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f85280ae000 tx=0x7f85280b5040 comp rx=0 tx=0).stop 2023-10-16T20:57:55.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.916+0000 7f853cca9640 1 -- 172.21.15.88:0/2297095704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85380c9bb0 msgr2=0x7f853815c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:55.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.916+0000 7f853cca9640 1 --2- 172.21.15.88:0/2297095704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85380c9bb0 0x7f853815c3b0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f852c00e930 tx=0x7f852c00edf0 comp rx=0 tx=0).stop 2023-10-16T20:57:55.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.916+0000 7f853cca9640 1 -- 172.21.15.88:0/2297095704 shutdown_connections 2023-10-16T20:57:55.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.916+0000 7f853cca9640 1 --2- 172.21.15.88:0/2297095704 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f85180664a0 0x7f8518068960 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:55.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.916+0000 7f853cca9640 1 --2- 172.21.15.88:0/2297095704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8538153770 0x7f853815c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:55.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.916+0000 7f853cca9640 1 --2- 172.21.15.88:0/2297095704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85380c9bb0 0x7f853815c3b0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:55.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.916+0000 7f853cca9640 1 --2- 172.21.15.88:0/2297095704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f85380c41f0 0x7f85380ce7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:55.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.916+0000 7f853cca9640 1 -- 172.21.15.88:0/2297095704 >> 172.21.15.88:0/2297095704 conn(0x7f85380715c0 msgr2=0x7f8538147190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:55.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.916+0000 7f853cca9640 1 -- 172.21.15.88:0/2297095704 shutdown_connections 2023-10-16T20:57:55.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:55.916+0000 7f853cca9640 1 -- 172.21.15.88:0/2297095704 wait complete. 2023-10-16T20:57:55.940 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-16T20:57:57.940 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:57:58.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.080+0000 7f063f232640 1 Processor -- start 2023-10-16T20:57:58.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.080+0000 7f063f232640 1 -- start start 2023-10-16T20:57:58.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.080+0000 7f063f232640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0638156820 0x7f06380c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:58.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.080+0000 7f063f232640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06380c5c60 con 0x7f0638150660 2023-10-16T20:57:58.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.080+0000 7f063f232640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06380c5370 con 0x7f0638150280 2023-10-16T20:57:58.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.080+0000 7f063f232640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0638156d10 con 0x7f0638156820 2023-10-16T20:57:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.080+0000 7f063d7cf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0638156820 0x7f06380c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.080+0000 7f0637fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0638150660 0x7f0638156110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36172/0 (socket says 172.21.15.88:36172) 2023-10-16T20:57:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.080+0000 7f063d7cf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0638156820 0x7f06380c60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60480/0 (socket says 172.21.15.88:60480) 2023-10-16T20:57:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.080+0000 7f0637fff640 1 -- 172.21.15.88:0/1599189430 learned_addr learned my addr 172.21.15.88:0/1599189430 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:57:58.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.084+0000 7f063d7cf640 1 -- 172.21.15.88:0/1599189430 >> v1:172.21.15.191:6789/0 conn(0x7f0638150280 legacy=0x7f063806d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:58.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.084+0000 7f063d7cf640 1 -- 172.21.15.88:0/1599189430 >> v1:172.21.15.88:6789/0 conn(0x7f0638150660 legacy=0x7f0638156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:58.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.084+0000 7f06377fe640 1 -- 172.21.15.88:0/1599189430 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242285068 0 0) 0x7f06380c5c60 con 0x7f0638150660 2023-10-16T20:57:58.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.084+0000 7f063d7cf640 1 -- 172.21.15.88:0/1599189430 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06380c5610 con 0x7f0638156820 2023-10-16T20:57:58.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.084+0000 7f063d7cf640 1 --2- 172.21.15.88:0/1599189430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0638156820 0x7f06380c60f0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f0630009d90 tx=0x7f063001d380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a81401445270e340 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:58.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.084+0000 7f06377fe640 1 -- 172.21.15.88:0/1599189430 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06300a5a00 con 0x7f0638156820 2023-10-16T20:57:58.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.084+0000 7f06377fe640 1 -- 172.21.15.88:0/1599189430 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06300a5b60 con 0x7f0638156820 2023-10-16T20:57:58.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.084+0000 7f06377fe640 1 -- 172.21.15.88:0/1599189430 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06300ad440 con 0x7f0638156820 2023-10-16T20:57:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.084+0000 7f063f232640 1 -- 172.21.15.88:0/1599189430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0638156820 msgr2=0x7f06380c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.084+0000 7f063f232640 1 --2- 172.21.15.88:0/1599189430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0638156820 0x7f06380c60f0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f0630009d90 tx=0x7f063001d380 comp rx=0 tx=0).stop 2023-10-16T20:57:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063f232640 1 -- 172.21.15.88:0/1599189430 shutdown_connections 2023-10-16T20:57:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063f232640 1 --2- 172.21.15.88:0/1599189430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0638156820 0x7f06380c60f0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063f232640 1 -- 172.21.15.88:0/1599189430 >> 172.21.15.88:0/1599189430 conn(0x7f06380cad50 msgr2=0x7f06380cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063f232640 1 -- 172.21.15.88:0/1599189430 shutdown_connections 2023-10-16T20:57:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063f232640 1 -- 172.21.15.88:0/1599189430 wait complete. 2023-10-16T20:57:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063f232640 1 Processor -- start 2023-10-16T20:57:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063f232640 1 -- start start 2023-10-16T20:57:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063f232640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0638150280 0x7f06381f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063f232640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0638150660 0x7f06380c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063f232640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0638156820 0x7f06380c8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063f232640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063813df80 con 0x7f0638150280 2023-10-16T20:57:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063f232640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063813dce0 con 0x7f0638156820 2023-10-16T20:57:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063f232640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06380ca340 con 0x7f0638150660 2023-10-16T20:57:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063cfce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0638150280 0x7f06381f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063cfce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0638150280 0x7f06381f2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60490/0 (socket says 172.21.15.88:60490) 2023-10-16T20:57:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063cfce640 1 -- 172.21.15.88:0/2661216658 learned_addr learned my addr 172.21.15.88:0/2661216658 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:57:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063d7cf640 1 --2- 172.21.15.88:0/2661216658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0638156820 0x7f06380c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f0637fff640 1 --2- 172.21.15.88:0/2661216658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0638150660 0x7f06380c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063cfce640 1 -- 172.21.15.88:0/2661216658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0638150660 msgr2=0x7f06380c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063cfce640 1 --2- 172.21.15.88:0/2661216658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0638150660 0x7f06380c9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063cfce640 1 -- 172.21.15.88:0/2661216658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0638156820 msgr2=0x7f06380c8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063cfce640 1 --2- 172.21.15.88:0/2661216658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0638156820 0x7f06380c8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063cfce640 1 -- 172.21.15.88:0/2661216658 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0630009a70 con 0x7f0638150280 2023-10-16T20:57:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.088+0000 7f063cfce640 1 --2- 172.21.15.88:0/2661216658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0638150280 0x7f06381f2c00 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f062c00c500 tx=0x7f062c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.092+0000 7f06357fa640 1 -- 172.21.15.88:0/2661216658 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c005220 con 0x7f0638150280 2023-10-16T20:57:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.092+0000 7f06357fa640 1 -- 172.21.15.88:0/2661216658 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062c005380 con 0x7f0638150280 2023-10-16T20:57:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.092+0000 7f063f232640 1 -- 172.21.15.88:0/2661216658 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06381f2120 con 0x7f0638150280 2023-10-16T20:57:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.092+0000 7f06357fa640 1 -- 172.21.15.88:0/2661216658 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c007270 con 0x7f0638150280 2023-10-16T20:57:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.092+0000 7f063f232640 1 -- 172.21.15.88:0/2661216658 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06381f26e0 con 0x7f0638150280 2023-10-16T20:57:58.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.092+0000 7f063f232640 1 -- 172.21.15.88:0/2661216658 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0638074700 con 0x7f0638150280 2023-10-16T20:57:58.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.092+0000 7f06357fa640 1 -- 172.21.15.88:0/2661216658 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f062c0073d0 con 0x7f0638150280 2023-10-16T20:57:58.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.092+0000 7f06357fa640 1 --2- 172.21.15.88:0/2661216658 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f061c0664a0 0x7f061c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:57:58.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.092+0000 7f06357fa640 1 -- 172.21.15.88:0/2661216658 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7f062c083d50 con 0x7f0638150280 2023-10-16T20:57:58.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.092+0000 7f0637fff640 1 --2- 172.21.15.88:0/2661216658 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f061c0664a0 0x7f061c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:57:58.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.096+0000 7f0637fff640 1 --2- 172.21.15.88:0/2661216658 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f061c0664a0 0x7f061c068960 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0638143020 tx=0x7f0624007240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:57:58.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.096+0000 7f06357fa640 1 -- 172.21.15.88:0/2661216658 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f062c050b30 con 0x7f0638150280 2023-10-16T20:57:58.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.280+0000 7f063f232640 1 -- 172.21.15.88:0/2661216658 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f063814dbb0 con 0x7f0638150280 2023-10-16T20:57:58.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.280+0000 7f06357fa640 1 -- 172.21.15.88:0/2661216658 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+6580 (secure 0 0 0) 0x7f062c054150 con 0x7f0638150280 2023-10-16T20:57:58.279 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:57:58.279 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6366,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:57:23.715986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":5378},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5378":{"gid":5378,"name":"f","rank":4,"incarnation":89,"state":"up:active","state_seq":32,"addr":"172.21.15.191:6835/3119045295","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3119045295},{"type":"v1","addr":"172.21.15.191:6835","nonce":3119045295}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:57:58.280 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 111 2023-10-16T20:57:58.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.280+0000 7f063f232640 1 -- 172.21.15.88:0/2661216658 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f061c0664a0 msgr2=0x7f061c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:58.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.280+0000 7f063f232640 1 --2- 172.21.15.88:0/2661216658 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f061c0664a0 0x7f061c068960 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0638143020 tx=0x7f0624007240 comp rx=0 tx=0).stop 2023-10-16T20:57:58.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.280+0000 7f063f232640 1 -- 172.21.15.88:0/2661216658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0638150280 msgr2=0x7f06381f2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:57:58.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.280+0000 7f063f232640 1 --2- 172.21.15.88:0/2661216658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0638150280 0x7f06381f2c00 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f062c00c500 tx=0x7f062c00c9c0 comp rx=0 tx=0).stop 2023-10-16T20:57:58.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.280+0000 7f063f232640 1 -- 172.21.15.88:0/2661216658 shutdown_connections 2023-10-16T20:57:58.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.280+0000 7f063f232640 1 --2- 172.21.15.88:0/2661216658 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f061c0664a0 0x7f061c068960 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:58.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.280+0000 7f063f232640 1 --2- 172.21.15.88:0/2661216658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0638156820 0x7f06380c8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:58.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.280+0000 7f063f232640 1 --2- 172.21.15.88:0/2661216658 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0638150660 0x7f06380c9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:58.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.280+0000 7f063f232640 1 --2- 172.21.15.88:0/2661216658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0638150280 0x7f06381f2c00 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:57:58.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.280+0000 7f063f232640 1 -- 172.21.15.88:0/2661216658 >> 172.21.15.88:0/2661216658 conn(0x7f06380cad50 msgr2=0x7f063806d200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:57:58.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.280+0000 7f063f232640 1 -- 172.21.15.88:0/2661216658 shutdown_connections 2023-10-16T20:57:58.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:57:58.280+0000 7f063f232640 1 -- 172.21.15.88:0/2661216658 wait complete. 2023-10-16T20:57:58.299 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-16T20:58:00.300 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:58:00.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa257765640 1 Processor -- start 2023-10-16T20:58:00.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa257765640 1 -- start start 2023-10-16T20:58:00.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa257765640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa250153770 0x7fa250159f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:00.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa257765640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa2500c9040 0x7fa2500c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:00.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa257765640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2500c6a20 0x7fa250071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:00.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa257765640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa250069e90 con 0x7fa250153770 2023-10-16T20:58:00.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa257765640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa25006aef0 con 0x7fa2500c9040 2023-10-16T20:58:00.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa257765640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25006a0c0 con 0x7fa2500c6a20 2023-10-16T20:58:00.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa255501640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa250153770 0x7fa250159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:00.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa255501640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa250153770 0x7fa250159f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33058/0 (socket says 172.21.15.88:33058) 2023-10-16T20:58:00.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa255501640 1 -- 172.21.15.88:0/1998266356 learned_addr learned my addr 172.21.15.88:0/1998266356 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:00.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa254d00640 1 --2- 172.21.15.88:0/1998266356 >> v2:172.21.15.191:3300/0 conn(0x7fa2500c9040 0x7fa2500c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:00.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa255d02640 1 --2- 172.21.15.88:0/1998266356 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2500c6a20 0x7fa250071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:00.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa255501640 1 -- 172.21.15.88:0/1998266356 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2500c6a20 msgr2=0x7fa250071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:00.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa255501640 1 --2- 172.21.15.88:0/1998266356 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2500c6a20 0x7fa250071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa255501640 1 -- 172.21.15.88:0/1998266356 >> v2:172.21.15.191:3300/0 conn(0x7fa2500c9040 msgr2=0x7fa2500c64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:00.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa255501640 1 --2- 172.21.15.88:0/1998266356 >> v2:172.21.15.191:3300/0 conn(0x7fa2500c9040 0x7fa2500c64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa255501640 1 -- 172.21.15.88:0/1998266356 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25006a360 con 0x7fa250153770 2023-10-16T20:58:00.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.472+0000 7fa255d02640 1 --2- 172.21.15.88:0/1998266356 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2500c6a20 0x7fa250071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:58:00.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.476+0000 7fa255501640 1 --2- 172.21.15.88:0/1998266356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa250153770 0x7fa250159f00 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fa2500c95d0 tx=0x7fa24c01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4585c8038e3f95c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:00.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.476+0000 7fa247fff640 1 -- 172.21.15.88:0/1998266356 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c0ac410 con 0x7fa250153770 2023-10-16T20:58:00.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.476+0000 7fa247fff640 1 -- 172.21.15.88:0/1998266356 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24c0ac570 con 0x7fa250153770 2023-10-16T20:58:00.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.476+0000 7fa247fff640 1 -- 172.21.15.88:0/1998266356 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c0ac900 con 0x7fa250153770 2023-10-16T20:58:00.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.476+0000 7fa257765640 1 -- 172.21.15.88:0/1998266356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa250153770 msgr2=0x7fa250159f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:00.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.476+0000 7fa257765640 1 --2- 172.21.15.88:0/1998266356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa250153770 0x7fa250159f00 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fa2500c95d0 tx=0x7fa24c01c4e0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.476+0000 7fa257765640 1 -- 172.21.15.88:0/1998266356 shutdown_connections 2023-10-16T20:58:00.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.476+0000 7fa257765640 1 --2- 172.21.15.88:0/1998266356 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa2500c6a20 0x7fa250071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.476+0000 7fa257765640 1 --2- 172.21.15.88:0/1998266356 >> v2:172.21.15.191:3300/0 conn(0x7fa2500c9040 0x7fa2500c64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.476+0000 7fa257765640 1 --2- 172.21.15.88:0/1998266356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa250153770 0x7fa250159f00 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.476+0000 7fa257765640 1 -- 172.21.15.88:0/1998266356 >> 172.21.15.88:0/1998266356 conn(0x7fa2500c84e0 msgr2=0x7fa2500c88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:00.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.476+0000 7fa257765640 1 -- 172.21.15.88:0/1998266356 shutdown_connections 2023-10-16T20:58:00.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.476+0000 7fa257765640 1 -- 172.21.15.88:0/1998266356 wait complete. 2023-10-16T20:58:00.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa257765640 1 Processor -- start 2023-10-16T20:58:00.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa257765640 1 -- start start 2023-10-16T20:58:00.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa257765640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2500c6a20 0x7fa25015dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:00.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa257765640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2500c9040 0x7fa25014ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:00.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa257765640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa250153770 0x7fa2500c3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:00.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa257765640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2501515b0 con 0x7fa2500c9040 2023-10-16T20:58:00.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa257765640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2501525b0 con 0x7fa2500c6a20 2023-10-16T20:58:00.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa257765640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa250152850 con 0x7fa250153770 2023-10-16T20:58:00.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa254d00640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2500c9040 0x7fa25014ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:00.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa254d00640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2500c9040 0x7fa25014ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33062/0 (socket says 172.21.15.88:33062) 2023-10-16T20:58:00.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa254d00640 1 -- 172.21.15.88:0/431261426 learned_addr learned my addr 172.21.15.88:0/431261426 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:00.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa255501640 1 --2- 172.21.15.88:0/431261426 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2500c6a20 0x7fa25015dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:00.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa255d02640 1 --2- 172.21.15.88:0/431261426 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa250153770 0x7fa2500c3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:00.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa254d00640 1 -- 172.21.15.88:0/431261426 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa250153770 msgr2=0x7fa2500c3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:00.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa254d00640 1 --2- 172.21.15.88:0/431261426 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa250153770 0x7fa2500c3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa254d00640 1 -- 172.21.15.88:0/431261426 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2500c6a20 msgr2=0x7fa25015dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:00.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa254d00640 1 --2- 172.21.15.88:0/431261426 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2500c6a20 0x7fa25015dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa254d00640 1 -- 172.21.15.88:0/431261426 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c0094d0 con 0x7fa2500c9040 2023-10-16T20:58:00.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa254d00640 1 --2- 172.21.15.88:0/431261426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2500c9040 0x7fa25014ef30 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fa24000e930 tx=0x7fa24000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:00.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa245ffb640 1 -- 172.21.15.88:0/431261426 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa240009690 con 0x7fa2500c9040 2023-10-16T20:58:00.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa245ffb640 1 -- 172.21.15.88:0/431261426 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24000ca00 con 0x7fa2500c9040 2023-10-16T20:58:00.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa245ffb640 1 -- 172.21.15.88:0/431261426 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24000cd70 con 0x7fa2500c9040 2023-10-16T20:58:00.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa257765640 1 -- 172.21.15.88:0/431261426 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa250152b10 con 0x7fa2500c9040 2023-10-16T20:58:00.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.480+0000 7fa257765640 1 -- 172.21.15.88:0/431261426 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa250151800 con 0x7fa2500c9040 2023-10-16T20:58:00.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.484+0000 7fa257765640 1 -- 172.21.15.88:0/431261426 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa250074700 con 0x7fa2500c9040 2023-10-16T20:58:00.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.484+0000 7fa245ffb640 1 -- 172.21.15.88:0/431261426 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa240013470 con 0x7fa2500c9040 2023-10-16T20:58:00.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.484+0000 7fa245ffb640 1 --2- 172.21.15.88:0/431261426 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa2340664d0 0x7fa234068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:00.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.484+0000 7fa245ffb640 1 -- 172.21.15.88:0/431261426 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7437+0+0 (secure 0 0 0) 0x7fa240084570 con 0x7fa2500c9040 2023-10-16T20:58:00.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.484+0000 7fa255501640 1 --2- 172.21.15.88:0/431261426 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa2340664d0 0x7fa234068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:00.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.484+0000 7fa255501640 1 --2- 172.21.15.88:0/431261426 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa2340664d0 0x7fa234068990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0ae000 tx=0x7fa24c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:00.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.488+0000 7fa245ffb640 1 -- 172.21.15.88:0/431261426 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa240051510 con 0x7fa2500c9040 2023-10-16T20:58:00.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.676+0000 7fa257765640 1 -- 172.21.15.88:0/431261426 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa250152390 con 0x7fa2500c9040 2023-10-16T20:58:00.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.676+0000 7fa245ffb640 1 -- 172.21.15.88:0/431261426 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5786 (secure 0 0 0) 0x7fa240054b30 con 0x7fa2500c9040 2023-10-16T20:58:00.675 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:00.675 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:58:00.653223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"c","rank":4,"incarnation":112,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:58:00.675 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 112 2023-10-16T20:58:00.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.676+0000 7fa257765640 1 -- 172.21.15.88:0/431261426 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa2340664d0 msgr2=0x7fa234068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:00.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.676+0000 7fa257765640 1 --2- 172.21.15.88:0/431261426 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa2340664d0 0x7fa234068990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0ae000 tx=0x7fa24c009040 comp rx=0 tx=0).stop 2023-10-16T20:58:00.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.676+0000 7fa257765640 1 -- 172.21.15.88:0/431261426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2500c9040 msgr2=0x7fa25014ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:00.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.676+0000 7fa257765640 1 --2- 172.21.15.88:0/431261426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2500c9040 0x7fa25014ef30 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fa24000e930 tx=0x7fa24000edf0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.676+0000 7fa257765640 1 -- 172.21.15.88:0/431261426 shutdown_connections 2023-10-16T20:58:00.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.676+0000 7fa257765640 1 --2- 172.21.15.88:0/431261426 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa2340664d0 0x7fa234068990 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.676+0000 7fa257765640 1 --2- 172.21.15.88:0/431261426 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa250153770 0x7fa2500c3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.676+0000 7fa257765640 1 --2- 172.21.15.88:0/431261426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2500c9040 0x7fa25014ef30 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.676+0000 7fa257765640 1 --2- 172.21.15.88:0/431261426 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2500c6a20 0x7fa25015dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.676+0000 7fa257765640 1 -- 172.21.15.88:0/431261426 >> 172.21.15.88:0/431261426 conn(0x7fa2500c84e0 msgr2=0x7fa25014f860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:00.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.676+0000 7fa257765640 1 -- 172.21.15.88:0/431261426 shutdown_connections 2023-10-16T20:58:00.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.676+0000 7fa257765640 1 -- 172.21.15.88:0/431261426 wait complete. 2023-10-16T20:58:00.706 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-16T20:58:00.706 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-16T20:58:00.706 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:58:00.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.856+0000 7fc445535640 1 Processor -- start 2023-10-16T20:58:00.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.856+0000 7fc445535640 1 -- start start 2023-10-16T20:58:00.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.860+0000 7fc445535640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc440153770 0x7fc440159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:00.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.860+0000 7fc445535640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4400cece0 0x7fc4400cf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:00.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.860+0000 7fc445535640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc44015a410 con 0x7fc4400c9bb0 2023-10-16T20:58:00.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.860+0000 7fc445535640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4400cc250 con 0x7fc440153770 2023-10-16T20:58:00.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.860+0000 7fc445535640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4400cc670 con 0x7fc4400cece0 2023-10-16T20:58:00.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.860+0000 7fc43effd640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc440153770 0x7fc440159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:00.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.860+0000 7fc43f7fe640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4400cece0 0x7fc4400cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:00.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.860+0000 7fc43e7fc640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fc4400c9bb0 0x7fc4400c3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:38918/0 (socket says 172.21.15.88:38918) 2023-10-16T20:58:00.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.860+0000 7fc43e7fc640 1 -- 172.21.15.88:0/4234672692 learned_addr learned my addr 172.21.15.88:0/4234672692 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:00.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.860+0000 7fc43effd640 1 -- 172.21.15.88:0/4234672692 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4400cece0 msgr2=0x7fc4400cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:00.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.860+0000 7fc43effd640 1 --2- 172.21.15.88:0/4234672692 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4400cece0 0x7fc4400cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.860+0000 7fc43effd640 1 -- 172.21.15.88:0/4234672692 >> v1:172.21.15.191:6789/0 conn(0x7fc4400c9bb0 legacy=0x7fc4400c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:00.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.860+0000 7fc43effd640 1 -- 172.21.15.88:0/4234672692 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc440153b50 con 0x7fc440153770 2023-10-16T20:58:00.861 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.860+0000 7fc43dffb640 1 -- 172.21.15.88:0/4234672692 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206615554 0 0) 0x7fc44015a410 con 0x7fc4400c9bb0 2023-10-16T20:58:00.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc43effd640 1 --2- 172.21.15.88:0/4234672692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc440153770 0x7fc440159ed0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc4300097a0 tx=0x7fc43001c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6406920946a28072 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:00.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc43dffb640 1 -- 172.21.15.88:0/4234672692 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4300ab660 con 0x7fc440153770 2023-10-16T20:58:00.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc43dffb640 1 -- 172.21.15.88:0/4234672692 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4300ab7c0 con 0x7fc440153770 2023-10-16T20:58:00.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc43dffb640 1 -- 172.21.15.88:0/4234672692 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4300abb50 con 0x7fc440153770 2023-10-16T20:58:00.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc445535640 1 -- 172.21.15.88:0/4234672692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc440153770 msgr2=0x7fc440159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:00.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc445535640 1 --2- 172.21.15.88:0/4234672692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc440153770 0x7fc440159ed0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc4300097a0 tx=0x7fc43001c460 comp rx=0 tx=0).stop 2023-10-16T20:58:00.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc445535640 1 -- 172.21.15.88:0/4234672692 shutdown_connections 2023-10-16T20:58:00.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc445535640 1 --2- 172.21.15.88:0/4234672692 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4400cece0 0x7fc4400cf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc445535640 1 --2- 172.21.15.88:0/4234672692 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc440153770 0x7fc440159ed0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc445535640 1 -- 172.21.15.88:0/4234672692 >> 172.21.15.88:0/4234672692 conn(0x7fc4400715c0 msgr2=0x7fc4400719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:00.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc445535640 1 -- 172.21.15.88:0/4234672692 shutdown_connections 2023-10-16T20:58:00.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc445535640 1 -- 172.21.15.88:0/4234672692 wait complete. 2023-10-16T20:58:00.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc445535640 1 Processor -- start 2023-10-16T20:58:00.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc445535640 1 -- start start 2023-10-16T20:58:00.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc445535640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4400c9bb0 0x7fc44013dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:00.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc445535640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4400cece0 0x7fc440149790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:00.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc445535640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc440153770 0x7fc440151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:00.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc445535640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4400c74c0 con 0x7fc440153770 2023-10-16T20:58:00.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc445535640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4401f4780 con 0x7fc4400c9bb0 2023-10-16T20:58:00.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.864+0000 7fc445535640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4401495b0 con 0x7fc4400cece0 2023-10-16T20:58:00.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc43f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc440153770 0x7fc440151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:00.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc43e7fc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4400cece0 0x7fc440149790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:00.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc43effd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4400c9bb0 0x7fc44013dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:00.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc43e7fc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4400cece0 0x7fc440149790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:56928/0 (socket says 172.21.15.88:56928) 2023-10-16T20:58:00.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc43effd640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4400c9bb0 0x7fc44013dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:34446/0 (socket says 172.21.15.88:34446) 2023-10-16T20:58:00.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc43f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc440153770 0x7fc440151fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33064/0 (socket says 172.21.15.88:33064) 2023-10-16T20:58:00.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc43e7fc640 1 -- 172.21.15.88:0/1159493962 learned_addr learned my addr 172.21.15.88:0/1159493962 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:00.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc43e7fc640 1 -- 172.21.15.88:0/1159493962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4400c9bb0 msgr2=0x7fc44013dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:00.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc43e7fc640 1 --2- 172.21.15.88:0/1159493962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4400c9bb0 0x7fc44013dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc43e7fc640 1 -- 172.21.15.88:0/1159493962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc440153770 msgr2=0x7fc440151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:00.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc43e7fc640 1 --2- 172.21.15.88:0/1159493962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc440153770 0x7fc440151fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:00.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc43e7fc640 1 -- 172.21.15.88:0/1159493962 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4300093b0 con 0x7fc4400cece0 2023-10-16T20:58:00.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc43e7fc640 1 --2- 172.21.15.88:0/1159493962 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4400cece0 0x7fc440149790 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc43400c4b0 tx=0x7fc43400c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:00.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc4277fe640 1 -- 172.21.15.88:0/1159493962 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4340074e0 con 0x7fc4400cece0 2023-10-16T20:58:00.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc4277fe640 1 -- 172.21.15.88:0/1159493962 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc434007660 con 0x7fc4400cece0 2023-10-16T20:58:00.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc4277fe640 1 -- 172.21.15.88:0/1159493962 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc434004a30 con 0x7fc4400cece0 2023-10-16T20:58:00.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc445535640 1 -- 172.21.15.88:0/1159493962 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4401f49b0 con 0x7fc4400cece0 2023-10-16T20:58:00.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc445535640 1 -- 172.21.15.88:0/1159493962 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4401f4f70 con 0x7fc4400cece0 2023-10-16T20:58:00.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc445535640 1 -- 172.21.15.88:0/1159493962 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc440074700 con 0x7fc4400cece0 2023-10-16T20:58:00.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc4277fe640 1 -- 172.21.15.88:0/1159493962 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc434004b90 con 0x7fc4400cece0 2023-10-16T20:58:00.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.868+0000 7fc4277fe640 1 --2- 172.21.15.88:0/1159493962 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc42c0664d0 0x7fc42c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:00.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.872+0000 7fc4277fe640 1 -- 172.21.15.88:0/1159493962 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7fc434083660 con 0x7fc4400cece0 2023-10-16T20:58:00.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.872+0000 7fc43effd640 1 --2- 172.21.15.88:0/1159493962 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc42c0664d0 0x7fc42c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:00.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.872+0000 7fc43effd640 1 --2- 172.21.15.88:0/1159493962 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc42c0664d0 0x7fc42c068990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc4300af000 tx=0x7fc4300a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:00.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:00.876+0000 7fc4277fe640 1 -- 172.21.15.88:0/1159493962 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc4340503d0 con 0x7fc4400cece0 2023-10-16T20:58:01.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:01.100+0000 7fc445535640 1 -- 172.21.15.88:0/1159493962 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4400ca740 con 0x7fc4400cece0 2023-10-16T20:58:01.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:01.100+0000 7fc4277fe640 1 -- 172.21.15.88:0/1159493962 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5786 (secure 0 0 0) 0x7fc4340539f0 con 0x7fc4400cece0 2023-10-16T20:58:01.099 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:01.099 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:58:00.653223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"c","rank":4,"incarnation":112,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:58:01.100 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 112 2023-10-16T20:58:01.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:01.100+0000 7fc445535640 1 -- 172.21.15.88:0/1159493962 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc42c0664d0 msgr2=0x7fc42c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:01.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:01.100+0000 7fc445535640 1 --2- 172.21.15.88:0/1159493962 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc42c0664d0 0x7fc42c068990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc4300af000 tx=0x7fc4300a7000 comp rx=0 tx=0).stop 2023-10-16T20:58:01.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:01.100+0000 7fc445535640 1 -- 172.21.15.88:0/1159493962 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4400cece0 msgr2=0x7fc440149790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:01.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:01.100+0000 7fc445535640 1 --2- 172.21.15.88:0/1159493962 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4400cece0 0x7fc440149790 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc43400c4b0 tx=0x7fc43400c970 comp rx=0 tx=0).stop 2023-10-16T20:58:01.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:01.100+0000 7fc445535640 1 -- 172.21.15.88:0/1159493962 shutdown_connections 2023-10-16T20:58:01.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:01.100+0000 7fc445535640 1 --2- 172.21.15.88:0/1159493962 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc42c0664d0 0x7fc42c068990 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:01.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:01.100+0000 7fc445535640 1 --2- 172.21.15.88:0/1159493962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc440153770 0x7fc440151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:01.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:01.100+0000 7fc445535640 1 --2- 172.21.15.88:0/1159493962 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc4400cece0 0x7fc440149790 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:01.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:01.100+0000 7fc445535640 1 --2- 172.21.15.88:0/1159493962 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4400c9bb0 0x7fc44013dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:01.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:01.100+0000 7fc445535640 1 -- 172.21.15.88:0/1159493962 >> 172.21.15.88:0/1159493962 conn(0x7fc4400715c0 msgr2=0x7fc4400ccc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:01.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:01.104+0000 7fc445535640 1 -- 172.21.15.88:0/1159493962 shutdown_connections 2023-10-16T20:58:01.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:01.104+0000 7fc445535640 1 -- 172.21.15.88:0/1159493962 wait complete. 2023-10-16T20:58:01.128 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 112, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 112, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5477": { "addr": "172.21.15.191:6837/2244284333", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6836", "nonce": 2244284333, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2244284333, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5477, "incarnation": 97, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 39 }, "gid_5535": { "addr": "172.21.15.88:6835/3476086754", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3476086754, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3476086754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5535, "incarnation": 52, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_5570": { "addr": "172.21.15.191:6833/3394064120", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 3394064120, "type": "v2" }, { "addr": "172.21.15.191:6833", "nonce": 3394064120, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5570, "incarnation": 103, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 13 }, "gid_5709": { "addr": "172.21.15.88:6838/2600891060", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 2600891060, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 2600891060, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5709, "incarnation": 67, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 37 }, "gid_6366": { "addr": "172.21.15.88:6839/2182739466", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6837", "nonce": 2182739466, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2182739466, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6366, "incarnation": 112, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:58:00.653223+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5477, "mds_1": 5709, "mds_2": 5535, "mds_3": 5570, "mds_4": 6366 } } } ], "standbys": [] } 2023-10-16T20:58:01.429 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:58:01.433 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:58:03.128 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:58:03.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.308+0000 7f7e7999d640 1 Processor -- start 2023-10-16T20:58:03.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.308+0000 7f7e7999d640 1 -- start start 2023-10-16T20:58:03.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.308+0000 7f7e7999d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7e741538f0 0x7f7e740c71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:03.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.308+0000 7f7e7999d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e740c7e40 0x7f7e740c9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:03.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.308+0000 7f7e7999d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7415cb30 con 0x7f7e740c9790 2023-10-16T20:58:03.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.308+0000 7f7e7999d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e74153cd0 con 0x7f7e741538f0 2023-10-16T20:58:03.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.308+0000 7f7e7999d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e740c8f50 con 0x7f7e740c7e40 2023-10-16T20:58:03.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.308+0000 7f7e73fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7e740c9790 0x7f7e7415cfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45796/0 (socket says 172.21.15.88:45796) 2023-10-16T20:58:03.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.308+0000 7f7e73fff640 1 -- 172.21.15.88:0/3594415128 learned_addr learned my addr 172.21.15.88:0/3594415128 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:03.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.308+0000 7f7e72ffd640 1 --2- 172.21.15.88:0/3594415128 >> v2:172.21.15.88:3300/0 conn(0x7f7e741538f0 0x7f7e740c71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:03.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.308+0000 7f7e727fc640 1 --2- 172.21.15.88:0/3594415128 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e740c7e40 0x7f7e740c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:03.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.308+0000 7f7e71ffb640 1 -- 172.21.15.88:0/3594415128 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731323712 0 0) 0x7f7e7415cb30 con 0x7f7e740c9790 2023-10-16T20:58:03.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.308+0000 7f7e71ffb640 1 -- 172.21.15.88:0/3594415128 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e5c003530 con 0x7f7e740c9790 2023-10-16T20:58:03.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.308+0000 7f7e727fc640 1 -- 172.21.15.88:0/3594415128 >> v2:172.21.15.88:3300/0 conn(0x7f7e741538f0 msgr2=0x7f7e740c71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:03.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.312+0000 7f7e71ffb640 1 -- 172.21.15.88:0/3594415128 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 324895065 0 0) 0x7f7e5c003530 con 0x7f7e740c9790 2023-10-16T20:58:03.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.312+0000 7f7e727fc640 1 --2- 172.21.15.88:0/3594415128 >> v2:172.21.15.88:3300/0 conn(0x7f7e741538f0 0x7f7e740c71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:03.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.312+0000 7f7e727fc640 1 -- 172.21.15.88:0/3594415128 >> v1:172.21.15.88:6789/0 conn(0x7f7e740c9790 legacy=0x7f7e7415cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:03.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.312+0000 7f7e727fc640 1 -- 172.21.15.88:0/3594415128 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7406a920 con 0x7f7e740c7e40 2023-10-16T20:58:03.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.312+0000 7f7e72ffd640 1 --2- 172.21.15.88:0/3594415128 >> v2:172.21.15.88:3300/0 conn(0x7f7e741538f0 0x7f7e740c71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:58:03.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.316+0000 7f7e727fc640 1 --2- 172.21.15.88:0/3594415128 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e740c7e40 0x7f7e740c9250 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7e680097c0 tx=0x7f7e6801eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d11c89098c62e03b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:03.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.316+0000 7f7e71ffb640 1 -- 172.21.15.88:0/3594415128 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c001a00 con 0x7f7e740c7e40 2023-10-16T20:58:03.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.316+0000 7f7e71ffb640 1 -- 172.21.15.88:0/3594415128 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e680a85d0 con 0x7f7e740c7e40 2023-10-16T20:58:03.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.316+0000 7f7e71ffb640 1 -- 172.21.15.88:0/3594415128 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e680a8960 con 0x7f7e740c7e40 2023-10-16T20:58:03.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.316+0000 7f7e7999d640 1 -- 172.21.15.88:0/3594415128 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e740c7e40 msgr2=0x7f7e740c9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:03.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.316+0000 7f7e7999d640 1 --2- 172.21.15.88:0/3594415128 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e740c7e40 0x7f7e740c9250 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7e680097c0 tx=0x7f7e6801eb90 comp rx=0 tx=0).stop 2023-10-16T20:58:03.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.316+0000 7f7e7999d640 1 -- 172.21.15.88:0/3594415128 shutdown_connections 2023-10-16T20:58:03.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.316+0000 7f7e7999d640 1 --2- 172.21.15.88:0/3594415128 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e740c7e40 0x7f7e740c9250 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:03.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.316+0000 7f7e7999d640 1 --2- 172.21.15.88:0/3594415128 >> v2:172.21.15.88:3300/0 conn(0x7f7e741538f0 0x7f7e740c71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:03.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.316+0000 7f7e7999d640 1 -- 172.21.15.88:0/3594415128 >> 172.21.15.88:0/3594415128 conn(0x7f7e7406f2c0 msgr2=0x7f7e7406f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:03.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.316+0000 7f7e7999d640 1 -- 172.21.15.88:0/3594415128 shutdown_connections 2023-10-16T20:58:03.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.316+0000 7f7e7999d640 1 -- 172.21.15.88:0/3594415128 wait complete. 2023-10-16T20:58:03.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e7999d640 1 Processor -- start 2023-10-16T20:58:03.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e7999d640 1 -- start start 2023-10-16T20:58:03.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e7999d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e740c7e40 0x7f7e7415b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:03.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e7999d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e740c9790 0x7f7e7414a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:03.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e7999d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e741538f0 0x7f7e741f23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:03.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e7999d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e740cb120 con 0x7f7e740c9790 2023-10-16T20:58:03.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e7999d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7415c280 con 0x7f7e741538f0 2023-10-16T20:58:03.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e7999d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7415bfa0 con 0x7f7e740c7e40 2023-10-16T20:58:03.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e727fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e740c9790 0x7f7e7414a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:03.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e73fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e741538f0 0x7f7e741f23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:03.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e72ffd640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e740c7e40 0x7f7e7415b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:03.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e727fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e740c9790 0x7f7e7414a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33092/0 (socket says 172.21.15.88:33092) 2023-10-16T20:58:03.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e73fff640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e741538f0 0x7f7e741f23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:34456/0 (socket says 172.21.15.88:34456) 2023-10-16T20:58:03.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e727fc640 1 -- 172.21.15.88:0/3448653321 learned_addr learned my addr 172.21.15.88:0/3448653321 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:03.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e72ffd640 1 -- 172.21.15.88:0/3448653321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e741538f0 msgr2=0x7f7e741f23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:03.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e72ffd640 1 --2- 172.21.15.88:0/3448653321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e741538f0 0x7f7e741f23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:03.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e72ffd640 1 -- 172.21.15.88:0/3448653321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e740c9790 msgr2=0x7f7e7414a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:03.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e72ffd640 1 --2- 172.21.15.88:0/3448653321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e740c9790 0x7f7e7414a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:03.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e72ffd640 1 -- 172.21.15.88:0/3448653321 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e64009420 con 0x7f7e740c7e40 2023-10-16T20:58:03.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e727fc640 1 --2- 172.21.15.88:0/3448653321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e740c9790 0x7f7e7414a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:58:03.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e72ffd640 1 --2- 172.21.15.88:0/3448653321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e740c7e40 0x7f7e7415b960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400e8e0 tx=0x7f7e6400eda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:03.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e5bfff640 1 -- 172.21.15.88:0/3448653321 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400c8e0 con 0x7f7e740c7e40 2023-10-16T20:58:03.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e5bfff640 1 -- 172.21.15.88:0/3448653321 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6400ca40 con 0x7f7e740c7e40 2023-10-16T20:58:03.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e5bfff640 1 -- 172.21.15.88:0/3448653321 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400cdf0 con 0x7f7e740c7e40 2023-10-16T20:58:03.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e7999d640 1 -- 172.21.15.88:0/3448653321 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e68009420 con 0x7f7e740c7e40 2023-10-16T20:58:03.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.320+0000 7f7e7999d640 1 -- 172.21.15.88:0/3448653321 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e740c58e0 con 0x7f7e740c7e40 2023-10-16T20:58:03.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.324+0000 7f7e7999d640 1 -- 172.21.15.88:0/3448653321 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e74074700 con 0x7f7e740c7e40 2023-10-16T20:58:03.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.324+0000 7f7e5bfff640 1 -- 172.21.15.88:0/3448653321 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7e64013470 con 0x7f7e740c7e40 2023-10-16T20:58:03.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.324+0000 7f7e5bfff640 1 --2- 172.21.15.88:0/3448653321 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7e600664a0 0x7f7e60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:03.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.324+0000 7f7e5bfff640 1 -- 172.21.15.88:0/3448653321 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f7e64084790 con 0x7f7e740c7e40 2023-10-16T20:58:03.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.328+0000 7f7e727fc640 1 --2- 172.21.15.88:0/3448653321 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7e600664a0 0x7f7e60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:03.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.328+0000 7f7e727fc640 1 --2- 172.21.15.88:0/3448653321 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7e600664a0 0x7f7e60068960 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7e680097c0 tx=0x7f7e68008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:03.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.328+0000 7f7e5bfff640 1 -- 172.21.15.88:0/3448653321 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7e64051580 con 0x7f7e740c7e40 2023-10-16T20:58:03.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.624+0000 7f7e7999d640 1 -- 172.21.15.88:0/3448653321 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e7406a350 con 0x7f7e740c7e40 2023-10-16T20:58:03.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.624+0000 7f7e5bfff640 1 -- 172.21.15.88:0/3448653321 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5786 (secure 0 0 0) 0x7f7e64054ba0 con 0x7f7e740c7e40 2023-10-16T20:58:03.625 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:03.625 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:58:00.653223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"c","rank":4,"incarnation":112,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:58:03.625 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 112 2023-10-16T20:58:03.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.628+0000 7f7e7999d640 1 -- 172.21.15.88:0/3448653321 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7e600664a0 msgr2=0x7f7e60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:03.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.628+0000 7f7e7999d640 1 --2- 172.21.15.88:0/3448653321 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7e600664a0 0x7f7e60068960 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7e680097c0 tx=0x7f7e68008040 comp rx=0 tx=0).stop 2023-10-16T20:58:03.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.628+0000 7f7e7999d640 1 -- 172.21.15.88:0/3448653321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e740c7e40 msgr2=0x7f7e7415b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:03.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.628+0000 7f7e7999d640 1 --2- 172.21.15.88:0/3448653321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e740c7e40 0x7f7e7415b960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400e8e0 tx=0x7f7e6400eda0 comp rx=0 tx=0).stop 2023-10-16T20:58:03.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.628+0000 7f7e7999d640 1 -- 172.21.15.88:0/3448653321 shutdown_connections 2023-10-16T20:58:03.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.628+0000 7f7e7999d640 1 --2- 172.21.15.88:0/3448653321 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7e600664a0 0x7f7e60068960 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:03.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.628+0000 7f7e7999d640 1 --2- 172.21.15.88:0/3448653321 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e741538f0 0x7f7e741f23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:03.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.628+0000 7f7e7999d640 1 --2- 172.21.15.88:0/3448653321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e740c9790 0x7f7e7414a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:03.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.628+0000 7f7e7999d640 1 --2- 172.21.15.88:0/3448653321 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7e740c7e40 0x7f7e7415b960 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:03.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.628+0000 7f7e7999d640 1 -- 172.21.15.88:0/3448653321 >> 172.21.15.88:0/3448653321 conn(0x7f7e7406f2c0 msgr2=0x7f7e740c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:03.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.628+0000 7f7e7999d640 1 -- 172.21.15.88:0/3448653321 shutdown_connections 2023-10-16T20:58:03.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:03.632+0000 7f7e7999d640 1 -- 172.21.15.88:0/3448653321 wait complete. 2023-10-16T20:58:03.659 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:58:05.662 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:58:05.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.808+0000 7ff5106ea640 1 Processor -- start 2023-10-16T20:58:05.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.808+0000 7ff5106ea640 1 -- start start 2023-10-16T20:58:05.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.812+0000 7ff5106ea640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff5080c9bb0 0x7ff5080c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:05.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.812+0000 7ff5106ea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff508153b50 con 0x7ff508153770 2023-10-16T20:58:05.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.812+0000 7ff5106ea640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5080cb2f0 con 0x7ff5080c41f0 2023-10-16T20:58:05.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.812+0000 7ff5106ea640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5080cc250 con 0x7ff5080c9bb0 2023-10-16T20:58:05.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.812+0000 7ff50e486640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff508153770 0x7ff508159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45800/0 (socket says 172.21.15.88:45800) 2023-10-16T20:58:05.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.812+0000 7ff50e486640 1 -- 172.21.15.88:0/837030150 learned_addr learned my addr 172.21.15.88:0/837030150 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:05.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.812+0000 7ff50dc85640 1 --2- 172.21.15.88:0/837030150 >> v2:172.21.15.191:3300/0 conn(0x7ff5080c9bb0 0x7ff5080c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:05.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.812+0000 7ff50d484640 1 -- 172.21.15.88:0/837030150 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973501538 0 0) 0x7ff508153b50 con 0x7ff508153770 2023-10-16T20:58:05.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.812+0000 7ff50d484640 1 -- 172.21.15.88:0/837030150 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4f4003530 con 0x7ff508153770 2023-10-16T20:58:05.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.812+0000 7ff50dc85640 1 -- 172.21.15.88:0/837030150 >> v1:172.21.15.191:6789/0 conn(0x7ff5080c41f0 legacy=0x7ff5080cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:05.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.812+0000 7ff50d484640 1 -- 172.21.15.88:0/837030150 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346775992 0 0) 0x7ff5080cb2f0 con 0x7ff5080c41f0 2023-10-16T20:58:05.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.812+0000 7ff50dc85640 1 -- 172.21.15.88:0/837030150 >> v1:172.21.15.88:6789/0 conn(0x7ff508153770 legacy=0x7ff508159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:05.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.812+0000 7ff50dc85640 1 -- 172.21.15.88:0/837030150 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5080cb5d0 con 0x7ff5080c9bb0 2023-10-16T20:58:05.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.816+0000 7ff50dc85640 1 --2- 172.21.15.88:0/837030150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5080c9bb0 0x7ff5080c3cb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff4f80097c0 tx=0x7ff4f801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54022b6c64da6fa2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:05.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.816+0000 7ff50d484640 1 -- 172.21.15.88:0/837030150 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f80a8050 con 0x7ff5080c9bb0 2023-10-16T20:58:05.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.816+0000 7ff50d484640 1 -- 172.21.15.88:0/837030150 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f80ac5f0 con 0x7ff5080c9bb0 2023-10-16T20:58:05.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.816+0000 7ff50d484640 1 -- 172.21.15.88:0/837030150 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f80ac980 con 0x7ff5080c9bb0 2023-10-16T20:58:05.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.816+0000 7ff5106ea640 1 -- 172.21.15.88:0/837030150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5080c9bb0 msgr2=0x7ff5080c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:05.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.816+0000 7ff5106ea640 1 --2- 172.21.15.88:0/837030150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5080c9bb0 0x7ff5080c3cb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff4f80097c0 tx=0x7ff4f801c4a0 comp rx=0 tx=0).stop 2023-10-16T20:58:05.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.816+0000 7ff5106ea640 1 -- 172.21.15.88:0/837030150 shutdown_connections 2023-10-16T20:58:05.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.816+0000 7ff5106ea640 1 --2- 172.21.15.88:0/837030150 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff5080c9bb0 0x7ff5080c3cb0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:05.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.816+0000 7ff5106ea640 1 -- 172.21.15.88:0/837030150 >> 172.21.15.88:0/837030150 conn(0x7ff5080715c0 msgr2=0x7ff5080719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:05.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.816+0000 7ff5106ea640 1 -- 172.21.15.88:0/837030150 shutdown_connections 2023-10-16T20:58:05.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.816+0000 7ff5106ea640 1 -- 172.21.15.88:0/837030150 wait complete. 2023-10-16T20:58:05.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff5106ea640 1 Processor -- start 2023-10-16T20:58:05.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff5106ea640 1 -- start start 2023-10-16T20:58:05.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff5106ea640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5080c41f0 0x7ff5080c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:05.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff5106ea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5080c9bb0 0x7ff5081f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:05.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff5106ea640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff508153770 0x7ff5081f35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:05.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff5106ea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50814d690 con 0x7ff5080c9bb0 2023-10-16T20:58:05.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff5106ea640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50814f8f0 con 0x7ff508153770 2023-10-16T20:58:05.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff5106ea640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50814d0d0 con 0x7ff5080c41f0 2023-10-16T20:58:05.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff50dc85640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5080c9bb0 0x7ff5081f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:05.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff50dc85640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5080c9bb0 0x7ff5081f3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33100/0 (socket says 172.21.15.88:33100) 2023-10-16T20:58:05.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff50dc85640 1 -- 172.21.15.88:0/810157745 learned_addr learned my addr 172.21.15.88:0/810157745 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:05.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff50ec87640 1 --2- 172.21.15.88:0/810157745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff508153770 0x7ff5081f35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:05.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff50e486640 1 --2- 172.21.15.88:0/810157745 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5080c41f0 0x7ff5080c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:05.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff50dc85640 1 -- 172.21.15.88:0/810157745 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5080c41f0 msgr2=0x7ff5080c84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:05.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff50dc85640 1 --2- 172.21.15.88:0/810157745 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5080c41f0 0x7ff5080c84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:05.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff50dc85640 1 -- 172.21.15.88:0/810157745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff508153770 msgr2=0x7ff5081f35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:05.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff50dc85640 1 --2- 172.21.15.88:0/810157745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff508153770 0x7ff5081f35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:05.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff50dc85640 1 -- 172.21.15.88:0/810157745 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f8009420 con 0x7ff5080c9bb0 2023-10-16T20:58:05.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.820+0000 7ff50dc85640 1 --2- 172.21.15.88:0/810157745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5080c9bb0 0x7ff5081f3090 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7ff4f80097c0 tx=0x7ff4f8019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:05.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.824+0000 7ff4feffd640 1 -- 172.21.15.88:0/810157745 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8006e20 con 0x7ff5080c9bb0 2023-10-16T20:58:05.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.824+0000 7ff4feffd640 1 -- 172.21.15.88:0/810157745 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f8004030 con 0x7ff5080c9bb0 2023-10-16T20:58:05.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.824+0000 7ff4feffd640 1 -- 172.21.15.88:0/810157745 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f80043c0 con 0x7ff5080c9bb0 2023-10-16T20:58:05.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.824+0000 7ff5106ea640 1 -- 172.21.15.88:0/810157745 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50814d3b0 con 0x7ff5080c9bb0 2023-10-16T20:58:05.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.824+0000 7ff5106ea640 1 -- 172.21.15.88:0/810157745 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5081f2b70 con 0x7ff5080c9bb0 2023-10-16T20:58:05.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.824+0000 7ff5106ea640 1 -- 172.21.15.88:0/810157745 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5081f26f0 con 0x7ff5080c9bb0 2023-10-16T20:58:05.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.824+0000 7ff4feffd640 1 -- 172.21.15.88:0/810157745 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff4f80b3040 con 0x7ff5080c9bb0 2023-10-16T20:58:05.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.824+0000 7ff4feffd640 1 --2- 172.21.15.88:0/810157745 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff4ec0664d0 0x7ff4ec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:05.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.824+0000 7ff4feffd640 1 -- 172.21.15.88:0/810157745 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7ff4f811d460 con 0x7ff5080c9bb0 2023-10-16T20:58:05.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.824+0000 7ff50e486640 1 --2- 172.21.15.88:0/810157745 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff4ec0664d0 0x7ff4ec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:05.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.828+0000 7ff50e486640 1 --2- 172.21.15.88:0/810157745 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff4ec0664d0 0x7ff4ec068990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff504001370 tx=0x7ff504009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:05.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:05.832+0000 7ff4feffd640 1 -- 172.21.15.88:0/810157745 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5081f26f0 con 0x7ff5080c9bb0 2023-10-16T20:58:06.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:06.052+0000 7ff5106ea640 1 -- 172.21.15.88:0/810157745 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5081f26f0 con 0x7ff5080c9bb0 2023-10-16T20:58:06.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:06.052+0000 7ff4feffd640 1 -- 172.21.15.88:0/810157745 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5786 (secure 0 0 0) 0x7ff5081f26f0 con 0x7ff5080c9bb0 2023-10-16T20:58:06.051 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:06.051 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:58:00.653223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"c","rank":4,"incarnation":112,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:58:06.052 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 112 2023-10-16T20:58:06.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:06.052+0000 7ff5106ea640 1 -- 172.21.15.88:0/810157745 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff4ec0664d0 msgr2=0x7ff4ec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:06.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:06.052+0000 7ff5106ea640 1 --2- 172.21.15.88:0/810157745 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff4ec0664d0 0x7ff4ec068990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff504001370 tx=0x7ff504009040 comp rx=0 tx=0).stop 2023-10-16T20:58:06.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:06.052+0000 7ff5106ea640 1 -- 172.21.15.88:0/810157745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5080c9bb0 msgr2=0x7ff5081f3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:06.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:06.052+0000 7ff5106ea640 1 --2- 172.21.15.88:0/810157745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5080c9bb0 0x7ff5081f3090 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7ff4f80097c0 tx=0x7ff4f8019240 comp rx=0 tx=0).stop 2023-10-16T20:58:06.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:06.052+0000 7ff5106ea640 1 -- 172.21.15.88:0/810157745 shutdown_connections 2023-10-16T20:58:06.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:06.052+0000 7ff5106ea640 1 --2- 172.21.15.88:0/810157745 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff4ec0664d0 0x7ff4ec068990 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:06.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:06.052+0000 7ff5106ea640 1 --2- 172.21.15.88:0/810157745 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff508153770 0x7ff5081f35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:06.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:06.052+0000 7ff5106ea640 1 --2- 172.21.15.88:0/810157745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5080c9bb0 0x7ff5081f3090 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:06.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:06.052+0000 7ff5106ea640 1 --2- 172.21.15.88:0/810157745 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff5080c41f0 0x7ff5080c84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:06.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:06.052+0000 7ff5106ea640 1 -- 172.21.15.88:0/810157745 >> 172.21.15.88:0/810157745 conn(0x7ff5080715c0 msgr2=0x7ff508150dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:06.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:06.052+0000 7ff5106ea640 1 -- 172.21.15.88:0/810157745 shutdown_connections 2023-10-16T20:58:06.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:06.052+0000 7ff5106ea640 1 -- 172.21.15.88:0/810157745 wait complete. 2023-10-16T20:58:06.071 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:58:08.072 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:58:08.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.264+0000 7feaac7aa640 1 Processor -- start 2023-10-16T20:58:08.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.264+0000 7feaac7aa640 1 -- start start 2023-10-16T20:58:08.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.268+0000 7feaac7aa640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaa40c9bb0 0x7feaa40c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:08.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.268+0000 7feaac7aa640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa4153b50 con 0x7feaa4153770 2023-10-16T20:58:08.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.268+0000 7feaac7aa640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa40cb2f0 con 0x7feaa40c41f0 2023-10-16T20:58:08.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.268+0000 7feaac7aa640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaa40cc250 con 0x7feaa40c9bb0 2023-10-16T20:58:08.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.268+0000 7feaaa546640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7feaa4153770 0x7feaa4159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45806/0 (socket says 172.21.15.88:45806) 2023-10-16T20:58:08.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.268+0000 7feaaa546640 1 -- 172.21.15.88:0/1578759242 learned_addr learned my addr 172.21.15.88:0/1578759242 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:08.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.268+0000 7feaa9d45640 1 --2- 172.21.15.88:0/1578759242 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaa40c9bb0 0x7feaa40c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:08.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.268+0000 7feaa9d45640 1 -- 172.21.15.88:0/1578759242 >> v1:172.21.15.191:6789/0 conn(0x7feaa40c41f0 legacy=0x7feaa40cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:08.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.268+0000 7feaa9544640 1 -- 172.21.15.88:0/1578759242 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779303257 0 0) 0x7feaa4153b50 con 0x7feaa4153770 2023-10-16T20:58:08.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.268+0000 7feaa9d45640 1 -- 172.21.15.88:0/1578759242 >> v1:172.21.15.88:6789/0 conn(0x7feaa4153770 legacy=0x7feaa4159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:08.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.268+0000 7feaa9d45640 1 -- 172.21.15.88:0/1578759242 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa40cb5d0 con 0x7feaa40c9bb0 2023-10-16T20:58:08.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.272+0000 7feaa9d45640 1 --2- 172.21.15.88:0/1578759242 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaa40c9bb0 0x7feaa40c3cb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feaa0009770 tx=0x7feaa001c2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb246b399b692e74 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:08.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.272+0000 7feaa9544640 1 -- 172.21.15.88:0/1578759242 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa00ab600 con 0x7feaa40c9bb0 2023-10-16T20:58:08.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.272+0000 7feaa9544640 1 -- 172.21.15.88:0/1578759242 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa00a7050 con 0x7feaa40c9bb0 2023-10-16T20:58:08.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.272+0000 7feaa9544640 1 -- 172.21.15.88:0/1578759242 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa00ab980 con 0x7feaa40c9bb0 2023-10-16T20:58:08.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.272+0000 7feaac7aa640 1 -- 172.21.15.88:0/1578759242 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaa40c9bb0 msgr2=0x7feaa40c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:08.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.272+0000 7feaac7aa640 1 --2- 172.21.15.88:0/1578759242 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaa40c9bb0 0x7feaa40c3cb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feaa0009770 tx=0x7feaa001c2f0 comp rx=0 tx=0).stop 2023-10-16T20:58:08.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.272+0000 7feaac7aa640 1 -- 172.21.15.88:0/1578759242 shutdown_connections 2023-10-16T20:58:08.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.272+0000 7feaac7aa640 1 --2- 172.21.15.88:0/1578759242 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaa40c9bb0 0x7feaa40c3cb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:08.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.272+0000 7feaac7aa640 1 -- 172.21.15.88:0/1578759242 >> 172.21.15.88:0/1578759242 conn(0x7feaa40715c0 msgr2=0x7feaa40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:08.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.272+0000 7feaac7aa640 1 -- 172.21.15.88:0/1578759242 shutdown_connections 2023-10-16T20:58:08.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.272+0000 7feaac7aa640 1 -- 172.21.15.88:0/1578759242 wait complete. 2023-10-16T20:58:08.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaac7aa640 1 Processor -- start 2023-10-16T20:58:08.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaac7aa640 1 -- start start 2023-10-16T20:58:08.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaac7aa640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feaa40c41f0 0x7feaa415c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:08.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaac7aa640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaa40c9bb0 0x7feaa40b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:08.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaac7aa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feaa4153770 0x7feaa4147370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:08.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaac7aa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa415cfe0 con 0x7feaa4153770 2023-10-16T20:58:08.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaac7aa640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa415a610 con 0x7feaa40c41f0 2023-10-16T20:58:08.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaac7aa640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaa4157940 con 0x7feaa40c9bb0 2023-10-16T20:58:08.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaaad47640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feaa4153770 0x7feaa4147370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:08.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaa9d45640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaa40c9bb0 0x7feaa40b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:08.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaaad47640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feaa4153770 0x7feaa4147370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33112/0 (socket says 172.21.15.88:33112) 2023-10-16T20:58:08.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaaad47640 1 -- 172.21.15.88:0/2547268870 learned_addr learned my addr 172.21.15.88:0/2547268870 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:08.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaaa546640 1 --2- 172.21.15.88:0/2547268870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feaa40c41f0 0x7feaa415c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:08.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaa9d45640 1 -- 172.21.15.88:0/2547268870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feaa40c41f0 msgr2=0x7feaa415c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:08.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaa9d45640 1 --2- 172.21.15.88:0/2547268870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feaa40c41f0 0x7feaa415c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:08.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaa9d45640 1 -- 172.21.15.88:0/2547268870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feaa4153770 msgr2=0x7feaa4147370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:08.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaa9d45640 1 --2- 172.21.15.88:0/2547268870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feaa4153770 0x7feaa4147370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:08.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.276+0000 7feaa9d45640 1 -- 172.21.15.88:0/2547268870 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa0009420 con 0x7feaa40c9bb0 2023-10-16T20:58:08.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.280+0000 7feaa9d45640 1 --2- 172.21.15.88:0/2547268870 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaa40c9bb0 0x7feaa40b39b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7feaa00ab000 tx=0x7feaa001ab30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:08.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.280+0000 7fea9affd640 1 -- 172.21.15.88:0/2547268870 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa00b7650 con 0x7feaa40c9bb0 2023-10-16T20:58:08.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.280+0000 7fea9affd640 1 -- 172.21.15.88:0/2547268870 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa00a4e90 con 0x7feaa40c9bb0 2023-10-16T20:58:08.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.280+0000 7feaac7aa640 1 -- 172.21.15.88:0/2547268870 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa40c9890 con 0x7feaa40c9bb0 2023-10-16T20:58:08.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.280+0000 7fea9affd640 1 -- 172.21.15.88:0/2547268870 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa00b79b0 con 0x7feaa40c9bb0 2023-10-16T20:58:08.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.280+0000 7feaac7aa640 1 -- 172.21.15.88:0/2547268870 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa415ab40 con 0x7feaa40c9bb0 2023-10-16T20:58:08.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.280+0000 7feaac7aa640 1 -- 172.21.15.88:0/2547268870 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa4074700 con 0x7feaa40c9bb0 2023-10-16T20:58:08.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.280+0000 7fea9affd640 1 -- 172.21.15.88:0/2547268870 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feaa00b7b10 con 0x7feaa40c9bb0 2023-10-16T20:58:08.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.284+0000 7fea9affd640 1 --2- 172.21.15.88:0/2547268870 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fea880664f0 0x7fea880689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:08.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.284+0000 7fea9affd640 1 -- 172.21.15.88:0/2547268870 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7feaa011f020 con 0x7feaa40c9bb0 2023-10-16T20:58:08.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.284+0000 7feaaa546640 1 --2- 172.21.15.88:0/2547268870 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fea880664f0 0x7fea880689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:08.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.284+0000 7feaaa546640 1 --2- 172.21.15.88:0/2547268870 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fea880664f0 0x7fea880689b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fea94000c10 tx=0x7fea94009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:08.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.288+0000 7fea9affd640 1 -- 172.21.15.88:0/2547268870 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feaa00ebfa0 con 0x7feaa40c9bb0 2023-10-16T20:58:08.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.584+0000 7feaac7aa640 1 -- 172.21.15.88:0/2547268870 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feaa40ca740 con 0x7feaa40c9bb0 2023-10-16T20:58:08.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.584+0000 7fea9affd640 1 -- 172.21.15.88:0/2547268870 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+5788 (secure 0 0 0) 0x7feaa00b0090 con 0x7feaa40c9bb0 2023-10-16T20:58:08.585 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:08.586 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:58:05.974113+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"c","rank":4,"incarnation":112,"state":"up:resolve","state_seq":17,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:58:08.586 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 113 2023-10-16T20:58:08.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.588+0000 7feaac7aa640 1 -- 172.21.15.88:0/2547268870 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fea880664f0 msgr2=0x7fea880689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:08.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.588+0000 7feaac7aa640 1 --2- 172.21.15.88:0/2547268870 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fea880664f0 0x7fea880689b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fea94000c10 tx=0x7fea94009040 comp rx=0 tx=0).stop 2023-10-16T20:58:08.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.588+0000 7feaac7aa640 1 -- 172.21.15.88:0/2547268870 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaa40c9bb0 msgr2=0x7feaa40b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:08.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.588+0000 7feaac7aa640 1 --2- 172.21.15.88:0/2547268870 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaa40c9bb0 0x7feaa40b39b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7feaa00ab000 tx=0x7feaa001ab30 comp rx=0 tx=0).stop 2023-10-16T20:58:08.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.588+0000 7feaac7aa640 1 -- 172.21.15.88:0/2547268870 shutdown_connections 2023-10-16T20:58:08.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.588+0000 7feaac7aa640 1 --2- 172.21.15.88:0/2547268870 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fea880664f0 0x7fea880689b0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:08.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.588+0000 7feaac7aa640 1 --2- 172.21.15.88:0/2547268870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feaa4153770 0x7feaa4147370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:08.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.588+0000 7feaac7aa640 1 --2- 172.21.15.88:0/2547268870 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feaa40c9bb0 0x7feaa40b39b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:08.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.588+0000 7feaac7aa640 1 --2- 172.21.15.88:0/2547268870 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feaa40c41f0 0x7feaa415c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:08.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.588+0000 7feaac7aa640 1 -- 172.21.15.88:0/2547268870 >> 172.21.15.88:0/2547268870 conn(0x7feaa40715c0 msgr2=0x7feaa40719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:08.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.588+0000 7feaac7aa640 1 -- 172.21.15.88:0/2547268870 shutdown_connections 2023-10-16T20:58:08.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:08.588+0000 7feaac7aa640 1 -- 172.21.15.88:0/2547268870 wait complete. 2023-10-16T20:58:08.613 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:58:10.614 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:58:10.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.755+0000 7fc975617640 1 Processor -- start 2023-10-16T20:58:10.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.755+0000 7fc975617640 1 -- start start 2023-10-16T20:58:10.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.755+0000 7fc975617640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc9700cece0 0x7fc9700cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:10.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.755+0000 7fc975617640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc970153b50 con 0x7fc970153770 2023-10-16T20:58:10.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.755+0000 7fc975617640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9700cb2f0 con 0x7fc9700c9bb0 2023-10-16T20:58:10.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.755+0000 7fc975617640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9700cc250 con 0x7fc9700cece0 2023-10-16T20:58:10.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.755+0000 7fc96effd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc970153770 0x7fc970159ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49460/0 (socket says 172.21.15.88:49460) 2023-10-16T20:58:10.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.755+0000 7fc96effd640 1 -- 172.21.15.88:0/2827610265 learned_addr learned my addr 172.21.15.88:0/2827610265 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:10.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.755+0000 7fc96f7fe640 1 --2- 172.21.15.88:0/2827610265 >> v2:172.21.15.191:3300/0 conn(0x7fc9700cece0 0x7fc9700cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:10.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.755+0000 7fc96e7fc640 1 -- 172.21.15.88:0/2827610265 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313239126 0 0) 0x7fc970153b50 con 0x7fc970153770 2023-10-16T20:58:10.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.755+0000 7fc96e7fc640 1 -- 172.21.15.88:0/2827610265 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc950003530 con 0x7fc970153770 2023-10-16T20:58:10.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.755+0000 7fc96f7fe640 1 -- 172.21.15.88:0/2827610265 >> v1:172.21.15.191:6789/0 conn(0x7fc9700c9bb0 legacy=0x7fc9700c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:10.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.755+0000 7fc96e7fc640 1 -- 172.21.15.88:0/2827610265 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406284949 0 0) 0x7fc9700cb2f0 con 0x7fc9700c9bb0 2023-10-16T20:58:10.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.755+0000 7fc96f7fe640 1 -- 172.21.15.88:0/2827610265 >> v1:172.21.15.88:6789/0 conn(0x7fc970153770 legacy=0x7fc970159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:10.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.755+0000 7fc96f7fe640 1 -- 172.21.15.88:0/2827610265 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9700cb5d0 con 0x7fc9700cece0 2023-10-16T20:58:10.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.759+0000 7fc96f7fe640 1 --2- 172.21.15.88:0/2827610265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9700cece0 0x7fc9700cf130 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc968009d90 tx=0x7fc96801c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=217f9ef4fd3bf77b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:10.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.759+0000 7fc96e7fc640 1 -- 172.21.15.88:0/2827610265 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9680a5920 con 0x7fc9700cece0 2023-10-16T20:58:10.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.759+0000 7fc96e7fc640 1 -- 172.21.15.88:0/2827610265 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9680a5a80 con 0x7fc9700cece0 2023-10-16T20:58:10.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.759+0000 7fc96e7fc640 1 -- 172.21.15.88:0/2827610265 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9680a5e10 con 0x7fc9700cece0 2023-10-16T20:58:10.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.759+0000 7fc975617640 1 -- 172.21.15.88:0/2827610265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9700cece0 msgr2=0x7fc9700cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:10.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.759+0000 7fc975617640 1 --2- 172.21.15.88:0/2827610265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9700cece0 0x7fc9700cf130 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc968009d90 tx=0x7fc96801c950 comp rx=0 tx=0).stop 2023-10-16T20:58:10.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.759+0000 7fc975617640 1 -- 172.21.15.88:0/2827610265 shutdown_connections 2023-10-16T20:58:10.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.759+0000 7fc975617640 1 --2- 172.21.15.88:0/2827610265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9700cece0 0x7fc9700cf130 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:10.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.759+0000 7fc975617640 1 -- 172.21.15.88:0/2827610265 >> 172.21.15.88:0/2827610265 conn(0x7fc9700715c0 msgr2=0x7fc9700719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:10.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.759+0000 7fc975617640 1 -- 172.21.15.88:0/2827610265 shutdown_connections 2023-10-16T20:58:10.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.759+0000 7fc975617640 1 -- 172.21.15.88:0/2827610265 wait complete. 2023-10-16T20:58:10.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc975617640 1 Processor -- start 2023-10-16T20:58:10.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc975617640 1 -- start start 2023-10-16T20:58:10.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc975617640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9700c9bb0 0x7fc97013c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:10.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc975617640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9700cece0 0x7fc9700b4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:10.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc975617640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc970153770 0x7fc9701f1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:10.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc975617640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97013cc70 con 0x7fc970153770 2023-10-16T20:58:10.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc975617640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9701f15e0 con 0x7fc9700c9bb0 2023-10-16T20:58:10.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc975617640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97013c9e0 con 0x7fc9700cece0 2023-10-16T20:58:10.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc96f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc970153770 0x7fc9701f1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:10.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc96f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc970153770 0x7fc9701f1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55086/0 (socket says 172.21.15.88:55086) 2023-10-16T20:58:10.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc96f7fe640 1 -- 172.21.15.88:0/2887741629 learned_addr learned my addr 172.21.15.88:0/2887741629 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:10.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc96effd640 1 --2- 172.21.15.88:0/2887741629 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9700c9bb0 0x7fc97013c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:10.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc9667fc640 1 --2- 172.21.15.88:0/2887741629 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9700cece0 0x7fc9700b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:10.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc96f7fe640 1 -- 172.21.15.88:0/2887741629 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9700cece0 msgr2=0x7fc9700b4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:10.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc96f7fe640 1 --2- 172.21.15.88:0/2887741629 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9700cece0 0x7fc9700b4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:10.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc96f7fe640 1 -- 172.21.15.88:0/2887741629 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9700c9bb0 msgr2=0x7fc97013c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:10.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc96f7fe640 1 --2- 172.21.15.88:0/2887741629 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9700c9bb0 0x7fc97013c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:10.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc96f7fe640 1 -- 172.21.15.88:0/2887741629 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc968009a70 con 0x7fc970153770 2023-10-16T20:58:10.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc96f7fe640 1 --2- 172.21.15.88:0/2887741629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc970153770 0x7fc9701f1aa0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc968009d90 tx=0x7fc9680196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:10.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc967fff640 1 -- 172.21.15.88:0/2887741629 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9680add50 con 0x7fc970153770 2023-10-16T20:58:10.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc967fff640 1 -- 172.21.15.88:0/2887741629 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc968009450 con 0x7fc970153770 2023-10-16T20:58:10.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc967fff640 1 -- 172.21.15.88:0/2887741629 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc968003ba0 con 0x7fc970153770 2023-10-16T20:58:10.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc975617640 1 -- 172.21.15.88:0/2887741629 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97013eea0 con 0x7fc970153770 2023-10-16T20:58:10.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc975617640 1 -- 172.21.15.88:0/2887741629 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9700b39d0 con 0x7fc970153770 2023-10-16T20:58:10.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.763+0000 7fc975617640 1 -- 172.21.15.88:0/2887741629 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc970074700 con 0x7fc970153770 2023-10-16T20:58:10.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.767+0000 7fc967fff640 1 -- 172.21.15.88:0/2887741629 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc9680c4450 con 0x7fc970153770 2023-10-16T20:58:10.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.767+0000 7fc967fff640 1 --2- 172.21.15.88:0/2887741629 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc9540664a0 0x7fc954068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:10.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.767+0000 7fc967fff640 1 -- 172.21.15.88:0/2887741629 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7fc96811f0e0 con 0x7fc970153770 2023-10-16T20:58:10.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.767+0000 7fc96effd640 1 --2- 172.21.15.88:0/2887741629 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc9540664a0 0x7fc954068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:10.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.767+0000 7fc96effd640 1 --2- 172.21.15.88:0/2887741629 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc9540664a0 0x7fc954068960 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc960001370 tx=0x7fc960009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:10.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.771+0000 7fc967fff640 1 -- 172.21.15.88:0/2887741629 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc9680ebdc0 con 0x7fc970153770 2023-10-16T20:58:10.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.955+0000 7fc975617640 1 -- 172.21.15.88:0/2887741629 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9701f1750 con 0x7fc970153770 2023-10-16T20:58:10.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.955+0000 7fc967fff640 1 -- 172.21.15.88:0/2887741629 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+5788 (secure 0 0 0) 0x7fc9680ef3e0 con 0x7fc970153770 2023-10-16T20:58:10.955 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:10.955 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:58:05.974113+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"c","rank":4,"incarnation":112,"state":"up:resolve","state_seq":17,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:58:10.956 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 113 2023-10-16T20:58:10.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.955+0000 7fc975617640 1 -- 172.21.15.88:0/2887741629 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc9540664a0 msgr2=0x7fc954068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:10.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.955+0000 7fc975617640 1 --2- 172.21.15.88:0/2887741629 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc9540664a0 0x7fc954068960 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc960001370 tx=0x7fc960009040 comp rx=0 tx=0).stop 2023-10-16T20:58:10.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.955+0000 7fc975617640 1 -- 172.21.15.88:0/2887741629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc970153770 msgr2=0x7fc9701f1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:10.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.955+0000 7fc975617640 1 --2- 172.21.15.88:0/2887741629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc970153770 0x7fc9701f1aa0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc968009d90 tx=0x7fc9680196f0 comp rx=0 tx=0).stop 2023-10-16T20:58:10.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.955+0000 7fc975617640 1 -- 172.21.15.88:0/2887741629 shutdown_connections 2023-10-16T20:58:10.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.955+0000 7fc975617640 1 --2- 172.21.15.88:0/2887741629 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc9540664a0 0x7fc954068960 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:10.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.955+0000 7fc975617640 1 --2- 172.21.15.88:0/2887741629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc970153770 0x7fc9701f1aa0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:10.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.955+0000 7fc975617640 1 --2- 172.21.15.88:0/2887741629 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc9700cece0 0x7fc9700b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:10.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.955+0000 7fc975617640 1 --2- 172.21.15.88:0/2887741629 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc9700c9bb0 0x7fc97013c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:10.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.955+0000 7fc975617640 1 -- 172.21.15.88:0/2887741629 >> 172.21.15.88:0/2887741629 conn(0x7fc9700715c0 msgr2=0x7fc9700cf0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:10.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.955+0000 7fc975617640 1 -- 172.21.15.88:0/2887741629 shutdown_connections 2023-10-16T20:58:10.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:10.955+0000 7fc975617640 1 -- 172.21.15.88:0/2887741629 wait complete. 2023-10-16T20:58:10.976 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:58:12.977 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:58:13.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.135+0000 7fad160dc640 1 Processor -- start 2023-10-16T20:58:13.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.135+0000 7fad160dc640 1 -- start start 2023-10-16T20:58:13.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.135+0000 7fad160dc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad100c9af0 0x7fad100cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:13.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.135+0000 7fad160dc640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad100c9fc0 con 0x7fad100cb580 2023-10-16T20:58:13.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.135+0000 7fad160dc640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad100c47d0 con 0x7fad100b1a60 2023-10-16T20:58:13.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.135+0000 7fad160dc640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad100b20c0 con 0x7fad100c9af0 2023-10-16T20:58:13.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.135+0000 7fad150da640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fad100cb580 0x7fad10159300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49472/0 (socket says 172.21.15.88:49472) 2023-10-16T20:58:13.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.135+0000 7fad150da640 1 -- 172.21.15.88:0/3388403779 learned_addr learned my addr 172.21.15.88:0/3388403779 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:13.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.135+0000 7fad148d9640 1 --2- 172.21.15.88:0/3388403779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad100c9af0 0x7fad100cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:13.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.135+0000 7fad07fff640 1 -- 172.21.15.88:0/3388403779 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715830885 0 0) 0x7fad100c47d0 con 0x7fad100b1a60 2023-10-16T20:58:13.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.135+0000 7fad07fff640 1 -- 172.21.15.88:0/3388403779 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facfc003530 con 0x7fad100b1a60 2023-10-16T20:58:13.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.135+0000 7fad07fff640 1 -- 172.21.15.88:0/3388403779 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718460674 0 0) 0x7fad100c9fc0 con 0x7fad100cb580 2023-10-16T20:58:13.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.135+0000 7fad148d9640 1 -- 172.21.15.88:0/3388403779 >> v1:172.21.15.191:6789/0 conn(0x7fad100b1a60 legacy=0x7fad10163b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:13.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.135+0000 7fad148d9640 1 -- 172.21.15.88:0/3388403779 >> v1:172.21.15.88:6789/0 conn(0x7fad100cb580 legacy=0x7fad10159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:13.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.135+0000 7fad148d9640 1 -- 172.21.15.88:0/3388403779 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad100c4ab0 con 0x7fad100c9af0 2023-10-16T20:58:13.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.139+0000 7fad148d9640 1 --2- 172.21.15.88:0/3388403779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad100c9af0 0x7fad100cb960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fad000097f0 tx=0x7fad0001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd2411b6a180931a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:13.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.139+0000 7fad07fff640 1 -- 172.21.15.88:0/3388403779 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad000ab600 con 0x7fad100c9af0 2023-10-16T20:58:13.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.139+0000 7fad07fff640 1 -- 172.21.15.88:0/3388403779 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad000a7050 con 0x7fad100c9af0 2023-10-16T20:58:13.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.139+0000 7fad07fff640 1 -- 172.21.15.88:0/3388403779 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad000ab980 con 0x7fad100c9af0 2023-10-16T20:58:13.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.143+0000 7fad160dc640 1 -- 172.21.15.88:0/3388403779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad100c9af0 msgr2=0x7fad100cb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:13.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.143+0000 7fad160dc640 1 --2- 172.21.15.88:0/3388403779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad100c9af0 0x7fad100cb960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fad000097f0 tx=0x7fad0001c450 comp rx=0 tx=0).stop 2023-10-16T20:58:13.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.143+0000 7fad160dc640 1 -- 172.21.15.88:0/3388403779 shutdown_connections 2023-10-16T20:58:13.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.143+0000 7fad160dc640 1 --2- 172.21.15.88:0/3388403779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad100c9af0 0x7fad100cb960 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:13.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.143+0000 7fad160dc640 1 -- 172.21.15.88:0/3388403779 >> 172.21.15.88:0/3388403779 conn(0x7fad1006bb40 msgr2=0x7fad1006bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:13.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.143+0000 7fad160dc640 1 -- 172.21.15.88:0/3388403779 shutdown_connections 2023-10-16T20:58:13.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.143+0000 7fad160dc640 1 -- 172.21.15.88:0/3388403779 wait complete. 2023-10-16T20:58:13.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.143+0000 7fad160dc640 1 Processor -- start 2023-10-16T20:58:13.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.143+0000 7fad160dc640 1 -- start start 2023-10-16T20:58:13.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.143+0000 7fad160dc640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad100b1a60 0x7fad10151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:13.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.143+0000 7fad160dc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad100c9af0 0x7fad10152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:13.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.143+0000 7fad160dc640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad100cb580 0x7fad101f3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:13.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.143+0000 7fad160dc640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad101445d0 con 0x7fad100c9af0 2023-10-16T20:58:13.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.143+0000 7fad160dc640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad100c5f90 con 0x7fad100cb580 2023-10-16T20:58:13.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.143+0000 7fad160dc640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad101f3370 con 0x7fad100b1a60 2023-10-16T20:58:13.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad148d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad100c9af0 0x7fad10152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:13.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad150da640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad100b1a60 0x7fad10151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:13.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad148d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad100c9af0 0x7fad10152330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55090/0 (socket says 172.21.15.88:55090) 2023-10-16T20:58:13.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad148d9640 1 -- 172.21.15.88:0/2537418410 learned_addr learned my addr 172.21.15.88:0/2537418410 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:13.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad158db640 1 --2- 172.21.15.88:0/2537418410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad100cb580 0x7fad101f3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:13.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad148d9640 1 -- 172.21.15.88:0/2537418410 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad100b1a60 msgr2=0x7fad10151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:13.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad148d9640 1 --2- 172.21.15.88:0/2537418410 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad100b1a60 0x7fad10151df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:13.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad148d9640 1 -- 172.21.15.88:0/2537418410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad100cb580 msgr2=0x7fad101f3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:13.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad148d9640 1 --2- 172.21.15.88:0/2537418410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad100cb580 0x7fad101f3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:13.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad148d9640 1 -- 172.21.15.88:0/2537418410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad00009420 con 0x7fad100c9af0 2023-10-16T20:58:13.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad148d9640 1 --2- 172.21.15.88:0/2537418410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad100c9af0 0x7fad10152330 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fad000ae040 tx=0x7fad0001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:13.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad05ffb640 1 -- 172.21.15.88:0/2537418410 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad000b7450 con 0x7fad100c9af0 2023-10-16T20:58:13.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad05ffb640 1 -- 172.21.15.88:0/2537418410 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad000b75d0 con 0x7fad100c9af0 2023-10-16T20:58:13.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad160dc640 1 -- 172.21.15.88:0/2537418410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad101447c0 con 0x7fad100c9af0 2023-10-16T20:58:13.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad05ffb640 1 -- 172.21.15.88:0/2537418410 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad000b7980 con 0x7fad100c9af0 2023-10-16T20:58:13.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad160dc640 1 -- 172.21.15.88:0/2537418410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad101f2d20 con 0x7fad100c9af0 2023-10-16T20:58:13.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad160dc640 1 -- 172.21.15.88:0/2537418410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad101f3500 con 0x7fad100c9af0 2023-10-16T20:58:13.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.147+0000 7fad05ffb640 1 -- 172.21.15.88:0/2537418410 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fad000b7ae0 con 0x7fad100c9af0 2023-10-16T20:58:13.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.151+0000 7fad05ffb640 1 --2- 172.21.15.88:0/2537418410 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7facf40664d0 0x7facf4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:13.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.151+0000 7fad05ffb640 1 -- 172.21.15.88:0/2537418410 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7fad0011edf0 con 0x7fad100c9af0 2023-10-16T20:58:13.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.151+0000 7fad150da640 1 --2- 172.21.15.88:0/2537418410 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7facf40664d0 0x7facf4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:13.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.151+0000 7fad150da640 1 --2- 172.21.15.88:0/2537418410 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7facf40664d0 0x7facf4068990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fad0c000f70 tx=0x7fad0c007240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:13.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.155+0000 7fad05ffb640 1 -- 172.21.15.88:0/2537418410 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fad000eba20 con 0x7fad100c9af0 2023-10-16T20:58:13.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.411+0000 7fad160dc640 1 -- 172.21.15.88:0/2537418410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad100ca2c0 con 0x7fad100c9af0 2023-10-16T20:58:13.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.415+0000 7fad05ffb640 1 -- 172.21.15.88:0/2537418410 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5790 (secure 0 0 0) 0x7fad000b0090 con 0x7fad100c9af0 2023-10-16T20:58:13.414 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:13.415 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:58:11.901154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"c","rank":4,"incarnation":112,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:58:13.415 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 114 2023-10-16T20:58:13.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.415+0000 7fad160dc640 1 -- 172.21.15.88:0/2537418410 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7facf40664d0 msgr2=0x7facf4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:13.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.415+0000 7fad160dc640 1 --2- 172.21.15.88:0/2537418410 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7facf40664d0 0x7facf4068990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fad0c000f70 tx=0x7fad0c007240 comp rx=0 tx=0).stop 2023-10-16T20:58:13.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.415+0000 7fad160dc640 1 -- 172.21.15.88:0/2537418410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad100c9af0 msgr2=0x7fad10152330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:13.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.415+0000 7fad160dc640 1 --2- 172.21.15.88:0/2537418410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad100c9af0 0x7fad10152330 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fad000ae040 tx=0x7fad0001ac90 comp rx=0 tx=0).stop 2023-10-16T20:58:13.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.415+0000 7fad160dc640 1 -- 172.21.15.88:0/2537418410 shutdown_connections 2023-10-16T20:58:13.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.415+0000 7fad160dc640 1 --2- 172.21.15.88:0/2537418410 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7facf40664d0 0x7facf4068990 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:13.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.415+0000 7fad160dc640 1 --2- 172.21.15.88:0/2537418410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fad100cb580 0x7fad101f3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:13.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.415+0000 7fad160dc640 1 --2- 172.21.15.88:0/2537418410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad100c9af0 0x7fad10152330 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:13.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.415+0000 7fad160dc640 1 --2- 172.21.15.88:0/2537418410 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fad100b1a60 0x7fad10151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:13.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.415+0000 7fad160dc640 1 -- 172.21.15.88:0/2537418410 >> 172.21.15.88:0/2537418410 conn(0x7fad1006bb40 msgr2=0x7fad100cdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:13.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.415+0000 7fad160dc640 1 -- 172.21.15.88:0/2537418410 shutdown_connections 2023-10-16T20:58:13.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:13.415+0000 7fad160dc640 1 -- 172.21.15.88:0/2537418410 wait complete. 2023-10-16T20:58:13.439 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 114, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-16T20:36:55.111692+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 114, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5477": { "addr": "172.21.15.191:6837/2244284333", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6836", "nonce": 2244284333, "type": "v2" }, { "addr": "172.21.15.191:6837", "nonce": 2244284333, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5477, "incarnation": 97, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 39 }, "gid_5535": { "addr": "172.21.15.88:6835/3476086754", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6834", "nonce": 3476086754, "type": "v2" }, { "addr": "172.21.15.88:6835", "nonce": 3476086754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5535, "incarnation": 52, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_5570": { "addr": "172.21.15.191:6833/3394064120", "addrs": { "addrvec": [ { "addr": "172.21.15.191:6832", "nonce": 3394064120, "type": "v2" }, { "addr": "172.21.15.191:6833", "nonce": 3394064120, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5570, "incarnation": 103, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 13 }, "gid_5709": { "addr": "172.21.15.88:6838/2600891060", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6836", "nonce": 2600891060, "type": "v2" }, { "addr": "172.21.15.88:6838", "nonce": 2600891060, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5709, "incarnation": 67, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 37 }, "gid_6366": { "addr": "172.21.15.88:6839/2182739466", "addrs": { "addrvec": [ { "addr": "172.21.15.88:6837", "nonce": 2182739466, "type": "v2" }, { "addr": "172.21.15.88:6839", "nonce": 2182739466, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6366, "incarnation": 112, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:reconnect", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-16T20:58:11.901154+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5477, "mds_1": 5709, "mds_2": 5535, "mds_3": 5570, "mds_4": 6366 } } } ], "standbys": [] } 2023-10-16T20:58:15.440 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:58:15.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.603+0000 7f84ef432640 1 Processor -- start 2023-10-16T20:58:15.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.603+0000 7f84ef432640 1 -- start start 2023-10-16T20:58:15.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.603+0000 7f84ef432640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f84e8153bc0 0x7f84e815cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:15.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.603+0000 7f84ef432640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f84e815aac0 0x7f84e80c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:15.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.603+0000 7f84ef432640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e8159bd0 con 0x7f84e81537e0 2023-10-16T20:58:15.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.603+0000 7f84ef432640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e815db10 con 0x7f84e8153bc0 2023-10-16T20:58:15.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.603+0000 7f84ef432640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e8159650 con 0x7f84e815aac0 2023-10-16T20:58:15.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.603+0000 7f84ec9cd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f84e8153bc0 0x7f84e815cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:15.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.603+0000 7f84ec9cd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f84e8153bc0 0x7f84e815cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55104/0 (socket says 172.21.15.88:55104) 2023-10-16T20:58:15.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.603+0000 7f84ed1ce640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f84e81537e0 0x7f84e815a060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49480/0 (socket says 172.21.15.88:49480) 2023-10-16T20:58:15.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.603+0000 7f84ec9cd640 1 -- 172.21.15.88:0/1358860419 learned_addr learned my addr 172.21.15.88:0/1358860419 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:15.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.603+0000 7f84ed9cf640 1 --2- 172.21.15.88:0/1358860419 >> v2:172.21.15.191:3300/0 conn(0x7f84e815aac0 0x7f84e80c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:15.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.607+0000 7f84ec9cd640 1 -- 172.21.15.88:0/1358860419 >> v2:172.21.15.191:3300/0 conn(0x7f84e815aac0 msgr2=0x7f84e80c9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:15.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.607+0000 7f84ec9cd640 1 --2- 172.21.15.88:0/1358860419 >> v2:172.21.15.191:3300/0 conn(0x7f84e815aac0 0x7f84e80c9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:15.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.607+0000 7f84ec9cd640 1 -- 172.21.15.88:0/1358860419 >> v1:172.21.15.88:6789/0 conn(0x7f84e81537e0 legacy=0x7f84e815a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:15.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.607+0000 7f84ec9cd640 1 -- 172.21.15.88:0/1358860419 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e815d270 con 0x7f84e8153bc0 2023-10-16T20:58:15.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.611+0000 7f84ec9cd640 1 --2- 172.21.15.88:0/1358860419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e8153bc0 0x7f84e815cbc0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f84e80c6190 tx=0x7f84d801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ee15d089fc5afe server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:15.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.611+0000 7f84dffff640 1 -- 172.21.15.88:0/1358860419 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d80a89f0 con 0x7f84e8153bc0 2023-10-16T20:58:15.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.611+0000 7f84dffff640 1 -- 172.21.15.88:0/1358860419 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d80a8b50 con 0x7f84e8153bc0 2023-10-16T20:58:15.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.611+0000 7f84dffff640 1 -- 172.21.15.88:0/1358860419 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d80b0440 con 0x7f84e8153bc0 2023-10-16T20:58:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.611+0000 7f84ef432640 1 -- 172.21.15.88:0/1358860419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e8153bc0 msgr2=0x7f84e815cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.611+0000 7f84ef432640 1 --2- 172.21.15.88:0/1358860419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e8153bc0 0x7f84e815cbc0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f84e80c6190 tx=0x7f84d801ecf0 comp rx=0 tx=0).stop 2023-10-16T20:58:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.611+0000 7f84ef432640 1 -- 172.21.15.88:0/1358860419 shutdown_connections 2023-10-16T20:58:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.611+0000 7f84ef432640 1 --2- 172.21.15.88:0/1358860419 >> v2:172.21.15.191:3300/0 conn(0x7f84e815aac0 0x7f84e80c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.611+0000 7f84ef432640 1 --2- 172.21.15.88:0/1358860419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e8153bc0 0x7f84e815cbc0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.611+0000 7f84ef432640 1 -- 172.21.15.88:0/1358860419 >> 172.21.15.88:0/1358860419 conn(0x7f84e80715c0 msgr2=0x7f84e80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.611+0000 7f84ef432640 1 -- 172.21.15.88:0/1358860419 shutdown_connections 2023-10-16T20:58:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.611+0000 7f84ef432640 1 -- 172.21.15.88:0/1358860419 wait complete. 2023-10-16T20:58:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ef432640 1 Processor -- start 2023-10-16T20:58:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ef432640 1 -- start start 2023-10-16T20:58:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ef432640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84e81537e0 0x7f84e80c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ef432640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e8153bc0 0x7f84e814ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ef432640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84e815aac0 0x7f84e814b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ef432640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e81f6430 con 0x7f84e8153bc0 2023-10-16T20:58:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ef432640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e8150b50 con 0x7f84e81537e0 2023-10-16T20:58:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ef432640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e81f66d0 con 0x7f84e815aac0 2023-10-16T20:58:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ed1ce640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84e81537e0 0x7f84e80c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ed1ce640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84e81537e0 0x7f84e80c5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:58084/0 (socket says 172.21.15.88:58084) 2023-10-16T20:58:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ed1ce640 1 -- 172.21.15.88:0/621333163 learned_addr learned my addr 172.21.15.88:0/621333163 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ed9cf640 1 --2- 172.21.15.88:0/621333163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84e815aac0 0x7f84e814b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ec9cd640 1 --2- 172.21.15.88:0/621333163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e8153bc0 0x7f84e814ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ed9cf640 1 -- 172.21.15.88:0/621333163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84e81537e0 msgr2=0x7f84e80c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ed9cf640 1 --2- 172.21.15.88:0/621333163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84e81537e0 0x7f84e80c5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ed9cf640 1 -- 172.21.15.88:0/621333163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e8153bc0 msgr2=0x7f84e814ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:15.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ed9cf640 1 --2- 172.21.15.88:0/621333163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e8153bc0 0x7f84e814ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:15.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ed9cf640 1 -- 172.21.15.88:0/621333163 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d80094d0 con 0x7f84e815aac0 2023-10-16T20:58:15.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ed9cf640 1 --2- 172.21.15.88:0/621333163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84e815aac0 0x7f84e814b2d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f84e000eb70 tx=0x7f84e000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:15.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ddffb640 1 -- 172.21.15.88:0/621333163 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e000cd20 con 0x7f84e815aac0 2023-10-16T20:58:15.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ddffb640 1 -- 172.21.15.88:0/621333163 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e000f040 con 0x7f84e815aac0 2023-10-16T20:58:15.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ef432640 1 -- 172.21.15.88:0/621333163 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e81504d0 con 0x7f84e815aac0 2023-10-16T20:58:15.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.615+0000 7f84ddffb640 1 -- 172.21.15.88:0/621333163 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e00146c0 con 0x7f84e815aac0 2023-10-16T20:58:15.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.619+0000 7f84ef432640 1 -- 172.21.15.88:0/621333163 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e8150870 con 0x7f84e815aac0 2023-10-16T20:58:15.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.619+0000 7f84ef432640 1 -- 172.21.15.88:0/621333163 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e8074700 con 0x7f84e815aac0 2023-10-16T20:58:15.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.619+0000 7f84ddffb640 1 -- 172.21.15.88:0/621333163 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f84e0014820 con 0x7f84e815aac0 2023-10-16T20:58:15.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.619+0000 7f84ddffb640 1 --2- 172.21.15.88:0/621333163 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f84cc0664f0 0x7f84cc0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:15.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.619+0000 7f84ddffb640 1 -- 172.21.15.88:0/621333163 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f84e0085260 con 0x7f84e815aac0 2023-10-16T20:58:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.619+0000 7f84ed1ce640 1 --2- 172.21.15.88:0/621333163 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f84cc0664f0 0x7f84cc0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.623+0000 7f84ed1ce640 1 --2- 172.21.15.88:0/621333163 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f84cc0664f0 0x7f84cc0689b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f84e4000f70 tx=0x7f84e4007f10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:15.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.627+0000 7f84ddffb640 1 -- 172.21.15.88:0/621333163 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f84e0051fa0 con 0x7f84e815aac0 2023-10-16T20:58:15.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.887+0000 7f84ef432640 1 -- 172.21.15.88:0/621333163 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84e81f6810 con 0x7f84e815aac0 2023-10-16T20:58:15.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.891+0000 7f84ddffb640 1 -- 172.21.15.88:0/621333163 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5790 (secure 0 0 0) 0x7f84e0018020 con 0x7f84e815aac0 2023-10-16T20:58:15.890 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:15.890 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:58:11.901154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"c","rank":4,"incarnation":112,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:58:15.890 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 114 2023-10-16T20:58:15.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.891+0000 7f84ef432640 1 -- 172.21.15.88:0/621333163 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f84cc0664f0 msgr2=0x7f84cc0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:15.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.891+0000 7f84ef432640 1 --2- 172.21.15.88:0/621333163 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f84cc0664f0 0x7f84cc0689b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f84e4000f70 tx=0x7f84e4007f10 comp rx=0 tx=0).stop 2023-10-16T20:58:15.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.891+0000 7f84ef432640 1 -- 172.21.15.88:0/621333163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84e815aac0 msgr2=0x7f84e814b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:15.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.891+0000 7f84ef432640 1 --2- 172.21.15.88:0/621333163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84e815aac0 0x7f84e814b2d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f84e000eb70 tx=0x7f84e000c490 comp rx=0 tx=0).stop 2023-10-16T20:58:15.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.891+0000 7f84ef432640 1 -- 172.21.15.88:0/621333163 shutdown_connections 2023-10-16T20:58:15.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.891+0000 7f84ef432640 1 --2- 172.21.15.88:0/621333163 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f84cc0664f0 0x7f84cc0689b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:15.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.891+0000 7f84ef432640 1 --2- 172.21.15.88:0/621333163 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f84e815aac0 0x7f84e814b2d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:15.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.891+0000 7f84ef432640 1 --2- 172.21.15.88:0/621333163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e8153bc0 0x7f84e814ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:15.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.891+0000 7f84ef432640 1 --2- 172.21.15.88:0/621333163 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84e81537e0 0x7f84e80c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:15.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.891+0000 7f84ef432640 1 -- 172.21.15.88:0/621333163 >> 172.21.15.88:0/621333163 conn(0x7f84e80715c0 msgr2=0x7f84e8157ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:15.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.891+0000 7f84ef432640 1 -- 172.21.15.88:0/621333163 shutdown_connections 2023-10-16T20:58:15.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:15.891+0000 7f84ef432640 1 -- 172.21.15.88:0/621333163 wait complete. 2023-10-16T20:58:15.917 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:58:17.918 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:58:18.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.099+0000 7fa9f0b9d640 1 Processor -- start 2023-10-16T20:58:18.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.099+0000 7fa9f0b9d640 1 -- start start 2023-10-16T20:58:18.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.099+0000 7fa9f0b9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa9ec0c9040 0x7fa9ec072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:18.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.099+0000 7fa9f0b9d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ec069e90 con 0x7fa9ec153770 2023-10-16T20:58:18.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.099+0000 7fa9f0b9d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ec06aef0 con 0x7fa9ec0c6bf0 2023-10-16T20:58:18.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.099+0000 7fa9f0b9d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9ec06a0c0 con 0x7fa9ec0c9040 2023-10-16T20:58:18.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.099+0000 7fa9ead9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa9ec0c9040 0x7fa9ec072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:18.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.099+0000 7fa9ea59c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa9ec153770 0x7fa9ec0c64e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49486/0 (socket says 172.21.15.88:49486) 2023-10-16T20:58:18.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.099+0000 7fa9ea59c640 1 -- 172.21.15.88:0/2958967376 learned_addr learned my addr 172.21.15.88:0/2958967376 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:18.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.099+0000 7fa9ead9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa9ec0c9040 0x7fa9ec072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55120/0 (socket says 172.21.15.88:55120) 2023-10-16T20:58:18.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.099+0000 7fa9ead9d640 1 -- 172.21.15.88:0/2958967376 >> v1:172.21.15.191:6789/0 conn(0x7fa9ec0c6bf0 legacy=0x7fa9ec071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:18.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.099+0000 7fa9ead9d640 1 -- 172.21.15.88:0/2958967376 >> v1:172.21.15.88:6789/0 conn(0x7fa9ec153770 legacy=0x7fa9ec0c64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:18.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.099+0000 7fa9ead9d640 1 -- 172.21.15.88:0/2958967376 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ec06a360 con 0x7fa9ec0c9040 2023-10-16T20:58:18.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.099+0000 7fa9e959a640 1 -- 172.21.15.88:0/2958967376 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862600920 0 0) 0x7fa9ec069e90 con 0x7fa9ec153770 2023-10-16T20:58:18.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.103+0000 7fa9ead9d640 1 --2- 172.21.15.88:0/2958967376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9ec0c9040 0x7fa9ec072230 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4009d90 tx=0x7fa9e401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77442a7d5b74435c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:18.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.103+0000 7fa9e959a640 1 -- 172.21.15.88:0/2958967376 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e40a5920 con 0x7fa9ec0c9040 2023-10-16T20:58:18.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.103+0000 7fa9e959a640 1 -- 172.21.15.88:0/2958967376 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e40a5a80 con 0x7fa9ec0c9040 2023-10-16T20:58:18.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.103+0000 7fa9e959a640 1 -- 172.21.15.88:0/2958967376 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e40a5e10 con 0x7fa9ec0c9040 2023-10-16T20:58:18.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.103+0000 7fa9f0b9d640 1 -- 172.21.15.88:0/2958967376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9ec0c9040 msgr2=0x7fa9ec072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:18.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.103+0000 7fa9f0b9d640 1 --2- 172.21.15.88:0/2958967376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9ec0c9040 0x7fa9ec072230 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4009d90 tx=0x7fa9e401c950 comp rx=0 tx=0).stop 2023-10-16T20:58:18.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.103+0000 7fa9f0b9d640 1 -- 172.21.15.88:0/2958967376 shutdown_connections 2023-10-16T20:58:18.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.103+0000 7fa9f0b9d640 1 --2- 172.21.15.88:0/2958967376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9ec0c9040 0x7fa9ec072230 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:18.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.103+0000 7fa9f0b9d640 1 -- 172.21.15.88:0/2958967376 >> 172.21.15.88:0/2958967376 conn(0x7fa9ec0c84e0 msgr2=0x7fa9ec0c88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:18.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.103+0000 7fa9f0b9d640 1 -- 172.21.15.88:0/2958967376 shutdown_connections 2023-10-16T20:58:18.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.103+0000 7fa9f0b9d640 1 -- 172.21.15.88:0/2958967376 wait complete. 2023-10-16T20:58:18.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9f0b9d640 1 Processor -- start 2023-10-16T20:58:18.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9f0b9d640 1 -- start start 2023-10-16T20:58:18.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9f0b9d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9ec0c6bf0 0x7fa9ec14eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:18.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9f0b9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9ec0c9040 0x7fa9ec153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:18.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9f0b9d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9ec153770 0x7fa9ec14a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:18.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9f0b9d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec159a60 con 0x7fa9ec0c9040 2023-10-16T20:58:18.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9f0b9d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec152430 con 0x7fa9ec0c6bf0 2023-10-16T20:58:18.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9f0b9d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec1510b0 con 0x7fa9ec153770 2023-10-16T20:58:18.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9e9d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9ec0c9040 0x7fa9ec153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:18.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9e9d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9ec0c9040 0x7fa9ec153d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55132/0 (socket says 172.21.15.88:55132) 2023-10-16T20:58:18.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9e9d9b640 1 -- 172.21.15.88:0/2567130919 learned_addr learned my addr 172.21.15.88:0/2567130919 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:18.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9ead9d640 1 --2- 172.21.15.88:0/2567130919 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9ec153770 0x7fa9ec14a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:18.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9ea59c640 1 --2- 172.21.15.88:0/2567130919 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9ec0c6bf0 0x7fa9ec14eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:18.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9e9d9b640 1 -- 172.21.15.88:0/2567130919 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9ec153770 msgr2=0x7fa9ec14a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:18.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9e9d9b640 1 --2- 172.21.15.88:0/2567130919 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9ec153770 0x7fa9ec14a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:18.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9e9d9b640 1 -- 172.21.15.88:0/2567130919 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9ec0c6bf0 msgr2=0x7fa9ec14eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:18.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9e9d9b640 1 --2- 172.21.15.88:0/2567130919 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9ec0c6bf0 0x7fa9ec14eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:18.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9e9d9b640 1 -- 172.21.15.88:0/2567130919 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e4009a70 con 0x7fa9ec0c9040 2023-10-16T20:58:18.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9e9d9b640 1 --2- 172.21.15.88:0/2567130919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9ec0c9040 0x7fa9ec153d90 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000c500 tx=0x7fa9e000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:18.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9daffd640 1 -- 172.21.15.88:0/2567130919 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0005220 con 0x7fa9ec0c9040 2023-10-16T20:58:18.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9daffd640 1 -- 172.21.15.88:0/2567130919 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e0005380 con 0x7fa9ec0c9040 2023-10-16T20:58:18.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9daffd640 1 -- 172.21.15.88:0/2567130919 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0007270 con 0x7fa9ec0c9040 2023-10-16T20:58:18.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9f0b9d640 1 -- 172.21.15.88:0/2567130919 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9ec151390 con 0x7fa9ec0c9040 2023-10-16T20:58:18.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9f0b9d640 1 -- 172.21.15.88:0/2567130919 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9ec151950 con 0x7fa9ec0c9040 2023-10-16T20:58:18.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.107+0000 7fa9f0b9d640 1 -- 172.21.15.88:0/2567130919 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9ec074700 con 0x7fa9ec0c9040 2023-10-16T20:58:18.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.111+0000 7fa9daffd640 1 -- 172.21.15.88:0/2567130919 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa9e00073d0 con 0x7fa9ec0c9040 2023-10-16T20:58:18.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.111+0000 7fa9daffd640 1 --2- 172.21.15.88:0/2567130919 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa9cc0664d0 0x7fa9cc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:18.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.111+0000 7fa9daffd640 1 -- 172.21.15.88:0/2567130919 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7fa9e0083da0 con 0x7fa9ec0c9040 2023-10-16T20:58:18.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.111+0000 7fa9ea59c640 1 --2- 172.21.15.88:0/2567130919 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa9cc0664d0 0x7fa9cc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:18.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.111+0000 7fa9ea59c640 1 --2- 172.21.15.88:0/2567130919 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa9cc0664d0 0x7fa9cc068990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc000f70 tx=0x7fa9dc007240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:18.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.115+0000 7fa9daffd640 1 -- 172.21.15.88:0/2567130919 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa9e0050b20 con 0x7fa9ec0c9040 2023-10-16T20:58:18.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.295+0000 7fa9f0b9d640 1 -- 172.21.15.88:0/2567130919 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9ec152fa0 con 0x7fa9ec0c9040 2023-10-16T20:58:18.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.295+0000 7fa9daffd640 1 -- 172.21.15.88:0/2567130919 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5787 (secure 0 0 0) 0x7fa9e0054140 con 0x7fa9ec0c9040 2023-10-16T20:58:18.294 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:18.295 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:58:17.694557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"c","rank":4,"incarnation":112,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:58:18.295 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 115 2023-10-16T20:58:18.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.295+0000 7fa9f0b9d640 1 -- 172.21.15.88:0/2567130919 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa9cc0664d0 msgr2=0x7fa9cc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:18.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.295+0000 7fa9f0b9d640 1 --2- 172.21.15.88:0/2567130919 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa9cc0664d0 0x7fa9cc068990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc000f70 tx=0x7fa9dc007240 comp rx=0 tx=0).stop 2023-10-16T20:58:18.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.295+0000 7fa9f0b9d640 1 -- 172.21.15.88:0/2567130919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9ec0c9040 msgr2=0x7fa9ec153d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:18.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.295+0000 7fa9f0b9d640 1 --2- 172.21.15.88:0/2567130919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9ec0c9040 0x7fa9ec153d90 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000c500 tx=0x7fa9e000c9c0 comp rx=0 tx=0).stop 2023-10-16T20:58:18.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.295+0000 7fa9f0b9d640 1 -- 172.21.15.88:0/2567130919 shutdown_connections 2023-10-16T20:58:18.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.295+0000 7fa9f0b9d640 1 --2- 172.21.15.88:0/2567130919 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa9cc0664d0 0x7fa9cc068990 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:18.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.295+0000 7fa9f0b9d640 1 --2- 172.21.15.88:0/2567130919 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa9ec153770 0x7fa9ec14a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:18.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.295+0000 7fa9f0b9d640 1 --2- 172.21.15.88:0/2567130919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa9ec0c9040 0x7fa9ec153d90 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:18.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.295+0000 7fa9f0b9d640 1 --2- 172.21.15.88:0/2567130919 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9ec0c6bf0 0x7fa9ec14eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:18.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.295+0000 7fa9f0b9d640 1 -- 172.21.15.88:0/2567130919 >> 172.21.15.88:0/2567130919 conn(0x7fa9ec0c84e0 msgr2=0x7fa9ec0c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:18.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.295+0000 7fa9f0b9d640 1 -- 172.21.15.88:0/2567130919 shutdown_connections 2023-10-16T20:58:18.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:18.295+0000 7fa9f0b9d640 1 -- 172.21.15.88:0/2567130919 wait complete. 2023-10-16T20:58:18.314 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:58:20.315 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:58:20.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9fd1d0640 1 Processor -- start 2023-10-16T20:58:20.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9fd1d0640 1 -- start start 2023-10-16T20:58:20.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9fd1d0640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb9f8150280 0x7fb9f80c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:20.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9fd1d0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb9f8150660 0x7fb9f80c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:20.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9fd1d0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9f80c6bc0 0x7fb9f806d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:20.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9fd1d0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9f80c5c60 con 0x7fb9f8150660 2023-10-16T20:58:20.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9fd1d0640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9f80c5370 con 0x7fb9f8150280 2023-10-16T20:58:20.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9fd1d0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f8156d10 con 0x7fb9f80c6bc0 2023-10-16T20:58:20.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9f659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb9f8150660 0x7fb9f80c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:20.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9f659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb9f8150660 0x7fb9f80c3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39892/0 (socket says 172.21.15.88:39892) 2023-10-16T20:58:20.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9f659c640 1 -- 172.21.15.88:0/2014409424 learned_addr learned my addr 172.21.15.88:0/2014409424 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:20.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9f6d9d640 1 --2- 172.21.15.88:0/2014409424 >> v2:172.21.15.191:3300/0 conn(0x7fb9f8150280 0x7fb9f80c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:20.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9f759e640 1 --2- 172.21.15.88:0/2014409424 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9f80c6bc0 0x7fb9f806d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:20.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9f659c640 1 -- 172.21.15.88:0/2014409424 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9f80c6bc0 msgr2=0x7fb9f806d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:20.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9f659c640 1 --2- 172.21.15.88:0/2014409424 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9f80c6bc0 0x7fb9f806d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:20.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9f659c640 1 -- 172.21.15.88:0/2014409424 >> v2:172.21.15.191:3300/0 conn(0x7fb9f8150280 msgr2=0x7fb9f80c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:20.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9f659c640 1 --2- 172.21.15.88:0/2014409424 >> v2:172.21.15.191:3300/0 conn(0x7fb9f8150280 0x7fb9f80c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:20.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.487+0000 7fb9f659c640 1 -- 172.21.15.88:0/2014409424 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f80c5610 con 0x7fb9f8150660 2023-10-16T20:58:20.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.491+0000 7fb9f659c640 1 --2- 172.21.15.88:0/2014409424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9f8150660 0x7fb9f80c3110 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb9f806f060 tx=0x7fb9ec01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff09c84bba56a1b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:20.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.491+0000 7fb9f5d9b640 1 -- 172.21.15.88:0/2014409424 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec0ac600 con 0x7fb9f8150660 2023-10-16T20:58:20.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.491+0000 7fb9f5d9b640 1 -- 172.21.15.88:0/2014409424 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9ec0a8050 con 0x7fb9f8150660 2023-10-16T20:58:20.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.491+0000 7fb9f5d9b640 1 -- 172.21.15.88:0/2014409424 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec0ac980 con 0x7fb9f8150660 2023-10-16T20:58:20.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.491+0000 7fb9fd1d0640 1 -- 172.21.15.88:0/2014409424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9f8150660 msgr2=0x7fb9f80c3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:20.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.491+0000 7fb9fd1d0640 1 --2- 172.21.15.88:0/2014409424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9f8150660 0x7fb9f80c3110 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb9f806f060 tx=0x7fb9ec01c4d0 comp rx=0 tx=0).stop 2023-10-16T20:58:20.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.491+0000 7fb9fd1d0640 1 -- 172.21.15.88:0/2014409424 shutdown_connections 2023-10-16T20:58:20.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.491+0000 7fb9fd1d0640 1 --2- 172.21.15.88:0/2014409424 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9f80c6bc0 0x7fb9f806d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:20.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.491+0000 7fb9fd1d0640 1 --2- 172.21.15.88:0/2014409424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9f8150660 0x7fb9f80c3110 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:20.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.491+0000 7fb9fd1d0640 1 --2- 172.21.15.88:0/2014409424 >> v2:172.21.15.191:3300/0 conn(0x7fb9f8150280 0x7fb9f80c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:20.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.491+0000 7fb9fd1d0640 1 -- 172.21.15.88:0/2014409424 >> 172.21.15.88:0/2014409424 conn(0x7fb9f80cad50 msgr2=0x7fb9f80cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:20.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.491+0000 7fb9fd1d0640 1 -- 172.21.15.88:0/2014409424 shutdown_connections 2023-10-16T20:58:20.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.491+0000 7fb9fd1d0640 1 -- 172.21.15.88:0/2014409424 wait complete. 2023-10-16T20:58:20.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9fd1d0640 1 Processor -- start 2023-10-16T20:58:20.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9fd1d0640 1 -- start start 2023-10-16T20:58:20.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9fd1d0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9f80c6bc0 0x7fb9f81f5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:20.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9fd1d0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9f8150280 0x7fb9f814ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:20.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9fd1d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9f8150660 0x7fb9f8158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9fd1d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f8151570 con 0x7fb9f8150660 2023-10-16T20:58:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9fd1d0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f8150a80 con 0x7fb9f8150280 2023-10-16T20:58:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9fd1d0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f81f59f0 con 0x7fb9f80c6bc0 2023-10-16T20:58:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9f759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9f8150660 0x7fb9f8158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9f759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9f8150660 0x7fb9f8158cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39906/0 (socket says 172.21.15.88:39906) 2023-10-16T20:58:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9f759e640 1 -- 172.21.15.88:0/3532429410 learned_addr learned my addr 172.21.15.88:0/3532429410 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9f659c640 1 --2- 172.21.15.88:0/3532429410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9f8150280 0x7fb9f814ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9f6d9d640 1 --2- 172.21.15.88:0/3532429410 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9f80c6bc0 0x7fb9f81f5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9f759e640 1 -- 172.21.15.88:0/3532429410 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9f80c6bc0 msgr2=0x7fb9f81f5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9f759e640 1 --2- 172.21.15.88:0/3532429410 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9f80c6bc0 0x7fb9f81f5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:20.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9f759e640 1 -- 172.21.15.88:0/3532429410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9f8150280 msgr2=0x7fb9f814ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:20.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9f759e640 1 --2- 172.21.15.88:0/3532429410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9f8150280 0x7fb9f814ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:20.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9f759e640 1 -- 172.21.15.88:0/3532429410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ec0094d0 con 0x7fb9f8150660 2023-10-16T20:58:20.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9f759e640 1 --2- 172.21.15.88:0/3532429410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9f8150660 0x7fb9f8158cd0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0007e70 tx=0x7fb9f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:20.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9df7fe640 1 -- 172.21.15.88:0/3532429410 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0014620 con 0x7fb9f8150660 2023-10-16T20:58:20.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9df7fe640 1 -- 172.21.15.88:0/3532429410 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f00147a0 con 0x7fb9f8150660 2023-10-16T20:58:20.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9fd1d0640 1 -- 172.21.15.88:0/3532429410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f81f63c0 con 0x7fb9f8150660 2023-10-16T20:58:20.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9df7fe640 1 -- 172.21.15.88:0/3532429410 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0014b50 con 0x7fb9f8150660 2023-10-16T20:58:20.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9fd1d0640 1 -- 172.21.15.88:0/3532429410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f8150fb0 con 0x7fb9f8150660 2023-10-16T20:58:20.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.495+0000 7fb9fd1d0640 1 -- 172.21.15.88:0/3532429410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f8074700 con 0x7fb9f8150660 2023-10-16T20:58:20.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.499+0000 7fb9df7fe640 1 -- 172.21.15.88:0/3532429410 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb9f0014cb0 con 0x7fb9f8150660 2023-10-16T20:58:20.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.499+0000 7fb9df7fe640 1 --2- 172.21.15.88:0/3532429410 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb9e40664d0 0x7fb9e4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:20.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.499+0000 7fb9df7fe640 1 -- 172.21.15.88:0/3532429410 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7fb9f0084f60 con 0x7fb9f8150660 2023-10-16T20:58:20.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.499+0000 7fb9f6d9d640 1 --2- 172.21.15.88:0/3532429410 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb9e40664d0 0x7fb9e4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:20.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.499+0000 7fb9f6d9d640 1 --2- 172.21.15.88:0/3532429410 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb9e40664d0 0x7fb9e4068990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8009ae0 tx=0x7fb9e8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:20.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.503+0000 7fb9df7fe640 1 -- 172.21.15.88:0/3532429410 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb9f0051c40 con 0x7fb9f8150660 2023-10-16T20:58:20.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.683+0000 7fb9fd1d0640 1 -- 172.21.15.88:0/3532429410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9f81f5b90 con 0x7fb9f8150660 2023-10-16T20:58:20.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.683+0000 7fb9df7fe640 1 -- 172.21.15.88:0/3532429410 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5787 (secure 0 0 0) 0x7fb9f0055260 con 0x7fb9f8150660 2023-10-16T20:58:20.684 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:20.684 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:58:17.694557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"c","rank":4,"incarnation":112,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:58:20.685 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 115 2023-10-16T20:58:20.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.687+0000 7fb9fd1d0640 1 -- 172.21.15.88:0/3532429410 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb9e40664d0 msgr2=0x7fb9e4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:20.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.687+0000 7fb9fd1d0640 1 --2- 172.21.15.88:0/3532429410 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb9e40664d0 0x7fb9e4068990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8009ae0 tx=0x7fb9e8009190 comp rx=0 tx=0).stop 2023-10-16T20:58:20.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.687+0000 7fb9fd1d0640 1 -- 172.21.15.88:0/3532429410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9f8150660 msgr2=0x7fb9f8158cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:20.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.687+0000 7fb9fd1d0640 1 --2- 172.21.15.88:0/3532429410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9f8150660 0x7fb9f8158cd0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0007e70 tx=0x7fb9f000e4b0 comp rx=0 tx=0).stop 2023-10-16T20:58:20.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.687+0000 7fb9fd1d0640 1 -- 172.21.15.88:0/3532429410 shutdown_connections 2023-10-16T20:58:20.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.687+0000 7fb9fd1d0640 1 --2- 172.21.15.88:0/3532429410 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb9e40664d0 0x7fb9e4068990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:20.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.687+0000 7fb9fd1d0640 1 --2- 172.21.15.88:0/3532429410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9f8150660 0x7fb9f8158cd0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:20.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.687+0000 7fb9fd1d0640 1 --2- 172.21.15.88:0/3532429410 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9f8150280 0x7fb9f814ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:20.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.687+0000 7fb9fd1d0640 1 --2- 172.21.15.88:0/3532429410 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb9f80c6bc0 0x7fb9f81f5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:20.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.687+0000 7fb9fd1d0640 1 -- 172.21.15.88:0/3532429410 >> 172.21.15.88:0/3532429410 conn(0x7fb9f80cad50 msgr2=0x7fb9f8155510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:20.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.687+0000 7fb9fd1d0640 1 -- 172.21.15.88:0/3532429410 shutdown_connections 2023-10-16T20:58:20.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:20.687+0000 7fb9fd1d0640 1 -- 172.21.15.88:0/3532429410 wait complete. 2023-10-16T20:58:20.704 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:58:22.706 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:58:22.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.883+0000 7f08a9846640 1 Processor -- start 2023-10-16T20:58:22.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.883+0000 7f08a9846640 1 -- start start 2023-10-16T20:58:22.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a9846640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08a41534f0 0x7f08a40c3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:22.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a9846640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f08a40c9a50 0x7f08a4071b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:22.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a9846640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f08a4072070 0x7f08a41592e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:22.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a9846640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a415d270 con 0x7f08a40c9a50 2023-10-16T20:58:22.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a9846640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a4159ec0 con 0x7f08a4072070 2023-10-16T20:58:22.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a9846640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a415a160 con 0x7f08a41534f0 2023-10-16T20:58:22.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a27fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f08a40c9a50 0x7f08a4071b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:22.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a27fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f08a40c9a50 0x7f08a4071b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39910/0 (socket says 172.21.15.88:39910) 2023-10-16T20:58:22.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a27fc640 1 -- 172.21.15.88:0/2003013570 learned_addr learned my addr 172.21.15.88:0/2003013570 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:22.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a37fe640 1 --2- 172.21.15.88:0/2003013570 >> v2:172.21.15.191:3300/0 conn(0x7f08a4072070 0x7f08a41592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:22.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a2ffd640 1 --2- 172.21.15.88:0/2003013570 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08a41534f0 0x7f08a40c3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:22.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a27fc640 1 -- 172.21.15.88:0/2003013570 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08a41534f0 msgr2=0x7f08a40c3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:22.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a27fc640 1 --2- 172.21.15.88:0/2003013570 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08a41534f0 0x7f08a40c3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:22.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a27fc640 1 -- 172.21.15.88:0/2003013570 >> v2:172.21.15.191:3300/0 conn(0x7f08a4072070 msgr2=0x7f08a41592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:22.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a27fc640 1 --2- 172.21.15.88:0/2003013570 >> v2:172.21.15.191:3300/0 conn(0x7f08a4072070 0x7f08a41592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:22.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a27fc640 1 -- 172.21.15.88:0/2003013570 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a415c8d0 con 0x7f08a40c9a50 2023-10-16T20:58:22.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a37fe640 1 --2- 172.21.15.88:0/2003013570 >> v2:172.21.15.191:3300/0 conn(0x7f08a4072070 0x7f08a41592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:58:22.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.887+0000 7f08a27fc640 1 --2- 172.21.15.88:0/2003013570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08a40c9a50 0x7f08a4071b30 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f08a415a450 tx=0x7f089801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc4d414096df621 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:22.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.891+0000 7f08a1ffb640 1 -- 172.21.15.88:0/2003013570 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08980a89b0 con 0x7f08a40c9a50 2023-10-16T20:58:22.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.891+0000 7f08a1ffb640 1 -- 172.21.15.88:0/2003013570 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08980a8b10 con 0x7f08a40c9a50 2023-10-16T20:58:22.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.891+0000 7f08a1ffb640 1 -- 172.21.15.88:0/2003013570 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08980a8ea0 con 0x7f08a40c9a50 2023-10-16T20:58:22.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.891+0000 7f08a9846640 1 -- 172.21.15.88:0/2003013570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08a40c9a50 msgr2=0x7f08a4071b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:22.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.891+0000 7f08a9846640 1 --2- 172.21.15.88:0/2003013570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08a40c9a50 0x7f08a4071b30 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f08a415a450 tx=0x7f089801ecf0 comp rx=0 tx=0).stop 2023-10-16T20:58:22.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.891+0000 7f08a9846640 1 -- 172.21.15.88:0/2003013570 shutdown_connections 2023-10-16T20:58:22.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.891+0000 7f08a9846640 1 --2- 172.21.15.88:0/2003013570 >> v2:172.21.15.191:3300/0 conn(0x7f08a4072070 0x7f08a41592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:22.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.891+0000 7f08a9846640 1 --2- 172.21.15.88:0/2003013570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08a40c9a50 0x7f08a4071b30 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:22.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.891+0000 7f08a9846640 1 --2- 172.21.15.88:0/2003013570 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08a41534f0 0x7f08a40c3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:22.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.891+0000 7f08a9846640 1 -- 172.21.15.88:0/2003013570 >> 172.21.15.88:0/2003013570 conn(0x7f08a40cf260 msgr2=0x7f08a40cf670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:22.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.891+0000 7f08a9846640 1 -- 172.21.15.88:0/2003013570 shutdown_connections 2023-10-16T20:58:22.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.891+0000 7f08a9846640 1 -- 172.21.15.88:0/2003013570 wait complete. 2023-10-16T20:58:22.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.891+0000 7f08a9846640 1 Processor -- start 2023-10-16T20:58:22.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.891+0000 7f08a9846640 1 -- start start 2023-10-16T20:58:22.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a9846640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08a4072070 0x7f08a40c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:22.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a9846640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08a40c9a50 0x7f08a41489e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:22.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a9846640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08a41534f0 0x7f08a4148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:22.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a9846640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a41e2ba0 con 0x7f08a40c9a50 2023-10-16T20:58:22.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a9846640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a40c92d0 con 0x7f08a41534f0 2023-10-16T20:58:22.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a9846640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a41e3ac0 con 0x7f08a4072070 2023-10-16T20:58:22.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a27fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08a40c9a50 0x7f08a41489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:22.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a27fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08a40c9a50 0x7f08a41489e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39918/0 (socket says 172.21.15.88:39918) 2023-10-16T20:58:22.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a27fc640 1 -- 172.21.15.88:0/783446105 learned_addr learned my addr 172.21.15.88:0/783446105 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:22.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a37fe640 1 --2- 172.21.15.88:0/783446105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08a41534f0 0x7f08a4148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:22.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a2ffd640 1 --2- 172.21.15.88:0/783446105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08a4072070 0x7f08a40c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:22.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a27fc640 1 -- 172.21.15.88:0/783446105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08a4072070 msgr2=0x7f08a40c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:22.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a27fc640 1 --2- 172.21.15.88:0/783446105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08a4072070 0x7f08a40c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:22.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a27fc640 1 -- 172.21.15.88:0/783446105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08a41534f0 msgr2=0x7f08a4148f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:22.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a27fc640 1 --2- 172.21.15.88:0/783446105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08a41534f0 0x7f08a4148f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:22.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a27fc640 1 -- 172.21.15.88:0/783446105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c009a70 con 0x7f08a40c9a50 2023-10-16T20:58:22.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a2ffd640 1 --2- 172.21.15.88:0/783446105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08a4072070 0x7f08a40c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:58:22.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a27fc640 1 --2- 172.21.15.88:0/783446105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08a40c9a50 0x7f08a41489e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f089801d3a0 tx=0x7f089801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:22.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f088bfff640 1 -- 172.21.15.88:0/783446105 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08980b0d40 con 0x7f08a40c9a50 2023-10-16T20:58:22.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f088bfff640 1 -- 172.21.15.88:0/783446105 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08980bb460 con 0x7f08a40c9a50 2023-10-16T20:58:22.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f088bfff640 1 -- 172.21.15.88:0/783446105 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08980bb810 con 0x7f08a40c9a50 2023-10-16T20:58:22.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a9846640 1 -- 172.21.15.88:0/783446105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08980094d0 con 0x7f08a40c9a50 2023-10-16T20:58:22.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a9846640 1 -- 172.21.15.88:0/783446105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a40c8ff0 con 0x7f08a40c9a50 2023-10-16T20:58:22.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f08a9846640 1 -- 172.21.15.88:0/783446105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a404e9c0 con 0x7f08a40c9a50 2023-10-16T20:58:22.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.895+0000 7f088bfff640 1 -- 172.21.15.88:0/783446105 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f08980b83f0 con 0x7f08a40c9a50 2023-10-16T20:58:22.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.899+0000 7f088bfff640 1 --2- 172.21.15.88:0/783446105 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08900664d0 0x7f0890068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:22.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.899+0000 7f088bfff640 1 -- 172.21.15.88:0/783446105 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f0898121de0 con 0x7f08a40c9a50 2023-10-16T20:58:22.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.899+0000 7f08a2ffd640 1 --2- 172.21.15.88:0/783446105 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08900664d0 0x7f0890068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:22.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.899+0000 7f08a2ffd640 1 --2- 172.21.15.88:0/783446105 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08900664d0 0x7f0890068990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0894009b10 tx=0x7f0894008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:22.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:22.903+0000 7f088bfff640 1 -- 172.21.15.88:0/783446105 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f08980eeaa0 con 0x7f08a40c9a50 2023-10-16T20:58:23.063 INFO:tasks.workunit.client.0.smithi088.stderr:Updating files: 2% (290/12063) Updating files: 3% (362/12063) Updating files: 3% (457/12063) Updating files: 4% (483/12063) Updating files: 5% (604/12063) Updating files: 5% (644/12063) Updating files: 6% (724/12063) Updating files: 7% (845/12063) Updating files: 7% (860/12063) Updating files: 8% (966/12063) Updating files: 8% (978/12063) Updating files: 9% (1086/12063) Updating files: 9% (1094/12063) Updating files: 9% (1135/12063) Updating files: 10% (1207/12063) Updating files: 10% (1223/12063) Updating files: 10% (1249/12063) Updating files: 11% (1327/12063) Updating files: 11% (1356/12063) Updating files: 11% (1439/12063) Updating files: 12% (1448/12063) Updating files: 12% (1541/12063) Updating files: 13% (1569/12063) Updating files: 14% (1689/12063) Updating files: 14% (1701/12063) Updating files: 14% (1713/12063) Updating files: 15% (1810/12063) Updating files: 15% (1820/12063) Updating files: 15% (1929/12063) Updating files: 16% (1931/12063) Updating files: 16% (1950/12063) Updating files: 17% (2051/12063) Updating files: 17% (2129/12063) Updating files: 18% (2172/12063) Updating files: 18% (2290/12063) Updating files: 19% (2292/12063) Updating files: 19% (2379/12063) Updating files: 20% (2413/12063) Updating files: 20% (2474/12063) Updating files: 20% (2497/12063) Updating files: 21% (2534/12063) Updating files: 21% (2564/12063) Updating files: 22% (2654/12063) Updating files: 22% (2712/12063) Updating files: 23% (2775/12063) Updating files: 23% (2870/12063) Updating files: 24% (2896/12063) Updating files: 25% (3016/12063) Updating files: 25% (3057/12063) Updating files: 25% (3130/12063) Updating files: 26% (3137/12063) Updating files: 27% (3258/12063) Updating files: 27% (3330/12063) Updating files: 28% (3378/12063) Updating files: 28% (3441/12063) Updating files: 28% (3498/12063) Updating files: 29% (3499/12063) Updating files: 29% (3500/12063) Updating files: 29% (3519/12063) Updating files: 29% (3572/12063) Updating files: 30% (3619/12063) Updating files: 30% (3646/12063) Updating files: 30% (3650/12063) Updating files: 30% (3707/12063) Updating files: 31% (3740/12063) Updating files: 31% (3790/12063) Updating files: 32% (3861/12063) Updating files: 32% (3878/12063) Updating files: 32% (3896/12063) Updating files: 33% (3981/12063) Updating files: 33% (4059/12063) Updating files: 34% (4102/12063) Updating files: 34% (4222/12063) Updating files: 35% (4223/12063) Updating files: 36% (4343/12063) Updating files: 36% (4365/12063) Updating files: 36% (4439/12063) Updating files: 37% (4464/12063) Updating files: 37% (4551/12063) Updating files: 38% (4584/12063) Updating files: 38% (4680/12063) Updating files: 39% (4705/12063) Updating files: 39% (4723/12063) Updating files: 40% (4826/12063) Updating files: 40% (4935/12063) Updating files: 41% (4946/12063) Updating files: 42% (5067/12063) Updating files: 42% (5108/12063) Updating files: 43% (5188/12063) Updating files: 43% (5240/12063) Updating files: 44% (5308/12063) Updating files: 45% (5429/12063) Updating files: 45% (5457/12063) Updating files: 46% (5549/12063) Updating files: 46% (5564/12063) Updating files: 47% (5670/12063) Updating files: 47% (5749/12063) Updating files: 48% (5791/12063) Updating files: 49% (5911/12063) Updating files: 49% (5992/12063) Updating files: 50% (6032/12063) Updating files: 50% (6086/12063) Updating files: 51% (6153/12063) Updating files: 52% (6273/12063) Updating files: 52% (6329/12063) Updating files: 53% (6394/12063) Updating files: 53% (6464/12063) Updating files: 54% (6515/12063) Updating files: 55% (6635/12063) Updating files: 55% (6662/12063) Updating files: 56% (6756/12063) Updating files: 56% (6772/12063) Updating files: 57% (6876/12063) Updating files: 58% (6997/12063) Updating files: 58% (7025/12063) Updating files: 59% (7118/12063) Updating files: 59% (7168/12063) Updating files: 60% (7238/12063) Updating files: 60% (7334/12063) Updating files: 61% (7359/12063) Updating files: 61% (7412/12063) Updating files: 61% (7413/12063) Updating files: 61% (7479/12063) Updating files: 62% (7480/12063) Updating files: 62% (7590/12063) Updating files: 63% (7600/12063) Updating files: 63% (7688/12063) Updating files: 64% (7721/12063) Updating files: 64% (7728/12063) Updating files: 64% (7831/12063) Updating files: 65% (7841/12063) Updating files: 65% (7896/12063) Updating files: 66% (7962/12063) Updating files: 66% (8051/12063) Updating files: 66% (8059/12063) Updating files: 67% (8083/12063) Updating files: 67% (8089/12063) Updating files: 67% (8178/12063) Updating files: 68% (8203/12063) Updating files: 68% (8255/12063) Updating files: 69% (8324/12063) Updating files: 69% (8372/12063) Updating files: 70% (8445/12063) Updating files: 70% (8462/12063) Updating files: 70% (8560/12063) Updating files: 71% (8565/12063) Updating files: 71% (8667/12063) Updating files: 71% (8684/12063) Updating files: 72% (8686/12063) Updating files: 72% (8707/12063) Updating files: 73% (8806/12063) Updating files: 73% (8816/12063) Updating files: 74% (8927/12063) Updating files: 74% (8951/12063) Updating files: 74% (9032/12063) Updating files: 75% (9048/12063) Updating files: 75% (9077/12063) Updating files: 76% (9168/12063) Updating files: 76% (9227/12063) Updating files: 77% (9289/12063) Updating files: 77% (9309/12063) Updating files: 77% (9352/12063) Updating files: 78% (9410/12063) Updating files: 78% (9411/12063) Updating files: 79% (9530/12063) Updating files: 79% (9540/12063) Updating files: 79% (9572/12063) Updating files: 79% (9606/12063) Updating files: 79% (9609/12063) Updating files: 79% (9620/12063) Updating files: 80% (9651/12063) Updating files: 80% (9687/12063) Updating files: 80% (9696/12063) Updating files: 81% (9772/12063) Updating files: 81% (9795/12063) Updating files: 81% (9847/12063) Updating files: 82% (9892/12063) Updating files: 82% (9977/12063) Updating files: 83% (10013/12063) Updating files: 83% (10032/12063) Updating files: 84% (10133/12063) Updating files: 84% (10165/12063) Updating files: 85% (10254/12063) Updating files: 86% (10375/12063) Updating files: 86% (10401/12063) Updating files: 86% (10470/12063) Updating files: 87% (10495/12063) Updating files: 87% (10538/12063) Updating files: 87% (10561/12063) Updating files: 88% (10616/12063) Updating files: 88% (10729/12063) Updating files: 89% (10737/12063) Updating files: 90% (10857/12063) Updating files: 90% (10863/12063) Updating files: 91% (10978/12063) Updating files: 91% (11071/12063) Updating files: 92% (11098/12063) Updating files: 92% (11198/12063) Updating files: 93% (11219/12063) Updating files: 94% (11340/12063) Updating files: 94% (11418/12063) Updating files: 95% (11460/12063) Updating files: 95% (11541/12063) Updating files: 96% (11581/12063) Updating files: 96% (11608/12063) Updating files: 97% (11702/12063) Updating files: 97% (11808/12063) Updating files: 98% (11822/12063) Updating files: 98% (11899/12063) Updating files: 98% (11904/12063) Updating files: 99% (11943/12063) Updating files: 99% (12045/12063) Updating files: 100% (12063/12063) Updating files: 100% (12063/12063), done. 2023-10-16T20:58:23.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:23.147+0000 7f08a9846640 1 -- 172.21.15.88:0/783446105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08a40c3d30 con 0x7f08a40c9a50 2023-10-16T20:58:23.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:23.147+0000 7f088bfff640 1 -- 172.21.15.88:0/783446105 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5787 (secure 0 0 0) 0x7f08980b5090 con 0x7f08a40c9a50 2023-10-16T20:58:23.150 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:23.150 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:58:17.694557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"c","rank":4,"incarnation":112,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:58:23.152 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 115 2023-10-16T20:58:23.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:23.151+0000 7f08a9846640 1 -- 172.21.15.88:0/783446105 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08900664d0 msgr2=0x7f0890068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:23.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:23.151+0000 7f08a9846640 1 --2- 172.21.15.88:0/783446105 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08900664d0 0x7f0890068990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0894009b10 tx=0x7f0894008040 comp rx=0 tx=0).stop 2023-10-16T20:58:23.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:23.151+0000 7f08a9846640 1 -- 172.21.15.88:0/783446105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08a40c9a50 msgr2=0x7f08a41489e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:23.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:23.151+0000 7f08a9846640 1 --2- 172.21.15.88:0/783446105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08a40c9a50 0x7f08a41489e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f089801d3a0 tx=0x7f089801ba90 comp rx=0 tx=0).stop 2023-10-16T20:58:23.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:23.151+0000 7f08a9846640 1 -- 172.21.15.88:0/783446105 shutdown_connections 2023-10-16T20:58:23.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:23.151+0000 7f08a9846640 1 --2- 172.21.15.88:0/783446105 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08900664d0 0x7f0890068990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:23.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:23.151+0000 7f08a9846640 1 --2- 172.21.15.88:0/783446105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08a41534f0 0x7f08a4148f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:23.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:23.151+0000 7f08a9846640 1 --2- 172.21.15.88:0/783446105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08a40c9a50 0x7f08a41489e0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:23.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:23.151+0000 7f08a9846640 1 --2- 172.21.15.88:0/783446105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08a4072070 0x7f08a40c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:23.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:23.151+0000 7f08a9846640 1 -- 172.21.15.88:0/783446105 >> 172.21.15.88:0/783446105 conn(0x7f08a40cf260 msgr2=0x7f08a41526c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:23.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:23.151+0000 7f08a9846640 1 -- 172.21.15.88:0/783446105 shutdown_connections 2023-10-16T20:58:23.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:23.151+0000 7f08a9846640 1 -- 172.21.15.88:0/783446105 wait complete. 2023-10-16T20:58:23.159 DEBUG:teuthology.orchestra.run:got remote process result: 128 2023-10-16T20:58:23.160 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2023-10-16T20:58:23.160 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-16T20:58:23.171 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-16T20:58:24.051 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_19fd96fe3581520b6d9f50ffdabf29051aee5d4d/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_19fd96fe3581520b6d9f50ffdabf29051aee5d4d/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_19fd96fe3581520b6d9f50ffdabf29051aee5d4d/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi088 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=19fd96fe3581520b6d9f50ffdabf29051aee5d4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-10-16T20:58:24.123 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3f584a96087948f59c1ae976fcc32472 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_19fd96fe3581520b6d9f50ffdabf29051aee5d4d/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_19fd96fe3581520b6d9f50ffdabf29051aee5d4d/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_19fd96fe3581520b6d9f50ffdabf29051aee5d4d/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi088 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=19fd96fe3581520b6d9f50ffdabf29051aee5d4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-10-16T20:58:24.126 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2023-10-16T20:58:24.136 INFO:tasks.mds_thrash:joining mds_thrasher 2023-10-16T20:58:25.171 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:58:25.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.315+0000 7f8ff530f640 1 Processor -- start 2023-10-16T20:58:25.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.315+0000 7f8ff530f640 1 -- start start 2023-10-16T20:58:25.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.319+0000 7f8ff530f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8ff0150660 0x7f8ff00c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:25.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.319+0000 7f8ff530f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff00c5c60 con 0x7f8ff00c6bc0 2023-10-16T20:58:25.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.319+0000 7f8ff530f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff00c5370 con 0x7f8ff0150280 2023-10-16T20:58:25.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.319+0000 7f8ff530f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff0156d10 con 0x7f8ff0150660 2023-10-16T20:58:25.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.319+0000 7f8fee7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8ff0150660 0x7f8ff00c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:25.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.319+0000 7f8fef7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8ff00c6bc0 0x7f8ff0156110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:38338/0 (socket says 172.21.15.88:38338) 2023-10-16T20:58:25.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.319+0000 7f8fef7fe640 1 -- 172.21.15.88:0/2576914656 learned_addr learned my addr 172.21.15.88:0/2576914656 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:25.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.319+0000 7f8fee7fc640 1 -- 172.21.15.88:0/2576914656 >> v1:172.21.15.191:6789/0 conn(0x7f8ff0150280 legacy=0x7f8ff006d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:25.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.319+0000 7f8fee7fc640 1 -- 172.21.15.88:0/2576914656 >> v1:172.21.15.88:6789/0 conn(0x7f8ff00c6bc0 legacy=0x7f8ff0156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:25.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.319+0000 7f8fedffb640 1 -- 172.21.15.88:0/2576914656 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908457854 0 0) 0x7f8ff00c5c60 con 0x7f8ff00c6bc0 2023-10-16T20:58:25.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.319+0000 7f8fee7fc640 1 -- 172.21.15.88:0/2576914656 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff00c5610 con 0x7f8ff0150660 2023-10-16T20:58:25.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.323+0000 7f8fee7fc640 1 --2- 172.21.15.88:0/2576914656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ff0150660 0x7f8ff00c60f0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f8fe40097c0 tx=0x7f8fe401ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d91c72ea512072a server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:25.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.323+0000 7f8fedffb640 1 -- 172.21.15.88:0/2576914656 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe40ab670 con 0x7f8ff0150660 2023-10-16T20:58:25.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.323+0000 7f8fedffb640 1 -- 172.21.15.88:0/2576914656 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe40ab7d0 con 0x7f8ff0150660 2023-10-16T20:58:25.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.323+0000 7f8fedffb640 1 -- 172.21.15.88:0/2576914656 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe40abb60 con 0x7f8ff0150660 2023-10-16T20:58:25.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.323+0000 7f8ff530f640 1 -- 172.21.15.88:0/2576914656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ff0150660 msgr2=0x7f8ff00c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:25.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.323+0000 7f8ff530f640 1 --2- 172.21.15.88:0/2576914656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ff0150660 0x7f8ff00c60f0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f8fe40097c0 tx=0x7f8fe401ceb0 comp rx=0 tx=0).stop 2023-10-16T20:58:25.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.323+0000 7f8ff530f640 1 -- 172.21.15.88:0/2576914656 shutdown_connections 2023-10-16T20:58:25.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.323+0000 7f8ff530f640 1 --2- 172.21.15.88:0/2576914656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ff0150660 0x7f8ff00c60f0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:25.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.323+0000 7f8ff530f640 1 -- 172.21.15.88:0/2576914656 >> 172.21.15.88:0/2576914656 conn(0x7f8ff00cad50 msgr2=0x7f8ff00cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:25.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.323+0000 7f8ff530f640 1 -- 172.21.15.88:0/2576914656 shutdown_connections 2023-10-16T20:58:25.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.323+0000 7f8ff530f640 1 -- 172.21.15.88:0/2576914656 wait complete. 2023-10-16T20:58:25.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.323+0000 7f8ff530f640 1 Processor -- start 2023-10-16T20:58:25.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.323+0000 7f8ff530f640 1 -- start start 2023-10-16T20:58:25.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8ff530f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff00c6bc0 0x7f8ff0146f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:25.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8ff530f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ff0150280 0x7f8ff006a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:25.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8ff530f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ff0150660 0x7f8ff01f58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:25.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8ff530f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff01517e0 con 0x7f8ff0150660 2023-10-16T20:58:25.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8ff530f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff0150c60 con 0x7f8ff00c6bc0 2023-10-16T20:58:25.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8ff530f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff0150f40 con 0x7f8ff0150280 2023-10-16T20:58:25.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8fef7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ff0150660 0x7f8ff01f58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:25.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8fef7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ff0150660 0x7f8ff01f58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39926/0 (socket says 172.21.15.88:39926) 2023-10-16T20:58:25.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8fef7fe640 1 -- 172.21.15.88:0/1297777358 learned_addr learned my addr 172.21.15.88:0/1297777358 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:25.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8feeffd640 1 --2- 172.21.15.88:0/1297777358 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff00c6bc0 0x7f8ff0146f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:25.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8fee7fc640 1 --2- 172.21.15.88:0/1297777358 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ff0150280 0x7f8ff006a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:25.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8fef7fe640 1 -- 172.21.15.88:0/1297777358 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ff0150280 msgr2=0x7f8ff006a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8fef7fe640 1 --2- 172.21.15.88:0/1297777358 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ff0150280 0x7f8ff006a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8fef7fe640 1 -- 172.21.15.88:0/1297777358 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff00c6bc0 msgr2=0x7f8ff0146f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8fef7fe640 1 --2- 172.21.15.88:0/1297777358 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff00c6bc0 0x7f8ff0146f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:25.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8fef7fe640 1 -- 172.21.15.88:0/1297777358 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe4009420 con 0x7f8ff0150660 2023-10-16T20:58:25.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8fef7fe640 1 --2- 172.21.15.88:0/1297777358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ff0150660 0x7f8ff01f58b0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8001570 tx=0x7f8fe800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:25.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8fd77fe640 1 -- 172.21.15.88:0/1297777358 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe800a640 con 0x7f8ff0150660 2023-10-16T20:58:25.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8fd77fe640 1 -- 172.21.15.88:0/1297777358 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe800a7c0 con 0x7f8ff0150660 2023-10-16T20:58:25.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8ff530f640 1 -- 172.21.15.88:0/1297777358 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff0151220 con 0x7f8ff0150660 2023-10-16T20:58:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8fd77fe640 1 -- 172.21.15.88:0/1297777358 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe800ab70 con 0x7f8ff0150660 2023-10-16T20:58:25.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8ff530f640 1 -- 172.21.15.88:0/1297777358 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff01489c0 con 0x7f8ff0150660 2023-10-16T20:58:25.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8ff530f640 1 -- 172.21.15.88:0/1297777358 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff0074700 con 0x7f8ff0150660 2023-10-16T20:58:25.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.327+0000 7f8fd77fe640 1 -- 172.21.15.88:0/1297777358 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8fe800acd0 con 0x7f8ff0150660 2023-10-16T20:58:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.331+0000 7f8fd77fe640 1 --2- 172.21.15.88:0/1297777358 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8fdc0664a0 0x7f8fdc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.331+0000 7f8fd77fe640 1 -- 172.21.15.88:0/1297777358 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f8fe8083dc0 con 0x7f8ff0150660 2023-10-16T20:58:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.331+0000 7f8feeffd640 1 --2- 172.21.15.88:0/1297777358 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8fdc0664a0 0x7f8fdc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:25.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.331+0000 7f8feeffd640 1 --2- 172.21.15.88:0/1297777358 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8fdc0664a0 0x7f8fdc068960 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0007fd0 tx=0x7f8fe0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:25.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.335+0000 7f8fd77fe640 1 -- 172.21.15.88:0/1297777358 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8fe8050bc0 con 0x7f8ff0150660 2023-10-16T20:58:25.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.515+0000 7f8ff530f640 1 -- 172.21.15.88:0/1297777358 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ff00c1b30 con 0x7f8ff0150660 2023-10-16T20:58:25.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.515+0000 7f8fd77fe640 1 -- 172.21.15.88:0/1297777358 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+5787 (secure 0 0 0) 0x7f8fe8018090 con 0x7f8ff0150660 2023-10-16T20:58:25.516 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:25.516 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:58:22.748501+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"c","rank":4,"incarnation":112,"state":"up:active","state_seq":23,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:58:25.517 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 116 2023-10-16T20:58:25.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.515+0000 7f8ff530f640 1 -- 172.21.15.88:0/1297777358 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8fdc0664a0 msgr2=0x7f8fdc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:25.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.515+0000 7f8ff530f640 1 --2- 172.21.15.88:0/1297777358 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8fdc0664a0 0x7f8fdc068960 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0007fd0 tx=0x7f8fe0009040 comp rx=0 tx=0).stop 2023-10-16T20:58:25.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.515+0000 7f8ff530f640 1 -- 172.21.15.88:0/1297777358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ff0150660 msgr2=0x7f8ff01f58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:25.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.515+0000 7f8ff530f640 1 --2- 172.21.15.88:0/1297777358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ff0150660 0x7f8ff01f58b0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8001570 tx=0x7f8fe800cca0 comp rx=0 tx=0).stop 2023-10-16T20:58:25.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.515+0000 7f8ff530f640 1 -- 172.21.15.88:0/1297777358 shutdown_connections 2023-10-16T20:58:25.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.515+0000 7f8ff530f640 1 --2- 172.21.15.88:0/1297777358 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8fdc0664a0 0x7f8fdc068960 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:25.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.515+0000 7f8ff530f640 1 --2- 172.21.15.88:0/1297777358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ff0150660 0x7f8ff01f58b0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:25.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.515+0000 7f8ff530f640 1 --2- 172.21.15.88:0/1297777358 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8ff0150280 0x7f8ff006a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:25.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.515+0000 7f8ff530f640 1 --2- 172.21.15.88:0/1297777358 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff00c6bc0 0x7f8ff0146f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:25.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.515+0000 7f8ff530f640 1 -- 172.21.15.88:0/1297777358 >> 172.21.15.88:0/1297777358 conn(0x7f8ff00cad50 msgr2=0x7f8ff00cb130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:25.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.515+0000 7f8ff530f640 1 -- 172.21.15.88:0/1297777358 shutdown_connections 2023-10-16T20:58:25.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:25.515+0000 7f8ff530f640 1 -- 172.21.15.88:0/1297777358 wait complete. 2023-10-16T20:58:25.538 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=4, replacing gid=5378 2023-10-16T20:58:25.538 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.f 2023-10-16T20:58:31.483 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:58:31.489 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:58:32.539 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-16T20:58:32.540 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-16T20:58:32.540 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-16T20:58:32.544 INFO:tasks.ceph.mds.f:Started 2023-10-16T20:58:32.545 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:58:32.640 INFO:tasks.ceph.mds.f.smithi191.stdout:starting mds.f at 2023-10-16T20:58:32.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.695+0000 7f9dc8e6b640 1 Processor -- start 2023-10-16T20:58:32.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.695+0000 7f9dc8e6b640 1 -- start start 2023-10-16T20:58:32.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.695+0000 7f9dc8e6b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9dc41406d0 0x7f9dc40c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:32.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.695+0000 7f9dc8e6b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9dc406a970 0x7f9dc4071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:32.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.695+0000 7f9dc8e6b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc415b3c0 con 0x7f9dc4071e60 2023-10-16T20:58:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.695+0000 7f9dc8e6b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc415b680 con 0x7f9dc41406d0 2023-10-16T20:58:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.695+0000 7f9dc8e6b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc4157570 con 0x7f9dc406a970 2023-10-16T20:58:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.695+0000 7f9dc259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9dc41406d0 0x7f9dc40c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.695+0000 7f9dc2d9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9dc4071e60 0x7f9dc415a810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35052/0 (socket says 172.21.15.88:35052) 2023-10-16T20:58:32.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.695+0000 7f9dc2d9d640 1 -- 172.21.15.88:0/3603556955 learned_addr learned my addr 172.21.15.88:0/3603556955 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:32.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.695+0000 7f9dc1d9b640 1 --2- 172.21.15.88:0/3603556955 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9dc406a970 0x7f9dc4071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:32.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.695+0000 7f9dc159a640 1 -- 172.21.15.88:0/3603556955 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973366967 0 0) 0x7f9dc415b3c0 con 0x7f9dc4071e60 2023-10-16T20:58:32.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.695+0000 7f9dc259c640 1 -- 172.21.15.88:0/3603556955 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9dc406a970 msgr2=0x7f9dc4071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:32.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.695+0000 7f9dc259c640 1 --2- 172.21.15.88:0/3603556955 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9dc406a970 0x7f9dc4071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:32.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.695+0000 7f9dc259c640 1 -- 172.21.15.88:0/3603556955 >> v1:172.21.15.88:6789/0 conn(0x7f9dc4071e60 legacy=0x7f9dc415a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:32.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.695+0000 7f9dc259c640 1 -- 172.21.15.88:0/3603556955 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc4157850 con 0x7f9dc41406d0 2023-10-16T20:58:32.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.699+0000 7f9dc259c640 1 --2- 172.21.15.88:0/3603556955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9dc41406d0 0x7f9dc40c1d50 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f9db40097c0 tx=0x7f9db401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2141679c16e69c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:32.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.699+0000 7f9dc159a640 1 -- 172.21.15.88:0/3603556955 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db40ab670 con 0x7f9dc41406d0 2023-10-16T20:58:32.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.699+0000 7f9dc159a640 1 -- 172.21.15.88:0/3603556955 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db40ab7d0 con 0x7f9dc41406d0 2023-10-16T20:58:32.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.699+0000 7f9dc159a640 1 -- 172.21.15.88:0/3603556955 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db40abb60 con 0x7f9dc41406d0 2023-10-16T20:58:32.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.699+0000 7f9dc8e6b640 1 -- 172.21.15.88:0/3603556955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9dc41406d0 msgr2=0x7f9dc40c1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:32.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.699+0000 7f9dc8e6b640 1 --2- 172.21.15.88:0/3603556955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9dc41406d0 0x7f9dc40c1d50 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f9db40097c0 tx=0x7f9db401c450 comp rx=0 tx=0).stop 2023-10-16T20:58:32.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.699+0000 7f9dc8e6b640 1 -- 172.21.15.88:0/3603556955 shutdown_connections 2023-10-16T20:58:32.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.699+0000 7f9dc8e6b640 1 --2- 172.21.15.88:0/3603556955 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9dc406a970 0x7f9dc4071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:32.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.699+0000 7f9dc8e6b640 1 --2- 172.21.15.88:0/3603556955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9dc41406d0 0x7f9dc40c1d50 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:32.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.699+0000 7f9dc8e6b640 1 -- 172.21.15.88:0/3603556955 >> 172.21.15.88:0/3603556955 conn(0x7f9dc40cad50 msgr2=0x7f9dc40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:32.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.699+0000 7f9dc8e6b640 1 -- 172.21.15.88:0/3603556955 shutdown_connections 2023-10-16T20:58:32.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.699+0000 7f9dc8e6b640 1 -- 172.21.15.88:0/3603556955 wait complete. 2023-10-16T20:58:32.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc8e6b640 1 Processor -- start 2023-10-16T20:58:32.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc8e6b640 1 -- start start 2023-10-16T20:58:32.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc8e6b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9dc406a970 0x7f9dc41ef950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:32.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc8e6b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9dc4071e60 0x7f9dc41efe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:32.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc8e6b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9dc41406d0 0x7f9dc413dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:32.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc8e6b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc41ef520 con 0x7f9dc4071e60 2023-10-16T20:58:32.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc8e6b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc413f430 con 0x7f9dc406a970 2023-10-16T20:58:32.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc8e6b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc41f35d0 con 0x7f9dc41406d0 2023-10-16T20:58:32.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc1d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9dc4071e60 0x7f9dc41efe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:32.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc1d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9dc4071e60 0x7f9dc41efe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34206/0 (socket says 172.21.15.88:34206) 2023-10-16T20:58:32.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc1d9b640 1 -- 172.21.15.88:0/3390147227 learned_addr learned my addr 172.21.15.88:0/3390147227 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:32.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc259c640 1 --2- 172.21.15.88:0/3390147227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9dc406a970 0x7f9dc41ef950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:32.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc2d9d640 1 --2- 172.21.15.88:0/3390147227 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9dc41406d0 0x7f9dc413dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:32.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc1d9b640 1 -- 172.21.15.88:0/3390147227 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9dc41406d0 msgr2=0x7f9dc413dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:32.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc1d9b640 1 --2- 172.21.15.88:0/3390147227 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9dc41406d0 0x7f9dc413dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:32.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc1d9b640 1 -- 172.21.15.88:0/3390147227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9dc406a970 msgr2=0x7f9dc41ef950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:32.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc1d9b640 1 --2- 172.21.15.88:0/3390147227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9dc406a970 0x7f9dc41ef950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:32.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc1d9b640 1 -- 172.21.15.88:0/3390147227 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db4009420 con 0x7f9dc4071e60 2023-10-16T20:58:32.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc2d9d640 1 --2- 172.21.15.88:0/3390147227 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9dc41406d0 0x7f9dc413dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:58:32.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc1d9b640 1 --2- 172.21.15.88:0/3390147227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9dc4071e60 0x7f9dc41efe90 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f9db800e960 tx=0x7f9db800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:32.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9db2ffd640 1 -- 172.21.15.88:0/3390147227 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db8009690 con 0x7f9dc4071e60 2023-10-16T20:58:32.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9db2ffd640 1 -- 172.21.15.88:0/3390147227 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db800ca00 con 0x7f9dc4071e60 2023-10-16T20:58:32.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9db2ffd640 1 -- 172.21.15.88:0/3390147227 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db800cd70 con 0x7f9dc4071e60 2023-10-16T20:58:32.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc8e6b640 1 -- 172.21.15.88:0/3390147227 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc41581d0 con 0x7f9dc4071e60 2023-10-16T20:58:32.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.703+0000 7f9dc8e6b640 1 -- 172.21.15.88:0/3390147227 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc4159ff0 con 0x7f9dc4071e60 2023-10-16T20:58:32.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.707+0000 7f9dc8e6b640 1 -- 172.21.15.88:0/3390147227 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc4074700 con 0x7f9dc4071e60 2023-10-16T20:58:32.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.707+0000 7f9db2ffd640 1 -- 172.21.15.88:0/3390147227 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9db8013470 con 0x7f9dc4071e60 2023-10-16T20:58:32.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.707+0000 7f9db2ffd640 1 --2- 172.21.15.88:0/3390147227 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9da40664a0 0x7f9da4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.707+0000 7f9db2ffd640 1 -- 172.21.15.88:0/3390147227 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f9db8084830 con 0x7f9dc4071e60 2023-10-16T20:58:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.707+0000 7f9dc259c640 1 --2- 172.21.15.88:0/3390147227 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9da40664a0 0x7f9da4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.707+0000 7f9dc259c640 1 --2- 172.21.15.88:0/3390147227 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9da40664a0 0x7f9da4068960 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9db40af000 tx=0x7f9db40a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.711+0000 7f9db2ffd640 1 -- 172.21.15.88:0/3390147227 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9db80515a0 con 0x7f9dc4071e60 2023-10-16T20:58:32.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.895+0000 7f9dc8e6b640 1 -- 172.21.15.88:0/3390147227 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9dc4154c40 con 0x7f9dc4071e60 2023-10-16T20:58:32.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.895+0000 7f9db2ffd640 1 -- 172.21.15.88:0/3390147227 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+6581 (secure 0 0 0) 0x7f9db8054bc0 con 0x7f9dc4071e60 2023-10-16T20:58:32.896 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:32.896 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6286,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2135868641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2135868641},{"type":"v1","addr":"172.21.15.191:6835","nonce":2135868641}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:58:22.748501+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"c","rank":4,"incarnation":112,"state":"up:active","state_seq":23,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:58:32.897 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 117 2023-10-16T20:58:32.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.895+0000 7f9dc8e6b640 1 -- 172.21.15.88:0/3390147227 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9da40664a0 msgr2=0x7f9da4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:32.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.895+0000 7f9dc8e6b640 1 --2- 172.21.15.88:0/3390147227 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9da40664a0 0x7f9da4068960 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9db40af000 tx=0x7f9db40a7000 comp rx=0 tx=0).stop 2023-10-16T20:58:32.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.895+0000 7f9dc8e6b640 1 -- 172.21.15.88:0/3390147227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9dc4071e60 msgr2=0x7f9dc41efe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:32.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.895+0000 7f9dc8e6b640 1 --2- 172.21.15.88:0/3390147227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9dc4071e60 0x7f9dc41efe90 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f9db800e960 tx=0x7f9db800ee20 comp rx=0 tx=0).stop 2023-10-16T20:58:32.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.895+0000 7f9dc8e6b640 1 -- 172.21.15.88:0/3390147227 shutdown_connections 2023-10-16T20:58:32.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.895+0000 7f9dc8e6b640 1 --2- 172.21.15.88:0/3390147227 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9da40664a0 0x7f9da4068960 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:32.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.895+0000 7f9dc8e6b640 1 --2- 172.21.15.88:0/3390147227 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9dc41406d0 0x7f9dc413dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:32.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.895+0000 7f9dc8e6b640 1 --2- 172.21.15.88:0/3390147227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9dc4071e60 0x7f9dc41efe90 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:32.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.895+0000 7f9dc8e6b640 1 --2- 172.21.15.88:0/3390147227 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9dc406a970 0x7f9dc41ef950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:32.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.895+0000 7f9dc8e6b640 1 -- 172.21.15.88:0/3390147227 >> 172.21.15.88:0/3390147227 conn(0x7f9dc40cad50 msgr2=0x7f9dc413f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:32.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.895+0000 7f9dc8e6b640 1 -- 172.21.15.88:0/3390147227 shutdown_connections 2023-10-16T20:58:32.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:32.895+0000 7f9dc8e6b640 1 -- 172.21.15.88:0/3390147227 wait complete. 2023-10-16T20:58:32.917 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-16T20:58:32.917 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 0 2023-10-16T20:58:32.917 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2023-10-16T20:58:32.918 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 10 2023-10-16T20:58:32.918 INFO:tasks.mds_thrash:done joining 2023-10-16T20:58:32.919 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-10-16T20:58:32.928 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-10-16T20:58:32.929 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:58:32.929 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T20:58:32.939 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:58:32.939 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T20:58:32.990 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:58:33.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.171+0000 7fec2c70f640 1 Processor -- start 2023-10-16T20:58:33.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.171+0000 7fec2c70f640 1 -- start start 2023-10-16T20:58:33.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.171+0000 7fec2c70f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fec240c9bb0 0x7fec240c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.171+0000 7fec2c70f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec240c41f0 0x7fec240cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.171+0000 7fec2c70f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec24153b50 con 0x7fec24153770 2023-10-16T20:58:33.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.171+0000 7fec2c70f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec240cc250 con 0x7fec240c9bb0 2023-10-16T20:58:33.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.171+0000 7fec2c70f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec240cc670 con 0x7fec240c41f0 2023-10-16T20:58:33.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.171+0000 7fec29caa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fec240c9bb0 0x7fec240c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.171+0000 7fec29caa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fec240c9bb0 0x7fec240c3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34210/0 (socket says 172.21.15.88:34210) 2023-10-16T20:58:33.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.171+0000 7fec29caa640 1 -- 172.21.15.88:0/966629023 learned_addr learned my addr 172.21.15.88:0/966629023 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:33.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.171+0000 7fec2acac640 1 --2- 172.21.15.88:0/966629023 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec240c41f0 0x7fec240cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.171+0000 7fec29caa640 1 -- 172.21.15.88:0/966629023 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec240c41f0 msgr2=0x7fec240cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:33.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.171+0000 7fec29caa640 1 --2- 172.21.15.88:0/966629023 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec240c41f0 0x7fec240cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.171+0000 7fec29caa640 1 -- 172.21.15.88:0/966629023 >> v1:172.21.15.191:6789/0 conn(0x7fec24153770 legacy=0x7fec24159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:33.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.171+0000 7fec29caa640 1 -- 172.21.15.88:0/966629023 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec240cb2f0 con 0x7fec240c9bb0 2023-10-16T20:58:33.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.175+0000 7fec29caa640 1 --2- 172.21.15.88:0/966629023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec240c9bb0 0x7fec240c3cb0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fec14009870 tx=0x7fec1401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab44a148206133ae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:33.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.175+0000 7fec294a9640 1 -- 172.21.15.88:0/966629023 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec140ab650 con 0x7fec240c9bb0 2023-10-16T20:58:33.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.175+0000 7fec294a9640 1 -- 172.21.15.88:0/966629023 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec140ab7b0 con 0x7fec240c9bb0 2023-10-16T20:58:33.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.175+0000 7fec294a9640 1 -- 172.21.15.88:0/966629023 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec140abb40 con 0x7fec240c9bb0 2023-10-16T20:58:33.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.175+0000 7fec2c70f640 1 -- 172.21.15.88:0/966629023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec240c9bb0 msgr2=0x7fec240c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:33.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.175+0000 7fec2c70f640 1 --2- 172.21.15.88:0/966629023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec240c9bb0 0x7fec240c3cb0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fec14009870 tx=0x7fec1401c540 comp rx=0 tx=0).stop 2023-10-16T20:58:33.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.175+0000 7fec2c70f640 1 -- 172.21.15.88:0/966629023 shutdown_connections 2023-10-16T20:58:33.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.175+0000 7fec2c70f640 1 --2- 172.21.15.88:0/966629023 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec240c41f0 0x7fec240cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.175+0000 7fec2c70f640 1 --2- 172.21.15.88:0/966629023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec240c9bb0 0x7fec240c3cb0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.175+0000 7fec2c70f640 1 -- 172.21.15.88:0/966629023 >> 172.21.15.88:0/966629023 conn(0x7fec240715c0 msgr2=0x7fec240719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:33.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.175+0000 7fec2c70f640 1 -- 172.21.15.88:0/966629023 shutdown_connections 2023-10-16T20:58:33.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.175+0000 7fec2c70f640 1 -- 172.21.15.88:0/966629023 wait complete. 2023-10-16T20:58:33.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2c70f640 1 Processor -- start 2023-10-16T20:58:33.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2c70f640 1 -- start start 2023-10-16T20:58:33.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2c70f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec240c41f0 0x7fec24148740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2c70f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec240c9bb0 0x7fec241f9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2c70f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec24153770 0x7fec24143870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2c70f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec241fa3a0 con 0x7fec24153770 2023-10-16T20:58:33.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2c70f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec24148df0 con 0x7fec240c41f0 2023-10-16T20:58:33.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2c70f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec241490d0 con 0x7fec240c9bb0 2023-10-16T20:58:33.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2acac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec24153770 0x7fec24143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2a4ab640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec240c41f0 0x7fec24148740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2acac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec24153770 0x7fec24143870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34222/0 (socket says 172.21.15.88:34222) 2023-10-16T20:58:33.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2acac640 1 -- 172.21.15.88:0/4061595753 learned_addr learned my addr 172.21.15.88:0/4061595753 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:33.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec29caa640 1 --2- 172.21.15.88:0/4061595753 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec240c9bb0 0x7fec241f9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2acac640 1 -- 172.21.15.88:0/4061595753 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec240c9bb0 msgr2=0x7fec241f9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:33.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2acac640 1 --2- 172.21.15.88:0/4061595753 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec240c9bb0 0x7fec241f9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2acac640 1 -- 172.21.15.88:0/4061595753 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec240c41f0 msgr2=0x7fec24148740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:33.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2acac640 1 --2- 172.21.15.88:0/4061595753 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec240c41f0 0x7fec24148740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2acac640 1 -- 172.21.15.88:0/4061595753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec140094d0 con 0x7fec24153770 2023-10-16T20:58:33.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2acac640 1 --2- 172.21.15.88:0/4061595753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec24153770 0x7fec24143870 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fec1c007e40 tx=0x7fec1c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:33.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec1affd640 1 -- 172.21.15.88:0/4061595753 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c014620 con 0x7fec24153770 2023-10-16T20:58:33.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec1affd640 1 -- 172.21.15.88:0/4061595753 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1c00ee90 con 0x7fec24153770 2023-10-16T20:58:33.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec1affd640 1 -- 172.21.15.88:0/4061595753 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c0149e0 con 0x7fec24153770 2023-10-16T20:58:33.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2c70f640 1 -- 172.21.15.88:0/4061595753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec241493b0 con 0x7fec24153770 2023-10-16T20:58:33.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.179+0000 7fec2c70f640 1 -- 172.21.15.88:0/4061595753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec241f9940 con 0x7fec24153770 2023-10-16T20:58:33.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.183+0000 7fec2c70f640 1 -- 172.21.15.88:0/4061595753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec241f94c0 con 0x7fec24153770 2023-10-16T20:58:33.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.183+0000 7fec1affd640 1 -- 172.21.15.88:0/4061595753 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fec1c014b40 con 0x7fec24153770 2023-10-16T20:58:33.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.183+0000 7fec1affd640 1 --2- 172.21.15.88:0/4061595753 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fec080664a0 0x7fec08068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.183+0000 7fec1affd640 1 -- 172.21.15.88:0/4061595753 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7fec1c084eb0 con 0x7fec24153770 2023-10-16T20:58:33.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.183+0000 7fec2a4ab640 1 --2- 172.21.15.88:0/4061595753 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fec080664a0 0x7fec08068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.183+0000 7fec2a4ab640 1 --2- 172.21.15.88:0/4061595753 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fec080664a0 0x7fec08068960 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fec20000fc0 tx=0x7fec20009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:33.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.187+0000 7fec1affd640 1 -- 172.21.15.88:0/4061595753 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fec241f94c0 con 0x7fec24153770 2023-10-16T20:58:33.338 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.339+0000 7fec2c70f640 1 -- 172.21.15.88:0/4061595753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec241f94c0 con 0x7fec24153770 2023-10-16T20:58:33.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.339+0000 7fec1affd640 1 -- 172.21.15.88:0/4061595753 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+20065 (secure 0 0 0) 0x7fec241f94c0 con 0x7fec24153770 2023-10-16T20:58:33.340 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:33.340 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":67,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:58:00.653255+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6839/2172973775":"2023-10-17T20:56:40.633674+0000","172.21.15.191:6833/3633326602":"2023-10-17T20:55:50.604940+0000","172.21.15.191:6836/1120825335":"2023-10-17T20:42:10.335721+0000","172.21.15.191:6832/3633326602":"2023-10-17T20:55:50.604940+0000","172.21.15.88:6835/518281867":"2023-10-17T20:44:25.369398+0000","172.21.15.191:6837/2468647900":"2023-10-17T20:53:25.553637+0000","172.21.15.88:6837/1987579860":"2023-10-17T20:43:25.353447+0000","172.21.15.191:6835/3119045295":"2023-10-17T20:58:00.653202+0000","172.21.15.88:6836/321173984":"2023-10-17T20:46:15.444657+0000","172.21.15.191:6837/1120825335":"2023-10-17T20:42:10.335721+0000","172.21.15.191:6832/2446850309":"2023-10-17T20:48:43.641901+0000","172.21.15.88:6837/2172973775":"2023-10-17T20:56:40.633674+0000","172.21.15.88:6839/1987579860":"2023-10-17T20:43:25.353447+0000","172.21.15.191:6834/3119045295":"2023-10-17T20:58:00.653202+0000","172.21.15.88:6834/518281867":"2023-10-17T20:44:25.369398+0000","172.21.15.88:6838/321173984":"2023-10-17T20:46:15.444657+0000","172.21.15.191:6834/3059699574":"2023-10-17T20:51:20.522638+0000","172.21.15.191:6833/2446850309":"2023-10-17T20:48:43.641901+0000","172.21.15.191:6835/3059699574":"2023-10-17T20:51:20.522638+0000","172.21.15.191:6836/2468647900":"2023-10-17T20:53:25.553637+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:58:33.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.339+0000 7fec2c70f640 1 -- 172.21.15.88:0/4061595753 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fec080664a0 msgr2=0x7fec08068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:33.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.339+0000 7fec2c70f640 1 --2- 172.21.15.88:0/4061595753 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fec080664a0 0x7fec08068960 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fec20000fc0 tx=0x7fec20009040 comp rx=0 tx=0).stop 2023-10-16T20:58:33.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.339+0000 7fec2c70f640 1 -- 172.21.15.88:0/4061595753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec24153770 msgr2=0x7fec24143870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:33.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.339+0000 7fec2c70f640 1 --2- 172.21.15.88:0/4061595753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec24153770 0x7fec24143870 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fec1c007e40 tx=0x7fec1c00e4b0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.339+0000 7fec2c70f640 1 -- 172.21.15.88:0/4061595753 shutdown_connections 2023-10-16T20:58:33.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.339+0000 7fec2c70f640 1 --2- 172.21.15.88:0/4061595753 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fec080664a0 0x7fec08068960 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.339+0000 7fec2c70f640 1 --2- 172.21.15.88:0/4061595753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec24153770 0x7fec24143870 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.339+0000 7fec2c70f640 1 --2- 172.21.15.88:0/4061595753 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fec240c9bb0 0x7fec241f9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.343+0000 7fec2c70f640 1 --2- 172.21.15.88:0/4061595753 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fec240c41f0 0x7fec24148740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.343+0000 7fec2c70f640 1 -- 172.21.15.88:0/4061595753 >> 172.21.15.88:0/4061595753 conn(0x7fec240715c0 msgr2=0x7fec24151b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:33.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.343+0000 7fec2c70f640 1 -- 172.21.15.88:0/4061595753 shutdown_connections 2023-10-16T20:58:33.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.343+0000 7fec2c70f640 1 -- 172.21.15.88:0/4061595753 wait complete. 2023-10-16T20:58:33.361 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T20:36:43.646418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T20:36:53.546513+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T20:36:54.143703+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 59, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T20:36:57.314642+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 59, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T20:58:33.362 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T20:58:33.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.503+0000 7fe908279640 1 Processor -- start 2023-10-16T20:58:33.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.503+0000 7fe908279640 1 -- start start 2023-10-16T20:58:33.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.503+0000 7fe908279640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe9000c66e0 0x7fe90015a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.503+0000 7fe908279640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe900070f40 con 0x7fe9000c6ac0 2023-10-16T20:58:33.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.503+0000 7fe908279640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90015a790 con 0x7fe9000cb5c0 2023-10-16T20:58:33.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.503+0000 7fe908279640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe90006a2d0 con 0x7fe9000c66e0 2023-10-16T20:58:33.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.503+0000 7fe905814640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe9000c6ac0 0x7fe900150150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35060/0 (socket says 172.21.15.88:35060) 2023-10-16T20:58:33.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.503+0000 7fe905814640 1 -- 172.21.15.88:0/2839689489 learned_addr learned my addr 172.21.15.88:0/2839689489 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:33.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe906015640 1 --2- 172.21.15.88:0/2839689489 >> v2:172.21.15.191:3300/0 conn(0x7fe9000c66e0 0x7fe90015a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe905013640 1 -- 172.21.15.88:0/2839689489 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111106095 0 0) 0x7fe900070f40 con 0x7fe9000c6ac0 2023-10-16T20:58:33.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe905013640 1 -- 172.21.15.88:0/2839689489 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8ec003530 con 0x7fe9000c6ac0 2023-10-16T20:58:33.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe905013640 1 -- 172.21.15.88:0/2839689489 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4081980688 0 0) 0x7fe8ec003530 con 0x7fe9000c6ac0 2023-10-16T20:58:33.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe905013640 1 -- 172.21.15.88:0/2839689489 >> v2:172.21.15.191:3300/0 conn(0x7fe9000c66e0 msgr2=0x7fe90015a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:33.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe905013640 1 --2- 172.21.15.88:0/2839689489 >> v2:172.21.15.191:3300/0 conn(0x7fe9000c66e0 0x7fe90015a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe905013640 1 -- 172.21.15.88:0/2839689489 >> v1:172.21.15.191:6789/0 conn(0x7fe9000cb5c0 legacy=0x7fe900071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:33.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe905013640 1 -- 172.21.15.88:0/2839689489 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90006a5b0 con 0x7fe9000c6ac0 2023-10-16T20:58:33.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe905013640 1 -- 172.21.15.88:0/2839689489 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7fe8f0001680 con 0x7fe9000c6ac0 2023-10-16T20:58:33.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe905013640 1 -- 172.21.15.88:0/2839689489 >> v1:172.21.15.88:6789/0 conn(0x7fe9000c6ac0 legacy=0x7fe900150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:33.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe905013640 1 --2- 172.21.15.88:0/2839689489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8ec003980 0x7fe8ec003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe905013640 1 --2- 172.21.15.88:0/2839689489 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8ec0047d0 0x7fe8ec004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe905013640 1 --2- 172.21.15.88:0/2839689489 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8ec005190 0x7fe8ec00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe905013640 1 -- 172.21.15.88:0/2839689489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90006a2d0 con 0x7fe8ec003980 2023-10-16T20:58:33.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe905013640 1 -- 172.21.15.88:0/2839689489 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec00c130 con 0x7fe8ec0047d0 2023-10-16T20:58:33.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe905013640 1 -- 172.21.15.88:0/2839689489 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec00c2f0 con 0x7fe8ec005190 2023-10-16T20:58:33.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe906015640 1 --2- 172.21.15.88:0/2839689489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8ec003980 0x7fe8ec003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe905814640 1 --2- 172.21.15.88:0/2839689489 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8ec0047d0 0x7fe8ec004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe906816640 1 --2- 172.21.15.88:0/2839689489 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8ec005190 0x7fe8ec00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe906015640 1 -- 172.21.15.88:0/2839689489 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8ec005190 msgr2=0x7fe8ec00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:33.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe906015640 1 --2- 172.21.15.88:0/2839689489 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8ec005190 0x7fe8ec00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe906015640 1 -- 172.21.15.88:0/2839689489 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8ec0047d0 msgr2=0x7fe8ec004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:33.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe906015640 1 --2- 172.21.15.88:0/2839689489 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe8ec0047d0 0x7fe8ec004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe906015640 1 -- 172.21.15.88:0/2839689489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ec003530 con 0x7fe8ec003980 2023-10-16T20:58:33.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.507+0000 7fe906816640 1 -- 172.21.15.88:0/2839689489 reap_dead start 2023-10-16T20:58:33.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.511+0000 7fe906015640 1 --2- 172.21.15.88:0/2839689489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8ec003980 0x7fe8ec003da0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009ae0 tx=0x7fe8fc01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd76dfefd930ed0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:33.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.511+0000 7fe905013640 1 -- 172.21.15.88:0/2839689489 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc0a6430 con 0x7fe8ec003980 2023-10-16T20:58:33.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.511+0000 7fe905013640 1 -- 172.21.15.88:0/2839689489 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8fc0a6590 con 0x7fe8ec003980 2023-10-16T20:58:33.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.511+0000 7fe905013640 1 -- 172.21.15.88:0/2839689489 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc0a6940 con 0x7fe8ec003980 2023-10-16T20:58:33.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.511+0000 7fe908279640 1 -- 172.21.15.88:0/2839689489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8ec003980 msgr2=0x7fe8ec003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:33.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.511+0000 7fe908279640 1 --2- 172.21.15.88:0/2839689489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8ec003980 0x7fe8ec003da0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009ae0 tx=0x7fe8fc01ca50 comp rx=0 tx=0).stop 2023-10-16T20:58:33.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.511+0000 7fe908279640 1 -- 172.21.15.88:0/2839689489 shutdown_connections 2023-10-16T20:58:33.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.511+0000 7fe908279640 1 --2- 172.21.15.88:0/2839689489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8ec003980 0x7fe8ec003da0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.511+0000 7fe908279640 1 -- 172.21.15.88:0/2839689489 >> 172.21.15.88:0/2839689489 conn(0x7fe9000ca9f0 msgr2=0x7fe9000cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:33.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.511+0000 7fe908279640 1 -- 172.21.15.88:0/2839689489 shutdown_connections 2023-10-16T20:58:33.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.511+0000 7fe908279640 1 -- 172.21.15.88:0/2839689489 wait complete. 2023-10-16T20:58:33.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.511+0000 7fe908279640 1 Processor -- start 2023-10-16T20:58:33.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.511+0000 7fe908279640 1 -- start start 2023-10-16T20:58:33.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.511+0000 7fe908279640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8ec003980 0x7fe900154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.511+0000 7fe908279640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe90014e450 0x7fe9001f53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe908279640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9000be300 0x7fe9000be780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe908279640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe900153b30 con 0x7fe90014e450 2023-10-16T20:58:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe908279640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe900154bc0 con 0x7fe9000be300 2023-10-16T20:58:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe908279640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9000becc0 con 0x7fe8ec003980 2023-10-16T20:58:33.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe905814640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe90014e450 0x7fe9001f53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe905814640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe90014e450 0x7fe9001f53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34242/0 (socket says 172.21.15.88:34242) 2023-10-16T20:58:33.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe905814640 1 -- 172.21.15.88:0/3776314818 learned_addr learned my addr 172.21.15.88:0/3776314818 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:33.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe906816640 1 --2- 172.21.15.88:0/3776314818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9000be300 0x7fe9000be780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe906015640 1 --2- 172.21.15.88:0/3776314818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8ec003980 0x7fe900154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe905814640 1 -- 172.21.15.88:0/3776314818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8ec003980 msgr2=0x7fe900154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe905814640 1 --2- 172.21.15.88:0/3776314818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8ec003980 0x7fe900154680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe905814640 1 -- 172.21.15.88:0/3776314818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9000be300 msgr2=0x7fe9000be780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:33.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe905814640 1 --2- 172.21.15.88:0/3776314818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9000be300 0x7fe9000be780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe905814640 1 -- 172.21.15.88:0/3776314818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9001f5d00 con 0x7fe90014e450 2023-10-16T20:58:33.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe905814640 1 --2- 172.21.15.88:0/3776314818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe90014e450 0x7fe9001f53c0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000e7b0 tx=0x7fe8f000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:33.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe8f6ffd640 1 -- 172.21.15.88:0/3776314818 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f000ee60 con 0x7fe90014e450 2023-10-16T20:58:33.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe8f6ffd640 1 -- 172.21.15.88:0/3776314818 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f000c8d0 con 0x7fe90014e450 2023-10-16T20:58:33.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe908279640 1 -- 172.21.15.88:0/3776314818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9000bfbd0 con 0x7fe90014e450 2023-10-16T20:58:33.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe8f6ffd640 1 -- 172.21.15.88:0/3776314818 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f000cc80 con 0x7fe90014e450 2023-10-16T20:58:33.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe908279640 1 -- 172.21.15.88:0/3776314818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9000c0150 con 0x7fe90014e450 2023-10-16T20:58:33.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe908279640 1 -- 172.21.15.88:0/3776314818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9000c7e30 con 0x7fe90014e450 2023-10-16T20:58:33.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe8f6ffd640 1 -- 172.21.15.88:0/3776314818 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe8f001d460 con 0x7fe90014e450 2023-10-16T20:58:33.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.515+0000 7fe8f6ffd640 1 --2- 172.21.15.88:0/3776314818 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe8e40664a0 0x7fe8e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.519+0000 7fe8f6ffd640 1 -- 172.21.15.88:0/3776314818 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7fe8f00843b0 con 0x7fe90014e450 2023-10-16T20:58:33.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.519+0000 7fe906015640 1 --2- 172.21.15.88:0/3776314818 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe8e40664a0 0x7fe8e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.519+0000 7fe906015640 1 --2- 172.21.15.88:0/3776314818 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe8e40664a0 0x7fe8e4068960 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc0ad000 tx=0x7fe8fc0b2040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:33.531 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.523+0000 7fe8f6ffd640 1 -- 172.21.15.88:0/3776314818 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe8f0051120 con 0x7fe90014e450 2023-10-16T20:58:33.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.671+0000 7fe908279640 1 -- 172.21.15.88:0/3776314818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe9000c5fa0 con 0x7fe90014e450 2023-10-16T20:58:33.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.671+0000 7fe8f6ffd640 1 -- 172.21.15.88:0/3776314818 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe8f0054740 con 0x7fe90014e450 2023-10-16T20:58:33.672 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-10-16T20:58:33.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.671+0000 7fe908279640 1 -- 172.21.15.88:0/3776314818 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe8e40664a0 msgr2=0x7fe8e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:33.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.671+0000 7fe908279640 1 --2- 172.21.15.88:0/3776314818 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe8e40664a0 0x7fe8e4068960 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc0ad000 tx=0x7fe8fc0b2040 comp rx=0 tx=0).stop 2023-10-16T20:58:33.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.671+0000 7fe908279640 1 -- 172.21.15.88:0/3776314818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe90014e450 msgr2=0x7fe9001f53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:33.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.671+0000 7fe908279640 1 --2- 172.21.15.88:0/3776314818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe90014e450 0x7fe9001f53c0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000e7b0 tx=0x7fe8f000ec70 comp rx=0 tx=0).stop 2023-10-16T20:58:33.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.671+0000 7fe908279640 1 -- 172.21.15.88:0/3776314818 shutdown_connections 2023-10-16T20:58:33.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.671+0000 7fe908279640 1 --2- 172.21.15.88:0/3776314818 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe8e40664a0 0x7fe8e4068960 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.671+0000 7fe908279640 1 --2- 172.21.15.88:0/3776314818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9000be300 0x7fe9000be780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.671+0000 7fe908279640 1 --2- 172.21.15.88:0/3776314818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe90014e450 0x7fe9001f53c0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.671+0000 7fe908279640 1 --2- 172.21.15.88:0/3776314818 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe8ec003980 0x7fe900154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.671+0000 7fe908279640 1 -- 172.21.15.88:0/3776314818 >> 172.21.15.88:0/3776314818 conn(0x7fe9000ca9f0 msgr2=0x7fe900159630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:33.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.671+0000 7fe908279640 1 -- 172.21.15.88:0/3776314818 shutdown_connections 2023-10-16T20:58:33.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.671+0000 7fe908279640 1 -- 172.21.15.88:0/3776314818 wait complete. 2023-10-16T20:58:33.693 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T20:58:33.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.835+0000 7fd0e471d640 1 Processor -- start 2023-10-16T20:58:33.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.835+0000 7fd0e471d640 1 -- start start 2023-10-16T20:58:33.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.835+0000 7fd0e471d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd0dc0c1520 0x7fd0dc164f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.835+0000 7fd0e471d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0dc0c1930 con 0x7fd0dc153a30 2023-10-16T20:58:33.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.835+0000 7fd0e471d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0dc1596e0 con 0x7fd0dc071450 2023-10-16T20:58:33.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.835+0000 7fd0e471d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0dc0c6cc0 con 0x7fd0dc0c1520 2023-10-16T20:58:33.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.835+0000 7fd0e2cba640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd0dc0c1520 0x7fd0dc164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.835+0000 7fd0e2cba640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd0dc0c1520 0x7fd0dc164f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34258/0 (socket says 172.21.15.88:34258) 2023-10-16T20:58:33.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.835+0000 7fd0e2cba640 1 -- 172.21.15.88:0/746091019 learned_addr learned my addr 172.21.15.88:0/746091019 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:33.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.835+0000 7fd0e24b9640 1 --1- 172.21.15.88:0/746091019 >> v1:172.21.15.88:6789/0 conn(0x7fd0dc153a30 0x7fd0dc0c65b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35062/0 (socket says 172.21.15.88:35062) 2023-10-16T20:58:33.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.839+0000 7fd0e2cba640 1 -- 172.21.15.88:0/746091019 >> v1:172.21.15.191:6789/0 conn(0x7fd0dc071450 legacy=0x7fd0dc071f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:33.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.839+0000 7fd0e2cba640 1 -- 172.21.15.88:0/746091019 >> v1:172.21.15.88:6789/0 conn(0x7fd0dc153a30 legacy=0x7fd0dc0c65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:33.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.839+0000 7fd0e2cba640 1 -- 172.21.15.88:0/746091019 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc0c2da0 con 0x7fd0dc0c1520 2023-10-16T20:58:33.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.839+0000 7fd0e2cba640 1 --2- 172.21.15.88:0/746091019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0dc0c1520 0x7fd0dc164f60 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009e50 tx=0x7fd0d401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7af3fb316412b72c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:33.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.839+0000 7fd0e14b7640 1 -- 172.21.15.88:0/746091019 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d40a5860 con 0x7fd0dc0c1520 2023-10-16T20:58:33.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.839+0000 7fd0e14b7640 1 -- 172.21.15.88:0/746091019 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d40a59c0 con 0x7fd0dc0c1520 2023-10-16T20:58:33.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.839+0000 7fd0e14b7640 1 -- 172.21.15.88:0/746091019 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d40a5d50 con 0x7fd0dc0c1520 2023-10-16T20:58:33.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.839+0000 7fd0e471d640 1 -- 172.21.15.88:0/746091019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0dc0c1520 msgr2=0x7fd0dc164f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:33.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.839+0000 7fd0e471d640 1 --2- 172.21.15.88:0/746091019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0dc0c1520 0x7fd0dc164f60 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009e50 tx=0x7fd0d401c9a0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e471d640 1 -- 172.21.15.88:0/746091019 shutdown_connections 2023-10-16T20:58:33.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e471d640 1 --2- 172.21.15.88:0/746091019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0dc0c1520 0x7fd0dc164f60 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e471d640 1 -- 172.21.15.88:0/746091019 >> 172.21.15.88:0/746091019 conn(0x7fd0dc0c86b0 msgr2=0x7fd0dc0c8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:33.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e471d640 1 -- 172.21.15.88:0/746091019 shutdown_connections 2023-10-16T20:58:33.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e471d640 1 -- 172.21.15.88:0/746091019 wait complete. 2023-10-16T20:58:33.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e471d640 1 Processor -- start 2023-10-16T20:58:33.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e471d640 1 -- start start 2023-10-16T20:58:33.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e471d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0dc071450 0x7fd0dc1f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e471d640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0dc0c1520 0x7fd0dc144f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e471d640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0dc153a30 0x7fd0dc14a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e471d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc1f2cf0 con 0x7fd0dc071450 2023-10-16T20:58:33.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e471d640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc143620 con 0x7fd0dc153a30 2023-10-16T20:58:33.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e471d640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc1438c0 con 0x7fd0dc0c1520 2023-10-16T20:58:33.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e24b9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0dc071450 0x7fd0dc1f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e24b9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0dc071450 0x7fd0dc1f27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34260/0 (socket says 172.21.15.88:34260) 2023-10-16T20:58:33.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e24b9640 1 -- 172.21.15.88:0/616553327 learned_addr learned my addr 172.21.15.88:0/616553327 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:33.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e2cba640 1 --2- 172.21.15.88:0/616553327 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0dc153a30 0x7fd0dc14a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e1cb8640 1 --2- 172.21.15.88:0/616553327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0dc0c1520 0x7fd0dc144f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e24b9640 1 -- 172.21.15.88:0/616553327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0dc0c1520 msgr2=0x7fd0dc144f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:33.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e24b9640 1 --2- 172.21.15.88:0/616553327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0dc0c1520 0x7fd0dc144f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e24b9640 1 -- 172.21.15.88:0/616553327 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0dc153a30 msgr2=0x7fd0dc14a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:33.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e24b9640 1 --2- 172.21.15.88:0/616553327 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0dc153a30 0x7fd0dc14a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:33.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e24b9640 1 -- 172.21.15.88:0/616553327 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d4009b30 con 0x7fd0dc071450 2023-10-16T20:58:33.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e1cb8640 1 --2- 172.21.15.88:0/616553327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0dc0c1520 0x7fd0dc144f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:58:33.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e24b9640 1 --2- 172.21.15.88:0/616553327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0dc071450 0x7fd0dc1f27b0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800c430 tx=0x7fd0d800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:33.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0d2ffd640 1 -- 172.21.15.88:0/616553327 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d80074e0 con 0x7fd0dc071450 2023-10-16T20:58:33.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0d2ffd640 1 -- 172.21.15.88:0/616553327 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d8007640 con 0x7fd0dc071450 2023-10-16T20:58:33.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0d2ffd640 1 -- 172.21.15.88:0/616553327 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8004a10 con 0x7fd0dc071450 2023-10-16T20:58:33.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.843+0000 7fd0e471d640 1 -- 172.21.15.88:0/616553327 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0dc144470 con 0x7fd0dc071450 2023-10-16T20:58:33.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.847+0000 7fd0e471d640 1 -- 172.21.15.88:0/616553327 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0dc144a30 con 0x7fd0dc071450 2023-10-16T20:58:33.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.847+0000 7fd0e471d640 1 -- 172.21.15.88:0/616553327 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0dc071230 con 0x7fd0dc071450 2023-10-16T20:58:33.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.847+0000 7fd0d2ffd640 1 -- 172.21.15.88:0/616553327 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd0d8004b70 con 0x7fd0dc071450 2023-10-16T20:58:33.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.847+0000 7fd0d2ffd640 1 --2- 172.21.15.88:0/616553327 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd0c00664a0 0x7fd0c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:33.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.847+0000 7fd0d2ffd640 1 -- 172.21.15.88:0/616553327 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7fd0d8083fe0 con 0x7fd0dc071450 2023-10-16T20:58:33.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.847+0000 7fd0e1cb8640 1 --2- 172.21.15.88:0/616553327 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd0c00664a0 0x7fd0c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:33.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.847+0000 7fd0e1cb8640 1 --2- 172.21.15.88:0/616553327 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd0c00664a0 0x7fd0c0068960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc148990 tx=0x7fd0cc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:33.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:33.851+0000 7fd0d2ffd640 1 -- 172.21.15.88:0/616553327 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd0d8050c70 con 0x7fd0dc071450 2023-10-16T20:58:34.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.003+0000 7fd0e471d640 1 -- 172.21.15.88:0/616553327 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd0dc0caee0 con 0x7fd0dc071450 2023-10-16T20:58:34.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.003+0000 7fd0d2ffd640 1 -- 172.21.15.88:0/616553327 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v67) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd0d8054290 con 0x7fd0dc071450 2023-10-16T20:58:34.004 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:58:34.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.003+0000 7fd0e471d640 1 -- 172.21.15.88:0/616553327 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd0c00664a0 msgr2=0x7fd0c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.003+0000 7fd0e471d640 1 --2- 172.21.15.88:0/616553327 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd0c00664a0 0x7fd0c0068960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc148990 tx=0x7fd0cc009040 comp rx=0 tx=0).stop 2023-10-16T20:58:34.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.003+0000 7fd0e471d640 1 -- 172.21.15.88:0/616553327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0dc071450 msgr2=0x7fd0dc1f27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.003+0000 7fd0e471d640 1 --2- 172.21.15.88:0/616553327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0dc071450 0x7fd0dc1f27b0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800c430 tx=0x7fd0d800c8f0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.003+0000 7fd0e471d640 1 -- 172.21.15.88:0/616553327 shutdown_connections 2023-10-16T20:58:34.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.003+0000 7fd0e471d640 1 --2- 172.21.15.88:0/616553327 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd0c00664a0 0x7fd0c0068960 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.003+0000 7fd0e471d640 1 --2- 172.21.15.88:0/616553327 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd0dc153a30 0x7fd0dc14a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.003+0000 7fd0e471d640 1 --2- 172.21.15.88:0/616553327 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd0dc0c1520 0x7fd0dc144f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.003+0000 7fd0e471d640 1 --2- 172.21.15.88:0/616553327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0dc071450 0x7fd0dc1f27b0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.003+0000 7fd0e471d640 1 -- 172.21.15.88:0/616553327 >> 172.21.15.88:0/616553327 conn(0x7fd0dc0c86b0 msgr2=0x7fd0dc0c8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:34.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.003+0000 7fd0e471d640 1 -- 172.21.15.88:0/616553327 shutdown_connections 2023-10-16T20:58:34.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.003+0000 7fd0e471d640 1 -- 172.21.15.88:0/616553327 wait complete. 2023-10-16T20:58:34.024 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T20:58:34.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.163+0000 7f4ffaf65640 1 Processor -- start 2023-10-16T20:58:34.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.163+0000 7f4ffaf65640 1 -- start start 2023-10-16T20:58:34.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4ffaf65640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f4ff4153a00 0x7f4ff40cb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4ffaf65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ff40ad870 0x7f4ff40cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4ffaf65640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff40c0230 con 0x7f4ff4153620 2023-10-16T20:58:34.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4ffaf65640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff41594f0 con 0x7f4ff4153a00 2023-10-16T20:58:34.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4ffaf65640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff40c0480 con 0x7f4ff40ad870 2023-10-16T20:58:34.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4ff8d01640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4ff4153620 0x7f4ff40c5c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35074/0 (socket says 172.21.15.88:35074) 2023-10-16T20:58:34.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4ff8d01640 1 -- 172.21.15.88:0/1779685774 learned_addr learned my addr 172.21.15.88:0/1779685774 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:34.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4febfff640 1 --2- 172.21.15.88:0/1779685774 >> v2:172.21.15.191:3300/0 conn(0x7f4ff4153a00 0x7f4ff40cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4feb7fe640 1 -- 172.21.15.88:0/1779685774 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824204050 0 0) 0x7f4ff40c0230 con 0x7f4ff4153620 2023-10-16T20:58:34.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4ff9502640 1 --2- 172.21.15.88:0/1779685774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ff40ad870 0x7f4ff40cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4feb7fe640 1 -- 172.21.15.88:0/1779685774 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fdc003530 con 0x7f4ff4153620 2023-10-16T20:58:34.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4feb7fe640 1 -- 172.21.15.88:0/1779685774 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4100115993 0 0) 0x7f4fdc003530 con 0x7f4ff4153620 2023-10-16T20:58:34.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4feb7fe640 1 -- 172.21.15.88:0/1779685774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ff40ad870 msgr2=0x7f4ff40cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:34.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4feb7fe640 1 --2- 172.21.15.88:0/1779685774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ff40ad870 0x7f4ff40cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4feb7fe640 1 -- 172.21.15.88:0/1779685774 >> v2:172.21.15.191:3300/0 conn(0x7f4ff4153a00 msgr2=0x7f4ff40cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:34.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4feb7fe640 1 --2- 172.21.15.88:0/1779685774 >> v2:172.21.15.191:3300/0 conn(0x7f4ff4153a00 0x7f4ff40cb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4feb7fe640 1 -- 172.21.15.88:0/1779685774 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff40c0760 con 0x7f4ff4153620 2023-10-16T20:58:34.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4feb7fe640 1 -- 172.21.15.88:0/1779685774 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f4ff0001680 con 0x7f4ff4153620 2023-10-16T20:58:34.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4feb7fe640 1 -- 172.21.15.88:0/1779685774 >> v1:172.21.15.88:6789/0 conn(0x7f4ff4153620 legacy=0x7f4ff40c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4feb7fe640 1 --2- 172.21.15.88:0/1779685774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fdc003980 0x7f4fdc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4feb7fe640 1 --2- 172.21.15.88:0/1779685774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fdc0047d0 0x7f4fdc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4feb7fe640 1 --2- 172.21.15.88:0/1779685774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ff40ad870 0x7f4fdc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4feb7fe640 1 -- 172.21.15.88:0/1779685774 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff41594f0 con 0x7f4fdc003980 2023-10-16T20:58:34.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4feb7fe640 1 -- 172.21.15.88:0/1779685774 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff40c0480 con 0x7f4fdc0047d0 2023-10-16T20:58:34.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4feb7fe640 1 -- 172.21.15.88:0/1779685774 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc00b9c0 con 0x7f4ff40ad870 2023-10-16T20:58:34.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4ff8d01640 1 --2- 172.21.15.88:0/1779685774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fdc003980 0x7f4fdc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4ff9502640 1 --2- 172.21.15.88:0/1779685774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ff40ad870 0x7f4fdc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4febfff640 1 --2- 172.21.15.88:0/1779685774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fdc0047d0 0x7f4fdc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4ff8d01640 1 -- 172.21.15.88:0/1779685774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ff40ad870 msgr2=0x7f4fdc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:34.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4ff8d01640 1 --2- 172.21.15.88:0/1779685774 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4ff40ad870 0x7f4fdc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4ff8d01640 1 -- 172.21.15.88:0/1779685774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fdc0047d0 msgr2=0x7f4fdc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:34.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4ff8d01640 1 --2- 172.21.15.88:0/1779685774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4fdc0047d0 0x7f4fdc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.167+0000 7f4ff8d01640 1 -- 172.21.15.88:0/1779685774 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc003530 con 0x7f4fdc003980 2023-10-16T20:58:34.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.171+0000 7f4ff8d01640 1 --2- 172.21.15.88:0/1779685774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fdc003980 0x7f4fdc003da0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4ff000c210 tx=0x7f4ff001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc35f46343e2241 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:34.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.171+0000 7f4feb7fe640 1 -- 172.21.15.88:0/1779685774 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff00058e0 con 0x7f4fdc003980 2023-10-16T20:58:34.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.171+0000 7f4feb7fe640 1 -- 172.21.15.88:0/1779685774 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff0005a40 con 0x7f4fdc003980 2023-10-16T20:58:34.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.171+0000 7f4feb7fe640 1 -- 172.21.15.88:0/1779685774 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0005df0 con 0x7f4fdc003980 2023-10-16T20:58:34.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.171+0000 7f4ffaf65640 1 -- 172.21.15.88:0/1779685774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fdc003980 msgr2=0x7f4fdc003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.171+0000 7f4ffaf65640 1 --2- 172.21.15.88:0/1779685774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fdc003980 0x7f4fdc003da0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4ff000c210 tx=0x7f4ff001cc00 comp rx=0 tx=0).stop 2023-10-16T20:58:34.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.171+0000 7f4ff9502640 1 -- 172.21.15.88:0/1779685774 reap_dead start 2023-10-16T20:58:34.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.171+0000 7f4ffaf65640 1 -- 172.21.15.88:0/1779685774 shutdown_connections 2023-10-16T20:58:34.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.171+0000 7f4ffaf65640 1 -- 172.21.15.88:0/1779685774 >> 172.21.15.88:0/1779685774 conn(0x7f4ff4069e90 msgr2=0x7f4ff406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:34.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.171+0000 7f4ffaf65640 1 -- 172.21.15.88:0/1779685774 shutdown_connections 2023-10-16T20:58:34.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.171+0000 7f4ffaf65640 1 -- 172.21.15.88:0/1779685774 wait complete. 2023-10-16T20:58:34.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4ffaf65640 1 Processor -- start 2023-10-16T20:58:34.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4ffaf65640 1 -- start start 2023-10-16T20:58:34.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4ffaf65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fdc003980 0x7f4ff413e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4ffaf65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ff415d480 0x7f4ff41f0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4ffaf65640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ff413ebb0 0x7f4ff41f5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4ffaf65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff40c5560 con 0x7f4ff415d480 2023-10-16T20:58:34.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4ffaf65640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff41f5580 con 0x7f4ff413ebb0 2023-10-16T20:58:34.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4ffaf65640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff4156f50 con 0x7f4fdc003980 2023-10-16T20:58:34.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4febfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ff415d480 0x7f4ff41f0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4febfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ff415d480 0x7f4ff41f0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34280/0 (socket says 172.21.15.88:34280) 2023-10-16T20:58:34.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4febfff640 1 -- 172.21.15.88:0/2867121000 learned_addr learned my addr 172.21.15.88:0/2867121000 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:34.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4ff9502640 1 --2- 172.21.15.88:0/2867121000 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ff413ebb0 0x7f4ff41f5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4ff8d01640 1 --2- 172.21.15.88:0/2867121000 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fdc003980 0x7f4ff413e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4febfff640 1 -- 172.21.15.88:0/2867121000 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fdc003980 msgr2=0x7f4ff413e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4febfff640 1 --2- 172.21.15.88:0/2867121000 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fdc003980 0x7f4ff413e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4febfff640 1 -- 172.21.15.88:0/2867121000 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ff413ebb0 msgr2=0x7f4ff41f5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4febfff640 1 --2- 172.21.15.88:0/2867121000 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ff413ebb0 0x7f4ff41f5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4febfff640 1 -- 172.21.15.88:0/2867121000 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff41f4220 con 0x7f4ff415d480 2023-10-16T20:58:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4ff8d01640 1 --2- 172.21.15.88:0/2867121000 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fdc003980 0x7f4ff413e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:58:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4febfff640 1 --2- 172.21.15.88:0/2867121000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ff415d480 0x7f4ff41f0190 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40119d0 tx=0x7f4fe4011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4fe97fa640 1 -- 172.21.15.88:0/2867121000 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe400faf0 con 0x7f4ff415d480 2023-10-16T20:58:34.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4fe97fa640 1 -- 172.21.15.88:0/2867121000 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe400fc50 con 0x7f4ff415d480 2023-10-16T20:58:34.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4fe97fa640 1 -- 172.21.15.88:0/2867121000 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4014440 con 0x7f4ff415d480 2023-10-16T20:58:34.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4ffaf65640 1 -- 172.21.15.88:0/2867121000 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff41f44e0 con 0x7f4ff415d480 2023-10-16T20:58:34.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4ffaf65640 1 -- 172.21.15.88:0/2867121000 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff41f4aa0 con 0x7f4ff415d480 2023-10-16T20:58:34.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.175+0000 7f4ffaf65640 1 -- 172.21.15.88:0/2867121000 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff415d8f0 con 0x7f4ff415d480 2023-10-16T20:58:34.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.179+0000 7f4fe97fa640 1 -- 172.21.15.88:0/2867121000 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4fe40145a0 con 0x7f4ff415d480 2023-10-16T20:58:34.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.179+0000 7f4fe97fa640 1 --2- 172.21.15.88:0/2867121000 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4fe00664d0 0x7f4fe0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.179+0000 7f4fe97fa640 1 -- 172.21.15.88:0/2867121000 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f4fe4084cd0 con 0x7f4ff415d480 2023-10-16T20:58:34.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.179+0000 7f4ff8d01640 1 --2- 172.21.15.88:0/2867121000 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4fe00664d0 0x7f4fe0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.179+0000 7f4ff8d01640 1 --2- 172.21.15.88:0/2867121000 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4fe00664d0 0x7f4fe0068990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00aa000 tx=0x7f4ff0001a10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.183+0000 7f4fe97fa640 1 -- 172.21.15.88:0/2867121000 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4fe4051a60 con 0x7f4ff415d480 2023-10-16T20:58:34.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.331+0000 7f4ffaf65640 1 -- 172.21.15.88:0/2867121000 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4ff4157140 con 0x7f4ff415d480 2023-10-16T20:58:34.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.331+0000 7f4fe97fa640 1 -- 172.21.15.88:0/2867121000 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v67) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4fe4022020 con 0x7f4ff415d480 2023-10-16T20:58:34.332 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2023-10-16T20:58:34.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.331+0000 7f4ffaf65640 1 -- 172.21.15.88:0/2867121000 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4fe00664d0 msgr2=0x7f4fe0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.331+0000 7f4ffaf65640 1 --2- 172.21.15.88:0/2867121000 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4fe00664d0 0x7f4fe0068990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00aa000 tx=0x7f4ff0001a10 comp rx=0 tx=0).stop 2023-10-16T20:58:34.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.331+0000 7f4ffaf65640 1 -- 172.21.15.88:0/2867121000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ff415d480 msgr2=0x7f4ff41f0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.331+0000 7f4ffaf65640 1 --2- 172.21.15.88:0/2867121000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ff415d480 0x7f4ff41f0190 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40119d0 tx=0x7f4fe4011e90 comp rx=0 tx=0).stop 2023-10-16T20:58:34.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.331+0000 7f4ffaf65640 1 -- 172.21.15.88:0/2867121000 shutdown_connections 2023-10-16T20:58:34.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.331+0000 7f4ffaf65640 1 --2- 172.21.15.88:0/2867121000 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4fe00664d0 0x7f4fe0068990 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.331+0000 7f4ffaf65640 1 --2- 172.21.15.88:0/2867121000 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4ff413ebb0 0x7f4ff41f5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.331+0000 7f4ffaf65640 1 --2- 172.21.15.88:0/2867121000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ff415d480 0x7f4ff41f0190 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.331+0000 7f4ffaf65640 1 --2- 172.21.15.88:0/2867121000 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4fdc003980 0x7f4ff413e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.331+0000 7f4ffaf65640 1 -- 172.21.15.88:0/2867121000 >> 172.21.15.88:0/2867121000 conn(0x7f4ff4069e90 msgr2=0x7f4ff40c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:34.336 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.335+0000 7f4ffaf65640 1 -- 172.21.15.88:0/2867121000 shutdown_connections 2023-10-16T20:58:34.337 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.335+0000 7f4ffaf65640 1 -- 172.21.15.88:0/2867121000 wait complete. 2023-10-16T20:58:34.355 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T20:58:34.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.495+0000 7f30c51a1640 1 Processor -- start 2023-10-16T20:58:34.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.495+0000 7f30c51a1640 1 -- start start 2023-10-16T20:58:34.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.495+0000 7f30c51a1640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f30c00c4640 0x7f30c01510d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.495+0000 7f30c51a1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c0069e90 con 0x7f30c00c4260 2023-10-16T20:58:34.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.495+0000 7f30c51a1640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c006aea0 con 0x7f30c0151610 2023-10-16T20:58:34.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.495+0000 7f30c51a1640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c00716a0 con 0x7f30c00c4640 2023-10-16T20:58:34.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.495+0000 7f30bed9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f30c00c4260 0x7f30c00c5520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35080/0 (socket says 172.21.15.88:35080) 2023-10-16T20:58:34.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.495+0000 7f30bed9d640 1 -- 172.21.15.88:0/2863975723 learned_addr learned my addr 172.21.15.88:0/2863975723 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:34.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.495+0000 7f30be59c640 1 -- 172.21.15.88:0/2863975723 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725808014 0 0) 0x7f30c0069e90 con 0x7f30c00c4260 2023-10-16T20:58:34.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.495+0000 7f30b7fff640 1 --2- 172.21.15.88:0/2863975723 >> v2:172.21.15.191:3300/0 conn(0x7f30c00c4640 0x7f30c01510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.495+0000 7f30be59c640 1 -- 172.21.15.88:0/2863975723 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30ac003530 con 0x7f30c00c4260 2023-10-16T20:58:34.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30be59c640 1 -- 172.21.15.88:0/2863975723 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3261520586 0 0) 0x7f30ac003530 con 0x7f30c00c4260 2023-10-16T20:58:34.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30be59c640 1 -- 172.21.15.88:0/2863975723 >> v2:172.21.15.191:3300/0 conn(0x7f30c00c4640 msgr2=0x7f30c01510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:34.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30be59c640 1 --2- 172.21.15.88:0/2863975723 >> v2:172.21.15.191:3300/0 conn(0x7f30c00c4640 0x7f30c01510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30be59c640 1 -- 172.21.15.88:0/2863975723 >> v1:172.21.15.191:6789/0 conn(0x7f30c0151610 legacy=0x7f30c01573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30be59c640 1 -- 172.21.15.88:0/2863975723 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c01581e0 con 0x7f30c00c4260 2023-10-16T20:58:34.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30be59c640 1 -- 172.21.15.88:0/2863975723 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f30b0001680 con 0x7f30c00c4260 2023-10-16T20:58:34.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30be59c640 1 -- 172.21.15.88:0/2863975723 >> v1:172.21.15.88:6789/0 conn(0x7f30c00c4260 legacy=0x7f30c00c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30be59c640 1 --2- 172.21.15.88:0/2863975723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30ac003980 0x7f30ac003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30be59c640 1 --2- 172.21.15.88:0/2863975723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30ac0047d0 0x7f30ac004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30be59c640 1 --2- 172.21.15.88:0/2863975723 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30ac0052b0 0x7f30ac00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30be59c640 1 -- 172.21.15.88:0/2863975723 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac00c150 con 0x7f30ac0047d0 2023-10-16T20:58:34.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30be59c640 1 -- 172.21.15.88:0/2863975723 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac00c2c0 con 0x7f30ac0052b0 2023-10-16T20:58:34.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30be59c640 1 -- 172.21.15.88:0/2863975723 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac00c480 con 0x7f30ac003980 2023-10-16T20:58:34.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30b7fff640 1 --2- 172.21.15.88:0/2863975723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30ac0047d0 0x7f30ac004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30bed9d640 1 --2- 172.21.15.88:0/2863975723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30ac003980 0x7f30ac003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30bf59e640 1 --2- 172.21.15.88:0/2863975723 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30ac0052b0 0x7f30ac00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30b7fff640 1 -- 172.21.15.88:0/2863975723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30ac003980 msgr2=0x7f30ac003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:34.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30b7fff640 1 --2- 172.21.15.88:0/2863975723 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30ac003980 0x7f30ac003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30b7fff640 1 -- 172.21.15.88:0/2863975723 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30ac0052b0 msgr2=0x7f30ac00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:34.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30b7fff640 1 --2- 172.21.15.88:0/2863975723 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30ac0052b0 0x7f30ac00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30b7fff640 1 -- 172.21.15.88:0/2863975723 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac003530 con 0x7f30ac0047d0 2023-10-16T20:58:34.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30bf59e640 1 -- 172.21.15.88:0/2863975723 reap_dead start 2023-10-16T20:58:34.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30b7fff640 1 --2- 172.21.15.88:0/2863975723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30ac0047d0 0x7f30ac004c50 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f30a8009ab0 tx=0x7f30a801ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e89f3934b8740e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:34.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30be59c640 1 -- 172.21.15.88:0/2863975723 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c00c5d30 con 0x7f30ac0047d0 2023-10-16T20:58:34.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30be59c640 1 -- 172.21.15.88:0/2863975723 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a80a64f0 con 0x7f30ac0047d0 2023-10-16T20:58:34.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.499+0000 7f30be59c640 1 -- 172.21.15.88:0/2863975723 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a80a68a0 con 0x7f30ac0047d0 2023-10-16T20:58:34.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30c51a1640 1 -- 172.21.15.88:0/2863975723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30ac0047d0 msgr2=0x7f30ac004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30c51a1640 1 --2- 172.21.15.88:0/2863975723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30ac0047d0 0x7f30ac004c50 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f30a8009ab0 tx=0x7f30a801ca00 comp rx=0 tx=0).stop 2023-10-16T20:58:34.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30c51a1640 1 -- 172.21.15.88:0/2863975723 shutdown_connections 2023-10-16T20:58:34.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30c51a1640 1 --2- 172.21.15.88:0/2863975723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30ac0047d0 0x7f30ac004c50 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30c51a1640 1 -- 172.21.15.88:0/2863975723 >> 172.21.15.88:0/2863975723 conn(0x7f30c00ca9f0 msgr2=0x7f30c00cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:34.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30c51a1640 1 -- 172.21.15.88:0/2863975723 shutdown_connections 2023-10-16T20:58:34.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30c51a1640 1 -- 172.21.15.88:0/2863975723 wait complete. 2023-10-16T20:58:34.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30c51a1640 1 Processor -- start 2023-10-16T20:58:34.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30c51a1640 1 -- start start 2023-10-16T20:58:34.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30c51a1640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30ac0047d0 0x7f30c01562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30c51a1640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30c01573c0 0x7f30c014b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30c51a1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30c014bf30 0x7f30c013d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30c51a1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c014c3b0 con 0x7f30c014bf30 2023-10-16T20:58:34.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30c51a1640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c0157f20 con 0x7f30ac0047d0 2023-10-16T20:58:34.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30c51a1640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c0156c40 con 0x7f30c01573c0 2023-10-16T20:58:34.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30bf59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30c014bf30 0x7f30c013d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30bf59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30c014bf30 0x7f30c013d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34306/0 (socket says 172.21.15.88:34306) 2023-10-16T20:58:34.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30bf59e640 1 -- 172.21.15.88:0/768141422 learned_addr learned my addr 172.21.15.88:0/768141422 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:34.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30bf59e640 1 -- 172.21.15.88:0/768141422 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30c01573c0 msgr2=0x7f30c014b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30bf59e640 1 --2- 172.21.15.88:0/768141422 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30c01573c0 0x7f30c014b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30bf59e640 1 -- 172.21.15.88:0/768141422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30ac0047d0 msgr2=0x7f30c01562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30bf59e640 1 --2- 172.21.15.88:0/768141422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30ac0047d0 0x7f30c01562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.503+0000 7f30bf59e640 1 -- 172.21.15.88:0/768141422 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c0157c40 con 0x7f30c014bf30 2023-10-16T20:58:34.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.507+0000 7f30bf59e640 1 --2- 172.21.15.88:0/768141422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30c014bf30 0x7f30c013d2b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f30c0071ca0 tx=0x7f30b800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:34.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.507+0000 7f30b77fe640 1 -- 172.21.15.88:0/768141422 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8006b60 con 0x7f30c014bf30 2023-10-16T20:58:34.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.507+0000 7f30b77fe640 1 -- 172.21.15.88:0/768141422 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b8006ce0 con 0x7f30c014bf30 2023-10-16T20:58:34.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.507+0000 7f30c51a1640 1 -- 172.21.15.88:0/768141422 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c0156ee0 con 0x7f30c014bf30 2023-10-16T20:58:34.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.507+0000 7f30b77fe640 1 -- 172.21.15.88:0/768141422 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8003bc0 con 0x7f30c014bf30 2023-10-16T20:58:34.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.507+0000 7f30c51a1640 1 -- 172.21.15.88:0/768141422 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c0144980 con 0x7f30c014bf30 2023-10-16T20:58:34.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.507+0000 7f30c51a1640 1 -- 172.21.15.88:0/768141422 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c0070ee0 con 0x7f30c014bf30 2023-10-16T20:58:34.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.507+0000 7f30b77fe640 1 -- 172.21.15.88:0/768141422 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f30b8003d20 con 0x7f30c014bf30 2023-10-16T20:58:34.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.507+0000 7f30b77fe640 1 --2- 172.21.15.88:0/768141422 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f30a00664a0 0x7f30a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.507+0000 7f30b77fe640 1 -- 172.21.15.88:0/768141422 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f30b8083f50 con 0x7f30c014bf30 2023-10-16T20:58:34.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.507+0000 7f30bed9d640 1 --2- 172.21.15.88:0/768141422 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f30a00664a0 0x7f30a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.511+0000 7f30bed9d640 1 --2- 172.21.15.88:0/768141422 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f30a00664a0 0x7f30a0068960 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f30c0069ae0 tx=0x7f30b0001e40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:34.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.511+0000 7f30b77fe640 1 -- 172.21.15.88:0/768141422 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f30b8050b80 con 0x7f30c014bf30 2023-10-16T20:58:34.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.663+0000 7f30c51a1640 1 -- 172.21.15.88:0/768141422 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f30c0070c00 con 0x7f30c014bf30 2023-10-16T20:58:34.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.663+0000 7f30b77fe640 1 -- 172.21.15.88:0/768141422 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v67) v1 ==== 103+0+11 (secure 0 0 0) 0x7f30b8019020 con 0x7f30c014bf30 2023-10-16T20:58:34.663 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:58:34.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.663+0000 7f30c51a1640 1 -- 172.21.15.88:0/768141422 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f30a00664a0 msgr2=0x7f30a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.663+0000 7f30c51a1640 1 --2- 172.21.15.88:0/768141422 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f30a00664a0 0x7f30a0068960 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f30c0069ae0 tx=0x7f30b0001e40 comp rx=0 tx=0).stop 2023-10-16T20:58:34.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.663+0000 7f30c51a1640 1 -- 172.21.15.88:0/768141422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30c014bf30 msgr2=0x7f30c013d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.663+0000 7f30c51a1640 1 --2- 172.21.15.88:0/768141422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30c014bf30 0x7f30c013d2b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f30c0071ca0 tx=0x7f30b800ec10 comp rx=0 tx=0).stop 2023-10-16T20:58:34.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.663+0000 7f30c51a1640 1 -- 172.21.15.88:0/768141422 shutdown_connections 2023-10-16T20:58:34.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.663+0000 7f30c51a1640 1 --2- 172.21.15.88:0/768141422 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f30a00664a0 0x7f30a0068960 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.663+0000 7f30c51a1640 1 --2- 172.21.15.88:0/768141422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30c014bf30 0x7f30c013d2b0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.663+0000 7f30c51a1640 1 --2- 172.21.15.88:0/768141422 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f30c01573c0 0x7f30c014b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.663+0000 7f30c51a1640 1 --2- 172.21.15.88:0/768141422 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f30ac0047d0 0x7f30c01562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.663+0000 7f30c51a1640 1 -- 172.21.15.88:0/768141422 >> 172.21.15.88:0/768141422 conn(0x7f30c00ca9f0 msgr2=0x7f30c01466e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:34.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.663+0000 7f30c51a1640 1 -- 172.21.15.88:0/768141422 shutdown_connections 2023-10-16T20:58:34.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.663+0000 7f30c51a1640 1 -- 172.21.15.88:0/768141422 wait complete. 2023-10-16T20:58:34.684 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T20:58:34.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.823+0000 7f5082e09640 1 Processor -- start 2023-10-16T20:58:34.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.823+0000 7f5082e09640 1 -- start start 2023-10-16T20:58:34.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.823+0000 7f5082e09640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f507c0c4640 0x7f507c1510d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.823+0000 7f5082e09640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f507c151610 0x7f507c1573e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.823+0000 7f5082e09640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507c069e90 con 0x7f507c0c4260 2023-10-16T20:58:34.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.823+0000 7f5082e09640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507c0716a0 con 0x7f507c0c4640 2023-10-16T20:58:34.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.823+0000 7f5082e09640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507c06aea0 con 0x7f507c151610 2023-10-16T20:58:34.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.823+0000 7f5073fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f507c0c4640 0x7f507c1510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.827+0000 7f5073fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f507c0c4640 0x7f507c1510d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34308/0 (socket says 172.21.15.88:34308) 2023-10-16T20:58:34.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.827+0000 7f5073fff640 1 -- 172.21.15.88:0/625238462 learned_addr learned my addr 172.21.15.88:0/625238462 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:34.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.827+0000 7f50813a6640 1 --2- 172.21.15.88:0/625238462 >> v2:172.21.15.191:3300/0 conn(0x7f507c151610 0x7f507c1573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.827+0000 7f5073fff640 1 -- 172.21.15.88:0/625238462 >> v2:172.21.15.191:3300/0 conn(0x7f507c151610 msgr2=0x7f507c1573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:34.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.827+0000 7f5073fff640 1 --2- 172.21.15.88:0/625238462 >> v2:172.21.15.191:3300/0 conn(0x7f507c151610 0x7f507c1573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.827+0000 7f5073fff640 1 -- 172.21.15.88:0/625238462 >> v1:172.21.15.191:6789/0 conn(0x7f507c0c4260 legacy=0x7f507c0c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.827+0000 7f5073fff640 1 -- 172.21.15.88:0/625238462 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c1581e0 con 0x7f507c0c4640 2023-10-16T20:58:34.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.827+0000 7f5073fff640 1 --2- 172.21.15.88:0/625238462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f507c0c4640 0x7f507c1510d0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f507c0c1cc0 tx=0x7f506c01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf2eb3dc29bd9f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:34.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.827+0000 7f50737fe640 1 -- 172.21.15.88:0/625238462 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c0ac410 con 0x7f507c0c4640 2023-10-16T20:58:34.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.827+0000 7f50737fe640 1 -- 172.21.15.88:0/625238462 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506c0ac570 con 0x7f507c0c4640 2023-10-16T20:58:34.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.827+0000 7f50737fe640 1 -- 172.21.15.88:0/625238462 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c0ac900 con 0x7f507c0c4640 2023-10-16T20:58:34.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5082e09640 1 -- 172.21.15.88:0/625238462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f507c0c4640 msgr2=0x7f507c1510d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5082e09640 1 --2- 172.21.15.88:0/625238462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f507c0c4640 0x7f507c1510d0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f507c0c1cc0 tx=0x7f506c01c3b0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5082e09640 1 -- 172.21.15.88:0/625238462 shutdown_connections 2023-10-16T20:58:34.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5082e09640 1 --2- 172.21.15.88:0/625238462 >> v2:172.21.15.191:3300/0 conn(0x7f507c151610 0x7f507c1573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5082e09640 1 --2- 172.21.15.88:0/625238462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f507c0c4640 0x7f507c1510d0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5082e09640 1 -- 172.21.15.88:0/625238462 >> 172.21.15.88:0/625238462 conn(0x7f507c0ca9f0 msgr2=0x7f507c0cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:34.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5082e09640 1 -- 172.21.15.88:0/625238462 shutdown_connections 2023-10-16T20:58:34.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5082e09640 1 -- 172.21.15.88:0/625238462 wait complete. 2023-10-16T20:58:34.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5082e09640 1 Processor -- start 2023-10-16T20:58:34.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5082e09640 1 -- start start 2023-10-16T20:58:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5082e09640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f507c0c4260 0x7f507c145c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5082e09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f507c0c4640 0x7f507c0c2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5082e09640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f507c151610 0x7f507c1611c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5082e09640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c1f5dd0 con 0x7f507c0c4640 2023-10-16T20:58:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5082e09640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c0c2c40 con 0x7f507c0c4260 2023-10-16T20:58:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5082e09640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507c1f4ee0 con 0x7f507c151610 2023-10-16T20:58:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5073fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f507c0c4640 0x7f507c0c2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5073fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f507c0c4640 0x7f507c0c2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34310/0 (socket says 172.21.15.88:34310) 2023-10-16T20:58:34.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5073fff640 1 -- 172.21.15.88:0/124637964 learned_addr learned my addr 172.21.15.88:0/124637964 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:34.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5080ba5640 1 --2- 172.21.15.88:0/124637964 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f507c0c4260 0x7f507c145c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.839 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f50813a6640 1 --2- 172.21.15.88:0/124637964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f507c151610 0x7f507c1611c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5073fff640 1 -- 172.21.15.88:0/124637964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f507c151610 msgr2=0x7f507c1611c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5073fff640 1 --2- 172.21.15.88:0/124637964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f507c151610 0x7f507c1611c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5073fff640 1 -- 172.21.15.88:0/124637964 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f507c0c4260 msgr2=0x7f507c145c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:34.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5073fff640 1 --2- 172.21.15.88:0/124637964 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f507c0c4260 0x7f507c145c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.831+0000 7f5073fff640 1 -- 172.21.15.88:0/124637964 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c0094d0 con 0x7f507c0c4640 2023-10-16T20:58:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.835+0000 7f5073fff640 1 --2- 172.21.15.88:0/124637964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f507c0c4640 0x7f507c0c2700 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f506c018fc0 tx=0x7f506c019150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.835+0000 7f50717fa640 1 -- 172.21.15.88:0/124637964 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c0ac410 con 0x7f507c0c4640 2023-10-16T20:58:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.835+0000 7f50717fa640 1 -- 172.21.15.88:0/124637964 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506c0b65f0 con 0x7f507c0c4640 2023-10-16T20:58:34.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.835+0000 7f50717fa640 1 -- 172.21.15.88:0/124637964 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c0b69c0 con 0x7f507c0c4640 2023-10-16T20:58:34.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.835+0000 7f5082e09640 1 -- 172.21.15.88:0/124637964 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507c1f51c0 con 0x7f507c0c4640 2023-10-16T20:58:34.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.835+0000 7f5082e09640 1 -- 172.21.15.88:0/124637964 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507c1f5780 con 0x7f507c0c4640 2023-10-16T20:58:34.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.835+0000 7f5082e09640 1 -- 172.21.15.88:0/124637964 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507c074700 con 0x7f507c0c4640 2023-10-16T20:58:34.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.835+0000 7f50717fa640 1 -- 172.21.15.88:0/124637964 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f506c0b6b20 con 0x7f507c0c4640 2023-10-16T20:58:34.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.835+0000 7f50717fa640 1 --2- 172.21.15.88:0/124637964 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f50680664a0 0x7f5068068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:34.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.835+0000 7f50717fa640 1 -- 172.21.15.88:0/124637964 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f506c11ef90 con 0x7f507c0c4640 2023-10-16T20:58:34.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.835+0000 7f5080ba5640 1 --2- 172.21.15.88:0/124637964 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f50680664a0 0x7f5068068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:34.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.839+0000 7f5080ba5640 1 --2- 172.21.15.88:0/124637964 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f50680664a0 0x7f5068068960 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5078007fd0 tx=0x7f5078009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:34.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.839+0000 7f50717fa640 1 -- 172.21.15.88:0/124637964 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f506c150090 con 0x7f507c0c4640 2023-10-16T20:58:35.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:34.999+0000 7f5082e09640 1 -- 172.21.15.88:0/124637964 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f507c0c8940 con 0x7f507c0c4640 2023-10-16T20:58:35.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.003+0000 7f50717fa640 1 -- 172.21.15.88:0/124637964 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 20 entries v67) v1 ==== 81+0+1228 (secure 0 0 0) 0x7f506c0ebea0 con 0x7f507c0c4640 2023-10-16T20:58:35.002 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6839/2172973775 2023-10-17T20:56:40.633674+0000 2023-10-16T20:58:35.003 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6833/3633326602 2023-10-17T20:55:50.604940+0000 2023-10-16T20:58:35.003 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6836/1120825335 2023-10-17T20:42:10.335721+0000 2023-10-16T20:58:35.003 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6832/3633326602 2023-10-17T20:55:50.604940+0000 2023-10-16T20:58:35.004 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/518281867 2023-10-17T20:44:25.369398+0000 2023-10-16T20:58:35.004 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6837/2468647900 2023-10-17T20:53:25.553637+0000 2023-10-16T20:58:35.004 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1987579860 2023-10-17T20:43:25.353447+0000 2023-10-16T20:58:35.004 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6835/3119045295 2023-10-17T20:58:00.653202+0000 2023-10-16T20:58:35.005 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/321173984 2023-10-17T20:46:15.444657+0000 2023-10-16T20:58:35.005 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6837/1120825335 2023-10-17T20:42:10.335721+0000 2023-10-16T20:58:35.005 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6832/2446850309 2023-10-17T20:48:43.641901+0000 2023-10-16T20:58:35.005 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2172973775 2023-10-17T20:56:40.633674+0000 2023-10-16T20:58:35.006 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6839/1987579860 2023-10-17T20:43:25.353447+0000 2023-10-16T20:58:35.006 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6834/3119045295 2023-10-17T20:58:00.653202+0000 2023-10-16T20:58:35.006 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/518281867 2023-10-17T20:44:25.369398+0000 2023-10-16T20:58:35.007 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6838/321173984 2023-10-17T20:46:15.444657+0000 2023-10-16T20:58:35.007 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6834/3059699574 2023-10-17T20:51:20.522638+0000 2023-10-16T20:58:35.007 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6833/2446850309 2023-10-17T20:48:43.641901+0000 2023-10-16T20:58:35.007 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6835/3059699574 2023-10-17T20:51:20.522638+0000 2023-10-16T20:58:35.008 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6836/2468647900 2023-10-17T20:53:25.553637+0000 2023-10-16T20:58:35.009 INFO:teuthology.orchestra.run.smithi088.stderr:listed 20 entries 2023-10-16T20:58:35.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.003+0000 7f5082e09640 1 -- 172.21.15.88:0/124637964 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f50680664a0 msgr2=0x7f5068068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.003+0000 7f5082e09640 1 --2- 172.21.15.88:0/124637964 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f50680664a0 0x7f5068068960 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5078007fd0 tx=0x7f5078009040 comp rx=0 tx=0).stop 2023-10-16T20:58:35.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.003+0000 7f5082e09640 1 -- 172.21.15.88:0/124637964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f507c0c4640 msgr2=0x7f507c0c2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.003+0000 7f5082e09640 1 --2- 172.21.15.88:0/124637964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f507c0c4640 0x7f507c0c2700 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f506c018fc0 tx=0x7f506c019150 comp rx=0 tx=0).stop 2023-10-16T20:58:35.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.003+0000 7f5082e09640 1 -- 172.21.15.88:0/124637964 shutdown_connections 2023-10-16T20:58:35.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.003+0000 7f5082e09640 1 --2- 172.21.15.88:0/124637964 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f50680664a0 0x7f5068068960 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.003+0000 7f5082e09640 1 --2- 172.21.15.88:0/124637964 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f507c151610 0x7f507c1611c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.003+0000 7f5082e09640 1 --2- 172.21.15.88:0/124637964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f507c0c4640 0x7f507c0c2700 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.003+0000 7f5082e09640 1 --2- 172.21.15.88:0/124637964 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f507c0c4260 0x7f507c145c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.003+0000 7f5082e09640 1 -- 172.21.15.88:0/124637964 >> 172.21.15.88:0/124637964 conn(0x7f507c0ca9f0 msgr2=0x7f507c070c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:35.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.003+0000 7f5082e09640 1 -- 172.21.15.88:0/124637964 shutdown_connections 2023-10-16T20:58:35.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.003+0000 7f5082e09640 1 -- 172.21.15.88:0/124637964 wait complete. 2023-10-16T20:58:35.023 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-10-16T20:58:35.023 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-16T20:58:35.033 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:58:35.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.211+0000 7f1c0593f640 1 Processor -- start 2023-10-16T20:58:35.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.211+0000 7f1c0593f640 1 -- start start 2023-10-16T20:58:35.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.211+0000 7f1c0593f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1c000cead0 0x7f1c000ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:35.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.211+0000 7f1c0593f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c000cae10 con 0x7f1c001537c0 2023-10-16T20:58:35.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.211+0000 7f1c0593f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c000cac30 con 0x7f1c000ca2f0 2023-10-16T20:58:35.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.211+0000 7f1c0593f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c000cf420 con 0x7f1c000cead0 2023-10-16T20:58:35.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.211+0000 7f1c0493d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1c001537c0 0x7f1c0015c870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35086/0 (socket says 172.21.15.88:35086) 2023-10-16T20:58:35.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.211+0000 7f1c0493d640 1 -- 172.21.15.88:0/3596018778 learned_addr learned my addr 172.21.15.88:0/3596018778 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:35.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.211+0000 7f1c0513e640 1 --2- 172.21.15.88:0/3596018778 >> v2:172.21.15.88:3300/0 conn(0x7f1c000cead0 0x7f1c000ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:35.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.211+0000 7f1bff7fe640 1 -- 172.21.15.88:0/3596018778 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082788230 0 0) 0x7f1c000cae10 con 0x7f1c001537c0 2023-10-16T20:58:35.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.211+0000 7f1bff7fe640 1 -- 172.21.15.88:0/3596018778 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf0003530 con 0x7f1c001537c0 2023-10-16T20:58:35.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.211+0000 7f1c0513e640 1 -- 172.21.15.88:0/3596018778 >> v1:172.21.15.191:6789/0 conn(0x7f1c000ca2f0 legacy=0x7f1c000c4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:35.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.211+0000 7f1c0513e640 1 -- 172.21.15.88:0/3596018778 >> v1:172.21.15.88:6789/0 conn(0x7f1c001537c0 legacy=0x7f1c0015c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.211+0000 7f1c0513e640 1 -- 172.21.15.88:0/3596018778 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0015d2b0 con 0x7f1c000cead0 2023-10-16T20:58:35.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.215+0000 7f1c0513e640 1 --2- 172.21.15.88:0/3596018778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c000cead0 0x7f1c000ceee0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8009d90 tx=0x7f1bf801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e887313a655e759 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:35.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.215+0000 7f1bff7fe640 1 -- 172.21.15.88:0/3596018778 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf80a59a0 con 0x7f1c000cead0 2023-10-16T20:58:35.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.215+0000 7f1bff7fe640 1 -- 172.21.15.88:0/3596018778 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf80a5b00 con 0x7f1c000cead0 2023-10-16T20:58:35.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.215+0000 7f1bff7fe640 1 -- 172.21.15.88:0/3596018778 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf80a5e90 con 0x7f1c000cead0 2023-10-16T20:58:35.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.215+0000 7f1c0593f640 1 -- 172.21.15.88:0/3596018778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c000cead0 msgr2=0x7f1c000ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.215+0000 7f1c0593f640 1 --2- 172.21.15.88:0/3596018778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c000cead0 0x7f1c000ceee0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8009d90 tx=0x7f1bf801c8e0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.215+0000 7f1c0593f640 1 -- 172.21.15.88:0/3596018778 shutdown_connections 2023-10-16T20:58:35.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.215+0000 7f1c0593f640 1 --2- 172.21.15.88:0/3596018778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c000cead0 0x7f1c000ceee0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.215+0000 7f1c0593f640 1 -- 172.21.15.88:0/3596018778 >> 172.21.15.88:0/3596018778 conn(0x7f1c0006c170 msgr2=0x7f1c000715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:35.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.215+0000 7f1c0593f640 1 -- 172.21.15.88:0/3596018778 shutdown_connections 2023-10-16T20:58:35.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.215+0000 7f1c0593f640 1 -- 172.21.15.88:0/3596018778 wait complete. 2023-10-16T20:58:35.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.215+0000 7f1c0593f640 1 Processor -- start 2023-10-16T20:58:35.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.215+0000 7f1c0593f640 1 -- start start 2023-10-16T20:58:35.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1c0593f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c000ca2f0 0x7f1c001f5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:35.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1c0593f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c000cead0 0x7f1c0014cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:35.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1c0593f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c001537c0 0x7f1c00147620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:35.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1c0593f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c00148bd0 con 0x7f1c000cead0 2023-10-16T20:58:35.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1c0593f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c00149780 con 0x7f1c001537c0 2023-10-16T20:58:35.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1c0593f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c001f5a60 con 0x7f1c000ca2f0 2023-10-16T20:58:35.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1bfffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c000cead0 0x7f1c0014cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:35.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1bfffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c000cead0 0x7f1c0014cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34330/0 (socket says 172.21.15.88:34330) 2023-10-16T20:58:35.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1bfffff640 1 -- 172.21.15.88:0/3943741485 learned_addr learned my addr 172.21.15.88:0/3943741485 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:35.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1bfffff640 1 -- 172.21.15.88:0/3943741485 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c000ca2f0 msgr2=0x7f1c001f5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1bfffff640 1 --2- 172.21.15.88:0/3943741485 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c000ca2f0 0x7f1c001f5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1bfffff640 1 -- 172.21.15.88:0/3943741485 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c001537c0 msgr2=0x7f1c00147620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1bfffff640 1 --2- 172.21.15.88:0/3943741485 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c001537c0 0x7f1c00147620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1bfffff640 1 -- 172.21.15.88:0/3943741485 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf8009a70 con 0x7f1c000cead0 2023-10-16T20:58:35.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1bfffff640 1 --2- 172.21.15.88:0/3943741485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c000cead0 0x7f1c0014cd70 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f1bec00c450 tx=0x7f1bec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:35.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1bfd7fa640 1 -- 172.21.15.88:0/3943741485 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec012830 con 0x7f1c000cead0 2023-10-16T20:58:35.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1bfd7fa640 1 -- 172.21.15.88:0/3943741485 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bec0129b0 con 0x7f1c000cead0 2023-10-16T20:58:35.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1bfd7fa640 1 -- 172.21.15.88:0/3943741485 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec012d60 con 0x7f1c000cead0 2023-10-16T20:58:35.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1c0593f640 1 -- 172.21.15.88:0/3943741485 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c001468a0 con 0x7f1c000cead0 2023-10-16T20:58:35.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1c0593f640 1 -- 172.21.15.88:0/3943741485 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c00146b40 con 0x7f1c000cead0 2023-10-16T20:58:35.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.219+0000 7f1c0593f640 1 -- 172.21.15.88:0/3943741485 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0004e9c0 con 0x7f1c000cead0 2023-10-16T20:58:35.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.223+0000 7f1bfd7fa640 1 -- 172.21.15.88:0/3943741485 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1bec01c460 con 0x7f1c000cead0 2023-10-16T20:58:35.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.223+0000 7f1bfd7fa640 1 --2- 172.21.15.88:0/3943741485 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1be40664a0 0x7f1be4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:35.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.223+0000 7f1bfd7fa640 1 -- 172.21.15.88:0/3943741485 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f1bec084460 con 0x7f1c000cead0 2023-10-16T20:58:35.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.223+0000 7f1c0493d640 1 --2- 172.21.15.88:0/3943741485 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1be40664a0 0x7f1be4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:35.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.223+0000 7f1c0493d640 1 --2- 172.21.15.88:0/3943741485 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1be40664a0 0x7f1be4068960 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4007fd0 tx=0x7f1bf4007940 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:35.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.227+0000 7f1bfd7fa640 1 -- 172.21.15.88:0/3943741485 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1bec051120 con 0x7f1c000cead0 2023-10-16T20:58:35.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.379+0000 7f1c0593f640 1 -- 172.21.15.88:0/3943741485 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c000ca0d0 con 0x7f1c000cead0 2023-10-16T20:58:35.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.383+0000 7f1bfd7fa640 1 -- 172.21.15.88:0/3943741485 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+20065 (secure 0 0 0) 0x7f1bec054740 con 0x7f1c000cead0 2023-10-16T20:58:35.383 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:35.383 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":67,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:58:00.653255+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6839/2172973775":"2023-10-17T20:56:40.633674+0000","172.21.15.191:6833/3633326602":"2023-10-17T20:55:50.604940+0000","172.21.15.191:6836/1120825335":"2023-10-17T20:42:10.335721+0000","172.21.15.191:6832/3633326602":"2023-10-17T20:55:50.604940+0000","172.21.15.88:6835/518281867":"2023-10-17T20:44:25.369398+0000","172.21.15.191:6837/2468647900":"2023-10-17T20:53:25.553637+0000","172.21.15.88:6837/1987579860":"2023-10-17T20:43:25.353447+0000","172.21.15.191:6835/3119045295":"2023-10-17T20:58:00.653202+0000","172.21.15.88:6836/321173984":"2023-10-17T20:46:15.444657+0000","172.21.15.191:6837/1120825335":"2023-10-17T20:42:10.335721+0000","172.21.15.191:6832/2446850309":"2023-10-17T20:48:43.641901+0000","172.21.15.88:6837/2172973775":"2023-10-17T20:56:40.633674+0000","172.21.15.88:6839/1987579860":"2023-10-17T20:43:25.353447+0000","172.21.15.191:6834/3119045295":"2023-10-17T20:58:00.653202+0000","172.21.15.88:6834/518281867":"2023-10-17T20:44:25.369398+0000","172.21.15.88:6838/321173984":"2023-10-17T20:46:15.444657+0000","172.21.15.191:6834/3059699574":"2023-10-17T20:51:20.522638+0000","172.21.15.191:6833/2446850309":"2023-10-17T20:48:43.641901+0000","172.21.15.191:6835/3059699574":"2023-10-17T20:51:20.522638+0000","172.21.15.191:6836/2468647900":"2023-10-17T20:53:25.553637+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:58:35.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.383+0000 7f1c0593f640 1 -- 172.21.15.88:0/3943741485 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1be40664a0 msgr2=0x7f1be4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.383+0000 7f1c0593f640 1 --2- 172.21.15.88:0/3943741485 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1be40664a0 0x7f1be4068960 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4007fd0 tx=0x7f1bf4007940 comp rx=0 tx=0).stop 2023-10-16T20:58:35.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.383+0000 7f1c0593f640 1 -- 172.21.15.88:0/3943741485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c000cead0 msgr2=0x7f1c0014cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.383+0000 7f1c0593f640 1 --2- 172.21.15.88:0/3943741485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c000cead0 0x7f1c0014cd70 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f1bec00c450 tx=0x7f1bec00c910 comp rx=0 tx=0).stop 2023-10-16T20:58:35.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.383+0000 7f1c0593f640 1 -- 172.21.15.88:0/3943741485 shutdown_connections 2023-10-16T20:58:35.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.383+0000 7f1c0593f640 1 --2- 172.21.15.88:0/3943741485 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1be40664a0 0x7f1be4068960 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.383+0000 7f1c0593f640 1 --2- 172.21.15.88:0/3943741485 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1c001537c0 0x7f1c00147620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.383+0000 7f1c0593f640 1 --2- 172.21.15.88:0/3943741485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c000cead0 0x7f1c0014cd70 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.383+0000 7f1c0593f640 1 --2- 172.21.15.88:0/3943741485 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1c000ca2f0 0x7f1c001f5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.383+0000 7f1c0593f640 1 -- 172.21.15.88:0/3943741485 >> 172.21.15.88:0/3943741485 conn(0x7f1c0006c170 msgr2=0x7f1c000c4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:35.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.383+0000 7f1c0593f640 1 -- 172.21.15.88:0/3943741485 shutdown_connections 2023-10-16T20:58:35.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.383+0000 7f1c0593f640 1 -- 172.21.15.88:0/3943741485 wait complete. 2023-10-16T20:58:35.404 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-16T20:36:43.646418+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-16T20:36:53.546513+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-16T20:36:54.143703+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 59, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-16T20:36:57.314642+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 59, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-16T20:58:35.405 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-16T20:58:35.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.547+0000 7f0d930a9640 1 Processor -- start 2023-10-16T20:58:35.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.547+0000 7f0d930a9640 1 -- start start 2023-10-16T20:58:35.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.547+0000 7f0d930a9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0d8c0cb5c0 0x7f0d8c071420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:35.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.547+0000 7f0d930a9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c070f40 con 0x7f0d8c0c6ac0 2023-10-16T20:58:35.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.547+0000 7f0d930a9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c06a2d0 con 0x7f0d8c0c66e0 2023-10-16T20:58:35.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.547+0000 7f0d930a9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d8c06a5b0 con 0x7f0d8c0cb5c0 2023-10-16T20:58:35.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.547+0000 7f0d91646640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0d8c0cb5c0 0x7f0d8c071420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:35.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.547+0000 7f0d83fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0d8c0c6ac0 0x7f0d8c150150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35098/0 (socket says 172.21.15.88:35098) 2023-10-16T20:58:35.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.547+0000 7f0d83fff640 1 -- 172.21.15.88:0/3162305052 learned_addr learned my addr 172.21.15.88:0/3162305052 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:35.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.551+0000 7f0d91646640 1 -- 172.21.15.88:0/3162305052 >> v1:172.21.15.191:6789/0 conn(0x7f0d8c0c66e0 legacy=0x7f0d8c15a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:35.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.551+0000 7f0d91646640 1 -- 172.21.15.88:0/3162305052 >> v1:172.21.15.88:6789/0 conn(0x7f0d8c0c6ac0 legacy=0x7f0d8c150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.551+0000 7f0d91646640 1 -- 172.21.15.88:0/3162305052 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8c06a890 con 0x7f0d8c0cb5c0 2023-10-16T20:58:35.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.551+0000 7f0d837fe640 1 -- 172.21.15.88:0/3162305052 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237779240 0 0) 0x7f0d8c070f40 con 0x7f0d8c0c6ac0 2023-10-16T20:58:35.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.551+0000 7f0d91646640 1 --2- 172.21.15.88:0/3162305052 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c0cb5c0 0x7f0d8c071420 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0d84009d90 tx=0x7f0d8401c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d3512ee3a07f477 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:35.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.551+0000 7f0d837fe640 1 -- 172.21.15.88:0/3162305052 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d840a5810 con 0x7f0d8c0cb5c0 2023-10-16T20:58:35.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.551+0000 7f0d837fe640 1 -- 172.21.15.88:0/3162305052 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d840a5970 con 0x7f0d8c0cb5c0 2023-10-16T20:58:35.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.551+0000 7f0d837fe640 1 -- 172.21.15.88:0/3162305052 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d840a5d00 con 0x7f0d8c0cb5c0 2023-10-16T20:58:35.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.551+0000 7f0d930a9640 1 -- 172.21.15.88:0/3162305052 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c0cb5c0 msgr2=0x7f0d8c071420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.551+0000 7f0d930a9640 1 --2- 172.21.15.88:0/3162305052 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c0cb5c0 0x7f0d8c071420 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0d84009d90 tx=0x7f0d8401c880 comp rx=0 tx=0).stop 2023-10-16T20:58:35.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.551+0000 7f0d930a9640 1 -- 172.21.15.88:0/3162305052 shutdown_connections 2023-10-16T20:58:35.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.551+0000 7f0d930a9640 1 --2- 172.21.15.88:0/3162305052 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c0cb5c0 0x7f0d8c071420 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.551+0000 7f0d930a9640 1 -- 172.21.15.88:0/3162305052 >> 172.21.15.88:0/3162305052 conn(0x7f0d8c0ca9f0 msgr2=0x7f0d8c0cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:35.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.551+0000 7f0d930a9640 1 -- 172.21.15.88:0/3162305052 shutdown_connections 2023-10-16T20:58:35.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.551+0000 7f0d930a9640 1 -- 172.21.15.88:0/3162305052 wait complete. 2023-10-16T20:58:35.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d930a9640 1 Processor -- start 2023-10-16T20:58:35.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d930a9640 1 -- start start 2023-10-16T20:58:35.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d930a9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c0c66e0 0x7f0d8c14ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:35.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d930a9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d8c0c6ac0 0x7f0d8c13f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:35.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d930a9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d8c0cb5c0 0x7f0d8c1f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:35.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d930a9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c14f460 con 0x7f0d8c0c66e0 2023-10-16T20:58:35.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d930a9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c165800 con 0x7f0d8c0cb5c0 2023-10-16T20:58:35.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d930a9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c166080 con 0x7f0d8c0c6ac0 2023-10-16T20:58:35.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d90e45640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c0c66e0 0x7f0d8c14ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:35.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d90e45640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c0c66e0 0x7f0d8c14ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34358/0 (socket says 172.21.15.88:34358) 2023-10-16T20:58:35.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d90e45640 1 -- 172.21.15.88:0/920465907 learned_addr learned my addr 172.21.15.88:0/920465907 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:35.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d90e45640 1 -- 172.21.15.88:0/920465907 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d8c0c6ac0 msgr2=0x7f0d8c13f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d90e45640 1 --2- 172.21.15.88:0/920465907 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d8c0c6ac0 0x7f0d8c13f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d90e45640 1 -- 172.21.15.88:0/920465907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d8c0cb5c0 msgr2=0x7f0d8c1f4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d90e45640 1 --2- 172.21.15.88:0/920465907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d8c0cb5c0 0x7f0d8c1f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d90e45640 1 -- 172.21.15.88:0/920465907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d84009a70 con 0x7f0d8c0c66e0 2023-10-16T20:58:35.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d90e45640 1 --2- 172.21.15.88:0/920465907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c0c66e0 0x7f0d8c14ef20 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800c480 tx=0x7f0d8800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:35.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.555+0000 7f0d817fa640 1 -- 172.21.15.88:0/920465907 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d88012830 con 0x7f0d8c0c66e0 2023-10-16T20:58:35.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.559+0000 7f0d817fa640 1 -- 172.21.15.88:0/920465907 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d88012990 con 0x7f0d8c0c66e0 2023-10-16T20:58:35.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.559+0000 7f0d817fa640 1 -- 172.21.15.88:0/920465907 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d88012d40 con 0x7f0d8c0c66e0 2023-10-16T20:58:35.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.559+0000 7f0d930a9640 1 -- 172.21.15.88:0/920465907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d8c14f890 con 0x7f0d8c0c66e0 2023-10-16T20:58:35.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.559+0000 7f0d930a9640 1 -- 172.21.15.88:0/920465907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d8c165a30 con 0x7f0d8c0c66e0 2023-10-16T20:58:35.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.559+0000 7f0d930a9640 1 -- 172.21.15.88:0/920465907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8c0c9fe0 con 0x7f0d8c0c66e0 2023-10-16T20:58:35.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.559+0000 7f0d817fa640 1 -- 172.21.15.88:0/920465907 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0d8801c460 con 0x7f0d8c0c66e0 2023-10-16T20:58:35.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.559+0000 7f0d817fa640 1 --2- 172.21.15.88:0/920465907 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0d780664a0 0x7f0d78068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:35.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.559+0000 7f0d817fa640 1 -- 172.21.15.88:0/920465907 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f0d88084450 con 0x7f0d8c0c66e0 2023-10-16T20:58:35.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.559+0000 7f0d83fff640 1 --2- 172.21.15.88:0/920465907 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0d780664a0 0x7f0d78068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:35.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.559+0000 7f0d83fff640 1 --2- 172.21.15.88:0/920465907 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0d780664a0 0x7f0d78068960 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0011f0 tx=0x7f0d7c007930 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:35.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.563+0000 7f0d817fa640 1 -- 172.21.15.88:0/920465907 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0d880511b0 con 0x7f0d8c0c66e0 2023-10-16T20:58:35.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.715+0000 7f0d930a9640 1 -- 172.21.15.88:0/920465907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0d8c14a470 con 0x7f0d8c0c66e0 2023-10-16T20:58:35.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.715+0000 7f0d817fa640 1 -- 172.21.15.88:0/920465907 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0d880547d0 con 0x7f0d8c0c66e0 2023-10-16T20:58:35.718 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-10-16T20:58:35.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.719+0000 7f0d930a9640 1 -- 172.21.15.88:0/920465907 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0d780664a0 msgr2=0x7f0d78068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.719+0000 7f0d930a9640 1 --2- 172.21.15.88:0/920465907 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0d780664a0 0x7f0d78068960 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0011f0 tx=0x7f0d7c007930 comp rx=0 tx=0).stop 2023-10-16T20:58:35.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.719+0000 7f0d930a9640 1 -- 172.21.15.88:0/920465907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c0c66e0 msgr2=0x7f0d8c14ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.719+0000 7f0d930a9640 1 --2- 172.21.15.88:0/920465907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c0c66e0 0x7f0d8c14ef20 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800c480 tx=0x7f0d8800c940 comp rx=0 tx=0).stop 2023-10-16T20:58:35.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.719+0000 7f0d930a9640 1 -- 172.21.15.88:0/920465907 shutdown_connections 2023-10-16T20:58:35.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.719+0000 7f0d930a9640 1 --2- 172.21.15.88:0/920465907 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f0d780664a0 0x7f0d78068960 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.719+0000 7f0d930a9640 1 --2- 172.21.15.88:0/920465907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d8c0cb5c0 0x7f0d8c1f4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.719+0000 7f0d930a9640 1 --2- 172.21.15.88:0/920465907 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f0d8c0c6ac0 0x7f0d8c13f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.719+0000 7f0d930a9640 1 --2- 172.21.15.88:0/920465907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c0c66e0 0x7f0d8c14ef20 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.719+0000 7f0d930a9640 1 -- 172.21.15.88:0/920465907 >> 172.21.15.88:0/920465907 conn(0x7f0d8c0ca9f0 msgr2=0x7f0d8c0c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:35.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.719+0000 7f0d930a9640 1 -- 172.21.15.88:0/920465907 shutdown_connections 2023-10-16T20:58:35.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.719+0000 7f0d930a9640 1 -- 172.21.15.88:0/920465907 wait complete. 2023-10-16T20:58:35.739 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-16T20:58:35.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.879+0000 7f9089dc4640 1 Processor -- start 2023-10-16T20:58:35.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.879+0000 7f9089dc4640 1 -- start start 2023-10-16T20:58:35.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.879+0000 7f9089dc4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f90840c4260 0x7f90840c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:35.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.879+0000 7f9089dc4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f90840c4640 0x7f90841573e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:35.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.879+0000 7f9089dc4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9084069e90 con 0x7f90840c5a60 2023-10-16T20:58:35.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.879+0000 7f9089dc4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90840716a0 con 0x7f90840c4260 2023-10-16T20:58:35.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.879+0000 7f9089dc4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908406aea0 con 0x7f90840c4640 2023-10-16T20:58:35.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.879+0000 7f90837fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f90840c4260 0x7f90840c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:35.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.879+0000 7f9082ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f90840c5a60 0x7f90841510d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35102/0 (socket says 172.21.15.88:35102) 2023-10-16T20:58:35.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.879+0000 7f90837fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f90840c4260 0x7f90840c5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34362/0 (socket says 172.21.15.88:34362) 2023-10-16T20:58:35.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.879+0000 7f9082ffd640 1 -- 172.21.15.88:0/747942195 learned_addr learned my addr 172.21.15.88:0/747942195 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:35.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.879+0000 7f90837fe640 1 -- 172.21.15.88:0/747942195 >> v2:172.21.15.191:3300/0 conn(0x7f90840c4640 msgr2=0x7f90841573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:35.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.879+0000 7f90837fe640 1 --2- 172.21.15.88:0/747942195 >> v2:172.21.15.191:3300/0 conn(0x7f90840c4640 0x7f90841573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.879+0000 7f90837fe640 1 -- 172.21.15.88:0/747942195 >> v1:172.21.15.88:6789/0 conn(0x7f90840c5a60 legacy=0x7f90841510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.879+0000 7f90837fe640 1 -- 172.21.15.88:0/747942195 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90841581e0 con 0x7f90840c4260 2023-10-16T20:58:35.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.879+0000 7f90827fc640 1 -- 172.21.15.88:0/747942195 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904579288 0 0) 0x7f9084069e90 con 0x7f90840c5a60 2023-10-16T20:58:35.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.883+0000 7f90837fe640 1 --2- 172.21.15.88:0/747942195 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90840c4260 0x7f90840c5520 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f90840c1cc0 tx=0x7f907401c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e685453cd500f2f server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:35.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.883+0000 7f90827fc640 1 -- 172.21.15.88:0/747942195 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90740ab600 con 0x7f90840c4260 2023-10-16T20:58:35.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.883+0000 7f90827fc640 1 -- 172.21.15.88:0/747942195 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90740a7050 con 0x7f90840c4260 2023-10-16T20:58:35.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.883+0000 7f90827fc640 1 -- 172.21.15.88:0/747942195 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90740ab980 con 0x7f90840c4260 2023-10-16T20:58:35.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.883+0000 7f9089dc4640 1 -- 172.21.15.88:0/747942195 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90840c4260 msgr2=0x7f90840c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.883+0000 7f9089dc4640 1 --2- 172.21.15.88:0/747942195 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90840c4260 0x7f90840c5520 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f90840c1cc0 tx=0x7f907401c310 comp rx=0 tx=0).stop 2023-10-16T20:58:35.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.883+0000 7f9089dc4640 1 -- 172.21.15.88:0/747942195 shutdown_connections 2023-10-16T20:58:35.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.883+0000 7f9089dc4640 1 --2- 172.21.15.88:0/747942195 >> v2:172.21.15.191:3300/0 conn(0x7f90840c4640 0x7f90841573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.883+0000 7f9089dc4640 1 --2- 172.21.15.88:0/747942195 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90840c4260 0x7f90840c5520 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.883+0000 7f9089dc4640 1 -- 172.21.15.88:0/747942195 >> 172.21.15.88:0/747942195 conn(0x7f90840ca9f0 msgr2=0x7f90840cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:35.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.883+0000 7f9089dc4640 1 -- 172.21.15.88:0/747942195 shutdown_connections 2023-10-16T20:58:35.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.883+0000 7f9089dc4640 1 -- 172.21.15.88:0/747942195 wait complete. 2023-10-16T20:58:35.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9089dc4640 1 Processor -- start 2023-10-16T20:58:35.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9089dc4640 1 -- start start 2023-10-16T20:58:35.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9089dc4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90840c4260 0x7f9084141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:35.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9089dc4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90840c4640 0x7f9084160590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:35.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9089dc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90840c5a60 0x7f9084160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:35.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9089dc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908413ea60 con 0x7f90840c5a60 2023-10-16T20:58:35.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9089dc4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908414e790 con 0x7f90840c4640 2023-10-16T20:58:35.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9089dc4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9084161b50 con 0x7f90840c4260 2023-10-16T20:58:35.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9083fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90840c5a60 0x7f9084160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:35.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9083fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90840c5a60 0x7f9084160ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34364/0 (socket says 172.21.15.88:34364) 2023-10-16T20:58:35.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9083fff640 1 -- 172.21.15.88:0/3035180773 learned_addr learned my addr 172.21.15.88:0/3035180773 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:35.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f90837fe640 1 --2- 172.21.15.88:0/3035180773 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90840c4260 0x7f9084141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:35.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9082ffd640 1 --2- 172.21.15.88:0/3035180773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90840c4640 0x7f9084160590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:35.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9083fff640 1 -- 172.21.15.88:0/3035180773 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90840c4260 msgr2=0x7f9084141f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9083fff640 1 --2- 172.21.15.88:0/3035180773 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90840c4260 0x7f9084141f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9083fff640 1 -- 172.21.15.88:0/3035180773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90840c4640 msgr2=0x7f9084160590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:35.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9083fff640 1 --2- 172.21.15.88:0/3035180773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90840c4640 0x7f9084160590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:35.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9083fff640 1 -- 172.21.15.88:0/3035180773 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9074009420 con 0x7f90840c5a60 2023-10-16T20:58:35.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9083fff640 1 --2- 172.21.15.88:0/3035180773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90840c5a60 0x7f9084160ad0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f907c00b970 tx=0x7f907c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:35.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f906bfff640 1 -- 172.21.15.88:0/3035180773 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c00cb40 con 0x7f90840c5a60 2023-10-16T20:58:35.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f906bfff640 1 -- 172.21.15.88:0/3035180773 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907c00ccc0 con 0x7f90840c5a60 2023-10-16T20:58:35.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f906bfff640 1 -- 172.21.15.88:0/3035180773 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c014680 con 0x7f90840c5a60 2023-10-16T20:58:35.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9089dc4640 1 -- 172.21.15.88:0/3035180773 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908413d5d0 con 0x7f90840c5a60 2023-10-16T20:58:35.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.887+0000 7f9089dc4640 1 -- 172.21.15.88:0/3035180773 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90841600b0 con 0x7f90840c5a60 2023-10-16T20:58:35.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.891+0000 7f9089dc4640 1 -- 172.21.15.88:0/3035180773 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f908415ff10 con 0x7f90840c5a60 2023-10-16T20:58:35.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.891+0000 7f906bfff640 1 -- 172.21.15.88:0/3035180773 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f907c01e440 con 0x7f90840c5a60 2023-10-16T20:58:35.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.891+0000 7f906bfff640 1 --2- 172.21.15.88:0/3035180773 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f90700664a0 0x7f9070068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:35.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.891+0000 7f906bfff640 1 -- 172.21.15.88:0/3035180773 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f907c084560 con 0x7f90840c5a60 2023-10-16T20:58:35.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.891+0000 7f90837fe640 1 --2- 172.21.15.88:0/3035180773 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f90700664a0 0x7f9070068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:35.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.891+0000 7f90837fe640 1 --2- 172.21.15.88:0/3035180773 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f90700664a0 0x7f9070068960 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f90740ad000 tx=0x7f90740a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:35.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:35.895+0000 7f906bfff640 1 -- 172.21.15.88:0/3035180773 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f908415ff10 con 0x7f90840c5a60 2023-10-16T20:58:36.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.047+0000 7f9089dc4640 1 -- 172.21.15.88:0/3035180773 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f90841602b0 con 0x7f90840c5a60 2023-10-16T20:58:36.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.047+0000 7f906bfff640 1 -- 172.21.15.88:0/3035180773 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v67) v1 ==== 104+0+11 (secure 0 0 0) 0x7f90841602b0 con 0x7f90840c5a60 2023-10-16T20:58:36.049 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:58:36.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.051+0000 7f9089dc4640 1 -- 172.21.15.88:0/3035180773 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f90700664a0 msgr2=0x7f9070068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.051+0000 7f9089dc4640 1 --2- 172.21.15.88:0/3035180773 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f90700664a0 0x7f9070068960 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f90740ad000 tx=0x7f90740a6000 comp rx=0 tx=0).stop 2023-10-16T20:58:36.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.051+0000 7f9089dc4640 1 -- 172.21.15.88:0/3035180773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90840c5a60 msgr2=0x7f9084160ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.051+0000 7f9089dc4640 1 --2- 172.21.15.88:0/3035180773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90840c5a60 0x7f9084160ad0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f907c00b970 tx=0x7f907c00be30 comp rx=0 tx=0).stop 2023-10-16T20:58:36.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.051+0000 7f9089dc4640 1 -- 172.21.15.88:0/3035180773 shutdown_connections 2023-10-16T20:58:36.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.051+0000 7f9089dc4640 1 --2- 172.21.15.88:0/3035180773 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f90700664a0 0x7f9070068960 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.051+0000 7f9089dc4640 1 --2- 172.21.15.88:0/3035180773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90840c5a60 0x7f9084160ad0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.051+0000 7f9089dc4640 1 --2- 172.21.15.88:0/3035180773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f90840c4640 0x7f9084160590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.051+0000 7f9089dc4640 1 --2- 172.21.15.88:0/3035180773 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f90840c4260 0x7f9084141f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.051+0000 7f9089dc4640 1 -- 172.21.15.88:0/3035180773 >> 172.21.15.88:0/3035180773 conn(0x7f90840ca9f0 msgr2=0x7f9084071f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:36.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.051+0000 7f9089dc4640 1 -- 172.21.15.88:0/3035180773 shutdown_connections 2023-10-16T20:58:36.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.051+0000 7f9089dc4640 1 -- 172.21.15.88:0/3035180773 wait complete. 2023-10-16T20:58:36.069 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-16T20:58:36.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.211+0000 7f4520417640 1 Processor -- start 2023-10-16T20:58:36.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.211+0000 7f4520417640 1 -- start start 2023-10-16T20:58:36.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.211+0000 7f4520417640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f45181537d0 0x7f451815aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.211+0000 7f4520417640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451806cbe0 con 0x7f45180cdb20 2023-10-16T20:58:36.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.211+0000 7f4520417640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451815d8b0 con 0x7f451806dd70 2023-10-16T20:58:36.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.211+0000 7f4520417640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f451806f5f0 con 0x7f45181537d0 2023-10-16T20:58:36.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.211+0000 7f451e1b3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f45181537d0 0x7f451815aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.211+0000 7f451e1b3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f45181537d0 0x7f451815aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34374/0 (socket says 172.21.15.88:34374) 2023-10-16T20:58:36.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.211+0000 7f451e1b3640 1 -- 172.21.15.88:0/4015053312 learned_addr learned my addr 172.21.15.88:0/4015053312 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:36.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.211+0000 7f451d9b2640 1 --1- 172.21.15.88:0/4015053312 >> v1:172.21.15.88:6789/0 conn(0x7f45180cdb20 0x7f45180ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35118/0 (socket says 172.21.15.88:35118) 2023-10-16T20:58:36.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.211+0000 7f451e1b3640 1 -- 172.21.15.88:0/4015053312 >> v1:172.21.15.191:6789/0 conn(0x7f451806dd70 legacy=0x7f4518163ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:36.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.211+0000 7f451d1b1640 1 -- 172.21.15.88:0/4015053312 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310818570 0 0) 0x7f451806cbe0 con 0x7f45180cdb20 2023-10-16T20:58:36.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.211+0000 7f451e1b3640 1 -- 172.21.15.88:0/4015053312 >> v1:172.21.15.88:6789/0 conn(0x7f45180cdb20 legacy=0x7f45180ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.211+0000 7f451e1b3640 1 -- 172.21.15.88:0/4015053312 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451806e830 con 0x7f45181537d0 2023-10-16T20:58:36.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.215+0000 7f451e1b3640 1 --2- 172.21.15.88:0/4015053312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45181537d0 0x7f451815aa60 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f4514009870 tx=0x7f451401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8bd5adc6c3e77a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:36.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.215+0000 7f451d1b1640 1 -- 172.21.15.88:0/4015053312 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45140ab600 con 0x7f45181537d0 2023-10-16T20:58:36.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.215+0000 7f451d1b1640 1 -- 172.21.15.88:0/4015053312 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45140a7050 con 0x7f45181537d0 2023-10-16T20:58:36.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.215+0000 7f451d1b1640 1 -- 172.21.15.88:0/4015053312 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45140ab980 con 0x7f45181537d0 2023-10-16T20:58:36.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.215+0000 7f4520417640 1 -- 172.21.15.88:0/4015053312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45181537d0 msgr2=0x7f451815aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.215+0000 7f4520417640 1 --2- 172.21.15.88:0/4015053312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45181537d0 0x7f451815aa60 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f4514009870 tx=0x7f451401c540 comp rx=0 tx=0).stop 2023-10-16T20:58:36.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.215+0000 7f4520417640 1 -- 172.21.15.88:0/4015053312 shutdown_connections 2023-10-16T20:58:36.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.215+0000 7f4520417640 1 --2- 172.21.15.88:0/4015053312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45181537d0 0x7f451815aa60 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.215+0000 7f4520417640 1 -- 172.21.15.88:0/4015053312 >> 172.21.15.88:0/4015053312 conn(0x7f451806f7d0 msgr2=0x7f451806fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:36.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.215+0000 7f4520417640 1 -- 172.21.15.88:0/4015053312 shutdown_connections 2023-10-16T20:58:36.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.215+0000 7f4520417640 1 -- 172.21.15.88:0/4015053312 wait complete. 2023-10-16T20:58:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f4520417640 1 Processor -- start 2023-10-16T20:58:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f4520417640 1 -- start start 2023-10-16T20:58:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f4520417640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f451806dd70 0x7f45181550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f4520417640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45180cdb20 0x7f45180c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f4520417640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45181537d0 0x7f45180c9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f4520417640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45180c8ee0 con 0x7f45181537d0 2023-10-16T20:58:36.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f4520417640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45181571e0 con 0x7f45180cdb20 2023-10-16T20:58:36.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f4520417640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4518155b20 con 0x7f451806dd70 2023-10-16T20:58:36.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f451e9b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45181537d0 0x7f45180c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f451e9b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45181537d0 0x7f45180c9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34388/0 (socket says 172.21.15.88:34388) 2023-10-16T20:58:36.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f451e9b4640 1 -- 172.21.15.88:0/809024535 learned_addr learned my addr 172.21.15.88:0/809024535 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:36.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f451e1b3640 1 --2- 172.21.15.88:0/809024535 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f451806dd70 0x7f45181550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f451d9b2640 1 --2- 172.21.15.88:0/809024535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45180cdb20 0x7f45180c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f451e9b4640 1 -- 172.21.15.88:0/809024535 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f451806dd70 msgr2=0x7f45181550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f451e9b4640 1 --2- 172.21.15.88:0/809024535 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f451806dd70 0x7f45181550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f451e9b4640 1 -- 172.21.15.88:0/809024535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45180cdb20 msgr2=0x7f45180c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f451e9b4640 1 --2- 172.21.15.88:0/809024535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45180cdb20 0x7f45180c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f451e9b4640 1 -- 172.21.15.88:0/809024535 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45140094d0 con 0x7f45181537d0 2023-10-16T20:58:36.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f451d9b2640 1 --2- 172.21.15.88:0/809024535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45180cdb20 0x7f45180c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:58:36.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f451e9b4640 1 --2- 172.21.15.88:0/809024535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45181537d0 0x7f45180c9840 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f451000c770 tx=0x7f451000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:36.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f450effd640 1 -- 172.21.15.88:0/809024535 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4510003b80 con 0x7f45181537d0 2023-10-16T20:58:36.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f450effd640 1 -- 172.21.15.88:0/809024535 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4510003ce0 con 0x7f45181537d0 2023-10-16T20:58:36.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f4520417640 1 -- 172.21.15.88:0/809024535 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4518156f00 con 0x7f45181537d0 2023-10-16T20:58:36.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f450effd640 1 -- 172.21.15.88:0/809024535 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451000aa90 con 0x7f45181537d0 2023-10-16T20:58:36.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.219+0000 7f4520417640 1 -- 172.21.15.88:0/809024535 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45180c8b00 con 0x7f45181537d0 2023-10-16T20:58:36.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.223+0000 7f4520417640 1 -- 172.21.15.88:0/809024535 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45180c9020 con 0x7f45181537d0 2023-10-16T20:58:36.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.223+0000 7f450effd640 1 -- 172.21.15.88:0/809024535 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f451000abf0 con 0x7f45181537d0 2023-10-16T20:58:36.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.223+0000 7f450effd640 1 --2- 172.21.15.88:0/809024535 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44fc0664a0 0x7f44fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.223+0000 7f450effd640 1 -- 172.21.15.88:0/809024535 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f4510084350 con 0x7f45181537d0 2023-10-16T20:58:36.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.223+0000 7f451e1b3640 1 --2- 172.21.15.88:0/809024535 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44fc0664a0 0x7f44fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.223+0000 7f451e1b3640 1 --2- 172.21.15.88:0/809024535 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44fc0664a0 0x7f44fc068960 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f451806c6d0 tx=0x7f45140a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:36.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.227+0000 7f450effd640 1 -- 172.21.15.88:0/809024535 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f45100510a0 con 0x7f45181537d0 2023-10-16T20:58:36.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.375+0000 7f4520417640 1 -- 172.21.15.88:0/809024535 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f451806e470 con 0x7f45181537d0 2023-10-16T20:58:36.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.375+0000 7f450effd640 1 -- 172.21.15.88:0/809024535 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v67) v1 ==== 100+0+11 (secure 0 0 0) 0x7f45100546c0 con 0x7f45181537d0 2023-10-16T20:58:36.377 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2023-10-16T20:58:36.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.379+0000 7f4520417640 1 -- 172.21.15.88:0/809024535 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44fc0664a0 msgr2=0x7f44fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.379+0000 7f4520417640 1 --2- 172.21.15.88:0/809024535 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44fc0664a0 0x7f44fc068960 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f451806c6d0 tx=0x7f45140a6000 comp rx=0 tx=0).stop 2023-10-16T20:58:36.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.379+0000 7f4520417640 1 -- 172.21.15.88:0/809024535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45181537d0 msgr2=0x7f45180c9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.379+0000 7f4520417640 1 --2- 172.21.15.88:0/809024535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45181537d0 0x7f45180c9840 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f451000c770 tx=0x7f451000cc30 comp rx=0 tx=0).stop 2023-10-16T20:58:36.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.379+0000 7f4520417640 1 -- 172.21.15.88:0/809024535 shutdown_connections 2023-10-16T20:58:36.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.379+0000 7f4520417640 1 --2- 172.21.15.88:0/809024535 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f44fc0664a0 0x7f44fc068960 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.379+0000 7f4520417640 1 --2- 172.21.15.88:0/809024535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45181537d0 0x7f45180c9840 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.379+0000 7f4520417640 1 --2- 172.21.15.88:0/809024535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f45180cdb20 0x7f45180c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.379+0000 7f4520417640 1 --2- 172.21.15.88:0/809024535 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f451806dd70 0x7f45181550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.379+0000 7f4520417640 1 -- 172.21.15.88:0/809024535 >> 172.21.15.88:0/809024535 conn(0x7f451806f7d0 msgr2=0x7f45180cc410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:36.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.379+0000 7f4520417640 1 -- 172.21.15.88:0/809024535 shutdown_connections 2023-10-16T20:58:36.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.379+0000 7f4520417640 1 -- 172.21.15.88:0/809024535 wait complete. 2023-10-16T20:58:36.397 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-16T20:58:36.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.539+0000 7feba5c9c640 1 Processor -- start 2023-10-16T20:58:36.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.539+0000 7feba5c9c640 1 -- start start 2023-10-16T20:58:36.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.539+0000 7feba5c9c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feba0153780 0x7feba00c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.539+0000 7feba5c9c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7feba0159490 0x7feba00c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.539+0000 7feba5c9c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba0159d90 con 0x7feba00c5930 2023-10-16T20:58:36.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.539+0000 7feba5c9c640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba015a070 con 0x7feba0159490 2023-10-16T20:58:36.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.539+0000 7feba5c9c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba015a350 con 0x7feba0153780 2023-10-16T20:58:36.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7feba00c5930 0x7feba00ced80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35122/0 (socket says 172.21.15.88:35122) 2023-10-16T20:58:36.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9ffff640 1 -- 172.21.15.88:0/1570292689 learned_addr learned my addr 172.21.15.88:0/1570292689 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:36.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb97fff640 1 --2- 172.21.15.88:0/1570292689 >> v2:172.21.15.191:3300/0 conn(0x7feba0159490 0x7feba00c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9f7fe640 1 --2- 172.21.15.88:0/1570292689 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feba0153780 0x7feba00c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9effd640 1 -- 172.21.15.88:0/1570292689 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197134685 0 0) 0x7feba0159d90 con 0x7feba00c5930 2023-10-16T20:58:36.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9effd640 1 -- 172.21.15.88:0/1570292689 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb8c003530 con 0x7feba00c5930 2023-10-16T20:58:36.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9effd640 1 -- 172.21.15.88:0/1570292689 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1726017405 0 0) 0x7feb8c003530 con 0x7feba00c5930 2023-10-16T20:58:36.547 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9effd640 1 -- 172.21.15.88:0/1570292689 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feba0153780 msgr2=0x7feba00c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:36.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9effd640 1 --2- 172.21.15.88:0/1570292689 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feba0153780 0x7feba00c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9effd640 1 -- 172.21.15.88:0/1570292689 >> v2:172.21.15.191:3300/0 conn(0x7feba0159490 msgr2=0x7feba00c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:36.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9effd640 1 --2- 172.21.15.88:0/1570292689 >> v2:172.21.15.191:3300/0 conn(0x7feba0159490 0x7feba00c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9effd640 1 -- 172.21.15.88:0/1570292689 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba00c4910 con 0x7feba00c5930 2023-10-16T20:58:36.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9effd640 1 -- 172.21.15.88:0/1570292689 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7feb98001a00 con 0x7feba00c5930 2023-10-16T20:58:36.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9effd640 1 -- 172.21.15.88:0/1570292689 >> v1:172.21.15.88:6789/0 conn(0x7feba00c5930 legacy=0x7feba00ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9effd640 1 --2- 172.21.15.88:0/1570292689 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb8c003980 0x7feb8c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9effd640 1 --2- 172.21.15.88:0/1570292689 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feba0153780 0x7feb8c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9effd640 1 --2- 172.21.15.88:0/1570292689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb8c004d10 0x7feb8c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9effd640 1 -- 172.21.15.88:0/1570292689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba015a070 con 0x7feb8c004d10 2023-10-16T20:58:36.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9effd640 1 -- 172.21.15.88:0/1570292689 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba015a350 con 0x7feb8c003980 2023-10-16T20:58:36.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9effd640 1 -- 172.21.15.88:0/1570292689 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c00b990 con 0x7feba0153780 2023-10-16T20:58:36.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9ffff640 1 --2- 172.21.15.88:0/1570292689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb8c004d10 0x7feb8c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.551 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9f7fe640 1 --2- 172.21.15.88:0/1570292689 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb8c003980 0x7feb8c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb97fff640 1 --2- 172.21.15.88:0/1570292689 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feba0153780 0x7feb8c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9ffff640 1 -- 172.21.15.88:0/1570292689 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feba0153780 msgr2=0x7feb8c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:36.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9ffff640 1 --2- 172.21.15.88:0/1570292689 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feba0153780 0x7feb8c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9ffff640 1 -- 172.21.15.88:0/1570292689 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb8c003980 msgr2=0x7feb8c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:36.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9ffff640 1 --2- 172.21.15.88:0/1570292689 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feb8c003980 0x7feb8c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.543+0000 7feb9ffff640 1 -- 172.21.15.88:0/1570292689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c003530 con 0x7feb8c004d10 2023-10-16T20:58:36.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.547+0000 7feb9ffff640 1 --2- 172.21.15.88:0/1570292689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb8c004d10 0x7feb8c005190 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7feba006ba60 tx=0x7feb9801d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e469aa1818a09f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:36.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.547+0000 7feb9effd640 1 -- 172.21.15.88:0/1570292689 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb980ad630 con 0x7feb8c004d10 2023-10-16T20:58:36.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.547+0000 7feb9effd640 1 -- 172.21.15.88:0/1570292689 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb980a8070 con 0x7feb8c004d10 2023-10-16T20:58:36.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.547+0000 7feb9effd640 1 -- 172.21.15.88:0/1570292689 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb980ad9f0 con 0x7feb8c004d10 2023-10-16T20:58:36.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.547+0000 7feba5c9c640 1 -- 172.21.15.88:0/1570292689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb8c004d10 msgr2=0x7feb8c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.547+0000 7feba5c9c640 1 --2- 172.21.15.88:0/1570292689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb8c004d10 0x7feb8c005190 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7feba006ba60 tx=0x7feb9801d300 comp rx=0 tx=0).stop 2023-10-16T20:58:36.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.547+0000 7feb9ffff640 1 -- 172.21.15.88:0/1570292689 reap_dead start 2023-10-16T20:58:36.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.547+0000 7feba5c9c640 1 -- 172.21.15.88:0/1570292689 shutdown_connections 2023-10-16T20:58:36.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.547+0000 7feba5c9c640 1 -- 172.21.15.88:0/1570292689 >> 172.21.15.88:0/1570292689 conn(0x7feba006bcb0 msgr2=0x7feba006c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:36.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.547+0000 7feba5c9c640 1 -- 172.21.15.88:0/1570292689 shutdown_connections 2023-10-16T20:58:36.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.547+0000 7feba5c9c640 1 -- 172.21.15.88:0/1570292689 wait complete. 2023-10-16T20:58:36.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.547+0000 7feba5c9c640 1 Processor -- start 2023-10-16T20:58:36.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.547+0000 7feba5c9c640 1 -- start start 2023-10-16T20:58:36.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.547+0000 7feba5c9c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb8c004d10 0x7feba0153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feba5c9c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feba0158ff0 0x7feba00c3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feba5c9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba01eea80 0x7feba01eef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feba5c9c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba01ef440 con 0x7feba01eea80 2023-10-16T20:58:36.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feba5c9c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba01f1ff0 con 0x7feba0158ff0 2023-10-16T20:58:36.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feba5c9c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba01f2b70 con 0x7feb8c004d10 2023-10-16T20:58:36.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feb9ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba01eea80 0x7feba01eef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feb9ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba01eea80 0x7feba01eef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34392/0 (socket says 172.21.15.88:34392) 2023-10-16T20:58:36.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feb9ffff640 1 -- 172.21.15.88:0/818035894 learned_addr learned my addr 172.21.15.88:0/818035894 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:36.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feb97fff640 1 --2- 172.21.15.88:0/818035894 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feba0158ff0 0x7feba00c3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feb9f7fe640 1 --2- 172.21.15.88:0/818035894 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb8c004d10 0x7feba0153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feb9ffff640 1 -- 172.21.15.88:0/818035894 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb8c004d10 msgr2=0x7feba0153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feb9ffff640 1 --2- 172.21.15.88:0/818035894 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb8c004d10 0x7feba0153ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feb9ffff640 1 -- 172.21.15.88:0/818035894 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feba0158ff0 msgr2=0x7feba00c3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feb9ffff640 1 --2- 172.21.15.88:0/818035894 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feba0158ff0 0x7feba00c3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feb9ffff640 1 -- 172.21.15.88:0/818035894 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba0158c10 con 0x7feba01eea80 2023-10-16T20:58:36.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feb9ffff640 1 --2- 172.21.15.88:0/818035894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba01eea80 0x7feba01eef00 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7feb98098d20 tx=0x7feb8c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:36.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feb9cff9640 1 -- 172.21.15.88:0/818035894 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb980017c0 con 0x7feba01eea80 2023-10-16T20:58:36.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feb9cff9640 1 -- 172.21.15.88:0/818035894 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb98001920 con 0x7feba01eea80 2023-10-16T20:58:36.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feb9cff9640 1 -- 172.21.15.88:0/818035894 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb98004d70 con 0x7feba01eea80 2023-10-16T20:58:36.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feba5c9c640 1 -- 172.21.15.88:0/818035894 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba0154550 con 0x7feba01eea80 2023-10-16T20:58:36.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feba5c9c640 1 -- 172.21.15.88:0/818035894 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba01f2520 con 0x7feba01eea80 2023-10-16T20:58:36.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feba5c9c640 1 -- 172.21.15.88:0/818035894 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba01f2d20 con 0x7feba01eea80 2023-10-16T20:58:36.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.551+0000 7feb9cff9640 1 -- 172.21.15.88:0/818035894 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feb980aa030 con 0x7feba01eea80 2023-10-16T20:58:36.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.555+0000 7feb9cff9640 1 --2- 172.21.15.88:0/818035894 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7feb800664d0 0x7feb80068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.555+0000 7feb9cff9640 1 -- 172.21.15.88:0/818035894 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7feb9811dda0 con 0x7feba01eea80 2023-10-16T20:58:36.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.555+0000 7feb9f7fe640 1 --2- 172.21.15.88:0/818035894 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7feb800664d0 0x7feb80068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.555+0000 7feb9f7fe640 1 --2- 172.21.15.88:0/818035894 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7feb800664d0 0x7feb80068990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7feb90009b40 tx=0x7feb90006f90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:36.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.559+0000 7feb9cff9640 1 -- 172.21.15.88:0/818035894 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feba01f2d20 con 0x7feba01eea80 2023-10-16T20:58:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.707+0000 7feba5c9c640 1 -- 172.21.15.88:0/818035894 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7feba00ca7d0 con 0x7feba01eea80 2023-10-16T20:58:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.707+0000 7feb9cff9640 1 -- 172.21.15.88:0/818035894 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v67) v1 ==== 103+0+11 (secure 0 0 0) 0x7feb980eabc0 con 0x7feba01eea80 2023-10-16T20:58:36.710 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-10-16T20:58:36.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.711+0000 7feba5c9c640 1 -- 172.21.15.88:0/818035894 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7feb800664d0 msgr2=0x7feb80068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.711+0000 7feba5c9c640 1 --2- 172.21.15.88:0/818035894 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7feb800664d0 0x7feb80068990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7feb90009b40 tx=0x7feb90006f90 comp rx=0 tx=0).stop 2023-10-16T20:58:36.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.711+0000 7feba5c9c640 1 -- 172.21.15.88:0/818035894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba01eea80 msgr2=0x7feba01eef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.711+0000 7feba5c9c640 1 --2- 172.21.15.88:0/818035894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba01eea80 0x7feba01eef00 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7feb98098d20 tx=0x7feb8c005800 comp rx=0 tx=0).stop 2023-10-16T20:58:36.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.711+0000 7feba5c9c640 1 -- 172.21.15.88:0/818035894 shutdown_connections 2023-10-16T20:58:36.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.711+0000 7feba5c9c640 1 --2- 172.21.15.88:0/818035894 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7feb800664d0 0x7feb80068990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.711+0000 7feba5c9c640 1 --2- 172.21.15.88:0/818035894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba01eea80 0x7feba01eef00 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.711+0000 7feba5c9c640 1 --2- 172.21.15.88:0/818035894 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feba0158ff0 0x7feba00c3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.711+0000 7feba5c9c640 1 --2- 172.21.15.88:0/818035894 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feb8c004d10 0x7feba0153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.711+0000 7feba5c9c640 1 -- 172.21.15.88:0/818035894 >> 172.21.15.88:0/818035894 conn(0x7feba006bcb0 msgr2=0x7feba00ce180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:36.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.711+0000 7feba5c9c640 1 -- 172.21.15.88:0/818035894 shutdown_connections 2023-10-16T20:58:36.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.711+0000 7feba5c9c640 1 -- 172.21.15.88:0/818035894 wait complete. 2023-10-16T20:58:36.731 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-16T20:58:36.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.871+0000 7f813ac9e640 1 Processor -- start 2023-10-16T20:58:36.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.871+0000 7f813ac9e640 1 -- start start 2023-10-16T20:58:36.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.871+0000 7f813ac9e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f81340cb990 0x7f81341599f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.871+0000 7f813ac9e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81340c0230 con 0x7f8134153a00 2023-10-16T20:58:36.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.871+0000 7f813ac9e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81341594f0 con 0x7f8134153620 2023-10-16T20:58:36.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.871+0000 7f813ac9e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81340c0480 con 0x7f81340cb990 2023-10-16T20:58:36.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.871+0000 7f8133fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8134153a00 0x7f81340cb280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35136/0 (socket says 172.21.15.88:35136) 2023-10-16T20:58:36.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.871+0000 7f8133fff640 1 -- 172.21.15.88:0/3759683202 learned_addr learned my addr 172.21.15.88:0/3759683202 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:36.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.871+0000 7f813923b640 1 --2- 172.21.15.88:0/3759683202 >> v2:172.21.15.88:3300/0 conn(0x7f81340cb990 0x7f81341599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.871+0000 7f813923b640 1 -- 172.21.15.88:0/3759683202 >> v1:172.21.15.191:6789/0 conn(0x7f8134153620 legacy=0x7f81340c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.871+0000 7f813923b640 1 -- 172.21.15.88:0/3759683202 >> v1:172.21.15.88:6789/0 conn(0x7f8134153a00 legacy=0x7f81340cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.871+0000 7f813923b640 1 -- 172.21.15.88:0/3759683202 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81340c0760 con 0x7f81340cb990 2023-10-16T20:58:36.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.875+0000 7f813923b640 1 --2- 172.21.15.88:0/3759683202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81340cb990 0x7f81341599f0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f812c009d90 tx=0x7f812c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9c41382e8bd96b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:36.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.875+0000 7f81337fe640 1 -- 172.21.15.88:0/3759683202 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c0a5900 con 0x7f81340cb990 2023-10-16T20:58:36.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.875+0000 7f81337fe640 1 -- 172.21.15.88:0/3759683202 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812c0a5a60 con 0x7f81340cb990 2023-10-16T20:58:36.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.875+0000 7f81337fe640 1 -- 172.21.15.88:0/3759683202 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c0a5df0 con 0x7f81340cb990 2023-10-16T20:58:36.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.875+0000 7f813ac9e640 1 -- 172.21.15.88:0/3759683202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81340cb990 msgr2=0x7f81341599f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.875+0000 7f813ac9e640 1 --2- 172.21.15.88:0/3759683202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81340cb990 0x7f81341599f0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f812c009d90 tx=0x7f812c01c950 comp rx=0 tx=0).stop 2023-10-16T20:58:36.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.875+0000 7f813ac9e640 1 -- 172.21.15.88:0/3759683202 shutdown_connections 2023-10-16T20:58:36.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.875+0000 7f813ac9e640 1 --2- 172.21.15.88:0/3759683202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81340cb990 0x7f81341599f0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.875+0000 7f813ac9e640 1 -- 172.21.15.88:0/3759683202 >> 172.21.15.88:0/3759683202 conn(0x7f8134069e90 msgr2=0x7f813406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:36.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.875+0000 7f813ac9e640 1 -- 172.21.15.88:0/3759683202 shutdown_connections 2023-10-16T20:58:36.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.875+0000 7f813ac9e640 1 -- 172.21.15.88:0/3759683202 wait complete. 2023-10-16T20:58:36.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.875+0000 7f813ac9e640 1 Processor -- start 2023-10-16T20:58:36.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.875+0000 7f813ac9e640 1 -- start start 2023-10-16T20:58:36.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f813ac9e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81340cb990 0x7f813413fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f813ac9e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8134153620 0x7f8134140300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f813ac9e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8134153a00 0x7f81340c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f813ac9e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81340c9200 con 0x7f8134153620 2023-10-16T20:58:36.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f813ac9e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8134140970 con 0x7f8134153a00 2023-10-16T20:58:36.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f813ac9e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81341f2770 con 0x7f81340cb990 2023-10-16T20:58:36.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f8133fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8134153620 0x7f8134140300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f8133fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8134153620 0x7f8134140300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34408/0 (socket says 172.21.15.88:34408) 2023-10-16T20:58:36.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f8133fff640 1 -- 172.21.15.88:0/4100033636 learned_addr learned my addr 172.21.15.88:0/4100033636 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:36.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f813923b640 1 --2- 172.21.15.88:0/4100033636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8134153a00 0x7f81340c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f8138a3a640 1 --2- 172.21.15.88:0/4100033636 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81340cb990 0x7f813413fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f8133fff640 1 -- 172.21.15.88:0/4100033636 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81340cb990 msgr2=0x7f813413fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f8133fff640 1 --2- 172.21.15.88:0/4100033636 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81340cb990 0x7f813413fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f8133fff640 1 -- 172.21.15.88:0/4100033636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8134153a00 msgr2=0x7f81340c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:36.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f8133fff640 1 --2- 172.21.15.88:0/4100033636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8134153a00 0x7f81340c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:36.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f8133fff640 1 -- 172.21.15.88:0/4100033636 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812c009a70 con 0x7f8134153620 2023-10-16T20:58:36.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f8133fff640 1 --2- 172.21.15.88:0/4100033636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8134153620 0x7f8134140300 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f812000c400 tx=0x7f812000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:36.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f81317fa640 1 -- 172.21.15.88:0/4100033636 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8120005cf0 con 0x7f8134153620 2023-10-16T20:58:36.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f81317fa640 1 -- 172.21.15.88:0/4100033636 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8120005e50 con 0x7f8134153620 2023-10-16T20:58:36.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f81317fa640 1 -- 172.21.15.88:0/4100033636 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81200072d0 con 0x7f8134153620 2023-10-16T20:58:36.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f813ac9e640 1 -- 172.21.15.88:0/4100033636 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81341f2a50 con 0x7f8134153620 2023-10-16T20:58:36.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f813ac9e640 1 -- 172.21.15.88:0/4100033636 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8134140f20 con 0x7f8134153620 2023-10-16T20:58:36.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.879+0000 7f813ac9e640 1 -- 172.21.15.88:0/4100033636 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81340c9340 con 0x7f8134153620 2023-10-16T20:58:36.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.883+0000 7f81317fa640 1 -- 172.21.15.88:0/4100033636 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8120007430 con 0x7f8134153620 2023-10-16T20:58:36.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.883+0000 7f81317fa640 1 --2- 172.21.15.88:0/4100033636 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f81180664d0 0x7f8118068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:36.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.883+0000 7f81317fa640 1 -- 172.21.15.88:0/4100033636 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f8120083a60 con 0x7f8134153620 2023-10-16T20:58:36.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.883+0000 7f8138a3a640 1 --2- 172.21.15.88:0/4100033636 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f81180664d0 0x7f8118068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:36.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.883+0000 7f8138a3a640 1 --2- 172.21.15.88:0/4100033636 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f81180664d0 0x7f8118068990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8128007fd0 tx=0x7f8128009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:36.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:36.887+0000 7f81317fa640 1 -- 172.21.15.88:0/4100033636 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f81340c9340 con 0x7f8134153620 2023-10-16T20:58:37.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:37.047+0000 7f813ac9e640 1 -- 172.21.15.88:0/4100033636 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f81340c9340 con 0x7f8134153620 2023-10-16T20:58:37.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:37.047+0000 7f81317fa640 1 -- 172.21.15.88:0/4100033636 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 20 entries v67) v1 ==== 81+0+1228 (secure 0 0 0) 0x7f81200174e0 con 0x7f8134153620 2023-10-16T20:58:37.049 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6839/2172973775 2023-10-17T20:56:40.633674+0000 2023-10-16T20:58:37.050 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6833/3633326602 2023-10-17T20:55:50.604940+0000 2023-10-16T20:58:37.050 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6836/1120825335 2023-10-17T20:42:10.335721+0000 2023-10-16T20:58:37.050 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6832/3633326602 2023-10-17T20:55:50.604940+0000 2023-10-16T20:58:37.050 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/518281867 2023-10-17T20:44:25.369398+0000 2023-10-16T20:58:37.051 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6837/2468647900 2023-10-17T20:53:25.553637+0000 2023-10-16T20:58:37.051 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1987579860 2023-10-17T20:43:25.353447+0000 2023-10-16T20:58:37.051 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6835/3119045295 2023-10-17T20:58:00.653202+0000 2023-10-16T20:58:37.051 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/321173984 2023-10-17T20:46:15.444657+0000 2023-10-16T20:58:37.052 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6837/1120825335 2023-10-17T20:42:10.335721+0000 2023-10-16T20:58:37.052 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6832/2446850309 2023-10-17T20:48:43.641901+0000 2023-10-16T20:58:37.052 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2172973775 2023-10-17T20:56:40.633674+0000 2023-10-16T20:58:37.053 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6839/1987579860 2023-10-17T20:43:25.353447+0000 2023-10-16T20:58:37.053 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6834/3119045295 2023-10-17T20:58:00.653202+0000 2023-10-16T20:58:37.053 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/518281867 2023-10-17T20:44:25.369398+0000 2023-10-16T20:58:37.053 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6838/321173984 2023-10-17T20:46:15.444657+0000 2023-10-16T20:58:37.054 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6834/3059699574 2023-10-17T20:51:20.522638+0000 2023-10-16T20:58:37.054 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6833/2446850309 2023-10-17T20:48:43.641901+0000 2023-10-16T20:58:37.054 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6835/3059699574 2023-10-17T20:51:20.522638+0000 2023-10-16T20:58:37.055 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.191:6836/2468647900 2023-10-17T20:53:25.553637+0000 2023-10-16T20:58:37.056 INFO:teuthology.orchestra.run.smithi088.stderr:listed 20 entries 2023-10-16T20:58:37.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:37.051+0000 7f813ac9e640 1 -- 172.21.15.88:0/4100033636 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f81180664d0 msgr2=0x7f8118068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:37.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:37.051+0000 7f813ac9e640 1 --2- 172.21.15.88:0/4100033636 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f81180664d0 0x7f8118068990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8128007fd0 tx=0x7f8128009040 comp rx=0 tx=0).stop 2023-10-16T20:58:37.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:37.051+0000 7f813ac9e640 1 -- 172.21.15.88:0/4100033636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8134153620 msgr2=0x7f8134140300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:37.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:37.051+0000 7f813ac9e640 1 --2- 172.21.15.88:0/4100033636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8134153620 0x7f8134140300 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f812000c400 tx=0x7f812000c8c0 comp rx=0 tx=0).stop 2023-10-16T20:58:37.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:37.051+0000 7f813ac9e640 1 -- 172.21.15.88:0/4100033636 shutdown_connections 2023-10-16T20:58:37.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:37.051+0000 7f813ac9e640 1 --2- 172.21.15.88:0/4100033636 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f81180664d0 0x7f8118068990 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:37.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:37.051+0000 7f813ac9e640 1 --2- 172.21.15.88:0/4100033636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8134153a00 0x7f81340c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:37.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:37.051+0000 7f813ac9e640 1 --2- 172.21.15.88:0/4100033636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8134153620 0x7f8134140300 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:37.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:37.051+0000 7f813ac9e640 1 --2- 172.21.15.88:0/4100033636 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f81340cb990 0x7f813413fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:37.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:37.051+0000 7f813ac9e640 1 -- 172.21.15.88:0/4100033636 >> 172.21.15.88:0/4100033636 conn(0x7f8134069e90 msgr2=0x7f81340c7560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:37.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:37.051+0000 7f813ac9e640 1 -- 172.21.15.88:0/4100033636 shutdown_connections 2023-10-16T20:58:37.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:37.051+0000 7f813ac9e640 1 -- 172.21.15.88:0/4100033636 wait complete. 2023-10-16T20:58:37.069 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi088.front.sepia.ceph.com... 2023-10-16T20:58:37.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:58:37.070 DEBUG:teuthology.orchestra.run.smithi088:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-16T20:58:37.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi088.stderr:ceph-fuse[46445]: fuse finished with error 0 and tester_r 0 2023-10-16T20:58:37.267 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T20:58:43.369 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2023-10-16T20:58:43.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:58:43.370 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-16T20:58:43.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:58:43.381 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2023-10-16T20:58:43.435 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-16T20:58:43.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:58:43.436 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-16T20:58:43.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-16T20:58:43.506 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link delete ceph-brx 2023-10-16T20:58:43.740 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-16T20:58:43.752 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-16T20:58:43.752 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T20:58:43.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.890+0000 7ff1ca5b5640 1 Processor -- start 2023-10-16T20:58:43.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.890+0000 7ff1ca5b5640 1 -- start start 2023-10-16T20:58:43.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.890+0000 7ff1ca5b5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1c40ca2f0 0x7ff1c40c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:43.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.890+0000 7ff1ca5b5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c40cae10 con 0x7ff1c40c51c0 2023-10-16T20:58:43.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.890+0000 7ff1ca5b5640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c40cac30 con 0x7ff1c41537c0 2023-10-16T20:58:43.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.890+0000 7ff1ca5b5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c415d2b0 con 0x7ff1c40ca2f0 2023-10-16T20:58:43.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.890+0000 7ff1c9db4640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff1c40c51c0 0x7ff1c40cead0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59564/0 (socket says 172.21.15.88:59564) 2023-10-16T20:58:43.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.890+0000 7ff1c9db4640 1 -- 172.21.15.88:0/2427820111 learned_addr learned my addr 172.21.15.88:0/2427820111 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:43.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1c8db2640 1 --2- 172.21.15.88:0/2427820111 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1c40ca2f0 0x7ff1c40c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:43.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1bbfff640 1 -- 172.21.15.88:0/2427820111 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233294351 0 0) 0x7ff1c40cae10 con 0x7ff1c40c51c0 2023-10-16T20:58:43.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1bbfff640 1 -- 172.21.15.88:0/2427820111 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1b0003530 con 0x7ff1c40c51c0 2023-10-16T20:58:43.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1bbfff640 1 -- 172.21.15.88:0/2427820111 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097551390 0 0) 0x7ff1c40cac30 con 0x7ff1c41537c0 2023-10-16T20:58:43.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1bbfff640 1 -- 172.21.15.88:0/2427820111 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c40cae10 con 0x7ff1c41537c0 2023-10-16T20:58:43.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1bbfff640 1 -- 172.21.15.88:0/2427820111 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 249649268 0 0) 0x7ff1b0003530 con 0x7ff1c40c51c0 2023-10-16T20:58:43.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1bbfff640 1 -- 172.21.15.88:0/2427820111 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1c40ca2f0 msgr2=0x7ff1c40c4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:43.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1bbfff640 1 --2- 172.21.15.88:0/2427820111 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1c40ca2f0 0x7ff1c40c4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:43.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1bbfff640 1 -- 172.21.15.88:0/2427820111 >> v1:172.21.15.191:6789/0 conn(0x7ff1c41537c0 legacy=0x7ff1c415c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:43.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1bbfff640 1 -- 172.21.15.88:0/2427820111 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c415d590 con 0x7ff1c40c51c0 2023-10-16T20:58:43.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1bbfff640 1 -- 172.21.15.88:0/2427820111 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7ff1bc001a00 con 0x7ff1c40c51c0 2023-10-16T20:58:43.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1bbfff640 1 -- 172.21.15.88:0/2427820111 >> v1:172.21.15.88:6789/0 conn(0x7ff1c40c51c0 legacy=0x7ff1c40cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:43.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1bbfff640 1 --2- 172.21.15.88:0/2427820111 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1b0003b40 0x7ff1b0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:43.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1bbfff640 1 --2- 172.21.15.88:0/2427820111 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1c40ca2f0 0x7ff1b00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:43.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1bbfff640 1 --2- 172.21.15.88:0/2427820111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1b0004d10 0x7ff1b0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:43.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1bbfff640 1 -- 172.21.15.88:0/2427820111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c415d2b0 con 0x7ff1b0004d10 2023-10-16T20:58:43.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1bbfff640 1 -- 172.21.15.88:0/2427820111 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b000ba40 con 0x7ff1b0003b40 2023-10-16T20:58:43.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1bbfff640 1 -- 172.21.15.88:0/2427820111 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b000bbb0 con 0x7ff1c40ca2f0 2023-10-16T20:58:43.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1c9db4640 1 --2- 172.21.15.88:0/2427820111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1b0004d10 0x7ff1b0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:43.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1c95b3640 1 --2- 172.21.15.88:0/2427820111 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1b0003b40 0x7ff1b0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:43.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1c8db2640 1 --2- 172.21.15.88:0/2427820111 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1c40ca2f0 0x7ff1b00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:43.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1c9db4640 1 -- 172.21.15.88:0/2427820111 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1c40ca2f0 msgr2=0x7ff1b00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:43.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1c9db4640 1 --2- 172.21.15.88:0/2427820111 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1c40ca2f0 0x7ff1b00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:43.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1c9db4640 1 -- 172.21.15.88:0/2427820111 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1b0003b40 msgr2=0x7ff1b0003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:43.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1c9db4640 1 --2- 172.21.15.88:0/2427820111 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1b0003b40 0x7ff1b0003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:43.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.894+0000 7ff1c9db4640 1 -- 172.21.15.88:0/2427820111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b0003530 con 0x7ff1b0004d10 2023-10-16T20:58:43.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1c9db4640 1 --2- 172.21.15.88:0/2427820111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1b0004d10 0x7ff1b0005190 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00a7a0 tx=0x7ff1bc01d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a586bab5ca83ec0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:43.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1bbfff640 1 -- 172.21.15.88:0/2427820111 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0ad630 con 0x7ff1b0004d10 2023-10-16T20:58:43.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1bbfff640 1 -- 172.21.15.88:0/2427820111 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1bc0a8070 con 0x7ff1b0004d10 2023-10-16T20:58:43.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1bbfff640 1 -- 172.21.15.88:0/2427820111 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0ad9f0 con 0x7ff1b0004d10 2023-10-16T20:58:43.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1ca5b5640 1 -- 172.21.15.88:0/2427820111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1b0004d10 msgr2=0x7ff1b0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:43.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1ca5b5640 1 --2- 172.21.15.88:0/2427820111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1b0004d10 0x7ff1b0005190 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00a7a0 tx=0x7ff1bc01d2a0 comp rx=0 tx=0).stop 2023-10-16T20:58:43.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1c9db4640 1 -- 172.21.15.88:0/2427820111 reap_dead start 2023-10-16T20:58:43.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1ca5b5640 1 -- 172.21.15.88:0/2427820111 shutdown_connections 2023-10-16T20:58:43.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1ca5b5640 1 -- 172.21.15.88:0/2427820111 >> 172.21.15.88:0/2427820111 conn(0x7ff1c406c170 msgr2=0x7ff1c40715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:43.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1ca5b5640 1 -- 172.21.15.88:0/2427820111 shutdown_connections 2023-10-16T20:58:43.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1ca5b5640 1 -- 172.21.15.88:0/2427820111 wait complete. 2023-10-16T20:58:43.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1ca5b5640 1 Processor -- start 2023-10-16T20:58:43.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1ca5b5640 1 -- start start 2023-10-16T20:58:43.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1ca5b5640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1b0004d10 0x7ff1c4156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:43.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1ca5b5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415c5c0 0x7ff1c415b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:43.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1ca5b5640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1c41f0d40 0x7ff1c41f11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:43.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1ca5b5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c4155330 con 0x7ff1c415c5c0 2023-10-16T20:58:43.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1ca5b5640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c4154680 con 0x7ff1c41f0d40 2023-10-16T20:58:43.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1ca5b5640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c41f1700 con 0x7ff1b0004d10 2023-10-16T20:58:43.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.898+0000 7ff1c8db2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415c5c0 0x7ff1c415b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:43.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1c8db2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415c5c0 0x7ff1c415b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54672/0 (socket says 172.21.15.88:54672) 2023-10-16T20:58:43.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1c8db2640 1 -- 172.21.15.88:0/3538372544 learned_addr learned my addr 172.21.15.88:0/3538372544 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:43.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1c95b3640 1 --2- 172.21.15.88:0/3538372544 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1b0004d10 0x7ff1c4156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:43.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1c9db4640 1 --2- 172.21.15.88:0/3538372544 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1c41f0d40 0x7ff1c41f11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:43.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1c8db2640 1 -- 172.21.15.88:0/3538372544 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1b0004d10 msgr2=0x7ff1c4156480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:43.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1c8db2640 1 --2- 172.21.15.88:0/3538372544 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1b0004d10 0x7ff1c4156480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:43.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1c8db2640 1 -- 172.21.15.88:0/3538372544 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1c41f0d40 msgr2=0x7ff1c41f11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:43.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1c8db2640 1 --2- 172.21.15.88:0/3538372544 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1c41f0d40 0x7ff1c41f11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:43.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1c8db2640 1 -- 172.21.15.88:0/3538372544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c41f0580 con 0x7ff1c415c5c0 2023-10-16T20:58:43.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1c8db2640 1 --2- 172.21.15.88:0/3538372544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415c5c0 0x7ff1c415b920 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4011a00 tx=0x7ff1b4011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:43.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1b9ffb640 1 -- 172.21.15.88:0/3538372544 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b400fb50 con 0x7ff1c415c5c0 2023-10-16T20:58:43.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1b9ffb640 1 -- 172.21.15.88:0/3538372544 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b400fcb0 con 0x7ff1c415c5c0 2023-10-16T20:58:43.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1b9ffb640 1 -- 172.21.15.88:0/3538372544 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4014440 con 0x7ff1c415c5c0 2023-10-16T20:58:43.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1ca5b5640 1 -- 172.21.15.88:0/3538372544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c41548d0 con 0x7ff1c415c5c0 2023-10-16T20:58:43.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1ca5b5640 1 -- 172.21.15.88:0/3538372544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c41f0820 con 0x7ff1c415c5c0 2023-10-16T20:58:43.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1ca5b5640 1 -- 172.21.15.88:0/3538372544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c40cc470 con 0x7ff1c415c5c0 2023-10-16T20:58:43.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1b9ffb640 1 -- 172.21.15.88:0/3538372544 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff1b40145a0 con 0x7ff1c415c5c0 2023-10-16T20:58:43.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1b9ffb640 1 --2- 172.21.15.88:0/3538372544 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff1a80664a0 0x7ff1a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:43.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.902+0000 7ff1b9ffb640 1 -- 172.21.15.88:0/3538372544 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7ff1b4084f50 con 0x7ff1c415c5c0 2023-10-16T20:58:43.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.906+0000 7ff1c95b3640 1 --2- 172.21.15.88:0/3538372544 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff1a80664a0 0x7ff1a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:43.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.906+0000 7ff1c95b3640 1 --2- 172.21.15.88:0/3538372544 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff1a80664a0 0x7ff1a8068960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0007fd0 tx=0x7ff1c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:43.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:43.910+0000 7ff1b9ffb640 1 -- 172.21.15.88:0/3538372544 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff1b4051cd0 con 0x7ff1c415c5c0 2023-10-16T20:58:44.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.054+0000 7ff1ca5b5640 1 -- 172.21.15.88:0/3538372544 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff1c40c4480 con 0x7ff1a80664a0 2023-10-16T20:58:44.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.070+0000 7ff1b9ffb640 1 -- 172.21.15.88:0/3538372544 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+374317 (secure 0 0 0) 0x7ff1c0001560 con 0x7ff1a80664a0 2023-10-16T20:58:44.075 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:44.078 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-16T20:58:44.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.074+0000 7ff1ca5b5640 1 -- 172.21.15.88:0/3538372544 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff1a80664a0 msgr2=0x7ff1a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:44.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.074+0000 7ff1ca5b5640 1 --2- 172.21.15.88:0/3538372544 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff1a80664a0 0x7ff1a8068960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0007fd0 tx=0x7ff1c0009040 comp rx=0 tx=0).stop 2023-10-16T20:58:44.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.074+0000 7ff1ca5b5640 1 -- 172.21.15.88:0/3538372544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415c5c0 msgr2=0x7ff1c415b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:44.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.074+0000 7ff1ca5b5640 1 --2- 172.21.15.88:0/3538372544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415c5c0 0x7ff1c415b920 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4011a00 tx=0x7ff1b4011ec0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.074+0000 7ff1ca5b5640 1 -- 172.21.15.88:0/3538372544 shutdown_connections 2023-10-16T20:58:44.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.074+0000 7ff1ca5b5640 1 --2- 172.21.15.88:0/3538372544 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff1a80664a0 0x7ff1a8068960 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.074+0000 7ff1ca5b5640 1 --2- 172.21.15.88:0/3538372544 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1c41f0d40 0x7ff1c41f11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.074+0000 7ff1ca5b5640 1 --2- 172.21.15.88:0/3538372544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415c5c0 0x7ff1c415b920 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.074+0000 7ff1ca5b5640 1 --2- 172.21.15.88:0/3538372544 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff1b0004d10 0x7ff1c4156480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.074+0000 7ff1ca5b5640 1 -- 172.21.15.88:0/3538372544 >> 172.21.15.88:0/3538372544 conn(0x7ff1c406c170 msgr2=0x7ff1c40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:44.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.074+0000 7ff1ca5b5640 1 -- 172.21.15.88:0/3538372544 shutdown_connections 2023-10-16T20:58:44.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.074+0000 7ff1ca5b5640 1 -- 172.21.15.88:0/3538372544 wait complete. 2023-10-16T20:58:44.095 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":722,"stamp":"2023-10-16T20:58:43.931001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1103869876,"num_objects":10494,"num_object_clones":0,"num_object_copies":41576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10494,"num_whiteouts":0,"num_read":31781,"num_read_kb":882609,"num_write":27510,"num_write_kb":1485755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":115,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34039,"ondisk_log_size":34039,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3214536,"kb_used_data":2277960,"kb_used_omap":12,"kb_used_meta":936499,"kb_avail":751760184,"statfs":{"total":773094113280,"available":769802428416,"internally_reserved":0,"allocated":2332631040,"data_stored":2329499415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":958975545},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,54,40],"upper_bound":128},"perf_stat":{"commit_latency_ms":145,"apply_latency_ms":145,"commit_latency_ns":145000000,"apply_latency_ns":145000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4254655,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":49,"num_write":147,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003434"},"pg_stats":[{"pgid":"2.1f","version":"67'323","reported_seq":411,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:39.469805+0000","last_change":"2023-10-16T20:36:54.763744+0000","last_active":"2023-10-16T20:58:39.469805+0000","last_peered":"2023-10-16T20:58:39.469805+0000","last_clean":"2023-10-16T20:58:39.469805+0000","last_became_active":"2023-10-16T20:36:54.763457+0000","last_became_peered":"2023-10-16T20:36:54.763457+0000","last_unstale":"2023-10-16T20:58:39.469805+0000","last_undegraded":"2023-10-16T20:58:39.469805+0000","last_fullsized":"2023-10-16T20:58:39.469805+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:05.774485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9864502,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":42,"num_read_kb":9548,"num_write":321,"num_write_kb":9823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"67'107","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.869873+0000","last_change":"2023-10-16T20:47:31.588044+0000","last_active":"2023-10-16T20:58:37.869873+0000","last_peered":"2023-10-16T20:58:37.869873+0000","last_clean":"2023-10-16T20:58:37.869873+0000","last_became_active":"2023-10-16T20:36:55.302519+0000","last_became_peered":"2023-10-16T20:36:55.302519+0000","last_unstale":"2023-10-16T20:58:37.869873+0000","last_undegraded":"2023-10-16T20:58:37.869873+0000","last_fullsized":"2023-10-16T20:58:37.869873+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:49.037032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007784165,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'672","reported_seq":3556,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.645718+0000","last_change":"2023-10-16T20:47:31.584576+0000","last_active":"2023-10-16T20:58:22.645718+0000","last_peered":"2023-10-16T20:58:22.645718+0000","last_clean":"2023-10-16T20:58:22.645718+0000","last_became_active":"2023-10-16T20:36:59.466306+0000","last_became_peered":"2023-10-16T20:36:59.466306+0000","last_unstale":"2023-10-16T20:58:22.645718+0000","last_undegraded":"2023-10-16T20:58:22.645718+0000","last_fullsized":"2023-10-16T20:58:22.645718+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:56.295661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073106899999999999,"stat_sum":{"num_bytes":36865036,"num_objects":299,"num_object_clones":0,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":299,"num_whiteouts":0,"num_read":779,"num_read_kb":25449,"num_write":671,"num_write_kb":56816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"67'207","reported_seq":277,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:00.972618+0000","last_change":"2023-10-16T20:36:54.759491+0000","last_active":"2023-10-16T20:58:00.972618+0000","last_peered":"2023-10-16T20:58:00.972618+0000","last_clean":"2023-10-16T20:58:00.972618+0000","last_became_active":"2023-10-16T20:36:54.759177+0000","last_became_peered":"2023-10-16T20:36:54.759177+0000","last_unstale":"2023-10-16T20:58:00.972618+0000","last_undegraded":"2023-10-16T20:58:00.972618+0000","last_fullsized":"2023-10-16T20:58:00.972618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:43:36.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":12288,"num_write":199,"num_write_kb":8312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"67'107","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889862+0000","last_change":"2023-10-16T20:47:31.488681+0000","last_active":"2023-10-16T20:58:37.889862+0000","last_peered":"2023-10-16T20:58:37.889862+0000","last_clean":"2023-10-16T20:58:37.889862+0000","last_became_active":"2023-10-16T20:36:55.291458+0000","last_became_peered":"2023-10-16T20:36:55.291458+0000","last_unstale":"2023-10-16T20:58:37.889862+0000","last_undegraded":"2023-10-16T20:58:37.889862+0000","last_fullsized":"2023-10-16T20:58:37.889862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:03:05.558721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087429459999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'682","reported_seq":3570,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.576451+0000","last_change":"2023-10-16T20:47:31.484150+0000","last_active":"2023-10-16T20:58:22.576451+0000","last_peered":"2023-10-16T20:58:22.576451+0000","last_clean":"2023-10-16T20:58:22.576451+0000","last_became_active":"2023-10-16T20:36:59.302968+0000","last_became_peered":"2023-10-16T20:36:59.302968+0000","last_unstale":"2023-10-16T20:58:22.576451+0000","last_undegraded":"2023-10-16T20:58:22.576451+0000","last_fullsized":"2023-10-16T20:58:22.576451+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:58:43.745409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027985639999999999,"stat_sum":{"num_bytes":24765378,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":783,"num_read_kb":25781,"num_write":678,"num_write_kb":40921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"67'340","reported_seq":437,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:33.688387+0000","last_change":"2023-10-16T20:36:54.762600+0000","last_active":"2023-10-16T20:58:33.688387+0000","last_peered":"2023-10-16T20:58:33.688387+0000","last_clean":"2023-10-16T20:58:33.688387+0000","last_became_active":"2023-10-16T20:36:54.762278+0000","last_became_peered":"2023-10-16T20:36:54.762278+0000","last_unstale":"2023-10-16T20:58:33.688387+0000","last_undegraded":"2023-10-16T20:58:33.688387+0000","last_fullsized":"2023-10-16T20:58:33.688387+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:46:03.836415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":85,"num_read_kb":16492,"num_write":374,"num_write_kb":12523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'96","reported_seq":2197,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.131579+0000","last_change":"2023-10-16T20:47:31.496372+0000","last_active":"2023-10-16T20:58:38.131579+0000","last_peered":"2023-10-16T20:58:38.131579+0000","last_clean":"2023-10-16T20:58:38.131579+0000","last_became_active":"2023-10-16T20:37:37.402987+0000","last_became_peered":"2023-10-16T20:37:37.402987+0000","last_unstale":"2023-10-16T20:58:38.131579+0000","last_undegraded":"2023-10-16T20:58:38.131579+0000","last_fullsized":"2023-10-16T20:58:38.131579+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:12:06.964995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082573900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"67'673","reported_seq":4224,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.678192+0000","last_change":"2023-10-16T20:47:31.485701+0000","last_active":"2023-10-16T20:58:22.678192+0000","last_peered":"2023-10-16T20:58:22.678192+0000","last_clean":"2023-10-16T20:58:22.678192+0000","last_became_active":"2023-10-16T20:36:59.303805+0000","last_became_peered":"2023-10-16T20:36:59.303805+0000","last_unstale":"2023-10-16T20:58:22.678192+0000","last_undegraded":"2023-10-16T20:58:22.678192+0000","last_fullsized":"2023-10-16T20:58:22.678192+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:28:11.659597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041657500000000002,"stat_sum":{"num_bytes":49691093,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":1446,"num_read_kb":31388,"num_write":673,"num_write_kb":52938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"67'172","reported_seq":252,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.691336+0000","last_change":"2023-10-16T20:36:54.749849+0000","last_active":"2023-10-16T20:58:28.691336+0000","last_peered":"2023-10-16T20:58:28.691336+0000","last_clean":"2023-10-16T20:58:28.691336+0000","last_became_active":"2023-10-16T20:36:54.749316+0000","last_became_peered":"2023-10-16T20:36:54.749316+0000","last_unstale":"2023-10-16T20:58:28.691336+0000","last_undegraded":"2023-10-16T20:58:28.691336+0000","last_fullsized":"2023-10-16T20:58:28.691336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:49:29.077126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":170,"num_write_kb":4198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"67'104","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.872186+0000","last_change":"2023-10-16T20:47:31.592091+0000","last_active":"2023-10-16T20:58:37.872186+0000","last_peered":"2023-10-16T20:58:37.872186+0000","last_clean":"2023-10-16T20:58:37.872186+0000","last_became_active":"2023-10-16T20:36:55.302387+0000","last_became_peered":"2023-10-16T20:36:55.302387+0000","last_unstale":"2023-10-16T20:58:37.872186+0000","last_undegraded":"2023-10-16T20:58:37.872186+0000","last_fullsized":"2023-10-16T20:58:37.872186+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:07:35.871205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011585343,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'641","reported_seq":3839,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.036707+0000","last_change":"2023-10-16T20:47:31.486382+0000","last_active":"2023-10-16T20:58:23.036707+0000","last_peered":"2023-10-16T20:58:23.036707+0000","last_clean":"2023-10-16T20:58:23.036707+0000","last_became_active":"2023-10-16T20:36:59.305228+0000","last_became_peered":"2023-10-16T20:36:59.305228+0000","last_unstale":"2023-10-16T20:58:23.036707+0000","last_undegraded":"2023-10-16T20:58:23.036707+0000","last_fullsized":"2023-10-16T20:58:23.036707+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":641,"log_dups_size":0,"ondisk_log_size":641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:32:37.261529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047877249999999996,"stat_sum":{"num_bytes":34900728,"num_objects":310,"num_object_clones":0,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":1093,"num_read_kb":21879,"num_write":638,"num_write_kb":46703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"67'104","reported_seq":2214,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.863809+0000","last_change":"2023-10-16T20:47:31.592641+0000","last_active":"2023-10-16T20:58:37.863809+0000","last_peered":"2023-10-16T20:58:37.863809+0000","last_clean":"2023-10-16T20:58:37.863809+0000","last_became_active":"2023-10-16T20:36:55.306427+0000","last_became_peered":"2023-10-16T20:36:55.306427+0000","last_unstale":"2023-10-16T20:58:37.863809+0000","last_undegraded":"2023-10-16T20:58:37.863809+0000","last_fullsized":"2023-10-16T20:58:37.863809+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:17:46.390047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013854674000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"67'187","reported_seq":258,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.360075+0000","last_change":"2023-10-16T20:36:54.768375+0000","last_active":"2023-10-16T20:58:05.360075+0000","last_peered":"2023-10-16T20:58:05.360075+0000","last_clean":"2023-10-16T20:58:05.360075+0000","last_became_active":"2023-10-16T20:36:54.767996+0000","last_became_peered":"2023-10-16T20:36:54.767996+0000","last_unstale":"2023-10-16T20:58:05.360075+0000","last_undegraded":"2023-10-16T20:58:05.360075+0000","last_fullsized":"2023-10-16T20:58:05.360075+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:55.303942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":8202,"num_write":173,"num_write_kb":8293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"67'648","reported_seq":4473,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.721394+0000","last_change":"2023-10-16T20:47:31.487925+0000","last_active":"2023-10-16T20:58:22.721394+0000","last_peered":"2023-10-16T20:58:22.721394+0000","last_clean":"2023-10-16T20:58:22.721394+0000","last_became_active":"2023-10-16T20:36:59.306640+0000","last_became_peered":"2023-10-16T20:36:59.306640+0000","last_unstale":"2023-10-16T20:58:22.721394+0000","last_undegraded":"2023-10-16T20:58:22.721394+0000","last_fullsized":"2023-10-16T20:58:22.721394+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:27:48.041001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054537470000000001,"stat_sum":{"num_bytes":41205181,"num_objects":318,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":1719,"num_read_kb":37802,"num_write":648,"num_write_kb":48776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"67'101","reported_seq":2210,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.891454+0000","last_change":"2023-10-16T20:47:31.587617+0000","last_active":"2023-10-16T20:58:37.891454+0000","last_peered":"2023-10-16T20:58:37.891454+0000","last_clean":"2023-10-16T20:58:37.891454+0000","last_became_active":"2023-10-16T20:36:55.275085+0000","last_became_peered":"2023-10-16T20:36:55.275085+0000","last_unstale":"2023-10-16T20:58:37.891454+0000","last_undegraded":"2023-10-16T20:58:37.891454+0000","last_fullsized":"2023-10-16T20:58:37.891454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:15.882497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087599630000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'182","reported_seq":251,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:41.511040+0000","last_change":"2023-10-16T20:36:54.756174+0000","last_active":"2023-10-16T20:58:41.511040+0000","last_peered":"2023-10-16T20:58:41.511040+0000","last_clean":"2023-10-16T20:58:41.511040+0000","last_became_active":"2023-10-16T20:36:54.755872+0000","last_became_peered":"2023-10-16T20:36:54.755872+0000","last_unstale":"2023-10-16T20:58:41.511040+0000","last_undegraded":"2023-10-16T20:58:41.511040+0000","last_fullsized":"2023-10-16T20:58:41.511040+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:32.741671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":10,"num_write":281,"num_write_kb":4257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"67'699","reported_seq":3454,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.718374+0000","last_change":"2023-10-16T20:47:31.486593+0000","last_active":"2023-10-16T20:58:22.718374+0000","last_peered":"2023-10-16T20:58:22.718374+0000","last_clean":"2023-10-16T20:58:22.718374+0000","last_became_active":"2023-10-16T20:36:59.302919+0000","last_became_peered":"2023-10-16T20:36:59.302919+0000","last_unstale":"2023-10-16T20:58:22.718374+0000","last_undegraded":"2023-10-16T20:58:22.718374+0000","last_fullsized":"2023-10-16T20:58:22.718374+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:48.779648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046546790000000001,"stat_sum":{"num_bytes":20793675,"num_objects":334,"num_object_clones":0,"num_object_copies":1336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":334,"num_whiteouts":0,"num_read":649,"num_read_kb":16394,"num_write":696,"num_write_kb":45235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"67'106","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.886436+0000","last_change":"2023-10-16T20:47:31.586816+0000","last_active":"2023-10-16T20:58:37.886436+0000","last_peered":"2023-10-16T20:58:37.886436+0000","last_clean":"2023-10-16T20:58:37.886436+0000","last_became_active":"2023-10-16T20:36:55.270455+0000","last_became_peered":"2023-10-16T20:36:55.270455+0000","last_unstale":"2023-10-16T20:58:37.886436+0000","last_undegraded":"2023-10-16T20:58:37.886436+0000","last_fullsized":"2023-10-16T20:58:37.886436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:37.316278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.008577889,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"66'49","reported_seq":127,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.707520+0000","last_change":"2023-10-16T20:36:54.744128+0000","last_active":"2023-10-16T20:58:28.707520+0000","last_peered":"2023-10-16T20:58:28.707520+0000","last_clean":"2023-10-16T20:58:28.707520+0000","last_became_active":"2023-10-16T20:36:54.743805+0000","last_became_peered":"2023-10-16T20:36:54.743805+0000","last_unstale":"2023-10-16T20:58:28.707520+0000","last_undegraded":"2023-10-16T20:58:28.707520+0000","last_fullsized":"2023-10-16T20:58:28.707520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:54:35.819820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":55,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'651","reported_seq":3692,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.352813+0000","last_change":"2023-10-16T20:47:31.485640+0000","last_active":"2023-10-16T20:58:22.352813+0000","last_peered":"2023-10-16T20:58:22.352813+0000","last_clean":"2023-10-16T20:58:22.352813+0000","last_became_active":"2023-10-16T20:36:59.465172+0000","last_became_peered":"2023-10-16T20:36:59.465172+0000","last_unstale":"2023-10-16T20:58:22.352813+0000","last_undegraded":"2023-10-16T20:58:22.352813+0000","last_fullsized":"2023-10-16T20:58:22.352813+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:18:14.804407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040107390000000001,"stat_sum":{"num_bytes":25409516,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":935,"num_read_kb":27643,"num_write":650,"num_write_kb":33262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"67'103","reported_seq":2213,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.849592+0000","last_change":"2023-10-16T20:47:31.487014+0000","last_active":"2023-10-16T20:58:37.849592+0000","last_peered":"2023-10-16T20:58:37.849592+0000","last_clean":"2023-10-16T20:58:37.849592+0000","last_became_active":"2023-10-16T20:36:55.278568+0000","last_became_peered":"2023-10-16T20:36:55.278568+0000","last_unstale":"2023-10-16T20:58:37.849592+0000","last_undegraded":"2023-10-16T20:58:37.849592+0000","last_fullsized":"2023-10-16T20:58:37.849592+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:01:00.125646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050399599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"66'287","reported_seq":358,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:01.079180+0000","last_change":"2023-10-16T20:36:54.758778+0000","last_active":"2023-10-16T20:58:01.079180+0000","last_peered":"2023-10-16T20:58:01.079180+0000","last_clean":"2023-10-16T20:58:01.079180+0000","last_became_active":"2023-10-16T20:36:54.758496+0000","last_became_peered":"2023-10-16T20:36:54.758496+0000","last_unstale":"2023-10-16T20:58:01.079180+0000","last_undegraded":"2023-10-16T20:58:01.079180+0000","last_fullsized":"2023-10-16T20:58:01.079180+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:36:00.515106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":12294,"num_write":291,"num_write_kb":4310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"67'586","reported_seq":3861,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.527756+0000","last_change":"2023-10-16T20:47:31.488200+0000","last_active":"2023-10-16T20:58:22.527756+0000","last_peered":"2023-10-16T20:58:22.527756+0000","last_clean":"2023-10-16T20:58:22.527756+0000","last_became_active":"2023-10-16T20:36:59.309068+0000","last_became_peered":"2023-10-16T20:36:59.309068+0000","last_unstale":"2023-10-16T20:58:22.527756+0000","last_undegraded":"2023-10-16T20:58:22.527756+0000","last_fullsized":"2023-10-16T20:58:22.527756+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:22:11.244977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055482439999999999,"stat_sum":{"num_bytes":36900597,"num_objects":303,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1169,"num_read_kb":30992,"num_write":585,"num_write_kb":40444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'101","reported_seq":2211,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889241+0000","last_change":"2023-10-16T20:47:31.492407+0000","last_active":"2023-10-16T20:58:37.889241+0000","last_peered":"2023-10-16T20:58:37.889241+0000","last_clean":"2023-10-16T20:58:37.889241+0000","last_became_active":"2023-10-16T20:36:55.272840+0000","last_became_peered":"2023-10-16T20:36:55.272840+0000","last_unstale":"2023-10-16T20:58:37.889241+0000","last_undegraded":"2023-10-16T20:58:37.889241+0000","last_fullsized":"2023-10-16T20:58:37.889241+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:35:42.200237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011897911000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"30'8","reported_seq":88,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.710153+0000","last_change":"2023-10-16T20:36:54.760681+0000","last_active":"2023-10-16T20:58:28.710153+0000","last_peered":"2023-10-16T20:58:28.710153+0000","last_clean":"2023-10-16T20:58:28.710153+0000","last_became_active":"2023-10-16T20:36:54.760397+0000","last_became_peered":"2023-10-16T20:36:54.760397+0000","last_unstale":"2023-10-16T20:58:28.710153+0000","last_undegraded":"2023-10-16T20:58:28.710153+0000","last_fullsized":"2023-10-16T20:58:28.710153+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:47:16.510382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"67'710","reported_seq":3798,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.025282+0000","last_change":"2023-10-16T20:47:31.485980+0000","last_active":"2023-10-16T20:58:23.025282+0000","last_peered":"2023-10-16T20:58:23.025282+0000","last_clean":"2023-10-16T20:58:23.025282+0000","last_became_active":"2023-10-16T20:36:59.465152+0000","last_became_peered":"2023-10-16T20:36:59.465152+0000","last_unstale":"2023-10-16T20:58:23.025282+0000","last_undegraded":"2023-10-16T20:58:23.025282+0000","last_fullsized":"2023-10-16T20:58:23.025282+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:48.800573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044307399999999999,"stat_sum":{"num_bytes":25809469,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":981,"num_read_kb":15497,"num_write":706,"num_write_kb":50141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'106","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.882745+0000","last_change":"2023-10-16T20:47:31.587090+0000","last_active":"2023-10-16T20:58:37.882745+0000","last_peered":"2023-10-16T20:58:37.882745+0000","last_clean":"2023-10-16T20:58:37.882745+0000","last_became_active":"2023-10-16T20:36:55.285566+0000","last_became_peered":"2023-10-16T20:36:55.285566+0000","last_unstale":"2023-10-16T20:58:37.882745+0000","last_undegraded":"2023-10-16T20:58:37.882745+0000","last_fullsized":"2023-10-16T20:58:37.882745+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:18.835080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085500479999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'101","reported_seq":177,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.657997+0000","last_change":"2023-10-16T20:36:54.758062+0000","last_active":"2023-10-16T20:58:37.657997+0000","last_peered":"2023-10-16T20:58:37.657997+0000","last_clean":"2023-10-16T20:58:37.657997+0000","last_became_active":"2023-10-16T20:36:54.757796+0000","last_became_peered":"2023-10-16T20:36:54.757796+0000","last_unstale":"2023-10-16T20:58:37.657997+0000","last_undegraded":"2023-10-16T20:58:37.657997+0000","last_fullsized":"2023-10-16T20:58:37.657997+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:02:09.953795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":254,"num_write_kb":1037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"67'686","reported_seq":3331,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.536581+0000","last_change":"2023-10-16T20:47:31.582989+0000","last_active":"2023-10-16T20:58:22.536581+0000","last_peered":"2023-10-16T20:58:22.536581+0000","last_clean":"2023-10-16T20:58:22.536581+0000","last_became_active":"2023-10-16T20:36:59.465221+0000","last_became_peered":"2023-10-16T20:36:59.465221+0000","last_unstale":"2023-10-16T20:58:22.536581+0000","last_undegraded":"2023-10-16T20:58:22.536581+0000","last_fullsized":"2023-10-16T20:58:22.536581+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:29:08.900666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059334330000000001,"stat_sum":{"num_bytes":20326999,"num_objects":343,"num_object_clones":0,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":538,"num_read_kb":13756,"num_write":685,"num_write_kb":32496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"67'101","reported_seq":2211,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.818833+0000","last_change":"2023-10-16T20:47:31.489294+0000","last_active":"2023-10-16T20:58:37.818833+0000","last_peered":"2023-10-16T20:58:37.818833+0000","last_clean":"2023-10-16T20:58:37.818833+0000","last_became_active":"2023-10-16T20:36:55.274432+0000","last_became_peered":"2023-10-16T20:36:55.274432+0000","last_unstale":"2023-10-16T20:58:37.818833+0000","last_undegraded":"2023-10-16T20:58:37.818833+0000","last_fullsized":"2023-10-16T20:58:37.818833+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:50:04.713610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057461259999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"66'34","reported_seq":109,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:01.080176+0000","last_change":"2023-10-16T20:36:54.762356+0000","last_active":"2023-10-16T20:58:01.080176+0000","last_peered":"2023-10-16T20:58:01.080176+0000","last_clean":"2023-10-16T20:58:01.080176+0000","last_became_active":"2023-10-16T20:36:54.762167+0000","last_became_peered":"2023-10-16T20:36:54.762167+0000","last_unstale":"2023-10-16T20:58:01.080176+0000","last_undegraded":"2023-10-16T20:58:01.080176+0000","last_fullsized":"2023-10-16T20:58:01.080176+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:45.164625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":34,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"67'725","reported_seq":3715,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.031400+0000","last_change":"2023-10-16T20:47:31.489099+0000","last_active":"2023-10-16T20:58:23.031400+0000","last_peered":"2023-10-16T20:58:23.031400+0000","last_clean":"2023-10-16T20:58:23.031400+0000","last_became_active":"2023-10-16T20:36:59.466876+0000","last_became_peered":"2023-10-16T20:36:59.466876+0000","last_unstale":"2023-10-16T20:58:23.031400+0000","last_undegraded":"2023-10-16T20:58:23.031400+0000","last_fullsized":"2023-10-16T20:58:23.031400+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":725,"log_dups_size":0,"ondisk_log_size":725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:45:55.969279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009058969,"stat_sum":{"num_bytes":44901233,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":884,"num_read_kb":20657,"num_write":720,"num_write_kb":60576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"67'98","reported_seq":2208,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.895976+0000","last_change":"2023-10-16T20:47:31.594422+0000","last_active":"2023-10-16T20:58:37.895976+0000","last_peered":"2023-10-16T20:58:37.895976+0000","last_clean":"2023-10-16T20:58:37.895976+0000","last_became_active":"2023-10-16T20:36:55.293228+0000","last_became_peered":"2023-10-16T20:36:55.293228+0000","last_unstale":"2023-10-16T20:58:37.895976+0000","last_undegraded":"2023-10-16T20:58:37.895976+0000","last_fullsized":"2023-10-16T20:58:37.895976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:33:58.039426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013567460999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"66'210","reported_seq":278,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:47.652674+0000","last_change":"2023-10-16T20:36:54.757588+0000","last_active":"2023-10-16T20:57:47.652674+0000","last_peered":"2023-10-16T20:57:47.652674+0000","last_clean":"2023-10-16T20:57:47.652674+0000","last_became_active":"2023-10-16T20:36:54.757308+0000","last_became_peered":"2023-10-16T20:36:54.757308+0000","last_unstale":"2023-10-16T20:57:47.652674+0000","last_undegraded":"2023-10-16T20:57:47.652674+0000","last_fullsized":"2023-10-16T20:57:47.652674+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:51.298239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":3049,"num_write":207,"num_write_kb":4195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"67'671","reported_seq":3425,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.522842+0000","last_change":"2023-10-16T20:47:31.485489+0000","last_active":"2023-10-16T20:58:22.522842+0000","last_peered":"2023-10-16T20:58:22.522842+0000","last_clean":"2023-10-16T20:58:22.522842+0000","last_became_active":"2023-10-16T20:36:59.303224+0000","last_became_peered":"2023-10-16T20:36:59.303224+0000","last_unstale":"2023-10-16T20:58:22.522842+0000","last_undegraded":"2023-10-16T20:58:22.522842+0000","last_fullsized":"2023-10-16T20:58:22.522842+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:03:55.965341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038422399999999998,"stat_sum":{"num_bytes":21817454,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":647,"num_read_kb":9306,"num_write":673,"num_write_kb":37999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"67'120","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889222+0000","last_change":"2023-10-16T20:47:31.487515+0000","last_active":"2023-10-16T20:58:37.889222+0000","last_peered":"2023-10-16T20:58:37.889222+0000","last_clean":"2023-10-16T20:58:37.889222+0000","last_became_active":"2023-10-16T20:36:55.277872+0000","last_became_peered":"2023-10-16T20:36:55.277872+0000","last_unstale":"2023-10-16T20:58:37.889222+0000","last_undegraded":"2023-10-16T20:58:37.889222+0000","last_fullsized":"2023-10-16T20:58:37.889222+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:46.751931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052323379999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"66'532","reported_seq":616,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.670124+0000","last_change":"2023-10-16T20:36:54.740644+0000","last_active":"2023-10-16T20:58:05.670124+0000","last_peered":"2023-10-16T20:58:05.670124+0000","last_clean":"2023-10-16T20:58:05.670124+0000","last_became_active":"2023-10-16T20:36:54.740340+0000","last_became_peered":"2023-10-16T20:36:54.740340+0000","last_unstale":"2023-10-16T20:58:05.670124+0000","last_undegraded":"2023-10-16T20:58:05.670124+0000","last_fullsized":"2023-10-16T20:58:05.670124+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:22.221727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21285,"num_write":524,"num_write_kb":16645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'623","reported_seq":3877,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.676798+0000","last_change":"2023-10-16T20:47:31.489650+0000","last_active":"2023-10-16T20:58:22.676798+0000","last_peered":"2023-10-16T20:58:22.676798+0000","last_clean":"2023-10-16T20:58:22.676798+0000","last_became_active":"2023-10-16T20:36:59.294548+0000","last_became_peered":"2023-10-16T20:36:59.294548+0000","last_unstale":"2023-10-16T20:58:22.676798+0000","last_undegraded":"2023-10-16T20:58:22.676798+0000","last_fullsized":"2023-10-16T20:58:22.676798+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:12.441064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062280529999999999,"stat_sum":{"num_bytes":29651112,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":1147,"num_read_kb":23932,"num_write":623,"num_write_kb":40690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"67'89","reported_seq":2199,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.807118+0000","last_change":"2023-10-16T20:47:31.484434+0000","last_active":"2023-10-16T20:58:37.807118+0000","last_peered":"2023-10-16T20:58:37.807118+0000","last_clean":"2023-10-16T20:58:37.807118+0000","last_became_active":"2023-10-16T20:36:55.269749+0000","last_became_peered":"2023-10-16T20:36:55.269749+0000","last_unstale":"2023-10-16T20:58:37.807118+0000","last_undegraded":"2023-10-16T20:58:37.807118+0000","last_fullsized":"2023-10-16T20:58:37.807118+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:21:13.945910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003009004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'188","reported_seq":376,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.714114+0000","last_change":"2023-10-16T20:36:54.757903+0000","last_active":"2023-10-16T20:58:37.714114+0000","last_peered":"2023-10-16T20:58:37.714114+0000","last_clean":"2023-10-16T20:58:37.714114+0000","last_became_active":"2023-10-16T20:36:54.757549+0000","last_became_peered":"2023-10-16T20:36:54.757549+0000","last_unstale":"2023-10-16T20:58:37.714114+0000","last_undegraded":"2023-10-16T20:58:37.714114+0000","last_fullsized":"2023-10-16T20:58:37.714114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:02:42.090848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1409828,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":124,"num_read_kb":239,"num_write":184,"num_write_kb":1568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'647","reported_seq":3934,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.025459+0000","last_change":"2023-10-16T20:47:31.487517+0000","last_active":"2023-10-16T20:58:23.025459+0000","last_peered":"2023-10-16T20:58:23.025459+0000","last_clean":"2023-10-16T20:58:23.025459+0000","last_became_active":"2023-10-16T20:36:59.301693+0000","last_became_peered":"2023-10-16T20:36:59.301693+0000","last_unstale":"2023-10-16T20:58:23.025459+0000","last_undegraded":"2023-10-16T20:58:23.025459+0000","last_fullsized":"2023-10-16T20:58:23.025459+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:04:52.612265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056692870000000003,"stat_sum":{"num_bytes":29916858,"num_objects":298,"num_object_clones":0,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":298,"num_whiteouts":0,"num_read":1180,"num_read_kb":25235,"num_write":643,"num_write_kb":45892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"67'104","reported_seq":2214,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888945+0000","last_change":"2023-10-16T20:47:31.487504+0000","last_active":"2023-10-16T20:58:37.888945+0000","last_peered":"2023-10-16T20:58:37.888945+0000","last_clean":"2023-10-16T20:58:37.888945+0000","last_became_active":"2023-10-16T20:36:55.276649+0000","last_became_peered":"2023-10-16T20:36:55.276649+0000","last_unstale":"2023-10-16T20:58:37.888945+0000","last_undegraded":"2023-10-16T20:58:37.888945+0000","last_fullsized":"2023-10-16T20:58:37.888945+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:12:33.903227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076292809999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'128","reported_seq":201,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:01.079234+0000","last_change":"2023-10-16T20:36:54.753592+0000","last_active":"2023-10-16T20:58:01.079234+0000","last_peered":"2023-10-16T20:58:01.079234+0000","last_clean":"2023-10-16T20:58:01.079234+0000","last_became_active":"2023-10-16T20:36:54.753237+0000","last_became_peered":"2023-10-16T20:36:54.753237+0000","last_unstale":"2023-10-16T20:58:01.079234+0000","last_undegraded":"2023-10-16T20:58:01.079234+0000","last_fullsized":"2023-10-16T20:58:01.079234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:53.005979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":20486,"num_write":119,"num_write_kb":8247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'673","reported_seq":3483,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.284780+0000","last_change":"2023-10-16T20:47:31.491504+0000","last_active":"2023-10-16T20:58:23.284780+0000","last_peered":"2023-10-16T20:58:23.284780+0000","last_clean":"2023-10-16T20:58:23.284780+0000","last_became_active":"2023-10-16T20:36:59.466708+0000","last_became_peered":"2023-10-16T20:36:59.466708+0000","last_unstale":"2023-10-16T20:58:23.284780+0000","last_undegraded":"2023-10-16T20:58:23.284780+0000","last_fullsized":"2023-10-16T20:58:23.284780+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:33:22.349924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011259905000000001,"stat_sum":{"num_bytes":19802424,"num_objects":318,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":703,"num_read_kb":14564,"num_write":670,"num_write_kb":31986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"67'99","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.897919+0000","last_change":"2023-10-16T20:47:31.585286+0000","last_active":"2023-10-16T20:58:37.897919+0000","last_peered":"2023-10-16T20:58:37.897919+0000","last_clean":"2023-10-16T20:58:37.897919+0000","last_became_active":"2023-10-16T20:36:55.294610+0000","last_became_peered":"2023-10-16T20:36:55.294610+0000","last_unstale":"2023-10-16T20:58:37.897919+0000","last_undegraded":"2023-10-16T20:58:37.897919+0000","last_fullsized":"2023-10-16T20:58:37.897919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:34:05.113883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075091560000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"66'7","reported_seq":83,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:00.736127+0000","last_change":"2023-10-16T20:36:54.741191+0000","last_active":"2023-10-16T20:58:00.736127+0000","last_peered":"2023-10-16T20:58:00.736127+0000","last_clean":"2023-10-16T20:58:00.736127+0000","last_became_active":"2023-10-16T20:36:54.740944+0000","last_became_peered":"2023-10-16T20:36:54.740944+0000","last_unstale":"2023-10-16T20:58:00.736127+0000","last_undegraded":"2023-10-16T20:58:00.736127+0000","last_fullsized":"2023-10-16T20:58:00.736127+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:05.961850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":23,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":219,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:56:52.651677+0000","last_change":"2023-10-16T20:36:46.390001+0000","last_active":"2023-10-16T20:56:52.651677+0000","last_peered":"2023-10-16T20:56:52.651677+0000","last_clean":"2023-10-16T20:56:52.651677+0000","last_became_active":"2023-10-16T20:36:46.388824+0000","last_became_peered":"2023-10-16T20:36:46.388824+0000","last_unstale":"2023-10-16T20:56:52.651677+0000","last_undegraded":"2023-10-16T20:56:52.651677+0000","last_fullsized":"2023-10-16T20:56:52.651677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_clean_scrub_stamp":"2023-10-16T20:36:44.575506+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:31.004318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"67'669","reported_seq":3425,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.621226+0000","last_change":"2023-10-16T20:47:31.593007+0000","last_active":"2023-10-16T20:58:22.621226+0000","last_peered":"2023-10-16T20:58:22.621226+0000","last_clean":"2023-10-16T20:58:22.621226+0000","last_became_active":"2023-10-16T20:36:59.468082+0000","last_became_peered":"2023-10-16T20:36:59.468082+0000","last_unstale":"2023-10-16T20:58:22.621226+0000","last_undegraded":"2023-10-16T20:58:22.621226+0000","last_fullsized":"2023-10-16T20:58:22.621226+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":669,"log_dups_size":0,"ondisk_log_size":669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:49:06.585943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013154572,"stat_sum":{"num_bytes":30910053,"num_objects":362,"num_object_clones":0,"num_object_copies":1448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":649,"num_read_kb":15567,"num_write":666,"num_write_kb":42817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'110","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.884585+0000","last_change":"2023-10-16T20:47:31.488408+0000","last_active":"2023-10-16T20:58:37.884585+0000","last_peered":"2023-10-16T20:58:37.884585+0000","last_clean":"2023-10-16T20:58:37.884585+0000","last_became_active":"2023-10-16T20:37:37.409004+0000","last_became_peered":"2023-10-16T20:37:37.409004+0000","last_unstale":"2023-10-16T20:58:37.884585+0000","last_undegraded":"2023-10-16T20:58:37.884585+0000","last_fullsized":"2023-10-16T20:58:37.884585+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:21:07.245091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060805590000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"66'68","reported_seq":138,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:19.377575+0000","last_change":"2023-10-16T20:36:54.746600+0000","last_active":"2023-10-16T20:57:19.377575+0000","last_peered":"2023-10-16T20:57:19.377575+0000","last_clean":"2023-10-16T20:57:19.377575+0000","last_became_active":"2023-10-16T20:36:54.746247+0000","last_became_peered":"2023-10-16T20:36:54.746247+0000","last_unstale":"2023-10-16T20:57:19.377575+0000","last_undegraded":"2023-10-16T20:57:19.377575+0000","last_fullsized":"2023-10-16T20:57:19.377575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:28:57.949487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":7042,"num_write":68,"num_write_kb":4224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"67'680","reported_seq":3679,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.607038+0000","last_change":"2023-10-16T20:47:31.590576+0000","last_active":"2023-10-16T20:58:22.607038+0000","last_peered":"2023-10-16T20:58:22.607038+0000","last_clean":"2023-10-16T20:58:22.607038+0000","last_became_active":"2023-10-16T20:37:37.467402+0000","last_became_peered":"2023-10-16T20:37:37.467402+0000","last_unstale":"2023-10-16T20:58:22.607038+0000","last_undegraded":"2023-10-16T20:58:22.607038+0000","last_fullsized":"2023-10-16T20:58:22.607038+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":680,"log_dups_size":0,"ondisk_log_size":680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:32:26.907296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012017773000000001,"stat_sum":{"num_bytes":36935626,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":884,"num_read_kb":21394,"num_write":679,"num_write_kb":40520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"67'106","reported_seq":2215,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.895855+0000","last_change":"2023-10-16T20:47:31.589500+0000","last_active":"2023-10-16T20:58:37.895855+0000","last_peered":"2023-10-16T20:58:37.895855+0000","last_clean":"2023-10-16T20:58:37.895855+0000","last_became_active":"2023-10-16T20:36:55.300958+0000","last_became_peered":"2023-10-16T20:36:55.300958+0000","last_unstale":"2023-10-16T20:58:37.895855+0000","last_undegraded":"2023-10-16T20:58:37.895855+0000","last_fullsized":"2023-10-16T20:58:37.895855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:29:17.323936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009734431,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"67'314","reported_seq":386,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.865738+0000","last_change":"2023-10-16T20:36:54.753770+0000","last_active":"2023-10-16T20:58:37.865738+0000","last_peered":"2023-10-16T20:58:37.865738+0000","last_clean":"2023-10-16T20:58:37.865738+0000","last_became_active":"2023-10-16T20:36:54.753443+0000","last_became_peered":"2023-10-16T20:36:54.753443+0000","last_unstale":"2023-10-16T20:58:37.865738+0000","last_undegraded":"2023-10-16T20:58:37.865738+0000","last_fullsized":"2023-10-16T20:58:37.865738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:21:26.392480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9664034,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":15859,"num_write":307,"num_write_kb":9612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'687","reported_seq":3976,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.741627+0000","last_change":"2023-10-16T20:47:31.492598+0000","last_active":"2023-10-16T20:58:22.741627+0000","last_peered":"2023-10-16T20:58:22.741627+0000","last_clean":"2023-10-16T20:58:22.741627+0000","last_became_active":"2023-10-16T20:36:59.302869+0000","last_became_peered":"2023-10-16T20:36:59.302869+0000","last_unstale":"2023-10-16T20:58:22.741627+0000","last_undegraded":"2023-10-16T20:58:22.741627+0000","last_fullsized":"2023-10-16T20:58:22.741627+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:37:25.463785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012136565,"stat_sum":{"num_bytes":23637013,"num_objects":352,"num_object_clones":0,"num_object_copies":1408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":1183,"num_read_kb":18772,"num_write":687,"num_write_kb":27489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"67'100","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.892995+0000","last_change":"2023-10-16T20:47:31.587528+0000","last_active":"2023-10-16T20:58:37.892995+0000","last_peered":"2023-10-16T20:58:37.892995+0000","last_clean":"2023-10-16T20:58:37.892995+0000","last_became_active":"2023-10-16T20:36:55.291633+0000","last_became_peered":"2023-10-16T20:36:55.291633+0000","last_unstale":"2023-10-16T20:58:37.892995+0000","last_undegraded":"2023-10-16T20:58:37.892995+0000","last_fullsized":"2023-10-16T20:58:37.892995+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:08.345584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085978630000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'120","reported_seq":195,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.767053+0000","last_change":"2023-10-16T20:36:54.748196+0000","last_active":"2023-10-16T20:58:23.767053+0000","last_peered":"2023-10-16T20:58:23.767053+0000","last_clean":"2023-10-16T20:58:23.767053+0000","last_became_active":"2023-10-16T20:36:54.747704+0000","last_became_peered":"2023-10-16T20:36:54.747704+0000","last_unstale":"2023-10-16T20:58:23.767053+0000","last_undegraded":"2023-10-16T20:58:23.767053+0000","last_fullsized":"2023-10-16T20:58:23.767053+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:55.471916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":116,"num_write_kb":4228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"67'663","reported_seq":3386,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.098033+0000","last_change":"2023-10-16T20:47:31.595065+0000","last_active":"2023-10-16T20:58:23.098033+0000","last_peered":"2023-10-16T20:58:23.098033+0000","last_clean":"2023-10-16T20:58:23.098033+0000","last_became_active":"2023-10-16T20:36:59.467291+0000","last_became_peered":"2023-10-16T20:36:59.467291+0000","last_unstale":"2023-10-16T20:58:23.098033+0000","last_undegraded":"2023-10-16T20:58:23.098033+0000","last_fullsized":"2023-10-16T20:58:23.098033+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:15.723182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0131915,"stat_sum":{"num_bytes":23136394,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":616,"num_read_kb":10065,"num_write":660,"num_write_kb":35181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"67'80","reported_seq":2189,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.876288+0000","last_change":"2023-10-16T20:47:31.589747+0000","last_active":"2023-10-16T20:58:37.876288+0000","last_peered":"2023-10-16T20:58:37.876288+0000","last_clean":"2023-10-16T20:58:37.876288+0000","last_became_active":"2023-10-16T20:36:55.290998+0000","last_became_peered":"2023-10-16T20:36:55.290998+0000","last_unstale":"2023-10-16T20:58:37.876288+0000","last_undegraded":"2023-10-16T20:58:37.876288+0000","last_fullsized":"2023-10-16T20:58:37.876288+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:52:52.835051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082191850000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'91","reported_seq":159,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:33.484841+0000","last_change":"2023-10-16T20:36:54.763818+0000","last_active":"2023-10-16T20:58:33.484841+0000","last_peered":"2023-10-16T20:58:33.484841+0000","last_clean":"2023-10-16T20:58:33.484841+0000","last_became_active":"2023-10-16T20:36:54.761337+0000","last_became_peered":"2023-10-16T20:36:54.761337+0000","last_unstale":"2023-10-16T20:58:33.484841+0000","last_undegraded":"2023-10-16T20:58:33.484841+0000","last_fullsized":"2023-10-16T20:58:33.484841+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:03.093189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":5,"num_write":137,"num_write_kb":4352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"67'666","reported_seq":3840,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.018714+0000","last_change":"2023-10-16T20:47:31.487063+0000","last_active":"2023-10-16T20:58:23.018714+0000","last_peered":"2023-10-16T20:58:23.018714+0000","last_clean":"2023-10-16T20:58:23.018714+0000","last_became_active":"2023-10-16T20:37:37.468309+0000","last_became_peered":"2023-10-16T20:37:37.468309+0000","last_unstale":"2023-10-16T20:58:23.018714+0000","last_undegraded":"2023-10-16T20:58:23.018714+0000","last_fullsized":"2023-10-16T20:58:23.018714+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:13.271803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058030460000000001,"stat_sum":{"num_bytes":20758781,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":1058,"num_read_kb":21621,"num_write":667,"num_write_kb":32853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"67'107","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.887159+0000","last_change":"2023-10-16T20:47:31.589398+0000","last_active":"2023-10-16T20:58:37.887159+0000","last_peered":"2023-10-16T20:58:37.887159+0000","last_clean":"2023-10-16T20:58:37.887159+0000","last_became_active":"2023-10-16T20:36:55.288354+0000","last_became_peered":"2023-10-16T20:36:55.288354+0000","last_unstale":"2023-10-16T20:58:37.887159+0000","last_undegraded":"2023-10-16T20:58:37.887159+0000","last_fullsized":"2023-10-16T20:58:37.887159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:09.359239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076786709999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"67'333","reported_seq":413,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.977642+0000","last_change":"2023-10-16T20:36:54.738216+0000","last_active":"2023-10-16T20:58:05.977642+0000","last_peered":"2023-10-16T20:58:05.977642+0000","last_clean":"2023-10-16T20:58:05.977642+0000","last_became_active":"2023-10-16T20:36:54.737876+0000","last_became_peered":"2023-10-16T20:36:54.737876+0000","last_unstale":"2023-10-16T20:58:05.977642+0000","last_undegraded":"2023-10-16T20:58:05.977642+0000","last_fullsized":"2023-10-16T20:58:05.977642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:28.214601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":39,"num_read_kb":4134,"num_write":333,"num_write_kb":8414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"67'666","reported_seq":3665,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.692242+0000","last_change":"2023-10-16T20:47:31.490637+0000","last_active":"2023-10-16T20:58:22.692242+0000","last_peered":"2023-10-16T20:58:22.692242+0000","last_clean":"2023-10-16T20:58:22.692242+0000","last_became_active":"2023-10-16T20:36:59.306801+0000","last_became_peered":"2023-10-16T20:36:59.306801+0000","last_unstale":"2023-10-16T20:58:22.692242+0000","last_undegraded":"2023-10-16T20:58:22.692242+0000","last_fullsized":"2023-10-16T20:58:22.692242+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:43:08.907279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010585667,"stat_sum":{"num_bytes":20444029,"num_objects":301,"num_object_clones":0,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":893,"num_read_kb":19660,"num_write":666,"num_write_kb":28474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"67'93","reported_seq":2202,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.873494+0000","last_change":"2023-10-16T20:47:31.590902+0000","last_active":"2023-10-16T20:58:37.873494+0000","last_peered":"2023-10-16T20:58:37.873494+0000","last_clean":"2023-10-16T20:58:37.873494+0000","last_became_active":"2023-10-16T20:36:55.287066+0000","last_became_peered":"2023-10-16T20:36:55.287066+0000","last_unstale":"2023-10-16T20:58:37.873494+0000","last_undegraded":"2023-10-16T20:58:37.873494+0000","last_fullsized":"2023-10-16T20:58:37.873494+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:36.165939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012121162,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"66'101","reported_seq":172,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:57.653854+0000","last_change":"2023-10-16T20:36:54.764693+0000","last_active":"2023-10-16T20:57:57.653854+0000","last_peered":"2023-10-16T20:57:57.653854+0000","last_clean":"2023-10-16T20:57:57.653854+0000","last_became_active":"2023-10-16T20:36:54.764417+0000","last_became_peered":"2023-10-16T20:36:54.764417+0000","last_unstale":"2023-10-16T20:57:57.653854+0000","last_undegraded":"2023-10-16T20:57:57.653854+0000","last_fullsized":"2023-10-16T20:57:57.653854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:56.653149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":4110,"num_write":99,"num_write_kb":4200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'634","reported_seq":3754,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:20.202595+0000","last_change":"2023-10-16T20:47:31.584541+0000","last_active":"2023-10-16T20:58:20.202595+0000","last_peered":"2023-10-16T20:58:20.202595+0000","last_clean":"2023-10-16T20:58:20.202595+0000","last_became_active":"2023-10-16T20:36:59.464626+0000","last_became_peered":"2023-10-16T20:36:59.464626+0000","last_unstale":"2023-10-16T20:58:20.202595+0000","last_undegraded":"2023-10-16T20:58:20.202595+0000","last_fullsized":"2023-10-16T20:58:20.202595+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":634,"log_dups_size":0,"ondisk_log_size":634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:18:35.242031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069938420000000001,"stat_sum":{"num_bytes":28405361,"num_objects":321,"num_object_clones":0,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":1015,"num_read_kb":24481,"num_write":633,"num_write_kb":44456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"67'113","reported_seq":2222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.886956+0000","last_change":"2023-10-16T20:47:31.589722+0000","last_active":"2023-10-16T20:58:37.886956+0000","last_peered":"2023-10-16T20:58:37.886956+0000","last_clean":"2023-10-16T20:58:37.886956+0000","last_became_active":"2023-10-16T20:36:55.291396+0000","last_became_peered":"2023-10-16T20:36:55.291396+0000","last_unstale":"2023-10-16T20:58:37.886956+0000","last_undegraded":"2023-10-16T20:58:37.886956+0000","last_fullsized":"2023-10-16T20:58:37.886956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:09:26.019798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011464676,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"67'184","reported_seq":258,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.657888+0000","last_change":"2023-10-16T20:36:54.750814+0000","last_active":"2023-10-16T20:58:05.657888+0000","last_peered":"2023-10-16T20:58:05.657888+0000","last_clean":"2023-10-16T20:58:05.657888+0000","last_became_active":"2023-10-16T20:36:54.750448+0000","last_became_peered":"2023-10-16T20:36:54.750448+0000","last_unstale":"2023-10-16T20:58:05.657888+0000","last_undegraded":"2023-10-16T20:58:05.657888+0000","last_fullsized":"2023-10-16T20:58:05.657888+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:41.981408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":3911,"num_write":180,"num_write_kb":4223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"67'608","reported_seq":3330,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.013307+0000","last_change":"2023-10-16T20:47:31.583645+0000","last_active":"2023-10-16T20:58:23.013307+0000","last_peered":"2023-10-16T20:58:23.013307+0000","last_clean":"2023-10-16T20:58:23.013307+0000","last_became_active":"2023-10-16T20:36:59.464578+0000","last_became_peered":"2023-10-16T20:36:59.464578+0000","last_unstale":"2023-10-16T20:58:23.013307+0000","last_undegraded":"2023-10-16T20:58:23.013307+0000","last_fullsized":"2023-10-16T20:58:23.013307+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:56.818326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064826880000000003,"stat_sum":{"num_bytes":21354704,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":616,"num_read_kb":9691,"num_write":609,"num_write_kb":35926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"67'101","reported_seq":2210,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889428+0000","last_change":"2023-10-16T20:47:31.490514+0000","last_active":"2023-10-16T20:58:37.889428+0000","last_peered":"2023-10-16T20:58:37.889428+0000","last_clean":"2023-10-16T20:58:37.889428+0000","last_became_active":"2023-10-16T20:36:55.285407+0000","last_became_peered":"2023-10-16T20:36:55.285407+0000","last_unstale":"2023-10-16T20:58:37.889428+0000","last_undegraded":"2023-10-16T20:58:37.889428+0000","last_fullsized":"2023-10-16T20:58:37.889428+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:31:00.135226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010336254,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"66'459","reported_seq":604,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.695585+0000","last_change":"2023-10-16T20:36:54.757012+0000","last_active":"2023-10-16T20:58:28.695585+0000","last_peered":"2023-10-16T20:58:28.695585+0000","last_clean":"2023-10-16T20:58:28.695585+0000","last_became_active":"2023-10-16T20:36:54.756697+0000","last_became_peered":"2023-10-16T20:36:54.756697+0000","last_unstale":"2023-10-16T20:58:28.695585+0000","last_undegraded":"2023-10-16T20:58:28.695585+0000","last_fullsized":"2023-10-16T20:58:28.695585+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":459,"log_dups_size":0,"ondisk_log_size":459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:20.259580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8464867,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":106,"num_read_kb":16561,"num_write":461,"num_write_kb":8611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'632","reported_seq":4046,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.561302+0000","last_change":"2023-10-16T20:47:31.589709+0000","last_active":"2023-10-16T20:58:22.561302+0000","last_peered":"2023-10-16T20:58:22.561302+0000","last_clean":"2023-10-16T20:58:22.561302+0000","last_became_active":"2023-10-16T20:36:59.468254+0000","last_became_peered":"2023-10-16T20:36:59.468254+0000","last_unstale":"2023-10-16T20:58:22.561302+0000","last_undegraded":"2023-10-16T20:58:22.561302+0000","last_fullsized":"2023-10-16T20:58:22.561302+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:09:21.589899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093892239999999998,"stat_sum":{"num_bytes":28874401,"num_objects":300,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":1309,"num_read_kb":15538,"num_write":630,"num_write_kb":28525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"67'100","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.880164+0000","last_change":"2023-10-16T20:47:31.588140+0000","last_active":"2023-10-16T20:58:37.880164+0000","last_peered":"2023-10-16T20:58:37.880164+0000","last_clean":"2023-10-16T20:58:37.880164+0000","last_became_active":"2023-10-16T20:36:55.288488+0000","last_became_peered":"2023-10-16T20:36:55.288488+0000","last_unstale":"2023-10-16T20:58:37.880164+0000","last_undegraded":"2023-10-16T20:58:37.880164+0000","last_fullsized":"2023-10-16T20:58:37.880164+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:40:33.001741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078405860000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"67'139","reported_seq":223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.709051+0000","last_change":"2023-10-16T20:36:54.765361+0000","last_active":"2023-10-16T20:58:28.709051+0000","last_peered":"2023-10-16T20:58:28.709051+0000","last_clean":"2023-10-16T20:58:28.709051+0000","last_became_active":"2023-10-16T20:36:54.764969+0000","last_became_peered":"2023-10-16T20:36:54.764969+0000","last_unstale":"2023-10-16T20:58:28.709051+0000","last_undegraded":"2023-10-16T20:58:28.709051+0000","last_fullsized":"2023-10-16T20:58:28.709051+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:21:21.183612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389280,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":45,"num_read_kb":8235,"num_write":134,"num_write_kb":8274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"67'666","reported_seq":4058,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.657743+0000","last_change":"2023-10-16T20:47:31.488335+0000","last_active":"2023-10-16T20:58:22.657743+0000","last_peered":"2023-10-16T20:58:22.657743+0000","last_clean":"2023-10-16T20:58:22.657743+0000","last_became_active":"2023-10-16T20:36:59.302698+0000","last_became_peered":"2023-10-16T20:36:59.302698+0000","last_unstale":"2023-10-16T20:58:22.657743+0000","last_undegraded":"2023-10-16T20:58:22.657743+0000","last_fullsized":"2023-10-16T20:58:22.657743+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:09:15.954887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064797020000000004,"stat_sum":{"num_bytes":31392251,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":1286,"num_read_kb":27897,"num_write":665,"num_write_kb":43250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'99","reported_seq":2208,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.882106+0000","last_change":"2023-10-16T20:47:31.487882+0000","last_active":"2023-10-16T20:58:37.882106+0000","last_peered":"2023-10-16T20:58:37.882106+0000","last_clean":"2023-10-16T20:58:37.882106+0000","last_became_active":"2023-10-16T20:36:55.290127+0000","last_became_peered":"2023-10-16T20:36:55.290127+0000","last_unstale":"2023-10-16T20:58:37.882106+0000","last_undegraded":"2023-10-16T20:58:37.882106+0000","last_fullsized":"2023-10-16T20:58:37.882106+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:23.063288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055494209999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"67'78","reported_seq":150,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:24.130464+0000","last_change":"2023-10-16T20:36:54.749120+0000","last_active":"2023-10-16T20:58:24.130464+0000","last_peered":"2023-10-16T20:58:24.130464+0000","last_clean":"2023-10-16T20:58:24.130464+0000","last_became_active":"2023-10-16T20:36:54.748788+0000","last_became_peered":"2023-10-16T20:36:54.748788+0000","last_unstale":"2023-10-16T20:58:24.130464+0000","last_undegraded":"2023-10-16T20:58:24.130464+0000","last_fullsized":"2023-10-16T20:58:24.130464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:11:07.257343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":15,"num_read_kb":23,"num_write":83,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"67'606","reported_seq":3860,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.746395+0000","last_change":"2023-10-16T20:47:31.486033+0000","last_active":"2023-10-16T20:58:22.746395+0000","last_peered":"2023-10-16T20:58:22.746395+0000","last_clean":"2023-10-16T20:58:22.746395+0000","last_became_active":"2023-10-16T20:36:59.302916+0000","last_became_peered":"2023-10-16T20:36:59.302916+0000","last_unstale":"2023-10-16T20:58:22.746395+0000","last_undegraded":"2023-10-16T20:58:22.746395+0000","last_fullsized":"2023-10-16T20:58:22.746395+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:09.867705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041353309999999999,"stat_sum":{"num_bytes":29291517,"num_objects":300,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":1148,"num_read_kb":24111,"num_write":607,"num_write_kb":41174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"67'107","reported_seq":2215,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889740+0000","last_change":"2023-10-16T20:47:31.490807+0000","last_active":"2023-10-16T20:58:37.889740+0000","last_peered":"2023-10-16T20:58:37.889740+0000","last_clean":"2023-10-16T20:58:37.889740+0000","last_became_active":"2023-10-16T20:36:55.291448+0000","last_became_peered":"2023-10-16T20:36:55.291448+0000","last_unstale":"2023-10-16T20:58:37.889740+0000","last_undegraded":"2023-10-16T20:58:37.889740+0000","last_fullsized":"2023-10-16T20:58:37.889740+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:03:32.439694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010585074,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"67'134","reported_seq":254,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:39.515133+0000","last_change":"2023-10-16T20:36:54.761824+0000","last_active":"2023-10-16T20:58:39.515133+0000","last_peered":"2023-10-16T20:58:39.515133+0000","last_clean":"2023-10-16T20:58:39.515133+0000","last_became_active":"2023-10-16T20:36:54.761541+0000","last_became_peered":"2023-10-16T20:36:54.761541+0000","last_unstale":"2023-10-16T20:58:39.515133+0000","last_undegraded":"2023-10-16T20:58:39.515133+0000","last_fullsized":"2023-10-16T20:58:39.515133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:34.512207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2461990,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":69,"num_read_kb":129,"num_write":135,"num_write_kb":2563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"67'663","reported_seq":3325,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.270542+0000","last_change":"2023-10-16T20:47:31.585744+0000","last_active":"2023-10-16T20:58:22.270542+0000","last_peered":"2023-10-16T20:58:22.270542+0000","last_clean":"2023-10-16T20:58:22.270542+0000","last_became_active":"2023-10-16T20:36:59.303773+0000","last_became_peered":"2023-10-16T20:36:59.303773+0000","last_unstale":"2023-10-16T20:58:22.270542+0000","last_undegraded":"2023-10-16T20:58:22.270542+0000","last_fullsized":"2023-10-16T20:58:22.270542+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:19:00.801685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007941142,"stat_sum":{"num_bytes":21664408,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":557,"num_read_kb":9585,"num_write":658,"num_write_kb":32916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"67'121","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.891891+0000","last_change":"2023-10-16T20:47:31.594424+0000","last_active":"2023-10-16T20:58:37.891891+0000","last_peered":"2023-10-16T20:58:37.891891+0000","last_clean":"2023-10-16T20:58:37.891891+0000","last_became_active":"2023-10-16T20:36:55.300535+0000","last_became_peered":"2023-10-16T20:36:55.300535+0000","last_unstale":"2023-10-16T20:58:37.891891+0000","last_undegraded":"2023-10-16T20:58:37.891891+0000","last_fullsized":"2023-10-16T20:58:37.891891+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:47:03.132953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013755641000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"67'55","reported_seq":124,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:30.004648+0000","last_change":"2023-10-16T20:36:54.758140+0000","last_active":"2023-10-16T20:58:30.004648+0000","last_peered":"2023-10-16T20:58:30.004648+0000","last_clean":"2023-10-16T20:58:30.004648+0000","last_became_active":"2023-10-16T20:36:54.757303+0000","last_became_peered":"2023-10-16T20:36:54.757303+0000","last_unstale":"2023-10-16T20:58:30.004648+0000","last_undegraded":"2023-10-16T20:58:30.004648+0000","last_fullsized":"2023-10-16T20:58:30.004648+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:45.944191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":56,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"67'687","reported_seq":4217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.643179+0000","last_change":"2023-10-16T20:47:31.586327+0000","last_active":"2023-10-16T20:58:22.643179+0000","last_peered":"2023-10-16T20:58:22.643179+0000","last_clean":"2023-10-16T20:58:22.643179+0000","last_became_active":"2023-10-16T20:36:59.465844+0000","last_became_peered":"2023-10-16T20:36:59.465844+0000","last_unstale":"2023-10-16T20:58:22.643179+0000","last_undegraded":"2023-10-16T20:58:22.643179+0000","last_fullsized":"2023-10-16T20:58:22.643179+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:54.762444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081168009999999999,"stat_sum":{"num_bytes":45881583,"num_objects":353,"num_object_clones":0,"num_object_copies":1412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":353,"num_whiteouts":0,"num_read":1424,"num_read_kb":44764,"num_write":688,"num_write_kb":65580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"67'661","reported_seq":4047,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.525871+0000","last_change":"2023-10-16T20:47:31.489178+0000","last_active":"2023-10-16T20:58:22.525871+0000","last_peered":"2023-10-16T20:58:22.525871+0000","last_clean":"2023-10-16T20:58:22.525871+0000","last_became_active":"2023-10-16T20:36:59.467356+0000","last_became_peered":"2023-10-16T20:36:59.467356+0000","last_unstale":"2023-10-16T20:58:22.525871+0000","last_undegraded":"2023-10-16T20:58:22.525871+0000","last_fullsized":"2023-10-16T20:58:22.525871+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:10:50.487421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093722700000000003,"stat_sum":{"num_bytes":37038710,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1279,"num_read_kb":28276,"num_write":659,"num_write_kb":40616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'108","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.864220+0000","last_change":"2023-10-16T20:47:31.484094+0000","last_active":"2023-10-16T20:58:37.864220+0000","last_peered":"2023-10-16T20:58:37.864220+0000","last_clean":"2023-10-16T20:58:37.864220+0000","last_became_active":"2023-10-16T20:36:55.280903+0000","last_became_peered":"2023-10-16T20:36:55.280903+0000","last_unstale":"2023-10-16T20:58:37.864220+0000","last_undegraded":"2023-10-16T20:58:37.864220+0000","last_fullsized":"2023-10-16T20:58:37.864220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:50:46.153222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029269690000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"67'457","reported_seq":528,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.761682+0000","last_change":"2023-10-16T20:36:54.757589+0000","last_active":"2023-10-16T20:58:23.761682+0000","last_peered":"2023-10-16T20:58:23.761682+0000","last_clean":"2023-10-16T20:58:23.761682+0000","last_became_active":"2023-10-16T20:36:54.756775+0000","last_became_peered":"2023-10-16T20:36:54.756775+0000","last_unstale":"2023-10-16T20:58:23.761682+0000","last_undegraded":"2023-10-16T20:58:23.761682+0000","last_fullsized":"2023-10-16T20:58:23.761682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:40.629540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583494,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":16386,"num_write":448,"num_write_kb":12544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"67'638","reported_seq":3559,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.378852+0000","last_change":"2023-10-16T20:47:31.593596+0000","last_active":"2023-10-16T20:58:22.378852+0000","last_peered":"2023-10-16T20:58:22.378852+0000","last_clean":"2023-10-16T20:58:22.378852+0000","last_became_active":"2023-10-16T20:37:37.438140+0000","last_became_peered":"2023-10-16T20:37:37.438140+0000","last_unstale":"2023-10-16T20:58:22.378852+0000","last_undegraded":"2023-10-16T20:58:22.378852+0000","last_fullsized":"2023-10-16T20:58:22.378852+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:41:29.134511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012287041,"stat_sum":{"num_bytes":24792014,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":805,"num_read_kb":12206,"num_write":635,"num_write_kb":32676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"67'121","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.897843+0000","last_change":"2023-10-16T20:47:31.585618+0000","last_active":"2023-10-16T20:58:37.897843+0000","last_peered":"2023-10-16T20:58:37.897843+0000","last_clean":"2023-10-16T20:58:37.897843+0000","last_became_active":"2023-10-16T20:36:55.273959+0000","last_became_peered":"2023-10-16T20:36:55.273959+0000","last_unstale":"2023-10-16T20:58:37.897843+0000","last_undegraded":"2023-10-16T20:58:37.897843+0000","last_fullsized":"2023-10-16T20:58:37.897843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:05:00.921391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076333039999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"67'135","reported_seq":276,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.678551+0000","last_change":"2023-10-16T20:36:54.759513+0000","last_active":"2023-10-16T20:58:28.678551+0000","last_peered":"2023-10-16T20:58:28.678551+0000","last_clean":"2023-10-16T20:58:28.678551+0000","last_became_active":"2023-10-16T20:36:54.759059+0000","last_became_peered":"2023-10-16T20:36:54.759059+0000","last_unstale":"2023-10-16T20:58:28.678551+0000","last_undegraded":"2023-10-16T20:58:28.678551+0000","last_fullsized":"2023-10-16T20:58:28.678551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:36.696053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4348959,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":88,"num_read_kb":239,"num_write":133,"num_write_kb":4341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"66'223","reported_seq":372,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:33.632797+0000","last_change":"2023-10-16T20:36:54.764780+0000","last_active":"2023-10-16T20:57:33.632797+0000","last_peered":"2023-10-16T20:57:33.632797+0000","last_clean":"2023-10-16T20:57:33.632797+0000","last_became_active":"2023-10-16T20:36:54.764436+0000","last_became_peered":"2023-10-16T20:36:54.764436+0000","last_unstale":"2023-10-16T20:57:33.632797+0000","last_undegraded":"2023-10-16T20:57:33.632797+0000","last_fullsized":"2023-10-16T20:57:33.632797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:55:31.816599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8524313,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":105,"num_read_kb":8412,"num_write":220,"num_write_kb":8480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"67'109","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888733+0000","last_change":"2023-10-16T20:47:31.492923+0000","last_active":"2023-10-16T20:58:37.888733+0000","last_peered":"2023-10-16T20:58:37.888733+0000","last_clean":"2023-10-16T20:58:37.888733+0000","last_became_active":"2023-10-16T20:36:55.277873+0000","last_became_peered":"2023-10-16T20:36:55.277873+0000","last_unstale":"2023-10-16T20:58:37.888733+0000","last_undegraded":"2023-10-16T20:58:37.888733+0000","last_fullsized":"2023-10-16T20:58:37.888733+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:24.211693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012247773,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'608","reported_seq":3232,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.002422+0000","last_change":"2023-10-16T20:47:31.496724+0000","last_active":"2023-10-16T20:58:23.002422+0000","last_peered":"2023-10-16T20:58:23.002422+0000","last_clean":"2023-10-16T20:58:23.002422+0000","last_became_active":"2023-10-16T20:36:59.302435+0000","last_became_peered":"2023-10-16T20:36:59.302435+0000","last_unstale":"2023-10-16T20:58:23.002422+0000","last_undegraded":"2023-10-16T20:58:23.002422+0000","last_fullsized":"2023-10-16T20:58:23.002422+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:34:39.329693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001061892,"stat_sum":{"num_bytes":16477397,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":519,"num_read_kb":10696,"num_write":606,"num_write_kb":24605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"66'232","reported_seq":301,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:33.636417+0000","last_change":"2023-10-16T20:36:54.756190+0000","last_active":"2023-10-16T20:57:33.636417+0000","last_peered":"2023-10-16T20:57:33.636417+0000","last_clean":"2023-10-16T20:57:33.636417+0000","last_became_active":"2023-10-16T20:36:54.755834+0000","last_became_peered":"2023-10-16T20:36:54.755834+0000","last_unstale":"2023-10-16T20:57:33.636417+0000","last_undegraded":"2023-10-16T20:57:33.636417+0000","last_fullsized":"2023-10-16T20:57:33.636417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:31.012441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582946,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":16385,"num_write":235,"num_write_kb":12609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"67'104","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.885029+0000","last_change":"2023-10-16T20:47:31.486761+0000","last_active":"2023-10-16T20:58:37.885029+0000","last_peered":"2023-10-16T20:58:37.885029+0000","last_clean":"2023-10-16T20:58:37.885029+0000","last_became_active":"2023-10-16T20:36:55.279333+0000","last_became_peered":"2023-10-16T20:36:55.279333+0000","last_unstale":"2023-10-16T20:58:37.885029+0000","last_undegraded":"2023-10-16T20:58:37.885029+0000","last_fullsized":"2023-10-16T20:58:37.885029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:30:26.879792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047896609999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'629","reported_seq":3442,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.721224+0000","last_change":"2023-10-16T20:47:31.586162+0000","last_active":"2023-10-16T20:58:22.721224+0000","last_peered":"2023-10-16T20:58:22.721224+0000","last_clean":"2023-10-16T20:58:22.721224+0000","last_became_active":"2023-10-16T20:36:59.301864+0000","last_became_peered":"2023-10-16T20:36:59.301864+0000","last_unstale":"2023-10-16T20:58:22.721224+0000","last_undegraded":"2023-10-16T20:58:22.721224+0000","last_fullsized":"2023-10-16T20:58:22.721224+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:44.207453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080450769999999994,"stat_sum":{"num_bytes":20568472,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":708,"num_read_kb":22370,"num_write":628,"num_write_kb":40892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"67'120","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.875503+0000","last_change":"2023-10-16T20:47:31.590065+0000","last_active":"2023-10-16T20:58:37.875503+0000","last_peered":"2023-10-16T20:58:37.875503+0000","last_clean":"2023-10-16T20:58:37.875503+0000","last_became_active":"2023-10-16T20:36:55.287442+0000","last_became_peered":"2023-10-16T20:36:55.287442+0000","last_unstale":"2023-10-16T20:58:37.875503+0000","last_undegraded":"2023-10-16T20:58:37.875503+0000","last_fullsized":"2023-10-16T20:58:37.875503+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:37:33.639292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011630351000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"67'131","reported_seq":2241,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.880621+0000","last_change":"2023-10-16T20:47:31.497003+0000","last_active":"2023-10-16T20:58:37.880621+0000","last_peered":"2023-10-16T20:58:37.880621+0000","last_clean":"2023-10-16T20:58:37.880621+0000","last_became_active":"2023-10-16T20:36:55.276276+0000","last_became_peered":"2023-10-16T20:36:55.276276+0000","last_unstale":"2023-10-16T20:58:37.880621+0000","last_undegraded":"2023-10-16T20:58:37.880621+0000","last_fullsized":"2023-10-16T20:58:37.880621+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:35:00.858846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091802500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"67'115","reported_seq":2232,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.886101+0000","last_change":"2023-10-16T20:47:31.592299+0000","last_active":"2023-10-16T20:58:37.886101+0000","last_peered":"2023-10-16T20:58:37.886101+0000","last_clean":"2023-10-16T20:58:37.886101+0000","last_became_active":"2023-10-16T20:37:37.413040+0000","last_became_peered":"2023-10-16T20:37:37.413040+0000","last_unstale":"2023-10-16T20:58:37.886101+0000","last_undegraded":"2023-10-16T20:58:37.886101+0000","last_fullsized":"2023-10-16T20:58:37.886101+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:50:46.069439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012618513,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"67'116","reported_seq":2226,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888796+0000","last_change":"2023-10-16T20:47:31.492800+0000","last_active":"2023-10-16T20:58:37.888796+0000","last_peered":"2023-10-16T20:58:37.888796+0000","last_clean":"2023-10-16T20:58:37.888796+0000","last_became_active":"2023-10-16T20:36:55.275349+0000","last_became_peered":"2023-10-16T20:36:55.275349+0000","last_unstale":"2023-10-16T20:58:37.888796+0000","last_undegraded":"2023-10-16T20:58:37.888796+0000","last_fullsized":"2023-10-16T20:58:37.888796+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:28:00.161390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012136073000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"67'88","reported_seq":2198,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.852714+0000","last_change":"2023-10-16T20:47:31.587774+0000","last_active":"2023-10-16T20:58:37.852714+0000","last_peered":"2023-10-16T20:58:37.852714+0000","last_clean":"2023-10-16T20:58:37.852714+0000","last_became_active":"2023-10-16T20:36:55.277043+0000","last_became_peered":"2023-10-16T20:36:55.277043+0000","last_unstale":"2023-10-16T20:58:37.852714+0000","last_undegraded":"2023-10-16T20:58:37.852714+0000","last_fullsized":"2023-10-16T20:58:37.852714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:32.439680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078065369999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"67'89","reported_seq":2199,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.863001+0000","last_change":"2023-10-16T20:47:31.489466+0000","last_active":"2023-10-16T20:58:37.863001+0000","last_peered":"2023-10-16T20:58:37.863001+0000","last_clean":"2023-10-16T20:58:37.863001+0000","last_became_active":"2023-10-16T20:36:55.278715+0000","last_became_peered":"2023-10-16T20:36:55.278715+0000","last_unstale":"2023-10-16T20:58:37.863001+0000","last_undegraded":"2023-10-16T20:58:37.863001+0000","last_fullsized":"2023-10-16T20:58:37.863001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:49:22.643504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067460719999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"67'124","reported_seq":2234,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.836534+0000","last_change":"2023-10-16T20:47:31.594095+0000","last_active":"2023-10-16T20:58:37.836534+0000","last_peered":"2023-10-16T20:58:37.836534+0000","last_clean":"2023-10-16T20:58:37.836534+0000","last_became_active":"2023-10-16T20:36:55.301604+0000","last_became_peered":"2023-10-16T20:36:55.301604+0000","last_unstale":"2023-10-16T20:58:37.836534+0000","last_undegraded":"2023-10-16T20:58:37.836534+0000","last_fullsized":"2023-10-16T20:58:37.836534+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:10.110081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012969489000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"67'99","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.858692+0000","last_change":"2023-10-16T20:47:31.587306+0000","last_active":"2023-10-16T20:58:37.858692+0000","last_peered":"2023-10-16T20:58:37.858692+0000","last_clean":"2023-10-16T20:58:37.858692+0000","last_became_active":"2023-10-16T20:36:55.276984+0000","last_became_peered":"2023-10-16T20:36:55.276984+0000","last_unstale":"2023-10-16T20:58:37.858692+0000","last_undegraded":"2023-10-16T20:58:37.858692+0000","last_fullsized":"2023-10-16T20:58:37.858692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:42:52.651942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086623480000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'111","reported_seq":2221,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.896916+0000","last_change":"2023-10-16T20:47:31.595489+0000","last_active":"2023-10-16T20:58:37.896916+0000","last_peered":"2023-10-16T20:58:37.896916+0000","last_clean":"2023-10-16T20:58:37.896916+0000","last_became_active":"2023-10-16T20:36:55.290023+0000","last_became_peered":"2023-10-16T20:36:55.290023+0000","last_unstale":"2023-10-16T20:58:37.896916+0000","last_undegraded":"2023-10-16T20:58:37.896916+0000","last_fullsized":"2023-10-16T20:58:37.896916+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:33.259824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013081514000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'124","reported_seq":2233,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.203227+0000","last_change":"2023-10-16T20:47:31.586678+0000","last_active":"2023-10-16T20:58:38.203227+0000","last_peered":"2023-10-16T20:58:38.203227+0000","last_clean":"2023-10-16T20:58:38.203227+0000","last_became_active":"2023-10-16T20:36:55.280186+0000","last_became_peered":"2023-10-16T20:36:55.280186+0000","last_unstale":"2023-10-16T20:58:38.203227+0000","last_undegraded":"2023-10-16T20:58:38.203227+0000","last_fullsized":"2023-10-16T20:58:38.203227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:22:01.009457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083925690000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"67'118","reported_seq":2227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.882744+0000","last_change":"2023-10-16T20:47:31.589254+0000","last_active":"2023-10-16T20:58:37.882744+0000","last_peered":"2023-10-16T20:58:37.882744+0000","last_clean":"2023-10-16T20:58:37.882744+0000","last_became_active":"2023-10-16T20:36:55.297253+0000","last_became_peered":"2023-10-16T20:36:55.297253+0000","last_unstale":"2023-10-16T20:58:37.882744+0000","last_undegraded":"2023-10-16T20:58:37.882744+0000","last_fullsized":"2023-10-16T20:58:37.882744+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:11.854849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097685800000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"67'108","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.891317+0000","last_change":"2023-10-16T20:47:31.492104+0000","last_active":"2023-10-16T20:58:37.891317+0000","last_peered":"2023-10-16T20:58:37.891317+0000","last_clean":"2023-10-16T20:58:37.891317+0000","last_became_active":"2023-10-16T20:36:55.286044+0000","last_became_peered":"2023-10-16T20:36:55.286044+0000","last_unstale":"2023-10-16T20:58:37.891317+0000","last_undegraded":"2023-10-16T20:58:37.891317+0000","last_fullsized":"2023-10-16T20:58:37.891317+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:45.304718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011681474000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"67'110","reported_seq":2219,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.865687+0000","last_change":"2023-10-16T20:47:31.592401+0000","last_active":"2023-10-16T20:58:37.865687+0000","last_peered":"2023-10-16T20:58:37.865687+0000","last_clean":"2023-10-16T20:58:37.865687+0000","last_became_active":"2023-10-16T20:36:55.286864+0000","last_became_peered":"2023-10-16T20:36:55.286864+0000","last_unstale":"2023-10-16T20:58:37.865687+0000","last_undegraded":"2023-10-16T20:58:37.865687+0000","last_fullsized":"2023-10-16T20:58:37.865687+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:31.381174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012885232999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"67'106","reported_seq":2215,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.872600+0000","last_change":"2023-10-16T20:47:31.588333+0000","last_active":"2023-10-16T20:58:37.872600+0000","last_peered":"2023-10-16T20:58:37.872600+0000","last_clean":"2023-10-16T20:58:37.872600+0000","last_became_active":"2023-10-16T20:36:55.292262+0000","last_became_peered":"2023-10-16T20:36:55.292262+0000","last_unstale":"2023-10-16T20:58:37.872600+0000","last_undegraded":"2023-10-16T20:58:37.872600+0000","last_fullsized":"2023-10-16T20:58:37.872600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:15.116061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077171749999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"67'107","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.832173+0000","last_change":"2023-10-16T20:47:31.496924+0000","last_active":"2023-10-16T20:58:37.832173+0000","last_peered":"2023-10-16T20:58:37.832173+0000","last_clean":"2023-10-16T20:58:37.832173+0000","last_became_active":"2023-10-16T20:36:55.282806+0000","last_became_peered":"2023-10-16T20:36:55.282806+0000","last_unstale":"2023-10-16T20:58:37.832173+0000","last_undegraded":"2023-10-16T20:58:37.832173+0000","last_fullsized":"2023-10-16T20:58:37.832173+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:51.395313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086132999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"67'109","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.887891+0000","last_change":"2023-10-16T20:47:31.594959+0000","last_active":"2023-10-16T20:58:37.887891+0000","last_peered":"2023-10-16T20:58:37.887891+0000","last_clean":"2023-10-16T20:58:37.887891+0000","last_became_active":"2023-10-16T20:36:55.303095+0000","last_became_peered":"2023-10-16T20:36:55.303095+0000","last_unstale":"2023-10-16T20:58:37.887891+0000","last_undegraded":"2023-10-16T20:58:37.887891+0000","last_fullsized":"2023-10-16T20:58:37.887891+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:00:51.726359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013746639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"67'119","reported_seq":2228,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.181755+0000","last_change":"2023-10-16T20:47:31.595380+0000","last_active":"2023-10-16T20:58:38.181755+0000","last_peered":"2023-10-16T20:58:38.181755+0000","last_clean":"2023-10-16T20:58:38.181755+0000","last_became_active":"2023-10-16T20:36:55.291447+0000","last_became_peered":"2023-10-16T20:36:55.291447+0000","last_unstale":"2023-10-16T20:58:38.181755+0000","last_undegraded":"2023-10-16T20:58:38.181755+0000","last_fullsized":"2023-10-16T20:58:38.181755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:20.797976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01315413,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"67'107","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.868160+0000","last_change":"2023-10-16T20:47:31.484708+0000","last_active":"2023-10-16T20:58:37.868160+0000","last_peered":"2023-10-16T20:58:37.868160+0000","last_clean":"2023-10-16T20:58:37.868160+0000","last_became_active":"2023-10-16T20:36:55.280605+0000","last_became_peered":"2023-10-16T20:36:55.280605+0000","last_unstale":"2023-10-16T20:58:37.868160+0000","last_undegraded":"2023-10-16T20:58:37.868160+0000","last_fullsized":"2023-10-16T20:58:37.868160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:39.852344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033133080000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"67'112","reported_seq":2221,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.891112+0000","last_change":"2023-10-16T20:47:31.492149+0000","last_active":"2023-10-16T20:58:37.891112+0000","last_peered":"2023-10-16T20:58:37.891112+0000","last_clean":"2023-10-16T20:58:37.891112+0000","last_became_active":"2023-10-16T20:36:55.277094+0000","last_became_peered":"2023-10-16T20:36:55.277094+0000","last_unstale":"2023-10-16T20:58:37.891112+0000","last_undegraded":"2023-10-16T20:58:37.891112+0000","last_fullsized":"2023-10-16T20:58:37.891112+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:01.971894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011816411000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"67'94","reported_seq":2203,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.879368+0000","last_change":"2023-10-16T20:47:31.587367+0000","last_active":"2023-10-16T20:58:37.879368+0000","last_peered":"2023-10-16T20:58:37.879368+0000","last_clean":"2023-10-16T20:58:37.879368+0000","last_became_active":"2023-10-16T20:36:55.297199+0000","last_became_peered":"2023-10-16T20:36:55.297199+0000","last_unstale":"2023-10-16T20:58:37.879368+0000","last_undegraded":"2023-10-16T20:58:37.879368+0000","last_fullsized":"2023-10-16T20:58:37.879368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:07:32.275973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086430910000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"67'90","reported_seq":2199,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.867006+0000","last_change":"2023-10-16T20:47:31.487461+0000","last_active":"2023-10-16T20:58:37.867006+0000","last_peered":"2023-10-16T20:58:37.867006+0000","last_clean":"2023-10-16T20:58:37.867006+0000","last_became_active":"2023-10-16T20:36:55.286964+0000","last_became_peered":"2023-10-16T20:36:55.286964+0000","last_unstale":"2023-10-16T20:58:37.867006+0000","last_undegraded":"2023-10-16T20:58:37.867006+0000","last_fullsized":"2023-10-16T20:58:37.867006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:57.164132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051857609999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"67'110","reported_seq":2226,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.834126+0000","last_change":"2023-10-16T20:47:31.594637+0000","last_active":"2023-10-16T20:58:37.834126+0000","last_peered":"2023-10-16T20:58:37.834126+0000","last_clean":"2023-10-16T20:58:37.834126+0000","last_became_active":"2023-10-16T20:37:37.413492+0000","last_became_peered":"2023-10-16T20:37:37.413492+0000","last_unstale":"2023-10-16T20:58:37.834126+0000","last_undegraded":"2023-10-16T20:58:37.834126+0000","last_fullsized":"2023-10-16T20:58:37.834126+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:07.365316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014529932000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"67'106","reported_seq":2215,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.850164+0000","last_change":"2023-10-16T20:47:31.593072+0000","last_active":"2023-10-16T20:58:37.850164+0000","last_peered":"2023-10-16T20:58:37.850164+0000","last_clean":"2023-10-16T20:58:37.850164+0000","last_became_active":"2023-10-16T20:36:55.302468+0000","last_became_peered":"2023-10-16T20:36:55.302468+0000","last_unstale":"2023-10-16T20:58:37.850164+0000","last_undegraded":"2023-10-16T20:58:37.850164+0000","last_fullsized":"2023-10-16T20:58:37.850164+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:54:24.682600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012359399,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"67'92","reported_seq":2200,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.894436+0000","last_change":"2023-10-16T20:47:31.585931+0000","last_active":"2023-10-16T20:58:37.894436+0000","last_peered":"2023-10-16T20:58:37.894436+0000","last_clean":"2023-10-16T20:58:37.894436+0000","last_became_active":"2023-10-16T20:36:55.296476+0000","last_became_peered":"2023-10-16T20:36:55.296476+0000","last_unstale":"2023-10-16T20:58:37.894436+0000","last_undegraded":"2023-10-16T20:58:37.894436+0000","last_fullsized":"2023-10-16T20:58:37.894436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:50:21.532799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072035780000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'95","reported_seq":2203,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.898420+0000","last_change":"2023-10-16T20:47:31.585218+0000","last_active":"2023-10-16T20:58:37.898420+0000","last_peered":"2023-10-16T20:58:37.898420+0000","last_clean":"2023-10-16T20:58:37.898420+0000","last_became_active":"2023-10-16T20:36:55.293687+0000","last_became_peered":"2023-10-16T20:36:55.293687+0000","last_unstale":"2023-10-16T20:58:37.898420+0000","last_undegraded":"2023-10-16T20:58:37.898420+0000","last_fullsized":"2023-10-16T20:58:37.898420+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:06:36.236196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074548399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"67'127","reported_seq":2235,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.887764+0000","last_change":"2023-10-16T20:47:31.588394+0000","last_active":"2023-10-16T20:58:37.887764+0000","last_peered":"2023-10-16T20:58:37.887764+0000","last_clean":"2023-10-16T20:58:37.887764+0000","last_became_active":"2023-10-16T20:36:55.280444+0000","last_became_peered":"2023-10-16T20:36:55.280444+0000","last_unstale":"2023-10-16T20:58:37.887764+0000","last_undegraded":"2023-10-16T20:58:37.887764+0000","last_fullsized":"2023-10-16T20:58:37.887764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:40:02.818328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068623299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'109","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888019+0000","last_change":"2023-10-16T20:47:31.588884+0000","last_active":"2023-10-16T20:58:37.888019+0000","last_peered":"2023-10-16T20:58:37.888019+0000","last_clean":"2023-10-16T20:58:37.888019+0000","last_became_active":"2023-10-16T20:36:55.296694+0000","last_became_peered":"2023-10-16T20:36:55.296694+0000","last_unstale":"2023-10-16T20:58:37.888019+0000","last_undegraded":"2023-10-16T20:58:37.888019+0000","last_fullsized":"2023-10-16T20:58:37.888019+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:39.793533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096291870000000009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"67'120","reported_seq":2228,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.879224+0000","last_change":"2023-10-16T20:47:31.585319+0000","last_active":"2023-10-16T20:58:37.879224+0000","last_peered":"2023-10-16T20:58:37.879224+0000","last_clean":"2023-10-16T20:58:37.879224+0000","last_became_active":"2023-10-16T20:36:55.291052+0000","last_became_peered":"2023-10-16T20:36:55.291052+0000","last_unstale":"2023-10-16T20:58:37.879224+0000","last_undegraded":"2023-10-16T20:58:37.879224+0000","last_fullsized":"2023-10-16T20:58:37.879224+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:45.283686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070559409999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"67'94","reported_seq":2204,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.895896+0000","last_change":"2023-10-16T20:47:31.588931+0000","last_active":"2023-10-16T20:58:37.895896+0000","last_peered":"2023-10-16T20:58:37.895896+0000","last_clean":"2023-10-16T20:58:37.895896+0000","last_became_active":"2023-10-16T20:36:55.290386+0000","last_became_peered":"2023-10-16T20:36:55.290386+0000","last_unstale":"2023-10-16T20:58:37.895896+0000","last_undegraded":"2023-10-16T20:58:37.895896+0000","last_fullsized":"2023-10-16T20:58:37.895896+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:56:59.640546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093217969999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"67'114","reported_seq":2224,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888639+0000","last_change":"2023-10-16T20:47:31.490951+0000","last_active":"2023-10-16T20:58:37.888639+0000","last_peered":"2023-10-16T20:58:37.888639+0000","last_clean":"2023-10-16T20:58:37.888639+0000","last_became_active":"2023-10-16T20:36:55.270787+0000","last_became_peered":"2023-10-16T20:36:55.270787+0000","last_unstale":"2023-10-16T20:58:37.888639+0000","last_undegraded":"2023-10-16T20:58:37.888639+0000","last_fullsized":"2023-10-16T20:58:37.888639+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:57:57.930582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010801276,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"67'94","reported_seq":2204,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.890538+0000","last_change":"2023-10-16T20:47:31.487902+0000","last_active":"2023-10-16T20:58:37.890538+0000","last_peered":"2023-10-16T20:58:37.890538+0000","last_clean":"2023-10-16T20:58:37.890538+0000","last_became_active":"2023-10-16T20:36:55.271373+0000","last_became_peered":"2023-10-16T20:36:55.271373+0000","last_unstale":"2023-10-16T20:58:37.890538+0000","last_undegraded":"2023-10-16T20:58:37.890538+0000","last_fullsized":"2023-10-16T20:58:37.890538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:37:10.982777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006091478,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"67'111","reported_seq":2221,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.890234+0000","last_change":"2023-10-16T20:47:31.587880+0000","last_active":"2023-10-16T20:58:37.890234+0000","last_peered":"2023-10-16T20:58:37.890234+0000","last_clean":"2023-10-16T20:58:37.890234+0000","last_became_active":"2023-10-16T20:36:55.291016+0000","last_became_peered":"2023-10-16T20:36:55.291016+0000","last_unstale":"2023-10-16T20:58:37.890234+0000","last_undegraded":"2023-10-16T20:58:37.890234+0000","last_fullsized":"2023-10-16T20:58:37.890234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:23:37.851121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009370148,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":924319467,"num_objects":10294,"num_object_clones":0,"num_object_copies":41176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10294,"num_whiteouts":0,"num_read":30602,"num_read_kb":676969,"num_write":21002,"num_write_kb":1302825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966899200,"data_stored":1966899200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21050,"ondisk_log_size":21050,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6794,"ondisk_log_size":6794,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":178960041,"num_objects":198,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1063,"num_read_kb":205540,"num_write":6349,"num_write_kb":180514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":115,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":358170624,"data_stored":357920082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6078,"ondisk_log_size":6078,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509707,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396772,"kb_used_data":280924,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93975068,"statfs":{"total":96636764160,"available":96230469632,"internally_reserved":0,"allocated":287666176,"data_stored":287268719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":118618571},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.52300000000000002,"15min":0.629},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67900000000000005,"5min":0.91400000000000003,"15min":2.4489999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54600000000000004,"15min":0.64700000000000002},"min":{"1min":0.378,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80200000000000005,"5min":0.95399999999999996,"15min":11.77},"last":0.73899999999999999}]},{"osd":1,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57699999999999996,"15min":0.65300000000000002},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73699999999999999,"5min":0.86199999999999999,"15min":4.5279999999999996},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58499999999999996,"15min":0.68400000000000005},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.84599999999999997,"5min":1.024,"15min":11.971},"last":0.77400000000000002}]},{"osd":2,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61599999999999999,"15min":0.67200000000000004},"min":{"1min":0.41999999999999998,"5min":0.40200000000000002,"15min":0.34300000000000003},"max":{"1min":0.81899999999999995,"5min":0.92200000000000004,"15min":7.1130000000000004},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.629,"15min":0.70999999999999996},"min":{"1min":0.51100000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.877,"5min":0.98299999999999998,"15min":9.4450000000000003},"last":0.69699999999999995}]},{"osd":3,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.64900000000000002,"15min":0.68300000000000005},"min":{"1min":0.439,"5min":0.437,"15min":0.41299999999999998},"max":{"1min":0.78000000000000003,"5min":1,"15min":3.363},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.68300000000000005,"15min":0.72699999999999998},"min":{"1min":0.50900000000000001,"5min":0.46899999999999997,"15min":0.42099999999999999},"max":{"1min":0.86299999999999999,"5min":1.0529999999999999,"15min":3.3980000000000001},"last":0.76300000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56699999999999995,"15min":0.61399999999999999},"min":{"1min":0.36399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81899999999999995,"5min":1.091,"15min":12.532999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.623,"15min":0.628},"min":{"1min":0.47099999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90600000000000003,"5min":1.107,"15min":3.8700000000000001},"last":0.52100000000000002}]},{"osd":5,"last update":"Mon Oct 16 20:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60999999999999999,"15min":0.66300000000000003},"min":{"1min":0.41399999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.85699999999999998,"5min":1.042,"15min":2.2010000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.63600000000000001,"15min":0.73699999999999999},"min":{"1min":0.47499999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94399999999999995,"5min":1.0369999999999999,"15min":12.353},"last":0.48899999999999999}]},{"osd":6,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65100000000000002,"15min":0.746},"min":{"1min":0.40799999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93799999999999994,"5min":1.1299999999999999,"15min":11.929},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66900000000000004,"15min":0.71399999999999997},"min":{"1min":0.41799999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92900000000000005,"5min":1.0620000000000001,"15min":6.0359999999999996},"last":0.58599999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509707,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421948,"kb_used_data":300532,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93949892,"statfs":{"total":96636764160,"available":96204689408,"internally_reserved":0,"allocated":307744768,"data_stored":307357593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124320199},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.54900000000000004,"15min":0.61499999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27800000000000002},"max":{"1min":0.69499999999999995,"5min":0.93799999999999994,"15min":4.5419999999999998},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.56499999999999995,"15min":0.624},"min":{"1min":0.33200000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.76000000000000001,"5min":2.0329999999999999,"15min":4.9699999999999998},"last":0.79900000000000004}]},{"osd":1,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.624,"15min":0.64100000000000001},"min":{"1min":0.433,"5min":0.39200000000000002,"15min":0.377},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":2.8610000000000002},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67000000000000004,"15min":0.69599999999999995},"min":{"1min":0.41499999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.94399999999999995,"5min":1.014,"15min":1.0309999999999999},"last":0.84499999999999997}]},{"osd":2,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.59699999999999998,"15min":0.65000000000000002},"min":{"1min":0.36399999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86599999999999999,"5min":1.0289999999999999,"15min":1.895},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.65500000000000003,"15min":0.68500000000000005},"min":{"1min":0.40799999999999997,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78500000000000003,"5min":0.96099999999999997,"15min":2.3340000000000001},"last":0.86499999999999999}]},{"osd":3,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.755,"15min":0.72099999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.36599999999999999},"max":{"1min":0.86699999999999999,"5min":4.6340000000000003,"15min":4.6340000000000003},"last":0.879},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.64100000000000001,"15min":0.69499999999999995},"min":{"1min":0.435,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.86099999999999999,"5min":1.163,"15min":5.1029999999999998},"last":0.67500000000000004}]},{"osd":4,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55500000000000005,"15min":0.58999999999999997},"min":{"1min":0.36299999999999999,"5min":0.34799999999999998,"15min":0.30299999999999999},"max":{"1min":0.80300000000000005,"5min":1.0189999999999999,"15min":4.8959999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.63500000000000001,"15min":0.63400000000000001},"min":{"1min":0.374,"5min":0.313,"15min":0.313},"max":{"1min":0.996,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.58999999999999997}]},{"osd":5,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.59799999999999998,"15min":0.623},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.82199999999999995,"5min":1.05,"15min":2.3769999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.626,"15min":0.66200000000000003},"min":{"1min":0.38200000000000001,"5min":0.35999999999999999,"15min":0.34499999999999997},"max":{"1min":0.84499999999999997,"5min":1.0700000000000001,"15min":4.9379999999999997},"last":0.82699999999999996}]},{"osd":7,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.71199999999999997,"15min":0.70999999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.034,"5min":1.425,"15min":4.5789999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.76800000000000002,"15min":0.748},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.39300000000000002},"max":{"1min":1.0600000000000001,"5min":1.6120000000000001,"15min":5.2460000000000004},"last":0.73999999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509709,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324840,"kb_used_data":225632,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94047000,"statfs":{"total":96636764160,"available":96304128000,"internally_reserved":0,"allocated":231047168,"data_stored":230645947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":101579207},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.57199999999999995,"15min":0.61299999999999999},"min":{"1min":0.30499999999999999,"5min":0.28199999999999997,"15min":0.25700000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":2.4580000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.59299999999999997,"15min":0.65800000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.94999999999999996,"5min":1.1799999999999999,"15min":7.181},"last":0.64600000000000002}]},{"osd":1,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":0.76800000000000002,"15min":0.69099999999999995},"min":{"1min":0.36699999999999999,"5min":0.33900000000000002,"15min":0.29899999999999999},"max":{"1min":16.655000000000001,"5min":16.655000000000001,"15min":16.655000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.5529999999999999,"5min":0.80100000000000005,"15min":0.71299999999999997},"min":{"1min":0.35099999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":16.289000000000001,"5min":16.289000000000001,"15min":16.289000000000001},"last":0.55700000000000005}]},{"osd":2,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.63200000000000001,"15min":0.67300000000000004},"min":{"1min":0.374,"5min":0.34599999999999997,"15min":0.308},"max":{"1min":1.1339999999999999,"5min":1.7110000000000001,"15min":3.3799999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67100000000000004,"15min":0.67700000000000005},"min":{"1min":0.42399999999999999,"5min":0.39800000000000002,"15min":0.38},"max":{"1min":1.1100000000000001,"5min":1.583,"15min":4.5369999999999999},"last":0.59399999999999997}]},{"osd":3,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.67400000000000004,"15min":0.69699999999999995},"min":{"1min":0.42399999999999999,"5min":0.39400000000000002,"15min":0.34899999999999998},"max":{"1min":1.0860000000000001,"5min":1.329,"15min":4.1070000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68600000000000005,"15min":0.69399999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.38},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":3.476},"last":0.69599999999999995}]},{"osd":4,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53100000000000003,"15min":0.55500000000000005},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.749,"5min":0.85099999999999998,"15min":2.415},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.59999999999999998,"15min":0.66300000000000003},"min":{"1min":0.34300000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.78900000000000003,"5min":1.155,"15min":3.2919999999999998},"last":0.45300000000000001}]},{"osd":6,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57299999999999995,"15min":0.59699999999999998},"min":{"1min":0.39000000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79400000000000004,"5min":0.93799999999999994,"15min":3.2400000000000002},"last":0.374},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.31},"max":{"1min":0.71799999999999997,"5min":1.036,"15min":1.2310000000000001},"last":0.48099999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.44600000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76300000000000001,"5min":1.23,"15min":5.2759999999999998},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69399999999999995,"15min":0.70699999999999996},"min":{"1min":0.45000000000000001,"5min":0.435,"15min":0.42299999999999999},"max":{"1min":0.83199999999999996,"5min":0.97699999999999998,"15min":3.3140000000000001},"last":0.63500000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509707,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420124,"kb_used_data":290388,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93951716,"statfs":{"total":96636764160,"available":96206557184,"internally_reserved":0,"allocated":297357312,"data_stored":296970571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56100000000000005,"15min":0.61299999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.83799999999999997,"5min":0.91400000000000003,"15min":1.524},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61699999999999999,"15min":0.69199999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.5649999999999999},"last":0.89200000000000002}]},{"osd":1,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.376,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.1749999999999998,"5min":4.1749999999999998,"15min":4.1749999999999998},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.625,"15min":0.66600000000000004},"min":{"1min":0.38200000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82599999999999996,"5min":0.94899999999999995,"15min":1.1020000000000001},"last":0.91400000000000003}]},{"osd":2,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.38900000000000001},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":2.3370000000000002},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.68300000000000005,"15min":0.71199999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.84499999999999997,"5min":1.2210000000000001,"15min":2.3660000000000001},"last":0.98999999999999999}]},{"osd":3,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.377},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":2.04},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":2.2240000000000002},"last":1.042}]},{"osd":5,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.58799999999999997,"15min":0.60299999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.9970000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63600000000000001,"15min":0.60499999999999998},"min":{"1min":0.41299999999999998,"5min":0.39500000000000002,"15min":0.33900000000000002},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.77900000000000003}]},{"osd":6,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.60299999999999998,"15min":0.61599999999999999},"min":{"1min":0.33900000000000002,"5min":0.315,"15min":0.312},"max":{"1min":1.2270000000000001,"5min":1.6339999999999999,"15min":2.2639999999999998},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.66200000000000003,"15min":0.67300000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.8300000000000001},"last":0.85299999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.69399999999999995,"15min":0.66800000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2969999999999999,"5min":1.347,"15min":1.347},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65500000000000003,"15min":0.67900000000000005},"min":{"1min":0.38600000000000001,"5min":0.35899999999999999,"15min":0.35599999999999998},"max":{"1min":1.266,"5min":1.266,"15min":2.2999999999999998},"last":1.008}]}]},{"osd":3,"up_from":15,"seq":64424509707,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485596,"kb_used_data":360020,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93886244,"statfs":{"total":96636764160,"available":96139513856,"internally_reserved":0,"allocated":368660480,"data_stored":368273200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":128580040},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.45200000000000001,"15min":0.433},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":0.76300000000000001,"5min":2.8109999999999999,"15min":4.5830000000000002},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.44700000000000001,"15min":0.44600000000000001},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.253},"max":{"1min":0.58199999999999996,"5min":1.147,"15min":1.2509999999999999},"last":0.40799999999999997}]},{"osd":1,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.49099999999999999,"15min":0.503},"min":{"1min":0.33500000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67000000000000004,"5min":1.411,"15min":4.6840000000000002},"last":0.495},{"interface":"front","average":{"1min":0.437,"5min":0.504,"15min":0.51000000000000001},"min":{"1min":0.29699999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61399999999999999,"5min":2.6360000000000001,"15min":2.7429999999999999},"last":0.38300000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.40000000000000002,"5min":0.34399999999999997,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":1.286,"15min":4.6449999999999996},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55700000000000005,"15min":0.51900000000000002},"min":{"1min":0.32200000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":1.1850000000000001,"15min":1.911},"last":0.47299999999999998}]},{"osd":4,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.72399999999999998,"15min":0.76200000000000001},"min":{"1min":0.52000000000000002,"5min":0.45400000000000001,"15min":0.44800000000000001},"max":{"1min":1.0620000000000001,"5min":1.538,"15min":1.7749999999999999},"last":0.877},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.72399999999999998,"15min":0.74199999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.155,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":0.91000000000000003}]},{"osd":5,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.74199999999999999,"15min":0.755},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0780000000000001,"5min":1.4970000000000001,"15min":4.8449999999999998},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.74099999999999999,"15min":0.745},"min":{"1min":0.48199999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1220000000000001,"5min":2.948,"15min":2.948},"last":0.65900000000000003}]},{"osd":6,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.76300000000000001,"15min":0.76600000000000001},"min":{"1min":0.48899999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.98399999999999999,"5min":1.633,"15min":4.8099999999999996},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.45400000000000001},"max":{"1min":0.93899999999999995,"5min":1.6559999999999999,"15min":2.339},"last":0.85799999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.79800000000000004,"15min":0.86499999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.46000000000000002},"max":{"1min":1.216,"5min":1.593,"15min":34.039000000000001},"last":0.753},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.76900000000000002,"15min":0.77700000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0249999999999999,"5min":2.984,"15min":2.984},"last":0.68899999999999995}]}]},{"osd":2,"up_from":15,"seq":64424509708,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303812,"kb_used_data":200444,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94068028,"statfs":{"total":96636764160,"available":96325660672,"internally_reserved":0,"allocated":205254656,"data_stored":204864239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42899999999999999,"15min":0.45000000000000001},"min":{"1min":0.27100000000000002,"5min":0.23400000000000001,"15min":0.20599999999999999},"max":{"1min":0.74299999999999999,"5min":0.92500000000000004,"15min":5.5499999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.52600000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.1180000000000001,"5min":1.1220000000000001,"15min":5.4160000000000004},"last":0.34799999999999998}]},{"osd":1,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.504,"15min":0.5},"min":{"1min":0.307,"5min":0.307,"15min":0.26000000000000001},"max":{"1min":0.81899999999999995,"5min":1.25,"15min":2.2349999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.501},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.27200000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":5.4359999999999999},"last":0.38600000000000001}]},{"osd":3,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.28399999999999997,"5min":0.26400000000000001,"15min":0.25800000000000001},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.312,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.83099999999999996,"5min":1.286,"15min":5.3789999999999996},"last":0.58699999999999997}]},{"osd":4,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.73099999999999998,"15min":0.80000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":3.464,"5min":3.464,"15min":16.905999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.70699999999999996,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.44700000000000001,"15min":0.441},"max":{"1min":0.92100000000000004,"5min":1.3100000000000001,"15min":10.064},"last":0.70699999999999996}]},{"osd":5,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70999999999999996,"15min":0.72999999999999998},"min":{"1min":0.45300000000000001,"5min":0.41099999999999998,"15min":0.32800000000000001},"max":{"1min":1.0960000000000001,"5min":1.304,"15min":3.2989999999999999},"last":0.627},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.73499999999999999,"15min":0.77100000000000002},"min":{"1min":0.52400000000000002,"5min":0.499,"15min":0.46700000000000003},"max":{"1min":1.0069999999999999,"5min":1.3640000000000001,"15min":5.3330000000000002},"last":0.66100000000000003}]},{"osd":6,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.76500000000000001,"15min":0.755},"min":{"1min":0.438,"5min":0.438,"15min":0.42299999999999999},"max":{"1min":3.5710000000000002,"5min":3.5710000000000002,"15min":3.5710000000000002},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.77900000000000003,"15min":0.77000000000000002},"min":{"1min":0.49099999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":5.4749999999999996},"last":0.75800000000000001}]},{"osd":7,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.47399999999999998,"5min":0.47199999999999998,"15min":0.44500000000000001},"max":{"1min":3.5089999999999999,"5min":3.5089999999999999,"15min":5.516},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77200000000000002,"15min":0.79100000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.41899999999999998},"max":{"1min":1.325,"5min":1.325,"15min":8.4610000000000003},"last":0.77600000000000002}]}]},{"osd":1,"up_from":15,"seq":64424509707,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490520,"kb_used_data":360784,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93881320,"statfs":{"total":96636764160,"available":96134471680,"internally_reserved":0,"allocated":369442816,"data_stored":369057373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.437,"15min":0.45700000000000002},"min":{"1min":0.245,"5min":0.219,"15min":0.21299999999999999},"max":{"1min":0.79800000000000004,"5min":0.93000000000000005,"15min":8.9339999999999993},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.34300000000000003,"5min":0.28999999999999998,"15min":0.23000000000000001},"max":{"1min":0.67200000000000004,"5min":1.6899999999999999,"15min":3.8809999999999998},"last":0.63600000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.502,"15min":0.46000000000000002},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.248},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.55900000000000005,"15min":0.55400000000000005},"min":{"1min":0.29699999999999999,"5min":0.29099999999999998,"15min":0.28499999999999998},"max":{"1min":0.88600000000000001,"5min":2.6739999999999999,"15min":9.2680000000000007},"last":0.69699999999999995}]},{"osd":3,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57499999999999996,"15min":0.54200000000000004},"min":{"1min":0.33000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":1.2030000000000001,"5min":1.9910000000000001,"15min":3.0539999999999998},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57999999999999996,"15min":0.57799999999999996},"min":{"1min":0.433,"5min":0.34200000000000003,"15min":0.313},"max":{"1min":0.84999999999999998,"5min":1.4119999999999999,"15min":6.9409999999999998},"last":0.73299999999999998}]},{"osd":4,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.68300000000000005,"15min":0.73199999999999998},"min":{"1min":0.34300000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.90400000000000003,"5min":1.962,"15min":5.6630000000000003},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.72399999999999998,"15min":0.75900000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.151,"5min":1.6719999999999999,"15min":8.8960000000000008},"last":0.88700000000000001}]},{"osd":5,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.76000000000000001,"15min":0.77700000000000002},"min":{"1min":0.48399999999999999,"5min":0.47299999999999998,"15min":0.40300000000000002},"max":{"1min":1.242,"5min":1.242,"15min":6.6159999999999997},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.77800000000000002,"15min":0.79500000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.1220000000000001,"5min":1.7130000000000001,"15min":5.7009999999999996},"last":0.93799999999999994}]},{"osd":6,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.73299999999999998,"15min":0.77300000000000002},"min":{"1min":0.47799999999999998,"5min":0.47299999999999998,"15min":0.39800000000000002},"max":{"1min":0.94199999999999995,"5min":2.0529999999999999,"15min":8.6039999999999992},"last":1.075},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.77200000000000002,"15min":0.75},"min":{"1min":0.50700000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98199999999999998,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.0369999999999999}]},{"osd":7,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76300000000000001,"15min":0.77800000000000002},"min":{"1min":0.46899999999999997,"5min":0.45200000000000001,"15min":0.44800000000000001},"max":{"1min":1.1639999999999999,"5min":1.7470000000000001,"15min":6.6600000000000001},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.78700000000000003,"15min":0.80600000000000005},"min":{"1min":0.50700000000000001,"5min":0.46400000000000002,"15min":0.46200000000000002},"max":{"1min":1.1990000000000001,"5min":1.774,"15min":8.8100000000000005},"last":0.998}]}]},{"osd":0,"up_from":13,"seq":55834575115,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":370924,"kb_used_data":259236,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94000916,"statfs":{"total":96636764160,"available":96256937984,"internally_reserved":0,"allocated":265457664,"data_stored":265061773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,54,40],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.39500000000000002,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.56299999999999994,"5min":1.3029999999999999,"15min":6.8170000000000002},"last":24.483000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.46100000000000002,"15min":0.47499999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.70199999999999996,"5min":1.591,"15min":6.2789999999999999},"last":91.462000000000003}]},{"osd":2,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.51800000000000002,"15min":0.495},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90100000000000002,"5min":1.169,"15min":3.1859999999999999},"last":91.384},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.47699999999999998,"15min":0.46500000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.97499999999999998,"5min":1.266,"15min":1.47},"last":11.782}]},{"osd":3,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56999999999999995,"15min":0.52100000000000002},"min":{"1min":0.40899999999999997,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.67900000000000005,"5min":1.423,"15min":1.8180000000000001},"last":91.557000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.35599999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.93300000000000005,"5min":1.4390000000000001,"15min":1.881},"last":11.835000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.68799999999999994,"15min":0.80700000000000005},"min":{"1min":0.46400000000000002,"5min":0.45100000000000001,"15min":0.45000000000000001},"max":{"1min":1.0529999999999999,"5min":1.3169999999999999,"15min":18.41},"last":13.169},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70399999999999996,"15min":0.81399999999999995},"min":{"1min":0.50800000000000001,"5min":0.46300000000000002,"15min":0.42399999999999999},"max":{"1min":1.097,"5min":1.409,"15min":18.998999999999999},"last":91.421999999999997}]},{"osd":5,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73699999999999999,"15min":0.76100000000000001},"min":{"1min":0.52400000000000002,"5min":0.50900000000000001,"15min":0.42799999999999999},"max":{"1min":1.1299999999999999,"5min":1.3460000000000001,"15min":3.5499999999999998},"last":79.207999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69999999999999996,"15min":0.73699999999999999},"min":{"1min":0.45900000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.996,"5min":1.4279999999999999,"15min":3.605},"last":91.587999999999994}]},{"osd":6,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.73999999999999999,"15min":0.754},"min":{"1min":0.52200000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0289999999999999,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":9.8290000000000006},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.76600000000000001,"15min":0.78200000000000003},"min":{"1min":0.502,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0840000000000001,"5min":2.367,"15min":2.367},"last":91.522999999999996}]},{"osd":7,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.49099999999999999},"max":{"1min":1.042,"5min":1.395,"15min":4.7839999999999998},"last":91.494},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73999999999999999,"15min":0.77000000000000002},"min":{"1min":0.57399999999999995,"5min":0.53200000000000003,"15min":0.45500000000000002},"max":{"1min":0.995,"5min":1.3759999999999999,"15min":2.0110000000000001},"last":91.346000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39333888,"data_stored":39301023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51634176,"data_stored":51607948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18219008,"data_stored":18187806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":47730688,"data_stored":47702623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56041472,"data_stored":56013946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":44449792,"data_stored":44407786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40529920,"data_stored":40501960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":60231680,"data_stored":60196990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":224890880,"data_stored":224890880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":316989440,"data_stored":316989440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186216448,"data_stored":186216448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":320110592,"data_stored":320110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":240496640,"data_stored":240496640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":185778176,"data_stored":185778176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266395648,"data_stored":266395648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":226021376,"data_stored":226021376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T20:58:44.103 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T20:58:44.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.278+0000 7fb86a079640 1 Processor -- start 2023-10-16T20:58:44.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.278+0000 7fb86a079640 1 -- start start 2023-10-16T20:58:44.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.282+0000 7fb86a079640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb8641406d0 0x7fb8640c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:44.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.282+0000 7fb86a079640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb86406a970 0x7fb864071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:44.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.282+0000 7fb86a079640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86415b3c0 con 0x7fb864071e60 2023-10-16T20:58:44.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.282+0000 7fb86a079640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86415b680 con 0x7fb8641406d0 2023-10-16T20:58:44.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.282+0000 7fb86a079640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb864157570 con 0x7fb86406a970 2023-10-16T20:58:44.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.282+0000 7fb8637fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb8641406d0 0x7fb8640c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:44.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.282+0000 7fb8637fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb8641406d0 0x7fb8640c1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54686/0 (socket says 172.21.15.88:54686) 2023-10-16T20:58:44.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.282+0000 7fb8637fe640 1 -- 172.21.15.88:0/44425696 learned_addr learned my addr 172.21.15.88:0/44425696 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:44.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.282+0000 7fb862ffd640 1 --2- 172.21.15.88:0/44425696 >> v2:172.21.15.191:3300/0 conn(0x7fb86406a970 0x7fb864071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:44.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.282+0000 7fb8637fe640 1 -- 172.21.15.88:0/44425696 >> v2:172.21.15.191:3300/0 conn(0x7fb86406a970 msgr2=0x7fb864071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:44.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.282+0000 7fb8637fe640 1 --2- 172.21.15.88:0/44425696 >> v2:172.21.15.191:3300/0 conn(0x7fb86406a970 0x7fb864071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.282+0000 7fb8637fe640 1 -- 172.21.15.88:0/44425696 >> v1:172.21.15.191:6789/0 conn(0x7fb864071e60 legacy=0x7fb86415a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:44.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.282+0000 7fb8637fe640 1 -- 172.21.15.88:0/44425696 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb864157850 con 0x7fb8641406d0 2023-10-16T20:58:44.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb8637fe640 1 --2- 172.21.15.88:0/44425696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8641406d0 0x7fb8640c1d50 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fb864151ab0 tx=0x7fb85401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e189935e9831ae4c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:44.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb8627fc640 1 -- 172.21.15.88:0/44425696 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8540ab670 con 0x7fb8641406d0 2023-10-16T20:58:44.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb8627fc640 1 -- 172.21.15.88:0/44425696 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8540ab7d0 con 0x7fb8641406d0 2023-10-16T20:58:44.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb8627fc640 1 -- 172.21.15.88:0/44425696 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8540abb60 con 0x7fb8641406d0 2023-10-16T20:58:44.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb86a079640 1 -- 172.21.15.88:0/44425696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8641406d0 msgr2=0x7fb8640c1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:44.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb86a079640 1 --2- 172.21.15.88:0/44425696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8641406d0 0x7fb8640c1d50 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fb864151ab0 tx=0x7fb85401c490 comp rx=0 tx=0).stop 2023-10-16T20:58:44.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb86a079640 1 -- 172.21.15.88:0/44425696 shutdown_connections 2023-10-16T20:58:44.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb86a079640 1 --2- 172.21.15.88:0/44425696 >> v2:172.21.15.191:3300/0 conn(0x7fb86406a970 0x7fb864071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb86a079640 1 --2- 172.21.15.88:0/44425696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8641406d0 0x7fb8640c1d50 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb86a079640 1 -- 172.21.15.88:0/44425696 >> 172.21.15.88:0/44425696 conn(0x7fb8640cad50 msgr2=0x7fb8640cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:44.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb86a079640 1 -- 172.21.15.88:0/44425696 shutdown_connections 2023-10-16T20:58:44.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb86a079640 1 -- 172.21.15.88:0/44425696 wait complete. 2023-10-16T20:58:44.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb86a079640 1 Processor -- start 2023-10-16T20:58:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb86a079640 1 -- start start 2023-10-16T20:58:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb86a079640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb86406a970 0x7fb8641599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb86a079640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb864071e60 0x7fb864155c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb86a079640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8641406d0 0x7fb8641361a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:44.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb86a079640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb864155780 con 0x7fb864071e60 2023-10-16T20:58:44.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb86a079640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86413eea0 con 0x7fb8641406d0 2023-10-16T20:58:44.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.286+0000 7fb86a079640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8640c6050 con 0x7fb86406a970 2023-10-16T20:58:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb862ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb864071e60 0x7fb864155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb862ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb864071e60 0x7fb864155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54702/0 (socket says 172.21.15.88:54702) 2023-10-16T20:58:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb862ffd640 1 -- 172.21.15.88:0/3573835136 learned_addr learned my addr 172.21.15.88:0/3573835136 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb8637fe640 1 --2- 172.21.15.88:0/3573835136 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb86406a970 0x7fb8641599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb863fff640 1 --2- 172.21.15.88:0/3573835136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8641406d0 0x7fb8641361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb862ffd640 1 -- 172.21.15.88:0/3573835136 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb86406a970 msgr2=0x7fb8641599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb862ffd640 1 --2- 172.21.15.88:0/3573835136 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb86406a970 0x7fb8641599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb862ffd640 1 -- 172.21.15.88:0/3573835136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8641406d0 msgr2=0x7fb8641361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:44.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb862ffd640 1 --2- 172.21.15.88:0/3573835136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8641406d0 0x7fb8641361a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb862ffd640 1 -- 172.21.15.88:0/3573835136 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8540094d0 con 0x7fb864071e60 2023-10-16T20:58:44.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb862ffd640 1 --2- 172.21.15.88:0/3573835136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb864071e60 0x7fb864155c10 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fb85800d800 tx=0x7fb85800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:44.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb84bfff640 1 -- 172.21.15.88:0/3573835136 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85800b6c0 con 0x7fb864071e60 2023-10-16T20:58:44.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb84bfff640 1 -- 172.21.15.88:0/3573835136 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85800b820 con 0x7fb864071e60 2023-10-16T20:58:44.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb86a079640 1 -- 172.21.15.88:0/3573835136 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8641581d0 con 0x7fb864071e60 2023-10-16T20:58:44.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb84bfff640 1 -- 172.21.15.88:0/3573835136 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85800bb90 con 0x7fb864071e60 2023-10-16T20:58:44.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb86a079640 1 -- 172.21.15.88:0/3573835136 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb864155130 con 0x7fb864071e60 2023-10-16T20:58:44.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb86a079640 1 -- 172.21.15.88:0/3573835136 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8641571c0 con 0x7fb864071e60 2023-10-16T20:58:44.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.290+0000 7fb84bfff640 1 -- 172.21.15.88:0/3573835136 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb85800bcf0 con 0x7fb864071e60 2023-10-16T20:58:44.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.294+0000 7fb84bfff640 1 --2- 172.21.15.88:0/3573835136 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb8500664a0 0x7fb850068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:44.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.294+0000 7fb84bfff640 1 -- 172.21.15.88:0/3573835136 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7fb858084a00 con 0x7fb864071e60 2023-10-16T20:58:44.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.294+0000 7fb8637fe640 1 --2- 172.21.15.88:0/3573835136 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb8500664a0 0x7fb850068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:44.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.294+0000 7fb8637fe640 1 --2- 172.21.15.88:0/3573835136 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb8500664a0 0x7fb850068960 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb8540af000 tx=0x7fb8540b6040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:44.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.298+0000 7fb84bfff640 1 -- 172.21.15.88:0/3573835136 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb858051790 con 0x7fb864071e60 2023-10-16T20:58:44.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.442+0000 7fb86a079640 1 -- 172.21.15.88:0/3573835136 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb864158050 con 0x7fb8500664a0 2023-10-16T20:58:44.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.454+0000 7fb84bfff640 1 -- 172.21.15.88:0/3573835136 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+374317 (secure 0 0 0) 0x7fb8540ab940 con 0x7fb8500664a0 2023-10-16T20:58:44.458 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:44.460 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-16T20:58:44.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.458+0000 7fb86a079640 1 -- 172.21.15.88:0/3573835136 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb8500664a0 msgr2=0x7fb850068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:44.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.458+0000 7fb86a079640 1 --2- 172.21.15.88:0/3573835136 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb8500664a0 0x7fb850068960 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb8540af000 tx=0x7fb8540b6040 comp rx=0 tx=0).stop 2023-10-16T20:58:44.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.458+0000 7fb86a079640 1 -- 172.21.15.88:0/3573835136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb864071e60 msgr2=0x7fb864155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:44.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.458+0000 7fb86a079640 1 --2- 172.21.15.88:0/3573835136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb864071e60 0x7fb864155c10 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fb85800d800 tx=0x7fb85800dcc0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.458+0000 7fb86a079640 1 -- 172.21.15.88:0/3573835136 shutdown_connections 2023-10-16T20:58:44.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.458+0000 7fb86a079640 1 --2- 172.21.15.88:0/3573835136 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fb8500664a0 0x7fb850068960 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.458+0000 7fb86a079640 1 --2- 172.21.15.88:0/3573835136 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb8641406d0 0x7fb8641361a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.458+0000 7fb86a079640 1 --2- 172.21.15.88:0/3573835136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb864071e60 0x7fb864155c10 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.458+0000 7fb86a079640 1 --2- 172.21.15.88:0/3573835136 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fb86406a970 0x7fb8641599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.458+0000 7fb86a079640 1 -- 172.21.15.88:0/3573835136 >> 172.21.15.88:0/3573835136 conn(0x7fb8640cad50 msgr2=0x7fb8640c9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:44.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.458+0000 7fb86a079640 1 -- 172.21.15.88:0/3573835136 shutdown_connections 2023-10-16T20:58:44.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.458+0000 7fb86a079640 1 -- 172.21.15.88:0/3573835136 wait complete. 2023-10-16T20:58:44.478 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":722,"stamp":"2023-10-16T20:58:43.931001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1103869876,"num_objects":10494,"num_object_clones":0,"num_object_copies":41576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10494,"num_whiteouts":0,"num_read":31781,"num_read_kb":882609,"num_write":27510,"num_write_kb":1485755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":115,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34039,"ondisk_log_size":34039,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3214536,"kb_used_data":2277960,"kb_used_omap":12,"kb_used_meta":936499,"kb_avail":751760184,"statfs":{"total":773094113280,"available":769802428416,"internally_reserved":0,"allocated":2332631040,"data_stored":2329499415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":958975545},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,54,40],"upper_bound":128},"perf_stat":{"commit_latency_ms":145,"apply_latency_ms":145,"commit_latency_ns":145000000,"apply_latency_ns":145000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4254655,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":49,"num_write":147,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003434"},"pg_stats":[{"pgid":"2.1f","version":"67'323","reported_seq":411,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:39.469805+0000","last_change":"2023-10-16T20:36:54.763744+0000","last_active":"2023-10-16T20:58:39.469805+0000","last_peered":"2023-10-16T20:58:39.469805+0000","last_clean":"2023-10-16T20:58:39.469805+0000","last_became_active":"2023-10-16T20:36:54.763457+0000","last_became_peered":"2023-10-16T20:36:54.763457+0000","last_unstale":"2023-10-16T20:58:39.469805+0000","last_undegraded":"2023-10-16T20:58:39.469805+0000","last_fullsized":"2023-10-16T20:58:39.469805+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:05.774485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9864502,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":42,"num_read_kb":9548,"num_write":321,"num_write_kb":9823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"67'107","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.869873+0000","last_change":"2023-10-16T20:47:31.588044+0000","last_active":"2023-10-16T20:58:37.869873+0000","last_peered":"2023-10-16T20:58:37.869873+0000","last_clean":"2023-10-16T20:58:37.869873+0000","last_became_active":"2023-10-16T20:36:55.302519+0000","last_became_peered":"2023-10-16T20:36:55.302519+0000","last_unstale":"2023-10-16T20:58:37.869873+0000","last_undegraded":"2023-10-16T20:58:37.869873+0000","last_fullsized":"2023-10-16T20:58:37.869873+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:49.037032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007784165,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'672","reported_seq":3556,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.645718+0000","last_change":"2023-10-16T20:47:31.584576+0000","last_active":"2023-10-16T20:58:22.645718+0000","last_peered":"2023-10-16T20:58:22.645718+0000","last_clean":"2023-10-16T20:58:22.645718+0000","last_became_active":"2023-10-16T20:36:59.466306+0000","last_became_peered":"2023-10-16T20:36:59.466306+0000","last_unstale":"2023-10-16T20:58:22.645718+0000","last_undegraded":"2023-10-16T20:58:22.645718+0000","last_fullsized":"2023-10-16T20:58:22.645718+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:56.295661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073106899999999999,"stat_sum":{"num_bytes":36865036,"num_objects":299,"num_object_clones":0,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":299,"num_whiteouts":0,"num_read":779,"num_read_kb":25449,"num_write":671,"num_write_kb":56816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"67'207","reported_seq":277,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:00.972618+0000","last_change":"2023-10-16T20:36:54.759491+0000","last_active":"2023-10-16T20:58:00.972618+0000","last_peered":"2023-10-16T20:58:00.972618+0000","last_clean":"2023-10-16T20:58:00.972618+0000","last_became_active":"2023-10-16T20:36:54.759177+0000","last_became_peered":"2023-10-16T20:36:54.759177+0000","last_unstale":"2023-10-16T20:58:00.972618+0000","last_undegraded":"2023-10-16T20:58:00.972618+0000","last_fullsized":"2023-10-16T20:58:00.972618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:43:36.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":12288,"num_write":199,"num_write_kb":8312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"67'107","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889862+0000","last_change":"2023-10-16T20:47:31.488681+0000","last_active":"2023-10-16T20:58:37.889862+0000","last_peered":"2023-10-16T20:58:37.889862+0000","last_clean":"2023-10-16T20:58:37.889862+0000","last_became_active":"2023-10-16T20:36:55.291458+0000","last_became_peered":"2023-10-16T20:36:55.291458+0000","last_unstale":"2023-10-16T20:58:37.889862+0000","last_undegraded":"2023-10-16T20:58:37.889862+0000","last_fullsized":"2023-10-16T20:58:37.889862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:03:05.558721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087429459999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'682","reported_seq":3570,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.576451+0000","last_change":"2023-10-16T20:47:31.484150+0000","last_active":"2023-10-16T20:58:22.576451+0000","last_peered":"2023-10-16T20:58:22.576451+0000","last_clean":"2023-10-16T20:58:22.576451+0000","last_became_active":"2023-10-16T20:36:59.302968+0000","last_became_peered":"2023-10-16T20:36:59.302968+0000","last_unstale":"2023-10-16T20:58:22.576451+0000","last_undegraded":"2023-10-16T20:58:22.576451+0000","last_fullsized":"2023-10-16T20:58:22.576451+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:58:43.745409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027985639999999999,"stat_sum":{"num_bytes":24765378,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":783,"num_read_kb":25781,"num_write":678,"num_write_kb":40921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"67'340","reported_seq":437,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:33.688387+0000","last_change":"2023-10-16T20:36:54.762600+0000","last_active":"2023-10-16T20:58:33.688387+0000","last_peered":"2023-10-16T20:58:33.688387+0000","last_clean":"2023-10-16T20:58:33.688387+0000","last_became_active":"2023-10-16T20:36:54.762278+0000","last_became_peered":"2023-10-16T20:36:54.762278+0000","last_unstale":"2023-10-16T20:58:33.688387+0000","last_undegraded":"2023-10-16T20:58:33.688387+0000","last_fullsized":"2023-10-16T20:58:33.688387+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:46:03.836415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":85,"num_read_kb":16492,"num_write":374,"num_write_kb":12523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'96","reported_seq":2197,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.131579+0000","last_change":"2023-10-16T20:47:31.496372+0000","last_active":"2023-10-16T20:58:38.131579+0000","last_peered":"2023-10-16T20:58:38.131579+0000","last_clean":"2023-10-16T20:58:38.131579+0000","last_became_active":"2023-10-16T20:37:37.402987+0000","last_became_peered":"2023-10-16T20:37:37.402987+0000","last_unstale":"2023-10-16T20:58:38.131579+0000","last_undegraded":"2023-10-16T20:58:38.131579+0000","last_fullsized":"2023-10-16T20:58:38.131579+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:12:06.964995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082573900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"67'673","reported_seq":4224,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.678192+0000","last_change":"2023-10-16T20:47:31.485701+0000","last_active":"2023-10-16T20:58:22.678192+0000","last_peered":"2023-10-16T20:58:22.678192+0000","last_clean":"2023-10-16T20:58:22.678192+0000","last_became_active":"2023-10-16T20:36:59.303805+0000","last_became_peered":"2023-10-16T20:36:59.303805+0000","last_unstale":"2023-10-16T20:58:22.678192+0000","last_undegraded":"2023-10-16T20:58:22.678192+0000","last_fullsized":"2023-10-16T20:58:22.678192+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:28:11.659597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041657500000000002,"stat_sum":{"num_bytes":49691093,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":1446,"num_read_kb":31388,"num_write":673,"num_write_kb":52938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"67'172","reported_seq":252,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.691336+0000","last_change":"2023-10-16T20:36:54.749849+0000","last_active":"2023-10-16T20:58:28.691336+0000","last_peered":"2023-10-16T20:58:28.691336+0000","last_clean":"2023-10-16T20:58:28.691336+0000","last_became_active":"2023-10-16T20:36:54.749316+0000","last_became_peered":"2023-10-16T20:36:54.749316+0000","last_unstale":"2023-10-16T20:58:28.691336+0000","last_undegraded":"2023-10-16T20:58:28.691336+0000","last_fullsized":"2023-10-16T20:58:28.691336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:49:29.077126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":170,"num_write_kb":4198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"67'104","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.872186+0000","last_change":"2023-10-16T20:47:31.592091+0000","last_active":"2023-10-16T20:58:37.872186+0000","last_peered":"2023-10-16T20:58:37.872186+0000","last_clean":"2023-10-16T20:58:37.872186+0000","last_became_active":"2023-10-16T20:36:55.302387+0000","last_became_peered":"2023-10-16T20:36:55.302387+0000","last_unstale":"2023-10-16T20:58:37.872186+0000","last_undegraded":"2023-10-16T20:58:37.872186+0000","last_fullsized":"2023-10-16T20:58:37.872186+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:07:35.871205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011585343,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'641","reported_seq":3839,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.036707+0000","last_change":"2023-10-16T20:47:31.486382+0000","last_active":"2023-10-16T20:58:23.036707+0000","last_peered":"2023-10-16T20:58:23.036707+0000","last_clean":"2023-10-16T20:58:23.036707+0000","last_became_active":"2023-10-16T20:36:59.305228+0000","last_became_peered":"2023-10-16T20:36:59.305228+0000","last_unstale":"2023-10-16T20:58:23.036707+0000","last_undegraded":"2023-10-16T20:58:23.036707+0000","last_fullsized":"2023-10-16T20:58:23.036707+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":641,"log_dups_size":0,"ondisk_log_size":641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:32:37.261529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047877249999999996,"stat_sum":{"num_bytes":34900728,"num_objects":310,"num_object_clones":0,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":1093,"num_read_kb":21879,"num_write":638,"num_write_kb":46703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"67'104","reported_seq":2214,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.863809+0000","last_change":"2023-10-16T20:47:31.592641+0000","last_active":"2023-10-16T20:58:37.863809+0000","last_peered":"2023-10-16T20:58:37.863809+0000","last_clean":"2023-10-16T20:58:37.863809+0000","last_became_active":"2023-10-16T20:36:55.306427+0000","last_became_peered":"2023-10-16T20:36:55.306427+0000","last_unstale":"2023-10-16T20:58:37.863809+0000","last_undegraded":"2023-10-16T20:58:37.863809+0000","last_fullsized":"2023-10-16T20:58:37.863809+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:17:46.390047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013854674000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"67'187","reported_seq":258,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.360075+0000","last_change":"2023-10-16T20:36:54.768375+0000","last_active":"2023-10-16T20:58:05.360075+0000","last_peered":"2023-10-16T20:58:05.360075+0000","last_clean":"2023-10-16T20:58:05.360075+0000","last_became_active":"2023-10-16T20:36:54.767996+0000","last_became_peered":"2023-10-16T20:36:54.767996+0000","last_unstale":"2023-10-16T20:58:05.360075+0000","last_undegraded":"2023-10-16T20:58:05.360075+0000","last_fullsized":"2023-10-16T20:58:05.360075+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:55.303942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":8202,"num_write":173,"num_write_kb":8293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"67'648","reported_seq":4473,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.721394+0000","last_change":"2023-10-16T20:47:31.487925+0000","last_active":"2023-10-16T20:58:22.721394+0000","last_peered":"2023-10-16T20:58:22.721394+0000","last_clean":"2023-10-16T20:58:22.721394+0000","last_became_active":"2023-10-16T20:36:59.306640+0000","last_became_peered":"2023-10-16T20:36:59.306640+0000","last_unstale":"2023-10-16T20:58:22.721394+0000","last_undegraded":"2023-10-16T20:58:22.721394+0000","last_fullsized":"2023-10-16T20:58:22.721394+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:27:48.041001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054537470000000001,"stat_sum":{"num_bytes":41205181,"num_objects":318,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":1719,"num_read_kb":37802,"num_write":648,"num_write_kb":48776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"67'101","reported_seq":2210,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.891454+0000","last_change":"2023-10-16T20:47:31.587617+0000","last_active":"2023-10-16T20:58:37.891454+0000","last_peered":"2023-10-16T20:58:37.891454+0000","last_clean":"2023-10-16T20:58:37.891454+0000","last_became_active":"2023-10-16T20:36:55.275085+0000","last_became_peered":"2023-10-16T20:36:55.275085+0000","last_unstale":"2023-10-16T20:58:37.891454+0000","last_undegraded":"2023-10-16T20:58:37.891454+0000","last_fullsized":"2023-10-16T20:58:37.891454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:15.882497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087599630000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'182","reported_seq":251,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:41.511040+0000","last_change":"2023-10-16T20:36:54.756174+0000","last_active":"2023-10-16T20:58:41.511040+0000","last_peered":"2023-10-16T20:58:41.511040+0000","last_clean":"2023-10-16T20:58:41.511040+0000","last_became_active":"2023-10-16T20:36:54.755872+0000","last_became_peered":"2023-10-16T20:36:54.755872+0000","last_unstale":"2023-10-16T20:58:41.511040+0000","last_undegraded":"2023-10-16T20:58:41.511040+0000","last_fullsized":"2023-10-16T20:58:41.511040+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:32.741671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":10,"num_write":281,"num_write_kb":4257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"67'699","reported_seq":3454,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.718374+0000","last_change":"2023-10-16T20:47:31.486593+0000","last_active":"2023-10-16T20:58:22.718374+0000","last_peered":"2023-10-16T20:58:22.718374+0000","last_clean":"2023-10-16T20:58:22.718374+0000","last_became_active":"2023-10-16T20:36:59.302919+0000","last_became_peered":"2023-10-16T20:36:59.302919+0000","last_unstale":"2023-10-16T20:58:22.718374+0000","last_undegraded":"2023-10-16T20:58:22.718374+0000","last_fullsized":"2023-10-16T20:58:22.718374+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:48.779648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046546790000000001,"stat_sum":{"num_bytes":20793675,"num_objects":334,"num_object_clones":0,"num_object_copies":1336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":334,"num_whiteouts":0,"num_read":649,"num_read_kb":16394,"num_write":696,"num_write_kb":45235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"67'106","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.886436+0000","last_change":"2023-10-16T20:47:31.586816+0000","last_active":"2023-10-16T20:58:37.886436+0000","last_peered":"2023-10-16T20:58:37.886436+0000","last_clean":"2023-10-16T20:58:37.886436+0000","last_became_active":"2023-10-16T20:36:55.270455+0000","last_became_peered":"2023-10-16T20:36:55.270455+0000","last_unstale":"2023-10-16T20:58:37.886436+0000","last_undegraded":"2023-10-16T20:58:37.886436+0000","last_fullsized":"2023-10-16T20:58:37.886436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:37.316278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.008577889,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"66'49","reported_seq":127,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.707520+0000","last_change":"2023-10-16T20:36:54.744128+0000","last_active":"2023-10-16T20:58:28.707520+0000","last_peered":"2023-10-16T20:58:28.707520+0000","last_clean":"2023-10-16T20:58:28.707520+0000","last_became_active":"2023-10-16T20:36:54.743805+0000","last_became_peered":"2023-10-16T20:36:54.743805+0000","last_unstale":"2023-10-16T20:58:28.707520+0000","last_undegraded":"2023-10-16T20:58:28.707520+0000","last_fullsized":"2023-10-16T20:58:28.707520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:54:35.819820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":55,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'651","reported_seq":3692,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.352813+0000","last_change":"2023-10-16T20:47:31.485640+0000","last_active":"2023-10-16T20:58:22.352813+0000","last_peered":"2023-10-16T20:58:22.352813+0000","last_clean":"2023-10-16T20:58:22.352813+0000","last_became_active":"2023-10-16T20:36:59.465172+0000","last_became_peered":"2023-10-16T20:36:59.465172+0000","last_unstale":"2023-10-16T20:58:22.352813+0000","last_undegraded":"2023-10-16T20:58:22.352813+0000","last_fullsized":"2023-10-16T20:58:22.352813+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:18:14.804407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040107390000000001,"stat_sum":{"num_bytes":25409516,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":935,"num_read_kb":27643,"num_write":650,"num_write_kb":33262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"67'103","reported_seq":2213,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.849592+0000","last_change":"2023-10-16T20:47:31.487014+0000","last_active":"2023-10-16T20:58:37.849592+0000","last_peered":"2023-10-16T20:58:37.849592+0000","last_clean":"2023-10-16T20:58:37.849592+0000","last_became_active":"2023-10-16T20:36:55.278568+0000","last_became_peered":"2023-10-16T20:36:55.278568+0000","last_unstale":"2023-10-16T20:58:37.849592+0000","last_undegraded":"2023-10-16T20:58:37.849592+0000","last_fullsized":"2023-10-16T20:58:37.849592+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:01:00.125646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050399599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"66'287","reported_seq":358,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:01.079180+0000","last_change":"2023-10-16T20:36:54.758778+0000","last_active":"2023-10-16T20:58:01.079180+0000","last_peered":"2023-10-16T20:58:01.079180+0000","last_clean":"2023-10-16T20:58:01.079180+0000","last_became_active":"2023-10-16T20:36:54.758496+0000","last_became_peered":"2023-10-16T20:36:54.758496+0000","last_unstale":"2023-10-16T20:58:01.079180+0000","last_undegraded":"2023-10-16T20:58:01.079180+0000","last_fullsized":"2023-10-16T20:58:01.079180+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:36:00.515106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":12294,"num_write":291,"num_write_kb":4310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"67'586","reported_seq":3861,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.527756+0000","last_change":"2023-10-16T20:47:31.488200+0000","last_active":"2023-10-16T20:58:22.527756+0000","last_peered":"2023-10-16T20:58:22.527756+0000","last_clean":"2023-10-16T20:58:22.527756+0000","last_became_active":"2023-10-16T20:36:59.309068+0000","last_became_peered":"2023-10-16T20:36:59.309068+0000","last_unstale":"2023-10-16T20:58:22.527756+0000","last_undegraded":"2023-10-16T20:58:22.527756+0000","last_fullsized":"2023-10-16T20:58:22.527756+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:22:11.244977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055482439999999999,"stat_sum":{"num_bytes":36900597,"num_objects":303,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1169,"num_read_kb":30992,"num_write":585,"num_write_kb":40444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'101","reported_seq":2211,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889241+0000","last_change":"2023-10-16T20:47:31.492407+0000","last_active":"2023-10-16T20:58:37.889241+0000","last_peered":"2023-10-16T20:58:37.889241+0000","last_clean":"2023-10-16T20:58:37.889241+0000","last_became_active":"2023-10-16T20:36:55.272840+0000","last_became_peered":"2023-10-16T20:36:55.272840+0000","last_unstale":"2023-10-16T20:58:37.889241+0000","last_undegraded":"2023-10-16T20:58:37.889241+0000","last_fullsized":"2023-10-16T20:58:37.889241+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:35:42.200237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011897911000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"30'8","reported_seq":88,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.710153+0000","last_change":"2023-10-16T20:36:54.760681+0000","last_active":"2023-10-16T20:58:28.710153+0000","last_peered":"2023-10-16T20:58:28.710153+0000","last_clean":"2023-10-16T20:58:28.710153+0000","last_became_active":"2023-10-16T20:36:54.760397+0000","last_became_peered":"2023-10-16T20:36:54.760397+0000","last_unstale":"2023-10-16T20:58:28.710153+0000","last_undegraded":"2023-10-16T20:58:28.710153+0000","last_fullsized":"2023-10-16T20:58:28.710153+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:47:16.510382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"67'710","reported_seq":3798,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.025282+0000","last_change":"2023-10-16T20:47:31.485980+0000","last_active":"2023-10-16T20:58:23.025282+0000","last_peered":"2023-10-16T20:58:23.025282+0000","last_clean":"2023-10-16T20:58:23.025282+0000","last_became_active":"2023-10-16T20:36:59.465152+0000","last_became_peered":"2023-10-16T20:36:59.465152+0000","last_unstale":"2023-10-16T20:58:23.025282+0000","last_undegraded":"2023-10-16T20:58:23.025282+0000","last_fullsized":"2023-10-16T20:58:23.025282+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:48.800573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044307399999999999,"stat_sum":{"num_bytes":25809469,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":981,"num_read_kb":15497,"num_write":706,"num_write_kb":50141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'106","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.882745+0000","last_change":"2023-10-16T20:47:31.587090+0000","last_active":"2023-10-16T20:58:37.882745+0000","last_peered":"2023-10-16T20:58:37.882745+0000","last_clean":"2023-10-16T20:58:37.882745+0000","last_became_active":"2023-10-16T20:36:55.285566+0000","last_became_peered":"2023-10-16T20:36:55.285566+0000","last_unstale":"2023-10-16T20:58:37.882745+0000","last_undegraded":"2023-10-16T20:58:37.882745+0000","last_fullsized":"2023-10-16T20:58:37.882745+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:18.835080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085500479999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'101","reported_seq":177,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.657997+0000","last_change":"2023-10-16T20:36:54.758062+0000","last_active":"2023-10-16T20:58:37.657997+0000","last_peered":"2023-10-16T20:58:37.657997+0000","last_clean":"2023-10-16T20:58:37.657997+0000","last_became_active":"2023-10-16T20:36:54.757796+0000","last_became_peered":"2023-10-16T20:36:54.757796+0000","last_unstale":"2023-10-16T20:58:37.657997+0000","last_undegraded":"2023-10-16T20:58:37.657997+0000","last_fullsized":"2023-10-16T20:58:37.657997+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:02:09.953795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":254,"num_write_kb":1037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"67'686","reported_seq":3331,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.536581+0000","last_change":"2023-10-16T20:47:31.582989+0000","last_active":"2023-10-16T20:58:22.536581+0000","last_peered":"2023-10-16T20:58:22.536581+0000","last_clean":"2023-10-16T20:58:22.536581+0000","last_became_active":"2023-10-16T20:36:59.465221+0000","last_became_peered":"2023-10-16T20:36:59.465221+0000","last_unstale":"2023-10-16T20:58:22.536581+0000","last_undegraded":"2023-10-16T20:58:22.536581+0000","last_fullsized":"2023-10-16T20:58:22.536581+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:29:08.900666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059334330000000001,"stat_sum":{"num_bytes":20326999,"num_objects":343,"num_object_clones":0,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":538,"num_read_kb":13756,"num_write":685,"num_write_kb":32496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"67'101","reported_seq":2211,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.818833+0000","last_change":"2023-10-16T20:47:31.489294+0000","last_active":"2023-10-16T20:58:37.818833+0000","last_peered":"2023-10-16T20:58:37.818833+0000","last_clean":"2023-10-16T20:58:37.818833+0000","last_became_active":"2023-10-16T20:36:55.274432+0000","last_became_peered":"2023-10-16T20:36:55.274432+0000","last_unstale":"2023-10-16T20:58:37.818833+0000","last_undegraded":"2023-10-16T20:58:37.818833+0000","last_fullsized":"2023-10-16T20:58:37.818833+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:50:04.713610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057461259999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"66'34","reported_seq":109,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:01.080176+0000","last_change":"2023-10-16T20:36:54.762356+0000","last_active":"2023-10-16T20:58:01.080176+0000","last_peered":"2023-10-16T20:58:01.080176+0000","last_clean":"2023-10-16T20:58:01.080176+0000","last_became_active":"2023-10-16T20:36:54.762167+0000","last_became_peered":"2023-10-16T20:36:54.762167+0000","last_unstale":"2023-10-16T20:58:01.080176+0000","last_undegraded":"2023-10-16T20:58:01.080176+0000","last_fullsized":"2023-10-16T20:58:01.080176+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:45.164625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":34,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"67'725","reported_seq":3715,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.031400+0000","last_change":"2023-10-16T20:47:31.489099+0000","last_active":"2023-10-16T20:58:23.031400+0000","last_peered":"2023-10-16T20:58:23.031400+0000","last_clean":"2023-10-16T20:58:23.031400+0000","last_became_active":"2023-10-16T20:36:59.466876+0000","last_became_peered":"2023-10-16T20:36:59.466876+0000","last_unstale":"2023-10-16T20:58:23.031400+0000","last_undegraded":"2023-10-16T20:58:23.031400+0000","last_fullsized":"2023-10-16T20:58:23.031400+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":725,"log_dups_size":0,"ondisk_log_size":725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:45:55.969279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009058969,"stat_sum":{"num_bytes":44901233,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":884,"num_read_kb":20657,"num_write":720,"num_write_kb":60576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"67'98","reported_seq":2208,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.895976+0000","last_change":"2023-10-16T20:47:31.594422+0000","last_active":"2023-10-16T20:58:37.895976+0000","last_peered":"2023-10-16T20:58:37.895976+0000","last_clean":"2023-10-16T20:58:37.895976+0000","last_became_active":"2023-10-16T20:36:55.293228+0000","last_became_peered":"2023-10-16T20:36:55.293228+0000","last_unstale":"2023-10-16T20:58:37.895976+0000","last_undegraded":"2023-10-16T20:58:37.895976+0000","last_fullsized":"2023-10-16T20:58:37.895976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:33:58.039426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013567460999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"66'210","reported_seq":278,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:47.652674+0000","last_change":"2023-10-16T20:36:54.757588+0000","last_active":"2023-10-16T20:57:47.652674+0000","last_peered":"2023-10-16T20:57:47.652674+0000","last_clean":"2023-10-16T20:57:47.652674+0000","last_became_active":"2023-10-16T20:36:54.757308+0000","last_became_peered":"2023-10-16T20:36:54.757308+0000","last_unstale":"2023-10-16T20:57:47.652674+0000","last_undegraded":"2023-10-16T20:57:47.652674+0000","last_fullsized":"2023-10-16T20:57:47.652674+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:51.298239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":3049,"num_write":207,"num_write_kb":4195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"67'671","reported_seq":3425,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.522842+0000","last_change":"2023-10-16T20:47:31.485489+0000","last_active":"2023-10-16T20:58:22.522842+0000","last_peered":"2023-10-16T20:58:22.522842+0000","last_clean":"2023-10-16T20:58:22.522842+0000","last_became_active":"2023-10-16T20:36:59.303224+0000","last_became_peered":"2023-10-16T20:36:59.303224+0000","last_unstale":"2023-10-16T20:58:22.522842+0000","last_undegraded":"2023-10-16T20:58:22.522842+0000","last_fullsized":"2023-10-16T20:58:22.522842+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:03:55.965341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038422399999999998,"stat_sum":{"num_bytes":21817454,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":647,"num_read_kb":9306,"num_write":673,"num_write_kb":37999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"67'120","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889222+0000","last_change":"2023-10-16T20:47:31.487515+0000","last_active":"2023-10-16T20:58:37.889222+0000","last_peered":"2023-10-16T20:58:37.889222+0000","last_clean":"2023-10-16T20:58:37.889222+0000","last_became_active":"2023-10-16T20:36:55.277872+0000","last_became_peered":"2023-10-16T20:36:55.277872+0000","last_unstale":"2023-10-16T20:58:37.889222+0000","last_undegraded":"2023-10-16T20:58:37.889222+0000","last_fullsized":"2023-10-16T20:58:37.889222+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:46.751931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052323379999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"66'532","reported_seq":616,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.670124+0000","last_change":"2023-10-16T20:36:54.740644+0000","last_active":"2023-10-16T20:58:05.670124+0000","last_peered":"2023-10-16T20:58:05.670124+0000","last_clean":"2023-10-16T20:58:05.670124+0000","last_became_active":"2023-10-16T20:36:54.740340+0000","last_became_peered":"2023-10-16T20:36:54.740340+0000","last_unstale":"2023-10-16T20:58:05.670124+0000","last_undegraded":"2023-10-16T20:58:05.670124+0000","last_fullsized":"2023-10-16T20:58:05.670124+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:22.221727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21285,"num_write":524,"num_write_kb":16645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'623","reported_seq":3877,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.676798+0000","last_change":"2023-10-16T20:47:31.489650+0000","last_active":"2023-10-16T20:58:22.676798+0000","last_peered":"2023-10-16T20:58:22.676798+0000","last_clean":"2023-10-16T20:58:22.676798+0000","last_became_active":"2023-10-16T20:36:59.294548+0000","last_became_peered":"2023-10-16T20:36:59.294548+0000","last_unstale":"2023-10-16T20:58:22.676798+0000","last_undegraded":"2023-10-16T20:58:22.676798+0000","last_fullsized":"2023-10-16T20:58:22.676798+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:12.441064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062280529999999999,"stat_sum":{"num_bytes":29651112,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":1147,"num_read_kb":23932,"num_write":623,"num_write_kb":40690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"67'89","reported_seq":2199,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.807118+0000","last_change":"2023-10-16T20:47:31.484434+0000","last_active":"2023-10-16T20:58:37.807118+0000","last_peered":"2023-10-16T20:58:37.807118+0000","last_clean":"2023-10-16T20:58:37.807118+0000","last_became_active":"2023-10-16T20:36:55.269749+0000","last_became_peered":"2023-10-16T20:36:55.269749+0000","last_unstale":"2023-10-16T20:58:37.807118+0000","last_undegraded":"2023-10-16T20:58:37.807118+0000","last_fullsized":"2023-10-16T20:58:37.807118+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:21:13.945910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003009004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'188","reported_seq":376,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.714114+0000","last_change":"2023-10-16T20:36:54.757903+0000","last_active":"2023-10-16T20:58:37.714114+0000","last_peered":"2023-10-16T20:58:37.714114+0000","last_clean":"2023-10-16T20:58:37.714114+0000","last_became_active":"2023-10-16T20:36:54.757549+0000","last_became_peered":"2023-10-16T20:36:54.757549+0000","last_unstale":"2023-10-16T20:58:37.714114+0000","last_undegraded":"2023-10-16T20:58:37.714114+0000","last_fullsized":"2023-10-16T20:58:37.714114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:02:42.090848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1409828,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":124,"num_read_kb":239,"num_write":184,"num_write_kb":1568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'647","reported_seq":3934,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.025459+0000","last_change":"2023-10-16T20:47:31.487517+0000","last_active":"2023-10-16T20:58:23.025459+0000","last_peered":"2023-10-16T20:58:23.025459+0000","last_clean":"2023-10-16T20:58:23.025459+0000","last_became_active":"2023-10-16T20:36:59.301693+0000","last_became_peered":"2023-10-16T20:36:59.301693+0000","last_unstale":"2023-10-16T20:58:23.025459+0000","last_undegraded":"2023-10-16T20:58:23.025459+0000","last_fullsized":"2023-10-16T20:58:23.025459+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:04:52.612265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056692870000000003,"stat_sum":{"num_bytes":29916858,"num_objects":298,"num_object_clones":0,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":298,"num_whiteouts":0,"num_read":1180,"num_read_kb":25235,"num_write":643,"num_write_kb":45892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"67'104","reported_seq":2214,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888945+0000","last_change":"2023-10-16T20:47:31.487504+0000","last_active":"2023-10-16T20:58:37.888945+0000","last_peered":"2023-10-16T20:58:37.888945+0000","last_clean":"2023-10-16T20:58:37.888945+0000","last_became_active":"2023-10-16T20:36:55.276649+0000","last_became_peered":"2023-10-16T20:36:55.276649+0000","last_unstale":"2023-10-16T20:58:37.888945+0000","last_undegraded":"2023-10-16T20:58:37.888945+0000","last_fullsized":"2023-10-16T20:58:37.888945+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:12:33.903227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076292809999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'128","reported_seq":201,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:01.079234+0000","last_change":"2023-10-16T20:36:54.753592+0000","last_active":"2023-10-16T20:58:01.079234+0000","last_peered":"2023-10-16T20:58:01.079234+0000","last_clean":"2023-10-16T20:58:01.079234+0000","last_became_active":"2023-10-16T20:36:54.753237+0000","last_became_peered":"2023-10-16T20:36:54.753237+0000","last_unstale":"2023-10-16T20:58:01.079234+0000","last_undegraded":"2023-10-16T20:58:01.079234+0000","last_fullsized":"2023-10-16T20:58:01.079234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:53.005979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":20486,"num_write":119,"num_write_kb":8247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'673","reported_seq":3483,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.284780+0000","last_change":"2023-10-16T20:47:31.491504+0000","last_active":"2023-10-16T20:58:23.284780+0000","last_peered":"2023-10-16T20:58:23.284780+0000","last_clean":"2023-10-16T20:58:23.284780+0000","last_became_active":"2023-10-16T20:36:59.466708+0000","last_became_peered":"2023-10-16T20:36:59.466708+0000","last_unstale":"2023-10-16T20:58:23.284780+0000","last_undegraded":"2023-10-16T20:58:23.284780+0000","last_fullsized":"2023-10-16T20:58:23.284780+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:33:22.349924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011259905000000001,"stat_sum":{"num_bytes":19802424,"num_objects":318,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":703,"num_read_kb":14564,"num_write":670,"num_write_kb":31986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"67'99","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.897919+0000","last_change":"2023-10-16T20:47:31.585286+0000","last_active":"2023-10-16T20:58:37.897919+0000","last_peered":"2023-10-16T20:58:37.897919+0000","last_clean":"2023-10-16T20:58:37.897919+0000","last_became_active":"2023-10-16T20:36:55.294610+0000","last_became_peered":"2023-10-16T20:36:55.294610+0000","last_unstale":"2023-10-16T20:58:37.897919+0000","last_undegraded":"2023-10-16T20:58:37.897919+0000","last_fullsized":"2023-10-16T20:58:37.897919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:34:05.113883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075091560000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"66'7","reported_seq":83,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:00.736127+0000","last_change":"2023-10-16T20:36:54.741191+0000","last_active":"2023-10-16T20:58:00.736127+0000","last_peered":"2023-10-16T20:58:00.736127+0000","last_clean":"2023-10-16T20:58:00.736127+0000","last_became_active":"2023-10-16T20:36:54.740944+0000","last_became_peered":"2023-10-16T20:36:54.740944+0000","last_unstale":"2023-10-16T20:58:00.736127+0000","last_undegraded":"2023-10-16T20:58:00.736127+0000","last_fullsized":"2023-10-16T20:58:00.736127+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:05.961850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":23,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":219,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:56:52.651677+0000","last_change":"2023-10-16T20:36:46.390001+0000","last_active":"2023-10-16T20:56:52.651677+0000","last_peered":"2023-10-16T20:56:52.651677+0000","last_clean":"2023-10-16T20:56:52.651677+0000","last_became_active":"2023-10-16T20:36:46.388824+0000","last_became_peered":"2023-10-16T20:36:46.388824+0000","last_unstale":"2023-10-16T20:56:52.651677+0000","last_undegraded":"2023-10-16T20:56:52.651677+0000","last_fullsized":"2023-10-16T20:56:52.651677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_clean_scrub_stamp":"2023-10-16T20:36:44.575506+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:31.004318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"67'669","reported_seq":3425,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.621226+0000","last_change":"2023-10-16T20:47:31.593007+0000","last_active":"2023-10-16T20:58:22.621226+0000","last_peered":"2023-10-16T20:58:22.621226+0000","last_clean":"2023-10-16T20:58:22.621226+0000","last_became_active":"2023-10-16T20:36:59.468082+0000","last_became_peered":"2023-10-16T20:36:59.468082+0000","last_unstale":"2023-10-16T20:58:22.621226+0000","last_undegraded":"2023-10-16T20:58:22.621226+0000","last_fullsized":"2023-10-16T20:58:22.621226+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":669,"log_dups_size":0,"ondisk_log_size":669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:49:06.585943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013154572,"stat_sum":{"num_bytes":30910053,"num_objects":362,"num_object_clones":0,"num_object_copies":1448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":649,"num_read_kb":15567,"num_write":666,"num_write_kb":42817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'110","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.884585+0000","last_change":"2023-10-16T20:47:31.488408+0000","last_active":"2023-10-16T20:58:37.884585+0000","last_peered":"2023-10-16T20:58:37.884585+0000","last_clean":"2023-10-16T20:58:37.884585+0000","last_became_active":"2023-10-16T20:37:37.409004+0000","last_became_peered":"2023-10-16T20:37:37.409004+0000","last_unstale":"2023-10-16T20:58:37.884585+0000","last_undegraded":"2023-10-16T20:58:37.884585+0000","last_fullsized":"2023-10-16T20:58:37.884585+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:21:07.245091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060805590000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"66'68","reported_seq":138,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:19.377575+0000","last_change":"2023-10-16T20:36:54.746600+0000","last_active":"2023-10-16T20:57:19.377575+0000","last_peered":"2023-10-16T20:57:19.377575+0000","last_clean":"2023-10-16T20:57:19.377575+0000","last_became_active":"2023-10-16T20:36:54.746247+0000","last_became_peered":"2023-10-16T20:36:54.746247+0000","last_unstale":"2023-10-16T20:57:19.377575+0000","last_undegraded":"2023-10-16T20:57:19.377575+0000","last_fullsized":"2023-10-16T20:57:19.377575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:28:57.949487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":7042,"num_write":68,"num_write_kb":4224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"67'680","reported_seq":3679,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.607038+0000","last_change":"2023-10-16T20:47:31.590576+0000","last_active":"2023-10-16T20:58:22.607038+0000","last_peered":"2023-10-16T20:58:22.607038+0000","last_clean":"2023-10-16T20:58:22.607038+0000","last_became_active":"2023-10-16T20:37:37.467402+0000","last_became_peered":"2023-10-16T20:37:37.467402+0000","last_unstale":"2023-10-16T20:58:22.607038+0000","last_undegraded":"2023-10-16T20:58:22.607038+0000","last_fullsized":"2023-10-16T20:58:22.607038+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":680,"log_dups_size":0,"ondisk_log_size":680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:32:26.907296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012017773000000001,"stat_sum":{"num_bytes":36935626,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":884,"num_read_kb":21394,"num_write":679,"num_write_kb":40520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"67'106","reported_seq":2215,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.895855+0000","last_change":"2023-10-16T20:47:31.589500+0000","last_active":"2023-10-16T20:58:37.895855+0000","last_peered":"2023-10-16T20:58:37.895855+0000","last_clean":"2023-10-16T20:58:37.895855+0000","last_became_active":"2023-10-16T20:36:55.300958+0000","last_became_peered":"2023-10-16T20:36:55.300958+0000","last_unstale":"2023-10-16T20:58:37.895855+0000","last_undegraded":"2023-10-16T20:58:37.895855+0000","last_fullsized":"2023-10-16T20:58:37.895855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:29:17.323936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009734431,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"67'314","reported_seq":386,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.865738+0000","last_change":"2023-10-16T20:36:54.753770+0000","last_active":"2023-10-16T20:58:37.865738+0000","last_peered":"2023-10-16T20:58:37.865738+0000","last_clean":"2023-10-16T20:58:37.865738+0000","last_became_active":"2023-10-16T20:36:54.753443+0000","last_became_peered":"2023-10-16T20:36:54.753443+0000","last_unstale":"2023-10-16T20:58:37.865738+0000","last_undegraded":"2023-10-16T20:58:37.865738+0000","last_fullsized":"2023-10-16T20:58:37.865738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:21:26.392480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9664034,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":15859,"num_write":307,"num_write_kb":9612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'687","reported_seq":3976,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.741627+0000","last_change":"2023-10-16T20:47:31.492598+0000","last_active":"2023-10-16T20:58:22.741627+0000","last_peered":"2023-10-16T20:58:22.741627+0000","last_clean":"2023-10-16T20:58:22.741627+0000","last_became_active":"2023-10-16T20:36:59.302869+0000","last_became_peered":"2023-10-16T20:36:59.302869+0000","last_unstale":"2023-10-16T20:58:22.741627+0000","last_undegraded":"2023-10-16T20:58:22.741627+0000","last_fullsized":"2023-10-16T20:58:22.741627+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:37:25.463785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012136565,"stat_sum":{"num_bytes":23637013,"num_objects":352,"num_object_clones":0,"num_object_copies":1408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":1183,"num_read_kb":18772,"num_write":687,"num_write_kb":27489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"67'100","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.892995+0000","last_change":"2023-10-16T20:47:31.587528+0000","last_active":"2023-10-16T20:58:37.892995+0000","last_peered":"2023-10-16T20:58:37.892995+0000","last_clean":"2023-10-16T20:58:37.892995+0000","last_became_active":"2023-10-16T20:36:55.291633+0000","last_became_peered":"2023-10-16T20:36:55.291633+0000","last_unstale":"2023-10-16T20:58:37.892995+0000","last_undegraded":"2023-10-16T20:58:37.892995+0000","last_fullsized":"2023-10-16T20:58:37.892995+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:08.345584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085978630000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'120","reported_seq":195,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.767053+0000","last_change":"2023-10-16T20:36:54.748196+0000","last_active":"2023-10-16T20:58:23.767053+0000","last_peered":"2023-10-16T20:58:23.767053+0000","last_clean":"2023-10-16T20:58:23.767053+0000","last_became_active":"2023-10-16T20:36:54.747704+0000","last_became_peered":"2023-10-16T20:36:54.747704+0000","last_unstale":"2023-10-16T20:58:23.767053+0000","last_undegraded":"2023-10-16T20:58:23.767053+0000","last_fullsized":"2023-10-16T20:58:23.767053+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:55.471916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":116,"num_write_kb":4228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"67'663","reported_seq":3386,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.098033+0000","last_change":"2023-10-16T20:47:31.595065+0000","last_active":"2023-10-16T20:58:23.098033+0000","last_peered":"2023-10-16T20:58:23.098033+0000","last_clean":"2023-10-16T20:58:23.098033+0000","last_became_active":"2023-10-16T20:36:59.467291+0000","last_became_peered":"2023-10-16T20:36:59.467291+0000","last_unstale":"2023-10-16T20:58:23.098033+0000","last_undegraded":"2023-10-16T20:58:23.098033+0000","last_fullsized":"2023-10-16T20:58:23.098033+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:15.723182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0131915,"stat_sum":{"num_bytes":23136394,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":616,"num_read_kb":10065,"num_write":660,"num_write_kb":35181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"67'80","reported_seq":2189,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.876288+0000","last_change":"2023-10-16T20:47:31.589747+0000","last_active":"2023-10-16T20:58:37.876288+0000","last_peered":"2023-10-16T20:58:37.876288+0000","last_clean":"2023-10-16T20:58:37.876288+0000","last_became_active":"2023-10-16T20:36:55.290998+0000","last_became_peered":"2023-10-16T20:36:55.290998+0000","last_unstale":"2023-10-16T20:58:37.876288+0000","last_undegraded":"2023-10-16T20:58:37.876288+0000","last_fullsized":"2023-10-16T20:58:37.876288+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:52:52.835051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082191850000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'91","reported_seq":159,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:33.484841+0000","last_change":"2023-10-16T20:36:54.763818+0000","last_active":"2023-10-16T20:58:33.484841+0000","last_peered":"2023-10-16T20:58:33.484841+0000","last_clean":"2023-10-16T20:58:33.484841+0000","last_became_active":"2023-10-16T20:36:54.761337+0000","last_became_peered":"2023-10-16T20:36:54.761337+0000","last_unstale":"2023-10-16T20:58:33.484841+0000","last_undegraded":"2023-10-16T20:58:33.484841+0000","last_fullsized":"2023-10-16T20:58:33.484841+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:03.093189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":5,"num_write":137,"num_write_kb":4352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"67'666","reported_seq":3840,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.018714+0000","last_change":"2023-10-16T20:47:31.487063+0000","last_active":"2023-10-16T20:58:23.018714+0000","last_peered":"2023-10-16T20:58:23.018714+0000","last_clean":"2023-10-16T20:58:23.018714+0000","last_became_active":"2023-10-16T20:37:37.468309+0000","last_became_peered":"2023-10-16T20:37:37.468309+0000","last_unstale":"2023-10-16T20:58:23.018714+0000","last_undegraded":"2023-10-16T20:58:23.018714+0000","last_fullsized":"2023-10-16T20:58:23.018714+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:13.271803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058030460000000001,"stat_sum":{"num_bytes":20758781,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":1058,"num_read_kb":21621,"num_write":667,"num_write_kb":32853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"67'107","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.887159+0000","last_change":"2023-10-16T20:47:31.589398+0000","last_active":"2023-10-16T20:58:37.887159+0000","last_peered":"2023-10-16T20:58:37.887159+0000","last_clean":"2023-10-16T20:58:37.887159+0000","last_became_active":"2023-10-16T20:36:55.288354+0000","last_became_peered":"2023-10-16T20:36:55.288354+0000","last_unstale":"2023-10-16T20:58:37.887159+0000","last_undegraded":"2023-10-16T20:58:37.887159+0000","last_fullsized":"2023-10-16T20:58:37.887159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:09.359239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076786709999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"67'333","reported_seq":413,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.977642+0000","last_change":"2023-10-16T20:36:54.738216+0000","last_active":"2023-10-16T20:58:05.977642+0000","last_peered":"2023-10-16T20:58:05.977642+0000","last_clean":"2023-10-16T20:58:05.977642+0000","last_became_active":"2023-10-16T20:36:54.737876+0000","last_became_peered":"2023-10-16T20:36:54.737876+0000","last_unstale":"2023-10-16T20:58:05.977642+0000","last_undegraded":"2023-10-16T20:58:05.977642+0000","last_fullsized":"2023-10-16T20:58:05.977642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:28.214601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":39,"num_read_kb":4134,"num_write":333,"num_write_kb":8414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"67'666","reported_seq":3665,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.692242+0000","last_change":"2023-10-16T20:47:31.490637+0000","last_active":"2023-10-16T20:58:22.692242+0000","last_peered":"2023-10-16T20:58:22.692242+0000","last_clean":"2023-10-16T20:58:22.692242+0000","last_became_active":"2023-10-16T20:36:59.306801+0000","last_became_peered":"2023-10-16T20:36:59.306801+0000","last_unstale":"2023-10-16T20:58:22.692242+0000","last_undegraded":"2023-10-16T20:58:22.692242+0000","last_fullsized":"2023-10-16T20:58:22.692242+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:43:08.907279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010585667,"stat_sum":{"num_bytes":20444029,"num_objects":301,"num_object_clones":0,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":893,"num_read_kb":19660,"num_write":666,"num_write_kb":28474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"67'93","reported_seq":2202,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.873494+0000","last_change":"2023-10-16T20:47:31.590902+0000","last_active":"2023-10-16T20:58:37.873494+0000","last_peered":"2023-10-16T20:58:37.873494+0000","last_clean":"2023-10-16T20:58:37.873494+0000","last_became_active":"2023-10-16T20:36:55.287066+0000","last_became_peered":"2023-10-16T20:36:55.287066+0000","last_unstale":"2023-10-16T20:58:37.873494+0000","last_undegraded":"2023-10-16T20:58:37.873494+0000","last_fullsized":"2023-10-16T20:58:37.873494+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:36.165939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012121162,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"66'101","reported_seq":172,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:57.653854+0000","last_change":"2023-10-16T20:36:54.764693+0000","last_active":"2023-10-16T20:57:57.653854+0000","last_peered":"2023-10-16T20:57:57.653854+0000","last_clean":"2023-10-16T20:57:57.653854+0000","last_became_active":"2023-10-16T20:36:54.764417+0000","last_became_peered":"2023-10-16T20:36:54.764417+0000","last_unstale":"2023-10-16T20:57:57.653854+0000","last_undegraded":"2023-10-16T20:57:57.653854+0000","last_fullsized":"2023-10-16T20:57:57.653854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:56.653149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":4110,"num_write":99,"num_write_kb":4200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'634","reported_seq":3754,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:20.202595+0000","last_change":"2023-10-16T20:47:31.584541+0000","last_active":"2023-10-16T20:58:20.202595+0000","last_peered":"2023-10-16T20:58:20.202595+0000","last_clean":"2023-10-16T20:58:20.202595+0000","last_became_active":"2023-10-16T20:36:59.464626+0000","last_became_peered":"2023-10-16T20:36:59.464626+0000","last_unstale":"2023-10-16T20:58:20.202595+0000","last_undegraded":"2023-10-16T20:58:20.202595+0000","last_fullsized":"2023-10-16T20:58:20.202595+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":634,"log_dups_size":0,"ondisk_log_size":634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:18:35.242031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069938420000000001,"stat_sum":{"num_bytes":28405361,"num_objects":321,"num_object_clones":0,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":1015,"num_read_kb":24481,"num_write":633,"num_write_kb":44456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"67'113","reported_seq":2222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.886956+0000","last_change":"2023-10-16T20:47:31.589722+0000","last_active":"2023-10-16T20:58:37.886956+0000","last_peered":"2023-10-16T20:58:37.886956+0000","last_clean":"2023-10-16T20:58:37.886956+0000","last_became_active":"2023-10-16T20:36:55.291396+0000","last_became_peered":"2023-10-16T20:36:55.291396+0000","last_unstale":"2023-10-16T20:58:37.886956+0000","last_undegraded":"2023-10-16T20:58:37.886956+0000","last_fullsized":"2023-10-16T20:58:37.886956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:09:26.019798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011464676,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"67'184","reported_seq":258,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.657888+0000","last_change":"2023-10-16T20:36:54.750814+0000","last_active":"2023-10-16T20:58:05.657888+0000","last_peered":"2023-10-16T20:58:05.657888+0000","last_clean":"2023-10-16T20:58:05.657888+0000","last_became_active":"2023-10-16T20:36:54.750448+0000","last_became_peered":"2023-10-16T20:36:54.750448+0000","last_unstale":"2023-10-16T20:58:05.657888+0000","last_undegraded":"2023-10-16T20:58:05.657888+0000","last_fullsized":"2023-10-16T20:58:05.657888+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:41.981408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":3911,"num_write":180,"num_write_kb":4223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"67'608","reported_seq":3330,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.013307+0000","last_change":"2023-10-16T20:47:31.583645+0000","last_active":"2023-10-16T20:58:23.013307+0000","last_peered":"2023-10-16T20:58:23.013307+0000","last_clean":"2023-10-16T20:58:23.013307+0000","last_became_active":"2023-10-16T20:36:59.464578+0000","last_became_peered":"2023-10-16T20:36:59.464578+0000","last_unstale":"2023-10-16T20:58:23.013307+0000","last_undegraded":"2023-10-16T20:58:23.013307+0000","last_fullsized":"2023-10-16T20:58:23.013307+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:56.818326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064826880000000003,"stat_sum":{"num_bytes":21354704,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":616,"num_read_kb":9691,"num_write":609,"num_write_kb":35926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"67'101","reported_seq":2210,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889428+0000","last_change":"2023-10-16T20:47:31.490514+0000","last_active":"2023-10-16T20:58:37.889428+0000","last_peered":"2023-10-16T20:58:37.889428+0000","last_clean":"2023-10-16T20:58:37.889428+0000","last_became_active":"2023-10-16T20:36:55.285407+0000","last_became_peered":"2023-10-16T20:36:55.285407+0000","last_unstale":"2023-10-16T20:58:37.889428+0000","last_undegraded":"2023-10-16T20:58:37.889428+0000","last_fullsized":"2023-10-16T20:58:37.889428+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:31:00.135226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010336254,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"66'459","reported_seq":604,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.695585+0000","last_change":"2023-10-16T20:36:54.757012+0000","last_active":"2023-10-16T20:58:28.695585+0000","last_peered":"2023-10-16T20:58:28.695585+0000","last_clean":"2023-10-16T20:58:28.695585+0000","last_became_active":"2023-10-16T20:36:54.756697+0000","last_became_peered":"2023-10-16T20:36:54.756697+0000","last_unstale":"2023-10-16T20:58:28.695585+0000","last_undegraded":"2023-10-16T20:58:28.695585+0000","last_fullsized":"2023-10-16T20:58:28.695585+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":459,"log_dups_size":0,"ondisk_log_size":459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:20.259580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8464867,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":106,"num_read_kb":16561,"num_write":461,"num_write_kb":8611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'632","reported_seq":4046,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.561302+0000","last_change":"2023-10-16T20:47:31.589709+0000","last_active":"2023-10-16T20:58:22.561302+0000","last_peered":"2023-10-16T20:58:22.561302+0000","last_clean":"2023-10-16T20:58:22.561302+0000","last_became_active":"2023-10-16T20:36:59.468254+0000","last_became_peered":"2023-10-16T20:36:59.468254+0000","last_unstale":"2023-10-16T20:58:22.561302+0000","last_undegraded":"2023-10-16T20:58:22.561302+0000","last_fullsized":"2023-10-16T20:58:22.561302+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:09:21.589899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093892239999999998,"stat_sum":{"num_bytes":28874401,"num_objects":300,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":1309,"num_read_kb":15538,"num_write":630,"num_write_kb":28525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"67'100","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.880164+0000","last_change":"2023-10-16T20:47:31.588140+0000","last_active":"2023-10-16T20:58:37.880164+0000","last_peered":"2023-10-16T20:58:37.880164+0000","last_clean":"2023-10-16T20:58:37.880164+0000","last_became_active":"2023-10-16T20:36:55.288488+0000","last_became_peered":"2023-10-16T20:36:55.288488+0000","last_unstale":"2023-10-16T20:58:37.880164+0000","last_undegraded":"2023-10-16T20:58:37.880164+0000","last_fullsized":"2023-10-16T20:58:37.880164+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:40:33.001741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078405860000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"67'139","reported_seq":223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.709051+0000","last_change":"2023-10-16T20:36:54.765361+0000","last_active":"2023-10-16T20:58:28.709051+0000","last_peered":"2023-10-16T20:58:28.709051+0000","last_clean":"2023-10-16T20:58:28.709051+0000","last_became_active":"2023-10-16T20:36:54.764969+0000","last_became_peered":"2023-10-16T20:36:54.764969+0000","last_unstale":"2023-10-16T20:58:28.709051+0000","last_undegraded":"2023-10-16T20:58:28.709051+0000","last_fullsized":"2023-10-16T20:58:28.709051+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:21:21.183612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389280,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":45,"num_read_kb":8235,"num_write":134,"num_write_kb":8274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"67'666","reported_seq":4058,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.657743+0000","last_change":"2023-10-16T20:47:31.488335+0000","last_active":"2023-10-16T20:58:22.657743+0000","last_peered":"2023-10-16T20:58:22.657743+0000","last_clean":"2023-10-16T20:58:22.657743+0000","last_became_active":"2023-10-16T20:36:59.302698+0000","last_became_peered":"2023-10-16T20:36:59.302698+0000","last_unstale":"2023-10-16T20:58:22.657743+0000","last_undegraded":"2023-10-16T20:58:22.657743+0000","last_fullsized":"2023-10-16T20:58:22.657743+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:09:15.954887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064797020000000004,"stat_sum":{"num_bytes":31392251,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":1286,"num_read_kb":27897,"num_write":665,"num_write_kb":43250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'99","reported_seq":2208,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.882106+0000","last_change":"2023-10-16T20:47:31.487882+0000","last_active":"2023-10-16T20:58:37.882106+0000","last_peered":"2023-10-16T20:58:37.882106+0000","last_clean":"2023-10-16T20:58:37.882106+0000","last_became_active":"2023-10-16T20:36:55.290127+0000","last_became_peered":"2023-10-16T20:36:55.290127+0000","last_unstale":"2023-10-16T20:58:37.882106+0000","last_undegraded":"2023-10-16T20:58:37.882106+0000","last_fullsized":"2023-10-16T20:58:37.882106+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:23.063288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055494209999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"67'78","reported_seq":150,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:24.130464+0000","last_change":"2023-10-16T20:36:54.749120+0000","last_active":"2023-10-16T20:58:24.130464+0000","last_peered":"2023-10-16T20:58:24.130464+0000","last_clean":"2023-10-16T20:58:24.130464+0000","last_became_active":"2023-10-16T20:36:54.748788+0000","last_became_peered":"2023-10-16T20:36:54.748788+0000","last_unstale":"2023-10-16T20:58:24.130464+0000","last_undegraded":"2023-10-16T20:58:24.130464+0000","last_fullsized":"2023-10-16T20:58:24.130464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:11:07.257343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":15,"num_read_kb":23,"num_write":83,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"67'606","reported_seq":3860,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.746395+0000","last_change":"2023-10-16T20:47:31.486033+0000","last_active":"2023-10-16T20:58:22.746395+0000","last_peered":"2023-10-16T20:58:22.746395+0000","last_clean":"2023-10-16T20:58:22.746395+0000","last_became_active":"2023-10-16T20:36:59.302916+0000","last_became_peered":"2023-10-16T20:36:59.302916+0000","last_unstale":"2023-10-16T20:58:22.746395+0000","last_undegraded":"2023-10-16T20:58:22.746395+0000","last_fullsized":"2023-10-16T20:58:22.746395+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:09.867705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041353309999999999,"stat_sum":{"num_bytes":29291517,"num_objects":300,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":1148,"num_read_kb":24111,"num_write":607,"num_write_kb":41174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"67'107","reported_seq":2215,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889740+0000","last_change":"2023-10-16T20:47:31.490807+0000","last_active":"2023-10-16T20:58:37.889740+0000","last_peered":"2023-10-16T20:58:37.889740+0000","last_clean":"2023-10-16T20:58:37.889740+0000","last_became_active":"2023-10-16T20:36:55.291448+0000","last_became_peered":"2023-10-16T20:36:55.291448+0000","last_unstale":"2023-10-16T20:58:37.889740+0000","last_undegraded":"2023-10-16T20:58:37.889740+0000","last_fullsized":"2023-10-16T20:58:37.889740+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:03:32.439694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010585074,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"67'134","reported_seq":254,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:39.515133+0000","last_change":"2023-10-16T20:36:54.761824+0000","last_active":"2023-10-16T20:58:39.515133+0000","last_peered":"2023-10-16T20:58:39.515133+0000","last_clean":"2023-10-16T20:58:39.515133+0000","last_became_active":"2023-10-16T20:36:54.761541+0000","last_became_peered":"2023-10-16T20:36:54.761541+0000","last_unstale":"2023-10-16T20:58:39.515133+0000","last_undegraded":"2023-10-16T20:58:39.515133+0000","last_fullsized":"2023-10-16T20:58:39.515133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:34.512207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2461990,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":69,"num_read_kb":129,"num_write":135,"num_write_kb":2563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"67'663","reported_seq":3325,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.270542+0000","last_change":"2023-10-16T20:47:31.585744+0000","last_active":"2023-10-16T20:58:22.270542+0000","last_peered":"2023-10-16T20:58:22.270542+0000","last_clean":"2023-10-16T20:58:22.270542+0000","last_became_active":"2023-10-16T20:36:59.303773+0000","last_became_peered":"2023-10-16T20:36:59.303773+0000","last_unstale":"2023-10-16T20:58:22.270542+0000","last_undegraded":"2023-10-16T20:58:22.270542+0000","last_fullsized":"2023-10-16T20:58:22.270542+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:19:00.801685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007941142,"stat_sum":{"num_bytes":21664408,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":557,"num_read_kb":9585,"num_write":658,"num_write_kb":32916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"67'121","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.891891+0000","last_change":"2023-10-16T20:47:31.594424+0000","last_active":"2023-10-16T20:58:37.891891+0000","last_peered":"2023-10-16T20:58:37.891891+0000","last_clean":"2023-10-16T20:58:37.891891+0000","last_became_active":"2023-10-16T20:36:55.300535+0000","last_became_peered":"2023-10-16T20:36:55.300535+0000","last_unstale":"2023-10-16T20:58:37.891891+0000","last_undegraded":"2023-10-16T20:58:37.891891+0000","last_fullsized":"2023-10-16T20:58:37.891891+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:47:03.132953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013755641000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"67'55","reported_seq":124,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:30.004648+0000","last_change":"2023-10-16T20:36:54.758140+0000","last_active":"2023-10-16T20:58:30.004648+0000","last_peered":"2023-10-16T20:58:30.004648+0000","last_clean":"2023-10-16T20:58:30.004648+0000","last_became_active":"2023-10-16T20:36:54.757303+0000","last_became_peered":"2023-10-16T20:36:54.757303+0000","last_unstale":"2023-10-16T20:58:30.004648+0000","last_undegraded":"2023-10-16T20:58:30.004648+0000","last_fullsized":"2023-10-16T20:58:30.004648+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:45.944191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":56,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"67'687","reported_seq":4217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.643179+0000","last_change":"2023-10-16T20:47:31.586327+0000","last_active":"2023-10-16T20:58:22.643179+0000","last_peered":"2023-10-16T20:58:22.643179+0000","last_clean":"2023-10-16T20:58:22.643179+0000","last_became_active":"2023-10-16T20:36:59.465844+0000","last_became_peered":"2023-10-16T20:36:59.465844+0000","last_unstale":"2023-10-16T20:58:22.643179+0000","last_undegraded":"2023-10-16T20:58:22.643179+0000","last_fullsized":"2023-10-16T20:58:22.643179+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:54.762444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081168009999999999,"stat_sum":{"num_bytes":45881583,"num_objects":353,"num_object_clones":0,"num_object_copies":1412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":353,"num_whiteouts":0,"num_read":1424,"num_read_kb":44764,"num_write":688,"num_write_kb":65580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"67'661","reported_seq":4047,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.525871+0000","last_change":"2023-10-16T20:47:31.489178+0000","last_active":"2023-10-16T20:58:22.525871+0000","last_peered":"2023-10-16T20:58:22.525871+0000","last_clean":"2023-10-16T20:58:22.525871+0000","last_became_active":"2023-10-16T20:36:59.467356+0000","last_became_peered":"2023-10-16T20:36:59.467356+0000","last_unstale":"2023-10-16T20:58:22.525871+0000","last_undegraded":"2023-10-16T20:58:22.525871+0000","last_fullsized":"2023-10-16T20:58:22.525871+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:10:50.487421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093722700000000003,"stat_sum":{"num_bytes":37038710,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1279,"num_read_kb":28276,"num_write":659,"num_write_kb":40616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'108","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.864220+0000","last_change":"2023-10-16T20:47:31.484094+0000","last_active":"2023-10-16T20:58:37.864220+0000","last_peered":"2023-10-16T20:58:37.864220+0000","last_clean":"2023-10-16T20:58:37.864220+0000","last_became_active":"2023-10-16T20:36:55.280903+0000","last_became_peered":"2023-10-16T20:36:55.280903+0000","last_unstale":"2023-10-16T20:58:37.864220+0000","last_undegraded":"2023-10-16T20:58:37.864220+0000","last_fullsized":"2023-10-16T20:58:37.864220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:50:46.153222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029269690000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"67'457","reported_seq":528,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.761682+0000","last_change":"2023-10-16T20:36:54.757589+0000","last_active":"2023-10-16T20:58:23.761682+0000","last_peered":"2023-10-16T20:58:23.761682+0000","last_clean":"2023-10-16T20:58:23.761682+0000","last_became_active":"2023-10-16T20:36:54.756775+0000","last_became_peered":"2023-10-16T20:36:54.756775+0000","last_unstale":"2023-10-16T20:58:23.761682+0000","last_undegraded":"2023-10-16T20:58:23.761682+0000","last_fullsized":"2023-10-16T20:58:23.761682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:40.629540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583494,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":16386,"num_write":448,"num_write_kb":12544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"67'638","reported_seq":3559,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.378852+0000","last_change":"2023-10-16T20:47:31.593596+0000","last_active":"2023-10-16T20:58:22.378852+0000","last_peered":"2023-10-16T20:58:22.378852+0000","last_clean":"2023-10-16T20:58:22.378852+0000","last_became_active":"2023-10-16T20:37:37.438140+0000","last_became_peered":"2023-10-16T20:37:37.438140+0000","last_unstale":"2023-10-16T20:58:22.378852+0000","last_undegraded":"2023-10-16T20:58:22.378852+0000","last_fullsized":"2023-10-16T20:58:22.378852+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:41:29.134511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012287041,"stat_sum":{"num_bytes":24792014,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":805,"num_read_kb":12206,"num_write":635,"num_write_kb":32676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"67'121","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.897843+0000","last_change":"2023-10-16T20:47:31.585618+0000","last_active":"2023-10-16T20:58:37.897843+0000","last_peered":"2023-10-16T20:58:37.897843+0000","last_clean":"2023-10-16T20:58:37.897843+0000","last_became_active":"2023-10-16T20:36:55.273959+0000","last_became_peered":"2023-10-16T20:36:55.273959+0000","last_unstale":"2023-10-16T20:58:37.897843+0000","last_undegraded":"2023-10-16T20:58:37.897843+0000","last_fullsized":"2023-10-16T20:58:37.897843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:05:00.921391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076333039999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"67'135","reported_seq":276,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.678551+0000","last_change":"2023-10-16T20:36:54.759513+0000","last_active":"2023-10-16T20:58:28.678551+0000","last_peered":"2023-10-16T20:58:28.678551+0000","last_clean":"2023-10-16T20:58:28.678551+0000","last_became_active":"2023-10-16T20:36:54.759059+0000","last_became_peered":"2023-10-16T20:36:54.759059+0000","last_unstale":"2023-10-16T20:58:28.678551+0000","last_undegraded":"2023-10-16T20:58:28.678551+0000","last_fullsized":"2023-10-16T20:58:28.678551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:36.696053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4348959,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":88,"num_read_kb":239,"num_write":133,"num_write_kb":4341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"66'223","reported_seq":372,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:33.632797+0000","last_change":"2023-10-16T20:36:54.764780+0000","last_active":"2023-10-16T20:57:33.632797+0000","last_peered":"2023-10-16T20:57:33.632797+0000","last_clean":"2023-10-16T20:57:33.632797+0000","last_became_active":"2023-10-16T20:36:54.764436+0000","last_became_peered":"2023-10-16T20:36:54.764436+0000","last_unstale":"2023-10-16T20:57:33.632797+0000","last_undegraded":"2023-10-16T20:57:33.632797+0000","last_fullsized":"2023-10-16T20:57:33.632797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:55:31.816599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8524313,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":105,"num_read_kb":8412,"num_write":220,"num_write_kb":8480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"67'109","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888733+0000","last_change":"2023-10-16T20:47:31.492923+0000","last_active":"2023-10-16T20:58:37.888733+0000","last_peered":"2023-10-16T20:58:37.888733+0000","last_clean":"2023-10-16T20:58:37.888733+0000","last_became_active":"2023-10-16T20:36:55.277873+0000","last_became_peered":"2023-10-16T20:36:55.277873+0000","last_unstale":"2023-10-16T20:58:37.888733+0000","last_undegraded":"2023-10-16T20:58:37.888733+0000","last_fullsized":"2023-10-16T20:58:37.888733+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:24.211693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012247773,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'608","reported_seq":3232,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.002422+0000","last_change":"2023-10-16T20:47:31.496724+0000","last_active":"2023-10-16T20:58:23.002422+0000","last_peered":"2023-10-16T20:58:23.002422+0000","last_clean":"2023-10-16T20:58:23.002422+0000","last_became_active":"2023-10-16T20:36:59.302435+0000","last_became_peered":"2023-10-16T20:36:59.302435+0000","last_unstale":"2023-10-16T20:58:23.002422+0000","last_undegraded":"2023-10-16T20:58:23.002422+0000","last_fullsized":"2023-10-16T20:58:23.002422+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:34:39.329693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001061892,"stat_sum":{"num_bytes":16477397,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":519,"num_read_kb":10696,"num_write":606,"num_write_kb":24605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"66'232","reported_seq":301,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:33.636417+0000","last_change":"2023-10-16T20:36:54.756190+0000","last_active":"2023-10-16T20:57:33.636417+0000","last_peered":"2023-10-16T20:57:33.636417+0000","last_clean":"2023-10-16T20:57:33.636417+0000","last_became_active":"2023-10-16T20:36:54.755834+0000","last_became_peered":"2023-10-16T20:36:54.755834+0000","last_unstale":"2023-10-16T20:57:33.636417+0000","last_undegraded":"2023-10-16T20:57:33.636417+0000","last_fullsized":"2023-10-16T20:57:33.636417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:31.012441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582946,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":16385,"num_write":235,"num_write_kb":12609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"67'104","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.885029+0000","last_change":"2023-10-16T20:47:31.486761+0000","last_active":"2023-10-16T20:58:37.885029+0000","last_peered":"2023-10-16T20:58:37.885029+0000","last_clean":"2023-10-16T20:58:37.885029+0000","last_became_active":"2023-10-16T20:36:55.279333+0000","last_became_peered":"2023-10-16T20:36:55.279333+0000","last_unstale":"2023-10-16T20:58:37.885029+0000","last_undegraded":"2023-10-16T20:58:37.885029+0000","last_fullsized":"2023-10-16T20:58:37.885029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:30:26.879792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047896609999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'629","reported_seq":3442,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.721224+0000","last_change":"2023-10-16T20:47:31.586162+0000","last_active":"2023-10-16T20:58:22.721224+0000","last_peered":"2023-10-16T20:58:22.721224+0000","last_clean":"2023-10-16T20:58:22.721224+0000","last_became_active":"2023-10-16T20:36:59.301864+0000","last_became_peered":"2023-10-16T20:36:59.301864+0000","last_unstale":"2023-10-16T20:58:22.721224+0000","last_undegraded":"2023-10-16T20:58:22.721224+0000","last_fullsized":"2023-10-16T20:58:22.721224+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:44.207453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080450769999999994,"stat_sum":{"num_bytes":20568472,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":708,"num_read_kb":22370,"num_write":628,"num_write_kb":40892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"67'120","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.875503+0000","last_change":"2023-10-16T20:47:31.590065+0000","last_active":"2023-10-16T20:58:37.875503+0000","last_peered":"2023-10-16T20:58:37.875503+0000","last_clean":"2023-10-16T20:58:37.875503+0000","last_became_active":"2023-10-16T20:36:55.287442+0000","last_became_peered":"2023-10-16T20:36:55.287442+0000","last_unstale":"2023-10-16T20:58:37.875503+0000","last_undegraded":"2023-10-16T20:58:37.875503+0000","last_fullsized":"2023-10-16T20:58:37.875503+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:37:33.639292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011630351000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"67'131","reported_seq":2241,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.880621+0000","last_change":"2023-10-16T20:47:31.497003+0000","last_active":"2023-10-16T20:58:37.880621+0000","last_peered":"2023-10-16T20:58:37.880621+0000","last_clean":"2023-10-16T20:58:37.880621+0000","last_became_active":"2023-10-16T20:36:55.276276+0000","last_became_peered":"2023-10-16T20:36:55.276276+0000","last_unstale":"2023-10-16T20:58:37.880621+0000","last_undegraded":"2023-10-16T20:58:37.880621+0000","last_fullsized":"2023-10-16T20:58:37.880621+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:35:00.858846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091802500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"67'115","reported_seq":2232,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.886101+0000","last_change":"2023-10-16T20:47:31.592299+0000","last_active":"2023-10-16T20:58:37.886101+0000","last_peered":"2023-10-16T20:58:37.886101+0000","last_clean":"2023-10-16T20:58:37.886101+0000","last_became_active":"2023-10-16T20:37:37.413040+0000","last_became_peered":"2023-10-16T20:37:37.413040+0000","last_unstale":"2023-10-16T20:58:37.886101+0000","last_undegraded":"2023-10-16T20:58:37.886101+0000","last_fullsized":"2023-10-16T20:58:37.886101+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:50:46.069439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012618513,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"67'116","reported_seq":2226,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888796+0000","last_change":"2023-10-16T20:47:31.492800+0000","last_active":"2023-10-16T20:58:37.888796+0000","last_peered":"2023-10-16T20:58:37.888796+0000","last_clean":"2023-10-16T20:58:37.888796+0000","last_became_active":"2023-10-16T20:36:55.275349+0000","last_became_peered":"2023-10-16T20:36:55.275349+0000","last_unstale":"2023-10-16T20:58:37.888796+0000","last_undegraded":"2023-10-16T20:58:37.888796+0000","last_fullsized":"2023-10-16T20:58:37.888796+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:28:00.161390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012136073000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"67'88","reported_seq":2198,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.852714+0000","last_change":"2023-10-16T20:47:31.587774+0000","last_active":"2023-10-16T20:58:37.852714+0000","last_peered":"2023-10-16T20:58:37.852714+0000","last_clean":"2023-10-16T20:58:37.852714+0000","last_became_active":"2023-10-16T20:36:55.277043+0000","last_became_peered":"2023-10-16T20:36:55.277043+0000","last_unstale":"2023-10-16T20:58:37.852714+0000","last_undegraded":"2023-10-16T20:58:37.852714+0000","last_fullsized":"2023-10-16T20:58:37.852714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:32.439680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078065369999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"67'89","reported_seq":2199,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.863001+0000","last_change":"2023-10-16T20:47:31.489466+0000","last_active":"2023-10-16T20:58:37.863001+0000","last_peered":"2023-10-16T20:58:37.863001+0000","last_clean":"2023-10-16T20:58:37.863001+0000","last_became_active":"2023-10-16T20:36:55.278715+0000","last_became_peered":"2023-10-16T20:36:55.278715+0000","last_unstale":"2023-10-16T20:58:37.863001+0000","last_undegraded":"2023-10-16T20:58:37.863001+0000","last_fullsized":"2023-10-16T20:58:37.863001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:49:22.643504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067460719999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"67'124","reported_seq":2234,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.836534+0000","last_change":"2023-10-16T20:47:31.594095+0000","last_active":"2023-10-16T20:58:37.836534+0000","last_peered":"2023-10-16T20:58:37.836534+0000","last_clean":"2023-10-16T20:58:37.836534+0000","last_became_active":"2023-10-16T20:36:55.301604+0000","last_became_peered":"2023-10-16T20:36:55.301604+0000","last_unstale":"2023-10-16T20:58:37.836534+0000","last_undegraded":"2023-10-16T20:58:37.836534+0000","last_fullsized":"2023-10-16T20:58:37.836534+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:10.110081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012969489000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"67'99","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.858692+0000","last_change":"2023-10-16T20:47:31.587306+0000","last_active":"2023-10-16T20:58:37.858692+0000","last_peered":"2023-10-16T20:58:37.858692+0000","last_clean":"2023-10-16T20:58:37.858692+0000","last_became_active":"2023-10-16T20:36:55.276984+0000","last_became_peered":"2023-10-16T20:36:55.276984+0000","last_unstale":"2023-10-16T20:58:37.858692+0000","last_undegraded":"2023-10-16T20:58:37.858692+0000","last_fullsized":"2023-10-16T20:58:37.858692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:42:52.651942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086623480000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'111","reported_seq":2221,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.896916+0000","last_change":"2023-10-16T20:47:31.595489+0000","last_active":"2023-10-16T20:58:37.896916+0000","last_peered":"2023-10-16T20:58:37.896916+0000","last_clean":"2023-10-16T20:58:37.896916+0000","last_became_active":"2023-10-16T20:36:55.290023+0000","last_became_peered":"2023-10-16T20:36:55.290023+0000","last_unstale":"2023-10-16T20:58:37.896916+0000","last_undegraded":"2023-10-16T20:58:37.896916+0000","last_fullsized":"2023-10-16T20:58:37.896916+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:33.259824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013081514000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'124","reported_seq":2233,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.203227+0000","last_change":"2023-10-16T20:47:31.586678+0000","last_active":"2023-10-16T20:58:38.203227+0000","last_peered":"2023-10-16T20:58:38.203227+0000","last_clean":"2023-10-16T20:58:38.203227+0000","last_became_active":"2023-10-16T20:36:55.280186+0000","last_became_peered":"2023-10-16T20:36:55.280186+0000","last_unstale":"2023-10-16T20:58:38.203227+0000","last_undegraded":"2023-10-16T20:58:38.203227+0000","last_fullsized":"2023-10-16T20:58:38.203227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:22:01.009457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083925690000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"67'118","reported_seq":2227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.882744+0000","last_change":"2023-10-16T20:47:31.589254+0000","last_active":"2023-10-16T20:58:37.882744+0000","last_peered":"2023-10-16T20:58:37.882744+0000","last_clean":"2023-10-16T20:58:37.882744+0000","last_became_active":"2023-10-16T20:36:55.297253+0000","last_became_peered":"2023-10-16T20:36:55.297253+0000","last_unstale":"2023-10-16T20:58:37.882744+0000","last_undegraded":"2023-10-16T20:58:37.882744+0000","last_fullsized":"2023-10-16T20:58:37.882744+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:11.854849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097685800000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"67'108","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.891317+0000","last_change":"2023-10-16T20:47:31.492104+0000","last_active":"2023-10-16T20:58:37.891317+0000","last_peered":"2023-10-16T20:58:37.891317+0000","last_clean":"2023-10-16T20:58:37.891317+0000","last_became_active":"2023-10-16T20:36:55.286044+0000","last_became_peered":"2023-10-16T20:36:55.286044+0000","last_unstale":"2023-10-16T20:58:37.891317+0000","last_undegraded":"2023-10-16T20:58:37.891317+0000","last_fullsized":"2023-10-16T20:58:37.891317+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:45.304718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011681474000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"67'110","reported_seq":2219,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.865687+0000","last_change":"2023-10-16T20:47:31.592401+0000","last_active":"2023-10-16T20:58:37.865687+0000","last_peered":"2023-10-16T20:58:37.865687+0000","last_clean":"2023-10-16T20:58:37.865687+0000","last_became_active":"2023-10-16T20:36:55.286864+0000","last_became_peered":"2023-10-16T20:36:55.286864+0000","last_unstale":"2023-10-16T20:58:37.865687+0000","last_undegraded":"2023-10-16T20:58:37.865687+0000","last_fullsized":"2023-10-16T20:58:37.865687+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:31.381174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012885232999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"67'106","reported_seq":2215,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.872600+0000","last_change":"2023-10-16T20:47:31.588333+0000","last_active":"2023-10-16T20:58:37.872600+0000","last_peered":"2023-10-16T20:58:37.872600+0000","last_clean":"2023-10-16T20:58:37.872600+0000","last_became_active":"2023-10-16T20:36:55.292262+0000","last_became_peered":"2023-10-16T20:36:55.292262+0000","last_unstale":"2023-10-16T20:58:37.872600+0000","last_undegraded":"2023-10-16T20:58:37.872600+0000","last_fullsized":"2023-10-16T20:58:37.872600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:15.116061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077171749999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"67'107","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.832173+0000","last_change":"2023-10-16T20:47:31.496924+0000","last_active":"2023-10-16T20:58:37.832173+0000","last_peered":"2023-10-16T20:58:37.832173+0000","last_clean":"2023-10-16T20:58:37.832173+0000","last_became_active":"2023-10-16T20:36:55.282806+0000","last_became_peered":"2023-10-16T20:36:55.282806+0000","last_unstale":"2023-10-16T20:58:37.832173+0000","last_undegraded":"2023-10-16T20:58:37.832173+0000","last_fullsized":"2023-10-16T20:58:37.832173+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:51.395313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086132999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"67'109","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.887891+0000","last_change":"2023-10-16T20:47:31.594959+0000","last_active":"2023-10-16T20:58:37.887891+0000","last_peered":"2023-10-16T20:58:37.887891+0000","last_clean":"2023-10-16T20:58:37.887891+0000","last_became_active":"2023-10-16T20:36:55.303095+0000","last_became_peered":"2023-10-16T20:36:55.303095+0000","last_unstale":"2023-10-16T20:58:37.887891+0000","last_undegraded":"2023-10-16T20:58:37.887891+0000","last_fullsized":"2023-10-16T20:58:37.887891+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:00:51.726359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013746639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"67'119","reported_seq":2228,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.181755+0000","last_change":"2023-10-16T20:47:31.595380+0000","last_active":"2023-10-16T20:58:38.181755+0000","last_peered":"2023-10-16T20:58:38.181755+0000","last_clean":"2023-10-16T20:58:38.181755+0000","last_became_active":"2023-10-16T20:36:55.291447+0000","last_became_peered":"2023-10-16T20:36:55.291447+0000","last_unstale":"2023-10-16T20:58:38.181755+0000","last_undegraded":"2023-10-16T20:58:38.181755+0000","last_fullsized":"2023-10-16T20:58:38.181755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:20.797976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01315413,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"67'107","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.868160+0000","last_change":"2023-10-16T20:47:31.484708+0000","last_active":"2023-10-16T20:58:37.868160+0000","last_peered":"2023-10-16T20:58:37.868160+0000","last_clean":"2023-10-16T20:58:37.868160+0000","last_became_active":"2023-10-16T20:36:55.280605+0000","last_became_peered":"2023-10-16T20:36:55.280605+0000","last_unstale":"2023-10-16T20:58:37.868160+0000","last_undegraded":"2023-10-16T20:58:37.868160+0000","last_fullsized":"2023-10-16T20:58:37.868160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:39.852344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033133080000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"67'112","reported_seq":2221,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.891112+0000","last_change":"2023-10-16T20:47:31.492149+0000","last_active":"2023-10-16T20:58:37.891112+0000","last_peered":"2023-10-16T20:58:37.891112+0000","last_clean":"2023-10-16T20:58:37.891112+0000","last_became_active":"2023-10-16T20:36:55.277094+0000","last_became_peered":"2023-10-16T20:36:55.277094+0000","last_unstale":"2023-10-16T20:58:37.891112+0000","last_undegraded":"2023-10-16T20:58:37.891112+0000","last_fullsized":"2023-10-16T20:58:37.891112+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:01.971894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011816411000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"67'94","reported_seq":2203,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.879368+0000","last_change":"2023-10-16T20:47:31.587367+0000","last_active":"2023-10-16T20:58:37.879368+0000","last_peered":"2023-10-16T20:58:37.879368+0000","last_clean":"2023-10-16T20:58:37.879368+0000","last_became_active":"2023-10-16T20:36:55.297199+0000","last_became_peered":"2023-10-16T20:36:55.297199+0000","last_unstale":"2023-10-16T20:58:37.879368+0000","last_undegraded":"2023-10-16T20:58:37.879368+0000","last_fullsized":"2023-10-16T20:58:37.879368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:07:32.275973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086430910000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"67'90","reported_seq":2199,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.867006+0000","last_change":"2023-10-16T20:47:31.487461+0000","last_active":"2023-10-16T20:58:37.867006+0000","last_peered":"2023-10-16T20:58:37.867006+0000","last_clean":"2023-10-16T20:58:37.867006+0000","last_became_active":"2023-10-16T20:36:55.286964+0000","last_became_peered":"2023-10-16T20:36:55.286964+0000","last_unstale":"2023-10-16T20:58:37.867006+0000","last_undegraded":"2023-10-16T20:58:37.867006+0000","last_fullsized":"2023-10-16T20:58:37.867006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:57.164132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051857609999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"67'110","reported_seq":2226,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.834126+0000","last_change":"2023-10-16T20:47:31.594637+0000","last_active":"2023-10-16T20:58:37.834126+0000","last_peered":"2023-10-16T20:58:37.834126+0000","last_clean":"2023-10-16T20:58:37.834126+0000","last_became_active":"2023-10-16T20:37:37.413492+0000","last_became_peered":"2023-10-16T20:37:37.413492+0000","last_unstale":"2023-10-16T20:58:37.834126+0000","last_undegraded":"2023-10-16T20:58:37.834126+0000","last_fullsized":"2023-10-16T20:58:37.834126+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:07.365316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014529932000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"67'106","reported_seq":2215,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.850164+0000","last_change":"2023-10-16T20:47:31.593072+0000","last_active":"2023-10-16T20:58:37.850164+0000","last_peered":"2023-10-16T20:58:37.850164+0000","last_clean":"2023-10-16T20:58:37.850164+0000","last_became_active":"2023-10-16T20:36:55.302468+0000","last_became_peered":"2023-10-16T20:36:55.302468+0000","last_unstale":"2023-10-16T20:58:37.850164+0000","last_undegraded":"2023-10-16T20:58:37.850164+0000","last_fullsized":"2023-10-16T20:58:37.850164+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:54:24.682600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012359399,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"67'92","reported_seq":2200,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.894436+0000","last_change":"2023-10-16T20:47:31.585931+0000","last_active":"2023-10-16T20:58:37.894436+0000","last_peered":"2023-10-16T20:58:37.894436+0000","last_clean":"2023-10-16T20:58:37.894436+0000","last_became_active":"2023-10-16T20:36:55.296476+0000","last_became_peered":"2023-10-16T20:36:55.296476+0000","last_unstale":"2023-10-16T20:58:37.894436+0000","last_undegraded":"2023-10-16T20:58:37.894436+0000","last_fullsized":"2023-10-16T20:58:37.894436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:50:21.532799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072035780000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'95","reported_seq":2203,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.898420+0000","last_change":"2023-10-16T20:47:31.585218+0000","last_active":"2023-10-16T20:58:37.898420+0000","last_peered":"2023-10-16T20:58:37.898420+0000","last_clean":"2023-10-16T20:58:37.898420+0000","last_became_active":"2023-10-16T20:36:55.293687+0000","last_became_peered":"2023-10-16T20:36:55.293687+0000","last_unstale":"2023-10-16T20:58:37.898420+0000","last_undegraded":"2023-10-16T20:58:37.898420+0000","last_fullsized":"2023-10-16T20:58:37.898420+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:06:36.236196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074548399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"67'127","reported_seq":2235,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.887764+0000","last_change":"2023-10-16T20:47:31.588394+0000","last_active":"2023-10-16T20:58:37.887764+0000","last_peered":"2023-10-16T20:58:37.887764+0000","last_clean":"2023-10-16T20:58:37.887764+0000","last_became_active":"2023-10-16T20:36:55.280444+0000","last_became_peered":"2023-10-16T20:36:55.280444+0000","last_unstale":"2023-10-16T20:58:37.887764+0000","last_undegraded":"2023-10-16T20:58:37.887764+0000","last_fullsized":"2023-10-16T20:58:37.887764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:40:02.818328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068623299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'109","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888019+0000","last_change":"2023-10-16T20:47:31.588884+0000","last_active":"2023-10-16T20:58:37.888019+0000","last_peered":"2023-10-16T20:58:37.888019+0000","last_clean":"2023-10-16T20:58:37.888019+0000","last_became_active":"2023-10-16T20:36:55.296694+0000","last_became_peered":"2023-10-16T20:36:55.296694+0000","last_unstale":"2023-10-16T20:58:37.888019+0000","last_undegraded":"2023-10-16T20:58:37.888019+0000","last_fullsized":"2023-10-16T20:58:37.888019+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:39.793533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096291870000000009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"67'120","reported_seq":2228,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.879224+0000","last_change":"2023-10-16T20:47:31.585319+0000","last_active":"2023-10-16T20:58:37.879224+0000","last_peered":"2023-10-16T20:58:37.879224+0000","last_clean":"2023-10-16T20:58:37.879224+0000","last_became_active":"2023-10-16T20:36:55.291052+0000","last_became_peered":"2023-10-16T20:36:55.291052+0000","last_unstale":"2023-10-16T20:58:37.879224+0000","last_undegraded":"2023-10-16T20:58:37.879224+0000","last_fullsized":"2023-10-16T20:58:37.879224+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:45.283686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070559409999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"67'94","reported_seq":2204,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.895896+0000","last_change":"2023-10-16T20:47:31.588931+0000","last_active":"2023-10-16T20:58:37.895896+0000","last_peered":"2023-10-16T20:58:37.895896+0000","last_clean":"2023-10-16T20:58:37.895896+0000","last_became_active":"2023-10-16T20:36:55.290386+0000","last_became_peered":"2023-10-16T20:36:55.290386+0000","last_unstale":"2023-10-16T20:58:37.895896+0000","last_undegraded":"2023-10-16T20:58:37.895896+0000","last_fullsized":"2023-10-16T20:58:37.895896+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:56:59.640546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093217969999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"67'114","reported_seq":2224,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888639+0000","last_change":"2023-10-16T20:47:31.490951+0000","last_active":"2023-10-16T20:58:37.888639+0000","last_peered":"2023-10-16T20:58:37.888639+0000","last_clean":"2023-10-16T20:58:37.888639+0000","last_became_active":"2023-10-16T20:36:55.270787+0000","last_became_peered":"2023-10-16T20:36:55.270787+0000","last_unstale":"2023-10-16T20:58:37.888639+0000","last_undegraded":"2023-10-16T20:58:37.888639+0000","last_fullsized":"2023-10-16T20:58:37.888639+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:57:57.930582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010801276,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"67'94","reported_seq":2204,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.890538+0000","last_change":"2023-10-16T20:47:31.487902+0000","last_active":"2023-10-16T20:58:37.890538+0000","last_peered":"2023-10-16T20:58:37.890538+0000","last_clean":"2023-10-16T20:58:37.890538+0000","last_became_active":"2023-10-16T20:36:55.271373+0000","last_became_peered":"2023-10-16T20:36:55.271373+0000","last_unstale":"2023-10-16T20:58:37.890538+0000","last_undegraded":"2023-10-16T20:58:37.890538+0000","last_fullsized":"2023-10-16T20:58:37.890538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:37:10.982777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006091478,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"67'111","reported_seq":2221,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.890234+0000","last_change":"2023-10-16T20:47:31.587880+0000","last_active":"2023-10-16T20:58:37.890234+0000","last_peered":"2023-10-16T20:58:37.890234+0000","last_clean":"2023-10-16T20:58:37.890234+0000","last_became_active":"2023-10-16T20:36:55.291016+0000","last_became_peered":"2023-10-16T20:36:55.291016+0000","last_unstale":"2023-10-16T20:58:37.890234+0000","last_undegraded":"2023-10-16T20:58:37.890234+0000","last_fullsized":"2023-10-16T20:58:37.890234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:23:37.851121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009370148,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":924319467,"num_objects":10294,"num_object_clones":0,"num_object_copies":41176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10294,"num_whiteouts":0,"num_read":30602,"num_read_kb":676969,"num_write":21002,"num_write_kb":1302825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966899200,"data_stored":1966899200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21050,"ondisk_log_size":21050,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6794,"ondisk_log_size":6794,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":178960041,"num_objects":198,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1063,"num_read_kb":205540,"num_write":6349,"num_write_kb":180514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":115,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":358170624,"data_stored":357920082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6078,"ondisk_log_size":6078,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509707,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396772,"kb_used_data":280924,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93975068,"statfs":{"total":96636764160,"available":96230469632,"internally_reserved":0,"allocated":287666176,"data_stored":287268719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":118618571},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.52300000000000002,"15min":0.629},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67900000000000005,"5min":0.91400000000000003,"15min":2.4489999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54600000000000004,"15min":0.64700000000000002},"min":{"1min":0.378,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80200000000000005,"5min":0.95399999999999996,"15min":11.77},"last":0.73899999999999999}]},{"osd":1,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57699999999999996,"15min":0.65300000000000002},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73699999999999999,"5min":0.86199999999999999,"15min":4.5279999999999996},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58499999999999996,"15min":0.68400000000000005},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.84599999999999997,"5min":1.024,"15min":11.971},"last":0.77400000000000002}]},{"osd":2,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61599999999999999,"15min":0.67200000000000004},"min":{"1min":0.41999999999999998,"5min":0.40200000000000002,"15min":0.34300000000000003},"max":{"1min":0.81899999999999995,"5min":0.92200000000000004,"15min":7.1130000000000004},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.629,"15min":0.70999999999999996},"min":{"1min":0.51100000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.877,"5min":0.98299999999999998,"15min":9.4450000000000003},"last":0.69699999999999995}]},{"osd":3,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.64900000000000002,"15min":0.68300000000000005},"min":{"1min":0.439,"5min":0.437,"15min":0.41299999999999998},"max":{"1min":0.78000000000000003,"5min":1,"15min":3.363},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.68300000000000005,"15min":0.72699999999999998},"min":{"1min":0.50900000000000001,"5min":0.46899999999999997,"15min":0.42099999999999999},"max":{"1min":0.86299999999999999,"5min":1.0529999999999999,"15min":3.3980000000000001},"last":0.76300000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56699999999999995,"15min":0.61399999999999999},"min":{"1min":0.36399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81899999999999995,"5min":1.091,"15min":12.532999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.623,"15min":0.628},"min":{"1min":0.47099999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90600000000000003,"5min":1.107,"15min":3.8700000000000001},"last":0.52100000000000002}]},{"osd":5,"last update":"Mon Oct 16 20:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60999999999999999,"15min":0.66300000000000003},"min":{"1min":0.41399999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.85699999999999998,"5min":1.042,"15min":2.2010000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.63600000000000001,"15min":0.73699999999999999},"min":{"1min":0.47499999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94399999999999995,"5min":1.0369999999999999,"15min":12.353},"last":0.48899999999999999}]},{"osd":6,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65100000000000002,"15min":0.746},"min":{"1min":0.40799999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93799999999999994,"5min":1.1299999999999999,"15min":11.929},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66900000000000004,"15min":0.71399999999999997},"min":{"1min":0.41799999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92900000000000005,"5min":1.0620000000000001,"15min":6.0359999999999996},"last":0.58599999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509707,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421948,"kb_used_data":300532,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93949892,"statfs":{"total":96636764160,"available":96204689408,"internally_reserved":0,"allocated":307744768,"data_stored":307357593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124320199},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.54900000000000004,"15min":0.61499999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27800000000000002},"max":{"1min":0.69499999999999995,"5min":0.93799999999999994,"15min":4.5419999999999998},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.56499999999999995,"15min":0.624},"min":{"1min":0.33200000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.76000000000000001,"5min":2.0329999999999999,"15min":4.9699999999999998},"last":0.79900000000000004}]},{"osd":1,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.624,"15min":0.64100000000000001},"min":{"1min":0.433,"5min":0.39200000000000002,"15min":0.377},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":2.8610000000000002},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67000000000000004,"15min":0.69599999999999995},"min":{"1min":0.41499999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.94399999999999995,"5min":1.014,"15min":1.0309999999999999},"last":0.84499999999999997}]},{"osd":2,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.59699999999999998,"15min":0.65000000000000002},"min":{"1min":0.36399999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86599999999999999,"5min":1.0289999999999999,"15min":1.895},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.65500000000000003,"15min":0.68500000000000005},"min":{"1min":0.40799999999999997,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78500000000000003,"5min":0.96099999999999997,"15min":2.3340000000000001},"last":0.86499999999999999}]},{"osd":3,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.755,"15min":0.72099999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.36599999999999999},"max":{"1min":0.86699999999999999,"5min":4.6340000000000003,"15min":4.6340000000000003},"last":0.879},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.64100000000000001,"15min":0.69499999999999995},"min":{"1min":0.435,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.86099999999999999,"5min":1.163,"15min":5.1029999999999998},"last":0.67500000000000004}]},{"osd":4,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55500000000000005,"15min":0.58999999999999997},"min":{"1min":0.36299999999999999,"5min":0.34799999999999998,"15min":0.30299999999999999},"max":{"1min":0.80300000000000005,"5min":1.0189999999999999,"15min":4.8959999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.63500000000000001,"15min":0.63400000000000001},"min":{"1min":0.374,"5min":0.313,"15min":0.313},"max":{"1min":0.996,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.58999999999999997}]},{"osd":5,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.59799999999999998,"15min":0.623},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.82199999999999995,"5min":1.05,"15min":2.3769999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.626,"15min":0.66200000000000003},"min":{"1min":0.38200000000000001,"5min":0.35999999999999999,"15min":0.34499999999999997},"max":{"1min":0.84499999999999997,"5min":1.0700000000000001,"15min":4.9379999999999997},"last":0.82699999999999996}]},{"osd":7,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.71199999999999997,"15min":0.70999999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.034,"5min":1.425,"15min":4.5789999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.76800000000000002,"15min":0.748},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.39300000000000002},"max":{"1min":1.0600000000000001,"5min":1.6120000000000001,"15min":5.2460000000000004},"last":0.73999999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509709,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324840,"kb_used_data":225632,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94047000,"statfs":{"total":96636764160,"available":96304128000,"internally_reserved":0,"allocated":231047168,"data_stored":230645947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":101579207},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.57199999999999995,"15min":0.61299999999999999},"min":{"1min":0.30499999999999999,"5min":0.28199999999999997,"15min":0.25700000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":2.4580000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.59299999999999997,"15min":0.65800000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.94999999999999996,"5min":1.1799999999999999,"15min":7.181},"last":0.64600000000000002}]},{"osd":1,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":0.76800000000000002,"15min":0.69099999999999995},"min":{"1min":0.36699999999999999,"5min":0.33900000000000002,"15min":0.29899999999999999},"max":{"1min":16.655000000000001,"5min":16.655000000000001,"15min":16.655000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.5529999999999999,"5min":0.80100000000000005,"15min":0.71299999999999997},"min":{"1min":0.35099999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":16.289000000000001,"5min":16.289000000000001,"15min":16.289000000000001},"last":0.55700000000000005}]},{"osd":2,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.63200000000000001,"15min":0.67300000000000004},"min":{"1min":0.374,"5min":0.34599999999999997,"15min":0.308},"max":{"1min":1.1339999999999999,"5min":1.7110000000000001,"15min":3.3799999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67100000000000004,"15min":0.67700000000000005},"min":{"1min":0.42399999999999999,"5min":0.39800000000000002,"15min":0.38},"max":{"1min":1.1100000000000001,"5min":1.583,"15min":4.5369999999999999},"last":0.59399999999999997}]},{"osd":3,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.67400000000000004,"15min":0.69699999999999995},"min":{"1min":0.42399999999999999,"5min":0.39400000000000002,"15min":0.34899999999999998},"max":{"1min":1.0860000000000001,"5min":1.329,"15min":4.1070000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68600000000000005,"15min":0.69399999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.38},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":3.476},"last":0.69599999999999995}]},{"osd":4,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53100000000000003,"15min":0.55500000000000005},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.749,"5min":0.85099999999999998,"15min":2.415},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.59999999999999998,"15min":0.66300000000000003},"min":{"1min":0.34300000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.78900000000000003,"5min":1.155,"15min":3.2919999999999998},"last":0.45300000000000001}]},{"osd":6,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57299999999999995,"15min":0.59699999999999998},"min":{"1min":0.39000000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79400000000000004,"5min":0.93799999999999994,"15min":3.2400000000000002},"last":0.374},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.31},"max":{"1min":0.71799999999999997,"5min":1.036,"15min":1.2310000000000001},"last":0.48099999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.44600000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76300000000000001,"5min":1.23,"15min":5.2759999999999998},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69399999999999995,"15min":0.70699999999999996},"min":{"1min":0.45000000000000001,"5min":0.435,"15min":0.42299999999999999},"max":{"1min":0.83199999999999996,"5min":0.97699999999999998,"15min":3.3140000000000001},"last":0.63500000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509707,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420124,"kb_used_data":290388,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93951716,"statfs":{"total":96636764160,"available":96206557184,"internally_reserved":0,"allocated":297357312,"data_stored":296970571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56100000000000005,"15min":0.61299999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.83799999999999997,"5min":0.91400000000000003,"15min":1.524},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61699999999999999,"15min":0.69199999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.5649999999999999},"last":0.89200000000000002}]},{"osd":1,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.376,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.1749999999999998,"5min":4.1749999999999998,"15min":4.1749999999999998},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.625,"15min":0.66600000000000004},"min":{"1min":0.38200000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82599999999999996,"5min":0.94899999999999995,"15min":1.1020000000000001},"last":0.91400000000000003}]},{"osd":2,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.38900000000000001},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":2.3370000000000002},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.68300000000000005,"15min":0.71199999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.84499999999999997,"5min":1.2210000000000001,"15min":2.3660000000000001},"last":0.98999999999999999}]},{"osd":3,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.377},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":2.04},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":2.2240000000000002},"last":1.042}]},{"osd":5,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.58799999999999997,"15min":0.60299999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.9970000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63600000000000001,"15min":0.60499999999999998},"min":{"1min":0.41299999999999998,"5min":0.39500000000000002,"15min":0.33900000000000002},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.77900000000000003}]},{"osd":6,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.60299999999999998,"15min":0.61599999999999999},"min":{"1min":0.33900000000000002,"5min":0.315,"15min":0.312},"max":{"1min":1.2270000000000001,"5min":1.6339999999999999,"15min":2.2639999999999998},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.66200000000000003,"15min":0.67300000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.8300000000000001},"last":0.85299999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.69399999999999995,"15min":0.66800000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2969999999999999,"5min":1.347,"15min":1.347},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65500000000000003,"15min":0.67900000000000005},"min":{"1min":0.38600000000000001,"5min":0.35899999999999999,"15min":0.35599999999999998},"max":{"1min":1.266,"5min":1.266,"15min":2.2999999999999998},"last":1.008}]}]},{"osd":3,"up_from":15,"seq":64424509707,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485596,"kb_used_data":360020,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93886244,"statfs":{"total":96636764160,"available":96139513856,"internally_reserved":0,"allocated":368660480,"data_stored":368273200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":128580040},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.45200000000000001,"15min":0.433},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":0.76300000000000001,"5min":2.8109999999999999,"15min":4.5830000000000002},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.44700000000000001,"15min":0.44600000000000001},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.253},"max":{"1min":0.58199999999999996,"5min":1.147,"15min":1.2509999999999999},"last":0.40799999999999997}]},{"osd":1,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.49099999999999999,"15min":0.503},"min":{"1min":0.33500000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67000000000000004,"5min":1.411,"15min":4.6840000000000002},"last":0.495},{"interface":"front","average":{"1min":0.437,"5min":0.504,"15min":0.51000000000000001},"min":{"1min":0.29699999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61399999999999999,"5min":2.6360000000000001,"15min":2.7429999999999999},"last":0.38300000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.40000000000000002,"5min":0.34399999999999997,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":1.286,"15min":4.6449999999999996},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55700000000000005,"15min":0.51900000000000002},"min":{"1min":0.32200000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":1.1850000000000001,"15min":1.911},"last":0.47299999999999998}]},{"osd":4,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.72399999999999998,"15min":0.76200000000000001},"min":{"1min":0.52000000000000002,"5min":0.45400000000000001,"15min":0.44800000000000001},"max":{"1min":1.0620000000000001,"5min":1.538,"15min":1.7749999999999999},"last":0.877},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.72399999999999998,"15min":0.74199999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.155,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":0.91000000000000003}]},{"osd":5,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.74199999999999999,"15min":0.755},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0780000000000001,"5min":1.4970000000000001,"15min":4.8449999999999998},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.74099999999999999,"15min":0.745},"min":{"1min":0.48199999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1220000000000001,"5min":2.948,"15min":2.948},"last":0.65900000000000003}]},{"osd":6,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.76300000000000001,"15min":0.76600000000000001},"min":{"1min":0.48899999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.98399999999999999,"5min":1.633,"15min":4.8099999999999996},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.45400000000000001},"max":{"1min":0.93899999999999995,"5min":1.6559999999999999,"15min":2.339},"last":0.85799999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.79800000000000004,"15min":0.86499999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.46000000000000002},"max":{"1min":1.216,"5min":1.593,"15min":34.039000000000001},"last":0.753},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.76900000000000002,"15min":0.77700000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0249999999999999,"5min":2.984,"15min":2.984},"last":0.68899999999999995}]}]},{"osd":2,"up_from":15,"seq":64424509708,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303812,"kb_used_data":200444,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94068028,"statfs":{"total":96636764160,"available":96325660672,"internally_reserved":0,"allocated":205254656,"data_stored":204864239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42899999999999999,"15min":0.45000000000000001},"min":{"1min":0.27100000000000002,"5min":0.23400000000000001,"15min":0.20599999999999999},"max":{"1min":0.74299999999999999,"5min":0.92500000000000004,"15min":5.5499999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.52600000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.1180000000000001,"5min":1.1220000000000001,"15min":5.4160000000000004},"last":0.34799999999999998}]},{"osd":1,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.504,"15min":0.5},"min":{"1min":0.307,"5min":0.307,"15min":0.26000000000000001},"max":{"1min":0.81899999999999995,"5min":1.25,"15min":2.2349999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.501},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.27200000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":5.4359999999999999},"last":0.38600000000000001}]},{"osd":3,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.28399999999999997,"5min":0.26400000000000001,"15min":0.25800000000000001},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.312,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.83099999999999996,"5min":1.286,"15min":5.3789999999999996},"last":0.58699999999999997}]},{"osd":4,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.73099999999999998,"15min":0.80000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":3.464,"5min":3.464,"15min":16.905999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.70699999999999996,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.44700000000000001,"15min":0.441},"max":{"1min":0.92100000000000004,"5min":1.3100000000000001,"15min":10.064},"last":0.70699999999999996}]},{"osd":5,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70999999999999996,"15min":0.72999999999999998},"min":{"1min":0.45300000000000001,"5min":0.41099999999999998,"15min":0.32800000000000001},"max":{"1min":1.0960000000000001,"5min":1.304,"15min":3.2989999999999999},"last":0.627},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.73499999999999999,"15min":0.77100000000000002},"min":{"1min":0.52400000000000002,"5min":0.499,"15min":0.46700000000000003},"max":{"1min":1.0069999999999999,"5min":1.3640000000000001,"15min":5.3330000000000002},"last":0.66100000000000003}]},{"osd":6,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.76500000000000001,"15min":0.755},"min":{"1min":0.438,"5min":0.438,"15min":0.42299999999999999},"max":{"1min":3.5710000000000002,"5min":3.5710000000000002,"15min":3.5710000000000002},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.77900000000000003,"15min":0.77000000000000002},"min":{"1min":0.49099999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":5.4749999999999996},"last":0.75800000000000001}]},{"osd":7,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.47399999999999998,"5min":0.47199999999999998,"15min":0.44500000000000001},"max":{"1min":3.5089999999999999,"5min":3.5089999999999999,"15min":5.516},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77200000000000002,"15min":0.79100000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.41899999999999998},"max":{"1min":1.325,"5min":1.325,"15min":8.4610000000000003},"last":0.77600000000000002}]}]},{"osd":1,"up_from":15,"seq":64424509707,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490520,"kb_used_data":360784,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93881320,"statfs":{"total":96636764160,"available":96134471680,"internally_reserved":0,"allocated":369442816,"data_stored":369057373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.437,"15min":0.45700000000000002},"min":{"1min":0.245,"5min":0.219,"15min":0.21299999999999999},"max":{"1min":0.79800000000000004,"5min":0.93000000000000005,"15min":8.9339999999999993},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.34300000000000003,"5min":0.28999999999999998,"15min":0.23000000000000001},"max":{"1min":0.67200000000000004,"5min":1.6899999999999999,"15min":3.8809999999999998},"last":0.63600000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.502,"15min":0.46000000000000002},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.248},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.55900000000000005,"15min":0.55400000000000005},"min":{"1min":0.29699999999999999,"5min":0.29099999999999998,"15min":0.28499999999999998},"max":{"1min":0.88600000000000001,"5min":2.6739999999999999,"15min":9.2680000000000007},"last":0.69699999999999995}]},{"osd":3,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57499999999999996,"15min":0.54200000000000004},"min":{"1min":0.33000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":1.2030000000000001,"5min":1.9910000000000001,"15min":3.0539999999999998},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57999999999999996,"15min":0.57799999999999996},"min":{"1min":0.433,"5min":0.34200000000000003,"15min":0.313},"max":{"1min":0.84999999999999998,"5min":1.4119999999999999,"15min":6.9409999999999998},"last":0.73299999999999998}]},{"osd":4,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.68300000000000005,"15min":0.73199999999999998},"min":{"1min":0.34300000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.90400000000000003,"5min":1.962,"15min":5.6630000000000003},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.72399999999999998,"15min":0.75900000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.151,"5min":1.6719999999999999,"15min":8.8960000000000008},"last":0.88700000000000001}]},{"osd":5,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.76000000000000001,"15min":0.77700000000000002},"min":{"1min":0.48399999999999999,"5min":0.47299999999999998,"15min":0.40300000000000002},"max":{"1min":1.242,"5min":1.242,"15min":6.6159999999999997},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.77800000000000002,"15min":0.79500000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.1220000000000001,"5min":1.7130000000000001,"15min":5.7009999999999996},"last":0.93799999999999994}]},{"osd":6,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.73299999999999998,"15min":0.77300000000000002},"min":{"1min":0.47799999999999998,"5min":0.47299999999999998,"15min":0.39800000000000002},"max":{"1min":0.94199999999999995,"5min":2.0529999999999999,"15min":8.6039999999999992},"last":1.075},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.77200000000000002,"15min":0.75},"min":{"1min":0.50700000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98199999999999998,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.0369999999999999}]},{"osd":7,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76300000000000001,"15min":0.77800000000000002},"min":{"1min":0.46899999999999997,"5min":0.45200000000000001,"15min":0.44800000000000001},"max":{"1min":1.1639999999999999,"5min":1.7470000000000001,"15min":6.6600000000000001},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.78700000000000003,"15min":0.80600000000000005},"min":{"1min":0.50700000000000001,"5min":0.46400000000000002,"15min":0.46200000000000002},"max":{"1min":1.1990000000000001,"5min":1.774,"15min":8.8100000000000005},"last":0.998}]}]},{"osd":0,"up_from":13,"seq":55834575115,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":370924,"kb_used_data":259236,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94000916,"statfs":{"total":96636764160,"available":96256937984,"internally_reserved":0,"allocated":265457664,"data_stored":265061773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,54,40],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.39500000000000002,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.56299999999999994,"5min":1.3029999999999999,"15min":6.8170000000000002},"last":24.483000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.46100000000000002,"15min":0.47499999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.70199999999999996,"5min":1.591,"15min":6.2789999999999999},"last":91.462000000000003}]},{"osd":2,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.51800000000000002,"15min":0.495},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90100000000000002,"5min":1.169,"15min":3.1859999999999999},"last":91.384},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.47699999999999998,"15min":0.46500000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.97499999999999998,"5min":1.266,"15min":1.47},"last":11.782}]},{"osd":3,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56999999999999995,"15min":0.52100000000000002},"min":{"1min":0.40899999999999997,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.67900000000000005,"5min":1.423,"15min":1.8180000000000001},"last":91.557000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.35599999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.93300000000000005,"5min":1.4390000000000001,"15min":1.881},"last":11.835000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.68799999999999994,"15min":0.80700000000000005},"min":{"1min":0.46400000000000002,"5min":0.45100000000000001,"15min":0.45000000000000001},"max":{"1min":1.0529999999999999,"5min":1.3169999999999999,"15min":18.41},"last":13.169},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70399999999999996,"15min":0.81399999999999995},"min":{"1min":0.50800000000000001,"5min":0.46300000000000002,"15min":0.42399999999999999},"max":{"1min":1.097,"5min":1.409,"15min":18.998999999999999},"last":91.421999999999997}]},{"osd":5,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73699999999999999,"15min":0.76100000000000001},"min":{"1min":0.52400000000000002,"5min":0.50900000000000001,"15min":0.42799999999999999},"max":{"1min":1.1299999999999999,"5min":1.3460000000000001,"15min":3.5499999999999998},"last":79.207999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69999999999999996,"15min":0.73699999999999999},"min":{"1min":0.45900000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.996,"5min":1.4279999999999999,"15min":3.605},"last":91.587999999999994}]},{"osd":6,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.73999999999999999,"15min":0.754},"min":{"1min":0.52200000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0289999999999999,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":9.8290000000000006},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.76600000000000001,"15min":0.78200000000000003},"min":{"1min":0.502,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0840000000000001,"5min":2.367,"15min":2.367},"last":91.522999999999996}]},{"osd":7,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.49099999999999999},"max":{"1min":1.042,"5min":1.395,"15min":4.7839999999999998},"last":91.494},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73999999999999999,"15min":0.77000000000000002},"min":{"1min":0.57399999999999995,"5min":0.53200000000000003,"15min":0.45500000000000002},"max":{"1min":0.995,"5min":1.3759999999999999,"15min":2.0110000000000001},"last":91.346000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39333888,"data_stored":39301023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51634176,"data_stored":51607948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18219008,"data_stored":18187806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":47730688,"data_stored":47702623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56041472,"data_stored":56013946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":44449792,"data_stored":44407786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40529920,"data_stored":40501960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":60231680,"data_stored":60196990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":224890880,"data_stored":224890880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":316989440,"data_stored":316989440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186216448,"data_stored":186216448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":320110592,"data_stored":320110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":240496640,"data_stored":240496640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":185778176,"data_stored":185778176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266395648,"data_stored":266395648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":226021376,"data_stored":226021376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T20:58:44.518 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-16T20:58:44.519 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T20:58:44.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.662+0000 7fcda9965640 1 Processor -- start 2023-10-16T20:58:44.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda9965640 1 -- start start 2023-10-16T20:58:44.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda9965640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcda4153770 0x7fcda4159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:44.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda9965640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcda40c9bb0 0x7fcda40c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:44.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda9965640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcda40c41f0 0x7fcda40cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:44.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda9965640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcda40cc250 con 0x7fcda40c9bb0 2023-10-16T20:58:44.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda9965640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcda40cc670 con 0x7fcda40c41f0 2023-10-16T20:58:44.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda9965640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda415a410 con 0x7fcda4153770 2023-10-16T20:58:44.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda27fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcda40c9bb0 0x7fcda40c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:44.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda27fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcda40c9bb0 0x7fcda40c3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54706/0 (socket says 172.21.15.88:54706) 2023-10-16T20:58:44.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda27fc640 1 -- 172.21.15.88:0/3687505124 learned_addr learned my addr 172.21.15.88:0/3687505124 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:44.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda2ffd640 1 --2- 172.21.15.88:0/3687505124 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcda4153770 0x7fcda4159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:44.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda37fe640 1 --2- 172.21.15.88:0/3687505124 >> v2:172.21.15.191:3300/0 conn(0x7fcda40c41f0 0x7fcda40cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:44.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda27fc640 1 -- 172.21.15.88:0/3687505124 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcda4153770 msgr2=0x7fcda4159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:44.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda27fc640 1 --2- 172.21.15.88:0/3687505124 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcda4153770 0x7fcda4159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda27fc640 1 -- 172.21.15.88:0/3687505124 >> v2:172.21.15.191:3300/0 conn(0x7fcda40c41f0 msgr2=0x7fcda40cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:44.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda27fc640 1 --2- 172.21.15.88:0/3687505124 >> v2:172.21.15.191:3300/0 conn(0x7fcda40c41f0 0x7fcda40cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda27fc640 1 -- 172.21.15.88:0/3687505124 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda4153b50 con 0x7fcda40c9bb0 2023-10-16T20:58:44.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.666+0000 7fcda37fe640 1 --2- 172.21.15.88:0/3687505124 >> v2:172.21.15.191:3300/0 conn(0x7fcda40c41f0 0x7fcda40cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:58:44.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.670+0000 7fcda27fc640 1 --2- 172.21.15.88:0/3687505124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcda40c9bb0 0x7fcda40c3cb0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fcda40b2dc0 tx=0x7fcd9801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9162a87b869577 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:44.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.670+0000 7fcda1ffb640 1 -- 172.21.15.88:0/3687505124 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd980ac630 con 0x7fcda40c9bb0 2023-10-16T20:58:44.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.670+0000 7fcda1ffb640 1 -- 172.21.15.88:0/3687505124 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd980ac790 con 0x7fcda40c9bb0 2023-10-16T20:58:44.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.670+0000 7fcda1ffb640 1 -- 172.21.15.88:0/3687505124 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd980acb20 con 0x7fcda40c9bb0 2023-10-16T20:58:44.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.670+0000 7fcda9965640 1 -- 172.21.15.88:0/3687505124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcda40c9bb0 msgr2=0x7fcda40c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:44.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.670+0000 7fcda9965640 1 --2- 172.21.15.88:0/3687505124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcda40c9bb0 0x7fcda40c3cb0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fcda40b2dc0 tx=0x7fcd9801c500 comp rx=0 tx=0).stop 2023-10-16T20:58:44.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.670+0000 7fcda9965640 1 -- 172.21.15.88:0/3687505124 shutdown_connections 2023-10-16T20:58:44.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.670+0000 7fcda9965640 1 --2- 172.21.15.88:0/3687505124 >> v2:172.21.15.191:3300/0 conn(0x7fcda40c41f0 0x7fcda40cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.670+0000 7fcda9965640 1 --2- 172.21.15.88:0/3687505124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcda40c9bb0 0x7fcda40c3cb0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.670+0000 7fcda9965640 1 --2- 172.21.15.88:0/3687505124 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcda4153770 0x7fcda4159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.670+0000 7fcda9965640 1 -- 172.21.15.88:0/3687505124 >> 172.21.15.88:0/3687505124 conn(0x7fcda40715c0 msgr2=0x7fcda40719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:44.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.670+0000 7fcda9965640 1 -- 172.21.15.88:0/3687505124 shutdown_connections 2023-10-16T20:58:44.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.670+0000 7fcda9965640 1 -- 172.21.15.88:0/3687505124 wait complete. 2023-10-16T20:58:44.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.670+0000 7fcda9965640 1 Processor -- start 2023-10-16T20:58:44.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.670+0000 7fcda9965640 1 -- start start 2023-10-16T20:58:44.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda9965640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcda40c41f0 0x7fcda415c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:44.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda9965640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcda40c9bb0 0x7fcda415ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:44.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda9965640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcda4153770 0x7fcda4140520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:44.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda9965640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda415bf50 con 0x7fcda4153770 2023-10-16T20:58:44.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda9965640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda415a970 con 0x7fcda40c41f0 2023-10-16T20:58:44.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda9965640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda415ac50 con 0x7fcda40c9bb0 2023-10-16T20:58:44.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcda4153770 0x7fcda4140520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:44.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcda4153770 0x7fcda4140520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54714/0 (socket says 172.21.15.88:54714) 2023-10-16T20:58:44.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda37fe640 1 -- 172.21.15.88:0/3730998250 learned_addr learned my addr 172.21.15.88:0/3730998250 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:44.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda27fc640 1 --2- 172.21.15.88:0/3730998250 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcda40c9bb0 0x7fcda415ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:44.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda2ffd640 1 --2- 172.21.15.88:0/3730998250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcda40c41f0 0x7fcda415c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:44.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda37fe640 1 -- 172.21.15.88:0/3730998250 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcda40c9bb0 msgr2=0x7fcda415ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:44.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda37fe640 1 --2- 172.21.15.88:0/3730998250 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcda40c9bb0 0x7fcda415ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda37fe640 1 -- 172.21.15.88:0/3730998250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcda40c41f0 msgr2=0x7fcda415c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:44.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda37fe640 1 --2- 172.21.15.88:0/3730998250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcda40c41f0 0x7fcda415c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda37fe640 1 -- 172.21.15.88:0/3730998250 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd980094d0 con 0x7fcda4153770 2023-10-16T20:58:44.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda2ffd640 1 --2- 172.21.15.88:0/3730998250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcda40c41f0 0x7fcda415c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:58:44.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda37fe640 1 --2- 172.21.15.88:0/3730998250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcda4153770 0x7fcda4140520 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c007db0 tx=0x7fcd9c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:44.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcd8bfff640 1 -- 172.21.15.88:0/3730998250 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c0039a0 con 0x7fcda4153770 2023-10-16T20:58:44.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcd8bfff640 1 -- 172.21.15.88:0/3730998250 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9c00f040 con 0x7fcda4153770 2023-10-16T20:58:44.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda9965640 1 -- 172.21.15.88:0/3730998250 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda415af30 con 0x7fcda4153770 2023-10-16T20:58:44.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcd8bfff640 1 -- 172.21.15.88:0/3730998250 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c003d60 con 0x7fcda4153770 2023-10-16T20:58:44.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda9965640 1 -- 172.21.15.88:0/3730998250 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda415b4f0 con 0x7fcda4153770 2023-10-16T20:58:44.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.674+0000 7fcda9965640 1 -- 172.21.15.88:0/3730998250 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda415b6e0 con 0x7fcda4153770 2023-10-16T20:58:44.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.678+0000 7fcd8bfff640 1 -- 172.21.15.88:0/3730998250 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcd9c004030 con 0x7fcda4153770 2023-10-16T20:58:44.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.678+0000 7fcd8bfff640 1 --2- 172.21.15.88:0/3730998250 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcd900664a0 0x7fcd90068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:44.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.678+0000 7fcd8bfff640 1 -- 172.21.15.88:0/3730998250 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7fcd9c0847f0 con 0x7fcda4153770 2023-10-16T20:58:44.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.678+0000 7fcda2ffd640 1 --2- 172.21.15.88:0/3730998250 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcd900664a0 0x7fcd90068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:44.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.678+0000 7fcda2ffd640 1 --2- 172.21.15.88:0/3730998250 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcd900664a0 0x7fcd90068960 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcd94006fd0 tx=0x7fcd94008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:44.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.682+0000 7fcd8bfff640 1 -- 172.21.15.88:0/3730998250 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcd9c051570 con 0x7fcda4153770 2023-10-16T20:58:44.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.830+0000 7fcda9965640 1 -- 172.21.15.88:0/3730998250 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcda40cbb90 con 0x7fcd900664a0 2023-10-16T20:58:44.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.838+0000 7fcd8bfff640 1 -- 172.21.15.88:0/3730998250 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+374317 (secure 0 0 0) 0x7fcd94007710 con 0x7fcd900664a0 2023-10-16T20:58:44.845 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:44.847 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-16T20:58:44.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.842+0000 7fcda9965640 1 -- 172.21.15.88:0/3730998250 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcd900664a0 msgr2=0x7fcd90068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:44.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.842+0000 7fcda9965640 1 --2- 172.21.15.88:0/3730998250 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcd900664a0 0x7fcd90068960 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcd94006fd0 tx=0x7fcd94008040 comp rx=0 tx=0).stop 2023-10-16T20:58:44.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.842+0000 7fcda9965640 1 -- 172.21.15.88:0/3730998250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcda4153770 msgr2=0x7fcda4140520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:44.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.842+0000 7fcda9965640 1 --2- 172.21.15.88:0/3730998250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcda4153770 0x7fcda4140520 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c007db0 tx=0x7fcd9c00d4a0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.842+0000 7fcda9965640 1 -- 172.21.15.88:0/3730998250 shutdown_connections 2023-10-16T20:58:44.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.842+0000 7fcda9965640 1 --2- 172.21.15.88:0/3730998250 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fcd900664a0 0x7fcd90068960 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.842+0000 7fcda9965640 1 --2- 172.21.15.88:0/3730998250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcda4153770 0x7fcda4140520 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.842+0000 7fcda9965640 1 --2- 172.21.15.88:0/3730998250 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fcda40c9bb0 0x7fcda415ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.842+0000 7fcda9965640 1 --2- 172.21.15.88:0/3730998250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcda40c41f0 0x7fcda415c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:44.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.842+0000 7fcda9965640 1 -- 172.21.15.88:0/3730998250 >> 172.21.15.88:0/3730998250 conn(0x7fcda40715c0 msgr2=0x7fcda40719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:44.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.846+0000 7fcda9965640 1 -- 172.21.15.88:0/3730998250 shutdown_connections 2023-10-16T20:58:44.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:44.846+0000 7fcda9965640 1 -- 172.21.15.88:0/3730998250 wait complete. 2023-10-16T20:58:44.865 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":722,"stamp":"2023-10-16T20:58:43.931001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1103869876,"num_objects":10494,"num_object_clones":0,"num_object_copies":41576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10494,"num_whiteouts":0,"num_read":31781,"num_read_kb":882609,"num_write":27510,"num_write_kb":1485755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":115,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34039,"ondisk_log_size":34039,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3214536,"kb_used_data":2277960,"kb_used_omap":12,"kb_used_meta":936499,"kb_avail":751760184,"statfs":{"total":773094113280,"available":769802428416,"internally_reserved":0,"allocated":2332631040,"data_stored":2329499415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":958975545},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,54,40],"upper_bound":128},"perf_stat":{"commit_latency_ms":145,"apply_latency_ms":145,"commit_latency_ns":145000000,"apply_latency_ns":145000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4254655,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":49,"num_write":147,"num_write_kb":4242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003434"},"pg_stats":[{"pgid":"2.1f","version":"67'323","reported_seq":411,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:39.469805+0000","last_change":"2023-10-16T20:36:54.763744+0000","last_active":"2023-10-16T20:58:39.469805+0000","last_peered":"2023-10-16T20:58:39.469805+0000","last_clean":"2023-10-16T20:58:39.469805+0000","last_became_active":"2023-10-16T20:36:54.763457+0000","last_became_peered":"2023-10-16T20:36:54.763457+0000","last_unstale":"2023-10-16T20:58:39.469805+0000","last_undegraded":"2023-10-16T20:58:39.469805+0000","last_fullsized":"2023-10-16T20:58:39.469805+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:05.774485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9864502,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":42,"num_read_kb":9548,"num_write":321,"num_write_kb":9823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"67'107","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.869873+0000","last_change":"2023-10-16T20:47:31.588044+0000","last_active":"2023-10-16T20:58:37.869873+0000","last_peered":"2023-10-16T20:58:37.869873+0000","last_clean":"2023-10-16T20:58:37.869873+0000","last_became_active":"2023-10-16T20:36:55.302519+0000","last_became_peered":"2023-10-16T20:36:55.302519+0000","last_unstale":"2023-10-16T20:58:37.869873+0000","last_undegraded":"2023-10-16T20:58:37.869873+0000","last_fullsized":"2023-10-16T20:58:37.869873+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:49.037032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007784165,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'672","reported_seq":3556,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.645718+0000","last_change":"2023-10-16T20:47:31.584576+0000","last_active":"2023-10-16T20:58:22.645718+0000","last_peered":"2023-10-16T20:58:22.645718+0000","last_clean":"2023-10-16T20:58:22.645718+0000","last_became_active":"2023-10-16T20:36:59.466306+0000","last_became_peered":"2023-10-16T20:36:59.466306+0000","last_unstale":"2023-10-16T20:58:22.645718+0000","last_undegraded":"2023-10-16T20:58:22.645718+0000","last_fullsized":"2023-10-16T20:58:22.645718+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:56.295661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073106899999999999,"stat_sum":{"num_bytes":36865036,"num_objects":299,"num_object_clones":0,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":299,"num_whiteouts":0,"num_read":779,"num_read_kb":25449,"num_write":671,"num_write_kb":56816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"67'207","reported_seq":277,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:00.972618+0000","last_change":"2023-10-16T20:36:54.759491+0000","last_active":"2023-10-16T20:58:00.972618+0000","last_peered":"2023-10-16T20:58:00.972618+0000","last_clean":"2023-10-16T20:58:00.972618+0000","last_became_active":"2023-10-16T20:36:54.759177+0000","last_became_peered":"2023-10-16T20:36:54.759177+0000","last_unstale":"2023-10-16T20:58:00.972618+0000","last_undegraded":"2023-10-16T20:58:00.972618+0000","last_fullsized":"2023-10-16T20:58:00.972618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:43:36.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":12288,"num_write":199,"num_write_kb":8312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"67'107","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889862+0000","last_change":"2023-10-16T20:47:31.488681+0000","last_active":"2023-10-16T20:58:37.889862+0000","last_peered":"2023-10-16T20:58:37.889862+0000","last_clean":"2023-10-16T20:58:37.889862+0000","last_became_active":"2023-10-16T20:36:55.291458+0000","last_became_peered":"2023-10-16T20:36:55.291458+0000","last_unstale":"2023-10-16T20:58:37.889862+0000","last_undegraded":"2023-10-16T20:58:37.889862+0000","last_fullsized":"2023-10-16T20:58:37.889862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:03:05.558721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087429459999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'682","reported_seq":3570,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.576451+0000","last_change":"2023-10-16T20:47:31.484150+0000","last_active":"2023-10-16T20:58:22.576451+0000","last_peered":"2023-10-16T20:58:22.576451+0000","last_clean":"2023-10-16T20:58:22.576451+0000","last_became_active":"2023-10-16T20:36:59.302968+0000","last_became_peered":"2023-10-16T20:36:59.302968+0000","last_unstale":"2023-10-16T20:58:22.576451+0000","last_undegraded":"2023-10-16T20:58:22.576451+0000","last_fullsized":"2023-10-16T20:58:22.576451+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:58:43.745409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027985639999999999,"stat_sum":{"num_bytes":24765378,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":783,"num_read_kb":25781,"num_write":678,"num_write_kb":40921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"67'340","reported_seq":437,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:33.688387+0000","last_change":"2023-10-16T20:36:54.762600+0000","last_active":"2023-10-16T20:58:33.688387+0000","last_peered":"2023-10-16T20:58:33.688387+0000","last_clean":"2023-10-16T20:58:33.688387+0000","last_became_active":"2023-10-16T20:36:54.762278+0000","last_became_peered":"2023-10-16T20:36:54.762278+0000","last_unstale":"2023-10-16T20:58:33.688387+0000","last_undegraded":"2023-10-16T20:58:33.688387+0000","last_fullsized":"2023-10-16T20:58:33.688387+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:46:03.836415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":85,"num_read_kb":16492,"num_write":374,"num_write_kb":12523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'96","reported_seq":2197,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.131579+0000","last_change":"2023-10-16T20:47:31.496372+0000","last_active":"2023-10-16T20:58:38.131579+0000","last_peered":"2023-10-16T20:58:38.131579+0000","last_clean":"2023-10-16T20:58:38.131579+0000","last_became_active":"2023-10-16T20:37:37.402987+0000","last_became_peered":"2023-10-16T20:37:37.402987+0000","last_unstale":"2023-10-16T20:58:38.131579+0000","last_undegraded":"2023-10-16T20:58:38.131579+0000","last_fullsized":"2023-10-16T20:58:38.131579+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:12:06.964995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082573900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"67'673","reported_seq":4224,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.678192+0000","last_change":"2023-10-16T20:47:31.485701+0000","last_active":"2023-10-16T20:58:22.678192+0000","last_peered":"2023-10-16T20:58:22.678192+0000","last_clean":"2023-10-16T20:58:22.678192+0000","last_became_active":"2023-10-16T20:36:59.303805+0000","last_became_peered":"2023-10-16T20:36:59.303805+0000","last_unstale":"2023-10-16T20:58:22.678192+0000","last_undegraded":"2023-10-16T20:58:22.678192+0000","last_fullsized":"2023-10-16T20:58:22.678192+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:28:11.659597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041657500000000002,"stat_sum":{"num_bytes":49691093,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":1446,"num_read_kb":31388,"num_write":673,"num_write_kb":52938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"67'172","reported_seq":252,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.691336+0000","last_change":"2023-10-16T20:36:54.749849+0000","last_active":"2023-10-16T20:58:28.691336+0000","last_peered":"2023-10-16T20:58:28.691336+0000","last_clean":"2023-10-16T20:58:28.691336+0000","last_became_active":"2023-10-16T20:36:54.749316+0000","last_became_peered":"2023-10-16T20:36:54.749316+0000","last_unstale":"2023-10-16T20:58:28.691336+0000","last_undegraded":"2023-10-16T20:58:28.691336+0000","last_fullsized":"2023-10-16T20:58:28.691336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:49:29.077126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":170,"num_write_kb":4198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"67'104","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.872186+0000","last_change":"2023-10-16T20:47:31.592091+0000","last_active":"2023-10-16T20:58:37.872186+0000","last_peered":"2023-10-16T20:58:37.872186+0000","last_clean":"2023-10-16T20:58:37.872186+0000","last_became_active":"2023-10-16T20:36:55.302387+0000","last_became_peered":"2023-10-16T20:36:55.302387+0000","last_unstale":"2023-10-16T20:58:37.872186+0000","last_undegraded":"2023-10-16T20:58:37.872186+0000","last_fullsized":"2023-10-16T20:58:37.872186+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:07:35.871205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011585343,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'641","reported_seq":3839,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.036707+0000","last_change":"2023-10-16T20:47:31.486382+0000","last_active":"2023-10-16T20:58:23.036707+0000","last_peered":"2023-10-16T20:58:23.036707+0000","last_clean":"2023-10-16T20:58:23.036707+0000","last_became_active":"2023-10-16T20:36:59.305228+0000","last_became_peered":"2023-10-16T20:36:59.305228+0000","last_unstale":"2023-10-16T20:58:23.036707+0000","last_undegraded":"2023-10-16T20:58:23.036707+0000","last_fullsized":"2023-10-16T20:58:23.036707+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":641,"log_dups_size":0,"ondisk_log_size":641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:32:37.261529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047877249999999996,"stat_sum":{"num_bytes":34900728,"num_objects":310,"num_object_clones":0,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":1093,"num_read_kb":21879,"num_write":638,"num_write_kb":46703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"67'104","reported_seq":2214,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.863809+0000","last_change":"2023-10-16T20:47:31.592641+0000","last_active":"2023-10-16T20:58:37.863809+0000","last_peered":"2023-10-16T20:58:37.863809+0000","last_clean":"2023-10-16T20:58:37.863809+0000","last_became_active":"2023-10-16T20:36:55.306427+0000","last_became_peered":"2023-10-16T20:36:55.306427+0000","last_unstale":"2023-10-16T20:58:37.863809+0000","last_undegraded":"2023-10-16T20:58:37.863809+0000","last_fullsized":"2023-10-16T20:58:37.863809+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:17:46.390047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013854674000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"67'187","reported_seq":258,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.360075+0000","last_change":"2023-10-16T20:36:54.768375+0000","last_active":"2023-10-16T20:58:05.360075+0000","last_peered":"2023-10-16T20:58:05.360075+0000","last_clean":"2023-10-16T20:58:05.360075+0000","last_became_active":"2023-10-16T20:36:54.767996+0000","last_became_peered":"2023-10-16T20:36:54.767996+0000","last_unstale":"2023-10-16T20:58:05.360075+0000","last_undegraded":"2023-10-16T20:58:05.360075+0000","last_fullsized":"2023-10-16T20:58:05.360075+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:55.303942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":8202,"num_write":173,"num_write_kb":8293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"67'648","reported_seq":4473,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.721394+0000","last_change":"2023-10-16T20:47:31.487925+0000","last_active":"2023-10-16T20:58:22.721394+0000","last_peered":"2023-10-16T20:58:22.721394+0000","last_clean":"2023-10-16T20:58:22.721394+0000","last_became_active":"2023-10-16T20:36:59.306640+0000","last_became_peered":"2023-10-16T20:36:59.306640+0000","last_unstale":"2023-10-16T20:58:22.721394+0000","last_undegraded":"2023-10-16T20:58:22.721394+0000","last_fullsized":"2023-10-16T20:58:22.721394+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:27:48.041001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054537470000000001,"stat_sum":{"num_bytes":41205181,"num_objects":318,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":1719,"num_read_kb":37802,"num_write":648,"num_write_kb":48776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"67'101","reported_seq":2210,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.891454+0000","last_change":"2023-10-16T20:47:31.587617+0000","last_active":"2023-10-16T20:58:37.891454+0000","last_peered":"2023-10-16T20:58:37.891454+0000","last_clean":"2023-10-16T20:58:37.891454+0000","last_became_active":"2023-10-16T20:36:55.275085+0000","last_became_peered":"2023-10-16T20:36:55.275085+0000","last_unstale":"2023-10-16T20:58:37.891454+0000","last_undegraded":"2023-10-16T20:58:37.891454+0000","last_fullsized":"2023-10-16T20:58:37.891454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:15.882497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087599630000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'182","reported_seq":251,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:41.511040+0000","last_change":"2023-10-16T20:36:54.756174+0000","last_active":"2023-10-16T20:58:41.511040+0000","last_peered":"2023-10-16T20:58:41.511040+0000","last_clean":"2023-10-16T20:58:41.511040+0000","last_became_active":"2023-10-16T20:36:54.755872+0000","last_became_peered":"2023-10-16T20:36:54.755872+0000","last_unstale":"2023-10-16T20:58:41.511040+0000","last_undegraded":"2023-10-16T20:58:41.511040+0000","last_fullsized":"2023-10-16T20:58:41.511040+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:05:32.741671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":10,"num_write":281,"num_write_kb":4257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"67'699","reported_seq":3454,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.718374+0000","last_change":"2023-10-16T20:47:31.486593+0000","last_active":"2023-10-16T20:58:22.718374+0000","last_peered":"2023-10-16T20:58:22.718374+0000","last_clean":"2023-10-16T20:58:22.718374+0000","last_became_active":"2023-10-16T20:36:59.302919+0000","last_became_peered":"2023-10-16T20:36:59.302919+0000","last_unstale":"2023-10-16T20:58:22.718374+0000","last_undegraded":"2023-10-16T20:58:22.718374+0000","last_fullsized":"2023-10-16T20:58:22.718374+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:48.779648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046546790000000001,"stat_sum":{"num_bytes":20793675,"num_objects":334,"num_object_clones":0,"num_object_copies":1336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":334,"num_whiteouts":0,"num_read":649,"num_read_kb":16394,"num_write":696,"num_write_kb":45235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"67'106","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.886436+0000","last_change":"2023-10-16T20:47:31.586816+0000","last_active":"2023-10-16T20:58:37.886436+0000","last_peered":"2023-10-16T20:58:37.886436+0000","last_clean":"2023-10-16T20:58:37.886436+0000","last_became_active":"2023-10-16T20:36:55.270455+0000","last_became_peered":"2023-10-16T20:36:55.270455+0000","last_unstale":"2023-10-16T20:58:37.886436+0000","last_undegraded":"2023-10-16T20:58:37.886436+0000","last_fullsized":"2023-10-16T20:58:37.886436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:37.316278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.008577889,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"66'49","reported_seq":127,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.707520+0000","last_change":"2023-10-16T20:36:54.744128+0000","last_active":"2023-10-16T20:58:28.707520+0000","last_peered":"2023-10-16T20:58:28.707520+0000","last_clean":"2023-10-16T20:58:28.707520+0000","last_became_active":"2023-10-16T20:36:54.743805+0000","last_became_peered":"2023-10-16T20:36:54.743805+0000","last_unstale":"2023-10-16T20:58:28.707520+0000","last_undegraded":"2023-10-16T20:58:28.707520+0000","last_fullsized":"2023-10-16T20:58:28.707520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:54:35.819820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":55,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'651","reported_seq":3692,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.352813+0000","last_change":"2023-10-16T20:47:31.485640+0000","last_active":"2023-10-16T20:58:22.352813+0000","last_peered":"2023-10-16T20:58:22.352813+0000","last_clean":"2023-10-16T20:58:22.352813+0000","last_became_active":"2023-10-16T20:36:59.465172+0000","last_became_peered":"2023-10-16T20:36:59.465172+0000","last_unstale":"2023-10-16T20:58:22.352813+0000","last_undegraded":"2023-10-16T20:58:22.352813+0000","last_fullsized":"2023-10-16T20:58:22.352813+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:18:14.804407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040107390000000001,"stat_sum":{"num_bytes":25409516,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":935,"num_read_kb":27643,"num_write":650,"num_write_kb":33262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"67'103","reported_seq":2213,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.849592+0000","last_change":"2023-10-16T20:47:31.487014+0000","last_active":"2023-10-16T20:58:37.849592+0000","last_peered":"2023-10-16T20:58:37.849592+0000","last_clean":"2023-10-16T20:58:37.849592+0000","last_became_active":"2023-10-16T20:36:55.278568+0000","last_became_peered":"2023-10-16T20:36:55.278568+0000","last_unstale":"2023-10-16T20:58:37.849592+0000","last_undegraded":"2023-10-16T20:58:37.849592+0000","last_fullsized":"2023-10-16T20:58:37.849592+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:01:00.125646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050399599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"66'287","reported_seq":358,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:01.079180+0000","last_change":"2023-10-16T20:36:54.758778+0000","last_active":"2023-10-16T20:58:01.079180+0000","last_peered":"2023-10-16T20:58:01.079180+0000","last_clean":"2023-10-16T20:58:01.079180+0000","last_became_active":"2023-10-16T20:36:54.758496+0000","last_became_peered":"2023-10-16T20:36:54.758496+0000","last_unstale":"2023-10-16T20:58:01.079180+0000","last_undegraded":"2023-10-16T20:58:01.079180+0000","last_fullsized":"2023-10-16T20:58:01.079180+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:36:00.515106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":12294,"num_write":291,"num_write_kb":4310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"67'586","reported_seq":3861,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.527756+0000","last_change":"2023-10-16T20:47:31.488200+0000","last_active":"2023-10-16T20:58:22.527756+0000","last_peered":"2023-10-16T20:58:22.527756+0000","last_clean":"2023-10-16T20:58:22.527756+0000","last_became_active":"2023-10-16T20:36:59.309068+0000","last_became_peered":"2023-10-16T20:36:59.309068+0000","last_unstale":"2023-10-16T20:58:22.527756+0000","last_undegraded":"2023-10-16T20:58:22.527756+0000","last_fullsized":"2023-10-16T20:58:22.527756+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:22:11.244977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055482439999999999,"stat_sum":{"num_bytes":36900597,"num_objects":303,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1169,"num_read_kb":30992,"num_write":585,"num_write_kb":40444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'101","reported_seq":2211,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889241+0000","last_change":"2023-10-16T20:47:31.492407+0000","last_active":"2023-10-16T20:58:37.889241+0000","last_peered":"2023-10-16T20:58:37.889241+0000","last_clean":"2023-10-16T20:58:37.889241+0000","last_became_active":"2023-10-16T20:36:55.272840+0000","last_became_peered":"2023-10-16T20:36:55.272840+0000","last_unstale":"2023-10-16T20:58:37.889241+0000","last_undegraded":"2023-10-16T20:58:37.889241+0000","last_fullsized":"2023-10-16T20:58:37.889241+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:35:42.200237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011897911000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"30'8","reported_seq":88,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.710153+0000","last_change":"2023-10-16T20:36:54.760681+0000","last_active":"2023-10-16T20:58:28.710153+0000","last_peered":"2023-10-16T20:58:28.710153+0000","last_clean":"2023-10-16T20:58:28.710153+0000","last_became_active":"2023-10-16T20:36:54.760397+0000","last_became_peered":"2023-10-16T20:36:54.760397+0000","last_unstale":"2023-10-16T20:58:28.710153+0000","last_undegraded":"2023-10-16T20:58:28.710153+0000","last_fullsized":"2023-10-16T20:58:28.710153+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:47:16.510382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"67'710","reported_seq":3798,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.025282+0000","last_change":"2023-10-16T20:47:31.485980+0000","last_active":"2023-10-16T20:58:23.025282+0000","last_peered":"2023-10-16T20:58:23.025282+0000","last_clean":"2023-10-16T20:58:23.025282+0000","last_became_active":"2023-10-16T20:36:59.465152+0000","last_became_peered":"2023-10-16T20:36:59.465152+0000","last_unstale":"2023-10-16T20:58:23.025282+0000","last_undegraded":"2023-10-16T20:58:23.025282+0000","last_fullsized":"2023-10-16T20:58:23.025282+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:48.800573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044307399999999999,"stat_sum":{"num_bytes":25809469,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":981,"num_read_kb":15497,"num_write":706,"num_write_kb":50141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'106","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.882745+0000","last_change":"2023-10-16T20:47:31.587090+0000","last_active":"2023-10-16T20:58:37.882745+0000","last_peered":"2023-10-16T20:58:37.882745+0000","last_clean":"2023-10-16T20:58:37.882745+0000","last_became_active":"2023-10-16T20:36:55.285566+0000","last_became_peered":"2023-10-16T20:36:55.285566+0000","last_unstale":"2023-10-16T20:58:37.882745+0000","last_undegraded":"2023-10-16T20:58:37.882745+0000","last_fullsized":"2023-10-16T20:58:37.882745+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:18.835080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085500479999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'101","reported_seq":177,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.657997+0000","last_change":"2023-10-16T20:36:54.758062+0000","last_active":"2023-10-16T20:58:37.657997+0000","last_peered":"2023-10-16T20:58:37.657997+0000","last_clean":"2023-10-16T20:58:37.657997+0000","last_became_active":"2023-10-16T20:36:54.757796+0000","last_became_peered":"2023-10-16T20:36:54.757796+0000","last_unstale":"2023-10-16T20:58:37.657997+0000","last_undegraded":"2023-10-16T20:58:37.657997+0000","last_fullsized":"2023-10-16T20:58:37.657997+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:02:09.953795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":254,"num_write_kb":1037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"67'686","reported_seq":3331,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.536581+0000","last_change":"2023-10-16T20:47:31.582989+0000","last_active":"2023-10-16T20:58:22.536581+0000","last_peered":"2023-10-16T20:58:22.536581+0000","last_clean":"2023-10-16T20:58:22.536581+0000","last_became_active":"2023-10-16T20:36:59.465221+0000","last_became_peered":"2023-10-16T20:36:59.465221+0000","last_unstale":"2023-10-16T20:58:22.536581+0000","last_undegraded":"2023-10-16T20:58:22.536581+0000","last_fullsized":"2023-10-16T20:58:22.536581+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:29:08.900666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059334330000000001,"stat_sum":{"num_bytes":20326999,"num_objects":343,"num_object_clones":0,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":538,"num_read_kb":13756,"num_write":685,"num_write_kb":32496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"67'101","reported_seq":2211,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.818833+0000","last_change":"2023-10-16T20:47:31.489294+0000","last_active":"2023-10-16T20:58:37.818833+0000","last_peered":"2023-10-16T20:58:37.818833+0000","last_clean":"2023-10-16T20:58:37.818833+0000","last_became_active":"2023-10-16T20:36:55.274432+0000","last_became_peered":"2023-10-16T20:36:55.274432+0000","last_unstale":"2023-10-16T20:58:37.818833+0000","last_undegraded":"2023-10-16T20:58:37.818833+0000","last_fullsized":"2023-10-16T20:58:37.818833+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:50:04.713610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057461259999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"66'34","reported_seq":109,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:01.080176+0000","last_change":"2023-10-16T20:36:54.762356+0000","last_active":"2023-10-16T20:58:01.080176+0000","last_peered":"2023-10-16T20:58:01.080176+0000","last_clean":"2023-10-16T20:58:01.080176+0000","last_became_active":"2023-10-16T20:36:54.762167+0000","last_became_peered":"2023-10-16T20:36:54.762167+0000","last_unstale":"2023-10-16T20:58:01.080176+0000","last_undegraded":"2023-10-16T20:58:01.080176+0000","last_fullsized":"2023-10-16T20:58:01.080176+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:45.164625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":34,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"67'725","reported_seq":3715,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.031400+0000","last_change":"2023-10-16T20:47:31.489099+0000","last_active":"2023-10-16T20:58:23.031400+0000","last_peered":"2023-10-16T20:58:23.031400+0000","last_clean":"2023-10-16T20:58:23.031400+0000","last_became_active":"2023-10-16T20:36:59.466876+0000","last_became_peered":"2023-10-16T20:36:59.466876+0000","last_unstale":"2023-10-16T20:58:23.031400+0000","last_undegraded":"2023-10-16T20:58:23.031400+0000","last_fullsized":"2023-10-16T20:58:23.031400+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":725,"log_dups_size":0,"ondisk_log_size":725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:45:55.969279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009058969,"stat_sum":{"num_bytes":44901233,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":884,"num_read_kb":20657,"num_write":720,"num_write_kb":60576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"67'98","reported_seq":2208,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.895976+0000","last_change":"2023-10-16T20:47:31.594422+0000","last_active":"2023-10-16T20:58:37.895976+0000","last_peered":"2023-10-16T20:58:37.895976+0000","last_clean":"2023-10-16T20:58:37.895976+0000","last_became_active":"2023-10-16T20:36:55.293228+0000","last_became_peered":"2023-10-16T20:36:55.293228+0000","last_unstale":"2023-10-16T20:58:37.895976+0000","last_undegraded":"2023-10-16T20:58:37.895976+0000","last_fullsized":"2023-10-16T20:58:37.895976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:33:58.039426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013567460999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"66'210","reported_seq":278,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:47.652674+0000","last_change":"2023-10-16T20:36:54.757588+0000","last_active":"2023-10-16T20:57:47.652674+0000","last_peered":"2023-10-16T20:57:47.652674+0000","last_clean":"2023-10-16T20:57:47.652674+0000","last_became_active":"2023-10-16T20:36:54.757308+0000","last_became_peered":"2023-10-16T20:36:54.757308+0000","last_unstale":"2023-10-16T20:57:47.652674+0000","last_undegraded":"2023-10-16T20:57:47.652674+0000","last_fullsized":"2023-10-16T20:57:47.652674+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:36:51.298239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":3049,"num_write":207,"num_write_kb":4195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"67'671","reported_seq":3425,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.522842+0000","last_change":"2023-10-16T20:47:31.485489+0000","last_active":"2023-10-16T20:58:22.522842+0000","last_peered":"2023-10-16T20:58:22.522842+0000","last_clean":"2023-10-16T20:58:22.522842+0000","last_became_active":"2023-10-16T20:36:59.303224+0000","last_became_peered":"2023-10-16T20:36:59.303224+0000","last_unstale":"2023-10-16T20:58:22.522842+0000","last_undegraded":"2023-10-16T20:58:22.522842+0000","last_fullsized":"2023-10-16T20:58:22.522842+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:03:55.965341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038422399999999998,"stat_sum":{"num_bytes":21817454,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":647,"num_read_kb":9306,"num_write":673,"num_write_kb":37999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"67'120","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889222+0000","last_change":"2023-10-16T20:47:31.487515+0000","last_active":"2023-10-16T20:58:37.889222+0000","last_peered":"2023-10-16T20:58:37.889222+0000","last_clean":"2023-10-16T20:58:37.889222+0000","last_became_active":"2023-10-16T20:36:55.277872+0000","last_became_peered":"2023-10-16T20:36:55.277872+0000","last_unstale":"2023-10-16T20:58:37.889222+0000","last_undegraded":"2023-10-16T20:58:37.889222+0000","last_fullsized":"2023-10-16T20:58:37.889222+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:46.751931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052323379999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"66'532","reported_seq":616,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.670124+0000","last_change":"2023-10-16T20:36:54.740644+0000","last_active":"2023-10-16T20:58:05.670124+0000","last_peered":"2023-10-16T20:58:05.670124+0000","last_clean":"2023-10-16T20:58:05.670124+0000","last_became_active":"2023-10-16T20:36:54.740340+0000","last_became_peered":"2023-10-16T20:36:54.740340+0000","last_unstale":"2023-10-16T20:58:05.670124+0000","last_undegraded":"2023-10-16T20:58:05.670124+0000","last_fullsized":"2023-10-16T20:58:05.670124+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:22.221727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21285,"num_write":524,"num_write_kb":16645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'623","reported_seq":3877,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.676798+0000","last_change":"2023-10-16T20:47:31.489650+0000","last_active":"2023-10-16T20:58:22.676798+0000","last_peered":"2023-10-16T20:58:22.676798+0000","last_clean":"2023-10-16T20:58:22.676798+0000","last_became_active":"2023-10-16T20:36:59.294548+0000","last_became_peered":"2023-10-16T20:36:59.294548+0000","last_unstale":"2023-10-16T20:58:22.676798+0000","last_undegraded":"2023-10-16T20:58:22.676798+0000","last_fullsized":"2023-10-16T20:58:22.676798+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:53:12.441064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062280529999999999,"stat_sum":{"num_bytes":29651112,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":1147,"num_read_kb":23932,"num_write":623,"num_write_kb":40690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"67'89","reported_seq":2199,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.807118+0000","last_change":"2023-10-16T20:47:31.484434+0000","last_active":"2023-10-16T20:58:37.807118+0000","last_peered":"2023-10-16T20:58:37.807118+0000","last_clean":"2023-10-16T20:58:37.807118+0000","last_became_active":"2023-10-16T20:36:55.269749+0000","last_became_peered":"2023-10-16T20:36:55.269749+0000","last_unstale":"2023-10-16T20:58:37.807118+0000","last_undegraded":"2023-10-16T20:58:37.807118+0000","last_fullsized":"2023-10-16T20:58:37.807118+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:21:13.945910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003009004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'188","reported_seq":376,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.714114+0000","last_change":"2023-10-16T20:36:54.757903+0000","last_active":"2023-10-16T20:58:37.714114+0000","last_peered":"2023-10-16T20:58:37.714114+0000","last_clean":"2023-10-16T20:58:37.714114+0000","last_became_active":"2023-10-16T20:36:54.757549+0000","last_became_peered":"2023-10-16T20:36:54.757549+0000","last_unstale":"2023-10-16T20:58:37.714114+0000","last_undegraded":"2023-10-16T20:58:37.714114+0000","last_fullsized":"2023-10-16T20:58:37.714114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:02:42.090848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1409828,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":124,"num_read_kb":239,"num_write":184,"num_write_kb":1568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'647","reported_seq":3934,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.025459+0000","last_change":"2023-10-16T20:47:31.487517+0000","last_active":"2023-10-16T20:58:23.025459+0000","last_peered":"2023-10-16T20:58:23.025459+0000","last_clean":"2023-10-16T20:58:23.025459+0000","last_became_active":"2023-10-16T20:36:59.301693+0000","last_became_peered":"2023-10-16T20:36:59.301693+0000","last_unstale":"2023-10-16T20:58:23.025459+0000","last_undegraded":"2023-10-16T20:58:23.025459+0000","last_fullsized":"2023-10-16T20:58:23.025459+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:04:52.612265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056692870000000003,"stat_sum":{"num_bytes":29916858,"num_objects":298,"num_object_clones":0,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":298,"num_whiteouts":0,"num_read":1180,"num_read_kb":25235,"num_write":643,"num_write_kb":45892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"67'104","reported_seq":2214,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888945+0000","last_change":"2023-10-16T20:47:31.487504+0000","last_active":"2023-10-16T20:58:37.888945+0000","last_peered":"2023-10-16T20:58:37.888945+0000","last_clean":"2023-10-16T20:58:37.888945+0000","last_became_active":"2023-10-16T20:36:55.276649+0000","last_became_peered":"2023-10-16T20:36:55.276649+0000","last_unstale":"2023-10-16T20:58:37.888945+0000","last_undegraded":"2023-10-16T20:58:37.888945+0000","last_fullsized":"2023-10-16T20:58:37.888945+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:12:33.903227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076292809999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'128","reported_seq":201,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:01.079234+0000","last_change":"2023-10-16T20:36:54.753592+0000","last_active":"2023-10-16T20:58:01.079234+0000","last_peered":"2023-10-16T20:58:01.079234+0000","last_clean":"2023-10-16T20:58:01.079234+0000","last_became_active":"2023-10-16T20:36:54.753237+0000","last_became_peered":"2023-10-16T20:36:54.753237+0000","last_unstale":"2023-10-16T20:58:01.079234+0000","last_undegraded":"2023-10-16T20:58:01.079234+0000","last_fullsized":"2023-10-16T20:58:01.079234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:53.005979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":20486,"num_write":119,"num_write_kb":8247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'673","reported_seq":3483,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.284780+0000","last_change":"2023-10-16T20:47:31.491504+0000","last_active":"2023-10-16T20:58:23.284780+0000","last_peered":"2023-10-16T20:58:23.284780+0000","last_clean":"2023-10-16T20:58:23.284780+0000","last_became_active":"2023-10-16T20:36:59.466708+0000","last_became_peered":"2023-10-16T20:36:59.466708+0000","last_unstale":"2023-10-16T20:58:23.284780+0000","last_undegraded":"2023-10-16T20:58:23.284780+0000","last_fullsized":"2023-10-16T20:58:23.284780+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:33:22.349924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011259905000000001,"stat_sum":{"num_bytes":19802424,"num_objects":318,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":703,"num_read_kb":14564,"num_write":670,"num_write_kb":31986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"67'99","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.897919+0000","last_change":"2023-10-16T20:47:31.585286+0000","last_active":"2023-10-16T20:58:37.897919+0000","last_peered":"2023-10-16T20:58:37.897919+0000","last_clean":"2023-10-16T20:58:37.897919+0000","last_became_active":"2023-10-16T20:36:55.294610+0000","last_became_peered":"2023-10-16T20:36:55.294610+0000","last_unstale":"2023-10-16T20:58:37.897919+0000","last_undegraded":"2023-10-16T20:58:37.897919+0000","last_fullsized":"2023-10-16T20:58:37.897919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:34:05.113883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075091560000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"66'7","reported_seq":83,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:00.736127+0000","last_change":"2023-10-16T20:36:54.741191+0000","last_active":"2023-10-16T20:58:00.736127+0000","last_peered":"2023-10-16T20:58:00.736127+0000","last_clean":"2023-10-16T20:58:00.736127+0000","last_became_active":"2023-10-16T20:36:54.740944+0000","last_became_peered":"2023-10-16T20:36:54.740944+0000","last_unstale":"2023-10-16T20:58:00.736127+0000","last_undegraded":"2023-10-16T20:58:00.736127+0000","last_fullsized":"2023-10-16T20:58:00.736127+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:05.961850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":23,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":219,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:56:52.651677+0000","last_change":"2023-10-16T20:36:46.390001+0000","last_active":"2023-10-16T20:56:52.651677+0000","last_peered":"2023-10-16T20:56:52.651677+0000","last_clean":"2023-10-16T20:56:52.651677+0000","last_became_active":"2023-10-16T20:36:46.388824+0000","last_became_peered":"2023-10-16T20:36:46.388824+0000","last_unstale":"2023-10-16T20:56:52.651677+0000","last_undegraded":"2023-10-16T20:56:52.651677+0000","last_fullsized":"2023-10-16T20:56:52.651677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_clean_scrub_stamp":"2023-10-16T20:36:44.575506+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:31.004318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"67'669","reported_seq":3425,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.621226+0000","last_change":"2023-10-16T20:47:31.593007+0000","last_active":"2023-10-16T20:58:22.621226+0000","last_peered":"2023-10-16T20:58:22.621226+0000","last_clean":"2023-10-16T20:58:22.621226+0000","last_became_active":"2023-10-16T20:36:59.468082+0000","last_became_peered":"2023-10-16T20:36:59.468082+0000","last_unstale":"2023-10-16T20:58:22.621226+0000","last_undegraded":"2023-10-16T20:58:22.621226+0000","last_fullsized":"2023-10-16T20:58:22.621226+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":669,"log_dups_size":0,"ondisk_log_size":669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:49:06.585943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013154572,"stat_sum":{"num_bytes":30910053,"num_objects":362,"num_object_clones":0,"num_object_copies":1448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":649,"num_read_kb":15567,"num_write":666,"num_write_kb":42817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'110","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.884585+0000","last_change":"2023-10-16T20:47:31.488408+0000","last_active":"2023-10-16T20:58:37.884585+0000","last_peered":"2023-10-16T20:58:37.884585+0000","last_clean":"2023-10-16T20:58:37.884585+0000","last_became_active":"2023-10-16T20:37:37.409004+0000","last_became_peered":"2023-10-16T20:37:37.409004+0000","last_unstale":"2023-10-16T20:58:37.884585+0000","last_undegraded":"2023-10-16T20:58:37.884585+0000","last_fullsized":"2023-10-16T20:58:37.884585+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:21:07.245091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060805590000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"66'68","reported_seq":138,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:19.377575+0000","last_change":"2023-10-16T20:36:54.746600+0000","last_active":"2023-10-16T20:57:19.377575+0000","last_peered":"2023-10-16T20:57:19.377575+0000","last_clean":"2023-10-16T20:57:19.377575+0000","last_became_active":"2023-10-16T20:36:54.746247+0000","last_became_peered":"2023-10-16T20:36:54.746247+0000","last_unstale":"2023-10-16T20:57:19.377575+0000","last_undegraded":"2023-10-16T20:57:19.377575+0000","last_fullsized":"2023-10-16T20:57:19.377575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:28:57.949487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":7042,"num_write":68,"num_write_kb":4224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"67'680","reported_seq":3679,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.607038+0000","last_change":"2023-10-16T20:47:31.590576+0000","last_active":"2023-10-16T20:58:22.607038+0000","last_peered":"2023-10-16T20:58:22.607038+0000","last_clean":"2023-10-16T20:58:22.607038+0000","last_became_active":"2023-10-16T20:37:37.467402+0000","last_became_peered":"2023-10-16T20:37:37.467402+0000","last_unstale":"2023-10-16T20:58:22.607038+0000","last_undegraded":"2023-10-16T20:58:22.607038+0000","last_fullsized":"2023-10-16T20:58:22.607038+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":680,"log_dups_size":0,"ondisk_log_size":680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:32:26.907296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012017773000000001,"stat_sum":{"num_bytes":36935626,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":884,"num_read_kb":21394,"num_write":679,"num_write_kb":40520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"67'106","reported_seq":2215,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.895855+0000","last_change":"2023-10-16T20:47:31.589500+0000","last_active":"2023-10-16T20:58:37.895855+0000","last_peered":"2023-10-16T20:58:37.895855+0000","last_clean":"2023-10-16T20:58:37.895855+0000","last_became_active":"2023-10-16T20:36:55.300958+0000","last_became_peered":"2023-10-16T20:36:55.300958+0000","last_unstale":"2023-10-16T20:58:37.895855+0000","last_undegraded":"2023-10-16T20:58:37.895855+0000","last_fullsized":"2023-10-16T20:58:37.895855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:29:17.323936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009734431,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"67'314","reported_seq":386,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.865738+0000","last_change":"2023-10-16T20:36:54.753770+0000","last_active":"2023-10-16T20:58:37.865738+0000","last_peered":"2023-10-16T20:58:37.865738+0000","last_clean":"2023-10-16T20:58:37.865738+0000","last_became_active":"2023-10-16T20:36:54.753443+0000","last_became_peered":"2023-10-16T20:36:54.753443+0000","last_unstale":"2023-10-16T20:58:37.865738+0000","last_undegraded":"2023-10-16T20:58:37.865738+0000","last_fullsized":"2023-10-16T20:58:37.865738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:21:26.392480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9664034,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":15859,"num_write":307,"num_write_kb":9612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'687","reported_seq":3976,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.741627+0000","last_change":"2023-10-16T20:47:31.492598+0000","last_active":"2023-10-16T20:58:22.741627+0000","last_peered":"2023-10-16T20:58:22.741627+0000","last_clean":"2023-10-16T20:58:22.741627+0000","last_became_active":"2023-10-16T20:36:59.302869+0000","last_became_peered":"2023-10-16T20:36:59.302869+0000","last_unstale":"2023-10-16T20:58:22.741627+0000","last_undegraded":"2023-10-16T20:58:22.741627+0000","last_fullsized":"2023-10-16T20:58:22.741627+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:37:25.463785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012136565,"stat_sum":{"num_bytes":23637013,"num_objects":352,"num_object_clones":0,"num_object_copies":1408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":1183,"num_read_kb":18772,"num_write":687,"num_write_kb":27489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"67'100","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.892995+0000","last_change":"2023-10-16T20:47:31.587528+0000","last_active":"2023-10-16T20:58:37.892995+0000","last_peered":"2023-10-16T20:58:37.892995+0000","last_clean":"2023-10-16T20:58:37.892995+0000","last_became_active":"2023-10-16T20:36:55.291633+0000","last_became_peered":"2023-10-16T20:36:55.291633+0000","last_unstale":"2023-10-16T20:58:37.892995+0000","last_undegraded":"2023-10-16T20:58:37.892995+0000","last_fullsized":"2023-10-16T20:58:37.892995+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:08.345584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085978630000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'120","reported_seq":195,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.767053+0000","last_change":"2023-10-16T20:36:54.748196+0000","last_active":"2023-10-16T20:58:23.767053+0000","last_peered":"2023-10-16T20:58:23.767053+0000","last_clean":"2023-10-16T20:58:23.767053+0000","last_became_active":"2023-10-16T20:36:54.747704+0000","last_became_peered":"2023-10-16T20:36:54.747704+0000","last_unstale":"2023-10-16T20:58:23.767053+0000","last_undegraded":"2023-10-16T20:58:23.767053+0000","last_fullsized":"2023-10-16T20:58:23.767053+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:55.471916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":116,"num_write_kb":4228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"67'663","reported_seq":3386,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.098033+0000","last_change":"2023-10-16T20:47:31.595065+0000","last_active":"2023-10-16T20:58:23.098033+0000","last_peered":"2023-10-16T20:58:23.098033+0000","last_clean":"2023-10-16T20:58:23.098033+0000","last_became_active":"2023-10-16T20:36:59.467291+0000","last_became_peered":"2023-10-16T20:36:59.467291+0000","last_unstale":"2023-10-16T20:58:23.098033+0000","last_undegraded":"2023-10-16T20:58:23.098033+0000","last_fullsized":"2023-10-16T20:58:23.098033+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:15.723182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0131915,"stat_sum":{"num_bytes":23136394,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":616,"num_read_kb":10065,"num_write":660,"num_write_kb":35181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"67'80","reported_seq":2189,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.876288+0000","last_change":"2023-10-16T20:47:31.589747+0000","last_active":"2023-10-16T20:58:37.876288+0000","last_peered":"2023-10-16T20:58:37.876288+0000","last_clean":"2023-10-16T20:58:37.876288+0000","last_became_active":"2023-10-16T20:36:55.290998+0000","last_became_peered":"2023-10-16T20:36:55.290998+0000","last_unstale":"2023-10-16T20:58:37.876288+0000","last_undegraded":"2023-10-16T20:58:37.876288+0000","last_fullsized":"2023-10-16T20:58:37.876288+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:52:52.835051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082191850000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'91","reported_seq":159,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:33.484841+0000","last_change":"2023-10-16T20:36:54.763818+0000","last_active":"2023-10-16T20:58:33.484841+0000","last_peered":"2023-10-16T20:58:33.484841+0000","last_clean":"2023-10-16T20:58:33.484841+0000","last_became_active":"2023-10-16T20:36:54.761337+0000","last_became_peered":"2023-10-16T20:36:54.761337+0000","last_unstale":"2023-10-16T20:58:33.484841+0000","last_undegraded":"2023-10-16T20:58:33.484841+0000","last_fullsized":"2023-10-16T20:58:33.484841+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:03.093189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":5,"num_write":137,"num_write_kb":4352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"67'666","reported_seq":3840,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.018714+0000","last_change":"2023-10-16T20:47:31.487063+0000","last_active":"2023-10-16T20:58:23.018714+0000","last_peered":"2023-10-16T20:58:23.018714+0000","last_clean":"2023-10-16T20:58:23.018714+0000","last_became_active":"2023-10-16T20:37:37.468309+0000","last_became_peered":"2023-10-16T20:37:37.468309+0000","last_unstale":"2023-10-16T20:58:23.018714+0000","last_undegraded":"2023-10-16T20:58:23.018714+0000","last_fullsized":"2023-10-16T20:58:23.018714+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:13.271803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058030460000000001,"stat_sum":{"num_bytes":20758781,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":1058,"num_read_kb":21621,"num_write":667,"num_write_kb":32853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"67'107","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.887159+0000","last_change":"2023-10-16T20:47:31.589398+0000","last_active":"2023-10-16T20:58:37.887159+0000","last_peered":"2023-10-16T20:58:37.887159+0000","last_clean":"2023-10-16T20:58:37.887159+0000","last_became_active":"2023-10-16T20:36:55.288354+0000","last_became_peered":"2023-10-16T20:36:55.288354+0000","last_unstale":"2023-10-16T20:58:37.887159+0000","last_undegraded":"2023-10-16T20:58:37.887159+0000","last_fullsized":"2023-10-16T20:58:37.887159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:09.359239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076786709999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"67'333","reported_seq":413,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.977642+0000","last_change":"2023-10-16T20:36:54.738216+0000","last_active":"2023-10-16T20:58:05.977642+0000","last_peered":"2023-10-16T20:58:05.977642+0000","last_clean":"2023-10-16T20:58:05.977642+0000","last_became_active":"2023-10-16T20:36:54.737876+0000","last_became_peered":"2023-10-16T20:36:54.737876+0000","last_unstale":"2023-10-16T20:58:05.977642+0000","last_undegraded":"2023-10-16T20:58:05.977642+0000","last_fullsized":"2023-10-16T20:58:05.977642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:28.214601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":39,"num_read_kb":4134,"num_write":333,"num_write_kb":8414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"67'666","reported_seq":3665,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.692242+0000","last_change":"2023-10-16T20:47:31.490637+0000","last_active":"2023-10-16T20:58:22.692242+0000","last_peered":"2023-10-16T20:58:22.692242+0000","last_clean":"2023-10-16T20:58:22.692242+0000","last_became_active":"2023-10-16T20:36:59.306801+0000","last_became_peered":"2023-10-16T20:36:59.306801+0000","last_unstale":"2023-10-16T20:58:22.692242+0000","last_undegraded":"2023-10-16T20:58:22.692242+0000","last_fullsized":"2023-10-16T20:58:22.692242+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:43:08.907279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010585667,"stat_sum":{"num_bytes":20444029,"num_objects":301,"num_object_clones":0,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":893,"num_read_kb":19660,"num_write":666,"num_write_kb":28474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"67'93","reported_seq":2202,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.873494+0000","last_change":"2023-10-16T20:47:31.590902+0000","last_active":"2023-10-16T20:58:37.873494+0000","last_peered":"2023-10-16T20:58:37.873494+0000","last_clean":"2023-10-16T20:58:37.873494+0000","last_became_active":"2023-10-16T20:36:55.287066+0000","last_became_peered":"2023-10-16T20:36:55.287066+0000","last_unstale":"2023-10-16T20:58:37.873494+0000","last_undegraded":"2023-10-16T20:58:37.873494+0000","last_fullsized":"2023-10-16T20:58:37.873494+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:36.165939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012121162,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"66'101","reported_seq":172,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:57.653854+0000","last_change":"2023-10-16T20:36:54.764693+0000","last_active":"2023-10-16T20:57:57.653854+0000","last_peered":"2023-10-16T20:57:57.653854+0000","last_clean":"2023-10-16T20:57:57.653854+0000","last_became_active":"2023-10-16T20:36:54.764417+0000","last_became_peered":"2023-10-16T20:36:54.764417+0000","last_unstale":"2023-10-16T20:57:57.653854+0000","last_undegraded":"2023-10-16T20:57:57.653854+0000","last_fullsized":"2023-10-16T20:57:57.653854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:56.653149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":4110,"num_write":99,"num_write_kb":4200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'634","reported_seq":3754,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:20.202595+0000","last_change":"2023-10-16T20:47:31.584541+0000","last_active":"2023-10-16T20:58:20.202595+0000","last_peered":"2023-10-16T20:58:20.202595+0000","last_clean":"2023-10-16T20:58:20.202595+0000","last_became_active":"2023-10-16T20:36:59.464626+0000","last_became_peered":"2023-10-16T20:36:59.464626+0000","last_unstale":"2023-10-16T20:58:20.202595+0000","last_undegraded":"2023-10-16T20:58:20.202595+0000","last_fullsized":"2023-10-16T20:58:20.202595+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":634,"log_dups_size":0,"ondisk_log_size":634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:18:35.242031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069938420000000001,"stat_sum":{"num_bytes":28405361,"num_objects":321,"num_object_clones":0,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":1015,"num_read_kb":24481,"num_write":633,"num_write_kb":44456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"67'113","reported_seq":2222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.886956+0000","last_change":"2023-10-16T20:47:31.589722+0000","last_active":"2023-10-16T20:58:37.886956+0000","last_peered":"2023-10-16T20:58:37.886956+0000","last_clean":"2023-10-16T20:58:37.886956+0000","last_became_active":"2023-10-16T20:36:55.291396+0000","last_became_peered":"2023-10-16T20:36:55.291396+0000","last_unstale":"2023-10-16T20:58:37.886956+0000","last_undegraded":"2023-10-16T20:58:37.886956+0000","last_fullsized":"2023-10-16T20:58:37.886956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:09:26.019798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011464676,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"67'184","reported_seq":258,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.657888+0000","last_change":"2023-10-16T20:36:54.750814+0000","last_active":"2023-10-16T20:58:05.657888+0000","last_peered":"2023-10-16T20:58:05.657888+0000","last_clean":"2023-10-16T20:58:05.657888+0000","last_became_active":"2023-10-16T20:36:54.750448+0000","last_became_peered":"2023-10-16T20:36:54.750448+0000","last_unstale":"2023-10-16T20:58:05.657888+0000","last_undegraded":"2023-10-16T20:58:05.657888+0000","last_fullsized":"2023-10-16T20:58:05.657888+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:41.981408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":3911,"num_write":180,"num_write_kb":4223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"67'608","reported_seq":3330,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.013307+0000","last_change":"2023-10-16T20:47:31.583645+0000","last_active":"2023-10-16T20:58:23.013307+0000","last_peered":"2023-10-16T20:58:23.013307+0000","last_clean":"2023-10-16T20:58:23.013307+0000","last_became_active":"2023-10-16T20:36:59.464578+0000","last_became_peered":"2023-10-16T20:36:59.464578+0000","last_unstale":"2023-10-16T20:58:23.013307+0000","last_undegraded":"2023-10-16T20:58:23.013307+0000","last_fullsized":"2023-10-16T20:58:23.013307+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:56.818326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064826880000000003,"stat_sum":{"num_bytes":21354704,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":616,"num_read_kb":9691,"num_write":609,"num_write_kb":35926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"67'101","reported_seq":2210,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889428+0000","last_change":"2023-10-16T20:47:31.490514+0000","last_active":"2023-10-16T20:58:37.889428+0000","last_peered":"2023-10-16T20:58:37.889428+0000","last_clean":"2023-10-16T20:58:37.889428+0000","last_became_active":"2023-10-16T20:36:55.285407+0000","last_became_peered":"2023-10-16T20:36:55.285407+0000","last_unstale":"2023-10-16T20:58:37.889428+0000","last_undegraded":"2023-10-16T20:58:37.889428+0000","last_fullsized":"2023-10-16T20:58:37.889428+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:31:00.135226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010336254,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"66'459","reported_seq":604,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.695585+0000","last_change":"2023-10-16T20:36:54.757012+0000","last_active":"2023-10-16T20:58:28.695585+0000","last_peered":"2023-10-16T20:58:28.695585+0000","last_clean":"2023-10-16T20:58:28.695585+0000","last_became_active":"2023-10-16T20:36:54.756697+0000","last_became_peered":"2023-10-16T20:36:54.756697+0000","last_unstale":"2023-10-16T20:58:28.695585+0000","last_undegraded":"2023-10-16T20:58:28.695585+0000","last_fullsized":"2023-10-16T20:58:28.695585+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":459,"log_dups_size":0,"ondisk_log_size":459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:39:20.259580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8464867,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":106,"num_read_kb":16561,"num_write":461,"num_write_kb":8611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'632","reported_seq":4046,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.561302+0000","last_change":"2023-10-16T20:47:31.589709+0000","last_active":"2023-10-16T20:58:22.561302+0000","last_peered":"2023-10-16T20:58:22.561302+0000","last_clean":"2023-10-16T20:58:22.561302+0000","last_became_active":"2023-10-16T20:36:59.468254+0000","last_became_peered":"2023-10-16T20:36:59.468254+0000","last_unstale":"2023-10-16T20:58:22.561302+0000","last_undegraded":"2023-10-16T20:58:22.561302+0000","last_fullsized":"2023-10-16T20:58:22.561302+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:09:21.589899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093892239999999998,"stat_sum":{"num_bytes":28874401,"num_objects":300,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":1309,"num_read_kb":15538,"num_write":630,"num_write_kb":28525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"67'100","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.880164+0000","last_change":"2023-10-16T20:47:31.588140+0000","last_active":"2023-10-16T20:58:37.880164+0000","last_peered":"2023-10-16T20:58:37.880164+0000","last_clean":"2023-10-16T20:58:37.880164+0000","last_became_active":"2023-10-16T20:36:55.288488+0000","last_became_peered":"2023-10-16T20:36:55.288488+0000","last_unstale":"2023-10-16T20:58:37.880164+0000","last_undegraded":"2023-10-16T20:58:37.880164+0000","last_fullsized":"2023-10-16T20:58:37.880164+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:40:33.001741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078405860000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"67'139","reported_seq":223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.709051+0000","last_change":"2023-10-16T20:36:54.765361+0000","last_active":"2023-10-16T20:58:28.709051+0000","last_peered":"2023-10-16T20:58:28.709051+0000","last_clean":"2023-10-16T20:58:28.709051+0000","last_became_active":"2023-10-16T20:36:54.764969+0000","last_became_peered":"2023-10-16T20:36:54.764969+0000","last_unstale":"2023-10-16T20:58:28.709051+0000","last_undegraded":"2023-10-16T20:58:28.709051+0000","last_fullsized":"2023-10-16T20:58:28.709051+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:21:21.183612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389280,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":45,"num_read_kb":8235,"num_write":134,"num_write_kb":8274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"67'666","reported_seq":4058,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.657743+0000","last_change":"2023-10-16T20:47:31.488335+0000","last_active":"2023-10-16T20:58:22.657743+0000","last_peered":"2023-10-16T20:58:22.657743+0000","last_clean":"2023-10-16T20:58:22.657743+0000","last_became_active":"2023-10-16T20:36:59.302698+0000","last_became_peered":"2023-10-16T20:36:59.302698+0000","last_unstale":"2023-10-16T20:58:22.657743+0000","last_undegraded":"2023-10-16T20:58:22.657743+0000","last_fullsized":"2023-10-16T20:58:22.657743+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:09:15.954887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064797020000000004,"stat_sum":{"num_bytes":31392251,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":1286,"num_read_kb":27897,"num_write":665,"num_write_kb":43250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'99","reported_seq":2208,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.882106+0000","last_change":"2023-10-16T20:47:31.487882+0000","last_active":"2023-10-16T20:58:37.882106+0000","last_peered":"2023-10-16T20:58:37.882106+0000","last_clean":"2023-10-16T20:58:37.882106+0000","last_became_active":"2023-10-16T20:36:55.290127+0000","last_became_peered":"2023-10-16T20:36:55.290127+0000","last_unstale":"2023-10-16T20:58:37.882106+0000","last_undegraded":"2023-10-16T20:58:37.882106+0000","last_fullsized":"2023-10-16T20:58:37.882106+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:23.063288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055494209999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"67'78","reported_seq":150,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:24.130464+0000","last_change":"2023-10-16T20:36:54.749120+0000","last_active":"2023-10-16T20:58:24.130464+0000","last_peered":"2023-10-16T20:58:24.130464+0000","last_clean":"2023-10-16T20:58:24.130464+0000","last_became_active":"2023-10-16T20:36:54.748788+0000","last_became_peered":"2023-10-16T20:36:54.748788+0000","last_unstale":"2023-10-16T20:58:24.130464+0000","last_undegraded":"2023-10-16T20:58:24.130464+0000","last_fullsized":"2023-10-16T20:58:24.130464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:11:07.257343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":15,"num_read_kb":23,"num_write":83,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"67'606","reported_seq":3860,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.746395+0000","last_change":"2023-10-16T20:47:31.486033+0000","last_active":"2023-10-16T20:58:22.746395+0000","last_peered":"2023-10-16T20:58:22.746395+0000","last_clean":"2023-10-16T20:58:22.746395+0000","last_became_active":"2023-10-16T20:36:59.302916+0000","last_became_peered":"2023-10-16T20:36:59.302916+0000","last_unstale":"2023-10-16T20:58:22.746395+0000","last_undegraded":"2023-10-16T20:58:22.746395+0000","last_fullsized":"2023-10-16T20:58:22.746395+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:09.867705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041353309999999999,"stat_sum":{"num_bytes":29291517,"num_objects":300,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":1148,"num_read_kb":24111,"num_write":607,"num_write_kb":41174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"67'107","reported_seq":2215,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889740+0000","last_change":"2023-10-16T20:47:31.490807+0000","last_active":"2023-10-16T20:58:37.889740+0000","last_peered":"2023-10-16T20:58:37.889740+0000","last_clean":"2023-10-16T20:58:37.889740+0000","last_became_active":"2023-10-16T20:36:55.291448+0000","last_became_peered":"2023-10-16T20:36:55.291448+0000","last_unstale":"2023-10-16T20:58:37.889740+0000","last_undegraded":"2023-10-16T20:58:37.889740+0000","last_fullsized":"2023-10-16T20:58:37.889740+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:03:32.439694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010585074,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"67'134","reported_seq":254,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:39.515133+0000","last_change":"2023-10-16T20:36:54.761824+0000","last_active":"2023-10-16T20:58:39.515133+0000","last_peered":"2023-10-16T20:58:39.515133+0000","last_clean":"2023-10-16T20:58:39.515133+0000","last_became_active":"2023-10-16T20:36:54.761541+0000","last_became_peered":"2023-10-16T20:36:54.761541+0000","last_unstale":"2023-10-16T20:58:39.515133+0000","last_undegraded":"2023-10-16T20:58:39.515133+0000","last_fullsized":"2023-10-16T20:58:39.515133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:34.512207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2461990,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":69,"num_read_kb":129,"num_write":135,"num_write_kb":2563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"67'663","reported_seq":3325,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.270542+0000","last_change":"2023-10-16T20:47:31.585744+0000","last_active":"2023-10-16T20:58:22.270542+0000","last_peered":"2023-10-16T20:58:22.270542+0000","last_clean":"2023-10-16T20:58:22.270542+0000","last_became_active":"2023-10-16T20:36:59.303773+0000","last_became_peered":"2023-10-16T20:36:59.303773+0000","last_unstale":"2023-10-16T20:58:22.270542+0000","last_undegraded":"2023-10-16T20:58:22.270542+0000","last_fullsized":"2023-10-16T20:58:22.270542+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:19:00.801685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007941142,"stat_sum":{"num_bytes":21664408,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":557,"num_read_kb":9585,"num_write":658,"num_write_kb":32916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"67'121","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.891891+0000","last_change":"2023-10-16T20:47:31.594424+0000","last_active":"2023-10-16T20:58:37.891891+0000","last_peered":"2023-10-16T20:58:37.891891+0000","last_clean":"2023-10-16T20:58:37.891891+0000","last_became_active":"2023-10-16T20:36:55.300535+0000","last_became_peered":"2023-10-16T20:36:55.300535+0000","last_unstale":"2023-10-16T20:58:37.891891+0000","last_undegraded":"2023-10-16T20:58:37.891891+0000","last_fullsized":"2023-10-16T20:58:37.891891+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:47:03.132953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013755641000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"67'55","reported_seq":124,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:30.004648+0000","last_change":"2023-10-16T20:36:54.758140+0000","last_active":"2023-10-16T20:58:30.004648+0000","last_peered":"2023-10-16T20:58:30.004648+0000","last_clean":"2023-10-16T20:58:30.004648+0000","last_became_active":"2023-10-16T20:36:54.757303+0000","last_became_peered":"2023-10-16T20:36:54.757303+0000","last_unstale":"2023-10-16T20:58:30.004648+0000","last_undegraded":"2023-10-16T20:58:30.004648+0000","last_fullsized":"2023-10-16T20:58:30.004648+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:45.944191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":56,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"67'687","reported_seq":4217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.643179+0000","last_change":"2023-10-16T20:47:31.586327+0000","last_active":"2023-10-16T20:58:22.643179+0000","last_peered":"2023-10-16T20:58:22.643179+0000","last_clean":"2023-10-16T20:58:22.643179+0000","last_became_active":"2023-10-16T20:36:59.465844+0000","last_became_peered":"2023-10-16T20:36:59.465844+0000","last_unstale":"2023-10-16T20:58:22.643179+0000","last_undegraded":"2023-10-16T20:58:22.643179+0000","last_fullsized":"2023-10-16T20:58:22.643179+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:54.762444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081168009999999999,"stat_sum":{"num_bytes":45881583,"num_objects":353,"num_object_clones":0,"num_object_copies":1412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":353,"num_whiteouts":0,"num_read":1424,"num_read_kb":44764,"num_write":688,"num_write_kb":65580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"67'661","reported_seq":4047,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.525871+0000","last_change":"2023-10-16T20:47:31.489178+0000","last_active":"2023-10-16T20:58:22.525871+0000","last_peered":"2023-10-16T20:58:22.525871+0000","last_clean":"2023-10-16T20:58:22.525871+0000","last_became_active":"2023-10-16T20:36:59.467356+0000","last_became_peered":"2023-10-16T20:36:59.467356+0000","last_unstale":"2023-10-16T20:58:22.525871+0000","last_undegraded":"2023-10-16T20:58:22.525871+0000","last_fullsized":"2023-10-16T20:58:22.525871+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:10:50.487421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093722700000000003,"stat_sum":{"num_bytes":37038710,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1279,"num_read_kb":28276,"num_write":659,"num_write_kb":40616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'108","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.864220+0000","last_change":"2023-10-16T20:47:31.484094+0000","last_active":"2023-10-16T20:58:37.864220+0000","last_peered":"2023-10-16T20:58:37.864220+0000","last_clean":"2023-10-16T20:58:37.864220+0000","last_became_active":"2023-10-16T20:36:55.280903+0000","last_became_peered":"2023-10-16T20:36:55.280903+0000","last_unstale":"2023-10-16T20:58:37.864220+0000","last_undegraded":"2023-10-16T20:58:37.864220+0000","last_fullsized":"2023-10-16T20:58:37.864220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:50:46.153222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029269690000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"67'457","reported_seq":528,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.761682+0000","last_change":"2023-10-16T20:36:54.757589+0000","last_active":"2023-10-16T20:58:23.761682+0000","last_peered":"2023-10-16T20:58:23.761682+0000","last_clean":"2023-10-16T20:58:23.761682+0000","last_became_active":"2023-10-16T20:36:54.756775+0000","last_became_peered":"2023-10-16T20:36:54.756775+0000","last_unstale":"2023-10-16T20:58:23.761682+0000","last_undegraded":"2023-10-16T20:58:23.761682+0000","last_fullsized":"2023-10-16T20:58:23.761682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:40.629540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583494,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":16386,"num_write":448,"num_write_kb":12544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"67'638","reported_seq":3559,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.378852+0000","last_change":"2023-10-16T20:47:31.593596+0000","last_active":"2023-10-16T20:58:22.378852+0000","last_peered":"2023-10-16T20:58:22.378852+0000","last_clean":"2023-10-16T20:58:22.378852+0000","last_became_active":"2023-10-16T20:37:37.438140+0000","last_became_peered":"2023-10-16T20:37:37.438140+0000","last_unstale":"2023-10-16T20:58:22.378852+0000","last_undegraded":"2023-10-16T20:58:22.378852+0000","last_fullsized":"2023-10-16T20:58:22.378852+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:41:29.134511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012287041,"stat_sum":{"num_bytes":24792014,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":805,"num_read_kb":12206,"num_write":635,"num_write_kb":32676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"67'121","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.897843+0000","last_change":"2023-10-16T20:47:31.585618+0000","last_active":"2023-10-16T20:58:37.897843+0000","last_peered":"2023-10-16T20:58:37.897843+0000","last_clean":"2023-10-16T20:58:37.897843+0000","last_became_active":"2023-10-16T20:36:55.273959+0000","last_became_peered":"2023-10-16T20:36:55.273959+0000","last_unstale":"2023-10-16T20:58:37.897843+0000","last_undegraded":"2023-10-16T20:58:37.897843+0000","last_fullsized":"2023-10-16T20:58:37.897843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:05:00.921391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076333039999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"67'135","reported_seq":276,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.678551+0000","last_change":"2023-10-16T20:36:54.759513+0000","last_active":"2023-10-16T20:58:28.678551+0000","last_peered":"2023-10-16T20:58:28.678551+0000","last_clean":"2023-10-16T20:58:28.678551+0000","last_became_active":"2023-10-16T20:36:54.759059+0000","last_became_peered":"2023-10-16T20:36:54.759059+0000","last_unstale":"2023-10-16T20:58:28.678551+0000","last_undegraded":"2023-10-16T20:58:28.678551+0000","last_fullsized":"2023-10-16T20:58:28.678551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:36.696053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4348959,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":88,"num_read_kb":239,"num_write":133,"num_write_kb":4341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"66'223","reported_seq":372,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:33.632797+0000","last_change":"2023-10-16T20:36:54.764780+0000","last_active":"2023-10-16T20:57:33.632797+0000","last_peered":"2023-10-16T20:57:33.632797+0000","last_clean":"2023-10-16T20:57:33.632797+0000","last_became_active":"2023-10-16T20:36:54.764436+0000","last_became_peered":"2023-10-16T20:36:54.764436+0000","last_unstale":"2023-10-16T20:57:33.632797+0000","last_undegraded":"2023-10-16T20:57:33.632797+0000","last_fullsized":"2023-10-16T20:57:33.632797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:55:31.816599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8524313,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":105,"num_read_kb":8412,"num_write":220,"num_write_kb":8480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"67'109","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888733+0000","last_change":"2023-10-16T20:47:31.492923+0000","last_active":"2023-10-16T20:58:37.888733+0000","last_peered":"2023-10-16T20:58:37.888733+0000","last_clean":"2023-10-16T20:58:37.888733+0000","last_became_active":"2023-10-16T20:36:55.277873+0000","last_became_peered":"2023-10-16T20:36:55.277873+0000","last_unstale":"2023-10-16T20:58:37.888733+0000","last_undegraded":"2023-10-16T20:58:37.888733+0000","last_fullsized":"2023-10-16T20:58:37.888733+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:10:24.211693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012247773,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'608","reported_seq":3232,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.002422+0000","last_change":"2023-10-16T20:47:31.496724+0000","last_active":"2023-10-16T20:58:23.002422+0000","last_peered":"2023-10-16T20:58:23.002422+0000","last_clean":"2023-10-16T20:58:23.002422+0000","last_became_active":"2023-10-16T20:36:59.302435+0000","last_became_peered":"2023-10-16T20:36:59.302435+0000","last_unstale":"2023-10-16T20:58:23.002422+0000","last_undegraded":"2023-10-16T20:58:23.002422+0000","last_fullsized":"2023-10-16T20:58:23.002422+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:34:39.329693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001061892,"stat_sum":{"num_bytes":16477397,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":519,"num_read_kb":10696,"num_write":606,"num_write_kb":24605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"66'232","reported_seq":301,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:33.636417+0000","last_change":"2023-10-16T20:36:54.756190+0000","last_active":"2023-10-16T20:57:33.636417+0000","last_peered":"2023-10-16T20:57:33.636417+0000","last_clean":"2023-10-16T20:57:33.636417+0000","last_became_active":"2023-10-16T20:36:54.755834+0000","last_became_peered":"2023-10-16T20:36:54.755834+0000","last_unstale":"2023-10-16T20:57:33.636417+0000","last_undegraded":"2023-10-16T20:57:33.636417+0000","last_fullsized":"2023-10-16T20:57:33.636417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:31.012441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582946,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":16385,"num_write":235,"num_write_kb":12609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"67'104","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.885029+0000","last_change":"2023-10-16T20:47:31.486761+0000","last_active":"2023-10-16T20:58:37.885029+0000","last_peered":"2023-10-16T20:58:37.885029+0000","last_clean":"2023-10-16T20:58:37.885029+0000","last_became_active":"2023-10-16T20:36:55.279333+0000","last_became_peered":"2023-10-16T20:36:55.279333+0000","last_unstale":"2023-10-16T20:58:37.885029+0000","last_undegraded":"2023-10-16T20:58:37.885029+0000","last_fullsized":"2023-10-16T20:58:37.885029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:30:26.879792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047896609999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'629","reported_seq":3442,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.721224+0000","last_change":"2023-10-16T20:47:31.586162+0000","last_active":"2023-10-16T20:58:22.721224+0000","last_peered":"2023-10-16T20:58:22.721224+0000","last_clean":"2023-10-16T20:58:22.721224+0000","last_became_active":"2023-10-16T20:36:59.301864+0000","last_became_peered":"2023-10-16T20:36:59.301864+0000","last_unstale":"2023-10-16T20:58:22.721224+0000","last_undegraded":"2023-10-16T20:58:22.721224+0000","last_fullsized":"2023-10-16T20:58:22.721224+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:44.207453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080450769999999994,"stat_sum":{"num_bytes":20568472,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":708,"num_read_kb":22370,"num_write":628,"num_write_kb":40892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"67'120","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.875503+0000","last_change":"2023-10-16T20:47:31.590065+0000","last_active":"2023-10-16T20:58:37.875503+0000","last_peered":"2023-10-16T20:58:37.875503+0000","last_clean":"2023-10-16T20:58:37.875503+0000","last_became_active":"2023-10-16T20:36:55.287442+0000","last_became_peered":"2023-10-16T20:36:55.287442+0000","last_unstale":"2023-10-16T20:58:37.875503+0000","last_undegraded":"2023-10-16T20:58:37.875503+0000","last_fullsized":"2023-10-16T20:58:37.875503+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:37:33.639292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011630351000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"67'131","reported_seq":2241,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.880621+0000","last_change":"2023-10-16T20:47:31.497003+0000","last_active":"2023-10-16T20:58:37.880621+0000","last_peered":"2023-10-16T20:58:37.880621+0000","last_clean":"2023-10-16T20:58:37.880621+0000","last_became_active":"2023-10-16T20:36:55.276276+0000","last_became_peered":"2023-10-16T20:36:55.276276+0000","last_unstale":"2023-10-16T20:58:37.880621+0000","last_undegraded":"2023-10-16T20:58:37.880621+0000","last_fullsized":"2023-10-16T20:58:37.880621+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:35:00.858846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091802500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"67'115","reported_seq":2232,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.886101+0000","last_change":"2023-10-16T20:47:31.592299+0000","last_active":"2023-10-16T20:58:37.886101+0000","last_peered":"2023-10-16T20:58:37.886101+0000","last_clean":"2023-10-16T20:58:37.886101+0000","last_became_active":"2023-10-16T20:37:37.413040+0000","last_became_peered":"2023-10-16T20:37:37.413040+0000","last_unstale":"2023-10-16T20:58:37.886101+0000","last_undegraded":"2023-10-16T20:58:37.886101+0000","last_fullsized":"2023-10-16T20:58:37.886101+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:50:46.069439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012618513,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"67'116","reported_seq":2226,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888796+0000","last_change":"2023-10-16T20:47:31.492800+0000","last_active":"2023-10-16T20:58:37.888796+0000","last_peered":"2023-10-16T20:58:37.888796+0000","last_clean":"2023-10-16T20:58:37.888796+0000","last_became_active":"2023-10-16T20:36:55.275349+0000","last_became_peered":"2023-10-16T20:36:55.275349+0000","last_unstale":"2023-10-16T20:58:37.888796+0000","last_undegraded":"2023-10-16T20:58:37.888796+0000","last_fullsized":"2023-10-16T20:58:37.888796+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:28:00.161390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012136073000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"67'88","reported_seq":2198,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.852714+0000","last_change":"2023-10-16T20:47:31.587774+0000","last_active":"2023-10-16T20:58:37.852714+0000","last_peered":"2023-10-16T20:58:37.852714+0000","last_clean":"2023-10-16T20:58:37.852714+0000","last_became_active":"2023-10-16T20:36:55.277043+0000","last_became_peered":"2023-10-16T20:36:55.277043+0000","last_unstale":"2023-10-16T20:58:37.852714+0000","last_undegraded":"2023-10-16T20:58:37.852714+0000","last_fullsized":"2023-10-16T20:58:37.852714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:32.439680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078065369999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"67'89","reported_seq":2199,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.863001+0000","last_change":"2023-10-16T20:47:31.489466+0000","last_active":"2023-10-16T20:58:37.863001+0000","last_peered":"2023-10-16T20:58:37.863001+0000","last_clean":"2023-10-16T20:58:37.863001+0000","last_became_active":"2023-10-16T20:36:55.278715+0000","last_became_peered":"2023-10-16T20:36:55.278715+0000","last_unstale":"2023-10-16T20:58:37.863001+0000","last_undegraded":"2023-10-16T20:58:37.863001+0000","last_fullsized":"2023-10-16T20:58:37.863001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:49:22.643504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067460719999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"67'124","reported_seq":2234,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.836534+0000","last_change":"2023-10-16T20:47:31.594095+0000","last_active":"2023-10-16T20:58:37.836534+0000","last_peered":"2023-10-16T20:58:37.836534+0000","last_clean":"2023-10-16T20:58:37.836534+0000","last_became_active":"2023-10-16T20:36:55.301604+0000","last_became_peered":"2023-10-16T20:36:55.301604+0000","last_unstale":"2023-10-16T20:58:37.836534+0000","last_undegraded":"2023-10-16T20:58:37.836534+0000","last_fullsized":"2023-10-16T20:58:37.836534+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:10.110081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012969489000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"67'99","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.858692+0000","last_change":"2023-10-16T20:47:31.587306+0000","last_active":"2023-10-16T20:58:37.858692+0000","last_peered":"2023-10-16T20:58:37.858692+0000","last_clean":"2023-10-16T20:58:37.858692+0000","last_became_active":"2023-10-16T20:36:55.276984+0000","last_became_peered":"2023-10-16T20:36:55.276984+0000","last_unstale":"2023-10-16T20:58:37.858692+0000","last_undegraded":"2023-10-16T20:58:37.858692+0000","last_fullsized":"2023-10-16T20:58:37.858692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:42:52.651942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086623480000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'111","reported_seq":2221,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.896916+0000","last_change":"2023-10-16T20:47:31.595489+0000","last_active":"2023-10-16T20:58:37.896916+0000","last_peered":"2023-10-16T20:58:37.896916+0000","last_clean":"2023-10-16T20:58:37.896916+0000","last_became_active":"2023-10-16T20:36:55.290023+0000","last_became_peered":"2023-10-16T20:36:55.290023+0000","last_unstale":"2023-10-16T20:58:37.896916+0000","last_undegraded":"2023-10-16T20:58:37.896916+0000","last_fullsized":"2023-10-16T20:58:37.896916+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:33.259824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013081514000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'124","reported_seq":2233,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.203227+0000","last_change":"2023-10-16T20:47:31.586678+0000","last_active":"2023-10-16T20:58:38.203227+0000","last_peered":"2023-10-16T20:58:38.203227+0000","last_clean":"2023-10-16T20:58:38.203227+0000","last_became_active":"2023-10-16T20:36:55.280186+0000","last_became_peered":"2023-10-16T20:36:55.280186+0000","last_unstale":"2023-10-16T20:58:38.203227+0000","last_undegraded":"2023-10-16T20:58:38.203227+0000","last_fullsized":"2023-10-16T20:58:38.203227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:22:01.009457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083925690000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"67'118","reported_seq":2227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.882744+0000","last_change":"2023-10-16T20:47:31.589254+0000","last_active":"2023-10-16T20:58:37.882744+0000","last_peered":"2023-10-16T20:58:37.882744+0000","last_clean":"2023-10-16T20:58:37.882744+0000","last_became_active":"2023-10-16T20:36:55.297253+0000","last_became_peered":"2023-10-16T20:36:55.297253+0000","last_unstale":"2023-10-16T20:58:37.882744+0000","last_undegraded":"2023-10-16T20:58:37.882744+0000","last_fullsized":"2023-10-16T20:58:37.882744+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:11.854849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097685800000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"67'108","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.891317+0000","last_change":"2023-10-16T20:47:31.492104+0000","last_active":"2023-10-16T20:58:37.891317+0000","last_peered":"2023-10-16T20:58:37.891317+0000","last_clean":"2023-10-16T20:58:37.891317+0000","last_became_active":"2023-10-16T20:36:55.286044+0000","last_became_peered":"2023-10-16T20:36:55.286044+0000","last_unstale":"2023-10-16T20:58:37.891317+0000","last_undegraded":"2023-10-16T20:58:37.891317+0000","last_fullsized":"2023-10-16T20:58:37.891317+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:45.304718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011681474000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"67'110","reported_seq":2219,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.865687+0000","last_change":"2023-10-16T20:47:31.592401+0000","last_active":"2023-10-16T20:58:37.865687+0000","last_peered":"2023-10-16T20:58:37.865687+0000","last_clean":"2023-10-16T20:58:37.865687+0000","last_became_active":"2023-10-16T20:36:55.286864+0000","last_became_peered":"2023-10-16T20:36:55.286864+0000","last_unstale":"2023-10-16T20:58:37.865687+0000","last_undegraded":"2023-10-16T20:58:37.865687+0000","last_fullsized":"2023-10-16T20:58:37.865687+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:31.381174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012885232999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"67'106","reported_seq":2215,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.872600+0000","last_change":"2023-10-16T20:47:31.588333+0000","last_active":"2023-10-16T20:58:37.872600+0000","last_peered":"2023-10-16T20:58:37.872600+0000","last_clean":"2023-10-16T20:58:37.872600+0000","last_became_active":"2023-10-16T20:36:55.292262+0000","last_became_peered":"2023-10-16T20:36:55.292262+0000","last_unstale":"2023-10-16T20:58:37.872600+0000","last_undegraded":"2023-10-16T20:58:37.872600+0000","last_fullsized":"2023-10-16T20:58:37.872600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:15.116061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077171749999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"67'107","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.832173+0000","last_change":"2023-10-16T20:47:31.496924+0000","last_active":"2023-10-16T20:58:37.832173+0000","last_peered":"2023-10-16T20:58:37.832173+0000","last_clean":"2023-10-16T20:58:37.832173+0000","last_became_active":"2023-10-16T20:36:55.282806+0000","last_became_peered":"2023-10-16T20:36:55.282806+0000","last_unstale":"2023-10-16T20:58:37.832173+0000","last_undegraded":"2023-10-16T20:58:37.832173+0000","last_fullsized":"2023-10-16T20:58:37.832173+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:51.395313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086132999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"67'109","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.887891+0000","last_change":"2023-10-16T20:47:31.594959+0000","last_active":"2023-10-16T20:58:37.887891+0000","last_peered":"2023-10-16T20:58:37.887891+0000","last_clean":"2023-10-16T20:58:37.887891+0000","last_became_active":"2023-10-16T20:36:55.303095+0000","last_became_peered":"2023-10-16T20:36:55.303095+0000","last_unstale":"2023-10-16T20:58:37.887891+0000","last_undegraded":"2023-10-16T20:58:37.887891+0000","last_fullsized":"2023-10-16T20:58:37.887891+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:00:51.726359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013746639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"67'119","reported_seq":2228,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.181755+0000","last_change":"2023-10-16T20:47:31.595380+0000","last_active":"2023-10-16T20:58:38.181755+0000","last_peered":"2023-10-16T20:58:38.181755+0000","last_clean":"2023-10-16T20:58:38.181755+0000","last_became_active":"2023-10-16T20:36:55.291447+0000","last_became_peered":"2023-10-16T20:36:55.291447+0000","last_unstale":"2023-10-16T20:58:38.181755+0000","last_undegraded":"2023-10-16T20:58:38.181755+0000","last_fullsized":"2023-10-16T20:58:38.181755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:20.797976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01315413,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"67'107","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.868160+0000","last_change":"2023-10-16T20:47:31.484708+0000","last_active":"2023-10-16T20:58:37.868160+0000","last_peered":"2023-10-16T20:58:37.868160+0000","last_clean":"2023-10-16T20:58:37.868160+0000","last_became_active":"2023-10-16T20:36:55.280605+0000","last_became_peered":"2023-10-16T20:36:55.280605+0000","last_unstale":"2023-10-16T20:58:37.868160+0000","last_undegraded":"2023-10-16T20:58:37.868160+0000","last_fullsized":"2023-10-16T20:58:37.868160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:06:39.852344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033133080000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"67'112","reported_seq":2221,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.891112+0000","last_change":"2023-10-16T20:47:31.492149+0000","last_active":"2023-10-16T20:58:37.891112+0000","last_peered":"2023-10-16T20:58:37.891112+0000","last_clean":"2023-10-16T20:58:37.891112+0000","last_became_active":"2023-10-16T20:36:55.277094+0000","last_became_peered":"2023-10-16T20:36:55.277094+0000","last_unstale":"2023-10-16T20:58:37.891112+0000","last_undegraded":"2023-10-16T20:58:37.891112+0000","last_fullsized":"2023-10-16T20:58:37.891112+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:35:01.971894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011816411000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"67'94","reported_seq":2203,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.879368+0000","last_change":"2023-10-16T20:47:31.587367+0000","last_active":"2023-10-16T20:58:37.879368+0000","last_peered":"2023-10-16T20:58:37.879368+0000","last_clean":"2023-10-16T20:58:37.879368+0000","last_became_active":"2023-10-16T20:36:55.297199+0000","last_became_peered":"2023-10-16T20:36:55.297199+0000","last_unstale":"2023-10-16T20:58:37.879368+0000","last_undegraded":"2023-10-16T20:58:37.879368+0000","last_fullsized":"2023-10-16T20:58:37.879368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:07:32.275973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086430910000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"67'90","reported_seq":2199,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.867006+0000","last_change":"2023-10-16T20:47:31.487461+0000","last_active":"2023-10-16T20:58:37.867006+0000","last_peered":"2023-10-16T20:58:37.867006+0000","last_clean":"2023-10-16T20:58:37.867006+0000","last_became_active":"2023-10-16T20:36:55.286964+0000","last_became_peered":"2023-10-16T20:36:55.286964+0000","last_unstale":"2023-10-16T20:58:37.867006+0000","last_undegraded":"2023-10-16T20:58:37.867006+0000","last_fullsized":"2023-10-16T20:58:37.867006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:57.164132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051857609999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"67'110","reported_seq":2226,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.834126+0000","last_change":"2023-10-16T20:47:31.594637+0000","last_active":"2023-10-16T20:58:37.834126+0000","last_peered":"2023-10-16T20:58:37.834126+0000","last_clean":"2023-10-16T20:58:37.834126+0000","last_became_active":"2023-10-16T20:37:37.413492+0000","last_became_peered":"2023-10-16T20:37:37.413492+0000","last_unstale":"2023-10-16T20:58:37.834126+0000","last_undegraded":"2023-10-16T20:58:37.834126+0000","last_fullsized":"2023-10-16T20:58:37.834126+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:07.365316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014529932000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"67'106","reported_seq":2215,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.850164+0000","last_change":"2023-10-16T20:47:31.593072+0000","last_active":"2023-10-16T20:58:37.850164+0000","last_peered":"2023-10-16T20:58:37.850164+0000","last_clean":"2023-10-16T20:58:37.850164+0000","last_became_active":"2023-10-16T20:36:55.302468+0000","last_became_peered":"2023-10-16T20:36:55.302468+0000","last_unstale":"2023-10-16T20:58:37.850164+0000","last_undegraded":"2023-10-16T20:58:37.850164+0000","last_fullsized":"2023-10-16T20:58:37.850164+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:54:24.682600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012359399,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"67'92","reported_seq":2200,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.894436+0000","last_change":"2023-10-16T20:47:31.585931+0000","last_active":"2023-10-16T20:58:37.894436+0000","last_peered":"2023-10-16T20:58:37.894436+0000","last_clean":"2023-10-16T20:58:37.894436+0000","last_became_active":"2023-10-16T20:36:55.296476+0000","last_became_peered":"2023-10-16T20:36:55.296476+0000","last_unstale":"2023-10-16T20:58:37.894436+0000","last_undegraded":"2023-10-16T20:58:37.894436+0000","last_fullsized":"2023-10-16T20:58:37.894436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:50:21.532799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072035780000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'95","reported_seq":2203,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.898420+0000","last_change":"2023-10-16T20:47:31.585218+0000","last_active":"2023-10-16T20:58:37.898420+0000","last_peered":"2023-10-16T20:58:37.898420+0000","last_clean":"2023-10-16T20:58:37.898420+0000","last_became_active":"2023-10-16T20:36:55.293687+0000","last_became_peered":"2023-10-16T20:36:55.293687+0000","last_unstale":"2023-10-16T20:58:37.898420+0000","last_undegraded":"2023-10-16T20:58:37.898420+0000","last_fullsized":"2023-10-16T20:58:37.898420+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:06:36.236196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074548399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"67'127","reported_seq":2235,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.887764+0000","last_change":"2023-10-16T20:47:31.588394+0000","last_active":"2023-10-16T20:58:37.887764+0000","last_peered":"2023-10-16T20:58:37.887764+0000","last_clean":"2023-10-16T20:58:37.887764+0000","last_became_active":"2023-10-16T20:36:55.280444+0000","last_became_peered":"2023-10-16T20:36:55.280444+0000","last_unstale":"2023-10-16T20:58:37.887764+0000","last_undegraded":"2023-10-16T20:58:37.887764+0000","last_fullsized":"2023-10-16T20:58:37.887764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:40:02.818328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068623299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'109","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888019+0000","last_change":"2023-10-16T20:47:31.588884+0000","last_active":"2023-10-16T20:58:37.888019+0000","last_peered":"2023-10-16T20:58:37.888019+0000","last_clean":"2023-10-16T20:58:37.888019+0000","last_became_active":"2023-10-16T20:36:55.296694+0000","last_became_peered":"2023-10-16T20:36:55.296694+0000","last_unstale":"2023-10-16T20:58:37.888019+0000","last_undegraded":"2023-10-16T20:58:37.888019+0000","last_fullsized":"2023-10-16T20:58:37.888019+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:39.793533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096291870000000009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"67'120","reported_seq":2228,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.879224+0000","last_change":"2023-10-16T20:47:31.585319+0000","last_active":"2023-10-16T20:58:37.879224+0000","last_peered":"2023-10-16T20:58:37.879224+0000","last_clean":"2023-10-16T20:58:37.879224+0000","last_became_active":"2023-10-16T20:36:55.291052+0000","last_became_peered":"2023-10-16T20:36:55.291052+0000","last_unstale":"2023-10-16T20:58:37.879224+0000","last_undegraded":"2023-10-16T20:58:37.879224+0000","last_fullsized":"2023-10-16T20:58:37.879224+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:45.283686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070559409999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"67'94","reported_seq":2204,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.895896+0000","last_change":"2023-10-16T20:47:31.588931+0000","last_active":"2023-10-16T20:58:37.895896+0000","last_peered":"2023-10-16T20:58:37.895896+0000","last_clean":"2023-10-16T20:58:37.895896+0000","last_became_active":"2023-10-16T20:36:55.290386+0000","last_became_peered":"2023-10-16T20:36:55.290386+0000","last_unstale":"2023-10-16T20:58:37.895896+0000","last_undegraded":"2023-10-16T20:58:37.895896+0000","last_fullsized":"2023-10-16T20:58:37.895896+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:56:59.640546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093217969999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"67'114","reported_seq":2224,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888639+0000","last_change":"2023-10-16T20:47:31.490951+0000","last_active":"2023-10-16T20:58:37.888639+0000","last_peered":"2023-10-16T20:58:37.888639+0000","last_clean":"2023-10-16T20:58:37.888639+0000","last_became_active":"2023-10-16T20:36:55.270787+0000","last_became_peered":"2023-10-16T20:36:55.270787+0000","last_unstale":"2023-10-16T20:58:37.888639+0000","last_undegraded":"2023-10-16T20:58:37.888639+0000","last_fullsized":"2023-10-16T20:58:37.888639+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:57:57.930582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010801276,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"67'94","reported_seq":2204,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.890538+0000","last_change":"2023-10-16T20:47:31.487902+0000","last_active":"2023-10-16T20:58:37.890538+0000","last_peered":"2023-10-16T20:58:37.890538+0000","last_clean":"2023-10-16T20:58:37.890538+0000","last_became_active":"2023-10-16T20:36:55.271373+0000","last_became_peered":"2023-10-16T20:36:55.271373+0000","last_unstale":"2023-10-16T20:58:37.890538+0000","last_undegraded":"2023-10-16T20:58:37.890538+0000","last_fullsized":"2023-10-16T20:58:37.890538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:37:10.982777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006091478,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"67'111","reported_seq":2221,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.890234+0000","last_change":"2023-10-16T20:47:31.587880+0000","last_active":"2023-10-16T20:58:37.890234+0000","last_peered":"2023-10-16T20:58:37.890234+0000","last_clean":"2023-10-16T20:58:37.890234+0000","last_became_active":"2023-10-16T20:36:55.291016+0000","last_became_peered":"2023-10-16T20:36:55.291016+0000","last_unstale":"2023-10-16T20:58:37.890234+0000","last_undegraded":"2023-10-16T20:58:37.890234+0000","last_fullsized":"2023-10-16T20:58:37.890234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:23:37.851121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009370148,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":924319467,"num_objects":10294,"num_object_clones":0,"num_object_copies":41176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10294,"num_whiteouts":0,"num_read":30602,"num_read_kb":676969,"num_write":21002,"num_write_kb":1302825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966899200,"data_stored":1966899200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21050,"ondisk_log_size":21050,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6794,"ondisk_log_size":6794,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":178960041,"num_objects":198,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1063,"num_read_kb":205540,"num_write":6349,"num_write_kb":180514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":115,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":358170624,"data_stored":357920082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6078,"ondisk_log_size":6078,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509707,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396772,"kb_used_data":280924,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93975068,"statfs":{"total":96636764160,"available":96230469632,"internally_reserved":0,"allocated":287666176,"data_stored":287268719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":118618571},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.52300000000000002,"15min":0.629},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67900000000000005,"5min":0.91400000000000003,"15min":2.4489999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54600000000000004,"15min":0.64700000000000002},"min":{"1min":0.378,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80200000000000005,"5min":0.95399999999999996,"15min":11.77},"last":0.73899999999999999}]},{"osd":1,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57699999999999996,"15min":0.65300000000000002},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73699999999999999,"5min":0.86199999999999999,"15min":4.5279999999999996},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58499999999999996,"15min":0.68400000000000005},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.84599999999999997,"5min":1.024,"15min":11.971},"last":0.77400000000000002}]},{"osd":2,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61599999999999999,"15min":0.67200000000000004},"min":{"1min":0.41999999999999998,"5min":0.40200000000000002,"15min":0.34300000000000003},"max":{"1min":0.81899999999999995,"5min":0.92200000000000004,"15min":7.1130000000000004},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.629,"15min":0.70999999999999996},"min":{"1min":0.51100000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.877,"5min":0.98299999999999998,"15min":9.4450000000000003},"last":0.69699999999999995}]},{"osd":3,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.64900000000000002,"15min":0.68300000000000005},"min":{"1min":0.439,"5min":0.437,"15min":0.41299999999999998},"max":{"1min":0.78000000000000003,"5min":1,"15min":3.363},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.68300000000000005,"15min":0.72699999999999998},"min":{"1min":0.50900000000000001,"5min":0.46899999999999997,"15min":0.42099999999999999},"max":{"1min":0.86299999999999999,"5min":1.0529999999999999,"15min":3.3980000000000001},"last":0.76300000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56699999999999995,"15min":0.61399999999999999},"min":{"1min":0.36399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81899999999999995,"5min":1.091,"15min":12.532999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.623,"15min":0.628},"min":{"1min":0.47099999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90600000000000003,"5min":1.107,"15min":3.8700000000000001},"last":0.52100000000000002}]},{"osd":5,"last update":"Mon Oct 16 20:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60999999999999999,"15min":0.66300000000000003},"min":{"1min":0.41399999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.85699999999999998,"5min":1.042,"15min":2.2010000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.63600000000000001,"15min":0.73699999999999999},"min":{"1min":0.47499999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94399999999999995,"5min":1.0369999999999999,"15min":12.353},"last":0.48899999999999999}]},{"osd":6,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65100000000000002,"15min":0.746},"min":{"1min":0.40799999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93799999999999994,"5min":1.1299999999999999,"15min":11.929},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66900000000000004,"15min":0.71399999999999997},"min":{"1min":0.41799999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92900000000000005,"5min":1.0620000000000001,"15min":6.0359999999999996},"last":0.58599999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509707,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421948,"kb_used_data":300532,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93949892,"statfs":{"total":96636764160,"available":96204689408,"internally_reserved":0,"allocated":307744768,"data_stored":307357593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124320199},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.54900000000000004,"15min":0.61499999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27800000000000002},"max":{"1min":0.69499999999999995,"5min":0.93799999999999994,"15min":4.5419999999999998},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.56499999999999995,"15min":0.624},"min":{"1min":0.33200000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.76000000000000001,"5min":2.0329999999999999,"15min":4.9699999999999998},"last":0.79900000000000004}]},{"osd":1,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.624,"15min":0.64100000000000001},"min":{"1min":0.433,"5min":0.39200000000000002,"15min":0.377},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":2.8610000000000002},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67000000000000004,"15min":0.69599999999999995},"min":{"1min":0.41499999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.94399999999999995,"5min":1.014,"15min":1.0309999999999999},"last":0.84499999999999997}]},{"osd":2,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.59699999999999998,"15min":0.65000000000000002},"min":{"1min":0.36399999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86599999999999999,"5min":1.0289999999999999,"15min":1.895},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.65500000000000003,"15min":0.68500000000000005},"min":{"1min":0.40799999999999997,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78500000000000003,"5min":0.96099999999999997,"15min":2.3340000000000001},"last":0.86499999999999999}]},{"osd":3,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.755,"15min":0.72099999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.36599999999999999},"max":{"1min":0.86699999999999999,"5min":4.6340000000000003,"15min":4.6340000000000003},"last":0.879},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.64100000000000001,"15min":0.69499999999999995},"min":{"1min":0.435,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.86099999999999999,"5min":1.163,"15min":5.1029999999999998},"last":0.67500000000000004}]},{"osd":4,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55500000000000005,"15min":0.58999999999999997},"min":{"1min":0.36299999999999999,"5min":0.34799999999999998,"15min":0.30299999999999999},"max":{"1min":0.80300000000000005,"5min":1.0189999999999999,"15min":4.8959999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.63500000000000001,"15min":0.63400000000000001},"min":{"1min":0.374,"5min":0.313,"15min":0.313},"max":{"1min":0.996,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.58999999999999997}]},{"osd":5,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.59799999999999998,"15min":0.623},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.82199999999999995,"5min":1.05,"15min":2.3769999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.626,"15min":0.66200000000000003},"min":{"1min":0.38200000000000001,"5min":0.35999999999999999,"15min":0.34499999999999997},"max":{"1min":0.84499999999999997,"5min":1.0700000000000001,"15min":4.9379999999999997},"last":0.82699999999999996}]},{"osd":7,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.71199999999999997,"15min":0.70999999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.034,"5min":1.425,"15min":4.5789999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.76800000000000002,"15min":0.748},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.39300000000000002},"max":{"1min":1.0600000000000001,"5min":1.6120000000000001,"15min":5.2460000000000004},"last":0.73999999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509709,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324840,"kb_used_data":225632,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94047000,"statfs":{"total":96636764160,"available":96304128000,"internally_reserved":0,"allocated":231047168,"data_stored":230645947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":101579207},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.57199999999999995,"15min":0.61299999999999999},"min":{"1min":0.30499999999999999,"5min":0.28199999999999997,"15min":0.25700000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":2.4580000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.59299999999999997,"15min":0.65800000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.94999999999999996,"5min":1.1799999999999999,"15min":7.181},"last":0.64600000000000002}]},{"osd":1,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":0.76800000000000002,"15min":0.69099999999999995},"min":{"1min":0.36699999999999999,"5min":0.33900000000000002,"15min":0.29899999999999999},"max":{"1min":16.655000000000001,"5min":16.655000000000001,"15min":16.655000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.5529999999999999,"5min":0.80100000000000005,"15min":0.71299999999999997},"min":{"1min":0.35099999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":16.289000000000001,"5min":16.289000000000001,"15min":16.289000000000001},"last":0.55700000000000005}]},{"osd":2,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.63200000000000001,"15min":0.67300000000000004},"min":{"1min":0.374,"5min":0.34599999999999997,"15min":0.308},"max":{"1min":1.1339999999999999,"5min":1.7110000000000001,"15min":3.3799999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67100000000000004,"15min":0.67700000000000005},"min":{"1min":0.42399999999999999,"5min":0.39800000000000002,"15min":0.38},"max":{"1min":1.1100000000000001,"5min":1.583,"15min":4.5369999999999999},"last":0.59399999999999997}]},{"osd":3,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.67400000000000004,"15min":0.69699999999999995},"min":{"1min":0.42399999999999999,"5min":0.39400000000000002,"15min":0.34899999999999998},"max":{"1min":1.0860000000000001,"5min":1.329,"15min":4.1070000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68600000000000005,"15min":0.69399999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.38},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":3.476},"last":0.69599999999999995}]},{"osd":4,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53100000000000003,"15min":0.55500000000000005},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.749,"5min":0.85099999999999998,"15min":2.415},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.59999999999999998,"15min":0.66300000000000003},"min":{"1min":0.34300000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.78900000000000003,"5min":1.155,"15min":3.2919999999999998},"last":0.45300000000000001}]},{"osd":6,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57299999999999995,"15min":0.59699999999999998},"min":{"1min":0.39000000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79400000000000004,"5min":0.93799999999999994,"15min":3.2400000000000002},"last":0.374},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.31},"max":{"1min":0.71799999999999997,"5min":1.036,"15min":1.2310000000000001},"last":0.48099999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.44600000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76300000000000001,"5min":1.23,"15min":5.2759999999999998},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69399999999999995,"15min":0.70699999999999996},"min":{"1min":0.45000000000000001,"5min":0.435,"15min":0.42299999999999999},"max":{"1min":0.83199999999999996,"5min":0.97699999999999998,"15min":3.3140000000000001},"last":0.63500000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509707,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420124,"kb_used_data":290388,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93951716,"statfs":{"total":96636764160,"available":96206557184,"internally_reserved":0,"allocated":297357312,"data_stored":296970571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56100000000000005,"15min":0.61299999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.83799999999999997,"5min":0.91400000000000003,"15min":1.524},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61699999999999999,"15min":0.69199999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.5649999999999999},"last":0.89200000000000002}]},{"osd":1,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.376,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.1749999999999998,"5min":4.1749999999999998,"15min":4.1749999999999998},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.625,"15min":0.66600000000000004},"min":{"1min":0.38200000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82599999999999996,"5min":0.94899999999999995,"15min":1.1020000000000001},"last":0.91400000000000003}]},{"osd":2,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.38900000000000001},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":2.3370000000000002},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.68300000000000005,"15min":0.71199999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.84499999999999997,"5min":1.2210000000000001,"15min":2.3660000000000001},"last":0.98999999999999999}]},{"osd":3,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.377},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":2.04},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":2.2240000000000002},"last":1.042}]},{"osd":5,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.58799999999999997,"15min":0.60299999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.9970000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63600000000000001,"15min":0.60499999999999998},"min":{"1min":0.41299999999999998,"5min":0.39500000000000002,"15min":0.33900000000000002},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.77900000000000003}]},{"osd":6,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.60299999999999998,"15min":0.61599999999999999},"min":{"1min":0.33900000000000002,"5min":0.315,"15min":0.312},"max":{"1min":1.2270000000000001,"5min":1.6339999999999999,"15min":2.2639999999999998},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.66200000000000003,"15min":0.67300000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.8300000000000001},"last":0.85299999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.69399999999999995,"15min":0.66800000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2969999999999999,"5min":1.347,"15min":1.347},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65500000000000003,"15min":0.67900000000000005},"min":{"1min":0.38600000000000001,"5min":0.35899999999999999,"15min":0.35599999999999998},"max":{"1min":1.266,"5min":1.266,"15min":2.2999999999999998},"last":1.008}]}]},{"osd":3,"up_from":15,"seq":64424509707,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485596,"kb_used_data":360020,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93886244,"statfs":{"total":96636764160,"available":96139513856,"internally_reserved":0,"allocated":368660480,"data_stored":368273200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":128580040},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.45200000000000001,"15min":0.433},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":0.76300000000000001,"5min":2.8109999999999999,"15min":4.5830000000000002},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.44700000000000001,"15min":0.44600000000000001},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.253},"max":{"1min":0.58199999999999996,"5min":1.147,"15min":1.2509999999999999},"last":0.40799999999999997}]},{"osd":1,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.49099999999999999,"15min":0.503},"min":{"1min":0.33500000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67000000000000004,"5min":1.411,"15min":4.6840000000000002},"last":0.495},{"interface":"front","average":{"1min":0.437,"5min":0.504,"15min":0.51000000000000001},"min":{"1min":0.29699999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61399999999999999,"5min":2.6360000000000001,"15min":2.7429999999999999},"last":0.38300000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.40000000000000002,"5min":0.34399999999999997,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":1.286,"15min":4.6449999999999996},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55700000000000005,"15min":0.51900000000000002},"min":{"1min":0.32200000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":1.1850000000000001,"15min":1.911},"last":0.47299999999999998}]},{"osd":4,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.72399999999999998,"15min":0.76200000000000001},"min":{"1min":0.52000000000000002,"5min":0.45400000000000001,"15min":0.44800000000000001},"max":{"1min":1.0620000000000001,"5min":1.538,"15min":1.7749999999999999},"last":0.877},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.72399999999999998,"15min":0.74199999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.155,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":0.91000000000000003}]},{"osd":5,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.74199999999999999,"15min":0.755},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0780000000000001,"5min":1.4970000000000001,"15min":4.8449999999999998},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.74099999999999999,"15min":0.745},"min":{"1min":0.48199999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1220000000000001,"5min":2.948,"15min":2.948},"last":0.65900000000000003}]},{"osd":6,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.76300000000000001,"15min":0.76600000000000001},"min":{"1min":0.48899999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.98399999999999999,"5min":1.633,"15min":4.8099999999999996},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.45400000000000001},"max":{"1min":0.93899999999999995,"5min":1.6559999999999999,"15min":2.339},"last":0.85799999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.79800000000000004,"15min":0.86499999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.46000000000000002},"max":{"1min":1.216,"5min":1.593,"15min":34.039000000000001},"last":0.753},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.76900000000000002,"15min":0.77700000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0249999999999999,"5min":2.984,"15min":2.984},"last":0.68899999999999995}]}]},{"osd":2,"up_from":15,"seq":64424509708,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303812,"kb_used_data":200444,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94068028,"statfs":{"total":96636764160,"available":96325660672,"internally_reserved":0,"allocated":205254656,"data_stored":204864239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42899999999999999,"15min":0.45000000000000001},"min":{"1min":0.27100000000000002,"5min":0.23400000000000001,"15min":0.20599999999999999},"max":{"1min":0.74299999999999999,"5min":0.92500000000000004,"15min":5.5499999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.52600000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.1180000000000001,"5min":1.1220000000000001,"15min":5.4160000000000004},"last":0.34799999999999998}]},{"osd":1,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.504,"15min":0.5},"min":{"1min":0.307,"5min":0.307,"15min":0.26000000000000001},"max":{"1min":0.81899999999999995,"5min":1.25,"15min":2.2349999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.501},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.27200000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":5.4359999999999999},"last":0.38600000000000001}]},{"osd":3,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.28399999999999997,"5min":0.26400000000000001,"15min":0.25800000000000001},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.312,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.83099999999999996,"5min":1.286,"15min":5.3789999999999996},"last":0.58699999999999997}]},{"osd":4,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.73099999999999998,"15min":0.80000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":3.464,"5min":3.464,"15min":16.905999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.70699999999999996,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.44700000000000001,"15min":0.441},"max":{"1min":0.92100000000000004,"5min":1.3100000000000001,"15min":10.064},"last":0.70699999999999996}]},{"osd":5,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70999999999999996,"15min":0.72999999999999998},"min":{"1min":0.45300000000000001,"5min":0.41099999999999998,"15min":0.32800000000000001},"max":{"1min":1.0960000000000001,"5min":1.304,"15min":3.2989999999999999},"last":0.627},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.73499999999999999,"15min":0.77100000000000002},"min":{"1min":0.52400000000000002,"5min":0.499,"15min":0.46700000000000003},"max":{"1min":1.0069999999999999,"5min":1.3640000000000001,"15min":5.3330000000000002},"last":0.66100000000000003}]},{"osd":6,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.76500000000000001,"15min":0.755},"min":{"1min":0.438,"5min":0.438,"15min":0.42299999999999999},"max":{"1min":3.5710000000000002,"5min":3.5710000000000002,"15min":3.5710000000000002},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.77900000000000003,"15min":0.77000000000000002},"min":{"1min":0.49099999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":5.4749999999999996},"last":0.75800000000000001}]},{"osd":7,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.47399999999999998,"5min":0.47199999999999998,"15min":0.44500000000000001},"max":{"1min":3.5089999999999999,"5min":3.5089999999999999,"15min":5.516},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77200000000000002,"15min":0.79100000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.41899999999999998},"max":{"1min":1.325,"5min":1.325,"15min":8.4610000000000003},"last":0.77600000000000002}]}]},{"osd":1,"up_from":15,"seq":64424509707,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490520,"kb_used_data":360784,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93881320,"statfs":{"total":96636764160,"available":96134471680,"internally_reserved":0,"allocated":369442816,"data_stored":369057373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.437,"15min":0.45700000000000002},"min":{"1min":0.245,"5min":0.219,"15min":0.21299999999999999},"max":{"1min":0.79800000000000004,"5min":0.93000000000000005,"15min":8.9339999999999993},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.34300000000000003,"5min":0.28999999999999998,"15min":0.23000000000000001},"max":{"1min":0.67200000000000004,"5min":1.6899999999999999,"15min":3.8809999999999998},"last":0.63600000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.502,"15min":0.46000000000000002},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.248},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.55900000000000005,"15min":0.55400000000000005},"min":{"1min":0.29699999999999999,"5min":0.29099999999999998,"15min":0.28499999999999998},"max":{"1min":0.88600000000000001,"5min":2.6739999999999999,"15min":9.2680000000000007},"last":0.69699999999999995}]},{"osd":3,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57499999999999996,"15min":0.54200000000000004},"min":{"1min":0.33000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":1.2030000000000001,"5min":1.9910000000000001,"15min":3.0539999999999998},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57999999999999996,"15min":0.57799999999999996},"min":{"1min":0.433,"5min":0.34200000000000003,"15min":0.313},"max":{"1min":0.84999999999999998,"5min":1.4119999999999999,"15min":6.9409999999999998},"last":0.73299999999999998}]},{"osd":4,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.68300000000000005,"15min":0.73199999999999998},"min":{"1min":0.34300000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.90400000000000003,"5min":1.962,"15min":5.6630000000000003},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.72399999999999998,"15min":0.75900000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.151,"5min":1.6719999999999999,"15min":8.8960000000000008},"last":0.88700000000000001}]},{"osd":5,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.76000000000000001,"15min":0.77700000000000002},"min":{"1min":0.48399999999999999,"5min":0.47299999999999998,"15min":0.40300000000000002},"max":{"1min":1.242,"5min":1.242,"15min":6.6159999999999997},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.77800000000000002,"15min":0.79500000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.1220000000000001,"5min":1.7130000000000001,"15min":5.7009999999999996},"last":0.93799999999999994}]},{"osd":6,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.73299999999999998,"15min":0.77300000000000002},"min":{"1min":0.47799999999999998,"5min":0.47299999999999998,"15min":0.39800000000000002},"max":{"1min":0.94199999999999995,"5min":2.0529999999999999,"15min":8.6039999999999992},"last":1.075},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.77200000000000002,"15min":0.75},"min":{"1min":0.50700000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98199999999999998,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.0369999999999999}]},{"osd":7,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76300000000000001,"15min":0.77800000000000002},"min":{"1min":0.46899999999999997,"5min":0.45200000000000001,"15min":0.44800000000000001},"max":{"1min":1.1639999999999999,"5min":1.7470000000000001,"15min":6.6600000000000001},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.78700000000000003,"15min":0.80600000000000005},"min":{"1min":0.50700000000000001,"5min":0.46400000000000002,"15min":0.46200000000000002},"max":{"1min":1.1990000000000001,"5min":1.774,"15min":8.8100000000000005},"last":0.998}]}]},{"osd":0,"up_from":13,"seq":55834575115,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":370924,"kb_used_data":259236,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94000916,"statfs":{"total":96636764160,"available":96256937984,"internally_reserved":0,"allocated":265457664,"data_stored":265061773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,54,40],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.39500000000000002,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.56299999999999994,"5min":1.3029999999999999,"15min":6.8170000000000002},"last":24.483000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.46100000000000002,"15min":0.47499999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.70199999999999996,"5min":1.591,"15min":6.2789999999999999},"last":91.462000000000003}]},{"osd":2,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.51800000000000002,"15min":0.495},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90100000000000002,"5min":1.169,"15min":3.1859999999999999},"last":91.384},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.47699999999999998,"15min":0.46500000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.97499999999999998,"5min":1.266,"15min":1.47},"last":11.782}]},{"osd":3,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56999999999999995,"15min":0.52100000000000002},"min":{"1min":0.40899999999999997,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.67900000000000005,"5min":1.423,"15min":1.8180000000000001},"last":91.557000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.35599999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.93300000000000005,"5min":1.4390000000000001,"15min":1.881},"last":11.835000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.68799999999999994,"15min":0.80700000000000005},"min":{"1min":0.46400000000000002,"5min":0.45100000000000001,"15min":0.45000000000000001},"max":{"1min":1.0529999999999999,"5min":1.3169999999999999,"15min":18.41},"last":13.169},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70399999999999996,"15min":0.81399999999999995},"min":{"1min":0.50800000000000001,"5min":0.46300000000000002,"15min":0.42399999999999999},"max":{"1min":1.097,"5min":1.409,"15min":18.998999999999999},"last":91.421999999999997}]},{"osd":5,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73699999999999999,"15min":0.76100000000000001},"min":{"1min":0.52400000000000002,"5min":0.50900000000000001,"15min":0.42799999999999999},"max":{"1min":1.1299999999999999,"5min":1.3460000000000001,"15min":3.5499999999999998},"last":79.207999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69999999999999996,"15min":0.73699999999999999},"min":{"1min":0.45900000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.996,"5min":1.4279999999999999,"15min":3.605},"last":91.587999999999994}]},{"osd":6,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.73999999999999999,"15min":0.754},"min":{"1min":0.52200000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0289999999999999,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":9.8290000000000006},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.76600000000000001,"15min":0.78200000000000003},"min":{"1min":0.502,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0840000000000001,"5min":2.367,"15min":2.367},"last":91.522999999999996}]},{"osd":7,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.49099999999999999},"max":{"1min":1.042,"5min":1.395,"15min":4.7839999999999998},"last":91.494},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73999999999999999,"15min":0.77000000000000002},"min":{"1min":0.57399999999999995,"5min":0.53200000000000003,"15min":0.45500000000000002},"max":{"1min":0.995,"5min":1.3759999999999999,"15min":2.0110000000000001},"last":91.346000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39333888,"data_stored":39301023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51634176,"data_stored":51607948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18219008,"data_stored":18187806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":47730688,"data_stored":47702623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56041472,"data_stored":56013946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":44449792,"data_stored":44407786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40529920,"data_stored":40501960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":60231680,"data_stored":60196990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":224890880,"data_stored":224890880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":316989440,"data_stored":316989440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186216448,"data_stored":186216448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":320110592,"data_stored":320110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":240496640,"data_stored":240496640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":185778176,"data_stored":185778176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266395648,"data_stored":266395648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":226021376,"data_stored":226021376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T20:58:44.873 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:58:45.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.046+0000 7fa4b7cc9640 1 Processor -- start 2023-10-16T20:58:45.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.046+0000 7fa4b7cc9640 1 -- start start 2023-10-16T20:58:45.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b7cc9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4b0150660 0x7fa4b00c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:45.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b7cc9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa4b00c6bc0 0x7fa4b00c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:45.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b7cc9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b00c5c60 con 0x7fa4b0150280 2023-10-16T20:58:45.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b7cc9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4b00c5370 con 0x7fa4b00c6bc0 2023-10-16T20:58:45.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b7cc9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b0156d10 con 0x7fa4b0150660 2023-10-16T20:58:45.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b5a65640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa4b0150280 0x7fa4b006d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59576/0 (socket says 172.21.15.88:59576) 2023-10-16T20:58:45.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b5a65640 1 -- 172.21.15.88:0/2739397609 learned_addr learned my addr 172.21.15.88:0/2739397609 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:45.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b6266640 1 --2- 172.21.15.88:0/2739397609 >> v2:172.21.15.191:3300/0 conn(0x7fa4b00c6bc0 0x7fa4b00c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:45.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b5264640 1 --2- 172.21.15.88:0/2739397609 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4b0150660 0x7fa4b00c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:45.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b4a63640 1 -- 172.21.15.88:0/2739397609 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378473570 0 0) 0x7fa4b00c5c60 con 0x7fa4b0150280 2023-10-16T20:58:45.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b4a63640 1 -- 172.21.15.88:0/2739397609 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa49c005740 con 0x7fa4b0150280 2023-10-16T20:58:45.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b4a63640 1 -- 172.21.15.88:0/2739397609 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 515383678 0 0) 0x7fa49c005740 con 0x7fa4b0150280 2023-10-16T20:58:45.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b4a63640 1 -- 172.21.15.88:0/2739397609 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4b0150660 msgr2=0x7fa4b00c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:45.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b4a63640 1 --2- 172.21.15.88:0/2739397609 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4b0150660 0x7fa4b00c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:45.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b4a63640 1 -- 172.21.15.88:0/2739397609 >> v2:172.21.15.191:3300/0 conn(0x7fa4b00c6bc0 msgr2=0x7fa4b00c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:45.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b4a63640 1 --2- 172.21.15.88:0/2739397609 >> v2:172.21.15.191:3300/0 conn(0x7fa4b00c6bc0 0x7fa4b00c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:45.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b4a63640 1 -- 172.21.15.88:0/2739397609 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b00c5610 con 0x7fa4b0150280 2023-10-16T20:58:45.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b4a63640 1 -- 172.21.15.88:0/2739397609 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7fa4ac001680 con 0x7fa4b0150280 2023-10-16T20:58:45.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b4a63640 1 -- 172.21.15.88:0/2739397609 >> v1:172.21.15.88:6789/0 conn(0x7fa4b0150280 legacy=0x7fa4b006d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:45.058 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b4a63640 1 --2- 172.21.15.88:0/2739397609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa49c005b90 0x7fa49c005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:45.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b4a63640 1 --2- 172.21.15.88:0/2739397609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa49c0069e0 0x7fa49c006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:45.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b4a63640 1 --2- 172.21.15.88:0/2739397609 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4b0150660 0x7fa49c0073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:45.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b4a63640 1 -- 172.21.15.88:0/2739397609 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b00c5370 con 0x7fa49c0069e0 2023-10-16T20:58:45.059 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b4a63640 1 -- 172.21.15.88:0/2739397609 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b0156d10 con 0x7fa49c005b90 2023-10-16T20:58:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b4a63640 1 -- 172.21.15.88:0/2739397609 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c00dbd0 con 0x7fa4b0150660 2023-10-16T20:58:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b5264640 1 --2- 172.21.15.88:0/2739397609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa49c0069e0 0x7fa49c006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b5a65640 1 --2- 172.21.15.88:0/2739397609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa49c005b90 0x7fa49c005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b6266640 1 --2- 172.21.15.88:0/2739397609 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4b0150660 0x7fa49c0073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b5264640 1 -- 172.21.15.88:0/2739397609 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4b0150660 msgr2=0x7fa49c0073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b5264640 1 --2- 172.21.15.88:0/2739397609 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa4b0150660 0x7fa49c0073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b5264640 1 -- 172.21.15.88:0/2739397609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa49c005b90 msgr2=0x7fa49c005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b5264640 1 --2- 172.21.15.88:0/2739397609 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa49c005b90 0x7fa49c005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.050+0000 7fa4b5264640 1 -- 172.21.15.88:0/2739397609 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c005740 con 0x7fa49c0069e0 2023-10-16T20:58:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.054+0000 7fa4b5264640 1 --2- 172.21.15.88:0/2739397609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa49c0069e0 0x7fa49c006e60 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0009ae0 tx=0x7fa4a001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f1478bf75cb553 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:45.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.054+0000 7fa4b4a63640 1 -- 172.21.15.88:0/2739397609 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a00a6430 con 0x7fa49c0069e0 2023-10-16T20:58:45.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.054+0000 7fa4b4a63640 1 -- 172.21.15.88:0/2739397609 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a00a6590 con 0x7fa49c0069e0 2023-10-16T20:58:45.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.054+0000 7fa4b4a63640 1 -- 172.21.15.88:0/2739397609 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a00a6940 con 0x7fa49c0069e0 2023-10-16T20:58:45.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.054+0000 7fa4b7cc9640 1 -- 172.21.15.88:0/2739397609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa49c0069e0 msgr2=0x7fa49c006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:45.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.054+0000 7fa4b7cc9640 1 --2- 172.21.15.88:0/2739397609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa49c0069e0 0x7fa49c006e60 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0009ae0 tx=0x7fa4a001c9c0 comp rx=0 tx=0).stop 2023-10-16T20:58:45.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.054+0000 7fa4b6266640 1 -- 172.21.15.88:0/2739397609 reap_dead start 2023-10-16T20:58:45.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.054+0000 7fa4b7cc9640 1 -- 172.21.15.88:0/2739397609 shutdown_connections 2023-10-16T20:58:45.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.054+0000 7fa4b7cc9640 1 -- 172.21.15.88:0/2739397609 >> 172.21.15.88:0/2739397609 conn(0x7fa4b00cad50 msgr2=0x7fa4b00cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:45.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.054+0000 7fa4b7cc9640 1 -- 172.21.15.88:0/2739397609 shutdown_connections 2023-10-16T20:58:45.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.054+0000 7fa4b7cc9640 1 -- 172.21.15.88:0/2739397609 wait complete. 2023-10-16T20:58:45.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b7cc9640 1 Processor -- start 2023-10-16T20:58:45.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b7cc9640 1 -- start start 2023-10-16T20:58:45.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b7cc9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa49c0069e0 0x7fa4b01f2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:45.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b7cc9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4b015c2a0 0x7fa4b00c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:45.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b7cc9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4b0141810 0x7fa4b00ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:45.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b7cc9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b01f5d20 con 0x7fa4b0141810 2023-10-16T20:58:45.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b7cc9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b013c3c0 con 0x7fa4b015c2a0 2023-10-16T20:58:45.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b7cc9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b01f2540 con 0x7fa49c0069e0 2023-10-16T20:58:45.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b6266640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4b0141810 0x7fa4b00ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:45.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b6266640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4b0141810 0x7fa4b00ca3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54730/0 (socket says 172.21.15.88:54730) 2023-10-16T20:58:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b6266640 1 -- 172.21.15.88:0/3654872630 learned_addr learned my addr 172.21.15.88:0/3654872630 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b5a65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa49c0069e0 0x7fa4b01f2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:45.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b5264640 1 --2- 172.21.15.88:0/3654872630 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4b015c2a0 0x7fa4b00c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b6266640 1 -- 172.21.15.88:0/3654872630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa49c0069e0 msgr2=0x7fa4b01f2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:45.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b6266640 1 --2- 172.21.15.88:0/3654872630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa49c0069e0 0x7fa4b01f2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b6266640 1 -- 172.21.15.88:0/3654872630 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4b015c2a0 msgr2=0x7fa4b00c9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b6266640 1 --2- 172.21.15.88:0/3654872630 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4b015c2a0 0x7fa4b00c9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:45.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b6266640 1 -- 172.21.15.88:0/3654872630 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b013cb50 con 0x7fa4b0141810 2023-10-16T20:58:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b6266640 1 --2- 172.21.15.88:0/3654872630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4b0141810 0x7fa4b00ca3e0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fa4b006f100 tx=0x7fa4a800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4a67fc640 1 -- 172.21.15.88:0/3654872630 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a800e040 con 0x7fa4b0141810 2023-10-16T20:58:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4a67fc640 1 -- 172.21.15.88:0/3654872630 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a800f040 con 0x7fa4b0141810 2023-10-16T20:58:45.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b7cc9640 1 -- 172.21.15.88:0/3654872630 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b01f56a0 con 0x7fa4b0141810 2023-10-16T20:58:45.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4a67fc640 1 -- 172.21.15.88:0/3654872630 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8015670 con 0x7fa4b0141810 2023-10-16T20:58:45.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b7cc9640 1 -- 172.21.15.88:0/3654872630 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b013c870 con 0x7fa4b0141810 2023-10-16T20:58:45.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.058+0000 7fa4b7cc9640 1 -- 172.21.15.88:0/3654872630 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b014d920 con 0x7fa4b0141810 2023-10-16T20:58:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.062+0000 7fa4a67fc640 1 -- 172.21.15.88:0/3654872630 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa4a80157d0 con 0x7fa4b0141810 2023-10-16T20:58:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.062+0000 7fa4a67fc640 1 --2- 172.21.15.88:0/3654872630 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa4940664a0 0x7fa494068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.062+0000 7fa4a67fc640 1 -- 172.21.15.88:0/3654872630 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7fa4a8084e70 con 0x7fa4b0141810 2023-10-16T20:58:45.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.062+0000 7fa4b5a65640 1 --2- 172.21.15.88:0/3654872630 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa4940664a0 0x7fa494068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.062+0000 7fa4b5a65640 1 --2- 172.21.15.88:0/3654872630 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa4940664a0 0x7fa494068960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac001d00 tx=0x7fa4ac007900 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:45.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.066+0000 7fa4a67fc640 1 -- 172.21.15.88:0/3654872630 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa4a8051d10 con 0x7fa4b0141810 2023-10-16T20:58:45.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.222+0000 7fa4b7cc9640 1 -- 172.21.15.88:0/3654872630 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4b01f5e60 con 0x7fa4b0141810 2023-10-16T20:58:45.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.222+0000 7fa4a67fc640 1 -- 172.21.15.88:0/3654872630 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+20065 (secure 0 0 0) 0x7fa4b01f5e60 con 0x7fa4b0141810 2023-10-16T20:58:45.224 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:45.224 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":67,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:58:00.653255+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6839/2172973775":"2023-10-17T20:56:40.633674+0000","172.21.15.191:6833/3633326602":"2023-10-17T20:55:50.604940+0000","172.21.15.191:6836/1120825335":"2023-10-17T20:42:10.335721+0000","172.21.15.191:6832/3633326602":"2023-10-17T20:55:50.604940+0000","172.21.15.88:6835/518281867":"2023-10-17T20:44:25.369398+0000","172.21.15.191:6837/2468647900":"2023-10-17T20:53:25.553637+0000","172.21.15.88:6837/1987579860":"2023-10-17T20:43:25.353447+0000","172.21.15.191:6835/3119045295":"2023-10-17T20:58:00.653202+0000","172.21.15.88:6836/321173984":"2023-10-17T20:46:15.444657+0000","172.21.15.191:6837/1120825335":"2023-10-17T20:42:10.335721+0000","172.21.15.191:6832/2446850309":"2023-10-17T20:48:43.641901+0000","172.21.15.88:6837/2172973775":"2023-10-17T20:56:40.633674+0000","172.21.15.88:6839/1987579860":"2023-10-17T20:43:25.353447+0000","172.21.15.191:6834/3119045295":"2023-10-17T20:58:00.653202+0000","172.21.15.88:6834/518281867":"2023-10-17T20:44:25.369398+0000","172.21.15.88:6838/321173984":"2023-10-17T20:46:15.444657+0000","172.21.15.191:6834/3059699574":"2023-10-17T20:51:20.522638+0000","172.21.15.191:6833/2446850309":"2023-10-17T20:48:43.641901+0000","172.21.15.191:6835/3059699574":"2023-10-17T20:51:20.522638+0000","172.21.15.191:6836/2468647900":"2023-10-17T20:53:25.553637+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:58:45.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.222+0000 7fa4b7cc9640 1 -- 172.21.15.88:0/3654872630 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa4940664a0 msgr2=0x7fa494068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:45.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.222+0000 7fa4b7cc9640 1 --2- 172.21.15.88:0/3654872630 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa4940664a0 0x7fa494068960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac001d00 tx=0x7fa4ac007900 comp rx=0 tx=0).stop 2023-10-16T20:58:45.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.222+0000 7fa4b7cc9640 1 -- 172.21.15.88:0/3654872630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4b0141810 msgr2=0x7fa4b00ca3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:45.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.222+0000 7fa4b7cc9640 1 --2- 172.21.15.88:0/3654872630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4b0141810 0x7fa4b00ca3e0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fa4b006f100 tx=0x7fa4a800f580 comp rx=0 tx=0).stop 2023-10-16T20:58:45.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.222+0000 7fa4b7cc9640 1 -- 172.21.15.88:0/3654872630 shutdown_connections 2023-10-16T20:58:45.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.222+0000 7fa4b7cc9640 1 --2- 172.21.15.88:0/3654872630 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa4940664a0 0x7fa494068960 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:45.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.222+0000 7fa4b7cc9640 1 --2- 172.21.15.88:0/3654872630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4b0141810 0x7fa4b00ca3e0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:45.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.222+0000 7fa4b7cc9640 1 --2- 172.21.15.88:0/3654872630 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa4b015c2a0 0x7fa4b00c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:45.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.226+0000 7fa4b7cc9640 1 --2- 172.21.15.88:0/3654872630 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa49c0069e0 0x7fa4b01f2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:45.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.226+0000 7fa4b7cc9640 1 -- 172.21.15.88:0/3654872630 >> 172.21.15.88:0/3654872630 conn(0x7fa4b00cad50 msgr2=0x7fa4b00c3110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:45.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.226+0000 7fa4b7cc9640 1 -- 172.21.15.88:0/3654872630 shutdown_connections 2023-10-16T20:58:45.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:45.226+0000 7fa4b7cc9640 1 -- 172.21.15.88:0/3654872630 wait complete. 2023-10-16T20:58:46.246 INFO:tasks.ceph:Scrubbing osd.0 2023-10-16T20:58:46.247 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-16T20:58:46.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.394+0000 7f11fe965640 1 Processor -- start 2023-10-16T20:58:46.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.394+0000 7f11fe965640 1 -- start start 2023-10-16T20:58:46.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.398+0000 7f11fe965640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f11f80c4260 0x7f11f80c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.398+0000 7f11fe965640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11f80c4640 0x7f11f81573e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.398+0000 7f11fe965640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f8069e90 con 0x7f11f80c5a60 2023-10-16T20:58:46.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.398+0000 7f11fe965640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11f80716a0 con 0x7f11f80c4260 2023-10-16T20:58:46.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.398+0000 7f11fe965640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f806aea0 con 0x7f11f80c4640 2023-10-16T20:58:46.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.398+0000 7f11fcf02640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11f80c4640 0x7f11f81573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.398+0000 7f11f7fff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f11f80c4260 0x7f11f80c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.398+0000 7f11f77fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f11f80c5a60 0x7f11f81510d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:49634/0 (socket says 172.21.15.88:49634) 2023-10-16T20:58:46.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.398+0000 7f11f77fe640 1 -- 172.21.15.88:0/1071435425 learned_addr learned my addr 172.21.15.88:0/1071435425 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:46.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.398+0000 7f11f6ffd640 1 -- 172.21.15.88:0/1071435425 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416010025 0 0) 0x7f11f8069e90 con 0x7f11f80c5a60 2023-10-16T20:58:46.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.398+0000 7f11f6ffd640 1 -- 172.21.15.88:0/1071435425 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11e0003530 con 0x7f11f80c5a60 2023-10-16T20:58:46.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.398+0000 7f11f7fff640 1 -- 172.21.15.88:0/1071435425 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11f80c4640 msgr2=0x7f11f81573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:46.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.398+0000 7f11f7fff640 1 --2- 172.21.15.88:0/1071435425 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11f80c4640 0x7f11f81573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.403 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.398+0000 7f11f7fff640 1 -- 172.21.15.88:0/1071435425 >> v1:172.21.15.191:6789/0 conn(0x7f11f80c5a60 legacy=0x7f11f81510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.398+0000 7f11f7fff640 1 -- 172.21.15.88:0/1071435425 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f81581e0 con 0x7f11f80c4260 2023-10-16T20:58:46.404 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.398+0000 7f11fcf02640 1 --2- 172.21.15.88:0/1071435425 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11f80c4640 0x7f11f81573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:58:46.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.402+0000 7f11f7fff640 1 --2- 172.21.15.88:0/1071435425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11f80c4260 0x7f11f80c5520 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f11e80097c0 tx=0x7f11e801c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee7b012e612ce9b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:46.405 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.402+0000 7f11f6ffd640 1 -- 172.21.15.88:0/1071435425 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e8004230 con 0x7f11f80c4260 2023-10-16T20:58:46.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.402+0000 7f11f6ffd640 1 -- 172.21.15.88:0/1071435425 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e80a7050 con 0x7f11f80c4260 2023-10-16T20:58:46.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.402+0000 7f11f6ffd640 1 -- 172.21.15.88:0/1071435425 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e8003620 con 0x7f11f80c4260 2023-10-16T20:58:46.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.402+0000 7f11fe965640 1 -- 172.21.15.88:0/1071435425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11f80c4260 msgr2=0x7f11f80c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.402+0000 7f11fe965640 1 --2- 172.21.15.88:0/1071435425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11f80c4260 0x7f11f80c5520 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f11e80097c0 tx=0x7f11e801c330 comp rx=0 tx=0).stop 2023-10-16T20:58:46.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.402+0000 7f11fe965640 1 -- 172.21.15.88:0/1071435425 shutdown_connections 2023-10-16T20:58:46.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.402+0000 7f11fe965640 1 --2- 172.21.15.88:0/1071435425 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11f80c4640 0x7f11f81573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.402+0000 7f11fe965640 1 --2- 172.21.15.88:0/1071435425 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11f80c4260 0x7f11f80c5520 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.402+0000 7f11fe965640 1 -- 172.21.15.88:0/1071435425 >> 172.21.15.88:0/1071435425 conn(0x7f11f80ca9f0 msgr2=0x7f11f80cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:46.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.402+0000 7f11fe965640 1 -- 172.21.15.88:0/1071435425 shutdown_connections 2023-10-16T20:58:46.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.402+0000 7f11fe965640 1 -- 172.21.15.88:0/1071435425 wait complete. 2023-10-16T20:58:46.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.402+0000 7f11fe965640 1 Processor -- start 2023-10-16T20:58:46.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fe965640 1 -- start start 2023-10-16T20:58:46.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fe965640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11f80c4260 0x7f11f8140510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fe965640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11f80c4640 0x7f11f8141980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fe965640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11f80c5a60 0x7f11f8141ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fe965640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f81efcb0 con 0x7f11f80c5a60 2023-10-16T20:58:46.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fe965640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f81ef3a0 con 0x7f11f80c4640 2023-10-16T20:58:46.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fe965640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f813d680 con 0x7f11f80c4260 2023-10-16T20:58:46.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fcf02640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11f80c5a60 0x7f11f8141ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fcf02640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11f80c5a60 0x7f11f8141ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54744/0 (socket says 172.21.15.88:54744) 2023-10-16T20:58:46.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11f7fff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11f80c4260 0x7f11f8140510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fcf02640 1 -- 172.21.15.88:0/1522791848 learned_addr learned my addr 172.21.15.88:0/1522791848 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:46.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11f77fe640 1 --2- 172.21.15.88:0/1522791848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11f80c4640 0x7f11f8141980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fcf02640 1 -- 172.21.15.88:0/1522791848 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11f80c4260 msgr2=0x7f11f8140510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fcf02640 1 --2- 172.21.15.88:0/1522791848 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f11f80c4260 0x7f11f8140510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fcf02640 1 -- 172.21.15.88:0/1522791848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11f80c4640 msgr2=0x7f11f8141980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fcf02640 1 --2- 172.21.15.88:0/1522791848 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11f80c4640 0x7f11f8141980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fcf02640 1 -- 172.21.15.88:0/1522791848 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f0009a70 con 0x7f11f80c5a60 2023-10-16T20:58:46.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fcf02640 1 --2- 172.21.15.88:0/1522791848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11f80c5a60 0x7f11f8141ec0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f11f000efc0 tx=0x7f11f0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:46.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11f4ff9640 1 -- 172.21.15.88:0/1522791848 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f0009d70 con 0x7f11f80c5a60 2023-10-16T20:58:46.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11f4ff9640 1 -- 172.21.15.88:0/1522791848 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f000f040 con 0x7f11f80c5a60 2023-10-16T20:58:46.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11f4ff9640 1 -- 172.21.15.88:0/1522791848 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f00039a0 con 0x7f11f80c5a60 2023-10-16T20:58:46.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fe965640 1 -- 172.21.15.88:0/1522791848 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e8009420 con 0x7f11f80c5a60 2023-10-16T20:58:46.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fe965640 1 -- 172.21.15.88:0/1522791848 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f81ef940 con 0x7f11f80c5a60 2023-10-16T20:58:46.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.406+0000 7f11fe965640 1 -- 172.21.15.88:0/1522791848 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f11f806a980 con 0x7f11f80c5a60 2023-10-16T20:58:46.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.410+0000 7f11f4ff9640 1 -- 172.21.15.88:0/1522791848 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f11f00144a0 con 0x7f11f80c5a60 2023-10-16T20:58:46.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.410+0000 7f11f4ff9640 1 --2- 172.21.15.88:0/1522791848 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f11e40664a0 0x7f11e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.410+0000 7f11f4ff9640 1 -- 172.21.15.88:0/1522791848 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f11f0085150 con 0x7f11f80c5a60 2023-10-16T20:58:46.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.410+0000 7f11f7fff640 1 --2- 172.21.15.88:0/1522791848 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f11e40664a0 0x7f11e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.410+0000 7f11f4ff9640 1 --2- 172.21.15.88:0/1522791848 >> [v2:172.21.15.88:6800/1130215823,v1:172.21.15.88:6801/1130215823] conn(0x7f11e4070520 0x7f11e40729c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.410+0000 7f11f4ff9640 1 -- 172.21.15.88:0/1522791848 --> [v2:172.21.15.88:6800/1130215823,v1:172.21.15.88:6801/1130215823] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11e4073090 con 0x7f11e4070520 2023-10-16T20:58:46.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.410+0000 7f11f4ff9640 1 -- 172.21.15.88:0/1522791848 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11f00876d0 con 0x7f11f80c5a60 2023-10-16T20:58:46.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.410+0000 7f11f77fe640 1 --2- 172.21.15.88:0/1522791848 >> [v2:172.21.15.88:6800/1130215823,v1:172.21.15.88:6801/1130215823] conn(0x7f11e4070520 0x7f11e40729c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.410+0000 7f11f7fff640 1 --2- 172.21.15.88:0/1522791848 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f11e40664a0 0x7f11e4068960 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f11e80ad000 tx=0x7f11e80a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:46.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.410+0000 7f11f77fe640 1 --2- 172.21.15.88:0/1522791848 >> [v2:172.21.15.88:6800/1130215823,v1:172.21.15.88:6801/1130215823] conn(0x7f11e4070520 0x7f11e40729c0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:46.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.410+0000 7f11f4ff9640 1 -- 172.21.15.88:0/1522791848 <== osd.0 v2:172.21.15.88:6800/1130215823 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f11e4073090 con 0x7f11e4070520 2023-10-16T20:58:46.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.422+0000 7f11fe965640 1 -- 172.21.15.88:0/1522791848 --> [v2:172.21.15.88:6800/1130215823,v1:172.21.15.88:6801/1130215823] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f11f806a6a0 con 0x7f11e4070520 2023-10-16T20:58:46.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.426+0000 7f11f4ff9640 1 -- 172.21.15.88:0/1522791848 <== osd.0 v2:172.21.15.88:6800/1130215823 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f11ec006b50 con 0x7f11e4070520 2023-10-16T20:58:46.426 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-10-16T20:58:46.426 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T20:58:46.427 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-10-16T20:58:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.426+0000 7f11fe965640 1 -- 172.21.15.88:0/1522791848 >> [v2:172.21.15.88:6800/1130215823,v1:172.21.15.88:6801/1130215823] conn(0x7f11e4070520 msgr2=0x7f11e40729c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.426+0000 7f11fe965640 1 --2- 172.21.15.88:0/1522791848 >> [v2:172.21.15.88:6800/1130215823,v1:172.21.15.88:6801/1130215823] conn(0x7f11e4070520 0x7f11e40729c0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.426+0000 7f11fe965640 1 -- 172.21.15.88:0/1522791848 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f11e40664a0 msgr2=0x7f11e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.426+0000 7f11fe965640 1 --2- 172.21.15.88:0/1522791848 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f11e40664a0 0x7f11e4068960 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f11e80ad000 tx=0x7f11e80a6000 comp rx=0 tx=0).stop 2023-10-16T20:58:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.426+0000 7f11fe965640 1 -- 172.21.15.88:0/1522791848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11f80c5a60 msgr2=0x7f11f8141ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.426+0000 7f11fe965640 1 --2- 172.21.15.88:0/1522791848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11f80c5a60 0x7f11f8141ec0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f11f000efc0 tx=0x7f11f0009f90 comp rx=0 tx=0).stop 2023-10-16T20:58:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.426+0000 7f11fcf02640 1 -- 172.21.15.88:0/1522791848 reap_dead start 2023-10-16T20:58:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.426+0000 7f11fe965640 1 -- 172.21.15.88:0/1522791848 shutdown_connections 2023-10-16T20:58:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.426+0000 7f11fe965640 1 -- 172.21.15.88:0/1522791848 >> 172.21.15.88:0/1522791848 conn(0x7f11f80ca9f0 msgr2=0x7f11f806f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.426+0000 7f11fe965640 1 -- 172.21.15.88:0/1522791848 shutdown_connections 2023-10-16T20:58:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.426+0000 7f11fe965640 1 -- 172.21.15.88:0/1522791848 wait complete. 2023-10-16T20:58:46.444 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-16T20:58:46.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.582+0000 7feba6846640 1 Processor -- start 2023-10-16T20:58:46.585 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.582+0000 7feba6846640 1 -- start start 2023-10-16T20:58:46.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.582+0000 7feba6846640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7feba0153780 0x7feba00c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.586 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.582+0000 7feba6846640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7feba01598c0 0x7feba00ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.582+0000 7feba6846640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba0159d90 con 0x7feba0159490 2023-10-16T20:58:46.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.582+0000 7feba6846640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba015a070 con 0x7feba0153780 2023-10-16T20:58:46.587 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.582+0000 7feba6846640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba015a350 con 0x7feba01598c0 2023-10-16T20:58:46.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.586+0000 7feb9ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7feba0153780 0x7feba00c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.586+0000 7feb9f7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7feba0159490 0x7feba00c53f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59588/0 (socket says 172.21.15.88:59588) 2023-10-16T20:58:46.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.586+0000 7feb9f7fe640 1 -- 172.21.15.88:0/2991528514 learned_addr learned my addr 172.21.15.88:0/2991528514 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:46.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.586+0000 7feba4de3640 1 --2- 172.21.15.88:0/2991528514 >> v2:172.21.15.191:3300/0 conn(0x7feba01598c0 0x7feba00ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.586+0000 7feb9ffff640 1 -- 172.21.15.88:0/2991528514 >> v2:172.21.15.191:3300/0 conn(0x7feba01598c0 msgr2=0x7feba00ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:46.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.586+0000 7feb9ffff640 1 --2- 172.21.15.88:0/2991528514 >> v2:172.21.15.191:3300/0 conn(0x7feba01598c0 0x7feba00ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.586+0000 7feb9ffff640 1 -- 172.21.15.88:0/2991528514 >> v1:172.21.15.88:6789/0 conn(0x7feba0159490 legacy=0x7feba00c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.586+0000 7feb9ffff640 1 -- 172.21.15.88:0/2991528514 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba00c4910 con 0x7feba0153780 2023-10-16T20:58:46.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.586+0000 7feba4de3640 1 --2- 172.21.15.88:0/2991528514 >> v2:172.21.15.191:3300/0 conn(0x7feba01598c0 0x7feba00ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:58:46.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.586+0000 7feb9ffff640 1 --2- 172.21.15.88:0/2991528514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba0153780 0x7feba00c9be0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7feba00b4fc0 tx=0x7feb9001ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fbbde82ef2c0f21 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:46.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.586+0000 7feb9effd640 1 -- 172.21.15.88:0/2991528514 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb900a8930 con 0x7feba0153780 2023-10-16T20:58:46.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.586+0000 7feb9effd640 1 -- 172.21.15.88:0/2991528514 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb900a8a90 con 0x7feba0153780 2023-10-16T20:58:46.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.586+0000 7feb9effd640 1 -- 172.21.15.88:0/2991528514 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb900a8e20 con 0x7feba0153780 2023-10-16T20:58:46.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feba6846640 1 -- 172.21.15.88:0/2991528514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba0153780 msgr2=0x7feba00c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feba6846640 1 --2- 172.21.15.88:0/2991528514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba0153780 0x7feba00c9be0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7feba00b4fc0 tx=0x7feb9001ec30 comp rx=0 tx=0).stop 2023-10-16T20:58:46.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feba6846640 1 -- 172.21.15.88:0/2991528514 shutdown_connections 2023-10-16T20:58:46.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feba6846640 1 --2- 172.21.15.88:0/2991528514 >> v2:172.21.15.191:3300/0 conn(0x7feba01598c0 0x7feba00ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feba6846640 1 --2- 172.21.15.88:0/2991528514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba0153780 0x7feba00c9be0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feba6846640 1 -- 172.21.15.88:0/2991528514 >> 172.21.15.88:0/2991528514 conn(0x7feba006bcb0 msgr2=0x7feba006c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:46.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feba6846640 1 -- 172.21.15.88:0/2991528514 shutdown_connections 2023-10-16T20:58:46.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feba6846640 1 -- 172.21.15.88:0/2991528514 wait complete. 2023-10-16T20:58:46.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feba6846640 1 Processor -- start 2023-10-16T20:58:46.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feba6846640 1 -- start start 2023-10-16T20:58:46.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feba6846640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feba0153780 0x7feba0147de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feba6846640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba0159490 0x7feba014f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feba6846640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feba01598c0 0x7feba0152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feba6846640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba0146460 con 0x7feba0159490 2023-10-16T20:58:46.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feba6846640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba0158a40 con 0x7feba01598c0 2023-10-16T20:58:46.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feba6846640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba0158d20 con 0x7feba0153780 2023-10-16T20:58:46.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feb9f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba0159490 0x7feba014f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feb9f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba0159490 0x7feba014f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54770/0 (socket says 172.21.15.88:54770) 2023-10-16T20:58:46.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feb9f7fe640 1 -- 172.21.15.88:0/2883037893 learned_addr learned my addr 172.21.15.88:0/2883037893 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feb9ffff640 1 --2- 172.21.15.88:0/2883037893 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feba0153780 0x7feba0147de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feba4de3640 1 --2- 172.21.15.88:0/2883037893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feba01598c0 0x7feba0152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feb9f7fe640 1 -- 172.21.15.88:0/2883037893 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feba0153780 msgr2=0x7feba0147de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feb9f7fe640 1 --2- 172.21.15.88:0/2883037893 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feba0153780 0x7feba0147de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feb9f7fe640 1 -- 172.21.15.88:0/2883037893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feba01598c0 msgr2=0x7feba0152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feb9f7fe640 1 --2- 172.21.15.88:0/2883037893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feba01598c0 0x7feba0152260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.590+0000 7feb9f7fe640 1 -- 172.21.15.88:0/2883037893 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb900093c0 con 0x7feba0159490 2023-10-16T20:58:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.594+0000 7feb9f7fe640 1 --2- 172.21.15.88:0/2883037893 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba0159490 0x7feba014f410 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7feb9400c500 tx=0x7feb9400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.594+0000 7feb9cff9640 1 -- 172.21.15.88:0/2883037893 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94005cf0 con 0x7feba0159490 2023-10-16T20:58:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.594+0000 7feb9cff9640 1 -- 172.21.15.88:0/2883037893 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb94005e50 con 0x7feba0159490 2023-10-16T20:58:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.594+0000 7feba6846640 1 -- 172.21.15.88:0/2883037893 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba0159000 con 0x7feba0159490 2023-10-16T20:58:46.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.594+0000 7feb9cff9640 1 -- 172.21.15.88:0/2883037893 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb940072f0 con 0x7feba0159490 2023-10-16T20:58:46.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.594+0000 7feba6846640 1 -- 172.21.15.88:0/2883037893 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba01527a0 con 0x7feba0159490 2023-10-16T20:58:46.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.594+0000 7feba6846640 1 -- 172.21.15.88:0/2883037893 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba0074700 con 0x7feba0159490 2023-10-16T20:58:46.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.594+0000 7feb9cff9640 1 -- 172.21.15.88:0/2883037893 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feb94007450 con 0x7feba0159490 2023-10-16T20:58:46.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.594+0000 7feb9cff9640 1 --2- 172.21.15.88:0/2883037893 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7feb8c0664a0 0x7feb8c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.594+0000 7feb9cff9640 1 -- 172.21.15.88:0/2883037893 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7feb940839a0 con 0x7feba0159490 2023-10-16T20:58:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.594+0000 7feb9ffff640 1 --2- 172.21.15.88:0/2883037893 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7feb8c0664a0 0x7feb8c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.594+0000 7feb9ffff640 1 --2- 172.21.15.88:0/2883037893 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7feb8c0664a0 0x7feb8c068960 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7feb900b2000 tx=0x7feb90008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.598+0000 7feb9cff9640 1 -- 172.21.15.88:0/2883037893 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feb94050840 con 0x7feba0159490 2023-10-16T20:58:46.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.754+0000 7feba6846640 1 -- 172.21.15.88:0/2883037893 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7feba0158bb0 con 0x7feb8c0664a0 2023-10-16T20:58:46.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.754+0000 7feb9cff9640 1 -- 172.21.15.88:0/2883037893 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7feba0158bb0 con 0x7feb8c0664a0 2023-10-16T20:58:46.758 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 0 to deep-scrub 2023-10-16T20:58:46.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.758+0000 7feba6846640 1 -- 172.21.15.88:0/2883037893 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7feb8c0664a0 msgr2=0x7feb8c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.758+0000 7feba6846640 1 --2- 172.21.15.88:0/2883037893 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7feb8c0664a0 0x7feb8c068960 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7feb900b2000 tx=0x7feb90008040 comp rx=0 tx=0).stop 2023-10-16T20:58:46.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.758+0000 7feba6846640 1 -- 172.21.15.88:0/2883037893 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba0159490 msgr2=0x7feba014f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.758+0000 7feba6846640 1 --2- 172.21.15.88:0/2883037893 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba0159490 0x7feba014f410 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7feb9400c500 tx=0x7feb9400c9c0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.758+0000 7feba6846640 1 -- 172.21.15.88:0/2883037893 shutdown_connections 2023-10-16T20:58:46.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.758+0000 7feba6846640 1 --2- 172.21.15.88:0/2883037893 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7feb8c0664a0 0x7feb8c068960 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.758+0000 7feba6846640 1 --2- 172.21.15.88:0/2883037893 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feba01598c0 0x7feba0152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.758+0000 7feba6846640 1 --2- 172.21.15.88:0/2883037893 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feba0159490 0x7feba014f410 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.758+0000 7feba6846640 1 --2- 172.21.15.88:0/2883037893 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7feba0153780 0x7feba0147de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.758+0000 7feba6846640 1 -- 172.21.15.88:0/2883037893 >> 172.21.15.88:0/2883037893 conn(0x7feba006bcb0 msgr2=0x7feba00ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:46.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.758+0000 7feba6846640 1 -- 172.21.15.88:0/2883037893 shutdown_connections 2023-10-16T20:58:46.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.758+0000 7feba6846640 1 -- 172.21.15.88:0/2883037893 wait complete. 2023-10-16T20:58:46.779 INFO:tasks.ceph:Scrubbing osd.1 2023-10-16T20:58:46.779 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-16T20:58:46.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a7e09640 1 Processor -- start 2023-10-16T20:58:46.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a7e09640 1 -- start start 2023-10-16T20:58:46.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a7e09640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f48a0153620 0x7f48a00c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a7e09640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48a0153a00 0x7f48a00c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a7e09640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a00c0230 con 0x7f48a00ad870 2023-10-16T20:58:46.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a7e09640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a01594f0 con 0x7f48a0153620 2023-10-16T20:58:46.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a7e09640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a00c0480 con 0x7f48a0153a00 2023-10-16T20:58:46.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a63a6640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f48a00ad870 0x7f48a00cb280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59592/0 (socket says 172.21.15.88:59592) 2023-10-16T20:58:46.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a63a6640 1 -- 172.21.15.88:0/1575760173 learned_addr learned my addr 172.21.15.88:0/1575760173 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:46.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a53a4640 1 --2- 172.21.15.88:0/1575760173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48a0153a00 0x7f48a00c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a5ba5640 1 --2- 172.21.15.88:0/1575760173 >> v2:172.21.15.191:3300/0 conn(0x7f48a0153620 0x7f48a00c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a4ba3640 1 -- 172.21.15.88:0/1575760173 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305252036 0 0) 0x7f48a00c0230 con 0x7f48a00ad870 2023-10-16T20:58:46.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a4ba3640 1 -- 172.21.15.88:0/1575760173 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f488c003530 con 0x7f48a00ad870 2023-10-16T20:58:46.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a4ba3640 1 -- 172.21.15.88:0/1575760173 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 377125067 0 0) 0x7f488c003530 con 0x7f48a00ad870 2023-10-16T20:58:46.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a4ba3640 1 -- 172.21.15.88:0/1575760173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48a0153a00 msgr2=0x7f48a00c61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:46.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a4ba3640 1 --2- 172.21.15.88:0/1575760173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48a0153a00 0x7f48a00c61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a4ba3640 1 -- 172.21.15.88:0/1575760173 >> v2:172.21.15.191:3300/0 conn(0x7f48a0153620 msgr2=0x7f48a00c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:46.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a4ba3640 1 --2- 172.21.15.88:0/1575760173 >> v2:172.21.15.191:3300/0 conn(0x7f48a0153620 0x7f48a00c5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a4ba3640 1 -- 172.21.15.88:0/1575760173 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a00c0760 con 0x7f48a00ad870 2023-10-16T20:58:46.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a4ba3640 1 -- 172.21.15.88:0/1575760173 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f4898001a00 con 0x7f48a00ad870 2023-10-16T20:58:46.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a4ba3640 1 -- 172.21.15.88:0/1575760173 >> v1:172.21.15.88:6789/0 conn(0x7f48a00ad870 legacy=0x7f48a00cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a4ba3640 1 --2- 172.21.15.88:0/1575760173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48a0153a00 0x7f488c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a4ba3640 1 --2- 172.21.15.88:0/1575760173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f488c003f00 0x7f488c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a4ba3640 1 --2- 172.21.15.88:0/1575760173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f488c004d10 0x7f488c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a4ba3640 1 -- 172.21.15.88:0/1575760173 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a01594f0 con 0x7f488c004d10 2023-10-16T20:58:46.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a4ba3640 1 -- 172.21.15.88:0/1575760173 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a00c0480 con 0x7f488c003f00 2023-10-16T20:58:46.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a4ba3640 1 -- 172.21.15.88:0/1575760173 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488c00b990 con 0x7f48a0153a00 2023-10-16T20:58:46.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a63a6640 1 --2- 172.21.15.88:0/1575760173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f488c004d10 0x7f488c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a53a4640 1 --2- 172.21.15.88:0/1575760173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f488c003f00 0x7f488c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a5ba5640 1 --2- 172.21.15.88:0/1575760173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48a0153a00 0x7f488c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a63a6640 1 -- 172.21.15.88:0/1575760173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48a0153a00 msgr2=0x7f488c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:46.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a63a6640 1 --2- 172.21.15.88:0/1575760173 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48a0153a00 0x7f488c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a63a6640 1 -- 172.21.15.88:0/1575760173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f488c003f00 msgr2=0x7f488c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:46.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a63a6640 1 --2- 172.21.15.88:0/1575760173 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f488c003f00 0x7f488c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.922+0000 7f48a63a6640 1 -- 172.21.15.88:0/1575760173 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c003530 con 0x7f488c004d10 2023-10-16T20:58:46.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.926+0000 7f48a63a6640 1 --2- 172.21.15.88:0/1575760173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f488c004d10 0x7f488c005190 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f489800a830 tx=0x7f489801d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea9a3cf13dc8b371 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:46.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.926+0000 7f48a4ba3640 1 -- 172.21.15.88:0/1575760173 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48980ad630 con 0x7f488c004d10 2023-10-16T20:58:46.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.926+0000 7f48a4ba3640 1 -- 172.21.15.88:0/1575760173 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48980a8070 con 0x7f488c004d10 2023-10-16T20:58:46.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.926+0000 7f48a4ba3640 1 -- 172.21.15.88:0/1575760173 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48980ad9d0 con 0x7f488c004d10 2023-10-16T20:58:46.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.926+0000 7f48a7e09640 1 -- 172.21.15.88:0/1575760173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f488c004d10 msgr2=0x7f488c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.926+0000 7f48a7e09640 1 --2- 172.21.15.88:0/1575760173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f488c004d10 0x7f488c005190 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f489800a830 tx=0x7f489801d2e0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.926+0000 7f48a63a6640 1 -- 172.21.15.88:0/1575760173 reap_dead start 2023-10-16T20:58:46.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.926+0000 7f48a7e09640 1 -- 172.21.15.88:0/1575760173 shutdown_connections 2023-10-16T20:58:46.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.926+0000 7f48a7e09640 1 -- 172.21.15.88:0/1575760173 >> 172.21.15.88:0/1575760173 conn(0x7f48a0069e90 msgr2=0x7f48a006a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:46.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.926+0000 7f48a7e09640 1 -- 172.21.15.88:0/1575760173 shutdown_connections 2023-10-16T20:58:46.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.926+0000 7f48a7e09640 1 -- 172.21.15.88:0/1575760173 wait complete. 2023-10-16T20:58:46.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a7e09640 1 Processor -- start 2023-10-16T20:58:46.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a7e09640 1 -- start start 2023-10-16T20:58:46.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a7e09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f488c004d10 0x7f48a013bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a7e09640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48a0158030 0x7f48a015a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a7e09640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48a015abf0 0x7f48a00c4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a7e09640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a013c4b0 con 0x7f488c004d10 2023-10-16T20:58:46.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a7e09640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a00c3e30 con 0x7f48a0158030 2023-10-16T20:58:46.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a7e09640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a00c3a30 con 0x7f48a015abf0 2023-10-16T20:58:46.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a5ba5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f488c004d10 0x7f48a013bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a5ba5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f488c004d10 0x7f48a013bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54792/0 (socket says 172.21.15.88:54792) 2023-10-16T20:58:46.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a5ba5640 1 -- 172.21.15.88:0/3625240957 learned_addr learned my addr 172.21.15.88:0/3625240957 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:46.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a63a6640 1 --2- 172.21.15.88:0/3625240957 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48a015abf0 0x7f48a00c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a53a4640 1 --2- 172.21.15.88:0/3625240957 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48a0158030 0x7f48a015a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a5ba5640 1 -- 172.21.15.88:0/3625240957 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48a015abf0 msgr2=0x7f48a00c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a5ba5640 1 --2- 172.21.15.88:0/3625240957 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48a015abf0 0x7f48a00c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a5ba5640 1 -- 172.21.15.88:0/3625240957 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48a0158030 msgr2=0x7f48a015a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a5ba5640 1 --2- 172.21.15.88:0/3625240957 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48a0158030 0x7f48a015a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a5ba5640 1 -- 172.21.15.88:0/3625240957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a015d8d0 con 0x7f488c004d10 2023-10-16T20:58:46.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a63a6640 1 --2- 172.21.15.88:0/3625240957 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f48a015abf0 0x7f48a00c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:58:46.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a5ba5640 1 --2- 172.21.15.88:0/3625240957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f488c004d10 0x7f48a013bf70 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f489c011aa0 tx=0x7f489c011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:46.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48967fc640 1 -- 172.21.15.88:0/3625240957 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c00fb80 con 0x7f488c004d10 2023-10-16T20:58:46.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48967fc640 1 -- 172.21.15.88:0/3625240957 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489c00fce0 con 0x7f488c004d10 2023-10-16T20:58:46.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a7e09640 1 -- 172.21.15.88:0/3625240957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a00c2900 con 0x7f488c004d10 2023-10-16T20:58:46.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48967fc640 1 -- 172.21.15.88:0/3625240957 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c014680 con 0x7f488c004d10 2023-10-16T20:58:46.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.930+0000 7f48a7e09640 1 -- 172.21.15.88:0/3625240957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a015d280 con 0x7f488c004d10 2023-10-16T20:58:46.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.934+0000 7f48a7e09640 1 -- 172.21.15.88:0/3625240957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f48a00c0d20 con 0x7f488c004d10 2023-10-16T20:58:46.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.934+0000 7f48967fc640 1 -- 172.21.15.88:0/3625240957 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f489c0147e0 con 0x7f488c004d10 2023-10-16T20:58:46.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.934+0000 7f48967fc640 1 --2- 172.21.15.88:0/3625240957 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f48840664a0 0x7f4884068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.934+0000 7f48967fc640 1 -- 172.21.15.88:0/3625240957 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f489c084ff0 con 0x7f488c004d10 2023-10-16T20:58:46.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.934+0000 7f48a53a4640 1 --2- 172.21.15.88:0/3625240957 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f48840664a0 0x7f4884068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.934+0000 7f48967fc640 1 --2- 172.21.15.88:0/3625240957 >> [v2:172.21.15.88:6804/3386972440,v1:172.21.15.88:6809/3386972440] conn(0x7f4884070520 0x7f48840729c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:46.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.934+0000 7f48967fc640 1 -- 172.21.15.88:0/3625240957 --> [v2:172.21.15.88:6804/3386972440,v1:172.21.15.88:6809/3386972440] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4884073090 con 0x7f4884070520 2023-10-16T20:58:46.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.934+0000 7f48967fc640 1 -- 172.21.15.88:0/3625240957 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f489c0875f0 con 0x7f488c004d10 2023-10-16T20:58:46.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.934+0000 7f48a63a6640 1 --2- 172.21.15.88:0/3625240957 >> [v2:172.21.15.88:6804/3386972440,v1:172.21.15.88:6809/3386972440] conn(0x7f4884070520 0x7f48840729c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:46.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.934+0000 7f48a53a4640 1 --2- 172.21.15.88:0/3625240957 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f48840664a0 0x7f4884068960 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f48a013e6c0 tx=0x7f4890008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:46.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.934+0000 7f48a63a6640 1 --2- 172.21.15.88:0/3625240957 >> [v2:172.21.15.88:6804/3386972440,v1:172.21.15.88:6809/3386972440] conn(0x7f4884070520 0x7f48840729c0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:46.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.938+0000 7f48967fc640 1 -- 172.21.15.88:0/3625240957 <== osd.1 v2:172.21.15.88:6804/3386972440 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f4884073090 con 0x7f4884070520 2023-10-16T20:58:46.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.950+0000 7f48a7e09640 1 -- 172.21.15.88:0/3625240957 --> [v2:172.21.15.88:6804/3386972440,v1:172.21.15.88:6809/3386972440] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f48a00c0a40 con 0x7f4884070520 2023-10-16T20:58:46.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.950+0000 7f48967fc640 1 -- 172.21.15.88:0/3625240957 <== osd.1 v2:172.21.15.88:6804/3386972440 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f4898002000 con 0x7f4884070520 2023-10-16T20:58:46.953 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-10-16T20:58:46.953 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T20:58:46.953 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-10-16T20:58:46.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.950+0000 7f48a7e09640 1 -- 172.21.15.88:0/3625240957 >> [v2:172.21.15.88:6804/3386972440,v1:172.21.15.88:6809/3386972440] conn(0x7f4884070520 msgr2=0x7f48840729c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.950+0000 7f48a7e09640 1 --2- 172.21.15.88:0/3625240957 >> [v2:172.21.15.88:6804/3386972440,v1:172.21.15.88:6809/3386972440] conn(0x7f4884070520 0x7f48840729c0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:46.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.950+0000 7f48a7e09640 1 -- 172.21.15.88:0/3625240957 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f48840664a0 msgr2=0x7f4884068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.950+0000 7f48a7e09640 1 --2- 172.21.15.88:0/3625240957 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f48840664a0 0x7f4884068960 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f48a013e6c0 tx=0x7f4890008040 comp rx=0 tx=0).stop 2023-10-16T20:58:46.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.950+0000 7f48a7e09640 1 -- 172.21.15.88:0/3625240957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f488c004d10 msgr2=0x7f48a013bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:46.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.950+0000 7f48a7e09640 1 --2- 172.21.15.88:0/3625240957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f488c004d10 0x7f48a013bf70 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f489c011aa0 tx=0x7f489c011f60 comp rx=0 tx=0).stop 2023-10-16T20:58:46.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.950+0000 7f48a63a6640 1 -- 172.21.15.88:0/3625240957 reap_dead start 2023-10-16T20:58:46.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.950+0000 7f48a7e09640 1 -- 172.21.15.88:0/3625240957 shutdown_connections 2023-10-16T20:58:46.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.950+0000 7f48a7e09640 1 -- 172.21.15.88:0/3625240957 >> 172.21.15.88:0/3625240957 conn(0x7f48a0069e90 msgr2=0x7f48a00ca780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:46.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.950+0000 7f48a7e09640 1 -- 172.21.15.88:0/3625240957 shutdown_connections 2023-10-16T20:58:46.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:46.950+0000 7f48a7e09640 1 -- 172.21.15.88:0/3625240957 wait complete. 2023-10-16T20:58:46.969 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-16T20:58:47.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.106+0000 7fa5398d6640 1 Processor -- start 2023-10-16T20:58:47.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.106+0000 7fa5398d6640 1 -- start start 2023-10-16T20:58:47.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.106+0000 7fa5398d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa5340c9eb0 0x7fa5340ca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.106+0000 7fa5398d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5340cb5e0 0x7fa53415cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.106+0000 7fa5398d6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fa53415d090 0x7fa53406a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.106+0000 7fa5398d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5340c5630 con 0x7fa5340c9eb0 2023-10-16T20:58:47.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.106+0000 7fa5398d6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5340ca7d0 con 0x7fa53415d090 2023-10-16T20:58:47.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.106+0000 7fa5398d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa534153b70 con 0x7fa5340cb5e0 2023-10-16T20:58:47.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.106+0000 7fa532ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa5340c9eb0 0x7fa5340ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.106+0000 7fa532ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa5340c9eb0 0x7fa5340ca290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54796/0 (socket says 172.21.15.88:54796) 2023-10-16T20:58:47.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.106+0000 7fa532ffd640 1 -- 172.21.15.88:0/1143138977 learned_addr learned my addr 172.21.15.88:0/1143138977 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:47.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.106+0000 7fa532ffd640 1 -- 172.21.15.88:0/1143138977 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5340cb5e0 msgr2=0x7fa53415cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:47.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.106+0000 7fa532ffd640 1 --2- 172.21.15.88:0/1143138977 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5340cb5e0 0x7fa53415cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.106+0000 7fa532ffd640 1 -- 172.21.15.88:0/1143138977 >> v2:172.21.15.191:3300/0 conn(0x7fa53415d090 msgr2=0x7fa53406a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:47.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.106+0000 7fa532ffd640 1 --2- 172.21.15.88:0/1143138977 >> v2:172.21.15.191:3300/0 conn(0x7fa53415d090 0x7fa53406a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.106+0000 7fa532ffd640 1 -- 172.21.15.88:0/1143138977 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53415a180 con 0x7fa5340c9eb0 2023-10-16T20:58:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.110+0000 7fa532ffd640 1 --2- 172.21.15.88:0/1143138977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5340c9eb0 0x7fa5340ca290 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fa5340cca40 tx=0x7fa52401c240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bcc952df97b7087 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:47.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.110+0000 7fa5327fc640 1 -- 172.21.15.88:0/1143138977 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5240ab600 con 0x7fa5340c9eb0 2023-10-16T20:58:47.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.110+0000 7fa5327fc640 1 -- 172.21.15.88:0/1143138977 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5240a7050 con 0x7fa5340c9eb0 2023-10-16T20:58:47.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.110+0000 7fa5327fc640 1 -- 172.21.15.88:0/1143138977 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5240ab980 con 0x7fa5340c9eb0 2023-10-16T20:58:47.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.110+0000 7fa5398d6640 1 -- 172.21.15.88:0/1143138977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5340c9eb0 msgr2=0x7fa5340ca290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.110+0000 7fa5398d6640 1 --2- 172.21.15.88:0/1143138977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5340c9eb0 0x7fa5340ca290 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fa5340cca40 tx=0x7fa52401c240 comp rx=0 tx=0).stop 2023-10-16T20:58:47.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.110+0000 7fa5398d6640 1 -- 172.21.15.88:0/1143138977 shutdown_connections 2023-10-16T20:58:47.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.110+0000 7fa5398d6640 1 --2- 172.21.15.88:0/1143138977 >> v2:172.21.15.191:3300/0 conn(0x7fa53415d090 0x7fa53406a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.110+0000 7fa5398d6640 1 --2- 172.21.15.88:0/1143138977 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5340cb5e0 0x7fa53415cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.110+0000 7fa5398d6640 1 --2- 172.21.15.88:0/1143138977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5340c9eb0 0x7fa5340ca290 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.110+0000 7fa5398d6640 1 -- 172.21.15.88:0/1143138977 >> 172.21.15.88:0/1143138977 conn(0x7fa53406bc00 msgr2=0x7fa53406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:47.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.110+0000 7fa5398d6640 1 -- 172.21.15.88:0/1143138977 shutdown_connections 2023-10-16T20:58:47.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.110+0000 7fa5398d6640 1 -- 172.21.15.88:0/1143138977 wait complete. 2023-10-16T20:58:47.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5398d6640 1 Processor -- start 2023-10-16T20:58:47.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5398d6640 1 -- start start 2023-10-16T20:58:47.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5398d6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5340c9eb0 0x7fa53415c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5398d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5340cb5e0 0x7fa534155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5398d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa53415d090 0x7fa534155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5398d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa534154e00 con 0x7fa53415d090 2023-10-16T20:58:47.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5398d6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa534155ff0 con 0x7fa5340c9eb0 2023-10-16T20:58:47.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5398d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5341f1f60 con 0x7fa5340cb5e0 2023-10-16T20:58:47.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5337fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa53415d090 0x7fa534155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5337fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa53415d090 0x7fa534155760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54808/0 (socket says 172.21.15.88:54808) 2023-10-16T20:58:47.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5337fe640 1 -- 172.21.15.88:0/1065482054 learned_addr learned my addr 172.21.15.88:0/1065482054 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:47.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa532ffd640 1 --2- 172.21.15.88:0/1065482054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5340c9eb0 0x7fa53415c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa52bfff640 1 --2- 172.21.15.88:0/1065482054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5340cb5e0 0x7fa534155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5337fe640 1 -- 172.21.15.88:0/1065482054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5340cb5e0 msgr2=0x7fa534155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5337fe640 1 --2- 172.21.15.88:0/1065482054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5340cb5e0 0x7fa534155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5337fe640 1 -- 172.21.15.88:0/1065482054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5340c9eb0 msgr2=0x7fa53415c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5337fe640 1 --2- 172.21.15.88:0/1065482054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5340c9eb0 0x7fa53415c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5337fe640 1 -- 172.21.15.88:0/1065482054 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5240094d0 con 0x7fa53415d090 2023-10-16T20:58:47.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5337fe640 1 --2- 172.21.15.88:0/1065482054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa53415d090 0x7fa534155760 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fa52c00b970 tx=0x7fa52c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:47.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa530ff9640 1 -- 172.21.15.88:0/1065482054 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c00cb40 con 0x7fa53415d090 2023-10-16T20:58:47.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa530ff9640 1 -- 172.21.15.88:0/1065482054 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52c00ccc0 con 0x7fa53415d090 2023-10-16T20:58:47.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5398d6640 1 -- 172.21.15.88:0/1065482054 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5341f2200 con 0x7fa53415d090 2023-10-16T20:58:47.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa530ff9640 1 -- 172.21.15.88:0/1065482054 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c014680 con 0x7fa53415d090 2023-10-16T20:58:47.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5398d6640 1 -- 172.21.15.88:0/1065482054 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5341f27c0 con 0x7fa53415d090 2023-10-16T20:58:47.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.114+0000 7fa5398d6640 1 -- 172.21.15.88:0/1065482054 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa534154f40 con 0x7fa53415d090 2023-10-16T20:58:47.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.118+0000 7fa530ff9640 1 -- 172.21.15.88:0/1065482054 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa52c01d3b0 con 0x7fa53415d090 2023-10-16T20:58:47.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.118+0000 7fa530ff9640 1 --2- 172.21.15.88:0/1065482054 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa5140664a0 0x7fa514068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.118+0000 7fa530ff9640 1 -- 172.21.15.88:0/1065482054 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7fa52c0847f0 con 0x7fa53415d090 2023-10-16T20:58:47.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.118+0000 7fa532ffd640 1 --2- 172.21.15.88:0/1065482054 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa5140664a0 0x7fa514068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.118+0000 7fa532ffd640 1 --2- 172.21.15.88:0/1065482054 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa5140664a0 0x7fa514068960 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa5240ad000 tx=0x7fa5240a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:47.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.122+0000 7fa530ff9640 1 -- 172.21.15.88:0/1065482054 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa52c051550 con 0x7fa53415d090 2023-10-16T20:58:47.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.270+0000 7fa5398d6640 1 -- 172.21.15.88:0/1065482054 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa534159850 con 0x7fa5140664a0 2023-10-16T20:58:47.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.270+0000 7fa530ff9640 1 -- 172.21.15.88:0/1065482054 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa5240a4e90 con 0x7fa5140664a0 2023-10-16T20:58:47.273 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 1 to deep-scrub 2023-10-16T20:58:47.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.274+0000 7fa5398d6640 1 -- 172.21.15.88:0/1065482054 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa5140664a0 msgr2=0x7fa514068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.274+0000 7fa5398d6640 1 --2- 172.21.15.88:0/1065482054 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa5140664a0 0x7fa514068960 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa5240ad000 tx=0x7fa5240a6000 comp rx=0 tx=0).stop 2023-10-16T20:58:47.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.274+0000 7fa5398d6640 1 -- 172.21.15.88:0/1065482054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa53415d090 msgr2=0x7fa534155760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.274+0000 7fa5398d6640 1 --2- 172.21.15.88:0/1065482054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa53415d090 0x7fa534155760 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fa52c00b970 tx=0x7fa52c00be30 comp rx=0 tx=0).stop 2023-10-16T20:58:47.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.274+0000 7fa5398d6640 1 -- 172.21.15.88:0/1065482054 shutdown_connections 2023-10-16T20:58:47.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.274+0000 7fa5398d6640 1 --2- 172.21.15.88:0/1065482054 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa5140664a0 0x7fa514068960 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.274+0000 7fa5398d6640 1 --2- 172.21.15.88:0/1065482054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa53415d090 0x7fa534155760 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.274+0000 7fa5398d6640 1 --2- 172.21.15.88:0/1065482054 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa5340cb5e0 0x7fa534155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.274+0000 7fa5398d6640 1 --2- 172.21.15.88:0/1065482054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5340c9eb0 0x7fa53415c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.274+0000 7fa5398d6640 1 -- 172.21.15.88:0/1065482054 >> 172.21.15.88:0/1065482054 conn(0x7fa53406bc00 msgr2=0x7fa53406bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:47.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.274+0000 7fa5398d6640 1 -- 172.21.15.88:0/1065482054 shutdown_connections 2023-10-16T20:58:47.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.274+0000 7fa5398d6640 1 -- 172.21.15.88:0/1065482054 wait complete. 2023-10-16T20:58:47.295 INFO:tasks.ceph:Scrubbing osd.2 2023-10-16T20:58:47.295 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-16T20:58:47.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c92c9640 1 Processor -- start 2023-10-16T20:58:47.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c92c9640 1 -- start start 2023-10-16T20:58:47.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c92c9640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f97c40ca9c0 0x7f97c40c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c92c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c40c4f70 con 0x7f97c41534f0 2023-10-16T20:58:47.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c92c9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c415d800 con 0x7f97c4159330 2023-10-16T20:58:47.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c92c9640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c40cade0 con 0x7f97c40ca9c0 2023-10-16T20:58:47.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c2ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f97c41534f0 0x7f97c415c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59598/0 (socket says 172.21.15.88:59598) 2023-10-16T20:58:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c2ffd640 1 -- 172.21.15.88:0/836120046 learned_addr learned my addr 172.21.15.88:0/836120046 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c37fe640 1 --2- 172.21.15.88:0/836120046 >> v2:172.21.15.191:3300/0 conn(0x7f97c40ca9c0 0x7f97c40c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c1ffb640 1 -- 172.21.15.88:0/836120046 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690669996 0 0) 0x7f97c40c4f70 con 0x7f97c41534f0 2023-10-16T20:58:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c1ffb640 1 -- 172.21.15.88:0/836120046 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97ac003530 con 0x7f97c41534f0 2023-10-16T20:58:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c1ffb640 1 -- 172.21.15.88:0/836120046 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3541707610 0 0) 0x7f97ac003530 con 0x7f97c41534f0 2023-10-16T20:58:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c1ffb640 1 -- 172.21.15.88:0/836120046 >> v2:172.21.15.191:3300/0 conn(0x7f97c40ca9c0 msgr2=0x7f97c40c4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c1ffb640 1 --2- 172.21.15.88:0/836120046 >> v2:172.21.15.191:3300/0 conn(0x7f97c40ca9c0 0x7f97c40c4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c1ffb640 1 -- 172.21.15.88:0/836120046 >> v1:172.21.15.191:6789/0 conn(0x7f97c4159330 legacy=0x7f97c40c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c1ffb640 1 -- 172.21.15.88:0/836120046 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c415d240 con 0x7f97c41534f0 2023-10-16T20:58:47.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c1ffb640 1 -- 172.21.15.88:0/836120046 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f97b4001680 con 0x7f97c41534f0 2023-10-16T20:58:47.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c1ffb640 1 -- 172.21.15.88:0/836120046 >> v1:172.21.15.88:6789/0 conn(0x7f97c41534f0 legacy=0x7f97c415c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c1ffb640 1 --2- 172.21.15.88:0/836120046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97ac003980 0x7f97ac003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c1ffb640 1 --2- 172.21.15.88:0/836120046 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97ac0047d0 0x7f97ac004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c1ffb640 1 --2- 172.21.15.88:0/836120046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97ac005190 0x7f97ac00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c1ffb640 1 -- 172.21.15.88:0/836120046 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c40cade0 con 0x7f97ac005190 2023-10-16T20:58:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c1ffb640 1 -- 172.21.15.88:0/836120046 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac00c030 con 0x7f97ac003980 2023-10-16T20:58:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c1ffb640 1 -- 172.21.15.88:0/836120046 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac00c1f0 con 0x7f97ac0047d0 2023-10-16T20:58:47.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c37fe640 1 --2- 172.21.15.88:0/836120046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97ac005190 0x7f97ac00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c2ffd640 1 --2- 172.21.15.88:0/836120046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97ac003980 0x7f97ac003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c27fc640 1 --2- 172.21.15.88:0/836120046 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97ac0047d0 0x7f97ac004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c37fe640 1 -- 172.21.15.88:0/836120046 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97ac0047d0 msgr2=0x7f97ac004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:47.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c37fe640 1 --2- 172.21.15.88:0/836120046 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97ac0047d0 0x7f97ac004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c37fe640 1 -- 172.21.15.88:0/836120046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97ac003980 msgr2=0x7f97ac003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:47.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c37fe640 1 --2- 172.21.15.88:0/836120046 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97ac003980 0x7f97ac003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.442+0000 7f97c37fe640 1 -- 172.21.15.88:0/836120046 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ac003530 con 0x7f97ac005190 2023-10-16T20:58:47.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.446+0000 7f97c37fe640 1 -- 172.21.15.88:0/836120046 reap_dead start 2023-10-16T20:58:47.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.446+0000 7f97c37fe640 1 --2- 172.21.15.88:0/836120046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97ac005190 0x7f97ac00b990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009fc0 tx=0x7f97bc01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e51680ccbc91849 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:47.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.446+0000 7f97c1ffb640 1 -- 172.21.15.88:0/836120046 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc0a5cb0 con 0x7f97ac005190 2023-10-16T20:58:47.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.446+0000 7f97c1ffb640 1 -- 172.21.15.88:0/836120046 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97bc00f040 con 0x7f97ac005190 2023-10-16T20:58:47.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.446+0000 7f97c1ffb640 1 -- 172.21.15.88:0/836120046 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc003be0 con 0x7f97ac005190 2023-10-16T20:58:47.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.446+0000 7f97c92c9640 1 -- 172.21.15.88:0/836120046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97ac005190 msgr2=0x7f97ac00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.446+0000 7f97c92c9640 1 --2- 172.21.15.88:0/836120046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97ac005190 0x7f97ac00b990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009fc0 tx=0x7f97bc01cca0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.446+0000 7f97c92c9640 1 -- 172.21.15.88:0/836120046 shutdown_connections 2023-10-16T20:58:47.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.446+0000 7f97c92c9640 1 --2- 172.21.15.88:0/836120046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97ac005190 0x7f97ac00b990 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.446+0000 7f97c92c9640 1 -- 172.21.15.88:0/836120046 >> 172.21.15.88:0/836120046 conn(0x7f97c406bbc0 msgr2=0x7f97c406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.446+0000 7f97c92c9640 1 -- 172.21.15.88:0/836120046 shutdown_connections 2023-10-16T20:58:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.446+0000 7f97c92c9640 1 -- 172.21.15.88:0/836120046 wait complete. 2023-10-16T20:58:47.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c92c9640 1 Processor -- start 2023-10-16T20:58:47.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c92c9640 1 -- start start 2023-10-16T20:58:47.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c92c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97ac005190 0x7f97c414e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c92c9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97c4148f40 0x7f97c4158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c92c9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97c41579d0 0x7f97c40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c92c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c40cd120 con 0x7f97ac005190 2023-10-16T20:58:47.463 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c92c9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c40739a0 con 0x7f97c4148f40 2023-10-16T20:58:47.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c92c9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c41f20f0 con 0x7f97c41579d0 2023-10-16T20:58:47.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97ac005190 0x7f97c414e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97ac005190 0x7f97c414e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54820/0 (socket says 172.21.15.88:54820) 2023-10-16T20:58:47.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c2ffd640 1 -- 172.21.15.88:0/2519173324 learned_addr learned my addr 172.21.15.88:0/2519173324 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c37fe640 1 --2- 172.21.15.88:0/2519173324 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97c41579d0 0x7f97c40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c27fc640 1 --2- 172.21.15.88:0/2519173324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97c4148f40 0x7f97c4158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c2ffd640 1 -- 172.21.15.88:0/2519173324 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97c41579d0 msgr2=0x7f97c40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c2ffd640 1 --2- 172.21.15.88:0/2519173324 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f97c41579d0 0x7f97c40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c2ffd640 1 -- 172.21.15.88:0/2519173324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97c4148f40 msgr2=0x7f97c4158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c2ffd640 1 --2- 172.21.15.88:0/2519173324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f97c4148f40 0x7f97c4158050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c2ffd640 1 -- 172.21.15.88:0/2519173324 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c40cd9f0 con 0x7f97ac005190 2023-10-16T20:58:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c2ffd640 1 --2- 172.21.15.88:0/2519173324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97ac005190 0x7f97c414e9c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f97b4007290 tx=0x7f97b4007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97ab7fe640 1 -- 172.21.15.88:0/2519173324 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b400f960 con 0x7f97ac005190 2023-10-16T20:58:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97ab7fe640 1 -- 172.21.15.88:0/2519173324 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b400fae0 con 0x7f97ac005190 2023-10-16T20:58:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97ab7fe640 1 -- 172.21.15.88:0/2519173324 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b400fe90 con 0x7f97ac005190 2023-10-16T20:58:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c92c9640 1 -- 172.21.15.88:0/2519173324 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c414f7d0 con 0x7f97ac005190 2023-10-16T20:58:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.450+0000 7f97c92c9640 1 -- 172.21.15.88:0/2519173324 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c414f4f0 con 0x7f97ac005190 2023-10-16T20:58:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.454+0000 7f97c92c9640 1 -- 172.21.15.88:0/2519173324 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f97c40c4230 con 0x7f97ac005190 2023-10-16T20:58:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.454+0000 7f97ab7fe640 1 -- 172.21.15.88:0/2519173324 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f97b401d460 con 0x7f97ac005190 2023-10-16T20:58:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.454+0000 7f97ab7fe640 1 --2- 172.21.15.88:0/2519173324 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f97b00664a0 0x7f97b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.454+0000 7f97ab7fe640 1 -- 172.21.15.88:0/2519173324 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f97b4084430 con 0x7f97ac005190 2023-10-16T20:58:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.454+0000 7f97c27fc640 1 --2- 172.21.15.88:0/2519173324 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f97b00664a0 0x7f97b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.454+0000 7f97ab7fe640 1 --2- 172.21.15.88:0/2519173324 >> [v2:172.21.15.88:6802/360619264,v1:172.21.15.88:6805/360619264] conn(0x7f97b0070520 0x7f97b00729c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.454+0000 7f97ab7fe640 1 -- 172.21.15.88:0/2519173324 --> [v2:172.21.15.88:6802/360619264,v1:172.21.15.88:6805/360619264] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97b0073090 con 0x7f97b0070520 2023-10-16T20:58:47.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.454+0000 7f97ab7fe640 1 -- 172.21.15.88:0/2519173324 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97b4086a10 con 0x7f97ac005190 2023-10-16T20:58:47.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.454+0000 7f97c37fe640 1 --2- 172.21.15.88:0/2519173324 >> [v2:172.21.15.88:6802/360619264,v1:172.21.15.88:6805/360619264] conn(0x7f97b0070520 0x7f97b00729c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.454+0000 7f97c27fc640 1 --2- 172.21.15.88:0/2519173324 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f97b00664a0 0x7f97b0068960 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f97c41ef4c0 tx=0x7f97b8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:47.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.454+0000 7f97c37fe640 1 --2- 172.21.15.88:0/2519173324 >> [v2:172.21.15.88:6802/360619264,v1:172.21.15.88:6805/360619264] conn(0x7f97b0070520 0x7f97b00729c0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:47.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.454+0000 7f97ab7fe640 1 -- 172.21.15.88:0/2519173324 <== osd.2 v2:172.21.15.88:6802/360619264 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f97b0073090 con 0x7f97b0070520 2023-10-16T20:58:47.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.470+0000 7f97c92c9640 1 -- 172.21.15.88:0/2519173324 --> [v2:172.21.15.88:6802/360619264,v1:172.21.15.88:6805/360619264] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f97c40c3f50 con 0x7f97b0070520 2023-10-16T20:58:47.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.470+0000 7f97ab7fe640 1 -- 172.21.15.88:0/2519173324 <== osd.2 v2:172.21.15.88:6802/360619264 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f97bc0a6be0 con 0x7f97b0070520 2023-10-16T20:58:47.473 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-10-16T20:58:47.473 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T20:58:47.474 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-10-16T20:58:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.470+0000 7f97c92c9640 1 -- 172.21.15.88:0/2519173324 >> [v2:172.21.15.88:6802/360619264,v1:172.21.15.88:6805/360619264] conn(0x7f97b0070520 msgr2=0x7f97b00729c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.470+0000 7f97c92c9640 1 --2- 172.21.15.88:0/2519173324 >> [v2:172.21.15.88:6802/360619264,v1:172.21.15.88:6805/360619264] conn(0x7f97b0070520 0x7f97b00729c0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.470+0000 7f97c92c9640 1 -- 172.21.15.88:0/2519173324 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f97b00664a0 msgr2=0x7f97b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.470+0000 7f97c92c9640 1 --2- 172.21.15.88:0/2519173324 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f97b00664a0 0x7f97b0068960 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f97c41ef4c0 tx=0x7f97b8009040 comp rx=0 tx=0).stop 2023-10-16T20:58:47.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.470+0000 7f97c92c9640 1 -- 172.21.15.88:0/2519173324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97ac005190 msgr2=0x7f97c414e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.470+0000 7f97c92c9640 1 --2- 172.21.15.88:0/2519173324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97ac005190 0x7f97c414e9c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f97b4007290 tx=0x7f97b4007750 comp rx=0 tx=0).stop 2023-10-16T20:58:47.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.470+0000 7f97c37fe640 1 -- 172.21.15.88:0/2519173324 reap_dead start 2023-10-16T20:58:47.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.470+0000 7f97c92c9640 1 -- 172.21.15.88:0/2519173324 shutdown_connections 2023-10-16T20:58:47.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.470+0000 7f97c92c9640 1 -- 172.21.15.88:0/2519173324 >> 172.21.15.88:0/2519173324 conn(0x7f97c406bbc0 msgr2=0x7f97c4150950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:47.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.470+0000 7f97c92c9640 1 -- 172.21.15.88:0/2519173324 shutdown_connections 2023-10-16T20:58:47.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.470+0000 7f97c92c9640 1 -- 172.21.15.88:0/2519173324 wait complete. 2023-10-16T20:58:47.490 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-16T20:58:47.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.626+0000 7f689546b640 1 Processor -- start 2023-10-16T20:58:47.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.626+0000 7f689546b640 1 -- start start 2023-10-16T20:58:47.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.626+0000 7f689546b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68900c66e0 0x7f689015a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.626+0000 7f689546b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f68900c6ac0 0x7f6890150150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.626+0000 7f689546b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6890070f40 con 0x7f68900cb5c0 2023-10-16T20:58:47.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.626+0000 7f689546b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f689015a790 con 0x7f68900c6ac0 2023-10-16T20:58:47.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.626+0000 7f689546b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f689006a2d0 con 0x7f68900c66e0 2023-10-16T20:58:47.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.626+0000 7f68867fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f68900c6ac0 0x7f6890150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.626+0000 7f68867fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f68900c6ac0 0x7f6890150150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54826/0 (socket says 172.21.15.88:54826) 2023-10-16T20:58:47.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.626+0000 7f68867fc640 1 -- 172.21.15.88:0/2193280981 learned_addr learned my addr 172.21.15.88:0/2193280981 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:47.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.626+0000 7f688effd640 1 --2- 172.21.15.88:0/2193280981 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68900c66e0 0x7f689015a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.630+0000 7f68867fc640 1 -- 172.21.15.88:0/2193280981 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68900c66e0 msgr2=0x7f689015a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:47.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.630+0000 7f68867fc640 1 --2- 172.21.15.88:0/2193280981 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68900c66e0 0x7f689015a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.630+0000 7f68867fc640 1 -- 172.21.15.88:0/2193280981 >> v1:172.21.15.191:6789/0 conn(0x7f68900cb5c0 legacy=0x7f6890071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.630+0000 7f68867fc640 1 -- 172.21.15.88:0/2193280981 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689006a5b0 con 0x7f68900c6ac0 2023-10-16T20:58:47.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.630+0000 7f68867fc640 1 --2- 172.21.15.88:0/2193280981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68900c6ac0 0x7f6890150150 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6878009870 tx=0x7f687801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edeac2a2677930a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:47.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.630+0000 7f688e7fc640 1 -- 172.21.15.88:0/2193280981 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68780ac600 con 0x7f68900c6ac0 2023-10-16T20:58:47.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.630+0000 7f688e7fc640 1 -- 172.21.15.88:0/2193280981 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68780a8050 con 0x7f68900c6ac0 2023-10-16T20:58:47.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.630+0000 7f688e7fc640 1 -- 172.21.15.88:0/2193280981 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68780ac980 con 0x7f68900c6ac0 2023-10-16T20:58:47.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.630+0000 7f689546b640 1 -- 172.21.15.88:0/2193280981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68900c6ac0 msgr2=0x7f6890150150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.630+0000 7f689546b640 1 --2- 172.21.15.88:0/2193280981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68900c6ac0 0x7f6890150150 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6878009870 tx=0x7f687801c4b0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.630+0000 7f689546b640 1 -- 172.21.15.88:0/2193280981 shutdown_connections 2023-10-16T20:58:47.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.630+0000 7f689546b640 1 --2- 172.21.15.88:0/2193280981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68900c6ac0 0x7f6890150150 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f689546b640 1 --2- 172.21.15.88:0/2193280981 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68900c66e0 0x7f689015a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f689546b640 1 -- 172.21.15.88:0/2193280981 >> 172.21.15.88:0/2193280981 conn(0x7f68900ca9f0 msgr2=0x7f68900cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:47.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f689546b640 1 -- 172.21.15.88:0/2193280981 shutdown_connections 2023-10-16T20:58:47.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f689546b640 1 -- 172.21.15.88:0/2193280981 wait complete. 2023-10-16T20:58:47.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f689546b640 1 Processor -- start 2023-10-16T20:58:47.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f689546b640 1 -- start start 2023-10-16T20:58:47.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f689546b640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68900c66e0 0x7f6890154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f689546b640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68900c6ac0 0x7f68901569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f689546b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68900cb5c0 0x7f6890162cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f689546b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6890156ee0 con 0x7f68900cb5c0 2023-10-16T20:58:47.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f689546b640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68901570e0 con 0x7f68900c6ac0 2023-10-16T20:58:47.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f689546b640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68901f4f40 con 0x7f68900c66e0 2023-10-16T20:58:47.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f688f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68900cb5c0 0x7f6890162cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f688f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68900cb5c0 0x7f6890162cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54832/0 (socket says 172.21.15.88:54832) 2023-10-16T20:58:47.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f688f7fe640 1 -- 172.21.15.88:0/1673129466 learned_addr learned my addr 172.21.15.88:0/1673129466 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:47.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f68867fc640 1 --2- 172.21.15.88:0/1673129466 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68900c6ac0 0x7f68901569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f688effd640 1 --2- 172.21.15.88:0/1673129466 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68900c66e0 0x7f6890154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f688f7fe640 1 -- 172.21.15.88:0/1673129466 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68900c66e0 msgr2=0x7f6890154f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f688f7fe640 1 --2- 172.21.15.88:0/1673129466 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68900c66e0 0x7f6890154f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f688f7fe640 1 -- 172.21.15.88:0/1673129466 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68900c6ac0 msgr2=0x7f68901569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f688f7fe640 1 --2- 172.21.15.88:0/1673129466 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68900c6ac0 0x7f68901569a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f688f7fe640 1 -- 172.21.15.88:0/1673129466 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68780094d0 con 0x7f68900cb5c0 2023-10-16T20:58:47.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.634+0000 7f688f7fe640 1 --2- 172.21.15.88:0/1673129466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68900cb5c0 0x7f6890162cd0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f688800c730 tx=0x7f688800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:47.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.638+0000 7f6887fff640 1 -- 172.21.15.88:0/1673129466 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6888003b80 con 0x7f68900cb5c0 2023-10-16T20:58:47.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.638+0000 7f6887fff640 1 -- 172.21.15.88:0/1673129466 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6888003ce0 con 0x7f68900cb5c0 2023-10-16T20:58:47.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.638+0000 7f689546b640 1 -- 172.21.15.88:0/1673129466 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68901f5220 con 0x7f68900cb5c0 2023-10-16T20:58:47.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.638+0000 7f6887fff640 1 -- 172.21.15.88:0/1673129466 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688800aa30 con 0x7f68900cb5c0 2023-10-16T20:58:47.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.638+0000 7f689546b640 1 -- 172.21.15.88:0/1673129466 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68901f57e0 con 0x7f68900cb5c0 2023-10-16T20:58:47.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.638+0000 7f689546b640 1 -- 172.21.15.88:0/1673129466 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6890074700 con 0x7f68900cb5c0 2023-10-16T20:58:47.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.638+0000 7f6887fff640 1 -- 172.21.15.88:0/1673129466 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f688800ab90 con 0x7f68900cb5c0 2023-10-16T20:58:47.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.638+0000 7f6887fff640 1 --2- 172.21.15.88:0/1673129466 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f68700664a0 0x7f6870068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.638+0000 7f6887fff640 1 -- 172.21.15.88:0/1673129466 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f6888084310 con 0x7f68900cb5c0 2023-10-16T20:58:47.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.638+0000 7f688effd640 1 --2- 172.21.15.88:0/1673129466 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f68700664a0 0x7f6870068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.638+0000 7f688effd640 1 --2- 172.21.15.88:0/1673129466 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f68700664a0 0x7f6870068960 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6880009b60 tx=0x7f6880009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:47.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.642+0000 7f6887fff640 1 -- 172.21.15.88:0/1673129466 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6888051010 con 0x7f68900cb5c0 2023-10-16T20:58:47.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.798+0000 7f689546b640 1 -- 172.21.15.88:0/1673129466 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f689006ae50 con 0x7f68700664a0 2023-10-16T20:58:47.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.798+0000 7f6887fff640 1 -- 172.21.15.88:0/1673129466 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6880011ac0 con 0x7f68700664a0 2023-10-16T20:58:47.799 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 2 to deep-scrub 2023-10-16T20:58:47.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.798+0000 7f689546b640 1 -- 172.21.15.88:0/1673129466 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f68700664a0 msgr2=0x7f6870068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.798+0000 7f689546b640 1 --2- 172.21.15.88:0/1673129466 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f68700664a0 0x7f6870068960 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6880009b60 tx=0x7f6880009190 comp rx=0 tx=0).stop 2023-10-16T20:58:47.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.798+0000 7f689546b640 1 -- 172.21.15.88:0/1673129466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68900cb5c0 msgr2=0x7f6890162cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.798+0000 7f689546b640 1 --2- 172.21.15.88:0/1673129466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68900cb5c0 0x7f6890162cd0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f688800c730 tx=0x7f688800cbf0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.798+0000 7f689546b640 1 -- 172.21.15.88:0/1673129466 shutdown_connections 2023-10-16T20:58:47.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.798+0000 7f689546b640 1 --2- 172.21.15.88:0/1673129466 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f68700664a0 0x7f6870068960 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.798+0000 7f689546b640 1 --2- 172.21.15.88:0/1673129466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68900cb5c0 0x7f6890162cd0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.798+0000 7f689546b640 1 --2- 172.21.15.88:0/1673129466 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68900c6ac0 0x7f68901569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.798+0000 7f689546b640 1 --2- 172.21.15.88:0/1673129466 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f68900c66e0 0x7f6890154f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.798+0000 7f689546b640 1 -- 172.21.15.88:0/1673129466 >> 172.21.15.88:0/1673129466 conn(0x7f68900ca9f0 msgr2=0x7f68900c8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:47.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.798+0000 7f689546b640 1 -- 172.21.15.88:0/1673129466 shutdown_connections 2023-10-16T20:58:47.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.798+0000 7f689546b640 1 -- 172.21.15.88:0/1673129466 wait complete. 2023-10-16T20:58:47.820 INFO:tasks.ceph:Scrubbing osd.3 2023-10-16T20:58:47.821 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-16T20:58:47.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.962+0000 7fd4aa53e640 1 Processor -- start 2023-10-16T20:58:47.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.962+0000 7fd4aa53e640 1 -- start start 2023-10-16T20:58:47.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.962+0000 7fd4aa53e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd4a40c2320 0x7fd4a4157e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.962+0000 7fd4aa53e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4a406a730 0x7fd4a406aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.962+0000 7fd4aa53e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4a4158390 con 0x7fd4a40c2700 2023-10-16T20:58:47.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.962+0000 7fd4aa53e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4a406bd50 con 0x7fd4a40c2320 2023-10-16T20:58:47.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.962+0000 7fd4aa53e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a40c1740 con 0x7fd4a406a730 2023-10-16T20:58:47.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.962+0000 7fd4a3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd4a40c2320 0x7fd4a4157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.962+0000 7fd4a3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd4a40c2320 0x7fd4a4157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54842/0 (socket says 172.21.15.88:54842) 2023-10-16T20:58:47.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.962+0000 7fd4a3fff640 1 -- 172.21.15.88:0/3452770369 learned_addr learned my addr 172.21.15.88:0/3452770369 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:47.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.962+0000 7fd4a8adb640 1 --2- 172.21.15.88:0/3452770369 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4a406a730 0x7fd4a406aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.962+0000 7fd4a3fff640 1 -- 172.21.15.88:0/3452770369 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4a406a730 msgr2=0x7fd4a406aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:47.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.962+0000 7fd4a3fff640 1 --2- 172.21.15.88:0/3452770369 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4a406a730 0x7fd4a406aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.962+0000 7fd4a3fff640 1 -- 172.21.15.88:0/3452770369 >> v1:172.21.15.88:6789/0 conn(0x7fd4a40c2700 legacy=0x7fd4a40c0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.962+0000 7fd4a3fff640 1 -- 172.21.15.88:0/3452770369 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a40c1a20 con 0x7fd4a40c2320 2023-10-16T20:58:47.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.962+0000 7fd4a8adb640 1 --2- 172.21.15.88:0/3452770369 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4a406a730 0x7fd4a406aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:58:47.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.966+0000 7fd4a3fff640 1 --2- 172.21.15.88:0/3452770369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4a40c2320 0x7fd4a4157e50 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fd494009870 tx=0x7fd49401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fbb161648633e63 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:47.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.966+0000 7fd4a2ffd640 1 -- 172.21.15.88:0/3452770369 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4940a8a30 con 0x7fd4a40c2320 2023-10-16T20:58:47.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.966+0000 7fd4a2ffd640 1 -- 172.21.15.88:0/3452770369 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4940a8b90 con 0x7fd4a40c2320 2023-10-16T20:58:47.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.966+0000 7fd4a2ffd640 1 -- 172.21.15.88:0/3452770369 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4940b0440 con 0x7fd4a40c2320 2023-10-16T20:58:47.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.966+0000 7fd4aa53e640 1 -- 172.21.15.88:0/3452770369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4a40c2320 msgr2=0x7fd4a4157e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.966+0000 7fd4aa53e640 1 --2- 172.21.15.88:0/3452770369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4a40c2320 0x7fd4a4157e50 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fd494009870 tx=0x7fd49401ed30 comp rx=0 tx=0).stop 2023-10-16T20:58:47.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.966+0000 7fd4aa53e640 1 -- 172.21.15.88:0/3452770369 shutdown_connections 2023-10-16T20:58:47.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.966+0000 7fd4aa53e640 1 --2- 172.21.15.88:0/3452770369 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4a406a730 0x7fd4a406aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.966+0000 7fd4aa53e640 1 --2- 172.21.15.88:0/3452770369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4a40c2320 0x7fd4a4157e50 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.966+0000 7fd4aa53e640 1 -- 172.21.15.88:0/3452770369 >> 172.21.15.88:0/3452770369 conn(0x7fd4a40c86b0 msgr2=0x7fd4a40c8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:47.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.966+0000 7fd4aa53e640 1 -- 172.21.15.88:0/3452770369 shutdown_connections 2023-10-16T20:58:47.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.966+0000 7fd4aa53e640 1 -- 172.21.15.88:0/3452770369 wait complete. 2023-10-16T20:58:47.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4aa53e640 1 Processor -- start 2023-10-16T20:58:47.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4aa53e640 1 -- start start 2023-10-16T20:58:47.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4aa53e640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4a406a730 0x7fd4a4071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4aa53e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4a40c2320 0x7fd4a40ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4aa53e640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4a40c2700 0x7fd4a414a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4aa53e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a41570f0 con 0x7fd4a40c2320 2023-10-16T20:58:47.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4aa53e640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a41ef810 con 0x7fd4a40c2700 2023-10-16T20:58:47.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4aa53e640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a4156e10 con 0x7fd4a406a730 2023-10-16T20:58:47.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4a37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4a40c2320 0x7fd4a40ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4a37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4a40c2320 0x7fd4a40ca940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54858/0 (socket says 172.21.15.88:54858) 2023-10-16T20:58:47.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4a37fe640 1 -- 172.21.15.88:0/238988047 learned_addr learned my addr 172.21.15.88:0/238988047 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:47.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4a8adb640 1 --2- 172.21.15.88:0/238988047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4a40c2700 0x7fd4a414a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4a3fff640 1 --2- 172.21.15.88:0/238988047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4a406a730 0x7fd4a4071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4a37fe640 1 -- 172.21.15.88:0/238988047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4a406a730 msgr2=0x7fd4a4071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4a37fe640 1 --2- 172.21.15.88:0/238988047 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd4a406a730 0x7fd4a4071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4a37fe640 1 -- 172.21.15.88:0/238988047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4a40c2700 msgr2=0x7fd4a414a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:47.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4a37fe640 1 --2- 172.21.15.88:0/238988047 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd4a40c2700 0x7fd4a414a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:47.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4a37fe640 1 -- 172.21.15.88:0/238988047 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4940094d0 con 0x7fd4a40c2320 2023-10-16T20:58:47.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4a37fe640 1 --2- 172.21.15.88:0/238988047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4a40c2320 0x7fd4a40ca940 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fd49800c3c0 tx=0x7fd49800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:47.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4a0ff9640 1 -- 172.21.15.88:0/238988047 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4980052d0 con 0x7fd4a40c2320 2023-10-16T20:58:47.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4a0ff9640 1 -- 172.21.15.88:0/238988047 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd498005430 con 0x7fd4a40c2320 2023-10-16T20:58:47.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4aa53e640 1 -- 172.21.15.88:0/238988047 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a41eea50 con 0x7fd4a40c2320 2023-10-16T20:58:47.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4a0ff9640 1 -- 172.21.15.88:0/238988047 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd498003840 con 0x7fd4a40c2320 2023-10-16T20:58:47.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4aa53e640 1 -- 172.21.15.88:0/238988047 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a41ef010 con 0x7fd4a40c2320 2023-10-16T20:58:47.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.970+0000 7fd4aa53e640 1 -- 172.21.15.88:0/238988047 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd4a406a210 con 0x7fd4a40c2320 2023-10-16T20:58:47.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.974+0000 7fd4a0ff9640 1 -- 172.21.15.88:0/238988047 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd4980039a0 con 0x7fd4a40c2320 2023-10-16T20:58:47.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.974+0000 7fd4a0ff9640 1 --2- 172.21.15.88:0/238988047 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd4900664a0 0x7fd490068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.974+0000 7fd4a0ff9640 1 -- 172.21.15.88:0/238988047 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7fd498083b80 con 0x7fd4a40c2320 2023-10-16T20:58:47.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.974+0000 7fd4a3fff640 1 --2- 172.21.15.88:0/238988047 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd4900664a0 0x7fd490068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:47.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.974+0000 7fd4a0ff9640 1 --2- 172.21.15.88:0/238988047 >> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] conn(0x7fd490070520 0x7fd4900729c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:47.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.974+0000 7fd4a0ff9640 1 -- 172.21.15.88:0/238988047 --> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd490073090 con 0x7fd490070520 2023-10-16T20:58:47.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.974+0000 7fd4a0ff9640 1 -- 172.21.15.88:0/238988047 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4980882a0 con 0x7fd4a40c2320 2023-10-16T20:58:47.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.974+0000 7fd4a8adb640 1 -- 172.21.15.88:0/238988047 >> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] conn(0x7fd490070520 msgr2=0x7fd4900729c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-16T20:58:47.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.974+0000 7fd4a8adb640 1 -- 172.21.15.88:0/238988047 >> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] conn(0x7fd490070520 msgr2=0x7fd4900729c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-16T20:58:47.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.974+0000 7fd4a8adb640 1 --2- 172.21.15.88:0/238988047 >> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] conn(0x7fd490070520 0x7fd4900729c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2023-10-16T20:58:47.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.974+0000 7fd4a8adb640 1 --2- 172.21.15.88:0/238988047 >> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] conn(0x7fd490070520 0x7fd4900729c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-16T20:58:47.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:47.974+0000 7fd4a3fff640 1 --2- 172.21.15.88:0/238988047 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd4900664a0 0x7fd490068960 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd4940b2000 tx=0x7fd494008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:48.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.178+0000 7fd4a8adb640 1 --2- 172.21.15.88:0/238988047 >> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] conn(0x7fd490070520 0x7fd4900729c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.178+0000 7fd4a8adb640 1 --2- 172.21.15.88:0/238988047 >> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] conn(0x7fd490070520 0x7fd4900729c0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:48.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.178+0000 7fd4a0ff9640 1 -- 172.21.15.88:0/238988047 <== osd.3 v2:172.21.15.88:6823/1519973662 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fd490073090 con 0x7fd490070520 2023-10-16T20:58:48.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.190+0000 7fd4aa53e640 1 -- 172.21.15.88:0/238988047 --> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd4a4069f30 con 0x7fd490070520 2023-10-16T20:58:48.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.190+0000 7fd4a0ff9640 1 -- 172.21.15.88:0/238988047 <== osd.3 v2:172.21.15.88:6823/1519973662 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fd49c009b40 con 0x7fd490070520 2023-10-16T20:58:48.193 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-10-16T20:58:48.193 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T20:58:48.193 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-10-16T20:58:48.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.190+0000 7fd4aa53e640 1 -- 172.21.15.88:0/238988047 >> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] conn(0x7fd490070520 msgr2=0x7fd4900729c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.190+0000 7fd4aa53e640 1 --2- 172.21.15.88:0/238988047 >> [v2:172.21.15.88:6823/1519973662,v1:172.21.15.88:6825/1519973662] conn(0x7fd490070520 0x7fd4900729c0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.190+0000 7fd4aa53e640 1 -- 172.21.15.88:0/238988047 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd4900664a0 msgr2=0x7fd490068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.190+0000 7fd4aa53e640 1 --2- 172.21.15.88:0/238988047 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd4900664a0 0x7fd490068960 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd4940b2000 tx=0x7fd494008040 comp rx=0 tx=0).stop 2023-10-16T20:58:48.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.190+0000 7fd4aa53e640 1 -- 172.21.15.88:0/238988047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4a40c2320 msgr2=0x7fd4a40ca940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.190+0000 7fd4aa53e640 1 --2- 172.21.15.88:0/238988047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4a40c2320 0x7fd4a40ca940 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fd49800c3c0 tx=0x7fd49800c880 comp rx=0 tx=0).stop 2023-10-16T20:58:48.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.190+0000 7fd4a8adb640 1 -- 172.21.15.88:0/238988047 reap_dead start 2023-10-16T20:58:48.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.190+0000 7fd4aa53e640 1 -- 172.21.15.88:0/238988047 shutdown_connections 2023-10-16T20:58:48.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.190+0000 7fd4aa53e640 1 -- 172.21.15.88:0/238988047 >> 172.21.15.88:0/238988047 conn(0x7fd4a40c86b0 msgr2=0x7fd4a40c8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:48.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.190+0000 7fd4aa53e640 1 -- 172.21.15.88:0/238988047 shutdown_connections 2023-10-16T20:58:48.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.190+0000 7fd4aa53e640 1 -- 172.21.15.88:0/238988047 wait complete. 2023-10-16T20:58:48.211 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-16T20:58:48.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.346+0000 7f56af49c640 1 Processor -- start 2023-10-16T20:58:48.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.346+0000 7f56af49c640 1 -- start start 2023-10-16T20:58:48.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.350+0000 7f56af49c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56a806a0f0 0x7f56a80c2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.350+0000 7f56af49c640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f56a806a4d0 0x7f56a8153410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.350+0000 7f56af49c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a815cd00 con 0x7f56a806ad20 2023-10-16T20:58:48.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.350+0000 7f56af49c640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a815d980 con 0x7f56a806a4d0 2023-10-16T20:58:48.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.350+0000 7f56af49c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a80c2000 con 0x7f56a806a0f0 2023-10-16T20:58:48.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.350+0000 7f56ad238640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56a806a0f0 0x7f56a80c2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.350+0000 7f56ada39640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f56a806ad20 0x7f56a80c30d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:49662/0 (socket says 172.21.15.88:49662) 2023-10-16T20:58:48.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.350+0000 7f56ada39640 1 -- 172.21.15.88:0/1313973189 learned_addr learned my addr 172.21.15.88:0/1313973189 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:48.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.350+0000 7f56aca37640 1 --2- 172.21.15.88:0/1313973189 >> v2:172.21.15.191:3300/0 conn(0x7f56a806a4d0 0x7f56a8153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.350+0000 7f569ffff640 1 -- 172.21.15.88:0/1313973189 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273789315 0 0) 0x7f56a815cd00 con 0x7f56a806ad20 2023-10-16T20:58:48.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.350+0000 7f569ffff640 1 -- 172.21.15.88:0/1313973189 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5694003530 con 0x7f56a806ad20 2023-10-16T20:58:48.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.350+0000 7f56ad238640 1 -- 172.21.15.88:0/1313973189 >> v2:172.21.15.191:3300/0 conn(0x7f56a806a4d0 msgr2=0x7f56a8153410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:48.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.350+0000 7f56ad238640 1 --2- 172.21.15.88:0/1313973189 >> v2:172.21.15.191:3300/0 conn(0x7f56a806a4d0 0x7f56a8153410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.350+0000 7f56ad238640 1 -- 172.21.15.88:0/1313973189 >> v1:172.21.15.191:6789/0 conn(0x7f56a806ad20 legacy=0x7f56a80c30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.350+0000 7f56ad238640 1 -- 172.21.15.88:0/1313973189 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a815cfe0 con 0x7f56a806a0f0 2023-10-16T20:58:48.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.350+0000 7f56aca37640 1 --2- 172.21.15.88:0/1313973189 >> v2:172.21.15.191:3300/0 conn(0x7f56a806a4d0 0x7f56a8153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:58:48.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.354+0000 7f56ad238640 1 --2- 172.21.15.88:0/1313973189 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56a806a0f0 0x7f56a80c2360 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f56a40097c0 tx=0x7f56a401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3686256667eafe5b server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:48.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.354+0000 7f569ffff640 1 -- 172.21.15.88:0/1313973189 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a40a8970 con 0x7f56a806a0f0 2023-10-16T20:58:48.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.354+0000 7f569ffff640 1 -- 172.21.15.88:0/1313973189 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a40a8ad0 con 0x7f56a806a0f0 2023-10-16T20:58:48.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.354+0000 7f569ffff640 1 -- 172.21.15.88:0/1313973189 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a40a8e60 con 0x7f56a806a0f0 2023-10-16T20:58:48.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.354+0000 7f56af49c640 1 -- 172.21.15.88:0/1313973189 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56a806a0f0 msgr2=0x7f56a80c2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.354+0000 7f56af49c640 1 --2- 172.21.15.88:0/1313973189 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56a806a0f0 0x7f56a80c2360 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f56a40097c0 tx=0x7f56a401ec70 comp rx=0 tx=0).stop 2023-10-16T20:58:48.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.354+0000 7f56af49c640 1 -- 172.21.15.88:0/1313973189 shutdown_connections 2023-10-16T20:58:48.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.354+0000 7f56af49c640 1 --2- 172.21.15.88:0/1313973189 >> v2:172.21.15.191:3300/0 conn(0x7f56a806a4d0 0x7f56a8153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.354+0000 7f56af49c640 1 --2- 172.21.15.88:0/1313973189 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56a806a0f0 0x7f56a80c2360 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.354+0000 7f56af49c640 1 -- 172.21.15.88:0/1313973189 >> 172.21.15.88:0/1313973189 conn(0x7f56a80ca9f0 msgr2=0x7f56a80cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:48.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.354+0000 7f56af49c640 1 -- 172.21.15.88:0/1313973189 shutdown_connections 2023-10-16T20:58:48.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.354+0000 7f56af49c640 1 -- 172.21.15.88:0/1313973189 wait complete. 2023-10-16T20:58:48.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.354+0000 7f56af49c640 1 Processor -- start 2023-10-16T20:58:48.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.354+0000 7f56af49c640 1 -- start start 2023-10-16T20:58:48.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56af49c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56a806a0f0 0x7f56a8159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56af49c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56a806a4d0 0x7f56a815bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56af49c640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56a806ad20 0x7f56a806f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56af49c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a8158d60 con 0x7f56a806a0f0 2023-10-16T20:58:48.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56af49c640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a81ee530 con 0x7f56a806ad20 2023-10-16T20:58:48.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56af49c640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a8157cc0 con 0x7f56a806a4d0 2023-10-16T20:58:48.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56ad238640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56a806a0f0 0x7f56a8159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56ad238640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56a806a0f0 0x7f56a8159ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54862/0 (socket says 172.21.15.88:54862) 2023-10-16T20:58:48.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56ad238640 1 -- 172.21.15.88:0/540104549 learned_addr learned my addr 172.21.15.88:0/540104549 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:48.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56ada39640 1 --2- 172.21.15.88:0/540104549 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56a806ad20 0x7f56a806f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56aca37640 1 --2- 172.21.15.88:0/540104549 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56a806a4d0 0x7f56a815bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56ad238640 1 -- 172.21.15.88:0/540104549 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56a806a4d0 msgr2=0x7f56a815bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56ad238640 1 --2- 172.21.15.88:0/540104549 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56a806a4d0 0x7f56a815bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56ad238640 1 -- 172.21.15.88:0/540104549 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56a806ad20 msgr2=0x7f56a806f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56ad238640 1 --2- 172.21.15.88:0/540104549 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56a806ad20 0x7f56a806f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56ad238640 1 -- 172.21.15.88:0/540104549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5698009420 con 0x7f56a806a0f0 2023-10-16T20:58:48.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56ada39640 1 --2- 172.21.15.88:0/540104549 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56a806ad20 0x7f56a806f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:58:48.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56ad238640 1 --2- 172.21.15.88:0/540104549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56a806a0f0 0x7f56a8159ab0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f56a401b880 tx=0x7f56a401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:48.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f569dffb640 1 -- 172.21.15.88:0/540104549 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a40b0d00 con 0x7f56a806a0f0 2023-10-16T20:58:48.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f569dffb640 1 -- 172.21.15.88:0/540104549 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56a40b0e60 con 0x7f56a806a0f0 2023-10-16T20:58:48.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f569dffb640 1 -- 172.21.15.88:0/540104549 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a40b9640 con 0x7f56a806a0f0 2023-10-16T20:58:48.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56af49c640 1 -- 172.21.15.88:0/540104549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a4009420 con 0x7f56a806a0f0 2023-10-16T20:58:48.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56af49c640 1 -- 172.21.15.88:0/540104549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a8158280 con 0x7f56a806a0f0 2023-10-16T20:58:48.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.358+0000 7f56af49c640 1 -- 172.21.15.88:0/540104549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a8074700 con 0x7f56a806a0f0 2023-10-16T20:58:48.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.362+0000 7f569dffb640 1 -- 172.21.15.88:0/540104549 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f56a40b7040 con 0x7f56a806a0f0 2023-10-16T20:58:48.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.362+0000 7f569dffb640 1 --2- 172.21.15.88:0/540104549 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f568c0664a0 0x7f568c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.362+0000 7f569dffb640 1 -- 172.21.15.88:0/540104549 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f56a4121f70 con 0x7f56a806a0f0 2023-10-16T20:58:48.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.362+0000 7f56aca37640 1 --2- 172.21.15.88:0/540104549 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f568c0664a0 0x7f568c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.362+0000 7f56aca37640 1 --2- 172.21.15.88:0/540104549 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f568c0664a0 0x7f568c068960 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f56a8151860 tx=0x7f5698008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:48.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.366+0000 7f569dffb640 1 -- 172.21.15.88:0/540104549 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f56a40eec30 con 0x7f56a806a0f0 2023-10-16T20:58:48.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.514+0000 7f56af49c640 1 -- 172.21.15.88:0/540104549 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f56a815d5a0 con 0x7f568c0664a0 2023-10-16T20:58:48.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.514+0000 7f569dffb640 1 -- 172.21.15.88:0/540104549 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5698011e60 con 0x7f568c0664a0 2023-10-16T20:58:48.518 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 3 to deep-scrub 2023-10-16T20:58:48.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.518+0000 7f56af49c640 1 -- 172.21.15.88:0/540104549 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f568c0664a0 msgr2=0x7f568c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.518+0000 7f56af49c640 1 --2- 172.21.15.88:0/540104549 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f568c0664a0 0x7f568c068960 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f56a8151860 tx=0x7f5698008040 comp rx=0 tx=0).stop 2023-10-16T20:58:48.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.518+0000 7f56af49c640 1 -- 172.21.15.88:0/540104549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56a806a0f0 msgr2=0x7f56a8159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.518+0000 7f56af49c640 1 --2- 172.21.15.88:0/540104549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56a806a0f0 0x7f56a8159ab0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f56a401b880 tx=0x7f56a401ba10 comp rx=0 tx=0).stop 2023-10-16T20:58:48.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.518+0000 7f56af49c640 1 -- 172.21.15.88:0/540104549 shutdown_connections 2023-10-16T20:58:48.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.518+0000 7f56af49c640 1 --2- 172.21.15.88:0/540104549 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f568c0664a0 0x7f568c068960 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.518+0000 7f56af49c640 1 --2- 172.21.15.88:0/540104549 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f56a806ad20 0x7f56a806f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.518+0000 7f56af49c640 1 --2- 172.21.15.88:0/540104549 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f56a806a4d0 0x7f56a815bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.518+0000 7f56af49c640 1 --2- 172.21.15.88:0/540104549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56a806a0f0 0x7f56a8159ab0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.518+0000 7f56af49c640 1 -- 172.21.15.88:0/540104549 >> 172.21.15.88:0/540104549 conn(0x7f56a80ca9f0 msgr2=0x7f56a80cadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:48.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.518+0000 7f56af49c640 1 -- 172.21.15.88:0/540104549 shutdown_connections 2023-10-16T20:58:48.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.518+0000 7f56af49c640 1 -- 172.21.15.88:0/540104549 wait complete. 2023-10-16T20:58:48.539 INFO:tasks.ceph:Scrubbing osd.4 2023-10-16T20:58:48.539 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-16T20:58:48.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.678+0000 7fbdc2faa640 1 Processor -- start 2023-10-16T20:58:48.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.678+0000 7fbdc2faa640 1 -- start start 2023-10-16T20:58:48.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.682+0000 7fbdc2faa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbdbc0cdb20 0x7fbdbc0ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.682+0000 7fbdc2faa640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdbc06cbe0 con 0x7fbdbc06dd70 2023-10-16T20:58:48.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.682+0000 7fbdc2faa640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdbc15d8b0 con 0x7fbdbc1537d0 2023-10-16T20:58:48.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.682+0000 7fbdc2faa640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdbc06f5f0 con 0x7fbdbc0cdb20 2023-10-16T20:58:48.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.682+0000 7fbdb3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbdbc0cdb20 0x7fbdbc0ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.682+0000 7fbdb3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbdbc0cdb20 0x7fbdbc0ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54878/0 (socket says 172.21.15.88:54878) 2023-10-16T20:58:48.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.682+0000 7fbdb3fff640 1 -- 172.21.15.88:0/1280780509 learned_addr learned my addr 172.21.15.88:0/1280780509 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:48.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.682+0000 7fbdc1547640 1 --1- 172.21.15.88:0/1280780509 >> v1:172.21.15.88:6789/0 conn(0x7fbdbc06dd70 0x7fbdbc163ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59614/0 (socket says 172.21.15.88:59614) 2023-10-16T20:58:48.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.682+0000 7fbdb3fff640 1 -- 172.21.15.88:0/1280780509 >> v1:172.21.15.191:6789/0 conn(0x7fbdbc1537d0 legacy=0x7fbdbc15aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.682+0000 7fbdb3fff640 1 -- 172.21.15.88:0/1280780509 >> v1:172.21.15.88:6789/0 conn(0x7fbdbc06dd70 legacy=0x7fbdbc163ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.682+0000 7fbdb3fff640 1 -- 172.21.15.88:0/1280780509 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdbc06e830 con 0x7fbdbc0cdb20 2023-10-16T20:58:48.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.686+0000 7fbdb3fff640 1 --2- 172.21.15.88:0/1280780509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdbc0cdb20 0x7fbdbc0ccdb0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fbdac009870 tx=0x7fbdac01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf38bb954ea1449 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:48.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.686+0000 7fbdb37fe640 1 -- 172.21.15.88:0/1280780509 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac0ab600 con 0x7fbdbc0cdb20 2023-10-16T20:58:48.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.686+0000 7fbdb37fe640 1 -- 172.21.15.88:0/1280780509 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdac0a7050 con 0x7fbdbc0cdb20 2023-10-16T20:58:48.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.686+0000 7fbdb37fe640 1 -- 172.21.15.88:0/1280780509 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac0ab980 con 0x7fbdbc0cdb20 2023-10-16T20:58:48.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.686+0000 7fbdc2faa640 1 -- 172.21.15.88:0/1280780509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdbc0cdb20 msgr2=0x7fbdbc0ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.686+0000 7fbdc2faa640 1 --2- 172.21.15.88:0/1280780509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdbc0cdb20 0x7fbdbc0ccdb0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fbdac009870 tx=0x7fbdac01c540 comp rx=0 tx=0).stop 2023-10-16T20:58:48.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.686+0000 7fbdc2faa640 1 -- 172.21.15.88:0/1280780509 shutdown_connections 2023-10-16T20:58:48.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.686+0000 7fbdc2faa640 1 --2- 172.21.15.88:0/1280780509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdbc0cdb20 0x7fbdbc0ccdb0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.686+0000 7fbdc2faa640 1 -- 172.21.15.88:0/1280780509 >> 172.21.15.88:0/1280780509 conn(0x7fbdbc06f7d0 msgr2=0x7fbdbc06fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:48.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.686+0000 7fbdc2faa640 1 -- 172.21.15.88:0/1280780509 shutdown_connections 2023-10-16T20:58:48.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.686+0000 7fbdc2faa640 1 -- 172.21.15.88:0/1280780509 wait complete. 2023-10-16T20:58:48.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.686+0000 7fbdc2faa640 1 Processor -- start 2023-10-16T20:58:48.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.686+0000 7fbdc2faa640 1 -- start start 2023-10-16T20:58:48.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdc2faa640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdbc06dd70 0x7fbdbc139b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdc2faa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdbc0cdb20 0x7fbdbc140d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdc2faa640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdbc1537d0 0x7fbdbc1f2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdc2faa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc13a080 con 0x7fbdbc0cdb20 2023-10-16T20:58:48.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdc2faa640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc1f5bf0 con 0x7fbdbc1537d0 2023-10-16T20:58:48.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdc2faa640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc14d700 con 0x7fbdbc06dd70 2023-10-16T20:58:48.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdb3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdbc0cdb20 0x7fbdbc140d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdb3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdbc0cdb20 0x7fbdbc140d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54880/0 (socket says 172.21.15.88:54880) 2023-10-16T20:58:48.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdb3fff640 1 -- 172.21.15.88:0/1830526037 learned_addr learned my addr 172.21.15.88:0/1830526037 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:48.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdc0d46640 1 --2- 172.21.15.88:0/1830526037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdbc06dd70 0x7fbdbc139b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdc1547640 1 --2- 172.21.15.88:0/1830526037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdbc1537d0 0x7fbdbc1f2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdb3fff640 1 -- 172.21.15.88:0/1830526037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdbc06dd70 msgr2=0x7fbdbc139b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdb3fff640 1 --2- 172.21.15.88:0/1830526037 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fbdbc06dd70 0x7fbdbc139b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdb3fff640 1 -- 172.21.15.88:0/1830526037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdbc1537d0 msgr2=0x7fbdbc1f2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdb3fff640 1 --2- 172.21.15.88:0/1830526037 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbdbc1537d0 0x7fbdbc1f2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdb3fff640 1 -- 172.21.15.88:0/1830526037 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdac0094d0 con 0x7fbdbc0cdb20 2023-10-16T20:58:48.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdb3fff640 1 --2- 172.21.15.88:0/1830526037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdbc0cdb20 0x7fbdbc140d60 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fbdac01a170 tx=0x7fbdac0192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:48.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdb17fa640 1 -- 172.21.15.88:0/1830526037 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac0b4620 con 0x7fbdbc0cdb20 2023-10-16T20:58:48.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdb17fa640 1 -- 172.21.15.88:0/1830526037 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdac0b47a0 con 0x7fbdbc0cdb20 2023-10-16T20:58:48.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdc2faa640 1 -- 172.21.15.88:0/1830526037 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdbc1f21b0 con 0x7fbdbc0cdb20 2023-10-16T20:58:48.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdb17fa640 1 -- 172.21.15.88:0/1830526037 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac0b4b50 con 0x7fbdbc0cdb20 2023-10-16T20:58:48.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdc2faa640 1 -- 172.21.15.88:0/1830526037 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdbc1f6170 con 0x7fbdbc0cdb20 2023-10-16T20:58:48.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdc2faa640 1 -- 172.21.15.88:0/1830526037 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbdbc15d570 con 0x7fbdbc0cdb20 2023-10-16T20:58:48.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.690+0000 7fbdb17fa640 1 -- 172.21.15.88:0/1830526037 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbdac0b2070 con 0x7fbdbc0cdb20 2023-10-16T20:58:48.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.694+0000 7fbdb17fa640 1 --2- 172.21.15.88:0/1830526037 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbda80664a0 0x7fbda8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.694+0000 7fbdb17fa640 1 -- 172.21.15.88:0/1830526037 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7fbdac11e4a0 con 0x7fbdbc0cdb20 2023-10-16T20:58:48.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.694+0000 7fbdc0d46640 1 --2- 172.21.15.88:0/1830526037 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbda80664a0 0x7fbda8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.694+0000 7fbdb17fa640 1 --2- 172.21.15.88:0/1830526037 >> [v2:172.21.15.191:6800/584586645,v1:172.21.15.191:6801/584586645] conn(0x7fbda80704a0 0x7fbda8072940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.694+0000 7fbdb17fa640 1 -- 172.21.15.88:0/1830526037 --> [v2:172.21.15.191:6800/584586645,v1:172.21.15.191:6801/584586645] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbda8073010 con 0x7fbda80704a0 2023-10-16T20:58:48.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.694+0000 7fbdc1547640 1 --2- 172.21.15.88:0/1830526037 >> [v2:172.21.15.191:6800/584586645,v1:172.21.15.191:6801/584586645] conn(0x7fbda80704a0 0x7fbda8072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.694+0000 7fbdc0d46640 1 --2- 172.21.15.88:0/1830526037 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbda80664a0 0x7fbda8068960 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8007fd0 tx=0x7fbdb8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:48.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.694+0000 7fbdb17fa640 1 -- 172.21.15.88:0/1830526037 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbdac0eb2a0 con 0x7fbdbc0cdb20 2023-10-16T20:58:48.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.694+0000 7fbdc1547640 1 --2- 172.21.15.88:0/1830526037 >> [v2:172.21.15.191:6800/584586645,v1:172.21.15.191:6801/584586645] conn(0x7fbda80704a0 0x7fbda8072940 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:48.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.694+0000 7fbdb17fa640 1 -- 172.21.15.88:0/1830526037 <== osd.4 v2:172.21.15.191:6800/584586645 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fbda8073010 con 0x7fbda80704a0 2023-10-16T20:58:48.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.710+0000 7fbdc2faa640 1 -- 172.21.15.88:0/1830526037 --> [v2:172.21.15.191:6800/584586645,v1:172.21.15.191:6801/584586645] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fbdbc074700 con 0x7fbda80704a0 2023-10-16T20:58:48.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.710+0000 7fbdb17fa640 1 -- 172.21.15.88:0/1830526037 <== osd.4 v2:172.21.15.191:6800/584586645 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fbdbc074700 con 0x7fbda80704a0 2023-10-16T20:58:48.711 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-10-16T20:58:48.711 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T20:58:48.712 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-10-16T20:58:48.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.710+0000 7fbdc2faa640 1 -- 172.21.15.88:0/1830526037 >> [v2:172.21.15.191:6800/584586645,v1:172.21.15.191:6801/584586645] conn(0x7fbda80704a0 msgr2=0x7fbda8072940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.710+0000 7fbdc2faa640 1 --2- 172.21.15.88:0/1830526037 >> [v2:172.21.15.191:6800/584586645,v1:172.21.15.191:6801/584586645] conn(0x7fbda80704a0 0x7fbda8072940 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.710+0000 7fbdc2faa640 1 -- 172.21.15.88:0/1830526037 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbda80664a0 msgr2=0x7fbda8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.710+0000 7fbdc2faa640 1 --2- 172.21.15.88:0/1830526037 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fbda80664a0 0x7fbda8068960 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8007fd0 tx=0x7fbdb8009040 comp rx=0 tx=0).stop 2023-10-16T20:58:48.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.710+0000 7fbdc2faa640 1 -- 172.21.15.88:0/1830526037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdbc0cdb20 msgr2=0x7fbdbc140d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.710+0000 7fbdc2faa640 1 --2- 172.21.15.88:0/1830526037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdbc0cdb20 0x7fbdbc140d60 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fbdac01a170 tx=0x7fbdac0192e0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.710+0000 7fbdc1547640 1 -- 172.21.15.88:0/1830526037 reap_dead start 2023-10-16T20:58:48.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.710+0000 7fbdc2faa640 1 -- 172.21.15.88:0/1830526037 shutdown_connections 2023-10-16T20:58:48.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.710+0000 7fbdc2faa640 1 -- 172.21.15.88:0/1830526037 >> 172.21.15.88:0/1830526037 conn(0x7fbdbc06f7d0 msgr2=0x7fbdbc150de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:48.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.710+0000 7fbdc2faa640 1 -- 172.21.15.88:0/1830526037 shutdown_connections 2023-10-16T20:58:48.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.710+0000 7fbdc2faa640 1 -- 172.21.15.88:0/1830526037 wait complete. 2023-10-16T20:58:48.729 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-16T20:58:48.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.866+0000 7ff928009640 1 Processor -- start 2023-10-16T20:58:48.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.866+0000 7ff928009640 1 -- start start 2023-10-16T20:58:48.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.870+0000 7ff928009640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff9200c66e0 0x7ff92015a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.870+0000 7ff928009640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9200cb5c0 0x7ff920071420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.870+0000 7ff928009640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff920070f40 con 0x7ff9200c6ac0 2023-10-16T20:58:48.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.870+0000 7ff928009640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff92015a790 con 0x7ff9200c66e0 2023-10-16T20:58:48.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.870+0000 7ff928009640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92006a2d0 con 0x7ff9200cb5c0 2023-10-16T20:58:48.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.870+0000 7ff925da5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff9200c66e0 0x7ff92015a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.870+0000 7ff925da5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff9200c66e0 0x7ff92015a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33974/0 (socket says 172.21.15.88:33974) 2023-10-16T20:58:48.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.870+0000 7ff925da5640 1 -- 172.21.15.88:0/3040046779 learned_addr learned my addr 172.21.15.88:0/3040046779 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:48.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.870+0000 7ff9265a6640 1 --2- 172.21.15.88:0/3040046779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9200cb5c0 0x7ff920071420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.870+0000 7ff925da5640 1 -- 172.21.15.88:0/3040046779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9200cb5c0 msgr2=0x7ff920071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:48.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.870+0000 7ff925da5640 1 --2- 172.21.15.88:0/3040046779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9200cb5c0 0x7ff920071420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.870+0000 7ff925da5640 1 -- 172.21.15.88:0/3040046779 >> v1:172.21.15.88:6789/0 conn(0x7ff9200c6ac0 legacy=0x7ff920150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.870+0000 7ff925da5640 1 -- 172.21.15.88:0/3040046779 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92006a5b0 con 0x7ff9200c66e0 2023-10-16T20:58:48.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.870+0000 7ff9265a6640 1 --2- 172.21.15.88:0/3040046779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9200cb5c0 0x7ff920071420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:58:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff925da5640 1 --2- 172.21.15.88:0/3040046779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9200c66e0 0x7ff92015a250 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ff91c00bb10 tx=0x7ff91c01eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b72a34c18d1350 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:48.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff924da3640 1 -- 172.21.15.88:0/3040046779 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c0a7a40 con 0x7ff9200c66e0 2023-10-16T20:58:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff924da3640 1 -- 172.21.15.88:0/3040046779 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91c0a7ba0 con 0x7ff9200c66e0 2023-10-16T20:58:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff924da3640 1 -- 172.21.15.88:0/3040046779 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c0af440 con 0x7ff9200c66e0 2023-10-16T20:58:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff928009640 1 -- 172.21.15.88:0/3040046779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9200c66e0 msgr2=0x7ff92015a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff928009640 1 --2- 172.21.15.88:0/3040046779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9200c66e0 0x7ff92015a250 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ff91c00bb10 tx=0x7ff91c01eb80 comp rx=0 tx=0).stop 2023-10-16T20:58:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff928009640 1 -- 172.21.15.88:0/3040046779 shutdown_connections 2023-10-16T20:58:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff928009640 1 --2- 172.21.15.88:0/3040046779 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9200cb5c0 0x7ff920071420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff928009640 1 --2- 172.21.15.88:0/3040046779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9200c66e0 0x7ff92015a250 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff928009640 1 -- 172.21.15.88:0/3040046779 >> 172.21.15.88:0/3040046779 conn(0x7ff9200ca9f0 msgr2=0x7ff9200cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff928009640 1 -- 172.21.15.88:0/3040046779 shutdown_connections 2023-10-16T20:58:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff928009640 1 -- 172.21.15.88:0/3040046779 wait complete. 2023-10-16T20:58:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff928009640 1 Processor -- start 2023-10-16T20:58:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff928009640 1 -- start start 2023-10-16T20:58:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff928009640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9200c66e0 0x7ff9200c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff928009640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9200c6ac0 0x7ff9200c2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff928009640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9200cb5c0 0x7ff920148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff928009640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9200c3730 con 0x7ff9200cb5c0 2023-10-16T20:58:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff928009640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9201f50a0 con 0x7ff9200c6ac0 2023-10-16T20:58:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff928009640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9201f5380 con 0x7ff9200c66e0 2023-10-16T20:58:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.874+0000 7ff9265a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9200cb5c0 0x7ff920148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff9265a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9200cb5c0 0x7ff920148b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33980/0 (socket says 172.21.15.88:33980) 2023-10-16T20:58:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff9265a6640 1 -- 172.21.15.88:0/812210005 learned_addr learned my addr 172.21.15.88:0/812210005 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff925da5640 1 --2- 172.21.15.88:0/812210005 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9200c66e0 0x7ff9200c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff9255a4640 1 --2- 172.21.15.88:0/812210005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9200c6ac0 0x7ff9200c2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff9265a6640 1 -- 172.21.15.88:0/812210005 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9200c66e0 msgr2=0x7ff9200c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff9265a6640 1 --2- 172.21.15.88:0/812210005 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9200c66e0 0x7ff9200c3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff9265a6640 1 -- 172.21.15.88:0/812210005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9200c6ac0 msgr2=0x7ff9200c2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff9265a6640 1 --2- 172.21.15.88:0/812210005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9200c6ac0 0x7ff9200c2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff9265a6640 1 -- 172.21.15.88:0/812210005 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c00b770 con 0x7ff9200cb5c0 2023-10-16T20:58:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff9265a6640 1 --2- 172.21.15.88:0/812210005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9200cb5c0 0x7ff920148b70 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7ff91800cd80 tx=0x7ff91800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff9167fc640 1 -- 172.21.15.88:0/812210005 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff918003ba0 con 0x7ff9200cb5c0 2023-10-16T20:58:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff9167fc640 1 -- 172.21.15.88:0/812210005 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff918003d20 con 0x7ff9200cb5c0 2023-10-16T20:58:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff9167fc640 1 -- 172.21.15.88:0/812210005 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff918004260 con 0x7ff9200cb5c0 2023-10-16T20:58:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff928009640 1 -- 172.21.15.88:0/812210005 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9201f5660 con 0x7ff9200cb5c0 2023-10-16T20:58:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff928009640 1 -- 172.21.15.88:0/812210005 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9201f5c20 con 0x7ff9200cb5c0 2023-10-16T20:58:48.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff928009640 1 -- 172.21.15.88:0/812210005 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9201590a0 con 0x7ff9200cb5c0 2023-10-16T20:58:48.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff9167fc640 1 -- 172.21.15.88:0/812210005 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff91800b040 con 0x7ff9200cb5c0 2023-10-16T20:58:48.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.878+0000 7ff9167fc640 1 --2- 172.21.15.88:0/812210005 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff9040664a0 0x7ff904068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:48.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.882+0000 7ff9167fc640 1 -- 172.21.15.88:0/812210005 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7ff918084c00 con 0x7ff9200cb5c0 2023-10-16T20:58:48.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.882+0000 7ff925da5640 1 --2- 172.21.15.88:0/812210005 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff9040664a0 0x7ff904068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:48.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.882+0000 7ff925da5640 1 --2- 172.21.15.88:0/812210005 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff9040664a0 0x7ff904068960 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff91c0b1000 tx=0x7ff91c0a9070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:48.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:48.882+0000 7ff9167fc640 1 -- 172.21.15.88:0/812210005 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff9201590a0 con 0x7ff9200cb5c0 2023-10-16T20:58:49.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.034+0000 7ff928009640 1 -- 172.21.15.88:0/812210005 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff92006ae50 con 0x7ff9040664a0 2023-10-16T20:58:49.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.034+0000 7ff9167fc640 1 -- 172.21.15.88:0/812210005 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff91c0af440 con 0x7ff9040664a0 2023-10-16T20:58:49.038 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 4 to deep-scrub 2023-10-16T20:58:49.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.038+0000 7ff928009640 1 -- 172.21.15.88:0/812210005 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff9040664a0 msgr2=0x7ff904068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.038+0000 7ff928009640 1 --2- 172.21.15.88:0/812210005 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff9040664a0 0x7ff904068960 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff91c0b1000 tx=0x7ff91c0a9070 comp rx=0 tx=0).stop 2023-10-16T20:58:49.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.038+0000 7ff928009640 1 -- 172.21.15.88:0/812210005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9200cb5c0 msgr2=0x7ff920148b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.038+0000 7ff928009640 1 --2- 172.21.15.88:0/812210005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9200cb5c0 0x7ff920148b70 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7ff91800cd80 tx=0x7ff91800a650 comp rx=0 tx=0).stop 2023-10-16T20:58:49.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.038+0000 7ff928009640 1 -- 172.21.15.88:0/812210005 shutdown_connections 2023-10-16T20:58:49.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.038+0000 7ff928009640 1 --2- 172.21.15.88:0/812210005 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff9040664a0 0x7ff904068960 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.038+0000 7ff928009640 1 --2- 172.21.15.88:0/812210005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9200cb5c0 0x7ff920148b70 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.038+0000 7ff928009640 1 --2- 172.21.15.88:0/812210005 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9200c6ac0 0x7ff9200c2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.038+0000 7ff928009640 1 --2- 172.21.15.88:0/812210005 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff9200c66e0 0x7ff9200c3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.038+0000 7ff928009640 1 -- 172.21.15.88:0/812210005 >> 172.21.15.88:0/812210005 conn(0x7ff9200ca9f0 msgr2=0x7ff920071ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:49.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.038+0000 7ff928009640 1 -- 172.21.15.88:0/812210005 shutdown_connections 2023-10-16T20:58:49.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.038+0000 7ff928009640 1 -- 172.21.15.88:0/812210005 wait complete. 2023-10-16T20:58:49.059 INFO:tasks.ceph:Scrubbing osd.5 2023-10-16T20:58:49.059 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-16T20:58:49.201 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.198+0000 7fa3529d9640 1 Processor -- start 2023-10-16T20:58:49.202 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.198+0000 7fa3529d9640 1 -- start start 2023-10-16T20:58:49.203 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.202+0000 7fa3529d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa34c06dd70 0x7fa34c164ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.202+0000 7fa3529d9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34c06cbe0 con 0x7fa34c0cdb20 2023-10-16T20:58:49.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.202+0000 7fa3529d9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34c15d8b0 con 0x7fa34c1537d0 2023-10-16T20:58:49.204 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.202+0000 7fa3529d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34c06f5f0 con 0x7fa34c06dd70 2023-10-16T20:58:49.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.202+0000 7fa350f76640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa34c06dd70 0x7fa34c164ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.202+0000 7fa350f76640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa34c06dd70 0x7fa34c164ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33982/0 (socket says 172.21.15.88:33982) 2023-10-16T20:58:49.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.202+0000 7fa350f76640 1 -- 172.21.15.88:0/1235910264 learned_addr learned my addr 172.21.15.88:0/1235910264 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:49.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.202+0000 7fa350f76640 1 -- 172.21.15.88:0/1235910264 >> v1:172.21.15.191:6789/0 conn(0x7fa34c1537d0 legacy=0x7fa34c15aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.202+0000 7fa350f76640 1 -- 172.21.15.88:0/1235910264 >> v1:172.21.15.88:6789/0 conn(0x7fa34c0cdb20 legacy=0x7fa34c0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.202+0000 7fa350f76640 1 -- 172.21.15.88:0/1235910264 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34c06e830 con 0x7fa34c06dd70 2023-10-16T20:58:49.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.202+0000 7fa350f76640 1 --2- 172.21.15.88:0/1235910264 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa34c06dd70 0x7fa34c164ee0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa344009e50 tx=0x7fa34401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff481ea7b2bd25 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:49.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa34affd640 1 -- 172.21.15.88:0/1235910264 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3440a57d0 con 0x7fa34c06dd70 2023-10-16T20:58:49.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa34affd640 1 -- 172.21.15.88:0/1235910264 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3440a5930 con 0x7fa34c06dd70 2023-10-16T20:58:49.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa34affd640 1 -- 172.21.15.88:0/1235910264 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3440a5cc0 con 0x7fa34c06dd70 2023-10-16T20:58:49.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa3529d9640 1 -- 172.21.15.88:0/1235910264 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa34c06dd70 msgr2=0x7fa34c164ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa3529d9640 1 --2- 172.21.15.88:0/1235910264 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa34c06dd70 0x7fa34c164ee0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa344009e50 tx=0x7fa34401ca10 comp rx=0 tx=0).stop 2023-10-16T20:58:49.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa3529d9640 1 -- 172.21.15.88:0/1235910264 shutdown_connections 2023-10-16T20:58:49.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa3529d9640 1 --2- 172.21.15.88:0/1235910264 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa34c06dd70 0x7fa34c164ee0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa3529d9640 1 -- 172.21.15.88:0/1235910264 >> 172.21.15.88:0/1235910264 conn(0x7fa34c06f7d0 msgr2=0x7fa34c06fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:49.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa3529d9640 1 -- 172.21.15.88:0/1235910264 shutdown_connections 2023-10-16T20:58:49.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa3529d9640 1 -- 172.21.15.88:0/1235910264 wait complete. 2023-10-16T20:58:49.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa3529d9640 1 Processor -- start 2023-10-16T20:58:49.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa3529d9640 1 -- start start 2023-10-16T20:58:49.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa3529d9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa34c06dd70 0x7fa34c13fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa3529d9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa34c0cdb20 0x7fa34c142180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa3529d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa34c1537d0 0x7fa34c1f4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa3529d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c0ca280 con 0x7fa34c1537d0 2023-10-16T20:58:49.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa3529d9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c159e20 con 0x7fa34c0cdb20 2023-10-16T20:58:49.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa3529d9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34c0c7bc0 con 0x7fa34c06dd70 2023-10-16T20:58:49.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.206+0000 7fa350f76640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa34c1537d0 0x7fa34c1f4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa350f76640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa34c1537d0 0x7fa34c1f4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33988/0 (socket says 172.21.15.88:33988) 2023-10-16T20:58:49.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa350f76640 1 -- 172.21.15.88:0/3126396815 learned_addr learned my addr 172.21.15.88:0/3126396815 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:49.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa34b7fe640 1 --2- 172.21.15.88:0/3126396815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa34c0cdb20 0x7fa34c142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa34bfff640 1 --2- 172.21.15.88:0/3126396815 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa34c06dd70 0x7fa34c13fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa350f76640 1 -- 172.21.15.88:0/3126396815 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa34c06dd70 msgr2=0x7fa34c13fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa350f76640 1 --2- 172.21.15.88:0/3126396815 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa34c06dd70 0x7fa34c13fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa350f76640 1 -- 172.21.15.88:0/3126396815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa34c0cdb20 msgr2=0x7fa34c142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa350f76640 1 --2- 172.21.15.88:0/3126396815 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa34c0cdb20 0x7fa34c142180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa350f76640 1 -- 172.21.15.88:0/3126396815 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa344009b30 con 0x7fa34c1537d0 2023-10-16T20:58:49.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa34bfff640 1 --2- 172.21.15.88:0/3126396815 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fa34c06dd70 0x7fa34c13fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:58:49.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa350f76640 1 --2- 172.21.15.88:0/3126396815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa34c1537d0 0x7fa34c1f4ea0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa3440af040 tx=0x7fa3440197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:49.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa348ff9640 1 -- 172.21.15.88:0/3126396815 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3440b8670 con 0x7fa34c1537d0 2023-10-16T20:58:49.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa348ff9640 1 -- 172.21.15.88:0/3126396815 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3440b87d0 con 0x7fa34c1537d0 2023-10-16T20:58:49.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa348ff9640 1 -- 172.21.15.88:0/3126396815 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3440b8b80 con 0x7fa34c1537d0 2023-10-16T20:58:49.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa3529d9640 1 -- 172.21.15.88:0/3126396815 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa34c140020 con 0x7fa34c1537d0 2023-10-16T20:58:49.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa3529d9640 1 -- 172.21.15.88:0/3126396815 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34c15a070 con 0x7fa34c1537d0 2023-10-16T20:58:49.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa3529d9640 1 -- 172.21.15.88:0/3126396815 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa34c15d570 con 0x7fa34c1537d0 2023-10-16T20:58:49.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa348ff9640 1 -- 172.21.15.88:0/3126396815 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa3440b8ce0 con 0x7fa34c1537d0 2023-10-16T20:58:49.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.210+0000 7fa348ff9640 1 --2- 172.21.15.88:0/3126396815 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa3380664a0 0x7fa338068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.214+0000 7fa348ff9640 1 -- 172.21.15.88:0/3126396815 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7fa34411fc80 con 0x7fa34c1537d0 2023-10-16T20:58:49.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.214+0000 7fa34bfff640 1 --2- 172.21.15.88:0/3126396815 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa3380664a0 0x7fa338068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.214+0000 7fa348ff9640 1 --2- 172.21.15.88:0/3126396815 >> [v2:172.21.15.191:6808/3826846674,v1:172.21.15.191:6809/3826846674] conn(0x7fa338070520 0x7fa3380729c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.214+0000 7fa348ff9640 1 -- 172.21.15.88:0/3126396815 --> [v2:172.21.15.191:6808/3826846674,v1:172.21.15.191:6809/3826846674] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa338073090 con 0x7fa338070520 2023-10-16T20:58:49.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.214+0000 7fa348ff9640 1 -- 172.21.15.88:0/3126396815 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3441242a0 con 0x7fa34c1537d0 2023-10-16T20:58:49.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.214+0000 7fa34bfff640 1 --2- 172.21.15.88:0/3126396815 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa3380664a0 0x7fa338068960 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa33c000f70 tx=0x7fa33c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:49.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.214+0000 7fa34b7fe640 1 --2- 172.21.15.88:0/3126396815 >> [v2:172.21.15.191:6808/3826846674,v1:172.21.15.191:6809/3826846674] conn(0x7fa338070520 0x7fa3380729c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.214+0000 7fa34b7fe640 1 --2- 172.21.15.88:0/3126396815 >> [v2:172.21.15.191:6808/3826846674,v1:172.21.15.191:6809/3826846674] conn(0x7fa338070520 0x7fa3380729c0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:49.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.214+0000 7fa348ff9640 1 -- 172.21.15.88:0/3126396815 <== osd.5 v2:172.21.15.191:6808/3826846674 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fa338073090 con 0x7fa338070520 2023-10-16T20:58:49.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.226+0000 7fa3529d9640 1 -- 172.21.15.88:0/3126396815 --> [v2:172.21.15.191:6808/3826846674,v1:172.21.15.191:6809/3826846674] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa34c0c8070 con 0x7fa338070520 2023-10-16T20:58:49.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.226+0000 7fa348ff9640 1 -- 172.21.15.88:0/3126396815 <== osd.5 v2:172.21.15.191:6808/3826846674 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fa34c0c8070 con 0x7fa338070520 2023-10-16T20:58:49.231 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-10-16T20:58:49.232 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T20:58:49.232 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-10-16T20:58:49.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.230+0000 7fa3529d9640 1 -- 172.21.15.88:0/3126396815 >> [v2:172.21.15.191:6808/3826846674,v1:172.21.15.191:6809/3826846674] conn(0x7fa338070520 msgr2=0x7fa3380729c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.230+0000 7fa3529d9640 1 --2- 172.21.15.88:0/3126396815 >> [v2:172.21.15.191:6808/3826846674,v1:172.21.15.191:6809/3826846674] conn(0x7fa338070520 0x7fa3380729c0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.230+0000 7fa3529d9640 1 -- 172.21.15.88:0/3126396815 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa3380664a0 msgr2=0x7fa338068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.230+0000 7fa3529d9640 1 --2- 172.21.15.88:0/3126396815 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fa3380664a0 0x7fa338068960 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa33c000f70 tx=0x7fa33c009040 comp rx=0 tx=0).stop 2023-10-16T20:58:49.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.230+0000 7fa3529d9640 1 -- 172.21.15.88:0/3126396815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa34c1537d0 msgr2=0x7fa34c1f4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.230+0000 7fa3529d9640 1 --2- 172.21.15.88:0/3126396815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa34c1537d0 0x7fa34c1f4ea0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa3440af040 tx=0x7fa3440197b0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.230+0000 7fa350f76640 1 -- 172.21.15.88:0/3126396815 reap_dead start 2023-10-16T20:58:49.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.230+0000 7fa3529d9640 1 -- 172.21.15.88:0/3126396815 shutdown_connections 2023-10-16T20:58:49.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.230+0000 7fa3529d9640 1 -- 172.21.15.88:0/3126396815 >> 172.21.15.88:0/3126396815 conn(0x7fa34c06f7d0 msgr2=0x7fa34c151a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:49.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.230+0000 7fa3529d9640 1 -- 172.21.15.88:0/3126396815 shutdown_connections 2023-10-16T20:58:49.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.230+0000 7fa3529d9640 1 -- 172.21.15.88:0/3126396815 wait complete. 2023-10-16T20:58:49.249 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-16T20:58:49.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.386+0000 7fd3ace32640 1 Processor -- start 2023-10-16T20:58:49.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.386+0000 7fd3ace32640 1 -- start start 2023-10-16T20:58:49.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.386+0000 7fd3ace32640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd3a814f7d0 0x7fd3a8156270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.386+0000 7fd3ace32640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3a80c1170 0x7fd3a80c15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.386+0000 7fd3ace32640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a80c72a0 con 0x7fd3a8158ee0 2023-10-16T20:58:49.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.386+0000 7fd3ace32640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a80c1ae0 con 0x7fd3a814f7d0 2023-10-16T20:58:49.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.386+0000 7fd3ace32640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80cb6a0 con 0x7fd3a80c1170 2023-10-16T20:58:49.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.386+0000 7fd3a659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd3a814f7d0 0x7fd3a8156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.386+0000 7fd3a659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd3a814f7d0 0x7fd3a8156270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33994/0 (socket says 172.21.15.88:33994) 2023-10-16T20:58:49.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.386+0000 7fd3a659c640 1 -- 172.21.15.88:0/2371518982 learned_addr learned my addr 172.21.15.88:0/2371518982 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:49.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.386+0000 7fd3a6d9d640 1 --2- 172.21.15.88:0/2371518982 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3a80c1170 0x7fd3a80c15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.386+0000 7fd3a659c640 1 -- 172.21.15.88:0/2371518982 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3a80c1170 msgr2=0x7fd3a80c15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:49.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.386+0000 7fd3a659c640 1 --2- 172.21.15.88:0/2371518982 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3a80c1170 0x7fd3a80c15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.386+0000 7fd3a659c640 1 -- 172.21.15.88:0/2371518982 >> v1:172.21.15.191:6789/0 conn(0x7fd3a8158ee0 legacy=0x7fd3a806a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:49.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.386+0000 7fd3a659c640 1 -- 172.21.15.88:0/2371518982 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a80c6a00 con 0x7fd3a814f7d0 2023-10-16T20:58:49.395 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.390+0000 7fd3a659c640 1 --2- 172.21.15.88:0/2371518982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3a814f7d0 0x7fd3a8156270 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fd398009870 tx=0x7fd39801fe40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f581b49056e8f7e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:49.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.390+0000 7fd3a5d9b640 1 -- 172.21.15.88:0/2371518982 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3980a9b40 con 0x7fd3a814f7d0 2023-10-16T20:58:49.396 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.390+0000 7fd3a5d9b640 1 -- 172.21.15.88:0/2371518982 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3980a9ca0 con 0x7fd3a814f7d0 2023-10-16T20:58:49.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.390+0000 7fd3a5d9b640 1 -- 172.21.15.88:0/2371518982 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3980b1660 con 0x7fd3a814f7d0 2023-10-16T20:58:49.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.390+0000 7fd3ace32640 1 -- 172.21.15.88:0/2371518982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3a814f7d0 msgr2=0x7fd3a8156270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.390+0000 7fd3ace32640 1 --2- 172.21.15.88:0/2371518982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3a814f7d0 0x7fd3a8156270 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fd398009870 tx=0x7fd39801fe40 comp rx=0 tx=0).stop 2023-10-16T20:58:49.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.390+0000 7fd3ace32640 1 -- 172.21.15.88:0/2371518982 shutdown_connections 2023-10-16T20:58:49.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.390+0000 7fd3ace32640 1 --2- 172.21.15.88:0/2371518982 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3a80c1170 0x7fd3a80c15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.399 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.390+0000 7fd3ace32640 1 --2- 172.21.15.88:0/2371518982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3a814f7d0 0x7fd3a8156270 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.390+0000 7fd3ace32640 1 -- 172.21.15.88:0/2371518982 >> 172.21.15.88:0/2371518982 conn(0x7fd3a80ca9f0 msgr2=0x7fd3a80cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:49.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.390+0000 7fd3ace32640 1 -- 172.21.15.88:0/2371518982 shutdown_connections 2023-10-16T20:58:49.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.390+0000 7fd3ace32640 1 -- 172.21.15.88:0/2371518982 wait complete. 2023-10-16T20:58:49.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd3ace32640 1 Processor -- start 2023-10-16T20:58:49.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd3ace32640 1 -- start start 2023-10-16T20:58:49.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd3ace32640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3a80c1170 0x7fd3a81eaab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd3ace32640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3a814f7d0 0x7fd3a814c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd3ace32640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3a8158ee0 0x7fd3a8141d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd3ace32640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a8071490 con 0x7fd3a814f7d0 2023-10-16T20:58:49.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd3ace32640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a8158460 con 0x7fd3a8158ee0 2023-10-16T20:58:49.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd3ace32640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a8158c30 con 0x7fd3a80c1170 2023-10-16T20:58:49.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd39dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3a814f7d0 0x7fd3a814c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd39dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3a814f7d0 0x7fd3a814c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34008/0 (socket says 172.21.15.88:34008) 2023-10-16T20:58:49.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd39dd9b640 1 -- 172.21.15.88:0/422001912 learned_addr learned my addr 172.21.15.88:0/422001912 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:49.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd3a659c640 1 --2- 172.21.15.88:0/422001912 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3a80c1170 0x7fd3a81eaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd3a6d9d640 1 --2- 172.21.15.88:0/422001912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3a8158ee0 0x7fd3a8141d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd39dd9b640 1 -- 172.21.15.88:0/422001912 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3a80c1170 msgr2=0x7fd3a81eaab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd39dd9b640 1 --2- 172.21.15.88:0/422001912 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3a80c1170 0x7fd3a81eaab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd39dd9b640 1 -- 172.21.15.88:0/422001912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3a8158ee0 msgr2=0x7fd3a8141d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd39dd9b640 1 --2- 172.21.15.88:0/422001912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3a8158ee0 0x7fd3a8141d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd39dd9b640 1 -- 172.21.15.88:0/422001912 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3980094d0 con 0x7fd3a814f7d0 2023-10-16T20:58:49.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd39dd9b640 1 --2- 172.21.15.88:0/422001912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3a814f7d0 0x7fd3a814c500 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fd39000c7a0 tx=0x7fd39000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:49.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd39f7fe640 1 -- 172.21.15.88:0/422001912 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd390003b80 con 0x7fd3a814f7d0 2023-10-16T20:58:49.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd39f7fe640 1 -- 172.21.15.88:0/422001912 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd390003ce0 con 0x7fd3a814f7d0 2023-10-16T20:58:49.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd39f7fe640 1 -- 172.21.15.88:0/422001912 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39000aa30 con 0x7fd3a814f7d0 2023-10-16T20:58:49.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.394+0000 7fd3ace32640 1 -- 172.21.15.88:0/422001912 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a81ea620 con 0x7fd3a814f7d0 2023-10-16T20:58:49.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.398+0000 7fd3ace32640 1 -- 172.21.15.88:0/422001912 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a8158950 con 0x7fd3a814f7d0 2023-10-16T20:58:49.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.398+0000 7fd3ace32640 1 -- 172.21.15.88:0/422001912 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a81ea7b0 con 0x7fd3a814f7d0 2023-10-16T20:58:49.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.398+0000 7fd39f7fe640 1 -- 172.21.15.88:0/422001912 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd39000ab90 con 0x7fd3a814f7d0 2023-10-16T20:58:49.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.398+0000 7fd39f7fe640 1 --2- 172.21.15.88:0/422001912 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd3880664a0 0x7fd388068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.398+0000 7fd39f7fe640 1 -- 172.21.15.88:0/422001912 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7fd390084200 con 0x7fd3a814f7d0 2023-10-16T20:58:49.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.398+0000 7fd3a659c640 1 --2- 172.21.15.88:0/422001912 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd3880664a0 0x7fd388068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.398+0000 7fd3a659c640 1 --2- 172.21.15.88:0/422001912 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd3880664a0 0x7fd388068960 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd3980b3000 tx=0x7fd3980ab070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:49.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.402+0000 7fd39f7fe640 1 -- 172.21.15.88:0/422001912 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd390051000 con 0x7fd3a814f7d0 2023-10-16T20:58:49.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.554+0000 7fd3ace32640 1 -- 172.21.15.88:0/422001912 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3a80c6fc0 con 0x7fd3880664a0 2023-10-16T20:58:49.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.554+0000 7fd39f7fe640 1 -- 172.21.15.88:0/422001912 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd3980bc450 con 0x7fd3880664a0 2023-10-16T20:58:49.556 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 5 to deep-scrub 2023-10-16T20:58:49.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.554+0000 7fd3ace32640 1 -- 172.21.15.88:0/422001912 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd3880664a0 msgr2=0x7fd388068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.554+0000 7fd3ace32640 1 --2- 172.21.15.88:0/422001912 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd3880664a0 0x7fd388068960 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd3980b3000 tx=0x7fd3980ab070 comp rx=0 tx=0).stop 2023-10-16T20:58:49.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.554+0000 7fd3ace32640 1 -- 172.21.15.88:0/422001912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3a814f7d0 msgr2=0x7fd3a814c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.554+0000 7fd3ace32640 1 --2- 172.21.15.88:0/422001912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3a814f7d0 0x7fd3a814c500 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fd39000c7a0 tx=0x7fd39000cc60 comp rx=0 tx=0).stop 2023-10-16T20:58:49.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.554+0000 7fd3ace32640 1 -- 172.21.15.88:0/422001912 shutdown_connections 2023-10-16T20:58:49.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.554+0000 7fd3ace32640 1 --2- 172.21.15.88:0/422001912 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fd3880664a0 0x7fd388068960 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.554+0000 7fd3ace32640 1 --2- 172.21.15.88:0/422001912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd3a8158ee0 0x7fd3a8141d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.554+0000 7fd3ace32640 1 --2- 172.21.15.88:0/422001912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3a814f7d0 0x7fd3a814c500 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.554+0000 7fd3ace32640 1 --2- 172.21.15.88:0/422001912 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fd3a80c1170 0x7fd3a81eaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.554+0000 7fd3ace32640 1 -- 172.21.15.88:0/422001912 >> 172.21.15.88:0/422001912 conn(0x7fd3a80ca9f0 msgr2=0x7fd3a80c4410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:49.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.554+0000 7fd3ace32640 1 -- 172.21.15.88:0/422001912 shutdown_connections 2023-10-16T20:58:49.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.554+0000 7fd3ace32640 1 -- 172.21.15.88:0/422001912 wait complete. 2023-10-16T20:58:49.577 INFO:tasks.ceph:Scrubbing osd.6 2023-10-16T20:58:49.578 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-16T20:58:49.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.718+0000 7f8f4e194640 1 Processor -- start 2023-10-16T20:58:49.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.718+0000 7f8f4e194640 1 -- start start 2023-10-16T20:58:49.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f4e194640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f480c4260 0x7f8f480c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f4e194640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8f480c5a60 0x7f8f481510d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f4e194640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f48069e90 con 0x7f8f480c4640 2023-10-16T20:58:49.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f4e194640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f480716a0 con 0x7f8f480c5a60 2023-10-16T20:58:49.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f4e194640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4806aea0 con 0x7f8f480c4260 2023-10-16T20:58:49.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f47fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8f480c4640 0x7f8f481573e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:32980/0 (socket says 172.21.15.88:32980) 2023-10-16T20:58:49.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f47fff640 1 -- 172.21.15.88:0/3105469023 learned_addr learned my addr 172.21.15.88:0/3105469023 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:49.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f477fe640 1 --2- 172.21.15.88:0/3105469023 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f480c4260 0x7f8f480c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f46ffd640 1 --2- 172.21.15.88:0/3105469023 >> v2:172.21.15.191:3300/0 conn(0x7f8f480c5a60 0x7f8f481510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f467fc640 1 -- 172.21.15.88:0/3105469023 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99787483 0 0) 0x7f8f48069e90 con 0x7f8f480c4640 2023-10-16T20:58:49.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f467fc640 1 -- 172.21.15.88:0/3105469023 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f30003530 con 0x7f8f480c4640 2023-10-16T20:58:49.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f467fc640 1 -- 172.21.15.88:0/3105469023 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4014624823 0 0) 0x7f8f30003530 con 0x7f8f480c4640 2023-10-16T20:58:49.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f467fc640 1 -- 172.21.15.88:0/3105469023 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f480c4260 msgr2=0x7f8f480c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:49.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f467fc640 1 --2- 172.21.15.88:0/3105469023 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f480c4260 0x7f8f480c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f467fc640 1 -- 172.21.15.88:0/3105469023 >> v2:172.21.15.191:3300/0 conn(0x7f8f480c5a60 msgr2=0x7f8f481510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:49.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f467fc640 1 --2- 172.21.15.88:0/3105469023 >> v2:172.21.15.191:3300/0 conn(0x7f8f480c5a60 0x7f8f481510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f467fc640 1 -- 172.21.15.88:0/3105469023 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f481581e0 con 0x7f8f480c4640 2023-10-16T20:58:49.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f467fc640 1 -- 172.21.15.88:0/3105469023 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 855422095 0 0) 0x7f8f40001a00 con 0x7f8f480c4640 2023-10-16T20:58:49.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f467fc640 1 -- 172.21.15.88:0/3105469023 >> v1:172.21.15.88:6789/0 conn(0x7f8f480c4640 legacy=0x7f8f481573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f467fc640 1 --2- 172.21.15.88:0/3105469023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f30003980 0x7f8f30003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f467fc640 1 --2- 172.21.15.88:0/3105469023 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f300047d0 0x7f8f30004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f467fc640 1 --2- 172.21.15.88:0/3105469023 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f30005190 0x7f8f3000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f467fc640 1 -- 172.21.15.88:0/3105469023 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4806aea0 con 0x7f8f30003980 2023-10-16T20:58:49.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f467fc640 1 -- 172.21.15.88:0/3105469023 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3000bf30 con 0x7f8f300047d0 2023-10-16T20:58:49.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f467fc640 1 -- 172.21.15.88:0/3105469023 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3000c0f0 con 0x7f8f30005190 2023-10-16T20:58:49.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f477fe640 1 --2- 172.21.15.88:0/3105469023 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f480c4260 0x7f8f480c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:58:49.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f477fe640 1 --2- 172.21.15.88:0/3105469023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f30003980 0x7f8f30003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f46ffd640 1 --2- 172.21.15.88:0/3105469023 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f300047d0 0x7f8f30004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f47fff640 1 --2- 172.21.15.88:0/3105469023 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f30005190 0x7f8f3000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f477fe640 1 -- 172.21.15.88:0/3105469023 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f30005190 msgr2=0x7f8f3000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:49.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f477fe640 1 --2- 172.21.15.88:0/3105469023 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f30005190 0x7f8f3000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f477fe640 1 -- 172.21.15.88:0/3105469023 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f300047d0 msgr2=0x7f8f30004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:49.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f477fe640 1 --2- 172.21.15.88:0/3105469023 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f300047d0 0x7f8f30004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.722+0000 7f8f477fe640 1 -- 172.21.15.88:0/3105469023 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f30003530 con 0x7f8f30003980 2023-10-16T20:58:49.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.726+0000 7f8f477fe640 1 --2- 172.21.15.88:0/3105469023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f30003980 0x7f8f30003da0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f8f48073ed0 tx=0x7f8f3801ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8075bcbbe19ed002 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:49.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.726+0000 7f8f467fc640 1 -- 172.21.15.88:0/3105469023 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38004030 con 0x7f8f30003980 2023-10-16T20:58:49.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.726+0000 7f8f467fc640 1 -- 172.21.15.88:0/3105469023 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f38004190 con 0x7f8f30003980 2023-10-16T20:58:49.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.726+0000 7f8f467fc640 1 -- 172.21.15.88:0/3105469023 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38004520 con 0x7f8f30003980 2023-10-16T20:58:49.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.726+0000 7f8f4e194640 1 -- 172.21.15.88:0/3105469023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f30003980 msgr2=0x7f8f30003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.726+0000 7f8f4e194640 1 --2- 172.21.15.88:0/3105469023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f30003980 0x7f8f30003da0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f8f48073ed0 tx=0x7f8f3801ca60 comp rx=0 tx=0).stop 2023-10-16T20:58:49.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.726+0000 7f8f47fff640 1 -- 172.21.15.88:0/3105469023 reap_dead start 2023-10-16T20:58:49.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.726+0000 7f8f4e194640 1 -- 172.21.15.88:0/3105469023 shutdown_connections 2023-10-16T20:58:49.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.726+0000 7f8f4e194640 1 -- 172.21.15.88:0/3105469023 >> 172.21.15.88:0/3105469023 conn(0x7f8f480ca9f0 msgr2=0x7f8f480cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:49.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.726+0000 7f8f4e194640 1 -- 172.21.15.88:0/3105469023 shutdown_connections 2023-10-16T20:58:49.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.726+0000 7f8f4e194640 1 -- 172.21.15.88:0/3105469023 wait complete. 2023-10-16T20:58:49.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.726+0000 7f8f4e194640 1 Processor -- start 2023-10-16T20:58:49.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f4e194640 1 -- start start 2023-10-16T20:58:49.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f4e194640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f30003980 0x7f8f48158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f4e194640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f481efb50 0x7f8f481effb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f4e194640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f481f30c0 0x7f8f481f3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f4e194640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f481f04f0 con 0x7f8f481f30c0 2023-10-16T20:58:49.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f4e194640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f48147560 con 0x7f8f481efb50 2023-10-16T20:58:49.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f4e194640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f48147ae0 con 0x7f8f30003980 2023-10-16T20:58:49.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f47fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f481f30c0 0x7f8f481f3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f47fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f481f30c0 0x7f8f481f3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34024/0 (socket says 172.21.15.88:34024) 2023-10-16T20:58:49.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f47fff640 1 -- 172.21.15.88:0/1386775105 learned_addr learned my addr 172.21.15.88:0/1386775105 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:49.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f46ffd640 1 --2- 172.21.15.88:0/1386775105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f481efb50 0x7f8f481effb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f477fe640 1 --2- 172.21.15.88:0/1386775105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f30003980 0x7f8f48158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f47fff640 1 -- 172.21.15.88:0/1386775105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f30003980 msgr2=0x7f8f48158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f47fff640 1 --2- 172.21.15.88:0/1386775105 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8f30003980 0x7f8f48158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f47fff640 1 -- 172.21.15.88:0/1386775105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f481efb50 msgr2=0x7f8f481effb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f47fff640 1 --2- 172.21.15.88:0/1386775105 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8f481efb50 0x7f8f481effb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f47fff640 1 -- 172.21.15.88:0/1386775105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f48147800 con 0x7f8f481f30c0 2023-10-16T20:58:49.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f47fff640 1 --2- 172.21.15.88:0/1386775105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f481f30c0 0x7f8f481f3540 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f8f4806f690 tx=0x7f8f40007960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:49.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f2ffff640 1 -- 172.21.15.88:0/1386775105 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4000aa40 con 0x7f8f481f30c0 2023-10-16T20:58:49.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f2ffff640 1 -- 172.21.15.88:0/1386775105 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4000abc0 con 0x7f8f481f30c0 2023-10-16T20:58:49.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f2ffff640 1 -- 172.21.15.88:0/1386775105 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f400133f0 con 0x7f8f481f30c0 2023-10-16T20:58:49.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f4e194640 1 -- 172.21.15.88:0/1386775105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f481491f0 con 0x7f8f481f30c0 2023-10-16T20:58:49.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f4e194640 1 -- 172.21.15.88:0/1386775105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f481497b0 con 0x7f8f481f30c0 2023-10-16T20:58:49.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.730+0000 7f8f4e194640 1 -- 172.21.15.88:0/1386775105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8f4806a980 con 0x7f8f481f30c0 2023-10-16T20:58:49.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.734+0000 7f8f2ffff640 1 -- 172.21.15.88:0/1386775105 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8f40013550 con 0x7f8f481f30c0 2023-10-16T20:58:49.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.734+0000 7f8f2ffff640 1 --2- 172.21.15.88:0/1386775105 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8f340664a0 0x7f8f34068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.734+0000 7f8f2ffff640 1 -- 172.21.15.88:0/1386775105 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f8f40084b10 con 0x7f8f481f30c0 2023-10-16T20:58:49.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.734+0000 7f8f477fe640 1 --2- 172.21.15.88:0/1386775105 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8f340664a0 0x7f8f34068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.734+0000 7f8f2ffff640 1 --2- 172.21.15.88:0/1386775105 >> [v2:172.21.15.191:6816/292421315,v1:172.21.15.191:6817/292421315] conn(0x7f8f34070520 0x7f8f340729c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.734+0000 7f8f2ffff640 1 -- 172.21.15.88:0/1386775105 --> [v2:172.21.15.191:6816/292421315,v1:172.21.15.191:6817/292421315] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8f34073090 con 0x7f8f34070520 2023-10-16T20:58:49.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.734+0000 7f8f2ffff640 1 -- 172.21.15.88:0/1386775105 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f400892c0 con 0x7f8f481f30c0 2023-10-16T20:58:49.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.734+0000 7f8f46ffd640 1 --2- 172.21.15.88:0/1386775105 >> [v2:172.21.15.191:6816/292421315,v1:172.21.15.191:6817/292421315] conn(0x7f8f34070520 0x7f8f340729c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.734+0000 7f8f477fe640 1 --2- 172.21.15.88:0/1386775105 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8f340664a0 0x7f8f34068960 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8f380ad000 tx=0x7f8f380af040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:49.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.734+0000 7f8f46ffd640 1 --2- 172.21.15.88:0/1386775105 >> [v2:172.21.15.191:6816/292421315,v1:172.21.15.191:6817/292421315] conn(0x7f8f34070520 0x7f8f340729c0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:49.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.734+0000 7f8f2ffff640 1 -- 172.21.15.88:0/1386775105 <== osd.6 v2:172.21.15.191:6816/292421315 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f8f34073090 con 0x7f8f34070520 2023-10-16T20:58:49.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.746+0000 7f8f4e194640 1 -- 172.21.15.88:0/1386775105 --> [v2:172.21.15.191:6816/292421315,v1:172.21.15.191:6817/292421315] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8f4806a6a0 con 0x7f8f34070520 2023-10-16T20:58:49.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.750+0000 7f8f2ffff640 1 -- 172.21.15.88:0/1386775105 <== osd.6 v2:172.21.15.191:6816/292421315 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f8f3c009770 con 0x7f8f34070520 2023-10-16T20:58:49.759 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-10-16T20:58:49.759 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T20:58:49.759 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-10-16T20:58:49.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.750+0000 7f8f4e194640 1 -- 172.21.15.88:0/1386775105 >> [v2:172.21.15.191:6816/292421315,v1:172.21.15.191:6817/292421315] conn(0x7f8f34070520 msgr2=0x7f8f340729c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.750+0000 7f8f4e194640 1 --2- 172.21.15.88:0/1386775105 >> [v2:172.21.15.191:6816/292421315,v1:172.21.15.191:6817/292421315] conn(0x7f8f34070520 0x7f8f340729c0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.750+0000 7f8f4e194640 1 -- 172.21.15.88:0/1386775105 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8f340664a0 msgr2=0x7f8f34068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.750+0000 7f8f4e194640 1 --2- 172.21.15.88:0/1386775105 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8f340664a0 0x7f8f34068960 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8f380ad000 tx=0x7f8f380af040 comp rx=0 tx=0).stop 2023-10-16T20:58:49.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.750+0000 7f8f4e194640 1 -- 172.21.15.88:0/1386775105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f481f30c0 msgr2=0x7f8f481f3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.750+0000 7f8f4e194640 1 --2- 172.21.15.88:0/1386775105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f481f30c0 0x7f8f481f3540 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f8f4806f690 tx=0x7f8f40007960 comp rx=0 tx=0).stop 2023-10-16T20:58:49.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.750+0000 7f8f47fff640 1 -- 172.21.15.88:0/1386775105 reap_dead start 2023-10-16T20:58:49.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.750+0000 7f8f4e194640 1 -- 172.21.15.88:0/1386775105 shutdown_connections 2023-10-16T20:58:49.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.750+0000 7f8f4e194640 1 -- 172.21.15.88:0/1386775105 >> 172.21.15.88:0/1386775105 conn(0x7f8f480ca9f0 msgr2=0x7f8f480c88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:49.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.750+0000 7f8f4e194640 1 -- 172.21.15.88:0/1386775105 shutdown_connections 2023-10-16T20:58:49.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.750+0000 7f8f4e194640 1 -- 172.21.15.88:0/1386775105 wait complete. 2023-10-16T20:58:49.770 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-16T20:58:49.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.906+0000 7f1e9b205640 1 Processor -- start 2023-10-16T20:58:49.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.906+0000 7f1e9b205640 1 -- start start 2023-10-16T20:58:49.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.910+0000 7f1e9b205640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1e94153620 0x7f1e940c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.910+0000 7f1e9b205640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e94153a00 0x7f1e940c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.910+0000 7f1e9b205640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e940c0230 con 0x7f1e940ad870 2023-10-16T20:58:49.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.910+0000 7f1e9b205640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e941594f0 con 0x7f1e94153620 2023-10-16T20:58:49.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.910+0000 7f1e9b205640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e940c0480 con 0x7f1e94153a00 2023-10-16T20:58:49.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.910+0000 7f1e98fa1640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1e94153620 0x7f1e940c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.910+0000 7f1e93fff640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e94153a00 0x7f1e940c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.910+0000 7f1e997a2640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f1e940ad870 0x7f1e940cb280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:36342/0 (socket says 172.21.15.88:36342) 2023-10-16T20:58:49.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.910+0000 7f1e997a2640 1 -- 172.21.15.88:0/2359364887 learned_addr learned my addr 172.21.15.88:0/2359364887 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:58:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.910+0000 7f1e93fff640 1 -- 172.21.15.88:0/2359364887 >> v2:172.21.15.191:3300/0 conn(0x7f1e94153620 msgr2=0x7f1e940c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.910+0000 7f1e93fff640 1 --2- 172.21.15.88:0/2359364887 >> v2:172.21.15.191:3300/0 conn(0x7f1e94153620 0x7f1e940c5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.910+0000 7f1e93fff640 1 -- 172.21.15.88:0/2359364887 >> v1:172.21.15.191:6789/0 conn(0x7f1e940ad870 legacy=0x7f1e940cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.910+0000 7f1e93fff640 1 -- 172.21.15.88:0/2359364887 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e940c0760 con 0x7f1e94153a00 2023-10-16T20:58:49.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.910+0000 7f1e93fff640 1 --2- 172.21.15.88:0/2359364887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e94153a00 0x7f1e940c61b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1e800097c0 tx=0x7f1e8001c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fa0b7ff629f5da3 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:49.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e937fe640 1 -- 172.21.15.88:0/2359364887 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e800ab600 con 0x7f1e94153a00 2023-10-16T20:58:49.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e937fe640 1 -- 172.21.15.88:0/2359364887 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e800a7050 con 0x7f1e94153a00 2023-10-16T20:58:49.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e937fe640 1 -- 172.21.15.88:0/2359364887 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e800ab980 con 0x7f1e94153a00 2023-10-16T20:58:49.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e9b205640 1 -- 172.21.15.88:0/2359364887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e94153a00 msgr2=0x7f1e940c61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e9b205640 1 --2- 172.21.15.88:0/2359364887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e94153a00 0x7f1e940c61b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1e800097c0 tx=0x7f1e8001c480 comp rx=0 tx=0).stop 2023-10-16T20:58:49.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e9b205640 1 -- 172.21.15.88:0/2359364887 shutdown_connections 2023-10-16T20:58:49.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e9b205640 1 --2- 172.21.15.88:0/2359364887 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e94153a00 0x7f1e940c61b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e9b205640 1 --2- 172.21.15.88:0/2359364887 >> v2:172.21.15.191:3300/0 conn(0x7f1e94153620 0x7f1e940c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e9b205640 1 -- 172.21.15.88:0/2359364887 >> 172.21.15.88:0/2359364887 conn(0x7f1e94069e90 msgr2=0x7f1e9406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:49.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e9b205640 1 -- 172.21.15.88:0/2359364887 shutdown_connections 2023-10-16T20:58:49.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e9b205640 1 -- 172.21.15.88:0/2359364887 wait complete. 2023-10-16T20:58:49.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e9b205640 1 Processor -- start 2023-10-16T20:58:49.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e9b205640 1 -- start start 2023-10-16T20:58:49.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e9b205640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e940ad870 0x7f1e9415a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e9b205640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e94153620 0x7f1e940c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e9b205640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e94153a00 0x7f1e9415bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e9b205640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9413dc80 con 0x7f1e94153a00 2023-10-16T20:58:49.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e9b205640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9413eec0 con 0x7f1e940ad870 2023-10-16T20:58:49.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e9b205640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9413f1a0 con 0x7f1e94153620 2023-10-16T20:58:49.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e997a2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e94153a00 0x7f1e9415bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e997a2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e94153a00 0x7f1e9415bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34032/0 (socket says 172.21.15.88:34032) 2023-10-16T20:58:49.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e997a2640 1 -- 172.21.15.88:0/704473230 learned_addr learned my addr 172.21.15.88:0/704473230 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:49.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.914+0000 7f1e98fa1640 1 --2- 172.21.15.88:0/704473230 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e940ad870 0x7f1e9415a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.918+0000 7f1e93fff640 1 --2- 172.21.15.88:0/704473230 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e94153620 0x7f1e940c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.918+0000 7f1e997a2640 1 -- 172.21.15.88:0/704473230 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e94153620 msgr2=0x7f1e940c4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.918+0000 7f1e997a2640 1 --2- 172.21.15.88:0/704473230 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e94153620 0x7f1e940c4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.918+0000 7f1e997a2640 1 -- 172.21.15.88:0/704473230 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e940ad870 msgr2=0x7f1e9415a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:49.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.918+0000 7f1e997a2640 1 --2- 172.21.15.88:0/704473230 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e940ad870 0x7f1e9415a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:49.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.918+0000 7f1e997a2640 1 -- 172.21.15.88:0/704473230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e80009420 con 0x7f1e94153a00 2023-10-16T20:58:49.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.918+0000 7f1e997a2640 1 --2- 172.21.15.88:0/704473230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e94153a00 0x7f1e9415bef0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00cbc0 tx=0x7f1e8c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:49.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.918+0000 7f1e917fa640 1 -- 172.21.15.88:0/704473230 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c00ad40 con 0x7f1e94153a00 2023-10-16T20:58:49.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.918+0000 7f1e917fa640 1 -- 172.21.15.88:0/704473230 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8c012400 con 0x7f1e94153a00 2023-10-16T20:58:49.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.918+0000 7f1e9b205640 1 -- 172.21.15.88:0/704473230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e9413f480 con 0x7f1e94153a00 2023-10-16T20:58:49.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.918+0000 7f1e917fa640 1 -- 172.21.15.88:0/704473230 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c00aea0 con 0x7f1e94153a00 2023-10-16T20:58:49.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.918+0000 7f1e9b205640 1 -- 172.21.15.88:0/704473230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e940712a0 con 0x7f1e94153a00 2023-10-16T20:58:49.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.918+0000 7f1e9b205640 1 -- 172.21.15.88:0/704473230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e9415bc10 con 0x7f1e94153a00 2023-10-16T20:58:49.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.918+0000 7f1e917fa640 1 -- 172.21.15.88:0/704473230 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1e8c007070 con 0x7f1e94153a00 2023-10-16T20:58:49.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.918+0000 7f1e917fa640 1 --2- 172.21.15.88:0/704473230 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1e780664d0 0x7f1e78068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:49.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.918+0000 7f1e917fa640 1 -- 172.21.15.88:0/704473230 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f1e8c084410 con 0x7f1e94153a00 2023-10-16T20:58:49.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.918+0000 7f1e98fa1640 1 --2- 172.21.15.88:0/704473230 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1e780664d0 0x7f1e78068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:49.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.922+0000 7f1e98fa1640 1 --2- 172.21.15.88:0/704473230 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1e780664d0 0x7f1e78068990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1e88009b60 tx=0x7f1e88009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:49.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:49.922+0000 7f1e917fa640 1 -- 172.21.15.88:0/704473230 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1e9415bc10 con 0x7f1e94153a00 2023-10-16T20:58:50.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.078+0000 7f1e9b205640 1 -- 172.21.15.88:0/704473230 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e9413f310 con 0x7f1e780664d0 2023-10-16T20:58:50.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.078+0000 7f1e917fa640 1 -- 172.21.15.88:0/704473230 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1e9413f310 con 0x7f1e780664d0 2023-10-16T20:58:50.079 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 6 to deep-scrub 2023-10-16T20:58:50.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.078+0000 7f1e9b205640 1 -- 172.21.15.88:0/704473230 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1e780664d0 msgr2=0x7f1e78068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.078+0000 7f1e9b205640 1 --2- 172.21.15.88:0/704473230 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1e780664d0 0x7f1e78068990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1e88009b60 tx=0x7f1e88009190 comp rx=0 tx=0).stop 2023-10-16T20:58:50.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.078+0000 7f1e9b205640 1 -- 172.21.15.88:0/704473230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e94153a00 msgr2=0x7f1e9415bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.078+0000 7f1e9b205640 1 --2- 172.21.15.88:0/704473230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e94153a00 0x7f1e9415bef0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00cbc0 tx=0x7f1e8c00a430 comp rx=0 tx=0).stop 2023-10-16T20:58:50.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.078+0000 7f1e9b205640 1 -- 172.21.15.88:0/704473230 shutdown_connections 2023-10-16T20:58:50.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.078+0000 7f1e9b205640 1 --2- 172.21.15.88:0/704473230 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f1e780664d0 0x7f1e78068990 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.078+0000 7f1e9b205640 1 --2- 172.21.15.88:0/704473230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e94153a00 0x7f1e9415bef0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.078+0000 7f1e9b205640 1 --2- 172.21.15.88:0/704473230 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f1e94153620 0x7f1e940c4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.078+0000 7f1e9b205640 1 --2- 172.21.15.88:0/704473230 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e940ad870 0x7f1e9415a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.078+0000 7f1e9b205640 1 -- 172.21.15.88:0/704473230 >> 172.21.15.88:0/704473230 conn(0x7f1e94069e90 msgr2=0x7f1e940caf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:50.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.078+0000 7f1e9b205640 1 -- 172.21.15.88:0/704473230 shutdown_connections 2023-10-16T20:58:50.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.078+0000 7f1e9b205640 1 -- 172.21.15.88:0/704473230 wait complete. 2023-10-16T20:58:50.101 INFO:tasks.ceph:Scrubbing osd.7 2023-10-16T20:58:50.101 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-16T20:58:50.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.242+0000 7f8d28394640 1 Processor -- start 2023-10-16T20:58:50.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.242+0000 7f8d28394640 1 -- start start 2023-10-16T20:58:50.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.242+0000 7f8d28394640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f8d20158ee0 0x7f8d2006a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.242+0000 7f8d28394640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8d2006ac60 0x7f8d200c1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.242+0000 7f8d28394640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d200c72a0 con 0x7f8d2014f7d0 2023-10-16T20:58:50.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.242+0000 7f8d28394640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d2006b070 con 0x7f8d2006ac60 2023-10-16T20:58:50.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.242+0000 7f8d28394640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d200cb6a0 con 0x7f8d20158ee0 2023-10-16T20:58:50.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.242+0000 7f8d26931640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8d2006ac60 0x7f8d200c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.242+0000 7f8d26931640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8d2006ac60 0x7f8d200c1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34042/0 (socket says 172.21.15.88:34042) 2023-10-16T20:58:50.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.242+0000 7f8d26931640 1 -- 172.21.15.88:0/1564003935 learned_addr learned my addr 172.21.15.88:0/1564003935 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:50.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.242+0000 7f8d2592f640 1 --2- 172.21.15.88:0/1564003935 >> v2:172.21.15.191:3300/0 conn(0x7f8d20158ee0 0x7f8d2006a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.242+0000 7f8d26931640 1 -- 172.21.15.88:0/1564003935 >> v2:172.21.15.191:3300/0 conn(0x7f8d20158ee0 msgr2=0x7f8d2006a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:50.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.242+0000 7f8d26931640 1 --2- 172.21.15.88:0/1564003935 >> v2:172.21.15.191:3300/0 conn(0x7f8d20158ee0 0x7f8d2006a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.242+0000 7f8d26931640 1 -- 172.21.15.88:0/1564003935 >> v1:172.21.15.191:6789/0 conn(0x7f8d2014f7d0 legacy=0x7f8d20156270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:50.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.242+0000 7f8d26931640 1 -- 172.21.15.88:0/1564003935 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d200c6a00 con 0x7f8d2006ac60 2023-10-16T20:58:50.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.246+0000 7f8d26931640 1 --2- 172.21.15.88:0/1564003935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d2006ac60 0x7f8d200c1170 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f8d200c5010 tx=0x7f8d1801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccefc378ee58ceae server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:50.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.246+0000 7f8d2512e640 1 -- 172.21.15.88:0/1564003935 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d180af400 con 0x7f8d2006ac60 2023-10-16T20:58:50.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.246+0000 7f8d2512e640 1 -- 172.21.15.88:0/1564003935 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d180ab050 con 0x7f8d2006ac60 2023-10-16T20:58:50.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.246+0000 7f8d2512e640 1 -- 172.21.15.88:0/1564003935 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d180af780 con 0x7f8d2006ac60 2023-10-16T20:58:50.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.246+0000 7f8d28394640 1 -- 172.21.15.88:0/1564003935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d2006ac60 msgr2=0x7f8d200c1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.246+0000 7f8d28394640 1 --2- 172.21.15.88:0/1564003935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d2006ac60 0x7f8d200c1170 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f8d200c5010 tx=0x7f8d1801f1c0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.246+0000 7f8d28394640 1 -- 172.21.15.88:0/1564003935 shutdown_connections 2023-10-16T20:58:50.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.246+0000 7f8d28394640 1 --2- 172.21.15.88:0/1564003935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d2006ac60 0x7f8d200c1170 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.255 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.246+0000 7f8d28394640 1 --2- 172.21.15.88:0/1564003935 >> v2:172.21.15.191:3300/0 conn(0x7f8d20158ee0 0x7f8d2006a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.246+0000 7f8d28394640 1 -- 172.21.15.88:0/1564003935 >> 172.21.15.88:0/1564003935 conn(0x7f8d200ca9f0 msgr2=0x7f8d200cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:50.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.246+0000 7f8d28394640 1 -- 172.21.15.88:0/1564003935 shutdown_connections 2023-10-16T20:58:50.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.246+0000 7f8d28394640 1 -- 172.21.15.88:0/1564003935 wait complete. 2023-10-16T20:58:50.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d28394640 1 Processor -- start 2023-10-16T20:58:50.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d28394640 1 -- start start 2023-10-16T20:58:50.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d28394640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d2006ac60 0x7f8d200706c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d28394640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d2014f7d0 0x7f8d200c1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d28394640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d20158ee0 0x7f8d2014a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d28394640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d20152170 con 0x7f8d2014f7d0 2023-10-16T20:58:50.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d28394640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d20153100 con 0x7f8d2006ac60 2023-10-16T20:58:50.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d28394640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d200c1f10 con 0x7f8d20158ee0 2023-10-16T20:58:50.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d2592f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d2014f7d0 0x7f8d200c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d2592f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d2014f7d0 0x7f8d200c1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34048/0 (socket says 172.21.15.88:34048) 2023-10-16T20:58:50.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d2592f640 1 -- 172.21.15.88:0/3621393751 learned_addr learned my addr 172.21.15.88:0/3621393751 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:50.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d26130640 1 --2- 172.21.15.88:0/3621393751 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d2006ac60 0x7f8d200706c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d26931640 1 --2- 172.21.15.88:0/3621393751 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d20158ee0 0x7f8d2014a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d2592f640 1 -- 172.21.15.88:0/3621393751 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d20158ee0 msgr2=0x7f8d2014a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d2592f640 1 --2- 172.21.15.88:0/3621393751 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d20158ee0 0x7f8d2014a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d2592f640 1 -- 172.21.15.88:0/3621393751 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d2006ac60 msgr2=0x7f8d200706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d2592f640 1 --2- 172.21.15.88:0/3621393751 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8d2006ac60 0x7f8d200706c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d2592f640 1 -- 172.21.15.88:0/3621393751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d18009b30 con 0x7f8d2014f7d0 2023-10-16T20:58:50.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d26931640 1 --2- 172.21.15.88:0/3621393751 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f8d20158ee0 0x7f8d2014a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:58:50.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d2592f640 1 --2- 172.21.15.88:0/3621393751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d2014f7d0 0x7f8d200c1540 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000d5f0 tx=0x7f8d1000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:50.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d16ffd640 1 -- 172.21.15.88:0/3621393751 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d100125f0 con 0x7f8d2014f7d0 2023-10-16T20:58:50.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d16ffd640 1 -- 172.21.15.88:0/3621393751 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d10012750 con 0x7f8d2014f7d0 2023-10-16T20:58:50.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d28394640 1 -- 172.21.15.88:0/3621393751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d201524f0 con 0x7f8d2014f7d0 2023-10-16T20:58:50.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d16ffd640 1 -- 172.21.15.88:0/3621393751 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10012ae0 con 0x7f8d2014f7d0 2023-10-16T20:58:50.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.250+0000 7f8d28394640 1 -- 172.21.15.88:0/3621393751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d20152ab0 con 0x7f8d2014f7d0 2023-10-16T20:58:50.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.254+0000 7f8d28394640 1 -- 172.21.15.88:0/3621393751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8d200c6fc0 con 0x7f8d2014f7d0 2023-10-16T20:58:50.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.254+0000 7f8d16ffd640 1 -- 172.21.15.88:0/3621393751 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8d10012c40 con 0x7f8d2014f7d0 2023-10-16T20:58:50.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.254+0000 7f8d16ffd640 1 --2- 172.21.15.88:0/3621393751 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8d040664d0 0x7f8d04068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.254+0000 7f8d16ffd640 1 -- 172.21.15.88:0/3621393751 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f8d10084a60 con 0x7f8d2014f7d0 2023-10-16T20:58:50.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.254+0000 7f8d26130640 1 --2- 172.21.15.88:0/3621393751 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8d040664d0 0x7f8d04068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.254+0000 7f8d16ffd640 1 --2- 172.21.15.88:0/3621393751 >> [v2:172.21.15.191:6824/4079140119,v1:172.21.15.191:6825/4079140119] conn(0x7f8d04070550 0x7f8d040729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.254+0000 7f8d16ffd640 1 -- 172.21.15.88:0/3621393751 --> [v2:172.21.15.191:6824/4079140119,v1:172.21.15.191:6825/4079140119] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d040730c0 con 0x7f8d04070550 2023-10-16T20:58:50.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.254+0000 7f8d16ffd640 1 -- 172.21.15.88:0/3621393751 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d100892a0 con 0x7f8d2014f7d0 2023-10-16T20:58:50.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.254+0000 7f8d26931640 1 --2- 172.21.15.88:0/3621393751 >> [v2:172.21.15.191:6824/4079140119,v1:172.21.15.191:6825/4079140119] conn(0x7f8d04070550 0x7f8d040729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.254+0000 7f8d26130640 1 --2- 172.21.15.88:0/3621393751 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8d040664d0 0x7f8d04068990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c000f70 tx=0x7f8d1c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:50.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.254+0000 7f8d26931640 1 --2- 172.21.15.88:0/3621393751 >> [v2:172.21.15.191:6824/4079140119,v1:172.21.15.191:6825/4079140119] conn(0x7f8d04070550 0x7f8d040729f0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:50.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.258+0000 7f8d16ffd640 1 -- 172.21.15.88:0/3621393751 <== osd.7 v2:172.21.15.191:6824/4079140119 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f8d040730c0 con 0x7f8d04070550 2023-10-16T20:58:50.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.270+0000 7f8d28394640 1 -- 172.21.15.88:0/3621393751 --> [v2:172.21.15.191:6824/4079140119,v1:172.21.15.191:6825/4079140119] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8d200c6ce0 con 0x7f8d04070550 2023-10-16T20:58:50.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.270+0000 7f8d16ffd640 1 -- 172.21.15.88:0/3621393751 <== osd.7 v2:172.21.15.191:6824/4079140119 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f8d180b9440 con 0x7f8d04070550 2023-10-16T20:58:50.274 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-10-16T20:58:50.275 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-16T20:58:50.275 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-10-16T20:58:50.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.270+0000 7f8d28394640 1 -- 172.21.15.88:0/3621393751 >> [v2:172.21.15.191:6824/4079140119,v1:172.21.15.191:6825/4079140119] conn(0x7f8d04070550 msgr2=0x7f8d040729f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.270+0000 7f8d28394640 1 --2- 172.21.15.88:0/3621393751 >> [v2:172.21.15.191:6824/4079140119,v1:172.21.15.191:6825/4079140119] conn(0x7f8d04070550 0x7f8d040729f0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.270+0000 7f8d28394640 1 -- 172.21.15.88:0/3621393751 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8d040664d0 msgr2=0x7f8d04068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.270+0000 7f8d28394640 1 --2- 172.21.15.88:0/3621393751 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f8d040664d0 0x7f8d04068990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c000f70 tx=0x7f8d1c009040 comp rx=0 tx=0).stop 2023-10-16T20:58:50.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.270+0000 7f8d28394640 1 -- 172.21.15.88:0/3621393751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d2014f7d0 msgr2=0x7f8d200c1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.270+0000 7f8d28394640 1 --2- 172.21.15.88:0/3621393751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d2014f7d0 0x7f8d200c1540 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000d5f0 tx=0x7f8d1000dab0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.270+0000 7f8d26931640 1 -- 172.21.15.88:0/3621393751 reap_dead start 2023-10-16T20:58:50.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.270+0000 7f8d28394640 1 -- 172.21.15.88:0/3621393751 shutdown_connections 2023-10-16T20:58:50.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.270+0000 7f8d28394640 1 -- 172.21.15.88:0/3621393751 >> 172.21.15.88:0/3621393751 conn(0x7f8d200ca9f0 msgr2=0x7f8d2014cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:50.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.270+0000 7f8d28394640 1 -- 172.21.15.88:0/3621393751 shutdown_connections 2023-10-16T20:58:50.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.270+0000 7f8d28394640 1 -- 172.21.15.88:0/3621393751 wait complete. 2023-10-16T20:58:50.291 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-16T20:58:50.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.426+0000 7f9c7b4d6640 1 Processor -- start 2023-10-16T20:58:50.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.426+0000 7f9c7b4d6640 1 -- start start 2023-10-16T20:58:50.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.430+0000 7f9c7b4d6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9c74153620 0x7f9c740c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.430+0000 7f9c7b4d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c74153a00 0x7f9c740c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.430+0000 7f9c7b4d6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c740c0230 con 0x7f9c740ad870 2023-10-16T20:58:50.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.430+0000 7f9c7b4d6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c741594f0 con 0x7f9c74153620 2023-10-16T20:58:50.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.430+0000 7f9c7b4d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c740c0480 con 0x7f9c74153a00 2023-10-16T20:58:50.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.430+0000 7f9c78a71640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c74153a00 0x7f9c740c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.430+0000 7f9c79272640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f9c74153620 0x7f9c740c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.430+0000 7f9c78a71640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c74153a00 0x7f9c740c61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:60384/0 (socket says 172.21.15.88:60384) 2023-10-16T20:58:50.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.430+0000 7f9c78a71640 1 -- 172.21.15.88:0/3508508136 learned_addr learned my addr 172.21.15.88:0/3508508136 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:50.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.430+0000 7f9c6bfff640 1 -- 172.21.15.88:0/3508508136 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197434315 0 0) 0x7f9c740c0230 con 0x7f9c740ad870 2023-10-16T20:58:50.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.430+0000 7f9c6bfff640 1 -- 172.21.15.88:0/3508508136 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c60003530 con 0x7f9c740ad870 2023-10-16T20:58:50.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.430+0000 7f9c78a71640 1 -- 172.21.15.88:0/3508508136 >> v2:172.21.15.191:3300/0 conn(0x7f9c74153620 msgr2=0x7f9c740c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:50.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.430+0000 7f9c78a71640 1 --2- 172.21.15.88:0/3508508136 >> v2:172.21.15.191:3300/0 conn(0x7f9c74153620 0x7f9c740c5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.430+0000 7f9c78a71640 1 -- 172.21.15.88:0/3508508136 >> v1:172.21.15.191:6789/0 conn(0x7f9c740ad870 legacy=0x7f9c740cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.430+0000 7f9c78a71640 1 -- 172.21.15.88:0/3508508136 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c740c0760 con 0x7f9c74153a00 2023-10-16T20:58:50.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.430+0000 7f9c79272640 1 --2- 172.21.15.88:0/3508508136 >> v2:172.21.15.191:3300/0 conn(0x7f9c74153620 0x7f9c740c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:58:50.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.434+0000 7f9c78a71640 1 --2- 172.21.15.88:0/3508508136 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c74153a00 0x7f9c740c61b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9c64009870 tx=0x7f9c6401c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e13a357be435615c server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:50.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.434+0000 7f9c6bfff640 1 -- 172.21.15.88:0/3508508136 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c640a7070 con 0x7f9c74153a00 2023-10-16T20:58:50.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.434+0000 7f9c6bfff640 1 -- 172.21.15.88:0/3508508136 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c640ab430 con 0x7f9c74153a00 2023-10-16T20:58:50.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.434+0000 7f9c6bfff640 1 -- 172.21.15.88:0/3508508136 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c640ab7c0 con 0x7f9c74153a00 2023-10-16T20:58:50.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.434+0000 7f9c7b4d6640 1 -- 172.21.15.88:0/3508508136 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c74153a00 msgr2=0x7f9c740c61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.434+0000 7f9c7b4d6640 1 --2- 172.21.15.88:0/3508508136 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c74153a00 0x7f9c740c61b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9c64009870 tx=0x7f9c6401c540 comp rx=0 tx=0).stop 2023-10-16T20:58:50.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.434+0000 7f9c7b4d6640 1 -- 172.21.15.88:0/3508508136 shutdown_connections 2023-10-16T20:58:50.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.434+0000 7f9c7b4d6640 1 --2- 172.21.15.88:0/3508508136 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c74153a00 0x7f9c740c61b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.434+0000 7f9c7b4d6640 1 --2- 172.21.15.88:0/3508508136 >> v2:172.21.15.191:3300/0 conn(0x7f9c74153620 0x7f9c740c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.434+0000 7f9c7b4d6640 1 -- 172.21.15.88:0/3508508136 >> 172.21.15.88:0/3508508136 conn(0x7f9c74069e90 msgr2=0x7f9c7406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:50.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.434+0000 7f9c7b4d6640 1 -- 172.21.15.88:0/3508508136 shutdown_connections 2023-10-16T20:58:50.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.434+0000 7f9c7b4d6640 1 -- 172.21.15.88:0/3508508136 wait complete. 2023-10-16T20:58:50.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.434+0000 7f9c7b4d6640 1 Processor -- start 2023-10-16T20:58:50.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.434+0000 7f9c7b4d6640 1 -- start start 2023-10-16T20:58:50.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c7b4d6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c740ad870 0x7f9c74156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c7b4d6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c74153620 0x7f9c74071770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c7b4d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c74153a00 0x7f9c741f2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c7b4d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7413c3e0 con 0x7f9c74153a00 2023-10-16T20:58:50.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c7b4d6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c740c7400 con 0x7f9c740ad870 2023-10-16T20:58:50.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c7b4d6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c741f1e90 con 0x7f9c74153620 2023-10-16T20:58:50.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c79a73640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c74153a00 0x7f9c741f2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c79a73640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c74153a00 0x7f9c741f2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34056/0 (socket says 172.21.15.88:34056) 2023-10-16T20:58:50.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c79a73640 1 -- 172.21.15.88:0/3084839443 learned_addr learned my addr 172.21.15.88:0/3084839443 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:50.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c79272640 1 --2- 172.21.15.88:0/3084839443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c740ad870 0x7f9c74156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c78a71640 1 --2- 172.21.15.88:0/3084839443 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c74153620 0x7f9c74071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c79a73640 1 -- 172.21.15.88:0/3084839443 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c74153620 msgr2=0x7f9c74071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c79a73640 1 --2- 172.21.15.88:0/3084839443 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c74153620 0x7f9c74071770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c79a73640 1 -- 172.21.15.88:0/3084839443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c740ad870 msgr2=0x7f9c74156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c79a73640 1 --2- 172.21.15.88:0/3084839443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c740ad870 0x7f9c74156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c79a73640 1 -- 172.21.15.88:0/3084839443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c70009420 con 0x7f9c74153a00 2023-10-16T20:58:50.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c79a73640 1 --2- 172.21.15.88:0/3084839443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c74153a00 0x7f9c741f2370 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c00c7c0 tx=0x7f9c6c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:50.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c69ffb640 1 -- 172.21.15.88:0/3084839443 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c003b80 con 0x7f9c74153a00 2023-10-16T20:58:50.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c69ffb640 1 -- 172.21.15.88:0/3084839443 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6c003ce0 con 0x7f9c74153a00 2023-10-16T20:58:50.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c7b4d6640 1 -- 172.21.15.88:0/3084839443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c640094d0 con 0x7f9c74153a00 2023-10-16T20:58:50.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c69ffb640 1 -- 172.21.15.88:0/3084839443 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c00aa50 con 0x7f9c74153a00 2023-10-16T20:58:50.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c7b4d6640 1 -- 172.21.15.88:0/3084839443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7413c8d0 con 0x7f9c74153a00 2023-10-16T20:58:50.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c7b4d6640 1 -- 172.21.15.88:0/3084839443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c741f2050 con 0x7f9c74153a00 2023-10-16T20:58:50.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.438+0000 7f9c69ffb640 1 -- 172.21.15.88:0/3084839443 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9c6c00abb0 con 0x7f9c74153a00 2023-10-16T20:58:50.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.442+0000 7f9c69ffb640 1 --2- 172.21.15.88:0/3084839443 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9c580664d0 0x7f9c58068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.442+0000 7f9c69ffb640 1 -- 172.21.15.88:0/3084839443 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f9c6c084270 con 0x7f9c74153a00 2023-10-16T20:58:50.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.442+0000 7f9c79272640 1 --2- 172.21.15.88:0/3084839443 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9c580664d0 0x7f9c58068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.442+0000 7f9c79272640 1 --2- 172.21.15.88:0/3084839443 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9c580664d0 0x7f9c58068990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9c70009cf0 tx=0x7f9c70009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:50.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.446+0000 7f9c69ffb640 1 -- 172.21.15.88:0/3084839443 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9c6c051110 con 0x7f9c74153a00 2023-10-16T20:58:50.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.598+0000 7f9c7b4d6640 1 -- 172.21.15.88:0/3084839443 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c74165e80 con 0x7f9c580664d0 2023-10-16T20:58:50.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.598+0000 7f9c69ffb640 1 -- 172.21.15.88:0/3084839443 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9c70009940 con 0x7f9c580664d0 2023-10-16T20:58:50.600 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 7 to deep-scrub 2023-10-16T20:58:50.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.598+0000 7f9c7b4d6640 1 -- 172.21.15.88:0/3084839443 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9c580664d0 msgr2=0x7f9c58068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.598+0000 7f9c7b4d6640 1 --2- 172.21.15.88:0/3084839443 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9c580664d0 0x7f9c58068990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9c70009cf0 tx=0x7f9c70009190 comp rx=0 tx=0).stop 2023-10-16T20:58:50.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.598+0000 7f9c7b4d6640 1 -- 172.21.15.88:0/3084839443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c74153a00 msgr2=0x7f9c741f2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.598+0000 7f9c7b4d6640 1 --2- 172.21.15.88:0/3084839443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c74153a00 0x7f9c741f2370 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c00c7c0 tx=0x7f9c6c00cc80 comp rx=0 tx=0).stop 2023-10-16T20:58:50.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.598+0000 7f9c7b4d6640 1 -- 172.21.15.88:0/3084839443 shutdown_connections 2023-10-16T20:58:50.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.598+0000 7f9c7b4d6640 1 --2- 172.21.15.88:0/3084839443 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f9c580664d0 0x7f9c58068990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.598+0000 7f9c7b4d6640 1 --2- 172.21.15.88:0/3084839443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c74153a00 0x7f9c741f2370 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.598+0000 7f9c7b4d6640 1 --2- 172.21.15.88:0/3084839443 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f9c74153620 0x7f9c74071770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.598+0000 7f9c7b4d6640 1 --2- 172.21.15.88:0/3084839443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c740ad870 0x7f9c74156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.598+0000 7f9c7b4d6640 1 -- 172.21.15.88:0/3084839443 >> 172.21.15.88:0/3084839443 conn(0x7f9c74069e90 msgr2=0x7f9c740c5b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:50.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.598+0000 7f9c7b4d6640 1 -- 172.21.15.88:0/3084839443 shutdown_connections 2023-10-16T20:58:50.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.598+0000 7f9c7b4d6640 1 -- 172.21.15.88:0/3084839443 wait complete. 2023-10-16T20:58:50.621 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T20:58:50.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.762+0000 7ff450cd6640 1 Processor -- start 2023-10-16T20:58:50.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.762+0000 7ff450cd6640 1 -- start start 2023-10-16T20:58:50.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.762+0000 7ff450cd6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff44c153770 0x7ff44c159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.762+0000 7ff450cd6640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7ff44c0c9bb0 0x7ff44c0c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.762+0000 7ff450cd6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff44c15a410 con 0x7ff44c0c41f0 2023-10-16T20:58:50.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.762+0000 7ff450cd6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff44c0cc250 con 0x7ff44c153770 2023-10-16T20:58:50.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.762+0000 7ff450cd6640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff44c0cc670 con 0x7ff44c0c9bb0 2023-10-16T20:58:50.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.762+0000 7ff44a59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff44c153770 0x7ff44c159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.762+0000 7ff44a59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff44c153770 0x7ff44c159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34060/0 (socket says 172.21.15.88:34060) 2023-10-16T20:58:50.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.762+0000 7ff44a59c640 1 -- 172.21.15.88:0/4198915884 learned_addr learned my addr 172.21.15.88:0/4198915884 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:50.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.762+0000 7ff449d9b640 1 --2- 172.21.15.88:0/4198915884 >> v2:172.21.15.191:3300/0 conn(0x7ff44c0c9bb0 0x7ff44c0c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.762+0000 7ff44a59c640 1 -- 172.21.15.88:0/4198915884 >> v2:172.21.15.191:3300/0 conn(0x7ff44c0c9bb0 msgr2=0x7ff44c0c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:58:50.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.762+0000 7ff44a59c640 1 --2- 172.21.15.88:0/4198915884 >> v2:172.21.15.191:3300/0 conn(0x7ff44c0c9bb0 0x7ff44c0c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.762+0000 7ff44a59c640 1 -- 172.21.15.88:0/4198915884 >> v1:172.21.15.191:6789/0 conn(0x7ff44c0c41f0 legacy=0x7ff44c0cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.762+0000 7ff44a59c640 1 -- 172.21.15.88:0/4198915884 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44c153b50 con 0x7ff44c153770 2023-10-16T20:58:50.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.766+0000 7ff44a59c640 1 --2- 172.21.15.88:0/4198915884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff44c153770 0x7ff44c159ed0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7ff44c0cc7b0 tx=0x7ff43c01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3664b212403babf1 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:50.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.766+0000 7ff44959a640 1 -- 172.21.15.88:0/4198915884 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43c0ab5f0 con 0x7ff44c153770 2023-10-16T20:58:50.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.766+0000 7ff44959a640 1 -- 172.21.15.88:0/4198915884 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff43c0ab750 con 0x7ff44c153770 2023-10-16T20:58:50.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.766+0000 7ff44959a640 1 -- 172.21.15.88:0/4198915884 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43c0abae0 con 0x7ff44c153770 2023-10-16T20:58:50.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.766+0000 7ff450cd6640 1 -- 172.21.15.88:0/4198915884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff44c153770 msgr2=0x7ff44c159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.766+0000 7ff450cd6640 1 --2- 172.21.15.88:0/4198915884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff44c153770 0x7ff44c159ed0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7ff44c0cc7b0 tx=0x7ff43c01c4e0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.766+0000 7ff450cd6640 1 -- 172.21.15.88:0/4198915884 shutdown_connections 2023-10-16T20:58:50.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.766+0000 7ff450cd6640 1 --2- 172.21.15.88:0/4198915884 >> v2:172.21.15.191:3300/0 conn(0x7ff44c0c9bb0 0x7ff44c0c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.766+0000 7ff450cd6640 1 --2- 172.21.15.88:0/4198915884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff44c153770 0x7ff44c159ed0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.766+0000 7ff450cd6640 1 -- 172.21.15.88:0/4198915884 >> 172.21.15.88:0/4198915884 conn(0x7ff44c0715c0 msgr2=0x7ff44c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:50.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.766+0000 7ff450cd6640 1 -- 172.21.15.88:0/4198915884 shutdown_connections 2023-10-16T20:58:50.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.766+0000 7ff450cd6640 1 -- 172.21.15.88:0/4198915884 wait complete. 2023-10-16T20:58:50.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff450cd6640 1 Processor -- start 2023-10-16T20:58:50.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff450cd6640 1 -- start start 2023-10-16T20:58:50.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff450cd6640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff44c0c41f0 0x7ff44c1499b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff450cd6640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff44c0c9bb0 0x7ff44c0c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff450cd6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff44c153770 0x7ff44c15b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff450cd6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c15ca40 con 0x7ff44c153770 2023-10-16T20:58:50.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff450cd6640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c15d540 con 0x7ff44c0c41f0 2023-10-16T20:58:50.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff450cd6640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c1f2190 con 0x7ff44c0c9bb0 2023-10-16T20:58:50.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff44ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff44c153770 0x7ff44c15b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff44ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff44c153770 0x7ff44c15b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:34068/0 (socket says 172.21.15.88:34068) 2023-10-16T20:58:50.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff44ad9d640 1 -- 172.21.15.88:0/285286329 learned_addr learned my addr 172.21.15.88:0/285286329 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:58:50.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff44a59c640 1 --2- 172.21.15.88:0/285286329 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff44c0c41f0 0x7ff44c1499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff449d9b640 1 --2- 172.21.15.88:0/285286329 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff44c0c9bb0 0x7ff44c0c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff44ad9d640 1 -- 172.21.15.88:0/285286329 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff44c0c9bb0 msgr2=0x7ff44c0c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff44ad9d640 1 --2- 172.21.15.88:0/285286329 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff44c0c9bb0 0x7ff44c0c78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff44ad9d640 1 -- 172.21.15.88:0/285286329 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff44c0c41f0 msgr2=0x7ff44c1499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff44ad9d640 1 --2- 172.21.15.88:0/285286329 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff44c0c41f0 0x7ff44c1499b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff44ad9d640 1 -- 172.21.15.88:0/285286329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43c009480 con 0x7ff44c153770 2023-10-16T20:58:50.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff44ad9d640 1 --2- 172.21.15.88:0/285286329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff44c153770 0x7ff44c15b850 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7ff44400c770 tx=0x7ff44400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:50.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff43affd640 1 -- 172.21.15.88:0/285286329 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff444003b80 con 0x7ff44c153770 2023-10-16T20:58:50.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff43affd640 1 -- 172.21.15.88:0/285286329 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff444003ce0 con 0x7ff44c153770 2023-10-16T20:58:50.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff43affd640 1 -- 172.21.15.88:0/285286329 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44400aa30 con 0x7ff44c153770 2023-10-16T20:58:50.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff450cd6640 1 -- 172.21.15.88:0/285286329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44c1f5b70 con 0x7ff44c153770 2023-10-16T20:58:50.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff450cd6640 1 -- 172.21.15.88:0/285286329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff44c1f2620 con 0x7ff44c153770 2023-10-16T20:58:50.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.770+0000 7ff450cd6640 1 -- 172.21.15.88:0/285286329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff44c1f2340 con 0x7ff44c153770 2023-10-16T20:58:50.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.774+0000 7ff43affd640 1 -- 172.21.15.88:0/285286329 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff44400ab90 con 0x7ff44c153770 2023-10-16T20:58:50.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.774+0000 7ff43affd640 1 --2- 172.21.15.88:0/285286329 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff42c0664d0 0x7ff42c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:58:50.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.774+0000 7ff43affd640 1 -- 172.21.15.88:0/285286329 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7ff444084180 con 0x7ff44c153770 2023-10-16T20:58:50.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.774+0000 7ff44a59c640 1 --2- 172.21.15.88:0/285286329 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff42c0664d0 0x7ff42c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:58:50.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.774+0000 7ff44a59c640 1 --2- 172.21.15.88:0/285286329 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff42c0664d0 0x7ff42c068990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0af000 tx=0x7ff43c0a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:58:50.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.778+0000 7ff43affd640 1 -- 172.21.15.88:0/285286329 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff44c1f2340 con 0x7ff44c153770 2023-10-16T20:58:50.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.926+0000 7ff450cd6640 1 -- 172.21.15.88:0/285286329 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff44c0cb8b0 con 0x7ff42c0664d0 2023-10-16T20:58:50.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.934+0000 7ff43affd640 1 -- 172.21.15.88:0/285286329 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+372829 (secure 0 0 0) 0x7ff43c0ab570 con 0x7ff42c0664d0 2023-10-16T20:58:50.940 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:58:50.942 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-16T20:58:50.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.938+0000 7ff450cd6640 1 -- 172.21.15.88:0/285286329 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff42c0664d0 msgr2=0x7ff42c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.938+0000 7ff450cd6640 1 --2- 172.21.15.88:0/285286329 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff42c0664d0 0x7ff42c068990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0af000 tx=0x7ff43c0a7000 comp rx=0 tx=0).stop 2023-10-16T20:58:50.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.938+0000 7ff450cd6640 1 -- 172.21.15.88:0/285286329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff44c153770 msgr2=0x7ff44c15b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:58:50.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.938+0000 7ff450cd6640 1 --2- 172.21.15.88:0/285286329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff44c153770 0x7ff44c15b850 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7ff44400c770 tx=0x7ff44400cc30 comp rx=0 tx=0).stop 2023-10-16T20:58:50.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.938+0000 7ff450cd6640 1 -- 172.21.15.88:0/285286329 shutdown_connections 2023-10-16T20:58:50.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.938+0000 7ff450cd6640 1 --2- 172.21.15.88:0/285286329 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff42c0664d0 0x7ff42c068990 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.938+0000 7ff450cd6640 1 --2- 172.21.15.88:0/285286329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff44c153770 0x7ff44c15b850 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.938+0000 7ff450cd6640 1 --2- 172.21.15.88:0/285286329 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff44c0c9bb0 0x7ff44c0c78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.938+0000 7ff450cd6640 1 --2- 172.21.15.88:0/285286329 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff44c0c41f0 0x7ff44c1499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:58:50.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.938+0000 7ff450cd6640 1 -- 172.21.15.88:0/285286329 >> 172.21.15.88:0/285286329 conn(0x7ff44c0715c0 msgr2=0x7ff44c0719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:58:50.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.938+0000 7ff450cd6640 1 -- 172.21.15.88:0/285286329 shutdown_connections 2023-10-16T20:58:50.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:58:50.938+0000 7ff450cd6640 1 -- 172.21.15.88:0/285286329 wait complete. 2023-10-16T20:58:50.961 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":725,"stamp":"2023-10-16T20:58:49.932640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1103877889,"num_objects":10494,"num_object_clones":0,"num_object_copies":41576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10494,"num_whiteouts":0,"num_read":31781,"num_read_kb":882609,"num_write":27514,"num_write_kb":1485766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":115,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34043,"ondisk_log_size":34043,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3214720,"kb_used_data":2278144,"kb_used_omap":12,"kb_used_meta":936499,"kb_avail":751760000,"statfs":{"total":773094113280,"available":769802240000,"internally_reserved":0,"allocated":2332819456,"data_stored":2329695924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":958975545},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":401512,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003460"},"pg_stats":[{"pgid":"2.1f","version":"67'326","reported_seq":415,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:46.510371+0000","last_change":"2023-10-16T20:36:54.763744+0000","last_active":"2023-10-16T20:58:46.510371+0000","last_peered":"2023-10-16T20:58:46.510371+0000","last_clean":"2023-10-16T20:58:46.510371+0000","last_became_active":"2023-10-16T20:36:54.763457+0000","last_became_peered":"2023-10-16T20:36:54.763457+0000","last_unstale":"2023-10-16T20:58:46.510371+0000","last_undegraded":"2023-10-16T20:58:46.510371+0000","last_fullsized":"2023-10-16T20:58:46.510371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9870219,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":42,"num_read_kb":9548,"num_write":324,"num_write_kb":9831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"67'107","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.869873+0000","last_change":"2023-10-16T20:47:31.588044+0000","last_active":"2023-10-16T20:58:37.869873+0000","last_peered":"2023-10-16T20:58:37.869873+0000","last_clean":"2023-10-16T20:58:37.869873+0000","last_became_active":"2023-10-16T20:36:55.302519+0000","last_became_peered":"2023-10-16T20:36:55.302519+0000","last_unstale":"2023-10-16T20:58:37.869873+0000","last_undegraded":"2023-10-16T20:58:37.869873+0000","last_fullsized":"2023-10-16T20:58:37.869873+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:49.037032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007784165,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'672","reported_seq":3557,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.056852+0000","last_change":"2023-10-16T20:47:31.584576+0000","last_active":"2023-10-16T20:58:23.056852+0000","last_peered":"2023-10-16T20:58:23.056852+0000","last_clean":"2023-10-16T20:58:23.056852+0000","last_became_active":"2023-10-16T20:36:59.466306+0000","last_became_peered":"2023-10-16T20:36:59.466306+0000","last_unstale":"2023-10-16T20:58:23.056852+0000","last_undegraded":"2023-10-16T20:58:23.056852+0000","last_fullsized":"2023-10-16T20:58:23.056852+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:57:56.295661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073106899999999999,"stat_sum":{"num_bytes":36865036,"num_objects":299,"num_object_clones":0,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":299,"num_whiteouts":0,"num_read":779,"num_read_kb":25449,"num_write":671,"num_write_kb":56816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"67'207","reported_seq":277,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:00.972618+0000","last_change":"2023-10-16T20:36:54.759491+0000","last_active":"2023-10-16T20:58:00.972618+0000","last_peered":"2023-10-16T20:58:00.972618+0000","last_clean":"2023-10-16T20:58:00.972618+0000","last_became_active":"2023-10-16T20:36:54.759177+0000","last_became_peered":"2023-10-16T20:36:54.759177+0000","last_unstale":"2023-10-16T20:58:00.972618+0000","last_undegraded":"2023-10-16T20:58:00.972618+0000","last_fullsized":"2023-10-16T20:58:00.972618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:43:36.547584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":12288,"num_write":199,"num_write_kb":8312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"67'107","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889932+0000","last_change":"2023-10-16T20:47:31.488681+0000","last_active":"2023-10-16T20:58:37.889932+0000","last_peered":"2023-10-16T20:58:37.889932+0000","last_clean":"2023-10-16T20:58:37.889932+0000","last_became_active":"2023-10-16T20:36:55.291458+0000","last_became_peered":"2023-10-16T20:36:55.291458+0000","last_unstale":"2023-10-16T20:58:37.889932+0000","last_undegraded":"2023-10-16T20:58:37.889932+0000","last_fullsized":"2023-10-16T20:58:37.889932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087429459999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'682","reported_seq":3571,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.040513+0000","last_change":"2023-10-16T20:47:31.484150+0000","last_active":"2023-10-16T20:58:23.040513+0000","last_peered":"2023-10-16T20:58:23.040513+0000","last_clean":"2023-10-16T20:58:23.040513+0000","last_became_active":"2023-10-16T20:36:59.302968+0000","last_became_peered":"2023-10-16T20:36:59.302968+0000","last_unstale":"2023-10-16T20:58:23.040513+0000","last_undegraded":"2023-10-16T20:58:23.040513+0000","last_fullsized":"2023-10-16T20:58:23.040513+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:58:43.745409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027985639999999999,"stat_sum":{"num_bytes":24765378,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":783,"num_read_kb":25781,"num_write":678,"num_write_kb":40921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"67'340","reported_seq":437,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:33.688387+0000","last_change":"2023-10-16T20:36:54.762600+0000","last_active":"2023-10-16T20:58:33.688387+0000","last_peered":"2023-10-16T20:58:33.688387+0000","last_clean":"2023-10-16T20:58:33.688387+0000","last_became_active":"2023-10-16T20:36:54.762278+0000","last_became_peered":"2023-10-16T20:36:54.762278+0000","last_unstale":"2023-10-16T20:58:33.688387+0000","last_undegraded":"2023-10-16T20:58:33.688387+0000","last_fullsized":"2023-10-16T20:58:33.688387+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:46:03.836415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":85,"num_read_kb":16492,"num_write":374,"num_write_kb":12523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'96","reported_seq":2197,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.131579+0000","last_change":"2023-10-16T20:47:31.496372+0000","last_active":"2023-10-16T20:58:38.131579+0000","last_peered":"2023-10-16T20:58:38.131579+0000","last_clean":"2023-10-16T20:58:38.131579+0000","last_became_active":"2023-10-16T20:37:37.402987+0000","last_became_peered":"2023-10-16T20:37:37.402987+0000","last_unstale":"2023-10-16T20:58:38.131579+0000","last_undegraded":"2023-10-16T20:58:38.131579+0000","last_fullsized":"2023-10-16T20:58:38.131579+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:12:06.964995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082573900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"67'673","reported_seq":4224,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.678192+0000","last_change":"2023-10-16T20:47:31.485701+0000","last_active":"2023-10-16T20:58:22.678192+0000","last_peered":"2023-10-16T20:58:22.678192+0000","last_clean":"2023-10-16T20:58:22.678192+0000","last_became_active":"2023-10-16T20:36:59.303805+0000","last_became_peered":"2023-10-16T20:36:59.303805+0000","last_unstale":"2023-10-16T20:58:22.678192+0000","last_undegraded":"2023-10-16T20:58:22.678192+0000","last_fullsized":"2023-10-16T20:58:22.678192+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:28:11.659597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041657500000000002,"stat_sum":{"num_bytes":49691093,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":1446,"num_read_kb":31388,"num_write":673,"num_write_kb":52938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"67'172","reported_seq":252,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.691336+0000","last_change":"2023-10-16T20:36:54.749849+0000","last_active":"2023-10-16T20:58:28.691336+0000","last_peered":"2023-10-16T20:58:28.691336+0000","last_clean":"2023-10-16T20:58:28.691336+0000","last_became_active":"2023-10-16T20:36:54.749316+0000","last_became_peered":"2023-10-16T20:36:54.749316+0000","last_unstale":"2023-10-16T20:58:28.691336+0000","last_undegraded":"2023-10-16T20:58:28.691336+0000","last_fullsized":"2023-10-16T20:58:28.691336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:49:29.077126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":170,"num_write_kb":4198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"67'104","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.872186+0000","last_change":"2023-10-16T20:47:31.592091+0000","last_active":"2023-10-16T20:58:37.872186+0000","last_peered":"2023-10-16T20:58:37.872186+0000","last_clean":"2023-10-16T20:58:37.872186+0000","last_became_active":"2023-10-16T20:36:55.302387+0000","last_became_peered":"2023-10-16T20:36:55.302387+0000","last_unstale":"2023-10-16T20:58:37.872186+0000","last_undegraded":"2023-10-16T20:58:37.872186+0000","last_fullsized":"2023-10-16T20:58:37.872186+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:07:35.871205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011585343,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'641","reported_seq":3839,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.036707+0000","last_change":"2023-10-16T20:47:31.486382+0000","last_active":"2023-10-16T20:58:23.036707+0000","last_peered":"2023-10-16T20:58:23.036707+0000","last_clean":"2023-10-16T20:58:23.036707+0000","last_became_active":"2023-10-16T20:36:59.305228+0000","last_became_peered":"2023-10-16T20:36:59.305228+0000","last_unstale":"2023-10-16T20:58:23.036707+0000","last_undegraded":"2023-10-16T20:58:23.036707+0000","last_fullsized":"2023-10-16T20:58:23.036707+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":641,"log_dups_size":0,"ondisk_log_size":641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:32:37.261529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047877249999999996,"stat_sum":{"num_bytes":34900728,"num_objects":310,"num_object_clones":0,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":1093,"num_read_kb":21879,"num_write":638,"num_write_kb":46703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"67'104","reported_seq":2214,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.863809+0000","last_change":"2023-10-16T20:47:31.592641+0000","last_active":"2023-10-16T20:58:37.863809+0000","last_peered":"2023-10-16T20:58:37.863809+0000","last_clean":"2023-10-16T20:58:37.863809+0000","last_became_active":"2023-10-16T20:36:55.306427+0000","last_became_peered":"2023-10-16T20:36:55.306427+0000","last_unstale":"2023-10-16T20:58:37.863809+0000","last_undegraded":"2023-10-16T20:58:37.863809+0000","last_fullsized":"2023-10-16T20:58:37.863809+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:17:46.390047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013854674000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"67'187","reported_seq":258,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.360075+0000","last_change":"2023-10-16T20:36:54.768375+0000","last_active":"2023-10-16T20:58:05.360075+0000","last_peered":"2023-10-16T20:58:05.360075+0000","last_clean":"2023-10-16T20:58:05.360075+0000","last_became_active":"2023-10-16T20:36:54.767996+0000","last_became_peered":"2023-10-16T20:36:54.767996+0000","last_unstale":"2023-10-16T20:58:05.360075+0000","last_undegraded":"2023-10-16T20:58:05.360075+0000","last_fullsized":"2023-10-16T20:58:05.360075+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:58:55.303942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":8202,"num_write":173,"num_write_kb":8293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"67'648","reported_seq":4473,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.721394+0000","last_change":"2023-10-16T20:47:31.487925+0000","last_active":"2023-10-16T20:58:22.721394+0000","last_peered":"2023-10-16T20:58:22.721394+0000","last_clean":"2023-10-16T20:58:22.721394+0000","last_became_active":"2023-10-16T20:36:59.306640+0000","last_became_peered":"2023-10-16T20:36:59.306640+0000","last_unstale":"2023-10-16T20:58:22.721394+0000","last_undegraded":"2023-10-16T20:58:22.721394+0000","last_fullsized":"2023-10-16T20:58:22.721394+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:27:48.041001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054537470000000001,"stat_sum":{"num_bytes":41205181,"num_objects":318,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":1719,"num_read_kb":37802,"num_write":648,"num_write_kb":48776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"67'101","reported_seq":2210,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.891454+0000","last_change":"2023-10-16T20:47:31.587617+0000","last_active":"2023-10-16T20:58:37.891454+0000","last_peered":"2023-10-16T20:58:37.891454+0000","last_clean":"2023-10-16T20:58:37.891454+0000","last_became_active":"2023-10-16T20:36:55.275085+0000","last_became_peered":"2023-10-16T20:36:55.275085+0000","last_unstale":"2023-10-16T20:58:37.891454+0000","last_undegraded":"2023-10-16T20:58:37.891454+0000","last_fullsized":"2023-10-16T20:58:37.891454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:45:15.882497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087599630000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'182","reported_seq":252,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:42.658621+0000","last_change":"2023-10-16T20:36:54.756174+0000","last_active":"2023-10-16T20:58:42.658621+0000","last_peered":"2023-10-16T20:58:42.658621+0000","last_clean":"2023-10-16T20:58:42.658621+0000","last_became_active":"2023-10-16T20:36:54.755872+0000","last_became_peered":"2023-10-16T20:36:54.755872+0000","last_unstale":"2023-10-16T20:58:42.658621+0000","last_undegraded":"2023-10-16T20:58:42.658621+0000","last_fullsized":"2023-10-16T20:58:42.658621+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":10,"num_write":281,"num_write_kb":4257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"67'699","reported_seq":3454,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.718374+0000","last_change":"2023-10-16T20:47:31.486593+0000","last_active":"2023-10-16T20:58:22.718374+0000","last_peered":"2023-10-16T20:58:22.718374+0000","last_clean":"2023-10-16T20:58:22.718374+0000","last_became_active":"2023-10-16T20:36:59.302919+0000","last_became_peered":"2023-10-16T20:36:59.302919+0000","last_unstale":"2023-10-16T20:58:22.718374+0000","last_undegraded":"2023-10-16T20:58:22.718374+0000","last_fullsized":"2023-10-16T20:58:22.718374+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:48.779648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046546790000000001,"stat_sum":{"num_bytes":20793675,"num_objects":334,"num_object_clones":0,"num_object_copies":1336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":334,"num_whiteouts":0,"num_read":649,"num_read_kb":16394,"num_write":696,"num_write_kb":45235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"67'106","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.886436+0000","last_change":"2023-10-16T20:47:31.586816+0000","last_active":"2023-10-16T20:58:37.886436+0000","last_peered":"2023-10-16T20:58:37.886436+0000","last_clean":"2023-10-16T20:58:37.886436+0000","last_became_active":"2023-10-16T20:36:55.270455+0000","last_became_peered":"2023-10-16T20:36:55.270455+0000","last_unstale":"2023-10-16T20:58:37.886436+0000","last_undegraded":"2023-10-16T20:58:37.886436+0000","last_fullsized":"2023-10-16T20:58:37.886436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:44:37.316278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.008577889,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"66'49","reported_seq":128,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.657919+0000","last_change":"2023-10-16T20:36:54.744128+0000","last_active":"2023-10-16T20:58:37.657919+0000","last_peered":"2023-10-16T20:58:37.657919+0000","last_clean":"2023-10-16T20:58:37.657919+0000","last_became_active":"2023-10-16T20:36:54.743805+0000","last_became_peered":"2023-10-16T20:36:54.743805+0000","last_unstale":"2023-10-16T20:58:37.657919+0000","last_undegraded":"2023-10-16T20:58:37.657919+0000","last_fullsized":"2023-10-16T20:58:37.657919+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":55,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'651","reported_seq":3693,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.438877+0000","last_change":"2023-10-16T20:47:31.485640+0000","last_active":"2023-10-16T20:58:22.438877+0000","last_peered":"2023-10-16T20:58:22.438877+0000","last_clean":"2023-10-16T20:58:22.438877+0000","last_became_active":"2023-10-16T20:36:59.465172+0000","last_became_peered":"2023-10-16T20:36:59.465172+0000","last_unstale":"2023-10-16T20:58:22.438877+0000","last_undegraded":"2023-10-16T20:58:22.438877+0000","last_fullsized":"2023-10-16T20:58:22.438877+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040107390000000001,"stat_sum":{"num_bytes":25409516,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":935,"num_read_kb":27643,"num_write":650,"num_write_kb":33262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"67'103","reported_seq":2213,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.849592+0000","last_change":"2023-10-16T20:47:31.487014+0000","last_active":"2023-10-16T20:58:37.849592+0000","last_peered":"2023-10-16T20:58:37.849592+0000","last_clean":"2023-10-16T20:58:37.849592+0000","last_became_active":"2023-10-16T20:36:55.278568+0000","last_became_peered":"2023-10-16T20:36:55.278568+0000","last_unstale":"2023-10-16T20:58:37.849592+0000","last_undegraded":"2023-10-16T20:58:37.849592+0000","last_fullsized":"2023-10-16T20:58:37.849592+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:01:00.125646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050399599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"66'287","reported_seq":358,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:01.079180+0000","last_change":"2023-10-16T20:36:54.758778+0000","last_active":"2023-10-16T20:58:01.079180+0000","last_peered":"2023-10-16T20:58:01.079180+0000","last_clean":"2023-10-16T20:58:01.079180+0000","last_became_active":"2023-10-16T20:36:54.758496+0000","last_became_peered":"2023-10-16T20:36:54.758496+0000","last_unstale":"2023-10-16T20:58:01.079180+0000","last_undegraded":"2023-10-16T20:58:01.079180+0000","last_fullsized":"2023-10-16T20:58:01.079180+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:36:00.515106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":12294,"num_write":291,"num_write_kb":4310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"67'586","reported_seq":3861,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.527756+0000","last_change":"2023-10-16T20:47:31.488200+0000","last_active":"2023-10-16T20:58:22.527756+0000","last_peered":"2023-10-16T20:58:22.527756+0000","last_clean":"2023-10-16T20:58:22.527756+0000","last_became_active":"2023-10-16T20:36:59.309068+0000","last_became_peered":"2023-10-16T20:36:59.309068+0000","last_unstale":"2023-10-16T20:58:22.527756+0000","last_undegraded":"2023-10-16T20:58:22.527756+0000","last_fullsized":"2023-10-16T20:58:22.527756+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:22:11.244977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055482439999999999,"stat_sum":{"num_bytes":36900597,"num_objects":303,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1169,"num_read_kb":30992,"num_write":585,"num_write_kb":40444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'101","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:39.656982+0000","last_change":"2023-10-16T20:47:31.492407+0000","last_active":"2023-10-16T20:58:39.656982+0000","last_peered":"2023-10-16T20:58:39.656982+0000","last_clean":"2023-10-16T20:58:39.656982+0000","last_became_active":"2023-10-16T20:36:55.272840+0000","last_became_peered":"2023-10-16T20:36:55.272840+0000","last_unstale":"2023-10-16T20:58:39.656982+0000","last_undegraded":"2023-10-16T20:58:39.656982+0000","last_fullsized":"2023-10-16T20:58:39.656982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011897911000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"30'8","reported_seq":88,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.710153+0000","last_change":"2023-10-16T20:36:54.760681+0000","last_active":"2023-10-16T20:58:28.710153+0000","last_peered":"2023-10-16T20:58:28.710153+0000","last_clean":"2023-10-16T20:58:28.710153+0000","last_became_active":"2023-10-16T20:36:54.760397+0000","last_became_peered":"2023-10-16T20:36:54.760397+0000","last_unstale":"2023-10-16T20:58:28.710153+0000","last_undegraded":"2023-10-16T20:58:28.710153+0000","last_fullsized":"2023-10-16T20:58:28.710153+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:47:16.510382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"67'710","reported_seq":3798,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.025282+0000","last_change":"2023-10-16T20:47:31.485980+0000","last_active":"2023-10-16T20:58:23.025282+0000","last_peered":"2023-10-16T20:58:23.025282+0000","last_clean":"2023-10-16T20:58:23.025282+0000","last_became_active":"2023-10-16T20:36:59.465152+0000","last_became_peered":"2023-10-16T20:36:59.465152+0000","last_unstale":"2023-10-16T20:58:23.025282+0000","last_undegraded":"2023-10-16T20:58:23.025282+0000","last_fullsized":"2023-10-16T20:58:23.025282+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:09:48.800573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044307399999999999,"stat_sum":{"num_bytes":25809469,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":981,"num_read_kb":15497,"num_write":706,"num_write_kb":50141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'106","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.882745+0000","last_change":"2023-10-16T20:47:31.587090+0000","last_active":"2023-10-16T20:58:37.882745+0000","last_peered":"2023-10-16T20:58:37.882745+0000","last_clean":"2023-10-16T20:58:37.882745+0000","last_became_active":"2023-10-16T20:36:55.285566+0000","last_became_peered":"2023-10-16T20:36:55.285566+0000","last_unstale":"2023-10-16T20:58:37.882745+0000","last_undegraded":"2023-10-16T20:58:37.882745+0000","last_fullsized":"2023-10-16T20:58:37.882745+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:18.835080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085500479999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'101","reported_seq":178,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.707570+0000","last_change":"2023-10-16T20:36:54.758062+0000","last_active":"2023-10-16T20:58:38.707570+0000","last_peered":"2023-10-16T20:58:38.707570+0000","last_clean":"2023-10-16T20:58:38.707570+0000","last_became_active":"2023-10-16T20:36:54.757796+0000","last_became_peered":"2023-10-16T20:36:54.757796+0000","last_unstale":"2023-10-16T20:58:38.707570+0000","last_undegraded":"2023-10-16T20:58:38.707570+0000","last_fullsized":"2023-10-16T20:58:38.707570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":254,"num_write_kb":1037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"67'686","reported_seq":3331,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.536581+0000","last_change":"2023-10-16T20:47:31.582989+0000","last_active":"2023-10-16T20:58:22.536581+0000","last_peered":"2023-10-16T20:58:22.536581+0000","last_clean":"2023-10-16T20:58:22.536581+0000","last_became_active":"2023-10-16T20:36:59.465221+0000","last_became_peered":"2023-10-16T20:36:59.465221+0000","last_unstale":"2023-10-16T20:58:22.536581+0000","last_undegraded":"2023-10-16T20:58:22.536581+0000","last_fullsized":"2023-10-16T20:58:22.536581+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:29:08.900666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059334330000000001,"stat_sum":{"num_bytes":20326999,"num_objects":343,"num_object_clones":0,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":538,"num_read_kb":13756,"num_write":685,"num_write_kb":32496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"67'101","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.818951+0000","last_change":"2023-10-16T20:47:31.489294+0000","last_active":"2023-10-16T20:58:37.818951+0000","last_peered":"2023-10-16T20:58:37.818951+0000","last_clean":"2023-10-16T20:58:37.818951+0000","last_became_active":"2023-10-16T20:36:55.274432+0000","last_became_peered":"2023-10-16T20:36:55.274432+0000","last_unstale":"2023-10-16T20:58:37.818951+0000","last_undegraded":"2023-10-16T20:58:37.818951+0000","last_fullsized":"2023-10-16T20:58:37.818951+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057461259999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"66'34","reported_seq":109,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:01.080176+0000","last_change":"2023-10-16T20:36:54.762356+0000","last_active":"2023-10-16T20:58:01.080176+0000","last_peered":"2023-10-16T20:58:01.080176+0000","last_clean":"2023-10-16T20:58:01.080176+0000","last_became_active":"2023-10-16T20:36:54.762167+0000","last_became_peered":"2023-10-16T20:36:54.762167+0000","last_unstale":"2023-10-16T20:58:01.080176+0000","last_undegraded":"2023-10-16T20:58:01.080176+0000","last_fullsized":"2023-10-16T20:58:01.080176+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:45.164625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":34,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"67'725","reported_seq":3716,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.045470+0000","last_change":"2023-10-16T20:47:31.489099+0000","last_active":"2023-10-16T20:58:23.045470+0000","last_peered":"2023-10-16T20:58:23.045470+0000","last_clean":"2023-10-16T20:58:23.045470+0000","last_became_active":"2023-10-16T20:36:59.466876+0000","last_became_peered":"2023-10-16T20:36:59.466876+0000","last_unstale":"2023-10-16T20:58:23.045470+0000","last_undegraded":"2023-10-16T20:58:23.045470+0000","last_fullsized":"2023-10-16T20:58:23.045470+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":725,"log_dups_size":0,"ondisk_log_size":725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009058969,"stat_sum":{"num_bytes":44901233,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":884,"num_read_kb":20657,"num_write":720,"num_write_kb":60576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"67'98","reported_seq":2208,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.895976+0000","last_change":"2023-10-16T20:47:31.594422+0000","last_active":"2023-10-16T20:58:37.895976+0000","last_peered":"2023-10-16T20:58:37.895976+0000","last_clean":"2023-10-16T20:58:37.895976+0000","last_became_active":"2023-10-16T20:36:55.293228+0000","last_became_peered":"2023-10-16T20:36:55.293228+0000","last_unstale":"2023-10-16T20:58:37.895976+0000","last_undegraded":"2023-10-16T20:58:37.895976+0000","last_fullsized":"2023-10-16T20:58:37.895976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:33:58.039426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013567460999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"66'210","reported_seq":279,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:00.665133+0000","last_change":"2023-10-16T20:36:54.757588+0000","last_active":"2023-10-16T20:58:00.665133+0000","last_peered":"2023-10-16T20:58:00.665133+0000","last_clean":"2023-10-16T20:58:00.665133+0000","last_became_active":"2023-10-16T20:36:54.757308+0000","last_became_peered":"2023-10-16T20:36:54.757308+0000","last_unstale":"2023-10-16T20:58:00.665133+0000","last_undegraded":"2023-10-16T20:58:00.665133+0000","last_fullsized":"2023-10-16T20:58:00.665133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":3049,"num_write":207,"num_write_kb":4195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"67'671","reported_seq":3426,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.059793+0000","last_change":"2023-10-16T20:47:31.485489+0000","last_active":"2023-10-16T20:58:23.059793+0000","last_peered":"2023-10-16T20:58:23.059793+0000","last_clean":"2023-10-16T20:58:23.059793+0000","last_became_active":"2023-10-16T20:36:59.303224+0000","last_became_peered":"2023-10-16T20:36:59.303224+0000","last_unstale":"2023-10-16T20:58:23.059793+0000","last_undegraded":"2023-10-16T20:58:23.059793+0000","last_fullsized":"2023-10-16T20:58:23.059793+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038422399999999998,"stat_sum":{"num_bytes":21817454,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":647,"num_read_kb":9306,"num_write":673,"num_write_kb":37999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"67'120","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889222+0000","last_change":"2023-10-16T20:47:31.487515+0000","last_active":"2023-10-16T20:58:37.889222+0000","last_peered":"2023-10-16T20:58:37.889222+0000","last_clean":"2023-10-16T20:58:37.889222+0000","last_became_active":"2023-10-16T20:36:55.277872+0000","last_became_peered":"2023-10-16T20:36:55.277872+0000","last_unstale":"2023-10-16T20:58:37.889222+0000","last_undegraded":"2023-10-16T20:58:37.889222+0000","last_fullsized":"2023-10-16T20:58:37.889222+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:47:46.751931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052323379999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"66'532","reported_seq":616,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.670124+0000","last_change":"2023-10-16T20:36:54.740644+0000","last_active":"2023-10-16T20:58:05.670124+0000","last_peered":"2023-10-16T20:58:05.670124+0000","last_clean":"2023-10-16T20:58:05.670124+0000","last_became_active":"2023-10-16T20:36:54.740340+0000","last_became_peered":"2023-10-16T20:36:54.740340+0000","last_unstale":"2023-10-16T20:58:05.670124+0000","last_undegraded":"2023-10-16T20:58:05.670124+0000","last_fullsized":"2023-10-16T20:58:05.670124+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:22.221727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21285,"num_write":524,"num_write_kb":16645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'623","reported_seq":3878,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.054189+0000","last_change":"2023-10-16T20:47:31.489650+0000","last_active":"2023-10-16T20:58:23.054189+0000","last_peered":"2023-10-16T20:58:23.054189+0000","last_clean":"2023-10-16T20:58:23.054189+0000","last_became_active":"2023-10-16T20:36:59.294548+0000","last_became_peered":"2023-10-16T20:36:59.294548+0000","last_unstale":"2023-10-16T20:58:23.054189+0000","last_undegraded":"2023-10-16T20:58:23.054189+0000","last_fullsized":"2023-10-16T20:58:23.054189+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062280529999999999,"stat_sum":{"num_bytes":29651112,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":1147,"num_read_kb":23932,"num_write":623,"num_write_kb":40690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"67'89","reported_seq":2200,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.807171+0000","last_change":"2023-10-16T20:47:31.484434+0000","last_active":"2023-10-16T20:58:37.807171+0000","last_peered":"2023-10-16T20:58:37.807171+0000","last_clean":"2023-10-16T20:58:37.807171+0000","last_became_active":"2023-10-16T20:36:55.269749+0000","last_became_peered":"2023-10-16T20:36:55.269749+0000","last_unstale":"2023-10-16T20:58:37.807171+0000","last_undegraded":"2023-10-16T20:58:37.807171+0000","last_fullsized":"2023-10-16T20:58:37.807171+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003009004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'188","reported_seq":376,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.714114+0000","last_change":"2023-10-16T20:36:54.757903+0000","last_active":"2023-10-16T20:58:37.714114+0000","last_peered":"2023-10-16T20:58:37.714114+0000","last_clean":"2023-10-16T20:58:37.714114+0000","last_became_active":"2023-10-16T20:36:54.757549+0000","last_became_peered":"2023-10-16T20:36:54.757549+0000","last_unstale":"2023-10-16T20:58:37.714114+0000","last_undegraded":"2023-10-16T20:58:37.714114+0000","last_fullsized":"2023-10-16T20:58:37.714114+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:02:42.090848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1409828,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":124,"num_read_kb":239,"num_write":184,"num_write_kb":1568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'647","reported_seq":3935,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:42.658770+0000","last_change":"2023-10-16T20:47:31.487517+0000","last_active":"2023-10-16T20:58:42.658770+0000","last_peered":"2023-10-16T20:58:42.658770+0000","last_clean":"2023-10-16T20:58:42.658770+0000","last_became_active":"2023-10-16T20:36:59.301693+0000","last_became_peered":"2023-10-16T20:36:59.301693+0000","last_unstale":"2023-10-16T20:58:42.658770+0000","last_undegraded":"2023-10-16T20:58:42.658770+0000","last_fullsized":"2023-10-16T20:58:42.658770+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056692870000000003,"stat_sum":{"num_bytes":29916858,"num_objects":298,"num_object_clones":0,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":298,"num_whiteouts":0,"num_read":1180,"num_read_kb":25235,"num_write":643,"num_write_kb":45892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"67'104","reported_seq":2215,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889004+0000","last_change":"2023-10-16T20:47:31.487504+0000","last_active":"2023-10-16T20:58:37.889004+0000","last_peered":"2023-10-16T20:58:37.889004+0000","last_clean":"2023-10-16T20:58:37.889004+0000","last_became_active":"2023-10-16T20:36:55.276649+0000","last_became_peered":"2023-10-16T20:36:55.276649+0000","last_unstale":"2023-10-16T20:58:37.889004+0000","last_undegraded":"2023-10-16T20:58:37.889004+0000","last_fullsized":"2023-10-16T20:58:37.889004+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076292809999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'128","reported_seq":201,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:01.079234+0000","last_change":"2023-10-16T20:36:54.753592+0000","last_active":"2023-10-16T20:58:01.079234+0000","last_peered":"2023-10-16T20:58:01.079234+0000","last_clean":"2023-10-16T20:58:01.079234+0000","last_became_active":"2023-10-16T20:36:54.753237+0000","last_became_peered":"2023-10-16T20:36:54.753237+0000","last_unstale":"2023-10-16T20:58:01.079234+0000","last_undegraded":"2023-10-16T20:58:01.079234+0000","last_fullsized":"2023-10-16T20:58:01.079234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:53.005979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":20486,"num_write":119,"num_write_kb":8247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'673","reported_seq":3484,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:39.656963+0000","last_change":"2023-10-16T20:47:31.491504+0000","last_active":"2023-10-16T20:58:39.656963+0000","last_peered":"2023-10-16T20:58:39.656963+0000","last_clean":"2023-10-16T20:58:39.656963+0000","last_became_active":"2023-10-16T20:36:59.466708+0000","last_became_peered":"2023-10-16T20:36:59.466708+0000","last_unstale":"2023-10-16T20:58:39.656963+0000","last_undegraded":"2023-10-16T20:58:39.656963+0000","last_fullsized":"2023-10-16T20:58:39.656963+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011259905000000001,"stat_sum":{"num_bytes":19802424,"num_objects":318,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":703,"num_read_kb":14564,"num_write":670,"num_write_kb":31986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"67'99","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.897919+0000","last_change":"2023-10-16T20:47:31.585286+0000","last_active":"2023-10-16T20:58:37.897919+0000","last_peered":"2023-10-16T20:58:37.897919+0000","last_clean":"2023-10-16T20:58:37.897919+0000","last_became_active":"2023-10-16T20:36:55.294610+0000","last_became_peered":"2023-10-16T20:36:55.294610+0000","last_unstale":"2023-10-16T20:58:37.897919+0000","last_undegraded":"2023-10-16T20:58:37.897919+0000","last_fullsized":"2023-10-16T20:58:37.897919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:34:05.113883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075091560000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"66'7","reported_seq":83,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:00.736127+0000","last_change":"2023-10-16T20:36:54.741191+0000","last_active":"2023-10-16T20:58:00.736127+0000","last_peered":"2023-10-16T20:58:00.736127+0000","last_clean":"2023-10-16T20:58:00.736127+0000","last_became_active":"2023-10-16T20:36:54.740944+0000","last_became_peered":"2023-10-16T20:36:54.740944+0000","last_unstale":"2023-10-16T20:58:00.736127+0000","last_undegraded":"2023-10-16T20:58:00.736127+0000","last_fullsized":"2023-10-16T20:58:00.736127+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:14:05.961850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":23,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":219,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:56:52.651677+0000","last_change":"2023-10-16T20:36:46.390001+0000","last_active":"2023-10-16T20:56:52.651677+0000","last_peered":"2023-10-16T20:56:52.651677+0000","last_clean":"2023-10-16T20:56:52.651677+0000","last_became_active":"2023-10-16T20:36:46.388824+0000","last_became_peered":"2023-10-16T20:36:46.388824+0000","last_unstale":"2023-10-16T20:56:52.651677+0000","last_undegraded":"2023-10-16T20:56:52.651677+0000","last_fullsized":"2023-10-16T20:56:52.651677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:44.575506+0000","last_clean_scrub_stamp":"2023-10-16T20:36:44.575506+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:07:31.004318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"67'669","reported_seq":3425,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.621226+0000","last_change":"2023-10-16T20:47:31.593007+0000","last_active":"2023-10-16T20:58:22.621226+0000","last_peered":"2023-10-16T20:58:22.621226+0000","last_clean":"2023-10-16T20:58:22.621226+0000","last_became_active":"2023-10-16T20:36:59.468082+0000","last_became_peered":"2023-10-16T20:36:59.468082+0000","last_unstale":"2023-10-16T20:58:22.621226+0000","last_undegraded":"2023-10-16T20:58:22.621226+0000","last_fullsized":"2023-10-16T20:58:22.621226+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":669,"log_dups_size":0,"ondisk_log_size":669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:49:06.585943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013154572,"stat_sum":{"num_bytes":30910053,"num_objects":362,"num_object_clones":0,"num_object_copies":1448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":649,"num_read_kb":15567,"num_write":666,"num_write_kb":42817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'110","reported_seq":2213,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.943568+0000","last_change":"2023-10-16T20:47:31.488408+0000","last_active":"2023-10-16T20:58:37.943568+0000","last_peered":"2023-10-16T20:58:37.943568+0000","last_clean":"2023-10-16T20:58:37.943568+0000","last_became_active":"2023-10-16T20:37:37.409004+0000","last_became_peered":"2023-10-16T20:37:37.409004+0000","last_unstale":"2023-10-16T20:58:37.943568+0000","last_undegraded":"2023-10-16T20:58:37.943568+0000","last_fullsized":"2023-10-16T20:58:37.943568+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060805590000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"66'68","reported_seq":138,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:19.377575+0000","last_change":"2023-10-16T20:36:54.746600+0000","last_active":"2023-10-16T20:57:19.377575+0000","last_peered":"2023-10-16T20:57:19.377575+0000","last_clean":"2023-10-16T20:57:19.377575+0000","last_became_active":"2023-10-16T20:36:54.746247+0000","last_became_peered":"2023-10-16T20:36:54.746247+0000","last_unstale":"2023-10-16T20:57:19.377575+0000","last_undegraded":"2023-10-16T20:57:19.377575+0000","last_fullsized":"2023-10-16T20:57:19.377575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:28:57.949487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":7042,"num_write":68,"num_write_kb":4224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"67'680","reported_seq":3680,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.986567+0000","last_change":"2023-10-16T20:47:31.590576+0000","last_active":"2023-10-16T20:58:22.986567+0000","last_peered":"2023-10-16T20:58:22.986567+0000","last_clean":"2023-10-16T20:58:22.986567+0000","last_became_active":"2023-10-16T20:37:37.467402+0000","last_became_peered":"2023-10-16T20:37:37.467402+0000","last_unstale":"2023-10-16T20:58:22.986567+0000","last_undegraded":"2023-10-16T20:58:22.986567+0000","last_fullsized":"2023-10-16T20:58:22.986567+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":680,"log_dups_size":0,"ondisk_log_size":680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:32:26.907296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012017773000000001,"stat_sum":{"num_bytes":36935626,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":884,"num_read_kb":21394,"num_write":679,"num_write_kb":40520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"67'106","reported_seq":2215,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.895855+0000","last_change":"2023-10-16T20:47:31.589500+0000","last_active":"2023-10-16T20:58:37.895855+0000","last_peered":"2023-10-16T20:58:37.895855+0000","last_clean":"2023-10-16T20:58:37.895855+0000","last_became_active":"2023-10-16T20:36:55.300958+0000","last_became_peered":"2023-10-16T20:36:55.300958+0000","last_unstale":"2023-10-16T20:58:37.895855+0000","last_undegraded":"2023-10-16T20:58:37.895855+0000","last_fullsized":"2023-10-16T20:58:37.895855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:29:17.323936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009734431,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"67'315","reported_seq":388,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:42.756619+0000","last_change":"2023-10-16T20:36:54.753770+0000","last_active":"2023-10-16T20:58:42.756619+0000","last_peered":"2023-10-16T20:58:42.756619+0000","last_clean":"2023-10-16T20:58:42.756619+0000","last_became_active":"2023-10-16T20:36:54.753443+0000","last_became_peered":"2023-10-16T20:36:54.753443+0000","last_unstale":"2023-10-16T20:58:42.756619+0000","last_undegraded":"2023-10-16T20:58:42.756619+0000","last_fullsized":"2023-10-16T20:58:42.756619+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9666330,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":15859,"num_write":308,"num_write_kb":9615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'687","reported_seq":3977,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.067835+0000","last_change":"2023-10-16T20:47:31.492598+0000","last_active":"2023-10-16T20:58:23.067835+0000","last_peered":"2023-10-16T20:58:23.067835+0000","last_clean":"2023-10-16T20:58:23.067835+0000","last_became_active":"2023-10-16T20:36:59.302869+0000","last_became_peered":"2023-10-16T20:36:59.302869+0000","last_unstale":"2023-10-16T20:58:23.067835+0000","last_undegraded":"2023-10-16T20:58:23.067835+0000","last_fullsized":"2023-10-16T20:58:23.067835+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012136565,"stat_sum":{"num_bytes":23637013,"num_objects":352,"num_object_clones":0,"num_object_copies":1408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":1183,"num_read_kb":18772,"num_write":687,"num_write_kb":27489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"67'100","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.892995+0000","last_change":"2023-10-16T20:47:31.587528+0000","last_active":"2023-10-16T20:58:37.892995+0000","last_peered":"2023-10-16T20:58:37.892995+0000","last_clean":"2023-10-16T20:58:37.892995+0000","last_became_active":"2023-10-16T20:36:55.291633+0000","last_became_peered":"2023-10-16T20:36:55.291633+0000","last_unstale":"2023-10-16T20:58:37.892995+0000","last_undegraded":"2023-10-16T20:58:37.892995+0000","last_fullsized":"2023-10-16T20:58:37.892995+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:08.345584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085978630000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'120","reported_seq":195,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.767053+0000","last_change":"2023-10-16T20:36:54.748196+0000","last_active":"2023-10-16T20:58:23.767053+0000","last_peered":"2023-10-16T20:58:23.767053+0000","last_clean":"2023-10-16T20:58:23.767053+0000","last_became_active":"2023-10-16T20:36:54.747704+0000","last_became_peered":"2023-10-16T20:36:54.747704+0000","last_unstale":"2023-10-16T20:58:23.767053+0000","last_undegraded":"2023-10-16T20:58:23.767053+0000","last_fullsized":"2023-10-16T20:58:23.767053+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:53:55.471916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":116,"num_write_kb":4228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"67'663","reported_seq":3386,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.098033+0000","last_change":"2023-10-16T20:47:31.595065+0000","last_active":"2023-10-16T20:58:23.098033+0000","last_peered":"2023-10-16T20:58:23.098033+0000","last_clean":"2023-10-16T20:58:23.098033+0000","last_became_active":"2023-10-16T20:36:59.467291+0000","last_became_peered":"2023-10-16T20:36:59.467291+0000","last_unstale":"2023-10-16T20:58:23.098033+0000","last_undegraded":"2023-10-16T20:58:23.098033+0000","last_fullsized":"2023-10-16T20:58:23.098033+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:15.723182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0131915,"stat_sum":{"num_bytes":23136394,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":616,"num_read_kb":10065,"num_write":660,"num_write_kb":35181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"67'80","reported_seq":2189,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.876288+0000","last_change":"2023-10-16T20:47:31.589747+0000","last_active":"2023-10-16T20:58:37.876288+0000","last_peered":"2023-10-16T20:58:37.876288+0000","last_clean":"2023-10-16T20:58:37.876288+0000","last_became_active":"2023-10-16T20:36:55.290998+0000","last_became_peered":"2023-10-16T20:36:55.290998+0000","last_unstale":"2023-10-16T20:58:37.876288+0000","last_undegraded":"2023-10-16T20:58:37.876288+0000","last_fullsized":"2023-10-16T20:58:37.876288+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:52:52.835051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082191850000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'91","reported_seq":159,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:33.484841+0000","last_change":"2023-10-16T20:36:54.763818+0000","last_active":"2023-10-16T20:58:33.484841+0000","last_peered":"2023-10-16T20:58:33.484841+0000","last_clean":"2023-10-16T20:58:33.484841+0000","last_became_active":"2023-10-16T20:36:54.761337+0000","last_became_peered":"2023-10-16T20:36:54.761337+0000","last_unstale":"2023-10-16T20:58:33.484841+0000","last_undegraded":"2023-10-16T20:58:33.484841+0000","last_fullsized":"2023-10-16T20:58:33.484841+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:33:03.093189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":5,"num_write":137,"num_write_kb":4352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"67'666","reported_seq":3840,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.018714+0000","last_change":"2023-10-16T20:47:31.487063+0000","last_active":"2023-10-16T20:58:23.018714+0000","last_peered":"2023-10-16T20:58:23.018714+0000","last_clean":"2023-10-16T20:58:23.018714+0000","last_became_active":"2023-10-16T20:37:37.468309+0000","last_became_peered":"2023-10-16T20:37:37.468309+0000","last_unstale":"2023-10-16T20:58:23.018714+0000","last_undegraded":"2023-10-16T20:58:23.018714+0000","last_fullsized":"2023-10-16T20:58:23.018714+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:27:13.271803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058030460000000001,"stat_sum":{"num_bytes":20758781,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":1058,"num_read_kb":21621,"num_write":667,"num_write_kb":32853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"67'107","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.887159+0000","last_change":"2023-10-16T20:47:31.589398+0000","last_active":"2023-10-16T20:58:37.887159+0000","last_peered":"2023-10-16T20:58:37.887159+0000","last_clean":"2023-10-16T20:58:37.887159+0000","last_became_active":"2023-10-16T20:36:55.288354+0000","last_became_peered":"2023-10-16T20:36:55.288354+0000","last_unstale":"2023-10-16T20:58:37.887159+0000","last_undegraded":"2023-10-16T20:58:37.887159+0000","last_fullsized":"2023-10-16T20:58:37.887159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:46:09.359239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076786709999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"67'333","reported_seq":413,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.977642+0000","last_change":"2023-10-16T20:36:54.738216+0000","last_active":"2023-10-16T20:58:05.977642+0000","last_peered":"2023-10-16T20:58:05.977642+0000","last_clean":"2023-10-16T20:58:05.977642+0000","last_became_active":"2023-10-16T20:36:54.737876+0000","last_became_peered":"2023-10-16T20:36:54.737876+0000","last_unstale":"2023-10-16T20:58:05.977642+0000","last_undegraded":"2023-10-16T20:58:05.977642+0000","last_fullsized":"2023-10-16T20:58:05.977642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:48:28.214601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":39,"num_read_kb":4134,"num_write":333,"num_write_kb":8414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"67'666","reported_seq":3666,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:39.656922+0000","last_change":"2023-10-16T20:47:31.490637+0000","last_active":"2023-10-16T20:58:39.656922+0000","last_peered":"2023-10-16T20:58:39.656922+0000","last_clean":"2023-10-16T20:58:39.656922+0000","last_became_active":"2023-10-16T20:36:59.306801+0000","last_became_peered":"2023-10-16T20:36:59.306801+0000","last_unstale":"2023-10-16T20:58:39.656922+0000","last_undegraded":"2023-10-16T20:58:39.656922+0000","last_fullsized":"2023-10-16T20:58:39.656922+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010585667,"stat_sum":{"num_bytes":20444029,"num_objects":301,"num_object_clones":0,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":893,"num_read_kb":19660,"num_write":666,"num_write_kb":28474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"67'93","reported_seq":2202,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.873494+0000","last_change":"2023-10-16T20:47:31.590902+0000","last_active":"2023-10-16T20:58:37.873494+0000","last_peered":"2023-10-16T20:58:37.873494+0000","last_clean":"2023-10-16T20:58:37.873494+0000","last_became_active":"2023-10-16T20:36:55.287066+0000","last_became_peered":"2023-10-16T20:36:55.287066+0000","last_unstale":"2023-10-16T20:58:37.873494+0000","last_undegraded":"2023-10-16T20:58:37.873494+0000","last_fullsized":"2023-10-16T20:58:37.873494+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:42:36.165939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012121162,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"66'101","reported_seq":172,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:57.653854+0000","last_change":"2023-10-16T20:36:54.764693+0000","last_active":"2023-10-16T20:57:57.653854+0000","last_peered":"2023-10-16T20:57:57.653854+0000","last_clean":"2023-10-16T20:57:57.653854+0000","last_became_active":"2023-10-16T20:36:54.764417+0000","last_became_peered":"2023-10-16T20:36:54.764417+0000","last_unstale":"2023-10-16T20:57:57.653854+0000","last_undegraded":"2023-10-16T20:57:57.653854+0000","last_fullsized":"2023-10-16T20:57:57.653854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:54:56.653149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":4110,"num_write":99,"num_write_kb":4200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'634","reported_seq":3755,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.016746+0000","last_change":"2023-10-16T20:47:31.584541+0000","last_active":"2023-10-16T20:58:23.016746+0000","last_peered":"2023-10-16T20:58:23.016746+0000","last_clean":"2023-10-16T20:58:23.016746+0000","last_became_active":"2023-10-16T20:36:59.464626+0000","last_became_peered":"2023-10-16T20:36:59.464626+0000","last_unstale":"2023-10-16T20:58:23.016746+0000","last_undegraded":"2023-10-16T20:58:23.016746+0000","last_fullsized":"2023-10-16T20:58:23.016746+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":634,"log_dups_size":0,"ondisk_log_size":634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:18:35.242031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069938420000000001,"stat_sum":{"num_bytes":28405361,"num_objects":321,"num_object_clones":0,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":1015,"num_read_kb":24481,"num_write":633,"num_write_kb":44456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"67'113","reported_seq":2222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.886956+0000","last_change":"2023-10-16T20:47:31.589722+0000","last_active":"2023-10-16T20:58:37.886956+0000","last_peered":"2023-10-16T20:58:37.886956+0000","last_clean":"2023-10-16T20:58:37.886956+0000","last_became_active":"2023-10-16T20:36:55.291396+0000","last_became_peered":"2023-10-16T20:36:55.291396+0000","last_unstale":"2023-10-16T20:58:37.886956+0000","last_undegraded":"2023-10-16T20:58:37.886956+0000","last_fullsized":"2023-10-16T20:58:37.886956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:09:26.019798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011464676,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"67'184","reported_seq":258,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.657888+0000","last_change":"2023-10-16T20:36:54.750814+0000","last_active":"2023-10-16T20:58:05.657888+0000","last_peered":"2023-10-16T20:58:05.657888+0000","last_clean":"2023-10-16T20:58:05.657888+0000","last_became_active":"2023-10-16T20:36:54.750448+0000","last_became_peered":"2023-10-16T20:36:54.750448+0000","last_unstale":"2023-10-16T20:58:05.657888+0000","last_undegraded":"2023-10-16T20:58:05.657888+0000","last_fullsized":"2023-10-16T20:58:05.657888+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:41.981408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":3911,"num_write":180,"num_write_kb":4223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"67'608","reported_seq":3330,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.013307+0000","last_change":"2023-10-16T20:47:31.583645+0000","last_active":"2023-10-16T20:58:23.013307+0000","last_peered":"2023-10-16T20:58:23.013307+0000","last_clean":"2023-10-16T20:58:23.013307+0000","last_became_active":"2023-10-16T20:36:59.464578+0000","last_became_peered":"2023-10-16T20:36:59.464578+0000","last_unstale":"2023-10-16T20:58:23.013307+0000","last_undegraded":"2023-10-16T20:58:23.013307+0000","last_fullsized":"2023-10-16T20:58:23.013307+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:56.818326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064826880000000003,"stat_sum":{"num_bytes":21354704,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":616,"num_read_kb":9691,"num_write":609,"num_write_kb":35926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"67'101","reported_seq":2211,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.759267+0000","last_change":"2023-10-16T20:47:31.490514+0000","last_active":"2023-10-16T20:58:38.759267+0000","last_peered":"2023-10-16T20:58:38.759267+0000","last_clean":"2023-10-16T20:58:38.759267+0000","last_became_active":"2023-10-16T20:36:55.285407+0000","last_became_peered":"2023-10-16T20:36:55.285407+0000","last_unstale":"2023-10-16T20:58:38.759267+0000","last_undegraded":"2023-10-16T20:58:38.759267+0000","last_fullsized":"2023-10-16T20:58:38.759267+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010336254,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"66'459","reported_seq":605,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.710652+0000","last_change":"2023-10-16T20:36:54.757012+0000","last_active":"2023-10-16T20:58:28.710652+0000","last_peered":"2023-10-16T20:58:28.710652+0000","last_clean":"2023-10-16T20:58:28.710652+0000","last_became_active":"2023-10-16T20:36:54.756697+0000","last_became_peered":"2023-10-16T20:36:54.756697+0000","last_unstale":"2023-10-16T20:58:28.710652+0000","last_undegraded":"2023-10-16T20:58:28.710652+0000","last_fullsized":"2023-10-16T20:58:28.710652+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":459,"log_dups_size":0,"ondisk_log_size":459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8464867,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":106,"num_read_kb":16561,"num_write":461,"num_write_kb":8611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'632","reported_seq":4047,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.726780+0000","last_change":"2023-10-16T20:47:31.589709+0000","last_active":"2023-10-16T20:58:22.726780+0000","last_peered":"2023-10-16T20:58:22.726780+0000","last_clean":"2023-10-16T20:58:22.726780+0000","last_became_active":"2023-10-16T20:36:59.468254+0000","last_became_peered":"2023-10-16T20:36:59.468254+0000","last_unstale":"2023-10-16T20:58:22.726780+0000","last_undegraded":"2023-10-16T20:58:22.726780+0000","last_fullsized":"2023-10-16T20:58:22.726780+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:09:21.589899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093892239999999998,"stat_sum":{"num_bytes":28874401,"num_objects":300,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":1309,"num_read_kb":15538,"num_write":630,"num_write_kb":28525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"67'100","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.880164+0000","last_change":"2023-10-16T20:47:31.588140+0000","last_active":"2023-10-16T20:58:37.880164+0000","last_peered":"2023-10-16T20:58:37.880164+0000","last_clean":"2023-10-16T20:58:37.880164+0000","last_became_active":"2023-10-16T20:36:55.288488+0000","last_became_peered":"2023-10-16T20:36:55.288488+0000","last_unstale":"2023-10-16T20:58:37.880164+0000","last_undegraded":"2023-10-16T20:58:37.880164+0000","last_fullsized":"2023-10-16T20:58:37.880164+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:40:33.001741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078405860000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"67'139","reported_seq":223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.709051+0000","last_change":"2023-10-16T20:36:54.765361+0000","last_active":"2023-10-16T20:58:28.709051+0000","last_peered":"2023-10-16T20:58:28.709051+0000","last_clean":"2023-10-16T20:58:28.709051+0000","last_became_active":"2023-10-16T20:36:54.764969+0000","last_became_peered":"2023-10-16T20:36:54.764969+0000","last_unstale":"2023-10-16T20:58:28.709051+0000","last_undegraded":"2023-10-16T20:58:28.709051+0000","last_fullsized":"2023-10-16T20:58:28.709051+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:21:21.183612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389280,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":45,"num_read_kb":8235,"num_write":134,"num_write_kb":8274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"67'666","reported_seq":4059,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.063602+0000","last_change":"2023-10-16T20:47:31.488335+0000","last_active":"2023-10-16T20:58:23.063602+0000","last_peered":"2023-10-16T20:58:23.063602+0000","last_clean":"2023-10-16T20:58:23.063602+0000","last_became_active":"2023-10-16T20:36:59.302698+0000","last_became_peered":"2023-10-16T20:36:59.302698+0000","last_unstale":"2023-10-16T20:58:23.063602+0000","last_undegraded":"2023-10-16T20:58:23.063602+0000","last_fullsized":"2023-10-16T20:58:23.063602+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064797020000000004,"stat_sum":{"num_bytes":31392251,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":1286,"num_read_kb":27897,"num_write":665,"num_write_kb":43250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'99","reported_seq":2208,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.882106+0000","last_change":"2023-10-16T20:47:31.487882+0000","last_active":"2023-10-16T20:58:37.882106+0000","last_peered":"2023-10-16T20:58:37.882106+0000","last_clean":"2023-10-16T20:58:37.882106+0000","last_became_active":"2023-10-16T20:36:55.290127+0000","last_became_peered":"2023-10-16T20:36:55.290127+0000","last_unstale":"2023-10-16T20:58:37.882106+0000","last_undegraded":"2023-10-16T20:58:37.882106+0000","last_fullsized":"2023-10-16T20:58:37.882106+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:23.063288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055494209999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"67'78","reported_seq":151,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:42.658704+0000","last_change":"2023-10-16T20:36:54.749120+0000","last_active":"2023-10-16T20:58:42.658704+0000","last_peered":"2023-10-16T20:58:42.658704+0000","last_clean":"2023-10-16T20:58:42.658704+0000","last_became_active":"2023-10-16T20:36:54.748788+0000","last_became_peered":"2023-10-16T20:36:54.748788+0000","last_unstale":"2023-10-16T20:58:42.658704+0000","last_undegraded":"2023-10-16T20:58:42.658704+0000","last_fullsized":"2023-10-16T20:58:42.658704+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":15,"num_read_kb":23,"num_write":83,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"67'606","reported_seq":3860,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.746395+0000","last_change":"2023-10-16T20:47:31.486033+0000","last_active":"2023-10-16T20:58:22.746395+0000","last_peered":"2023-10-16T20:58:22.746395+0000","last_clean":"2023-10-16T20:58:22.746395+0000","last_became_active":"2023-10-16T20:36:59.302916+0000","last_became_peered":"2023-10-16T20:36:59.302916+0000","last_unstale":"2023-10-16T20:58:22.746395+0000","last_undegraded":"2023-10-16T20:58:22.746395+0000","last_fullsized":"2023-10-16T20:58:22.746395+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:47:09.867705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041353309999999999,"stat_sum":{"num_bytes":29291517,"num_objects":300,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":1148,"num_read_kb":24111,"num_write":607,"num_write_kb":41174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"67'107","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:44.657550+0000","last_change":"2023-10-16T20:47:31.490807+0000","last_active":"2023-10-16T20:58:44.657550+0000","last_peered":"2023-10-16T20:58:44.657550+0000","last_clean":"2023-10-16T20:58:44.657550+0000","last_became_active":"2023-10-16T20:36:55.291448+0000","last_became_peered":"2023-10-16T20:36:55.291448+0000","last_unstale":"2023-10-16T20:58:44.657550+0000","last_undegraded":"2023-10-16T20:58:44.657550+0000","last_fullsized":"2023-10-16T20:58:44.657550+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010585074,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"67'134","reported_seq":254,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:39.515133+0000","last_change":"2023-10-16T20:36:54.761824+0000","last_active":"2023-10-16T20:58:39.515133+0000","last_peered":"2023-10-16T20:58:39.515133+0000","last_clean":"2023-10-16T20:58:39.515133+0000","last_became_active":"2023-10-16T20:36:54.761541+0000","last_became_peered":"2023-10-16T20:36:54.761541+0000","last_unstale":"2023-10-16T20:58:39.515133+0000","last_undegraded":"2023-10-16T20:58:39.515133+0000","last_fullsized":"2023-10-16T20:58:39.515133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:54:34.512207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2461990,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":69,"num_read_kb":129,"num_write":135,"num_write_kb":2563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"67'663","reported_seq":3325,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.270542+0000","last_change":"2023-10-16T20:47:31.585744+0000","last_active":"2023-10-16T20:58:22.270542+0000","last_peered":"2023-10-16T20:58:22.270542+0000","last_clean":"2023-10-16T20:58:22.270542+0000","last_became_active":"2023-10-16T20:36:59.303773+0000","last_became_peered":"2023-10-16T20:36:59.303773+0000","last_unstale":"2023-10-16T20:58:22.270542+0000","last_undegraded":"2023-10-16T20:58:22.270542+0000","last_fullsized":"2023-10-16T20:58:22.270542+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:19:00.801685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007941142,"stat_sum":{"num_bytes":21664408,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":557,"num_read_kb":9585,"num_write":658,"num_write_kb":32916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"67'121","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.891891+0000","last_change":"2023-10-16T20:47:31.594424+0000","last_active":"2023-10-16T20:58:37.891891+0000","last_peered":"2023-10-16T20:58:37.891891+0000","last_clean":"2023-10-16T20:58:37.891891+0000","last_became_active":"2023-10-16T20:36:55.300535+0000","last_became_peered":"2023-10-16T20:36:55.300535+0000","last_unstale":"2023-10-16T20:58:37.891891+0000","last_undegraded":"2023-10-16T20:58:37.891891+0000","last_fullsized":"2023-10-16T20:58:37.891891+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:47:03.132953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013755641000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"67'55","reported_seq":124,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:30.004648+0000","last_change":"2023-10-16T20:36:54.758140+0000","last_active":"2023-10-16T20:58:30.004648+0000","last_peered":"2023-10-16T20:58:30.004648+0000","last_clean":"2023-10-16T20:58:30.004648+0000","last_became_active":"2023-10-16T20:36:54.757303+0000","last_became_peered":"2023-10-16T20:36:54.757303+0000","last_unstale":"2023-10-16T20:58:30.004648+0000","last_undegraded":"2023-10-16T20:58:30.004648+0000","last_fullsized":"2023-10-16T20:58:30.004648+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:36:45.944191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":56,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"67'687","reported_seq":4217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.643179+0000","last_change":"2023-10-16T20:47:31.586327+0000","last_active":"2023-10-16T20:58:22.643179+0000","last_peered":"2023-10-16T20:58:22.643179+0000","last_clean":"2023-10-16T20:58:22.643179+0000","last_became_active":"2023-10-16T20:36:59.465844+0000","last_became_peered":"2023-10-16T20:36:59.465844+0000","last_unstale":"2023-10-16T20:58:22.643179+0000","last_undegraded":"2023-10-16T20:58:22.643179+0000","last_fullsized":"2023-10-16T20:58:22.643179+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:54.762444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081168009999999999,"stat_sum":{"num_bytes":45881583,"num_objects":353,"num_object_clones":0,"num_object_copies":1412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":353,"num_whiteouts":0,"num_read":1424,"num_read_kb":44764,"num_write":688,"num_write_kb":65580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"67'661","reported_seq":4048,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.629546+0000","last_change":"2023-10-16T20:47:31.489178+0000","last_active":"2023-10-16T20:58:22.629546+0000","last_peered":"2023-10-16T20:58:22.629546+0000","last_clean":"2023-10-16T20:58:22.629546+0000","last_became_active":"2023-10-16T20:36:59.467356+0000","last_became_peered":"2023-10-16T20:36:59.467356+0000","last_unstale":"2023-10-16T20:58:22.629546+0000","last_undegraded":"2023-10-16T20:58:22.629546+0000","last_fullsized":"2023-10-16T20:58:22.629546+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093722700000000003,"stat_sum":{"num_bytes":37038710,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1279,"num_read_kb":28276,"num_write":659,"num_write_kb":40616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'108","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:47.276397+0000","last_change":"2023-10-16T20:47:31.484094+0000","last_active":"2023-10-16T20:58:47.276397+0000","last_peered":"2023-10-16T20:58:47.276397+0000","last_clean":"2023-10-16T20:58:47.276397+0000","last_became_active":"2023-10-16T20:36:55.280903+0000","last_became_peered":"2023-10-16T20:36:55.280903+0000","last_unstale":"2023-10-16T20:58:47.276397+0000","last_undegraded":"2023-10-16T20:58:47.276397+0000","last_fullsized":"2023-10-16T20:58:47.276397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029269690000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"67'457","reported_seq":528,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.761682+0000","last_change":"2023-10-16T20:36:54.757589+0000","last_active":"2023-10-16T20:58:23.761682+0000","last_peered":"2023-10-16T20:58:23.761682+0000","last_clean":"2023-10-16T20:58:23.761682+0000","last_became_active":"2023-10-16T20:36:54.756775+0000","last_became_peered":"2023-10-16T20:36:54.756775+0000","last_unstale":"2023-10-16T20:58:23.761682+0000","last_undegraded":"2023-10-16T20:58:23.761682+0000","last_fullsized":"2023-10-16T20:58:23.761682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:54:40.629540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583494,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":16386,"num_write":448,"num_write_kb":12544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"67'638","reported_seq":3559,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.378852+0000","last_change":"2023-10-16T20:47:31.593596+0000","last_active":"2023-10-16T20:58:22.378852+0000","last_peered":"2023-10-16T20:58:22.378852+0000","last_clean":"2023-10-16T20:58:22.378852+0000","last_became_active":"2023-10-16T20:37:37.438140+0000","last_became_peered":"2023-10-16T20:37:37.438140+0000","last_unstale":"2023-10-16T20:58:22.378852+0000","last_undegraded":"2023-10-16T20:58:22.378852+0000","last_fullsized":"2023-10-16T20:58:22.378852+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:41:29.134511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012287041,"stat_sum":{"num_bytes":24792014,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":805,"num_read_kb":12206,"num_write":635,"num_write_kb":32676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"67'121","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.897843+0000","last_change":"2023-10-16T20:47:31.585618+0000","last_active":"2023-10-16T20:58:37.897843+0000","last_peered":"2023-10-16T20:58:37.897843+0000","last_clean":"2023-10-16T20:58:37.897843+0000","last_became_active":"2023-10-16T20:36:55.273959+0000","last_became_peered":"2023-10-16T20:36:55.273959+0000","last_unstale":"2023-10-16T20:58:37.897843+0000","last_undegraded":"2023-10-16T20:58:37.897843+0000","last_fullsized":"2023-10-16T20:58:37.897843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:05:00.921391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076333039999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"67'135","reported_seq":276,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.678551+0000","last_change":"2023-10-16T20:36:54.759513+0000","last_active":"2023-10-16T20:58:28.678551+0000","last_peered":"2023-10-16T20:58:28.678551+0000","last_clean":"2023-10-16T20:58:28.678551+0000","last_became_active":"2023-10-16T20:36:54.759059+0000","last_became_peered":"2023-10-16T20:36:54.759059+0000","last_unstale":"2023-10-16T20:58:28.678551+0000","last_undegraded":"2023-10-16T20:58:28.678551+0000","last_fullsized":"2023-10-16T20:58:28.678551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:59:36.696053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4348959,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":88,"num_read_kb":239,"num_write":133,"num_write_kb":4341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"66'223","reported_seq":372,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:33.632797+0000","last_change":"2023-10-16T20:36:54.764780+0000","last_active":"2023-10-16T20:57:33.632797+0000","last_peered":"2023-10-16T20:57:33.632797+0000","last_clean":"2023-10-16T20:57:33.632797+0000","last_became_active":"2023-10-16T20:36:54.764436+0000","last_became_peered":"2023-10-16T20:36:54.764436+0000","last_unstale":"2023-10-16T20:57:33.632797+0000","last_undegraded":"2023-10-16T20:57:33.632797+0000","last_fullsized":"2023-10-16T20:57:33.632797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:55:31.816599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8524313,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":105,"num_read_kb":8412,"num_write":220,"num_write_kb":8480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"67'109","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.891992+0000","last_change":"2023-10-16T20:47:31.492923+0000","last_active":"2023-10-16T20:58:37.891992+0000","last_peered":"2023-10-16T20:58:37.891992+0000","last_clean":"2023-10-16T20:58:37.891992+0000","last_became_active":"2023-10-16T20:36:55.277873+0000","last_became_peered":"2023-10-16T20:36:55.277873+0000","last_unstale":"2023-10-16T20:58:37.891992+0000","last_undegraded":"2023-10-16T20:58:37.891992+0000","last_fullsized":"2023-10-16T20:58:37.891992+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012247773,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'608","reported_seq":3232,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.002422+0000","last_change":"2023-10-16T20:47:31.496724+0000","last_active":"2023-10-16T20:58:23.002422+0000","last_peered":"2023-10-16T20:58:23.002422+0000","last_clean":"2023-10-16T20:58:23.002422+0000","last_became_active":"2023-10-16T20:36:59.302435+0000","last_became_peered":"2023-10-16T20:36:59.302435+0000","last_unstale":"2023-10-16T20:58:23.002422+0000","last_undegraded":"2023-10-16T20:58:23.002422+0000","last_fullsized":"2023-10-16T20:58:23.002422+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:34:39.329693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001061892,"stat_sum":{"num_bytes":16477397,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":519,"num_read_kb":10696,"num_write":606,"num_write_kb":24605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"66'232","reported_seq":301,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-10-16T20:57:33.636417+0000","last_change":"2023-10-16T20:36:54.756190+0000","last_active":"2023-10-16T20:57:33.636417+0000","last_peered":"2023-10-16T20:57:33.636417+0000","last_clean":"2023-10-16T20:57:33.636417+0000","last_became_active":"2023-10-16T20:36:54.755834+0000","last_became_peered":"2023-10-16T20:36:54.755834+0000","last_unstale":"2023-10-16T20:57:33.636417+0000","last_undegraded":"2023-10-16T20:57:33.636417+0000","last_fullsized":"2023-10-16T20:57:33.636417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:22:31.012441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582946,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":16385,"num_write":235,"num_write_kb":12609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"67'104","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.885029+0000","last_change":"2023-10-16T20:47:31.486761+0000","last_active":"2023-10-16T20:58:37.885029+0000","last_peered":"2023-10-16T20:58:37.885029+0000","last_clean":"2023-10-16T20:58:37.885029+0000","last_became_active":"2023-10-16T20:36:55.279333+0000","last_became_peered":"2023-10-16T20:36:55.279333+0000","last_unstale":"2023-10-16T20:58:37.885029+0000","last_undegraded":"2023-10-16T20:58:37.885029+0000","last_fullsized":"2023-10-16T20:58:37.885029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:30:26.879792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047896609999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'629","reported_seq":3442,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.721224+0000","last_change":"2023-10-16T20:47:31.586162+0000","last_active":"2023-10-16T20:58:22.721224+0000","last_peered":"2023-10-16T20:58:22.721224+0000","last_clean":"2023-10-16T20:58:22.721224+0000","last_became_active":"2023-10-16T20:36:59.301864+0000","last_became_peered":"2023-10-16T20:36:59.301864+0000","last_unstale":"2023-10-16T20:58:22.721224+0000","last_undegraded":"2023-10-16T20:58:22.721224+0000","last_fullsized":"2023-10-16T20:58:22.721224+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:07:44.207453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080450769999999994,"stat_sum":{"num_bytes":20568472,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":708,"num_read_kb":22370,"num_write":628,"num_write_kb":40892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"67'120","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.875503+0000","last_change":"2023-10-16T20:47:31.590065+0000","last_active":"2023-10-16T20:58:37.875503+0000","last_peered":"2023-10-16T20:58:37.875503+0000","last_clean":"2023-10-16T20:58:37.875503+0000","last_became_active":"2023-10-16T20:36:55.287442+0000","last_became_peered":"2023-10-16T20:36:55.287442+0000","last_unstale":"2023-10-16T20:58:37.875503+0000","last_undegraded":"2023-10-16T20:58:37.875503+0000","last_fullsized":"2023-10-16T20:58:37.875503+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:37:33.639292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011630351000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"67'131","reported_seq":2241,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.880621+0000","last_change":"2023-10-16T20:47:31.497003+0000","last_active":"2023-10-16T20:58:37.880621+0000","last_peered":"2023-10-16T20:58:37.880621+0000","last_clean":"2023-10-16T20:58:37.880621+0000","last_became_active":"2023-10-16T20:36:55.276276+0000","last_became_peered":"2023-10-16T20:36:55.276276+0000","last_unstale":"2023-10-16T20:58:37.880621+0000","last_undegraded":"2023-10-16T20:58:37.880621+0000","last_fullsized":"2023-10-16T20:58:37.880621+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:35:00.858846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091802500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"67'115","reported_seq":2232,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.886101+0000","last_change":"2023-10-16T20:47:31.592299+0000","last_active":"2023-10-16T20:58:37.886101+0000","last_peered":"2023-10-16T20:58:37.886101+0000","last_clean":"2023-10-16T20:58:37.886101+0000","last_became_active":"2023-10-16T20:37:37.413040+0000","last_became_peered":"2023-10-16T20:37:37.413040+0000","last_unstale":"2023-10-16T20:58:37.886101+0000","last_undegraded":"2023-10-16T20:58:37.886101+0000","last_fullsized":"2023-10-16T20:58:37.886101+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:50:46.069439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012618513,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"67'116","reported_seq":2227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.892711+0000","last_change":"2023-10-16T20:47:31.492800+0000","last_active":"2023-10-16T20:58:37.892711+0000","last_peered":"2023-10-16T20:58:37.892711+0000","last_clean":"2023-10-16T20:58:37.892711+0000","last_became_active":"2023-10-16T20:36:55.275349+0000","last_became_peered":"2023-10-16T20:36:55.275349+0000","last_unstale":"2023-10-16T20:58:37.892711+0000","last_undegraded":"2023-10-16T20:58:37.892711+0000","last_fullsized":"2023-10-16T20:58:37.892711+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012136073000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"67'88","reported_seq":2198,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.852714+0000","last_change":"2023-10-16T20:47:31.587774+0000","last_active":"2023-10-16T20:58:37.852714+0000","last_peered":"2023-10-16T20:58:37.852714+0000","last_clean":"2023-10-16T20:58:37.852714+0000","last_became_active":"2023-10-16T20:36:55.277043+0000","last_became_peered":"2023-10-16T20:36:55.277043+0000","last_unstale":"2023-10-16T20:58:37.852714+0000","last_undegraded":"2023-10-16T20:58:37.852714+0000","last_fullsized":"2023-10-16T20:58:37.852714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:10:32.439680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078065369999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"67'89","reported_seq":2200,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.863323+0000","last_change":"2023-10-16T20:47:31.489466+0000","last_active":"2023-10-16T20:58:37.863323+0000","last_peered":"2023-10-16T20:58:37.863323+0000","last_clean":"2023-10-16T20:58:37.863323+0000","last_became_active":"2023-10-16T20:36:55.278715+0000","last_became_peered":"2023-10-16T20:36:55.278715+0000","last_unstale":"2023-10-16T20:58:37.863323+0000","last_undegraded":"2023-10-16T20:58:37.863323+0000","last_fullsized":"2023-10-16T20:58:37.863323+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067460719999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"67'124","reported_seq":2234,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.836534+0000","last_change":"2023-10-16T20:47:31.594095+0000","last_active":"2023-10-16T20:58:37.836534+0000","last_peered":"2023-10-16T20:58:37.836534+0000","last_clean":"2023-10-16T20:58:37.836534+0000","last_became_active":"2023-10-16T20:36:55.301604+0000","last_became_peered":"2023-10-16T20:36:55.301604+0000","last_unstale":"2023-10-16T20:58:37.836534+0000","last_undegraded":"2023-10-16T20:58:37.836534+0000","last_fullsized":"2023-10-16T20:58:37.836534+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:10.110081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012969489000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"67'99","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.858692+0000","last_change":"2023-10-16T20:47:31.587306+0000","last_active":"2023-10-16T20:58:37.858692+0000","last_peered":"2023-10-16T20:58:37.858692+0000","last_clean":"2023-10-16T20:58:37.858692+0000","last_became_active":"2023-10-16T20:36:55.276984+0000","last_became_peered":"2023-10-16T20:36:55.276984+0000","last_unstale":"2023-10-16T20:58:37.858692+0000","last_undegraded":"2023-10-16T20:58:37.858692+0000","last_fullsized":"2023-10-16T20:58:37.858692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:42:52.651942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086623480000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'111","reported_seq":2221,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.896916+0000","last_change":"2023-10-16T20:47:31.595489+0000","last_active":"2023-10-16T20:58:37.896916+0000","last_peered":"2023-10-16T20:58:37.896916+0000","last_clean":"2023-10-16T20:58:37.896916+0000","last_became_active":"2023-10-16T20:36:55.290023+0000","last_became_peered":"2023-10-16T20:36:55.290023+0000","last_unstale":"2023-10-16T20:58:37.896916+0000","last_undegraded":"2023-10-16T20:58:37.896916+0000","last_fullsized":"2023-10-16T20:58:37.896916+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:16:33.259824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013081514000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'124","reported_seq":2233,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.203227+0000","last_change":"2023-10-16T20:47:31.586678+0000","last_active":"2023-10-16T20:58:38.203227+0000","last_peered":"2023-10-16T20:58:38.203227+0000","last_clean":"2023-10-16T20:58:38.203227+0000","last_became_active":"2023-10-16T20:36:55.280186+0000","last_became_peered":"2023-10-16T20:36:55.280186+0000","last_unstale":"2023-10-16T20:58:38.203227+0000","last_undegraded":"2023-10-16T20:58:38.203227+0000","last_fullsized":"2023-10-16T20:58:38.203227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:22:01.009457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083925690000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"67'118","reported_seq":2227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.882744+0000","last_change":"2023-10-16T20:47:31.589254+0000","last_active":"2023-10-16T20:58:37.882744+0000","last_peered":"2023-10-16T20:58:37.882744+0000","last_clean":"2023-10-16T20:58:37.882744+0000","last_became_active":"2023-10-16T20:36:55.297253+0000","last_became_peered":"2023-10-16T20:36:55.297253+0000","last_unstale":"2023-10-16T20:58:37.882744+0000","last_undegraded":"2023-10-16T20:58:37.882744+0000","last_fullsized":"2023-10-16T20:58:37.882744+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:41:11.854849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097685800000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"67'108","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.893558+0000","last_change":"2023-10-16T20:47:31.492104+0000","last_active":"2023-10-16T20:58:37.893558+0000","last_peered":"2023-10-16T20:58:37.893558+0000","last_clean":"2023-10-16T20:58:37.893558+0000","last_became_active":"2023-10-16T20:36:55.286044+0000","last_became_peered":"2023-10-16T20:36:55.286044+0000","last_unstale":"2023-10-16T20:58:37.893558+0000","last_undegraded":"2023-10-16T20:58:37.893558+0000","last_fullsized":"2023-10-16T20:58:37.893558+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011681474000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"67'110","reported_seq":2219,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.865687+0000","last_change":"2023-10-16T20:47:31.592401+0000","last_active":"2023-10-16T20:58:37.865687+0000","last_peered":"2023-10-16T20:58:37.865687+0000","last_clean":"2023-10-16T20:58:37.865687+0000","last_became_active":"2023-10-16T20:36:55.286864+0000","last_became_peered":"2023-10-16T20:36:55.286864+0000","last_unstale":"2023-10-16T20:58:37.865687+0000","last_undegraded":"2023-10-16T20:58:37.865687+0000","last_fullsized":"2023-10-16T20:58:37.865687+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:36:31.381174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012885232999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"67'106","reported_seq":2215,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.872600+0000","last_change":"2023-10-16T20:47:31.588333+0000","last_active":"2023-10-16T20:58:37.872600+0000","last_peered":"2023-10-16T20:58:37.872600+0000","last_clean":"2023-10-16T20:58:37.872600+0000","last_became_active":"2023-10-16T20:36:55.292262+0000","last_became_peered":"2023-10-16T20:36:55.292262+0000","last_unstale":"2023-10-16T20:58:37.872600+0000","last_undegraded":"2023-10-16T20:58:37.872600+0000","last_fullsized":"2023-10-16T20:58:37.872600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T20:55:15.116061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077171749999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"67'107","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.832173+0000","last_change":"2023-10-16T20:47:31.496924+0000","last_active":"2023-10-16T20:58:37.832173+0000","last_peered":"2023-10-16T20:58:37.832173+0000","last_clean":"2023-10-16T20:58:37.832173+0000","last_became_active":"2023-10-16T20:36:55.282806+0000","last_became_peered":"2023-10-16T20:36:55.282806+0000","last_unstale":"2023-10-16T20:58:37.832173+0000","last_undegraded":"2023-10-16T20:58:37.832173+0000","last_fullsized":"2023-10-16T20:58:37.832173+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:19:51.395313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086132999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"67'109","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.887891+0000","last_change":"2023-10-16T20:47:31.594959+0000","last_active":"2023-10-16T20:58:37.887891+0000","last_peered":"2023-10-16T20:58:37.887891+0000","last_clean":"2023-10-16T20:58:37.887891+0000","last_became_active":"2023-10-16T20:36:55.303095+0000","last_became_peered":"2023-10-16T20:36:55.303095+0000","last_unstale":"2023-10-16T20:58:37.887891+0000","last_undegraded":"2023-10-16T20:58:37.887891+0000","last_fullsized":"2023-10-16T20:58:37.887891+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:00:51.726359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013746639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"67'119","reported_seq":2228,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.181755+0000","last_change":"2023-10-16T20:47:31.595380+0000","last_active":"2023-10-16T20:58:38.181755+0000","last_peered":"2023-10-16T20:58:38.181755+0000","last_clean":"2023-10-16T20:58:38.181755+0000","last_became_active":"2023-10-16T20:36:55.291447+0000","last_became_peered":"2023-10-16T20:36:55.291447+0000","last_unstale":"2023-10-16T20:58:38.181755+0000","last_undegraded":"2023-10-16T20:58:38.181755+0000","last_fullsized":"2023-10-16T20:58:38.181755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:25:20.797976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01315413,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"67'107","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.868260+0000","last_change":"2023-10-16T20:47:31.484708+0000","last_active":"2023-10-16T20:58:37.868260+0000","last_peered":"2023-10-16T20:58:37.868260+0000","last_clean":"2023-10-16T20:58:37.868260+0000","last_became_active":"2023-10-16T20:36:55.280605+0000","last_became_peered":"2023-10-16T20:36:55.280605+0000","last_unstale":"2023-10-16T20:58:37.868260+0000","last_undegraded":"2023-10-16T20:58:37.868260+0000","last_fullsized":"2023-10-16T20:58:37.868260+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033133080000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"67'112","reported_seq":2222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.891293+0000","last_change":"2023-10-16T20:47:31.492149+0000","last_active":"2023-10-16T20:58:37.891293+0000","last_peered":"2023-10-16T20:58:37.891293+0000","last_clean":"2023-10-16T20:58:37.891293+0000","last_became_active":"2023-10-16T20:36:55.277094+0000","last_became_peered":"2023-10-16T20:36:55.277094+0000","last_unstale":"2023-10-16T20:58:37.891293+0000","last_undegraded":"2023-10-16T20:58:37.891293+0000","last_fullsized":"2023-10-16T20:58:37.891293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011816411000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"67'94","reported_seq":2203,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.879368+0000","last_change":"2023-10-16T20:47:31.587367+0000","last_active":"2023-10-16T20:58:37.879368+0000","last_peered":"2023-10-16T20:58:37.879368+0000","last_clean":"2023-10-16T20:58:37.879368+0000","last_became_active":"2023-10-16T20:36:55.297199+0000","last_became_peered":"2023-10-16T20:36:55.297199+0000","last_unstale":"2023-10-16T20:58:37.879368+0000","last_undegraded":"2023-10-16T20:58:37.879368+0000","last_fullsized":"2023-10-16T20:58:37.879368+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:07:32.275973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086430910000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"67'90","reported_seq":2199,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.867006+0000","last_change":"2023-10-16T20:47:31.487461+0000","last_active":"2023-10-16T20:58:37.867006+0000","last_peered":"2023-10-16T20:58:37.867006+0000","last_clean":"2023-10-16T20:58:37.867006+0000","last_became_active":"2023-10-16T20:36:55.286964+0000","last_became_peered":"2023-10-16T20:36:55.286964+0000","last_unstale":"2023-10-16T20:58:37.867006+0000","last_undegraded":"2023-10-16T20:58:37.867006+0000","last_fullsized":"2023-10-16T20:58:37.867006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:26:57.164132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051857609999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"67'110","reported_seq":2226,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.834126+0000","last_change":"2023-10-16T20:47:31.594637+0000","last_active":"2023-10-16T20:58:37.834126+0000","last_peered":"2023-10-16T20:58:37.834126+0000","last_clean":"2023-10-16T20:58:37.834126+0000","last_became_active":"2023-10-16T20:37:37.413492+0000","last_became_peered":"2023-10-16T20:37:37.413492+0000","last_unstale":"2023-10-16T20:58:37.834126+0000","last_undegraded":"2023-10-16T20:58:37.834126+0000","last_fullsized":"2023-10-16T20:58:37.834126+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:08:07.365316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014529932000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"67'106","reported_seq":2215,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.850164+0000","last_change":"2023-10-16T20:47:31.593072+0000","last_active":"2023-10-16T20:58:37.850164+0000","last_peered":"2023-10-16T20:58:37.850164+0000","last_clean":"2023-10-16T20:58:37.850164+0000","last_became_active":"2023-10-16T20:36:55.302468+0000","last_became_peered":"2023-10-16T20:36:55.302468+0000","last_unstale":"2023-10-16T20:58:37.850164+0000","last_undegraded":"2023-10-16T20:58:37.850164+0000","last_fullsized":"2023-10-16T20:58:37.850164+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:54:24.682600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012359399,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"67'92","reported_seq":2201,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.894576+0000","last_change":"2023-10-16T20:47:31.585931+0000","last_active":"2023-10-16T20:58:37.894576+0000","last_peered":"2023-10-16T20:58:37.894576+0000","last_clean":"2023-10-16T20:58:37.894576+0000","last_became_active":"2023-10-16T20:36:55.296476+0000","last_became_peered":"2023-10-16T20:36:55.296476+0000","last_unstale":"2023-10-16T20:58:37.894576+0000","last_undegraded":"2023-10-16T20:58:37.894576+0000","last_fullsized":"2023-10-16T20:58:37.894576+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:50:21.532799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072035780000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'95","reported_seq":2203,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.898420+0000","last_change":"2023-10-16T20:47:31.585218+0000","last_active":"2023-10-16T20:58:37.898420+0000","last_peered":"2023-10-16T20:58:37.898420+0000","last_clean":"2023-10-16T20:58:37.898420+0000","last_became_active":"2023-10-16T20:36:55.293687+0000","last_became_peered":"2023-10-16T20:36:55.293687+0000","last_unstale":"2023-10-16T20:58:37.898420+0000","last_undegraded":"2023-10-16T20:58:37.898420+0000","last_fullsized":"2023-10-16T20:58:37.898420+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:06:36.236196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074548399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"67'127","reported_seq":2235,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.887764+0000","last_change":"2023-10-16T20:47:31.588394+0000","last_active":"2023-10-16T20:58:37.887764+0000","last_peered":"2023-10-16T20:58:37.887764+0000","last_clean":"2023-10-16T20:58:37.887764+0000","last_became_active":"2023-10-16T20:36:55.280444+0000","last_became_peered":"2023-10-16T20:36:55.280444+0000","last_unstale":"2023-10-16T20:58:37.887764+0000","last_undegraded":"2023-10-16T20:58:37.887764+0000","last_fullsized":"2023-10-16T20:58:37.887764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:40:02.818328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068623299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'109","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888019+0000","last_change":"2023-10-16T20:47:31.588884+0000","last_active":"2023-10-16T20:58:37.888019+0000","last_peered":"2023-10-16T20:58:37.888019+0000","last_clean":"2023-10-16T20:58:37.888019+0000","last_became_active":"2023-10-16T20:36:55.296694+0000","last_became_peered":"2023-10-16T20:36:55.296694+0000","last_unstale":"2023-10-16T20:58:37.888019+0000","last_undegraded":"2023-10-16T20:58:37.888019+0000","last_fullsized":"2023-10-16T20:58:37.888019+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:35:39.793533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096291870000000009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"67'120","reported_seq":2228,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.879224+0000","last_change":"2023-10-16T20:47:31.585319+0000","last_active":"2023-10-16T20:58:37.879224+0000","last_peered":"2023-10-16T20:58:37.879224+0000","last_clean":"2023-10-16T20:58:37.879224+0000","last_became_active":"2023-10-16T20:36:55.291052+0000","last_became_peered":"2023-10-16T20:36:55.291052+0000","last_unstale":"2023-10-16T20:58:37.879224+0000","last_undegraded":"2023-10-16T20:58:37.879224+0000","last_fullsized":"2023-10-16T20:58:37.879224+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:19:45.283686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070559409999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"67'94","reported_seq":2204,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.895896+0000","last_change":"2023-10-16T20:47:31.588931+0000","last_active":"2023-10-16T20:58:37.895896+0000","last_peered":"2023-10-16T20:58:37.895896+0000","last_clean":"2023-10-16T20:58:37.895896+0000","last_became_active":"2023-10-16T20:36:55.290386+0000","last_became_peered":"2023-10-16T20:36:55.290386+0000","last_unstale":"2023-10-16T20:58:37.895896+0000","last_undegraded":"2023-10-16T20:58:37.895896+0000","last_fullsized":"2023-10-16T20:58:37.895896+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:56:59.640546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093217969999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"67'114","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888703+0000","last_change":"2023-10-16T20:47:31.490951+0000","last_active":"2023-10-16T20:58:37.888703+0000","last_peered":"2023-10-16T20:58:37.888703+0000","last_clean":"2023-10-16T20:58:37.888703+0000","last_became_active":"2023-10-16T20:36:55.270787+0000","last_became_peered":"2023-10-16T20:36:55.270787+0000","last_unstale":"2023-10-16T20:58:37.888703+0000","last_undegraded":"2023-10-16T20:58:37.888703+0000","last_fullsized":"2023-10-16T20:58:37.888703+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010801276,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"67'94","reported_seq":2205,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.890633+0000","last_change":"2023-10-16T20:47:31.487902+0000","last_active":"2023-10-16T20:58:37.890633+0000","last_peered":"2023-10-16T20:58:37.890633+0000","last_clean":"2023-10-16T20:58:37.890633+0000","last_became_active":"2023-10-16T20:36:55.271373+0000","last_became_peered":"2023-10-16T20:36:55.271373+0000","last_unstale":"2023-10-16T20:58:37.890633+0000","last_undegraded":"2023-10-16T20:58:37.890633+0000","last_fullsized":"2023-10-16T20:58:37.890633+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006091478,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"67'111","reported_seq":2221,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.890234+0000","last_change":"2023-10-16T20:47:31.587880+0000","last_active":"2023-10-16T20:58:37.890234+0000","last_peered":"2023-10-16T20:58:37.890234+0000","last_clean":"2023-10-16T20:58:37.890234+0000","last_became_active":"2023-10-16T20:36:55.291016+0000","last_became_peered":"2023-10-16T20:36:55.291016+0000","last_unstale":"2023-10-16T20:58:37.890234+0000","last_undegraded":"2023-10-16T20:58:37.890234+0000","last_fullsized":"2023-10-16T20:58:37.890234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:23:37.851121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009370148,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":924319467,"num_objects":10294,"num_object_clones":0,"num_object_copies":41176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10294,"num_whiteouts":0,"num_read":30602,"num_read_kb":676969,"num_write":21002,"num_write_kb":1302825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966899200,"data_stored":1966899200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21050,"ondisk_log_size":21050,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6794,"ondisk_log_size":6794,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":178968054,"num_objects":198,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1063,"num_read_kb":205540,"num_write":6353,"num_write_kb":180525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":115,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":358178816,"data_stored":357936108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6082,"ondisk_log_size":6082,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509708,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396776,"kb_used_data":280928,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93975064,"statfs":{"total":96636764160,"available":96230465536,"internally_reserved":0,"allocated":287670272,"data_stored":287274436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":118618571},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.52300000000000002,"15min":0.629},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67900000000000005,"5min":0.91400000000000003,"15min":2.4489999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54600000000000004,"15min":0.64700000000000002},"min":{"1min":0.378,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80200000000000005,"5min":0.95399999999999996,"15min":11.77},"last":0.40999999999999998}]},{"osd":1,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57699999999999996,"15min":0.65300000000000002},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73699999999999999,"5min":0.86199999999999999,"15min":4.5279999999999996},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58499999999999996,"15min":0.68400000000000005},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.84599999999999997,"5min":1.024,"15min":11.971},"last":0.47099999999999997}]},{"osd":2,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61599999999999999,"15min":0.67200000000000004},"min":{"1min":0.41999999999999998,"5min":0.40200000000000002,"15min":0.34300000000000003},"max":{"1min":0.81899999999999995,"5min":0.92200000000000004,"15min":7.1130000000000004},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.629,"15min":0.70999999999999996},"min":{"1min":0.51100000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.877,"5min":0.98299999999999998,"15min":9.4450000000000003},"last":0.63}]},{"osd":3,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.64900000000000002,"15min":0.68300000000000005},"min":{"1min":0.439,"5min":0.437,"15min":0.41299999999999998},"max":{"1min":0.78000000000000003,"5min":1,"15min":3.363},"last":0.438},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.68300000000000005,"15min":0.72699999999999998},"min":{"1min":0.50900000000000001,"5min":0.46899999999999997,"15min":0.42099999999999999},"max":{"1min":0.86299999999999999,"5min":1.0529999999999999,"15min":3.3980000000000001},"last":0.59799999999999998}]},{"osd":4,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56699999999999995,"15min":0.61399999999999999},"min":{"1min":0.36399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81899999999999995,"5min":1.091,"15min":12.532999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.623,"15min":0.628},"min":{"1min":0.47099999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90600000000000003,"5min":1.107,"15min":3.8700000000000001},"last":0.52700000000000002}]},{"osd":5,"last update":"Mon Oct 16 20:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60999999999999999,"15min":0.66300000000000003},"min":{"1min":0.41399999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.85699999999999998,"5min":1.042,"15min":2.2010000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.63600000000000001,"15min":0.73699999999999999},"min":{"1min":0.47499999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94399999999999995,"5min":1.0369999999999999,"15min":12.353},"last":0.56599999999999995}]},{"osd":6,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65100000000000002,"15min":0.746},"min":{"1min":0.40799999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93799999999999994,"5min":1.1299999999999999,"15min":11.929},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66900000000000004,"15min":0.71399999999999997},"min":{"1min":0.41799999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92900000000000005,"5min":1.0620000000000001,"15min":6.0359999999999996},"last":0.68200000000000005}]}]},{"osd":6,"up_from":15,"seq":64424509708,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":421948,"kb_used_data":300532,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93949892,"statfs":{"total":96636764160,"available":96204689408,"internally_reserved":0,"allocated":307744768,"data_stored":307357593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124320199},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.54900000000000004,"15min":0.61499999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27800000000000002},"max":{"1min":0.69499999999999995,"5min":0.93799999999999994,"15min":4.5419999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.56499999999999995,"15min":0.624},"min":{"1min":0.33200000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.76000000000000001,"5min":2.0329999999999999,"15min":4.9699999999999998},"last":0.81599999999999995}]},{"osd":1,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.624,"15min":0.64100000000000001},"min":{"1min":0.433,"5min":0.39200000000000002,"15min":0.377},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":2.8610000000000002},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67000000000000004,"15min":0.69599999999999995},"min":{"1min":0.41499999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.94399999999999995,"5min":1.014,"15min":1.0309999999999999},"last":0.69499999999999995}]},{"osd":2,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.59699999999999998,"15min":0.65000000000000002},"min":{"1min":0.36399999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86599999999999999,"5min":1.0289999999999999,"15min":1.895},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.65500000000000003,"15min":0.68500000000000005},"min":{"1min":0.40799999999999997,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78500000000000003,"5min":0.96099999999999997,"15min":2.3340000000000001},"last":0.56100000000000005}]},{"osd":3,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.755,"15min":0.72099999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.36599999999999999},"max":{"1min":0.86699999999999999,"5min":4.6340000000000003,"15min":4.6340000000000003},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.64100000000000001,"15min":0.69499999999999995},"min":{"1min":0.435,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.86099999999999999,"5min":1.163,"15min":5.1029999999999998},"last":0.497}]},{"osd":4,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55500000000000005,"15min":0.58999999999999997},"min":{"1min":0.36299999999999999,"5min":0.34799999999999998,"15min":0.30299999999999999},"max":{"1min":0.80300000000000005,"5min":1.0189999999999999,"15min":4.8959999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.63500000000000001,"15min":0.63400000000000001},"min":{"1min":0.374,"5min":0.313,"15min":0.313},"max":{"1min":0.996,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.59299999999999997}]},{"osd":5,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.59799999999999998,"15min":0.623},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.82199999999999995,"5min":1.05,"15min":2.3769999999999998},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.626,"15min":0.66200000000000003},"min":{"1min":0.38200000000000001,"5min":0.35999999999999999,"15min":0.34499999999999997},"max":{"1min":0.84499999999999997,"5min":1.0700000000000001,"15min":4.9379999999999997},"last":0.67600000000000005}]},{"osd":7,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.71199999999999997,"15min":0.70999999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.034,"5min":1.425,"15min":4.5789999999999997},"last":0.754},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.76800000000000002,"15min":0.748},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.39300000000000002},"max":{"1min":1.0600000000000001,"5min":1.6120000000000001,"15min":5.2460000000000004},"last":0.72499999999999998}]}]},{"osd":5,"up_from":15,"seq":64424509710,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324840,"kb_used_data":225632,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94047000,"statfs":{"total":96636764160,"available":96304128000,"internally_reserved":0,"allocated":231047168,"data_stored":230645947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":101579207},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.57199999999999995,"15min":0.61299999999999999},"min":{"1min":0.30499999999999999,"5min":0.28199999999999997,"15min":0.25700000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":2.4580000000000002},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.59299999999999997,"15min":0.65800000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.94999999999999996,"5min":1.1799999999999999,"15min":7.181},"last":0.502}]},{"osd":1,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":0.76800000000000002,"15min":0.69099999999999995},"min":{"1min":0.36699999999999999,"5min":0.33900000000000002,"15min":0.29899999999999999},"max":{"1min":16.655000000000001,"5min":16.655000000000001,"15min":16.655000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.5529999999999999,"5min":0.80100000000000005,"15min":0.71299999999999997},"min":{"1min":0.35099999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":16.289000000000001,"5min":16.289000000000001,"15min":16.289000000000001},"last":0.57999999999999996}]},{"osd":2,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.63200000000000001,"15min":0.67300000000000004},"min":{"1min":0.374,"5min":0.34599999999999997,"15min":0.308},"max":{"1min":1.1339999999999999,"5min":1.7110000000000001,"15min":3.3799999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67100000000000004,"15min":0.67700000000000005},"min":{"1min":0.42399999999999999,"5min":0.39800000000000002,"15min":0.38},"max":{"1min":1.1100000000000001,"5min":1.583,"15min":4.5369999999999999},"last":0.69199999999999995}]},{"osd":3,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.67400000000000004,"15min":0.69699999999999995},"min":{"1min":0.42399999999999999,"5min":0.39400000000000002,"15min":0.34899999999999998},"max":{"1min":1.0860000000000001,"5min":1.329,"15min":4.1070000000000002},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68600000000000005,"15min":0.69399999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.38},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":3.476},"last":0.74099999999999999}]},{"osd":4,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53100000000000003,"15min":0.55500000000000005},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.749,"5min":0.85099999999999998,"15min":2.415},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.59999999999999998,"15min":0.66300000000000003},"min":{"1min":0.34300000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.78900000000000003,"5min":1.155,"15min":3.2919999999999998},"last":0.81999999999999995}]},{"osd":6,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57299999999999995,"15min":0.59699999999999998},"min":{"1min":0.39000000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79400000000000004,"5min":0.93799999999999994,"15min":3.2400000000000002},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.31},"max":{"1min":0.71799999999999997,"5min":1.036,"15min":1.2310000000000001},"last":0.80200000000000005}]},{"osd":7,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.44600000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76300000000000001,"5min":1.23,"15min":5.2759999999999998},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69399999999999995,"15min":0.70699999999999996},"min":{"1min":0.45000000000000001,"5min":0.435,"15min":0.42299999999999999},"max":{"1min":0.83199999999999996,"5min":0.97699999999999998,"15min":3.3140000000000001},"last":0.91600000000000004}]}]},{"osd":4,"up_from":15,"seq":64424509708,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420124,"kb_used_data":290388,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93951716,"statfs":{"total":96636764160,"available":96206557184,"internally_reserved":0,"allocated":297357312,"data_stored":296972867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56100000000000005,"15min":0.61299999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.83799999999999997,"5min":0.91400000000000003,"15min":1.524},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61699999999999999,"15min":0.69199999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.5649999999999999},"last":0.98199999999999998}]},{"osd":1,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.376,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.1749999999999998,"5min":4.1749999999999998,"15min":4.1749999999999998},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.625,"15min":0.66600000000000004},"min":{"1min":0.38200000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82599999999999996,"5min":0.94899999999999995,"15min":1.1020000000000001},"last":1.04}]},{"osd":2,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.38900000000000001},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":2.3370000000000002},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.63,"5min":0.68300000000000005,"15min":0.71199999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.84499999999999997,"5min":1.2210000000000001,"15min":2.3660000000000001},"last":0.85799999999999998}]},{"osd":3,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.377},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":2.04},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":2.2240000000000002},"last":1.0109999999999999}]},{"osd":5,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.58799999999999997,"15min":0.60299999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.9970000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63600000000000001,"15min":0.60499999999999998},"min":{"1min":0.41299999999999998,"5min":0.39500000000000002,"15min":0.33900000000000002},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":1.113}]},{"osd":6,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.60299999999999998,"15min":0.61599999999999999},"min":{"1min":0.33900000000000002,"5min":0.315,"15min":0.312},"max":{"1min":1.2270000000000001,"5min":1.6339999999999999,"15min":2.2639999999999998},"last":1.151},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.66200000000000003,"15min":0.67300000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.8300000000000001},"last":1.056}]},{"osd":7,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.69399999999999995,"15min":0.66800000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2969999999999999,"5min":1.347,"15min":1.347},"last":1.272},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65500000000000003,"15min":0.67900000000000005},"min":{"1min":0.38600000000000001,"5min":0.35899999999999999,"15min":0.35599999999999998},"max":{"1min":1.266,"5min":1.266,"15min":2.2999999999999998},"last":1.123}]}]},{"osd":3,"up_from":15,"seq":64424509708,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485596,"kb_used_data":360020,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93886244,"statfs":{"total":96636764160,"available":96139513856,"internally_reserved":0,"allocated":368660480,"data_stored":368273200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":128580040},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.45200000000000001,"15min":0.433},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":0.76300000000000001,"5min":2.8109999999999999,"15min":4.5830000000000002},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.44700000000000001,"15min":0.44600000000000001},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.253},"max":{"1min":0.58199999999999996,"5min":1.147,"15min":1.2509999999999999},"last":0.57399999999999995}]},{"osd":1,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.49099999999999999,"15min":0.503},"min":{"1min":0.33500000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67000000000000004,"5min":1.411,"15min":4.6840000000000002},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.437,"5min":0.504,"15min":0.51000000000000001},"min":{"1min":0.29699999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61399999999999999,"5min":2.6360000000000001,"15min":2.7429999999999999},"last":0.63600000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.40000000000000002,"5min":0.34399999999999997,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":1.286,"15min":4.6449999999999996},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55700000000000005,"15min":0.51900000000000002},"min":{"1min":0.32200000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":1.1850000000000001,"15min":1.911},"last":0.80400000000000005}]},{"osd":4,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.72399999999999998,"15min":0.76200000000000001},"min":{"1min":0.52000000000000002,"5min":0.45400000000000001,"15min":0.44800000000000001},"max":{"1min":1.0620000000000001,"5min":1.538,"15min":1.7749999999999999},"last":1.1240000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.72399999999999998,"15min":0.74199999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.155,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":1.1499999999999999}]},{"osd":5,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.74199999999999999,"15min":0.755},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0780000000000001,"5min":1.4970000000000001,"15min":4.8449999999999998},"last":1.0369999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.74099999999999999,"15min":0.745},"min":{"1min":0.48199999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1220000000000001,"5min":2.948,"15min":2.948},"last":1.1319999999999999}]},{"osd":6,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.76300000000000001,"15min":0.76600000000000001},"min":{"1min":0.48899999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.98399999999999999,"5min":1.633,"15min":4.8099999999999996},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.45400000000000001},"max":{"1min":0.93899999999999995,"5min":1.6559999999999999,"15min":2.339},"last":1.0680000000000001}]},{"osd":7,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.79800000000000004,"15min":0.86499999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.46000000000000002},"max":{"1min":1.216,"5min":1.593,"15min":34.039000000000001},"last":1.1080000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.76900000000000002,"15min":0.77700000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0249999999999999,"5min":2.984,"15min":2.984},"last":1.0920000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509709,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303812,"kb_used_data":200444,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94068028,"statfs":{"total":96636764160,"available":96325660672,"internally_reserved":0,"allocated":205254656,"data_stored":204864239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42899999999999999,"15min":0.45000000000000001},"min":{"1min":0.27100000000000002,"5min":0.23400000000000001,"15min":0.20599999999999999},"max":{"1min":0.74299999999999999,"5min":0.92500000000000004,"15min":5.5499999999999998},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.52600000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.1180000000000001,"5min":1.1220000000000001,"15min":5.4160000000000004},"last":0.47499999999999998}]},{"osd":1,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.504,"15min":0.5},"min":{"1min":0.307,"5min":0.307,"15min":0.26000000000000001},"max":{"1min":0.81899999999999995,"5min":1.25,"15min":2.2349999999999999},"last":0.373},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.501},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.27200000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":5.4359999999999999},"last":0.40500000000000003}]},{"osd":3,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.28399999999999997,"5min":0.26400000000000001,"15min":0.25800000000000001},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.312,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.83099999999999996,"5min":1.286,"15min":5.3789999999999996},"last":0.49099999999999999}]},{"osd":4,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.73099999999999998,"15min":0.80000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":3.464,"5min":3.464,"15min":16.905999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.70699999999999996,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.44700000000000001,"15min":0.441},"max":{"1min":0.92100000000000004,"5min":1.3100000000000001,"15min":10.064},"last":0.68000000000000005}]},{"osd":5,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70999999999999996,"15min":0.72999999999999998},"min":{"1min":0.45300000000000001,"5min":0.41099999999999998,"15min":0.32800000000000001},"max":{"1min":1.0960000000000001,"5min":1.304,"15min":3.2989999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.73499999999999999,"15min":0.77100000000000002},"min":{"1min":0.52400000000000002,"5min":0.499,"15min":0.46700000000000003},"max":{"1min":1.0069999999999999,"5min":1.3640000000000001,"15min":5.3330000000000002},"last":0.748}]},{"osd":6,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.76500000000000001,"15min":0.755},"min":{"1min":0.438,"5min":0.438,"15min":0.42299999999999999},"max":{"1min":3.5710000000000002,"5min":3.5710000000000002,"15min":3.5710000000000002},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.77900000000000003,"15min":0.77000000000000002},"min":{"1min":0.49099999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":5.4749999999999996},"last":0.81100000000000005}]},{"osd":7,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.47399999999999998,"5min":0.47199999999999998,"15min":0.44500000000000001},"max":{"1min":3.5089999999999999,"5min":3.5089999999999999,"15min":5.516},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77200000000000002,"15min":0.79100000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.41899999999999998},"max":{"1min":1.325,"5min":1.325,"15min":8.4610000000000003},"last":0.86699999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509708,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490520,"kb_used_data":360784,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93881320,"statfs":{"total":96636764160,"available":96134471680,"internally_reserved":0,"allocated":369442816,"data_stored":369059669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.437,"15min":0.45700000000000002},"min":{"1min":0.245,"5min":0.219,"15min":0.21299999999999999},"max":{"1min":0.79800000000000004,"5min":0.93000000000000005,"15min":8.9339999999999993},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.34300000000000003,"5min":0.28999999999999998,"15min":0.23000000000000001},"max":{"1min":0.67200000000000004,"5min":1.6899999999999999,"15min":3.8809999999999998},"last":0.45200000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.502,"15min":0.46000000000000002},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.248},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.55900000000000005,"15min":0.55400000000000005},"min":{"1min":0.29699999999999999,"5min":0.29099999999999998,"15min":0.28499999999999998},"max":{"1min":0.88600000000000001,"5min":2.6739999999999999,"15min":9.2680000000000007},"last":0.42699999999999999}]},{"osd":3,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57499999999999996,"15min":0.54200000000000004},"min":{"1min":0.33000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":1.2030000000000001,"5min":1.9910000000000001,"15min":3.0539999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57999999999999996,"15min":0.57799999999999996},"min":{"1min":0.433,"5min":0.34200000000000003,"15min":0.313},"max":{"1min":0.84999999999999998,"5min":1.4119999999999999,"15min":6.9409999999999998},"last":0.49099999999999999}]},{"osd":4,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.68300000000000005,"15min":0.73199999999999998},"min":{"1min":0.34300000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.90400000000000003,"5min":1.962,"15min":5.6630000000000003},"last":0.502},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.72399999999999998,"15min":0.75900000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.151,"5min":1.6719999999999999,"15min":8.8960000000000008},"last":0.47499999999999998}]},{"osd":5,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.76000000000000001,"15min":0.77700000000000002},"min":{"1min":0.48399999999999999,"5min":0.47299999999999998,"15min":0.40300000000000002},"max":{"1min":1.242,"5min":1.242,"15min":6.6159999999999997},"last":0.623},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.77800000000000002,"15min":0.79500000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.1220000000000001,"5min":1.7130000000000001,"15min":5.7009999999999996},"last":0.65900000000000003}]},{"osd":6,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.73299999999999998,"15min":0.77300000000000002},"min":{"1min":0.47799999999999998,"5min":0.47299999999999998,"15min":0.39800000000000002},"max":{"1min":0.94199999999999995,"5min":2.0529999999999999,"15min":8.6039999999999992},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.77200000000000002,"15min":0.75},"min":{"1min":0.50700000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98199999999999998,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":0.53900000000000003}]},{"osd":7,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76300000000000001,"15min":0.77800000000000002},"min":{"1min":0.46899999999999997,"5min":0.45200000000000001,"15min":0.44800000000000001},"max":{"1min":1.1639999999999999,"5min":1.7470000000000001,"15min":6.6600000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.78700000000000003,"15min":0.80600000000000005},"min":{"1min":0.50700000000000001,"5min":0.46400000000000002,"15min":0.46200000000000002},"max":{"1min":1.1990000000000001,"5min":1.774,"15min":8.8100000000000005},"last":0.52900000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575117,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371104,"kb_used_data":259416,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94000736,"statfs":{"total":96636764160,"available":96256753664,"internally_reserved":0,"allocated":265641984,"data_stored":265247973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.39500000000000002,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.56299999999999994,"5min":1.3029999999999999,"15min":6.8170000000000002},"last":0.307},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.46100000000000002,"15min":0.47499999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.70199999999999996,"5min":1.591,"15min":6.2789999999999999},"last":0.372}]},{"osd":2,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.51800000000000002,"15min":0.495},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90100000000000002,"5min":1.169,"15min":3.1859999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.47699999999999998,"15min":0.46500000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.97499999999999998,"5min":1.266,"15min":1.47},"last":0.46200000000000002}]},{"osd":3,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56999999999999995,"15min":0.52100000000000002},"min":{"1min":0.40899999999999997,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.67900000000000005,"5min":1.423,"15min":1.8180000000000001},"last":0.438},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.35599999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.93300000000000005,"5min":1.4390000000000001,"15min":1.881},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.68799999999999994,"15min":0.80700000000000005},"min":{"1min":0.46400000000000002,"5min":0.45100000000000001,"15min":0.45000000000000001},"max":{"1min":1.0529999999999999,"5min":1.3169999999999999,"15min":18.41},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70399999999999996,"15min":0.81399999999999995},"min":{"1min":0.50800000000000001,"5min":0.46300000000000002,"15min":0.42399999999999999},"max":{"1min":1.097,"5min":1.409,"15min":18.998999999999999},"last":0.61799999999999999}]},{"osd":5,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73699999999999999,"15min":0.76100000000000001},"min":{"1min":0.52400000000000002,"5min":0.50900000000000001,"15min":0.42799999999999999},"max":{"1min":1.1299999999999999,"5min":1.3460000000000001,"15min":3.5499999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69999999999999996,"15min":0.73699999999999999},"min":{"1min":0.45900000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.996,"5min":1.4279999999999999,"15min":3.605},"last":0.52100000000000002}]},{"osd":6,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.73999999999999999,"15min":0.754},"min":{"1min":0.52200000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0289999999999999,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.76600000000000001,"15min":0.78200000000000003},"min":{"1min":0.502,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0840000000000001,"5min":2.367,"15min":2.367},"last":0.65200000000000002}]},{"osd":7,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.49099999999999999},"max":{"1min":1.042,"5min":1.395,"15min":4.7839999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73999999999999999,"15min":0.77000000000000002},"min":{"1min":0.57399999999999995,"5min":0.53200000000000003,"15min":0.45500000000000002},"max":{"1min":0.995,"5min":1.3759999999999999,"15min":2.0110000000000001},"last":0.67600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39337984,"data_stored":39306740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51634176,"data_stored":51610244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18219008,"data_stored":18187806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":47730688,"data_stored":47702623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56041472,"data_stored":56016242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":44449792,"data_stored":44407786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40529920,"data_stored":40501960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":60235776,"data_stored":60202707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":224890880,"data_stored":224890880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":316989440,"data_stored":316989440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186216448,"data_stored":186216448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":320110592,"data_stored":320110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":240496640,"data_stored":240496640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":185778176,"data_stored":185778176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266395648,"data_stored":266395648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":226021376,"data_stored":226021376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T20:58:50.975 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.976 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.976 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.976 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.977 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.977 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.977 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.978 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.978 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.978 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.979 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.979 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.980 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.980 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.980 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.981 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.981 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.981 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.982 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.982 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.983 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.983 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.983 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.984 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.984 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.984 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.985 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.985 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.985 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.986 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.986 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.986 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.987 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.987 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.987 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.988 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.988 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.989 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.989 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.990 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.990 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.990 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.991 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.991 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.991 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.992 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.992 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.993 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-16T20:36:44.575506+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=44, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.993 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.993 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.994 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.994 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.995 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.995 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.995 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.996 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.996 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.996 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.997 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.997 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.998 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.998 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.998 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.999 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.999 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:50.999 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.000 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.000 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.001 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.001 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.001 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.002 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.002 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.003 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.003 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.003 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.004 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.004 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.004 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.005 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.005 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.006 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.006 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.006 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.007 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.007 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.007 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.008 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.008 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.008 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.009 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.009 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.010 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.010 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.010 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.011 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.011 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.012 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.012 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.012 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.013 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.013 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.013 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.014 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.014 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.014 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.015 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.015 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.016 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.016 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.016 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.017 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.017 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.017 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.018 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.018 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.018 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.019 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.019 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.019 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.020 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.020 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.021 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.021 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.021 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.022 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.022 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.022 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.023 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:58:51.023 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-16T20:59:01.680 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:59:01.684 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:59:11.025 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T20:59:11.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.169+0000 7f345ab65640 1 Processor -- start 2023-10-16T20:59:11.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.169+0000 7f345ab65640 1 -- start start 2023-10-16T20:59:11.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.173+0000 7f345ab65640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f345406a970 0x7f3454071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:11.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.173+0000 7f345ab65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3454071e60 0x7f345415a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:11.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.173+0000 7f345ab65640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f345415b3c0 con 0x7f34541406d0 2023-10-16T20:59:11.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.173+0000 7f345ab65640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34540c2460 con 0x7f345406a970 2023-10-16T20:59:11.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.173+0000 7f345ab65640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345415b680 con 0x7f3454071e60 2023-10-16T20:59:11.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.173+0000 7f3458901640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f34541406d0 0x7f34540c1d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49164/0 (socket says 172.21.15.88:49164) 2023-10-16T20:59:11.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.173+0000 7f3458901640 1 -- 172.21.15.88:0/1176727040 learned_addr learned my addr 172.21.15.88:0/1176727040 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:59:11.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.173+0000 7f3459102640 1 --2- 172.21.15.88:0/1176727040 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3454071e60 0x7f345415a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:11.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.173+0000 7f344bfff640 1 --2- 172.21.15.88:0/1176727040 >> v2:172.21.15.191:3300/0 conn(0x7f345406a970 0x7f3454071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:11.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.173+0000 7f344b7fe640 1 -- 172.21.15.88:0/1176727040 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097879059 0 0) 0x7f345415b3c0 con 0x7f34541406d0 2023-10-16T20:59:11.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.173+0000 7f344b7fe640 1 -- 172.21.15.88:0/1176727040 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f343c003530 con 0x7f34541406d0 2023-10-16T20:59:11.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.173+0000 7f3459102640 1 -- 172.21.15.88:0/1176727040 >> v2:172.21.15.191:3300/0 conn(0x7f345406a970 msgr2=0x7f3454071920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:59:11.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.173+0000 7f3459102640 1 --2- 172.21.15.88:0/1176727040 >> v2:172.21.15.191:3300/0 conn(0x7f345406a970 0x7f3454071920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:11.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.173+0000 7f3459102640 1 -- 172.21.15.88:0/1176727040 >> v1:172.21.15.88:6789/0 conn(0x7f34541406d0 legacy=0x7f34540c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:11.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.173+0000 7f344b7fe640 1 -- 172.21.15.88:0/1176727040 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3608405352 0 0) 0x7f343c003530 con 0x7f34541406d0 2023-10-16T20:59:11.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.173+0000 7f3459102640 1 -- 172.21.15.88:0/1176727040 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3454157570 con 0x7f3454071e60 2023-10-16T20:59:11.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.177+0000 7f3459102640 1 --2- 172.21.15.88:0/1176727040 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3454071e60 0x7f345415a810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f344c009d90 tx=0x7f344c01c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f973fda47670fc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:11.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.177+0000 7f344b7fe640 1 -- 172.21.15.88:0/1176727040 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3450001680 con 0x7f3454071e60 2023-10-16T20:59:11.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.177+0000 7f344b7fe640 1 -- 172.21.15.88:0/1176727040 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344c0a6480 con 0x7f3454071e60 2023-10-16T20:59:11.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.177+0000 7f344b7fe640 1 -- 172.21.15.88:0/1176727040 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c0a6810 con 0x7f3454071e60 2023-10-16T20:59:11.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.177+0000 7f345ab65640 1 -- 172.21.15.88:0/1176727040 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3454071e60 msgr2=0x7f345415a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:11.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.177+0000 7f345ab65640 1 --2- 172.21.15.88:0/1176727040 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3454071e60 0x7f345415a810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f344c009d90 tx=0x7f344c01c870 comp rx=0 tx=0).stop 2023-10-16T20:59:11.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.177+0000 7f345ab65640 1 -- 172.21.15.88:0/1176727040 shutdown_connections 2023-10-16T20:59:11.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.177+0000 7f345ab65640 1 --2- 172.21.15.88:0/1176727040 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f3454071e60 0x7f345415a810 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:11.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.177+0000 7f345ab65640 1 --2- 172.21.15.88:0/1176727040 >> v2:172.21.15.191:3300/0 conn(0x7f345406a970 0x7f3454071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:11.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.177+0000 7f345ab65640 1 -- 172.21.15.88:0/1176727040 >> 172.21.15.88:0/1176727040 conn(0x7f34540cad50 msgr2=0x7f34540cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:11.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.177+0000 7f345ab65640 1 -- 172.21.15.88:0/1176727040 shutdown_connections 2023-10-16T20:59:11.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.177+0000 7f345ab65640 1 -- 172.21.15.88:0/1176727040 wait complete. 2023-10-16T20:59:11.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.177+0000 7f345ab65640 1 Processor -- start 2023-10-16T20:59:11.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.177+0000 7f345ab65640 1 -- start start 2023-10-16T20:59:11.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f345ab65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f345406a970 0x7f34541f5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:11.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f345ab65640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3454071e60 0x7f34541f1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:11.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f345ab65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34541406d0 0x7f3454156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:11.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f345ab65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34541f51a0 con 0x7f34541406d0 2023-10-16T20:59:11.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f345ab65640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3454155fa0 con 0x7f3454071e60 2023-10-16T20:59:11.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f345ab65640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3454156280 con 0x7f345406a970 2023-10-16T20:59:11.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f3459102640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34541406d0 0x7f3454156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:11.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f3459102640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34541406d0 0x7f3454156a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60708/0 (socket says 172.21.15.88:60708) 2023-10-16T20:59:11.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f3459102640 1 -- 172.21.15.88:0/4092181488 learned_addr learned my addr 172.21.15.88:0/4092181488 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:11.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f344bfff640 1 --2- 172.21.15.88:0/4092181488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3454071e60 0x7f34541f1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:11.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f3458901640 1 --2- 172.21.15.88:0/4092181488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f345406a970 0x7f34541f5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:11.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f3459102640 1 -- 172.21.15.88:0/4092181488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f345406a970 msgr2=0x7f34541f5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:11.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f3459102640 1 --2- 172.21.15.88:0/4092181488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f345406a970 0x7f34541f5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:11.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f3459102640 1 -- 172.21.15.88:0/4092181488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3454071e60 msgr2=0x7f34541f1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:11.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f3459102640 1 --2- 172.21.15.88:0/4092181488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3454071e60 0x7f34541f1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:11.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f3459102640 1 -- 172.21.15.88:0/4092181488 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344c009a70 con 0x7f34541406d0 2023-10-16T20:59:11.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f344bfff640 1 --2- 172.21.15.88:0/4092181488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3454071e60 0x7f34541f1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:59:11.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f3459102640 1 --2- 172.21.15.88:0/4092181488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34541406d0 0x7f3454156a40 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f344c009d90 tx=0x7f344c019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:11.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f34497fa640 1 -- 172.21.15.88:0/4092181488 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c0b6070 con 0x7f34541406d0 2023-10-16T20:59:11.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f34497fa640 1 -- 172.21.15.88:0/4092181488 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344c009450 con 0x7f34541406d0 2023-10-16T20:59:11.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f34497fa640 1 -- 172.21.15.88:0/4092181488 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c0a76a0 con 0x7f34541406d0 2023-10-16T20:59:11.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f345ab65640 1 -- 172.21.15.88:0/4092181488 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34541f4b50 con 0x7f34541406d0 2023-10-16T20:59:11.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f345ab65640 1 -- 172.21.15.88:0/4092181488 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3454156f80 con 0x7f34541406d0 2023-10-16T20:59:11.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.181+0000 7f345ab65640 1 -- 172.21.15.88:0/4092181488 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3454157200 con 0x7f34541406d0 2023-10-16T20:59:11.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.185+0000 7f34497fa640 1 -- 172.21.15.88:0/4092181488 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f344c0a7800 con 0x7f34541406d0 2023-10-16T20:59:11.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.185+0000 7f34497fa640 1 --2- 172.21.15.88:0/4092181488 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34400664d0 0x7f3440068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:11.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.185+0000 7f34497fa640 1 -- 172.21.15.88:0/4092181488 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f344c11e9c0 con 0x7f34541406d0 2023-10-16T20:59:11.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.185+0000 7f3458901640 1 --2- 172.21.15.88:0/4092181488 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34400664d0 0x7f3440068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:11.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.185+0000 7f3458901640 1 --2- 172.21.15.88:0/4092181488 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34400664d0 0x7f3440068990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3450001ba0 tx=0x7f3450009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:11.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.189+0000 7f34497fa640 1 -- 172.21.15.88:0/4092181488 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3454157200 con 0x7f34541406d0 2023-10-16T20:59:11.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.337+0000 7f345ab65640 1 -- 172.21.15.88:0/4092181488 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3454140ab0 con 0x7f34400664d0 2023-10-16T20:59:11.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.349+0000 7f34497fa640 1 -- 172.21.15.88:0/4092181488 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+372824 (secure 0 0 0) 0x7f34500058e0 con 0x7f34400664d0 2023-10-16T20:59:11.354 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:59:11.357 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-16T20:59:11.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.353+0000 7f345ab65640 1 -- 172.21.15.88:0/4092181488 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34400664d0 msgr2=0x7f3440068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:11.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.353+0000 7f345ab65640 1 --2- 172.21.15.88:0/4092181488 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34400664d0 0x7f3440068990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3450001ba0 tx=0x7f3450009040 comp rx=0 tx=0).stop 2023-10-16T20:59:11.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.353+0000 7f345ab65640 1 -- 172.21.15.88:0/4092181488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34541406d0 msgr2=0x7f3454156a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:11.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.353+0000 7f345ab65640 1 --2- 172.21.15.88:0/4092181488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34541406d0 0x7f3454156a40 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f344c009d90 tx=0x7f344c019610 comp rx=0 tx=0).stop 2023-10-16T20:59:11.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.353+0000 7f345ab65640 1 -- 172.21.15.88:0/4092181488 shutdown_connections 2023-10-16T20:59:11.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.353+0000 7f345ab65640 1 --2- 172.21.15.88:0/4092181488 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f34400664d0 0x7f3440068990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:11.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.353+0000 7f345ab65640 1 --2- 172.21.15.88:0/4092181488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34541406d0 0x7f3454156a40 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:11.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.353+0000 7f345ab65640 1 --2- 172.21.15.88:0/4092181488 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3454071e60 0x7f34541f1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:11.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.353+0000 7f345ab65640 1 --2- 172.21.15.88:0/4092181488 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f345406a970 0x7f34541f5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:11.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.353+0000 7f345ab65640 1 -- 172.21.15.88:0/4092181488 >> 172.21.15.88:0/4092181488 conn(0x7f34540cad50 msgr2=0x7f345413f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:11.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.353+0000 7f345ab65640 1 -- 172.21.15.88:0/4092181488 shutdown_connections 2023-10-16T20:59:11.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:11.353+0000 7f345ab65640 1 -- 172.21.15.88:0/4092181488 wait complete. 2023-10-16T20:59:11.375 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":735,"stamp":"2023-10-16T20:59:09.938449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1104159081,"num_objects":10494,"num_object_clones":0,"num_object_copies":41576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10494,"num_whiteouts":0,"num_read":31781,"num_read_kb":882609,"num_write":27534,"num_write_kb":1486054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":115,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":399398,"num_omap_keys":761,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34210,"ondisk_log_size":34210,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3215280,"kb_used_data":2278704,"kb_used_omap":12,"kb_used_meta":936499,"kb_avail":751759440,"statfs":{"total":773094113280,"available":769801666560,"internally_reserved":0,"allocated":2333392896,"data_stored":2330258308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":958975545},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":113663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":131300,"num_omap_keys":251,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003571"},"pg_stats":[{"pgid":"2.1f","version":"67'338","reported_seq":435,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:58.690133+0000","last_change":"2023-10-16T20:58:53.179067+0000","last_active":"2023-10-16T20:58:58.690133+0000","last_peered":"2023-10-16T20:58:58.690133+0000","last_clean":"2023-10-16T20:58:58.690133+0000","last_became_active":"2023-10-16T20:36:54.763457+0000","last_became_peered":"2023-10-16T20:36:54.763457+0000","last_unstale":"2023-10-16T20:58:58.690133+0000","last_undegraded":"2023-10-16T20:58:58.690133+0000","last_fullsized":"2023-10-16T20:58:58.690133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'334","last_scrub_stamp":"2023-10-16T20:58:53.178846+0000","last_deep_scrub":"67'334","last_deep_scrub_stamp":"2023-10-16T20:58:53.178846+0000","last_clean_scrub_stamp":"2023-10-16T20:58:53.178846+0000","objects_scrubbed":12,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:41:18.102491+0000","scrub_duration":0.029593793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9874237,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":42,"num_read_kb":9548,"num_write":328,"num_write_kb":9838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":24974,"num_omap_keys":48,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"67'107","reported_seq":2219,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:48.521033+0000","last_change":"2023-10-16T20:47:31.588044+0000","last_active":"2023-10-16T20:58:48.521033+0000","last_peered":"2023-10-16T20:58:48.521033+0000","last_clean":"2023-10-16T20:58:48.521033+0000","last_became_active":"2023-10-16T20:36:55.302519+0000","last_became_peered":"2023-10-16T20:36:55.302519+0000","last_unstale":"2023-10-16T20:58:48.521033+0000","last_undegraded":"2023-10-16T20:58:48.521033+0000","last_fullsized":"2023-10-16T20:58:48.521033+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007784165,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'672","reported_seq":3558,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:47.854072+0000","last_change":"2023-10-16T20:47:31.584576+0000","last_active":"2023-10-16T20:58:47.854072+0000","last_peered":"2023-10-16T20:58:47.854072+0000","last_clean":"2023-10-16T20:58:47.854072+0000","last_became_active":"2023-10-16T20:36:59.466306+0000","last_became_peered":"2023-10-16T20:36:59.466306+0000","last_unstale":"2023-10-16T20:58:47.854072+0000","last_undegraded":"2023-10-16T20:58:47.854072+0000","last_fullsized":"2023-10-16T20:58:47.854072+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073106899999999999,"stat_sum":{"num_bytes":36865036,"num_objects":299,"num_object_clones":0,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":299,"num_whiteouts":0,"num_read":779,"num_read_kb":25449,"num_write":671,"num_write_kb":56816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"67'207","reported_seq":279,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:48.520972+0000","last_change":"2023-10-16T20:36:54.759491+0000","last_active":"2023-10-16T20:58:48.520972+0000","last_peered":"2023-10-16T20:58:48.520972+0000","last_clean":"2023-10-16T20:58:48.520972+0000","last_became_active":"2023-10-16T20:36:54.759177+0000","last_became_peered":"2023-10-16T20:36:54.759177+0000","last_unstale":"2023-10-16T20:58:48.520972+0000","last_undegraded":"2023-10-16T20:58:48.520972+0000","last_fullsized":"2023-10-16T20:58:48.520972+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":12288,"num_write":199,"num_write_kb":8312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"67'107","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.889932+0000","last_change":"2023-10-16T20:47:31.488681+0000","last_active":"2023-10-16T20:58:37.889932+0000","last_peered":"2023-10-16T20:58:37.889932+0000","last_clean":"2023-10-16T20:58:37.889932+0000","last_became_active":"2023-10-16T20:36:55.291458+0000","last_became_peered":"2023-10-16T20:36:55.291458+0000","last_unstale":"2023-10-16T20:58:37.889932+0000","last_undegraded":"2023-10-16T20:58:37.889932+0000","last_fullsized":"2023-10-16T20:58:37.889932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087429459999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'682","reported_seq":3572,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:47.660250+0000","last_change":"2023-10-16T20:47:31.484150+0000","last_active":"2023-10-16T20:58:47.660250+0000","last_peered":"2023-10-16T20:58:47.660250+0000","last_clean":"2023-10-16T20:58:47.660250+0000","last_became_active":"2023-10-16T20:36:59.302968+0000","last_became_peered":"2023-10-16T20:36:59.302968+0000","last_unstale":"2023-10-16T20:58:47.660250+0000","last_undegraded":"2023-10-16T20:58:47.660250+0000","last_fullsized":"2023-10-16T20:58:47.660250+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027985639999999999,"stat_sum":{"num_bytes":24765378,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":783,"num_read_kb":25781,"num_write":678,"num_write_kb":40921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"67'340","reported_seq":438,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.686988+0000","last_change":"2023-10-16T20:36:54.762600+0000","last_active":"2023-10-16T20:58:38.686988+0000","last_peered":"2023-10-16T20:58:38.686988+0000","last_clean":"2023-10-16T20:58:38.686988+0000","last_became_active":"2023-10-16T20:36:54.762278+0000","last_became_peered":"2023-10-16T20:36:54.762278+0000","last_unstale":"2023-10-16T20:58:38.686988+0000","last_undegraded":"2023-10-16T20:58:38.686988+0000","last_fullsized":"2023-10-16T20:58:38.686988+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":85,"num_read_kb":16492,"num_write":374,"num_write_kb":12523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'96","reported_seq":2198,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.347585+0000","last_change":"2023-10-16T20:47:31.496372+0000","last_active":"2023-10-16T20:58:38.347585+0000","last_peered":"2023-10-16T20:58:38.347585+0000","last_clean":"2023-10-16T20:58:38.347585+0000","last_became_active":"2023-10-16T20:37:37.402987+0000","last_became_peered":"2023-10-16T20:37:37.402987+0000","last_unstale":"2023-10-16T20:58:38.347585+0000","last_undegraded":"2023-10-16T20:58:38.347585+0000","last_fullsized":"2023-10-16T20:58:38.347585+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082573900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"67'673","reported_seq":4225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.695810+0000","last_change":"2023-10-16T20:47:31.485701+0000","last_active":"2023-10-16T20:58:22.695810+0000","last_peered":"2023-10-16T20:58:22.695810+0000","last_clean":"2023-10-16T20:58:22.695810+0000","last_became_active":"2023-10-16T20:36:59.303805+0000","last_became_peered":"2023-10-16T20:36:59.303805+0000","last_unstale":"2023-10-16T20:58:22.695810+0000","last_undegraded":"2023-10-16T20:58:22.695810+0000","last_fullsized":"2023-10-16T20:58:22.695810+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041657500000000002,"stat_sum":{"num_bytes":49691093,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":1446,"num_read_kb":31388,"num_write":673,"num_write_kb":52938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"67'172","reported_seq":253,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:28.700466+0000","last_change":"2023-10-16T20:36:54.749849+0000","last_active":"2023-10-16T20:58:28.700466+0000","last_peered":"2023-10-16T20:58:28.700466+0000","last_clean":"2023-10-16T20:58:28.700466+0000","last_became_active":"2023-10-16T20:36:54.749316+0000","last_became_peered":"2023-10-16T20:36:54.749316+0000","last_unstale":"2023-10-16T20:58:28.700466+0000","last_undegraded":"2023-10-16T20:58:28.700466+0000","last_fullsized":"2023-10-16T20:58:28.700466+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":170,"num_write_kb":4198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"67'104","reported_seq":2213,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.891882+0000","last_change":"2023-10-16T20:47:31.592091+0000","last_active":"2023-10-16T20:58:37.891882+0000","last_peered":"2023-10-16T20:58:37.891882+0000","last_clean":"2023-10-16T20:58:37.891882+0000","last_became_active":"2023-10-16T20:36:55.302387+0000","last_became_peered":"2023-10-16T20:36:55.302387+0000","last_unstale":"2023-10-16T20:58:37.891882+0000","last_undegraded":"2023-10-16T20:58:37.891882+0000","last_fullsized":"2023-10-16T20:58:37.891882+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011585343,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'641","reported_seq":3840,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.066860+0000","last_change":"2023-10-16T20:47:31.486382+0000","last_active":"2023-10-16T20:58:23.066860+0000","last_peered":"2023-10-16T20:58:23.066860+0000","last_clean":"2023-10-16T20:58:23.066860+0000","last_became_active":"2023-10-16T20:36:59.305228+0000","last_became_peered":"2023-10-16T20:36:59.305228+0000","last_unstale":"2023-10-16T20:58:23.066860+0000","last_undegraded":"2023-10-16T20:58:23.066860+0000","last_fullsized":"2023-10-16T20:58:23.066860+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":641,"log_dups_size":0,"ondisk_log_size":641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047877249999999996,"stat_sum":{"num_bytes":34900728,"num_objects":310,"num_object_clones":0,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":1093,"num_read_kb":21879,"num_write":638,"num_write_kb":46703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"67'104","reported_seq":2215,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:42.659947+0000","last_change":"2023-10-16T20:47:31.592641+0000","last_active":"2023-10-16T20:58:42.659947+0000","last_peered":"2023-10-16T20:58:42.659947+0000","last_clean":"2023-10-16T20:58:42.659947+0000","last_became_active":"2023-10-16T20:36:55.306427+0000","last_became_peered":"2023-10-16T20:36:55.306427+0000","last_unstale":"2023-10-16T20:58:42.659947+0000","last_undegraded":"2023-10-16T20:58:42.659947+0000","last_fullsized":"2023-10-16T20:58:42.659947+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013854674000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"67'187","reported_seq":259,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.970965+0000","last_change":"2023-10-16T20:36:54.768375+0000","last_active":"2023-10-16T20:58:05.970965+0000","last_peered":"2023-10-16T20:58:05.970965+0000","last_clean":"2023-10-16T20:58:05.970965+0000","last_became_active":"2023-10-16T20:36:54.767996+0000","last_became_peered":"2023-10-16T20:36:54.767996+0000","last_unstale":"2023-10-16T20:58:05.970965+0000","last_undegraded":"2023-10-16T20:58:05.970965+0000","last_fullsized":"2023-10-16T20:58:05.970965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":8202,"num_write":173,"num_write_kb":8293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"67'648","reported_seq":4474,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.017793+0000","last_change":"2023-10-16T20:47:31.487925+0000","last_active":"2023-10-16T20:58:23.017793+0000","last_peered":"2023-10-16T20:58:23.017793+0000","last_clean":"2023-10-16T20:58:23.017793+0000","last_became_active":"2023-10-16T20:36:59.306640+0000","last_became_peered":"2023-10-16T20:36:59.306640+0000","last_unstale":"2023-10-16T20:58:23.017793+0000","last_undegraded":"2023-10-16T20:58:23.017793+0000","last_fullsized":"2023-10-16T20:58:23.017793+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054537470000000001,"stat_sum":{"num_bytes":41205181,"num_objects":318,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":1719,"num_read_kb":37802,"num_write":648,"num_write_kb":48776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"67'101","reported_seq":2211,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.020674+0000","last_change":"2023-10-16T20:47:31.587617+0000","last_active":"2023-10-16T20:58:38.020674+0000","last_peered":"2023-10-16T20:58:38.020674+0000","last_clean":"2023-10-16T20:58:38.020674+0000","last_became_active":"2023-10-16T20:36:55.275085+0000","last_became_peered":"2023-10-16T20:36:55.275085+0000","last_unstale":"2023-10-16T20:58:38.020674+0000","last_undegraded":"2023-10-16T20:58:38.020674+0000","last_fullsized":"2023-10-16T20:58:38.020674+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087599630000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'183","reported_seq":261,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:57.085209+0000","last_change":"2023-10-16T20:58:57.085283+0000","last_active":"2023-10-16T20:58:57.085209+0000","last_peered":"2023-10-16T20:58:57.085209+0000","last_clean":"2023-10-16T20:58:57.085209+0000","last_became_active":"2023-10-16T20:36:54.755872+0000","last_became_peered":"2023-10-16T20:36:54.755872+0000","last_unstale":"2023-10-16T20:58:57.085209+0000","last_undegraded":"2023-10-16T20:58:57.085209+0000","last_fullsized":"2023-10-16T20:58:57.085209+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'183","last_scrub_stamp":"2023-10-16T20:58:57.085118+0000","last_deep_scrub":"67'183","last_deep_scrub_stamp":"2023-10-16T20:58:57.085118+0000","last_clean_scrub_stamp":"2023-10-16T20:58:57.085118+0000","objects_scrubbed":3,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:40:14.300010+0000","scrub_duration":0.017757339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":10,"num_write":281,"num_write_kb":4257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"67'699","reported_seq":3455,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.001720+0000","last_change":"2023-10-16T20:47:31.486593+0000","last_active":"2023-10-16T20:58:23.001720+0000","last_peered":"2023-10-16T20:58:23.001720+0000","last_clean":"2023-10-16T20:58:23.001720+0000","last_became_active":"2023-10-16T20:36:59.302919+0000","last_became_peered":"2023-10-16T20:36:59.302919+0000","last_unstale":"2023-10-16T20:58:23.001720+0000","last_undegraded":"2023-10-16T20:58:23.001720+0000","last_fullsized":"2023-10-16T20:58:23.001720+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046546790000000001,"stat_sum":{"num_bytes":20793675,"num_objects":334,"num_object_clones":0,"num_object_copies":1336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":334,"num_whiteouts":0,"num_read":649,"num_read_kb":16394,"num_write":696,"num_write_kb":45235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"67'106","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.886671+0000","last_change":"2023-10-16T20:47:31.586816+0000","last_active":"2023-10-16T20:58:37.886671+0000","last_peered":"2023-10-16T20:58:37.886671+0000","last_clean":"2023-10-16T20:58:37.886671+0000","last_became_active":"2023-10-16T20:36:55.270455+0000","last_became_peered":"2023-10-16T20:36:55.270455+0000","last_unstale":"2023-10-16T20:58:37.886671+0000","last_undegraded":"2023-10-16T20:58:37.886671+0000","last_fullsized":"2023-10-16T20:58:37.886671+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.008577889,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"67'55","reported_seq":142,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:02.072253+0000","last_change":"2023-10-16T20:59:02.072347+0000","last_active":"2023-10-16T20:59:02.072253+0000","last_peered":"2023-10-16T20:59:02.072253+0000","last_clean":"2023-10-16T20:59:02.072253+0000","last_became_active":"2023-10-16T20:36:54.743805+0000","last_became_peered":"2023-10-16T20:36:54.743805+0000","last_unstale":"2023-10-16T20:59:02.072253+0000","last_undegraded":"2023-10-16T20:59:02.072253+0000","last_fullsized":"2023-10-16T20:59:02.072253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'55","last_scrub_stamp":"2023-10-16T20:59:02.072155+0000","last_deep_scrub":"67'55","last_deep_scrub_stamp":"2023-10-16T20:59:02.072155+0000","last_clean_scrub_stamp":"2023-10-16T20:59:02.072155+0000","objects_scrubbed":6,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:40.321154+0000","scrub_duration":0.011965646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":55,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":19110,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'651","reported_seq":3693,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.438877+0000","last_change":"2023-10-16T20:47:31.485640+0000","last_active":"2023-10-16T20:58:22.438877+0000","last_peered":"2023-10-16T20:58:22.438877+0000","last_clean":"2023-10-16T20:58:22.438877+0000","last_became_active":"2023-10-16T20:36:59.465172+0000","last_became_peered":"2023-10-16T20:36:59.465172+0000","last_unstale":"2023-10-16T20:58:22.438877+0000","last_undegraded":"2023-10-16T20:58:22.438877+0000","last_fullsized":"2023-10-16T20:58:22.438877+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040107390000000001,"stat_sum":{"num_bytes":25409516,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":935,"num_read_kb":27643,"num_write":650,"num_write_kb":33262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"67'103","reported_seq":2214,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.855988+0000","last_change":"2023-10-16T20:47:31.487014+0000","last_active":"2023-10-16T20:58:37.855988+0000","last_peered":"2023-10-16T20:58:37.855988+0000","last_clean":"2023-10-16T20:58:37.855988+0000","last_became_active":"2023-10-16T20:36:55.278568+0000","last_became_peered":"2023-10-16T20:36:55.278568+0000","last_unstale":"2023-10-16T20:58:37.855988+0000","last_undegraded":"2023-10-16T20:58:37.855988+0000","last_fullsized":"2023-10-16T20:58:37.855988+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0050399599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"67'293","reported_seq":373,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:55.917267+0000","last_change":"2023-10-16T20:58:55.917322+0000","last_active":"2023-10-16T20:58:55.917267+0000","last_peered":"2023-10-16T20:58:55.917267+0000","last_clean":"2023-10-16T20:58:55.917267+0000","last_became_active":"2023-10-16T20:36:54.758496+0000","last_became_peered":"2023-10-16T20:36:54.758496+0000","last_unstale":"2023-10-16T20:58:55.917267+0000","last_undegraded":"2023-10-16T20:58:55.917267+0000","last_fullsized":"2023-10-16T20:58:55.917267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'293","last_scrub_stamp":"2023-10-16T20:58:55.917187+0000","last_deep_scrub":"67'293","last_deep_scrub_stamp":"2023-10-16T20:58:55.917187+0000","last_clean_scrub_stamp":"2023-10-16T20:58:55.917187+0000","objects_scrubbed":6,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:17:21.018502+0000","scrub_duration":0.017810975999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":12294,"num_write":291,"num_write_kb":4310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":34642,"num_omap_keys":66,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"67'586","reported_seq":3862,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.663154+0000","last_change":"2023-10-16T20:47:31.488200+0000","last_active":"2023-10-16T20:58:22.663154+0000","last_peered":"2023-10-16T20:58:22.663154+0000","last_clean":"2023-10-16T20:58:22.663154+0000","last_became_active":"2023-10-16T20:36:59.309068+0000","last_became_peered":"2023-10-16T20:36:59.309068+0000","last_unstale":"2023-10-16T20:58:22.663154+0000","last_undegraded":"2023-10-16T20:58:22.663154+0000","last_fullsized":"2023-10-16T20:58:22.663154+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055482439999999999,"stat_sum":{"num_bytes":36900597,"num_objects":303,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1169,"num_read_kb":30992,"num_write":585,"num_write_kb":40444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'101","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:39.656982+0000","last_change":"2023-10-16T20:47:31.492407+0000","last_active":"2023-10-16T20:58:39.656982+0000","last_peered":"2023-10-16T20:58:39.656982+0000","last_clean":"2023-10-16T20:58:39.656982+0000","last_became_active":"2023-10-16T20:36:55.272840+0000","last_became_peered":"2023-10-16T20:36:55.272840+0000","last_unstale":"2023-10-16T20:58:39.656982+0000","last_undegraded":"2023-10-16T20:58:39.656982+0000","last_fullsized":"2023-10-16T20:58:39.656982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011897911000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"67'12","reported_seq":101,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:51.472708+0000","last_change":"2023-10-16T20:58:51.472802+0000","last_active":"2023-10-16T20:58:51.472708+0000","last_peered":"2023-10-16T20:58:51.472708+0000","last_clean":"2023-10-16T20:58:51.472708+0000","last_became_active":"2023-10-16T20:36:54.760397+0000","last_became_peered":"2023-10-16T20:36:54.760397+0000","last_unstale":"2023-10-16T20:58:51.472708+0000","last_undegraded":"2023-10-16T20:58:51.472708+0000","last_fullsized":"2023-10-16T20:58:51.472708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2023-10-16T20:58:51.472601+0000","last_deep_scrub":"67'12","last_deep_scrub_stamp":"2023-10-16T20:58:51.472601+0000","last_clean_scrub_stamp":"2023-10-16T20:58:51.472601+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:36.331637+0000","scrub_duration":0.0080542610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"67'710","reported_seq":3799,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.048832+0000","last_change":"2023-10-16T20:47:31.485980+0000","last_active":"2023-10-16T20:58:23.048832+0000","last_peered":"2023-10-16T20:58:23.048832+0000","last_clean":"2023-10-16T20:58:23.048832+0000","last_became_active":"2023-10-16T20:36:59.465152+0000","last_became_peered":"2023-10-16T20:36:59.465152+0000","last_unstale":"2023-10-16T20:58:23.048832+0000","last_undegraded":"2023-10-16T20:58:23.048832+0000","last_fullsized":"2023-10-16T20:58:23.048832+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044307399999999999,"stat_sum":{"num_bytes":25809469,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":981,"num_read_kb":15497,"num_write":706,"num_write_kb":50141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'106","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.882833+0000","last_change":"2023-10-16T20:47:31.587090+0000","last_active":"2023-10-16T20:58:37.882833+0000","last_peered":"2023-10-16T20:58:37.882833+0000","last_clean":"2023-10-16T20:58:37.882833+0000","last_became_active":"2023-10-16T20:36:55.285566+0000","last_became_peered":"2023-10-16T20:36:55.285566+0000","last_unstale":"2023-10-16T20:58:37.882833+0000","last_undegraded":"2023-10-16T20:58:37.882833+0000","last_fullsized":"2023-10-16T20:58:37.882833+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085500479999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'104","reported_seq":189,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:01.068372+0000","last_change":"2023-10-16T20:59:01.068434+0000","last_active":"2023-10-16T20:59:01.068372+0000","last_peered":"2023-10-16T20:59:01.068372+0000","last_clean":"2023-10-16T20:59:01.068372+0000","last_became_active":"2023-10-16T20:36:54.757796+0000","last_became_peered":"2023-10-16T20:36:54.757796+0000","last_unstale":"2023-10-16T20:59:01.068372+0000","last_undegraded":"2023-10-16T20:59:01.068372+0000","last_fullsized":"2023-10-16T20:59:01.068372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'104","last_scrub_stamp":"2023-10-16T20:59:01.068280+0000","last_deep_scrub":"67'104","last_deep_scrub_stamp":"2023-10-16T20:59:01.068280+0000","last_clean_scrub_stamp":"2023-10-16T20:59:01.068280+0000","objects_scrubbed":4,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:13:55.968912+0000","scrub_duration":0.035122747000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":254,"num_write_kb":1037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"67'686","reported_seq":3333,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:48.521168+0000","last_change":"2023-10-16T20:47:31.582989+0000","last_active":"2023-10-16T20:58:48.521168+0000","last_peered":"2023-10-16T20:58:48.521168+0000","last_clean":"2023-10-16T20:58:48.521168+0000","last_became_active":"2023-10-16T20:36:59.465221+0000","last_became_peered":"2023-10-16T20:36:59.465221+0000","last_unstale":"2023-10-16T20:58:48.521168+0000","last_undegraded":"2023-10-16T20:58:48.521168+0000","last_fullsized":"2023-10-16T20:58:48.521168+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059334330000000001,"stat_sum":{"num_bytes":20326999,"num_objects":343,"num_object_clones":0,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":538,"num_read_kb":13756,"num_write":685,"num_write_kb":32496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"67'101","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.818951+0000","last_change":"2023-10-16T20:47:31.489294+0000","last_active":"2023-10-16T20:58:37.818951+0000","last_peered":"2023-10-16T20:58:37.818951+0000","last_clean":"2023-10-16T20:58:37.818951+0000","last_became_active":"2023-10-16T20:36:55.274432+0000","last_became_peered":"2023-10-16T20:36:55.274432+0000","last_unstale":"2023-10-16T20:58:37.818951+0000","last_undegraded":"2023-10-16T20:58:37.818951+0000","last_fullsized":"2023-10-16T20:58:37.818951+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057461259999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"66'34","reported_seq":110,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:05.657899+0000","last_change":"2023-10-16T20:36:54.762356+0000","last_active":"2023-10-16T20:58:05.657899+0000","last_peered":"2023-10-16T20:58:05.657899+0000","last_clean":"2023-10-16T20:58:05.657899+0000","last_became_active":"2023-10-16T20:36:54.762167+0000","last_became_peered":"2023-10-16T20:36:54.762167+0000","last_unstale":"2023-10-16T20:58:05.657899+0000","last_undegraded":"2023-10-16T20:58:05.657899+0000","last_fullsized":"2023-10-16T20:58:05.657899+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":34,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"67'725","reported_seq":3716,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.045470+0000","last_change":"2023-10-16T20:47:31.489099+0000","last_active":"2023-10-16T20:58:23.045470+0000","last_peered":"2023-10-16T20:58:23.045470+0000","last_clean":"2023-10-16T20:58:23.045470+0000","last_became_active":"2023-10-16T20:36:59.466876+0000","last_became_peered":"2023-10-16T20:36:59.466876+0000","last_unstale":"2023-10-16T20:58:23.045470+0000","last_undegraded":"2023-10-16T20:58:23.045470+0000","last_fullsized":"2023-10-16T20:58:23.045470+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":725,"log_dups_size":0,"ondisk_log_size":725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009058969,"stat_sum":{"num_bytes":44901233,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":884,"num_read_kb":20657,"num_write":720,"num_write_kb":60576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"67'98","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.896340+0000","last_change":"2023-10-16T20:47:31.594422+0000","last_active":"2023-10-16T20:58:37.896340+0000","last_peered":"2023-10-16T20:58:37.896340+0000","last_clean":"2023-10-16T20:58:37.896340+0000","last_became_active":"2023-10-16T20:36:55.293228+0000","last_became_peered":"2023-10-16T20:36:55.293228+0000","last_unstale":"2023-10-16T20:58:37.896340+0000","last_undegraded":"2023-10-16T20:58:37.896340+0000","last_fullsized":"2023-10-16T20:58:37.896340+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013567460999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"67'212","reported_seq":289,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:59.097576+0000","last_change":"2023-10-16T20:58:59.097630+0000","last_active":"2023-10-16T20:58:59.097576+0000","last_peered":"2023-10-16T20:58:59.097576+0000","last_clean":"2023-10-16T20:58:59.097576+0000","last_became_active":"2023-10-16T20:36:54.757308+0000","last_became_peered":"2023-10-16T20:36:54.757308+0000","last_unstale":"2023-10-16T20:58:59.097576+0000","last_undegraded":"2023-10-16T20:58:59.097576+0000","last_fullsized":"2023-10-16T20:58:59.097576+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'212","last_scrub_stamp":"2023-10-16T20:58:59.097485+0000","last_deep_scrub":"67'212","last_deep_scrub_stamp":"2023-10-16T20:58:59.097485+0000","last_clean_scrub_stamp":"2023-10-16T20:58:59.097485+0000","objects_scrubbed":2,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:26:44.246482+0000","scrub_duration":0.017169976999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":3049,"num_write":207,"num_write_kb":4195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9334,"num_omap_keys":18,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"67'671","reported_seq":3426,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.059793+0000","last_change":"2023-10-16T20:47:31.485489+0000","last_active":"2023-10-16T20:58:23.059793+0000","last_peered":"2023-10-16T20:58:23.059793+0000","last_clean":"2023-10-16T20:58:23.059793+0000","last_became_active":"2023-10-16T20:36:59.303224+0000","last_became_peered":"2023-10-16T20:36:59.303224+0000","last_unstale":"2023-10-16T20:58:23.059793+0000","last_undegraded":"2023-10-16T20:58:23.059793+0000","last_fullsized":"2023-10-16T20:58:23.059793+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038422399999999998,"stat_sum":{"num_bytes":21817454,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":647,"num_read_kb":9306,"num_write":673,"num_write_kb":37999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"67'120","reported_seq":2239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:58.334392+0000","last_change":"2023-10-16T20:58:58.334443+0000","last_active":"2023-10-16T20:58:58.334392+0000","last_peered":"2023-10-16T20:58:58.334392+0000","last_clean":"2023-10-16T20:58:58.334392+0000","last_became_active":"2023-10-16T20:36:55.277872+0000","last_became_peered":"2023-10-16T20:36:55.277872+0000","last_unstale":"2023-10-16T20:58:58.334392+0000","last_undegraded":"2023-10-16T20:58:58.334392+0000","last_fullsized":"2023-10-16T20:58:58.334392+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'120","last_scrub_stamp":"2023-10-16T20:58:58.334264+0000","last_deep_scrub":"67'120","last_deep_scrub_stamp":"2023-10-16T20:58:58.334264+0000","last_clean_scrub_stamp":"2023-10-16T20:58:58.334264+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:35:22.356584+0000","scrub_duration":0.0023669569999999998,"objects_trimmed":0,"snaptrim_duration":0.0052323379999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"67'537","reported_seq":630,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:54.485905+0000","last_change":"2023-10-16T20:58:54.485966+0000","last_active":"2023-10-16T20:58:54.485905+0000","last_peered":"2023-10-16T20:58:54.485905+0000","last_clean":"2023-10-16T20:58:54.485905+0000","last_became_active":"2023-10-16T20:36:54.740340+0000","last_became_peered":"2023-10-16T20:36:54.740340+0000","last_unstale":"2023-10-16T20:58:54.485905+0000","last_undegraded":"2023-10-16T20:58:54.485905+0000","last_fullsized":"2023-10-16T20:58:54.485905+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'537","last_scrub_stamp":"2023-10-16T20:58:54.485786+0000","last_deep_scrub":"67'537","last_deep_scrub_stamp":"2023-10-16T20:58:54.485786+0000","last_clean_scrub_stamp":"2023-10-16T20:58:54.485786+0000","objects_scrubbed":5,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:18.710024+0000","scrub_duration":0.040724903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21285,"num_write":524,"num_write_kb":16645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'623","reported_seq":3878,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.054189+0000","last_change":"2023-10-16T20:47:31.489650+0000","last_active":"2023-10-16T20:58:23.054189+0000","last_peered":"2023-10-16T20:58:23.054189+0000","last_clean":"2023-10-16T20:58:23.054189+0000","last_became_active":"2023-10-16T20:36:59.294548+0000","last_became_peered":"2023-10-16T20:36:59.294548+0000","last_unstale":"2023-10-16T20:58:23.054189+0000","last_undegraded":"2023-10-16T20:58:23.054189+0000","last_fullsized":"2023-10-16T20:58:23.054189+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062280529999999999,"stat_sum":{"num_bytes":29651112,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":1147,"num_read_kb":23932,"num_write":623,"num_write_kb":40690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"67'89","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:04.003786+0000","last_change":"2023-10-16T20:59:04.003844+0000","last_active":"2023-10-16T20:59:04.003786+0000","last_peered":"2023-10-16T20:59:04.003786+0000","last_clean":"2023-10-16T20:59:04.003786+0000","last_became_active":"2023-10-16T20:36:55.269749+0000","last_became_peered":"2023-10-16T20:36:55.269749+0000","last_unstale":"2023-10-16T20:59:04.003786+0000","last_undegraded":"2023-10-16T20:59:04.003786+0000","last_fullsized":"2023-10-16T20:59:04.003786+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'89","last_scrub_stamp":"2023-10-16T20:59:04.003679+0000","last_deep_scrub":"67'89","last_deep_scrub_stamp":"2023-10-16T20:59:04.003679+0000","last_clean_scrub_stamp":"2023-10-16T20:59:04.003679+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:45:42.918860+0000","scrub_duration":0.0021523760000000001,"objects_trimmed":0,"snaptrim_duration":0.003009004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'198","reported_seq":395,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:55.620844+0000","last_change":"2023-10-16T20:58:55.620915+0000","last_active":"2023-10-16T20:58:55.620844+0000","last_peered":"2023-10-16T20:58:55.620844+0000","last_clean":"2023-10-16T20:58:55.620844+0000","last_became_active":"2023-10-16T20:36:54.757549+0000","last_became_peered":"2023-10-16T20:36:54.757549+0000","last_unstale":"2023-10-16T20:58:55.620844+0000","last_undegraded":"2023-10-16T20:58:55.620844+0000","last_fullsized":"2023-10-16T20:58:55.620844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'198","last_scrub_stamp":"2023-10-16T20:58:55.620756+0000","last_deep_scrub":"67'198","last_deep_scrub_stamp":"2023-10-16T20:58:55.620756+0000","last_clean_scrub_stamp":"2023-10-16T20:58:55.620756+0000","objects_scrubbed":10,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:40.908940+0000","scrub_duration":0.012899747,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1410402,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":124,"num_read_kb":239,"num_write":186,"num_write_kb":1570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":50816,"num_omap_keys":97,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'647","reported_seq":3935,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:42.658770+0000","last_change":"2023-10-16T20:47:31.487517+0000","last_active":"2023-10-16T20:58:42.658770+0000","last_peered":"2023-10-16T20:58:42.658770+0000","last_clean":"2023-10-16T20:58:42.658770+0000","last_became_active":"2023-10-16T20:36:59.301693+0000","last_became_peered":"2023-10-16T20:36:59.301693+0000","last_unstale":"2023-10-16T20:58:42.658770+0000","last_undegraded":"2023-10-16T20:58:42.658770+0000","last_fullsized":"2023-10-16T20:58:42.658770+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056692870000000003,"stat_sum":{"num_bytes":29916858,"num_objects":298,"num_object_clones":0,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":298,"num_whiteouts":0,"num_read":1180,"num_read_kb":25235,"num_write":643,"num_write_kb":45892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"67'104","reported_seq":2223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:58.176470+0000","last_change":"2023-10-16T20:58:58.176549+0000","last_active":"2023-10-16T20:58:58.176470+0000","last_peered":"2023-10-16T20:58:58.176470+0000","last_clean":"2023-10-16T20:58:58.176470+0000","last_became_active":"2023-10-16T20:36:55.276649+0000","last_became_peered":"2023-10-16T20:36:55.276649+0000","last_unstale":"2023-10-16T20:58:58.176470+0000","last_undegraded":"2023-10-16T20:58:58.176470+0000","last_fullsized":"2023-10-16T20:58:58.176470+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'104","last_scrub_stamp":"2023-10-16T20:58:58.176362+0000","last_deep_scrub":"67'104","last_deep_scrub_stamp":"2023-10-16T20:58:58.176362+0000","last_clean_scrub_stamp":"2023-10-16T20:58:58.176362+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:39:24.099280+0000","scrub_duration":0.0024904129999999999,"objects_trimmed":0,"snaptrim_duration":0.0076292809999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'131","reported_seq":214,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:56.907025+0000","last_change":"2023-10-16T20:58:56.907025+0000","last_active":"2023-10-16T20:58:56.907025+0000","last_peered":"2023-10-16T20:58:56.907025+0000","last_clean":"2023-10-16T20:58:56.907025+0000","last_became_active":"2023-10-16T20:36:54.753237+0000","last_became_peered":"2023-10-16T20:36:54.753237+0000","last_unstale":"2023-10-16T20:58:56.907025+0000","last_undegraded":"2023-10-16T20:58:56.907025+0000","last_fullsized":"2023-10-16T20:58:56.907025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'131","last_scrub_stamp":"2023-10-16T20:58:56.906872+0000","last_deep_scrub":"67'131","last_deep_scrub_stamp":"2023-10-16T20:58:56.906872+0000","last_clean_scrub_stamp":"2023-10-16T20:58:56.906872+0000","objects_scrubbed":3,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:45.121322+0000","scrub_duration":0.023535402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":20486,"num_write":119,"num_write_kb":8247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'673","reported_seq":3484,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:39.656963+0000","last_change":"2023-10-16T20:47:31.491504+0000","last_active":"2023-10-16T20:58:39.656963+0000","last_peered":"2023-10-16T20:58:39.656963+0000","last_clean":"2023-10-16T20:58:39.656963+0000","last_became_active":"2023-10-16T20:36:59.466708+0000","last_became_peered":"2023-10-16T20:36:59.466708+0000","last_unstale":"2023-10-16T20:58:39.656963+0000","last_undegraded":"2023-10-16T20:58:39.656963+0000","last_fullsized":"2023-10-16T20:58:39.656963+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011259905000000001,"stat_sum":{"num_bytes":19802424,"num_objects":318,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":703,"num_read_kb":14564,"num_write":670,"num_write_kb":31986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"67'99","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:02.278130+0000","last_change":"2023-10-16T20:59:02.278178+0000","last_active":"2023-10-16T20:59:02.278130+0000","last_peered":"2023-10-16T20:59:02.278130+0000","last_clean":"2023-10-16T20:59:02.278130+0000","last_became_active":"2023-10-16T20:36:55.294610+0000","last_became_peered":"2023-10-16T20:36:55.294610+0000","last_unstale":"2023-10-16T20:59:02.278130+0000","last_undegraded":"2023-10-16T20:59:02.278130+0000","last_fullsized":"2023-10-16T20:59:02.278130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'99","last_scrub_stamp":"2023-10-16T20:59:02.278035+0000","last_deep_scrub":"67'99","last_deep_scrub_stamp":"2023-10-16T20:59:02.278035+0000","last_clean_scrub_stamp":"2023-10-16T20:59:02.278035+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:05.752220+0000","scrub_duration":0.0025758619999999999,"objects_trimmed":0,"snaptrim_duration":0.0075091560000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"67'12","reported_seq":97,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:55.494605+0000","last_change":"2023-10-16T20:58:55.494660+0000","last_active":"2023-10-16T20:58:55.494605+0000","last_peered":"2023-10-16T20:58:55.494605+0000","last_clean":"2023-10-16T20:58:55.494605+0000","last_became_active":"2023-10-16T20:36:54.740944+0000","last_became_peered":"2023-10-16T20:36:54.740944+0000","last_unstale":"2023-10-16T20:58:55.494605+0000","last_undegraded":"2023-10-16T20:58:55.494605+0000","last_fullsized":"2023-10-16T20:58:55.494605+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2023-10-16T20:58:55.494532+0000","last_deep_scrub":"67'12","last_deep_scrub_stamp":"2023-10-16T20:58:55.494532+0000","last_clean_scrub_stamp":"2023-10-16T20:58:55.494532+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:08:42.299214+0000","scrub_duration":0.0059841030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":23,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"67'119","reported_seq":231,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:54.954307+0000","last_change":"2023-10-16T20:58:54.954375+0000","last_active":"2023-10-16T20:58:54.954307+0000","last_peered":"2023-10-16T20:58:54.954307+0000","last_clean":"2023-10-16T20:58:54.954307+0000","last_became_active":"2023-10-16T20:36:46.388824+0000","last_became_peered":"2023-10-16T20:36:46.388824+0000","last_unstale":"2023-10-16T20:58:54.954307+0000","last_undegraded":"2023-10-16T20:58:54.954307+0000","last_fullsized":"2023-10-16T20:58:54.954307+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'119","last_scrub_stamp":"2023-10-16T20:58:54.954227+0000","last_deep_scrub":"67'119","last_deep_scrub_stamp":"2023-10-16T20:58:54.954227+0000","last_clean_scrub_stamp":"2023-10-16T20:58:54.954227+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:25:20.529327+0000","scrub_duration":0.0064233470000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"67'669","reported_seq":3426,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.684418+0000","last_change":"2023-10-16T20:47:31.593007+0000","last_active":"2023-10-16T20:58:22.684418+0000","last_peered":"2023-10-16T20:58:22.684418+0000","last_clean":"2023-10-16T20:58:22.684418+0000","last_became_active":"2023-10-16T20:36:59.468082+0000","last_became_peered":"2023-10-16T20:36:59.468082+0000","last_unstale":"2023-10-16T20:58:22.684418+0000","last_undegraded":"2023-10-16T20:58:22.684418+0000","last_fullsized":"2023-10-16T20:58:22.684418+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":669,"log_dups_size":0,"ondisk_log_size":669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013154572,"stat_sum":{"num_bytes":30910053,"num_objects":362,"num_object_clones":0,"num_object_copies":1448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":649,"num_read_kb":15567,"num_write":666,"num_write_kb":42817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'110","reported_seq":2213,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.943568+0000","last_change":"2023-10-16T20:47:31.488408+0000","last_active":"2023-10-16T20:58:37.943568+0000","last_peered":"2023-10-16T20:58:37.943568+0000","last_clean":"2023-10-16T20:58:37.943568+0000","last_became_active":"2023-10-16T20:37:37.409004+0000","last_became_peered":"2023-10-16T20:37:37.409004+0000","last_unstale":"2023-10-16T20:58:37.943568+0000","last_undegraded":"2023-10-16T20:58:37.943568+0000","last_fullsized":"2023-10-16T20:58:37.943568+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060805590000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"67'73","reported_seq":152,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.517624+0000","last_change":"2023-10-16T20:58:50.517695+0000","last_active":"2023-10-16T20:58:50.517624+0000","last_peered":"2023-10-16T20:58:50.517624+0000","last_clean":"2023-10-16T20:58:50.517624+0000","last_became_active":"2023-10-16T20:36:54.746247+0000","last_became_peered":"2023-10-16T20:36:54.746247+0000","last_unstale":"2023-10-16T20:58:50.517624+0000","last_undegraded":"2023-10-16T20:58:50.517624+0000","last_fullsized":"2023-10-16T20:58:50.517624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'73","last_scrub_stamp":"2023-10-16T20:58:50.517537+0000","last_deep_scrub":"67'73","last_deep_scrub_stamp":"2023-10-16T20:58:50.517537+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.517537+0000","objects_scrubbed":5,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:02:38.977902+0000","scrub_duration":0.021023528999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":7042,"num_write":68,"num_write_kb":4224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7644,"num_omap_keys":14,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"67'680","reported_seq":3681,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:47.660539+0000","last_change":"2023-10-16T20:47:31.590576+0000","last_active":"2023-10-16T20:58:47.660539+0000","last_peered":"2023-10-16T20:58:47.660539+0000","last_clean":"2023-10-16T20:58:47.660539+0000","last_became_active":"2023-10-16T20:37:37.467402+0000","last_became_peered":"2023-10-16T20:37:37.467402+0000","last_unstale":"2023-10-16T20:58:47.660539+0000","last_undegraded":"2023-10-16T20:58:47.660539+0000","last_fullsized":"2023-10-16T20:58:47.660539+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":680,"log_dups_size":0,"ondisk_log_size":680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012017773000000001,"stat_sum":{"num_bytes":36935626,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":884,"num_read_kb":21394,"num_write":679,"num_write_kb":40520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"67'106","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.896016+0000","last_change":"2023-10-16T20:47:31.589500+0000","last_active":"2023-10-16T20:58:37.896016+0000","last_peered":"2023-10-16T20:58:37.896016+0000","last_clean":"2023-10-16T20:58:37.896016+0000","last_became_active":"2023-10-16T20:36:55.300958+0000","last_became_peered":"2023-10-16T20:36:55.300958+0000","last_unstale":"2023-10-16T20:58:37.896016+0000","last_undegraded":"2023-10-16T20:58:37.896016+0000","last_fullsized":"2023-10-16T20:58:37.896016+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009734431,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"67'323","reported_seq":404,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:51.074256+0000","last_change":"2023-10-16T20:58:51.074256+0000","last_active":"2023-10-16T20:58:51.074256+0000","last_peered":"2023-10-16T20:58:51.074256+0000","last_clean":"2023-10-16T20:58:51.074256+0000","last_became_active":"2023-10-16T20:36:54.753443+0000","last_became_peered":"2023-10-16T20:36:54.753443+0000","last_unstale":"2023-10-16T20:58:51.074256+0000","last_undegraded":"2023-10-16T20:58:51.074256+0000","last_fullsized":"2023-10-16T20:58:51.074256+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'322","last_scrub_stamp":"2023-10-16T20:58:51.074122+0000","last_deep_scrub":"67'322","last_deep_scrub_stamp":"2023-10-16T20:58:51.074122+0000","last_clean_scrub_stamp":"2023-10-16T20:58:51.074122+0000","objects_scrubbed":7,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:03.275883+0000","scrub_duration":0.031482483999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9668880,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":15859,"num_write":310,"num_write_kb":9619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":30112,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'687","reported_seq":3977,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.067835+0000","last_change":"2023-10-16T20:47:31.492598+0000","last_active":"2023-10-16T20:58:23.067835+0000","last_peered":"2023-10-16T20:58:23.067835+0000","last_clean":"2023-10-16T20:58:23.067835+0000","last_became_active":"2023-10-16T20:36:59.302869+0000","last_became_peered":"2023-10-16T20:36:59.302869+0000","last_unstale":"2023-10-16T20:58:23.067835+0000","last_undegraded":"2023-10-16T20:58:23.067835+0000","last_fullsized":"2023-10-16T20:58:23.067835+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012136565,"stat_sum":{"num_bytes":23637013,"num_objects":352,"num_object_clones":0,"num_object_copies":1408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":1183,"num_read_kb":18772,"num_write":687,"num_write_kb":27489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"67'100","reported_seq":2210,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.893042+0000","last_change":"2023-10-16T20:47:31.587528+0000","last_active":"2023-10-16T20:58:37.893042+0000","last_peered":"2023-10-16T20:58:37.893042+0000","last_clean":"2023-10-16T20:58:37.893042+0000","last_became_active":"2023-10-16T20:36:55.291633+0000","last_became_peered":"2023-10-16T20:36:55.291633+0000","last_unstale":"2023-10-16T20:58:37.893042+0000","last_undegraded":"2023-10-16T20:58:37.893042+0000","last_fullsized":"2023-10-16T20:58:37.893042+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085978630000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'126","reported_seq":210,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:54.604075+0000","last_change":"2023-10-16T20:58:54.604136+0000","last_active":"2023-10-16T20:58:54.604075+0000","last_peered":"2023-10-16T20:58:54.604075+0000","last_clean":"2023-10-16T20:58:54.604075+0000","last_became_active":"2023-10-16T20:36:54.747704+0000","last_became_peered":"2023-10-16T20:36:54.747704+0000","last_unstale":"2023-10-16T20:58:54.604075+0000","last_undegraded":"2023-10-16T20:58:54.604075+0000","last_fullsized":"2023-10-16T20:58:54.604075+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'126","last_scrub_stamp":"2023-10-16T20:58:54.603981+0000","last_deep_scrub":"67'126","last_deep_scrub_stamp":"2023-10-16T20:58:54.603981+0000","last_clean_scrub_stamp":"2023-10-16T20:58:54.603981+0000","objects_scrubbed":6,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:18:13.372332+0000","scrub_duration":0.020068987999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":116,"num_write_kb":4228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20215,"num_omap_keys":38,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"67'663","reported_seq":3387,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:42.664360+0000","last_change":"2023-10-16T20:47:31.595065+0000","last_active":"2023-10-16T20:58:42.664360+0000","last_peered":"2023-10-16T20:58:42.664360+0000","last_clean":"2023-10-16T20:58:42.664360+0000","last_became_active":"2023-10-16T20:36:59.467291+0000","last_became_peered":"2023-10-16T20:36:59.467291+0000","last_unstale":"2023-10-16T20:58:42.664360+0000","last_undegraded":"2023-10-16T20:58:42.664360+0000","last_fullsized":"2023-10-16T20:58:42.664360+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0131915,"stat_sum":{"num_bytes":23136394,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":616,"num_read_kb":10065,"num_write":660,"num_write_kb":35181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"67'80","reported_seq":2190,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.882561+0000","last_change":"2023-10-16T20:47:31.589747+0000","last_active":"2023-10-16T20:58:37.882561+0000","last_peered":"2023-10-16T20:58:37.882561+0000","last_clean":"2023-10-16T20:58:37.882561+0000","last_became_active":"2023-10-16T20:36:55.290998+0000","last_became_peered":"2023-10-16T20:36:55.290998+0000","last_unstale":"2023-10-16T20:58:37.882561+0000","last_undegraded":"2023-10-16T20:58:37.882561+0000","last_fullsized":"2023-10-16T20:58:37.882561+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082191850000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'95","reported_seq":172,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.292497+0000","last_change":"2023-10-16T20:58:50.292551+0000","last_active":"2023-10-16T20:58:50.292497+0000","last_peered":"2023-10-16T20:58:50.292497+0000","last_clean":"2023-10-16T20:58:50.292497+0000","last_became_active":"2023-10-16T20:36:54.761337+0000","last_became_peered":"2023-10-16T20:36:54.761337+0000","last_unstale":"2023-10-16T20:58:50.292497+0000","last_undegraded":"2023-10-16T20:58:50.292497+0000","last_fullsized":"2023-10-16T20:58:50.292497+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'95","last_scrub_stamp":"2023-10-16T20:58:50.292415+0000","last_deep_scrub":"67'95","last_deep_scrub_stamp":"2023-10-16T20:58:50.292415+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.292415+0000","objects_scrubbed":5,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:51:30.957479+0000","scrub_duration":0.018399241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":5,"num_write":137,"num_write_kb":4352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18880,"num_omap_keys":36,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"67'666","reported_seq":3841,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:42.659648+0000","last_change":"2023-10-16T20:47:31.487063+0000","last_active":"2023-10-16T20:58:42.659648+0000","last_peered":"2023-10-16T20:58:42.659648+0000","last_clean":"2023-10-16T20:58:42.659648+0000","last_became_active":"2023-10-16T20:37:37.468309+0000","last_became_peered":"2023-10-16T20:37:37.468309+0000","last_unstale":"2023-10-16T20:58:42.659648+0000","last_undegraded":"2023-10-16T20:58:42.659648+0000","last_fullsized":"2023-10-16T20:58:42.659648+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058030460000000001,"stat_sum":{"num_bytes":20758781,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":1058,"num_read_kb":21621,"num_write":667,"num_write_kb":32853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"67'107","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.887254+0000","last_change":"2023-10-16T20:47:31.589398+0000","last_active":"2023-10-16T20:58:37.887254+0000","last_peered":"2023-10-16T20:58:37.887254+0000","last_clean":"2023-10-16T20:58:37.887254+0000","last_became_active":"2023-10-16T20:36:55.288354+0000","last_became_peered":"2023-10-16T20:36:55.288354+0000","last_unstale":"2023-10-16T20:58:37.887254+0000","last_undegraded":"2023-10-16T20:58:37.887254+0000","last_fullsized":"2023-10-16T20:58:37.887254+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076786709999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"67'348","reported_seq":437,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:57.706996+0000","last_change":"2023-10-16T20:58:57.707108+0000","last_active":"2023-10-16T20:58:57.706996+0000","last_peered":"2023-10-16T20:58:57.706996+0000","last_clean":"2023-10-16T20:58:57.706996+0000","last_became_active":"2023-10-16T20:36:54.737876+0000","last_became_peered":"2023-10-16T20:36:54.737876+0000","last_unstale":"2023-10-16T20:58:57.706996+0000","last_undegraded":"2023-10-16T20:58:57.706996+0000","last_fullsized":"2023-10-16T20:58:57.706996+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'348","last_scrub_stamp":"2023-10-16T20:58:57.706891+0000","last_deep_scrub":"67'348","last_deep_scrub_stamp":"2023-10-16T20:58:57.706891+0000","last_clean_scrub_stamp":"2023-10-16T20:58:57.706891+0000","objects_scrubbed":15,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:40:24.491212+0000","scrub_duration":0.033886146999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":39,"num_read_kb":4134,"num_write":333,"num_write_kb":8414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":43828,"num_omap_keys":84,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"67'666","reported_seq":3666,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:39.656922+0000","last_change":"2023-10-16T20:47:31.490637+0000","last_active":"2023-10-16T20:58:39.656922+0000","last_peered":"2023-10-16T20:58:39.656922+0000","last_clean":"2023-10-16T20:58:39.656922+0000","last_became_active":"2023-10-16T20:36:59.306801+0000","last_became_peered":"2023-10-16T20:36:59.306801+0000","last_unstale":"2023-10-16T20:58:39.656922+0000","last_undegraded":"2023-10-16T20:58:39.656922+0000","last_fullsized":"2023-10-16T20:58:39.656922+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010585667,"stat_sum":{"num_bytes":20444029,"num_objects":301,"num_object_clones":0,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":893,"num_read_kb":19660,"num_write":666,"num_write_kb":28474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"67'93","reported_seq":2211,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:06.354741+0000","last_change":"2023-10-16T20:59:06.354789+0000","last_active":"2023-10-16T20:59:06.354741+0000","last_peered":"2023-10-16T20:59:06.354741+0000","last_clean":"2023-10-16T20:59:06.354741+0000","last_became_active":"2023-10-16T20:36:55.287066+0000","last_became_peered":"2023-10-16T20:36:55.287066+0000","last_unstale":"2023-10-16T20:59:06.354741+0000","last_undegraded":"2023-10-16T20:59:06.354741+0000","last_fullsized":"2023-10-16T20:59:06.354741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'93","last_scrub_stamp":"2023-10-16T20:59:06.354644+0000","last_deep_scrub":"67'93","last_deep_scrub_stamp":"2023-10-16T20:59:06.354644+0000","last_clean_scrub_stamp":"2023-10-16T20:59:06.354644+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:18:40.827931+0000","scrub_duration":0.0027107630000000001,"objects_trimmed":0,"snaptrim_duration":0.012121162,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"67'105","reported_seq":185,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:55.485180+0000","last_change":"2023-10-16T20:58:55.485242+0000","last_active":"2023-10-16T20:58:55.485180+0000","last_peered":"2023-10-16T20:58:55.485180+0000","last_clean":"2023-10-16T20:58:55.485180+0000","last_became_active":"2023-10-16T20:36:54.764417+0000","last_became_peered":"2023-10-16T20:36:54.764417+0000","last_unstale":"2023-10-16T20:58:55.485180+0000","last_undegraded":"2023-10-16T20:58:55.485180+0000","last_fullsized":"2023-10-16T20:58:55.485180+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'105","last_scrub_stamp":"2023-10-16T20:58:55.485084+0000","last_deep_scrub":"67'105","last_deep_scrub_stamp":"2023-10-16T20:58:55.485084+0000","last_clean_scrub_stamp":"2023-10-16T20:58:55.485084+0000","objects_scrubbed":4,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:13:49.097422+0000","scrub_duration":0.015871666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":4110,"num_write":99,"num_write_kb":4200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'634","reported_seq":3756,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:47.854168+0000","last_change":"2023-10-16T20:47:31.584541+0000","last_active":"2023-10-16T20:58:47.854168+0000","last_peered":"2023-10-16T20:58:47.854168+0000","last_clean":"2023-10-16T20:58:47.854168+0000","last_became_active":"2023-10-16T20:36:59.464626+0000","last_became_peered":"2023-10-16T20:36:59.464626+0000","last_unstale":"2023-10-16T20:58:47.854168+0000","last_undegraded":"2023-10-16T20:58:47.854168+0000","last_fullsized":"2023-10-16T20:58:47.854168+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":634,"log_dups_size":0,"ondisk_log_size":634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069938420000000001,"stat_sum":{"num_bytes":28405361,"num_objects":321,"num_object_clones":0,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":1015,"num_read_kb":24481,"num_write":633,"num_write_kb":44456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"67'113","reported_seq":2223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.887053+0000","last_change":"2023-10-16T20:47:31.589722+0000","last_active":"2023-10-16T20:58:37.887053+0000","last_peered":"2023-10-16T20:58:37.887053+0000","last_clean":"2023-10-16T20:58:37.887053+0000","last_became_active":"2023-10-16T20:36:55.291396+0000","last_became_peered":"2023-10-16T20:36:55.291396+0000","last_unstale":"2023-10-16T20:58:37.887053+0000","last_undegraded":"2023-10-16T20:58:37.887053+0000","last_fullsized":"2023-10-16T20:58:37.887053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011464676,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"67'189","reported_seq":272,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:56.473518+0000","last_change":"2023-10-16T20:58:56.473573+0000","last_active":"2023-10-16T20:58:56.473518+0000","last_peered":"2023-10-16T20:58:56.473518+0000","last_clean":"2023-10-16T20:58:56.473518+0000","last_became_active":"2023-10-16T20:36:54.750448+0000","last_became_peered":"2023-10-16T20:36:54.750448+0000","last_unstale":"2023-10-16T20:58:56.473518+0000","last_undegraded":"2023-10-16T20:58:56.473518+0000","last_fullsized":"2023-10-16T20:58:56.473518+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'189","last_scrub_stamp":"2023-10-16T20:58:56.473428+0000","last_deep_scrub":"67'189","last_deep_scrub_stamp":"2023-10-16T20:58:56.473428+0000","last_clean_scrub_stamp":"2023-10-16T20:58:56.473428+0000","objects_scrubbed":5,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:18:30.642536+0000","scrub_duration":0.01623637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":3911,"num_write":180,"num_write_kb":4223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12239,"num_omap_keys":24,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"67'608","reported_seq":3331,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.255862+0000","last_change":"2023-10-16T20:47:31.583645+0000","last_active":"2023-10-16T20:58:23.255862+0000","last_peered":"2023-10-16T20:58:23.255862+0000","last_clean":"2023-10-16T20:58:23.255862+0000","last_became_active":"2023-10-16T20:36:59.464578+0000","last_became_peered":"2023-10-16T20:36:59.464578+0000","last_unstale":"2023-10-16T20:58:23.255862+0000","last_undegraded":"2023-10-16T20:58:23.255862+0000","last_fullsized":"2023-10-16T20:58:23.255862+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064826880000000003,"stat_sum":{"num_bytes":21354704,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":616,"num_read_kb":9691,"num_write":609,"num_write_kb":35926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"67'101","reported_seq":2219,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:56.192318+0000","last_change":"2023-10-16T20:58:56.192363+0000","last_active":"2023-10-16T20:58:56.192318+0000","last_peered":"2023-10-16T20:58:56.192318+0000","last_clean":"2023-10-16T20:58:56.192318+0000","last_became_active":"2023-10-16T20:36:55.285407+0000","last_became_peered":"2023-10-16T20:36:55.285407+0000","last_unstale":"2023-10-16T20:58:56.192318+0000","last_undegraded":"2023-10-16T20:58:56.192318+0000","last_fullsized":"2023-10-16T20:58:56.192318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'101","last_scrub_stamp":"2023-10-16T20:58:56.192235+0000","last_deep_scrub":"67'101","last_deep_scrub_stamp":"2023-10-16T20:58:56.192235+0000","last_clean_scrub_stamp":"2023-10-16T20:58:56.192235+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:39:41.514891+0000","scrub_duration":0.0021418700000000001,"objects_trimmed":0,"snaptrim_duration":0.010336254,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"67'469","reported_seq":623,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:52.147902+0000","last_change":"2023-10-16T20:58:52.147973+0000","last_active":"2023-10-16T20:58:52.147902+0000","last_peered":"2023-10-16T20:58:52.147902+0000","last_clean":"2023-10-16T20:58:52.147902+0000","last_became_active":"2023-10-16T20:36:54.756697+0000","last_became_peered":"2023-10-16T20:36:54.756697+0000","last_unstale":"2023-10-16T20:58:52.147902+0000","last_undegraded":"2023-10-16T20:58:52.147902+0000","last_fullsized":"2023-10-16T20:58:52.147902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'469","last_scrub_stamp":"2023-10-16T20:58:52.147814+0000","last_deep_scrub":"67'469","last_deep_scrub_stamp":"2023-10-16T20:58:52.147814+0000","last_clean_scrub_stamp":"2023-10-16T20:58:52.147814+0000","objects_scrubbed":10,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:59:37.271178+0000","scrub_duration":0.031542023000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8464867,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":106,"num_read_kb":16561,"num_write":461,"num_write_kb":8611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11094,"num_omap_keys":22,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'632","reported_seq":4048,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:47.665085+0000","last_change":"2023-10-16T20:47:31.589709+0000","last_active":"2023-10-16T20:58:47.665085+0000","last_peered":"2023-10-16T20:58:47.665085+0000","last_clean":"2023-10-16T20:58:47.665085+0000","last_became_active":"2023-10-16T20:36:59.468254+0000","last_became_peered":"2023-10-16T20:36:59.468254+0000","last_unstale":"2023-10-16T20:58:47.665085+0000","last_undegraded":"2023-10-16T20:58:47.665085+0000","last_fullsized":"2023-10-16T20:58:47.665085+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093892239999999998,"stat_sum":{"num_bytes":28874401,"num_objects":300,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":1309,"num_read_kb":15538,"num_write":630,"num_write_kb":28525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"67'100","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:53.035022+0000","last_change":"2023-10-16T20:58:53.035079+0000","last_active":"2023-10-16T20:58:53.035022+0000","last_peered":"2023-10-16T20:58:53.035022+0000","last_clean":"2023-10-16T20:58:53.035022+0000","last_became_active":"2023-10-16T20:36:55.288488+0000","last_became_peered":"2023-10-16T20:36:55.288488+0000","last_unstale":"2023-10-16T20:58:53.035022+0000","last_undegraded":"2023-10-16T20:58:53.035022+0000","last_fullsized":"2023-10-16T20:58:53.035022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'100","last_scrub_stamp":"2023-10-16T20:58:53.034921+0000","last_deep_scrub":"67'100","last_deep_scrub_stamp":"2023-10-16T20:58:53.034921+0000","last_clean_scrub_stamp":"2023-10-16T20:58:53.034921+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:10.663084+0000","scrub_duration":0.002664118,"objects_trimmed":0,"snaptrim_duration":0.0078405860000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"67'146","reported_seq":239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:01.469380+0000","last_change":"2023-10-16T20:59:01.469451+0000","last_active":"2023-10-16T20:59:01.469380+0000","last_peered":"2023-10-16T20:59:01.469380+0000","last_clean":"2023-10-16T20:59:01.469380+0000","last_became_active":"2023-10-16T20:36:54.764969+0000","last_became_peered":"2023-10-16T20:36:54.764969+0000","last_unstale":"2023-10-16T20:59:01.469380+0000","last_undegraded":"2023-10-16T20:59:01.469380+0000","last_fullsized":"2023-10-16T20:59:01.469380+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'146","last_scrub_stamp":"2023-10-16T20:59:01.469291+0000","last_deep_scrub":"67'146","last_deep_scrub_stamp":"2023-10-16T20:59:01.469291+0000","last_clean_scrub_stamp":"2023-10-16T20:59:01.469291+0000","objects_scrubbed":7,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:10.085576+0000","scrub_duration":0.028155902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389280,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":45,"num_read_kb":8235,"num_write":135,"num_write_kb":8275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"67'666","reported_seq":4059,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.063602+0000","last_change":"2023-10-16T20:47:31.488335+0000","last_active":"2023-10-16T20:58:23.063602+0000","last_peered":"2023-10-16T20:58:23.063602+0000","last_clean":"2023-10-16T20:58:23.063602+0000","last_became_active":"2023-10-16T20:36:59.302698+0000","last_became_peered":"2023-10-16T20:36:59.302698+0000","last_unstale":"2023-10-16T20:58:23.063602+0000","last_undegraded":"2023-10-16T20:58:23.063602+0000","last_fullsized":"2023-10-16T20:58:23.063602+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064797020000000004,"stat_sum":{"num_bytes":31392251,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":1286,"num_read_kb":27897,"num_write":665,"num_write_kb":43250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'99","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:59.336824+0000","last_change":"2023-10-16T20:58:59.336882+0000","last_active":"2023-10-16T20:58:59.336824+0000","last_peered":"2023-10-16T20:58:59.336824+0000","last_clean":"2023-10-16T20:58:59.336824+0000","last_became_active":"2023-10-16T20:36:55.290127+0000","last_became_peered":"2023-10-16T20:36:55.290127+0000","last_unstale":"2023-10-16T20:58:59.336824+0000","last_undegraded":"2023-10-16T20:58:59.336824+0000","last_fullsized":"2023-10-16T20:58:59.336824+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'99","last_scrub_stamp":"2023-10-16T20:58:59.336718+0000","last_deep_scrub":"67'99","last_deep_scrub_stamp":"2023-10-16T20:58:59.336718+0000","last_clean_scrub_stamp":"2023-10-16T20:58:59.336718+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:51:20.952065+0000","scrub_duration":0.0019832140000000001,"objects_trimmed":0,"snaptrim_duration":0.0055494209999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"67'84","reported_seq":165,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:00.071444+0000","last_change":"2023-10-16T20:59:00.071528+0000","last_active":"2023-10-16T20:59:00.071444+0000","last_peered":"2023-10-16T20:59:00.071444+0000","last_clean":"2023-10-16T20:59:00.071444+0000","last_became_active":"2023-10-16T20:36:54.748788+0000","last_became_peered":"2023-10-16T20:36:54.748788+0000","last_unstale":"2023-10-16T20:59:00.071444+0000","last_undegraded":"2023-10-16T20:59:00.071444+0000","last_fullsized":"2023-10-16T20:59:00.071444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'84","last_scrub_stamp":"2023-10-16T20:59:00.071354+0000","last_deep_scrub":"67'84","last_deep_scrub_stamp":"2023-10-16T20:59:00.071354+0000","last_clean_scrub_stamp":"2023-10-16T20:59:00.071354+0000","objects_scrubbed":6,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:33:58.876619+0000","scrub_duration":0.006295775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":15,"num_read_kb":23,"num_write":83,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9178,"num_omap_keys":18,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"67'606","reported_seq":3861,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.047896+0000","last_change":"2023-10-16T20:47:31.486033+0000","last_active":"2023-10-16T20:58:23.047896+0000","last_peered":"2023-10-16T20:58:23.047896+0000","last_clean":"2023-10-16T20:58:23.047896+0000","last_became_active":"2023-10-16T20:36:59.302916+0000","last_became_peered":"2023-10-16T20:36:59.302916+0000","last_unstale":"2023-10-16T20:58:23.047896+0000","last_undegraded":"2023-10-16T20:58:23.047896+0000","last_fullsized":"2023-10-16T20:58:23.047896+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041353309999999999,"stat_sum":{"num_bytes":29291517,"num_objects":300,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":1148,"num_read_kb":24111,"num_write":607,"num_write_kb":41174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"67'107","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:01.224253+0000","last_change":"2023-10-16T20:59:01.224305+0000","last_active":"2023-10-16T20:59:01.224253+0000","last_peered":"2023-10-16T20:59:01.224253+0000","last_clean":"2023-10-16T20:59:01.224253+0000","last_became_active":"2023-10-16T20:36:55.291448+0000","last_became_peered":"2023-10-16T20:36:55.291448+0000","last_unstale":"2023-10-16T20:59:01.224253+0000","last_undegraded":"2023-10-16T20:59:01.224253+0000","last_fullsized":"2023-10-16T20:59:01.224253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'107","last_scrub_stamp":"2023-10-16T20:59:01.224155+0000","last_deep_scrub":"67'107","last_deep_scrub_stamp":"2023-10-16T20:59:01.224155+0000","last_clean_scrub_stamp":"2023-10-16T20:59:01.224155+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:59:11.517988+0000","scrub_duration":0.0022855140000000002,"objects_trimmed":0,"snaptrim_duration":0.010585074,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"67'148","reported_seq":277,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:05.396217+0000","last_change":"2023-10-16T20:59:05.396279+0000","last_active":"2023-10-16T20:59:05.396217+0000","last_peered":"2023-10-16T20:59:05.396217+0000","last_clean":"2023-10-16T20:59:05.396217+0000","last_became_active":"2023-10-16T20:36:54.761541+0000","last_became_peered":"2023-10-16T20:36:54.761541+0000","last_unstale":"2023-10-16T20:59:05.396217+0000","last_undegraded":"2023-10-16T20:59:05.396217+0000","last_fullsized":"2023-10-16T20:59:05.396217+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'148","last_scrub_stamp":"2023-10-16T20:59:05.396099+0000","last_deep_scrub":"67'148","last_deep_scrub_stamp":"2023-10-16T20:59:05.396099+0000","last_clean_scrub_stamp":"2023-10-16T20:59:05.396099+0000","objects_scrubbed":7,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:44:03.980765+0000","scrub_duration":0.01804234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2736040,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":69,"num_read_kb":129,"num_write":143,"num_write_kb":2834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":19476,"num_omap_keys":37,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"67'663","reported_seq":3326,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.288364+0000","last_change":"2023-10-16T20:47:31.585744+0000","last_active":"2023-10-16T20:58:22.288364+0000","last_peered":"2023-10-16T20:58:22.288364+0000","last_clean":"2023-10-16T20:58:22.288364+0000","last_became_active":"2023-10-16T20:36:59.303773+0000","last_became_peered":"2023-10-16T20:36:59.303773+0000","last_unstale":"2023-10-16T20:58:22.288364+0000","last_undegraded":"2023-10-16T20:58:22.288364+0000","last_fullsized":"2023-10-16T20:58:22.288364+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007941142,"stat_sum":{"num_bytes":21664408,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":557,"num_read_kb":9585,"num_write":658,"num_write_kb":32916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"67'121","reported_seq":2231,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:42.664503+0000","last_change":"2023-10-16T20:47:31.594424+0000","last_active":"2023-10-16T20:58:42.664503+0000","last_peered":"2023-10-16T20:58:42.664503+0000","last_clean":"2023-10-16T20:58:42.664503+0000","last_became_active":"2023-10-16T20:36:55.300535+0000","last_became_peered":"2023-10-16T20:36:55.300535+0000","last_unstale":"2023-10-16T20:58:42.664503+0000","last_undegraded":"2023-10-16T20:58:42.664503+0000","last_fullsized":"2023-10-16T20:58:42.664503+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013755641000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"67'61","reported_seq":139,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:52.484963+0000","last_change":"2023-10-16T20:58:52.484963+0000","last_active":"2023-10-16T20:58:52.484963+0000","last_peered":"2023-10-16T20:58:52.484963+0000","last_clean":"2023-10-16T20:58:52.484963+0000","last_became_active":"2023-10-16T20:36:54.757303+0000","last_became_peered":"2023-10-16T20:36:54.757303+0000","last_unstale":"2023-10-16T20:58:52.484963+0000","last_undegraded":"2023-10-16T20:58:52.484963+0000","last_fullsized":"2023-10-16T20:58:52.484963+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'60","last_scrub_stamp":"2023-10-16T20:58:52.484737+0000","last_deep_scrub":"67'60","last_deep_scrub_stamp":"2023-10-16T20:58:52.484737+0000","last_clean_scrub_stamp":"2023-10-16T20:58:52.484737+0000","objects_scrubbed":6,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:17:16.465796+0000","scrub_duration":0.0082689740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":59,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":40464,"num_omap_keys":76,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"67'687","reported_seq":4218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.665131+0000","last_change":"2023-10-16T20:47:31.586327+0000","last_active":"2023-10-16T20:58:22.665131+0000","last_peered":"2023-10-16T20:58:22.665131+0000","last_clean":"2023-10-16T20:58:22.665131+0000","last_became_active":"2023-10-16T20:36:59.465844+0000","last_became_peered":"2023-10-16T20:36:59.465844+0000","last_unstale":"2023-10-16T20:58:22.665131+0000","last_undegraded":"2023-10-16T20:58:22.665131+0000","last_fullsized":"2023-10-16T20:58:22.665131+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081168009999999999,"stat_sum":{"num_bytes":45881583,"num_objects":353,"num_object_clones":0,"num_object_copies":1412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":353,"num_whiteouts":0,"num_read":1424,"num_read_kb":44764,"num_write":688,"num_write_kb":65580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"67'661","reported_seq":4049,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:46.761134+0000","last_change":"2023-10-16T20:47:31.489178+0000","last_active":"2023-10-16T20:58:46.761134+0000","last_peered":"2023-10-16T20:58:46.761134+0000","last_clean":"2023-10-16T20:58:46.761134+0000","last_became_active":"2023-10-16T20:36:59.467356+0000","last_became_peered":"2023-10-16T20:36:59.467356+0000","last_unstale":"2023-10-16T20:58:46.761134+0000","last_undegraded":"2023-10-16T20:58:46.761134+0000","last_fullsized":"2023-10-16T20:58:46.761134+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093722700000000003,"stat_sum":{"num_bytes":37038710,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1279,"num_read_kb":28276,"num_write":659,"num_write_kb":40616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'108","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:47.276397+0000","last_change":"2023-10-16T20:47:31.484094+0000","last_active":"2023-10-16T20:58:47.276397+0000","last_peered":"2023-10-16T20:58:47.276397+0000","last_clean":"2023-10-16T20:58:47.276397+0000","last_became_active":"2023-10-16T20:36:55.280903+0000","last_became_peered":"2023-10-16T20:36:55.280903+0000","last_unstale":"2023-10-16T20:58:47.276397+0000","last_undegraded":"2023-10-16T20:58:47.276397+0000","last_fullsized":"2023-10-16T20:58:47.276397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029269690000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"67'463","reported_seq":543,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:04.451251+0000","last_change":"2023-10-16T20:59:04.451310+0000","last_active":"2023-10-16T20:59:04.451251+0000","last_peered":"2023-10-16T20:59:04.451251+0000","last_clean":"2023-10-16T20:59:04.451251+0000","last_became_active":"2023-10-16T20:36:54.756775+0000","last_became_peered":"2023-10-16T20:36:54.756775+0000","last_unstale":"2023-10-16T20:59:04.451251+0000","last_undegraded":"2023-10-16T20:59:04.451251+0000","last_fullsized":"2023-10-16T20:59:04.451251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'463","last_scrub_stamp":"2023-10-16T20:59:04.451169+0000","last_deep_scrub":"67'463","last_deep_scrub_stamp":"2023-10-16T20:59:04.451169+0000","last_clean_scrub_stamp":"2023-10-16T20:59:04.451169+0000","objects_scrubbed":6,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:16.153884+0000","scrub_duration":0.039845845999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583494,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":16386,"num_write":448,"num_write_kb":12544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20150,"num_omap_keys":39,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"67'638","reported_seq":3560,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.627190+0000","last_change":"2023-10-16T20:47:31.593596+0000","last_active":"2023-10-16T20:58:22.627190+0000","last_peered":"2023-10-16T20:58:22.627190+0000","last_clean":"2023-10-16T20:58:22.627190+0000","last_became_active":"2023-10-16T20:37:37.438140+0000","last_became_peered":"2023-10-16T20:37:37.438140+0000","last_unstale":"2023-10-16T20:58:22.627190+0000","last_undegraded":"2023-10-16T20:58:22.627190+0000","last_fullsized":"2023-10-16T20:58:22.627190+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012287041,"stat_sum":{"num_bytes":24792014,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":805,"num_read_kb":12206,"num_write":635,"num_write_kb":32676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"67'121","reported_seq":2231,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:42.853484+0000","last_change":"2023-10-16T20:47:31.585618+0000","last_active":"2023-10-16T20:58:42.853484+0000","last_peered":"2023-10-16T20:58:42.853484+0000","last_clean":"2023-10-16T20:58:42.853484+0000","last_became_active":"2023-10-16T20:36:55.273959+0000","last_became_peered":"2023-10-16T20:36:55.273959+0000","last_unstale":"2023-10-16T20:58:42.853484+0000","last_undegraded":"2023-10-16T20:58:42.853484+0000","last_fullsized":"2023-10-16T20:58:42.853484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076333039999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"67'140","reported_seq":290,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:07.224275+0000","last_change":"2023-10-16T20:59:07.224354+0000","last_active":"2023-10-16T20:59:07.224275+0000","last_peered":"2023-10-16T20:59:07.224275+0000","last_clean":"2023-10-16T20:59:07.224275+0000","last_became_active":"2023-10-16T20:36:54.759059+0000","last_became_peered":"2023-10-16T20:36:54.759059+0000","last_unstale":"2023-10-16T20:59:07.224275+0000","last_undegraded":"2023-10-16T20:59:07.224275+0000","last_fullsized":"2023-10-16T20:59:07.224275+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'140","last_scrub_stamp":"2023-10-16T20:59:07.224189+0000","last_deep_scrub":"67'140","last_deep_scrub_stamp":"2023-10-16T20:59:07.224189+0000","last_clean_scrub_stamp":"2023-10-16T20:59:07.224189+0000","objects_scrubbed":5,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:11:29.789893+0000","scrub_duration":0.016670114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4348959,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":88,"num_read_kb":239,"num_write":133,"num_write_kb":4341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10224,"num_omap_keys":20,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"67'230","reported_seq":388,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.505423+0000","last_change":"2023-10-16T20:58:50.505513+0000","last_active":"2023-10-16T20:58:50.505423+0000","last_peered":"2023-10-16T20:58:50.505423+0000","last_clean":"2023-10-16T20:58:50.505423+0000","last_became_active":"2023-10-16T20:36:54.764436+0000","last_became_peered":"2023-10-16T20:36:54.764436+0000","last_unstale":"2023-10-16T20:58:50.505423+0000","last_undegraded":"2023-10-16T20:58:50.505423+0000","last_fullsized":"2023-10-16T20:58:50.505423+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'230","last_scrub_stamp":"2023-10-16T20:58:50.505320+0000","last_deep_scrub":"67'230","last_deep_scrub_stamp":"2023-10-16T20:58:50.505320+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.505320+0000","objects_scrubbed":7,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:05:38.516764+0000","scrub_duration":0.030122286000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8524313,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":105,"num_read_kb":8412,"num_write":220,"num_write_kb":8480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12018,"num_omap_keys":23,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"67'109","reported_seq":2226,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:59.179914+0000","last_change":"2023-10-16T20:58:59.179962+0000","last_active":"2023-10-16T20:58:59.179914+0000","last_peered":"2023-10-16T20:58:59.179914+0000","last_clean":"2023-10-16T20:58:59.179914+0000","last_became_active":"2023-10-16T20:36:55.277873+0000","last_became_peered":"2023-10-16T20:36:55.277873+0000","last_unstale":"2023-10-16T20:58:59.179914+0000","last_undegraded":"2023-10-16T20:58:59.179914+0000","last_fullsized":"2023-10-16T20:58:59.179914+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'109","last_scrub_stamp":"2023-10-16T20:58:59.179829+0000","last_deep_scrub":"67'109","last_deep_scrub_stamp":"2023-10-16T20:58:59.179829+0000","last_clean_scrub_stamp":"2023-10-16T20:58:59.179829+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:34:00.424120+0000","scrub_duration":0.0027646039999999999,"objects_trimmed":0,"snaptrim_duration":0.012247773,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'608","reported_seq":3243,"reported_epoch":67,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-16T20:59:07.652973+0000","last_change":"2023-10-16T20:59:07.596221+0000","last_active":"2023-10-16T20:59:07.652973+0000","last_peered":"2023-10-16T20:59:07.652973+0000","last_clean":"2023-10-16T20:59:07.652973+0000","last_became_active":"2023-10-16T20:36:59.302435+0000","last_became_peered":"2023-10-16T20:36:59.302435+0000","last_unstale":"2023-10-16T20:59:07.652973+0000","last_undegraded":"2023-10-16T20:59:07.652973+0000","last_fullsized":"2023-10-16T20:59:07.652973+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":149,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001061892,"stat_sum":{"num_bytes":16477397,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":519,"num_read_kb":10696,"num_write":606,"num_write_kb":24605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"66'232","reported_seq":302,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:00.971393+0000","last_change":"2023-10-16T20:36:54.756190+0000","last_active":"2023-10-16T20:58:00.971393+0000","last_peered":"2023-10-16T20:58:00.971393+0000","last_clean":"2023-10-16T20:58:00.971393+0000","last_became_active":"2023-10-16T20:36:54.755834+0000","last_became_peered":"2023-10-16T20:36:54.755834+0000","last_unstale":"2023-10-16T20:58:00.971393+0000","last_undegraded":"2023-10-16T20:58:00.971393+0000","last_fullsized":"2023-10-16T20:58:00.971393+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:53.711958+0000","last_clean_scrub_stamp":"2023-10-16T20:36:53.711958+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582946,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":16385,"num_write":235,"num_write_kb":12609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"67'104","reported_seq":2221,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:51.283330+0000","last_change":"2023-10-16T20:58:51.283381+0000","last_active":"2023-10-16T20:58:51.283330+0000","last_peered":"2023-10-16T20:58:51.283330+0000","last_clean":"2023-10-16T20:58:51.283330+0000","last_became_active":"2023-10-16T20:36:55.279333+0000","last_became_peered":"2023-10-16T20:36:55.279333+0000","last_unstale":"2023-10-16T20:58:51.283330+0000","last_undegraded":"2023-10-16T20:58:51.283330+0000","last_fullsized":"2023-10-16T20:58:51.283330+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'104","last_scrub_stamp":"2023-10-16T20:58:51.283239+0000","last_deep_scrub":"67'104","last_deep_scrub_stamp":"2023-10-16T20:58:51.283239+0000","last_clean_scrub_stamp":"2023-10-16T20:58:51.283239+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:12.385879+0000","scrub_duration":0.0020579999999999999,"objects_trimmed":0,"snaptrim_duration":0.0047896609999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'629","reported_seq":3443,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.755040+0000","last_change":"2023-10-16T20:47:31.586162+0000","last_active":"2023-10-16T20:58:22.755040+0000","last_peered":"2023-10-16T20:58:22.755040+0000","last_clean":"2023-10-16T20:58:22.755040+0000","last_became_active":"2023-10-16T20:36:59.301864+0000","last_became_peered":"2023-10-16T20:36:59.301864+0000","last_unstale":"2023-10-16T20:58:22.755040+0000","last_undegraded":"2023-10-16T20:58:22.755040+0000","last_fullsized":"2023-10-16T20:58:22.755040+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080450769999999994,"stat_sum":{"num_bytes":20568472,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":708,"num_read_kb":22370,"num_write":628,"num_write_kb":40892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"67'120","reported_seq":2231,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.886218+0000","last_change":"2023-10-16T20:47:31.590065+0000","last_active":"2023-10-16T20:58:37.886218+0000","last_peered":"2023-10-16T20:58:37.886218+0000","last_clean":"2023-10-16T20:58:37.886218+0000","last_became_active":"2023-10-16T20:36:55.287442+0000","last_became_peered":"2023-10-16T20:36:55.287442+0000","last_unstale":"2023-10-16T20:58:37.886218+0000","last_undegraded":"2023-10-16T20:58:37.886218+0000","last_fullsized":"2023-10-16T20:58:37.886218+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011630351000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"67'131","reported_seq":2250,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:58.645708+0000","last_change":"2023-10-16T20:58:58.645760+0000","last_active":"2023-10-16T20:58:58.645708+0000","last_peered":"2023-10-16T20:58:58.645708+0000","last_clean":"2023-10-16T20:58:58.645708+0000","last_became_active":"2023-10-16T20:36:55.276276+0000","last_became_peered":"2023-10-16T20:36:55.276276+0000","last_unstale":"2023-10-16T20:58:58.645708+0000","last_undegraded":"2023-10-16T20:58:58.645708+0000","last_fullsized":"2023-10-16T20:58:58.645708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'131","last_scrub_stamp":"2023-10-16T20:58:58.645616+0000","last_deep_scrub":"67'131","last_deep_scrub_stamp":"2023-10-16T20:58:58.645616+0000","last_clean_scrub_stamp":"2023-10-16T20:58:58.645616+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:28.761446+0000","scrub_duration":0.0024049459999999998,"objects_trimmed":0,"snaptrim_duration":0.00091802500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"67'115","reported_seq":2233,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.886238+0000","last_change":"2023-10-16T20:47:31.592299+0000","last_active":"2023-10-16T20:58:37.886238+0000","last_peered":"2023-10-16T20:58:37.886238+0000","last_clean":"2023-10-16T20:58:37.886238+0000","last_became_active":"2023-10-16T20:37:37.413040+0000","last_became_peered":"2023-10-16T20:37:37.413040+0000","last_unstale":"2023-10-16T20:58:37.886238+0000","last_undegraded":"2023-10-16T20:58:37.886238+0000","last_fullsized":"2023-10-16T20:58:37.886238+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012618513,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"67'116","reported_seq":2235,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:04.262507+0000","last_change":"2023-10-16T20:59:04.262559+0000","last_active":"2023-10-16T20:59:04.262507+0000","last_peered":"2023-10-16T20:59:04.262507+0000","last_clean":"2023-10-16T20:59:04.262507+0000","last_became_active":"2023-10-16T20:36:55.275349+0000","last_became_peered":"2023-10-16T20:36:55.275349+0000","last_unstale":"2023-10-16T20:59:04.262507+0000","last_undegraded":"2023-10-16T20:59:04.262507+0000","last_fullsized":"2023-10-16T20:59:04.262507+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'116","last_scrub_stamp":"2023-10-16T20:59:04.262407+0000","last_deep_scrub":"67'116","last_deep_scrub_stamp":"2023-10-16T20:59:04.262407+0000","last_clean_scrub_stamp":"2023-10-16T20:59:04.262407+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:42.718879+0000","scrub_duration":0.0019239579999999999,"objects_trimmed":0,"snaptrim_duration":0.012136073000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"67'88","reported_seq":2207,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:53.160971+0000","last_change":"2023-10-16T20:58:53.161022+0000","last_active":"2023-10-16T20:58:53.160971+0000","last_peered":"2023-10-16T20:58:53.160971+0000","last_clean":"2023-10-16T20:58:53.160971+0000","last_became_active":"2023-10-16T20:36:55.277043+0000","last_became_peered":"2023-10-16T20:36:55.277043+0000","last_unstale":"2023-10-16T20:58:53.160971+0000","last_undegraded":"2023-10-16T20:58:53.160971+0000","last_fullsized":"2023-10-16T20:58:53.160971+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'88","last_scrub_stamp":"2023-10-16T20:58:53.160862+0000","last_deep_scrub":"67'88","last_deep_scrub_stamp":"2023-10-16T20:58:53.160862+0000","last_clean_scrub_stamp":"2023-10-16T20:58:53.160862+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:19.651642+0000","scrub_duration":0.001900938,"objects_trimmed":0,"snaptrim_duration":0.0078065369999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"67'89","reported_seq":2208,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:03.022307+0000","last_change":"2023-10-16T20:59:03.022367+0000","last_active":"2023-10-16T20:59:03.022307+0000","last_peered":"2023-10-16T20:59:03.022307+0000","last_clean":"2023-10-16T20:59:03.022307+0000","last_became_active":"2023-10-16T20:36:55.278715+0000","last_became_peered":"2023-10-16T20:36:55.278715+0000","last_unstale":"2023-10-16T20:59:03.022307+0000","last_undegraded":"2023-10-16T20:59:03.022307+0000","last_fullsized":"2023-10-16T20:59:03.022307+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'89","last_scrub_stamp":"2023-10-16T20:59:03.022209+0000","last_deep_scrub":"67'89","last_deep_scrub_stamp":"2023-10-16T20:59:03.022209+0000","last_clean_scrub_stamp":"2023-10-16T20:59:03.022209+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:31.713609+0000","scrub_duration":0.0019901680000000001,"objects_trimmed":0,"snaptrim_duration":0.0067460719999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"67'124","reported_seq":2243,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:03.386234+0000","last_change":"2023-10-16T20:59:03.386284+0000","last_active":"2023-10-16T20:59:03.386234+0000","last_peered":"2023-10-16T20:59:03.386234+0000","last_clean":"2023-10-16T20:59:03.386234+0000","last_became_active":"2023-10-16T20:36:55.301604+0000","last_became_peered":"2023-10-16T20:36:55.301604+0000","last_unstale":"2023-10-16T20:59:03.386234+0000","last_undegraded":"2023-10-16T20:59:03.386234+0000","last_fullsized":"2023-10-16T20:59:03.386234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'124","last_scrub_stamp":"2023-10-16T20:59:03.386139+0000","last_deep_scrub":"67'124","last_deep_scrub_stamp":"2023-10-16T20:59:03.386139+0000","last_clean_scrub_stamp":"2023-10-16T20:59:03.386139+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:18.278253+0000","scrub_duration":0.0020316280000000002,"objects_trimmed":0,"snaptrim_duration":0.012969489000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"67'99","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:01.252753+0000","last_change":"2023-10-16T20:59:01.252799+0000","last_active":"2023-10-16T20:59:01.252753+0000","last_peered":"2023-10-16T20:59:01.252753+0000","last_clean":"2023-10-16T20:59:01.252753+0000","last_became_active":"2023-10-16T20:36:55.276984+0000","last_became_peered":"2023-10-16T20:36:55.276984+0000","last_unstale":"2023-10-16T20:59:01.252753+0000","last_undegraded":"2023-10-16T20:59:01.252753+0000","last_fullsized":"2023-10-16T20:59:01.252753+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'99","last_scrub_stamp":"2023-10-16T20:59:01.252663+0000","last_deep_scrub":"67'99","last_deep_scrub_stamp":"2023-10-16T20:59:01.252663+0000","last_clean_scrub_stamp":"2023-10-16T20:59:01.252663+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:32:54.405775+0000","scrub_duration":0.0025568829999999998,"objects_trimmed":0,"snaptrim_duration":0.0086623480000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'111","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:07.294638+0000","last_change":"2023-10-16T20:59:07.294686+0000","last_active":"2023-10-16T20:59:07.294638+0000","last_peered":"2023-10-16T20:59:07.294638+0000","last_clean":"2023-10-16T20:59:07.294638+0000","last_became_active":"2023-10-16T20:36:55.290023+0000","last_became_peered":"2023-10-16T20:36:55.290023+0000","last_unstale":"2023-10-16T20:59:07.294638+0000","last_undegraded":"2023-10-16T20:59:07.294638+0000","last_fullsized":"2023-10-16T20:59:07.294638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'111","last_scrub_stamp":"2023-10-16T20:59:07.294551+0000","last_deep_scrub":"67'111","last_deep_scrub_stamp":"2023-10-16T20:59:07.294551+0000","last_clean_scrub_stamp":"2023-10-16T20:59:07.294551+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:48:23.755124+0000","scrub_duration":0.0022375450000000001,"objects_trimmed":0,"snaptrim_duration":0.013081514000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'124","reported_seq":2234,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.526976+0000","last_change":"2023-10-16T20:47:31.586678+0000","last_active":"2023-10-16T20:58:38.526976+0000","last_peered":"2023-10-16T20:58:38.526976+0000","last_clean":"2023-10-16T20:58:38.526976+0000","last_became_active":"2023-10-16T20:36:55.280186+0000","last_became_peered":"2023-10-16T20:36:55.280186+0000","last_unstale":"2023-10-16T20:58:38.526976+0000","last_undegraded":"2023-10-16T20:58:38.526976+0000","last_fullsized":"2023-10-16T20:58:38.526976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083925690000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"67'118","reported_seq":2228,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.892253+0000","last_change":"2023-10-16T20:47:31.589254+0000","last_active":"2023-10-16T20:58:37.892253+0000","last_peered":"2023-10-16T20:58:37.892253+0000","last_clean":"2023-10-16T20:58:37.892253+0000","last_became_active":"2023-10-16T20:36:55.297253+0000","last_became_peered":"2023-10-16T20:36:55.297253+0000","last_unstale":"2023-10-16T20:58:37.892253+0000","last_undegraded":"2023-10-16T20:58:37.892253+0000","last_fullsized":"2023-10-16T20:58:37.892253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097685800000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"67'108","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.893558+0000","last_change":"2023-10-16T20:47:31.492104+0000","last_active":"2023-10-16T20:58:37.893558+0000","last_peered":"2023-10-16T20:58:37.893558+0000","last_clean":"2023-10-16T20:58:37.893558+0000","last_became_active":"2023-10-16T20:36:55.286044+0000","last_became_peered":"2023-10-16T20:36:55.286044+0000","last_unstale":"2023-10-16T20:58:37.893558+0000","last_undegraded":"2023-10-16T20:58:37.893558+0000","last_fullsized":"2023-10-16T20:58:37.893558+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011681474000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"67'110","reported_seq":2220,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.865807+0000","last_change":"2023-10-16T20:47:31.592401+0000","last_active":"2023-10-16T20:58:37.865807+0000","last_peered":"2023-10-16T20:58:37.865807+0000","last_clean":"2023-10-16T20:58:37.865807+0000","last_became_active":"2023-10-16T20:36:55.286864+0000","last_became_peered":"2023-10-16T20:36:55.286864+0000","last_unstale":"2023-10-16T20:58:37.865807+0000","last_undegraded":"2023-10-16T20:58:37.865807+0000","last_fullsized":"2023-10-16T20:58:37.865807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012885232999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"67'106","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.872663+0000","last_change":"2023-10-16T20:47:31.588333+0000","last_active":"2023-10-16T20:58:37.872663+0000","last_peered":"2023-10-16T20:58:37.872663+0000","last_clean":"2023-10-16T20:58:37.872663+0000","last_became_active":"2023-10-16T20:36:55.292262+0000","last_became_peered":"2023-10-16T20:36:55.292262+0000","last_unstale":"2023-10-16T20:58:37.872663+0000","last_undegraded":"2023-10-16T20:58:37.872663+0000","last_fullsized":"2023-10-16T20:58:37.872663+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077171749999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"67'107","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:00.659400+0000","last_change":"2023-10-16T20:59:00.659448+0000","last_active":"2023-10-16T20:59:00.659400+0000","last_peered":"2023-10-16T20:59:00.659400+0000","last_clean":"2023-10-16T20:59:00.659400+0000","last_became_active":"2023-10-16T20:36:55.282806+0000","last_became_peered":"2023-10-16T20:36:55.282806+0000","last_unstale":"2023-10-16T20:59:00.659400+0000","last_undegraded":"2023-10-16T20:59:00.659400+0000","last_fullsized":"2023-10-16T20:59:00.659400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'107","last_scrub_stamp":"2023-10-16T20:59:00.659305+0000","last_deep_scrub":"67'107","last_deep_scrub_stamp":"2023-10-16T20:59:00.659305+0000","last_clean_scrub_stamp":"2023-10-16T20:59:00.659305+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:01:49.319877+0000","scrub_duration":0.0024992080000000002,"objects_trimmed":0,"snaptrim_duration":0.00086132999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"67'109","reported_seq":2219,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.888026+0000","last_change":"2023-10-16T20:47:31.594959+0000","last_active":"2023-10-16T20:58:37.888026+0000","last_peered":"2023-10-16T20:58:37.888026+0000","last_clean":"2023-10-16T20:58:37.888026+0000","last_became_active":"2023-10-16T20:36:55.303095+0000","last_became_peered":"2023-10-16T20:36:55.303095+0000","last_unstale":"2023-10-16T20:58:37.888026+0000","last_undegraded":"2023-10-16T20:58:37.888026+0000","last_fullsized":"2023-10-16T20:58:37.888026+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013746639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"67'119","reported_seq":2229,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.357033+0000","last_change":"2023-10-16T20:47:31.595380+0000","last_active":"2023-10-16T20:58:38.357033+0000","last_peered":"2023-10-16T20:58:38.357033+0000","last_clean":"2023-10-16T20:58:38.357033+0000","last_became_active":"2023-10-16T20:36:55.291447+0000","last_became_peered":"2023-10-16T20:36:55.291447+0000","last_unstale":"2023-10-16T20:58:38.357033+0000","last_undegraded":"2023-10-16T20:58:38.357033+0000","last_fullsized":"2023-10-16T20:58:38.357033+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01315413,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"67'107","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:48.072269+0000","last_change":"2023-10-16T20:58:48.072313+0000","last_active":"2023-10-16T20:58:48.072269+0000","last_peered":"2023-10-16T20:58:48.072269+0000","last_clean":"2023-10-16T20:58:48.072269+0000","last_became_active":"2023-10-16T20:36:55.280605+0000","last_became_peered":"2023-10-16T20:36:55.280605+0000","last_unstale":"2023-10-16T20:58:48.072269+0000","last_undegraded":"2023-10-16T20:58:48.072269+0000","last_fullsized":"2023-10-16T20:58:48.072269+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'107","last_scrub_stamp":"2023-10-16T20:58:48.072178+0000","last_deep_scrub":"67'107","last_deep_scrub_stamp":"2023-10-16T20:58:48.072178+0000","last_clean_scrub_stamp":"2023-10-16T20:58:48.072178+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:58:57.439416+0000","scrub_duration":0.0025767939999999999,"objects_trimmed":0,"snaptrim_duration":0.0033133080000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"67'112","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:57.143403+0000","last_change":"2023-10-16T20:58:57.143460+0000","last_active":"2023-10-16T20:58:57.143403+0000","last_peered":"2023-10-16T20:58:57.143403+0000","last_clean":"2023-10-16T20:58:57.143403+0000","last_became_active":"2023-10-16T20:36:55.277094+0000","last_became_peered":"2023-10-16T20:36:55.277094+0000","last_unstale":"2023-10-16T20:58:57.143403+0000","last_undegraded":"2023-10-16T20:58:57.143403+0000","last_fullsized":"2023-10-16T20:58:57.143403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'112","last_scrub_stamp":"2023-10-16T20:58:57.143322+0000","last_deep_scrub":"67'112","last_deep_scrub_stamp":"2023-10-16T20:58:57.143322+0000","last_clean_scrub_stamp":"2023-10-16T20:58:57.143322+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:52:54.678403+0000","scrub_duration":0.0016325790000000001,"objects_trimmed":0,"snaptrim_duration":0.011816411000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"67'94","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.120683+0000","last_change":"2023-10-16T20:58:50.120723+0000","last_active":"2023-10-16T20:58:50.120683+0000","last_peered":"2023-10-16T20:58:50.120683+0000","last_clean":"2023-10-16T20:58:50.120683+0000","last_became_active":"2023-10-16T20:36:55.297199+0000","last_became_peered":"2023-10-16T20:36:55.297199+0000","last_unstale":"2023-10-16T20:58:50.120683+0000","last_undegraded":"2023-10-16T20:58:50.120683+0000","last_fullsized":"2023-10-16T20:58:50.120683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'94","last_scrub_stamp":"2023-10-16T20:58:50.120603+0000","last_deep_scrub":"67'94","last_deep_scrub_stamp":"2023-10-16T20:58:50.120603+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.120603+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:29:38.916058+0000","scrub_duration":0.0022327750000000002,"objects_trimmed":0,"snaptrim_duration":0.0086430910000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"67'90","reported_seq":2208,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:52.287328+0000","last_change":"2023-10-16T20:58:52.287408+0000","last_active":"2023-10-16T20:58:52.287328+0000","last_peered":"2023-10-16T20:58:52.287328+0000","last_clean":"2023-10-16T20:58:52.287328+0000","last_became_active":"2023-10-16T20:36:55.286964+0000","last_became_peered":"2023-10-16T20:36:55.286964+0000","last_unstale":"2023-10-16T20:58:52.287328+0000","last_undegraded":"2023-10-16T20:58:52.287328+0000","last_fullsized":"2023-10-16T20:58:52.287328+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'90","last_scrub_stamp":"2023-10-16T20:58:52.287221+0000","last_deep_scrub":"67'90","last_deep_scrub_stamp":"2023-10-16T20:58:52.287221+0000","last_clean_scrub_stamp":"2023-10-16T20:58:52.287221+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:06.754055+0000","scrub_duration":0.0018582900000000001,"objects_trimmed":0,"snaptrim_duration":0.0051857609999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"67'110","reported_seq":2227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:42.659992+0000","last_change":"2023-10-16T20:47:31.594637+0000","last_active":"2023-10-16T20:58:42.659992+0000","last_peered":"2023-10-16T20:58:42.659992+0000","last_clean":"2023-10-16T20:58:42.659992+0000","last_became_active":"2023-10-16T20:37:37.413492+0000","last_became_peered":"2023-10-16T20:37:37.413492+0000","last_unstale":"2023-10-16T20:58:42.659992+0000","last_undegraded":"2023-10-16T20:58:42.659992+0000","last_fullsized":"2023-10-16T20:58:42.659992+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014529932000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"67'106","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:42.660003+0000","last_change":"2023-10-16T20:47:31.593072+0000","last_active":"2023-10-16T20:58:42.660003+0000","last_peered":"2023-10-16T20:58:42.660003+0000","last_clean":"2023-10-16T20:58:42.660003+0000","last_became_active":"2023-10-16T20:36:55.302468+0000","last_became_peered":"2023-10-16T20:36:55.302468+0000","last_unstale":"2023-10-16T20:58:42.660003+0000","last_undegraded":"2023-10-16T20:58:42.660003+0000","last_fullsized":"2023-10-16T20:58:42.660003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012359399,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"67'92","reported_seq":2202,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:47.665090+0000","last_change":"2023-10-16T20:47:31.585931+0000","last_active":"2023-10-16T20:58:47.665090+0000","last_peered":"2023-10-16T20:58:47.665090+0000","last_clean":"2023-10-16T20:58:47.665090+0000","last_became_active":"2023-10-16T20:36:55.296476+0000","last_became_peered":"2023-10-16T20:36:55.296476+0000","last_unstale":"2023-10-16T20:58:47.665090+0000","last_undegraded":"2023-10-16T20:58:47.665090+0000","last_fullsized":"2023-10-16T20:58:47.665090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072035780000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'95","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:56.206037+0000","last_change":"2023-10-16T20:58:56.206084+0000","last_active":"2023-10-16T20:58:56.206037+0000","last_peered":"2023-10-16T20:58:56.206037+0000","last_clean":"2023-10-16T20:58:56.206037+0000","last_became_active":"2023-10-16T20:36:55.293687+0000","last_became_peered":"2023-10-16T20:36:55.293687+0000","last_unstale":"2023-10-16T20:58:56.206037+0000","last_undegraded":"2023-10-16T20:58:56.206037+0000","last_fullsized":"2023-10-16T20:58:56.206037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'95","last_scrub_stamp":"2023-10-16T20:58:56.205959+0000","last_deep_scrub":"67'95","last_deep_scrub_stamp":"2023-10-16T20:58:56.205959+0000","last_clean_scrub_stamp":"2023-10-16T20:58:56.205959+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:53.185697+0000","scrub_duration":0.0025542830000000001,"objects_trimmed":0,"snaptrim_duration":0.0074548399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"67'127","reported_seq":2236,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.968924+0000","last_change":"2023-10-16T20:47:31.588394+0000","last_active":"2023-10-16T20:58:37.968924+0000","last_peered":"2023-10-16T20:58:37.968924+0000","last_clean":"2023-10-16T20:58:37.968924+0000","last_became_active":"2023-10-16T20:36:55.280444+0000","last_became_peered":"2023-10-16T20:36:55.280444+0000","last_unstale":"2023-10-16T20:58:37.968924+0000","last_undegraded":"2023-10-16T20:58:37.968924+0000","last_fullsized":"2023-10-16T20:58:37.968924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068623299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'109","reported_seq":2226,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.544588+0000","last_change":"2023-10-16T20:58:50.544632+0000","last_active":"2023-10-16T20:58:50.544588+0000","last_peered":"2023-10-16T20:58:50.544588+0000","last_clean":"2023-10-16T20:58:50.544588+0000","last_became_active":"2023-10-16T20:36:55.296694+0000","last_became_peered":"2023-10-16T20:36:55.296694+0000","last_unstale":"2023-10-16T20:58:50.544588+0000","last_undegraded":"2023-10-16T20:58:50.544588+0000","last_fullsized":"2023-10-16T20:58:50.544588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'109","last_scrub_stamp":"2023-10-16T20:58:50.544515+0000","last_deep_scrub":"67'109","last_deep_scrub_stamp":"2023-10-16T20:58:50.544515+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.544515+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:15:02.193326+0000","scrub_duration":0.001832127,"objects_trimmed":0,"snaptrim_duration":0.0096291870000000009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"67'120","reported_seq":2229,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:38.513162+0000","last_change":"2023-10-16T20:47:31.585319+0000","last_active":"2023-10-16T20:58:38.513162+0000","last_peered":"2023-10-16T20:58:38.513162+0000","last_clean":"2023-10-16T20:58:38.513162+0000","last_became_active":"2023-10-16T20:36:55.291052+0000","last_became_peered":"2023-10-16T20:36:55.291052+0000","last_unstale":"2023-10-16T20:58:38.513162+0000","last_undegraded":"2023-10-16T20:58:38.513162+0000","last_fullsized":"2023-10-16T20:58:38.513162+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070559409999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"67'94","reported_seq":2214,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:49.588755+0000","last_change":"2023-10-16T20:58:49.588755+0000","last_active":"2023-10-16T20:58:49.588755+0000","last_peered":"2023-10-16T20:58:49.588755+0000","last_clean":"2023-10-16T20:58:49.588755+0000","last_became_active":"2023-10-16T20:36:55.290386+0000","last_became_peered":"2023-10-16T20:36:55.290386+0000","last_unstale":"2023-10-16T20:58:49.588755+0000","last_undegraded":"2023-10-16T20:58:49.588755+0000","last_fullsized":"2023-10-16T20:58:49.588755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'94","last_scrub_stamp":"2023-10-16T20:58:49.588589+0000","last_deep_scrub":"67'94","last_deep_scrub_stamp":"2023-10-16T20:58:49.588589+0000","last_clean_scrub_stamp":"2023-10-16T20:58:49.588589+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:41:38.561038+0000","scrub_duration":0.0020095009999999999,"objects_trimmed":0,"snaptrim_duration":0.0093217969999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"67'114","reported_seq":2233,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.138708+0000","last_change":"2023-10-16T20:58:50.138746+0000","last_active":"2023-10-16T20:58:50.138708+0000","last_peered":"2023-10-16T20:58:50.138708+0000","last_clean":"2023-10-16T20:58:50.138708+0000","last_became_active":"2023-10-16T20:36:55.270787+0000","last_became_peered":"2023-10-16T20:36:55.270787+0000","last_unstale":"2023-10-16T20:58:50.138708+0000","last_undegraded":"2023-10-16T20:58:50.138708+0000","last_fullsized":"2023-10-16T20:58:50.138708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'114","last_scrub_stamp":"2023-10-16T20:58:50.138632+0000","last_deep_scrub":"67'114","last_deep_scrub_stamp":"2023-10-16T20:58:50.138632+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.138632+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:35:28.601665+0000","scrub_duration":0.0015404730000000001,"objects_trimmed":0,"snaptrim_duration":0.010801276,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"67'94","reported_seq":2213,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.047134+0000","last_change":"2023-10-16T20:58:50.047177+0000","last_active":"2023-10-16T20:58:50.047134+0000","last_peered":"2023-10-16T20:58:50.047134+0000","last_clean":"2023-10-16T20:58:50.047134+0000","last_became_active":"2023-10-16T20:36:55.271373+0000","last_became_peered":"2023-10-16T20:36:55.271373+0000","last_unstale":"2023-10-16T20:58:50.047134+0000","last_undegraded":"2023-10-16T20:58:50.047134+0000","last_fullsized":"2023-10-16T20:58:50.047134+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'94","last_scrub_stamp":"2023-10-16T20:58:50.047059+0000","last_deep_scrub":"67'94","last_deep_scrub_stamp":"2023-10-16T20:58:50.047059+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.047059+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:24.904268+0000","scrub_duration":0.0017358,"objects_trimmed":0,"snaptrim_duration":0.006091478,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"67'111","reported_seq":2222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.890295+0000","last_change":"2023-10-16T20:47:31.587880+0000","last_active":"2023-10-16T20:58:37.890295+0000","last_peered":"2023-10-16T20:58:37.890295+0000","last_clean":"2023-10-16T20:58:37.890295+0000","last_became_active":"2023-10-16T20:36:55.291016+0000","last_became_peered":"2023-10-16T20:36:55.291016+0000","last_unstale":"2023-10-16T20:58:37.890295+0000","last_undegraded":"2023-10-16T20:58:37.890295+0000","last_fullsized":"2023-10-16T20:58:37.890295+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.009370148,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":924319467,"num_objects":10294,"num_object_clones":0,"num_object_copies":41176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10294,"num_whiteouts":0,"num_read":30602,"num_read_kb":676969,"num_write":21002,"num_write_kb":1302825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966899200,"data_stored":1966899200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21050,"ondisk_log_size":21050,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6794,"ondisk_log_size":6794,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":179249246,"num_objects":198,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1063,"num_read_kb":205540,"num_write":6373,"num_write_kb":180813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":115,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":399398,"num_omap_keys":761,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":358752256,"data_stored":358498492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6247,"ondisk_log_size":6247,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509712,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396784,"kb_used_data":280936,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93975056,"statfs":{"total":96636764160,"available":96230457344,"internally_reserved":0,"allocated":287678464,"data_stored":287278454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":118618571},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.52300000000000002,"15min":0.629},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67900000000000005,"5min":0.91400000000000003,"15min":2.4489999999999998},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54600000000000004,"15min":0.64700000000000002},"min":{"1min":0.378,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80200000000000005,"5min":0.95399999999999996,"15min":11.77},"last":0.75600000000000001}]},{"osd":1,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57699999999999996,"15min":0.65300000000000002},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73699999999999999,"5min":0.86199999999999999,"15min":4.5279999999999996},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.58499999999999996,"15min":0.68400000000000005},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.84599999999999997,"5min":1.024,"15min":11.971},"last":0.67700000000000005}]},{"osd":2,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61599999999999999,"15min":0.67200000000000004},"min":{"1min":0.41999999999999998,"5min":0.40200000000000002,"15min":0.34300000000000003},"max":{"1min":0.81899999999999995,"5min":0.92200000000000004,"15min":7.1130000000000004},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.629,"15min":0.70999999999999996},"min":{"1min":0.51100000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.877,"5min":0.98299999999999998,"15min":9.4450000000000003},"last":0.81999999999999995}]},{"osd":3,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.64900000000000002,"15min":0.68300000000000005},"min":{"1min":0.439,"5min":0.437,"15min":0.41299999999999998},"max":{"1min":0.78000000000000003,"5min":1,"15min":3.363},"last":0.627},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.68300000000000005,"15min":0.72699999999999998},"min":{"1min":0.50900000000000001,"5min":0.46899999999999997,"15min":0.42099999999999999},"max":{"1min":0.86299999999999999,"5min":1.0529999999999999,"15min":3.3980000000000001},"last":0.85499999999999998}]},{"osd":4,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56699999999999995,"15min":0.61399999999999999},"min":{"1min":0.36399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81899999999999995,"5min":1.091,"15min":12.532999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.623,"15min":0.628},"min":{"1min":0.47099999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.90600000000000003,"5min":1.107,"15min":3.8700000000000001},"last":0.875}]},{"osd":5,"last update":"Mon Oct 16 20:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60999999999999999,"15min":0.66300000000000003},"min":{"1min":0.41399999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.85699999999999998,"5min":1.042,"15min":2.2010000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.63600000000000001,"15min":0.73699999999999999},"min":{"1min":0.47499999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94399999999999995,"5min":1.0369999999999999,"15min":12.353},"last":0.51800000000000002}]},{"osd":6,"last update":"Mon Oct 16 20:58:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.65100000000000002,"15min":0.746},"min":{"1min":0.40799999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93799999999999994,"5min":1.1299999999999999,"15min":11.929},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66900000000000004,"15min":0.71399999999999997},"min":{"1min":0.41799999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.92900000000000005,"5min":1.0620000000000001,"15min":6.0359999999999996},"last":0.90500000000000003}]}]},{"osd":6,"up_from":15,"seq":64424509712,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422216,"kb_used_data":300800,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93949624,"statfs":{"total":96636764160,"available":96204414976,"internally_reserved":0,"allocated":308019200,"data_stored":307631643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124320199},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.54900000000000004,"15min":0.61499999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27800000000000002},"max":{"1min":0.69499999999999995,"5min":0.93799999999999994,"15min":4.5419999999999998},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.56499999999999995,"15min":0.624},"min":{"1min":0.33200000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.76000000000000001,"5min":2.0329999999999999,"15min":4.9699999999999998},"last":0.59099999999999997}]},{"osd":1,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.624,"15min":0.64100000000000001},"min":{"1min":0.433,"5min":0.39200000000000002,"15min":0.377},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":2.8610000000000002},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67000000000000004,"15min":0.69599999999999995},"min":{"1min":0.41499999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.94399999999999995,"5min":1.014,"15min":1.0309999999999999},"last":0.72299999999999998}]},{"osd":2,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.59699999999999998,"15min":0.65000000000000002},"min":{"1min":0.36399999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86599999999999999,"5min":1.0289999999999999,"15min":1.895},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.65500000000000003,"15min":0.68500000000000005},"min":{"1min":0.40799999999999997,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78500000000000003,"5min":0.96099999999999997,"15min":2.3340000000000001},"last":0.73999999999999999}]},{"osd":3,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.755,"15min":0.72099999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.36599999999999999},"max":{"1min":0.86699999999999999,"5min":4.6340000000000003,"15min":4.6340000000000003},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.64100000000000001,"15min":0.69499999999999995},"min":{"1min":0.435,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.86099999999999999,"5min":1.163,"15min":5.1029999999999998},"last":0.70299999999999996}]},{"osd":4,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55500000000000005,"15min":0.58999999999999997},"min":{"1min":0.36299999999999999,"5min":0.34799999999999998,"15min":0.30299999999999999},"max":{"1min":0.80300000000000005,"5min":1.0189999999999999,"15min":4.8959999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.63500000000000001,"15min":0.63400000000000001},"min":{"1min":0.374,"5min":0.313,"15min":0.313},"max":{"1min":0.996,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.63100000000000001}]},{"osd":5,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.59799999999999998,"15min":0.623},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.82199999999999995,"5min":1.05,"15min":2.3769999999999998},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.626,"15min":0.66200000000000003},"min":{"1min":0.38200000000000001,"5min":0.35999999999999999,"15min":0.34499999999999997},"max":{"1min":0.84499999999999997,"5min":1.0700000000000001,"15min":4.9379999999999997},"last":0.88700000000000001}]},{"osd":7,"last update":"Mon Oct 16 20:58:12 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.71199999999999997,"15min":0.70999999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.034,"5min":1.425,"15min":4.5789999999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.76800000000000002,"15min":0.748},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.39300000000000002},"max":{"1min":1.0600000000000001,"5min":1.6120000000000001,"15min":5.2460000000000004},"last":0.76400000000000001}]}]},{"osd":5,"up_from":15,"seq":64424509714,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325108,"kb_used_data":225900,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94046732,"statfs":{"total":96636764160,"available":96303853568,"internally_reserved":0,"allocated":231321600,"data_stored":230919997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":101579207},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.57199999999999995,"15min":0.61299999999999999},"min":{"1min":0.30499999999999999,"5min":0.28199999999999997,"15min":0.25700000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":2.4580000000000002},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.59299999999999997,"15min":0.65800000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.94999999999999996,"5min":1.1799999999999999,"15min":7.181},"last":0.65600000000000003}]},{"osd":1,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":0.76800000000000002,"15min":0.69099999999999995},"min":{"1min":0.36699999999999999,"5min":0.33900000000000002,"15min":0.29899999999999999},"max":{"1min":16.655000000000001,"5min":16.655000000000001,"15min":16.655000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.5529999999999999,"5min":0.80100000000000005,"15min":0.71299999999999997},"min":{"1min":0.35099999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":16.289000000000001,"5min":16.289000000000001,"15min":16.289000000000001},"last":0.77900000000000003}]},{"osd":2,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.63200000000000001,"15min":0.67300000000000004},"min":{"1min":0.374,"5min":0.34599999999999997,"15min":0.308},"max":{"1min":1.1339999999999999,"5min":1.7110000000000001,"15min":3.3799999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67100000000000004,"15min":0.67700000000000005},"min":{"1min":0.42399999999999999,"5min":0.39800000000000002,"15min":0.38},"max":{"1min":1.1100000000000001,"5min":1.583,"15min":4.5369999999999999},"last":0.70499999999999996}]},{"osd":3,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.67400000000000004,"15min":0.69699999999999995},"min":{"1min":0.42399999999999999,"5min":0.39400000000000002,"15min":0.34899999999999998},"max":{"1min":1.0860000000000001,"5min":1.329,"15min":4.1070000000000002},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68600000000000005,"15min":0.69399999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.38},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":3.476},"last":0.751}]},{"osd":4,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53100000000000003,"15min":0.55500000000000005},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.749,"5min":0.85099999999999998,"15min":2.415},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.59999999999999998,"15min":0.66300000000000003},"min":{"1min":0.34300000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.78900000000000003,"5min":1.155,"15min":3.2919999999999998},"last":0.496}]},{"osd":6,"last update":"Mon Oct 16 20:58:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.57299999999999995,"15min":0.59699999999999998},"min":{"1min":0.39000000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79400000000000004,"5min":0.93799999999999994,"15min":3.2400000000000002},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.31},"max":{"1min":0.71799999999999997,"5min":1.036,"15min":1.2310000000000001},"last":0.47199999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.44600000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76300000000000001,"5min":1.23,"15min":5.2759999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69399999999999995,"15min":0.70699999999999996},"min":{"1min":0.45000000000000001,"5min":0.435,"15min":0.42299999999999999},"max":{"1min":0.83199999999999996,"5min":0.97699999999999998,"15min":3.3140000000000001},"last":0.53300000000000003}]}]},{"osd":4,"up_from":15,"seq":64424509712,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420128,"kb_used_data":290392,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93951712,"statfs":{"total":96636764160,"available":96206553088,"internally_reserved":0,"allocated":297361408,"data_stored":296975417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56100000000000005,"15min":0.61299999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.83799999999999997,"5min":0.91400000000000003,"15min":1.524},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61699999999999999,"15min":0.69199999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.5649999999999999},"last":0.85299999999999998}]},{"osd":1,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.376,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.1749999999999998,"5min":4.1749999999999998,"15min":4.1749999999999998},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.625,"15min":0.66600000000000004},"min":{"1min":0.38200000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82599999999999996,"5min":0.94899999999999995,"15min":1.1020000000000001},"last":0.80600000000000005}]},{"osd":2,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.38900000000000001},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":2.3370000000000002},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.68300000000000005,"15min":0.71199999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.84499999999999997,"5min":1.2210000000000001,"15min":2.3660000000000001},"last":0.90900000000000003}]},{"osd":3,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.377},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":2.04},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":2.2240000000000002},"last":0.88400000000000001}]},{"osd":5,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.58799999999999997,"15min":0.60299999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.9970000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63600000000000001,"15min":0.60499999999999998},"min":{"1min":0.41299999999999998,"5min":0.39500000000000002,"15min":0.33900000000000002},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.74299999999999999}]},{"osd":6,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.60299999999999998,"15min":0.61599999999999999},"min":{"1min":0.33900000000000002,"5min":0.315,"15min":0.312},"max":{"1min":1.2270000000000001,"5min":1.6339999999999999,"15min":2.2639999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.66200000000000003,"15min":0.67300000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.8300000000000001},"last":0.61499999999999999}]},{"osd":7,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.69399999999999995,"15min":0.66800000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2969999999999999,"5min":1.347,"15min":1.347},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65500000000000003,"15min":0.67900000000000005},"min":{"1min":0.38600000000000001,"5min":0.35899999999999999,"15min":0.35599999999999998},"max":{"1min":1.266,"5min":1.266,"15min":2.2999999999999998},"last":0.68200000000000005}]}]},{"osd":3,"up_from":15,"seq":64424509712,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485596,"kb_used_data":360020,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93886244,"statfs":{"total":96636764160,"available":96139513856,"internally_reserved":0,"allocated":368660480,"data_stored":368273774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":128580040},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.45200000000000001,"15min":0.433},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":0.76300000000000001,"5min":2.8109999999999999,"15min":4.5830000000000002},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.44700000000000001,"15min":0.44600000000000001},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.253},"max":{"1min":0.58199999999999996,"5min":1.147,"15min":1.2509999999999999},"last":0.52800000000000002}]},{"osd":1,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.49099999999999999,"15min":0.503},"min":{"1min":0.33500000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67000000000000004,"5min":1.411,"15min":4.6840000000000002},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.504,"15min":0.51000000000000001},"min":{"1min":0.29699999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61399999999999999,"5min":2.6360000000000001,"15min":2.7429999999999999},"last":0.45800000000000002}]},{"osd":2,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.40000000000000002,"5min":0.34399999999999997,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":1.286,"15min":4.6449999999999996},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55700000000000005,"15min":0.51900000000000002},"min":{"1min":0.32200000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":1.1850000000000001,"15min":1.911},"last":0.47999999999999998}]},{"osd":4,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.72399999999999998,"15min":0.76200000000000001},"min":{"1min":0.52000000000000002,"5min":0.45400000000000001,"15min":0.44800000000000001},"max":{"1min":1.0620000000000001,"5min":1.538,"15min":1.7749999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.72399999999999998,"15min":0.74199999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.155,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":0.90000000000000002}]},{"osd":5,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.74199999999999999,"15min":0.755},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0780000000000001,"5min":1.4970000000000001,"15min":4.8449999999999998},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.627,"5min":0.74099999999999999,"15min":0.745},"min":{"1min":0.48199999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.1220000000000001,"5min":2.948,"15min":2.948},"last":0.88900000000000001}]},{"osd":6,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.76300000000000001,"15min":0.76600000000000001},"min":{"1min":0.48899999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.98399999999999999,"5min":1.633,"15min":4.8099999999999996},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.77000000000000002,"15min":0.76800000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.45400000000000001},"max":{"1min":0.93899999999999995,"5min":1.6559999999999999,"15min":2.339},"last":0.84199999999999997}]},{"osd":7,"last update":"Mon Oct 16 20:58:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.79800000000000004,"15min":0.86499999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.46000000000000002},"max":{"1min":1.216,"5min":1.593,"15min":34.039000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.76900000000000002,"15min":0.77700000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0249999999999999,"5min":2.984,"15min":2.984},"last":0.77100000000000002}]}]},{"osd":2,"up_from":15,"seq":64424509713,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303812,"kb_used_data":200444,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94068028,"statfs":{"total":96636764160,"available":96325660672,"internally_reserved":0,"allocated":205254656,"data_stored":204864813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42899999999999999,"15min":0.45000000000000001},"min":{"1min":0.27100000000000002,"5min":0.23400000000000001,"15min":0.20599999999999999},"max":{"1min":0.74299999999999999,"5min":0.92500000000000004,"15min":5.5499999999999998},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.52600000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.1180000000000001,"5min":1.1220000000000001,"15min":5.4160000000000004},"last":0.56499999999999995}]},{"osd":1,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.504,"15min":0.5},"min":{"1min":0.307,"5min":0.307,"15min":0.26000000000000001},"max":{"1min":0.81899999999999995,"5min":1.25,"15min":2.2349999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.501},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.27200000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":5.4359999999999999},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.28399999999999997,"5min":0.26400000000000001,"15min":0.25800000000000001},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.312,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.83099999999999996,"5min":1.286,"15min":5.3789999999999996},"last":0.52300000000000002}]},{"osd":4,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.73099999999999998,"15min":0.80000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":3.464,"5min":3.464,"15min":16.905999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.70699999999999996,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.44700000000000001,"15min":0.441},"max":{"1min":0.92100000000000004,"5min":1.3100000000000001,"15min":10.064},"last":0.68200000000000005}]},{"osd":5,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70999999999999996,"15min":0.72999999999999998},"min":{"1min":0.45300000000000001,"5min":0.41099999999999998,"15min":0.32800000000000001},"max":{"1min":1.0960000000000001,"5min":1.304,"15min":3.2989999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.73499999999999999,"15min":0.77100000000000002},"min":{"1min":0.52400000000000002,"5min":0.499,"15min":0.46700000000000003},"max":{"1min":1.0069999999999999,"5min":1.3640000000000001,"15min":5.3330000000000002},"last":0.78400000000000003}]},{"osd":6,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.76500000000000001,"15min":0.755},"min":{"1min":0.438,"5min":0.438,"15min":0.42299999999999999},"max":{"1min":3.5710000000000002,"5min":3.5710000000000002,"15min":3.5710000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.77900000000000003,"15min":0.77000000000000002},"min":{"1min":0.49099999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":5.4749999999999996},"last":0.72799999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.47399999999999998,"5min":0.47199999999999998,"15min":0.44500000000000001},"max":{"1min":3.5089999999999999,"5min":3.5089999999999999,"15min":5.516},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77200000000000002,"15min":0.79100000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.41899999999999998},"max":{"1min":1.325,"5min":1.325,"15min":8.4610000000000003},"last":0.71099999999999997}]}]},{"osd":1,"up_from":15,"seq":64424509712,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490524,"kb_used_data":360788,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93881316,"statfs":{"total":96636764160,"available":96134467584,"internally_reserved":0,"allocated":369446912,"data_stored":369062219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.437,"15min":0.45700000000000002},"min":{"1min":0.245,"5min":0.219,"15min":0.21299999999999999},"max":{"1min":0.79800000000000004,"5min":0.93000000000000005,"15min":8.9339999999999993},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.34300000000000003,"5min":0.28999999999999998,"15min":0.23000000000000001},"max":{"1min":0.67200000000000004,"5min":1.6899999999999999,"15min":3.8809999999999998},"last":0.67100000000000004}]},{"osd":2,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.502,"15min":0.46000000000000002},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.248},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.497},{"interface":"front","average":{"1min":0.496,"5min":0.55900000000000005,"15min":0.55400000000000005},"min":{"1min":0.29699999999999999,"5min":0.29099999999999998,"15min":0.28499999999999998},"max":{"1min":0.88600000000000001,"5min":2.6739999999999999,"15min":9.2680000000000007},"last":0.61399999999999999}]},{"osd":3,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57499999999999996,"15min":0.54200000000000004},"min":{"1min":0.33000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":1.2030000000000001,"5min":1.9910000000000001,"15min":3.0539999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57999999999999996,"15min":0.57799999999999996},"min":{"1min":0.433,"5min":0.34200000000000003,"15min":0.313},"max":{"1min":0.84999999999999998,"5min":1.4119999999999999,"15min":6.9409999999999998},"last":0.69599999999999995}]},{"osd":4,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.68300000000000005,"15min":0.73199999999999998},"min":{"1min":0.34300000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.90400000000000003,"5min":1.962,"15min":5.6630000000000003},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.72399999999999998,"15min":0.75900000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.151,"5min":1.6719999999999999,"15min":8.8960000000000008},"last":0.873}]},{"osd":5,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.76000000000000001,"15min":0.77700000000000002},"min":{"1min":0.48399999999999999,"5min":0.47299999999999998,"15min":0.40300000000000002},"max":{"1min":1.242,"5min":1.242,"15min":6.6159999999999997},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.77800000000000002,"15min":0.79500000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.1220000000000001,"5min":1.7130000000000001,"15min":5.7009999999999996},"last":0.999}]},{"osd":6,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.73299999999999998,"15min":0.77300000000000002},"min":{"1min":0.47799999999999998,"5min":0.47299999999999998,"15min":0.39800000000000002},"max":{"1min":0.94199999999999995,"5min":2.0529999999999999,"15min":8.6039999999999992},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.77200000000000002,"15min":0.75},"min":{"1min":0.50700000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98199999999999998,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":0.90000000000000002}]},{"osd":7,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76300000000000001,"15min":0.77800000000000002},"min":{"1min":0.46899999999999997,"5min":0.45200000000000001,"15min":0.44800000000000001},"max":{"1min":1.1639999999999999,"5min":1.7470000000000001,"15min":6.6600000000000001},"last":1.038},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.78700000000000003,"15min":0.80600000000000005},"min":{"1min":0.50700000000000001,"5min":0.46400000000000002,"15min":0.46200000000000002},"max":{"1min":1.1990000000000001,"5min":1.774,"15min":8.8100000000000005},"last":0.95399999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575121,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371112,"kb_used_data":259424,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94000728,"statfs":{"total":96636764160,"available":96256745472,"internally_reserved":0,"allocated":265650176,"data_stored":265251991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.39500000000000002,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.56299999999999994,"5min":1.3029999999999999,"15min":6.8170000000000002},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.46100000000000002,"15min":0.47499999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.70199999999999996,"5min":1.591,"15min":6.2789999999999999},"last":0.502}]},{"osd":2,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.51800000000000002,"15min":0.495},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90100000000000002,"5min":1.169,"15min":3.1859999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.47699999999999998,"15min":0.46500000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.97499999999999998,"5min":1.266,"15min":1.47},"last":0.55400000000000005}]},{"osd":3,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56999999999999995,"15min":0.52100000000000002},"min":{"1min":0.40899999999999997,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.67900000000000005,"5min":1.423,"15min":1.8180000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.35599999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.93300000000000005,"5min":1.4390000000000001,"15min":1.881},"last":0.66400000000000003}]},{"osd":4,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.68799999999999994,"15min":0.80700000000000005},"min":{"1min":0.46400000000000002,"5min":0.45100000000000001,"15min":0.45000000000000001},"max":{"1min":1.0529999999999999,"5min":1.3169999999999999,"15min":18.41},"last":1.056},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70399999999999996,"15min":0.81399999999999995},"min":{"1min":0.50800000000000001,"5min":0.46300000000000002,"15min":0.42399999999999999},"max":{"1min":1.097,"5min":1.409,"15min":18.998999999999999},"last":1.083}]},{"osd":5,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73699999999999999,"15min":0.76100000000000001},"min":{"1min":0.52400000000000002,"5min":0.50900000000000001,"15min":0.42799999999999999},"max":{"1min":1.1299999999999999,"5min":1.3460000000000001,"15min":3.5499999999999998},"last":1.004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69999999999999996,"15min":0.73699999999999999},"min":{"1min":0.45900000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.996,"5min":1.4279999999999999,"15min":3.605},"last":1.1299999999999999}]},{"osd":6,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.73999999999999999,"15min":0.754},"min":{"1min":0.52200000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0289999999999999,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.76600000000000001,"15min":0.78200000000000003},"min":{"1min":0.502,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0840000000000001,"5min":2.367,"15min":2.367},"last":1.2150000000000001}]},{"osd":7,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.49099999999999999},"max":{"1min":1.042,"5min":1.395,"15min":4.7839999999999998},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73999999999999999,"15min":0.77000000000000002},"min":{"1min":0.57399999999999995,"5min":0.53200000000000003,"15min":0.45500000000000002},"max":{"1min":0.995,"5min":1.3759999999999999,"15min":2.0110000000000001},"last":1.1699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39346176,"data_stored":39310758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51638272,"data_stored":51612794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18219008,"data_stored":18188380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":47730688,"data_stored":47703197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56045568,"data_stored":56018792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":44724224,"data_stored":44681836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40804352,"data_stored":40776010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":60243968,"data_stored":60206725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":224890880,"data_stored":224890880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":316989440,"data_stored":316989440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186216448,"data_stored":186216448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":320110592,"data_stored":320110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":240496640,"data_stored":240496640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":185778176,"data_stored":185778176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266395648,"data_stored":266395648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":226021376,"data_stored":226021376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T20:59:11.384 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.384 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.385 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.385 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.385 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.386 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.386 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.387 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.387 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.387 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.388 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.388 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.389 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.389 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.389 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.390 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.390 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.390 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.391 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.391 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.392 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.392 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.392 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.393 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.393 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.393 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.394 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.394 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.395 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.395 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.396 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.396 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.397 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.397 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.398 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.398 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.398 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.399 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.399 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.400 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.400 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.400 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.401 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.401 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.401 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.402 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.402 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.403 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.403 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.404 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.404 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.404 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.405 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.405 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.406 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.406 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.406 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.407 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2023-10-16T20:36:53.711958+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=53, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.407 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.407 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.408 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.408 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.409 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.409 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.410 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.410 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.410 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.411 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.412 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.412 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.412 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.413 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.413 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.414 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:11.414 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-16T20:59:31.415 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T20:59:31.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.557+0000 7fc10c805640 1 Processor -- start 2023-10-16T20:59:31.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.557+0000 7fc10c805640 1 -- start start 2023-10-16T20:59:31.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.557+0000 7fc10c805640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1080c9bb0 0x7fc1080c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:31.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.557+0000 7fc10c805640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc108153b50 con 0x7fc1080c41f0 2023-10-16T20:59:31.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.557+0000 7fc10c805640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1080cb2f0 con 0x7fc108153770 2023-10-16T20:59:31.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.557+0000 7fc10c805640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1080cc250 con 0x7fc1080c9bb0 2023-10-16T20:59:31.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.557+0000 7fc106d9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc1080c41f0 0x7fc1080cece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45820/0 (socket says 172.21.15.88:45820) 2023-10-16T20:59:31.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.557+0000 7fc106d9d640 1 -- 172.21.15.88:0/2471776318 learned_addr learned my addr 172.21.15.88:0/2471776318 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:59:31.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.557+0000 7fc0fdd9b640 1 --2- 172.21.15.88:0/2471776318 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1080c9bb0 0x7fc1080c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:31.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.557+0000 7fc105d9b640 1 -- 172.21.15.88:0/2471776318 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985575038 0 0) 0x7fc108153b50 con 0x7fc1080c41f0 2023-10-16T20:59:31.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.557+0000 7fc105d9b640 1 -- 172.21.15.88:0/2471776318 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0f4003530 con 0x7fc1080c41f0 2023-10-16T20:59:31.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.557+0000 7fc105d9b640 1 -- 172.21.15.88:0/2471776318 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865550470 0 0) 0x7fc1080cb2f0 con 0x7fc108153770 2023-10-16T20:59:31.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.557+0000 7fc105d9b640 1 -- 172.21.15.88:0/2471776318 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc108153b50 con 0x7fc108153770 2023-10-16T20:59:31.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.557+0000 7fc0fdd9b640 1 -- 172.21.15.88:0/2471776318 >> v1:172.21.15.191:6789/0 conn(0x7fc108153770 legacy=0x7fc108159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:31.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.557+0000 7fc0fdd9b640 1 -- 172.21.15.88:0/2471776318 >> v1:172.21.15.88:6789/0 conn(0x7fc1080c41f0 legacy=0x7fc1080cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:31.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.557+0000 7fc0fdd9b640 1 -- 172.21.15.88:0/2471776318 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1080cb5d0 con 0x7fc1080c9bb0 2023-10-16T20:59:31.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.561+0000 7fc0fdd9b640 1 --2- 172.21.15.88:0/2471776318 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1080c9bb0 0x7fc1080c3cb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc0f00097f0 tx=0x7fc0f001c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebcd6c9147f358ec server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:31.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.561+0000 7fc105d9b640 1 -- 172.21.15.88:0/2471776318 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f00ac630 con 0x7fc1080c9bb0 2023-10-16T20:59:31.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.561+0000 7fc105d9b640 1 -- 172.21.15.88:0/2471776318 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f00ac790 con 0x7fc1080c9bb0 2023-10-16T20:59:31.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.561+0000 7fc105d9b640 1 -- 172.21.15.88:0/2471776318 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f00acb20 con 0x7fc1080c9bb0 2023-10-16T20:59:31.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.561+0000 7fc10c805640 1 -- 172.21.15.88:0/2471776318 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1080c9bb0 msgr2=0x7fc1080c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:31.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.561+0000 7fc10c805640 1 --2- 172.21.15.88:0/2471776318 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1080c9bb0 0x7fc1080c3cb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc0f00097f0 tx=0x7fc0f001c4c0 comp rx=0 tx=0).stop 2023-10-16T20:59:31.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.561+0000 7fc10c805640 1 -- 172.21.15.88:0/2471776318 shutdown_connections 2023-10-16T20:59:31.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.561+0000 7fc10c805640 1 --2- 172.21.15.88:0/2471776318 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1080c9bb0 0x7fc1080c3cb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:31.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.561+0000 7fc10c805640 1 -- 172.21.15.88:0/2471776318 >> 172.21.15.88:0/2471776318 conn(0x7fc1080715c0 msgr2=0x7fc1080719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:31.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.561+0000 7fc10c805640 1 -- 172.21.15.88:0/2471776318 shutdown_connections 2023-10-16T20:59:31.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.561+0000 7fc10c805640 1 -- 172.21.15.88:0/2471776318 wait complete. 2023-10-16T20:59:31.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc10c805640 1 Processor -- start 2023-10-16T20:59:31.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc10c805640 1 -- start start 2023-10-16T20:59:31.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc10c805640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1080c41f0 0x7fc108142940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:31.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc10c805640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1080c9bb0 0x7fc10815bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:31.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc10c805640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc108153770 0x7fc108151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:31.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc10c805640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10815d440 con 0x7fc1080c9bb0 2023-10-16T20:59:31.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc10c805640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1081489e0 con 0x7fc108153770 2023-10-16T20:59:31.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc10c805640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1080c67c0 con 0x7fc1080c41f0 2023-10-16T20:59:31.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc0fdd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1080c9bb0 0x7fc10815bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:31.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc0fdd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1080c9bb0 0x7fc10815bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43670/0 (socket says 172.21.15.88:43670) 2023-10-16T20:59:31.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc0fdd9b640 1 -- 172.21.15.88:0/1862309971 learned_addr learned my addr 172.21.15.88:0/1862309971 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:31.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc106d9d640 1 --2- 172.21.15.88:0/1862309971 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc108153770 0x7fc108151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:31.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc10659c640 1 --2- 172.21.15.88:0/1862309971 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1080c41f0 0x7fc108142940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:31.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc0fdd9b640 1 -- 172.21.15.88:0/1862309971 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1080c41f0 msgr2=0x7fc108142940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc0fdd9b640 1 --2- 172.21.15.88:0/1862309971 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1080c41f0 0x7fc108142940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc0fdd9b640 1 -- 172.21.15.88:0/1862309971 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc108153770 msgr2=0x7fc108151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc0fdd9b640 1 --2- 172.21.15.88:0/1862309971 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc108153770 0x7fc108151dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc0fdd9b640 1 -- 172.21.15.88:0/1862309971 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f0009420 con 0x7fc1080c9bb0 2023-10-16T20:59:31.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc0fdd9b640 1 --2- 172.21.15.88:0/1862309971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1080c9bb0 0x7fc10815bc80 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fc0f00097f0 tx=0x7fc0f0019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:31.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.565+0000 7fc0ff7fe640 1 -- 172.21.15.88:0/1862309971 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f00b5610 con 0x7fc1080c9bb0 2023-10-16T20:59:31.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.569+0000 7fc0ff7fe640 1 -- 172.21.15.88:0/1862309971 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f00b5770 con 0x7fc1080c9bb0 2023-10-16T20:59:31.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.569+0000 7fc0ff7fe640 1 -- 172.21.15.88:0/1862309971 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f00b5b20 con 0x7fc1080c9bb0 2023-10-16T20:59:31.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.569+0000 7fc10c805640 1 -- 172.21.15.88:0/1862309971 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1081496f0 con 0x7fc1080c9bb0 2023-10-16T20:59:31.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.569+0000 7fc10c805640 1 -- 172.21.15.88:0/1862309971 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc10815a740 con 0x7fc1080c9bb0 2023-10-16T20:59:31.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.569+0000 7fc10c805640 1 -- 172.21.15.88:0/1862309971 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc10815a5a0 con 0x7fc1080c9bb0 2023-10-16T20:59:31.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.569+0000 7fc0ff7fe640 1 -- 172.21.15.88:0/1862309971 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc0f00b3040 con 0x7fc1080c9bb0 2023-10-16T20:59:31.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.569+0000 7fc0ff7fe640 1 --2- 172.21.15.88:0/1862309971 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0e80664a0 0x7fc0e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:31.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.569+0000 7fc0ff7fe640 1 -- 172.21.15.88:0/1862309971 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7fc0f011e2a0 con 0x7fc1080c9bb0 2023-10-16T20:59:31.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.569+0000 7fc10659c640 1 --2- 172.21.15.88:0/1862309971 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0e80664a0 0x7fc0e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:31.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.573+0000 7fc10659c640 1 --2- 172.21.15.88:0/1862309971 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0e80664a0 0x7fc0e8068960 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fc0f80015c0 tx=0x7fc0f8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:31.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.573+0000 7fc0ff7fe640 1 -- 172.21.15.88:0/1862309971 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc10815a5a0 con 0x7fc1080c9bb0 2023-10-16T20:59:31.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.725+0000 7fc10c805640 1 -- 172.21.15.88:0/1862309971 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1080cbe70 con 0x7fc0e80664a0 2023-10-16T20:59:31.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.733+0000 7fc0ff7fe640 1 -- 172.21.15.88:0/1862309971 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+376041 (secure 0 0 0) 0x7fc0f800fe00 con 0x7fc0e80664a0 2023-10-16T20:59:31.738 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:59:31.740 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-16T20:59:31.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.741+0000 7fc10c805640 1 -- 172.21.15.88:0/1862309971 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0e80664a0 msgr2=0x7fc0e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:31.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.741+0000 7fc10c805640 1 --2- 172.21.15.88:0/1862309971 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0e80664a0 0x7fc0e8068960 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fc0f80015c0 tx=0x7fc0f8009040 comp rx=0 tx=0).stop 2023-10-16T20:59:31.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.741+0000 7fc10c805640 1 -- 172.21.15.88:0/1862309971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1080c9bb0 msgr2=0x7fc10815bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:31.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.741+0000 7fc10c805640 1 --2- 172.21.15.88:0/1862309971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1080c9bb0 0x7fc10815bc80 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fc0f00097f0 tx=0x7fc0f0019260 comp rx=0 tx=0).stop 2023-10-16T20:59:31.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.741+0000 7fc10c805640 1 -- 172.21.15.88:0/1862309971 shutdown_connections 2023-10-16T20:59:31.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.741+0000 7fc10c805640 1 --2- 172.21.15.88:0/1862309971 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc0e80664a0 0x7fc0e8068960 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:31.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.741+0000 7fc10c805640 1 --2- 172.21.15.88:0/1862309971 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc108153770 0x7fc108151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:31.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.741+0000 7fc10c805640 1 --2- 172.21.15.88:0/1862309971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1080c9bb0 0x7fc10815bc80 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:31.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.741+0000 7fc10c805640 1 --2- 172.21.15.88:0/1862309971 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc1080c41f0 0x7fc108142940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:31.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.741+0000 7fc10c805640 1 -- 172.21.15.88:0/1862309971 >> 172.21.15.88:0/1862309971 conn(0x7fc1080715c0 msgr2=0x7fc1080b3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:31.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.741+0000 7fc10c805640 1 -- 172.21.15.88:0/1862309971 shutdown_connections 2023-10-16T20:59:31.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:31.741+0000 7fc10c805640 1 -- 172.21.15.88:0/1862309971 wait complete. 2023-10-16T20:59:31.753 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:59:31.757 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T20:59:31.759 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":745,"stamp":"2023-10-16T20:59:29.943729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1104211726,"num_objects":10494,"num_object_clones":0,"num_object_copies":41576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10494,"num_whiteouts":0,"num_read":31781,"num_read_kb":882609,"num_write":27536,"num_write_kb":1486107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":115,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":468586,"num_omap_keys":892,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34249,"ondisk_log_size":34249,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3215384,"kb_used_data":2278808,"kb_used_omap":12,"kb_used_meta":936499,"kb_avail":751759336,"statfs":{"total":773094113280,"available":769801560064,"internally_reserved":0,"allocated":2333499392,"data_stored":2330363598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":958975545},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16998,"num_omap_keys":32,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003292"},"pg_stats":[{"pgid":"2.1f","version":"67'338","reported_seq":435,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:58.690133+0000","last_change":"2023-10-16T20:58:53.179067+0000","last_active":"2023-10-16T20:58:58.690133+0000","last_peered":"2023-10-16T20:58:58.690133+0000","last_clean":"2023-10-16T20:58:58.690133+0000","last_became_active":"2023-10-16T20:36:54.763457+0000","last_became_peered":"2023-10-16T20:36:54.763457+0000","last_unstale":"2023-10-16T20:58:58.690133+0000","last_undegraded":"2023-10-16T20:58:58.690133+0000","last_fullsized":"2023-10-16T20:58:58.690133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'334","last_scrub_stamp":"2023-10-16T20:58:53.178846+0000","last_deep_scrub":"67'334","last_deep_scrub_stamp":"2023-10-16T20:58:53.178846+0000","last_clean_scrub_stamp":"2023-10-16T20:58:53.178846+0000","objects_scrubbed":12,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:41:18.102491+0000","scrub_duration":0.029593793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9874237,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":42,"num_read_kb":9548,"num_write":328,"num_write_kb":9838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":24974,"num_omap_keys":48,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"67'107","reported_seq":2227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:23.206008+0000","last_change":"2023-10-16T20:59:23.206053+0000","last_active":"2023-10-16T20:59:23.206008+0000","last_peered":"2023-10-16T20:59:23.206008+0000","last_clean":"2023-10-16T20:59:23.206008+0000","last_became_active":"2023-10-16T20:36:55.302519+0000","last_became_peered":"2023-10-16T20:36:55.302519+0000","last_unstale":"2023-10-16T20:59:23.206008+0000","last_undegraded":"2023-10-16T20:59:23.206008+0000","last_fullsized":"2023-10-16T20:59:23.206008+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'107","last_scrub_stamp":"2023-10-16T20:59:23.205910+0000","last_deep_scrub":"67'107","last_deep_scrub_stamp":"2023-10-16T20:59:23.205910+0000","last_clean_scrub_stamp":"2023-10-16T20:59:23.205910+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:09.164505+0000","scrub_duration":0.0026049739999999999,"objects_trimmed":0,"snaptrim_duration":0.007784165,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'672","reported_seq":3558,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:47.854072+0000","last_change":"2023-10-16T20:47:31.584576+0000","last_active":"2023-10-16T20:58:47.854072+0000","last_peered":"2023-10-16T20:58:47.854072+0000","last_clean":"2023-10-16T20:58:47.854072+0000","last_became_active":"2023-10-16T20:36:59.466306+0000","last_became_peered":"2023-10-16T20:36:59.466306+0000","last_unstale":"2023-10-16T20:58:47.854072+0000","last_undegraded":"2023-10-16T20:58:47.854072+0000","last_fullsized":"2023-10-16T20:58:47.854072+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073106899999999999,"stat_sum":{"num_bytes":36865036,"num_objects":299,"num_object_clones":0,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":299,"num_whiteouts":0,"num_read":779,"num_read_kb":25449,"num_write":671,"num_write_kb":56816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"67'210","reported_seq":290,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:08.200626+0000","last_change":"2023-10-16T20:59:08.200674+0000","last_active":"2023-10-16T20:59:08.200626+0000","last_peered":"2023-10-16T20:59:08.200626+0000","last_clean":"2023-10-16T20:59:08.200626+0000","last_became_active":"2023-10-16T20:36:54.759177+0000","last_became_peered":"2023-10-16T20:36:54.759177+0000","last_unstale":"2023-10-16T20:59:08.200626+0000","last_undegraded":"2023-10-16T20:59:08.200626+0000","last_fullsized":"2023-10-16T20:59:08.200626+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'210","last_scrub_stamp":"2023-10-16T20:59:08.200542+0000","last_deep_scrub":"67'210","last_deep_scrub_stamp":"2023-10-16T20:59:08.200542+0000","last_clean_scrub_stamp":"2023-10-16T20:59:08.200542+0000","objects_scrubbed":3,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:06:26.078845+0000","scrub_duration":0.024305964999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":12288,"num_write":199,"num_write_kb":8312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9282,"num_omap_keys":17,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"67'107","reported_seq":2226,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:16.207741+0000","last_change":"2023-10-16T20:59:16.207806+0000","last_active":"2023-10-16T20:59:16.207741+0000","last_peered":"2023-10-16T20:59:16.207741+0000","last_clean":"2023-10-16T20:59:16.207741+0000","last_became_active":"2023-10-16T20:36:55.291458+0000","last_became_peered":"2023-10-16T20:36:55.291458+0000","last_unstale":"2023-10-16T20:59:16.207741+0000","last_undegraded":"2023-10-16T20:59:16.207741+0000","last_fullsized":"2023-10-16T20:59:16.207741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'107","last_scrub_stamp":"2023-10-16T20:59:16.207636+0000","last_deep_scrub":"67'107","last_deep_scrub_stamp":"2023-10-16T20:59:16.207636+0000","last_clean_scrub_stamp":"2023-10-16T20:59:16.207636+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:09:59.716717+0000","scrub_duration":0.0019653269999999998,"objects_trimmed":0,"snaptrim_duration":0.0087429459999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'682","reported_seq":3572,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:47.660250+0000","last_change":"2023-10-16T20:47:31.484150+0000","last_active":"2023-10-16T20:58:47.660250+0000","last_peered":"2023-10-16T20:58:47.660250+0000","last_clean":"2023-10-16T20:58:47.660250+0000","last_became_active":"2023-10-16T20:36:59.302968+0000","last_became_peered":"2023-10-16T20:36:59.302968+0000","last_unstale":"2023-10-16T20:58:47.660250+0000","last_undegraded":"2023-10-16T20:58:47.660250+0000","last_fullsized":"2023-10-16T20:58:47.660250+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027985639999999999,"stat_sum":{"num_bytes":24765378,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":783,"num_read_kb":25781,"num_write":678,"num_write_kb":40921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"67'351","reported_seq":458,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:16.994190+0000","last_change":"2023-10-16T20:59:16.994190+0000","last_active":"2023-10-16T20:59:16.994190+0000","last_peered":"2023-10-16T20:59:16.994190+0000","last_clean":"2023-10-16T20:59:16.994190+0000","last_became_active":"2023-10-16T20:36:54.762278+0000","last_became_peered":"2023-10-16T20:36:54.762278+0000","last_unstale":"2023-10-16T20:59:16.994190+0000","last_undegraded":"2023-10-16T20:59:16.994190+0000","last_fullsized":"2023-10-16T20:59:16.994190+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'351","last_scrub_stamp":"2023-10-16T20:59:16.994034+0000","last_deep_scrub":"67'351","last_deep_scrub_stamp":"2023-10-16T20:59:16.994034+0000","last_clean_scrub_stamp":"2023-10-16T20:59:16.994034+0000","objects_scrubbed":11,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:42:30.755177+0000","scrub_duration":0.038213219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":85,"num_read_kb":16492,"num_write":374,"num_write_kb":12523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10532,"num_omap_keys":21,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'96","reported_seq":2206,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:18.558055+0000","last_change":"2023-10-16T20:59:18.558106+0000","last_active":"2023-10-16T20:59:18.558055+0000","last_peered":"2023-10-16T20:59:18.558055+0000","last_clean":"2023-10-16T20:59:18.558055+0000","last_became_active":"2023-10-16T20:37:37.402987+0000","last_became_peered":"2023-10-16T20:37:37.402987+0000","last_unstale":"2023-10-16T20:59:18.558055+0000","last_undegraded":"2023-10-16T20:59:18.558055+0000","last_fullsized":"2023-10-16T20:59:18.558055+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'96","last_scrub_stamp":"2023-10-16T20:59:18.557959+0000","last_deep_scrub":"67'96","last_deep_scrub_stamp":"2023-10-16T20:59:18.557959+0000","last_clean_scrub_stamp":"2023-10-16T20:59:18.557959+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:00.882012+0000","scrub_duration":0.0024629650000000001,"objects_trimmed":0,"snaptrim_duration":0.00082573900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"67'673","reported_seq":4245,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:22.287023+0000","last_change":"2023-10-16T20:59:22.287085+0000","last_active":"2023-10-16T20:59:22.287023+0000","last_peered":"2023-10-16T20:59:22.287023+0000","last_clean":"2023-10-16T20:59:22.287023+0000","last_became_active":"2023-10-16T20:36:59.303805+0000","last_became_peered":"2023-10-16T20:36:59.303805+0000","last_unstale":"2023-10-16T20:59:22.287023+0000","last_undegraded":"2023-10-16T20:59:22.287023+0000","last_fullsized":"2023-10-16T20:59:22.287023+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'673","last_scrub_stamp":"2023-10-16T20:59:22.286926+0000","last_deep_scrub":"67'673","last_deep_scrub_stamp":"2023-10-16T20:59:22.286926+0000","last_clean_scrub_stamp":"2023-10-16T20:59:22.286926+0000","objects_scrubbed":304,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:33.137048+0000","scrub_duration":0.12974059900000001,"objects_trimmed":0,"snaptrim_duration":0.0041657500000000002,"stat_sum":{"num_bytes":49691093,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":1446,"num_read_kb":31388,"num_write":673,"num_write_kb":52938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"67'178","reported_seq":268,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:09.911993+0000","last_change":"2023-10-16T20:59:09.911993+0000","last_active":"2023-10-16T20:59:09.911993+0000","last_peered":"2023-10-16T20:59:09.911993+0000","last_clean":"2023-10-16T20:59:09.911993+0000","last_became_active":"2023-10-16T20:36:54.749316+0000","last_became_peered":"2023-10-16T20:36:54.749316+0000","last_unstale":"2023-10-16T20:59:09.911993+0000","last_undegraded":"2023-10-16T20:59:09.911993+0000","last_fullsized":"2023-10-16T20:59:09.911993+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'178","last_scrub_stamp":"2023-10-16T20:59:09.911761+0000","last_deep_scrub":"67'178","last_deep_scrub_stamp":"2023-10-16T20:59:09.911761+0000","last_clean_scrub_stamp":"2023-10-16T20:59:09.911761+0000","objects_scrubbed":6,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:22.957274+0000","scrub_duration":0.018582283000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":170,"num_write_kb":4198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20168,"num_omap_keys":38,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"67'104","reported_seq":2222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:14.206196+0000","last_change":"2023-10-16T20:59:14.206196+0000","last_active":"2023-10-16T20:59:14.206196+0000","last_peered":"2023-10-16T20:59:14.206196+0000","last_clean":"2023-10-16T20:59:14.206196+0000","last_became_active":"2023-10-16T20:36:55.302387+0000","last_became_peered":"2023-10-16T20:36:55.302387+0000","last_unstale":"2023-10-16T20:59:14.206196+0000","last_undegraded":"2023-10-16T20:59:14.206196+0000","last_fullsized":"2023-10-16T20:59:14.206196+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'104","last_scrub_stamp":"2023-10-16T20:59:14.206041+0000","last_deep_scrub":"67'104","last_deep_scrub_stamp":"2023-10-16T20:59:14.206041+0000","last_clean_scrub_stamp":"2023-10-16T20:59:14.206041+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:30.710991+0000","scrub_duration":0.0028641040000000001,"objects_trimmed":0,"snaptrim_duration":0.011585343,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'641","reported_seq":3860,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:16.317006+0000","last_change":"2023-10-16T20:59:16.317087+0000","last_active":"2023-10-16T20:59:16.317006+0000","last_peered":"2023-10-16T20:59:16.317006+0000","last_clean":"2023-10-16T20:59:16.317006+0000","last_became_active":"2023-10-16T20:36:59.305228+0000","last_became_peered":"2023-10-16T20:36:59.305228+0000","last_unstale":"2023-10-16T20:59:16.317006+0000","last_undegraded":"2023-10-16T20:59:16.317006+0000","last_fullsized":"2023-10-16T20:59:16.317006+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'641","last_scrub_stamp":"2023-10-16T20:59:16.316909+0000","last_deep_scrub":"67'641","last_deep_scrub_stamp":"2023-10-16T20:59:16.316909+0000","last_clean_scrub_stamp":"2023-10-16T20:59:16.316909+0000","objects_scrubbed":310,"log_size":641,"log_dups_size":0,"ondisk_log_size":641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:57.953725+0000","scrub_duration":0.12335723,"objects_trimmed":0,"snaptrim_duration":0.0047877249999999996,"stat_sum":{"num_bytes":34900728,"num_objects":310,"num_object_clones":0,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":1093,"num_read_kb":21879,"num_write":638,"num_write_kb":46703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"67'104","reported_seq":2223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:18.508850+0000","last_change":"2023-10-16T20:59:18.508908+0000","last_active":"2023-10-16T20:59:18.508850+0000","last_peered":"2023-10-16T20:59:18.508850+0000","last_clean":"2023-10-16T20:59:18.508850+0000","last_became_active":"2023-10-16T20:36:55.306427+0000","last_became_peered":"2023-10-16T20:36:55.306427+0000","last_unstale":"2023-10-16T20:59:18.508850+0000","last_undegraded":"2023-10-16T20:59:18.508850+0000","last_fullsized":"2023-10-16T20:59:18.508850+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'104","last_scrub_stamp":"2023-10-16T20:59:18.508734+0000","last_deep_scrub":"67'104","last_deep_scrub_stamp":"2023-10-16T20:59:18.508734+0000","last_clean_scrub_stamp":"2023-10-16T20:59:18.508734+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:58:43.422816+0000","scrub_duration":0.0019135370000000001,"objects_trimmed":0,"snaptrim_duration":0.013854674000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"67'191","reported_seq":272,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:10.947598+0000","last_change":"2023-10-16T20:59:10.947598+0000","last_active":"2023-10-16T20:59:10.947598+0000","last_peered":"2023-10-16T20:59:10.947598+0000","last_clean":"2023-10-16T20:59:10.947598+0000","last_became_active":"2023-10-16T20:36:54.767996+0000","last_became_peered":"2023-10-16T20:36:54.767996+0000","last_unstale":"2023-10-16T20:59:10.947598+0000","last_undegraded":"2023-10-16T20:59:10.947598+0000","last_fullsized":"2023-10-16T20:59:10.947598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'191","last_scrub_stamp":"2023-10-16T20:59:10.947456+0000","last_deep_scrub":"67'191","last_deep_scrub_stamp":"2023-10-16T20:59:10.947456+0000","last_clean_scrub_stamp":"2023-10-16T20:59:10.947456+0000","objects_scrubbed":4,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:10.801672+0000","scrub_duration":0.024723371000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":8202,"num_write":173,"num_write_kb":8293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"67'648","reported_seq":4474,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.017793+0000","last_change":"2023-10-16T20:47:31.487925+0000","last_active":"2023-10-16T20:58:23.017793+0000","last_peered":"2023-10-16T20:58:23.017793+0000","last_clean":"2023-10-16T20:58:23.017793+0000","last_became_active":"2023-10-16T20:36:59.306640+0000","last_became_peered":"2023-10-16T20:36:59.306640+0000","last_unstale":"2023-10-16T20:58:23.017793+0000","last_undegraded":"2023-10-16T20:58:23.017793+0000","last_fullsized":"2023-10-16T20:58:23.017793+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054537470000000001,"stat_sum":{"num_bytes":41205181,"num_objects":318,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":1719,"num_read_kb":37802,"num_write":648,"num_write_kb":48776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"67'101","reported_seq":2220,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:14.250365+0000","last_change":"2023-10-16T20:59:14.250365+0000","last_active":"2023-10-16T20:59:14.250365+0000","last_peered":"2023-10-16T20:59:14.250365+0000","last_clean":"2023-10-16T20:59:14.250365+0000","last_became_active":"2023-10-16T20:36:55.275085+0000","last_became_peered":"2023-10-16T20:36:55.275085+0000","last_unstale":"2023-10-16T20:59:14.250365+0000","last_undegraded":"2023-10-16T20:59:14.250365+0000","last_fullsized":"2023-10-16T20:59:14.250365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'101","last_scrub_stamp":"2023-10-16T20:59:14.250203+0000","last_deep_scrub":"67'101","last_deep_scrub_stamp":"2023-10-16T20:59:14.250203+0000","last_clean_scrub_stamp":"2023-10-16T20:59:14.250203+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:20:58.657625+0000","scrub_duration":0.001852245,"objects_trimmed":0,"snaptrim_duration":0.0087599630000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'183","reported_seq":261,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:57.085209+0000","last_change":"2023-10-16T20:58:57.085283+0000","last_active":"2023-10-16T20:58:57.085209+0000","last_peered":"2023-10-16T20:58:57.085209+0000","last_clean":"2023-10-16T20:58:57.085209+0000","last_became_active":"2023-10-16T20:36:54.755872+0000","last_became_peered":"2023-10-16T20:36:54.755872+0000","last_unstale":"2023-10-16T20:58:57.085209+0000","last_undegraded":"2023-10-16T20:58:57.085209+0000","last_fullsized":"2023-10-16T20:58:57.085209+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'183","last_scrub_stamp":"2023-10-16T20:58:57.085118+0000","last_deep_scrub":"67'183","last_deep_scrub_stamp":"2023-10-16T20:58:57.085118+0000","last_clean_scrub_stamp":"2023-10-16T20:58:57.085118+0000","objects_scrubbed":3,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:40:14.300010+0000","scrub_duration":0.017757339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":10,"num_write":281,"num_write_kb":4257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"67'699","reported_seq":3476,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:26.358137+0000","last_change":"2023-10-16T20:59:26.358241+0000","last_active":"2023-10-16T20:59:26.358137+0000","last_peered":"2023-10-16T20:59:26.358137+0000","last_clean":"2023-10-16T20:59:26.358137+0000","last_became_active":"2023-10-16T20:36:59.302919+0000","last_became_peered":"2023-10-16T20:36:59.302919+0000","last_unstale":"2023-10-16T20:59:26.358137+0000","last_undegraded":"2023-10-16T20:59:26.358137+0000","last_fullsized":"2023-10-16T20:59:26.358137+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'699","last_scrub_stamp":"2023-10-16T20:59:26.358029+0000","last_deep_scrub":"67'699","last_deep_scrub_stamp":"2023-10-16T20:59:26.358029+0000","last_clean_scrub_stamp":"2023-10-16T20:59:26.358029+0000","objects_scrubbed":334,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:30:05.906486+0000","scrub_duration":0.129970374,"objects_trimmed":0,"snaptrim_duration":0.0046546790000000001,"stat_sum":{"num_bytes":20793675,"num_objects":334,"num_object_clones":0,"num_object_copies":1336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":334,"num_whiteouts":0,"num_read":649,"num_read_kb":16394,"num_write":696,"num_write_kb":45235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"67'106","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:19.261082+0000","last_change":"2023-10-16T20:59:19.261129+0000","last_active":"2023-10-16T20:59:19.261082+0000","last_peered":"2023-10-16T20:59:19.261082+0000","last_clean":"2023-10-16T20:59:19.261082+0000","last_became_active":"2023-10-16T20:36:55.270455+0000","last_became_peered":"2023-10-16T20:36:55.270455+0000","last_unstale":"2023-10-16T20:59:19.261082+0000","last_undegraded":"2023-10-16T20:59:19.261082+0000","last_fullsized":"2023-10-16T20:59:19.261082+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'106","last_scrub_stamp":"2023-10-16T20:59:19.261013+0000","last_deep_scrub":"67'106","last_deep_scrub_stamp":"2023-10-16T20:59:19.261013+0000","last_clean_scrub_stamp":"2023-10-16T20:59:19.261013+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:49:53.748721+0000","scrub_duration":0.0014257040000000001,"objects_trimmed":0,"snaptrim_duration":0.008577889,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"67'55","reported_seq":142,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:02.072253+0000","last_change":"2023-10-16T20:59:02.072347+0000","last_active":"2023-10-16T20:59:02.072253+0000","last_peered":"2023-10-16T20:59:02.072253+0000","last_clean":"2023-10-16T20:59:02.072253+0000","last_became_active":"2023-10-16T20:36:54.743805+0000","last_became_peered":"2023-10-16T20:36:54.743805+0000","last_unstale":"2023-10-16T20:59:02.072253+0000","last_undegraded":"2023-10-16T20:59:02.072253+0000","last_fullsized":"2023-10-16T20:59:02.072253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'55","last_scrub_stamp":"2023-10-16T20:59:02.072155+0000","last_deep_scrub":"67'55","last_deep_scrub_stamp":"2023-10-16T20:59:02.072155+0000","last_clean_scrub_stamp":"2023-10-16T20:59:02.072155+0000","objects_scrubbed":6,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:40.321154+0000","scrub_duration":0.011965646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":55,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":19110,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'651","reported_seq":3713,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:22.025771+0000","last_change":"2023-10-16T20:59:22.025877+0000","last_active":"2023-10-16T20:59:22.025771+0000","last_peered":"2023-10-16T20:59:22.025771+0000","last_clean":"2023-10-16T20:59:22.025771+0000","last_became_active":"2023-10-16T20:36:59.465172+0000","last_became_peered":"2023-10-16T20:36:59.465172+0000","last_unstale":"2023-10-16T20:59:22.025771+0000","last_undegraded":"2023-10-16T20:59:22.025771+0000","last_fullsized":"2023-10-16T20:59:22.025771+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'651","last_scrub_stamp":"2023-10-16T20:59:22.025668+0000","last_deep_scrub":"67'651","last_deep_scrub_stamp":"2023-10-16T20:59:22.025668+0000","last_clean_scrub_stamp":"2023-10-16T20:59:22.025668+0000","objects_scrubbed":314,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:10.904294+0000","scrub_duration":0.12543077799999999,"objects_trimmed":0,"snaptrim_duration":0.0040107390000000001,"stat_sum":{"num_bytes":25409516,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":935,"num_read_kb":27643,"num_write":650,"num_write_kb":33262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"67'103","reported_seq":2222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:08.331174+0000","last_change":"2023-10-16T20:59:08.331222+0000","last_active":"2023-10-16T20:59:08.331174+0000","last_peered":"2023-10-16T20:59:08.331174+0000","last_clean":"2023-10-16T20:59:08.331174+0000","last_became_active":"2023-10-16T20:36:55.278568+0000","last_became_peered":"2023-10-16T20:36:55.278568+0000","last_unstale":"2023-10-16T20:59:08.331174+0000","last_undegraded":"2023-10-16T20:59:08.331174+0000","last_fullsized":"2023-10-16T20:59:08.331174+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'103","last_scrub_stamp":"2023-10-16T20:59:08.331083+0000","last_deep_scrub":"67'103","last_deep_scrub_stamp":"2023-10-16T20:59:08.331083+0000","last_clean_scrub_stamp":"2023-10-16T20:59:08.331083+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:39:29.469121+0000","scrub_duration":0.0021561620000000001,"objects_trimmed":0,"snaptrim_duration":0.0050399599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"67'293","reported_seq":373,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:55.917267+0000","last_change":"2023-10-16T20:58:55.917322+0000","last_active":"2023-10-16T20:58:55.917267+0000","last_peered":"2023-10-16T20:58:55.917267+0000","last_clean":"2023-10-16T20:58:55.917267+0000","last_became_active":"2023-10-16T20:36:54.758496+0000","last_became_peered":"2023-10-16T20:36:54.758496+0000","last_unstale":"2023-10-16T20:58:55.917267+0000","last_undegraded":"2023-10-16T20:58:55.917267+0000","last_fullsized":"2023-10-16T20:58:55.917267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'293","last_scrub_stamp":"2023-10-16T20:58:55.917187+0000","last_deep_scrub":"67'293","last_deep_scrub_stamp":"2023-10-16T20:58:55.917187+0000","last_clean_scrub_stamp":"2023-10-16T20:58:55.917187+0000","objects_scrubbed":6,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:17:21.018502+0000","scrub_duration":0.017810975999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":12294,"num_write":291,"num_write_kb":4310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":34642,"num_omap_keys":66,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"67'586","reported_seq":3862,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.663154+0000","last_change":"2023-10-16T20:47:31.488200+0000","last_active":"2023-10-16T20:58:22.663154+0000","last_peered":"2023-10-16T20:58:22.663154+0000","last_clean":"2023-10-16T20:58:22.663154+0000","last_became_active":"2023-10-16T20:36:59.309068+0000","last_became_peered":"2023-10-16T20:36:59.309068+0000","last_unstale":"2023-10-16T20:58:22.663154+0000","last_undegraded":"2023-10-16T20:58:22.663154+0000","last_fullsized":"2023-10-16T20:58:22.663154+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055482439999999999,"stat_sum":{"num_bytes":36900597,"num_objects":303,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1169,"num_read_kb":30992,"num_write":585,"num_write_kb":40444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'101","reported_seq":2220,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:14.223878+0000","last_change":"2023-10-16T20:59:14.223927+0000","last_active":"2023-10-16T20:59:14.223878+0000","last_peered":"2023-10-16T20:59:14.223878+0000","last_clean":"2023-10-16T20:59:14.223878+0000","last_became_active":"2023-10-16T20:36:55.272840+0000","last_became_peered":"2023-10-16T20:36:55.272840+0000","last_unstale":"2023-10-16T20:59:14.223878+0000","last_undegraded":"2023-10-16T20:59:14.223878+0000","last_fullsized":"2023-10-16T20:59:14.223878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'101","last_scrub_stamp":"2023-10-16T20:59:14.223774+0000","last_deep_scrub":"67'101","last_deep_scrub_stamp":"2023-10-16T20:59:14.223774+0000","last_clean_scrub_stamp":"2023-10-16T20:59:14.223774+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:44:52.868862+0000","scrub_duration":0.0020408700000000002,"objects_trimmed":0,"snaptrim_duration":0.011897911000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"67'12","reported_seq":101,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:51.472708+0000","last_change":"2023-10-16T20:58:51.472802+0000","last_active":"2023-10-16T20:58:51.472708+0000","last_peered":"2023-10-16T20:58:51.472708+0000","last_clean":"2023-10-16T20:58:51.472708+0000","last_became_active":"2023-10-16T20:36:54.760397+0000","last_became_peered":"2023-10-16T20:36:54.760397+0000","last_unstale":"2023-10-16T20:58:51.472708+0000","last_undegraded":"2023-10-16T20:58:51.472708+0000","last_fullsized":"2023-10-16T20:58:51.472708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2023-10-16T20:58:51.472601+0000","last_deep_scrub":"67'12","last_deep_scrub_stamp":"2023-10-16T20:58:51.472601+0000","last_clean_scrub_stamp":"2023-10-16T20:58:51.472601+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:36.331637+0000","scrub_duration":0.0080542610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"67'710","reported_seq":3820,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:12.359908+0000","last_change":"2023-10-16T20:59:12.360004+0000","last_active":"2023-10-16T20:59:12.359908+0000","last_peered":"2023-10-16T20:59:12.359908+0000","last_clean":"2023-10-16T20:59:12.359908+0000","last_became_active":"2023-10-16T20:36:59.465152+0000","last_became_peered":"2023-10-16T20:36:59.465152+0000","last_unstale":"2023-10-16T20:59:12.359908+0000","last_undegraded":"2023-10-16T20:59:12.359908+0000","last_fullsized":"2023-10-16T20:59:12.359908+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'710","last_scrub_stamp":"2023-10-16T20:59:12.359810+0000","last_deep_scrub":"67'710","last_deep_scrub_stamp":"2023-10-16T20:59:12.359810+0000","last_clean_scrub_stamp":"2023-10-16T20:59:12.359810+0000","objects_scrubbed":344,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:51:44.766685+0000","scrub_duration":0.14617412099999999,"objects_trimmed":0,"snaptrim_duration":0.0044307399999999999,"stat_sum":{"num_bytes":25809469,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":981,"num_read_kb":15497,"num_write":706,"num_write_kb":50141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'106","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:26.290296+0000","last_change":"2023-10-16T20:59:26.290343+0000","last_active":"2023-10-16T20:59:26.290296+0000","last_peered":"2023-10-16T20:59:26.290296+0000","last_clean":"2023-10-16T20:59:26.290296+0000","last_became_active":"2023-10-16T20:36:55.285566+0000","last_became_peered":"2023-10-16T20:36:55.285566+0000","last_unstale":"2023-10-16T20:59:26.290296+0000","last_undegraded":"2023-10-16T20:59:26.290296+0000","last_fullsized":"2023-10-16T20:59:26.290296+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'106","last_scrub_stamp":"2023-10-16T20:59:26.290208+0000","last_deep_scrub":"67'106","last_deep_scrub_stamp":"2023-10-16T20:59:26.290208+0000","last_clean_scrub_stamp":"2023-10-16T20:59:26.290208+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:36:47.364832+0000","scrub_duration":0.0019590620000000001,"objects_trimmed":0,"snaptrim_duration":0.0085500479999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'104","reported_seq":189,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:01.068372+0000","last_change":"2023-10-16T20:59:01.068434+0000","last_active":"2023-10-16T20:59:01.068372+0000","last_peered":"2023-10-16T20:59:01.068372+0000","last_clean":"2023-10-16T20:59:01.068372+0000","last_became_active":"2023-10-16T20:36:54.757796+0000","last_became_peered":"2023-10-16T20:36:54.757796+0000","last_unstale":"2023-10-16T20:59:01.068372+0000","last_undegraded":"2023-10-16T20:59:01.068372+0000","last_fullsized":"2023-10-16T20:59:01.068372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'104","last_scrub_stamp":"2023-10-16T20:59:01.068280+0000","last_deep_scrub":"67'104","last_deep_scrub_stamp":"2023-10-16T20:59:01.068280+0000","last_clean_scrub_stamp":"2023-10-16T20:59:01.068280+0000","objects_scrubbed":4,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:13:55.968912+0000","scrub_duration":0.035122747000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":254,"num_write_kb":1037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"67'686","reported_seq":3333,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:48.521168+0000","last_change":"2023-10-16T20:47:31.582989+0000","last_active":"2023-10-16T20:58:48.521168+0000","last_peered":"2023-10-16T20:58:48.521168+0000","last_clean":"2023-10-16T20:58:48.521168+0000","last_became_active":"2023-10-16T20:36:59.465221+0000","last_became_peered":"2023-10-16T20:36:59.465221+0000","last_unstale":"2023-10-16T20:58:48.521168+0000","last_undegraded":"2023-10-16T20:58:48.521168+0000","last_fullsized":"2023-10-16T20:58:48.521168+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059334330000000001,"stat_sum":{"num_bytes":20326999,"num_objects":343,"num_object_clones":0,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":538,"num_read_kb":13756,"num_write":685,"num_write_kb":32496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"67'101","reported_seq":2220,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:10.927031+0000","last_change":"2023-10-16T20:59:10.927098+0000","last_active":"2023-10-16T20:59:10.927031+0000","last_peered":"2023-10-16T20:59:10.927031+0000","last_clean":"2023-10-16T20:59:10.927031+0000","last_became_active":"2023-10-16T20:36:55.274432+0000","last_became_peered":"2023-10-16T20:36:55.274432+0000","last_unstale":"2023-10-16T20:59:10.927031+0000","last_undegraded":"2023-10-16T20:59:10.927031+0000","last_fullsized":"2023-10-16T20:59:10.927031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'101","last_scrub_stamp":"2023-10-16T20:59:10.926915+0000","last_deep_scrub":"67'101","last_deep_scrub_stamp":"2023-10-16T20:59:10.926915+0000","last_clean_scrub_stamp":"2023-10-16T20:59:10.926915+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:40.067704+0000","scrub_duration":0.0018791190000000001,"objects_trimmed":0,"snaptrim_duration":0.0057461259999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"67'39","reported_seq":123,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:17.915784+0000","last_change":"2023-10-16T20:59:17.915835+0000","last_active":"2023-10-16T20:59:17.915784+0000","last_peered":"2023-10-16T20:59:17.915784+0000","last_clean":"2023-10-16T20:59:17.915784+0000","last_became_active":"2023-10-16T20:36:54.762167+0000","last_became_peered":"2023-10-16T20:36:54.762167+0000","last_unstale":"2023-10-16T20:59:17.915784+0000","last_undegraded":"2023-10-16T20:59:17.915784+0000","last_fullsized":"2023-10-16T20:59:17.915784+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'39","last_scrub_stamp":"2023-10-16T20:59:17.915708+0000","last_deep_scrub":"67'39","last_deep_scrub_stamp":"2023-10-16T20:59:17.915708+0000","last_clean_scrub_stamp":"2023-10-16T20:59:17.915708+0000","objects_scrubbed":5,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:18:11.275071+0000","scrub_duration":0.0057107759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":34,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16998,"num_omap_keys":32,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"67'725","reported_seq":3737,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:24.401605+0000","last_change":"2023-10-16T20:59:24.401720+0000","last_active":"2023-10-16T20:59:24.401605+0000","last_peered":"2023-10-16T20:59:24.401605+0000","last_clean":"2023-10-16T20:59:24.401605+0000","last_became_active":"2023-10-16T20:36:59.466876+0000","last_became_peered":"2023-10-16T20:36:59.466876+0000","last_unstale":"2023-10-16T20:59:24.401605+0000","last_undegraded":"2023-10-16T20:59:24.401605+0000","last_fullsized":"2023-10-16T20:59:24.401605+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'725","last_scrub_stamp":"2023-10-16T20:59:24.401499+0000","last_deep_scrub":"67'725","last_deep_scrub_stamp":"2023-10-16T20:59:24.401499+0000","last_clean_scrub_stamp":"2023-10-16T20:59:24.401499+0000","objects_scrubbed":340,"log_size":725,"log_dups_size":0,"ondisk_log_size":725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:16:44.535101+0000","scrub_duration":0.141220765,"objects_trimmed":0,"snaptrim_duration":0.009058969,"stat_sum":{"num_bytes":44901233,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":884,"num_read_kb":20657,"num_write":720,"num_write_kb":60576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"67'98","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:13.164714+0000","last_change":"2023-10-16T20:59:13.164786+0000","last_active":"2023-10-16T20:59:13.164714+0000","last_peered":"2023-10-16T20:59:13.164714+0000","last_clean":"2023-10-16T20:59:13.164714+0000","last_became_active":"2023-10-16T20:36:55.293228+0000","last_became_peered":"2023-10-16T20:36:55.293228+0000","last_unstale":"2023-10-16T20:59:13.164714+0000","last_undegraded":"2023-10-16T20:59:13.164714+0000","last_fullsized":"2023-10-16T20:59:13.164714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'98","last_scrub_stamp":"2023-10-16T20:59:13.164628+0000","last_deep_scrub":"67'98","last_deep_scrub_stamp":"2023-10-16T20:59:13.164628+0000","last_clean_scrub_stamp":"2023-10-16T20:59:13.164628+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:49:21.419640+0000","scrub_duration":0.001733166,"objects_trimmed":0,"snaptrim_duration":0.013567460999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"67'212","reported_seq":289,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:59.097576+0000","last_change":"2023-10-16T20:58:59.097630+0000","last_active":"2023-10-16T20:58:59.097576+0000","last_peered":"2023-10-16T20:58:59.097576+0000","last_clean":"2023-10-16T20:58:59.097576+0000","last_became_active":"2023-10-16T20:36:54.757308+0000","last_became_peered":"2023-10-16T20:36:54.757308+0000","last_unstale":"2023-10-16T20:58:59.097576+0000","last_undegraded":"2023-10-16T20:58:59.097576+0000","last_fullsized":"2023-10-16T20:58:59.097576+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'212","last_scrub_stamp":"2023-10-16T20:58:59.097485+0000","last_deep_scrub":"67'212","last_deep_scrub_stamp":"2023-10-16T20:58:59.097485+0000","last_clean_scrub_stamp":"2023-10-16T20:58:59.097485+0000","objects_scrubbed":2,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:26:44.246482+0000","scrub_duration":0.017169976999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":3049,"num_write":207,"num_write_kb":4195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9334,"num_omap_keys":18,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"67'671","reported_seq":3446,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:15.193409+0000","last_change":"2023-10-16T20:59:15.193523+0000","last_active":"2023-10-16T20:59:15.193409+0000","last_peered":"2023-10-16T20:59:15.193409+0000","last_clean":"2023-10-16T20:59:15.193409+0000","last_became_active":"2023-10-16T20:36:59.303224+0000","last_became_peered":"2023-10-16T20:36:59.303224+0000","last_unstale":"2023-10-16T20:59:15.193409+0000","last_undegraded":"2023-10-16T20:59:15.193409+0000","last_fullsized":"2023-10-16T20:59:15.193409+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'671","last_scrub_stamp":"2023-10-16T20:59:15.193311+0000","last_deep_scrub":"67'671","last_deep_scrub_stamp":"2023-10-16T20:59:15.193311+0000","last_clean_scrub_stamp":"2023-10-16T20:59:15.193311+0000","objects_scrubbed":323,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:37:00.217159+0000","scrub_duration":0.12305557,"objects_trimmed":0,"snaptrim_duration":0.0038422399999999998,"stat_sum":{"num_bytes":21817454,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":647,"num_read_kb":9306,"num_write":673,"num_write_kb":37999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"67'120","reported_seq":2239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:58.334392+0000","last_change":"2023-10-16T20:58:58.334443+0000","last_active":"2023-10-16T20:58:58.334392+0000","last_peered":"2023-10-16T20:58:58.334392+0000","last_clean":"2023-10-16T20:58:58.334392+0000","last_became_active":"2023-10-16T20:36:55.277872+0000","last_became_peered":"2023-10-16T20:36:55.277872+0000","last_unstale":"2023-10-16T20:58:58.334392+0000","last_undegraded":"2023-10-16T20:58:58.334392+0000","last_fullsized":"2023-10-16T20:58:58.334392+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'120","last_scrub_stamp":"2023-10-16T20:58:58.334264+0000","last_deep_scrub":"67'120","last_deep_scrub_stamp":"2023-10-16T20:58:58.334264+0000","last_clean_scrub_stamp":"2023-10-16T20:58:58.334264+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:35:22.356584+0000","scrub_duration":0.0023669569999999998,"objects_trimmed":0,"snaptrim_duration":0.0052323379999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"67'537","reported_seq":630,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:54.485905+0000","last_change":"2023-10-16T20:58:54.485966+0000","last_active":"2023-10-16T20:58:54.485905+0000","last_peered":"2023-10-16T20:58:54.485905+0000","last_clean":"2023-10-16T20:58:54.485905+0000","last_became_active":"2023-10-16T20:36:54.740340+0000","last_became_peered":"2023-10-16T20:36:54.740340+0000","last_unstale":"2023-10-16T20:58:54.485905+0000","last_undegraded":"2023-10-16T20:58:54.485905+0000","last_fullsized":"2023-10-16T20:58:54.485905+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'537","last_scrub_stamp":"2023-10-16T20:58:54.485786+0000","last_deep_scrub":"67'537","last_deep_scrub_stamp":"2023-10-16T20:58:54.485786+0000","last_clean_scrub_stamp":"2023-10-16T20:58:54.485786+0000","objects_scrubbed":5,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:18.710024+0000","scrub_duration":0.040724903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21285,"num_write":524,"num_write_kb":16645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'623","reported_seq":3898,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:21.069874+0000","last_change":"2023-10-16T20:59:21.069939+0000","last_active":"2023-10-16T20:59:21.069874+0000","last_peered":"2023-10-16T20:59:21.069874+0000","last_clean":"2023-10-16T20:59:21.069874+0000","last_became_active":"2023-10-16T20:36:59.294548+0000","last_became_peered":"2023-10-16T20:36:59.294548+0000","last_unstale":"2023-10-16T20:59:21.069874+0000","last_undegraded":"2023-10-16T20:59:21.069874+0000","last_fullsized":"2023-10-16T20:59:21.069874+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'623","last_scrub_stamp":"2023-10-16T20:59:21.069779+0000","last_deep_scrub":"67'623","last_deep_scrub_stamp":"2023-10-16T20:59:21.069779+0000","last_clean_scrub_stamp":"2023-10-16T20:59:21.069779+0000","objects_scrubbed":322,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:38.618092+0000","scrub_duration":0.12688919600000001,"objects_trimmed":0,"snaptrim_duration":0.0062280529999999999,"stat_sum":{"num_bytes":29651112,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":1147,"num_read_kb":23932,"num_write":623,"num_write_kb":40690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"67'89","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:04.003786+0000","last_change":"2023-10-16T20:59:04.003844+0000","last_active":"2023-10-16T20:59:04.003786+0000","last_peered":"2023-10-16T20:59:04.003786+0000","last_clean":"2023-10-16T20:59:04.003786+0000","last_became_active":"2023-10-16T20:36:55.269749+0000","last_became_peered":"2023-10-16T20:36:55.269749+0000","last_unstale":"2023-10-16T20:59:04.003786+0000","last_undegraded":"2023-10-16T20:59:04.003786+0000","last_fullsized":"2023-10-16T20:59:04.003786+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'89","last_scrub_stamp":"2023-10-16T20:59:04.003679+0000","last_deep_scrub":"67'89","last_deep_scrub_stamp":"2023-10-16T20:59:04.003679+0000","last_clean_scrub_stamp":"2023-10-16T20:59:04.003679+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:45:42.918860+0000","scrub_duration":0.0021523760000000001,"objects_trimmed":0,"snaptrim_duration":0.003009004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'198","reported_seq":395,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:55.620844+0000","last_change":"2023-10-16T20:58:55.620915+0000","last_active":"2023-10-16T20:58:55.620844+0000","last_peered":"2023-10-16T20:58:55.620844+0000","last_clean":"2023-10-16T20:58:55.620844+0000","last_became_active":"2023-10-16T20:36:54.757549+0000","last_became_peered":"2023-10-16T20:36:54.757549+0000","last_unstale":"2023-10-16T20:58:55.620844+0000","last_undegraded":"2023-10-16T20:58:55.620844+0000","last_fullsized":"2023-10-16T20:58:55.620844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'198","last_scrub_stamp":"2023-10-16T20:58:55.620756+0000","last_deep_scrub":"67'198","last_deep_scrub_stamp":"2023-10-16T20:58:55.620756+0000","last_clean_scrub_stamp":"2023-10-16T20:58:55.620756+0000","objects_scrubbed":10,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:40.908940+0000","scrub_duration":0.012899747,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1410402,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":124,"num_read_kb":239,"num_write":186,"num_write_kb":1570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":50816,"num_omap_keys":97,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'647","reported_seq":3954,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:19.082046+0000","last_change":"2023-10-16T20:59:19.082138+0000","last_active":"2023-10-16T20:59:19.082046+0000","last_peered":"2023-10-16T20:59:19.082046+0000","last_clean":"2023-10-16T20:59:19.082046+0000","last_became_active":"2023-10-16T20:36:59.301693+0000","last_became_peered":"2023-10-16T20:36:59.301693+0000","last_unstale":"2023-10-16T20:59:19.082046+0000","last_undegraded":"2023-10-16T20:59:19.082046+0000","last_fullsized":"2023-10-16T20:59:19.082046+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'647","last_scrub_stamp":"2023-10-16T20:59:19.081936+0000","last_deep_scrub":"67'647","last_deep_scrub_stamp":"2023-10-16T20:59:19.081936+0000","last_clean_scrub_stamp":"2023-10-16T20:59:19.081936+0000","objects_scrubbed":298,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:47:03.996136+0000","scrub_duration":0.124113616,"objects_trimmed":0,"snaptrim_duration":0.0056692870000000003,"stat_sum":{"num_bytes":29916858,"num_objects":298,"num_object_clones":0,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":298,"num_whiteouts":0,"num_read":1180,"num_read_kb":25235,"num_write":643,"num_write_kb":45892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"67'104","reported_seq":2223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:58.176470+0000","last_change":"2023-10-16T20:58:58.176549+0000","last_active":"2023-10-16T20:58:58.176470+0000","last_peered":"2023-10-16T20:58:58.176470+0000","last_clean":"2023-10-16T20:58:58.176470+0000","last_became_active":"2023-10-16T20:36:55.276649+0000","last_became_peered":"2023-10-16T20:36:55.276649+0000","last_unstale":"2023-10-16T20:58:58.176470+0000","last_undegraded":"2023-10-16T20:58:58.176470+0000","last_fullsized":"2023-10-16T20:58:58.176470+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'104","last_scrub_stamp":"2023-10-16T20:58:58.176362+0000","last_deep_scrub":"67'104","last_deep_scrub_stamp":"2023-10-16T20:58:58.176362+0000","last_clean_scrub_stamp":"2023-10-16T20:58:58.176362+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:39:24.099280+0000","scrub_duration":0.0024904129999999999,"objects_trimmed":0,"snaptrim_duration":0.0076292809999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'131","reported_seq":214,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:56.907025+0000","last_change":"2023-10-16T20:58:56.907025+0000","last_active":"2023-10-16T20:58:56.907025+0000","last_peered":"2023-10-16T20:58:56.907025+0000","last_clean":"2023-10-16T20:58:56.907025+0000","last_became_active":"2023-10-16T20:36:54.753237+0000","last_became_peered":"2023-10-16T20:36:54.753237+0000","last_unstale":"2023-10-16T20:58:56.907025+0000","last_undegraded":"2023-10-16T20:58:56.907025+0000","last_fullsized":"2023-10-16T20:58:56.907025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'131","last_scrub_stamp":"2023-10-16T20:58:56.906872+0000","last_deep_scrub":"67'131","last_deep_scrub_stamp":"2023-10-16T20:58:56.906872+0000","last_clean_scrub_stamp":"2023-10-16T20:58:56.906872+0000","objects_scrubbed":3,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:45.121322+0000","scrub_duration":0.023535402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":20486,"num_write":119,"num_write_kb":8247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'673","reported_seq":3504,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:27.360002+0000","last_change":"2023-10-16T20:59:27.360070+0000","last_active":"2023-10-16T20:59:27.360002+0000","last_peered":"2023-10-16T20:59:27.360002+0000","last_clean":"2023-10-16T20:59:27.360002+0000","last_became_active":"2023-10-16T20:36:59.466708+0000","last_became_peered":"2023-10-16T20:36:59.466708+0000","last_unstale":"2023-10-16T20:59:27.360002+0000","last_undegraded":"2023-10-16T20:59:27.360002+0000","last_fullsized":"2023-10-16T20:59:27.360002+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'673","last_scrub_stamp":"2023-10-16T20:59:27.359906+0000","last_deep_scrub":"67'673","last_deep_scrub_stamp":"2023-10-16T20:59:27.359906+0000","last_clean_scrub_stamp":"2023-10-16T20:59:27.359906+0000","objects_scrubbed":318,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:25:43.177804+0000","scrub_duration":0.12333601900000001,"objects_trimmed":0,"snaptrim_duration":0.011259905000000001,"stat_sum":{"num_bytes":19802424,"num_objects":318,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":703,"num_read_kb":14564,"num_write":670,"num_write_kb":31986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"67'99","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:02.278130+0000","last_change":"2023-10-16T20:59:02.278178+0000","last_active":"2023-10-16T20:59:02.278130+0000","last_peered":"2023-10-16T20:59:02.278130+0000","last_clean":"2023-10-16T20:59:02.278130+0000","last_became_active":"2023-10-16T20:36:55.294610+0000","last_became_peered":"2023-10-16T20:36:55.294610+0000","last_unstale":"2023-10-16T20:59:02.278130+0000","last_undegraded":"2023-10-16T20:59:02.278130+0000","last_fullsized":"2023-10-16T20:59:02.278130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'99","last_scrub_stamp":"2023-10-16T20:59:02.278035+0000","last_deep_scrub":"67'99","last_deep_scrub_stamp":"2023-10-16T20:59:02.278035+0000","last_clean_scrub_stamp":"2023-10-16T20:59:02.278035+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:05.752220+0000","scrub_duration":0.0025758619999999999,"objects_trimmed":0,"snaptrim_duration":0.0075091560000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"67'12","reported_seq":97,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:55.494605+0000","last_change":"2023-10-16T20:58:55.494660+0000","last_active":"2023-10-16T20:58:55.494605+0000","last_peered":"2023-10-16T20:58:55.494605+0000","last_clean":"2023-10-16T20:58:55.494605+0000","last_became_active":"2023-10-16T20:36:54.740944+0000","last_became_peered":"2023-10-16T20:36:54.740944+0000","last_unstale":"2023-10-16T20:58:55.494605+0000","last_undegraded":"2023-10-16T20:58:55.494605+0000","last_fullsized":"2023-10-16T20:58:55.494605+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2023-10-16T20:58:55.494532+0000","last_deep_scrub":"67'12","last_deep_scrub_stamp":"2023-10-16T20:58:55.494532+0000","last_clean_scrub_stamp":"2023-10-16T20:58:55.494532+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:08:42.299214+0000","scrub_duration":0.0059841030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":23,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"67'119","reported_seq":231,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:54.954307+0000","last_change":"2023-10-16T20:58:54.954375+0000","last_active":"2023-10-16T20:58:54.954307+0000","last_peered":"2023-10-16T20:58:54.954307+0000","last_clean":"2023-10-16T20:58:54.954307+0000","last_became_active":"2023-10-16T20:36:46.388824+0000","last_became_peered":"2023-10-16T20:36:46.388824+0000","last_unstale":"2023-10-16T20:58:54.954307+0000","last_undegraded":"2023-10-16T20:58:54.954307+0000","last_fullsized":"2023-10-16T20:58:54.954307+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'119","last_scrub_stamp":"2023-10-16T20:58:54.954227+0000","last_deep_scrub":"67'119","last_deep_scrub_stamp":"2023-10-16T20:58:54.954227+0000","last_clean_scrub_stamp":"2023-10-16T20:58:54.954227+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:25:20.529327+0000","scrub_duration":0.0064233470000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"67'669","reported_seq":3426,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.684418+0000","last_change":"2023-10-16T20:47:31.593007+0000","last_active":"2023-10-16T20:58:22.684418+0000","last_peered":"2023-10-16T20:58:22.684418+0000","last_clean":"2023-10-16T20:58:22.684418+0000","last_became_active":"2023-10-16T20:36:59.468082+0000","last_became_peered":"2023-10-16T20:36:59.468082+0000","last_unstale":"2023-10-16T20:58:22.684418+0000","last_undegraded":"2023-10-16T20:58:22.684418+0000","last_fullsized":"2023-10-16T20:58:22.684418+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":669,"log_dups_size":0,"ondisk_log_size":669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013154572,"stat_sum":{"num_bytes":30910053,"num_objects":362,"num_object_clones":0,"num_object_copies":1448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":649,"num_read_kb":15567,"num_write":666,"num_write_kb":42817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'110","reported_seq":2221,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:25.926391+0000","last_change":"2023-10-16T20:59:25.926438+0000","last_active":"2023-10-16T20:59:25.926391+0000","last_peered":"2023-10-16T20:59:25.926391+0000","last_clean":"2023-10-16T20:59:25.926391+0000","last_became_active":"2023-10-16T20:37:37.409004+0000","last_became_peered":"2023-10-16T20:37:37.409004+0000","last_unstale":"2023-10-16T20:59:25.926391+0000","last_undegraded":"2023-10-16T20:59:25.926391+0000","last_fullsized":"2023-10-16T20:59:25.926391+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'110","last_scrub_stamp":"2023-10-16T20:59:25.926296+0000","last_deep_scrub":"67'110","last_deep_scrub_stamp":"2023-10-16T20:59:25.926296+0000","last_clean_scrub_stamp":"2023-10-16T20:59:25.926296+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:03:54.206289+0000","scrub_duration":0.002525621,"objects_trimmed":0,"snaptrim_duration":0.0060805590000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"67'73","reported_seq":153,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.517695+0000","last_change":"2023-10-16T20:58:50.517695+0000","last_active":"2023-10-16T20:58:50.517695+0000","last_peered":"2023-10-16T20:58:50.517695+0000","last_clean":"2023-10-16T20:58:50.517695+0000","last_became_active":"2023-10-16T20:36:54.746247+0000","last_became_peered":"2023-10-16T20:36:54.746247+0000","last_unstale":"2023-10-16T20:58:50.517695+0000","last_undegraded":"2023-10-16T20:58:50.517695+0000","last_fullsized":"2023-10-16T20:58:50.517695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'73","last_scrub_stamp":"2023-10-16T20:58:50.517537+0000","last_deep_scrub":"67'73","last_deep_scrub_stamp":"2023-10-16T20:58:50.517537+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.517537+0000","objects_scrubbed":5,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:02:38.977902+0000","scrub_duration":0.021023528999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":7042,"num_write":68,"num_write_kb":4224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7644,"num_omap_keys":14,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"67'680","reported_seq":3681,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:47.660539+0000","last_change":"2023-10-16T20:47:31.590576+0000","last_active":"2023-10-16T20:58:47.660539+0000","last_peered":"2023-10-16T20:58:47.660539+0000","last_clean":"2023-10-16T20:58:47.660539+0000","last_became_active":"2023-10-16T20:37:37.467402+0000","last_became_peered":"2023-10-16T20:37:37.467402+0000","last_unstale":"2023-10-16T20:58:47.660539+0000","last_undegraded":"2023-10-16T20:58:47.660539+0000","last_fullsized":"2023-10-16T20:58:47.660539+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":680,"log_dups_size":0,"ondisk_log_size":680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012017773000000001,"stat_sum":{"num_bytes":36935626,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":884,"num_read_kb":21394,"num_write":679,"num_write_kb":40520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"67'106","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:19.134997+0000","last_change":"2023-10-16T20:59:19.134997+0000","last_active":"2023-10-16T20:59:19.134997+0000","last_peered":"2023-10-16T20:59:19.134997+0000","last_clean":"2023-10-16T20:59:19.134997+0000","last_became_active":"2023-10-16T20:36:55.300958+0000","last_became_peered":"2023-10-16T20:36:55.300958+0000","last_unstale":"2023-10-16T20:59:19.134997+0000","last_undegraded":"2023-10-16T20:59:19.134997+0000","last_fullsized":"2023-10-16T20:59:19.134997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'106","last_scrub_stamp":"2023-10-16T20:59:19.134877+0000","last_deep_scrub":"67'106","last_deep_scrub_stamp":"2023-10-16T20:59:19.134877+0000","last_clean_scrub_stamp":"2023-10-16T20:59:19.134877+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:46.731251+0000","scrub_duration":0.0019278839999999999,"objects_trimmed":0,"snaptrim_duration":0.009734431,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"67'323","reported_seq":405,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:02.756886+0000","last_change":"2023-10-16T20:58:51.074256+0000","last_active":"2023-10-16T20:59:02.756886+0000","last_peered":"2023-10-16T20:59:02.756886+0000","last_clean":"2023-10-16T20:59:02.756886+0000","last_became_active":"2023-10-16T20:36:54.753443+0000","last_became_peered":"2023-10-16T20:36:54.753443+0000","last_unstale":"2023-10-16T20:59:02.756886+0000","last_undegraded":"2023-10-16T20:59:02.756886+0000","last_fullsized":"2023-10-16T20:59:02.756886+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'322","last_scrub_stamp":"2023-10-16T20:58:51.074122+0000","last_deep_scrub":"67'322","last_deep_scrub_stamp":"2023-10-16T20:58:51.074122+0000","last_clean_scrub_stamp":"2023-10-16T20:58:51.074122+0000","objects_scrubbed":7,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:03.275883+0000","scrub_duration":0.031482483999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9668880,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":15859,"num_write":310,"num_write_kb":9619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":30112,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'687","reported_seq":4000,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:17.390664+0000","last_change":"2023-10-16T20:59:17.390664+0000","last_active":"2023-10-16T20:59:17.390664+0000","last_peered":"2023-10-16T20:59:17.390664+0000","last_clean":"2023-10-16T20:59:17.390664+0000","last_became_active":"2023-10-16T20:36:59.302869+0000","last_became_peered":"2023-10-16T20:36:59.302869+0000","last_unstale":"2023-10-16T20:59:17.390664+0000","last_undegraded":"2023-10-16T20:59:17.390664+0000","last_fullsized":"2023-10-16T20:59:17.390664+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'687","last_scrub_stamp":"2023-10-16T20:59:17.390491+0000","last_deep_scrub":"67'687","last_deep_scrub_stamp":"2023-10-16T20:59:17.390491+0000","last_clean_scrub_stamp":"2023-10-16T20:59:17.390491+0000","objects_scrubbed":352,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:35:09.376960+0000","scrub_duration":0.137334489,"objects_trimmed":0,"snaptrim_duration":0.012136565,"stat_sum":{"num_bytes":23637013,"num_objects":352,"num_object_clones":0,"num_object_copies":1408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":1183,"num_read_kb":18772,"num_write":687,"num_write_kb":27489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"67'100","reported_seq":2210,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.893042+0000","last_change":"2023-10-16T20:47:31.587528+0000","last_active":"2023-10-16T20:58:37.893042+0000","last_peered":"2023-10-16T20:58:37.893042+0000","last_clean":"2023-10-16T20:58:37.893042+0000","last_became_active":"2023-10-16T20:36:55.291633+0000","last_became_peered":"2023-10-16T20:36:55.291633+0000","last_unstale":"2023-10-16T20:58:37.893042+0000","last_undegraded":"2023-10-16T20:58:37.893042+0000","last_fullsized":"2023-10-16T20:58:37.893042+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085978630000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'126","reported_seq":211,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:54.604136+0000","last_change":"2023-10-16T20:58:54.604136+0000","last_active":"2023-10-16T20:58:54.604136+0000","last_peered":"2023-10-16T20:58:54.604136+0000","last_clean":"2023-10-16T20:58:54.604136+0000","last_became_active":"2023-10-16T20:36:54.747704+0000","last_became_peered":"2023-10-16T20:36:54.747704+0000","last_unstale":"2023-10-16T20:58:54.604136+0000","last_undegraded":"2023-10-16T20:58:54.604136+0000","last_fullsized":"2023-10-16T20:58:54.604136+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'126","last_scrub_stamp":"2023-10-16T20:58:54.603981+0000","last_deep_scrub":"67'126","last_deep_scrub_stamp":"2023-10-16T20:58:54.603981+0000","last_clean_scrub_stamp":"2023-10-16T20:58:54.603981+0000","objects_scrubbed":6,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:18:13.372332+0000","scrub_duration":0.020068987999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":116,"num_write_kb":4228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20215,"num_omap_keys":38,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"67'663","reported_seq":3387,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:42.664360+0000","last_change":"2023-10-16T20:47:31.595065+0000","last_active":"2023-10-16T20:58:42.664360+0000","last_peered":"2023-10-16T20:58:42.664360+0000","last_clean":"2023-10-16T20:58:42.664360+0000","last_became_active":"2023-10-16T20:36:59.467291+0000","last_became_peered":"2023-10-16T20:36:59.467291+0000","last_unstale":"2023-10-16T20:58:42.664360+0000","last_undegraded":"2023-10-16T20:58:42.664360+0000","last_fullsized":"2023-10-16T20:58:42.664360+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0131915,"stat_sum":{"num_bytes":23136394,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":616,"num_read_kb":10065,"num_write":660,"num_write_kb":35181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"67'80","reported_seq":2199,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:20.894073+0000","last_change":"2023-10-16T20:59:20.894073+0000","last_active":"2023-10-16T20:59:20.894073+0000","last_peered":"2023-10-16T20:59:20.894073+0000","last_clean":"2023-10-16T20:59:20.894073+0000","last_became_active":"2023-10-16T20:36:55.290998+0000","last_became_peered":"2023-10-16T20:36:55.290998+0000","last_unstale":"2023-10-16T20:59:20.894073+0000","last_undegraded":"2023-10-16T20:59:20.894073+0000","last_fullsized":"2023-10-16T20:59:20.894073+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'80","last_scrub_stamp":"2023-10-16T20:59:20.893925+0000","last_deep_scrub":"67'80","last_deep_scrub_stamp":"2023-10-16T20:59:20.893925+0000","last_clean_scrub_stamp":"2023-10-16T20:59:20.893925+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:33:27.367641+0000","scrub_duration":0.001815213,"objects_trimmed":0,"snaptrim_duration":0.0082191850000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'95","reported_seq":172,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.292497+0000","last_change":"2023-10-16T20:58:50.292551+0000","last_active":"2023-10-16T20:58:50.292497+0000","last_peered":"2023-10-16T20:58:50.292497+0000","last_clean":"2023-10-16T20:58:50.292497+0000","last_became_active":"2023-10-16T20:36:54.761337+0000","last_became_peered":"2023-10-16T20:36:54.761337+0000","last_unstale":"2023-10-16T20:58:50.292497+0000","last_undegraded":"2023-10-16T20:58:50.292497+0000","last_fullsized":"2023-10-16T20:58:50.292497+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'95","last_scrub_stamp":"2023-10-16T20:58:50.292415+0000","last_deep_scrub":"67'95","last_deep_scrub_stamp":"2023-10-16T20:58:50.292415+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.292415+0000","objects_scrubbed":5,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:51:30.957479+0000","scrub_duration":0.018399241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":5,"num_write":137,"num_write_kb":4352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18880,"num_omap_keys":36,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"67'666","reported_seq":3841,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:42.659648+0000","last_change":"2023-10-16T20:47:31.487063+0000","last_active":"2023-10-16T20:58:42.659648+0000","last_peered":"2023-10-16T20:58:42.659648+0000","last_clean":"2023-10-16T20:58:42.659648+0000","last_became_active":"2023-10-16T20:37:37.468309+0000","last_became_peered":"2023-10-16T20:37:37.468309+0000","last_unstale":"2023-10-16T20:58:42.659648+0000","last_undegraded":"2023-10-16T20:58:42.659648+0000","last_fullsized":"2023-10-16T20:58:42.659648+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058030460000000001,"stat_sum":{"num_bytes":20758781,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":1058,"num_read_kb":21621,"num_write":667,"num_write_kb":32853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"67'107","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:23.969170+0000","last_change":"2023-10-16T20:59:23.969232+0000","last_active":"2023-10-16T20:59:23.969170+0000","last_peered":"2023-10-16T20:59:23.969170+0000","last_clean":"2023-10-16T20:59:23.969170+0000","last_became_active":"2023-10-16T20:36:55.288354+0000","last_became_peered":"2023-10-16T20:36:55.288354+0000","last_unstale":"2023-10-16T20:59:23.969170+0000","last_undegraded":"2023-10-16T20:59:23.969170+0000","last_fullsized":"2023-10-16T20:59:23.969170+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'107","last_scrub_stamp":"2023-10-16T20:59:23.969053+0000","last_deep_scrub":"67'107","last_deep_scrub_stamp":"2023-10-16T20:59:23.969053+0000","last_clean_scrub_stamp":"2023-10-16T20:59:23.969053+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:35:11.681865+0000","scrub_duration":0.0020025540000000001,"objects_trimmed":0,"snaptrim_duration":0.0076786709999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"67'348","reported_seq":437,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:57.706996+0000","last_change":"2023-10-16T20:58:57.707108+0000","last_active":"2023-10-16T20:58:57.706996+0000","last_peered":"2023-10-16T20:58:57.706996+0000","last_clean":"2023-10-16T20:58:57.706996+0000","last_became_active":"2023-10-16T20:36:54.737876+0000","last_became_peered":"2023-10-16T20:36:54.737876+0000","last_unstale":"2023-10-16T20:58:57.706996+0000","last_undegraded":"2023-10-16T20:58:57.706996+0000","last_fullsized":"2023-10-16T20:58:57.706996+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'348","last_scrub_stamp":"2023-10-16T20:58:57.706891+0000","last_deep_scrub":"67'348","last_deep_scrub_stamp":"2023-10-16T20:58:57.706891+0000","last_clean_scrub_stamp":"2023-10-16T20:58:57.706891+0000","objects_scrubbed":15,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:40:24.491212+0000","scrub_duration":0.033886146999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":39,"num_read_kb":4134,"num_write":333,"num_write_kb":8414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":43828,"num_omap_keys":84,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"67'666","reported_seq":3686,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:28.325879+0000","last_change":"2023-10-16T20:59:28.325991+0000","last_active":"2023-10-16T20:59:28.325879+0000","last_peered":"2023-10-16T20:59:28.325879+0000","last_clean":"2023-10-16T20:59:28.325879+0000","last_became_active":"2023-10-16T20:36:59.306801+0000","last_became_peered":"2023-10-16T20:36:59.306801+0000","last_unstale":"2023-10-16T20:59:28.325879+0000","last_undegraded":"2023-10-16T20:59:28.325879+0000","last_fullsized":"2023-10-16T20:59:28.325879+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'666","last_scrub_stamp":"2023-10-16T20:59:28.325767+0000","last_deep_scrub":"67'666","last_deep_scrub_stamp":"2023-10-16T20:59:28.325767+0000","last_clean_scrub_stamp":"2023-10-16T20:59:28.325767+0000","objects_scrubbed":301,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:38.407734+0000","scrub_duration":0.112555059,"objects_trimmed":0,"snaptrim_duration":0.010585667,"stat_sum":{"num_bytes":20444029,"num_objects":301,"num_object_clones":0,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":893,"num_read_kb":19660,"num_write":666,"num_write_kb":28474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"67'93","reported_seq":2211,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:06.354741+0000","last_change":"2023-10-16T20:59:06.354789+0000","last_active":"2023-10-16T20:59:06.354741+0000","last_peered":"2023-10-16T20:59:06.354741+0000","last_clean":"2023-10-16T20:59:06.354741+0000","last_became_active":"2023-10-16T20:36:55.287066+0000","last_became_peered":"2023-10-16T20:36:55.287066+0000","last_unstale":"2023-10-16T20:59:06.354741+0000","last_undegraded":"2023-10-16T20:59:06.354741+0000","last_fullsized":"2023-10-16T20:59:06.354741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'93","last_scrub_stamp":"2023-10-16T20:59:06.354644+0000","last_deep_scrub":"67'93","last_deep_scrub_stamp":"2023-10-16T20:59:06.354644+0000","last_clean_scrub_stamp":"2023-10-16T20:59:06.354644+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:18:40.827931+0000","scrub_duration":0.0027107630000000001,"objects_trimmed":0,"snaptrim_duration":0.012121162,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"67'105","reported_seq":185,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:55.485180+0000","last_change":"2023-10-16T20:58:55.485242+0000","last_active":"2023-10-16T20:58:55.485180+0000","last_peered":"2023-10-16T20:58:55.485180+0000","last_clean":"2023-10-16T20:58:55.485180+0000","last_became_active":"2023-10-16T20:36:54.764417+0000","last_became_peered":"2023-10-16T20:36:54.764417+0000","last_unstale":"2023-10-16T20:58:55.485180+0000","last_undegraded":"2023-10-16T20:58:55.485180+0000","last_fullsized":"2023-10-16T20:58:55.485180+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'105","last_scrub_stamp":"2023-10-16T20:58:55.485084+0000","last_deep_scrub":"67'105","last_deep_scrub_stamp":"2023-10-16T20:58:55.485084+0000","last_clean_scrub_stamp":"2023-10-16T20:58:55.485084+0000","objects_scrubbed":4,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:13:49.097422+0000","scrub_duration":0.015871666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":4110,"num_write":99,"num_write_kb":4200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'634","reported_seq":3756,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:47.854168+0000","last_change":"2023-10-16T20:47:31.584541+0000","last_active":"2023-10-16T20:58:47.854168+0000","last_peered":"2023-10-16T20:58:47.854168+0000","last_clean":"2023-10-16T20:58:47.854168+0000","last_became_active":"2023-10-16T20:36:59.464626+0000","last_became_peered":"2023-10-16T20:36:59.464626+0000","last_unstale":"2023-10-16T20:58:47.854168+0000","last_undegraded":"2023-10-16T20:58:47.854168+0000","last_fullsized":"2023-10-16T20:58:47.854168+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":634,"log_dups_size":0,"ondisk_log_size":634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069938420000000001,"stat_sum":{"num_bytes":28405361,"num_objects":321,"num_object_clones":0,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":1015,"num_read_kb":24481,"num_write":633,"num_write_kb":44456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"67'113","reported_seq":2223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.887053+0000","last_change":"2023-10-16T20:47:31.589722+0000","last_active":"2023-10-16T20:58:37.887053+0000","last_peered":"2023-10-16T20:58:37.887053+0000","last_clean":"2023-10-16T20:58:37.887053+0000","last_became_active":"2023-10-16T20:36:55.291396+0000","last_became_peered":"2023-10-16T20:36:55.291396+0000","last_unstale":"2023-10-16T20:58:37.887053+0000","last_undegraded":"2023-10-16T20:58:37.887053+0000","last_fullsized":"2023-10-16T20:58:37.887053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011464676,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"67'189","reported_seq":272,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:56.473518+0000","last_change":"2023-10-16T20:58:56.473573+0000","last_active":"2023-10-16T20:58:56.473518+0000","last_peered":"2023-10-16T20:58:56.473518+0000","last_clean":"2023-10-16T20:58:56.473518+0000","last_became_active":"2023-10-16T20:36:54.750448+0000","last_became_peered":"2023-10-16T20:36:54.750448+0000","last_unstale":"2023-10-16T20:58:56.473518+0000","last_undegraded":"2023-10-16T20:58:56.473518+0000","last_fullsized":"2023-10-16T20:58:56.473518+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'189","last_scrub_stamp":"2023-10-16T20:58:56.473428+0000","last_deep_scrub":"67'189","last_deep_scrub_stamp":"2023-10-16T20:58:56.473428+0000","last_clean_scrub_stamp":"2023-10-16T20:58:56.473428+0000","objects_scrubbed":5,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:18:30.642536+0000","scrub_duration":0.01623637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":3911,"num_write":180,"num_write_kb":4223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12239,"num_omap_keys":24,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"67'608","reported_seq":3331,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:23.255862+0000","last_change":"2023-10-16T20:47:31.583645+0000","last_active":"2023-10-16T20:58:23.255862+0000","last_peered":"2023-10-16T20:58:23.255862+0000","last_clean":"2023-10-16T20:58:23.255862+0000","last_became_active":"2023-10-16T20:36:59.464578+0000","last_became_peered":"2023-10-16T20:36:59.464578+0000","last_unstale":"2023-10-16T20:58:23.255862+0000","last_undegraded":"2023-10-16T20:58:23.255862+0000","last_fullsized":"2023-10-16T20:58:23.255862+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064826880000000003,"stat_sum":{"num_bytes":21354704,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":616,"num_read_kb":9691,"num_write":609,"num_write_kb":35926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"67'101","reported_seq":2219,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:56.192318+0000","last_change":"2023-10-16T20:58:56.192363+0000","last_active":"2023-10-16T20:58:56.192318+0000","last_peered":"2023-10-16T20:58:56.192318+0000","last_clean":"2023-10-16T20:58:56.192318+0000","last_became_active":"2023-10-16T20:36:55.285407+0000","last_became_peered":"2023-10-16T20:36:55.285407+0000","last_unstale":"2023-10-16T20:58:56.192318+0000","last_undegraded":"2023-10-16T20:58:56.192318+0000","last_fullsized":"2023-10-16T20:58:56.192318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'101","last_scrub_stamp":"2023-10-16T20:58:56.192235+0000","last_deep_scrub":"67'101","last_deep_scrub_stamp":"2023-10-16T20:58:56.192235+0000","last_clean_scrub_stamp":"2023-10-16T20:58:56.192235+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:39:41.514891+0000","scrub_duration":0.0021418700000000001,"objects_trimmed":0,"snaptrim_duration":0.010336254,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"67'469","reported_seq":623,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:52.147902+0000","last_change":"2023-10-16T20:58:52.147973+0000","last_active":"2023-10-16T20:58:52.147902+0000","last_peered":"2023-10-16T20:58:52.147902+0000","last_clean":"2023-10-16T20:58:52.147902+0000","last_became_active":"2023-10-16T20:36:54.756697+0000","last_became_peered":"2023-10-16T20:36:54.756697+0000","last_unstale":"2023-10-16T20:58:52.147902+0000","last_undegraded":"2023-10-16T20:58:52.147902+0000","last_fullsized":"2023-10-16T20:58:52.147902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'469","last_scrub_stamp":"2023-10-16T20:58:52.147814+0000","last_deep_scrub":"67'469","last_deep_scrub_stamp":"2023-10-16T20:58:52.147814+0000","last_clean_scrub_stamp":"2023-10-16T20:58:52.147814+0000","objects_scrubbed":10,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:59:37.271178+0000","scrub_duration":0.031542023000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8464867,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":106,"num_read_kb":16561,"num_write":461,"num_write_kb":8611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11094,"num_omap_keys":22,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'632","reported_seq":4048,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:47.665085+0000","last_change":"2023-10-16T20:47:31.589709+0000","last_active":"2023-10-16T20:58:47.665085+0000","last_peered":"2023-10-16T20:58:47.665085+0000","last_clean":"2023-10-16T20:58:47.665085+0000","last_became_active":"2023-10-16T20:36:59.468254+0000","last_became_peered":"2023-10-16T20:36:59.468254+0000","last_unstale":"2023-10-16T20:58:47.665085+0000","last_undegraded":"2023-10-16T20:58:47.665085+0000","last_fullsized":"2023-10-16T20:58:47.665085+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093892239999999998,"stat_sum":{"num_bytes":28874401,"num_objects":300,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":1309,"num_read_kb":15538,"num_write":630,"num_write_kb":28525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"67'100","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:53.035022+0000","last_change":"2023-10-16T20:58:53.035079+0000","last_active":"2023-10-16T20:58:53.035022+0000","last_peered":"2023-10-16T20:58:53.035022+0000","last_clean":"2023-10-16T20:58:53.035022+0000","last_became_active":"2023-10-16T20:36:55.288488+0000","last_became_peered":"2023-10-16T20:36:55.288488+0000","last_unstale":"2023-10-16T20:58:53.035022+0000","last_undegraded":"2023-10-16T20:58:53.035022+0000","last_fullsized":"2023-10-16T20:58:53.035022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'100","last_scrub_stamp":"2023-10-16T20:58:53.034921+0000","last_deep_scrub":"67'100","last_deep_scrub_stamp":"2023-10-16T20:58:53.034921+0000","last_clean_scrub_stamp":"2023-10-16T20:58:53.034921+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:10.663084+0000","scrub_duration":0.002664118,"objects_trimmed":0,"snaptrim_duration":0.0078405860000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"67'147","reported_seq":240,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:01.469451+0000","last_change":"2023-10-16T20:59:01.469451+0000","last_active":"2023-10-16T20:59:01.469451+0000","last_peered":"2023-10-16T20:59:01.469451+0000","last_clean":"2023-10-16T20:59:01.469451+0000","last_became_active":"2023-10-16T20:36:54.764969+0000","last_became_peered":"2023-10-16T20:36:54.764969+0000","last_unstale":"2023-10-16T20:59:01.469451+0000","last_undegraded":"2023-10-16T20:59:01.469451+0000","last_fullsized":"2023-10-16T20:59:01.469451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'146","last_scrub_stamp":"2023-10-16T20:59:01.469291+0000","last_deep_scrub":"67'146","last_deep_scrub_stamp":"2023-10-16T20:59:01.469291+0000","last_clean_scrub_stamp":"2023-10-16T20:59:01.469291+0000","objects_scrubbed":7,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:10.085576+0000","scrub_duration":0.028155902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389280,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":45,"num_read_kb":8235,"num_write":136,"num_write_kb":8276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"67'666","reported_seq":4080,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:13.118241+0000","last_change":"2023-10-16T20:59:13.118315+0000","last_active":"2023-10-16T20:59:13.118241+0000","last_peered":"2023-10-16T20:59:13.118241+0000","last_clean":"2023-10-16T20:59:13.118241+0000","last_became_active":"2023-10-16T20:36:59.302698+0000","last_became_peered":"2023-10-16T20:36:59.302698+0000","last_unstale":"2023-10-16T20:59:13.118241+0000","last_undegraded":"2023-10-16T20:59:13.118241+0000","last_fullsized":"2023-10-16T20:59:13.118241+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'666","last_scrub_stamp":"2023-10-16T20:59:13.118142+0000","last_deep_scrub":"67'666","last_deep_scrub_stamp":"2023-10-16T20:59:13.118142+0000","last_clean_scrub_stamp":"2023-10-16T20:59:13.118142+0000","objects_scrubbed":333,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:54.067602+0000","scrub_duration":0.142898739,"objects_trimmed":0,"snaptrim_duration":0.0064797020000000004,"stat_sum":{"num_bytes":31392251,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":1286,"num_read_kb":27897,"num_write":665,"num_write_kb":43250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'99","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:59.336824+0000","last_change":"2023-10-16T20:58:59.336882+0000","last_active":"2023-10-16T20:58:59.336824+0000","last_peered":"2023-10-16T20:58:59.336824+0000","last_clean":"2023-10-16T20:58:59.336824+0000","last_became_active":"2023-10-16T20:36:55.290127+0000","last_became_peered":"2023-10-16T20:36:55.290127+0000","last_unstale":"2023-10-16T20:58:59.336824+0000","last_undegraded":"2023-10-16T20:58:59.336824+0000","last_fullsized":"2023-10-16T20:58:59.336824+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'99","last_scrub_stamp":"2023-10-16T20:58:59.336718+0000","last_deep_scrub":"67'99","last_deep_scrub_stamp":"2023-10-16T20:58:59.336718+0000","last_clean_scrub_stamp":"2023-10-16T20:58:59.336718+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:51:20.952065+0000","scrub_duration":0.0019832140000000001,"objects_trimmed":0,"snaptrim_duration":0.0055494209999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"67'84","reported_seq":165,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:00.071444+0000","last_change":"2023-10-16T20:59:00.071528+0000","last_active":"2023-10-16T20:59:00.071444+0000","last_peered":"2023-10-16T20:59:00.071444+0000","last_clean":"2023-10-16T20:59:00.071444+0000","last_became_active":"2023-10-16T20:36:54.748788+0000","last_became_peered":"2023-10-16T20:36:54.748788+0000","last_unstale":"2023-10-16T20:59:00.071444+0000","last_undegraded":"2023-10-16T20:59:00.071444+0000","last_fullsized":"2023-10-16T20:59:00.071444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'84","last_scrub_stamp":"2023-10-16T20:59:00.071354+0000","last_deep_scrub":"67'84","last_deep_scrub_stamp":"2023-10-16T20:59:00.071354+0000","last_clean_scrub_stamp":"2023-10-16T20:59:00.071354+0000","objects_scrubbed":6,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:33:58.876619+0000","scrub_duration":0.006295775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":15,"num_read_kb":23,"num_write":83,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9178,"num_omap_keys":18,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"67'606","reported_seq":3881,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:13.298004+0000","last_change":"2023-10-16T20:59:13.298078+0000","last_active":"2023-10-16T20:59:13.298004+0000","last_peered":"2023-10-16T20:59:13.298004+0000","last_clean":"2023-10-16T20:59:13.298004+0000","last_became_active":"2023-10-16T20:36:59.302916+0000","last_became_peered":"2023-10-16T20:36:59.302916+0000","last_unstale":"2023-10-16T20:59:13.298004+0000","last_undegraded":"2023-10-16T20:59:13.298004+0000","last_fullsized":"2023-10-16T20:59:13.298004+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'606","last_scrub_stamp":"2023-10-16T20:59:13.297907+0000","last_deep_scrub":"67'606","last_deep_scrub_stamp":"2023-10-16T20:59:13.297907+0000","last_clean_scrub_stamp":"2023-10-16T20:59:13.297907+0000","objects_scrubbed":300,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:10:39.220250+0000","scrub_duration":0.11959800800000001,"objects_trimmed":0,"snaptrim_duration":0.0041353309999999999,"stat_sum":{"num_bytes":29291517,"num_objects":300,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":1148,"num_read_kb":24111,"num_write":607,"num_write_kb":41174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"67'107","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:01.224253+0000","last_change":"2023-10-16T20:59:01.224305+0000","last_active":"2023-10-16T20:59:01.224253+0000","last_peered":"2023-10-16T20:59:01.224253+0000","last_clean":"2023-10-16T20:59:01.224253+0000","last_became_active":"2023-10-16T20:36:55.291448+0000","last_became_peered":"2023-10-16T20:36:55.291448+0000","last_unstale":"2023-10-16T20:59:01.224253+0000","last_undegraded":"2023-10-16T20:59:01.224253+0000","last_fullsized":"2023-10-16T20:59:01.224253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'107","last_scrub_stamp":"2023-10-16T20:59:01.224155+0000","last_deep_scrub":"67'107","last_deep_scrub_stamp":"2023-10-16T20:59:01.224155+0000","last_clean_scrub_stamp":"2023-10-16T20:59:01.224155+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:59:11.517988+0000","scrub_duration":0.0022855140000000002,"objects_trimmed":0,"snaptrim_duration":0.010585074,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"67'149","reported_seq":278,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:05.396279+0000","last_change":"2023-10-16T20:59:05.396279+0000","last_active":"2023-10-16T20:59:05.396279+0000","last_peered":"2023-10-16T20:59:05.396279+0000","last_clean":"2023-10-16T20:59:05.396279+0000","last_became_active":"2023-10-16T20:36:54.761541+0000","last_became_peered":"2023-10-16T20:36:54.761541+0000","last_unstale":"2023-10-16T20:59:05.396279+0000","last_undegraded":"2023-10-16T20:59:05.396279+0000","last_fullsized":"2023-10-16T20:59:05.396279+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'148","last_scrub_stamp":"2023-10-16T20:59:05.396099+0000","last_deep_scrub":"67'148","last_deep_scrub_stamp":"2023-10-16T20:59:05.396099+0000","last_clean_scrub_stamp":"2023-10-16T20:59:05.396099+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:44:03.980765+0000","scrub_duration":0.01804234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2788685,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":69,"num_read_kb":129,"num_write":144,"num_write_kb":2886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":19476,"num_omap_keys":37,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"67'663","reported_seq":3326,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.288364+0000","last_change":"2023-10-16T20:47:31.585744+0000","last_active":"2023-10-16T20:58:22.288364+0000","last_peered":"2023-10-16T20:58:22.288364+0000","last_clean":"2023-10-16T20:58:22.288364+0000","last_became_active":"2023-10-16T20:36:59.303773+0000","last_became_peered":"2023-10-16T20:36:59.303773+0000","last_unstale":"2023-10-16T20:58:22.288364+0000","last_undegraded":"2023-10-16T20:58:22.288364+0000","last_fullsized":"2023-10-16T20:58:22.288364+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007941142,"stat_sum":{"num_bytes":21664408,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":557,"num_read_kb":9585,"num_write":658,"num_write_kb":32916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"67'121","reported_seq":2239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:26.217163+0000","last_change":"2023-10-16T20:59:26.217216+0000","last_active":"2023-10-16T20:59:26.217163+0000","last_peered":"2023-10-16T20:59:26.217163+0000","last_clean":"2023-10-16T20:59:26.217163+0000","last_became_active":"2023-10-16T20:36:55.300535+0000","last_became_peered":"2023-10-16T20:36:55.300535+0000","last_unstale":"2023-10-16T20:59:26.217163+0000","last_undegraded":"2023-10-16T20:59:26.217163+0000","last_fullsized":"2023-10-16T20:59:26.217163+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'121","last_scrub_stamp":"2023-10-16T20:59:26.217064+0000","last_deep_scrub":"67'121","last_deep_scrub_stamp":"2023-10-16T20:59:26.217064+0000","last_clean_scrub_stamp":"2023-10-16T20:59:26.217064+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:41:02.094148+0000","scrub_duration":0.0019061880000000001,"objects_trimmed":0,"snaptrim_duration":0.013755641000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"67'61","reported_seq":139,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:52.484963+0000","last_change":"2023-10-16T20:58:52.484963+0000","last_active":"2023-10-16T20:58:52.484963+0000","last_peered":"2023-10-16T20:58:52.484963+0000","last_clean":"2023-10-16T20:58:52.484963+0000","last_became_active":"2023-10-16T20:36:54.757303+0000","last_became_peered":"2023-10-16T20:36:54.757303+0000","last_unstale":"2023-10-16T20:58:52.484963+0000","last_undegraded":"2023-10-16T20:58:52.484963+0000","last_fullsized":"2023-10-16T20:58:52.484963+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'60","last_scrub_stamp":"2023-10-16T20:58:52.484737+0000","last_deep_scrub":"67'60","last_deep_scrub_stamp":"2023-10-16T20:58:52.484737+0000","last_clean_scrub_stamp":"2023-10-16T20:58:52.484737+0000","objects_scrubbed":6,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:17:16.465796+0000","scrub_duration":0.0082689740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":59,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":40464,"num_omap_keys":76,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"67'687","reported_seq":4218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.665131+0000","last_change":"2023-10-16T20:47:31.586327+0000","last_active":"2023-10-16T20:58:22.665131+0000","last_peered":"2023-10-16T20:58:22.665131+0000","last_clean":"2023-10-16T20:58:22.665131+0000","last_became_active":"2023-10-16T20:36:59.465844+0000","last_became_peered":"2023-10-16T20:36:59.465844+0000","last_unstale":"2023-10-16T20:58:22.665131+0000","last_undegraded":"2023-10-16T20:58:22.665131+0000","last_fullsized":"2023-10-16T20:58:22.665131+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081168009999999999,"stat_sum":{"num_bytes":45881583,"num_objects":353,"num_object_clones":0,"num_object_copies":1412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":353,"num_whiteouts":0,"num_read":1424,"num_read_kb":44764,"num_write":688,"num_write_kb":65580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"67'661","reported_seq":4049,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:46.761134+0000","last_change":"2023-10-16T20:47:31.489178+0000","last_active":"2023-10-16T20:58:46.761134+0000","last_peered":"2023-10-16T20:58:46.761134+0000","last_clean":"2023-10-16T20:58:46.761134+0000","last_became_active":"2023-10-16T20:36:59.467356+0000","last_became_peered":"2023-10-16T20:36:59.467356+0000","last_unstale":"2023-10-16T20:58:46.761134+0000","last_undegraded":"2023-10-16T20:58:46.761134+0000","last_fullsized":"2023-10-16T20:58:46.761134+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093722700000000003,"stat_sum":{"num_bytes":37038710,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1279,"num_read_kb":28276,"num_write":659,"num_write_kb":40616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'108","reported_seq":2226,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:11.931941+0000","last_change":"2023-10-16T20:59:11.931993+0000","last_active":"2023-10-16T20:59:11.931941+0000","last_peered":"2023-10-16T20:59:11.931941+0000","last_clean":"2023-10-16T20:59:11.931941+0000","last_became_active":"2023-10-16T20:36:55.280903+0000","last_became_peered":"2023-10-16T20:36:55.280903+0000","last_unstale":"2023-10-16T20:59:11.931941+0000","last_undegraded":"2023-10-16T20:59:11.931941+0000","last_fullsized":"2023-10-16T20:59:11.931941+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'108","last_scrub_stamp":"2023-10-16T20:59:11.931838+0000","last_deep_scrub":"67'108","last_deep_scrub_stamp":"2023-10-16T20:59:11.931838+0000","last_clean_scrub_stamp":"2023-10-16T20:59:11.931838+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:53:40.858669+0000","scrub_duration":0.0017970499999999999,"objects_trimmed":0,"snaptrim_duration":0.0029269690000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"67'463","reported_seq":543,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:04.451251+0000","last_change":"2023-10-16T20:59:04.451310+0000","last_active":"2023-10-16T20:59:04.451251+0000","last_peered":"2023-10-16T20:59:04.451251+0000","last_clean":"2023-10-16T20:59:04.451251+0000","last_became_active":"2023-10-16T20:36:54.756775+0000","last_became_peered":"2023-10-16T20:36:54.756775+0000","last_unstale":"2023-10-16T20:59:04.451251+0000","last_undegraded":"2023-10-16T20:59:04.451251+0000","last_fullsized":"2023-10-16T20:59:04.451251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'463","last_scrub_stamp":"2023-10-16T20:59:04.451169+0000","last_deep_scrub":"67'463","last_deep_scrub_stamp":"2023-10-16T20:59:04.451169+0000","last_clean_scrub_stamp":"2023-10-16T20:59:04.451169+0000","objects_scrubbed":6,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:16.153884+0000","scrub_duration":0.039845845999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583494,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":16386,"num_write":448,"num_write_kb":12544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20150,"num_omap_keys":39,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"67'638","reported_seq":3560,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.627190+0000","last_change":"2023-10-16T20:47:31.593596+0000","last_active":"2023-10-16T20:58:22.627190+0000","last_peered":"2023-10-16T20:58:22.627190+0000","last_clean":"2023-10-16T20:58:22.627190+0000","last_became_active":"2023-10-16T20:37:37.438140+0000","last_became_peered":"2023-10-16T20:37:37.438140+0000","last_unstale":"2023-10-16T20:58:22.627190+0000","last_undegraded":"2023-10-16T20:58:22.627190+0000","last_fullsized":"2023-10-16T20:58:22.627190+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012287041,"stat_sum":{"num_bytes":24792014,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":805,"num_read_kb":12206,"num_write":635,"num_write_kb":32676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"67'121","reported_seq":2239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:12.182911+0000","last_change":"2023-10-16T20:59:12.182960+0000","last_active":"2023-10-16T20:59:12.182911+0000","last_peered":"2023-10-16T20:59:12.182911+0000","last_clean":"2023-10-16T20:59:12.182911+0000","last_became_active":"2023-10-16T20:36:55.273959+0000","last_became_peered":"2023-10-16T20:36:55.273959+0000","last_unstale":"2023-10-16T20:59:12.182911+0000","last_undegraded":"2023-10-16T20:59:12.182911+0000","last_fullsized":"2023-10-16T20:59:12.182911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'121","last_scrub_stamp":"2023-10-16T20:59:12.182820+0000","last_deep_scrub":"67'121","last_deep_scrub_stamp":"2023-10-16T20:59:12.182820+0000","last_clean_scrub_stamp":"2023-10-16T20:59:12.182820+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:30.216780+0000","scrub_duration":0.0019525829999999999,"objects_trimmed":0,"snaptrim_duration":0.0076333039999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"67'140","reported_seq":290,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:07.224275+0000","last_change":"2023-10-16T20:59:07.224354+0000","last_active":"2023-10-16T20:59:07.224275+0000","last_peered":"2023-10-16T20:59:07.224275+0000","last_clean":"2023-10-16T20:59:07.224275+0000","last_became_active":"2023-10-16T20:36:54.759059+0000","last_became_peered":"2023-10-16T20:36:54.759059+0000","last_unstale":"2023-10-16T20:59:07.224275+0000","last_undegraded":"2023-10-16T20:59:07.224275+0000","last_fullsized":"2023-10-16T20:59:07.224275+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'140","last_scrub_stamp":"2023-10-16T20:59:07.224189+0000","last_deep_scrub":"67'140","last_deep_scrub_stamp":"2023-10-16T20:59:07.224189+0000","last_clean_scrub_stamp":"2023-10-16T20:59:07.224189+0000","objects_scrubbed":5,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:11:29.789893+0000","scrub_duration":0.016670114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4348959,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":88,"num_read_kb":239,"num_write":133,"num_write_kb":4341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10224,"num_omap_keys":20,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"67'230","reported_seq":389,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.505513+0000","last_change":"2023-10-16T20:58:50.505513+0000","last_active":"2023-10-16T20:58:50.505513+0000","last_peered":"2023-10-16T20:58:50.505513+0000","last_clean":"2023-10-16T20:58:50.505513+0000","last_became_active":"2023-10-16T20:36:54.764436+0000","last_became_peered":"2023-10-16T20:36:54.764436+0000","last_unstale":"2023-10-16T20:58:50.505513+0000","last_undegraded":"2023-10-16T20:58:50.505513+0000","last_fullsized":"2023-10-16T20:58:50.505513+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'230","last_scrub_stamp":"2023-10-16T20:58:50.505320+0000","last_deep_scrub":"67'230","last_deep_scrub_stamp":"2023-10-16T20:58:50.505320+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.505320+0000","objects_scrubbed":7,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:05:38.516764+0000","scrub_duration":0.030122286000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8524313,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":105,"num_read_kb":8412,"num_write":220,"num_write_kb":8480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12018,"num_omap_keys":23,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"67'109","reported_seq":2227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:59.179962+0000","last_change":"2023-10-16T20:58:59.179962+0000","last_active":"2023-10-16T20:58:59.179962+0000","last_peered":"2023-10-16T20:58:59.179962+0000","last_clean":"2023-10-16T20:58:59.179962+0000","last_became_active":"2023-10-16T20:36:55.277873+0000","last_became_peered":"2023-10-16T20:36:55.277873+0000","last_unstale":"2023-10-16T20:58:59.179962+0000","last_undegraded":"2023-10-16T20:58:59.179962+0000","last_fullsized":"2023-10-16T20:58:59.179962+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'109","last_scrub_stamp":"2023-10-16T20:58:59.179829+0000","last_deep_scrub":"67'109","last_deep_scrub_stamp":"2023-10-16T20:58:59.179829+0000","last_clean_scrub_stamp":"2023-10-16T20:58:59.179829+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:34:00.424120+0000","scrub_duration":0.0027646039999999999,"objects_trimmed":0,"snaptrim_duration":0.012247773,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'608","reported_seq":3254,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:07.721115+0000","last_change":"2023-10-16T20:59:07.721210+0000","last_active":"2023-10-16T20:59:07.721115+0000","last_peered":"2023-10-16T20:59:07.721115+0000","last_clean":"2023-10-16T20:59:07.721115+0000","last_became_active":"2023-10-16T20:36:59.302435+0000","last_became_peered":"2023-10-16T20:36:59.302435+0000","last_unstale":"2023-10-16T20:59:07.721115+0000","last_undegraded":"2023-10-16T20:59:07.721115+0000","last_fullsized":"2023-10-16T20:59:07.721115+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'608","last_scrub_stamp":"2023-10-16T20:59:07.721011+0000","last_deep_scrub":"67'608","last_deep_scrub_stamp":"2023-10-16T20:59:07.721011+0000","last_clean_scrub_stamp":"2023-10-16T20:59:07.721011+0000","objects_scrubbed":323,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:33:34.190513+0000","scrub_duration":0.124636293,"objects_trimmed":0,"snaptrim_duration":0.001061892,"stat_sum":{"num_bytes":16477397,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":519,"num_read_kb":10696,"num_write":606,"num_write_kb":24605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"67'240","reported_seq":319,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:09.221980+0000","last_change":"2023-10-16T20:59:09.221980+0000","last_active":"2023-10-16T20:59:09.221980+0000","last_peered":"2023-10-16T20:59:09.221980+0000","last_clean":"2023-10-16T20:59:09.221980+0000","last_became_active":"2023-10-16T20:36:54.755834+0000","last_became_peered":"2023-10-16T20:36:54.755834+0000","last_unstale":"2023-10-16T20:59:09.221980+0000","last_undegraded":"2023-10-16T20:59:09.221980+0000","last_fullsized":"2023-10-16T20:59:09.221980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'240","last_scrub_stamp":"2023-10-16T20:59:09.221846+0000","last_deep_scrub":"67'240","last_deep_scrub_stamp":"2023-10-16T20:59:09.221846+0000","last_clean_scrub_stamp":"2023-10-16T20:59:09.221846+0000","objects_scrubbed":8,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:30.882044+0000","scrub_duration":0.036256330000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582946,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":16385,"num_write":235,"num_write_kb":12609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12208,"num_omap_keys":23,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"67'104","reported_seq":2222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:51.283381+0000","last_change":"2023-10-16T20:58:51.283381+0000","last_active":"2023-10-16T20:58:51.283381+0000","last_peered":"2023-10-16T20:58:51.283381+0000","last_clean":"2023-10-16T20:58:51.283381+0000","last_became_active":"2023-10-16T20:36:55.279333+0000","last_became_peered":"2023-10-16T20:36:55.279333+0000","last_unstale":"2023-10-16T20:58:51.283381+0000","last_undegraded":"2023-10-16T20:58:51.283381+0000","last_fullsized":"2023-10-16T20:58:51.283381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'104","last_scrub_stamp":"2023-10-16T20:58:51.283239+0000","last_deep_scrub":"67'104","last_deep_scrub_stamp":"2023-10-16T20:58:51.283239+0000","last_clean_scrub_stamp":"2023-10-16T20:58:51.283239+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:12.385879+0000","scrub_duration":0.0020579999999999999,"objects_trimmed":0,"snaptrim_duration":0.0047896609999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'629","reported_seq":3443,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:22.755040+0000","last_change":"2023-10-16T20:47:31.586162+0000","last_active":"2023-10-16T20:58:22.755040+0000","last_peered":"2023-10-16T20:58:22.755040+0000","last_clean":"2023-10-16T20:58:22.755040+0000","last_became_active":"2023-10-16T20:36:59.301864+0000","last_became_peered":"2023-10-16T20:36:59.301864+0000","last_unstale":"2023-10-16T20:58:22.755040+0000","last_undegraded":"2023-10-16T20:58:22.755040+0000","last_fullsized":"2023-10-16T20:58:22.755040+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:58.266878+0000","last_clean_scrub_stamp":"2023-10-16T20:36:58.266878+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080450769999999994,"stat_sum":{"num_bytes":20568472,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":708,"num_read_kb":22370,"num_write":628,"num_write_kb":40892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"67'120","reported_seq":2239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:14.471167+0000","last_change":"2023-10-16T20:59:14.471217+0000","last_active":"2023-10-16T20:59:14.471167+0000","last_peered":"2023-10-16T20:59:14.471167+0000","last_clean":"2023-10-16T20:59:14.471167+0000","last_became_active":"2023-10-16T20:36:55.287442+0000","last_became_peered":"2023-10-16T20:36:55.287442+0000","last_unstale":"2023-10-16T20:59:14.471167+0000","last_undegraded":"2023-10-16T20:59:14.471167+0000","last_fullsized":"2023-10-16T20:59:14.471167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'120","last_scrub_stamp":"2023-10-16T20:59:14.471080+0000","last_deep_scrub":"67'120","last_deep_scrub_stamp":"2023-10-16T20:59:14.471080+0000","last_clean_scrub_stamp":"2023-10-16T20:59:14.471080+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:17.500010+0000","scrub_duration":0.0021998500000000002,"objects_trimmed":0,"snaptrim_duration":0.011630351000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"67'131","reported_seq":2250,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:58.645708+0000","last_change":"2023-10-16T20:58:58.645760+0000","last_active":"2023-10-16T20:58:58.645708+0000","last_peered":"2023-10-16T20:58:58.645708+0000","last_clean":"2023-10-16T20:58:58.645708+0000","last_became_active":"2023-10-16T20:36:55.276276+0000","last_became_peered":"2023-10-16T20:36:55.276276+0000","last_unstale":"2023-10-16T20:58:58.645708+0000","last_undegraded":"2023-10-16T20:58:58.645708+0000","last_fullsized":"2023-10-16T20:58:58.645708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'131","last_scrub_stamp":"2023-10-16T20:58:58.645616+0000","last_deep_scrub":"67'131","last_deep_scrub_stamp":"2023-10-16T20:58:58.645616+0000","last_clean_scrub_stamp":"2023-10-16T20:58:58.645616+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:28.761446+0000","scrub_duration":0.0024049459999999998,"objects_trimmed":0,"snaptrim_duration":0.00091802500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"67'115","reported_seq":2241,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:12.209102+0000","last_change":"2023-10-16T20:59:12.209156+0000","last_active":"2023-10-16T20:59:12.209102+0000","last_peered":"2023-10-16T20:59:12.209102+0000","last_clean":"2023-10-16T20:59:12.209102+0000","last_became_active":"2023-10-16T20:37:37.413040+0000","last_became_peered":"2023-10-16T20:37:37.413040+0000","last_unstale":"2023-10-16T20:59:12.209102+0000","last_undegraded":"2023-10-16T20:59:12.209102+0000","last_fullsized":"2023-10-16T20:59:12.209102+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'115","last_scrub_stamp":"2023-10-16T20:59:12.208990+0000","last_deep_scrub":"67'115","last_deep_scrub_stamp":"2023-10-16T20:59:12.208990+0000","last_clean_scrub_stamp":"2023-10-16T20:59:12.208990+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:34:02.568264+0000","scrub_duration":0.0026514730000000001,"objects_trimmed":0,"snaptrim_duration":0.012618513,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"67'116","reported_seq":2235,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:04.262507+0000","last_change":"2023-10-16T20:59:04.262559+0000","last_active":"2023-10-16T20:59:04.262507+0000","last_peered":"2023-10-16T20:59:04.262507+0000","last_clean":"2023-10-16T20:59:04.262507+0000","last_became_active":"2023-10-16T20:36:55.275349+0000","last_became_peered":"2023-10-16T20:36:55.275349+0000","last_unstale":"2023-10-16T20:59:04.262507+0000","last_undegraded":"2023-10-16T20:59:04.262507+0000","last_fullsized":"2023-10-16T20:59:04.262507+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'116","last_scrub_stamp":"2023-10-16T20:59:04.262407+0000","last_deep_scrub":"67'116","last_deep_scrub_stamp":"2023-10-16T20:59:04.262407+0000","last_clean_scrub_stamp":"2023-10-16T20:59:04.262407+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:42.718879+0000","scrub_duration":0.0019239579999999999,"objects_trimmed":0,"snaptrim_duration":0.012136073000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"67'88","reported_seq":2207,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:53.160971+0000","last_change":"2023-10-16T20:58:53.161022+0000","last_active":"2023-10-16T20:58:53.160971+0000","last_peered":"2023-10-16T20:58:53.160971+0000","last_clean":"2023-10-16T20:58:53.160971+0000","last_became_active":"2023-10-16T20:36:55.277043+0000","last_became_peered":"2023-10-16T20:36:55.277043+0000","last_unstale":"2023-10-16T20:58:53.160971+0000","last_undegraded":"2023-10-16T20:58:53.160971+0000","last_fullsized":"2023-10-16T20:58:53.160971+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'88","last_scrub_stamp":"2023-10-16T20:58:53.160862+0000","last_deep_scrub":"67'88","last_deep_scrub_stamp":"2023-10-16T20:58:53.160862+0000","last_clean_scrub_stamp":"2023-10-16T20:58:53.160862+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:19.651642+0000","scrub_duration":0.001900938,"objects_trimmed":0,"snaptrim_duration":0.0078065369999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"67'89","reported_seq":2208,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:03.022307+0000","last_change":"2023-10-16T20:59:03.022367+0000","last_active":"2023-10-16T20:59:03.022307+0000","last_peered":"2023-10-16T20:59:03.022307+0000","last_clean":"2023-10-16T20:59:03.022307+0000","last_became_active":"2023-10-16T20:36:55.278715+0000","last_became_peered":"2023-10-16T20:36:55.278715+0000","last_unstale":"2023-10-16T20:59:03.022307+0000","last_undegraded":"2023-10-16T20:59:03.022307+0000","last_fullsized":"2023-10-16T20:59:03.022307+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'89","last_scrub_stamp":"2023-10-16T20:59:03.022209+0000","last_deep_scrub":"67'89","last_deep_scrub_stamp":"2023-10-16T20:59:03.022209+0000","last_clean_scrub_stamp":"2023-10-16T20:59:03.022209+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:31.713609+0000","scrub_duration":0.0019901680000000001,"objects_trimmed":0,"snaptrim_duration":0.0067460719999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"67'124","reported_seq":2243,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:03.386234+0000","last_change":"2023-10-16T20:59:03.386284+0000","last_active":"2023-10-16T20:59:03.386234+0000","last_peered":"2023-10-16T20:59:03.386234+0000","last_clean":"2023-10-16T20:59:03.386234+0000","last_became_active":"2023-10-16T20:36:55.301604+0000","last_became_peered":"2023-10-16T20:36:55.301604+0000","last_unstale":"2023-10-16T20:59:03.386234+0000","last_undegraded":"2023-10-16T20:59:03.386234+0000","last_fullsized":"2023-10-16T20:59:03.386234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'124","last_scrub_stamp":"2023-10-16T20:59:03.386139+0000","last_deep_scrub":"67'124","last_deep_scrub_stamp":"2023-10-16T20:59:03.386139+0000","last_clean_scrub_stamp":"2023-10-16T20:59:03.386139+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:18.278253+0000","scrub_duration":0.0020316280000000002,"objects_trimmed":0,"snaptrim_duration":0.012969489000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"67'99","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:01.252753+0000","last_change":"2023-10-16T20:59:01.252799+0000","last_active":"2023-10-16T20:59:01.252753+0000","last_peered":"2023-10-16T20:59:01.252753+0000","last_clean":"2023-10-16T20:59:01.252753+0000","last_became_active":"2023-10-16T20:36:55.276984+0000","last_became_peered":"2023-10-16T20:36:55.276984+0000","last_unstale":"2023-10-16T20:59:01.252753+0000","last_undegraded":"2023-10-16T20:59:01.252753+0000","last_fullsized":"2023-10-16T20:59:01.252753+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'99","last_scrub_stamp":"2023-10-16T20:59:01.252663+0000","last_deep_scrub":"67'99","last_deep_scrub_stamp":"2023-10-16T20:59:01.252663+0000","last_clean_scrub_stamp":"2023-10-16T20:59:01.252663+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:32:54.405775+0000","scrub_duration":0.0025568829999999998,"objects_trimmed":0,"snaptrim_duration":0.0086623480000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'111","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:07.294638+0000","last_change":"2023-10-16T20:59:07.294686+0000","last_active":"2023-10-16T20:59:07.294638+0000","last_peered":"2023-10-16T20:59:07.294638+0000","last_clean":"2023-10-16T20:59:07.294638+0000","last_became_active":"2023-10-16T20:36:55.290023+0000","last_became_peered":"2023-10-16T20:36:55.290023+0000","last_unstale":"2023-10-16T20:59:07.294638+0000","last_undegraded":"2023-10-16T20:59:07.294638+0000","last_fullsized":"2023-10-16T20:59:07.294638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'111","last_scrub_stamp":"2023-10-16T20:59:07.294551+0000","last_deep_scrub":"67'111","last_deep_scrub_stamp":"2023-10-16T20:59:07.294551+0000","last_clean_scrub_stamp":"2023-10-16T20:59:07.294551+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:48:23.755124+0000","scrub_duration":0.0022375450000000001,"objects_trimmed":0,"snaptrim_duration":0.013081514000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'124","reported_seq":2243,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:10.167167+0000","last_change":"2023-10-16T20:59:10.167167+0000","last_active":"2023-10-16T20:59:10.167167+0000","last_peered":"2023-10-16T20:59:10.167167+0000","last_clean":"2023-10-16T20:59:10.167167+0000","last_became_active":"2023-10-16T20:36:55.280186+0000","last_became_peered":"2023-10-16T20:36:55.280186+0000","last_unstale":"2023-10-16T20:59:10.167167+0000","last_undegraded":"2023-10-16T20:59:10.167167+0000","last_fullsized":"2023-10-16T20:59:10.167167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'124","last_scrub_stamp":"2023-10-16T20:59:10.167005+0000","last_deep_scrub":"67'124","last_deep_scrub_stamp":"2023-10-16T20:59:10.167005+0000","last_clean_scrub_stamp":"2023-10-16T20:59:10.167005+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:34.469234+0000","scrub_duration":0.0025810059999999998,"objects_trimmed":0,"snaptrim_duration":0.0083925690000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"67'118","reported_seq":2237,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:09.250466+0000","last_change":"2023-10-16T20:59:09.250466+0000","last_active":"2023-10-16T20:59:09.250466+0000","last_peered":"2023-10-16T20:59:09.250466+0000","last_clean":"2023-10-16T20:59:09.250466+0000","last_became_active":"2023-10-16T20:36:55.297253+0000","last_became_peered":"2023-10-16T20:36:55.297253+0000","last_unstale":"2023-10-16T20:59:09.250466+0000","last_undegraded":"2023-10-16T20:59:09.250466+0000","last_fullsized":"2023-10-16T20:59:09.250466+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'118","last_scrub_stamp":"2023-10-16T20:59:09.250291+0000","last_deep_scrub":"67'118","last_deep_scrub_stamp":"2023-10-16T20:59:09.250291+0000","last_clean_scrub_stamp":"2023-10-16T20:59:09.250291+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:37.809041+0000","scrub_duration":0.0016459720000000001,"objects_trimmed":0,"snaptrim_duration":0.0097685800000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"67'108","reported_seq":2227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:10.288057+0000","last_change":"2023-10-16T20:59:10.288057+0000","last_active":"2023-10-16T20:59:10.288057+0000","last_peered":"2023-10-16T20:59:10.288057+0000","last_clean":"2023-10-16T20:59:10.288057+0000","last_became_active":"2023-10-16T20:36:55.286044+0000","last_became_peered":"2023-10-16T20:36:55.286044+0000","last_unstale":"2023-10-16T20:59:10.288057+0000","last_undegraded":"2023-10-16T20:59:10.288057+0000","last_fullsized":"2023-10-16T20:59:10.288057+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'108","last_scrub_stamp":"2023-10-16T20:59:10.287858+0000","last_deep_scrub":"67'108","last_deep_scrub_stamp":"2023-10-16T20:59:10.287858+0000","last_clean_scrub_stamp":"2023-10-16T20:59:10.287858+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:31:52.358547+0000","scrub_duration":0.002902193,"objects_trimmed":0,"snaptrim_duration":0.011681474000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"67'110","reported_seq":2229,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:10.363741+0000","last_change":"2023-10-16T20:59:10.363741+0000","last_active":"2023-10-16T20:59:10.363741+0000","last_peered":"2023-10-16T20:59:10.363741+0000","last_clean":"2023-10-16T20:59:10.363741+0000","last_became_active":"2023-10-16T20:36:55.286864+0000","last_became_peered":"2023-10-16T20:36:55.286864+0000","last_unstale":"2023-10-16T20:59:10.363741+0000","last_undegraded":"2023-10-16T20:59:10.363741+0000","last_fullsized":"2023-10-16T20:59:10.363741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'110","last_scrub_stamp":"2023-10-16T20:59:10.363590+0000","last_deep_scrub":"67'110","last_deep_scrub_stamp":"2023-10-16T20:59:10.363590+0000","last_clean_scrub_stamp":"2023-10-16T20:59:10.363590+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:11.804155+0000","scrub_duration":0.0018541569999999999,"objects_trimmed":0,"snaptrim_duration":0.012885232999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"67'106","reported_seq":2216,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:37.872663+0000","last_change":"2023-10-16T20:47:31.588333+0000","last_active":"2023-10-16T20:58:37.872663+0000","last_peered":"2023-10-16T20:58:37.872663+0000","last_clean":"2023-10-16T20:58:37.872663+0000","last_became_active":"2023-10-16T20:36:55.292262+0000","last_became_peered":"2023-10-16T20:36:55.292262+0000","last_unstale":"2023-10-16T20:58:37.872663+0000","last_undegraded":"2023-10-16T20:58:37.872663+0000","last_fullsized":"2023-10-16T20:58:37.872663+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0077171749999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"67'107","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:00.659400+0000","last_change":"2023-10-16T20:59:00.659448+0000","last_active":"2023-10-16T20:59:00.659400+0000","last_peered":"2023-10-16T20:59:00.659400+0000","last_clean":"2023-10-16T20:59:00.659400+0000","last_became_active":"2023-10-16T20:36:55.282806+0000","last_became_peered":"2023-10-16T20:36:55.282806+0000","last_unstale":"2023-10-16T20:59:00.659400+0000","last_undegraded":"2023-10-16T20:59:00.659400+0000","last_fullsized":"2023-10-16T20:59:00.659400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'107","last_scrub_stamp":"2023-10-16T20:59:00.659305+0000","last_deep_scrub":"67'107","last_deep_scrub_stamp":"2023-10-16T20:59:00.659305+0000","last_clean_scrub_stamp":"2023-10-16T20:59:00.659305+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:01:49.319877+0000","scrub_duration":0.0024992080000000002,"objects_trimmed":0,"snaptrim_duration":0.00086132999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"67'109","reported_seq":2227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:24.189217+0000","last_change":"2023-10-16T20:59:24.189272+0000","last_active":"2023-10-16T20:59:24.189217+0000","last_peered":"2023-10-16T20:59:24.189217+0000","last_clean":"2023-10-16T20:59:24.189217+0000","last_became_active":"2023-10-16T20:36:55.303095+0000","last_became_peered":"2023-10-16T20:36:55.303095+0000","last_unstale":"2023-10-16T20:59:24.189217+0000","last_undegraded":"2023-10-16T20:59:24.189217+0000","last_fullsized":"2023-10-16T20:59:24.189217+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'109","last_scrub_stamp":"2023-10-16T20:59:24.189109+0000","last_deep_scrub":"67'109","last_deep_scrub_stamp":"2023-10-16T20:59:24.189109+0000","last_clean_scrub_stamp":"2023-10-16T20:59:24.189109+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:42:36.202135+0000","scrub_duration":0.0019574549999999999,"objects_trimmed":0,"snaptrim_duration":0.013746639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"67'119","reported_seq":2237,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:08.257529+0000","last_change":"2023-10-16T20:59:08.257579+0000","last_active":"2023-10-16T20:59:08.257529+0000","last_peered":"2023-10-16T20:59:08.257529+0000","last_clean":"2023-10-16T20:59:08.257529+0000","last_became_active":"2023-10-16T20:36:55.291447+0000","last_became_peered":"2023-10-16T20:36:55.291447+0000","last_unstale":"2023-10-16T20:59:08.257529+0000","last_undegraded":"2023-10-16T20:59:08.257529+0000","last_fullsized":"2023-10-16T20:59:08.257529+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'119","last_scrub_stamp":"2023-10-16T20:59:08.257424+0000","last_deep_scrub":"67'119","last_deep_scrub_stamp":"2023-10-16T20:59:08.257424+0000","last_clean_scrub_stamp":"2023-10-16T20:59:08.257424+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:42.198263+0000","scrub_duration":0.002719015,"objects_trimmed":0,"snaptrim_duration":0.01315413,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"67'107","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:48.072269+0000","last_change":"2023-10-16T20:58:48.072313+0000","last_active":"2023-10-16T20:58:48.072269+0000","last_peered":"2023-10-16T20:58:48.072269+0000","last_clean":"2023-10-16T20:58:48.072269+0000","last_became_active":"2023-10-16T20:36:55.280605+0000","last_became_peered":"2023-10-16T20:36:55.280605+0000","last_unstale":"2023-10-16T20:58:48.072269+0000","last_undegraded":"2023-10-16T20:58:48.072269+0000","last_fullsized":"2023-10-16T20:58:48.072269+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'107","last_scrub_stamp":"2023-10-16T20:58:48.072178+0000","last_deep_scrub":"67'107","last_deep_scrub_stamp":"2023-10-16T20:58:48.072178+0000","last_clean_scrub_stamp":"2023-10-16T20:58:48.072178+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:58:57.439416+0000","scrub_duration":0.0025767939999999999,"objects_trimmed":0,"snaptrim_duration":0.0033133080000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"67'112","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:57.143403+0000","last_change":"2023-10-16T20:58:57.143460+0000","last_active":"2023-10-16T20:58:57.143403+0000","last_peered":"2023-10-16T20:58:57.143403+0000","last_clean":"2023-10-16T20:58:57.143403+0000","last_became_active":"2023-10-16T20:36:55.277094+0000","last_became_peered":"2023-10-16T20:36:55.277094+0000","last_unstale":"2023-10-16T20:58:57.143403+0000","last_undegraded":"2023-10-16T20:58:57.143403+0000","last_fullsized":"2023-10-16T20:58:57.143403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'112","last_scrub_stamp":"2023-10-16T20:58:57.143322+0000","last_deep_scrub":"67'112","last_deep_scrub_stamp":"2023-10-16T20:58:57.143322+0000","last_clean_scrub_stamp":"2023-10-16T20:58:57.143322+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:52:54.678403+0000","scrub_duration":0.0016325790000000001,"objects_trimmed":0,"snaptrim_duration":0.011816411000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"67'94","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.120683+0000","last_change":"2023-10-16T20:58:50.120723+0000","last_active":"2023-10-16T20:58:50.120683+0000","last_peered":"2023-10-16T20:58:50.120683+0000","last_clean":"2023-10-16T20:58:50.120683+0000","last_became_active":"2023-10-16T20:36:55.297199+0000","last_became_peered":"2023-10-16T20:36:55.297199+0000","last_unstale":"2023-10-16T20:58:50.120683+0000","last_undegraded":"2023-10-16T20:58:50.120683+0000","last_fullsized":"2023-10-16T20:58:50.120683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'94","last_scrub_stamp":"2023-10-16T20:58:50.120603+0000","last_deep_scrub":"67'94","last_deep_scrub_stamp":"2023-10-16T20:58:50.120603+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.120603+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:29:38.916058+0000","scrub_duration":0.0022327750000000002,"objects_trimmed":0,"snaptrim_duration":0.0086430910000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"67'90","reported_seq":2208,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:52.287328+0000","last_change":"2023-10-16T20:58:52.287408+0000","last_active":"2023-10-16T20:58:52.287328+0000","last_peered":"2023-10-16T20:58:52.287328+0000","last_clean":"2023-10-16T20:58:52.287328+0000","last_became_active":"2023-10-16T20:36:55.286964+0000","last_became_peered":"2023-10-16T20:36:55.286964+0000","last_unstale":"2023-10-16T20:58:52.287328+0000","last_undegraded":"2023-10-16T20:58:52.287328+0000","last_fullsized":"2023-10-16T20:58:52.287328+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'90","last_scrub_stamp":"2023-10-16T20:58:52.287221+0000","last_deep_scrub":"67'90","last_deep_scrub_stamp":"2023-10-16T20:58:52.287221+0000","last_clean_scrub_stamp":"2023-10-16T20:58:52.287221+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:06.754055+0000","scrub_duration":0.0018582900000000001,"objects_trimmed":0,"snaptrim_duration":0.0051857609999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"67'110","reported_seq":2235,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:08.351680+0000","last_change":"2023-10-16T20:59:08.351732+0000","last_active":"2023-10-16T20:59:08.351680+0000","last_peered":"2023-10-16T20:59:08.351680+0000","last_clean":"2023-10-16T20:59:08.351680+0000","last_became_active":"2023-10-16T20:37:37.413492+0000","last_became_peered":"2023-10-16T20:37:37.413492+0000","last_unstale":"2023-10-16T20:59:08.351680+0000","last_undegraded":"2023-10-16T20:59:08.351680+0000","last_fullsized":"2023-10-16T20:59:08.351680+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'110","last_scrub_stamp":"2023-10-16T20:59:08.351593+0000","last_deep_scrub":"67'110","last_deep_scrub_stamp":"2023-10-16T20:59:08.351593+0000","last_clean_scrub_stamp":"2023-10-16T20:59:08.351593+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:24:02.460822+0000","scrub_duration":0.0019714580000000002,"objects_trimmed":0,"snaptrim_duration":0.014529932000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"67'106","reported_seq":2224,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:22.427898+0000","last_change":"2023-10-16T20:59:22.427941+0000","last_active":"2023-10-16T20:59:22.427898+0000","last_peered":"2023-10-16T20:59:22.427898+0000","last_clean":"2023-10-16T20:59:22.427898+0000","last_became_active":"2023-10-16T20:36:55.302468+0000","last_became_peered":"2023-10-16T20:36:55.302468+0000","last_unstale":"2023-10-16T20:59:22.427898+0000","last_undegraded":"2023-10-16T20:59:22.427898+0000","last_fullsized":"2023-10-16T20:59:22.427898+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'106","last_scrub_stamp":"2023-10-16T20:59:22.427826+0000","last_deep_scrub":"67'106","last_deep_scrub_stamp":"2023-10-16T20:59:22.427826+0000","last_clean_scrub_stamp":"2023-10-16T20:59:22.427826+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:26.180362+0000","scrub_duration":0.001392236,"objects_trimmed":0,"snaptrim_duration":0.012359399,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"67'92","reported_seq":2210,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:26.003456+0000","last_change":"2023-10-16T20:59:26.003513+0000","last_active":"2023-10-16T20:59:26.003456+0000","last_peered":"2023-10-16T20:59:26.003456+0000","last_clean":"2023-10-16T20:59:26.003456+0000","last_became_active":"2023-10-16T20:36:55.296476+0000","last_became_peered":"2023-10-16T20:36:55.296476+0000","last_unstale":"2023-10-16T20:59:26.003456+0000","last_undegraded":"2023-10-16T20:59:26.003456+0000","last_fullsized":"2023-10-16T20:59:26.003456+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'92","last_scrub_stamp":"2023-10-16T20:59:26.003338+0000","last_deep_scrub":"67'92","last_deep_scrub_stamp":"2023-10-16T20:59:26.003338+0000","last_clean_scrub_stamp":"2023-10-16T20:59:26.003338+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:59.321272+0000","scrub_duration":0.001948794,"objects_trimmed":0,"snaptrim_duration":0.0072035780000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'95","reported_seq":2213,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:56.206084+0000","last_change":"2023-10-16T20:58:56.206084+0000","last_active":"2023-10-16T20:58:56.206084+0000","last_peered":"2023-10-16T20:58:56.206084+0000","last_clean":"2023-10-16T20:58:56.206084+0000","last_became_active":"2023-10-16T20:36:55.293687+0000","last_became_peered":"2023-10-16T20:36:55.293687+0000","last_unstale":"2023-10-16T20:58:56.206084+0000","last_undegraded":"2023-10-16T20:58:56.206084+0000","last_fullsized":"2023-10-16T20:58:56.206084+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'95","last_scrub_stamp":"2023-10-16T20:58:56.205959+0000","last_deep_scrub":"67'95","last_deep_scrub_stamp":"2023-10-16T20:58:56.205959+0000","last_clean_scrub_stamp":"2023-10-16T20:58:56.205959+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:53.185697+0000","scrub_duration":0.0025542830000000001,"objects_trimmed":0,"snaptrim_duration":0.0074548399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"67'127","reported_seq":2245,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:19.869758+0000","last_change":"2023-10-16T20:59:19.869758+0000","last_active":"2023-10-16T20:59:19.869758+0000","last_peered":"2023-10-16T20:59:19.869758+0000","last_clean":"2023-10-16T20:59:19.869758+0000","last_became_active":"2023-10-16T20:36:55.280444+0000","last_became_peered":"2023-10-16T20:36:55.280444+0000","last_unstale":"2023-10-16T20:59:19.869758+0000","last_undegraded":"2023-10-16T20:59:19.869758+0000","last_fullsized":"2023-10-16T20:59:19.869758+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'127","last_scrub_stamp":"2023-10-16T20:59:19.869537+0000","last_deep_scrub":"67'127","last_deep_scrub_stamp":"2023-10-16T20:59:19.869537+0000","last_clean_scrub_stamp":"2023-10-16T20:59:19.869537+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:58:41.974943+0000","scrub_duration":0.0024841920000000001,"objects_trimmed":0,"snaptrim_duration":0.0068623299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'109","reported_seq":2227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.544632+0000","last_change":"2023-10-16T20:58:50.544632+0000","last_active":"2023-10-16T20:58:50.544632+0000","last_peered":"2023-10-16T20:58:50.544632+0000","last_clean":"2023-10-16T20:58:50.544632+0000","last_became_active":"2023-10-16T20:36:55.296694+0000","last_became_peered":"2023-10-16T20:36:55.296694+0000","last_unstale":"2023-10-16T20:58:50.544632+0000","last_undegraded":"2023-10-16T20:58:50.544632+0000","last_fullsized":"2023-10-16T20:58:50.544632+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'109","last_scrub_stamp":"2023-10-16T20:58:50.544515+0000","last_deep_scrub":"67'109","last_deep_scrub_stamp":"2023-10-16T20:58:50.544515+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.544515+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:15:02.193326+0000","scrub_duration":0.001832127,"objects_trimmed":0,"snaptrim_duration":0.0096291870000000009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"67'120","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.600847+0000","last_change":"2023-10-16T20:47:31.585319+0000","last_active":"2023-10-16T20:58:50.600847+0000","last_peered":"2023-10-16T20:58:50.600847+0000","last_clean":"2023-10-16T20:58:50.600847+0000","last_became_active":"2023-10-16T20:36:55.291052+0000","last_became_peered":"2023-10-16T20:36:55.291052+0000","last_unstale":"2023-10-16T20:58:50.600847+0000","last_undegraded":"2023-10-16T20:58:50.600847+0000","last_fullsized":"2023-10-16T20:58:50.600847+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T20:36:54.718951+0000","last_clean_scrub_stamp":"2023-10-16T20:36:54.718951+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070559409999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"67'94","reported_seq":2214,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:49.588755+0000","last_change":"2023-10-16T20:58:49.588755+0000","last_active":"2023-10-16T20:58:49.588755+0000","last_peered":"2023-10-16T20:58:49.588755+0000","last_clean":"2023-10-16T20:58:49.588755+0000","last_became_active":"2023-10-16T20:36:55.290386+0000","last_became_peered":"2023-10-16T20:36:55.290386+0000","last_unstale":"2023-10-16T20:58:49.588755+0000","last_undegraded":"2023-10-16T20:58:49.588755+0000","last_fullsized":"2023-10-16T20:58:49.588755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'94","last_scrub_stamp":"2023-10-16T20:58:49.588589+0000","last_deep_scrub":"67'94","last_deep_scrub_stamp":"2023-10-16T20:58:49.588589+0000","last_clean_scrub_stamp":"2023-10-16T20:58:49.588589+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:41:38.561038+0000","scrub_duration":0.0020095009999999999,"objects_trimmed":0,"snaptrim_duration":0.0093217969999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"67'114","reported_seq":2233,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.138708+0000","last_change":"2023-10-16T20:58:50.138746+0000","last_active":"2023-10-16T20:58:50.138708+0000","last_peered":"2023-10-16T20:58:50.138708+0000","last_clean":"2023-10-16T20:58:50.138708+0000","last_became_active":"2023-10-16T20:36:55.270787+0000","last_became_peered":"2023-10-16T20:36:55.270787+0000","last_unstale":"2023-10-16T20:58:50.138708+0000","last_undegraded":"2023-10-16T20:58:50.138708+0000","last_fullsized":"2023-10-16T20:58:50.138708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'114","last_scrub_stamp":"2023-10-16T20:58:50.138632+0000","last_deep_scrub":"67'114","last_deep_scrub_stamp":"2023-10-16T20:58:50.138632+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.138632+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:35:28.601665+0000","scrub_duration":0.0015404730000000001,"objects_trimmed":0,"snaptrim_duration":0.010801276,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"67'94","reported_seq":2213,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.047134+0000","last_change":"2023-10-16T20:58:50.047177+0000","last_active":"2023-10-16T20:58:50.047134+0000","last_peered":"2023-10-16T20:58:50.047134+0000","last_clean":"2023-10-16T20:58:50.047134+0000","last_became_active":"2023-10-16T20:36:55.271373+0000","last_became_peered":"2023-10-16T20:36:55.271373+0000","last_unstale":"2023-10-16T20:58:50.047134+0000","last_undegraded":"2023-10-16T20:58:50.047134+0000","last_fullsized":"2023-10-16T20:58:50.047134+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'94","last_scrub_stamp":"2023-10-16T20:58:50.047059+0000","last_deep_scrub":"67'94","last_deep_scrub_stamp":"2023-10-16T20:58:50.047059+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.047059+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:24.904268+0000","scrub_duration":0.0017358,"objects_trimmed":0,"snaptrim_duration":0.006091478,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"67'111","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:26.976263+0000","last_change":"2023-10-16T20:59:26.976305+0000","last_active":"2023-10-16T20:59:26.976263+0000","last_peered":"2023-10-16T20:59:26.976263+0000","last_clean":"2023-10-16T20:59:26.976263+0000","last_became_active":"2023-10-16T20:36:55.291016+0000","last_became_peered":"2023-10-16T20:36:55.291016+0000","last_unstale":"2023-10-16T20:59:26.976263+0000","last_undegraded":"2023-10-16T20:59:26.976263+0000","last_fullsized":"2023-10-16T20:59:26.976263+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'111","last_scrub_stamp":"2023-10-16T20:59:26.976174+0000","last_deep_scrub":"67'111","last_deep_scrub_stamp":"2023-10-16T20:59:26.976174+0000","last_clean_scrub_stamp":"2023-10-16T20:59:26.976174+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:14:47.120578+0000","scrub_duration":0.001725011,"objects_trimmed":0,"snaptrim_duration":0.009370148,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":924319467,"num_objects":10294,"num_object_clones":0,"num_object_copies":41176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10294,"num_whiteouts":0,"num_read":30602,"num_read_kb":676969,"num_write":21002,"num_write_kb":1302825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966899200,"data_stored":1966899200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21050,"ondisk_log_size":21050,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6794,"ondisk_log_size":6794,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":179301891,"num_objects":198,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1063,"num_read_kb":205540,"num_write":6375,"num_write_kb":180866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":115,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":468586,"num_omap_keys":892,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":358858752,"data_stored":358603782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6286,"ondisk_log_size":6286,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509716,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396784,"kb_used_data":280936,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93975056,"statfs":{"total":96636764160,"available":96230457344,"internally_reserved":0,"allocated":287678464,"data_stored":287278454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":118618571},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.56399999999999995,"15min":0.628},"min":{"1min":0.49199999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.113,"5min":1.113,"15min":2.4489999999999998},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.58199999999999996,"15min":0.64900000000000002},"min":{"1min":0.40999999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":11.77},"last":0.58299999999999996}]},{"osd":1,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.59299999999999997,"15min":0.65300000000000002},"min":{"1min":0.38400000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":4.5279999999999996},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.61599999999999999,"15min":0.68500000000000005},"min":{"1min":0.39400000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.006,"5min":1.024,"15min":11.971},"last":0.81799999999999995}]},{"osd":2,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.63500000000000001,"15min":0.67400000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34300000000000003},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":7.1130000000000004},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.65700000000000003,"15min":0.71299999999999997},"min":{"1min":0.44500000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":9.4450000000000003},"last":0.85999999999999999}]},{"osd":3,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.65700000000000003,"15min":0.68400000000000005},"min":{"1min":0.438,"5min":0.437,"15min":0.41299999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":3.363},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.69999999999999996,"15min":0.73099999999999998},"min":{"1min":0.57099999999999995,"5min":0.46899999999999997,"15min":0.42099999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":3.3980000000000001},"last":0.879}]},{"osd":4,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55500000000000005,"15min":0.61399999999999999},"min":{"1min":0.32900000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81399999999999995,"5min":0.84099999999999997,"15min":12.532999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.42299999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.875,"5min":0.90600000000000003,"15min":3.8700000000000001},"last":0.72699999999999998}]},{"osd":5,"last update":"Mon Oct 16 20:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60999999999999999,"15min":0.66300000000000003},"min":{"1min":0.41399999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.85699999999999998,"5min":1.042,"15min":2.2010000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.63600000000000001,"15min":0.73699999999999999},"min":{"1min":0.47499999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94399999999999995,"5min":1.0369999999999999,"15min":12.353},"last":0.92400000000000004}]},{"osd":6,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.64500000000000002,"15min":0.748},"min":{"1min":0.44600000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90400000000000003,"5min":0.93799999999999994,"15min":11.929},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.67800000000000005,"15min":0.71899999999999997},"min":{"1min":0.47899999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.026,"5min":1.026,"15min":6.0359999999999996},"last":0.89200000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509716,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422268,"kb_used_data":300852,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93949572,"statfs":{"total":96636764160,"available":96204361728,"internally_reserved":0,"allocated":308072448,"data_stored":307684288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124320199},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:59:15 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.57699999999999996,"15min":0.61599999999999999},"min":{"1min":0.35099999999999998,"5min":0.27900000000000003,"15min":0.27800000000000002},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":4.5419999999999998},"last":0.498},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.59099999999999997,"15min":0.625},"min":{"1min":0.36899999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.90200000000000002,"5min":2.0329999999999999,"15min":4.9699999999999998},"last":0.60399999999999998}]},{"osd":1,"last update":"Mon Oct 16 20:59:15 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.64600000000000002,"15min":0.64300000000000002},"min":{"1min":0.43099999999999999,"5min":0.39200000000000002,"15min":0.377},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":2.8610000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.41399999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.95299999999999996,"5min":1.014,"15min":1.0309999999999999},"last":0.63600000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:59:15 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.61099999999999999,"15min":0.64500000000000002},"min":{"1min":0.40799999999999997,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.02,"5min":1.0289999999999999,"15min":1.895},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.68000000000000005,"15min":0.68700000000000006},"min":{"1min":0.48799999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":2.3340000000000001},"last":0.621}]},{"osd":3,"last update":"Mon Oct 16 20:59:15 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.77300000000000002,"15min":0.72799999999999998},"min":{"1min":0.60699999999999998,"5min":0.45500000000000002,"15min":0.42299999999999999},"max":{"1min":1.242,"5min":4.6340000000000003,"15min":4.6340000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.65200000000000002,"15min":0.69199999999999995},"min":{"1min":0.48399999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.96699999999999997,"5min":1.163,"15min":5.1029999999999998},"last":0.65300000000000002}]},{"osd":4,"last update":"Mon Oct 16 20:59:15 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.56999999999999995,"15min":0.59599999999999997},"min":{"1min":0.441,"5min":0.34799999999999998,"15min":0.30299999999999999},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":4.8959999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.63,"15min":0.63700000000000001},"min":{"1min":0.39300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":1.1000000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.45800000000000002}]},{"osd":5,"last update":"Mon Oct 16 20:59:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.59899999999999998,"15min":0.625},"min":{"1min":0.374,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":2.3769999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.63900000000000001,"15min":0.66600000000000004},"min":{"1min":0.46000000000000002,"5min":0.38200000000000001,"15min":0.34499999999999997},"max":{"1min":1.196,"5min":1.196,"15min":4.9379999999999997},"last":0.67600000000000005}]},{"osd":7,"last update":"Mon Oct 16 20:59:15 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.70399999999999996,"15min":0.71399999999999997},"min":{"1min":0.435,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.147,"5min":1.425,"15min":4.5789999999999997},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.76500000000000001,"15min":0.751},"min":{"1min":0.47899999999999998,"5min":0.44600000000000001,"15min":0.39300000000000002},"max":{"1min":1.206,"5min":1.6120000000000001,"15min":5.2460000000000004},"last":0.66400000000000003}]}]},{"osd":5,"up_from":15,"seq":64424509718,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325160,"kb_used_data":225952,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94046680,"statfs":{"total":96636764160,"available":96303800320,"internally_reserved":0,"allocated":231374848,"data_stored":230972642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":101579207},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.57199999999999995,"15min":0.61299999999999999},"min":{"1min":0.30499999999999999,"5min":0.28199999999999997,"15min":0.25700000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":2.4580000000000002},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.59299999999999997,"15min":0.65800000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.94999999999999996,"5min":1.1799999999999999,"15min":7.181},"last":0.80600000000000005}]},{"osd":1,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":0.76800000000000002,"15min":0.69099999999999995},"min":{"1min":0.36699999999999999,"5min":0.33900000000000002,"15min":0.29899999999999999},"max":{"1min":16.655000000000001,"5min":16.655000000000001,"15min":16.655000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.5529999999999999,"5min":0.80100000000000005,"15min":0.71299999999999997},"min":{"1min":0.35099999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":16.289000000000001,"5min":16.289000000000001,"15min":16.289000000000001},"last":0.71799999999999997}]},{"osd":2,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.64900000000000002,"15min":0.67300000000000004},"min":{"1min":0.51800000000000002,"5min":0.34599999999999997,"15min":0.308},"max":{"1min":0.86699999999999999,"5min":1.7110000000000001,"15min":3.3799999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.67500000000000004,"15min":0.67800000000000005},"min":{"1min":0.55500000000000005,"5min":0.39800000000000002,"15min":0.38800000000000001},"max":{"1min":0.86899999999999999,"5min":1.583,"15min":4.5369999999999999},"last":0.86899999999999999}]},{"osd":3,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.67700000000000005,"15min":0.69999999999999996},"min":{"1min":0.495,"5min":0.39400000000000002,"15min":0.34899999999999998},"max":{"1min":0.91700000000000004,"5min":1.329,"15min":4.1070000000000002},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68100000000000005,"15min":0.69299999999999995},"min":{"1min":0.52800000000000002,"5min":0.434,"15min":0.38},"max":{"1min":0.83099999999999996,"5min":1.1679999999999999,"15min":3.476},"last":0.83099999999999996}]},{"osd":4,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51600000000000001,"15min":0.55500000000000005},"min":{"1min":0.35299999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.873,"5min":0.873,"15min":2.415},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57299999999999995,"15min":0.65600000000000003},"min":{"1min":0.41999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81999999999999995,"5min":1.155,"15min":3.2919999999999998},"last":0.67700000000000005}]},{"osd":6,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.56799999999999995,"15min":0.59399999999999997},"min":{"1min":0.374,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.91200000000000003,"5min":0.93799999999999994,"15min":3.2400000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.40000000000000002,"5min":0.38500000000000001,"15min":0.31},"max":{"1min":0.80200000000000005,"5min":1.036,"15min":1.2310000000000001},"last":0.57499999999999996}]},{"osd":7,"last update":"Mon Oct 16 20:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.44600000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76300000000000001,"5min":1.23,"15min":5.2759999999999998},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.69399999999999995,"15min":0.70699999999999996},"min":{"1min":0.45000000000000001,"5min":0.435,"15min":0.42299999999999999},"max":{"1min":0.83199999999999996,"5min":0.97699999999999998,"15min":3.3140000000000001},"last":0.91200000000000003}]}]},{"osd":4,"up_from":15,"seq":64424509716,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420128,"kb_used_data":290392,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93951712,"statfs":{"total":96636764160,"available":96206553088,"internally_reserved":0,"allocated":297361408,"data_stored":296975417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.56100000000000005,"15min":0.61299999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.83799999999999997,"5min":0.91400000000000003,"15min":1.524},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61699999999999999,"15min":0.69199999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":1.5649999999999999},"last":0.47299999999999998}]},{"osd":1,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.376,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.1749999999999998,"5min":4.1749999999999998,"15min":4.1749999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.625,"15min":0.66600000000000004},"min":{"1min":0.38200000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82599999999999996,"5min":0.94899999999999995,"15min":1.1020000000000001},"last":0.55300000000000005}]},{"osd":2,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65500000000000003,"15min":0.66300000000000003},"min":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.38900000000000001},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":2.3370000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.68300000000000005,"15min":0.71199999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.84499999999999997,"5min":1.2210000000000001,"15min":2.3660000000000001},"last":0.56100000000000005}]},{"osd":3,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.377},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":2.04},"last":0.44},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":2.2240000000000002},"last":0.49199999999999999}]},{"osd":5,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.58799999999999997,"15min":0.60299999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.9970000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.63600000000000001,"15min":0.60499999999999998},"min":{"1min":0.41299999999999998,"5min":0.39500000000000002,"15min":0.33900000000000002},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.60899999999999999}]},{"osd":6,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.60299999999999998,"15min":0.61599999999999999},"min":{"1min":0.33900000000000002,"5min":0.315,"15min":0.312},"max":{"1min":1.2270000000000001,"5min":1.6339999999999999,"15min":2.2639999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.66200000000000003,"15min":0.67300000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.8300000000000001},"last":0.51100000000000001}]},{"osd":7,"last update":"Mon Oct 16 20:58:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.69399999999999995,"15min":0.66800000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2969999999999999,"5min":1.347,"15min":1.347},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.65500000000000003,"15min":0.67900000000000005},"min":{"1min":0.38600000000000001,"5min":0.35899999999999999,"15min":0.35599999999999998},"max":{"1min":1.266,"5min":1.266,"15min":2.2999999999999998},"last":0.58899999999999997}]}]},{"osd":3,"up_from":15,"seq":64424509716,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485596,"kb_used_data":360020,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93886244,"statfs":{"total":96636764160,"available":96139513856,"internally_reserved":0,"allocated":368660480,"data_stored":368273774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":128580040},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.442,"15min":0.434},"min":{"1min":0.27400000000000002,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":0.69099999999999995,"5min":1.371,"15min":4.5830000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.499,"5min":0.46600000000000003,"15min":0.44900000000000001},"min":{"1min":0.35399999999999998,"5min":0.27100000000000002,"15min":0.25700000000000001},"max":{"1min":0.69299999999999995,"5min":1.147,"15min":1.2509999999999999},"last":0.313}]},{"osd":1,"last update":"Mon Oct 16 20:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.41199999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.82899999999999996,"5min":1.411,"15min":4.6840000000000002},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.51700000000000002,"15min":0.50700000000000001},"min":{"1min":0.36299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71599999999999997,"5min":2.6360000000000001,"15min":2.7429999999999999},"last":0.46700000000000003}]},{"osd":2,"last update":"Mon Oct 16 20:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57399999999999995,"15min":0.55400000000000005},"min":{"1min":0.48199999999999998,"5min":0.34399999999999997,"15min":0.311},"max":{"1min":0.70999999999999996,"5min":1.286,"15min":4.6449999999999996},"last":0.503},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.56499999999999995,"15min":0.52200000000000002},"min":{"1min":0.38200000000000001,"5min":0.32200000000000001,"15min":0.30399999999999999},"max":{"1min":0.80400000000000005,"5min":1.1850000000000001,"15min":1.911},"last":0.45000000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.75,"15min":0.76400000000000001},"min":{"1min":0.60299999999999998,"5min":0.45400000000000001,"15min":0.44800000000000001},"max":{"1min":1.1240000000000001,"5min":1.538,"15min":1.7749999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.76200000000000001,"15min":0.745},"min":{"1min":0.58899999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.212,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":0.879}]},{"osd":5,"last update":"Mon Oct 16 20:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.76800000000000002,"15min":0.75600000000000001},"min":{"1min":0.53000000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.1519999999999999,"5min":1.4970000000000001,"15min":4.8449999999999998},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.75700000000000001,"15min":0.74399999999999999},"min":{"1min":0.48899999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.161,"5min":2.948,"15min":2.948},"last":0.754}]},{"osd":6,"last update":"Mon Oct 16 20:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78200000000000003,"15min":0.76500000000000001},"min":{"1min":0.60299999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.044,"5min":1.633,"15min":4.8099999999999996},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.79500000000000004,"15min":0.76900000000000002},"min":{"1min":0.57999999999999996,"5min":0.46400000000000002,"15min":0.45400000000000001},"max":{"1min":1.0680000000000001,"5min":1.6559999999999999,"15min":2.339},"last":0.82799999999999996}]},{"osd":7,"last update":"Mon Oct 16 20:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.81799999999999995,"15min":0.86899999999999999},"min":{"1min":0.56899999999999995,"5min":0.5,"15min":0.46000000000000002},"max":{"1min":1.179,"5min":1.593,"15min":34.039000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.79400000000000004,"15min":0.77500000000000002},"min":{"1min":0.58999999999999997,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.198,"5min":2.984,"15min":2.984},"last":0.86299999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509717,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303812,"kb_used_data":200444,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94068028,"statfs":{"total":96636764160,"available":96325660672,"internally_reserved":0,"allocated":205254656,"data_stored":204864813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42899999999999999,"15min":0.45000000000000001},"min":{"1min":0.27100000000000002,"5min":0.23400000000000001,"15min":0.20599999999999999},"max":{"1min":0.74299999999999999,"5min":0.92500000000000004,"15min":5.5499999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.52600000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.1180000000000001,"5min":1.1220000000000001,"15min":5.4160000000000004},"last":0.49199999999999999}]},{"osd":1,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.504,"15min":0.5},"min":{"1min":0.307,"5min":0.307,"15min":0.26000000000000001},"max":{"1min":0.81899999999999995,"5min":1.25,"15min":2.2349999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.501},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.27200000000000002},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":5.4359999999999999},"last":0.42899999999999999}]},{"osd":3,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.28399999999999997,"5min":0.26400000000000001,"15min":0.25800000000000001},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.312,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.83099999999999996,"5min":1.286,"15min":5.3789999999999996},"last":0.47099999999999997}]},{"osd":4,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.73099999999999998,"15min":0.80000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":3.464,"5min":3.464,"15min":16.905999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.70699999999999996,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.44700000000000001,"15min":0.441},"max":{"1min":0.92100000000000004,"5min":1.3100000000000001,"15min":10.064},"last":0.86899999999999999}]},{"osd":5,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70999999999999996,"15min":0.72999999999999998},"min":{"1min":0.45300000000000001,"5min":0.41099999999999998,"15min":0.32800000000000001},"max":{"1min":1.0960000000000001,"5min":1.304,"15min":3.2989999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.73499999999999999,"15min":0.77100000000000002},"min":{"1min":0.52400000000000002,"5min":0.499,"15min":0.46700000000000003},"max":{"1min":1.0069999999999999,"5min":1.3640000000000001,"15min":5.3330000000000002},"last":0.82499999999999996}]},{"osd":6,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.76500000000000001,"15min":0.755},"min":{"1min":0.438,"5min":0.438,"15min":0.42299999999999999},"max":{"1min":3.5710000000000002,"5min":3.5710000000000002,"15min":3.5710000000000002},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.77900000000000003,"15min":0.77000000000000002},"min":{"1min":0.49099999999999999,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":5.4749999999999996},"last":0.92300000000000004}]},{"osd":7,"last update":"Mon Oct 16 20:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.47399999999999998,"5min":0.47199999999999998,"15min":0.44500000000000001},"max":{"1min":3.5089999999999999,"5min":3.5089999999999999,"15min":5.516},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77200000000000002,"15min":0.79100000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.41899999999999998},"max":{"1min":1.325,"5min":1.325,"15min":8.4610000000000003},"last":0.84499999999999997}]}]},{"osd":1,"up_from":15,"seq":64424509716,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490524,"kb_used_data":360788,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93881316,"statfs":{"total":96636764160,"available":96134467584,"internally_reserved":0,"allocated":369446912,"data_stored":369062219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.437,"15min":0.45700000000000002},"min":{"1min":0.245,"5min":0.219,"15min":0.21299999999999999},"max":{"1min":0.79800000000000004,"5min":0.93000000000000005,"15min":8.9339999999999993},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.34300000000000003,"5min":0.28999999999999998,"15min":0.23000000000000001},"max":{"1min":0.67200000000000004,"5min":1.6899999999999999,"15min":3.8809999999999998},"last":0.39300000000000002}]},{"osd":2,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.502,"15min":0.46000000000000002},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.248},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.55900000000000005,"15min":0.55400000000000005},"min":{"1min":0.29699999999999999,"5min":0.29099999999999998,"15min":0.28499999999999998},"max":{"1min":0.88600000000000001,"5min":2.6739999999999999,"15min":9.2680000000000007},"last":0.44800000000000001}]},{"osd":3,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57499999999999996,"15min":0.54200000000000004},"min":{"1min":0.33000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":1.2030000000000001,"5min":1.9910000000000001,"15min":3.0539999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57999999999999996,"15min":0.57799999999999996},"min":{"1min":0.433,"5min":0.34200000000000003,"15min":0.313},"max":{"1min":0.84999999999999998,"5min":1.4119999999999999,"15min":6.9409999999999998},"last":0.53000000000000003}]},{"osd":4,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.68300000000000005,"15min":0.73199999999999998},"min":{"1min":0.34300000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.90400000000000003,"5min":1.962,"15min":5.6630000000000003},"last":0.501},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.72399999999999998,"15min":0.75900000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.151,"5min":1.6719999999999999,"15min":8.8960000000000008},"last":0.59799999999999998}]},{"osd":5,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.76000000000000001,"15min":0.77700000000000002},"min":{"1min":0.48399999999999999,"5min":0.47299999999999998,"15min":0.40300000000000002},"max":{"1min":1.242,"5min":1.242,"15min":6.6159999999999997},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.77800000000000002,"15min":0.79500000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.1220000000000001,"5min":1.7130000000000001,"15min":5.7009999999999996},"last":0.62}]},{"osd":6,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.73299999999999998,"15min":0.77300000000000002},"min":{"1min":0.47799999999999998,"5min":0.47299999999999998,"15min":0.39800000000000002},"max":{"1min":0.94199999999999995,"5min":2.0529999999999999,"15min":8.6039999999999992},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.77200000000000002,"15min":0.75},"min":{"1min":0.50700000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98199999999999998,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":0.71299999999999997}]},{"osd":7,"last update":"Mon Oct 16 20:58:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76300000000000001,"15min":0.77800000000000002},"min":{"1min":0.46899999999999997,"5min":0.45200000000000001,"15min":0.44800000000000001},"max":{"1min":1.1639999999999999,"5min":1.7470000000000001,"15min":6.6600000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.78700000000000003,"15min":0.80600000000000005},"min":{"1min":0.50700000000000001,"5min":0.46400000000000002,"15min":0.46200000000000002},"max":{"1min":1.1990000000000001,"5min":1.774,"15min":8.8100000000000005},"last":0.58499999999999996}]}]},{"osd":0,"up_from":13,"seq":55834575125,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371112,"kb_used_data":259424,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94000728,"statfs":{"total":96636764160,"available":96256745472,"internally_reserved":0,"allocated":265650176,"data_stored":265251991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.39500000000000002,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.56299999999999994,"5min":1.3029999999999999,"15min":6.8170000000000002},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.46100000000000002,"15min":0.47499999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.70199999999999996,"5min":1.591,"15min":6.2789999999999999},"last":0.47299999999999998}]},{"osd":2,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.51800000000000002,"15min":0.495},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90100000000000002,"5min":1.169,"15min":3.1859999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.47699999999999998,"15min":0.46500000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.97499999999999998,"5min":1.266,"15min":1.47},"last":0.35999999999999999}]},{"osd":3,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56999999999999995,"15min":0.52100000000000002},"min":{"1min":0.40899999999999997,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":0.67900000000000005,"5min":1.423,"15min":1.8180000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.35599999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.93300000000000005,"5min":1.4390000000000001,"15min":1.881},"last":0.44500000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.68799999999999994,"15min":0.80700000000000005},"min":{"1min":0.46400000000000002,"5min":0.45100000000000001,"15min":0.45000000000000001},"max":{"1min":1.0529999999999999,"5min":1.3169999999999999,"15min":18.41},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70399999999999996,"15min":0.81399999999999995},"min":{"1min":0.50800000000000001,"5min":0.46300000000000002,"15min":0.42399999999999999},"max":{"1min":1.097,"5min":1.409,"15min":18.998999999999999},"last":0.71199999999999997}]},{"osd":5,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73699999999999999,"15min":0.76100000000000001},"min":{"1min":0.52400000000000002,"5min":0.50900000000000001,"15min":0.42799999999999999},"max":{"1min":1.1299999999999999,"5min":1.3460000000000001,"15min":3.5499999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69999999999999996,"15min":0.73699999999999999},"min":{"1min":0.45900000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.996,"5min":1.4279999999999999,"15min":3.605},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.73999999999999999,"15min":0.754},"min":{"1min":0.52200000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0289999999999999,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.76600000000000001,"15min":0.78200000000000003},"min":{"1min":0.502,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0840000000000001,"5min":2.367,"15min":2.367},"last":0.65900000000000003}]},{"osd":7,"last update":"Mon Oct 16 20:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.49099999999999999},"max":{"1min":1.042,"5min":1.395,"15min":4.7839999999999998},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73999999999999999,"15min":0.77000000000000002},"min":{"1min":0.57399999999999995,"5min":0.53200000000000003,"15min":0.45500000000000002},"max":{"1min":0.995,"5min":1.3759999999999999,"15min":2.0110000000000001},"last":0.73399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39346176,"data_stored":39310758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51638272,"data_stored":51612794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18219008,"data_stored":18188380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":47730688,"data_stored":47703197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56045568,"data_stored":56018792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":44777472,"data_stored":44734481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40857600,"data_stored":40828655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":60243968,"data_stored":60206725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":224890880,"data_stored":224890880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":316989440,"data_stored":316989440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186216448,"data_stored":186216448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":320110592,"data_stored":320110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":240496640,"data_stored":240496640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":185778176,"data_stored":185778176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266395648,"data_stored":266395648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":226021376,"data_stored":226021376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T20:59:31.766 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.767 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.767 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.768 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.768 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.769 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.770 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.771 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.771 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.772 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.772 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.773 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.773 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.773 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.774 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.774 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.775 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.775 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.776 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2023-10-16T20:36:58.266878+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=58, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.776 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.777 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2023-10-16T20:36:54.718951+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=36, tm_sec=54, tm_wday=0, tm_yday=289, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=16, tm_hour=20, tm_min=58, tm_sec=45, tm_wday=0, tm_yday=289, tm_isdst=0) 2023-10-16T20:59:31.777 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-16T20:59:51.779 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T20:59:51.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.924+0000 7f08dc816640 1 Processor -- start 2023-10-16T20:59:51.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.924+0000 7f08dc816640 1 -- start start 2023-10-16T20:59:51.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08dc816640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08d81537c0 0x7f08d815c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:51.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08dc816640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f08d815cdb0 0x7f08d80c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:51.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08dc816640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f08d80ca2f0 0x7f08d80c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:51.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08dc816640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08d80cae10 con 0x7f08d815cdb0 2023-10-16T20:59:51.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08dc816640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08d80cac30 con 0x7f08d80ca2f0 2023-10-16T20:59:51.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08dc816640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d815d2b0 con 0x7f08d81537c0 2023-10-16T20:59:51.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08d659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f08d815cdb0 0x7f08d80c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:51.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08d659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f08d815cdb0 0x7f08d80c4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57564/0 (socket says 172.21.15.88:57564) 2023-10-16T20:59:51.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08d659c640 1 -- 172.21.15.88:0/634020513 learned_addr learned my addr 172.21.15.88:0/634020513 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:51.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08d6d9d640 1 --2- 172.21.15.88:0/634020513 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08d81537c0 0x7f08d815c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:51.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08d759e640 1 --2- 172.21.15.88:0/634020513 >> v2:172.21.15.191:3300/0 conn(0x7f08d80ca2f0 0x7f08d80c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:51.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08d659c640 1 -- 172.21.15.88:0/634020513 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08d81537c0 msgr2=0x7f08d815c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:59:51.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08d659c640 1 --2- 172.21.15.88:0/634020513 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08d81537c0 0x7f08d815c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:51.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08d659c640 1 -- 172.21.15.88:0/634020513 >> v2:172.21.15.191:3300/0 conn(0x7f08d80ca2f0 msgr2=0x7f08d80c51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:59:51.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08d659c640 1 --2- 172.21.15.88:0/634020513 >> v2:172.21.15.191:3300/0 conn(0x7f08d80ca2f0 0x7f08d80c51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:51.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08d659c640 1 -- 172.21.15.88:0/634020513 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d815d590 con 0x7f08d815cdb0 2023-10-16T20:59:51.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08d6d9d640 1 --2- 172.21.15.88:0/634020513 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08d81537c0 0x7f08d815c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:59:51.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.928+0000 7f08d759e640 1 --2- 172.21.15.88:0/634020513 >> v2:172.21.15.191:3300/0 conn(0x7f08d80ca2f0 0x7f08d80c51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:59:51.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.932+0000 7f08d659c640 1 --2- 172.21.15.88:0/634020513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08d815cdb0 0x7f08d80c4c80 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f08d80cc9f0 tx=0x7f08cc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=136f00f2eb92f4d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:51.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.932+0000 7f08d5d9b640 1 -- 172.21.15.88:0/634020513 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc0a89b0 con 0x7f08d815cdb0 2023-10-16T20:59:51.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.932+0000 7f08d5d9b640 1 -- 172.21.15.88:0/634020513 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08cc0a8b10 con 0x7f08d815cdb0 2023-10-16T20:59:51.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.932+0000 7f08d5d9b640 1 -- 172.21.15.88:0/634020513 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc0a8ea0 con 0x7f08d815cdb0 2023-10-16T20:59:51.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.932+0000 7f08dc816640 1 -- 172.21.15.88:0/634020513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08d815cdb0 msgr2=0x7f08d80c4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:51.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.932+0000 7f08dc816640 1 --2- 172.21.15.88:0/634020513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08d815cdb0 0x7f08d80c4c80 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f08d80cc9f0 tx=0x7f08cc01ecf0 comp rx=0 tx=0).stop 2023-10-16T20:59:51.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.932+0000 7f08dc816640 1 -- 172.21.15.88:0/634020513 shutdown_connections 2023-10-16T20:59:51.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.932+0000 7f08dc816640 1 --2- 172.21.15.88:0/634020513 >> v2:172.21.15.191:3300/0 conn(0x7f08d80ca2f0 0x7f08d80c51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:51.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.932+0000 7f08dc816640 1 --2- 172.21.15.88:0/634020513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08d815cdb0 0x7f08d80c4c80 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:51.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.932+0000 7f08dc816640 1 --2- 172.21.15.88:0/634020513 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08d81537c0 0x7f08d815c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:51.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.932+0000 7f08dc816640 1 -- 172.21.15.88:0/634020513 >> 172.21.15.88:0/634020513 conn(0x7f08d806c170 msgr2=0x7f08d80715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:51.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.932+0000 7f08dc816640 1 -- 172.21.15.88:0/634020513 shutdown_connections 2023-10-16T20:59:51.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.932+0000 7f08dc816640 1 -- 172.21.15.88:0/634020513 wait complete. 2023-10-16T20:59:51.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.932+0000 7f08dc816640 1 Processor -- start 2023-10-16T20:59:51.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.932+0000 7f08dc816640 1 -- start start 2023-10-16T20:59:51.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08dc816640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08d80ca2f0 0x7f08d80c7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:51.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08dc816640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08d81537c0 0x7f08d80c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:51.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08dc816640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08d815cdb0 0x7f08d80c4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:51.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08dc816640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d80c62a0 con 0x7f08d80ca2f0 2023-10-16T20:59:51.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08dc816640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d80c64d0 con 0x7f08d815cdb0 2023-10-16T20:59:51.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08dc816640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d80c67b0 con 0x7f08d81537c0 2023-10-16T20:59:51.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08d6d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08d80ca2f0 0x7f08d80c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:51.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08d6d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08d80ca2f0 0x7f08d80c7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57570/0 (socket says 172.21.15.88:57570) 2023-10-16T20:59:51.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08d6d9d640 1 -- 172.21.15.88:0/2583803704 learned_addr learned my addr 172.21.15.88:0/2583803704 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:51.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08d659c640 1 --2- 172.21.15.88:0/2583803704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08d81537c0 0x7f08d80c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:51.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08d759e640 1 --2- 172.21.15.88:0/2583803704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08d815cdb0 0x7f08d80c4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:51.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08d6d9d640 1 -- 172.21.15.88:0/2583803704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08d81537c0 msgr2=0x7f08d80c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:51.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08d6d9d640 1 --2- 172.21.15.88:0/2583803704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08d81537c0 0x7f08d80c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:51.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08d6d9d640 1 -- 172.21.15.88:0/2583803704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08d815cdb0 msgr2=0x7f08d80c4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:51.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08d6d9d640 1 --2- 172.21.15.88:0/2583803704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08d815cdb0 0x7f08d80c4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:51.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08d6d9d640 1 -- 172.21.15.88:0/2583803704 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08cc0094d0 con 0x7f08d80ca2f0 2023-10-16T20:59:51.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08d759e640 1 --2- 172.21.15.88:0/2583803704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08d815cdb0 0x7f08d80c4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:59:51.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08d6d9d640 1 --2- 172.21.15.88:0/2583803704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08d80ca2f0 0x7f08d80c7290 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f08c800c7c0 tx=0x7f08c800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:51.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08bf7fe640 1 -- 172.21.15.88:0/2583803704 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c8007a30 con 0x7f08d80ca2f0 2023-10-16T20:59:51.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08bf7fe640 1 -- 172.21.15.88:0/2583803704 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c8007b90 con 0x7f08d80ca2f0 2023-10-16T20:59:51.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08dc816640 1 -- 172.21.15.88:0/2583803704 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d80c7ea0 con 0x7f08d80ca2f0 2023-10-16T20:59:51.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08bf7fe640 1 -- 172.21.15.88:0/2583803704 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c800ce70 con 0x7f08d80ca2f0 2023-10-16T20:59:51.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08dc816640 1 -- 172.21.15.88:0/2583803704 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d80c8460 con 0x7f08d80ca2f0 2023-10-16T20:59:51.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.936+0000 7f08dc816640 1 -- 172.21.15.88:0/2583803704 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d80c4180 con 0x7f08d80ca2f0 2023-10-16T20:59:51.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.940+0000 7f08bf7fe640 1 -- 172.21.15.88:0/2583803704 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f08c80113f0 con 0x7f08d80ca2f0 2023-10-16T20:59:51.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.940+0000 7f08bf7fe640 1 --2- 172.21.15.88:0/2583803704 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08c40664a0 0x7f08c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:51.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.940+0000 7f08bf7fe640 1 -- 172.21.15.88:0/2583803704 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f08c80849f0 con 0x7f08d80ca2f0 2023-10-16T20:59:51.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.940+0000 7f08d659c640 1 --2- 172.21.15.88:0/2583803704 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08c40664a0 0x7f08c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:51.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.940+0000 7f08d659c640 1 --2- 172.21.15.88:0/2583803704 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08c40664a0 0x7f08c4068960 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f08cc0b2000 tx=0x7f08cc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:51.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:51.944+0000 7f08bf7fe640 1 -- 172.21.15.88:0/2583803704 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f08d80c4180 con 0x7f08d80ca2f0 2023-10-16T20:59:52.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.092+0000 7f08dc816640 1 -- 172.21.15.88:0/2583803704 --> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f08d80c4480 con 0x7f08c40664a0 2023-10-16T20:59:52.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.100+0000 7f08bf7fe640 1 -- 172.21.15.88:0/2583803704 <== mgr.4110 v2:172.21.15.88:6832/1504518179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+377096 (secure 0 0 0) 0x7f08cc0b0540 con 0x7f08c40664a0 2023-10-16T20:59:52.106 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:59:52.109 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-10-16T20:59:52.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.108+0000 7f08dc816640 1 -- 172.21.15.88:0/2583803704 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08c40664a0 msgr2=0x7f08c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:52.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.108+0000 7f08dc816640 1 --2- 172.21.15.88:0/2583803704 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08c40664a0 0x7f08c4068960 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f08cc0b2000 tx=0x7f08cc009040 comp rx=0 tx=0).stop 2023-10-16T20:59:52.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.108+0000 7f08dc816640 1 -- 172.21.15.88:0/2583803704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08d80ca2f0 msgr2=0x7f08d80c7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:52.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.108+0000 7f08dc816640 1 --2- 172.21.15.88:0/2583803704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08d80ca2f0 0x7f08d80c7290 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f08c800c7c0 tx=0x7f08c800cc80 comp rx=0 tx=0).stop 2023-10-16T20:59:52.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.108+0000 7f08dc816640 1 -- 172.21.15.88:0/2583803704 shutdown_connections 2023-10-16T20:59:52.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.108+0000 7f08dc816640 1 --2- 172.21.15.88:0/2583803704 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f08c40664a0 0x7f08c4068960 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.108+0000 7f08dc816640 1 --2- 172.21.15.88:0/2583803704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f08d815cdb0 0x7f08d80c4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.108+0000 7f08dc816640 1 --2- 172.21.15.88:0/2583803704 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f08d81537c0 0x7f08d80c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.108+0000 7f08dc816640 1 --2- 172.21.15.88:0/2583803704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08d80ca2f0 0x7f08d80c7290 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.108+0000 7f08dc816640 1 -- 172.21.15.88:0/2583803704 >> 172.21.15.88:0/2583803704 conn(0x7f08d806c170 msgr2=0x7f08d815c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:52.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.108+0000 7f08dc816640 1 -- 172.21.15.88:0/2583803704 shutdown_connections 2023-10-16T20:59:52.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.108+0000 7f08dc816640 1 -- 172.21.15.88:0/2583803704 wait complete. 2023-10-16T20:59:52.126 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":756,"stamp":"2023-10-16T20:59:51.949501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1104211726,"num_objects":10494,"num_object_clones":0,"num_object_copies":41576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10494,"num_whiteouts":0,"num_read":31781,"num_read_kb":882609,"num_write":27536,"num_write_kb":1486107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":115,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":468586,"num_omap_keys":892,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34249,"ondisk_log_size":34249,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":3215384,"kb_used_data":2278808,"kb_used_omap":12,"kb_used_meta":936499,"kb_avail":751759336,"statfs":{"total":773094113280,"available":769801560064,"internally_reserved":0,"allocated":2333499392,"data_stored":2330363598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":958975545},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003391"},"pg_stats":[{"pgid":"2.1f","version":"67'338","reported_seq":436,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:58.692768+0000","last_change":"2023-10-16T20:58:53.179067+0000","last_active":"2023-10-16T20:58:58.692768+0000","last_peered":"2023-10-16T20:58:58.692768+0000","last_clean":"2023-10-16T20:58:58.692768+0000","last_became_active":"2023-10-16T20:36:54.763457+0000","last_became_peered":"2023-10-16T20:36:54.763457+0000","last_unstale":"2023-10-16T20:58:58.692768+0000","last_undegraded":"2023-10-16T20:58:58.692768+0000","last_fullsized":"2023-10-16T20:58:58.692768+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'334","last_scrub_stamp":"2023-10-16T20:58:53.178846+0000","last_deep_scrub":"67'334","last_deep_scrub_stamp":"2023-10-16T20:58:53.178846+0000","last_clean_scrub_stamp":"2023-10-16T20:58:53.178846+0000","objects_scrubbed":12,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:41:18.102491+0000","scrub_duration":0.029593793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9874237,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":42,"num_read_kb":9548,"num_write":328,"num_write_kb":9838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":24974,"num_omap_keys":48,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"67'107","reported_seq":2227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:23.206008+0000","last_change":"2023-10-16T20:59:23.206053+0000","last_active":"2023-10-16T20:59:23.206008+0000","last_peered":"2023-10-16T20:59:23.206008+0000","last_clean":"2023-10-16T20:59:23.206008+0000","last_became_active":"2023-10-16T20:36:55.302519+0000","last_became_peered":"2023-10-16T20:36:55.302519+0000","last_unstale":"2023-10-16T20:59:23.206008+0000","last_undegraded":"2023-10-16T20:59:23.206008+0000","last_fullsized":"2023-10-16T20:59:23.206008+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'107","last_scrub_stamp":"2023-10-16T20:59:23.205910+0000","last_deep_scrub":"67'107","last_deep_scrub_stamp":"2023-10-16T20:59:23.205910+0000","last_clean_scrub_stamp":"2023-10-16T20:59:23.205910+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:39:09.164505+0000","scrub_duration":0.0026049739999999999,"objects_trimmed":0,"snaptrim_duration":0.007784165,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'672","reported_seq":3577,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:33.273450+0000","last_change":"2023-10-16T20:59:33.273577+0000","last_active":"2023-10-16T20:59:33.273450+0000","last_peered":"2023-10-16T20:59:33.273450+0000","last_clean":"2023-10-16T20:59:33.273450+0000","last_became_active":"2023-10-16T20:36:59.466306+0000","last_became_peered":"2023-10-16T20:36:59.466306+0000","last_unstale":"2023-10-16T20:59:33.273450+0000","last_undegraded":"2023-10-16T20:59:33.273450+0000","last_fullsized":"2023-10-16T20:59:33.273450+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'672","last_scrub_stamp":"2023-10-16T20:59:33.273341+0000","last_deep_scrub":"67'672","last_deep_scrub_stamp":"2023-10-16T20:59:33.273341+0000","last_clean_scrub_stamp":"2023-10-16T20:59:33.273341+0000","objects_scrubbed":299,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:18:41.326475+0000","scrub_duration":0.12294205800000001,"objects_trimmed":0,"snaptrim_duration":0.0073106899999999999,"stat_sum":{"num_bytes":36865036,"num_objects":299,"num_object_clones":0,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":299,"num_whiteouts":0,"num_read":779,"num_read_kb":25449,"num_write":671,"num_write_kb":56816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"67'210","reported_seq":290,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:08.200626+0000","last_change":"2023-10-16T20:59:08.200674+0000","last_active":"2023-10-16T20:59:08.200626+0000","last_peered":"2023-10-16T20:59:08.200626+0000","last_clean":"2023-10-16T20:59:08.200626+0000","last_became_active":"2023-10-16T20:36:54.759177+0000","last_became_peered":"2023-10-16T20:36:54.759177+0000","last_unstale":"2023-10-16T20:59:08.200626+0000","last_undegraded":"2023-10-16T20:59:08.200626+0000","last_fullsized":"2023-10-16T20:59:08.200626+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'210","last_scrub_stamp":"2023-10-16T20:59:08.200542+0000","last_deep_scrub":"67'210","last_deep_scrub_stamp":"2023-10-16T20:59:08.200542+0000","last_clean_scrub_stamp":"2023-10-16T20:59:08.200542+0000","objects_scrubbed":3,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:06:26.078845+0000","scrub_duration":0.024305964999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":12288,"num_write":199,"num_write_kb":8312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9282,"num_omap_keys":17,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"67'107","reported_seq":2226,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:16.207741+0000","last_change":"2023-10-16T20:59:16.207806+0000","last_active":"2023-10-16T20:59:16.207741+0000","last_peered":"2023-10-16T20:59:16.207741+0000","last_clean":"2023-10-16T20:59:16.207741+0000","last_became_active":"2023-10-16T20:36:55.291458+0000","last_became_peered":"2023-10-16T20:36:55.291458+0000","last_unstale":"2023-10-16T20:59:16.207741+0000","last_undegraded":"2023-10-16T20:59:16.207741+0000","last_fullsized":"2023-10-16T20:59:16.207741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'107","last_scrub_stamp":"2023-10-16T20:59:16.207636+0000","last_deep_scrub":"67'107","last_deep_scrub_stamp":"2023-10-16T20:59:16.207636+0000","last_clean_scrub_stamp":"2023-10-16T20:59:16.207636+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:09:59.716717+0000","scrub_duration":0.0019653269999999998,"objects_trimmed":0,"snaptrim_duration":0.0087429459999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'682","reported_seq":3593,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:37.432193+0000","last_change":"2023-10-16T20:59:37.432278+0000","last_active":"2023-10-16T20:59:37.432193+0000","last_peered":"2023-10-16T20:59:37.432193+0000","last_clean":"2023-10-16T20:59:37.432193+0000","last_became_active":"2023-10-16T20:36:59.302968+0000","last_became_peered":"2023-10-16T20:36:59.302968+0000","last_unstale":"2023-10-16T20:59:37.432193+0000","last_undegraded":"2023-10-16T20:59:37.432193+0000","last_fullsized":"2023-10-16T20:59:37.432193+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'682","last_scrub_stamp":"2023-10-16T20:59:37.432091+0000","last_deep_scrub":"67'682","last_deep_scrub_stamp":"2023-10-16T20:59:37.432091+0000","last_clean_scrub_stamp":"2023-10-16T20:59:37.432091+0000","objects_scrubbed":344,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:53:27.461153+0000","scrub_duration":0.12678435499999999,"objects_trimmed":0,"snaptrim_duration":0.0027985639999999999,"stat_sum":{"num_bytes":24765378,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":783,"num_read_kb":25781,"num_write":678,"num_write_kb":40921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"67'351","reported_seq":458,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:16.994190+0000","last_change":"2023-10-16T20:59:16.994190+0000","last_active":"2023-10-16T20:59:16.994190+0000","last_peered":"2023-10-16T20:59:16.994190+0000","last_clean":"2023-10-16T20:59:16.994190+0000","last_became_active":"2023-10-16T20:36:54.762278+0000","last_became_peered":"2023-10-16T20:36:54.762278+0000","last_unstale":"2023-10-16T20:59:16.994190+0000","last_undegraded":"2023-10-16T20:59:16.994190+0000","last_fullsized":"2023-10-16T20:59:16.994190+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'351","last_scrub_stamp":"2023-10-16T20:59:16.994034+0000","last_deep_scrub":"67'351","last_deep_scrub_stamp":"2023-10-16T20:59:16.994034+0000","last_clean_scrub_stamp":"2023-10-16T20:59:16.994034+0000","objects_scrubbed":11,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:42:30.755177+0000","scrub_duration":0.038213219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":85,"num_read_kb":16492,"num_write":374,"num_write_kb":12523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10532,"num_omap_keys":21,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"67'96","reported_seq":2206,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:18.558055+0000","last_change":"2023-10-16T20:59:18.558106+0000","last_active":"2023-10-16T20:59:18.558055+0000","last_peered":"2023-10-16T20:59:18.558055+0000","last_clean":"2023-10-16T20:59:18.558055+0000","last_became_active":"2023-10-16T20:37:37.402987+0000","last_became_peered":"2023-10-16T20:37:37.402987+0000","last_unstale":"2023-10-16T20:59:18.558055+0000","last_undegraded":"2023-10-16T20:59:18.558055+0000","last_fullsized":"2023-10-16T20:59:18.558055+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'96","last_scrub_stamp":"2023-10-16T20:59:18.557959+0000","last_deep_scrub":"67'96","last_deep_scrub_stamp":"2023-10-16T20:59:18.557959+0000","last_clean_scrub_stamp":"2023-10-16T20:59:18.557959+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:00.882012+0000","scrub_duration":0.0024629650000000001,"objects_trimmed":0,"snaptrim_duration":0.00082573900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"67'673","reported_seq":4246,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:22.287085+0000","last_change":"2023-10-16T20:59:22.287085+0000","last_active":"2023-10-16T20:59:22.287085+0000","last_peered":"2023-10-16T20:59:22.287085+0000","last_clean":"2023-10-16T20:59:22.287085+0000","last_became_active":"2023-10-16T20:36:59.303805+0000","last_became_peered":"2023-10-16T20:36:59.303805+0000","last_unstale":"2023-10-16T20:59:22.287085+0000","last_undegraded":"2023-10-16T20:59:22.287085+0000","last_fullsized":"2023-10-16T20:59:22.287085+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'673","last_scrub_stamp":"2023-10-16T20:59:22.286926+0000","last_deep_scrub":"67'673","last_deep_scrub_stamp":"2023-10-16T20:59:22.286926+0000","last_clean_scrub_stamp":"2023-10-16T20:59:22.286926+0000","objects_scrubbed":304,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:17:33.137048+0000","scrub_duration":0.12974059900000001,"objects_trimmed":0,"snaptrim_duration":0.0041657500000000002,"stat_sum":{"num_bytes":49691093,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":1446,"num_read_kb":31388,"num_write":673,"num_write_kb":52938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"67'178","reported_seq":268,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:09.911993+0000","last_change":"2023-10-16T20:59:09.911993+0000","last_active":"2023-10-16T20:59:09.911993+0000","last_peered":"2023-10-16T20:59:09.911993+0000","last_clean":"2023-10-16T20:59:09.911993+0000","last_became_active":"2023-10-16T20:36:54.749316+0000","last_became_peered":"2023-10-16T20:36:54.749316+0000","last_unstale":"2023-10-16T20:59:09.911993+0000","last_undegraded":"2023-10-16T20:59:09.911993+0000","last_fullsized":"2023-10-16T20:59:09.911993+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'178","last_scrub_stamp":"2023-10-16T20:59:09.911761+0000","last_deep_scrub":"67'178","last_deep_scrub_stamp":"2023-10-16T20:59:09.911761+0000","last_clean_scrub_stamp":"2023-10-16T20:59:09.911761+0000","objects_scrubbed":6,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:22.957274+0000","scrub_duration":0.018582283000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":170,"num_write_kb":4198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20168,"num_omap_keys":38,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"67'104","reported_seq":2222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:14.206196+0000","last_change":"2023-10-16T20:59:14.206196+0000","last_active":"2023-10-16T20:59:14.206196+0000","last_peered":"2023-10-16T20:59:14.206196+0000","last_clean":"2023-10-16T20:59:14.206196+0000","last_became_active":"2023-10-16T20:36:55.302387+0000","last_became_peered":"2023-10-16T20:36:55.302387+0000","last_unstale":"2023-10-16T20:59:14.206196+0000","last_undegraded":"2023-10-16T20:59:14.206196+0000","last_fullsized":"2023-10-16T20:59:14.206196+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'104","last_scrub_stamp":"2023-10-16T20:59:14.206041+0000","last_deep_scrub":"67'104","last_deep_scrub_stamp":"2023-10-16T20:59:14.206041+0000","last_clean_scrub_stamp":"2023-10-16T20:59:14.206041+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:26:30.710991+0000","scrub_duration":0.0028641040000000001,"objects_trimmed":0,"snaptrim_duration":0.011585343,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'641","reported_seq":3861,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:16.317087+0000","last_change":"2023-10-16T20:59:16.317087+0000","last_active":"2023-10-16T20:59:16.317087+0000","last_peered":"2023-10-16T20:59:16.317087+0000","last_clean":"2023-10-16T20:59:16.317087+0000","last_became_active":"2023-10-16T20:36:59.305228+0000","last_became_peered":"2023-10-16T20:36:59.305228+0000","last_unstale":"2023-10-16T20:59:16.317087+0000","last_undegraded":"2023-10-16T20:59:16.317087+0000","last_fullsized":"2023-10-16T20:59:16.317087+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'641","last_scrub_stamp":"2023-10-16T20:59:16.316909+0000","last_deep_scrub":"67'641","last_deep_scrub_stamp":"2023-10-16T20:59:16.316909+0000","last_clean_scrub_stamp":"2023-10-16T20:59:16.316909+0000","objects_scrubbed":310,"log_size":641,"log_dups_size":0,"ondisk_log_size":641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:48:57.953725+0000","scrub_duration":0.12335723,"objects_trimmed":0,"snaptrim_duration":0.0047877249999999996,"stat_sum":{"num_bytes":34900728,"num_objects":310,"num_object_clones":0,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":1093,"num_read_kb":21879,"num_write":638,"num_write_kb":46703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"67'104","reported_seq":2223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:18.508850+0000","last_change":"2023-10-16T20:59:18.508908+0000","last_active":"2023-10-16T20:59:18.508850+0000","last_peered":"2023-10-16T20:59:18.508850+0000","last_clean":"2023-10-16T20:59:18.508850+0000","last_became_active":"2023-10-16T20:36:55.306427+0000","last_became_peered":"2023-10-16T20:36:55.306427+0000","last_unstale":"2023-10-16T20:59:18.508850+0000","last_undegraded":"2023-10-16T20:59:18.508850+0000","last_fullsized":"2023-10-16T20:59:18.508850+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'104","last_scrub_stamp":"2023-10-16T20:59:18.508734+0000","last_deep_scrub":"67'104","last_deep_scrub_stamp":"2023-10-16T20:59:18.508734+0000","last_clean_scrub_stamp":"2023-10-16T20:59:18.508734+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:58:43.422816+0000","scrub_duration":0.0019135370000000001,"objects_trimmed":0,"snaptrim_duration":0.013854674000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"67'191","reported_seq":272,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:10.947598+0000","last_change":"2023-10-16T20:59:10.947598+0000","last_active":"2023-10-16T20:59:10.947598+0000","last_peered":"2023-10-16T20:59:10.947598+0000","last_clean":"2023-10-16T20:59:10.947598+0000","last_became_active":"2023-10-16T20:36:54.767996+0000","last_became_peered":"2023-10-16T20:36:54.767996+0000","last_unstale":"2023-10-16T20:59:10.947598+0000","last_undegraded":"2023-10-16T20:59:10.947598+0000","last_fullsized":"2023-10-16T20:59:10.947598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'191","last_scrub_stamp":"2023-10-16T20:59:10.947456+0000","last_deep_scrub":"67'191","last_deep_scrub_stamp":"2023-10-16T20:59:10.947456+0000","last_clean_scrub_stamp":"2023-10-16T20:59:10.947456+0000","objects_scrubbed":4,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:33:10.801672+0000","scrub_duration":0.024723371000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":8202,"num_write":173,"num_write_kb":8293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"67'648","reported_seq":4495,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:33.433914+0000","last_change":"2023-10-16T20:59:33.434020+0000","last_active":"2023-10-16T20:59:33.433914+0000","last_peered":"2023-10-16T20:59:33.433914+0000","last_clean":"2023-10-16T20:59:33.433914+0000","last_became_active":"2023-10-16T20:36:59.306640+0000","last_became_peered":"2023-10-16T20:36:59.306640+0000","last_unstale":"2023-10-16T20:59:33.433914+0000","last_undegraded":"2023-10-16T20:59:33.433914+0000","last_fullsized":"2023-10-16T20:59:33.433914+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'648","last_scrub_stamp":"2023-10-16T20:59:33.433820+0000","last_deep_scrub":"67'648","last_deep_scrub_stamp":"2023-10-16T20:59:33.433820+0000","last_clean_scrub_stamp":"2023-10-16T20:59:33.433820+0000","objects_scrubbed":318,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:46:33.861581+0000","scrub_duration":0.13662969799999999,"objects_trimmed":0,"snaptrim_duration":0.0054537470000000001,"stat_sum":{"num_bytes":41205181,"num_objects":318,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":1719,"num_read_kb":37802,"num_write":648,"num_write_kb":48776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"67'101","reported_seq":2220,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:14.250365+0000","last_change":"2023-10-16T20:59:14.250365+0000","last_active":"2023-10-16T20:59:14.250365+0000","last_peered":"2023-10-16T20:59:14.250365+0000","last_clean":"2023-10-16T20:59:14.250365+0000","last_became_active":"2023-10-16T20:36:55.275085+0000","last_became_peered":"2023-10-16T20:36:55.275085+0000","last_unstale":"2023-10-16T20:59:14.250365+0000","last_undegraded":"2023-10-16T20:59:14.250365+0000","last_fullsized":"2023-10-16T20:59:14.250365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'101","last_scrub_stamp":"2023-10-16T20:59:14.250203+0000","last_deep_scrub":"67'101","last_deep_scrub_stamp":"2023-10-16T20:59:14.250203+0000","last_clean_scrub_stamp":"2023-10-16T20:59:14.250203+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:20:58.657625+0000","scrub_duration":0.001852245,"objects_trimmed":0,"snaptrim_duration":0.0087599630000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"67'183","reported_seq":261,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:57.085209+0000","last_change":"2023-10-16T20:58:57.085283+0000","last_active":"2023-10-16T20:58:57.085209+0000","last_peered":"2023-10-16T20:58:57.085209+0000","last_clean":"2023-10-16T20:58:57.085209+0000","last_became_active":"2023-10-16T20:36:54.755872+0000","last_became_peered":"2023-10-16T20:36:54.755872+0000","last_unstale":"2023-10-16T20:58:57.085209+0000","last_undegraded":"2023-10-16T20:58:57.085209+0000","last_fullsized":"2023-10-16T20:58:57.085209+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'183","last_scrub_stamp":"2023-10-16T20:58:57.085118+0000","last_deep_scrub":"67'183","last_deep_scrub_stamp":"2023-10-16T20:58:57.085118+0000","last_clean_scrub_stamp":"2023-10-16T20:58:57.085118+0000","objects_scrubbed":3,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:40:14.300010+0000","scrub_duration":0.017757339,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":10,"num_write":281,"num_write_kb":4257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"67'699","reported_seq":3477,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:26.358241+0000","last_change":"2023-10-16T20:59:26.358241+0000","last_active":"2023-10-16T20:59:26.358241+0000","last_peered":"2023-10-16T20:59:26.358241+0000","last_clean":"2023-10-16T20:59:26.358241+0000","last_became_active":"2023-10-16T20:36:59.302919+0000","last_became_peered":"2023-10-16T20:36:59.302919+0000","last_unstale":"2023-10-16T20:59:26.358241+0000","last_undegraded":"2023-10-16T20:59:26.358241+0000","last_fullsized":"2023-10-16T20:59:26.358241+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'699","last_scrub_stamp":"2023-10-16T20:59:26.358029+0000","last_deep_scrub":"67'699","last_deep_scrub_stamp":"2023-10-16T20:59:26.358029+0000","last_clean_scrub_stamp":"2023-10-16T20:59:26.358029+0000","objects_scrubbed":334,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:30:05.906486+0000","scrub_duration":0.129970374,"objects_trimmed":0,"snaptrim_duration":0.0046546790000000001,"stat_sum":{"num_bytes":20793675,"num_objects":334,"num_object_clones":0,"num_object_copies":1336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":334,"num_whiteouts":0,"num_read":649,"num_read_kb":16394,"num_write":696,"num_write_kb":45235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"67'106","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:19.261082+0000","last_change":"2023-10-16T20:59:19.261129+0000","last_active":"2023-10-16T20:59:19.261082+0000","last_peered":"2023-10-16T20:59:19.261082+0000","last_clean":"2023-10-16T20:59:19.261082+0000","last_became_active":"2023-10-16T20:36:55.270455+0000","last_became_peered":"2023-10-16T20:36:55.270455+0000","last_unstale":"2023-10-16T20:59:19.261082+0000","last_undegraded":"2023-10-16T20:59:19.261082+0000","last_fullsized":"2023-10-16T20:59:19.261082+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'106","last_scrub_stamp":"2023-10-16T20:59:19.261013+0000","last_deep_scrub":"67'106","last_deep_scrub_stamp":"2023-10-16T20:59:19.261013+0000","last_clean_scrub_stamp":"2023-10-16T20:59:19.261013+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:49:53.748721+0000","scrub_duration":0.0014257040000000001,"objects_trimmed":0,"snaptrim_duration":0.008577889,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"67'55","reported_seq":142,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:02.072253+0000","last_change":"2023-10-16T20:59:02.072347+0000","last_active":"2023-10-16T20:59:02.072253+0000","last_peered":"2023-10-16T20:59:02.072253+0000","last_clean":"2023-10-16T20:59:02.072253+0000","last_became_active":"2023-10-16T20:36:54.743805+0000","last_became_peered":"2023-10-16T20:36:54.743805+0000","last_unstale":"2023-10-16T20:59:02.072253+0000","last_undegraded":"2023-10-16T20:59:02.072253+0000","last_fullsized":"2023-10-16T20:59:02.072253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'55","last_scrub_stamp":"2023-10-16T20:59:02.072155+0000","last_deep_scrub":"67'55","last_deep_scrub_stamp":"2023-10-16T20:59:02.072155+0000","last_clean_scrub_stamp":"2023-10-16T20:59:02.072155+0000","objects_scrubbed":6,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:18:40.321154+0000","scrub_duration":0.011965646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":55,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":19110,"num_omap_keys":35,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'651","reported_seq":3714,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:22.025877+0000","last_change":"2023-10-16T20:59:22.025877+0000","last_active":"2023-10-16T20:59:22.025877+0000","last_peered":"2023-10-16T20:59:22.025877+0000","last_clean":"2023-10-16T20:59:22.025877+0000","last_became_active":"2023-10-16T20:36:59.465172+0000","last_became_peered":"2023-10-16T20:36:59.465172+0000","last_unstale":"2023-10-16T20:59:22.025877+0000","last_undegraded":"2023-10-16T20:59:22.025877+0000","last_fullsized":"2023-10-16T20:59:22.025877+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'651","last_scrub_stamp":"2023-10-16T20:59:22.025668+0000","last_deep_scrub":"67'651","last_deep_scrub_stamp":"2023-10-16T20:59:22.025668+0000","last_clean_scrub_stamp":"2023-10-16T20:59:22.025668+0000","objects_scrubbed":314,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:38:10.904294+0000","scrub_duration":0.12543077799999999,"objects_trimmed":0,"snaptrim_duration":0.0040107390000000001,"stat_sum":{"num_bytes":25409516,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":935,"num_read_kb":27643,"num_write":650,"num_write_kb":33262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"67'103","reported_seq":2222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:08.331174+0000","last_change":"2023-10-16T20:59:08.331222+0000","last_active":"2023-10-16T20:59:08.331174+0000","last_peered":"2023-10-16T20:59:08.331174+0000","last_clean":"2023-10-16T20:59:08.331174+0000","last_became_active":"2023-10-16T20:36:55.278568+0000","last_became_peered":"2023-10-16T20:36:55.278568+0000","last_unstale":"2023-10-16T20:59:08.331174+0000","last_undegraded":"2023-10-16T20:59:08.331174+0000","last_fullsized":"2023-10-16T20:59:08.331174+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'103","last_scrub_stamp":"2023-10-16T20:59:08.331083+0000","last_deep_scrub":"67'103","last_deep_scrub_stamp":"2023-10-16T20:59:08.331083+0000","last_clean_scrub_stamp":"2023-10-16T20:59:08.331083+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:39:29.469121+0000","scrub_duration":0.0021561620000000001,"objects_trimmed":0,"snaptrim_duration":0.0050399599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"67'293","reported_seq":373,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:55.917267+0000","last_change":"2023-10-16T20:58:55.917322+0000","last_active":"2023-10-16T20:58:55.917267+0000","last_peered":"2023-10-16T20:58:55.917267+0000","last_clean":"2023-10-16T20:58:55.917267+0000","last_became_active":"2023-10-16T20:36:54.758496+0000","last_became_peered":"2023-10-16T20:36:54.758496+0000","last_unstale":"2023-10-16T20:58:55.917267+0000","last_undegraded":"2023-10-16T20:58:55.917267+0000","last_fullsized":"2023-10-16T20:58:55.917267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'293","last_scrub_stamp":"2023-10-16T20:58:55.917187+0000","last_deep_scrub":"67'293","last_deep_scrub_stamp":"2023-10-16T20:58:55.917187+0000","last_clean_scrub_stamp":"2023-10-16T20:58:55.917187+0000","objects_scrubbed":6,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:17:21.018502+0000","scrub_duration":0.017810975999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":12294,"num_write":291,"num_write_kb":4310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":34642,"num_omap_keys":66,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"67'586","reported_seq":3883,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:34.445350+0000","last_change":"2023-10-16T20:59:34.445448+0000","last_active":"2023-10-16T20:59:34.445350+0000","last_peered":"2023-10-16T20:59:34.445350+0000","last_clean":"2023-10-16T20:59:34.445350+0000","last_became_active":"2023-10-16T20:36:59.309068+0000","last_became_peered":"2023-10-16T20:36:59.309068+0000","last_unstale":"2023-10-16T20:59:34.445350+0000","last_undegraded":"2023-10-16T20:59:34.445350+0000","last_fullsized":"2023-10-16T20:59:34.445350+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'586","last_scrub_stamp":"2023-10-16T20:59:34.445220+0000","last_deep_scrub":"67'586","last_deep_scrub_stamp":"2023-10-16T20:59:34.445220+0000","last_clean_scrub_stamp":"2023-10-16T20:59:34.445220+0000","objects_scrubbed":303,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:23:26.856676+0000","scrub_duration":0.12469408799999999,"objects_trimmed":0,"snaptrim_duration":0.0055482439999999999,"stat_sum":{"num_bytes":36900597,"num_objects":303,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1169,"num_read_kb":30992,"num_write":585,"num_write_kb":40444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'101","reported_seq":2220,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:14.223878+0000","last_change":"2023-10-16T20:59:14.223927+0000","last_active":"2023-10-16T20:59:14.223878+0000","last_peered":"2023-10-16T20:59:14.223878+0000","last_clean":"2023-10-16T20:59:14.223878+0000","last_became_active":"2023-10-16T20:36:55.272840+0000","last_became_peered":"2023-10-16T20:36:55.272840+0000","last_unstale":"2023-10-16T20:59:14.223878+0000","last_undegraded":"2023-10-16T20:59:14.223878+0000","last_fullsized":"2023-10-16T20:59:14.223878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'101","last_scrub_stamp":"2023-10-16T20:59:14.223774+0000","last_deep_scrub":"67'101","last_deep_scrub_stamp":"2023-10-16T20:59:14.223774+0000","last_clean_scrub_stamp":"2023-10-16T20:59:14.223774+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:44:52.868862+0000","scrub_duration":0.0020408700000000002,"objects_trimmed":0,"snaptrim_duration":0.011897911000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"67'12","reported_seq":101,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:51.472708+0000","last_change":"2023-10-16T20:58:51.472802+0000","last_active":"2023-10-16T20:58:51.472708+0000","last_peered":"2023-10-16T20:58:51.472708+0000","last_clean":"2023-10-16T20:58:51.472708+0000","last_became_active":"2023-10-16T20:36:54.760397+0000","last_became_peered":"2023-10-16T20:36:54.760397+0000","last_unstale":"2023-10-16T20:58:51.472708+0000","last_undegraded":"2023-10-16T20:58:51.472708+0000","last_fullsized":"2023-10-16T20:58:51.472708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2023-10-16T20:58:51.472601+0000","last_deep_scrub":"67'12","last_deep_scrub_stamp":"2023-10-16T20:58:51.472601+0000","last_clean_scrub_stamp":"2023-10-16T20:58:51.472601+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:05:36.331637+0000","scrub_duration":0.0080542610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"67'710","reported_seq":3820,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:12.359908+0000","last_change":"2023-10-16T20:59:12.360004+0000","last_active":"2023-10-16T20:59:12.359908+0000","last_peered":"2023-10-16T20:59:12.359908+0000","last_clean":"2023-10-16T20:59:12.359908+0000","last_became_active":"2023-10-16T20:36:59.465152+0000","last_became_peered":"2023-10-16T20:36:59.465152+0000","last_unstale":"2023-10-16T20:59:12.359908+0000","last_undegraded":"2023-10-16T20:59:12.359908+0000","last_fullsized":"2023-10-16T20:59:12.359908+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'710","last_scrub_stamp":"2023-10-16T20:59:12.359810+0000","last_deep_scrub":"67'710","last_deep_scrub_stamp":"2023-10-16T20:59:12.359810+0000","last_clean_scrub_stamp":"2023-10-16T20:59:12.359810+0000","objects_scrubbed":344,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:51:44.766685+0000","scrub_duration":0.14617412099999999,"objects_trimmed":0,"snaptrim_duration":0.0044307399999999999,"stat_sum":{"num_bytes":25809469,"num_objects":344,"num_object_clones":0,"num_object_copies":1376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":981,"num_read_kb":15497,"num_write":706,"num_write_kb":50141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'106","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:26.290296+0000","last_change":"2023-10-16T20:59:26.290343+0000","last_active":"2023-10-16T20:59:26.290296+0000","last_peered":"2023-10-16T20:59:26.290296+0000","last_clean":"2023-10-16T20:59:26.290296+0000","last_became_active":"2023-10-16T20:36:55.285566+0000","last_became_peered":"2023-10-16T20:36:55.285566+0000","last_unstale":"2023-10-16T20:59:26.290296+0000","last_undegraded":"2023-10-16T20:59:26.290296+0000","last_fullsized":"2023-10-16T20:59:26.290296+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'106","last_scrub_stamp":"2023-10-16T20:59:26.290208+0000","last_deep_scrub":"67'106","last_deep_scrub_stamp":"2023-10-16T20:59:26.290208+0000","last_clean_scrub_stamp":"2023-10-16T20:59:26.290208+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:36:47.364832+0000","scrub_duration":0.0019590620000000001,"objects_trimmed":0,"snaptrim_duration":0.0085500479999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"67'104","reported_seq":189,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:01.068372+0000","last_change":"2023-10-16T20:59:01.068434+0000","last_active":"2023-10-16T20:59:01.068372+0000","last_peered":"2023-10-16T20:59:01.068372+0000","last_clean":"2023-10-16T20:59:01.068372+0000","last_became_active":"2023-10-16T20:36:54.757796+0000","last_became_peered":"2023-10-16T20:36:54.757796+0000","last_unstale":"2023-10-16T20:59:01.068372+0000","last_undegraded":"2023-10-16T20:59:01.068372+0000","last_fullsized":"2023-10-16T20:59:01.068372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'104","last_scrub_stamp":"2023-10-16T20:59:01.068280+0000","last_deep_scrub":"67'104","last_deep_scrub_stamp":"2023-10-16T20:59:01.068280+0000","last_clean_scrub_stamp":"2023-10-16T20:59:01.068280+0000","objects_scrubbed":4,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:13:55.968912+0000","scrub_duration":0.035122747000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":254,"num_write_kb":1037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"67'686","reported_seq":3354,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:37.233416+0000","last_change":"2023-10-16T20:59:37.233515+0000","last_active":"2023-10-16T20:59:37.233416+0000","last_peered":"2023-10-16T20:59:37.233416+0000","last_clean":"2023-10-16T20:59:37.233416+0000","last_became_active":"2023-10-16T20:36:59.465221+0000","last_became_peered":"2023-10-16T20:36:59.465221+0000","last_unstale":"2023-10-16T20:59:37.233416+0000","last_undegraded":"2023-10-16T20:59:37.233416+0000","last_fullsized":"2023-10-16T20:59:37.233416+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'686","last_scrub_stamp":"2023-10-16T20:59:37.233315+0000","last_deep_scrub":"67'686","last_deep_scrub_stamp":"2023-10-16T20:59:37.233315+0000","last_clean_scrub_stamp":"2023-10-16T20:59:37.233315+0000","objects_scrubbed":343,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:22:03.162842+0000","scrub_duration":0.138060398,"objects_trimmed":0,"snaptrim_duration":0.0059334330000000001,"stat_sum":{"num_bytes":20326999,"num_objects":343,"num_object_clones":0,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":538,"num_read_kb":13756,"num_write":685,"num_write_kb":32496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"67'101","reported_seq":2220,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:10.927031+0000","last_change":"2023-10-16T20:59:10.927098+0000","last_active":"2023-10-16T20:59:10.927031+0000","last_peered":"2023-10-16T20:59:10.927031+0000","last_clean":"2023-10-16T20:59:10.927031+0000","last_became_active":"2023-10-16T20:36:55.274432+0000","last_became_peered":"2023-10-16T20:36:55.274432+0000","last_unstale":"2023-10-16T20:59:10.927031+0000","last_undegraded":"2023-10-16T20:59:10.927031+0000","last_fullsized":"2023-10-16T20:59:10.927031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'101","last_scrub_stamp":"2023-10-16T20:59:10.926915+0000","last_deep_scrub":"67'101","last_deep_scrub_stamp":"2023-10-16T20:59:10.926915+0000","last_clean_scrub_stamp":"2023-10-16T20:59:10.926915+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:52:40.067704+0000","scrub_duration":0.0018791190000000001,"objects_trimmed":0,"snaptrim_duration":0.0057461259999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"67'39","reported_seq":123,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:17.915784+0000","last_change":"2023-10-16T20:59:17.915835+0000","last_active":"2023-10-16T20:59:17.915784+0000","last_peered":"2023-10-16T20:59:17.915784+0000","last_clean":"2023-10-16T20:59:17.915784+0000","last_became_active":"2023-10-16T20:36:54.762167+0000","last_became_peered":"2023-10-16T20:36:54.762167+0000","last_unstale":"2023-10-16T20:59:17.915784+0000","last_undegraded":"2023-10-16T20:59:17.915784+0000","last_fullsized":"2023-10-16T20:59:17.915784+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'39","last_scrub_stamp":"2023-10-16T20:59:17.915708+0000","last_deep_scrub":"67'39","last_deep_scrub_stamp":"2023-10-16T20:59:17.915708+0000","last_clean_scrub_stamp":"2023-10-16T20:59:17.915708+0000","objects_scrubbed":5,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:18:11.275071+0000","scrub_duration":0.0057107759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":34,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":16998,"num_omap_keys":32,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"67'725","reported_seq":3737,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:24.401605+0000","last_change":"2023-10-16T20:59:24.401720+0000","last_active":"2023-10-16T20:59:24.401605+0000","last_peered":"2023-10-16T20:59:24.401605+0000","last_clean":"2023-10-16T20:59:24.401605+0000","last_became_active":"2023-10-16T20:36:59.466876+0000","last_became_peered":"2023-10-16T20:36:59.466876+0000","last_unstale":"2023-10-16T20:59:24.401605+0000","last_undegraded":"2023-10-16T20:59:24.401605+0000","last_fullsized":"2023-10-16T20:59:24.401605+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'725","last_scrub_stamp":"2023-10-16T20:59:24.401499+0000","last_deep_scrub":"67'725","last_deep_scrub_stamp":"2023-10-16T20:59:24.401499+0000","last_clean_scrub_stamp":"2023-10-16T20:59:24.401499+0000","objects_scrubbed":340,"log_size":725,"log_dups_size":0,"ondisk_log_size":725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:16:44.535101+0000","scrub_duration":0.141220765,"objects_trimmed":0,"snaptrim_duration":0.009058969,"stat_sum":{"num_bytes":44901233,"num_objects":340,"num_object_clones":0,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":884,"num_read_kb":20657,"num_write":720,"num_write_kb":60576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"67'98","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:13.164714+0000","last_change":"2023-10-16T20:59:13.164786+0000","last_active":"2023-10-16T20:59:13.164714+0000","last_peered":"2023-10-16T20:59:13.164714+0000","last_clean":"2023-10-16T20:59:13.164714+0000","last_became_active":"2023-10-16T20:36:55.293228+0000","last_became_peered":"2023-10-16T20:36:55.293228+0000","last_unstale":"2023-10-16T20:59:13.164714+0000","last_undegraded":"2023-10-16T20:59:13.164714+0000","last_fullsized":"2023-10-16T20:59:13.164714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'98","last_scrub_stamp":"2023-10-16T20:59:13.164628+0000","last_deep_scrub":"67'98","last_deep_scrub_stamp":"2023-10-16T20:59:13.164628+0000","last_clean_scrub_stamp":"2023-10-16T20:59:13.164628+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:49:21.419640+0000","scrub_duration":0.001733166,"objects_trimmed":0,"snaptrim_duration":0.013567460999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"67'212","reported_seq":289,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:59.097576+0000","last_change":"2023-10-16T20:58:59.097630+0000","last_active":"2023-10-16T20:58:59.097576+0000","last_peered":"2023-10-16T20:58:59.097576+0000","last_clean":"2023-10-16T20:58:59.097576+0000","last_became_active":"2023-10-16T20:36:54.757308+0000","last_became_peered":"2023-10-16T20:36:54.757308+0000","last_unstale":"2023-10-16T20:58:59.097576+0000","last_undegraded":"2023-10-16T20:58:59.097576+0000","last_fullsized":"2023-10-16T20:58:59.097576+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'212","last_scrub_stamp":"2023-10-16T20:58:59.097485+0000","last_deep_scrub":"67'212","last_deep_scrub_stamp":"2023-10-16T20:58:59.097485+0000","last_clean_scrub_stamp":"2023-10-16T20:58:59.097485+0000","objects_scrubbed":2,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:26:44.246482+0000","scrub_duration":0.017169976999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":3049,"num_write":207,"num_write_kb":4195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9334,"num_omap_keys":18,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"67'671","reported_seq":3446,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:15.193409+0000","last_change":"2023-10-16T20:59:15.193523+0000","last_active":"2023-10-16T20:59:15.193409+0000","last_peered":"2023-10-16T20:59:15.193409+0000","last_clean":"2023-10-16T20:59:15.193409+0000","last_became_active":"2023-10-16T20:36:59.303224+0000","last_became_peered":"2023-10-16T20:36:59.303224+0000","last_unstale":"2023-10-16T20:59:15.193409+0000","last_undegraded":"2023-10-16T20:59:15.193409+0000","last_fullsized":"2023-10-16T20:59:15.193409+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'671","last_scrub_stamp":"2023-10-16T20:59:15.193311+0000","last_deep_scrub":"67'671","last_deep_scrub_stamp":"2023-10-16T20:59:15.193311+0000","last_clean_scrub_stamp":"2023-10-16T20:59:15.193311+0000","objects_scrubbed":323,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:37:00.217159+0000","scrub_duration":0.12305557,"objects_trimmed":0,"snaptrim_duration":0.0038422399999999998,"stat_sum":{"num_bytes":21817454,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":647,"num_read_kb":9306,"num_write":673,"num_write_kb":37999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"67'120","reported_seq":2239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:58.334392+0000","last_change":"2023-10-16T20:58:58.334443+0000","last_active":"2023-10-16T20:58:58.334392+0000","last_peered":"2023-10-16T20:58:58.334392+0000","last_clean":"2023-10-16T20:58:58.334392+0000","last_became_active":"2023-10-16T20:36:55.277872+0000","last_became_peered":"2023-10-16T20:36:55.277872+0000","last_unstale":"2023-10-16T20:58:58.334392+0000","last_undegraded":"2023-10-16T20:58:58.334392+0000","last_fullsized":"2023-10-16T20:58:58.334392+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'120","last_scrub_stamp":"2023-10-16T20:58:58.334264+0000","last_deep_scrub":"67'120","last_deep_scrub_stamp":"2023-10-16T20:58:58.334264+0000","last_clean_scrub_stamp":"2023-10-16T20:58:58.334264+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:35:22.356584+0000","scrub_duration":0.0023669569999999998,"objects_trimmed":0,"snaptrim_duration":0.0052323379999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"67'537","reported_seq":630,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:54.485905+0000","last_change":"2023-10-16T20:58:54.485966+0000","last_active":"2023-10-16T20:58:54.485905+0000","last_peered":"2023-10-16T20:58:54.485905+0000","last_clean":"2023-10-16T20:58:54.485905+0000","last_became_active":"2023-10-16T20:36:54.740340+0000","last_became_peered":"2023-10-16T20:36:54.740340+0000","last_unstale":"2023-10-16T20:58:54.485905+0000","last_undegraded":"2023-10-16T20:58:54.485905+0000","last_fullsized":"2023-10-16T20:58:54.485905+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'537","last_scrub_stamp":"2023-10-16T20:58:54.485786+0000","last_deep_scrub":"67'537","last_deep_scrub_stamp":"2023-10-16T20:58:54.485786+0000","last_clean_scrub_stamp":"2023-10-16T20:58:54.485786+0000","objects_scrubbed":5,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:18.710024+0000","scrub_duration":0.040724903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21285,"num_write":524,"num_write_kb":16645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'623","reported_seq":3898,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:21.069874+0000","last_change":"2023-10-16T20:59:21.069939+0000","last_active":"2023-10-16T20:59:21.069874+0000","last_peered":"2023-10-16T20:59:21.069874+0000","last_clean":"2023-10-16T20:59:21.069874+0000","last_became_active":"2023-10-16T20:36:59.294548+0000","last_became_peered":"2023-10-16T20:36:59.294548+0000","last_unstale":"2023-10-16T20:59:21.069874+0000","last_undegraded":"2023-10-16T20:59:21.069874+0000","last_fullsized":"2023-10-16T20:59:21.069874+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'623","last_scrub_stamp":"2023-10-16T20:59:21.069779+0000","last_deep_scrub":"67'623","last_deep_scrub_stamp":"2023-10-16T20:59:21.069779+0000","last_clean_scrub_stamp":"2023-10-16T20:59:21.069779+0000","objects_scrubbed":322,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:56:38.618092+0000","scrub_duration":0.12688919600000001,"objects_trimmed":0,"snaptrim_duration":0.0062280529999999999,"stat_sum":{"num_bytes":29651112,"num_objects":322,"num_object_clones":0,"num_object_copies":1288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":1147,"num_read_kb":23932,"num_write":623,"num_write_kb":40690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"67'89","reported_seq":2209,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:04.003786+0000","last_change":"2023-10-16T20:59:04.003844+0000","last_active":"2023-10-16T20:59:04.003786+0000","last_peered":"2023-10-16T20:59:04.003786+0000","last_clean":"2023-10-16T20:59:04.003786+0000","last_became_active":"2023-10-16T20:36:55.269749+0000","last_became_peered":"2023-10-16T20:36:55.269749+0000","last_unstale":"2023-10-16T20:59:04.003786+0000","last_undegraded":"2023-10-16T20:59:04.003786+0000","last_fullsized":"2023-10-16T20:59:04.003786+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'89","last_scrub_stamp":"2023-10-16T20:59:04.003679+0000","last_deep_scrub":"67'89","last_deep_scrub_stamp":"2023-10-16T20:59:04.003679+0000","last_clean_scrub_stamp":"2023-10-16T20:59:04.003679+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:45:42.918860+0000","scrub_duration":0.0021523760000000001,"objects_trimmed":0,"snaptrim_duration":0.003009004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'198","reported_seq":395,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:55.620844+0000","last_change":"2023-10-16T20:58:55.620915+0000","last_active":"2023-10-16T20:58:55.620844+0000","last_peered":"2023-10-16T20:58:55.620844+0000","last_clean":"2023-10-16T20:58:55.620844+0000","last_became_active":"2023-10-16T20:36:54.757549+0000","last_became_peered":"2023-10-16T20:36:54.757549+0000","last_unstale":"2023-10-16T20:58:55.620844+0000","last_undegraded":"2023-10-16T20:58:55.620844+0000","last_fullsized":"2023-10-16T20:58:55.620844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'198","last_scrub_stamp":"2023-10-16T20:58:55.620756+0000","last_deep_scrub":"67'198","last_deep_scrub_stamp":"2023-10-16T20:58:55.620756+0000","last_clean_scrub_stamp":"2023-10-16T20:58:55.620756+0000","objects_scrubbed":10,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:24:40.908940+0000","scrub_duration":0.012899747,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1410402,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":124,"num_read_kb":239,"num_write":186,"num_write_kb":1570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":50816,"num_omap_keys":97,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'647","reported_seq":3954,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:19.082046+0000","last_change":"2023-10-16T20:59:19.082138+0000","last_active":"2023-10-16T20:59:19.082046+0000","last_peered":"2023-10-16T20:59:19.082046+0000","last_clean":"2023-10-16T20:59:19.082046+0000","last_became_active":"2023-10-16T20:36:59.301693+0000","last_became_peered":"2023-10-16T20:36:59.301693+0000","last_unstale":"2023-10-16T20:59:19.082046+0000","last_undegraded":"2023-10-16T20:59:19.082046+0000","last_fullsized":"2023-10-16T20:59:19.082046+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'647","last_scrub_stamp":"2023-10-16T20:59:19.081936+0000","last_deep_scrub":"67'647","last_deep_scrub_stamp":"2023-10-16T20:59:19.081936+0000","last_clean_scrub_stamp":"2023-10-16T20:59:19.081936+0000","objects_scrubbed":298,"log_size":647,"log_dups_size":0,"ondisk_log_size":647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:47:03.996136+0000","scrub_duration":0.124113616,"objects_trimmed":0,"snaptrim_duration":0.0056692870000000003,"stat_sum":{"num_bytes":29916858,"num_objects":298,"num_object_clones":0,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":298,"num_whiteouts":0,"num_read":1180,"num_read_kb":25235,"num_write":643,"num_write_kb":45892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"67'104","reported_seq":2223,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:58.176470+0000","last_change":"2023-10-16T20:58:58.176549+0000","last_active":"2023-10-16T20:58:58.176470+0000","last_peered":"2023-10-16T20:58:58.176470+0000","last_clean":"2023-10-16T20:58:58.176470+0000","last_became_active":"2023-10-16T20:36:55.276649+0000","last_became_peered":"2023-10-16T20:36:55.276649+0000","last_unstale":"2023-10-16T20:58:58.176470+0000","last_undegraded":"2023-10-16T20:58:58.176470+0000","last_fullsized":"2023-10-16T20:58:58.176470+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'104","last_scrub_stamp":"2023-10-16T20:58:58.176362+0000","last_deep_scrub":"67'104","last_deep_scrub_stamp":"2023-10-16T20:58:58.176362+0000","last_clean_scrub_stamp":"2023-10-16T20:58:58.176362+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:39:24.099280+0000","scrub_duration":0.0024904129999999999,"objects_trimmed":0,"snaptrim_duration":0.0076292809999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'131","reported_seq":214,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:56.907025+0000","last_change":"2023-10-16T20:58:56.907025+0000","last_active":"2023-10-16T20:58:56.907025+0000","last_peered":"2023-10-16T20:58:56.907025+0000","last_clean":"2023-10-16T20:58:56.907025+0000","last_became_active":"2023-10-16T20:36:54.753237+0000","last_became_peered":"2023-10-16T20:36:54.753237+0000","last_unstale":"2023-10-16T20:58:56.907025+0000","last_undegraded":"2023-10-16T20:58:56.907025+0000","last_fullsized":"2023-10-16T20:58:56.907025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'131","last_scrub_stamp":"2023-10-16T20:58:56.906872+0000","last_deep_scrub":"67'131","last_deep_scrub_stamp":"2023-10-16T20:58:56.906872+0000","last_clean_scrub_stamp":"2023-10-16T20:58:56.906872+0000","objects_scrubbed":3,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:38:45.121322+0000","scrub_duration":0.023535402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":20486,"num_write":119,"num_write_kb":8247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'673","reported_seq":3504,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:27.360002+0000","last_change":"2023-10-16T20:59:27.360070+0000","last_active":"2023-10-16T20:59:27.360002+0000","last_peered":"2023-10-16T20:59:27.360002+0000","last_clean":"2023-10-16T20:59:27.360002+0000","last_became_active":"2023-10-16T20:36:59.466708+0000","last_became_peered":"2023-10-16T20:36:59.466708+0000","last_unstale":"2023-10-16T20:59:27.360002+0000","last_undegraded":"2023-10-16T20:59:27.360002+0000","last_fullsized":"2023-10-16T20:59:27.360002+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'673","last_scrub_stamp":"2023-10-16T20:59:27.359906+0000","last_deep_scrub":"67'673","last_deep_scrub_stamp":"2023-10-16T20:59:27.359906+0000","last_clean_scrub_stamp":"2023-10-16T20:59:27.359906+0000","objects_scrubbed":318,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:25:43.177804+0000","scrub_duration":0.12333601900000001,"objects_trimmed":0,"snaptrim_duration":0.011259905000000001,"stat_sum":{"num_bytes":19802424,"num_objects":318,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":703,"num_read_kb":14564,"num_write":670,"num_write_kb":31986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"67'99","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:02.278130+0000","last_change":"2023-10-16T20:59:02.278178+0000","last_active":"2023-10-16T20:59:02.278130+0000","last_peered":"2023-10-16T20:59:02.278130+0000","last_clean":"2023-10-16T20:59:02.278130+0000","last_became_active":"2023-10-16T20:36:55.294610+0000","last_became_peered":"2023-10-16T20:36:55.294610+0000","last_unstale":"2023-10-16T20:59:02.278130+0000","last_undegraded":"2023-10-16T20:59:02.278130+0000","last_fullsized":"2023-10-16T20:59:02.278130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'99","last_scrub_stamp":"2023-10-16T20:59:02.278035+0000","last_deep_scrub":"67'99","last_deep_scrub_stamp":"2023-10-16T20:59:02.278035+0000","last_clean_scrub_stamp":"2023-10-16T20:59:02.278035+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:52:05.752220+0000","scrub_duration":0.0025758619999999999,"objects_trimmed":0,"snaptrim_duration":0.0075091560000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"67'12","reported_seq":97,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:55.494605+0000","last_change":"2023-10-16T20:58:55.494660+0000","last_active":"2023-10-16T20:58:55.494605+0000","last_peered":"2023-10-16T20:58:55.494605+0000","last_clean":"2023-10-16T20:58:55.494605+0000","last_became_active":"2023-10-16T20:36:54.740944+0000","last_became_peered":"2023-10-16T20:36:54.740944+0000","last_unstale":"2023-10-16T20:58:55.494605+0000","last_undegraded":"2023-10-16T20:58:55.494605+0000","last_fullsized":"2023-10-16T20:58:55.494605+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'12","last_scrub_stamp":"2023-10-16T20:58:55.494532+0000","last_deep_scrub":"67'12","last_deep_scrub_stamp":"2023-10-16T20:58:55.494532+0000","last_clean_scrub_stamp":"2023-10-16T20:58:55.494532+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:08:42.299214+0000","scrub_duration":0.0059841030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":23,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"67'119","reported_seq":231,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:54.954307+0000","last_change":"2023-10-16T20:58:54.954375+0000","last_active":"2023-10-16T20:58:54.954307+0000","last_peered":"2023-10-16T20:58:54.954307+0000","last_clean":"2023-10-16T20:58:54.954307+0000","last_became_active":"2023-10-16T20:36:46.388824+0000","last_became_peered":"2023-10-16T20:36:46.388824+0000","last_unstale":"2023-10-16T20:58:54.954307+0000","last_undegraded":"2023-10-16T20:58:54.954307+0000","last_fullsized":"2023-10-16T20:58:54.954307+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'119","last_scrub_stamp":"2023-10-16T20:58:54.954227+0000","last_deep_scrub":"67'119","last_deep_scrub_stamp":"2023-10-16T20:58:54.954227+0000","last_clean_scrub_stamp":"2023-10-16T20:58:54.954227+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:25:20.529327+0000","scrub_duration":0.0064233470000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"67'669","reported_seq":3448,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:30.743797+0000","last_change":"2023-10-16T20:59:30.743897+0000","last_active":"2023-10-16T20:59:30.743797+0000","last_peered":"2023-10-16T20:59:30.743797+0000","last_clean":"2023-10-16T20:59:30.743797+0000","last_became_active":"2023-10-16T20:36:59.468082+0000","last_became_peered":"2023-10-16T20:36:59.468082+0000","last_unstale":"2023-10-16T20:59:30.743797+0000","last_undegraded":"2023-10-16T20:59:30.743797+0000","last_fullsized":"2023-10-16T20:59:30.743797+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'669","last_scrub_stamp":"2023-10-16T20:59:30.743694+0000","last_deep_scrub":"67'669","last_deep_scrub_stamp":"2023-10-16T20:59:30.743694+0000","last_clean_scrub_stamp":"2023-10-16T20:59:30.743694+0000","objects_scrubbed":362,"log_size":669,"log_dups_size":0,"ondisk_log_size":669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:37:46.002990+0000","scrub_duration":0.14589089999999999,"objects_trimmed":0,"snaptrim_duration":0.013154572,"stat_sum":{"num_bytes":30910053,"num_objects":362,"num_object_clones":0,"num_object_copies":1448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":649,"num_read_kb":15567,"num_write":666,"num_write_kb":42817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'110","reported_seq":2221,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:25.926391+0000","last_change":"2023-10-16T20:59:25.926438+0000","last_active":"2023-10-16T20:59:25.926391+0000","last_peered":"2023-10-16T20:59:25.926391+0000","last_clean":"2023-10-16T20:59:25.926391+0000","last_became_active":"2023-10-16T20:37:37.409004+0000","last_became_peered":"2023-10-16T20:37:37.409004+0000","last_unstale":"2023-10-16T20:59:25.926391+0000","last_undegraded":"2023-10-16T20:59:25.926391+0000","last_fullsized":"2023-10-16T20:59:25.926391+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'110","last_scrub_stamp":"2023-10-16T20:59:25.926296+0000","last_deep_scrub":"67'110","last_deep_scrub_stamp":"2023-10-16T20:59:25.926296+0000","last_clean_scrub_stamp":"2023-10-16T20:59:25.926296+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:03:54.206289+0000","scrub_duration":0.002525621,"objects_trimmed":0,"snaptrim_duration":0.0060805590000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"67'73","reported_seq":153,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.517695+0000","last_change":"2023-10-16T20:58:50.517695+0000","last_active":"2023-10-16T20:58:50.517695+0000","last_peered":"2023-10-16T20:58:50.517695+0000","last_clean":"2023-10-16T20:58:50.517695+0000","last_became_active":"2023-10-16T20:36:54.746247+0000","last_became_peered":"2023-10-16T20:36:54.746247+0000","last_unstale":"2023-10-16T20:58:50.517695+0000","last_undegraded":"2023-10-16T20:58:50.517695+0000","last_fullsized":"2023-10-16T20:58:50.517695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'73","last_scrub_stamp":"2023-10-16T20:58:50.517537+0000","last_deep_scrub":"67'73","last_deep_scrub_stamp":"2023-10-16T20:58:50.517537+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.517537+0000","objects_scrubbed":5,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:02:38.977902+0000","scrub_duration":0.021023528999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":7042,"num_write":68,"num_write_kb":4224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7644,"num_omap_keys":14,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"67'680","reported_seq":3702,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:29.696376+0000","last_change":"2023-10-16T20:59:29.696444+0000","last_active":"2023-10-16T20:59:29.696376+0000","last_peered":"2023-10-16T20:59:29.696376+0000","last_clean":"2023-10-16T20:59:29.696376+0000","last_became_active":"2023-10-16T20:37:37.467402+0000","last_became_peered":"2023-10-16T20:37:37.467402+0000","last_unstale":"2023-10-16T20:59:29.696376+0000","last_undegraded":"2023-10-16T20:59:29.696376+0000","last_fullsized":"2023-10-16T20:59:29.696376+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'680","last_scrub_stamp":"2023-10-16T20:59:29.696283+0000","last_deep_scrub":"67'680","last_deep_scrub_stamp":"2023-10-16T20:59:29.696283+0000","last_clean_scrub_stamp":"2023-10-16T20:59:29.696283+0000","objects_scrubbed":333,"log_size":680,"log_dups_size":0,"ondisk_log_size":680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:41:51.780950+0000","scrub_duration":0.143599907,"objects_trimmed":0,"snaptrim_duration":0.012017773000000001,"stat_sum":{"num_bytes":36935626,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":884,"num_read_kb":21394,"num_write":679,"num_write_kb":40520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2,0],"acting":[6,7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"67'106","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:19.134997+0000","last_change":"2023-10-16T20:59:19.134997+0000","last_active":"2023-10-16T20:59:19.134997+0000","last_peered":"2023-10-16T20:59:19.134997+0000","last_clean":"2023-10-16T20:59:19.134997+0000","last_became_active":"2023-10-16T20:36:55.300958+0000","last_became_peered":"2023-10-16T20:36:55.300958+0000","last_unstale":"2023-10-16T20:59:19.134997+0000","last_undegraded":"2023-10-16T20:59:19.134997+0000","last_fullsized":"2023-10-16T20:59:19.134997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'106","last_scrub_stamp":"2023-10-16T20:59:19.134877+0000","last_deep_scrub":"67'106","last_deep_scrub_stamp":"2023-10-16T20:59:19.134877+0000","last_clean_scrub_stamp":"2023-10-16T20:59:19.134877+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:06:46.731251+0000","scrub_duration":0.0019278839999999999,"objects_trimmed":0,"snaptrim_duration":0.009734431,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"67'323","reported_seq":405,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:02.756886+0000","last_change":"2023-10-16T20:58:51.074256+0000","last_active":"2023-10-16T20:59:02.756886+0000","last_peered":"2023-10-16T20:59:02.756886+0000","last_clean":"2023-10-16T20:59:02.756886+0000","last_became_active":"2023-10-16T20:36:54.753443+0000","last_became_peered":"2023-10-16T20:36:54.753443+0000","last_unstale":"2023-10-16T20:59:02.756886+0000","last_undegraded":"2023-10-16T20:59:02.756886+0000","last_fullsized":"2023-10-16T20:59:02.756886+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'322","last_scrub_stamp":"2023-10-16T20:58:51.074122+0000","last_deep_scrub":"67'322","last_deep_scrub_stamp":"2023-10-16T20:58:51.074122+0000","last_clean_scrub_stamp":"2023-10-16T20:58:51.074122+0000","objects_scrubbed":7,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:15:03.275883+0000","scrub_duration":0.031482483999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9668880,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":15859,"num_write":310,"num_write_kb":9619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":30112,"num_omap_keys":56,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'687","reported_seq":4000,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:17.390664+0000","last_change":"2023-10-16T20:59:17.390664+0000","last_active":"2023-10-16T20:59:17.390664+0000","last_peered":"2023-10-16T20:59:17.390664+0000","last_clean":"2023-10-16T20:59:17.390664+0000","last_became_active":"2023-10-16T20:36:59.302869+0000","last_became_peered":"2023-10-16T20:36:59.302869+0000","last_unstale":"2023-10-16T20:59:17.390664+0000","last_undegraded":"2023-10-16T20:59:17.390664+0000","last_fullsized":"2023-10-16T20:59:17.390664+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'687","last_scrub_stamp":"2023-10-16T20:59:17.390491+0000","last_deep_scrub":"67'687","last_deep_scrub_stamp":"2023-10-16T20:59:17.390491+0000","last_clean_scrub_stamp":"2023-10-16T20:59:17.390491+0000","objects_scrubbed":352,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:35:09.376960+0000","scrub_duration":0.137334489,"objects_trimmed":0,"snaptrim_duration":0.012136565,"stat_sum":{"num_bytes":23637013,"num_objects":352,"num_object_clones":0,"num_object_copies":1408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":1183,"num_read_kb":18772,"num_write":687,"num_write_kb":27489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"67'100","reported_seq":2219,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:45.793646+0000","last_change":"2023-10-16T20:59:45.793725+0000","last_active":"2023-10-16T20:59:45.793646+0000","last_peered":"2023-10-16T20:59:45.793646+0000","last_clean":"2023-10-16T20:59:45.793646+0000","last_became_active":"2023-10-16T20:36:55.291633+0000","last_became_peered":"2023-10-16T20:36:55.291633+0000","last_unstale":"2023-10-16T20:59:45.793646+0000","last_undegraded":"2023-10-16T20:59:45.793646+0000","last_fullsized":"2023-10-16T20:59:45.793646+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'100","last_scrub_stamp":"2023-10-16T20:59:45.793545+0000","last_deep_scrub":"67'100","last_deep_scrub_stamp":"2023-10-16T20:59:45.793545+0000","last_clean_scrub_stamp":"2023-10-16T20:59:45.793545+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:21:25.751082+0000","scrub_duration":0.001857809,"objects_trimmed":0,"snaptrim_duration":0.0085978630000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"67'126","reported_seq":211,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:54.604136+0000","last_change":"2023-10-16T20:58:54.604136+0000","last_active":"2023-10-16T20:58:54.604136+0000","last_peered":"2023-10-16T20:58:54.604136+0000","last_clean":"2023-10-16T20:58:54.604136+0000","last_became_active":"2023-10-16T20:36:54.747704+0000","last_became_peered":"2023-10-16T20:36:54.747704+0000","last_unstale":"2023-10-16T20:58:54.604136+0000","last_undegraded":"2023-10-16T20:58:54.604136+0000","last_fullsized":"2023-10-16T20:58:54.604136+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'126","last_scrub_stamp":"2023-10-16T20:58:54.603981+0000","last_deep_scrub":"67'126","last_deep_scrub_stamp":"2023-10-16T20:58:54.603981+0000","last_clean_scrub_stamp":"2023-10-16T20:58:54.603981+0000","objects_scrubbed":6,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:18:13.372332+0000","scrub_duration":0.020068987999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":116,"num_write_kb":4228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20215,"num_omap_keys":38,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"67'663","reported_seq":3408,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:31.329291+0000","last_change":"2023-10-16T20:59:31.329364+0000","last_active":"2023-10-16T20:59:31.329291+0000","last_peered":"2023-10-16T20:59:31.329291+0000","last_clean":"2023-10-16T20:59:31.329291+0000","last_became_active":"2023-10-16T20:36:59.467291+0000","last_became_peered":"2023-10-16T20:36:59.467291+0000","last_unstale":"2023-10-16T20:59:31.329291+0000","last_undegraded":"2023-10-16T20:59:31.329291+0000","last_fullsized":"2023-10-16T20:59:31.329291+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'663","last_scrub_stamp":"2023-10-16T20:59:31.329191+0000","last_deep_scrub":"67'663","last_deep_scrub_stamp":"2023-10-16T20:59:31.329191+0000","last_clean_scrub_stamp":"2023-10-16T20:59:31.329191+0000","objects_scrubbed":327,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:08:51.268721+0000","scrub_duration":0.13195290300000001,"objects_trimmed":0,"snaptrim_duration":0.0131915,"stat_sum":{"num_bytes":23136394,"num_objects":327,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":616,"num_read_kb":10065,"num_write":660,"num_write_kb":35181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"67'80","reported_seq":2199,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:20.894073+0000","last_change":"2023-10-16T20:59:20.894073+0000","last_active":"2023-10-16T20:59:20.894073+0000","last_peered":"2023-10-16T20:59:20.894073+0000","last_clean":"2023-10-16T20:59:20.894073+0000","last_became_active":"2023-10-16T20:36:55.290998+0000","last_became_peered":"2023-10-16T20:36:55.290998+0000","last_unstale":"2023-10-16T20:59:20.894073+0000","last_undegraded":"2023-10-16T20:59:20.894073+0000","last_fullsized":"2023-10-16T20:59:20.894073+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'80","last_scrub_stamp":"2023-10-16T20:59:20.893925+0000","last_deep_scrub":"67'80","last_deep_scrub_stamp":"2023-10-16T20:59:20.893925+0000","last_clean_scrub_stamp":"2023-10-16T20:59:20.893925+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:33:27.367641+0000","scrub_duration":0.001815213,"objects_trimmed":0,"snaptrim_duration":0.0082191850000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'95","reported_seq":173,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.292551+0000","last_change":"2023-10-16T20:58:50.292551+0000","last_active":"2023-10-16T20:58:50.292551+0000","last_peered":"2023-10-16T20:58:50.292551+0000","last_clean":"2023-10-16T20:58:50.292551+0000","last_became_active":"2023-10-16T20:36:54.761337+0000","last_became_peered":"2023-10-16T20:36:54.761337+0000","last_unstale":"2023-10-16T20:58:50.292551+0000","last_undegraded":"2023-10-16T20:58:50.292551+0000","last_fullsized":"2023-10-16T20:58:50.292551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'95","last_scrub_stamp":"2023-10-16T20:58:50.292415+0000","last_deep_scrub":"67'95","last_deep_scrub_stamp":"2023-10-16T20:58:50.292415+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.292415+0000","objects_scrubbed":5,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:51:30.957479+0000","scrub_duration":0.018399241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194326,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":5,"num_write":137,"num_write_kb":4352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18880,"num_omap_keys":36,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"67'666","reported_seq":3861,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:38.425058+0000","last_change":"2023-10-16T20:59:38.425167+0000","last_active":"2023-10-16T20:59:38.425058+0000","last_peered":"2023-10-16T20:59:38.425058+0000","last_clean":"2023-10-16T20:59:38.425058+0000","last_became_active":"2023-10-16T20:37:37.468309+0000","last_became_peered":"2023-10-16T20:37:37.468309+0000","last_unstale":"2023-10-16T20:59:38.425058+0000","last_undegraded":"2023-10-16T20:59:38.425058+0000","last_fullsized":"2023-10-16T20:59:38.425058+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'666","last_scrub_stamp":"2023-10-16T20:59:38.424960+0000","last_deep_scrub":"67'666","last_deep_scrub_stamp":"2023-10-16T20:59:38.424960+0000","last_clean_scrub_stamp":"2023-10-16T20:59:38.424960+0000","objects_scrubbed":315,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:58:08.663348+0000","scrub_duration":0.119144721,"objects_trimmed":0,"snaptrim_duration":0.0058030460000000001,"stat_sum":{"num_bytes":20758781,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":1058,"num_read_kb":21621,"num_write":667,"num_write_kb":32853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3,6],"acting":[4,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"67'107","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:23.969170+0000","last_change":"2023-10-16T20:59:23.969232+0000","last_active":"2023-10-16T20:59:23.969170+0000","last_peered":"2023-10-16T20:59:23.969170+0000","last_clean":"2023-10-16T20:59:23.969170+0000","last_became_active":"2023-10-16T20:36:55.288354+0000","last_became_peered":"2023-10-16T20:36:55.288354+0000","last_unstale":"2023-10-16T20:59:23.969170+0000","last_undegraded":"2023-10-16T20:59:23.969170+0000","last_fullsized":"2023-10-16T20:59:23.969170+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'107","last_scrub_stamp":"2023-10-16T20:59:23.969053+0000","last_deep_scrub":"67'107","last_deep_scrub_stamp":"2023-10-16T20:59:23.969053+0000","last_clean_scrub_stamp":"2023-10-16T20:59:23.969053+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:35:11.681865+0000","scrub_duration":0.0020025540000000001,"objects_trimmed":0,"snaptrim_duration":0.0076786709999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"67'348","reported_seq":437,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:57.706996+0000","last_change":"2023-10-16T20:58:57.707108+0000","last_active":"2023-10-16T20:58:57.706996+0000","last_peered":"2023-10-16T20:58:57.706996+0000","last_clean":"2023-10-16T20:58:57.706996+0000","last_became_active":"2023-10-16T20:36:54.737876+0000","last_became_peered":"2023-10-16T20:36:54.737876+0000","last_unstale":"2023-10-16T20:58:57.706996+0000","last_undegraded":"2023-10-16T20:58:57.706996+0000","last_fullsized":"2023-10-16T20:58:57.706996+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'348","last_scrub_stamp":"2023-10-16T20:58:57.706891+0000","last_deep_scrub":"67'348","last_deep_scrub_stamp":"2023-10-16T20:58:57.706891+0000","last_clean_scrub_stamp":"2023-10-16T20:58:57.706891+0000","objects_scrubbed":15,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:40:24.491212+0000","scrub_duration":0.033886146999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388720,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":39,"num_read_kb":4134,"num_write":333,"num_write_kb":8414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":43828,"num_omap_keys":84,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"67'666","reported_seq":3686,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:28.325879+0000","last_change":"2023-10-16T20:59:28.325991+0000","last_active":"2023-10-16T20:59:28.325879+0000","last_peered":"2023-10-16T20:59:28.325879+0000","last_clean":"2023-10-16T20:59:28.325879+0000","last_became_active":"2023-10-16T20:36:59.306801+0000","last_became_peered":"2023-10-16T20:36:59.306801+0000","last_unstale":"2023-10-16T20:59:28.325879+0000","last_undegraded":"2023-10-16T20:59:28.325879+0000","last_fullsized":"2023-10-16T20:59:28.325879+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'666","last_scrub_stamp":"2023-10-16T20:59:28.325767+0000","last_deep_scrub":"67'666","last_deep_scrub_stamp":"2023-10-16T20:59:28.325767+0000","last_clean_scrub_stamp":"2023-10-16T20:59:28.325767+0000","objects_scrubbed":301,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:19:38.407734+0000","scrub_duration":0.112555059,"objects_trimmed":0,"snaptrim_duration":0.010585667,"stat_sum":{"num_bytes":20444029,"num_objects":301,"num_object_clones":0,"num_object_copies":1204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":893,"num_read_kb":19660,"num_write":666,"num_write_kb":28474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"67'93","reported_seq":2211,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:06.354741+0000","last_change":"2023-10-16T20:59:06.354789+0000","last_active":"2023-10-16T20:59:06.354741+0000","last_peered":"2023-10-16T20:59:06.354741+0000","last_clean":"2023-10-16T20:59:06.354741+0000","last_became_active":"2023-10-16T20:36:55.287066+0000","last_became_peered":"2023-10-16T20:36:55.287066+0000","last_unstale":"2023-10-16T20:59:06.354741+0000","last_undegraded":"2023-10-16T20:59:06.354741+0000","last_fullsized":"2023-10-16T20:59:06.354741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'93","last_scrub_stamp":"2023-10-16T20:59:06.354644+0000","last_deep_scrub":"67'93","last_deep_scrub_stamp":"2023-10-16T20:59:06.354644+0000","last_clean_scrub_stamp":"2023-10-16T20:59:06.354644+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:18:40.827931+0000","scrub_duration":0.0027107630000000001,"objects_trimmed":0,"snaptrim_duration":0.012121162,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"67'105","reported_seq":185,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:55.485180+0000","last_change":"2023-10-16T20:58:55.485242+0000","last_active":"2023-10-16T20:58:55.485180+0000","last_peered":"2023-10-16T20:58:55.485180+0000","last_clean":"2023-10-16T20:58:55.485180+0000","last_became_active":"2023-10-16T20:36:54.764417+0000","last_became_peered":"2023-10-16T20:36:54.764417+0000","last_unstale":"2023-10-16T20:58:55.485180+0000","last_undegraded":"2023-10-16T20:58:55.485180+0000","last_fullsized":"2023-10-16T20:58:55.485180+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'105","last_scrub_stamp":"2023-10-16T20:58:55.485084+0000","last_deep_scrub":"67'105","last_deep_scrub_stamp":"2023-10-16T20:58:55.485084+0000","last_clean_scrub_stamp":"2023-10-16T20:58:55.485084+0000","objects_scrubbed":4,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:13:49.097422+0000","scrub_duration":0.015871666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":4110,"num_write":99,"num_write_kb":4200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'634","reported_seq":3776,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:47.175552+0000","last_change":"2023-10-16T20:59:47.175642+0000","last_active":"2023-10-16T20:59:47.175552+0000","last_peered":"2023-10-16T20:59:47.175552+0000","last_clean":"2023-10-16T20:59:47.175552+0000","last_became_active":"2023-10-16T20:36:59.464626+0000","last_became_peered":"2023-10-16T20:36:59.464626+0000","last_unstale":"2023-10-16T20:59:47.175552+0000","last_undegraded":"2023-10-16T20:59:47.175552+0000","last_fullsized":"2023-10-16T20:59:47.175552+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'634","last_scrub_stamp":"2023-10-16T20:59:47.175450+0000","last_deep_scrub":"67'634","last_deep_scrub_stamp":"2023-10-16T20:59:47.175450+0000","last_clean_scrub_stamp":"2023-10-16T20:59:47.175450+0000","objects_scrubbed":321,"log_size":634,"log_dups_size":0,"ondisk_log_size":634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:42:23.547688+0000","scrub_duration":0.12749555900000001,"objects_trimmed":0,"snaptrim_duration":0.0069938420000000001,"stat_sum":{"num_bytes":28405361,"num_objects":321,"num_object_clones":0,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":1015,"num_read_kb":24481,"num_write":633,"num_write_kb":44456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"67'113","reported_seq":2231,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:44.786327+0000","last_change":"2023-10-16T20:59:44.786374+0000","last_active":"2023-10-16T20:59:44.786327+0000","last_peered":"2023-10-16T20:59:44.786327+0000","last_clean":"2023-10-16T20:59:44.786327+0000","last_became_active":"2023-10-16T20:36:55.291396+0000","last_became_peered":"2023-10-16T20:36:55.291396+0000","last_unstale":"2023-10-16T20:59:44.786327+0000","last_undegraded":"2023-10-16T20:59:44.786327+0000","last_fullsized":"2023-10-16T20:59:44.786327+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'113","last_scrub_stamp":"2023-10-16T20:59:44.786224+0000","last_deep_scrub":"67'113","last_deep_scrub_stamp":"2023-10-16T20:59:44.786224+0000","last_clean_scrub_stamp":"2023-10-16T20:59:44.786224+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:14:59.277249+0000","scrub_duration":0.0018387779999999999,"objects_trimmed":0,"snaptrim_duration":0.011464676,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"67'189","reported_seq":272,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:56.473518+0000","last_change":"2023-10-16T20:58:56.473573+0000","last_active":"2023-10-16T20:58:56.473518+0000","last_peered":"2023-10-16T20:58:56.473518+0000","last_clean":"2023-10-16T20:58:56.473518+0000","last_became_active":"2023-10-16T20:36:54.750448+0000","last_became_peered":"2023-10-16T20:36:54.750448+0000","last_unstale":"2023-10-16T20:58:56.473518+0000","last_undegraded":"2023-10-16T20:58:56.473518+0000","last_fullsized":"2023-10-16T20:58:56.473518+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'189","last_scrub_stamp":"2023-10-16T20:58:56.473428+0000","last_deep_scrub":"67'189","last_deep_scrub_stamp":"2023-10-16T20:58:56.473428+0000","last_clean_scrub_stamp":"2023-10-16T20:58:56.473428+0000","objects_scrubbed":5,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:18:30.642536+0000","scrub_duration":0.01623637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194394,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":3911,"num_write":180,"num_write_kb":4223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12239,"num_omap_keys":24,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"67'608","reported_seq":3351,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:39.167886+0000","last_change":"2023-10-16T20:59:39.168011+0000","last_active":"2023-10-16T20:59:39.167886+0000","last_peered":"2023-10-16T20:59:39.167886+0000","last_clean":"2023-10-16T20:59:39.167886+0000","last_became_active":"2023-10-16T20:36:59.464578+0000","last_became_peered":"2023-10-16T20:36:59.464578+0000","last_unstale":"2023-10-16T20:59:39.167886+0000","last_undegraded":"2023-10-16T20:59:39.167886+0000","last_fullsized":"2023-10-16T20:59:39.167886+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'608","last_scrub_stamp":"2023-10-16T20:59:39.167773+0000","last_deep_scrub":"67'608","last_deep_scrub_stamp":"2023-10-16T20:59:39.167773+0000","last_clean_scrub_stamp":"2023-10-16T20:59:39.167773+0000","objects_scrubbed":314,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:13:35.592473+0000","scrub_duration":0.121012868,"objects_trimmed":0,"snaptrim_duration":0.0064826880000000003,"stat_sum":{"num_bytes":21354704,"num_objects":314,"num_object_clones":0,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":616,"num_read_kb":9691,"num_write":609,"num_write_kb":35926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"67'101","reported_seq":2219,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:56.192318+0000","last_change":"2023-10-16T20:58:56.192363+0000","last_active":"2023-10-16T20:58:56.192318+0000","last_peered":"2023-10-16T20:58:56.192318+0000","last_clean":"2023-10-16T20:58:56.192318+0000","last_became_active":"2023-10-16T20:36:55.285407+0000","last_became_peered":"2023-10-16T20:36:55.285407+0000","last_unstale":"2023-10-16T20:58:56.192318+0000","last_undegraded":"2023-10-16T20:58:56.192318+0000","last_fullsized":"2023-10-16T20:58:56.192318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'101","last_scrub_stamp":"2023-10-16T20:58:56.192235+0000","last_deep_scrub":"67'101","last_deep_scrub_stamp":"2023-10-16T20:58:56.192235+0000","last_clean_scrub_stamp":"2023-10-16T20:58:56.192235+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:39:41.514891+0000","scrub_duration":0.0021418700000000001,"objects_trimmed":0,"snaptrim_duration":0.010336254,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"67'469","reported_seq":623,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:52.147902+0000","last_change":"2023-10-16T20:58:52.147973+0000","last_active":"2023-10-16T20:58:52.147902+0000","last_peered":"2023-10-16T20:58:52.147902+0000","last_clean":"2023-10-16T20:58:52.147902+0000","last_became_active":"2023-10-16T20:36:54.756697+0000","last_became_peered":"2023-10-16T20:36:54.756697+0000","last_unstale":"2023-10-16T20:58:52.147902+0000","last_undegraded":"2023-10-16T20:58:52.147902+0000","last_fullsized":"2023-10-16T20:58:52.147902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'469","last_scrub_stamp":"2023-10-16T20:58:52.147814+0000","last_deep_scrub":"67'469","last_deep_scrub_stamp":"2023-10-16T20:58:52.147814+0000","last_clean_scrub_stamp":"2023-10-16T20:58:52.147814+0000","objects_scrubbed":10,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:59:37.271178+0000","scrub_duration":0.031542023000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8464867,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":106,"num_read_kb":16561,"num_write":461,"num_write_kb":8611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11094,"num_omap_keys":22,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'632","reported_seq":4068,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:30.366503+0000","last_change":"2023-10-16T20:59:30.366601+0000","last_active":"2023-10-16T20:59:30.366503+0000","last_peered":"2023-10-16T20:59:30.366503+0000","last_clean":"2023-10-16T20:59:30.366503+0000","last_became_active":"2023-10-16T20:36:59.468254+0000","last_became_peered":"2023-10-16T20:36:59.468254+0000","last_unstale":"2023-10-16T20:59:30.366503+0000","last_undegraded":"2023-10-16T20:59:30.366503+0000","last_fullsized":"2023-10-16T20:59:30.366503+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'632","last_scrub_stamp":"2023-10-16T20:59:30.366394+0000","last_deep_scrub":"67'632","last_deep_scrub_stamp":"2023-10-16T20:59:30.366394+0000","last_clean_scrub_stamp":"2023-10-16T20:59:30.366394+0000","objects_scrubbed":300,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:26:37.125587+0000","scrub_duration":0.15197577300000001,"objects_trimmed":0,"snaptrim_duration":0.0093892239999999998,"stat_sum":{"num_bytes":28874401,"num_objects":300,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":1309,"num_read_kb":15538,"num_write":630,"num_write_kb":28525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"67'100","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:53.035022+0000","last_change":"2023-10-16T20:58:53.035079+0000","last_active":"2023-10-16T20:58:53.035022+0000","last_peered":"2023-10-16T20:58:53.035022+0000","last_clean":"2023-10-16T20:58:53.035022+0000","last_became_active":"2023-10-16T20:36:55.288488+0000","last_became_peered":"2023-10-16T20:36:55.288488+0000","last_unstale":"2023-10-16T20:58:53.035022+0000","last_undegraded":"2023-10-16T20:58:53.035022+0000","last_fullsized":"2023-10-16T20:58:53.035022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'100","last_scrub_stamp":"2023-10-16T20:58:53.034921+0000","last_deep_scrub":"67'100","last_deep_scrub_stamp":"2023-10-16T20:58:53.034921+0000","last_clean_scrub_stamp":"2023-10-16T20:58:53.034921+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:48:10.663084+0000","scrub_duration":0.002664118,"objects_trimmed":0,"snaptrim_duration":0.0078405860000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"67'147","reported_seq":240,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:01.469451+0000","last_change":"2023-10-16T20:59:01.469451+0000","last_active":"2023-10-16T20:59:01.469451+0000","last_peered":"2023-10-16T20:59:01.469451+0000","last_clean":"2023-10-16T20:59:01.469451+0000","last_became_active":"2023-10-16T20:36:54.764969+0000","last_became_peered":"2023-10-16T20:36:54.764969+0000","last_unstale":"2023-10-16T20:59:01.469451+0000","last_undegraded":"2023-10-16T20:59:01.469451+0000","last_fullsized":"2023-10-16T20:59:01.469451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'146","last_scrub_stamp":"2023-10-16T20:59:01.469291+0000","last_deep_scrub":"67'146","last_deep_scrub_stamp":"2023-10-16T20:59:01.469291+0000","last_clean_scrub_stamp":"2023-10-16T20:59:01.469291+0000","objects_scrubbed":7,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:33:10.085576+0000","scrub_duration":0.028155902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8389280,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":45,"num_read_kb":8235,"num_write":136,"num_write_kb":8276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"67'666","reported_seq":4080,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:13.118241+0000","last_change":"2023-10-16T20:59:13.118315+0000","last_active":"2023-10-16T20:59:13.118241+0000","last_peered":"2023-10-16T20:59:13.118241+0000","last_clean":"2023-10-16T20:59:13.118241+0000","last_became_active":"2023-10-16T20:36:59.302698+0000","last_became_peered":"2023-10-16T20:36:59.302698+0000","last_unstale":"2023-10-16T20:59:13.118241+0000","last_undegraded":"2023-10-16T20:59:13.118241+0000","last_fullsized":"2023-10-16T20:59:13.118241+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'666","last_scrub_stamp":"2023-10-16T20:59:13.118142+0000","last_deep_scrub":"67'666","last_deep_scrub_stamp":"2023-10-16T20:59:13.118142+0000","last_clean_scrub_stamp":"2023-10-16T20:59:13.118142+0000","objects_scrubbed":333,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:36:54.067602+0000","scrub_duration":0.142898739,"objects_trimmed":0,"snaptrim_duration":0.0064797020000000004,"stat_sum":{"num_bytes":31392251,"num_objects":333,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":1286,"num_read_kb":27897,"num_write":665,"num_write_kb":43250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'99","reported_seq":2217,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:59.336824+0000","last_change":"2023-10-16T20:58:59.336882+0000","last_active":"2023-10-16T20:58:59.336824+0000","last_peered":"2023-10-16T20:58:59.336824+0000","last_clean":"2023-10-16T20:58:59.336824+0000","last_became_active":"2023-10-16T20:36:55.290127+0000","last_became_peered":"2023-10-16T20:36:55.290127+0000","last_unstale":"2023-10-16T20:58:59.336824+0000","last_undegraded":"2023-10-16T20:58:59.336824+0000","last_fullsized":"2023-10-16T20:58:59.336824+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'99","last_scrub_stamp":"2023-10-16T20:58:59.336718+0000","last_deep_scrub":"67'99","last_deep_scrub_stamp":"2023-10-16T20:58:59.336718+0000","last_clean_scrub_stamp":"2023-10-16T20:58:59.336718+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:51:20.952065+0000","scrub_duration":0.0019832140000000001,"objects_trimmed":0,"snaptrim_duration":0.0055494209999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"67'84","reported_seq":165,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:00.071444+0000","last_change":"2023-10-16T20:59:00.071528+0000","last_active":"2023-10-16T20:59:00.071444+0000","last_peered":"2023-10-16T20:59:00.071444+0000","last_clean":"2023-10-16T20:59:00.071444+0000","last_became_active":"2023-10-16T20:36:54.748788+0000","last_became_peered":"2023-10-16T20:36:54.748788+0000","last_unstale":"2023-10-16T20:59:00.071444+0000","last_undegraded":"2023-10-16T20:59:00.071444+0000","last_fullsized":"2023-10-16T20:59:00.071444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'84","last_scrub_stamp":"2023-10-16T20:59:00.071354+0000","last_deep_scrub":"67'84","last_deep_scrub_stamp":"2023-10-16T20:59:00.071354+0000","last_clean_scrub_stamp":"2023-10-16T20:59:00.071354+0000","objects_scrubbed":6,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:33:58.876619+0000","scrub_duration":0.006295775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":15,"num_read_kb":23,"num_write":83,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9178,"num_omap_keys":18,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"67'606","reported_seq":3881,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:13.298004+0000","last_change":"2023-10-16T20:59:13.298078+0000","last_active":"2023-10-16T20:59:13.298004+0000","last_peered":"2023-10-16T20:59:13.298004+0000","last_clean":"2023-10-16T20:59:13.298004+0000","last_became_active":"2023-10-16T20:36:59.302916+0000","last_became_peered":"2023-10-16T20:36:59.302916+0000","last_unstale":"2023-10-16T20:59:13.298004+0000","last_undegraded":"2023-10-16T20:59:13.298004+0000","last_fullsized":"2023-10-16T20:59:13.298004+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'606","last_scrub_stamp":"2023-10-16T20:59:13.297907+0000","last_deep_scrub":"67'606","last_deep_scrub_stamp":"2023-10-16T20:59:13.297907+0000","last_clean_scrub_stamp":"2023-10-16T20:59:13.297907+0000","objects_scrubbed":300,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:10:39.220250+0000","scrub_duration":0.11959800800000001,"objects_trimmed":0,"snaptrim_duration":0.0041353309999999999,"stat_sum":{"num_bytes":29291517,"num_objects":300,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":1148,"num_read_kb":24111,"num_write":607,"num_write_kb":41174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"67'107","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:01.224253+0000","last_change":"2023-10-16T20:59:01.224305+0000","last_active":"2023-10-16T20:59:01.224253+0000","last_peered":"2023-10-16T20:59:01.224253+0000","last_clean":"2023-10-16T20:59:01.224253+0000","last_became_active":"2023-10-16T20:36:55.291448+0000","last_became_peered":"2023-10-16T20:36:55.291448+0000","last_unstale":"2023-10-16T20:59:01.224253+0000","last_undegraded":"2023-10-16T20:59:01.224253+0000","last_fullsized":"2023-10-16T20:59:01.224253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'107","last_scrub_stamp":"2023-10-16T20:59:01.224155+0000","last_deep_scrub":"67'107","last_deep_scrub_stamp":"2023-10-16T20:59:01.224155+0000","last_clean_scrub_stamp":"2023-10-16T20:59:01.224155+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:59:11.517988+0000","scrub_duration":0.0022855140000000002,"objects_trimmed":0,"snaptrim_duration":0.010585074,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"67'149","reported_seq":278,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:05.396279+0000","last_change":"2023-10-16T20:59:05.396279+0000","last_active":"2023-10-16T20:59:05.396279+0000","last_peered":"2023-10-16T20:59:05.396279+0000","last_clean":"2023-10-16T20:59:05.396279+0000","last_became_active":"2023-10-16T20:36:54.761541+0000","last_became_peered":"2023-10-16T20:36:54.761541+0000","last_unstale":"2023-10-16T20:59:05.396279+0000","last_undegraded":"2023-10-16T20:59:05.396279+0000","last_fullsized":"2023-10-16T20:59:05.396279+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'148","last_scrub_stamp":"2023-10-16T20:59:05.396099+0000","last_deep_scrub":"67'148","last_deep_scrub_stamp":"2023-10-16T20:59:05.396099+0000","last_clean_scrub_stamp":"2023-10-16T20:59:05.396099+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:44:03.980765+0000","scrub_duration":0.01804234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2788685,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":69,"num_read_kb":129,"num_write":144,"num_write_kb":2886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":19476,"num_omap_keys":37,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"67'663","reported_seq":3346,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:32.319125+0000","last_change":"2023-10-16T20:59:32.319249+0000","last_active":"2023-10-16T20:59:32.319125+0000","last_peered":"2023-10-16T20:59:32.319125+0000","last_clean":"2023-10-16T20:59:32.319125+0000","last_became_active":"2023-10-16T20:36:59.303773+0000","last_became_peered":"2023-10-16T20:36:59.303773+0000","last_unstale":"2023-10-16T20:59:32.319125+0000","last_undegraded":"2023-10-16T20:59:32.319125+0000","last_fullsized":"2023-10-16T20:59:32.319125+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'663","last_scrub_stamp":"2023-10-16T20:59:32.319015+0000","last_deep_scrub":"67'663","last_deep_scrub_stamp":"2023-10-16T20:59:32.319015+0000","last_clean_scrub_stamp":"2023-10-16T20:59:32.319015+0000","objects_scrubbed":315,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:14:50.523968+0000","scrub_duration":0.121881516,"objects_trimmed":0,"snaptrim_duration":0.007941142,"stat_sum":{"num_bytes":21664408,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":557,"num_read_kb":9585,"num_write":658,"num_write_kb":32916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"67'121","reported_seq":2239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:26.217163+0000","last_change":"2023-10-16T20:59:26.217216+0000","last_active":"2023-10-16T20:59:26.217163+0000","last_peered":"2023-10-16T20:59:26.217163+0000","last_clean":"2023-10-16T20:59:26.217163+0000","last_became_active":"2023-10-16T20:36:55.300535+0000","last_became_peered":"2023-10-16T20:36:55.300535+0000","last_unstale":"2023-10-16T20:59:26.217163+0000","last_undegraded":"2023-10-16T20:59:26.217163+0000","last_fullsized":"2023-10-16T20:59:26.217163+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'121","last_scrub_stamp":"2023-10-16T20:59:26.217064+0000","last_deep_scrub":"67'121","last_deep_scrub_stamp":"2023-10-16T20:59:26.217064+0000","last_clean_scrub_stamp":"2023-10-16T20:59:26.217064+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:41:02.094148+0000","scrub_duration":0.0019061880000000001,"objects_trimmed":0,"snaptrim_duration":0.013755641000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"67'61","reported_seq":139,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:52.484963+0000","last_change":"2023-10-16T20:58:52.484963+0000","last_active":"2023-10-16T20:58:52.484963+0000","last_peered":"2023-10-16T20:58:52.484963+0000","last_clean":"2023-10-16T20:58:52.484963+0000","last_became_active":"2023-10-16T20:36:54.757303+0000","last_became_peered":"2023-10-16T20:36:54.757303+0000","last_unstale":"2023-10-16T20:58:52.484963+0000","last_undegraded":"2023-10-16T20:58:52.484963+0000","last_fullsized":"2023-10-16T20:58:52.484963+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'60","last_scrub_stamp":"2023-10-16T20:58:52.484737+0000","last_deep_scrub":"67'60","last_deep_scrub_stamp":"2023-10-16T20:58:52.484737+0000","last_clean_scrub_stamp":"2023-10-16T20:58:52.484737+0000","objects_scrubbed":6,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:17:16.465796+0000","scrub_duration":0.0082689740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":59,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":40464,"num_omap_keys":76,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"67'687","reported_seq":4240,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:30.401172+0000","last_change":"2023-10-16T20:59:30.401234+0000","last_active":"2023-10-16T20:59:30.401172+0000","last_peered":"2023-10-16T20:59:30.401172+0000","last_clean":"2023-10-16T20:59:30.401172+0000","last_became_active":"2023-10-16T20:36:59.465844+0000","last_became_peered":"2023-10-16T20:36:59.465844+0000","last_unstale":"2023-10-16T20:59:30.401172+0000","last_undegraded":"2023-10-16T20:59:30.401172+0000","last_fullsized":"2023-10-16T20:59:30.401172+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'687","last_scrub_stamp":"2023-10-16T20:59:30.401099+0000","last_deep_scrub":"67'687","last_deep_scrub_stamp":"2023-10-16T20:59:30.401099+0000","last_clean_scrub_stamp":"2023-10-16T20:59:30.401099+0000","objects_scrubbed":353,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T23:12:35.314763+0000","scrub_duration":0.183474843,"objects_trimmed":0,"snaptrim_duration":0.0081168009999999999,"stat_sum":{"num_bytes":45881583,"num_objects":353,"num_object_clones":0,"num_object_copies":1412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":353,"num_whiteouts":0,"num_read":1424,"num_read_kb":44764,"num_write":688,"num_write_kb":65580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"67'661","reported_seq":4070,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:30.379036+0000","last_change":"2023-10-16T20:59:30.379144+0000","last_active":"2023-10-16T20:59:30.379036+0000","last_peered":"2023-10-16T20:59:30.379036+0000","last_clean":"2023-10-16T20:59:30.379036+0000","last_became_active":"2023-10-16T20:36:59.467356+0000","last_became_peered":"2023-10-16T20:36:59.467356+0000","last_unstale":"2023-10-16T20:59:30.379036+0000","last_undegraded":"2023-10-16T20:59:30.379036+0000","last_fullsized":"2023-10-16T20:59:30.379036+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'661","last_scrub_stamp":"2023-10-16T20:59:30.378951+0000","last_deep_scrub":"67'661","last_deep_scrub_stamp":"2023-10-16T20:59:30.378951+0000","last_clean_scrub_stamp":"2023-10-16T20:59:30.378951+0000","objects_scrubbed":335,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:47:25.326887+0000","scrub_duration":0.17385836700000001,"objects_trimmed":0,"snaptrim_duration":0.0093722700000000003,"stat_sum":{"num_bytes":37038710,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1279,"num_read_kb":28276,"num_write":659,"num_write_kb":40616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'108","reported_seq":2226,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:11.931941+0000","last_change":"2023-10-16T20:59:11.931993+0000","last_active":"2023-10-16T20:59:11.931941+0000","last_peered":"2023-10-16T20:59:11.931941+0000","last_clean":"2023-10-16T20:59:11.931941+0000","last_became_active":"2023-10-16T20:36:55.280903+0000","last_became_peered":"2023-10-16T20:36:55.280903+0000","last_unstale":"2023-10-16T20:59:11.931941+0000","last_undegraded":"2023-10-16T20:59:11.931941+0000","last_fullsized":"2023-10-16T20:59:11.931941+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'108","last_scrub_stamp":"2023-10-16T20:59:11.931838+0000","last_deep_scrub":"67'108","last_deep_scrub_stamp":"2023-10-16T20:59:11.931838+0000","last_clean_scrub_stamp":"2023-10-16T20:59:11.931838+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:53:40.858669+0000","scrub_duration":0.0017970499999999999,"objects_trimmed":0,"snaptrim_duration":0.0029269690000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"67'463","reported_seq":543,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:04.451251+0000","last_change":"2023-10-16T20:59:04.451310+0000","last_active":"2023-10-16T20:59:04.451251+0000","last_peered":"2023-10-16T20:59:04.451251+0000","last_clean":"2023-10-16T20:59:04.451251+0000","last_became_active":"2023-10-16T20:36:54.756775+0000","last_became_peered":"2023-10-16T20:36:54.756775+0000","last_unstale":"2023-10-16T20:59:04.451251+0000","last_undegraded":"2023-10-16T20:59:04.451251+0000","last_fullsized":"2023-10-16T20:59:04.451251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'463","last_scrub_stamp":"2023-10-16T20:59:04.451169+0000","last_deep_scrub":"67'463","last_deep_scrub_stamp":"2023-10-16T20:59:04.451169+0000","last_clean_scrub_stamp":"2023-10-16T20:59:04.451169+0000","objects_scrubbed":6,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:16.153884+0000","scrub_duration":0.039845845999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12583494,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":16386,"num_write":448,"num_write_kb":12544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":20150,"num_omap_keys":39,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"67'638","reported_seq":3579,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:39.766061+0000","last_change":"2023-10-16T20:59:39.766180+0000","last_active":"2023-10-16T20:59:39.766061+0000","last_peered":"2023-10-16T20:59:39.766061+0000","last_clean":"2023-10-16T20:59:39.766061+0000","last_became_active":"2023-10-16T20:37:37.438140+0000","last_became_peered":"2023-10-16T20:37:37.438140+0000","last_unstale":"2023-10-16T20:59:39.766061+0000","last_undegraded":"2023-10-16T20:59:39.766061+0000","last_fullsized":"2023-10-16T20:59:39.766061+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'638","last_scrub_stamp":"2023-10-16T20:59:39.765954+0000","last_deep_scrub":"67'638","last_deep_scrub_stamp":"2023-10-16T20:59:39.765954+0000","last_clean_scrub_stamp":"2023-10-16T20:59:39.765954+0000","objects_scrubbed":292,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:34:36.944127+0000","scrub_duration":0.11271870000000001,"objects_trimmed":0,"snaptrim_duration":0.012287041,"stat_sum":{"num_bytes":24792014,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":805,"num_read_kb":12206,"num_write":635,"num_write_kb":32676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2,0],"acting":[6,5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"67'121","reported_seq":2239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:12.182911+0000","last_change":"2023-10-16T20:59:12.182960+0000","last_active":"2023-10-16T20:59:12.182911+0000","last_peered":"2023-10-16T20:59:12.182911+0000","last_clean":"2023-10-16T20:59:12.182911+0000","last_became_active":"2023-10-16T20:36:55.273959+0000","last_became_peered":"2023-10-16T20:36:55.273959+0000","last_unstale":"2023-10-16T20:59:12.182911+0000","last_undegraded":"2023-10-16T20:59:12.182911+0000","last_fullsized":"2023-10-16T20:59:12.182911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'121","last_scrub_stamp":"2023-10-16T20:59:12.182820+0000","last_deep_scrub":"67'121","last_deep_scrub_stamp":"2023-10-16T20:59:12.182820+0000","last_clean_scrub_stamp":"2023-10-16T20:59:12.182820+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:24:30.216780+0000","scrub_duration":0.0019525829999999999,"objects_trimmed":0,"snaptrim_duration":0.0076333039999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"67'140","reported_seq":290,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:07.224275+0000","last_change":"2023-10-16T20:59:07.224354+0000","last_active":"2023-10-16T20:59:07.224275+0000","last_peered":"2023-10-16T20:59:07.224275+0000","last_clean":"2023-10-16T20:59:07.224275+0000","last_became_active":"2023-10-16T20:36:54.759059+0000","last_became_peered":"2023-10-16T20:36:54.759059+0000","last_unstale":"2023-10-16T20:59:07.224275+0000","last_undegraded":"2023-10-16T20:59:07.224275+0000","last_fullsized":"2023-10-16T20:59:07.224275+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'140","last_scrub_stamp":"2023-10-16T20:59:07.224189+0000","last_deep_scrub":"67'140","last_deep_scrub_stamp":"2023-10-16T20:59:07.224189+0000","last_clean_scrub_stamp":"2023-10-16T20:59:07.224189+0000","objects_scrubbed":5,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:11:29.789893+0000","scrub_duration":0.016670114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4348959,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":88,"num_read_kb":239,"num_write":133,"num_write_kb":4341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10224,"num_omap_keys":20,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"67'230","reported_seq":389,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.505513+0000","last_change":"2023-10-16T20:58:50.505513+0000","last_active":"2023-10-16T20:58:50.505513+0000","last_peered":"2023-10-16T20:58:50.505513+0000","last_clean":"2023-10-16T20:58:50.505513+0000","last_became_active":"2023-10-16T20:36:54.764436+0000","last_became_peered":"2023-10-16T20:36:54.764436+0000","last_unstale":"2023-10-16T20:58:50.505513+0000","last_undegraded":"2023-10-16T20:58:50.505513+0000","last_fullsized":"2023-10-16T20:58:50.505513+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'230","last_scrub_stamp":"2023-10-16T20:58:50.505320+0000","last_deep_scrub":"67'230","last_deep_scrub_stamp":"2023-10-16T20:58:50.505320+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.505320+0000","objects_scrubbed":7,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:05:38.516764+0000","scrub_duration":0.030122286000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8524313,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":105,"num_read_kb":8412,"num_write":220,"num_write_kb":8480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12018,"num_omap_keys":23,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"67'109","reported_seq":2227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:59.179962+0000","last_change":"2023-10-16T20:58:59.179962+0000","last_active":"2023-10-16T20:58:59.179962+0000","last_peered":"2023-10-16T20:58:59.179962+0000","last_clean":"2023-10-16T20:58:59.179962+0000","last_became_active":"2023-10-16T20:36:55.277873+0000","last_became_peered":"2023-10-16T20:36:55.277873+0000","last_unstale":"2023-10-16T20:58:59.179962+0000","last_undegraded":"2023-10-16T20:58:59.179962+0000","last_fullsized":"2023-10-16T20:58:59.179962+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'109","last_scrub_stamp":"2023-10-16T20:58:59.179829+0000","last_deep_scrub":"67'109","last_deep_scrub_stamp":"2023-10-16T20:58:59.179829+0000","last_clean_scrub_stamp":"2023-10-16T20:58:59.179829+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:34:00.424120+0000","scrub_duration":0.0027646039999999999,"objects_trimmed":0,"snaptrim_duration":0.012247773,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'608","reported_seq":3254,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:07.721115+0000","last_change":"2023-10-16T20:59:07.721210+0000","last_active":"2023-10-16T20:59:07.721115+0000","last_peered":"2023-10-16T20:59:07.721115+0000","last_clean":"2023-10-16T20:59:07.721115+0000","last_became_active":"2023-10-16T20:36:59.302435+0000","last_became_peered":"2023-10-16T20:36:59.302435+0000","last_unstale":"2023-10-16T20:59:07.721115+0000","last_undegraded":"2023-10-16T20:59:07.721115+0000","last_fullsized":"2023-10-16T20:59:07.721115+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'608","last_scrub_stamp":"2023-10-16T20:59:07.721011+0000","last_deep_scrub":"67'608","last_deep_scrub_stamp":"2023-10-16T20:59:07.721011+0000","last_clean_scrub_stamp":"2023-10-16T20:59:07.721011+0000","objects_scrubbed":323,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:33:34.190513+0000","scrub_duration":0.124636293,"objects_trimmed":0,"snaptrim_duration":0.001061892,"stat_sum":{"num_bytes":16477397,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":519,"num_read_kb":10696,"num_write":606,"num_write_kb":24605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"67'240","reported_seq":319,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:09.221980+0000","last_change":"2023-10-16T20:59:09.221980+0000","last_active":"2023-10-16T20:59:09.221980+0000","last_peered":"2023-10-16T20:59:09.221980+0000","last_clean":"2023-10-16T20:59:09.221980+0000","last_became_active":"2023-10-16T20:36:54.755834+0000","last_became_peered":"2023-10-16T20:36:54.755834+0000","last_unstale":"2023-10-16T20:59:09.221980+0000","last_undegraded":"2023-10-16T20:59:09.221980+0000","last_fullsized":"2023-10-16T20:59:09.221980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'240","last_scrub_stamp":"2023-10-16T20:59:09.221846+0000","last_deep_scrub":"67'240","last_deep_scrub_stamp":"2023-10-16T20:59:09.221846+0000","last_clean_scrub_stamp":"2023-10-16T20:59:09.221846+0000","objects_scrubbed":8,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:35:30.882044+0000","scrub_duration":0.036256330000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582946,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":16385,"num_write":235,"num_write_kb":12609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":12208,"num_omap_keys":23,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"67'104","reported_seq":2222,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:51.283381+0000","last_change":"2023-10-16T20:58:51.283381+0000","last_active":"2023-10-16T20:58:51.283381+0000","last_peered":"2023-10-16T20:58:51.283381+0000","last_clean":"2023-10-16T20:58:51.283381+0000","last_became_active":"2023-10-16T20:36:55.279333+0000","last_became_peered":"2023-10-16T20:36:55.279333+0000","last_unstale":"2023-10-16T20:58:51.283381+0000","last_undegraded":"2023-10-16T20:58:51.283381+0000","last_fullsized":"2023-10-16T20:58:51.283381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'104","last_scrub_stamp":"2023-10-16T20:58:51.283239+0000","last_deep_scrub":"67'104","last_deep_scrub_stamp":"2023-10-16T20:58:51.283239+0000","last_clean_scrub_stamp":"2023-10-16T20:58:51.283239+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:14:12.385879+0000","scrub_duration":0.0020579999999999999,"objects_trimmed":0,"snaptrim_duration":0.0047896609999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'629","reported_seq":3464,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:42.174888+0000","last_change":"2023-10-16T20:59:42.174982+0000","last_active":"2023-10-16T20:59:42.174888+0000","last_peered":"2023-10-16T20:59:42.174888+0000","last_clean":"2023-10-16T20:59:42.174888+0000","last_became_active":"2023-10-16T20:36:59.301864+0000","last_became_peered":"2023-10-16T20:36:59.301864+0000","last_unstale":"2023-10-16T20:59:42.174888+0000","last_undegraded":"2023-10-16T20:59:42.174888+0000","last_fullsized":"2023-10-16T20:59:42.174888+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'629","last_scrub_stamp":"2023-10-16T20:59:42.174789+0000","last_deep_scrub":"67'629","last_deep_scrub_stamp":"2023-10-16T20:59:42.174789+0000","last_clean_scrub_stamp":"2023-10-16T20:59:42.174789+0000","objects_scrubbed":304,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:00:53.749784+0000","scrub_duration":0.118588633,"objects_trimmed":0,"snaptrim_duration":0.0080450769999999994,"stat_sum":{"num_bytes":20568472,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":708,"num_read_kb":22370,"num_write":628,"num_write_kb":40892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"67'120","reported_seq":2239,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:14.471167+0000","last_change":"2023-10-16T20:59:14.471217+0000","last_active":"2023-10-16T20:59:14.471167+0000","last_peered":"2023-10-16T20:59:14.471167+0000","last_clean":"2023-10-16T20:59:14.471167+0000","last_became_active":"2023-10-16T20:36:55.287442+0000","last_became_peered":"2023-10-16T20:36:55.287442+0000","last_unstale":"2023-10-16T20:59:14.471167+0000","last_undegraded":"2023-10-16T20:59:14.471167+0000","last_fullsized":"2023-10-16T20:59:14.471167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'120","last_scrub_stamp":"2023-10-16T20:59:14.471080+0000","last_deep_scrub":"67'120","last_deep_scrub_stamp":"2023-10-16T20:59:14.471080+0000","last_clean_scrub_stamp":"2023-10-16T20:59:14.471080+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:07:17.500010+0000","scrub_duration":0.0021998500000000002,"objects_trimmed":0,"snaptrim_duration":0.011630351000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"67'131","reported_seq":2250,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:58.645708+0000","last_change":"2023-10-16T20:58:58.645760+0000","last_active":"2023-10-16T20:58:58.645708+0000","last_peered":"2023-10-16T20:58:58.645708+0000","last_clean":"2023-10-16T20:58:58.645708+0000","last_became_active":"2023-10-16T20:36:55.276276+0000","last_became_peered":"2023-10-16T20:36:55.276276+0000","last_unstale":"2023-10-16T20:58:58.645708+0000","last_undegraded":"2023-10-16T20:58:58.645708+0000","last_fullsized":"2023-10-16T20:58:58.645708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'131","last_scrub_stamp":"2023-10-16T20:58:58.645616+0000","last_deep_scrub":"67'131","last_deep_scrub_stamp":"2023-10-16T20:58:58.645616+0000","last_clean_scrub_stamp":"2023-10-16T20:58:58.645616+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:39:28.761446+0000","scrub_duration":0.0024049459999999998,"objects_trimmed":0,"snaptrim_duration":0.00091802500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"67'115","reported_seq":2241,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:12.209102+0000","last_change":"2023-10-16T20:59:12.209156+0000","last_active":"2023-10-16T20:59:12.209102+0000","last_peered":"2023-10-16T20:59:12.209102+0000","last_clean":"2023-10-16T20:59:12.209102+0000","last_became_active":"2023-10-16T20:37:37.413040+0000","last_became_peered":"2023-10-16T20:37:37.413040+0000","last_unstale":"2023-10-16T20:59:12.209102+0000","last_undegraded":"2023-10-16T20:59:12.209102+0000","last_fullsized":"2023-10-16T20:59:12.209102+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'115","last_scrub_stamp":"2023-10-16T20:59:12.208990+0000","last_deep_scrub":"67'115","last_deep_scrub_stamp":"2023-10-16T20:59:12.208990+0000","last_clean_scrub_stamp":"2023-10-16T20:59:12.208990+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:34:02.568264+0000","scrub_duration":0.0026514730000000001,"objects_trimmed":0,"snaptrim_duration":0.012618513,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"67'116","reported_seq":2235,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:04.262507+0000","last_change":"2023-10-16T20:59:04.262559+0000","last_active":"2023-10-16T20:59:04.262507+0000","last_peered":"2023-10-16T20:59:04.262507+0000","last_clean":"2023-10-16T20:59:04.262507+0000","last_became_active":"2023-10-16T20:36:55.275349+0000","last_became_peered":"2023-10-16T20:36:55.275349+0000","last_unstale":"2023-10-16T20:59:04.262507+0000","last_undegraded":"2023-10-16T20:59:04.262507+0000","last_fullsized":"2023-10-16T20:59:04.262507+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'116","last_scrub_stamp":"2023-10-16T20:59:04.262407+0000","last_deep_scrub":"67'116","last_deep_scrub_stamp":"2023-10-16T20:59:04.262407+0000","last_clean_scrub_stamp":"2023-10-16T20:59:04.262407+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:14:42.718879+0000","scrub_duration":0.0019239579999999999,"objects_trimmed":0,"snaptrim_duration":0.012136073000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"67'88","reported_seq":2207,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:53.160971+0000","last_change":"2023-10-16T20:58:53.161022+0000","last_active":"2023-10-16T20:58:53.160971+0000","last_peered":"2023-10-16T20:58:53.160971+0000","last_clean":"2023-10-16T20:58:53.160971+0000","last_became_active":"2023-10-16T20:36:55.277043+0000","last_became_peered":"2023-10-16T20:36:55.277043+0000","last_unstale":"2023-10-16T20:58:53.160971+0000","last_undegraded":"2023-10-16T20:58:53.160971+0000","last_fullsized":"2023-10-16T20:58:53.160971+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'88","last_scrub_stamp":"2023-10-16T20:58:53.160862+0000","last_deep_scrub":"67'88","last_deep_scrub_stamp":"2023-10-16T20:58:53.160862+0000","last_clean_scrub_stamp":"2023-10-16T20:58:53.160862+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:34:19.651642+0000","scrub_duration":0.001900938,"objects_trimmed":0,"snaptrim_duration":0.0078065369999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"67'89","reported_seq":2208,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:03.022307+0000","last_change":"2023-10-16T20:59:03.022367+0000","last_active":"2023-10-16T20:59:03.022307+0000","last_peered":"2023-10-16T20:59:03.022307+0000","last_clean":"2023-10-16T20:59:03.022307+0000","last_became_active":"2023-10-16T20:36:55.278715+0000","last_became_peered":"2023-10-16T20:36:55.278715+0000","last_unstale":"2023-10-16T20:59:03.022307+0000","last_undegraded":"2023-10-16T20:59:03.022307+0000","last_fullsized":"2023-10-16T20:59:03.022307+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'89","last_scrub_stamp":"2023-10-16T20:59:03.022209+0000","last_deep_scrub":"67'89","last_deep_scrub_stamp":"2023-10-16T20:59:03.022209+0000","last_clean_scrub_stamp":"2023-10-16T20:59:03.022209+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:09:31.713609+0000","scrub_duration":0.0019901680000000001,"objects_trimmed":0,"snaptrim_duration":0.0067460719999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"67'124","reported_seq":2243,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:03.386234+0000","last_change":"2023-10-16T20:59:03.386284+0000","last_active":"2023-10-16T20:59:03.386234+0000","last_peered":"2023-10-16T20:59:03.386234+0000","last_clean":"2023-10-16T20:59:03.386234+0000","last_became_active":"2023-10-16T20:36:55.301604+0000","last_became_peered":"2023-10-16T20:36:55.301604+0000","last_unstale":"2023-10-16T20:59:03.386234+0000","last_undegraded":"2023-10-16T20:59:03.386234+0000","last_fullsized":"2023-10-16T20:59:03.386234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'124","last_scrub_stamp":"2023-10-16T20:59:03.386139+0000","last_deep_scrub":"67'124","last_deep_scrub_stamp":"2023-10-16T20:59:03.386139+0000","last_clean_scrub_stamp":"2023-10-16T20:59:03.386139+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:16:18.278253+0000","scrub_duration":0.0020316280000000002,"objects_trimmed":0,"snaptrim_duration":0.012969489000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"67'99","reported_seq":2218,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:01.252753+0000","last_change":"2023-10-16T20:59:01.252799+0000","last_active":"2023-10-16T20:59:01.252753+0000","last_peered":"2023-10-16T20:59:01.252753+0000","last_clean":"2023-10-16T20:59:01.252753+0000","last_became_active":"2023-10-16T20:36:55.276984+0000","last_became_peered":"2023-10-16T20:36:55.276984+0000","last_unstale":"2023-10-16T20:59:01.252753+0000","last_undegraded":"2023-10-16T20:59:01.252753+0000","last_fullsized":"2023-10-16T20:59:01.252753+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'99","last_scrub_stamp":"2023-10-16T20:59:01.252663+0000","last_deep_scrub":"67'99","last_deep_scrub_stamp":"2023-10-16T20:59:01.252663+0000","last_clean_scrub_stamp":"2023-10-16T20:59:01.252663+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:32:54.405775+0000","scrub_duration":0.0025568829999999998,"objects_trimmed":0,"snaptrim_duration":0.0086623480000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"67'111","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:07.294638+0000","last_change":"2023-10-16T20:59:07.294686+0000","last_active":"2023-10-16T20:59:07.294638+0000","last_peered":"2023-10-16T20:59:07.294638+0000","last_clean":"2023-10-16T20:59:07.294638+0000","last_became_active":"2023-10-16T20:36:55.290023+0000","last_became_peered":"2023-10-16T20:36:55.290023+0000","last_unstale":"2023-10-16T20:59:07.294638+0000","last_undegraded":"2023-10-16T20:59:07.294638+0000","last_fullsized":"2023-10-16T20:59:07.294638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'111","last_scrub_stamp":"2023-10-16T20:59:07.294551+0000","last_deep_scrub":"67'111","last_deep_scrub_stamp":"2023-10-16T20:59:07.294551+0000","last_clean_scrub_stamp":"2023-10-16T20:59:07.294551+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:48:23.755124+0000","scrub_duration":0.0022375450000000001,"objects_trimmed":0,"snaptrim_duration":0.013081514000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"67'124","reported_seq":2243,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:10.167167+0000","last_change":"2023-10-16T20:59:10.167167+0000","last_active":"2023-10-16T20:59:10.167167+0000","last_peered":"2023-10-16T20:59:10.167167+0000","last_clean":"2023-10-16T20:59:10.167167+0000","last_became_active":"2023-10-16T20:36:55.280186+0000","last_became_peered":"2023-10-16T20:36:55.280186+0000","last_unstale":"2023-10-16T20:59:10.167167+0000","last_undegraded":"2023-10-16T20:59:10.167167+0000","last_fullsized":"2023-10-16T20:59:10.167167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'124","last_scrub_stamp":"2023-10-16T20:59:10.167005+0000","last_deep_scrub":"67'124","last_deep_scrub_stamp":"2023-10-16T20:59:10.167005+0000","last_clean_scrub_stamp":"2023-10-16T20:59:10.167005+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T07:49:34.469234+0000","scrub_duration":0.0025810059999999998,"objects_trimmed":0,"snaptrim_duration":0.0083925690000000008,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"67'118","reported_seq":2237,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:09.250466+0000","last_change":"2023-10-16T20:59:09.250466+0000","last_active":"2023-10-16T20:59:09.250466+0000","last_peered":"2023-10-16T20:59:09.250466+0000","last_clean":"2023-10-16T20:59:09.250466+0000","last_became_active":"2023-10-16T20:36:55.297253+0000","last_became_peered":"2023-10-16T20:36:55.297253+0000","last_unstale":"2023-10-16T20:59:09.250466+0000","last_undegraded":"2023-10-16T20:59:09.250466+0000","last_fullsized":"2023-10-16T20:59:09.250466+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'118","last_scrub_stamp":"2023-10-16T20:59:09.250291+0000","last_deep_scrub":"67'118","last_deep_scrub_stamp":"2023-10-16T20:59:09.250291+0000","last_clean_scrub_stamp":"2023-10-16T20:59:09.250291+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T01:36:37.809041+0000","scrub_duration":0.0016459720000000001,"objects_trimmed":0,"snaptrim_duration":0.0097685800000000007,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"67'108","reported_seq":2227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:10.288057+0000","last_change":"2023-10-16T20:59:10.288057+0000","last_active":"2023-10-16T20:59:10.288057+0000","last_peered":"2023-10-16T20:59:10.288057+0000","last_clean":"2023-10-16T20:59:10.288057+0000","last_became_active":"2023-10-16T20:36:55.286044+0000","last_became_peered":"2023-10-16T20:36:55.286044+0000","last_unstale":"2023-10-16T20:59:10.288057+0000","last_undegraded":"2023-10-16T20:59:10.288057+0000","last_fullsized":"2023-10-16T20:59:10.288057+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'108","last_scrub_stamp":"2023-10-16T20:59:10.287858+0000","last_deep_scrub":"67'108","last_deep_scrub_stamp":"2023-10-16T20:59:10.287858+0000","last_clean_scrub_stamp":"2023-10-16T20:59:10.287858+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T04:31:52.358547+0000","scrub_duration":0.002902193,"objects_trimmed":0,"snaptrim_duration":0.011681474000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"67'110","reported_seq":2229,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:10.363741+0000","last_change":"2023-10-16T20:59:10.363741+0000","last_active":"2023-10-16T20:59:10.363741+0000","last_peered":"2023-10-16T20:59:10.363741+0000","last_clean":"2023-10-16T20:59:10.363741+0000","last_became_active":"2023-10-16T20:36:55.286864+0000","last_became_peered":"2023-10-16T20:36:55.286864+0000","last_unstale":"2023-10-16T20:59:10.363741+0000","last_undegraded":"2023-10-16T20:59:10.363741+0000","last_fullsized":"2023-10-16T20:59:10.363741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'110","last_scrub_stamp":"2023-10-16T20:59:10.363590+0000","last_deep_scrub":"67'110","last_deep_scrub_stamp":"2023-10-16T20:59:10.363590+0000","last_clean_scrub_stamp":"2023-10-16T20:59:10.363590+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:01:11.804155+0000","scrub_duration":0.0018541569999999999,"objects_trimmed":0,"snaptrim_duration":0.012885232999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"67'106","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:43.738263+0000","last_change":"2023-10-16T20:59:43.738317+0000","last_active":"2023-10-16T20:59:43.738263+0000","last_peered":"2023-10-16T20:59:43.738263+0000","last_clean":"2023-10-16T20:59:43.738263+0000","last_became_active":"2023-10-16T20:36:55.292262+0000","last_became_peered":"2023-10-16T20:36:55.292262+0000","last_unstale":"2023-10-16T20:59:43.738263+0000","last_undegraded":"2023-10-16T20:59:43.738263+0000","last_fullsized":"2023-10-16T20:59:43.738263+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'106","last_scrub_stamp":"2023-10-16T20:59:43.738154+0000","last_deep_scrub":"67'106","last_deep_scrub_stamp":"2023-10-16T20:59:43.738154+0000","last_clean_scrub_stamp":"2023-10-16T20:59:43.738154+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:20:28.580919+0000","scrub_duration":0.002255721,"objects_trimmed":0,"snaptrim_duration":0.0077171749999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"67'107","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:00.659400+0000","last_change":"2023-10-16T20:59:00.659448+0000","last_active":"2023-10-16T20:59:00.659400+0000","last_peered":"2023-10-16T20:59:00.659400+0000","last_clean":"2023-10-16T20:59:00.659400+0000","last_became_active":"2023-10-16T20:36:55.282806+0000","last_became_peered":"2023-10-16T20:36:55.282806+0000","last_unstale":"2023-10-16T20:59:00.659400+0000","last_undegraded":"2023-10-16T20:59:00.659400+0000","last_fullsized":"2023-10-16T20:59:00.659400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'107","last_scrub_stamp":"2023-10-16T20:59:00.659305+0000","last_deep_scrub":"67'107","last_deep_scrub_stamp":"2023-10-16T20:59:00.659305+0000","last_clean_scrub_stamp":"2023-10-16T20:59:00.659305+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:01:49.319877+0000","scrub_duration":0.0024992080000000002,"objects_trimmed":0,"snaptrim_duration":0.00086132999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"67'109","reported_seq":2227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:24.189217+0000","last_change":"2023-10-16T20:59:24.189272+0000","last_active":"2023-10-16T20:59:24.189217+0000","last_peered":"2023-10-16T20:59:24.189217+0000","last_clean":"2023-10-16T20:59:24.189217+0000","last_became_active":"2023-10-16T20:36:55.303095+0000","last_became_peered":"2023-10-16T20:36:55.303095+0000","last_unstale":"2023-10-16T20:59:24.189217+0000","last_undegraded":"2023-10-16T20:59:24.189217+0000","last_fullsized":"2023-10-16T20:59:24.189217+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'109","last_scrub_stamp":"2023-10-16T20:59:24.189109+0000","last_deep_scrub":"67'109","last_deep_scrub_stamp":"2023-10-16T20:59:24.189109+0000","last_clean_scrub_stamp":"2023-10-16T20:59:24.189109+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:42:36.202135+0000","scrub_duration":0.0019574549999999999,"objects_trimmed":0,"snaptrim_duration":0.013746639999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"67'119","reported_seq":2237,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:08.257529+0000","last_change":"2023-10-16T20:59:08.257579+0000","last_active":"2023-10-16T20:59:08.257529+0000","last_peered":"2023-10-16T20:59:08.257529+0000","last_clean":"2023-10-16T20:59:08.257529+0000","last_became_active":"2023-10-16T20:36:55.291447+0000","last_became_peered":"2023-10-16T20:36:55.291447+0000","last_unstale":"2023-10-16T20:59:08.257529+0000","last_undegraded":"2023-10-16T20:59:08.257529+0000","last_fullsized":"2023-10-16T20:59:08.257529+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'119","last_scrub_stamp":"2023-10-16T20:59:08.257424+0000","last_deep_scrub":"67'119","last_deep_scrub_stamp":"2023-10-16T20:59:08.257424+0000","last_clean_scrub_stamp":"2023-10-16T20:59:08.257424+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:17:42.198263+0000","scrub_duration":0.002719015,"objects_trimmed":0,"snaptrim_duration":0.01315413,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"67'107","reported_seq":2225,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:48.072269+0000","last_change":"2023-10-16T20:58:48.072313+0000","last_active":"2023-10-16T20:58:48.072269+0000","last_peered":"2023-10-16T20:58:48.072269+0000","last_clean":"2023-10-16T20:58:48.072269+0000","last_became_active":"2023-10-16T20:36:55.280605+0000","last_became_peered":"2023-10-16T20:36:55.280605+0000","last_unstale":"2023-10-16T20:58:48.072269+0000","last_undegraded":"2023-10-16T20:58:48.072269+0000","last_fullsized":"2023-10-16T20:58:48.072269+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'107","last_scrub_stamp":"2023-10-16T20:58:48.072178+0000","last_deep_scrub":"67'107","last_deep_scrub_stamp":"2023-10-16T20:58:48.072178+0000","last_clean_scrub_stamp":"2023-10-16T20:58:48.072178+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:58:57.439416+0000","scrub_duration":0.0025767939999999999,"objects_trimmed":0,"snaptrim_duration":0.0033133080000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"67'112","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:57.143403+0000","last_change":"2023-10-16T20:58:57.143460+0000","last_active":"2023-10-16T20:58:57.143403+0000","last_peered":"2023-10-16T20:58:57.143403+0000","last_clean":"2023-10-16T20:58:57.143403+0000","last_became_active":"2023-10-16T20:36:55.277094+0000","last_became_peered":"2023-10-16T20:36:55.277094+0000","last_unstale":"2023-10-16T20:58:57.143403+0000","last_undegraded":"2023-10-16T20:58:57.143403+0000","last_fullsized":"2023-10-16T20:58:57.143403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'112","last_scrub_stamp":"2023-10-16T20:58:57.143322+0000","last_deep_scrub":"67'112","last_deep_scrub_stamp":"2023-10-16T20:58:57.143322+0000","last_clean_scrub_stamp":"2023-10-16T20:58:57.143322+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T06:52:54.678403+0000","scrub_duration":0.0016325790000000001,"objects_trimmed":0,"snaptrim_duration":0.011816411000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"67'94","reported_seq":2212,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.120683+0000","last_change":"2023-10-16T20:58:50.120723+0000","last_active":"2023-10-16T20:58:50.120683+0000","last_peered":"2023-10-16T20:58:50.120683+0000","last_clean":"2023-10-16T20:58:50.120683+0000","last_became_active":"2023-10-16T20:36:55.297199+0000","last_became_peered":"2023-10-16T20:36:55.297199+0000","last_unstale":"2023-10-16T20:58:50.120683+0000","last_undegraded":"2023-10-16T20:58:50.120683+0000","last_fullsized":"2023-10-16T20:58:50.120683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'94","last_scrub_stamp":"2023-10-16T20:58:50.120603+0000","last_deep_scrub":"67'94","last_deep_scrub_stamp":"2023-10-16T20:58:50.120603+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.120603+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:29:38.916058+0000","scrub_duration":0.0022327750000000002,"objects_trimmed":0,"snaptrim_duration":0.0086430910000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"67'90","reported_seq":2208,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:52.287328+0000","last_change":"2023-10-16T20:58:52.287408+0000","last_active":"2023-10-16T20:58:52.287328+0000","last_peered":"2023-10-16T20:58:52.287328+0000","last_clean":"2023-10-16T20:58:52.287328+0000","last_became_active":"2023-10-16T20:36:55.286964+0000","last_became_peered":"2023-10-16T20:36:55.286964+0000","last_unstale":"2023-10-16T20:58:52.287328+0000","last_undegraded":"2023-10-16T20:58:52.287328+0000","last_fullsized":"2023-10-16T20:58:52.287328+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'90","last_scrub_stamp":"2023-10-16T20:58:52.287221+0000","last_deep_scrub":"67'90","last_deep_scrub_stamp":"2023-10-16T20:58:52.287221+0000","last_clean_scrub_stamp":"2023-10-16T20:58:52.287221+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:06.754055+0000","scrub_duration":0.0018582900000000001,"objects_trimmed":0,"snaptrim_duration":0.0051857609999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"67'110","reported_seq":2235,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:08.351680+0000","last_change":"2023-10-16T20:59:08.351732+0000","last_active":"2023-10-16T20:59:08.351680+0000","last_peered":"2023-10-16T20:59:08.351680+0000","last_clean":"2023-10-16T20:59:08.351680+0000","last_became_active":"2023-10-16T20:37:37.413492+0000","last_became_peered":"2023-10-16T20:37:37.413492+0000","last_unstale":"2023-10-16T20:59:08.351680+0000","last_undegraded":"2023-10-16T20:59:08.351680+0000","last_fullsized":"2023-10-16T20:59:08.351680+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'110","last_scrub_stamp":"2023-10-16T20:59:08.351593+0000","last_deep_scrub":"67'110","last_deep_scrub_stamp":"2023-10-16T20:59:08.351593+0000","last_clean_scrub_stamp":"2023-10-16T20:59:08.351593+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T08:24:02.460822+0000","scrub_duration":0.0019714580000000002,"objects_trimmed":0,"snaptrim_duration":0.014529932000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"67'106","reported_seq":2224,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:22.427898+0000","last_change":"2023-10-16T20:59:22.427941+0000","last_active":"2023-10-16T20:59:22.427898+0000","last_peered":"2023-10-16T20:59:22.427898+0000","last_clean":"2023-10-16T20:59:22.427898+0000","last_became_active":"2023-10-16T20:36:55.302468+0000","last_became_peered":"2023-10-16T20:36:55.302468+0000","last_unstale":"2023-10-16T20:59:22.427898+0000","last_undegraded":"2023-10-16T20:59:22.427898+0000","last_fullsized":"2023-10-16T20:59:22.427898+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'106","last_scrub_stamp":"2023-10-16T20:59:22.427826+0000","last_deep_scrub":"67'106","last_deep_scrub_stamp":"2023-10-16T20:59:22.427826+0000","last_clean_scrub_stamp":"2023-10-16T20:59:22.427826+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T21:41:26.180362+0000","scrub_duration":0.001392236,"objects_trimmed":0,"snaptrim_duration":0.012359399,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"67'92","reported_seq":2210,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:26.003456+0000","last_change":"2023-10-16T20:59:26.003513+0000","last_active":"2023-10-16T20:59:26.003456+0000","last_peered":"2023-10-16T20:59:26.003456+0000","last_clean":"2023-10-16T20:59:26.003456+0000","last_became_active":"2023-10-16T20:36:55.296476+0000","last_became_peered":"2023-10-16T20:36:55.296476+0000","last_unstale":"2023-10-16T20:59:26.003456+0000","last_undegraded":"2023-10-16T20:59:26.003456+0000","last_fullsized":"2023-10-16T20:59:26.003456+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'92","last_scrub_stamp":"2023-10-16T20:59:26.003338+0000","last_deep_scrub":"67'92","last_deep_scrub_stamp":"2023-10-16T20:59:26.003338+0000","last_clean_scrub_stamp":"2023-10-16T20:59:26.003338+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:08:59.321272+0000","scrub_duration":0.001948794,"objects_trimmed":0,"snaptrim_duration":0.0072035780000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"67'95","reported_seq":2213,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:56.206084+0000","last_change":"2023-10-16T20:58:56.206084+0000","last_active":"2023-10-16T20:58:56.206084+0000","last_peered":"2023-10-16T20:58:56.206084+0000","last_clean":"2023-10-16T20:58:56.206084+0000","last_became_active":"2023-10-16T20:36:55.293687+0000","last_became_peered":"2023-10-16T20:36:55.293687+0000","last_unstale":"2023-10-16T20:58:56.206084+0000","last_undegraded":"2023-10-16T20:58:56.206084+0000","last_fullsized":"2023-10-16T20:58:56.206084+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'95","last_scrub_stamp":"2023-10-16T20:58:56.205959+0000","last_deep_scrub":"67'95","last_deep_scrub_stamp":"2023-10-16T20:58:56.205959+0000","last_clean_scrub_stamp":"2023-10-16T20:58:56.205959+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:29:53.185697+0000","scrub_duration":0.0025542830000000001,"objects_trimmed":0,"snaptrim_duration":0.0074548399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"67'127","reported_seq":2245,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:19.869758+0000","last_change":"2023-10-16T20:59:19.869758+0000","last_active":"2023-10-16T20:59:19.869758+0000","last_peered":"2023-10-16T20:59:19.869758+0000","last_clean":"2023-10-16T20:59:19.869758+0000","last_became_active":"2023-10-16T20:36:55.280444+0000","last_became_peered":"2023-10-16T20:36:55.280444+0000","last_unstale":"2023-10-16T20:59:19.869758+0000","last_undegraded":"2023-10-16T20:59:19.869758+0000","last_fullsized":"2023-10-16T20:59:19.869758+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'127","last_scrub_stamp":"2023-10-16T20:59:19.869537+0000","last_deep_scrub":"67'127","last_deep_scrub_stamp":"2023-10-16T20:59:19.869537+0000","last_clean_scrub_stamp":"2023-10-16T20:59:19.869537+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:58:41.974943+0000","scrub_duration":0.0024841920000000001,"objects_trimmed":0,"snaptrim_duration":0.0068623299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"67'109","reported_seq":2227,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.544632+0000","last_change":"2023-10-16T20:58:50.544632+0000","last_active":"2023-10-16T20:58:50.544632+0000","last_peered":"2023-10-16T20:58:50.544632+0000","last_clean":"2023-10-16T20:58:50.544632+0000","last_became_active":"2023-10-16T20:36:55.296694+0000","last_became_peered":"2023-10-16T20:36:55.296694+0000","last_unstale":"2023-10-16T20:58:50.544632+0000","last_undegraded":"2023-10-16T20:58:50.544632+0000","last_fullsized":"2023-10-16T20:58:50.544632+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'109","last_scrub_stamp":"2023-10-16T20:58:50.544515+0000","last_deep_scrub":"67'109","last_deep_scrub_stamp":"2023-10-16T20:58:50.544515+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.544515+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T05:15:02.193326+0000","scrub_duration":0.001832127,"objects_trimmed":0,"snaptrim_duration":0.0096291870000000009,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"67'120","reported_seq":2238,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:34.866076+0000","last_change":"2023-10-16T20:59:34.866132+0000","last_active":"2023-10-16T20:59:34.866076+0000","last_peered":"2023-10-16T20:59:34.866076+0000","last_clean":"2023-10-16T20:59:34.866076+0000","last_became_active":"2023-10-16T20:36:55.291052+0000","last_became_peered":"2023-10-16T20:36:55.291052+0000","last_unstale":"2023-10-16T20:59:34.866076+0000","last_undegraded":"2023-10-16T20:59:34.866076+0000","last_fullsized":"2023-10-16T20:59:34.866076+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'120","last_scrub_stamp":"2023-10-16T20:59:34.865956+0000","last_deep_scrub":"67'120","last_deep_scrub_stamp":"2023-10-16T20:59:34.865956+0000","last_clean_scrub_stamp":"2023-10-16T20:59:34.865956+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T03:44:28.508046+0000","scrub_duration":0.0025743160000000001,"objects_trimmed":0,"snaptrim_duration":0.0070559409999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"67'94","reported_seq":2214,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:49.588755+0000","last_change":"2023-10-16T20:58:49.588755+0000","last_active":"2023-10-16T20:58:49.588755+0000","last_peered":"2023-10-16T20:58:49.588755+0000","last_clean":"2023-10-16T20:58:49.588755+0000","last_became_active":"2023-10-16T20:36:55.290386+0000","last_became_peered":"2023-10-16T20:36:55.290386+0000","last_unstale":"2023-10-16T20:58:49.588755+0000","last_undegraded":"2023-10-16T20:58:49.588755+0000","last_fullsized":"2023-10-16T20:58:49.588755+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'94","last_scrub_stamp":"2023-10-16T20:58:49.588589+0000","last_deep_scrub":"67'94","last_deep_scrub_stamp":"2023-10-16T20:58:49.588589+0000","last_clean_scrub_stamp":"2023-10-16T20:58:49.588589+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:41:38.561038+0000","scrub_duration":0.0020095009999999999,"objects_trimmed":0,"snaptrim_duration":0.0093217969999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"67'114","reported_seq":2233,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.138708+0000","last_change":"2023-10-16T20:58:50.138746+0000","last_active":"2023-10-16T20:58:50.138708+0000","last_peered":"2023-10-16T20:58:50.138708+0000","last_clean":"2023-10-16T20:58:50.138708+0000","last_became_active":"2023-10-16T20:36:55.270787+0000","last_became_peered":"2023-10-16T20:36:55.270787+0000","last_unstale":"2023-10-16T20:58:50.138708+0000","last_undegraded":"2023-10-16T20:58:50.138708+0000","last_fullsized":"2023-10-16T20:58:50.138708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'114","last_scrub_stamp":"2023-10-16T20:58:50.138632+0000","last_deep_scrub":"67'114","last_deep_scrub_stamp":"2023-10-16T20:58:50.138632+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.138632+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T02:35:28.601665+0000","scrub_duration":0.0015404730000000001,"objects_trimmed":0,"snaptrim_duration":0.010801276,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"67'94","reported_seq":2213,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:58:50.047134+0000","last_change":"2023-10-16T20:58:50.047177+0000","last_active":"2023-10-16T20:58:50.047134+0000","last_peered":"2023-10-16T20:58:50.047134+0000","last_clean":"2023-10-16T20:58:50.047134+0000","last_became_active":"2023-10-16T20:36:55.271373+0000","last_became_peered":"2023-10-16T20:36:55.271373+0000","last_unstale":"2023-10-16T20:58:50.047134+0000","last_undegraded":"2023-10-16T20:58:50.047134+0000","last_fullsized":"2023-10-16T20:58:50.047134+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'94","last_scrub_stamp":"2023-10-16T20:58:50.047059+0000","last_deep_scrub":"67'94","last_deep_scrub_stamp":"2023-10-16T20:58:50.047059+0000","last_clean_scrub_stamp":"2023-10-16T20:58:50.047059+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-18T00:50:24.904268+0000","scrub_duration":0.0017358,"objects_trimmed":0,"snaptrim_duration":0.006091478,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"67'111","reported_seq":2230,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-16T20:59:26.976263+0000","last_change":"2023-10-16T20:59:26.976305+0000","last_active":"2023-10-16T20:59:26.976263+0000","last_peered":"2023-10-16T20:59:26.976263+0000","last_clean":"2023-10-16T20:59:26.976263+0000","last_became_active":"2023-10-16T20:36:55.291016+0000","last_became_peered":"2023-10-16T20:36:55.291016+0000","last_unstale":"2023-10-16T20:59:26.976263+0000","last_undegraded":"2023-10-16T20:59:26.976263+0000","last_fullsized":"2023-10-16T20:59:26.976263+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'111","last_scrub_stamp":"2023-10-16T20:59:26.976174+0000","last_deep_scrub":"67'111","last_deep_scrub_stamp":"2023-10-16T20:59:26.976174+0000","last_clean_scrub_stamp":"2023-10-16T20:59:26.976174+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-17T22:14:47.120578+0000","scrub_duration":0.001725011,"objects_trimmed":0,"snaptrim_duration":0.009370148,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":924319467,"num_objects":10294,"num_object_clones":0,"num_object_copies":41176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10294,"num_whiteouts":0,"num_read":30602,"num_read_kb":676969,"num_write":21002,"num_write_kb":1302825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966899200,"data_stored":1966899200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21050,"ondisk_log_size":21050,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6794,"ondisk_log_size":6794,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":179301891,"num_objects":198,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1063,"num_read_kb":205540,"num_write":6375,"num_write_kb":180866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":115,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":468586,"num_omap_keys":892,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":358858752,"data_stored":358603782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6286,"ondisk_log_size":6286,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509720,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396784,"kb_used_data":280936,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93975056,"statfs":{"total":96636764160,"available":96230457344,"internally_reserved":0,"allocated":287678464,"data_stored":287278454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":118618571},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.56399999999999995,"15min":0.628},"min":{"1min":0.49199999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.113,"5min":1.113,"15min":2.4489999999999998},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.58199999999999996,"15min":0.64900000000000002},"min":{"1min":0.40999999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":11.77},"last":0.82299999999999995}]},{"osd":1,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.59299999999999997,"15min":0.65300000000000002},"min":{"1min":0.38400000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":4.5279999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.61599999999999999,"15min":0.68500000000000005},"min":{"1min":0.39400000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.006,"5min":1.024,"15min":11.971},"last":0.85099999999999998}]},{"osd":2,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.63500000000000001,"15min":0.67400000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34300000000000003},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":7.1130000000000004},"last":0.87},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.65700000000000003,"15min":0.71299999999999997},"min":{"1min":0.44500000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":9.4450000000000003},"last":0.73899999999999999}]},{"osd":3,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.65700000000000003,"15min":0.68400000000000005},"min":{"1min":0.438,"5min":0.437,"15min":0.41299999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":3.363},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.69999999999999996,"15min":0.73099999999999998},"min":{"1min":0.57099999999999995,"5min":0.46899999999999997,"15min":0.42099999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":3.3980000000000001},"last":0.76000000000000001}]},{"osd":4,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55500000000000005,"15min":0.61399999999999999},"min":{"1min":0.32900000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81399999999999995,"5min":0.84099999999999997,"15min":12.532999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.42299999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.875,"5min":0.90600000000000003,"15min":3.8700000000000001},"last":0.47699999999999998}]},{"osd":5,"last update":"Mon Oct 16 20:59:31 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.60099999999999998,"15min":0.65700000000000003},"min":{"1min":0.41299999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.94999999999999996,"5min":1.016,"15min":2.2010000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.63600000000000001,"15min":0.73699999999999999},"min":{"1min":0.48899999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.92400000000000004,"5min":1.0369999999999999,"15min":12.353},"last":0.78000000000000003}]},{"osd":6,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.64500000000000002,"15min":0.748},"min":{"1min":0.44600000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90400000000000003,"5min":0.93799999999999994,"15min":11.929},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.67800000000000005,"15min":0.71899999999999997},"min":{"1min":0.47899999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.026,"5min":1.026,"15min":6.0359999999999996},"last":0.89300000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509720,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422268,"kb_used_data":300852,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93949572,"statfs":{"total":96636764160,"available":96204361728,"internally_reserved":0,"allocated":308072448,"data_stored":307684288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124320199},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:59:15 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.57699999999999996,"15min":0.61599999999999999},"min":{"1min":0.35099999999999998,"5min":0.27900000000000003,"15min":0.27800000000000002},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":4.5419999999999998},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.59099999999999997,"15min":0.625},"min":{"1min":0.36899999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.90200000000000002,"5min":2.0329999999999999,"15min":4.9699999999999998},"last":0.88400000000000001}]},{"osd":1,"last update":"Mon Oct 16 20:59:15 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.64600000000000002,"15min":0.64300000000000002},"min":{"1min":0.43099999999999999,"5min":0.39200000000000002,"15min":0.377},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":2.8610000000000002},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.41399999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.95299999999999996,"5min":1.014,"15min":1.0309999999999999},"last":0.99399999999999999}]},{"osd":2,"last update":"Mon Oct 16 20:59:15 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.61099999999999999,"15min":0.64500000000000002},"min":{"1min":0.40799999999999997,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.02,"5min":1.0289999999999999,"15min":1.895},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.68000000000000005,"15min":0.68700000000000006},"min":{"1min":0.48799999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":2.3340000000000001},"last":1.014}]},{"osd":3,"last update":"Mon Oct 16 20:59:15 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.77300000000000002,"15min":0.72799999999999998},"min":{"1min":0.60699999999999998,"5min":0.45500000000000002,"15min":0.42299999999999999},"max":{"1min":1.242,"5min":4.6340000000000003,"15min":4.6340000000000003},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.65200000000000002,"15min":0.69199999999999995},"min":{"1min":0.48399999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.96699999999999997,"5min":1.163,"15min":5.1029999999999998},"last":0.97699999999999998}]},{"osd":4,"last update":"Mon Oct 16 20:59:15 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.56999999999999995,"15min":0.59599999999999997},"min":{"1min":0.441,"5min":0.34799999999999998,"15min":0.30299999999999999},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":4.8959999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.63,"15min":0.63700000000000001},"min":{"1min":0.39300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":1.1000000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.752}]},{"osd":5,"last update":"Mon Oct 16 20:59:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.59899999999999998,"15min":0.625},"min":{"1min":0.374,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":2.3769999999999998},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.63900000000000001,"15min":0.66600000000000004},"min":{"1min":0.46000000000000002,"5min":0.38200000000000001,"15min":0.34499999999999997},"max":{"1min":1.196,"5min":1.196,"15min":4.9379999999999997},"last":0.96299999999999997}]},{"osd":7,"last update":"Mon Oct 16 20:59:15 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.70399999999999996,"15min":0.71399999999999997},"min":{"1min":0.435,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.147,"5min":1.425,"15min":4.5789999999999997},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.76500000000000001,"15min":0.751},"min":{"1min":0.47899999999999998,"5min":0.44600000000000001,"15min":0.39300000000000002},"max":{"1min":1.206,"5min":1.6120000000000001,"15min":5.2460000000000004},"last":1.0589999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509722,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325160,"kb_used_data":225952,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94046680,"statfs":{"total":96636764160,"available":96303800320,"internally_reserved":0,"allocated":231374848,"data_stored":230972642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":101579207},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.59399999999999997,"15min":0.61699999999999999},"min":{"1min":0.48399999999999999,"5min":0.28199999999999997,"15min":0.25700000000000001},"max":{"1min":0.89200000000000002,"5min":1.0089999999999999,"15min":2.4580000000000002},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.61699999999999999,"15min":0.65700000000000003},"min":{"1min":0.48299999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.83999999999999997,"5min":1.1799999999999999,"15min":7.181},"last":0.66300000000000003}]},{"osd":1,"last update":"Mon Oct 16 20:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.78700000000000003,"15min":0.69099999999999995},"min":{"1min":0.53100000000000003,"5min":0.33900000000000002,"15min":0.29899999999999999},"max":{"1min":0.84999999999999998,"5min":16.655000000000001,"15min":16.655000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.81000000000000005,"15min":0.71299999999999997},"min":{"1min":0.51900000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.88100000000000001,"5min":16.289000000000001,"15min":16.289000000000001},"last":0.63400000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.64900000000000002,"15min":0.67300000000000004},"min":{"1min":0.51800000000000002,"5min":0.34599999999999997,"15min":0.308},"max":{"1min":0.86699999999999999,"5min":1.7110000000000001,"15min":3.3799999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.67500000000000004,"15min":0.67800000000000005},"min":{"1min":0.55500000000000005,"5min":0.39800000000000002,"15min":0.38800000000000001},"max":{"1min":0.86899999999999999,"5min":1.583,"15min":4.5369999999999999},"last":0.81899999999999995}]},{"osd":3,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.67700000000000005,"15min":0.69999999999999996},"min":{"1min":0.495,"5min":0.39400000000000002,"15min":0.34899999999999998},"max":{"1min":0.91700000000000004,"5min":1.329,"15min":4.1070000000000002},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.68100000000000005,"15min":0.69299999999999995},"min":{"1min":0.52800000000000002,"5min":0.434,"15min":0.38},"max":{"1min":0.83099999999999996,"5min":1.1679999999999999,"15min":3.476},"last":0.68400000000000005}]},{"osd":4,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51600000000000001,"15min":0.55500000000000005},"min":{"1min":0.35299999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.873,"5min":0.873,"15min":2.415},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57299999999999995,"15min":0.65600000000000003},"min":{"1min":0.41999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81999999999999995,"5min":1.155,"15min":3.2919999999999998},"last":0.45300000000000001}]},{"osd":6,"last update":"Mon Oct 16 20:59:25 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.56799999999999995,"15min":0.59399999999999997},"min":{"1min":0.374,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.91200000000000003,"5min":0.93799999999999994,"15min":3.2400000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.40000000000000002,"5min":0.38500000000000001,"15min":0.31},"max":{"1min":0.80200000000000005,"5min":1.036,"15min":1.2310000000000001},"last":0.54800000000000004}]},{"osd":7,"last update":"Mon Oct 16 20:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.48999999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.89300000000000002,"5min":1.23,"15min":5.2759999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.68799999999999994,"15min":0.70899999999999996},"min":{"1min":0.50800000000000001,"5min":0.435,"15min":0.42299999999999999},"max":{"1min":0.91600000000000004,"5min":0.97699999999999998,"15min":3.3140000000000001},"last":0.73299999999999998}]}]},{"osd":4,"up_from":15,"seq":64424509720,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":420128,"kb_used_data":290392,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93951712,"statfs":{"total":96636764160,"available":96206553088,"internally_reserved":0,"allocated":297361408,"data_stored":296975417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:59:37 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.58699999999999997,"15min":0.61699999999999999},"min":{"1min":0.35199999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.524},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.63400000000000001,"15min":0.69299999999999995},"min":{"1min":0.42399999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":1.5649999999999999},"last":0.63400000000000001}]},{"osd":1,"last update":"Mon Oct 16 20:59:37 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.68100000000000005,"15min":0.67900000000000005},"min":{"1min":0.34999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.95599999999999996,"5min":4.1749999999999998,"15min":4.1749999999999998},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.629,"15min":0.66900000000000004},"min":{"1min":0.50600000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.1020000000000001},"last":0.73499999999999999}]},{"osd":2,"last update":"Mon Oct 16 20:59:37 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.68300000000000005,"15min":0.67100000000000004},"min":{"1min":0.52600000000000002,"5min":0.42499999999999999,"15min":0.38900000000000001},"max":{"1min":0.97699999999999998,"5min":1.1140000000000001,"15min":2.3370000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.69899999999999995,"15min":0.71499999999999997},"min":{"1min":0.48499999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.98999999999999999,"5min":1.085,"15min":2.3660000000000001},"last":0.76500000000000001}]},{"osd":3,"last update":"Mon Oct 16 20:59:37 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.66600000000000004,"15min":0.66100000000000003},"min":{"1min":0.42199999999999999,"5min":0.38500000000000001,"15min":0.377},"max":{"1min":1.0720000000000001,"5min":1.1439999999999999,"15min":2.04},"last":0.746},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.71399999999999997,"15min":0.71199999999999997},"min":{"1min":0.46400000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.042,"5min":1.2529999999999999,"15min":2.2240000000000002},"last":0.71899999999999997}]},{"osd":5,"last update":"Mon Oct 16 20:59:37 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57899999999999996,"15min":0.60199999999999998},"min":{"1min":0.40600000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.96299999999999997,"5min":1.085,"15min":1.9970000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.64200000000000002,"15min":0.61299999999999999},"min":{"1min":0.52500000000000002,"5min":0.40799999999999997,"15min":0.33900000000000002},"max":{"1min":1.113,"5min":1.206,"15min":1.206},"last":0.56599999999999995}]},{"osd":6,"last update":"Mon Oct 16 20:59:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61199999999999999,"15min":0.61799999999999999},"min":{"1min":0.40600000000000003,"5min":0.315,"15min":0.312},"max":{"1min":1.151,"5min":1.2270000000000001,"15min":2.2639999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.66300000000000003,"15min":0.67500000000000004},"min":{"1min":0.47899999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.056,"5min":1.1819999999999999,"15min":1.8300000000000001},"last":0.69299999999999995}]},{"osd":7,"last update":"Mon Oct 16 20:59:37 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.69599999999999995,"15min":0.67400000000000004},"min":{"1min":0.57199999999999995,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.272,"5min":1.2969999999999999,"15min":1.347},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.68000000000000005,"15min":0.68300000000000005},"min":{"1min":0.56499999999999995,"5min":0.35899999999999999,"15min":0.35599999999999998},"max":{"1min":1.123,"5min":1.266,"15min":2.2999999999999998},"last":0.59899999999999998}]}]},{"osd":3,"up_from":15,"seq":64424509720,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485596,"kb_used_data":360020,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93886244,"statfs":{"total":96636764160,"available":96139513856,"internally_reserved":0,"allocated":368660480,"data_stored":368273774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":128580040},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.442,"15min":0.434},"min":{"1min":0.27400000000000002,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":0.69099999999999995,"5min":1.371,"15min":4.5830000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.46600000000000003,"15min":0.44900000000000001},"min":{"1min":0.35399999999999998,"5min":0.27100000000000002,"15min":0.25700000000000001},"max":{"1min":0.69299999999999995,"5min":1.147,"15min":1.2509999999999999},"last":0.55100000000000005}]},{"osd":1,"last update":"Mon Oct 16 20:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.41199999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.82899999999999996,"5min":1.411,"15min":4.6840000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.51700000000000002,"15min":0.50700000000000001},"min":{"1min":0.36299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71599999999999997,"5min":2.6360000000000001,"15min":2.7429999999999999},"last":0.433}]},{"osd":2,"last update":"Mon Oct 16 20:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57399999999999995,"15min":0.55400000000000005},"min":{"1min":0.48199999999999998,"5min":0.34399999999999997,"15min":0.311},"max":{"1min":0.70999999999999996,"5min":1.286,"15min":4.6449999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.56499999999999995,"15min":0.52200000000000002},"min":{"1min":0.38200000000000001,"5min":0.32200000000000001,"15min":0.30399999999999999},"max":{"1min":0.80400000000000005,"5min":1.1850000000000001,"15min":1.911},"last":0.60899999999999999}]},{"osd":4,"last update":"Mon Oct 16 20:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.75,"15min":0.76400000000000001},"min":{"1min":0.60299999999999998,"5min":0.45400000000000001,"15min":0.44800000000000001},"max":{"1min":1.1240000000000001,"5min":1.538,"15min":1.7749999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.76200000000000001,"15min":0.745},"min":{"1min":0.58899999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.212,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":0.97399999999999998}]},{"osd":5,"last update":"Mon Oct 16 20:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.76800000000000002,"15min":0.75600000000000001},"min":{"1min":0.53000000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.1519999999999999,"5min":1.4970000000000001,"15min":4.8449999999999998},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.75700000000000001,"15min":0.74399999999999999},"min":{"1min":0.48899999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.161,"5min":2.948,"15min":2.948},"last":0.86399999999999999}]},{"osd":6,"last update":"Mon Oct 16 20:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78200000000000003,"15min":0.76500000000000001},"min":{"1min":0.60299999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.044,"5min":1.633,"15min":4.8099999999999996},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.79500000000000004,"15min":0.76900000000000002},"min":{"1min":0.57999999999999996,"5min":0.46400000000000002,"15min":0.45400000000000001},"max":{"1min":1.0680000000000001,"5min":1.6559999999999999,"15min":2.339},"last":0.93400000000000005}]},{"osd":7,"last update":"Mon Oct 16 20:59:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.81799999999999995,"15min":0.86899999999999999},"min":{"1min":0.56899999999999995,"5min":0.5,"15min":0.46000000000000002},"max":{"1min":1.179,"5min":1.593,"15min":34.039000000000001},"last":0.749},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.79400000000000004,"15min":0.77500000000000002},"min":{"1min":0.58999999999999997,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.198,"5min":2.984,"15min":2.984},"last":0.90600000000000003}]}]},{"osd":2,"up_from":15,"seq":64424509721,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303812,"kb_used_data":200444,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94068028,"statfs":{"total":96636764160,"available":96325660672,"internally_reserved":0,"allocated":205254656,"data_stored":204864813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.436,"15min":0.45100000000000001},"min":{"1min":0.26800000000000002,"5min":0.23400000000000001,"15min":0.20599999999999999},"max":{"1min":0.623,"5min":0.92500000000000004,"15min":5.5499999999999998},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.51400000000000001,"15min":0.52700000000000002},"min":{"1min":0.32100000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.73999999999999999,"5min":1.1220000000000001,"15min":5.4160000000000004},"last":0.60899999999999999}]},{"osd":1,"last update":"Mon Oct 16 20:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.498,"15min":0.501},"min":{"1min":0.33700000000000002,"5min":0.307,"15min":0.26000000000000001},"max":{"1min":0.622,"5min":1.25,"15min":2.2349999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.51800000000000002,"15min":0.502},"min":{"1min":0.34699999999999998,"5min":0.28399999999999997,"15min":0.27200000000000002},"max":{"1min":0.64800000000000002,"5min":1.2849999999999999,"15min":5.4359999999999999},"last":0.68899999999999995}]},{"osd":3,"last update":"Mon Oct 16 20:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.51500000000000001,"15min":0.48999999999999999},"min":{"1min":0.317,"5min":0.26400000000000001,"15min":0.25800000000000001},"max":{"1min":0.65200000000000002,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56000000000000005,"15min":0.56699999999999995},"min":{"1min":0.40799999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.82899999999999996,"5min":1.286,"15min":5.3789999999999996},"last":0.745}]},{"osd":4,"last update":"Mon Oct 16 20:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.73699999999999999,"15min":0.80300000000000005},"min":{"1min":0.55200000000000005,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.006,"5min":3.464,"15min":16.905999999999999},"last":0.878},{"interface":"front","average":{"1min":0.745,"5min":0.70699999999999996,"15min":0.77800000000000002},"min":{"1min":0.51500000000000001,"5min":0.44700000000000001,"15min":0.441},"max":{"1min":0.89600000000000002,"5min":1.3100000000000001,"15min":10.064},"last":0.91300000000000003}]},{"osd":5,"last update":"Mon Oct 16 20:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.71999999999999997,"15min":0.73099999999999998},"min":{"1min":0.51600000000000001,"5min":0.41099999999999998,"15min":0.32800000000000001},"max":{"1min":0.91700000000000004,"5min":1.304,"15min":3.2989999999999999},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.74199999999999999,"15min":0.77400000000000002},"min":{"1min":0.55700000000000005,"5min":0.499,"15min":0.46700000000000003},"max":{"1min":0.96999999999999997,"5min":1.3640000000000001,"15min":5.3330000000000002},"last":1}]},{"osd":6,"last update":"Mon Oct 16 20:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.77400000000000002,"15min":0.75800000000000001},"min":{"1min":0.501,"5min":0.438,"15min":0.42299999999999999},"max":{"1min":1.0509999999999999,"5min":3.5710000000000002,"15min":3.5710000000000002},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79000000000000004,"15min":0.77500000000000002},"min":{"1min":0.53300000000000003,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.97299999999999998,"5min":3.7639999999999998,"15min":5.4749999999999996},"last":0.93500000000000005}]},{"osd":7,"last update":"Mon Oct 16 20:59:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.77800000000000002,"15min":0.78400000000000003},"min":{"1min":0.503,"5min":0.47199999999999998,"15min":0.44500000000000001},"max":{"1min":0.996,"5min":3.5089999999999999,"15min":5.516},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.77500000000000002,"15min":0.79200000000000004},"min":{"1min":0.58499999999999996,"5min":0.46600000000000003,"15min":0.41899999999999998},"max":{"1min":0.90300000000000002,"5min":1.325,"15min":8.4610000000000003},"last":0.88800000000000001}]}]},{"osd":1,"up_from":15,"seq":64424509720,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490524,"kb_used_data":360788,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93881316,"statfs":{"total":96636764160,"available":96134467584,"internally_reserved":0,"allocated":369446912,"data_stored":369062219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 16 20:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44700000000000001,"15min":0.45800000000000002},"min":{"1min":0.27200000000000002,"5min":0.219,"15min":0.21299999999999999},"max":{"1min":0.74199999999999999,"5min":0.93000000000000005,"15min":8.9339999999999993},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49299999999999999,"15min":0.47699999999999998},"min":{"1min":0.35499999999999998,"5min":0.34100000000000003,"15min":0.23000000000000001},"max":{"1min":0.67100000000000004,"5min":1.6899999999999999,"15min":3.8809999999999998},"last":0.70199999999999996}]},{"osd":2,"last update":"Mon Oct 16 20:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.52400000000000002,"15min":0.46000000000000002},"min":{"1min":0.28599999999999998,"5min":0.27500000000000002,"15min":0.248},"max":{"1min":0.66600000000000004,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.69699999999999995,"5min":2.6739999999999999,"15min":9.2680000000000007},"last":0.68200000000000005}]},{"osd":3,"last update":"Mon Oct 16 20:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.59099999999999997,"15min":0.54000000000000004},"min":{"1min":0.39600000000000002,"5min":0.33000000000000002,"15min":0.315},"max":{"1min":0.76800000000000002,"5min":1.9910000000000001,"15min":3.0539999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58799999999999997,"15min":0.57699999999999996},"min":{"1min":0.40699999999999997,"5min":0.34200000000000003,"15min":0.313},"max":{"1min":0.73299999999999998,"5min":1.4119999999999999,"15min":6.9409999999999998},"last":0.61699999999999999}]},{"osd":4,"last update":"Mon Oct 16 20:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68600000000000005,"15min":0.72699999999999998},"min":{"1min":0.47799999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.90800000000000003,"5min":1.962,"15min":5.6630000000000003},"last":1.002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72999999999999998,"15min":0.75800000000000001},"min":{"1min":0.47499999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.88700000000000001,"5min":1.6719999999999999,"15min":8.8960000000000008},"last":0.89400000000000002}]},{"osd":5,"last update":"Mon Oct 16 20:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76200000000000001,"15min":0.77500000000000002},"min":{"1min":0.623,"5min":0.47299999999999998,"15min":0.40300000000000002},"max":{"1min":1.0149999999999999,"5min":1.242,"15min":6.6159999999999997},"last":1.0900000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.77300000000000002,"15min":0.79100000000000004},"min":{"1min":0.54900000000000004,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.999,"5min":1.7130000000000001,"15min":5.7009999999999996},"last":1.0549999999999999}]},{"osd":6,"last update":"Mon Oct 16 20:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.73899999999999999,"15min":0.77300000000000002},"min":{"1min":0.56799999999999995,"5min":0.47299999999999998,"15min":0.39800000000000002},"max":{"1min":1.075,"5min":2.0529999999999999,"15min":8.6039999999999992},"last":1.073},{"interface":"front","average":{"1min":0.746,"5min":0.77600000000000002,"15min":0.749},"min":{"1min":0.51100000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.0369999999999999,"5min":2.1019999999999999,"15min":2.1019999999999999},"last":1.038}]},{"osd":7,"last update":"Mon Oct 16 20:59:30 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.76800000000000002,"15min":0.77500000000000002},"min":{"1min":0.52500000000000002,"5min":0.46899999999999997,"15min":0.44800000000000001},"max":{"1min":1.0580000000000001,"5min":1.7470000000000001,"15min":6.6600000000000001},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.79100000000000004,"15min":0.80400000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.46200000000000002},"max":{"1min":0.998,"5min":1.774,"15min":8.8100000000000005},"last":1.1220000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575129,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371112,"kb_used_data":259424,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94000728,"statfs":{"total":96636764160,"available":96256745472,"internally_reserved":0,"allocated":265650176,"data_stored":265251991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 16 20:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":0.64700000000000002,"15min":0.51100000000000001},"min":{"1min":0.28599999999999998,"5min":0.221,"15min":0.221},"max":{"1min":24.483000000000001,"5min":24.483000000000001,"15min":24.483000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":4.8319999999999999,"5min":1.3440000000000001,"15min":0.76600000000000001},"min":{"1min":0.32000000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":91.462000000000003,"5min":91.462000000000003,"15min":91.462000000000003},"last":0.39100000000000001}]},{"osd":2,"last update":"Mon Oct 16 20:59:32 2023","interfaces":[{"interface":"back","average":{"1min":4.8170000000000002,"5min":1.387,"15min":0.78500000000000003},"min":{"1min":0.33300000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":91.384,"5min":91.384,"15min":91.384},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.61799999999999999,"15min":0.504},"min":{"1min":0.35999999999999999,"5min":0.251,"15min":0.251},"max":{"1min":11.782,"5min":11.782,"15min":11.782},"last":0.56699999999999995}]},{"osd":3,"last update":"Mon Oct 16 20:59:32 2023","interfaces":[{"interface":"back","average":{"1min":4.923,"5min":1.4550000000000001,"15min":0.81699999999999995},"min":{"1min":0.41899999999999998,"5min":0.34300000000000003,"15min":0.28399999999999997},"max":{"1min":91.557000000000002,"5min":91.557000000000002,"15min":91.557000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.056,"5min":0.65900000000000003,"15min":0.57999999999999996},"min":{"1min":0.375,"5min":0.33500000000000002,"15min":0.317},"max":{"1min":11.835000000000001,"5min":11.835000000000001,"15min":11.835000000000001},"last":0.54600000000000004}]},{"osd":4,"last update":"Mon Oct 16 20:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":0.81499999999999995,"15min":0.84099999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":13.169,"5min":13.169,"15min":18.41},"last":0.74199999999999999},{"interface":"front","average":{"1min":5.0880000000000001,"5min":1.579,"15min":1.107},"min":{"1min":0.499,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":91.421999999999997,"5min":91.421999999999997,"15min":91.421999999999997},"last":0.88800000000000001}]},{"osd":5,"last update":"Mon Oct 16 20:59:32 2023","interfaces":[{"interface":"back","average":{"1min":4.5,"5min":1.49,"15min":1.014},"min":{"1min":0.52800000000000002,"5min":0.50900000000000001,"15min":0.42799999999999999},"max":{"1min":79.207999999999998,"5min":79.207999999999998,"15min":79.207999999999998},"last":0.71299999999999997},{"interface":"front","average":{"1min":5.1159999999999997,"5min":1.5920000000000001,"15min":1.0269999999999999},"min":{"1min":0.52100000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":91.587999999999994,"5min":91.587999999999994,"15min":91.587999999999994},"last":0.72199999999999998}]},{"osd":6,"last update":"Mon Oct 16 20:59:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":0.81200000000000006,"15min":0.78000000000000003},"min":{"1min":0.49299999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":9.8290000000000006,"5min":9.8290000000000006,"15min":9.8290000000000006},"last":0.83399999999999996},{"interface":"front","average":{"1min":5.1550000000000002,"5min":1.6279999999999999,"15min":1.073},"min":{"1min":0.53600000000000003,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":91.522999999999996,"5min":91.522999999999996,"15min":91.522999999999996},"last":0.85599999999999998}]},{"osd":7,"last update":"Mon Oct 16 20:59:32 2023","interfaces":[{"interface":"back","average":{"1min":5.1200000000000001,"5min":1.6439999999999999,"15min":1.0680000000000001},"min":{"1min":0.56000000000000005,"5min":0.502,"15min":0.49099999999999999},"max":{"1min":91.494,"5min":91.494,"15min":91.494},"last":0.81799999999999995},{"interface":"front","average":{"1min":5.1660000000000004,"5min":1.6359999999999999,"15min":1.0600000000000001},"min":{"1min":0.60499999999999998,"5min":0.54500000000000004,"15min":0.45500000000000002},"max":{"1min":91.346000000000004,"5min":91.346000000000004,"15min":91.346000000000004},"last":0.78300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39346176,"data_stored":39310758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51638272,"data_stored":51612794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18219008,"data_stored":18188380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":47730688,"data_stored":47703197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56045568,"data_stored":56018792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":44777472,"data_stored":44734481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40857600,"data_stored":40828655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":60243968,"data_stored":60206725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":224890880,"data_stored":224890880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":316989440,"data_stored":316989440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186216448,"data_stored":186216448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":320110592,"data_stored":320110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":240496640,"data_stored":240496640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":185778176,"data_stored":185778176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266395648,"data_stored":266395648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":226021376,"data_stored":226021376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-16T20:59:52.136 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-16T20:59:52.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.300+0000 7f75e73d0640 1 Processor -- start 2023-10-16T20:59:52.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.300+0000 7f75e73d0640 1 -- start start 2023-10-16T20:59:52.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.300+0000 7f75e73d0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f75e0157b20 0x7f75e0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:52.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.300+0000 7f75e73d0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75e006ae00 0x7f75e00c20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:52.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.300+0000 7f75e73d0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e015dba0 con 0x7f75e0157f00 2023-10-16T20:59:52.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.300+0000 7f75e73d0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75e015d9b0 con 0x7f75e0157b20 2023-10-16T20:59:52.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.300+0000 7f75e73d0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e0072a10 con 0x7f75e006ae00 2023-10-16T20:59:52.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.304+0000 7f75e516c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f75e0157b20 0x7f75e0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:52.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.304+0000 7f75e516c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f75e0157b20 0x7f75e0069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57576/0 (socket says 172.21.15.88:57576) 2023-10-16T20:59:52.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.304+0000 7f75e516c640 1 -- 172.21.15.88:0/3647774391 learned_addr learned my addr 172.21.15.88:0/3647774391 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:52.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.304+0000 7f75e496b640 1 --2- 172.21.15.88:0/3647774391 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75e006ae00 0x7f75e00c20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:52.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.304+0000 7f75e516c640 1 -- 172.21.15.88:0/3647774391 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75e006ae00 msgr2=0x7f75e00c20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:59:52.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.304+0000 7f75e516c640 1 --2- 172.21.15.88:0/3647774391 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75e006ae00 0x7f75e00c20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.304+0000 7f75e516c640 1 -- 172.21.15.88:0/3647774391 >> v1:172.21.15.88:6789/0 conn(0x7f75e0157f00 legacy=0x7f75e00c5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:52.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.304+0000 7f75e516c640 1 -- 172.21.15.88:0/3647774391 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e015de80 con 0x7f75e0157b20 2023-10-16T20:59:52.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.304+0000 7f75e516c640 1 --2- 172.21.15.88:0/3647774391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75e0157b20 0x7f75e0069e90 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f75dc0098a0 tx=0x7f75dc01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aa899181b153e96 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:52.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.304+0000 7f75d7fff640 1 -- 172.21.15.88:0/3647774391 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc0a8a10 con 0x7f75e0157b20 2023-10-16T20:59:52.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75d7fff640 1 -- 172.21.15.88:0/3647774391 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75dc0a8b70 con 0x7f75e0157b20 2023-10-16T20:59:52.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75d7fff640 1 -- 172.21.15.88:0/3647774391 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc0b0440 con 0x7f75e0157b20 2023-10-16T20:59:52.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e73d0640 1 -- 172.21.15.88:0/3647774391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75e0157b20 msgr2=0x7f75e0069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:52.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e73d0640 1 --2- 172.21.15.88:0/3647774391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75e0157b20 0x7f75e0069e90 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f75dc0098a0 tx=0x7f75dc01ed10 comp rx=0 tx=0).stop 2023-10-16T20:59:52.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e73d0640 1 -- 172.21.15.88:0/3647774391 shutdown_connections 2023-10-16T20:59:52.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e73d0640 1 --2- 172.21.15.88:0/3647774391 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75e006ae00 0x7f75e00c20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e73d0640 1 --2- 172.21.15.88:0/3647774391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75e0157b20 0x7f75e0069e90 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e73d0640 1 -- 172.21.15.88:0/3647774391 >> 172.21.15.88:0/3647774391 conn(0x7f75e00ca9f0 msgr2=0x7f75e00cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:52.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e73d0640 1 -- 172.21.15.88:0/3647774391 shutdown_connections 2023-10-16T20:59:52.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e73d0640 1 -- 172.21.15.88:0/3647774391 wait complete. 2023-10-16T20:59:52.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e73d0640 1 Processor -- start 2023-10-16T20:59:52.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e73d0640 1 -- start start 2023-10-16T20:59:52.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e73d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75e006ae00 0x7f75e00c91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:52.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e73d0640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75e0157b20 0x7f75e0158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:52.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e73d0640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75e0157f00 0x7f75e01f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:52.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e73d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e01f2ee0 con 0x7f75e006ae00 2023-10-16T20:59:52.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e73d0640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e01f31c0 con 0x7f75e0157b20 2023-10-16T20:59:52.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e73d0640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e01f34a0 con 0x7f75e0157f00 2023-10-16T20:59:52.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e516c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75e006ae00 0x7f75e00c91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:52.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e516c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75e006ae00 0x7f75e00c91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57586/0 (socket says 172.21.15.88:57586) 2023-10-16T20:59:52.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e516c640 1 -- 172.21.15.88:0/1539979548 learned_addr learned my addr 172.21.15.88:0/1539979548 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:52.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.308+0000 7f75e596d640 1 --2- 172.21.15.88:0/1539979548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75e0157f00 0x7f75e01f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:52.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75e496b640 1 --2- 172.21.15.88:0/1539979548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75e0157b20 0x7f75e0158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:52.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75e516c640 1 -- 172.21.15.88:0/1539979548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75e0157f00 msgr2=0x7f75e01f66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:52.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75e516c640 1 --2- 172.21.15.88:0/1539979548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75e0157f00 0x7f75e01f66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75e516c640 1 -- 172.21.15.88:0/1539979548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75e0157b20 msgr2=0x7f75e0158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:52.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75e516c640 1 --2- 172.21.15.88:0/1539979548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75e0157b20 0x7f75e0158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75e516c640 1 -- 172.21.15.88:0/1539979548 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc0094d0 con 0x7f75e006ae00 2023-10-16T20:59:52.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75e496b640 1 --2- 172.21.15.88:0/1539979548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75e0157b20 0x7f75e0158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:59:52.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75e596d640 1 --2- 172.21.15.88:0/1539979548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75e0157f00 0x7f75e01f66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:59:52.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75e516c640 1 --2- 172.21.15.88:0/1539979548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75e006ae00 0x7f75e00c91f0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f75dc0b3040 tx=0x7f75dc01bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:52.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75d5ffb640 1 -- 172.21.15.88:0/1539979548 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc0be070 con 0x7f75e006ae00 2023-10-16T20:59:52.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75d5ffb640 1 -- 172.21.15.88:0/1539979548 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75dc0ba400 con 0x7f75e006ae00 2023-10-16T20:59:52.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75d5ffb640 1 -- 172.21.15.88:0/1539979548 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc0ba7d0 con 0x7f75e006ae00 2023-10-16T20:59:52.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75e73d0640 1 -- 172.21.15.88:0/1539979548 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e01f3780 con 0x7f75e006ae00 2023-10-16T20:59:52.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75e73d0640 1 -- 172.21.15.88:0/1539979548 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e01f61c0 con 0x7f75e006ae00 2023-10-16T20:59:52.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75e73d0640 1 -- 172.21.15.88:0/1539979548 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e0074700 con 0x7f75e006ae00 2023-10-16T20:59:52.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75d5ffb640 1 -- 172.21.15.88:0/1539979548 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f75dc0c3410 con 0x7f75e006ae00 2023-10-16T20:59:52.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75d5ffb640 1 --2- 172.21.15.88:0/1539979548 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f75c40664a0 0x7f75c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:52.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75d5ffb640 1 -- 172.21.15.88:0/1539979548 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f75dc1221d0 con 0x7f75e006ae00 2023-10-16T20:59:52.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.312+0000 7f75e496b640 1 --2- 172.21.15.88:0/1539979548 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f75c40664a0 0x7f75c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:52.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.316+0000 7f75e496b640 1 --2- 172.21.15.88:0/1539979548 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f75c40664a0 0x7f75c4068960 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f75e01f07e0 tx=0x7f75d0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:52.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.320+0000 7f75d5ffb640 1 -- 172.21.15.88:0/1539979548 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f75dc0eef10 con 0x7f75e006ae00 2023-10-16T20:59:52.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.468+0000 7f75e73d0640 1 -- 172.21.15.88:0/1539979548 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f75e00c9fe0 con 0x7f75e006ae00 2023-10-16T20:59:52.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.472+0000 7f75d5ffb640 1 -- 172.21.15.88:0/1539979548 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) v1 ==== 123+0+0 (secure 0 0 0) 0x7f75dc0f2530 con 0x7f75e006ae00 2023-10-16T20:59:52.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.476+0000 7f75d5ffb640 1 -- 172.21.15.88:0/1539979548 <== mon.0 v2:172.21.15.88:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f75dc0eef10 con 0x7f75e006ae00 2023-10-16T20:59:52.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.476+0000 7f75e73d0640 1 -- 172.21.15.88:0/1539979548 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f75c40664a0 msgr2=0x7f75c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:52.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.476+0000 7f75e73d0640 1 --2- 172.21.15.88:0/1539979548 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f75c40664a0 0x7f75c4068960 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f75e01f07e0 tx=0x7f75d0008040 comp rx=0 tx=0).stop 2023-10-16T20:59:52.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.476+0000 7f75e73d0640 1 -- 172.21.15.88:0/1539979548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75e006ae00 msgr2=0x7f75e00c91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:52.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.476+0000 7f75e73d0640 1 --2- 172.21.15.88:0/1539979548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75e006ae00 0x7f75e00c91f0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f75dc0b3040 tx=0x7f75dc01bab0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.476+0000 7f75e73d0640 1 -- 172.21.15.88:0/1539979548 shutdown_connections 2023-10-16T20:59:52.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.476+0000 7f75e73d0640 1 --2- 172.21.15.88:0/1539979548 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f75c40664a0 0x7f75c4068960 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.476+0000 7f75e73d0640 1 --2- 172.21.15.88:0/1539979548 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f75e0157f00 0x7f75e01f66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.476+0000 7f75e73d0640 1 --2- 172.21.15.88:0/1539979548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75e0157b20 0x7f75e0158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.476+0000 7f75e73d0640 1 --2- 172.21.15.88:0/1539979548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75e006ae00 0x7f75e00c91f0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.476+0000 7f75e73d0640 1 -- 172.21.15.88:0/1539979548 >> 172.21.15.88:0/1539979548 conn(0x7f75e00ca9f0 msgr2=0x7f75e00cadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:52.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.476+0000 7f75e73d0640 1 -- 172.21.15.88:0/1539979548 shutdown_connections 2023-10-16T20:59:52.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.476+0000 7f75e73d0640 1 -- 172.21.15.88:0/1539979548 wait complete. 2023-10-16T20:59:52.496 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-16T20:59:52.497 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:59:52.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.636+0000 7f7a8eb65640 1 Processor -- start 2023-10-16T20:59:52.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.636+0000 7f7a8eb65640 1 -- start start 2023-10-16T20:59:52.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.636+0000 7f7a8eb65640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7a880c9bb0 0x7f7a880c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:52.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.636+0000 7f7a8eb65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a880c41f0 0x7f7a880cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:52.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.636+0000 7f7a8eb65640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a88153b50 con 0x7f7a88153770 2023-10-16T20:59:52.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.636+0000 7f7a8eb65640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a880cc250 con 0x7f7a880c9bb0 2023-10-16T20:59:52.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.636+0000 7f7a8eb65640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a880cc670 con 0x7f7a880c41f0 2023-10-16T20:59:52.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.636+0000 7f7a7ffff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7a880c9bb0 0x7f7a880c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:52.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.636+0000 7f7a8d102640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a880c41f0 0x7f7a880cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:52.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.636+0000 7f7a7ffff640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f7a880c9bb0 0x7f7a880c3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:58678/0 (socket says 172.21.15.88:58678) 2023-10-16T20:59:52.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.636+0000 7f7a7ffff640 1 -- 172.21.15.88:0/2746363010 learned_addr learned my addr 172.21.15.88:0/2746363010 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:52.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.636+0000 7f7a8d102640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a880c41f0 0x7f7a880cece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:32916/0 (socket says 172.21.15.88:32916) 2023-10-16T20:59:52.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.636+0000 7f7a7ffff640 1 -- 172.21.15.88:0/2746363010 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a880c41f0 msgr2=0x7f7a880cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:59:52.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.636+0000 7f7a7ffff640 1 --2- 172.21.15.88:0/2746363010 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a880c41f0 0x7f7a880cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.636+0000 7f7a7ffff640 1 -- 172.21.15.88:0/2746363010 >> v1:172.21.15.191:6789/0 conn(0x7f7a88153770 legacy=0x7f7a88159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:52.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.636+0000 7f7a7ffff640 1 -- 172.21.15.88:0/2746363010 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a880cb2f0 con 0x7f7a880c9bb0 2023-10-16T20:59:52.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.636+0000 7f7a8d102640 1 --2- 172.21.15.88:0/2746363010 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a880c41f0 0x7f7a880cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:59:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.640+0000 7f7a7ffff640 1 --2- 172.21.15.88:0/2746363010 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a880c9bb0 0x7f7a880c3cb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7a78009870 tx=0x7f7a7801c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c324d419413172e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.640+0000 7f7a7f7fe640 1 -- 172.21.15.88:0/2746363010 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a780ab650 con 0x7f7a880c9bb0 2023-10-16T20:59:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.640+0000 7f7a7f7fe640 1 -- 172.21.15.88:0/2746363010 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f7a780ab7b0 con 0x7f7a880c9bb0 2023-10-16T20:59:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.640+0000 7f7a7f7fe640 1 -- 172.21.15.88:0/2746363010 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a780abb60 con 0x7f7a880c9bb0 2023-10-16T20:59:52.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.640+0000 7f7a8eb65640 1 -- 172.21.15.88:0/2746363010 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a880c9bb0 msgr2=0x7f7a880c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:52.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.640+0000 7f7a8eb65640 1 --2- 172.21.15.88:0/2746363010 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a880c9bb0 0x7f7a880c3cb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7a78009870 tx=0x7f7a7801c540 comp rx=0 tx=0).stop 2023-10-16T20:59:52.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.640+0000 7f7a8eb65640 1 -- 172.21.15.88:0/2746363010 shutdown_connections 2023-10-16T20:59:52.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.640+0000 7f7a8eb65640 1 --2- 172.21.15.88:0/2746363010 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a880c41f0 0x7f7a880cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.640+0000 7f7a8eb65640 1 --2- 172.21.15.88:0/2746363010 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a880c9bb0 0x7f7a880c3cb0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.640+0000 7f7a8eb65640 1 -- 172.21.15.88:0/2746363010 >> 172.21.15.88:0/2746363010 conn(0x7f7a880715c0 msgr2=0x7f7a880719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:52.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.640+0000 7f7a8eb65640 1 -- 172.21.15.88:0/2746363010 shutdown_connections 2023-10-16T20:59:52.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.640+0000 7f7a8eb65640 1 -- 172.21.15.88:0/2746363010 wait complete. 2023-10-16T20:59:52.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8eb65640 1 Processor -- start 2023-10-16T20:59:52.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8eb65640 1 -- start start 2023-10-16T20:59:52.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8eb65640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a880c41f0 0x7f7a880c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:52.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8eb65640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a880c9bb0 0x7f7a881eb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:52.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8eb65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a88153770 0x7f7a880c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:52.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8eb65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8815b120 con 0x7f7a88153770 2023-10-16T20:59:52.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8eb65640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a880c8050 con 0x7f7a880c9bb0 2023-10-16T20:59:52.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8eb65640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a880b4db0 con 0x7f7a880c41f0 2023-10-16T20:59:52.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8d102640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a88153770 0x7f7a880c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:52.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8d102640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a88153770 0x7f7a880c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57592/0 (socket says 172.21.15.88:57592) 2023-10-16T20:59:52.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8d102640 1 -- 172.21.15.88:0/3653925021 learned_addr learned my addr 172.21.15.88:0/3653925021 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:52.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a7ffff640 1 --2- 172.21.15.88:0/3653925021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a880c9bb0 0x7f7a881eb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:52.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8c901640 1 --2- 172.21.15.88:0/3653925021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a880c41f0 0x7f7a880c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:52.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8d102640 1 -- 172.21.15.88:0/3653925021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a880c41f0 msgr2=0x7f7a880c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:52.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8d102640 1 --2- 172.21.15.88:0/3653925021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a880c41f0 0x7f7a880c7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8d102640 1 -- 172.21.15.88:0/3653925021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a880c9bb0 msgr2=0x7f7a881eb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:52.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8d102640 1 --2- 172.21.15.88:0/3653925021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a880c9bb0 0x7f7a881eb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8d102640 1 -- 172.21.15.88:0/3653925021 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a80009a70 con 0x7f7a88153770 2023-10-16T20:59:52.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8d102640 1 --2- 172.21.15.88:0/3653925021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a88153770 0x7f7a880c8760 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009f00 tx=0x7f7a8000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:52.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a7d7fa640 1 -- 172.21.15.88:0/3653925021 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80009d70 con 0x7f7a88153770 2023-10-16T20:59:52.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a7d7fa640 1 -- 172.21.15.88:0/3653925021 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f7a8000f040 con 0x7f7a88153770 2023-10-16T20:59:52.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a7d7fa640 1 -- 172.21.15.88:0/3653925021 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80003be0 con 0x7f7a88153770 2023-10-16T20:59:52.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8eb65640 1 -- 172.21.15.88:0/3653925021 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a780094d0 con 0x7f7a88153770 2023-10-16T20:59:52.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.644+0000 7f7a8eb65640 1 -- 172.21.15.88:0/3653925021 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a880c82d0 con 0x7f7a88153770 2023-10-16T20:59:52.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.648+0000 7f7a8eb65640 1 -- 172.21.15.88:0/3653925021 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a880c8460 con 0x7f7a88153770 2023-10-16T20:59:52.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.648+0000 7f7a7d7fa640 1 -- 172.21.15.88:0/3653925021 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7a80014480 con 0x7f7a88153770 2023-10-16T20:59:52.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.648+0000 7f7a7d7fa640 1 --2- 172.21.15.88:0/3653925021 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7a740664d0 0x7f7a74068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:52.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.648+0000 7f7a7d7fa640 1 -- 172.21.15.88:0/3653925021 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f7a80085260 con 0x7f7a88153770 2023-10-16T20:59:52.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.648+0000 7f7a8c901640 1 --2- 172.21.15.88:0/3653925021 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7a740664d0 0x7f7a74068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:52.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.648+0000 7f7a8c901640 1 --2- 172.21.15.88:0/3653925021 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7a740664d0 0x7f7a74068990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7a84000fc0 tx=0x7f7a84009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:52.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.652+0000 7f7a7d7fa640 1 -- 172.21.15.88:0/3653925021 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7a80051eb0 con 0x7f7a88153770 2023-10-16T20:59:52.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.836+0000 7f7a8eb65640 1 -- 172.21.15.88:0/3653925021 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a881ebe30 con 0x7f7a88153770 2023-10-16T20:59:52.837 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.836+0000 7f7a7d7fa640 1 -- 172.21.15.88:0/3653925021 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+6581 (secure 0 0 0) 0x7f7a80053de0 con 0x7f7a88153770 2023-10-16T20:59:52.838 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:59:52.838 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6286,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2135868641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2135868641},{"type":"v1","addr":"172.21.15.191:6835","nonce":2135868641}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:58:22.748501+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"c","rank":4,"incarnation":112,"state":"up:active","state_seq":23,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:59:52.839 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 117 2023-10-16T20:59:52.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.840+0000 7f7a8eb65640 1 -- 172.21.15.88:0/3653925021 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7a740664d0 msgr2=0x7f7a74068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:52.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.840+0000 7f7a8eb65640 1 --2- 172.21.15.88:0/3653925021 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7a740664d0 0x7f7a74068990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7a84000fc0 tx=0x7f7a84009040 comp rx=0 tx=0).stop 2023-10-16T20:59:52.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.840+0000 7f7a8eb65640 1 -- 172.21.15.88:0/3653925021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a88153770 msgr2=0x7f7a880c8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:52.840 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.840+0000 7f7a8eb65640 1 --2- 172.21.15.88:0/3653925021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a88153770 0x7f7a880c8760 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009f00 tx=0x7f7a8000c490 comp rx=0 tx=0).stop 2023-10-16T20:59:52.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.840+0000 7f7a8eb65640 1 -- 172.21.15.88:0/3653925021 shutdown_connections 2023-10-16T20:59:52.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.840+0000 7f7a8eb65640 1 --2- 172.21.15.88:0/3653925021 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f7a740664d0 0x7f7a74068990 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.840+0000 7f7a8eb65640 1 --2- 172.21.15.88:0/3653925021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a88153770 0x7f7a880c8760 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.840+0000 7f7a8eb65640 1 --2- 172.21.15.88:0/3653925021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7a880c9bb0 0x7f7a881eb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.840+0000 7f7a8eb65640 1 --2- 172.21.15.88:0/3653925021 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f7a880c41f0 0x7f7a880c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:52.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.840+0000 7f7a8eb65640 1 -- 172.21.15.88:0/3653925021 >> 172.21.15.88:0/3653925021 conn(0x7f7a880715c0 msgr2=0x7f7a880719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:52.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.840+0000 7f7a8eb65640 1 -- 172.21.15.88:0/3653925021 shutdown_connections 2023-10-16T20:59:52.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.840+0000 7f7a8eb65640 1 -- 172.21.15.88:0/3653925021 wait complete. 2023-10-16T20:59:52.859 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-16T20:59:52.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.996+0000 7f965272a640 1 Processor -- start 2023-10-16T20:59:52.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.996+0000 7f965272a640 1 -- start start 2023-10-16T20:59:52.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.996+0000 7f965272a640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f964c14fd10 0x7f964c070c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:52.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.996+0000 7f965272a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f964c06f710 0x7f964c06fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:52.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.996+0000 7f965272a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964c156430 con 0x7f964c071170 2023-10-16T20:59:52.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.996+0000 7f965272a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f964c070500 con 0x7f964c06f710 2023-10-16T20:59:52.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.996+0000 7f965272a640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f964c155b00 con 0x7f964c14fd10 2023-10-16T20:59:52.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.996+0000 7f9650cc7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f964c06f710 0x7f964c06fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:52.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.996+0000 7f9650cc7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f964c06f710 0x7f964c06fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57602/0 (socket says 172.21.15.88:57602) 2023-10-16T20:59:52.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.996+0000 7f9650cc7640 1 -- 172.21.15.88:0/141747745 learned_addr learned my addr 172.21.15.88:0/141747745 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:53.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.996+0000 7f964bfff640 1 --2- 172.21.15.88:0/141747745 >> v2:172.21.15.191:3300/0 conn(0x7f964c14fd10 0x7f964c070c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:53.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.996+0000 7f9650cc7640 1 -- 172.21.15.88:0/141747745 >> v2:172.21.15.191:3300/0 conn(0x7f964c14fd10 msgr2=0x7f964c070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:59:53.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.996+0000 7f9650cc7640 1 --2- 172.21.15.88:0/141747745 >> v2:172.21.15.191:3300/0 conn(0x7f964c14fd10 0x7f964c070c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.996+0000 7f9650cc7640 1 -- 172.21.15.88:0/141747745 >> v1:172.21.15.88:6789/0 conn(0x7f964c071170 legacy=0x7f964c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:52.996+0000 7f9650cc7640 1 -- 172.21.15.88:0/141747745 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c155de0 con 0x7f964c06f710 2023-10-16T20:59:53.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.000+0000 7f9650cc7640 1 --2- 172.21.15.88:0/141747745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f964c06f710 0x7f964c06fb40 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f964c072840 tx=0x7f96440213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69fc3bec3594cb2d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:53.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.000+0000 7f964affd640 1 -- 172.21.15.88:0/141747745 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964400b2d0 con 0x7f964c06f710 2023-10-16T20:59:53.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.000+0000 7f964affd640 1 -- 172.21.15.88:0/141747745 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f96440ad050 con 0x7f964c06f710 2023-10-16T20:59:53.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.000+0000 7f964affd640 1 -- 172.21.15.88:0/141747745 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9644009840 con 0x7f964c06f710 2023-10-16T20:59:53.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.000+0000 7f965272a640 1 -- 172.21.15.88:0/141747745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f964c06f710 msgr2=0x7f964c06fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.000+0000 7f965272a640 1 --2- 172.21.15.88:0/141747745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f964c06f710 0x7f964c06fb40 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f964c072840 tx=0x7f96440213c0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.000+0000 7f965272a640 1 -- 172.21.15.88:0/141747745 shutdown_connections 2023-10-16T20:59:53.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.000+0000 7f965272a640 1 --2- 172.21.15.88:0/141747745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f964c06f710 0x7f964c06fb40 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.000+0000 7f965272a640 1 --2- 172.21.15.88:0/141747745 >> v2:172.21.15.191:3300/0 conn(0x7f964c14fd10 0x7f964c070c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.000+0000 7f965272a640 1 -- 172.21.15.88:0/141747745 >> 172.21.15.88:0/141747745 conn(0x7f964c0cad50 msgr2=0x7f964c0cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:53.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.000+0000 7f965272a640 1 -- 172.21.15.88:0/141747745 shutdown_connections 2023-10-16T20:59:53.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.000+0000 7f965272a640 1 -- 172.21.15.88:0/141747745 wait complete. 2023-10-16T20:59:53.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f965272a640 1 Processor -- start 2023-10-16T20:59:53.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f965272a640 1 -- start start 2023-10-16T20:59:53.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f965272a640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f964c06f710 0x7f964c154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:53.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f965272a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f964c071170 0x7f964c1f2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:53.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f965272a640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f964c14fd10 0x7f964c1f3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:53.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f965272a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c0ca7d0 con 0x7f964c071170 2023-10-16T20:59:53.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f965272a640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c0c9c50 con 0x7f964c14fd10 2023-10-16T20:59:53.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f965272a640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964c0c9f30 con 0x7f964c06f710 2023-10-16T20:59:53.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f964b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f964c071170 0x7f964c1f2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:53.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f964b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f964c071170 0x7f964c1f2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57606/0 (socket says 172.21.15.88:57606) 2023-10-16T20:59:53.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f964b7fe640 1 -- 172.21.15.88:0/2190291093 learned_addr learned my addr 172.21.15.88:0/2190291093 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:53.010 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f964bfff640 1 --2- 172.21.15.88:0/2190291093 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f964c06f710 0x7f964c154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:53.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f9650cc7640 1 --2- 172.21.15.88:0/2190291093 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f964c14fd10 0x7f964c1f3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:53.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f964b7fe640 1 -- 172.21.15.88:0/2190291093 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f964c06f710 msgr2=0x7f964c154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f964b7fe640 1 --2- 172.21.15.88:0/2190291093 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f964c06f710 0x7f964c154f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f964b7fe640 1 -- 172.21.15.88:0/2190291093 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f964c14fd10 msgr2=0x7f964c1f3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f964b7fe640 1 --2- 172.21.15.88:0/2190291093 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f964c14fd10 0x7f964c1f3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f964b7fe640 1 -- 172.21.15.88:0/2190291093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964400bd40 con 0x7f964c071170 2023-10-16T20:59:53.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f9650cc7640 1 --2- 172.21.15.88:0/2190291093 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f964c14fd10 0x7f964c1f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:59:53.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f964bfff640 1 --2- 172.21.15.88:0/2190291093 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f964c06f710 0x7f964c154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:59:53.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f964b7fe640 1 --2- 172.21.15.88:0/2190291093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f964c071170 0x7f964c1f2ad0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f964000c4d0 tx=0x7f964000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:53.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f9648ff9640 1 -- 172.21.15.88:0/2190291093 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96400052d0 con 0x7f964c071170 2023-10-16T20:59:53.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f9648ff9640 1 -- 172.21.15.88:0/2190291093 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f9640005430 con 0x7f964c071170 2023-10-16T20:59:53.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f965272a640 1 -- 172.21.15.88:0/2190291093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964c0ca210 con 0x7f964c071170 2023-10-16T20:59:53.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f9648ff9640 1 -- 172.21.15.88:0/2190291093 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640012850 con 0x7f964c071170 2023-10-16T20:59:53.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f965272a640 1 -- 172.21.15.88:0/2190291093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f964c14e9b0 con 0x7f964c071170 2023-10-16T20:59:53.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.004+0000 7f965272a640 1 -- 172.21.15.88:0/2190291093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f964c074700 con 0x7f964c071170 2023-10-16T20:59:53.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.008+0000 7f9648ff9640 1 -- 172.21.15.88:0/2190291093 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f96400129b0 con 0x7f964c071170 2023-10-16T20:59:53.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.008+0000 7f9648ff9640 1 --2- 172.21.15.88:0/2190291093 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f96380664a0 0x7f9638068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:53.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.008+0000 7f9648ff9640 1 -- 172.21.15.88:0/2190291093 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f96400842c0 con 0x7f964c071170 2023-10-16T20:59:53.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.008+0000 7f964bfff640 1 --2- 172.21.15.88:0/2190291093 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f96380664a0 0x7f9638068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:53.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.008+0000 7f964bfff640 1 --2- 172.21.15.88:0/2190291093 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f96380664a0 0x7f9638068960 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f963c00b9f0 tx=0x7f963c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:53.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.012+0000 7f9648ff9640 1 -- 172.21.15.88:0/2190291093 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9640050e60 con 0x7f964c071170 2023-10-16T20:59:53.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.192+0000 7f965272a640 1 -- 172.21.15.88:0/2190291093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f964c148710 con 0x7f964c071170 2023-10-16T20:59:53.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.192+0000 7f9648ff9640 1 -- 172.21.15.88:0/2190291093 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+6581 (secure 0 0 0) 0x7f9640052d90 con 0x7f964c071170 2023-10-16T20:59:53.192 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:59:53.193 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6286,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/2135868641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":2135868641},{"type":"v1","addr":"172.21.15.191:6835","nonce":2135868641}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-16T20:36:55.111692+0000","modified":"2023-10-16T20:58:22.748501+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5477,"mds_1":5709,"mds_2":5535,"mds_3":5570,"mds_4":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5477":{"gid":5477,"name":"b","rank":0,"incarnation":97,"state":"up:active","state_seq":39,"addr":"172.21.15.191:6837/2244284333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2244284333},{"type":"v1","addr":"172.21.15.191:6837","nonce":2244284333}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5535":{"gid":5535,"name":"a","rank":2,"incarnation":52,"state":"up:active","state_seq":32,"addr":"172.21.15.88:6835/3476086754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3476086754},{"type":"v1","addr":"172.21.15.88:6835","nonce":3476086754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5570":{"gid":5570,"name":"d","rank":3,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.191:6833/3394064120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":3394064120},{"type":"v1","addr":"172.21.15.191:6833","nonce":3394064120}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5709":{"gid":5709,"name":"e","rank":1,"incarnation":67,"state":"up:active","state_seq":37,"addr":"172.21.15.88:6838/2600891060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":2600891060},{"type":"v1","addr":"172.21.15.88:6838","nonce":2600891060}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6366":{"gid":6366,"name":"c","rank":4,"incarnation":112,"state":"up:active","state_seq":23,"addr":"172.21.15.88:6839/2182739466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6837","nonce":2182739466},{"type":"v1","addr":"172.21.15.88:6839","nonce":2182739466}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-16T20:59:53.194 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 117 2023-10-16T20:59:53.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.192+0000 7f965272a640 1 -- 172.21.15.88:0/2190291093 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f96380664a0 msgr2=0x7f9638068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.192+0000 7f965272a640 1 --2- 172.21.15.88:0/2190291093 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f96380664a0 0x7f9638068960 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f963c00b9f0 tx=0x7f963c008040 comp rx=0 tx=0).stop 2023-10-16T20:59:53.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.192+0000 7f965272a640 1 -- 172.21.15.88:0/2190291093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f964c071170 msgr2=0x7f964c1f2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.192+0000 7f965272a640 1 --2- 172.21.15.88:0/2190291093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f964c071170 0x7f964c1f2ad0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f964000c4d0 tx=0x7f964000c990 comp rx=0 tx=0).stop 2023-10-16T20:59:53.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.192+0000 7f965272a640 1 -- 172.21.15.88:0/2190291093 shutdown_connections 2023-10-16T20:59:53.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.192+0000 7f965272a640 1 --2- 172.21.15.88:0/2190291093 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f96380664a0 0x7f9638068960 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.192+0000 7f965272a640 1 --2- 172.21.15.88:0/2190291093 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f964c14fd10 0x7f964c1f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.192+0000 7f965272a640 1 --2- 172.21.15.88:0/2190291093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f964c071170 0x7f964c1f2ad0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.192+0000 7f965272a640 1 --2- 172.21.15.88:0/2190291093 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f964c06f710 0x7f964c154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.192+0000 7f965272a640 1 -- 172.21.15.88:0/2190291093 >> 172.21.15.88:0/2190291093 conn(0x7f964c0cad50 msgr2=0x7f964c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:53.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.192+0000 7f965272a640 1 -- 172.21.15.88:0/2190291093 shutdown_connections 2023-10-16T20:59:53.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.192+0000 7f965272a640 1 -- 172.21.15.88:0/2190291093 wait complete. 2023-10-16T20:59:53.213 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T20:59:53.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.348+0000 7f4f09dd9640 1 Processor -- start 2023-10-16T20:59:53.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.348+0000 7f4f09dd9640 1 -- start start 2023-10-16T20:59:53.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.352+0000 7f4f09dd9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f04150660 0x7f4f040c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:53.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.352+0000 7f4f09dd9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f040c5c60 con 0x7f4f04150280 2023-10-16T20:59:53.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.352+0000 7f4f09dd9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f040c5370 con 0x7f4f040c6bc0 2023-10-16T20:59:53.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.352+0000 7f4f09dd9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f04156d10 con 0x7f4f04150660 2023-10-16T20:59:53.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.352+0000 7f4f02ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f04150660 0x7f4f040c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:53.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.352+0000 7f4f02ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f04150660 0x7f4f040c60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57608/0 (socket says 172.21.15.88:57608) 2023-10-16T20:59:53.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.352+0000 7f4f02ffd640 1 -- 172.21.15.88:0/2475707657 learned_addr learned my addr 172.21.15.88:0/2475707657 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:53.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.352+0000 7f4f02ffd640 1 -- 172.21.15.88:0/2475707657 >> v1:172.21.15.191:6789/0 conn(0x7f4f040c6bc0 legacy=0x7f4f04156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.352+0000 7f4f02ffd640 1 -- 172.21.15.88:0/2475707657 >> v1:172.21.15.88:6789/0 conn(0x7f4f04150280 legacy=0x7f4f0406d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.352+0000 7f4f02ffd640 1 -- 172.21.15.88:0/2475707657 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f040c5610 con 0x7f4f04150660 2023-10-16T20:59:53.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.356+0000 7f4f02ffd640 1 --2- 172.21.15.88:0/2475707657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f04150660 0x7f4f040c60f0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8009870 tx=0x7f4ef801cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5400c7beaea7b13 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:53.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.356+0000 7f4f027fc640 1 -- 172.21.15.88:0/2475707657 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef80a95f0 con 0x7f4f04150660 2023-10-16T20:59:53.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.356+0000 7f4f027fc640 1 -- 172.21.15.88:0/2475707657 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f4ef80a9750 con 0x7f4f04150660 2023-10-16T20:59:53.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.356+0000 7f4f027fc640 1 -- 172.21.15.88:0/2475707657 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef80a9b00 con 0x7f4f04150660 2023-10-16T20:59:53.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.356+0000 7f4f09dd9640 1 -- 172.21.15.88:0/2475707657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f04150660 msgr2=0x7f4f040c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.356+0000 7f4f09dd9640 1 --2- 172.21.15.88:0/2475707657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f04150660 0x7f4f040c60f0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8009870 tx=0x7f4ef801cf70 comp rx=0 tx=0).stop 2023-10-16T20:59:53.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.356+0000 7f4f09dd9640 1 -- 172.21.15.88:0/2475707657 shutdown_connections 2023-10-16T20:59:53.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.356+0000 7f4f09dd9640 1 --2- 172.21.15.88:0/2475707657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f04150660 0x7f4f040c60f0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.356+0000 7f4f09dd9640 1 -- 172.21.15.88:0/2475707657 >> 172.21.15.88:0/2475707657 conn(0x7f4f040cad50 msgr2=0x7f4f040cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:53.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.356+0000 7f4f09dd9640 1 -- 172.21.15.88:0/2475707657 shutdown_connections 2023-10-16T20:59:53.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.356+0000 7f4f09dd9640 1 -- 172.21.15.88:0/2475707657 wait complete. 2023-10-16T20:59:53.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.356+0000 7f4f09dd9640 1 Processor -- start 2023-10-16T20:59:53.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.356+0000 7f4f09dd9640 1 -- start start 2023-10-16T20:59:53.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f09dd9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f040c6bc0 0x7f4f04150a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:53.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f09dd9640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f04150280 0x7f4f041e9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:53.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f09dd9640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f04150660 0x7f4f041f2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:53.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f09dd9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f04152590 con 0x7f4f040c6bc0 2023-10-16T20:59:53.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f09dd9640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f04151130 con 0x7f4f04150280 2023-10-16T20:59:53.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f09dd9640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f04151410 con 0x7f4f04150660 2023-10-16T20:59:53.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f037fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f040c6bc0 0x7f4f04150a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:53.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f037fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f040c6bc0 0x7f4f04150a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57612/0 (socket says 172.21.15.88:57612) 2023-10-16T20:59:53.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f037fe640 1 -- 172.21.15.88:0/3139068106 learned_addr learned my addr 172.21.15.88:0/3139068106 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:53.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f02ffd640 1 --2- 172.21.15.88:0/3139068106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f04150280 0x7f4f041e9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:53.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f03fff640 1 --2- 172.21.15.88:0/3139068106 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f04150660 0x7f4f041f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:53.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f037fe640 1 -- 172.21.15.88:0/3139068106 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f04150660 msgr2=0x7f4f041f2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f037fe640 1 --2- 172.21.15.88:0/3139068106 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f04150660 0x7f4f041f2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f037fe640 1 -- 172.21.15.88:0/3139068106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f04150280 msgr2=0x7f4f041e9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f037fe640 1 --2- 172.21.15.88:0/3139068106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f04150280 0x7f4f041e9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f037fe640 1 -- 172.21.15.88:0/3139068106 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef80094d0 con 0x7f4f040c6bc0 2023-10-16T20:59:53.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f03fff640 1 --2- 172.21.15.88:0/3139068106 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f04150660 0x7f4f041f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:59:53.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f037fe640 1 --2- 172.21.15.88:0/3139068106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f040c6bc0 0x7f4f04150a80 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400c500 tx=0x7f4ef400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:53.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4eebfff640 1 -- 172.21.15.88:0/3139068106 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4005220 con 0x7f4f040c6bc0 2023-10-16T20:59:53.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4eebfff640 1 -- 172.21.15.88:0/3139068106 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f4ef4005380 con 0x7f4f040c6bc0 2023-10-16T20:59:53.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4eebfff640 1 -- 172.21.15.88:0/3139068106 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4007290 con 0x7f4f040c6bc0 2023-10-16T20:59:53.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f09dd9640 1 -- 172.21.15.88:0/3139068106 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f04151bf0 con 0x7f4f040c6bc0 2023-10-16T20:59:53.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f09dd9640 1 -- 172.21.15.88:0/3139068106 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f041521b0 con 0x7f4f040c6bc0 2023-10-16T20:59:53.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4f09dd9640 1 -- 172.21.15.88:0/3139068106 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f040c1bc0 con 0x7f4f040c6bc0 2023-10-16T20:59:53.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.360+0000 7f4eebfff640 1 -- 172.21.15.88:0/3139068106 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4ef40073f0 con 0x7f4f040c6bc0 2023-10-16T20:59:53.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.364+0000 7f4eebfff640 1 --2- 172.21.15.88:0/3139068106 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4ef00664a0 0x7f4ef0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:53.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.364+0000 7f4eebfff640 1 -- 172.21.15.88:0/3139068106 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7f4ef4083f20 con 0x7f4f040c6bc0 2023-10-16T20:59:53.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.364+0000 7f4f02ffd640 1 --2- 172.21.15.88:0/3139068106 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4ef00664a0 0x7f4ef0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:53.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.364+0000 7f4f02ffd640 1 --2- 172.21.15.88:0/3139068106 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4ef00664a0 0x7f4ef0068960 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80ae000 tx=0x7f4ef80b3040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:53.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.368+0000 7f4eebfff640 1 -- 172.21.15.88:0/3139068106 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4ef4050bf0 con 0x7f4f040c6bc0 2023-10-16T20:59:53.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.520+0000 7f4f09dd9640 1 -- 172.21.15.88:0/3139068106 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f04074700 con 0x7f4f040c6bc0 2023-10-16T20:59:53.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.524+0000 7f4eebfff640 1 -- 172.21.15.88:0/3139068106 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+20065 (secure 0 0 0) 0x7f4ef4054210 con 0x7f4f040c6bc0 2023-10-16T20:59:53.524 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-10-16T20:59:53.524 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":67,"fsid":"8cf7342f-a394-4b4c-ac35-9931ac9d9e57","created":"2023-10-16T20:36:30.476427+0000","modified":"2023-10-16T20:58:00.653255+0000","last_up_change":"2023-10-16T20:36:42.412256+0000","last_in_change":"2023-10-16T20:36:33.014460+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T20:36:43.646418+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-16T20:36:53.546513+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-16T20:36:54.143703+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-16T20:36:57.314642+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":59,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8f96fb46-c190-4aae-85f6-7fc0bf73047b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6801","nonce":1130215823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6803","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6806","nonce":1130215823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6816","nonce":1130215823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1130215823},{"type":"v1","addr":"172.21.15.88:6811","nonce":1130215823}]},"public_addr":"172.21.15.88:6801/1130215823","cluster_addr":"172.21.15.88:6806/1130215823","heartbeat_back_addr":"172.21.15.88:6816/1130215823","heartbeat_front_addr":"172.21.15.88:6811/1130215823","state":["exists","up"]},{"osd":1,"uuid":"f4a636df-8b38-4af2-a0a2-59dc457d22bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6809","nonce":3386972440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6813","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6817","nonce":3386972440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6824","nonce":3386972440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6819","nonce":3386972440},{"type":"v1","addr":"172.21.15.88:6821","nonce":3386972440}]},"public_addr":"172.21.15.88:6809/3386972440","cluster_addr":"172.21.15.88:6817/3386972440","heartbeat_back_addr":"172.21.15.88:6824/3386972440","heartbeat_front_addr":"172.21.15.88:6821/3386972440","state":["exists","up"]},{"osd":2,"uuid":"38af5d72-deaa-45f1-86b8-920f2698202c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6805","nonce":360619264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6807","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6810","nonce":360619264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6820","nonce":360619264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":360619264},{"type":"v1","addr":"172.21.15.88:6815","nonce":360619264}]},"public_addr":"172.21.15.88:6805/360619264","cluster_addr":"172.21.15.88:6810/360619264","heartbeat_back_addr":"172.21.15.88:6820/360619264","heartbeat_front_addr":"172.21.15.88:6815/360619264","state":["exists","up"]},{"osd":3,"uuid":"9cd29885-fd6d-4419-aae7-e9d4065bbc31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6823","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6825","nonce":1519973662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6827","nonce":1519973662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6831","nonce":1519973662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":1519973662},{"type":"v1","addr":"172.21.15.88:6829","nonce":1519973662}]},"public_addr":"172.21.15.88:6825/1519973662","cluster_addr":"172.21.15.88:6827/1519973662","heartbeat_back_addr":"172.21.15.88:6831/1519973662","heartbeat_front_addr":"172.21.15.88:6829/1519973662","state":["exists","up"]},{"osd":4,"uuid":"81025152-01c0-4624-be28-68cc5c853e79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6801","nonce":584586645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6803","nonce":584586645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6807","nonce":584586645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":584586645},{"type":"v1","addr":"172.21.15.191:6805","nonce":584586645}]},"public_addr":"172.21.15.191:6801/584586645","cluster_addr":"172.21.15.191:6803/584586645","heartbeat_back_addr":"172.21.15.191:6807/584586645","heartbeat_front_addr":"172.21.15.191:6805/584586645","state":["exists","up"]},{"osd":5,"uuid":"6e32466e-2f1f-4220-9f02-30708ad99202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6809","nonce":3826846674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6811","nonce":3826846674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6815","nonce":3826846674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3826846674},{"type":"v1","addr":"172.21.15.191:6813","nonce":3826846674}]},"public_addr":"172.21.15.191:6809/3826846674","cluster_addr":"172.21.15.191:6811/3826846674","heartbeat_back_addr":"172.21.15.191:6815/3826846674","heartbeat_front_addr":"172.21.15.191:6813/3826846674","state":["exists","up"]},{"osd":6,"uuid":"d82bbafe-7801-495b-95ce-768a512980fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6817","nonce":292421315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6819","nonce":292421315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6823","nonce":292421315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":292421315},{"type":"v1","addr":"172.21.15.191:6821","nonce":292421315}]},"public_addr":"172.21.15.191:6817/292421315","cluster_addr":"172.21.15.191:6819/292421315","heartbeat_back_addr":"172.21.15.191:6823/292421315","heartbeat_front_addr":"172.21.15.191:6821/292421315","state":["exists","up"]},{"osd":7,"uuid":"d10aba86-e78a-4f6d-a15c-fe5dffd0aac6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6825","nonce":4079140119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6827","nonce":4079140119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6831","nonce":4079140119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":4079140119},{"type":"v1","addr":"172.21.15.191:6829","nonce":4079140119}]},"public_addr":"172.21.15.191:6825/4079140119","cluster_addr":"172.21.15.191:6827/4079140119","heartbeat_back_addr":"172.21.15.191:6831/4079140119","heartbeat_front_addr":"172.21.15.191:6829/4079140119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.585476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.785646+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.758265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.793711+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.637894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.764919+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.759163+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-16T20:36:38.769553+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6839/2172973775":"2023-10-17T20:56:40.633674+0000","172.21.15.191:6833/3633326602":"2023-10-17T20:55:50.604940+0000","172.21.15.191:6836/1120825335":"2023-10-17T20:42:10.335721+0000","172.21.15.191:6832/3633326602":"2023-10-17T20:55:50.604940+0000","172.21.15.88:6835/518281867":"2023-10-17T20:44:25.369398+0000","172.21.15.191:6837/2468647900":"2023-10-17T20:53:25.553637+0000","172.21.15.88:6837/1987579860":"2023-10-17T20:43:25.353447+0000","172.21.15.191:6835/3119045295":"2023-10-17T20:58:00.653202+0000","172.21.15.88:6836/321173984":"2023-10-17T20:46:15.444657+0000","172.21.15.191:6837/1120825335":"2023-10-17T20:42:10.335721+0000","172.21.15.191:6832/2446850309":"2023-10-17T20:48:43.641901+0000","172.21.15.88:6837/2172973775":"2023-10-17T20:56:40.633674+0000","172.21.15.88:6839/1987579860":"2023-10-17T20:43:25.353447+0000","172.21.15.191:6834/3119045295":"2023-10-17T20:58:00.653202+0000","172.21.15.88:6834/518281867":"2023-10-17T20:44:25.369398+0000","172.21.15.88:6838/321173984":"2023-10-17T20:46:15.444657+0000","172.21.15.191:6834/3059699574":"2023-10-17T20:51:20.522638+0000","172.21.15.191:6833/2446850309":"2023-10-17T20:48:43.641901+0000","172.21.15.191:6835/3059699574":"2023-10-17T20:51:20.522638+0000","172.21.15.191:6836/2468647900":"2023-10-17T20:53:25.553637+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T20:59:53.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.524+0000 7f4f09dd9640 1 -- 172.21.15.88:0/3139068106 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4ef00664a0 msgr2=0x7f4ef0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.524+0000 7f4f09dd9640 1 --2- 172.21.15.88:0/3139068106 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4ef00664a0 0x7f4ef0068960 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80ae000 tx=0x7f4ef80b3040 comp rx=0 tx=0).stop 2023-10-16T20:59:53.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.524+0000 7f4f09dd9640 1 -- 172.21.15.88:0/3139068106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f040c6bc0 msgr2=0x7f4f04150a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.524+0000 7f4f09dd9640 1 --2- 172.21.15.88:0/3139068106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f040c6bc0 0x7f4f04150a80 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400c500 tx=0x7f4ef400c9c0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.524+0000 7f4f09dd9640 1 -- 172.21.15.88:0/3139068106 shutdown_connections 2023-10-16T20:59:53.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.524+0000 7f4f09dd9640 1 --2- 172.21.15.88:0/3139068106 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f4ef00664a0 0x7f4ef0068960 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.524+0000 7f4f09dd9640 1 --2- 172.21.15.88:0/3139068106 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f4f04150660 0x7f4f041f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.524+0000 7f4f09dd9640 1 --2- 172.21.15.88:0/3139068106 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4f04150280 0x7f4f041e9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.524+0000 7f4f09dd9640 1 --2- 172.21.15.88:0/3139068106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f040c6bc0 0x7f4f04150a80 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.524+0000 7f4f09dd9640 1 -- 172.21.15.88:0/3139068106 >> 172.21.15.88:0/3139068106 conn(0x7f4f040cad50 msgr2=0x7f4f0414f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:53.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.524+0000 7f4f09dd9640 1 -- 172.21.15.88:0/3139068106 shutdown_connections 2023-10-16T20:59:53.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.524+0000 7f4f09dd9640 1 -- 172.21.15.88:0/3139068106 wait complete. 2023-10-16T20:59:53.547 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-16T20:59:53.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.684+0000 7ff6cecea640 1 Processor -- start 2023-10-16T20:59:53.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.684+0000 7ff6cecea640 1 -- start start 2023-10-16T20:59:53.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.684+0000 7ff6cecea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff6c80c66e0 0x7ff6c815a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:53.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.684+0000 7ff6cecea640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6c80cb5c0 0x7ff6c8071420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:53.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.688+0000 7ff6cecea640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c8070f40 con 0x7ff6c80c6ac0 2023-10-16T20:59:53.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.688+0000 7ff6cecea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6c815a790 con 0x7ff6c80c66e0 2023-10-16T20:59:53.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.688+0000 7ff6cecea640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c806a2d0 con 0x7ff6c80cb5c0 2023-10-16T20:59:53.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.688+0000 7ff6cca86640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff6c80c66e0 0x7ff6c815a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:53.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.688+0000 7ff6cca86640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff6c80c66e0 0x7ff6c815a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57620/0 (socket says 172.21.15.88:57620) 2023-10-16T20:59:53.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.688+0000 7ff6cca86640 1 -- 172.21.15.88:0/1540661928 learned_addr learned my addr 172.21.15.88:0/1540661928 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:53.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.688+0000 7ff6cd287640 1 --2- 172.21.15.88:0/1540661928 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6c80cb5c0 0x7ff6c8071420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:53.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.688+0000 7ff6cca86640 1 -- 172.21.15.88:0/1540661928 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6c80cb5c0 msgr2=0x7ff6c8071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:59:53.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.688+0000 7ff6cca86640 1 --2- 172.21.15.88:0/1540661928 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6c80cb5c0 0x7ff6c8071420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.688+0000 7ff6cca86640 1 -- 172.21.15.88:0/1540661928 >> v1:172.21.15.88:6789/0 conn(0x7ff6c80c6ac0 legacy=0x7ff6c8150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.688+0000 7ff6cca86640 1 -- 172.21.15.88:0/1540661928 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c806a5b0 con 0x7ff6c80c66e0 2023-10-16T20:59:53.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.688+0000 7ff6cca86640 1 --2- 172.21.15.88:0/1540661928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c80c66e0 0x7ff6c815a250 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc009870 tx=0x7ff6bc01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64fc523bcf05e787 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:53.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6c77fe640 1 -- 172.21.15.88:0/1540661928 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc0ac600 con 0x7ff6c80c66e0 2023-10-16T20:59:53.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6c77fe640 1 -- 172.21.15.88:0/1540661928 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7ff6bc0a8050 con 0x7ff6c80c66e0 2023-10-16T20:59:53.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6c77fe640 1 -- 172.21.15.88:0/1540661928 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc0ac9a0 con 0x7ff6c80c66e0 2023-10-16T20:59:53.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6cecea640 1 -- 172.21.15.88:0/1540661928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c80c66e0 msgr2=0x7ff6c815a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6cecea640 1 --2- 172.21.15.88:0/1540661928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c80c66e0 0x7ff6c815a250 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc009870 tx=0x7ff6bc01c4b0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6cecea640 1 -- 172.21.15.88:0/1540661928 shutdown_connections 2023-10-16T20:59:53.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6cecea640 1 --2- 172.21.15.88:0/1540661928 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6c80cb5c0 0x7ff6c8071420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6cecea640 1 --2- 172.21.15.88:0/1540661928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c80c66e0 0x7ff6c815a250 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6cecea640 1 -- 172.21.15.88:0/1540661928 >> 172.21.15.88:0/1540661928 conn(0x7ff6c80ca9f0 msgr2=0x7ff6c80cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:53.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6cecea640 1 -- 172.21.15.88:0/1540661928 shutdown_connections 2023-10-16T20:59:53.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6cecea640 1 -- 172.21.15.88:0/1540661928 wait complete. 2023-10-16T20:59:53.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6cecea640 1 Processor -- start 2023-10-16T20:59:53.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6cecea640 1 -- start start 2023-10-16T20:59:53.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6cecea640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6c80c66e0 0x7ff6c81436a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:53.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6cecea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c80c6ac0 0x7ff6c814e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:53.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6cecea640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6c80cb5c0 0x7ff6c814e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:53.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6cecea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c814bcc0 con 0x7ff6c80c6ac0 2023-10-16T20:59:53.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6cecea640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c814bf10 con 0x7ff6c80c66e0 2023-10-16T20:59:53.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6cecea640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c814c1f0 con 0x7ff6c80cb5c0 2023-10-16T20:59:53.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6c7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c80c6ac0 0x7ff6c814e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:53.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6c7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c80c6ac0 0x7ff6c814e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57624/0 (socket says 172.21.15.88:57624) 2023-10-16T20:59:53.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.692+0000 7ff6c7fff640 1 -- 172.21.15.88:0/1550557237 learned_addr learned my addr 172.21.15.88:0/1550557237 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:53.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.696+0000 7ff6c7fff640 1 -- 172.21.15.88:0/1550557237 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6c80cb5c0 msgr2=0x7ff6c814e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.696+0000 7ff6c7fff640 1 --2- 172.21.15.88:0/1550557237 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6c80cb5c0 0x7ff6c814e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.696+0000 7ff6cca86640 1 --2- 172.21.15.88:0/1550557237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6c80c66e0 0x7ff6c81436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:53.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.696+0000 7ff6c7fff640 1 -- 172.21.15.88:0/1550557237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6c80c66e0 msgr2=0x7ff6c81436a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.696+0000 7ff6c7fff640 1 --2- 172.21.15.88:0/1550557237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6c80c66e0 0x7ff6c81436a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.696+0000 7ff6c7fff640 1 -- 172.21.15.88:0/1550557237 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6bc0094d0 con 0x7ff6c80c6ac0 2023-10-16T20:59:53.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.696+0000 7ff6cca86640 1 --2- 172.21.15.88:0/1550557237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6c80c66e0 0x7ff6c81436a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-16T20:59:53.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.696+0000 7ff6c7fff640 1 --2- 172.21.15.88:0/1550557237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c80c6ac0 0x7ff6c814e060 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7ff6b400c4d0 tx=0x7ff6b400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:53.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.696+0000 7ff6c57fa640 1 -- 172.21.15.88:0/1550557237 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b40052d0 con 0x7ff6c80c6ac0 2023-10-16T20:59:53.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.696+0000 7ff6c57fa640 1 -- 172.21.15.88:0/1550557237 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7ff6b4005430 con 0x7ff6c80c6ac0 2023-10-16T20:59:53.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.696+0000 7ff6c57fa640 1 -- 172.21.15.88:0/1550557237 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b4012850 con 0x7ff6c80c6ac0 2023-10-16T20:59:53.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.696+0000 7ff6cecea640 1 -- 172.21.15.88:0/1550557237 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c814c4d0 con 0x7ff6c80c6ac0 2023-10-16T20:59:53.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.696+0000 7ff6cecea640 1 -- 172.21.15.88:0/1550557237 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c8143be0 con 0x7ff6c80c6ac0 2023-10-16T20:59:53.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.696+0000 7ff6cecea640 1 -- 172.21.15.88:0/1550557237 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c815eda0 con 0x7ff6c80c6ac0 2023-10-16T20:59:53.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.696+0000 7ff6c57fa640 1 -- 172.21.15.88:0/1550557237 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff6b40129b0 con 0x7ff6c80c6ac0 2023-10-16T20:59:53.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.700+0000 7ff6c57fa640 1 --2- 172.21.15.88:0/1550557237 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff6ac0664a0 0x7ff6ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:53.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.700+0000 7ff6c57fa640 1 -- 172.21.15.88:0/1550557237 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7523+0+0 (secure 0 0 0) 0x7ff6b4084170 con 0x7ff6c80c6ac0 2023-10-16T20:59:53.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.700+0000 7ff6cca86640 1 --2- 172.21.15.88:0/1550557237 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff6ac0664a0 0x7ff6ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:53.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.700+0000 7ff6cca86640 1 --2- 172.21.15.88:0/1550557237 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff6ac0664a0 0x7ff6ac068960 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0ae000 tx=0x7ff6bc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:53.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.704+0000 7ff6c57fa640 1 -- 172.21.15.88:0/1550557237 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff6c815eda0 con 0x7ff6c80c6ac0 2023-10-16T20:59:53.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.880+0000 7ff6cecea640 1 -- 172.21.15.88:0/1550557237 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff6c815eda0 con 0x7ff6c80c6ac0 2023-10-16T20:59:53.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.888+0000 7ff6c57fa640 1 -- 172.21.15.88:0/1550557237 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v118) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff6c815eda0 con 0x7ff6c80c6ac0 2023-10-16T20:59:53.887 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-16T20:59:53.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.888+0000 7ff6cecea640 1 -- 172.21.15.88:0/1550557237 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff6ac0664a0 msgr2=0x7ff6ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.888+0000 7ff6cecea640 1 --2- 172.21.15.88:0/1550557237 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff6ac0664a0 0x7ff6ac068960 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0ae000 tx=0x7ff6bc008040 comp rx=0 tx=0).stop 2023-10-16T20:59:53.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.888+0000 7ff6cecea640 1 -- 172.21.15.88:0/1550557237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c80c6ac0 msgr2=0x7ff6c814e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:53.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.888+0000 7ff6cecea640 1 --2- 172.21.15.88:0/1550557237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c80c6ac0 0x7ff6c814e060 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7ff6b400c4d0 tx=0x7ff6b400c990 comp rx=0 tx=0).stop 2023-10-16T20:59:53.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.888+0000 7ff6cecea640 1 -- 172.21.15.88:0/1550557237 shutdown_connections 2023-10-16T20:59:53.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.888+0000 7ff6cecea640 1 --2- 172.21.15.88:0/1550557237 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7ff6ac0664a0 0x7ff6ac068960 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.888+0000 7ff6cecea640 1 --2- 172.21.15.88:0/1550557237 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7ff6c80cb5c0 0x7ff6c814e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.888+0000 7ff6cecea640 1 --2- 172.21.15.88:0/1550557237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c80c6ac0 0x7ff6c814e060 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.888+0000 7ff6cecea640 1 --2- 172.21.15.88:0/1550557237 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff6c80c66e0 0x7ff6c81436a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:53.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.888+0000 7ff6cecea640 1 -- 172.21.15.88:0/1550557237 >> 172.21.15.88:0/1550557237 conn(0x7ff6c80ca9f0 msgr2=0x7ff6c80c2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:53.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.888+0000 7ff6cecea640 1 -- 172.21.15.88:0/1550557237 shutdown_connections 2023-10-16T20:59:53.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:53.888+0000 7ff6cecea640 1 -- 172.21.15.88:0/1550557237 wait complete. 2023-10-16T20:59:53.914 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-16T20:59:53.961 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2023-10-16T20:59:53.962 INFO:tasks.ceph.mds.d.smithi191.stdout:starting mds.d at 2023-10-16T20:59:53.964 INFO:tasks.ceph.mds.b.smithi191.stdout:starting mds.b at 2023-10-16T20:59:53.985 INFO:tasks.ceph.mds.e.smithi088.stdout:starting mds.e at 2023-10-16T20:59:53.991 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2023-10-16T20:59:54.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.068+0000 7f37635c4640 1 Processor -- start 2023-10-16T20:59:54.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.068+0000 7f37635c4640 1 -- start start 2023-10-16T20:59:54.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.072+0000 7f37635c4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f375c153a00 0x7f375c0cb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:54.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.072+0000 7f37635c4640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f375c0ad870 0x7f375c0cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:54.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.072+0000 7f37635c4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375c0c0230 con 0x7f375c153620 2023-10-16T20:59:54.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.072+0000 7f37635c4640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f375c1594f0 con 0x7f375c0ad870 2023-10-16T20:59:54.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.072+0000 7f37635c4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f375c0c0480 con 0x7f375c153a00 2023-10-16T20:59:54.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.072+0000 7f3761b61640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f375c0ad870 0x7f375c0cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:54.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.072+0000 7f3760b5f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f375c153a00 0x7f375c0cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:54.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.072+0000 7f3760b5f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f375c153a00 0x7f375c0cb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3301/0 says I am v2:172.21.15.88:32988/0 (socket says 172.21.15.88:32988) 2023-10-16T20:59:54.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.072+0000 7f3761b61640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f375c0ad870 0x7f375c0cb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:58776/0 (socket says 172.21.15.88:58776) 2023-10-16T20:59:54.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.072+0000 7f3760b5f640 1 -- 172.21.15.88:0/3137017580 learned_addr learned my addr 172.21.15.88:0/3137017580 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:54.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.072+0000 7f3761360640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f375c153620 0x7f375c0c5c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.88:40244/0 (socket says 172.21.15.88:40244) 2023-10-16T20:59:54.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.072+0000 7f3760b5f640 1 -- 172.21.15.88:0/3137017580 >> v2:172.21.15.191:3300/0 conn(0x7f375c0ad870 msgr2=0x7f375c0cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:59:54.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.072+0000 7f3760b5f640 1 --2- 172.21.15.88:0/3137017580 >> v2:172.21.15.191:3300/0 conn(0x7f375c0ad870 0x7f375c0cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:54.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.072+0000 7f3760b5f640 1 -- 172.21.15.88:0/3137017580 >> v1:172.21.15.191:6789/0 conn(0x7f375c153620 legacy=0x7f375c0c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:54.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.072+0000 7f3760b5f640 1 -- 172.21.15.88:0/3137017580 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375c0c0760 con 0x7f375c153a00 2023-10-16T20:59:54.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.072+0000 7f3760b5f640 1 --2- 172.21.15.88:0/3137017580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f375c153a00 0x7f375c0cb280 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f374c009870 tx=0x7f374c01c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf28071a3d376e6e server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:54.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f3753fff640 1 -- 172.21.15.88:0/3137017580 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374c0a7070 con 0x7f375c153a00 2023-10-16T20:59:54.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f3753fff640 1 -- 172.21.15.88:0/3137017580 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f374c0ab430 con 0x7f375c153a00 2023-10-16T20:59:54.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f3753fff640 1 -- 172.21.15.88:0/3137017580 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374c0ab7e0 con 0x7f375c153a00 2023-10-16T20:59:54.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f37635c4640 1 -- 172.21.15.88:0/3137017580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f375c153a00 msgr2=0x7f375c0cb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:54.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f37635c4640 1 --2- 172.21.15.88:0/3137017580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f375c153a00 0x7f375c0cb280 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f374c009870 tx=0x7f374c01c540 comp rx=0 tx=0).stop 2023-10-16T20:59:54.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f37635c4640 1 -- 172.21.15.88:0/3137017580 shutdown_connections 2023-10-16T20:59:54.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f37635c4640 1 --2- 172.21.15.88:0/3137017580 >> v2:172.21.15.191:3300/0 conn(0x7f375c0ad870 0x7f375c0cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:54.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f37635c4640 1 --2- 172.21.15.88:0/3137017580 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f375c153a00 0x7f375c0cb280 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:54.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f37635c4640 1 -- 172.21.15.88:0/3137017580 >> 172.21.15.88:0/3137017580 conn(0x7f375c069e90 msgr2=0x7f375c06a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:54.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f37635c4640 1 -- 172.21.15.88:0/3137017580 shutdown_connections 2023-10-16T20:59:54.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f37635c4640 1 -- 172.21.15.88:0/3137017580 wait complete. 2023-10-16T20:59:54.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f37635c4640 1 Processor -- start 2023-10-16T20:59:54.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f37635c4640 1 -- start start 2023-10-16T20:59:54.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f37635c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f375c0ad870 0x7f375c1f2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:54.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f37635c4640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f375c153620 0x7f375c141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:54.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f37635c4640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f375c153a00 0x7f375c13ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:54.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f37635c4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c15c3c0 con 0x7f375c0ad870 2023-10-16T20:59:54.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f37635c4640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c15a9f0 con 0x7f375c153620 2023-10-16T20:59:54.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.076+0000 7f37635c4640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f375c15a750 con 0x7f375c153a00 2023-10-16T20:59:54.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f3761360640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f375c0ad870 0x7f375c1f2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:54.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f3760b5f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f375c153620 0x7f375c141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:54.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f3761360640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f375c0ad870 0x7f375c1f2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57678/0 (socket says 172.21.15.88:57678) 2023-10-16T20:59:54.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f3760b5f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f375c153620 0x7f375c141ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.88:58784/0 (socket says 172.21.15.88:58784) 2023-10-16T20:59:54.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f3761360640 1 -- 172.21.15.88:0/2004254210 learned_addr learned my addr 172.21.15.88:0/2004254210 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:54.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f3761b61640 1 --2- 172.21.15.88:0/2004254210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f375c153a00 0x7f375c13ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:54.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f3760b5f640 1 -- 172.21.15.88:0/2004254210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f375c153a00 msgr2=0x7f375c13ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:54.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f3760b5f640 1 --2- 172.21.15.88:0/2004254210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f375c153a00 0x7f375c13ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:54.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f3760b5f640 1 -- 172.21.15.88:0/2004254210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f375c0ad870 msgr2=0x7f375c1f2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:54.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f3760b5f640 1 --2- 172.21.15.88:0/2004254210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f375c0ad870 0x7f375c1f2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:54.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f3760b5f640 1 -- 172.21.15.88:0/2004254210 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374c0094d0 con 0x7f375c153620 2023-10-16T20:59:54.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f3760b5f640 1 --2- 172.21.15.88:0/2004254210 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f375c153620 0x7f375c141ae0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f374c0ac000 tx=0x7f374c0192e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:54.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f3751ffb640 1 -- 172.21.15.88:0/2004254210 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374c0b4800 con 0x7f375c153620 2023-10-16T20:59:54.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f3751ffb640 1 -- 172.21.15.88:0/2004254210 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f374c0b49a0 con 0x7f375c153620 2023-10-16T20:59:54.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f3751ffb640 1 -- 172.21.15.88:0/2004254210 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374c0b4d70 con 0x7f375c153620 2023-10-16T20:59:54.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f37635c4640 1 -- 172.21.15.88:0/2004254210 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f375c15bad0 con 0x7f375c153620 2023-10-16T20:59:54.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f37635c4640 1 -- 172.21.15.88:0/2004254210 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375c15bd70 con 0x7f375c153620 2023-10-16T20:59:54.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f37635c4640 1 -- 172.21.15.88:0/2004254210 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f375c15c530 con 0x7f375c153620 2023-10-16T20:59:54.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.080+0000 7f3751ffb640 1 -- 172.21.15.88:0/2004254210 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f374c0be400 con 0x7f375c153620 2023-10-16T20:59:54.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.084+0000 7f3751ffb640 1 --2- 172.21.15.88:0/2004254210 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f37400664a0 0x7f3740068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:54.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.084+0000 7f3751ffb640 1 -- 172.21.15.88:0/2004254210 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7953+0+0 (secure 0 0 0) 0x7f374c11f250 con 0x7f375c153620 2023-10-16T20:59:54.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.084+0000 7f3761360640 1 --2- 172.21.15.88:0/2004254210 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f37400664a0 0x7f3740068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:54.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.084+0000 7f3761360640 1 --2- 172.21.15.88:0/2004254210 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f37400664a0 0x7f3740068960 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3758000f70 tx=0x7f3758007f10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:54.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.084+0000 7f3751ffb640 1 -- 172.21.15.88:0/2004254210 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f374c0ebd70 con 0x7f375c153620 2023-10-16T20:59:54.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.268+0000 7f37635c4640 1 -- 172.21.15.88:0/2004254210 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f375c074700 con 0x7f375c153620 2023-10-16T20:59:54.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.896+0000 7f3751ffb640 1 -- 172.21.15.88:0/2004254210 <== mon.1 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v119) v1 ==== 104+0+0 (secure 0 0 0) 0x7f374c0edca0 con 0x7f375c153620 2023-10-16T20:59:54.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.900+0000 7f37635c4640 1 -- 172.21.15.88:0/2004254210 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f37400664a0 msgr2=0x7f3740068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:54.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.900+0000 7f37635c4640 1 --2- 172.21.15.88:0/2004254210 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f37400664a0 0x7f3740068960 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3758000f70 tx=0x7f3758007f10 comp rx=0 tx=0).stop 2023-10-16T20:59:54.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.900+0000 7f37635c4640 1 -- 172.21.15.88:0/2004254210 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f375c153620 msgr2=0x7f375c141ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:54.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.900+0000 7f37635c4640 1 --2- 172.21.15.88:0/2004254210 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f375c153620 0x7f375c141ae0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f374c0ac000 tx=0x7f374c0192e0 comp rx=0 tx=0).stop 2023-10-16T20:59:54.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.900+0000 7f37635c4640 1 -- 172.21.15.88:0/2004254210 shutdown_connections 2023-10-16T20:59:54.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.900+0000 7f37635c4640 1 --2- 172.21.15.88:0/2004254210 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f37400664a0 0x7f3740068960 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:54.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.900+0000 7f37635c4640 1 --2- 172.21.15.88:0/2004254210 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f375c153a00 0x7f375c13ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:54.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.900+0000 7f37635c4640 1 --2- 172.21.15.88:0/2004254210 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f375c153620 0x7f375c141ae0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:54.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.900+0000 7f37635c4640 1 --2- 172.21.15.88:0/2004254210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f375c0ad870 0x7f375c1f2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:54.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.900+0000 7f37635c4640 1 -- 172.21.15.88:0/2004254210 >> 172.21.15.88:0/2004254210 conn(0x7f375c069e90 msgr2=0x7f375c06fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:54.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.900+0000 7f37635c4640 1 -- 172.21.15.88:0/2004254210 shutdown_connections 2023-10-16T20:59:54.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:54.900+0000 7f37635c4640 1 -- 172.21.15.88:0/2004254210 wait complete. 2023-10-16T20:59:54.925 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-16T20:59:55.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.108+0000 7f64ae30f640 1 Processor -- start 2023-10-16T20:59:55.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.108+0000 7f64ae30f640 1 -- start start 2023-10-16T20:59:55.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.112+0000 7f64ae30f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f64a80cdb20 0x7f64a80ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:55.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.112+0000 7f64ae30f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a806dd70 0x7f64a80cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:55.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.112+0000 7f64ae30f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a806cbe0 con 0x7f64a81537d0 2023-10-16T20:59:55.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.112+0000 7f64ae30f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a806f5f0 con 0x7f64a80cdb20 2023-10-16T20:59:55.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.112+0000 7f64ae30f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a815d8b0 con 0x7f64a806dd70 2023-10-16T20:59:55.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.112+0000 7f64a77fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f64a80cdb20 0x7f64a80ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:55.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.112+0000 7f64a77fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f64a80cdb20 0x7f64a80ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57684/0 (socket says 172.21.15.88:57684) 2023-10-16T20:59:55.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.112+0000 7f64a77fe640 1 -- 172.21.15.88:0/3279443240 learned_addr learned my addr 172.21.15.88:0/3279443240 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:55.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.112+0000 7f64ac8ac640 1 --2- 172.21.15.88:0/3279443240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a806dd70 0x7f64a80cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:55.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.112+0000 7f64a77fe640 1 -- 172.21.15.88:0/3279443240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a806dd70 msgr2=0x7f64a80cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:59:55.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.112+0000 7f64a77fe640 1 --2- 172.21.15.88:0/3279443240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a806dd70 0x7f64a80cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:55.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.112+0000 7f64a77fe640 1 -- 172.21.15.88:0/3279443240 >> v1:172.21.15.191:6789/0 conn(0x7f64a81537d0 legacy=0x7f64a815aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:55.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.112+0000 7f64a77fe640 1 -- 172.21.15.88:0/3279443240 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a806e830 con 0x7f64a80cdb20 2023-10-16T20:59:55.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.112+0000 7f64a77fe640 1 --2- 172.21.15.88:0/3279443240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a80cdb20 0x7f64a80ccdb0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f649c009870 tx=0x7f649c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cc539805d73a90 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:55.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.112+0000 7f64a6ffd640 1 -- 172.21.15.88:0/3279443240 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c0a7070 con 0x7f64a80cdb20 2023-10-16T20:59:55.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64a6ffd640 1 -- 172.21.15.88:0/3279443240 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f649c0ab430 con 0x7f64a80cdb20 2023-10-16T20:59:55.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64a6ffd640 1 -- 172.21.15.88:0/3279443240 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c0ab7e0 con 0x7f64a80cdb20 2023-10-16T20:59:55.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ae30f640 1 -- 172.21.15.88:0/3279443240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a80cdb20 msgr2=0x7f64a80ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:55.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ae30f640 1 --2- 172.21.15.88:0/3279443240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a80cdb20 0x7f64a80ccdb0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f649c009870 tx=0x7f649c01c540 comp rx=0 tx=0).stop 2023-10-16T20:59:55.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ae30f640 1 -- 172.21.15.88:0/3279443240 shutdown_connections 2023-10-16T20:59:55.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ae30f640 1 --2- 172.21.15.88:0/3279443240 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a806dd70 0x7f64a80cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:55.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ae30f640 1 --2- 172.21.15.88:0/3279443240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a80cdb20 0x7f64a80ccdb0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:55.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ae30f640 1 -- 172.21.15.88:0/3279443240 >> 172.21.15.88:0/3279443240 conn(0x7f64a806f7d0 msgr2=0x7f64a806fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:55.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ae30f640 1 -- 172.21.15.88:0/3279443240 shutdown_connections 2023-10-16T20:59:55.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ae30f640 1 -- 172.21.15.88:0/3279443240 wait complete. 2023-10-16T20:59:55.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ae30f640 1 Processor -- start 2023-10-16T20:59:55.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ae30f640 1 -- start start 2023-10-16T20:59:55.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ae30f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a806dd70 0x7f64a81467e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:55.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ae30f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64a80cdb20 0x7f64a81f2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:55.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ae30f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a81537d0 0x7f64a8144150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:55.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ae30f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a814fc60 con 0x7f64a81537d0 2023-10-16T20:59:55.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ae30f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a81f18f0 con 0x7f64a80cdb20 2023-10-16T20:59:55.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ae30f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a81f5900 con 0x7f64a806dd70 2023-10-16T20:59:55.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ac8ac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a81537d0 0x7f64a8144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:55.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ac8ac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a81537d0 0x7f64a8144150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57692/0 (socket says 172.21.15.88:57692) 2023-10-16T20:59:55.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.116+0000 7f64ac8ac640 1 -- 172.21.15.88:0/3583860601 learned_addr learned my addr 172.21.15.88:0/3583860601 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:55.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64a7fff640 1 --2- 172.21.15.88:0/3583860601 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a806dd70 0x7f64a81467e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:55.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64a77fe640 1 --2- 172.21.15.88:0/3583860601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64a80cdb20 0x7f64a81f2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:55.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64ac8ac640 1 -- 172.21.15.88:0/3583860601 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a806dd70 msgr2=0x7f64a81467e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:55.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64ac8ac640 1 --2- 172.21.15.88:0/3583860601 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a806dd70 0x7f64a81467e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:55.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64ac8ac640 1 -- 172.21.15.88:0/3583860601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64a80cdb20 msgr2=0x7f64a81f2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:55.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64ac8ac640 1 --2- 172.21.15.88:0/3583860601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64a80cdb20 0x7f64a81f2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:55.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64ac8ac640 1 -- 172.21.15.88:0/3583860601 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c0094d0 con 0x7f64a81537d0 2023-10-16T20:59:55.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64a77fe640 1 --2- 172.21.15.88:0/3583860601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64a80cdb20 0x7f64a81f2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:59:55.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64ac8ac640 1 --2- 172.21.15.88:0/3583860601 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a81537d0 0x7f64a8144150 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f64a0007e40 tx=0x7f64a000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:55.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64a4ff9640 1 -- 172.21.15.88:0/3583860601 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0014620 con 0x7f64a81537d0 2023-10-16T20:59:55.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64a4ff9640 1 -- 172.21.15.88:0/3583860601 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f64a000ee90 con 0x7f64a81537d0 2023-10-16T20:59:55.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64ae30f640 1 -- 172.21.15.88:0/3583860601 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a81f1b90 con 0x7f64a81537d0 2023-10-16T20:59:55.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64a4ff9640 1 -- 172.21.15.88:0/3583860601 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0014a00 con 0x7f64a81537d0 2023-10-16T20:59:55.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64ae30f640 1 -- 172.21.15.88:0/3583860601 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a81f2150 con 0x7f64a81537d0 2023-10-16T20:59:55.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64ae30f640 1 -- 172.21.15.88:0/3583860601 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a81f2340 con 0x7f64a81537d0 2023-10-16T20:59:55.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64a4ff9640 1 -- 172.21.15.88:0/3583860601 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f64a0014b60 con 0x7f64a81537d0 2023-10-16T20:59:55.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64a4ff9640 1 --2- 172.21.15.88:0/3583860601 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f64940664a0 0x7f6494068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:55.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64a4ff9640 1 -- 172.21.15.88:0/3583860601 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7953+0+0 (secure 0 0 0) 0x7f64a00852b0 con 0x7f64a81537d0 2023-10-16T20:59:55.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.120+0000 7f64a7fff640 1 --2- 172.21.15.88:0/3583860601 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f64940664a0 0x7f6494068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:55.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.124+0000 7f64a7fff640 1 --2- 172.21.15.88:0/3583860601 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f64940664a0 0x7f6494068960 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6498000f70 tx=0x7f6498009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:55.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.124+0000 7f64a4ff9640 1 -- 172.21.15.88:0/3583860601 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f64a0051ca0 con 0x7f64a81537d0 2023-10-16T20:59:55.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.280+0000 7f64ae30f640 1 -- 172.21.15.88:0/3583860601 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f64a8074700 con 0x7f64a81537d0 2023-10-16T20:59:55.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.896+0000 7f64a4ff9640 1 -- 172.21.15.88:0/3583860601 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v69) v1 ==== 181+0+0 (secure 0 0 0) 0x7f64a0053bd0 con 0x7f64a81537d0 2023-10-16T20:59:55.898 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' removed 2023-10-16T20:59:55.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.900+0000 7f64ae30f640 1 -- 172.21.15.88:0/3583860601 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f64940664a0 msgr2=0x7f6494068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:55.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.900+0000 7f64ae30f640 1 --2- 172.21.15.88:0/3583860601 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f64940664a0 0x7f6494068960 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6498000f70 tx=0x7f6498009040 comp rx=0 tx=0).stop 2023-10-16T20:59:55.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.900+0000 7f64ae30f640 1 -- 172.21.15.88:0/3583860601 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a81537d0 msgr2=0x7f64a8144150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:55.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.900+0000 7f64ae30f640 1 --2- 172.21.15.88:0/3583860601 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a81537d0 0x7f64a8144150 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f64a0007e40 tx=0x7f64a000e4b0 comp rx=0 tx=0).stop 2023-10-16T20:59:55.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.900+0000 7f64ae30f640 1 -- 172.21.15.88:0/3583860601 shutdown_connections 2023-10-16T20:59:55.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.900+0000 7f64ae30f640 1 --2- 172.21.15.88:0/3583860601 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7f64940664a0 0x7f6494068960 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:55.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.900+0000 7f64ae30f640 1 --2- 172.21.15.88:0/3583860601 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a81537d0 0x7f64a8144150 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:55.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.900+0000 7f64ae30f640 1 --2- 172.21.15.88:0/3583860601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f64a80cdb20 0x7f64a81f2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:55.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.900+0000 7f64ae30f640 1 --2- 172.21.15.88:0/3583860601 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7f64a806dd70 0x7f64a81467e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:55.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.900+0000 7f64ae30f640 1 -- 172.21.15.88:0/3583860601 >> 172.21.15.88:0/3583860601 conn(0x7f64a806f7d0 msgr2=0x7f64a80cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:55.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.900+0000 7f64ae30f640 1 -- 172.21.15.88:0/3583860601 shutdown_connections 2023-10-16T20:59:55.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:55.900+0000 7f64ae30f640 1 -- 172.21.15.88:0/3583860601 wait complete. 2023-10-16T20:59:55.928 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-16T20:59:56.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.076+0000 7fe77110f640 1 Processor -- start 2023-10-16T20:59:56.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.076+0000 7fe77110f640 1 -- start start 2023-10-16T20:59:56.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.080+0000 7fe77110f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe76c0c4640 0x7fe76c1510d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:56.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.080+0000 7fe77110f640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe76c151610 0x7fe76c1573e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:56.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.080+0000 7fe77110f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76c069e90 con 0x7fe76c0c4260 2023-10-16T20:59:56.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.080+0000 7fe77110f640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe76c0716a0 con 0x7fe76c151610 2023-10-16T20:59:56.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.080+0000 7fe77110f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c06aea0 con 0x7fe76c0c4640 2023-10-16T20:59:56.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.080+0000 7fe76a59c640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe76c0c4640 0x7fe76c1510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:56.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.080+0000 7fe76ad9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe76c0c4260 0x7fe76c0c5520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45726/0 (socket says 172.21.15.88:45726) 2023-10-16T20:59:56.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.080+0000 7fe76ad9d640 1 -- 172.21.15.88:0/3900945505 learned_addr learned my addr 172.21.15.88:0/3900945505 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:59:56.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.080+0000 7fe76b59e640 1 --2- 172.21.15.88:0/3900945505 >> v2:172.21.15.191:3300/0 conn(0x7fe76c151610 0x7fe76c1573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:56.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.080+0000 7fe769d9b640 1 -- 172.21.15.88:0/3900945505 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422934336 0 0) 0x7fe76c069e90 con 0x7fe76c0c4260 2023-10-16T20:59:56.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.080+0000 7fe769d9b640 1 -- 172.21.15.88:0/3900945505 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe758003530 con 0x7fe76c0c4260 2023-10-16T20:59:56.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.080+0000 7fe76b59e640 1 -- 172.21.15.88:0/3900945505 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe76c0c4640 msgr2=0x7fe76c1510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:59:56.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.080+0000 7fe76b59e640 1 --2- 172.21.15.88:0/3900945505 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe76c0c4640 0x7fe76c1510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:56.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.080+0000 7fe76b59e640 1 -- 172.21.15.88:0/3900945505 >> v1:172.21.15.88:6789/0 conn(0x7fe76c0c4260 legacy=0x7fe76c0c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:56.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.080+0000 7fe76b59e640 1 -- 172.21.15.88:0/3900945505 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c1581e0 con 0x7fe76c151610 2023-10-16T20:59:56.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.080+0000 7fe76a59c640 1 --2- 172.21.15.88:0/3900945505 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe76c0c4640 0x7fe76c1510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:59:56.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.084+0000 7fe76b59e640 1 --2- 172.21.15.88:0/3900945505 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe76c151610 0x7fe76c1573e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe7640097c0 tx=0x7fe76401c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ea46a1f7dd52cb0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:56.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.084+0000 7fe769d9b640 1 -- 172.21.15.88:0/3900945505 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7640ab600 con 0x7fe76c151610 2023-10-16T20:59:56.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.084+0000 7fe769d9b640 1 -- 172.21.15.88:0/3900945505 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe7640a7050 con 0x7fe76c151610 2023-10-16T20:59:56.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.084+0000 7fe769d9b640 1 -- 172.21.15.88:0/3900945505 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7640ab960 con 0x7fe76c151610 2023-10-16T20:59:56.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.088+0000 7fe77110f640 1 -- 172.21.15.88:0/3900945505 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe76c151610 msgr2=0x7fe76c1573e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:56.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.088+0000 7fe77110f640 1 --2- 172.21.15.88:0/3900945505 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe76c151610 0x7fe76c1573e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe7640097c0 tx=0x7fe76401c330 comp rx=0 tx=0).stop 2023-10-16T20:59:56.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.088+0000 7fe77110f640 1 -- 172.21.15.88:0/3900945505 shutdown_connections 2023-10-16T20:59:56.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.088+0000 7fe77110f640 1 --2- 172.21.15.88:0/3900945505 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe76c151610 0x7fe76c1573e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:56.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.088+0000 7fe77110f640 1 --2- 172.21.15.88:0/3900945505 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe76c0c4640 0x7fe76c1510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:56.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.088+0000 7fe77110f640 1 -- 172.21.15.88:0/3900945505 >> 172.21.15.88:0/3900945505 conn(0x7fe76c0ca9f0 msgr2=0x7fe76c0cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:56.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.088+0000 7fe77110f640 1 -- 172.21.15.88:0/3900945505 shutdown_connections 2023-10-16T20:59:56.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.088+0000 7fe77110f640 1 -- 172.21.15.88:0/3900945505 wait complete. 2023-10-16T20:59:56.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.088+0000 7fe77110f640 1 Processor -- start 2023-10-16T20:59:56.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.088+0000 7fe77110f640 1 -- start start 2023-10-16T20:59:56.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe77110f640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe76c0c4260 0x7fe76c14fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:56.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe77110f640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe76c0c4640 0x7fe76c147c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:56.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe77110f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe76c151610 0x7fe76c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:56.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe77110f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c14f940 con 0x7fe76c151610 2023-10-16T20:59:56.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe77110f640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c1481d0 con 0x7fe76c0c4260 2023-10-16T20:59:56.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe77110f640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c1473d0 con 0x7fe76c0c4640 2023-10-16T20:59:56.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe76b59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe76c151610 0x7fe76c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:56.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe76b59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe76c151610 0x7fe76c13f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57708/0 (socket says 172.21.15.88:57708) 2023-10-16T20:59:56.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe76b59e640 1 -- 172.21.15.88:0/174174775 learned_addr learned my addr 172.21.15.88:0/174174775 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:56.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe76ad9d640 1 --2- 172.21.15.88:0/174174775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe76c0c4260 0x7fe76c14fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:56.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe76a59c640 1 --2- 172.21.15.88:0/174174775 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe76c0c4640 0x7fe76c147c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:56.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe76a59c640 1 -- 172.21.15.88:0/174174775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe76c0c4260 msgr2=0x7fe76c14fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:56.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe76a59c640 1 --2- 172.21.15.88:0/174174775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe76c0c4260 0x7fe76c14fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:56.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe76a59c640 1 -- 172.21.15.88:0/174174775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe76c151610 msgr2=0x7fe76c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:56.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe76a59c640 1 --2- 172.21.15.88:0/174174775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe76c151610 0x7fe76c13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:56.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe76a59c640 1 -- 172.21.15.88:0/174174775 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe754009a70 con 0x7fe76c0c4640 2023-10-16T20:59:56.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe76ad9d640 1 --2- 172.21.15.88:0/174174775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe76c0c4260 0x7fe76c14fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:59:56.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe76b59e640 1 --2- 172.21.15.88:0/174174775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe76c151610 0x7fe76c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-16T20:59:56.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe76a59c640 1 --2- 172.21.15.88:0/174174775 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe76c0c4640 0x7fe76c147c90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe75400ef90 tx=0x7fe75400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:56.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe7637fe640 1 -- 172.21.15.88:0/174174775 <== mon.2 v2:172.21.15.191:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe754014680 con 0x7fe76c0c4640 2023-10-16T20:59:56.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe7637fe640 1 -- 172.21.15.88:0/174174775 <== mon.2 v2:172.21.15.191:3301/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe75400f040 con 0x7fe76c0c4640 2023-10-16T20:59:56.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe7637fe640 1 -- 172.21.15.88:0/174174775 <== mon.2 v2:172.21.15.191:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe754014a40 con 0x7fe76c0c4640 2023-10-16T20:59:56.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe77110f640 1 -- 172.21.15.88:0/174174775 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe764009420 con 0x7fe76c0c4640 2023-10-16T20:59:56.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.092+0000 7fe77110f640 1 -- 172.21.15.88:0/174174775 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76c14e9b0 con 0x7fe76c0c4640 2023-10-16T20:59:56.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.096+0000 7fe77110f640 1 -- 172.21.15.88:0/174174775 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76c074700 con 0x7fe76c0c4640 2023-10-16T20:59:56.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.096+0000 7fe7637fe640 1 -- 172.21.15.88:0/174174775 <== mon.2 v2:172.21.15.191:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe754014ba0 con 0x7fe76c0c4640 2023-10-16T20:59:56.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.096+0000 7fe7637fe640 1 --2- 172.21.15.88:0/174174775 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe74c0664a0 0x7fe74c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:56.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.096+0000 7fe7637fe640 1 -- 172.21.15.88:0/174174775 <== mon.2 v2:172.21.15.191:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7515+0+0 (secure 0 0 0) 0x7fe754085580 con 0x7fe76c0c4640 2023-10-16T20:59:56.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.096+0000 7fe76ad9d640 1 --2- 172.21.15.88:0/174174775 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe74c0664a0 0x7fe74c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:56.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.096+0000 7fe76ad9d640 1 --2- 172.21.15.88:0/174174775 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe74c0664a0 0x7fe74c068960 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0011f0 tx=0x7fe75c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:56.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.100+0000 7fe7637fe640 1 -- 172.21.15.88:0/174174775 <== mon.2 v2:172.21.15.191:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe7540521c0 con 0x7fe76c0c4640 2023-10-16T20:59:56.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.276+0000 7fe77110f640 1 -- 172.21.15.88:0/174174775 --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe76c0c6e80 con 0x7fe76c0c4640 2023-10-16T20:59:56.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.912+0000 7fe7637fe640 1 -- 172.21.15.88:0/174174775 <== mon.2 v2:172.21.15.191:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v70) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe7540557e0 con 0x7fe76c0c4640 2023-10-16T20:59:56.913 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' removed 2023-10-16T20:59:56.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.916+0000 7fe77110f640 1 -- 172.21.15.88:0/174174775 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe74c0664a0 msgr2=0x7fe74c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:56.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.916+0000 7fe77110f640 1 --2- 172.21.15.88:0/174174775 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe74c0664a0 0x7fe74c068960 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0011f0 tx=0x7fe75c009040 comp rx=0 tx=0).stop 2023-10-16T20:59:56.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.916+0000 7fe77110f640 1 -- 172.21.15.88:0/174174775 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe76c0c4640 msgr2=0x7fe76c147c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:56.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.916+0000 7fe77110f640 1 --2- 172.21.15.88:0/174174775 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe76c0c4640 0x7fe76c147c90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe75400ef90 tx=0x7fe75400c490 comp rx=0 tx=0).stop 2023-10-16T20:59:56.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.916+0000 7fe77110f640 1 -- 172.21.15.88:0/174174775 shutdown_connections 2023-10-16T20:59:56.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.916+0000 7fe77110f640 1 --2- 172.21.15.88:0/174174775 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fe74c0664a0 0x7fe74c068960 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:56.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.916+0000 7fe77110f640 1 --2- 172.21.15.88:0/174174775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe76c151610 0x7fe76c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:56.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.916+0000 7fe77110f640 1 --2- 172.21.15.88:0/174174775 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fe76c0c4640 0x7fe76c147c90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:56.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.916+0000 7fe77110f640 1 --2- 172.21.15.88:0/174174775 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe76c0c4260 0x7fe76c14fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:56.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.916+0000 7fe77110f640 1 -- 172.21.15.88:0/174174775 >> 172.21.15.88:0/174174775 conn(0x7fe76c0ca9f0 msgr2=0x7fe76c0c87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:56.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.916+0000 7fe77110f640 1 -- 172.21.15.88:0/174174775 shutdown_connections 2023-10-16T20:59:56.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:56.916+0000 7fe77110f640 1 -- 172.21.15.88:0/174174775 wait complete. 2023-10-16T20:59:56.945 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-10-16T20:59:57.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.088+0000 7fc667046640 1 Processor -- start 2023-10-16T20:59:57.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.088+0000 7fc667046640 1 -- start start 2023-10-16T20:59:57.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.092+0000 7fc667046640 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc66006a0f0 0x7fc6600c2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:57.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.092+0000 7fc667046640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc660153b20 0x7fc6600c30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:57.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.092+0000 7fc667046640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66015cd00 con 0x7fc66006a4d0 2023-10-16T20:59:57.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.092+0000 7fc667046640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66015d980 con 0x7fc660153b20 2023-10-16T20:59:57.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.092+0000 7fc667046640 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6600c2000 con 0x7fc66006a0f0 2023-10-16T20:59:57.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.092+0000 7fc657fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc66006a4d0 0x7fc660153410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45740/0 (socket says 172.21.15.88:45740) 2023-10-16T20:59:57.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.092+0000 7fc657fff640 1 -- 172.21.15.88:0/724951329 learned_addr learned my addr 172.21.15.88:0/724951329 (peer_addr_for_me v1:172.21.15.88:0/0) 2023-10-16T20:59:57.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.092+0000 7fc6655e3640 1 --2- 172.21.15.88:0/724951329 >> v2:172.21.15.88:3300/0 conn(0x7fc660153b20 0x7fc6600c30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:57.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.092+0000 7fc664de2640 1 --2- 172.21.15.88:0/724951329 >> v2:172.21.15.191:3300/0 conn(0x7fc66006a0f0 0x7fc6600c2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:57.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.092+0000 7fc6577fe640 1 -- 172.21.15.88:0/724951329 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735868596 0 0) 0x7fc66015cd00 con 0x7fc66006a4d0 2023-10-16T20:59:57.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.092+0000 7fc6655e3640 1 -- 172.21.15.88:0/724951329 >> v2:172.21.15.191:3300/0 conn(0x7fc66006a0f0 msgr2=0x7fc6600c2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-16T20:59:57.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.092+0000 7fc6655e3640 1 --2- 172.21.15.88:0/724951329 >> v2:172.21.15.191:3300/0 conn(0x7fc66006a0f0 0x7fc6600c2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:57.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.092+0000 7fc6655e3640 1 -- 172.21.15.88:0/724951329 >> v1:172.21.15.88:6789/0 conn(0x7fc66006a4d0 legacy=0x7fc660153410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:57.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.092+0000 7fc6655e3640 1 -- 172.21.15.88:0/724951329 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66015cfe0 con 0x7fc660153b20 2023-10-16T20:59:57.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.096+0000 7fc6655e3640 1 --2- 172.21.15.88:0/724951329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc660153b20 0x7fc6600c30d0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc660067290 tx=0x7fc65801f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecc678ff814d7d5d server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:57.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.096+0000 7fc6577fe640 1 -- 172.21.15.88:0/724951329 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6580b0440 con 0x7fc660153b20 2023-10-16T20:59:57.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.096+0000 7fc6577fe640 1 -- 172.21.15.88:0/724951329 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fc6580b05a0 con 0x7fc660153b20 2023-10-16T20:59:57.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.096+0000 7fc6577fe640 1 -- 172.21.15.88:0/724951329 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6580b0950 con 0x7fc660153b20 2023-10-16T20:59:57.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.096+0000 7fc667046640 1 -- 172.21.15.88:0/724951329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc660153b20 msgr2=0x7fc6600c30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:57.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.096+0000 7fc667046640 1 --2- 172.21.15.88:0/724951329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc660153b20 0x7fc6600c30d0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc660067290 tx=0x7fc65801f0f0 comp rx=0 tx=0).stop 2023-10-16T20:59:57.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.096+0000 7fc667046640 1 -- 172.21.15.88:0/724951329 shutdown_connections 2023-10-16T20:59:57.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.096+0000 7fc667046640 1 --2- 172.21.15.88:0/724951329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc660153b20 0x7fc6600c30d0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:57.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.096+0000 7fc667046640 1 --2- 172.21.15.88:0/724951329 >> v2:172.21.15.191:3300/0 conn(0x7fc66006a0f0 0x7fc6600c2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:57.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.096+0000 7fc667046640 1 -- 172.21.15.88:0/724951329 >> 172.21.15.88:0/724951329 conn(0x7fc6600ca9f0 msgr2=0x7fc6600cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:57.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.096+0000 7fc667046640 1 -- 172.21.15.88:0/724951329 shutdown_connections 2023-10-16T20:59:57.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.096+0000 7fc667046640 1 -- 172.21.15.88:0/724951329 wait complete. 2023-10-16T20:59:57.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.096+0000 7fc667046640 1 Processor -- start 2023-10-16T20:59:57.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.096+0000 7fc667046640 1 -- start start 2023-10-16T20:59:57.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.096+0000 7fc667046640 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc66006a0f0 0x7fc6600c9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:57.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc667046640 1 --2- >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc66006a4d0 0x7fc6601f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:57.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc667046640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc660153b20 0x7fc6601ee830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:57.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc667046640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc660158fd0 con 0x7fc660153b20 2023-10-16T20:59:57.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc667046640 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600c63a0 con 0x7fc66006a0f0 2023-10-16T20:59:57.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc667046640 1 -- --> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6601f18d0 con 0x7fc66006a4d0 2023-10-16T20:59:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc6655e3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc660153b20 0x7fc6601ee830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc6655e3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc660153b20 0x7fc6601ee830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:57720/0 (socket says 172.21.15.88:57720) 2023-10-16T20:59:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc6655e3640 1 -- 172.21.15.88:0/753486033 learned_addr learned my addr 172.21.15.88:0/753486033 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-10-16T20:59:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc664de2640 1 --2- 172.21.15.88:0/753486033 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc66006a0f0 0x7fc6600c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:57.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc657fff640 1 --2- 172.21.15.88:0/753486033 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc66006a4d0 0x7fc6601f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:57.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc6655e3640 1 -- 172.21.15.88:0/753486033 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc66006a4d0 msgr2=0x7fc6601f2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:57.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc6655e3640 1 --2- 172.21.15.88:0/753486033 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc66006a4d0 0x7fc6601f2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:57.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc6655e3640 1 -- 172.21.15.88:0/753486033 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc66006a0f0 msgr2=0x7fc6600c9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:57.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc6655e3640 1 --2- 172.21.15.88:0/753486033 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc66006a0f0 0x7fc6600c9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:57.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc6655e3640 1 -- 172.21.15.88:0/753486033 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc658009a70 con 0x7fc660153b20 2023-10-16T20:59:57.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc657fff640 1 --2- 172.21.15.88:0/753486033 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc66006a4d0 0x7fc6601f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-16T20:59:57.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc6655e3640 1 --2- 172.21.15.88:0/753486033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc660153b20 0x7fc6601ee830 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fc6580b3040 tx=0x7fc65801be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:57.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc6557fa640 1 -- 172.21.15.88:0/753486033 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6580bc860 con 0x7fc660153b20 2023-10-16T20:59:57.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc6557fa640 1 -- 172.21.15.88:0/753486033 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fc6580b0e90 con 0x7fc660153b20 2023-10-16T20:59:57.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc667046640 1 -- 172.21.15.88:0/753486033 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc66015c740 con 0x7fc660153b20 2023-10-16T20:59:57.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc6557fa640 1 -- 172.21.15.88:0/753486033 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6580bcc20 con 0x7fc660153b20 2023-10-16T20:59:57.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc667046640 1 -- 172.21.15.88:0/753486033 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc660159f00 con 0x7fc660153b20 2023-10-16T20:59:57.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc667046640 1 -- 172.21.15.88:0/753486033 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc660074700 con 0x7fc660153b20 2023-10-16T20:59:57.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.100+0000 7fc6557fa640 1 -- 172.21.15.88:0/753486033 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc6580c6410 con 0x7fc660153b20 2023-10-16T20:59:57.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.104+0000 7fc6557fa640 1 --2- 172.21.15.88:0/753486033 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc64c0664d0 0x7fc64c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-16T20:59:57.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.104+0000 7fc6557fa640 1 -- 172.21.15.88:0/753486033 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6985+0+0 (secure 0 0 0) 0x7fc658122ae0 con 0x7fc660153b20 2023-10-16T20:59:57.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.104+0000 7fc664de2640 1 --2- 172.21.15.88:0/753486033 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc64c0664d0 0x7fc64c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-16T20:59:57.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.104+0000 7fc664de2640 1 --2- 172.21.15.88:0/753486033 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc64c0664d0 0x7fc64c068990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc65c007a60 tx=0x7fc65c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-16T20:59:57.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.108+0000 7fc6557fa640 1 -- 172.21.15.88:0/753486033 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc6580ef9c0 con 0x7fc660153b20 2023-10-16T20:59:57.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.256+0000 7fc667046640 1 -- 172.21.15.88:0/753486033 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc660152bb0 con 0x7fc660153b20 2023-10-16T20:59:57.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.908+0000 7fc6557fa640 1 -- 172.21.15.88:0/753486033 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v71) v1 ==== 178+0+0 (secure 0 0 0) 0x7fc658009450 con 0x7fc660153b20 2023-10-16T20:59:57.911 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data_ec' removed 2023-10-16T20:59:57.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.912+0000 7fc667046640 1 -- 172.21.15.88:0/753486033 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc64c0664d0 msgr2=0x7fc64c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.912+0000 7fc667046640 1 --2- 172.21.15.88:0/753486033 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc64c0664d0 0x7fc64c068990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc65c007a60 tx=0x7fc65c008040 comp rx=0 tx=0).stop 2023-10-16T20:59:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.912+0000 7fc667046640 1 -- 172.21.15.88:0/753486033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc660153b20 msgr2=0x7fc6601ee830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-16T20:59:57.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.912+0000 7fc667046640 1 --2- 172.21.15.88:0/753486033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc660153b20 0x7fc6601ee830 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fc6580b3040 tx=0x7fc65801be90 comp rx=0 tx=0).stop 2023-10-16T20:59:57.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.912+0000 7fc667046640 1 -- 172.21.15.88:0/753486033 shutdown_connections 2023-10-16T20:59:57.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.912+0000 7fc667046640 1 --2- 172.21.15.88:0/753486033 >> [v2:172.21.15.88:6832/1504518179,v1:172.21.15.88:6833/1504518179] conn(0x7fc64c0664d0 0x7fc64c068990 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:57.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.912+0000 7fc667046640 1 --2- 172.21.15.88:0/753486033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc660153b20 0x7fc6601ee830 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:57.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.912+0000 7fc667046640 1 --2- 172.21.15.88:0/753486033 >> [v2:172.21.15.191:3301/0,v1:172.21.15.191:6790/0] conn(0x7fc66006a4d0 0x7fc6601f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:57.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.912+0000 7fc667046640 1 --2- 172.21.15.88:0/753486033 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc66006a0f0 0x7fc6600c9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-16T20:59:57.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.912+0000 7fc667046640 1 -- 172.21.15.88:0/753486033 >> 172.21.15.88:0/753486033 conn(0x7fc6600ca9f0 msgr2=0x7fc660150d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-16T20:59:57.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.912+0000 7fc667046640 1 -- 172.21.15.88:0/753486033 shutdown_connections 2023-10-16T20:59:57.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-10-16T20:59:57.912+0000 7fc667046640 1 -- 172.21.15.88:0/753486033 wait complete. 2023-10-16T20:59:57.949 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-16T20:59:57.950 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-16T20:59:57.950 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T20:59:57.968 INFO:tasks.ceph.mds.a:Stopped 2023-10-16T20:59:57.969 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-16T20:59:57.969 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T20:59:58.040 INFO:tasks.ceph.mds.c:Stopped 2023-10-16T20:59:58.041 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-16T20:59:58.041 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T21:00:00.693 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~0s 2023-10-16T21:00:01.849 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T21:00:01.854 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T21:00:04.119 INFO:tasks.ceph.mds.e:Stopped 2023-10-16T21:00:04.120 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-16T21:00:04.121 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T21:00:04.134 INFO:tasks.ceph.mds.b:Stopped 2023-10-16T21:00:04.135 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-16T21:00:04.135 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T21:00:04.197 INFO:tasks.ceph.mds.d:Stopped 2023-10-16T21:00:04.197 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-16T21:00:04.198 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T21:00:04.257 INFO:tasks.ceph.mds.f:Stopped 2023-10-16T21:00:04.257 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-16T21:00:04.258 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-16T21:00:04.258 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T21:00:04.350 INFO:tasks.ceph.osd.0:Stopped 2023-10-16T21:00:04.351 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-16T21:00:04.351 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T21:00:06.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-16T21:00:06.803 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e has been restored 2023-10-16T21:00:10.453 INFO:tasks.ceph.osd.1:Stopped 2023-10-16T21:00:10.454 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-16T21:00:10.455 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T21:00:10.537 INFO:tasks.ceph.osd.2:Stopped 2023-10-16T21:00:10.538 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-16T21:00:10.538 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T21:00:10.623 INFO:tasks.ceph.osd.3:Stopped 2023-10-16T21:00:10.623 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-16T21:00:10.623 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T21:00:10.711 INFO:tasks.ceph.osd.4:Stopped 2023-10-16T21:00:10.712 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-16T21:00:10.712 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T21:00:12.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-10-16T21:00:12.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-16T21:00:16.814 INFO:tasks.ceph.osd.5:Stopped 2023-10-16T21:00:16.815 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-16T21:00:16.816 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T21:00:16.913 INFO:tasks.ceph.osd.6:Stopped 2023-10-16T21:00:16.913 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-16T21:00:16.914 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T21:00:17.014 INFO:tasks.ceph.osd.7:Stopped 2023-10-16T21:00:17.015 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-16T21:00:17.015 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-16T21:00:17.016 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T21:00:17.073 INFO:tasks.ceph.mgr.y:Stopped 2023-10-16T21:00:17.074 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-16T21:00:17.074 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T21:00:17.125 INFO:tasks.ceph.mgr.x:Stopped 2023-10-16T21:00:17.126 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-16T21:00:17.126 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-16T21:00:17.127 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T21:00:17.146 INFO:tasks.ceph.mon.a:Stopped 2023-10-16T21:00:17.147 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-16T21:00:17.147 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T21:00:17.162 INFO:tasks.ceph.mon.b:Stopped 2023-10-16T21:00:17.162 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-16T21:00:17.162 INFO:teuthology.orchestra.run:waiting for 300 2023-10-16T21:00:17.219 INFO:tasks.ceph.mon.c:Stopped 2023-10-16T21:00:17.219 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-16T21:00:17.220 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v 'Replacing daemon mds' | head -n 1 2023-10-16T21:00:17.263 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-16T21:00:17.263 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-16T21:00:17.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-10-16T21:00:18.495 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-16T21:00:18.495 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-16T21:00:18.698 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-16T21:00:18.699 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-16T21:00:18.861 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi088.front.sepia.ceph.com 2023-10-16T21:00:18.862 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-16T21:00:19.023 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi191.front.sepia.ceph.com 2023-10-16T21:00:19.023 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-16T21:00:19.916 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi191.front.sepia.ceph.com 2023-10-16T21:00:19.917 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-16T21:00:20.074 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi191.front.sepia.ceph.com 2023-10-16T21:00:20.075 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-16T21:00:20.276 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi191.front.sepia.ceph.com 2023-10-16T21:00:20.276 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-16T21:00:20.477 INFO:tasks.ceph:Archiving mon data... 2023-10-16T21:00:20.479 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi/7430114/data/mon.a.tgz 2023-10-16T21:00:20.480 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2023-10-16T21:00:20.491 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.SCF7UbvYJK 2023-10-16T21:00:20.492 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.SCF7UbvYJK 2023-10-16T21:00:20.631 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.SCF7UbvYJK 2023-10-16T21:00:20.711 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.SCF7UbvYJK is 389KB 2023-10-16T21:00:20.777 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.SCF7UbvYJK 2023-10-16T21:00:20.785 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi/7430114/data/mon.b.tgz 2023-10-16T21:00:20.785 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2023-10-16T21:00:20.795 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.zwY02wHUSN 2023-10-16T21:00:20.795 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.zwY02wHUSN 2023-10-16T21:00:20.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.zwY02wHUSN 2023-10-16T21:00:21.015 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.zwY02wHUSN is 395KB 2023-10-16T21:00:21.081 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.zwY02wHUSN 2023-10-16T21:00:21.091 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi/7430114/data/mon.c.tgz 2023-10-16T21:00:21.092 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2023-10-16T21:00:21.146 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.I5rAtT6d8L 2023-10-16T21:00:21.147 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.I5rAtT6d8L 2023-10-16T21:00:21.282 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.I5rAtT6d8L 2023-10-16T21:00:21.359 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.I5rAtT6d8L is 396KB 2023-10-16T21:00:21.421 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.I5rAtT6d8L 2023-10-16T21:00:21.430 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-16T21:00:21.430 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-16T21:00:21.434 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-16T21:00:21.490 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-16T21:00:21.491 DEBUG:teuthology.orchestra.run.smithi088:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-16T21:00:21.501 DEBUG:teuthology.orchestra.run.smithi191:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-16T21:00:21.536 INFO:teuthology.orchestra.run.smithi088.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-16T21:00:21.565 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-16T21:00:21.572 INFO:tasks.ceph:Archiving crash dumps... 2023-10-16T21:00:21.574 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi/7430114/remote/smithi088/crash 2023-10-16T21:00:21.575 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-16T21:00:21.607 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi/7430114/remote/smithi191/crash 2023-10-16T21:00:21.608 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-16T21:00:21.638 INFO:tasks.ceph:Shutting down logrotate 2023-10-16T21:00:21.638 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T21:00:21.657 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-16T21:00:21.722 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-16T21:00:21.744 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-16T21:00:21.790 INFO:tasks.ceph:Compressing logs... 2023-10-16T21:00:21.790 DEBUG:teuthology.orchestra.run.smithi088:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-16T21:00:21.794 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-16T21:00:21.811 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49515.log 2023-10-16T21:00:21.812 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45128.log 2023-10-16T21:00:21.814 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48849.log 2023-10-16T21:00:21.814 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49515.log.gz 2023-10-16T21:00:21.817 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44371.log 2023-10-16T21:00:21.818 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45128.log.gz 2023-10-16T21:00:21.818 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53853.log 2023-10-16T21:00:21.819 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48849.log.gz 2023-10-16T21:00:21.819 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44663.log 2023-10-16T21:00:21.820 INFO:teuthology.orchestra.run.smithi088.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44371.log.gz 2023-10-16T21:00:21.823 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45618.log 2023-10-16T21:00:21.823 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53853.log.gz 2023-10-16T21:00:21.823 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56182.log 2023-10-16T21:00:21.824 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44663.log.gz 2023-10-16T21:00:21.824 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51037.log 2023-10-16T21:00:21.824 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45618.log.gz 2023-10-16T21:00:21.825 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.56182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43609.log 2023-10-16T21:00:21.825 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56182.log.gz 2023-10-16T21:00:21.825 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48147.log 2023-10-16T21:00:21.826 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51037.log.gz 2023-10-16T21:00:21.826 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54362.log 2023-10-16T21:00:21.826 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43609.log.gz 2023-10-16T21:00:21.829 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54411.log 2023-10-16T21:00:21.829 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48147.log.gz 2023-10-16T21:00:21.830 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53722.log 2023-10-16T21:00:21.830 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54362.log.gz 2023-10-16T21:00:21.830 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55496.log 2023-10-16T21:00:21.830 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54411.log.gz 2023-10-16T21:00:21.831 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54279.log 2023-10-16T21:00:21.831 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53722.log.gz 2023-10-16T21:00:21.831 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55496.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.33247.log 2023-10-16T21:00:21.832 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55496.log.gz 2023-10-16T21:00:21.832 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54279.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-16T21:00:21.832 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54279.log.gz 2023-10-16T21:00:21.833 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.tmp-client.admin.33247.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.33247.log.gz 2023-10-16T21:00:21.833 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44528.log 2023-10-16T21:00:21.836 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48317.log 2023-10-16T21:00:21.836 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54655.log 2023-10-16T21:00:21.836 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44528.log.gz 2023-10-16T21:00:21.837 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46968.log 2023-10-16T21:00:21.837 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48317.log.gz 2023-10-16T21:00:21.837 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45644.log 2023-10-16T21:00:21.838 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54655.log.gz 2023-10-16T21:00:21.838 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53197.log 2023-10-16T21:00:21.838 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46968.log.gz 2023-10-16T21:00:21.839 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45644.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45644.log.gz 2023-10-16T21:00:21.839 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48823.log 2023-10-16T21:00:21.843 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53197.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53197.log.gz 2023-10-16T21:00:21.843 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53668.log 2023-10-16T21:00:21.844 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48823.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48823.log.gz 2023-10-16T21:00:21.844 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46011.log 2023-10-16T21:00:21.844 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53668.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53668.log.gz 2023-10-16T21:00:21.845 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53455.log 2023-10-16T21:00:21.845 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46011.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46011.log.gz 2023-10-16T21:00:21.845 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54712.log 2023-10-16T21:00:21.846 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53455.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53455.log.gz 2023-10-16T21:00:21.846 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55626.log 2023-10-16T21:00:21.847 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54712.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54712.log.gz 2023-10-16T21:00:21.847 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45336.log 2023-10-16T21:00:21.848 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55626.log: gzip 87.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.55626.log.gz --verbose 2023-10-16T21:00:21.848 INFO:teuthology.orchestra.run.smithi088.stderr: -- /var/log/ceph/ceph-client.admin.43301.log 2023-10-16T21:00:21.857 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45336.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45336.log.gz 2023-10-16T21:00:21.858 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43847.log 2023-10-16T21:00:21.858 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43301.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43301.log.gz 2023-10-16T21:00:21.859 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51839.log 2023-10-16T21:00:21.859 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43847.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43847.log.gz 2023-10-16T21:00:21.860 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55808.log 2023-10-16T21:00:21.860 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51839.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51839.log.gz 2023-10-16T21:00:21.860 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45127.log 2023-10-16T21:00:21.861 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55808.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55808.log.gz 2023-10-16T21:00:21.861 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54463.log 2023-10-16T21:00:21.862 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45127.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45127.log.gz 2023-10-16T21:00:21.862 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-16T21:00:21.863 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54463.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54463.log.gz 2023-10-16T21:00:21.863 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43233.log 2023-10-16T21:00:21.863 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55782.log 2023-10-16T21:00:21.864 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43233.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43233.log.gz 2023-10-16T21:00:21.864 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46550.log 2023-10-16T21:00:21.865 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55782.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55782.log.gz 2023-10-16T21:00:21.865 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45880.log 2023-10-16T21:00:21.866 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46654.log 2023-10-16T21:00:21.866 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46550.log.gz 2023-10-16T21:00:21.866 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45880.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45880.log.gz 2023-10-16T21:00:21.867 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46890.log 2023-10-16T21:00:21.867 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46654.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46654.log.gz 2023-10-16T21:00:21.867 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38173.log 2023-10-16T21:00:21.873 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46890.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46890.log.gz 2023-10-16T21:00:21.873 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53351.log 2023-10-16T21:00:21.873 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38173.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38173.log.gz 2023-10-16T21:00:21.873 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50657.log 2023-10-16T21:00:21.874 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53351.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53351.log.gz 2023-10-16T21:00:21.874 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46498.log 2023-10-16T21:00:21.874 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50657.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50657.log.gz 2023-10-16T21:00:21.875 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51089.log 2023-10-16T21:00:21.875 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46498.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46498.log.gz 2023-10-16T21:00:21.875 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52008.log 2023-10-16T21:00:21.876 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51089.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51089.log.gz 2023-10-16T21:00:21.876 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49784.log 2023-10-16T21:00:21.876 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52008.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52008.log.gz 2023-10-16T21:00:21.876 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38054.log 2023-10-16T21:00:21.877 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49784.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49784.log.gz 2023-10-16T21:00:21.877 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48929.log 2023-10-16T21:00:21.877 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38054.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38054.log.gz 2023-10-16T21:00:21.878 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49007.log 2023-10-16T21:00:21.878 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48929.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48929.log.gz 2023-10-16T21:00:21.878 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54227.log 2023-10-16T21:00:21.879 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49007.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49007.log.gz 2023-10-16T21:00:21.879 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48451.log 2023-10-16T21:00:21.879 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54227.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54227.log.gz 2023-10-16T21:00:21.880 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53093.log 2023-10-16T21:00:21.880 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48451.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48451.log.gz 2023-10-16T21:00:21.880 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45322.log 2023-10-16T21:00:21.881 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53093.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53093.log.gz 2023-10-16T21:00:21.881 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53145.log 2023-10-16T21:00:21.887 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38287.log 2023-10-16T21:00:21.887 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-16T21:00:21.887 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.38287.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-16T21:00:21.888 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38287.log.gz 2023-10-16T21:00:21.888 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38335.log 2023-10-16T21:00:21.888 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38263.log 2023-10-16T21:00:21.888 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.38335.log: gzip -5 --verbose -- 87.6% /var/log/ceph/ceph-osd.5.log 2023-10-16T21:00:21.889 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38335.log.gz 2023-10-16T21:00:21.889 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.38263.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-16T21:00:21.889 INFO:teuthology.orchestra.run.smithi191.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38263.log.gz 2023-10-16T21:00:21.890 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2023-10-16T21:00:21.890 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-16T21:00:21.890 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38167.log 2023-10-16T21:00:21.895 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-16T21:00:21.895 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.38167.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38167.log.gz 2023-10-16T21:00:21.895 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-16T21:00:21.896 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38239.log 2023-10-16T21:00:21.896 INFO:teuthology.orchestra.run.smithi191.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-16T21:00:21.896 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.38239.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38239.log.gz 2023-10-16T21:00:21.897 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38191.log 2023-10-16T21:00:21.901 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.38191.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38191.log.gz 2023-10-16T21:00:21.902 INFO:teuthology.orchestra.run.smithi191.stderr: 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-16T21:00:21.902 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-10-16T21:00:21.902 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-10-16T21:00:21.902 INFO:teuthology.orchestra.run.smithi191.stderr: 95.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz/var/log/ceph/ceph-mds.d.log: 2023-10-16T21:00:21.904 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45322.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45322.log.gz 2023-10-16T21:00:21.904 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54594.log 2023-10-16T21:00:21.905 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53145.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53145.log.gz 2023-10-16T21:00:21.905 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49755.log 2023-10-16T21:00:21.905 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54594.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54594.log.gz 2023-10-16T21:00:21.905 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46576.log 2023-10-16T21:00:21.906 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49755.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49755.log.gz 2023-10-16T21:00:21.906 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54683.log 2023-10-16T21:00:21.906 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46576.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46576.log.gz 2023-10-16T21:00:21.907 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46732.log 2023-10-16T21:00:21.907 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54683.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54683.log.gz 2023-10-16T21:00:21.907 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49089.log 2023-10-16T21:00:21.908 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46732.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46732.log.gz 2023-10-16T21:00:21.908 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50683.log 2023-10-16T21:00:21.908 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49089.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49089.log.gz 2023-10-16T21:00:21.908 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55114.log 2023-10-16T21:00:21.909 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50683.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50683.log.gz 2023-10-16T21:00:21.909 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53223.log 2023-10-16T21:00:21.909 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55114.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55114.log.gz 2023-10-16T21:00:21.910 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54516.log 2023-10-16T21:00:21.910 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53223.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53223.log.gz 2023-10-16T21:00:21.910 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45132.log 2023-10-16T21:00:21.911 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54516.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54516.log.gz 2023-10-16T21:00:21.911 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54013.log 2023-10-16T21:00:21.911 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45132.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45132.log.gz 2023-10-16T21:00:21.911 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50822.log 2023-10-16T21:00:21.912 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54013.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54013.log.gz 2023-10-16T21:00:21.912 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45802.log 2023-10-16T21:00:21.912 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50822.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50822.log.gz 2023-10-16T21:00:21.913 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45133.log 2023-10-16T21:00:21.913 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45802.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45802.log.gz 2023-10-16T21:00:21.913 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44917.log 2023-10-16T21:00:21.913 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45133.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45133.log.gz 2023-10-16T21:00:21.914 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49060.log 2023-10-16T21:00:21.914 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44917.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44917.log.gz 2023-10-16T21:00:21.914 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54899.log 2023-10-16T21:00:21.915 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49060.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49060.log.gz 2023-10-16T21:00:21.915 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52769.log 2023-10-16T21:00:21.915 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54899.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54899.log.gz 2023-10-16T21:00:21.915 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52830.log 2023-10-16T21:00:21.916 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52769.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52769.log.gz 2023-10-16T21:00:21.916 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52632.log 2023-10-16T21:00:21.916 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52830.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52830.log.gz 2023-10-16T21:00:21.917 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43925.log 2023-10-16T21:00:21.917 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52632.log: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.52632.log.gz 2023-10-16T21:00:21.917 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43717.log 2023-10-16T21:00:21.917 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43925.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-16T21:00:21.918 INFO:teuthology.orchestra.run.smithi088.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.43925.log.gz 2023-10-16T21:00:21.919 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43717.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43717.log.gz 2023-10-16T21:00:21.920 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53302.log 2023-10-16T21:00:21.920 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43689.log 2023-10-16T21:00:21.920 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53302.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53302.log.gz 2023-10-16T21:00:21.921 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51813.log 2023-10-16T21:00:21.921 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43689.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43689.log.gz 2023-10-16T21:00:21.921 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51865.log 2023-10-16T21:00:21.924 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-16T21:00:21.925 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mds.b.log: /var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-16T21:00:21.925 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38311.log 2023-10-16T21:00:21.926 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.38311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38215.log 2023-10-16T21:00:21.926 INFO:teuthology.orchestra.run.smithi191.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38311.log.gz 2023-10-16T21:00:21.926 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.38215.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38215.log.gz 2023-10-16T21:00:21.931 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51813.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51813.log.gz 2023-10-16T21:00:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52168.log 2023-10-16T21:00:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44267.log 2023-10-16T21:00:21.932 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51865.log.gz 2023-10-16T21:00:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48875.log 2023-10-16T21:00:21.933 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52168.log.gz 2023-10-16T21:00:21.933 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44267.log: gzip -5 89.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.44267.log.gz 2023-10-16T21:00:21.933 INFO:teuthology.orchestra.run.smithi088.stderr: -- /var/log/ceph/ceph-client.admin.44241.log 2023-10-16T21:00:21.933 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55418.log 2023-10-16T21:00:21.934 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48875.log.gz 2023-10-16T21:00:21.934 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44241.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44241.log.gz 2023-10-16T21:00:21.934 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55365.log 2023-10-16T21:00:21.935 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55418.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55418.log.gz 2023-10-16T21:00:21.935 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49702.log 2023-10-16T21:00:21.935 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52550.log 2023-10-16T21:00:21.936 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55365.log.gz 2023-10-16T21:00:21.936 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49702.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49702.log.gz 2023-10-16T21:00:21.936 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51953.log 2023-10-16T21:00:21.936 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52550.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52550.log.gz 2023-10-16T21:00:21.937 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44293.log 2023-10-16T21:00:21.937 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51953.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51953.log.gz 2023-10-16T21:00:21.937 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-16T21:00:21.937 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44293.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44293.log.gz 2023-10-16T21:00:21.938 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52795.log 2023-10-16T21:00:21.938 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52604.log 2023-10-16T21:00:21.938 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46706.log 2023-10-16T21:00:21.939 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52795.log.gz 2023-10-16T21:00:21.939 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43501.log 2023-10-16T21:00:21.939 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52604.log.gz 2023-10-16T21:00:21.940 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52116.log 2023-10-16T21:00:21.940 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46706.log.gz 2023-10-16T21:00:21.941 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45672.log 2023-10-16T21:00:21.941 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43501.log.gz 2023-10-16T21:00:21.942 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55969.log 2023-10-16T21:00:21.942 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52116.log.gz 2023-10-16T21:00:21.943 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49570.log 2023-10-16T21:00:21.943 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45672.log.gz 2023-10-16T21:00:21.949 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44871.log 2023-10-16T21:00:21.950 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55969.log.gz 2023-10-16T21:00:21.950 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49570.log: gzip -5 88.7% --verbose -- replaced with /var/log/ceph/ceph-client.admin.49570.log.gz -- 2023-10-16T21:00:21.951 INFO:teuthology.orchestra.run.smithi088.stderr: /var/log/ceph/ceph-client.admin.53171.log 2023-10-16T21:00:21.951 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51761.log 2023-10-16T21:00:21.952 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44871.log.gz 2023-10-16T21:00:21.952 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45334.log 2023-10-16T21:00:21.953 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53171.log.gz 2023-10-16T21:00:21.953 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50796.log 2023-10-16T21:00:21.954 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51761.log.gz 2023-10-16T21:00:21.954 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43579.log 2023-10-16T21:00:21.955 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45334.log.gz 2023-10-16T21:00:21.955 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54568.log 2023-10-16T21:00:21.956 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50796.log.gz 2023-10-16T21:00:21.956 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48121.log 2023-10-16T21:00:21.957 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43579.log.gz 2023-10-16T21:00:21.957 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45111.log 2023-10-16T21:00:21.957 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54568.log.gz 2023-10-16T21:00:21.958 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52987.log 2023-10-16T21:00:21.959 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48121.log.gz 2023-10-16T21:00:21.959 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44793.log 2023-10-16T21:00:21.960 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45111.log.gz 2023-10-16T21:00:21.960 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52987.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52987.log.gz -5 2023-10-16T21:00:21.960 INFO:teuthology.orchestra.run.smithi088.stderr: --verbose -- /var/log/ceph/ceph-client.admin.56208.log 2023-10-16T21:00:21.961 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55704.log 2023-10-16T21:00:21.961 INFO:teuthology.orchestra.run.smithi088.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44793.log.gz 2023-10-16T21:00:21.967 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.56208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50877.log 2023-10-16T21:00:21.968 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56208.log.gz 2023-10-16T21:00:21.968 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46325.log 2023-10-16T21:00:21.968 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55704.log.gz 2023-10-16T21:00:21.969 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44398.log 2023-10-16T21:00:21.969 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50877.log.gz 2023-10-16T21:00:21.970 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45750.log 2023-10-16T21:00:21.970 INFO:teuthology.orchestra.run.smithi088.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.46325.log.gz 2023-10-16T21:00:21.971 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46445.log 88.6% 2023-10-16T21:00:21.971 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44398.log.gz 2023-10-16T21:00:21.972 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43821.log 2023-10-16T21:00:21.972 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45750.log.gz 2023-10-16T21:00:21.973 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.46445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45349.log 2023-10-16T21:00:21.973 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56156.log 2023-10-16T21:00:21.974 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43821.log.gz 2023-10-16T21:00:21.974 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45349.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45349.log.gz 2023-10-16T21:00:21.975 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55036.log 2023-10-16T21:00:21.975 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.56156.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56156.log.gz 2023-10-16T21:00:21.976 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53879.log 2023-10-16T21:00:21.976 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53377.log 2023-10-16T21:00:21.977 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55036.log.gz 2023-10-16T21:00:21.977 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51116.log 2023-10-16T21:00:21.977 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53879.log.gz 2023-10-16T21:00:21.983 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54847.log 2023-10-16T21:00:21.984 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53377.log.gz 2023-10-16T21:00:21.984 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51116.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51116.log.gz 2023-10-16T21:00:21.985 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54847.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54847.log.gz 2023-10-16T21:00:21.985 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47020.log 2023-10-16T21:00:21.986 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55652.log 2023-10-16T21:00:21.986 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.47020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55548.log 2023-10-16T21:00:21.987 INFO:teuthology.orchestra.run.smithi088.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.47020.log.gz 2023-10-16T21:00:21.987 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53276.log 2023-10-16T21:00:21.988 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55652.log.gz 2023-10-16T21:00:21.988 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48043.log 2023-10-16T21:00:21.989 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55548.log.gz 2023-10-16T21:00:21.989 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48715.log 2023-10-16T21:00:21.990 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53276.log.gz 2023-10-16T21:00:21.990 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46602.log 2023-10-16T21:00:21.991 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48043.log.gz 2023-10-16T21:00:21.991 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48715.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48715.log.gz 2023-10-16T21:00:21.992 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46602.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46602.log.gz 2023-10-16T21:00:21.992 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54765.log 2023-10-16T21:00:21.996 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52935.log 2023-10-16T21:00:21.997 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55574.log 2023-10-16T21:00:21.997 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54765.log.gz 2023-10-16T21:00:21.998 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54873.log 2023-10-16T21:00:21.998 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52935.log.gz 2023-10-16T21:00:21.999 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49650.log 2023-10-16T21:00:21.999 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55574.log.gz 2023-10-16T21:00:22.000 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54873.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54873.log.gzgzip 2023-10-16T21:00:22.000 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52524.log 2023-10-16T21:00:22.000 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48767.log 2023-10-16T21:00:22.001 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49650.log.gz 2023-10-16T21:00:22.001 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52524.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52524.log.gz 2023-10-16T21:00:22.001 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48095.log 2023-10-16T21:00:22.001 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48767.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48767.log.gz 2023-10-16T21:00:22.002 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46164.log 2023-10-16T21:00:22.004 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53429.log 2023-10-16T21:00:22.004 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48095.log.gz 2023-10-16T21:00:22.004 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46264.log 2023-10-16T21:00:22.005 INFO:teuthology.orchestra.run.smithi088.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.46164.log.gz 2023-10-16T21:00:22.005 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48345.log 2023-10-16T21:00:22.005 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53429.log.gz 2023-10-16T21:00:22.005 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43421.log 2023-10-16T21:00:22.006 INFO:teuthology.orchestra.run.smithi088.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.46264.log.gz 2023-10-16T21:00:22.006 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45124.log 2023-10-16T21:00:22.006 INFO:teuthology.orchestra.run.smithi088.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48345.log.gz 2023-10-16T21:00:22.007 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43421.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43421.log.gz 2023-10-16T21:00:22.008 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45124.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45124.log.gz 2023-10-16T21:00:22.008 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53013.log 2023-10-16T21:00:22.008 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52142.log 2023-10-16T21:00:22.009 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53013.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-16T21:00:22.009 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53013.log.gz 2023-10-16T21:00:22.010 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44611.log 2023-10-16T21:00:22.011 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52142.log.gz 2023-10-16T21:00:22.012 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46063.log 2023-10-16T21:00:22.014 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44611.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44611.log.gz 2023-10-16T21:00:22.014 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51926.log 2023-10-16T21:00:22.015 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46063.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.46063.log.gz 2023-10-16T21:00:22.017 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51926.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51926.log.gz 2023-10-16T21:00:22.017 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43899.log 2023-10-16T21:00:22.019 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53905.log 2023-10-16T21:00:22.020 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43899.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43899.log.gz 2023-10-16T21:00:22.021 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53905.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53905.log.gz 2023-10-16T21:00:22.022 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45118.log 2023-10-16T21:00:22.024 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54793.log 2023-10-16T21:00:22.025 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45118.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45118.log.gz 2023-10-16T21:00:22.026 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54793.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54793.log.gz 2023-10-16T21:00:22.027 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43337.log 2023-10-16T21:00:22.029 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48663.log 2023-10-16T21:00:22.030 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43337.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43337.log.gz 2023-10-16T21:00:22.031 INFO:teuthology.orchestra.run.smithi088.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-16T21:00:22.032 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48663.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48663.log.gz 2023-10-16T21:00:22.032 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53588.log 2023-10-16T21:00:22.033 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51011.log 2023-10-16T21:00:22.034 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53588.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53588.log.gz 2023-10-16T21:00:22.034 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45932.log 2023-10-16T21:00:22.035 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51011.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51011.log.gz 2023-10-16T21:00:22.036 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50959.log 2023-10-16T21:00:22.037 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52195.log 2023-10-16T21:00:22.037 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45932.log.gz 2023-10-16T21:00:22.038 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50959.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50959.log.gz 2023-10-16T21:00:22.039 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44559.log 2023-10-16T21:00:22.040 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52195.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52195.log.gz 2023-10-16T21:00:22.040 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53640.log 2023-10-16T21:00:22.041 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44559.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44559.log.gz 2023-10-16T21:00:22.042 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44187.log 2023-10-16T21:00:22.043 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45698.log 2023-10-16T21:00:22.044 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53640.log.gz 2023-10-16T21:00:22.045 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44319.log 2023-10-16T21:00:22.046 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45698.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.44187.log.gz 2023-10-16T21:00:22.046 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45698.log.gz 2023-10-16T21:00:22.047 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52090.log 2023-10-16T21:00:22.048 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44319.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44319.log.gz 2023-10-16T21:00:22.048 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54739.log 2023-10-16T21:00:22.050 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55271.log 2023-10-16T21:00:22.050 INFO:teuthology.orchestra.run.smithi088.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.52090.log.gz 2023-10-16T21:00:22.051 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54739.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54739.log.gz 2023-10-16T21:00:22.052 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55271.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55271.log.gz 2023-10-16T21:00:22.053 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48503.log 2023-10-16T21:00:22.054 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46942.log 2023-10-16T21:00:22.055 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48503.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48503.log.gz 2023-10-16T21:00:22.056 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44637.log 2023-10-16T21:00:22.057 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46784.log 2023-10-16T21:00:22.057 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46942.log.gz 2023-10-16T21:00:22.059 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46214.log 2023-10-16T21:00:22.059 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44637.log.gz 2023-10-16T21:00:22.060 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46784.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46784.log.gz 2023-10-16T21:00:22.060 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53536.log 2023-10-16T21:00:22.062 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55470.log 2023-10-16T21:00:22.062 INFO:teuthology.orchestra.run.smithi088.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.46214.log.gz 2023-10-16T21:00:22.063 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53536.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53536.log.gz 2023-10-16T21:00:22.064 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55470.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55470.log.gz 2023-10-16T21:00:22.065 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46114.log 2023-10-16T21:00:22.066 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52961.log 2023-10-16T21:00:22.068 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46114.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-16T21:00:22.068 INFO:teuthology.orchestra.run.smithi088.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.46114.log.gz 2023-10-16T21:00:22.070 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52961.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52961.log.gz 2023-10-16T21:00:22.075 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55193.log 2023-10-16T21:00:22.076 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44908.log 2023-10-16T21:00:22.078 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55193.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55193.log.gz 2023-10-16T21:00:22.078 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49676.log 2023-10-16T21:00:22.079 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44908.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44908.log.gz 2023-10-16T21:00:22.080 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50579.log 2023-10-16T21:00:22.081 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49676.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49676.log.gz 2023-10-16T21:00:22.082 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52340.log 2023-10-16T21:00:22.083 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50579.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50579.log.gz 2023-10-16T21:00:22.084 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46810.log 2023-10-16T21:00:22.085 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52340.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52340.log.gz 2023-10-16T21:00:22.086 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48069.log 2023-10-16T21:00:22.087 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46810.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46810.log.gz 2023-10-16T21:00:22.088 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45828.log 2023-10-16T21:00:22.090 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48069.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48069.log.gz 2023-10-16T21:00:22.091 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45985.log 2023-10-16T21:00:22.091 INFO:teuthology.orchestra.run.smithi088.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.45828.log.gz 2023-10-16T21:00:22.092 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55834.log 2023-10-16T21:00:22.094 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45985.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45985.log.gz 2023-10-16T21:00:22.094 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43637.log 2023-10-16T21:00:22.095 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55834.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55834.log.gz 2023-10-16T21:00:22.096 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45776.log 2023-10-16T21:00:22.097 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43637.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43637.log.gz 2023-10-16T21:00:22.098 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54542.log 2023-10-16T21:00:22.099 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45776.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45776.log.gz 2023-10-16T21:00:22.100 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53774.log 2023-10-16T21:00:22.101 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54542.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54542.log.gz 2023-10-16T21:00:22.102 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54307.log 2023-10-16T21:00:22.103 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53774.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53774.log.gz 2023-10-16T21:00:22.104 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53933.log 2023-10-16T21:00:22.105 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54307.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54307.log.gz 2023-10-16T21:00:22.106 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44061.log 2023-10-16T21:00:22.108 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53933.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53933.log.gz 2023-10-16T21:00:22.108 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53562.log 2023-10-16T21:00:22.109 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44061.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44061.log.gz 2023-10-16T21:00:22.110 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54821.log 2023-10-16T21:00:22.111 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53562.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53562.log.gz 2023-10-16T21:00:22.112 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49238.log 2023-10-16T21:00:22.113 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54821.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54821.log.gz 2023-10-16T21:00:22.114 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55862.log 2023-10-16T21:00:22.115 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49238.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49238.log.gz 2023-10-16T21:00:22.116 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53119.log 2023-10-16T21:00:22.117 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55862.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55862.log.gz 2023-10-16T21:00:22.118 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43873.log 2023-10-16T21:00:22.119 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53119.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53119.log.gz 2023-10-16T21:00:22.120 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52658.log 2023-10-16T21:00:22.121 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43873.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43873.log.gz 2023-10-16T21:00:22.121 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52418.log 2023-10-16T21:00:22.123 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52658.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52658.log.gz 2023-10-16T21:00:22.124 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49346.log 2023-10-16T21:00:22.125 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52418.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52418.log.gz 2023-10-16T21:00:22.126 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53065.log 2023-10-16T21:00:22.127 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49346.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49346.log.gz 2023-10-16T21:00:22.127 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45906.log 2023-10-16T21:00:22.129 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53065.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53065.log.gz 2023-10-16T21:00:22.129 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44424.log 2023-10-16T21:00:22.130 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45906.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45906.log.gz 2023-10-16T21:00:22.131 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43475.log 2023-10-16T21:00:22.133 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44424.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44424.log.gz 2023-10-16T21:00:22.133 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55339.log 2023-10-16T21:00:22.134 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43475.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43475.log.gz 2023-10-16T21:00:22.135 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-16T21:00:22.136 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55339.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55339.log.gz 2023-10-16T21:00:22.137 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52062.log 2023-10-16T21:00:22.141 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.52062.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52062.log.gz 2023-10-16T21:00:22.141 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49622.log 2023-10-16T21:00:22.142 INFO:teuthology.orchestra.run.smithi088.stderr: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-16T21:00:22.143 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45530.log 2023-10-16T21:00:22.144 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49622.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49622.log.gz 2023-10-16T21:00:22.145 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52314.log 2023-10-16T21:00:22.146 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45530.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45530.log.gz 2023-10-16T21:00:22.147 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49729.log 2023-10-16T21:00:22.148 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52314.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52314.log.gz 2023-10-16T21:00:22.148 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48637.log 2023-10-16T21:00:22.150 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49729.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49729.log.gz 2023-10-16T21:00:22.151 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46758.log 2023-10-16T21:00:22.151 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48637.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48637.log.gz 2023-10-16T21:00:22.152 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50553.log 2023-10-16T21:00:22.154 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46758.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46758.log.gz 2023-10-16T21:00:22.154 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53961.log 2023-10-16T21:00:22.155 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50553.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50553.log.gz 2023-10-16T21:00:22.156 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45556.log 2023-10-16T21:00:22.158 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53961.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53961.log.gz 2023-10-16T21:00:22.158 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53987.log 2023-10-16T21:00:22.159 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45556.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45556.log.gz 2023-10-16T21:00:22.160 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48210.log 2023-10-16T21:00:22.162 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53987.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53987.log.gz 2023-10-16T21:00:22.162 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44905.log 2023-10-16T21:00:22.164 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48210.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48210.log.gz 2023-10-16T21:00:22.164 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51147.log 2023-10-16T21:00:22.165 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44905.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44905.log.gz 2023-10-16T21:00:22.166 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49832.log 2023-10-16T21:00:22.167 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51147.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51147.log.gz 2023-10-16T21:00:22.168 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53696.log 2023-10-16T21:00:22.169 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49832.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49832.log.gz 2023-10-16T21:00:22.171 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44918.log 2023-10-16T21:00:22.171 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53696.log.gz 2023-10-16T21:00:22.172 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48477.log 2023-10-16T21:00:22.173 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44918.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44918.log.gz 2023-10-16T21:00:22.174 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55600.log 2023-10-16T21:00:22.175 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48477.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48477.log.gz 2023-10-16T21:00:22.176 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56073.log 2023-10-16T21:00:22.177 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55600.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55600.log.gz 2023-10-16T21:00:22.178 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49541.log 2023-10-16T21:00:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.56073.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56073.log.gz 2023-10-16T21:00:22.180 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52392.log 2023-10-16T21:00:22.181 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49541.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49541.log.gz 2023-10-16T21:00:22.182 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55919.log 2023-10-16T21:00:22.183 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52392.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52392.log.gz 2023-10-16T21:00:22.184 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50770.log 2023-10-16T21:00:22.185 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55919.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55919.log.gz 2023-10-16T21:00:22.186 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43743.log 2023-10-16T21:00:22.187 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50770.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50770.log.gz 2023-10-16T21:00:22.188 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52882.log 2023-10-16T21:00:22.189 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43743.log.gz 2023-10-16T21:00:22.189 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53482.log 2023-10-16T21:00:22.191 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52882.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52882.log.gz 2023-10-16T21:00:22.192 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55995.log 2023-10-16T21:00:22.193 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53482.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53482.log.gz 2023-10-16T21:00:22.194 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48017.log 2023-10-16T21:00:22.195 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55995.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55995.log.gz 2023-10-16T21:00:22.196 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50631.log 2023-10-16T21:00:22.197 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48017.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48017.log.gz 2023-10-16T21:00:22.197 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49596.log 2023-10-16T21:00:22.199 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50631.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50631.log.gz 2023-10-16T21:00:22.200 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49034.log 2023-10-16T21:00:22.201 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49596.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49596.log.gz 2023-10-16T21:00:22.201 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52262.log 2023-10-16T21:00:22.203 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49034.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49034.log.gz 2023-10-16T21:00:22.203 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55890.log 2023-10-16T21:00:22.204 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52262.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52262.log.gz 2023-10-16T21:00:22.205 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52366.log 2023-10-16T21:00:22.207 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55890.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55890.log.gz 2023-10-16T21:00:22.207 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51707.log 2023-10-16T21:00:22.208 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52366.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52366.log.gz 2023-10-16T21:00:22.209 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54437.log 2023-10-16T21:00:22.211 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51707.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51707.log.gz 2023-10-16T21:00:22.212 INFO:teuthology.orchestra.run.smithi088.stderr:gzip/var/log/ceph/ceph-client.admin.54437.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53614.log 2023-10-16T21:00:22.212 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54437.log.gz 2023-10-16T21:00:22.213 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52578.log 2023-10-16T21:00:22.215 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53614.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53614.log.gz 2023-10-16T21:00:22.215 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45582.log 2023-10-16T21:00:22.217 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52578.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52578.log.gz 2023-10-16T21:00:22.217 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50605.log 2023-10-16T21:00:22.218 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45582.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45582.log.gz 2023-10-16T21:00:22.219 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56047.log 2023-10-16T21:00:22.220 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50605.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50605.log.gz 2023-10-16T21:00:22.221 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48585.log 2023-10-16T21:00:22.222 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.56047.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56047.log.gz 2023-10-16T21:00:22.223 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51681.log 2023-10-16T21:00:22.224 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48585.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48585.log.gz 2023-10-16T21:00:22.225 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54173.log 2023-10-16T21:00:22.226 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51681.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51681.log.gz 2023-10-16T21:00:22.227 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52736.log 2023-10-16T21:00:22.228 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54173.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54173.log.gz 2023-10-16T21:00:22.229 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54199.log 2023-10-16T21:00:22.230 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52736.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52736.log.gz 2023-10-16T21:00:22.231 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49398.log 2023-10-16T21:00:22.232 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54199.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54199.log.gz 2023-10-16T21:00:22.233 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52444.log 2023-10-16T21:00:22.234 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49398.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49398.log.gz 2023-10-16T21:00:22.235 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54984.log 2023-10-16T21:00:22.236 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52444.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52444.log.gz 2023-10-16T21:00:22.237 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55391.log 2023-10-16T21:00:22.238 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54984.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54984.log.gz 2023-10-16T21:00:22.238 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45958.log 2023-10-16T21:00:22.240 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55391.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55391.log.gz 2023-10-16T21:00:22.241 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51787.log 2023-10-16T21:00:22.242 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45958.log.gz 2023-10-16T21:00:22.242 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48741.log 2023-10-16T21:00:22.244 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51787.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51787.log.gz 2023-10-16T21:00:22.244 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50735.log 2023-10-16T21:00:22.245 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48741.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48741.log.gz 2023-10-16T21:00:22.246 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48236.log 2023-10-16T21:00:22.248 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50735.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50735.log.gz 2023-10-16T21:00:22.248 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47989.log 2023-10-16T21:00:22.250 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48236.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48236.log.gz 2023-10-16T21:00:22.250 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44767.log 2023-10-16T21:00:22.251 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.47989.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47989.log.gz 2023-10-16T21:00:22.252 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50985.log 2023-10-16T21:00:22.254 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44767.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44767.log.gz 2023-10-16T21:00:22.254 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44450.log 2023-10-16T21:00:22.255 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50985.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50985.log.gz 2023-10-16T21:00:22.256 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52472.log 2023-10-16T21:00:22.258 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44450.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44450.log.gz 2023-10-16T21:00:22.258 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52684.log 2023-10-16T21:00:22.259 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52472.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.52472.log.gz 2023-10-16T21:00:22.260 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55522.log 2023-10-16T21:00:22.261 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52684.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52684.log.gz 2023-10-16T21:00:22.262 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-16T21:00:22.264 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55522.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55522.log.gz 2023-10-16T21:00:22.264 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55062.log 2023-10-16T21:00:22.267 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.55062.log: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-16T21:00:22.268 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54120.log 2023-10-16T21:00:22.268 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55062.log.gz 2023-10-16T21:00:22.268 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-10-16T21:00:22.270 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54120.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54120.log.gz 2023-10-16T21:00:22.271 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48399.log 2023-10-16T21:00:22.274 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.48399.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48399.log.gz 2023-10-16T21:00:22.275 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44913.log 2023-10-16T21:00:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44913.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44913.log.gz 2023-10-16T21:00:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54927.log 2023-10-16T21:00:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54927.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54927.log.gz 2023-10-16T21:00:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-10-16T21:00:22.287 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48611.log 2023-10-16T21:00:22.292 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56021.log 2023-10-16T21:00:22.293 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48611.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48611.log.gz 2023-10-16T21:00:22.299 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.56021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43553.log 2023-10-16T21:00:22.299 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56021.log.gz 2023-10-16T21:00:22.304 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44819.log 2023-10-16T21:00:22.305 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43553.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43553.log.gz 2023-10-16T21:00:22.309 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44819.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44819.log.gz 2023-10-16T21:00:22.310 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48559.log 2023-10-16T21:00:22.316 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52498.log 2023-10-16T21:00:22.317 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48559.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48559.log.gz 2023-10-16T21:00:22.320 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46524.log 2023-10-16T21:00:22.321 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52498.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.52498.log.gz 2023-10-16T21:00:22.329 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46524.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46524.log.gz 2023-10-16T21:00:22.331 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54620.log 2023-10-16T21:00:22.336 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54956.log 2023-10-16T21:00:22.337 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54620.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54620.log.gz 2023-10-16T21:00:22.339 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44925.log 2023-10-16T21:00:22.345 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54956.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54956.log.gz 2023-10-16T21:00:22.345 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54040.log 2023-10-16T21:00:22.346 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44925.log.gz 2023-10-16T21:00:22.350 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44919.log 2023-10-16T21:00:22.350 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54040.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54040.log.gz 2023-10-16T21:00:22.355 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44919.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44919.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54333.log 2023-10-16T21:00:22.355 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-10-16T21:00:22.359 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53508.log 2023-10-16T21:00:22.361 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54333.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54333.log.gz 2023-10-16T21:00:22.364 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45854.log 2023-10-16T21:00:22.365 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53508.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53508.log.gz 2023-10-16T21:00:22.366 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46680.log 2023-10-16T21:00:22.368 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45854.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45854.log.gz 2023-10-16T21:00:22.372 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55943.log 2023-10-16T21:00:22.373 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46680.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46680.log.gz 2023-10-16T21:00:22.374 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52710.log 2023-10-16T21:00:22.376 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55943.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55943.log.gz 2023-10-16T21:00:22.381 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52710.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52710.log.gz 2023-10-16T21:00:22.383 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46864.log 2023-10-16T21:00:22.388 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53827.log 2023-10-16T21:00:22.389 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46864.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46864.log.gz 2023-10-16T21:00:22.392 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53748.log 2023-10-16T21:00:22.393 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53827.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53827.log.gz 2023-10-16T21:00:22.394 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44476.log 2023-10-16T21:00:22.396 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53748.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53748.log.gz 2023-10-16T21:00:22.400 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46994.log 2023-10-16T21:00:22.401 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44476.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44476.log.gz 2023-10-16T21:00:22.407 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53039.log 2023-10-16T21:00:22.407 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46994.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46994.log.gz 2023-10-16T21:00:22.409 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53039.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53039.log.gz 2023-10-16T21:00:22.411 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49424.log 2023-10-16T21:00:22.416 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45125.log 2023-10-16T21:00:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49424.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49424.log.gz 2023-10-16T21:00:22.420 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55678.log 2023-10-16T21:00:22.421 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45125.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45125.log.gz 2023-10-16T21:00:22.423 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43769.log 2023-10-16T21:00:22.425 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55678.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55678.log.gz 2023-10-16T21:00:22.428 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48797.log 2023-10-16T21:00:22.429 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43769.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43769.log.gz 2023-10-16T21:00:22.432 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48797.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48797.log.gz 2023-10-16T21:00:22.439 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45345.log 2023-10-16T21:00:22.444 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52288.log 2023-10-16T21:00:22.445 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45345.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45345.log.gz 2023-10-16T21:00:22.448 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44502.log 2023-10-16T21:00:22.449 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52288.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52288.log.gz 2023-10-16T21:00:22.452 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55219.log 2023-10-16T21:00:22.453 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44502.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44502.log.gz 2023-10-16T21:00:22.459 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45343.log 2023-10-16T21:00:22.461 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55219.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55219.log.gz 2023-10-16T21:00:22.464 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48371.log 2023-10-16T21:00:22.465 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45343.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45343.log.gz 2023-10-16T21:00:22.468 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49489.log 2023-10-16T21:00:22.469 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48371.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48371.log.gz 2023-10-16T21:00:22.472 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44689.log 2023-10-16T21:00:22.475 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49489.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49489.log.gz 2023-10-16T21:00:22.479 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44689.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46916.log 2023-10-16T21:00:22.483 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44689.log.gz 2023-10-16T21:00:22.484 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54253.log 2023-10-16T21:00:22.485 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46916.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46916.log.gz 2023-10-16T21:00:22.488 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55313.log 2023-10-16T21:00:22.495 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54253.log: 89.3%/var/log/ceph/ceph-client.admin.55313.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54253.log.gz 2023-10-16T21:00:22.495 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48262.log 2023-10-16T21:00:22.497 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55313.log.gz 2023-10-16T21:00:22.500 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55245.log 2023-10-16T21:00:22.502 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48262.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48262.log.gz 2023-10-16T21:00:22.504 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50709.log 2023-10-16T21:00:22.505 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55245.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55245.log.gz 2023-10-16T21:00:22.508 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54147.log 2023-10-16T21:00:22.509 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50709.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50709.log.gz 2023-10-16T21:00:22.515 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52034.log 2023-10-16T21:00:22.517 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54147.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54147.log.gz 2023-10-16T21:00:22.520 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45350.log 2023-10-16T21:00:22.521 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52034.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52034.log.gz 2023-10-16T21:00:22.525 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45350.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45350.log.gz 2023-10-16T21:00:22.527 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53800.log 2023-10-16T21:00:22.528 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53403.log 2023-10-16T21:00:22.529 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53800.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53800.log.gz 2023-10-16T21:00:22.532 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48173.log 2023-10-16T21:00:22.533 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53403.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53403.log.gz 2023-10-16T21:00:22.536 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50933.log 2023-10-16T21:00:22.537 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48173.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48173.log.gz 2023-10-16T21:00:22.543 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46628.log 2023-10-16T21:00:22.545 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50933.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50933.log.gz 2023-10-16T21:00:22.548 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43527.log 2023-10-16T21:00:22.549 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46628.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46628.log.gz 2023-10-16T21:00:22.555 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43527.log: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55088.log 2023-10-16T21:00:22.559 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43527.log.gz 2023-10-16T21:00:22.560 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55444.log 2023-10-16T21:00:22.561 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55088.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55088.log.gz 2023-10-16T21:00:22.565 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55444.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55444.log.gz 2023-10-16T21:00:22.567 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51900.log 2023-10-16T21:00:22.572 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45724.log 2023-10-16T21:00:22.573 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51900.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51900.log.gz 2023-10-16T21:00:22.576 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55756.log 2023-10-16T21:00:22.577 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45724.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45724.log.gz 2023-10-16T21:00:22.580 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48529.log 2023-10-16T21:00:22.581 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55756.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55756.log.gz 2023-10-16T21:00:22.583 INFO:teuthology.orchestra.run.smithi191.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-16T21:00:22.584 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54094.log 2023-10-16T21:00:22.585 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48529.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48529.log.gz 2023-10-16T21:00:22.588 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45348.log 2023-10-16T21:00:22.590 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54094.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54094.log.gz 2023-10-16T21:00:22.590 INFO:teuthology.orchestra.run.smithi191.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-16T21:00:22.593 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45348.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45348.log.gz 2023-10-16T21:00:22.595 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49137.log 2023-10-16T21:00:22.596 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48291.log 2023-10-16T21:00:22.597 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49137.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49137.log.gz 2023-10-16T21:00:22.600 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53250.log 2023-10-16T21:00:22.601 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48291.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48291.log.gz 2023-10-16T21:00:22.604 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48425.log 2023-10-16T21:00:22.605 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53250.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53250.log.gz 2023-10-16T21:00:22.608 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47963.log 2023-10-16T21:00:22.609 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48425.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48425.log.gz 2023-10-16T21:00:22.612 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43984.log 2023-10-16T21:00:22.613 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.47963.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47963.log.gz 2023-10-16T21:00:22.617 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43984.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43984.log.gz 2023-10-16T21:00:22.619 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50851.log 2023-10-16T21:00:22.624 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52856.log 2023-10-16T21:00:22.625 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50851.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50851.log.gz 2023-10-16T21:00:22.628 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38273.log 2023-10-16T21:00:22.629 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52856.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52856.log.gz 2023-10-16T21:00:22.632 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43447.log 2023-10-16T21:00:22.633 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.38273.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38273.log.gz 2023-10-16T21:00:22.635 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55730.log 2023-10-16T21:00:22.641 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43447.log: 87.6%gzip/var/log/ceph/ceph-client.admin.55730.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55730.log.gz 2023-10-16T21:00:22.643 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43447.log.gz -5 --verbose 2023-10-16T21:00:22.643 INFO:teuthology.orchestra.run.smithi088.stderr: -- /var/log/ceph/ceph-client.admin.43795.log 2023-10-16T21:00:22.648 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48981.log 2023-10-16T21:00:22.649 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43795.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43795.log.gz 2023-10-16T21:00:22.652 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55167.log 2023-10-16T21:00:22.653 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48981.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48981.log.gz 2023-10-16T21:00:22.656 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54490.log 2023-10-16T21:00:22.657 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55167.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55167.log.gz 2023-10-16T21:00:22.659 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50903.log 2023-10-16T21:00:22.664 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54490.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48955.log 2023-10-16T21:00:22.665 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50903.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50903.log.gz 2023-10-16T21:00:22.667 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54490.log.gz 2023-10-16T21:00:22.669 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48955.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48955.log.gz 2023-10-16T21:00:22.671 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46037.log 2023-10-16T21:00:22.672 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48901.log 2023-10-16T21:00:22.674 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46037.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46037.log.gz 2023-10-16T21:00:22.676 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49372.log 2023-10-16T21:00:22.677 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48901.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48901.log.gz 2023-10-16T21:00:22.680 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44715.log 2023-10-16T21:00:22.681 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49372.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49372.log.gz 2023-10-16T21:00:22.684 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2023-10-16T21:00:22.685 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44715.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44715.log.gz 2023-10-16T21:00:22.687 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44011.log 2023-10-16T21:00:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mds.e.log: /var/log/ceph/ceph-client.admin.44011.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.44011.log.gz 2023-10-16T21:00:22.695 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55010.log 2023-10-16T21:00:22.700 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48689.log 2023-10-16T21:00:22.701 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55010.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55010.log.gz 2023-10-16T21:00:22.704 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46838.log 2023-10-16T21:00:22.705 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48689.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48689.log.gz 2023-10-16T21:00:22.711 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54068.log 2023-10-16T21:00:22.713 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46838.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46838.log.gz 2023-10-16T21:00:22.716 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44845.log 2023-10-16T21:00:22.717 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54068.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54068.log.gz 2023-10-16T21:00:22.720 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44585.log 2023-10-16T21:00:22.721 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44845.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44845.log.gz 2023-10-16T21:00:22.727 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44585.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44585.log.gz 2023-10-16T21:00:22.727 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44741.log 2023-10-16T21:00:22.732 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51735.log 2023-10-16T21:00:22.733 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44741.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44741.log.gz 2023-10-16T21:00:22.736 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49450.log 2023-10-16T21:00:22.737 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51735.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51735.log.gz 2023-10-16T21:00:22.740 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55141.log 2023-10-16T21:00:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49450.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49450.log.gz 2023-10-16T21:00:22.747 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55141.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55141.log.gz 2023-10-16T21:00:22.747 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43663.log 2023-10-16T21:00:22.752 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44345.log 2023-10-16T21:00:22.753 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43663.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43663.log.gz 2023-10-16T21:00:22.757 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51982.log 2023-10-16T21:00:22.758 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44345.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44345.log.gz 2023-10-16T21:00:22.760 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51063.log 2023-10-16T21:00:22.761 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51982.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51982.log.gz 2023-10-16T21:00:22.767 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44924.log 2023-10-16T21:00:22.769 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51063.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51063.log.gz 2023-10-16T21:00:22.772 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52221.log 2023-10-16T21:00:22.773 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44924.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44924.log.gz 2023-10-16T21:00:22.777 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52221.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52221.log.gz 2023-10-16T21:00:22.779 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52909.log 2023-10-16T21:00:22.780 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49275.log 2023-10-16T21:00:22.782 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52909.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52909.log.gz 2023-10-16T21:00:22.785 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49275.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49275.log.gz 2023-10-16T21:00:23.314 INFO:teuthology.orchestra.run.smithi088.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-16T21:00:25.826 INFO:teuthology.orchestra.run.smithi088.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-10-16T21:00:26.519 INFO:teuthology.orchestra.run.smithi088.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2023-10-16T21:00:27.744 INFO:teuthology.orchestra.run.smithi088.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-10-16T21:00:28.724 INFO:teuthology.orchestra.run.smithi191.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-10-16T21:00:29.920 INFO:teuthology.orchestra.run.smithi088.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-16T21:00:32.261 INFO:teuthology.orchestra.run.smithi191.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-16T21:00:34.343 INFO:teuthology.orchestra.run.smithi191.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-16T21:00:35.682 INFO:teuthology.orchestra.run.smithi088.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-16T21:00:35.978 INFO:teuthology.orchestra.run.smithi191.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-16T21:00:37.006 INFO:teuthology.orchestra.run.smithi191.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-10-16T21:00:37.533 INFO:teuthology.orchestra.run.smithi088.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-16T21:00:38.353 INFO:teuthology.orchestra.run.smithi088.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-16T21:00:40.178 INFO:teuthology.orchestra.run.smithi191.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-16T21:00:40.305 INFO:teuthology.orchestra.run.smithi191.stderr: 94.7% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2023-10-16T21:00:40.307 INFO:teuthology.orchestra.run.smithi191.stderr: 2023-10-16T21:00:40.307 INFO:teuthology.orchestra.run.smithi191.stderr:real 0m18.461s 2023-10-16T21:00:40.307 INFO:teuthology.orchestra.run.smithi191.stderr:user 1m29.786s 2023-10-16T21:00:40.308 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m6.548s 2023-10-16T21:01:07.236 INFO:teuthology.orchestra.run.smithi088.stderr: 95.9% -- replaced with /var/log/ceph/ceph-client.0.46445.log.gz 2023-10-16T21:01:07.237 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-10-16T21:01:07.237 INFO:teuthology.orchestra.run.smithi088.stderr:real 0m45.439s 2023-10-16T21:01:07.238 INFO:teuthology.orchestra.run.smithi088.stderr:user 1m44.436s 2023-10-16T21:01:07.238 INFO:teuthology.orchestra.run.smithi088.stderr:sys 0m9.027s 2023-10-16T21:01:07.239 INFO:tasks.ceph:Archiving logs... 2023-10-16T21:01:07.240 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi/7430114/remote/smithi088/log 2023-10-16T21:01:07.241 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-16T21:01:18.035 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi/7430114/remote/smithi191/log 2023-10-16T21:01:18.036 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-16T21:01:26.782 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-16T21:01:26.792 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-16T21:01:26.793 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-16T21:01:26.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-16T21:01:26.883 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-16T21:01:26.893 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-16T21:01:26.893 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T21:01:26.897 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T21:01:26.909 INFO:teuthology.orchestra.run.smithi088.stdout: remote refid st t when poll reach delay offset jitter 2023-10-16T21:01:26.910 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================== 2023-10-16T21:01:26.910 INFO:teuthology.orchestra.run.smithi088.stdout:*hv01.front.sepi 44.4.53.2 2 u 59 64 377 0.084 -4.312 1.041 2023-10-16T21:01:26.910 INFO:teuthology.orchestra.run.smithi088.stdout:+hv02.front.sepi 63.231.80.2 3 u 54 64 377 0.070 -3.922 1.168 2023-10-16T21:01:26.911 INFO:teuthology.orchestra.run.smithi088.stdout:+hv03.front.sepi 74.6.168.72 3 u 55 64 377 0.100 -11.521 1.839 2023-10-16T21:01:26.911 INFO:teuthology.orchestra.run.smithi088.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-16T21:01:26.913 INFO:teuthology.orchestra.run.smithi191.stdout: remote refid st t when poll reach delay offset jitter 2023-10-16T21:01:26.914 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================== 2023-10-16T21:01:26.914 INFO:teuthology.orchestra.run.smithi191.stdout:*hv01.front.sepi 44.4.53.2 2 u 51 64 377 0.073 -1.872 0.818 2023-10-16T21:01:26.914 INFO:teuthology.orchestra.run.smithi191.stdout:+hv02.front.sepi 63.231.80.2 3 u 63 64 377 0.102 -0.467 0.753 2023-10-16T21:01:26.915 INFO:teuthology.orchestra.run.smithi191.stdout:+hv03.front.sepi 74.6.168.72 3 u - 64 377 0.084 -7.652 0.873 2023-10-16T21:01:26.915 INFO:teuthology.orchestra.run.smithi191.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-16T21:01:26.916 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-16T21:01:26.925 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-16T21:01:26.925 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-16T21:01:26.933 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-16T21:01:26.941 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-16T21:01:26.950 INFO:teuthology.task.internal:Duration was 2221.366299 seconds 2023-10-16T21:01:26.951 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-16T21:01:26.960 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-16T21:01:26.961 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-16T21:01:26.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-16T21:01:27.069 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-16T21:01:27.069 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2023-10-16T21:01:27.070 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-16T21:01:27.093 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2023-10-16T21:01:27.094 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-16T21:01:27.117 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-16T21:01:27.118 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-16T21:01:27.142 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-16T21:01:27.194 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-16T21:01:27.195 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-16T21:01:27.226 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-16T21:01:27.300 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-16T21:01:27.310 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-16T21:01:27.311 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-16T21:01:27.332 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-16T21:01:27.354 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-16T21:01:27.364 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-16T21:01:27.378 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-16T21:01:27.393 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2023-10-16T21:01:27.416 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2023-10-16T21:01:27.449 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-16T21:01:27.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T21:01:27.463 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-16T21:01:27.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T21:01:27.502 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-16T21:01:27.514 INFO:teuthology.task.internal:Transferring archived files... 2023-10-16T21:01:27.515 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi/7430114/remote/smithi088 2023-10-16T21:01:27.515 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-16T21:01:27.542 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi/7430114/remote/smithi191 2023-10-16T21:01:27.542 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-16T21:01:27.575 INFO:teuthology.task.internal:Removing archive directory... 2023-10-16T21:01:27.576 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-16T21:01:27.582 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-16T21:01:27.623 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-16T21:01:27.634 INFO:teuthology.task.internal:Not uploading archives. 2023-10-16T21:01:27.635 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-16T21:01:27.645 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-16T21:01:27.645 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-16T21:01:27.649 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-16T21:01:27.654 INFO:teuthology.orchestra.run.smithi088.stdout: 657971 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 16 21:01 /home/ubuntu/cephtest 2023-10-16T21:01:27.675 INFO:teuthology.orchestra.run.smithi191.stdout: 657969 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 16 21:01 /home/ubuntu/cephtest 2023-10-16T21:01:27.677 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-16T21:01:27.688 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-16T21:01:27.710 INFO:teuthology.nuke:Checking targets against current locks 2023-10-16T21:01:27.741 DEBUG:teuthology.nuke:shortname: smithi088 2023-10-16T21:01:27.741 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-16T21:01:27.766 DEBUG:teuthology.nuke:shortname: smithi191 2023-10-16T21:01:27.767 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-16T21:01:27.770 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi/7430114', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 20:13:16.460231', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO3ZZJl3J3J2FRWmXEHKzqpNP/UrrVcr/DKpKW9bbB30t5/mQLXo9MV/4v88nZ3Aidf1/I++8C8HmPysMI48YZU='} 2023-10-16T21:01:27.792 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-10-16_18:49:15-fs-rishabh-2023oct13-testing-default-smithi/7430114', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 20:13:16.458789', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBENSQMED5hBpKUrlawu3SsqoqtNsEGYfWpGz3nN8acKssOxqU8AqvVaZ6pkCVsiVeDFvVF1lai5z+ACojtAJSDc='} 2023-10-16T21:01:27.812 INFO:teuthology.orchestra.console.smithi088:Power off 2023-10-16T21:01:27.813 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-16T21:01:27.837 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off 2023-10-16T21:01:27.838 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T21:01:27.857 INFO:teuthology.orchestra.console.smithi191:Power off 2023-10-16T21:01:27.858 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-16T21:01:27.870 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2023-10-16T21:01:27.883 DEBUG:teuthology.orchestra.console.smithi191:power off output: Chassis Power Control: Down/Off 2023-10-16T21:01:27.883 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T21:01:27.907 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2023-10-16T21:01:31.872 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T21:01:31.908 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T21:01:31.982 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2023-10-16T21:01:32.019 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2023-10-16T21:01:35.986 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T21:01:36.020 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T21:01:36.098 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2023-10-16T21:01:36.133 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off 2023-10-16T21:01:36.234 INFO:teuthology.orchestra.console.smithi191:Power off completed 2023-10-16T21:01:40.102 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-16T21:01:40.212 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off 2023-10-16T21:01:40.313 INFO:teuthology.orchestra.console.smithi088:Power off completed 2023-10-16T21:01:40.420 INFO:teuthology.run:Summary data: description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 2221.366298675537 failure_reason: 'Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi088 with status 128: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=19fd96fe3581520b6d9f50ffdabf29051aee5d4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh''' flavor: default mon.a-kernel-sha1: 47672119149b103ec7cec2cc65c063ccbb001ac7 mon.b-kernel-sha1: 47672119149b103ec7cec2cc65c063ccbb001ac7 owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3f584a96087948f59c1ae976fcc32472 status: fail success: false 2023-10-16T21:01:40.421 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-16T21:01:40.496 INFO:teuthology.run:FAIL